Robot Companion Helps the Memory-Impaired Find Lost Objects
Research interests for Ali Ayub, a post-doctorate student at the Social and Intelligent Robotics Research Lab at the University of Waterloo, include developing autonomous robots that can assist older adults and people with disabilities, helping to create a more independent and fulfilling lifestyle.
As Ayub was developing an assistive robot for moving around in a home, he realized that the robot has to visualize and understand its environment to perform its functions, and that this information is stored in the robot’s memory. “That’s when it occurred to me that this memory could help users track salient items in the house—such as a cell phone—and search the robot’s memory to find these items if they get lost,” he said.
Become a Member: How to Join ASME
This capability would be especially helpful for people with dementia, who often repeatedly forget the location of everyday objects, which diminishes their quality of life and places additional burdens on caregivers. Ayub’s goal then became to build a companion robot with its own episodic memory that can be programmed to help people locate important objects.
The main challenge for Ayub’s team was being able to accurately detect different items in the house using the robot’s camera. With a wide variety of items in a sometimes-cluttered residence, “it can be quite challenging to detect and track items using artificial intelligence [AI] algorithms on the video-based input from the robot’s camera,” he said. “Furthermore, bad lighting, very small objects, and camera movement are additional factors that make this a challenging task.”
The research team began with a Fetch mobile manipulator robot, which has a camera for evaluating its surroundings. Next, using an object-detection algorithm, the team programmed the robot to detect, track, and keep a memory log of specific objects in its camera view, through stored video. Because it can distinguish one object from another, the robot can record the time and date objects enter or leave its view.
Ayub then developed a graphical interface to enable users to select the objects they want to be tracked and, after typing in the object names, search for them on a smartphone app or computer. Once that happens, the robot can indicate when and where it last observed the specific object.
“The biggest surprise for us was how difficult it was to integrate different AI algorithms together with a graphical user interface and a real robot into a complete system,” said Ayub. “Many of the algorithms used in our system have been tested as standalone systems and not integrated into an episodic memory system, which made this an interesting engineering problem to solve.”
More for You: How to Talk Robot
Testing has shown the system to be highly accurate. And although some people with dementia might find the process daunting, Ayub believes caregivers could readily use it.
“We designed a complete system that can be integrated with a robot and be deployed in an unconstrained environment,” added Ayub. “Particularly, we designed a robust approach for detecting and tracking salient objects, even when they are moving in an environment where distractor objects are present.”
Ayub and his team are now working on the design of the robot and the positioning of the robot’s camera. “Currently, our robot is quite bulky, and users have raised issues that this could be a problem in a house,” he said. “Therefore, we are interested in developing more compact robots for the same task.”
The team is also taking steps to improve the graphical user interface to make it more user-friendly and improve the robustness of our system. Ayub will also conduct user studies with people with disabilities to understand how his system performs in a real-world environment.
“Our episodic memory system could potentially be used in other domains, such as smart refrigerators, and autonomous cars,” he said. "The long-term impact of this is really exciting—a user can be involved not just with a companion robot, but a personalized companion robot that can give them more independence in their daily living.”
Mark Crawford is a technology writer in Corrales, N.M.
As Ayub was developing an assistive robot for moving around in a home, he realized that the robot has to visualize and understand its environment to perform its functions, and that this information is stored in the robot’s memory. “That’s when it occurred to me that this memory could help users track salient items in the house—such as a cell phone—and search the robot’s memory to find these items if they get lost,” he said.
Become a Member: How to Join ASME
This capability would be especially helpful for people with dementia, who often repeatedly forget the location of everyday objects, which diminishes their quality of life and places additional burdens on caregivers. Ayub’s goal then became to build a companion robot with its own episodic memory that can be programmed to help people locate important objects.
Integrating AI algorithms
The main challenge for Ayub’s team was being able to accurately detect different items in the house using the robot’s camera. With a wide variety of items in a sometimes-cluttered residence, “it can be quite challenging to detect and track items using artificial intelligence [AI] algorithms on the video-based input from the robot’s camera,” he said. “Furthermore, bad lighting, very small objects, and camera movement are additional factors that make this a challenging task.”
The research team began with a Fetch mobile manipulator robot, which has a camera for evaluating its surroundings. Next, using an object-detection algorithm, the team programmed the robot to detect, track, and keep a memory log of specific objects in its camera view, through stored video. Because it can distinguish one object from another, the robot can record the time and date objects enter or leave its view.
Ayub then developed a graphical interface to enable users to select the objects they want to be tracked and, after typing in the object names, search for them on a smartphone app or computer. Once that happens, the robot can indicate when and where it last observed the specific object.
“The biggest surprise for us was how difficult it was to integrate different AI algorithms together with a graphical user interface and a real robot into a complete system,” said Ayub. “Many of the algorithms used in our system have been tested as standalone systems and not integrated into an episodic memory system, which made this an interesting engineering problem to solve.”
More for You: How to Talk Robot
Testing has shown the system to be highly accurate. And although some people with dementia might find the process daunting, Ayub believes caregivers could readily use it.
“We designed a complete system that can be integrated with a robot and be deployed in an unconstrained environment,” added Ayub. “Particularly, we designed a robust approach for detecting and tracking salient objects, even when they are moving in an environment where distractor objects are present.”
More iterations to come
Ayub and his team are now working on the design of the robot and the positioning of the robot’s camera. “Currently, our robot is quite bulky, and users have raised issues that this could be a problem in a house,” he said. “Therefore, we are interested in developing more compact robots for the same task.”
The team is also taking steps to improve the graphical user interface to make it more user-friendly and improve the robustness of our system. Ayub will also conduct user studies with people with disabilities to understand how his system performs in a real-world environment.
“Our episodic memory system could potentially be used in other domains, such as smart refrigerators, and autonomous cars,” he said. "The long-term impact of this is really exciting—a user can be involved not just with a companion robot, but a personalized companion robot that can give them more independence in their daily living.”
Mark Crawford is a technology writer in Corrales, N.M.
The use of robotics in industrial settings continues to accelerate at a rapid pace. Explore this collection to learn more about automation, cobots, AI-enabled robots, customized robots, and drones.