Giving Robots the Power of Touch
Imagine trying to pick up a slippery glass of water while wearing thick oven mitts and a blindfold. Even for a human, it’s nearly impossible!
Robots face this problem every single day. While robots have great "eyes" (cameras), they are often very clumsy because they don't "feel" the world like we do.
The Super-Brain Solution
The Problem: Robots Are Clumsy
Scientists are now building a "super-brain" for robots that combines vision and touch. In a recent test, robots trying to put a peg into a hole only succeeded 50% of the time using just their eyes.
The Human Blueprint
To solve this, researchers looked at how our own brains work. They found that in humans, the perfect teamwork between eyes and hands usually finishes developing between ages 8 and 10.
Navarro-Guerrero
et al.
"The integration of multiple sensory modalities at the level of a single neuron has been studied in the cat superior colliculus (Stein et al, 2014); while we do so, the neurons in our brain organize among themselves, a process which has been termed input-driven self-organization."
How They Gave Robots "Feeling"
The Key: Visuo-Haptic Integration
This is like a conversation between your eyes and your fingertips. When scientists gave robots this ability, their peg-in-hole success rate jumped to 75%. This matters because if we want robots to help us fold laundry or perform surgery, they need to know if an object is heavy, slippery, or about to break.
The Method: Midst-Mapping
This is like having two students—one who is an expert at looking and one who is an expert at touching—work together on a project and share notes halfway through, instead of only talking at the very end.
The High-Tech Senses
To give robots this power, scientists use incredible sensors:
- The BioTac sensor can feel a tiny force of just 1 mN—that’s like feeling a single grain of sand land on your skin.
- Others use special light to "see" depth at 750–1400 nm, which are wavelengths of light that human eyes can't even spot.
The Incredible Results
Object Recognition
A robot using GFK learning achieved 94.7% accuracy in recognizing objects.
Zero-Shot Learning
Even when shown a totally new object it had never seen or touched before, the robot guessed correctly 72% of the time.
The Reality Gap & Next Steps
There is still work to do. Robots are currently struggling with the "reality gap," which is like being a pro at a soccer video game but then tripping over the ball in real life.
Furthermore, while we have billions of pictures to teach robots, one haptic dataset called PHAC-2 only has 60 objects and 4 ways to touch them. Scientists are now working on ways to let robots learn safely without breaking their expensive fingers.
Source: Navarro-Guerrero, N., Toprak, S., Josifovski, J., & Jamone, L. (2022). Visuo-Haptic Object Perception for Robots: An Overview. Springer Nature / arXiv:2203.11544v3 [cs.RO].