RatioLogo
Back

The Robot Brain Upgrade: Talking to Machines Like a Friend

Have you ever tried to talk to a robot and ended up feeling like you were arguing with a brick wall? Usually, you have to use very specific, boring words like "Move Forward" or "Pick Up Object." If you use the wrong word, the robot just blinks at you, totally confused.

But what if you could talk to a robot just like you talk to your best friend? A team of scientists just tested a way to give robots a "brain upgrade" using GPT-4—the same super-smart AI that helps people write stories and code games.


The Goal: A Robot with Training Wheels

Variable Autonomy

They wanted to see if robots could handle variable autonomy. Think of this like a bicycle with training wheels that you can flip up or down; sometimes you want to steer everything yourself, and sometimes you want the bike to help you stay balanced.


The Virtual Experiment

The scientists gathered 12 people, aged 20 to 68, and put them inside a Virtual Reality (VR) headset. Inside this digital world, they met three robot helpers:

  • A heavy-lifter named Jupiter
  • A flyer named Pluto
  • A little wheeled robot named Neptune

The Human-Robot Translator

Instead of typing in computer code, the humans just spoke out loud. Their voices were turned into text by a system called Whisper v2. Then, the "brain" (GPT-4) turned those sentences into function calling.

This is like a translator that takes a messy human sentence like "Hey, can you go toss those three plates?" and turns it into a neat to-do list for the robot:

  1. Move
  2. Grab
  3. Drop

Younes

Lakhnati

Younes

"Our findings suggest that users may have preconceived expectations on how to converse with robots and seldom try to explore the actual language and cognitive capabilities of their simulated robot collaborators."


The Results: A Huge "Thumbs Up"

The experiment lasted an average of 27:28 minutes, and the results were a huge "thumbs up" for the future. The humans successfully finished 7 different tricky tasks just by chatting.


The Reality Check

Latency

On a scale of 1 to 7, the robots got a 5.25 for "latency." Latency is essentially the "loading bar" of real life—it’s the annoying pause between when you say "Go!" and when the robot actually moves.

Hallucinations

The robots suffered from "hallucinations." This is like the robot dreaming with its eyes open; it might insist a door is locked even after you’ve already opened it!

Language Rigidity

Sometimes, the robots were even too picky. If a human said "blue trash can" but the robot’s code said "trash bin," the robot might refuse to move.


The Path Forward: Because this was a small test with only 12 people, scientists need to do much more work before these robots leave the VR world. For now, the "brain" lives in the cloud (faraway computers), which makes it slow. To make these robots safe and fast enough for the real world, that big AI brain needs to fit right inside the robot's own head.

Reference: Exploring a GPT-based Large Language Model for Variable Autonomy in a VR-based Human-Robot Teaming Simulation — Younes Lakhnati, Max Pascher, and Jens Gerken (arXiv:2312.07214v3).