Agent-Guided Failure Recovery for Autonomous Robotic Manipulation

Isabella Liu1, An-Chieh Cheng1, Johan Bjorck2, Zhiding Yu2, Hongxu Yin2, Jan Kautz2, Linxi Fan2, Yuke Zhu3, Sifei Liu2

1 University of California, San Diego

2 NVIDIA

3 University of Texas at Austin, Department of Computer Science

Manipulation policies can complete a task yet still depend on human intervention when a failure leaves the scene in an unfamiliar state. RecoverLoop connects recovery during execution with learning between deployments. An agent builds a digital twin from camera observations and explores the task in simulation, collecting successful trajectories and developing recovery policies or executable recovery programs. During real-world rollouts, the agent monitors execution, invokes a recovery behavior when needed, and requests human DAgger assistance when autonomous recovery is unsuccessful. The resulting experience follows two learning paths: successful task rollouts and task demonstrations refine the task policy, while human recovery demonstrations improve the recovery policy. By treating restoration of a workable scene as an explicit objective, RecoverLoop aims to increase autonomous recovery and task success while reducing human intervention over time.

RecoverLoop workflow RecoverLoop builds and explores a digital twin and delegates to the base policy. Simulation experience prepares the base policy and recovery skills. Failures invoke recovery skills, which restore the scene and resume the base policy. Unresolved failures request human assistance, whose recovery data improves recovery skills and returns control to RecoverLoop. Successful task data improves the base policy. Expected trends are increasing recovery success and decreasing human intervention. Build &explore Delegate Simexperience Sim recovery Failure Unresolved Resume Success data Recovery data Digital twinSimulation Base policy Recovery skillsPolicy / program Humanassistance RecoverLoop Recovery success Humanintervention

02 / RECOVERY SKILLS

Recovery skills are discovered as our framework attempts tasks in a simulated digital twin. During simulation rollouts, we collect not only successful trajectories but also analyze failures, prompting our agent to develop skills that recover from them. This recovery knowledge then guides our real-world rollouts.

03 / REAL → SIM

Our framework builds a simulated digital twin directly from camera observations, then attempts to solve the task inside the simulation. This lets the agent practice, fail, and learn to recover safely before acting in the real world.

Loading the digital twin…

GREEN CIRCLE

Drag to orbit · Scroll to zoom

Top camera
0:002× speed

04 / AUTO DATA COLLECTION

10× speed

RecoverLoop orchestrates policy rollouts across four YAM arms, invokes recovery policies when needed, and hands control to a human operator for DAgger corrections when autonomous recovery cannot restore the scene.

05 / SIMULATION BENCHMARKS

PRELIMINARY RESULTS

Success rates on MolmoSpaces and LIBERO-Pro, transcribed from the current paper draft. Evaluation details and final results are being completed.

Reported average success rate

SUCCESS RATE (%)
View all reported numbers