# Learning Human-to-Robot Handovers from Point Clouds

**Authors:** Sammy Christen, Wei Yang, Claudia Pérez-D'Arpino, Otmar Hilliges, Dieter Fox, Yu-Wei Chao  
**arXiv:** [2303.17592](https://arxiv.org/abs/2303.17592)  
**Date:** 2023-03-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.17592v1/

## Key points

- The paper proposes a framework for learning control policies for vision-based human-to-robot handovers
- It uses a two-stage teacher-student framework that combines motion and grasp planning, reinforcement learning, and self-supervision to train the model with a human demonstrator providing feedback on successful handovers during training
- Point clouds obtained from depth cameras mounted on the robot's end effector and the human's wrist are used to align the robot's end effector with the human's wrist while avoiding collisions during motion planning in the first stage of the framework
- In the second stage of the framework, grasp planning is performed to determine how the robot should grasp an object handed over by the human
- The authors show significant performance gains over baselines on simulation benchmarking, sim-to-sim transfer, and sim-to-real transfer
- Experiments demonstrate improved performance in terms of success rate and time taken for successful handovers in both simulation and real world settings as well as improved performance when fine tuning in simulation is applied to real world scenarios.

## Summary

This paper proposes a framework for learning control policies for vision-based human-to-robot handovers, which is a critical task in human-robot interaction. Recent research has developed realistic simulated environments for human-to-robot handovers and the proposed method leverages this result. It uses a two-stage teacher-student framework that combines motion and grasp planning, reinforcement learning, and self-supervision to train the model with a human demonstrator providing feedback on successful handovers during training. The authors show significant performance gains over baselines on simulation benchmarking, sim-to-sim transfer, and sim-to-real transfer. Point clouds obtained from depth cameras mounted on the robot's end effector and the human's wrist are used to align the robot's end effector with the human's wrist while avoiding collisions during motion planning in the first stage of the framework. In the second stage of the framework, grasp planning is performed to determine how the robot should grasp an object handed over by the human. Experiments demonstrate improved performance in terms of success rate and time taken for successful handovers in both simulation and real world settings as well as improved performance when fine tuning in simulation is applied to real world scenarios. This work presents an important step towards improving human robot interaction by developing effective control policies for vision based handovers that can be applied in real world scenarios.