ESE Spring Seminar – “Learning to Interact With the World: When Generality Meets Precision”

Robotics stands as one of the most impactful and promising endeavors of our times. Learning to interact with the world is fundamental for solving some of our most pressing societal challenges: from taking care of our aging population and aiding with labor-intensive jobs to assisting in climate-related disasters and rescue emergencies. In this talk, I will argue that such a level of autonomy and performance requires robots that can excel across diverse settings while remaining accurate and reliable.

My talk will focus on how we can develop learning algorithms that foster robotic generalization while ensuring the desired task performance. First, I will present a learning approach to pose estimation for novel objects based on visuo-tactile sensing that doesn’t rely on real data and results in accurate pose distributions. Then, I will demonstrate how this approach enables precise robotic pick-and-place using task-aware grasping. The robotic system reasons over the models for grasping, planning, and perception in order to optimize its actions based only on simulated data. In real experiments, we demonstrate that our approach learned purely in simulation, allows robots to successfully manipulate new objects and perform highly accurate placements.