- This event has passed.
Spring 2026 GRASP SFI: Junyao Shi, University of Pennsylvania, “Unlocking Generalist Robots with Human Data and Foundation Models”
March 2 at 2:00 PM - 3:00 PM
This presenter is one of the winners of the 2025 GRASP vote for internal PhD or postdoc SFI Speakers!
This will be a hybrid event with in-person attendance in Levine 307 and virtual attendance on Zoom.
ABSTRACT
Building general-purpose robots remains fundamentally constrained by data scarcity and labor-intensive engineering. Unlike vision and language, robotics lacks large, diverse datasets spanning tasks, environments, and embodiments, limiting both scalability and generalization.
This talk explores how human data and foundation models trained at scale can help overcome these bottlenecks. I will discuss recent progress in leveraging in-the-wild human videos to transfer manipulation skills and provide rich signals for robot learning. I will also highlight how vision and language models can automate traditionally hand-engineered components of robotics pipelines, including modular system construction, data collection, reward design, and simulation construction.
Together, these directions point toward a new paradigm for robot learning: one that replaces task-specific engineering with scalable learning from non-robot knowledge and data, bringing us closer to adaptable, generalist robots.
Junyao Shi
University of Pennsylvania
Hi! My name is Junyao Shi (施钧耀). I’m a final-year PhD student in Computer and Information Science (CIS) at University of Pennsylvania GRASP Laboratory, advised by Dinesh Jayaraman. I am also currently a Research Intern at Skild AI where I work on training robust long-horizon robot manipulation policies. My research at Penn focuses on robot learning, with a particular emphasis on leveraging human data and foundation models for building general-purpose robots. Previously, I received my B.S. in Computer Science from Columbia University, where I worked with Peter Allen on brain-signal guided robot learning and Tony Dear on reinforcement learning for snake robot locomotion.