CIS Doctoral Dissertation Defense: “Enabling Tight Perception-Action Loops for Autonomous Robots with Event-based Vision”

CIS Doctoral Dissertation Defense: “Enabling Tight Perception-Action Loops for Autonomous Robots with Event-based Vision”

Anish Bhattacharya
Advisors: Nikolai Matni, Vijay Kumar
Zoom: https://upenn.zoom.us/j/96649115114

Abstract:

This dissertation addresses the challenge of enabling fast, efficient perception-action loops for agile robots using event-based vision. Building on an initial demonstration that end-to-end learned policies—trained entirely in simulation with vision transformers—can transfer zero-shot to real-world flight with strong benefits over previous robotics methods, this work develops a series of methods that leverage event cameras to advance agile aerial autonomy. We introduce a framework for the first event-driven quadrotor obstacle avoidance demonstration that bridges the large simulation-to-real gap of event cameras using a cross-modal post-training scheme. Counter to traditional camera-based approaches, we show improving performance with increasing flight speeds, and demonstrate flight in extreme low-light conditions. To address the scarcity of event-vision datasets in robotics, we next present EvDNeRF, which applies neural radiance fields directly to event data to synthesize novel-view, variable-temporal-resolution event streams from limited recordings. Finally, we explore neural controlled differential equations as a framework for processing the irregularly sampled, near-continuous nature of event streams, exploiting symmetries to achieve improved sample efficiency and generalization.