MEAM Master’s Thesis Defense: “Gaussian Process-Based Active Exploration Strategies in Vision and Touch”

Robots struggle to understand object properties like shape, material, and semantics due to limited prior knowledge, hindering manipulation in unstructured environments. In contrast, humans learn these properties through interactive multi-sensor exploration. This work proposes fusing visual and tactile observations into a unified Gaussian Process Distance Field (GPDF) representation for active perception of object properties. While primarily focusing on geometry, this approach also demonstrates potential for modeling surface properties beyond geometry.

The GPDF encodes signed distance, gradients, and uncertainty estimates. Starting with an initial visual shape estimate, the framework iteratively refines the geometry by integrating dense vision measurements using differentiable rendering and tactile measurements at uncertain regions. By quantifying multi-sensor uncertainties, it plans exploratory motions to maximize information gain for recovering precise 3D structures. To improve scalability, it investigates approximation methods like inducing point parameterization for Gaussian Processes. This probabilistic multi-modal fusion enables active exploration and mapping of complex object geometries.