IRIM Fall 2024 Seminar | Unifying Semantic and Physical Intelligence for Generalist Humanoid Robots

Abstract: Humanoid robots offer two unparalleled advantages in general-purpose embodied intelligence. First, humanoids are built as generalist robots that can potentially do all the tasks humans can do in complex environments. Second, the embodiment alignment between humans and humanoids allows for the seamless integration of human cognitive skills with versatile humanoid capabilities. To build generalist humanoids, there are three critical aspects of intelligence: (1) Semantic intelligence (how the robot understands the world and reasons); (2) Physical/Motion intelligence (locomotion and manipulation skills); and (3) Mechanical/Hardware intelligence (how the robot actuates and senses). In this talk, I will present some recent works (H2O, OmniH2O, WoCoCo, ABS) that aim to unify semantic and physical intelligence for humanoid robots. In particular, H2O and OmniH2O provide a universal and dexterous interface that enables diverse human control (e.g., VR, RGB) and autonomy (e.g., using imitation learning or VLMs) methods for humanoids, WoCoCo provides an efficient framework for loco-manipulation skill learning without motion priors, and ABS provides safety guarantees for agile vision-based locomotion control. Finally, I will briefly discuss how to combine learning-based control approaches and traditional model-based control approaches to get the best of two worlds.

Bio: Guanya Shi is an Assistant Professor at the Robotics Institute at Carnegie Mellon University (CMU). He completed his Ph.D. in Control and Dynamical Systems in 2022 from Caltech. Before joining CMU, he was a postdoctoral scholar at the University of Washington. He is broadly interested in the intersection of machine learning and control, spanning the entire spectrum from theory and foundation, algorithm design, to real-world agile robotics. Guanya was the recipient of several awards, including the RSS Outstanding Student Paper Award Finalist, the Ben P.C. Chou Doctoral Prize from Caltech, and the Rising Star in Data Science. Guanya is an Associate Editor of IEEE Robotics and Automation Letters.