Stanford Seminar - Connecting Robotics and Foundation Models, Brian Ichter of Google DeepMind

Foundation models offer transformative potential for robotics by providing general reasoning and semantic knowledge, yet they require physical grounding to be actionable. Bridging this gap involves techniques like SACAN and Grounded Decoding, which align model outputs with environmental constraints and robot capabilities. Generating executable code enables robots to perform complex, long-horizon tasks that are difficult to specify through natural language alone. Training on heterogeneous datasets—including simulated and multi-embodiment data—facilitates cross-domain generalization and prevents catastrophic forgetting, as seen in models like RT1 and PaLM-E. While foundation models have significantly improved high-level planning, the primary bottleneck in robotics remains the physical interaction and low-level control. Future advancements depend on integrating these models with robust, high-frequency control systems to achieve reliable, real-world manipulation.