Moonlake: Causal World Models should be Multimodal, Interactive, and Efficient — with Chris Manning and Fan-yun Sun

Chris Manning discusses the challenges of creating effective benchmarks for emerging AI models, including text-based and world models, because current AI applications are moving beyond simple question-answering tasks. Swyx introduces Moonlake and its founders, Fan-yun Sun and Chris Manning, highlighting their innovative approach to world models.

Chris Manning discusses Moonlake's goal of pursuing AI beyond language by incorporating vision and sound. He notes that despite significant investment, computer vision understanding has stalled, with language models dominating vision-language tasks. Moonlake aims to address this by creating a richer connection between symbolic understanding and visual domains, emphasizing structure over scale. He defines world models as needing action-conditioned capabilities, predicting how actions change the world over longer timescales, requiring abstracted semantic models.

Chris Manning addresses the "bitter lesson" of scaling data, questioning whether simply collecting more video data will solve world modeling. He emphasizes the importance of action-conditioned video data, where actions are known and linked to video changes, unlike observational video. Collecting action-conditioned data is challenging, driving interest in simulations. While scale is important, leveraging abstractions can significantly reduce the data needed, offering a more efficient path to progress.

Chris Manning argues that human cognition relies on semantic abstractions, processing only what is focused on and using abstracted descriptions for the surrounding world. This approach is vital for real-time worlds, long-term planning, and consistency. He references a blog post from Physical Intelligence, noting their strategy of storing texts to maintain a long-term memory of world events, rather than relying solely on pixel-level data.

Fan-yun Sun clarifies that Moonlake's abstraction approach aligns with the "bitter lesson," focusing on the right abstraction level for efficiency. Chris Manning discusses philosophical differences with Yann LeCun, who prioritizes visual data over language and symbolic representations. Manning argues that language is a powerful cognitive tool that enabled significant advancements in human intelligence, providing a symbolic knowledge representation and reasoning level.

Chris Manning discusses the concept of joint embeddings for consistent world models, questioning whether autoregressive language models can achieve this. Fan-yun Sun walks through reasoning traces, showcasing how Moonlake's models create interactive game demos with causality and consistency. Swyx notes that Google's Genie and World Labs' Marble lack interactive worlds, highlighting the advantage of Moonlake's reasoning model.

Chris Manning emphasizes the importance of interacting with objects in a world model and observing the correct consequences of actions. Fan-yun Sun explains that Moonlake's models can employ tools like physics engines as cognitive tools to achieve specific goals. She clarifies that writing code for Unity is similar to a tool that their model can use, but the goal is for the model to take a representation-conditioned reasoning approach internally.

Fan-yun Sun reveals that Moonlake's models can configure multiplayer settings and create databases. She discusses data and fidelity constraints, introducing Reverie, a model designed to enhance pixel fidelity. Reverie restyles persistent representations generated by the multimodal reasoning model into photorealistic styles, respecting the interactivity of the world.

Vibhu Sapra raises the question of how to evaluate world models, considering factors like adherence to prompts and simulation logic. Fan-yun Sun emphasizes that evaluation depends on the purpose of the world model, with metrics varying for games and embodied AI. Chris Manning notes the difficulty of creating benchmarks for world models, similar to the challenges faced with language models.

Chris Manning suggests that evaluating world models will involve users "walking with their feet," similar to how people choose language models based on perceived utility. He emphasizes that while great visuals are appealing, gameplay and concept are crucial for successful games. Keeping these axes separate is important when considering what matters in a world model for different uses.

Swyx discusses creating alternative worlds by changing one aspect of reality and observing the consequences. Chris Manning suggests that Moonlake's technology is well-suited for this. Fan-yun Sun explains that code-based models offer greater flexibility compared to models trained on real-world data, making it easier to change fundamental aspects like gravity.

Vibhu Sapra notes that many models are overtrained on one style, making it difficult to extract diversity. Fan-yun Sun emphasizes that there are many types of world simulators, and while pixel coherency is useful for games and marketing, it is less valuable for causal reasoning and embodied AI. She argues that a disproportionately large share of value will come from tasks where high-resolution pixel fidelity is not needed.

Swyx expresses his desire to see Moonlake's technology productized. Fan-yun Sun outlines a vision for the platform in three years, where users can specify end goals, and the world model will generate environments for training and evaluation. This approach aims to provide a world model that allows people to train policies that can act in multimodal environments.

Swyx asks whether Moonlake's approach makes it harder to reward hack. Chris Manning suggests that it does not necessarily solve this problem. Swyx suggests building a better video generation model. Chris Manning emphasizes the importance of compelling gameplay, arguing that while Sora can create visually appealing worlds, it lacks the ability to implement gameplay mechanics and maintain long-term consistency.

Swyx questions the inclusion of spatial 3D audio, noting its complexity. Fan-yun Sun explains that spatial audio benefits from using a game engine as a tool, with the code underlying the simulation providing this capability. Chris Manning emphasizes that Moonlake's integrated audio model exploits the understanding and semantics of the world, unlike GenAI video models that lack integration across modalities.

Vibhu Sapra acknowledges Chris Manning's contributions to NLP and asks about his transition to world models. Chris Manning explains that his interest in visual question answering led him to recognize the limitations of visual understanding. This, combined with the enthusiasm of his students and work with generative AI, motivated his shift towards world models.

Swyx thanks the guests for their contributions to world modeling. Fan-yun Sun states that Moonlake is hiring people with knowledge in code generation, computer vision, and graphics to set up a self-improving system. She emphasizes the need for individuals with backgrounds in both computer vision and graphics.