Gemini Co-Lead on World Models, RL's Next Domains & Continual Learning

Introduction and Multimodal Models

Gemini co-lead Oriol Vinyals discusses advancements in multimodal AI, focusing on world models and their usability.

World Models and Concept Extraction

The core challenge in AI is extracting world knowledge from modalities like video and images without explicit language links, which is a key research area.

Evaluating Physics and Agent Capabilities

Evaluating physics in models is challenging, and while Spark demonstrates impressive consumer agent capabilities, research is ongoing to generalize these systems.

Scaffolding, Agents, and Memory

The future of AI systems may involve models writing their own scaffolds, with progress in agentic reliability driven by model and system improvements, and memory systems evolving beyond simple working memory.

Continual Learning and Organizational Strategy

Google's strategy combines innovation with scalability, leveraging its end-to-end infrastructure to invest in diverse AI research areas, including frontier models and robotics.

Post-Training and Meta Capabilities

While coding and math have seen significant RL progress, the focus is shifting to meta-capabilities like efficient learning and instruction following, which are key to intelligence.

Generalization and Evaluation

Generalization from domain-specific RL, particularly in math and coding, is showing promise in other areas, though evaluating solutions remains more challenging than creating them.

Value of Evaluation and Specialization

Founders should focus on creating robust evaluation metrics and valuable data, as these aspects are crucial for progress, even when building on top of existing models.

Future Capabilities and Innovation

The most fascinating capability is meta-learning, and while true innovation by AI is still developing, advancements in productivity tools and research are expected to continue.