Andrew Trask, Senior Research Scientist at Google DeepMind and Executive Director of OpenMind, discusses the limits of embedded evaluators, double-blind frontier model testing via confidential compute, and why ensemble routing could decentralize the AI landscape. Check out our sponsor Lovable: https://lovable.dev. Turn ideas into software people love.

MTS is a live news and interview show covering technology, business, politics, and culture as it happens.

Follow MTS:

X | ⁠https://twitter.com/MTSlive⁠

YouTube | ⁠https://www.youtube.com/@mtsituation⁠

Spotify | ⁠https://open.spotify.com/show/4HUNNmV1pjV7CW6Lphm1rM⁠

Apple | ⁠https://podcasts.apple.com/us/podcast/mts/id1891088763⁠

Substack | ⁠https://mtslive.substack.com/⁠

Interested in sponsoring the show? ⁠⁠⁠⁠⁠⁠sponsors@mts.now⁠⁠

--

Brought to you by:

Mongo DB - Anyone can prototype an agent. Learn how to get them into production on September 30th in New York - register at https://www.mongodb.com/events/mongodb-local/nyc

Lovable - Turn ideas into software people love with Lovable. Start today at https://lovable.dev/

Larridin - Finally prove what your AI investment is worth. Head to https://larridin.com/ today to start measuring your AI’s bottom line.

Neon (by Databricks) - Get up to $100K in credits for your startup at ⁠https://neon.com/mts⁠

ElevenLabs - AI that communicates at human level across every channel and modality. https://elevenlabs.io/mts

Arena - Measuring AI performance in the real world. ⁠https://arena.ai/⁠

Kong - The AI connectivity platform. Connect APIs, LLMs, agents, and systems with serious security and governance. https://konghq.com

Today’s stream is sponsored by VCX, the public ticker for private tech. Head over to http://getvcx.com to learn more

Blitzy - Autonomous software development for enterprise codebases. Ship 5x faster. https://blitzy.com

AdQuick - Make your brand a billboard. Out-of-home advertising as easy to scale as digital. https://adquick.com

Chargebee - Billing and monetization infrastructure for the AI era. https://www.chargebee.com/

--

Timestamps:

(00:00) Introduction

(00:40) Scaling Embedded Evaluators

(03:54) Addressing Evaluator Bias

(08:22) Day-to-Day Evaluator Workflows

(12:08) Preventing Benchmark Gaming

(17:45) Solving the Evaluator Talent Bottleneck

(22:14) Ensemble Routing vs. Scaling Laws

--

Note: This podcast is not investment, legal, or tax advice, and is intended for informational and entertainment purposes only. Hosts and guests may hold positions in the companies and securities discussed; do your own research before acting on anything you hear.

Millions of Small AIs Could Beat One Giant Model | Andrew Trask

MTS

Millions of Small AIs Could Beat One Giant Model | Andrew Trask