Retrieval After RAG: Hybrid Search, Agents, and Database Design — Simon Hørup Eskildsen of Turbopuffer

Speaker 1 recounts a conversation with Locky, an investor, where he proposed returning the investment if Turbopuffer didn't achieve product-market fit by year's end. Speaker 1 and Justine were only willing to continue if the project showed real promise and were committed to giving it their best shot, including hiring aggressively. Locky was surprised by the offer, noting he had never heard anyone say that before. The podcast introduces Alessio and Swyx, hosts of the Latent Space Podcast, and their guest, Simon Eskildsen of Turbopuffer.

Swyx notes Turbopuffer's rapid growth and welcomes Simon Eskildsen, highlighting his Danish background and connection to the "Aarhus mafia," a group of influential programmers including Bjørn Strøstrup, Rasmus Lerdof, and members of the V8 and Google Maps teams. Despite now being mostly Canadian, Eskildsen acknowledges his Danish upbringing and its influence, particularly the "ruthless pragmatism" and focus on aesthetics. He expresses humility about being considered part of the Aarhus mafia compared to its legendary members.

Speaker 1 recounts his decade at Shopify, focusing on infrastructure scaling challenges, particularly with Elasticsearch. After leaving Shopify, he consulted for Readwise, where he developed a recommendation engine using embeddings. The engine worked well, even suggesting articles about having a child to one of the co-founders, but the infrastructure costs were prohibitively high at $30,000 per month for one feature. This experience highlighted the latent demand for a more cost-effective solution, sparking the initial idea for Turbopuffer.

Speaker 1 describes his "napkin math" calculations that led to the idea of building a database using object storage, NVMe SSDs, and DRAM. The concept involves storing everything on object storage and "puffing" data into NVMe or DRAM as needed. This approach minimizes round trips and leverages the capabilities of modern hardware and cloud infrastructure. He notes that S3 consistency and NVMe SSD availability in the cloud are recent developments that make this architecture feasible.

The discussion shifts to the specific workloads Turbopuffer handles for Notion, emphasizing the importance of low latency. Speaker 1 explains that Notion engineers had considered building a similar system internally but chose to partner with Turbopuffer due to time constraints and the need for a specialized team. This highlights the evolving "buy versus build" equation in the age of AI, where speed and expertise are critical factors.

Speaker 1 recounts how Cursor became an early customer, emphasizing the scrappy beginnings of Turbopuffer. After Cursor reached out, Speaker 1 visited their office and offered complete support. This led to a 95% cost reduction for Cursor, which significantly improved their per-user economics. This success prompted Speaker 1 to bring on Justine as a co-founder to help manage the workload and ensure ongoing support.

The discussion shifts to the evolution of search in AI, from single searches to build context to concurrent agent queries. Speaker 1 notes that LLMs are increasingly used for reasoning, with Turbopuffer serving as a tool call. He observes a growing demand for concurrency, with users driving multiple queries simultaneously. In response, Turbopuffer is reducing query pricing by 5x to accommodate these new workloads.

Speaker 1 discusses Turbopuffer's pricing strategy, which initially started as a "vibe price" based on a rough estimate of costs. The pricing was storage, writes, and query based. The company optimized to ensure a small margin, driven by the high GCP bills. Turbopuffer offers SaaS, single-tenant cluster, and BYOC deployment options.

The discussion turns to team building, particularly the concept of the "P99 engineer." Speaker 1 emphasizes the importance of a talent-dense company and describes the traits they look for in candidates. He explains that the default decision is to reject a candidate, requiring strong advocacy from the team to justify a hire.

Speaker 1 elaborates on the traits of a P99 engineer, including the ability to "bend" software to their will, citing the example of Nathan, who enabled Turbopuffer to search 100 billion vectors with low latency. He also jokingly mentions a love of maps as a common trait, emphasizing the importance of being obsessive and detail-oriented.

Speaker 1 outlines the future of Turbopuffer, describing Act 1 as vector search and Act 2 as full-text search. The company is focused on improving full-text search features, scaling to handle larger datasets, and enhancing the dashboard. He expresses a desire to bring back the simplicity and functionality of PHPMyAdmin.

Speaker 1 discusses potential Act 3 candidates for Turbopuffer, including simpler OLAP queries, traces and logging, and time series. He emphasizes the importance of focus and avoiding overextension. The company is looking for patterns in customer usage to guide its future development.

The conversation shifts to Speaker 1's love of tea, specifically Yabukita Kamairicha from the Green Tea Shop. He describes his air table of 200 teas and his preference for Chinese green tea. He notes that the best time to get this tea is during the spring harvest.