Designing Data-intensive Applications with Martin Kleppmann

The evolution of data-intensive systems centers on the transition from traditional, machine-bound architectures to cloud-native models that leverage primitives like object storage. This shift reduces the burden of manual capacity planning but introduces new complexities in system design and operational reliability. While batch processing tools like MapReduce have largely become obsolete, modern infrastructure now prioritizes stream processing and vector indexes to support AI-driven applications. Formal verification remains a critical, albeit labor-intensive, strategy for ensuring the correctness of high-stakes algorithms, with potential for increased adoption as LLMs simplify the proof-writing process. Furthermore, the push for "local-first" software challenges the dominance of centralized cloud services by prioritizing user data autonomy. This approach necessitates solving difficult engineering problems regarding decentralized access control and consistency, offering a principled alternative to the standard commercial models that rely on vendor lock-in.