Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith

Artificial Analysis founders Micah-Hill Smith and George Cameron discuss the evolution and future of AI benchmarking with Swyx. They detail their journey from a side project to a company providing independent AI model analysis, emphasizing the importance of objective metrics. They cover their business model, which includes enterprise subscriptions and private benchmarking, and the tech stack behind their public benchmarks. The conversation explores the nuances of AI model evaluation, including cost considerations, the challenges of parsing model responses, and the importance of controlling for variance in benchmarks. They also introduce new metrics like the Omniscience Index for measuring hallucination and discuss the trend of decreasing costs for AI intelligence alongside increasing overall spending due to new use cases.