The Unreasonable Effectiveness of Prompt Learning – Aparna Dhinakaran, Arize

Aparna discusses prompt learning and its application to coding agents, particularly Claude and Klein. She draws an analogy between reinforcement learning (RL) and prompt learning, highlighting the benefits of using English feedback to improve system prompts. The discussion covers a process involving coding agents, unit tests, and LLM-as-a-Judge evals to refine prompts, and compares the results of prompt learning with GEPA, emphasizing the importance of well-developed evals. The presentation includes results from benchmarking Claude and Klein on SweBench, showing improvements in resolving GitHub issues through prompt learning.