Extreme Harness Engineering for Token Billionaires: 1M LOC, 1B toks/day, 0% human code, 0% human review — Ryan Lopopolo, OpenAI Frontier & Symphony

Ryan Lopopolo from OpenAI discusses the potential of Codex and harness engineering in building AI products. He emphasizes the momentum around improving models for coding and the ability to translate product ideas into code using the Codex Harness. Ryan works on Frontier product exploration, focusing on novel ways to deploy OpenAI models into enterprise solutions. His team aims to create packaged products that enterprises can use to safely deploy agents at scale with good governance.

Ryan describes his experience at OpenAI, highlighting the company's AI-maximalist environment and internal resources. He recounts building an internal tool with zero lines of code written by himself, resulting in a million-line codebase. This approach was reportedly 10x faster than traditional methods. Starting with early versions of Codex CLI, the team adopted a strategy of breaking down complex tasks into smaller, manageable building blocks for the model to assemble.

The discussion centers on how humans became the bottleneck in the AI-driven development process, despite a small team producing a million lines of code. The focus shifted to providing the model with observability, such as the graph in the article, to improve its performance. Instead of setting up an environment for the coding agent, the agent itself is the entry point, equipped with skills and scripts to boot the stack. The models crave text, so the team found ways to inject text into the system.

Concerns about autonomous merging by coding review agents are addressed, with a discussion on how the agents are instructed to acknowledge and respond to feedback. The prompts allow the agents to push back, and the reviewer agents are biased toward merging. The agents handle a wide range of tasks, including product development, code and tests, CI configuration, documentation, and production dashboard definitions. The team uses Codex to author JSON for Grafana dashboards and respond to pages.

The conversation explores how the team adapted to the model's preferred way of writing software, prioritizing agent legibility over human legibility. Ryan shares his mindset of being removed from the process, similar to a group tech lead for a large organization. He emphasizes the importance of a Command base class for repeatable business logic with built-in tracing and metrics. The discussion touches on how models are improving at proposing abstractions, allowing humans to focus on higher-level strategic issues.

People are reportedly giving the article on Harness engineering to AI models like Pi or Codex, and it's proving wildly effective. Brett Taylor's response is discussed, focusing on the idea that software dependencies are going away and can be vendored. Ryan agrees, stating that the complexity of dependencies that can be internalized is currently low to medium. He also notes that security can be improved by deeply reviewing and changing internalized dependencies.

The team had deployed their app to the first dozen users internally, had some performance issues, and asked them to export a trace for them. The on-call engineer worked with Codex to build a local DevTool Next.js app that visualizes the entire trace. The team is distributing Symphony as a spec, which some are calling ghost libraries. The team is spinning up a new repo, asking Codex to write the spec, and then having another Codex review the implementation.

The discussion transitions to Symphony, an Elixir-based system designed to automate the development workflow. The model chose Elixir because the process supervision and gen servers are amenable to the type of process orchestration that the team is doing. Symphony aims to remove the need for humans to sit in front of their terminals, allowing them to be more latency insensitive and less attached to the code.

The team uses Linear as their issue tracker and Slack for communication. The team fires off Codex to do low-success-y fix-offs to sink that knowledge into the repository. The team discusses the need for collaboration tooling that allows agents to naturally collaborate with humans. The team gives the agent full accessibility over its domain.

The team has been adapting non-textual things to that shape in order to improve model behavior. Agents do not perceive visually in the same way that humans do. If the team wants to actually make it see the layout, it's almost easier to rasterize that image to ASCII architecture and feed it in to the agent.

The coordination layer was a tricky piece to get right. The model takes a shortcut and uses the primitives that it can make use of in the runtime that has native process supervision. The team gives the agent the GH CLI with some text that says CI has to pass. The agents are good at following instructions, so give them instructions and it will improve the reliability of the result.

The team is at the computer with windows popping up all over the place and getting captured and files appearing on the desktop. The team discusses the different models and how to deploy them. The models are not there yet on being able to go from new product idea to prototype.

The discussion transitions to OpenAI's Frontier, the platform by which OpenAI wants to do AI transformation of every enterprise. The goal is to make it easy to deploy, highly observable, safe, controlled, identifiable agents into the workplace. The platform will work with company native IAM stacks and plug into security tooling.

The demo videos are an example of very large scale agent management. The dashboard is for IT, GRC and governments folks, AI innovation office, and the security team. The data is the feedback layer, and it needs to be solved first in order to have the product's feedback loop closed.

The team has skills for how to properly generate deep fried memes and have ReactG culture and Slack. There's a fundamental tension between whether or not to invest deeper into the harness or invest deeper into the training process to get the model to do more of this by default. The team is building an on policy harness, which is already within distribution and modifying it from there.