Nvidia research: it's the harness, not the AI model, driving agent performance
Nvidia researchers found that wrapping an AI model in a custom support system pushed Claude Opus 5 to a perfect score on a tough game-based benchmark, up from 30% without it. The findings suggest the surrounding infrastructure matters more than the model itself.
Nvidia published research on Friday indicating that for long-horizon AI tasks, the system built around a model — known as the "harness" — matters more than the underlying model itself. By using a custom harness designed to better manage memory and by adding a "supervisor" component, researchers got Claude Opus 5 to score a perfect 100% on ARC-AGI-3, an interactive reasoning benchmark built from simple 2D games with no instructions, where success requires the model to figure out how to play and win.
Without the custom harness, the same model scored only 30% — still the best result among all models tested, but far below the perfect score achieved with the added scaffolding.
Adel El Hallack, vice president of product in Nvidia's AI unit, explained that an agent isn't just an interface to a model but includes the scaffolding, tools, runtime, and skills surrounding it. The most notable addition was a supervising agent that monitors the main agent's work and nudges it back on track when it gets stuck, explores dead ends, or repeats previously tried paths.
OpenAI had previously been troubled by its own models scoring below 10% on the same benchmark. Last month, OpenAI conducted its own research and found that adjusting two harness settings tripled its models' scores, though none reached the perfect score Nvidia achieved.
Nvidia's harness, called Agentic Variation Operators, is not a standalone commercial product but part of the open tooling Nvidia releases under its Nemo brand. Similar conclusions were reached in July by Databricks, whose CEO Ali Ghodsi noted that choosing the wrong harness for the same model can double AI usage costs.
El Hallack argued that open, adjustable harnesses give users far more control over accuracy and security than they may realize, compared to relying solely on closed systems.
