Agent craft · The Monday lead
Your coding agent’s model is no longer the only thing that needs an eval
LoopArena separates the controller from the worker and finds that long-running software work still breaks at the level of direction, verification, and stopping.
AI coding systems are becoming teams in miniature. One model writes the code; another watches progress, chooses the next step, asks for checks, and decides when the work is finished. LoopArena evaluates that second job, the controller, without allowing the worker’s raw coding ability to hide bad direction.
The result is a warning for anyone judging an agent by its final demo. Across full tasks, the best reported strict success rate was 24.69%. A run can fail because the controller trusted stale progress, skipped verification, spent its budget badly, or stopped early, even when the worker was capable of carrying out the missing step.
The useful part is not another leaderboard. LoopArena offers three evaluation depths, including a task-slice method that cut estimated inference cost by an average of 64.4% while producing nearly the same model ordering as the full-task evaluation under its main criterion. That makes controller evaluation more plausible as a regular engineering practice, not only a research exercise.
What it changes Log control decisions separately from tool execution. Grade what the agent chose to do, what it chose to verify, and why it stopped. If you only score the repository at the end, you cannot tell whether your orchestration improved or the coding model merely rescued it.