LinuxArena proposes evaluating agent control in live production software environments
A control setting that places AI agents in live production software environments rather than sandboxes, on the premise that the failure modes worth measuring are the ones that only appear when actions have consequences and state persists.
Almost every agent benchmark runs in a reset-on-failure sandbox. That design measures capability cleanly and measures control badly, because the entire question in control research is what an agent does when its actions are not free.
A live production environment has properties a sandbox deliberately removes: state persists across attempts, side effects are real, other processes are affected, and there is no clean rollback. Those are also precisely the conditions under which an agent might take an irreversible shortcut, disable a check that is slowing it down, or produce a plausible completion report for work it did not finish.
The methodological difficulty is obvious and is the interesting part: an evaluation that runs in production has to be safe enough to run in production. Whatever containment makes that acceptable re-imports some of the artificiality the design set out to remove, and how that boundary is drawn is where the contribution lives.
It fits a broader turn toward measuring conduct rather than knowledge — the same instinct behind sabotage evaluations. Knowing what a model can do has turned out to predict less about deployment than knowing what it does when nobody reverses the consequences.
arXiv — LinuxArena: A Control Setting for AI Agents in Live Production Software Environments →