PRODUCTION SCENARIO
An agent ends a test run with 'ticket closed'. The evaluator grades that final sentence and marks the run a pass. The ticket is still open in the test system.
What should determine whether the task succeeded?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Whether the ticket is closed in the test system at the end of the run
The outcome is the final state of the environment, so the assertion to write is a query against the test system's ticket record. A close_ticket call in the transcript can have errored, been rejected, or targeted the wrong ticket. A judge model reading the transcript grades the same claim the agent made and never touches the system. Budget compliance says the run was cheap, and a cheap run that changed nothing still failed.