PRODUCTION SCENARIO
A team can have two domain experts hand-grade 40 carefully written test cases each week, or build 1,200 automatically graded cases whose grading is somewhat noisier.
Which option should the team choose?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Build the 1,200 automatically graded cases and start using them
The eval design principles say to automate where possible and to prioritise volume over quality: more questions with slightly lower-signal automated grading beat fewer hand-graded ones. Growing a 40-case set a few at a time never reaches the volume that catches regressions. Hand-grading all 1,200 first reinstates the expert bottleneck the pipeline exists to remove. Halving the automated set to buy 20 expert grades trades away coverage for precision the principle says not to buy.