PRODUCTION SCENARIO
A team can either have two domain experts hand-grade 40 carefully written test cases each week, or build 1,200 automatically graded cases whose grading is somewhat noisier.
Which option should the team choose?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: The 1,200 automatically graded cases, since volume beats a handful of perfect grades
The eval design principles say to automate when possible and to prioritize volume over quality: more questions with slightly lower-signal automated grading beat fewer high-quality hand-graded evals.