PRODUCTION SCENARIO
A Generative AI Engineer wrote a class-based MLflow Scorer subclass that checks whether each response contains a ticket number. It runs correctly in mlflow.genai.evaluate, but registering it for production monitoring of live traces is rejected.
What should the engineer change?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Rewrite the check as a function decorated with @scorer, then register it
Production monitoring accepts only @scorer decorator-based scorers, so the same deterministic logic must be expressed as a decorated function, registered with the experiment, and started with a sampling configuration. An LLM judge adds cost for a deterministic format check, and sampling or storage changes do not affect registration.