PRODUCTION SCENARIO
A Generative AI Engineer wrote a class-based MLflow Scorer subclass that checks whether each response contains a ticket number. It runs correctly in mlflow.genai.evaluate, but registering it for production monitoring of live traces is rejected.
What should the engineer change?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Express the same check as an @scorer-decorated function and register it
Production monitoring supports only @scorer decorator-based scorers, and class-based Scorer subclasses cannot be registered for it, so the same deterministic logic has to be expressed as a decorated function, registered with the MLflow experiment, and started with a sampling configuration. Scorers register with the experiment rather than with the serving endpoint, an LLM-graded Guidelines judge adds cost and variance to a check a string match settles, and a scheduled evaluation run scores a batch after the fact instead of live traces.