PRODUCTION SCENARIO
A churn model reaches 0.99 AUC on its validation split but performs close to random in production. The engineer reviews how each input feature is produced: the billing system writes account_closure_reason when an account is cancelled, the activity service recomputes days_since_last_login every night, the billing system writes monthly_charge on each invoice, and the ticketing service aggregates support_tickets_90d over a rolling window.
Which feature should be removed from the training set?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: account_closure_reason, written by the billing system when an account is cancelled
account_closure_reason is written only when an account is cancelled, so it is a consequence of the label rather than a predictor of it, and no value exists when a live customer is scored. The other three are produced on a nightly, per-invoice or rolling schedule, so each carries a value at prediction time and none of them encodes the outcome the model is trying to predict.