PRODUCTION SCENARIO
An ML engineer trains a SageMaker XGBoost model. Training error keeps falling while validation error rises after round 60. Every hyperparameter is at its default except max_depth, which is set to 14, and subsample and colsample_bytree, which are both 0.6.
Which hyperparameter change is MOST likely to improve validation performance?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Lower max_depth and raise lambda
Rising validation error against falling training error is overfitting, and max_depth at 14 is the cause: increasing max_depth makes a model more complex and likely to overfit, while raising lambda makes it conservative. Raising depth and lowering min_child_weight loosens both limits, lower eta with more rounds leaves depth untouched, and raising the two sampling ratios back toward 1 removes the randomization that is currently holding overfitting back.