PRODUCTION SCENARIO
A data engineering team already operates Apache Airflow DAGs for the upstream ETL that feeds a model, with an established alerting path and on-call rotation. The ML steps must run inside that same dependency graph, and the team wants one scheduler to operate.
Which orchestrator meets these requirements with the LEAST added operational surface?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Amazon MWAA, extending the existing DAGs with SageMaker operators
MWAA runs the Airflow the team already operates and ships operators for SageMaker AI, so the ML steps join the existing DAGs under one scheduler, one alerting path and one rotation. Pipelines, Step Functions and CodePipeline would each orchestrate the ML steps competently, but each one adds a second scheduler alongside Airflow and a second place to look when a nightly run fails.