Quick Reference
| Topic | File | Key Trap |
|---|
| ------- | ------ | ---------- |
| CI/CD and DAGs | pipelines.md | Coupling training/inference deps |
| Model serving | serving.md | Cold start with large models |
| Drift and alerts | monitoring.md | Only technical metrics |
| Versioning | reproducibility.md | Not versioning preprocessing |
| GPU infrastructure | gpu.md | GPU request = full device |
Critical Traps
Training-Serving Skew:
- Preprocessing in notebook ≠ preprocessing in service → silent bugs
- Pandas in notebook → memory leaks in production (use native types)
- Feature store values at training time ≠ serving time without proper joins
GPU Memory:
requests.nvidia.com/gpu: 1 reserves ENTIRE GPU, not partial memory- MIG/MPS sharing has real limitations (not plug-and-play)
- OOM on GPU kills pod with no useful logs
Model Versioning ≠ Code Versioning:
- Model artifacts need separate versioning (MLflow, W&B, DVC)
- Training data version + preprocessing version + code version = reproducibility
- Rollback requires keeping old model versions deployable
Drift Detection Timing:
- Retraining trigger isn't just "drift > threshold" → cost/benefit matters
- Delayed ground truth makes concept drift detection lag weeks
- Upstream data pipeline changes cause drift without model issues
Scope
This skill ONLY covers:
- CI/CD pipelines for models
- Model serving and scaling
- Monitoring and drift detection
- Reproducibility practices
- GPU infrastructure patterns
Does NOT cover: ML algorithms, feature engineering, hyperparameter tuning.