The 5 failure modes of production ML
Production ML systems fail in predictable ways. We catalogue the top 5 — feature drift, label drift, concept drift, infrastructure drift, eval drift — with real anonymized client examples.
- · Feature drift: input distribution changes (often during external events)
- · Concept drift: the relationship between features and label changes
- · Label drift: ground truth definition changes (annotation policy shifts)
- · Infrastructure drift: silent dependency upgrades break inference
- · Eval drift: holdout set becomes stale and stops representing reality