Computer Pathshalaकंप्यूटर पाठशाला
Start free
videoLesson 01· 16 minFree previewContent ready

Why most ML projects never reach production

Video coming soon. The lesson script and lab are ready; recording is in production.voiceover script available ↓

Why most ML projects never reach production

There's a number that gets quoted in every keynote about machine learning in industry: 60 to 80 percent of ML projects never reach production. Gartner, VentureBeat, NewVantage Partners, Algorithmia — every survey agrees on the order of magnitude even when the exact figure varies. The projects don't fail because the models were inaccurate. They fail at the boring stuff in front of and behind the model: getting the data in, deploying the model, watching it after deployment, deciding who owns the incident pager when it drifts.

This lesson maps that failure pattern so the rest of the course makes sense. We're not here to teach you how to train a slightly-better XGBoost. We're here to teach you the discipline that turns a Jupyter notebook into a thing your company will pay engineering salaries to keep running.

The shape of the failure

If you've worked in a company that tried to ship ML, the story is probably familiar. A data scientist gets a problem — fraud detection, churn prediction, lead scoring. They pull a CSV of historical data into a notebook. They try four model families. One of them gets impressive numbers on the held-out set. They write it up. Leadership is excited. Someone says "let's productionise this." Six months later, the project is dead.

Where did it die? Almost never in the model. It died at one of these gates:

  1. Data pipeline. The notebook ran on a hand-curated CSV. Production needs that same data refreshed daily, joined with three other tables, validated for schema drift, and reproducible. Nobody built that.
  2. Reproducibility. The notebook used pd.read_csv('/Users/anya/Desktop/v3_FINAL_use_this_one.csv'). Six months later, nobody can recreate the training set. Even Anya can't — she changed laptops.
  3. Deployment. The data scientist handed over a model.pkl to the platform team and said "you serve this." The platform team has no idea what features the model expects, what version of scikit-learn it was trained with, or what to do when an inference request takes 4 seconds.
  4. Monitoring. The model went live. Three months later, conversion dropped 12% and nobody noticed for two weeks. The model had drifted — inference inputs no longer matched the training distribution. Nobody had set up a drift detector.
  5. Ownership. When the model misbehaves, who fixes it? The data scientist who built it is on another project. The platform team owns the service but not the model. The business owns the outcome but doesn't know what an F1 score is. The model rots in a no-man's-land.

Notice that "model not accurate enough" isn't a node on that graph. Accuracy is rarely what kills these projects. The boring 70% — the data pipeline, the deployment surface, the monitoring, the ownership — is what kills them.

The 'POC purgatory' anti-pattern

The Proof-of-Concept Purgatory pattern goes like this: a company decides to "do AI." They hire a data science team. The team builds POCs — sentiment analysis, lead scoring, fraud detection. Each POC works in the notebook. Each gets a 20-slide deck. Each is presented to leadership. Each is declared "promising." None of them ever go live.

After 18 months, the data science team has 12 POCs and zero shipped products. The CTO looks at the budget and asks where the value is. The team gets disbanded or repositioned to "data analytics." This is the rule, not the exception, in companies that approach ML without engineering rigour.

What's actually happening: the team is producing demos, not products. A demo proves the model can work under controlled conditions. A product works in the wild, repeatedly, when conditions change. The gap between the two is engineering — and it's what this course is about.

Why a SageMaker notebook is not a production system

You'll hear "SageMaker is the production ML platform from AWS" and it's easy to assume that just by using SageMaker you're doing MLOps. You're not. SageMaker is a toolbox of services. The toolbox includes:

Most teams use only the first one. They train in a Studio notebook, save the model artifact, and either email it to the platform team or stand up an endpoint by hand from the notebook. They've used one tool out of six and none of the discipline the other five enforce. The notebook is the part of SageMaker that looks most like what they already know (Jupyter), so they stop there.

This course teaches you to use all six in a way that fits together. The through-line: a model goes from data → training → registered version → deployed endpoint → monitored in flight → re-trained when drift is detected — and all of it lives in code in your git repo, not in a senior data scientist's laptop.

Eval suites in CI: the discipline that separates research from product

Here's a discipline almost no team has, and the one that most changes the quality of a production ML system: evaluation suites that run in CI.

In conventional software, you don't merge code without tests passing. The tests give you a guarantee that the new code doesn't break behaviour customers depend on. In ML, the equivalent is the eval suite: a fixed set of inputs with known correct outputs, run against the model on every proposed change, with a pass/fail threshold.

Without an eval suite, every model change is a leap of faith. The training set metric went up — does the model actually do the job better? You don't know. Without an eval suite, you can't refactor your training pipeline without risking a silent regression. You can't tighten one slice of the data without checking that another slice didn't degrade. You can't distinguish "the new model is better" from "the new model is better on the test set we happen to have."

Teams that have eval suites can ship 10x more model versions per quarter than teams that don't, because each ship is a routine push instead of a nail-biting deploy-and-pray.

Drift detection: feature drift vs concept drift vs label drift

Three flavours of drift kill production models. They're often conflated; it matters that you can tell them apart because the fix is different for each:

We'll wire all three in week 7 of the course. For now, just know that "monitoring an ML system" is at least three monitoring jobs, not one.

Who owns a deployed model — the org chart trap

The hardest problem isn't technical. It's the org chart.

In most companies, models are owned by data scientists, who don't carry the pager. Endpoints are owned by platform engineers, who don't understand the model. Business outcomes are owned by product managers, who don't know which model is in production this week. When something breaks, you have three groups who each can see only part of the problem.

The MLOps move is to put ownership in one place: a single team that owns both the model and the service that serves it, with on-call rotation covering both. The team needs members fluent enough in both ML and platform engineering that they can debug a feature-drift incident at 2 AM without five Slack threads. This is rare; building toward it is the real organisational work of doing ML in production.

Throughout this course, when we deploy something, we'll be explicit about who owns it and what their pager looks like. The technical pattern always implies an org pattern. If your org can't support the pattern, the pattern won't ship.

Key takeaways

What's next

In Lesson 02 we'll go hands-on: build a complete SageMaker Pipeline that trains a model on S3 data, registers a versioned artifact in the Model Registry, and deploys it to a real-time endpoint with shadow traffic. By the end of that lesson you'll have seen — end to end — what every later week of the course builds on.

Further reading