What AIOps actually means (and what it doesn't)

AIOps is more often a vendor pitch than a working pattern. Walk any DevOps World expo floor and every booth claims AI-driven everything. Most of it is regression with a marketing budget. Some of it is genuinely useful.

This lesson separates the real wins from the marketing. By the end of it, you'll know which AIOps capabilities are worth investing in for your team, which ones are not, and what data prerequisites have to be in place before any of them work.

Three real AIOps wins

Of the dozens of capabilities sold under the AIOps banner, three have actually delivered measurable value at scale across multiple teams we've worked with:

Noise reduction — alert clustering and de-duplication. When a service degrades, you typically don't get one alert; you get fifty. The same root cause fires alerts on the service itself, on every dependency that timed out, on every dashboard that breached its SLO. A good AIOps tool clusters these into one incident with one notification. This is the highest-ROI capability, and it's mostly deterministic — nearest-neighbour clustering on (timestamp, service, metric_name) triples, not deep learning.

Anomaly detection on metrics — finding spikes and dips that wouldn't have triggered a static threshold but are statistically unusual for that metric. The classic example: latency that doubles at 3 AM is below any reasonable SLO threshold but is anomalous because 3 AM latency is usually low. Time-series decomposition (trend + seasonality + residual) plus a confidence-interval threshold catches these. We build one of these in Lesson 02.

Root-cause hints — when an incident fires, the AIOps tool surfaces "this metric also changed at the same time" or "this deployment happened 8 minutes earlier in this service's dependency chain." It doesn't tell you the root cause. It points at three things to look at first. This is a real time-saver during a 2 AM page.

Three things sold as AIOps that mostly aren't

"AI-driven self-healing." The vendor pitch: the AIOps system detects an anomaly and automatically restarts the affected service / scales up the cluster / fails over to a replica. Reality: this works for the trivial 5% of incidents (memory leak → restart → fixed). For the other 95%, the same action that "heals" the trivial case causes or masks a worse incident. Cascading restarts hide a thundering-herd bug. Auto-scaling hides a runaway query. The teams that have working "self-healing" have it because they hand-coded the runbook actions and put each one behind a circuit breaker. There's no AI deciding.

"Predictive maintenance for cloud workloads." Predictive maintenance is a real thing for physical systems (turbines, hard drives, EV batteries) with measurable degradation curves. Web services don't have those curves. A stateless Node.js container either works or doesn't; "predicting" failures based on CPU patterns is mostly fantasy. The exceptions: disk failures on bare-metal servers (SMART data + ML can genuinely warn you 30+ days out), and GPU thermal degradation in HPC clusters.

"AI-detected business anomalies." The pitch: the tool watches your business metrics (signups, revenue, conversions) and alerts on unusual changes. Reality: it does, but the alerts are useless because business metrics are dominated by external factors (marketing campaigns, holidays, competitor launches). A "drop in signups" alert that fires every time you change your homepage isn't an anomaly detector — it's a regression detector you didn't ask for.

The data prerequisite

No AIOps tool works without good observability data underneath it. This is the most-skipped step in every AIOps adoption we've seen. Teams buy the AIOps tool first; then spend six months getting their data into shape; then quietly forget they bought the tool.

What "good data" means:

• Structured logs in JSON or another structured format. Plain- text logs are unsearchable in volume; alert correlation requires fields you can group by.
• Metrics with consistent labels. A service label that's consistent across HTTP metrics, JVM metrics, database metrics, Kubernetes metrics. Without it, alert correlation can't connect the dots.
• Distributed traces with consistent trace IDs propagated across service boundaries. RCA hints come from saying "here's a slow trace; the latency originated in service B." Without traces, you're guessing.
• A unified time-series store. Prometheus + Loki + Tempo, or Datadog, or New Relic — pick one stack and standardise. AIOps tools that have to crawl across three or four data stores are fragile and slow.

If your team doesn't have all four in place, stop. The first six weeks of any AIOps initiative are observability hygiene, not AI. We spend Week 2-3 of this course on the data side; only then do we get back to anomaly detection.

Build vs buy: where AIOps tooling lives in 2026

Three categories of tool:

Hosted (Datadog Watchdog, Splunk ITSI, New Relic AI, BigPanda). These are the big commercial AIOps offerings. They work well if you're already on their observability platform. They're expensive ($$$$$ per node + per data volume). The ML models they run are mostly black boxes; you can't tune them, you take what the vendor ships.

Self-hosted (PrometheusAlertManager + custom routing, Karma, Falcon, NetSentinel-style stacks). For noise reduction specifically, AlertManager's clustering + inhibition + grouping is ~80% of what you'd get from a commercial tool, at zero license cost. We'll use it in Lesson 02's stretch goals.

DIY ML on your own data (Prophet, sktime, statsmodels, Anomaly Detection libraries on top of Prometheus/Datadog metrics). Good for anomaly detection where the rules are domain-specific and the commercial tools' generic models don't fit. More work; more control; more skill required on the team.

Our recommendation by team size:

• <50 engineers, basic needs: AlertManager + a custom Prometheus query → Slack flow. Don't buy AIOps yet.
• 50-500 engineers, multi-service: Self-hosted noise reduction
  • Prophet/statsmodels anomaly detection on key metrics. Commercial AIOps if you're already paying for Datadog and the marginal cost of Watchdog is small.
• 500+ engineers, regulated industry: Commercial AIOps starts paying off because the cost of incidents is high enough to justify $200k+ in tooling. Even then, layer DIY anomaly detection on top for the domain-specific signals.

Key takeaways

• The three real AIOps wins are noise reduction, anomaly detection, and RCA hints. Most of "noise reduction" isn't even ML — it's clustering + grouping.
• "AI-driven self-healing," "predictive maintenance for stateless workloads," and "AI-detected business anomalies" are mostly marketing.
• Observability data prerequisite: structured logs, metrics with consistent labels, traces with propagated IDs, unified store. No AIOps works without these.
• Build vs buy depends on team size and existing tooling. Small teams: AlertManager + custom scripts. Mid: hybrid. Large + regulated: commercial AIOps + DIY on top.

What's next

Lesson 02 — anomaly detection on metrics, the practical version. We build a working anomaly detector on real Prometheus latency data using Prophet, compare it to Datadog's Watchdog, and wire the alerts into PagerDuty. By the end of the lesson you'll have one working detector and the framework to build others.