Anomaly detection on metrics — the practical version

Anomaly detection is the most concrete of the three real AIOps wins. Latency that doubles at 3 AM. Error rate that triples on a Tuesday afternoon that's otherwise calm. Memory usage that creeps up over 72 hours when it usually doesn't. None of these would trigger a static threshold; all of them are anomalies your team would want to know about.

This lesson teaches you to build a working anomaly detector on real Prometheus latency metrics. We'll use Prophet (Facebook's time-series library) for the model, compare it to a simple z-score baseline, and to Datadog Watchdog. Then we'll wire the alerts into Slack and PagerDuty with severity-aware routing.

Time-series decomposition: what we're actually modelling

Almost every operational metric decomposes into three components:

Trend is the slow background change — "p95 latency has been creeping up 5ms per week as user load grows." Seasonality is the predictable repeating pattern — "weekday traffic is 3x weekend traffic; latency at 9 AM IST is 2x latency at 3 AM." Residual is what's left after subtracting trend and seasonality: the noise plus any anomalies.

A reasonable anomaly is a residual far enough from zero that it wouldn't have occurred by chance under the model's noise distribution. Concretely: residual outside the model's 99% confidence interval.

Static thresholds don't decompose. "Alert when p95 > 500ms" fires at 9 AM every day because real-load latency is higher than 3 AM latency. Anomaly detection on the residual fires only when latency is unusual for the time of day.

Why Prophet (and when not to)

Prophet, originally from Facebook, is the most usable open-source time-series library for operational metrics. The model fits trend + seasonality + holidays automatically; you don't need to be a statistician to use it.

Use Prophet when:

• The metric has visible daily or weekly seasonality
• You have ≥ 14 days of data history
• You can tolerate per-metric model fits (each metric gets its own model; that's hundreds of fits across a service catalogue)

Don't use Prophet when:

• The metric is mostly flat (no seasonality). A z-score with a rolling mean is simpler and works just as well.
• You have < 7 days of history. The seasonality fit is unreliable on too little data.
• You need sub-minute anomaly detection. Prophet's fit granularity is hourly at the finest; for second-level metrics, use a streaming detector (e.g. numenta-htm or Cassandra-based approaches).

A practical rule: start with Prophet for HTTP latency, error rate, queue depth, and database connection counts. Use z-score for CPU, memory, disk utilisation (mostly flat at the minute level). Use streaming for anything with second-resolution.

Building it: end-to-end code

We'll fit Prophet on the last 30 days of p95_latency_ms for a service called payment-api, then score the last hour of data for anomalies.

Step 1 — pull data from Prometheus

import pandas as pd
import requests
from datetime import datetime, timedelta

PROM_URL = "http://prometheus:9090"

def pull_metric(query: str, days: int) -> pd.DataFrame:
    end = datetime.utcnow()
    start = end - timedelta(days=days)
    response = requests.get(
        f"{PROM_URL}/api/v1/query_range",
        params={
            "query": query,
            "start": start.timestamp(),
            "end": end.timestamp(),
            "step": "5m",
        },
    )
    data = response.json()["data"]["result"][0]["values"]
    df = pd.DataFrame(data, columns=["ts", "value"])
    df["ds"] = pd.to_datetime(df["ts"], unit="s")
    df["y"] = df["value"].astype(float)
    return df[["ds", "y"]]

train_df = pull_metric(
    'histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{service="payment-api"}[5m]))',
    days=30,
)

Prophet wants columns named ds (timestamp) and y (value); this data shape is what the rest of the code assumes.

Step 2 — fit Prophet

from prophet import Prophet

model = Prophet(
    interval_width=0.99,            # 99% confidence interval
    daily_seasonality=True,
    weekly_seasonality=True,
    yearly_seasonality=False,       # we only have 30 days of data
    changepoint_prior_scale=0.05,   # mild trend changes
)
model.fit(train_df)

interval_width=0.99 controls the band around the fitted line we treat as "normal." Setting this to 0.95 produces more alerts; 0.999 produces fewer. Tune against your team's alert tolerance — we'll explain how below.

Step 3 — score the last hour

forecast_df = model.make_future_dataframe(
    periods=12,                    # 12 × 5-min steps = next hour
    freq="5min",
    include_history=True,
)
forecast = model.predict(forecast_df)

# Pull the most recent observed values
recent = pull_metric(
    'histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{service="payment-api"}[5m]))',
    days=1 / 24,   # last hour
)

# Join forecast confidence interval to observed values
joined = recent.merge(
    forecast[["ds", "yhat", "yhat_lower", "yhat_upper"]],
    on="ds",
    how="left",
)

# Mark anomalies
joined["is_anomaly"] = (joined["y"] > joined["yhat_upper"]) | (joined["y"] < joined["yhat_lower"])

print(joined[joined["is_anomaly"]])

If joined[joined["is_anomaly"]] returns rows, those rows are time points where the observed value fell outside the 99% confidence band. Alert on them.

Calibration: balancing precision vs recall (and why recall usually wins)

The single most important AIOps decision: how do you tune the alert threshold?

The conventional wisdom — "minimise false positives" — is wrong for operational alerts. You want high recall (catch real incidents) even at the cost of some false positives, because:

1. A missed real incident is much more expensive than a false alert. A missed-incident SLA breach costs the company customer compensation, regulatory penalties, customer-trust erosion. A false alert costs an on-call engineer 5 minutes of attention.
2. False alerts can be muted by runbook. If 9 AM Monday morning reliably fires a "latency anomaly" alert that turns out to be the week's traffic ramp, write a runbook entry — "9 AM Monday is expected; investigate only if it persists past 9:30" — and route the alert to a low-priority Slack channel.
3. Missed alerts have no equivalent muting mechanism. You can't un-miss an incident.

The discipline: tune for ~90% recall; accept ~10-15% false positive rate; lean on runbooks and severity routing to absorb the noise.

Wiring alerts into PagerDuty + Slack with severity routing

The detector above produces a list of (timestamp, metric, value, predicted_band). Three routing tiers:

SEVERITY_THRESHOLDS = {
    "critical": 3.0,    # observed value > 3x upper bound
    "high":     1.5,    # 1.5-3x upper bound
    "informational": 1.0,
}

def severity(observed: float, upper_bound: float) -> str:
    if upper_bound <= 0:
        return "informational"
    ratio = observed / upper_bound
    if ratio > SEVERITY_THRESHOLDS["critical"]:
        return "critical"
    if ratio > SEVERITY_THRESHOLDS["high"]:
        return "high"
    return "informational"

Then route:

• critical → PagerDuty (wake someone up)
• high → Slack #ops-alerts, with an @here mention
• informational → Slack #ops-noise, no mention (a low-priority channel humans look at during the workday but don't follow live)

The routing is what makes high-recall detection survivable. Without it, every false positive wakes someone up and the detector gets turned off.

Avoiding the 'alert fatigue → ignored alerts → outage' loop

The pattern that kills more AIOps initiatives than any other:

1. Team deploys anomaly detection.
2. First week: 200 alerts, most are noise.
3. Engineers stop reading the alerts.
4. Two months later, a real outage fires the same kind of alert that's been ignored for weeks.
5. Engineer ignores it for 30 minutes assuming it's noise.
6. Outage compounds. Customer-facing impact.
7. The post-incident review concludes "the detector was too noisy." Detector is turned off.

Prevention:

• Severity routing from day one, not as a follow-up. Even week 1, route low-confidence to a low-priority channel.
• Auto-mute after N false positives. If the same alert fires 5x in 24h and is manually marked as a false positive each time, auto-suppress it for 48 hours and notify the on-call team that it's muted.
• Weekly alert review. Look at every alert that fired last week; decide whether the threshold should tighten, the routing should change, or the alert should be deleted.

The discipline: anomaly detection is not "set it and forget it." It's a living system that needs weekly attention. Teams that treat it that way get value out of it; teams that don't, don't.

Comparing to Datadog Watchdog

If you're on Datadog, Watchdog gives you anomaly detection out of the box with no Prophet code to maintain. Trade-offs:

Watchdog pros:

• No code; flip a toggle per metric.
• Datadog has more training data across customers; its model generalises better on common metrics (CPU, HTTP latency, RPS).
• Auto-correlates with deployments, log spikes, etc.

Watchdog cons:

• Black box; you can't tune the model.
• Pricing scales with data volume; can get expensive at scale.
• Misses domain-specific signals that Prophet on your custom metric would catch.

Realistic stack: use Watchdog for breadth on common metrics. Layer custom Prophet detectors on top for the 3-5 metrics that are most important to your business (e.g. "successful checkout count" — a domain metric Watchdog doesn't have prior knowledge of).

Key takeaways

• Every operational metric decomposes into trend + seasonality + residual. Anomalies are residuals outside the noise distribution.
• Prophet is the right choice for metrics with daily/weekly seasonality and ≥ 14 days of history. z-score for flat metrics; streaming detectors for sub-minute resolution.
• The right calibration target is high recall (catch real incidents) with severity routing absorbing the false-positive noise.
• Severity routing on day one + auto-mute after N false positives + weekly alert review are the practices that keep the detector alive long-term.
• Datadog Watchdog is fine for breadth on common metrics; layer custom Prophet detectors on top for domain-specific signals.

What's next

The week 1 lab — train a Prophet model on real Prometheus latency data from the course sandbox and trigger Slack alerts on detected anomalies. By the end you'll have one working detector and the muscle memory to build similar ones for any metric your team cares about.