Cardinality is a budget, not an accident

Every observability platform we are asked to rescue has the same shape. It worked beautifully for eighteen months. Then dashboards started timing out, the ingest bill grew faster than traffic, and somebody added a retention flag to make the symptoms go away.

The cause is almost always the same, and it is not volume. It is cardinality.

What cardinality actually costs

A time series is a unique combination of metric name and label values. http_requests_total{method="GET", status="200", service="api"} is one series. Change any label value and it is a different one.

The cost is multiplicative. A metric with 5 methods, 12 status codes and 40 services is 2,400 series — entirely fine. Add pod with 200 values and it is 480,000. Add user_id and you have built an unbounded system that will fall over on a date you cannot predict.

What makes this specifically painful is where the cost lands:

• Memory. Prometheus holds an index of every active series. Roughly 3–4 KB each in practice, so five million active series is about 16 GB of RAM before samples.
• Query latency. A query touching a million series is slow no matter how much hardware you throw at it, because the index lookup itself is the bottleneck.
• Ingest cost. Every hosted backend prices on series or samples. Cardinality is the multiplier on your invoice.

Set the budget explicitly

Cardinality is only ever a problem when nobody owns the number. So make it a number someone owns.

We start every engagement with a written budget, allocated per team, and enforced at ingest:

The number is less important than the fact that it exists. A team with a budget makes different choices about label design than a team with none, and they make those choices before the metric ships rather than during an incident.

Enforce it in Mimir per tenant so exceeding the budget is a visible rejection rather than a silent cost:

overrides:
  payments-team:
    max_global_series_per_user: 250000
    max_global_series_per_metric: 20000
    ingestion_rate: 50000
    ingestion_burst_size: 500000
    max_label_names_per_series: 30
    # Reject expensive queries rather than letting them starve the read path.
    max_fetched_series_per_query: 500000
    max_query_lookback: 90d

max_global_series_per_metric is the one that catches real mistakes early. A single metric with 20,000 series is nearly always a label that should not be there, and rejecting it names the metric in the error — which is far more useful than a tenant-level limit that just says "you are over."

Kill the offending labels at ingest

Most cardinality arrives from three sources, and all three are fixable with relabelling.

Unbounded identifiers

User IDs, request IDs, session IDs, full URL paths. These belong in traces and logs, which are built for high-cardinality lookup. They must never become metric labels.

metric_relabel_configs:
  # Drop known-unbounded labels outright.
  - regex: "(user_id|request_id|session_id|trace_id|email)"
    action: labeldrop
 
  # Collapse URL paths to route templates: /orders/8a3f/items -> /orders/:id/items
  - source_labels: [path]
    regex: "(/[^/]+)/[0-9a-f]{8,}(/.*)?"
    target_label: path
    replacement: "${1}/:id${2}"
 
  # Bucket status codes; 2xx is almost always enough.
  - source_labels: [status_code]
    regex: "([0-9])[0-9][0-9]"
    target_label: status_class
    replacement: "${1}xx"
  - regex: "status_code"
    action: labeldrop

Path templating is the single highest-value rule on most clusters. An API emitting raw paths generates a new series for every resource ID it has ever served, and the series never expire from the index within the retention window.

Churning pod names

Every deploy replaces every pod, and every replacement is a new value of the pod label. With daily deploys, a 200-pod service generates 6,000 distinct pod values a month, all retained.

Keep pod only where you genuinely debug per-pod — usually resource saturation, which cAdvisor and kube-state-metrics already cover — and drop it from application metrics, where the aggregate is what you actually query.

scrape_configs:
  # Application metrics: aggregate view only. Per-pod resource detail
  # comes from the kubelet job, so it is redundant here.
  - job_name: app-metrics
    kubernetes_sd_configs:
      - role: pod
    metric_relabel_configs:
      - regex: "pod|instance|pod_template_hash|controller_revision_hash"
        action: labeldrop

Histograms nobody sized

A histogram with 20 buckets is 22 series per label combination, before you multiply by anything. Three histograms with 20 buckets each across 40 services and 3 environments is 7,920 series from three metrics.

Most services need 8 to 10 buckets, chosen around the SLO threshold rather than spread evenly across a log scale. If your latency objective is 300 ms, you need resolution near 300 ms and almost none at 10 seconds.

Recording rules for anything a human waits on

A dashboard query that scans a million series will be slow forever. Precompute it.

groups:
  - name: slo-burn
    interval: 30s
    rules:
      - record: service:http_requests:rate5m
        expr: sum by (service, status_class) (rate(http_requests_total[5m]))
 
      - record: service:http_errors:ratio5m
        expr: |
          sum by (service) (rate(http_requests_total{status_class="5xx"}[5m]))
            /
          sum by (service) (rate(http_requests_total[5m]))
 
      - record: service:latency:p99_5m
        expr: |
          histogram_quantile(0.99,
            sum by (service, le) (rate(http_request_duration_seconds_bucket[5m])))

Two rules of thumb keep this honest. Anything on a dashboard someone opens during an incident should be a recording rule — dashboard load time is incident response time. And name them with the level:metric:operation convention so the aggregation level is legible from the name alone.

The saving compounds: a recording rule evaluates once every 30 seconds regardless of how many engineers have the dashboard open, and during an incident that number is much larger than one.

Alert on burn rate, not thresholds

Threshold alerts produce the worst possible pairing: pages for brief blips, and silence during slow degradation. Multi-window burn-rate alerting fixes both.

groups:
  - name: slo-alerts
    rules:
      # Fast burn: at this rate the monthly budget is gone in ~2 days. Page.
      - alert: ErrorBudgetBurnFast
        expr: |
          service:http_errors:ratio5m > (14.4 * 0.001)
            and
          service:http_errors:ratio1h > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.service }} is burning error budget 14x too fast"
 
      # Slow burn: real, but it can wait for business hours. Ticket.
      - alert: ErrorBudgetBurnSlow
        expr: |
          service:http_errors:ratio6h > (6 * 0.001)
            and
          service:http_errors:ratio3d > (6 * 0.001)
        for: 1h
        labels:
          severity: ticket

The and across two windows is what removes the false pages. A 30-second spike moves the 5-minute window but not the 1-hour one, so it never fires. A genuine degradation moves both.

0.001 is the error budget for a 99.9% objective. Change the objective and both thresholds follow from it, which means the alert is derived from a commitment rather than from a number someone picked in a meeting.

Retention should be tiered, not chosen

The instinct when costs rise is to cut retention globally. That is the wrong lever, because the queries you actually run at different ages are different queries.

• 0–15 days, full resolution. Incident response and debugging. Everything, at native scrape interval.
• 15–90 days, 5-minute downsample. Capacity planning and trend analysis. Nobody looks at 15-second resolution from six weeks ago.
• 90 days–13 months, 1-hour downsample, recorded series only. Quarterly reviews and year-over-year comparison. Keep the SLO recording rules, drop the raw series.

Mimir and Thanos both implement this as compactor configuration. It typically removes 60–80% of long-term storage cost while preserving every query anyone actually runs, and it is a much better answer than telling teams they can no longer see last month.

The review that keeps it working

Cardinality is not a project. Put thirty minutes in the calendar every month and look at four things:

# 1. Which metrics are the most expensive right now?
topk(20, count by (__name__)({__name__=~".+"}))
 
# 2. Which label is doing the damage on the worst offender?
count(count by (pod) (container_memory_usage_bytes))
 
# 3. Are we trending toward the budget?
sum(prometheus_tsdb_head_series) by (tenant)
 
# 4. What did we ingest and never query?
# (Grafana usage insights, or Mimir query-frontend logs by metric name)

That fourth one is consistently the most surprising. In most estates, a meaningful share of ingested series have never appeared in a single query or alert. They are pure cost, they are usually the output of a default exporter configuration nobody trimmed, and deleting them is the easiest budget win available.

Cardinality growth is not a failure of discipline. It is the natural result of many people making individually reasonable decisions with no shared budget. Give them the budget, enforce it at ingest, and the problem stops being an emergency and becomes a number on a dashboard — which is, after all, what you built the platform for.