Time Series Data at Scale

You start with PostgreSQL because you already run it. The monitoring data goes into a metrics table with a timestamp column, a metric name, a few label columns, and a value. Works fine for the first month when you’re tracking 500 time series from 10 services. A filing cabinet for a small office.

Then the Kubernetes migration happens. Suddenly 200 pods are emitting 30 metrics each at 15-second intervals. The metrics table hits 400 million rows. The filing cabinet for a small office is now handling the mail for a stadium. VACUUM takes 45 minutes. Dashboard queries that returned in 200ms now take 12 seconds. B-tree indexes are fighting the write load for I/O. Your DBA sends a pointed email about that one table eating most of the database’s I/O budget.

Why General-Purpose Databases Break Under Time Series Workloads

A medium infrastructure footprint: 100 nodes, 50 pods each, 20 metrics at 15-second intervals. That generates 576 million data points per day from monitoring alone.

B-tree indexes were designed for random access patterns, not append-only, time-ordered, high-throughput writes. Forcing time series data through a relational engine means you’re fighting the storage architecture itself. Purpose-built TSDBs use time-based partitioning, in-memory write buffers, and compression algorithms (delta encoding on timestamps, XOR on values) for far better storage efficiency. A PostgreSQL row storing a single metric sample consumes roughly 16 bytes. VictoriaMetrics compresses that same sample to 1-2 bytes.

Cardinality: The Silent TSDB Killer

Unbounded labels are the root cause of almost every TSDB outage. user_id, session_id, request_id. These fields create millions of new series per day. The correct architecture for request-level granularity is distributed tracing, not metrics. Metrics are for aggregated trends. Traces handle individual requests. Blurring that boundary is the most common mistake in observability implementations.

Labels multiply, not add. One unbounded label (user_id, request_id, trace_id) turns a 20-series metric into 200 million. The TSDB falls over, and nobody understands why until they check cardinality.

Controls that catch it before it happens:

Query Optimization and Recording Rules

A Grafana panel scanning 30 days of raw data with no aggregation will DDoS your own monitoring stack. During an incident. When you need it most.

Three query disciplines prevent this. Aggregation functions must match the visualization. A 7-day chart doesn’t need 40 million raw points. Label matchers reduce scan scope early by filtering on specific services or endpoints before aggregation. Step intervals should align to the time range: 15-second steps for a 1-hour window, 5-minute steps for a 24-hour window, 1-hour steps for a 7-day view.

Recording rules are the underrated performance lever. They pre-compute expensive aggregations and store results as new time series. A dashboard panel that runs rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m]) across 200 services on every page load can be replaced by a recording rule that computes and stores service:http_request_duration_seconds:mean5m continuously. Multi-second dashboard loads become sub-second responses. An afternoon of configuration for a huge improvement in usability.

groups:
  - name: service_latency_aggregations
    interval: 30s
    rules:
      - record: service:http_request_duration_seconds:mean5m
        expr: |
          rate(http_request_duration_seconds_sum[5m])
            / rate(http_request_duration_seconds_count[5m])
      - record: service:http_request_duration_seconds:p99_5m
        expr: |
          histogram_quantile(0.99,
            rate(http_request_duration_seconds_bucket[5m]))

The full end-to-end architecture:

Downsampling and Retention Tiers

10,000 active series at 15-second resolution produces 21 billion data points per year. Without retention policies, storage costs just keep growing. Three tiers solve this.

Configure downsampling before the storage bill arrives, not after. VictoriaMetrics supports it natively with -retentionPeriod and downsampling flags. Thanos uses a compactor against object storage. Data engineering teams should plan compute capacity for the downsampling pipeline at setup time, not as an afterthought when storage costs spike.

Alerting Architecture on Time Series Data

Alert on symptoms, not causes. An error rate exceeding 1% captures every failure mode, including ones nobody anticipated. A log-based alert for a specific error string captures only the failure someone predicted.

Multi-burn-rate alerting gets rid of the false-positive noise that comes with simple threshold alerts. A 5-minute window detects acute spikes. A 1-hour window detects sustained degradation. Requiring both windows to breach before paging cuts noise way down. Google’s SRE workbook formalizes this approach, and it works in practice as well as on paper.

When the alert queue is always full, investigation quality collapses. Teams stop triaging and start ignoring. Every alert should have a documented response. If an alert fires repeatedly without requiring human action, either auto-remediate it or eliminate it. Alerts that train people to ignore alerts are worse than no alerts at all.

That metrics table devouring your database’s I/O budget? With the right TSDB, tiered retention, cardinality controls, and SLO-based alerting, your monitoring scales with the infrastructure instead of against it. Performance and capacity engineering turns metric thresholds into capacity decisions before they become incidents. The DBA stops sending pointed emails. The dashboards load in under a second. And the 576 million daily data points become a well-managed pipeline instead of a storage crisis.