Home  /  Clusters  /  Troubleshoot clusters

Cluster memory spike troubleshooting

View as Markdown

A memory spike is a sudden increase in a cluster replica’s memory usage. A spike that outgrows the replica’s heap limit makes Materialize spill to disk, which slows the cluster down, or triggers an out-of-memory (OOM) kill and replica restart. This guide helps you find what caused a spike and how to prevent it recurring.

For sustained CPU/memory pressure and OOM crash loops on an undersized cluster, see Check cluster health. This guide focuses on spikes: sudden increases against an otherwise healthy baseline.

Step 1: Find when the spike happened

Check historical spike

mz_internal.mz_cluster_replica_metrics_history retains per-replica memory samples across restarts (at least 30 days by default), so you can find a past spike even if the replica has since recovered or restarted:

SELECT
    c.name AS cluster_name,
    r.name AS replica_name,
    h.process_id,
    max(h.heap_bytes) AS peak_heap_bytes,
    max(h.heap_limit) AS heap_limit_bytes,
    min(h.occurred_at) FILTER (WHERE h.heap_bytes > 0.9 * h.heap_limit) AS first_above_90pct
FROM mz_internal.mz_cluster_replica_metrics_history h
JOIN mz_catalog.mz_cluster_replicas r ON r.id = h.replica_id
JOIN mz_catalog.mz_clusters c ON c.id = r.cluster_id
WHERE h.occurred_at > now() - INTERVAL '7 days'
GROUP BY c.name, r.name, h.process_id
HAVING max(h.heap_bytes) > 0.9 * max(h.heap_limit)
ORDER BY peak_heap_bytes DESC;

 cluster_name | replica_name | process_id | peak_heap_bytes | heap_limit_bytes |   first_above_90pct
--------------+--------------+------------+-----------------+-------------------+------------------------
 prod_compute | r1           |          0 |      15461882265 |       17179869184 | 2026-09-08 15:49:03+00
(1 row)

Check a live spike

To check whether a replica is under memory pressure right now, query mz_internal.mz_cluster_replica_metrics:

SELECT
    c.name AS cluster_name,
    r.name AS replica_name,
    m.process_id,
    m.heap_bytes,
    m.heap_limit,
    round(100 * m.heap_bytes::numeric / m.heap_limit, 1) AS heap_pct
FROM mz_internal.mz_cluster_replica_metrics m
JOIN mz_catalog.mz_cluster_replicas r ON r.id = m.replica_id
JOIN mz_catalog.mz_clusters c ON c.id = r.cluster_id
ORDER BY heap_pct DESC;

 cluster_name | replica_name | process_id | heap_bytes  |  heap_limit | heap_pct
--------------+--------------+------------+-------------+-------------+----------
 prod_compute | r1           |          0 | 15461882265 | 17179869184 |     90.0
 prod_sources | r1           |          0 |  3221225472 | 17179869184 |     18.8
(2 rows)

For sub-minute detail and high-water marks that survive a spike shorter than the sampling interval, see Replica resource usage.

Step 2: Match the spike to a cause

Cross-reference the spike window against these common causes.

A new object hydrated on the cluster

Creating an index, materialized view, or source loads its full result set into memory: hydration. If the new object landed on a cluster that already runs production workloads, its hydration competes with that cluster’s steady-state memory.

Check mz_catalog.mz_audit_events for create events on the cluster around the spike window:

SELECT occurred_at, event_type, object_type, details
FROM mz_catalog.mz_audit_events
WHERE occurred_at BETWEEN '<spike_start>' AND '<spike_end>'
  AND object_type IN ('index', 'materialized-view', 'source')
ORDER BY occurred_at;

See also Check for DDL or deploy activity.

Resolution: Put new objects on their own cluster and cut over with a blue/green deployment instead of adding them to an existing production cluster. See Operational guidelines and Hydration strategies.

An ad-hoc query stood up a dataflow

If a query cannot be served by an existing index or materialized view, a new dataflow is created to serve it. This is similar to hydrating a new index; it causes memory usage to increase. See Dataflows: mental model and basic terminology for background.

Use EXPLAIN PLAN FOR <query> to check whether a query hits an existing index or builds new operators, then use Dataflow troubleshooting to identify which dataflow or operator is consuming memory.

Resolution: Rewrite the query to reuse an existing index, or add one, rather than relying on ad-hoc plans for hot-path queries. Some query patterns force a new dataflow even though they look selective, for example a WHERE clause using <> ALL (...) instead of NOT IN (...).

Many concurrent SUBSCRIBEs restarted at once

Like an ad-hoc query, an active SUBSCRIBE runs as its own dataflow. A cluster serving many concurrent subscribes (for example, one per connected UI session) can multiply its restart cost: every subscribe re-establishes its own snapshot when its replica comes back online, on top of the cluster rebuilding its indexes. That combined restart cost can exceed what the same subscribes cost at steady state, and, if it exceeds the replica’s memory limit, produce a restart loop that looks unrecoverable because every restart recreates the same spike.

Check how many subscribes are active on a cluster, using mz_internal.mz_subscriptions:

SELECT count(*) AS active_subscribes
FROM mz_internal.mz_subscriptions s
JOIN mz_catalog.mz_clusters c ON c.id = s.cluster_id
WHERE c.name = '<cluster_name>';

Resolution:

  • Add jittered backoff to client reconnect logic so subscribes re-establish gradually instead of all at once.
  • As a stopgap, moving the workload to a new cluster lets clients reconnect gradually against already-hydrated indexes, avoiding the simultaneous spike while you fix the reconnect pattern above.

The working set outgrew memory

If a cluster’s data no longer fits in memory, Materialize spills arrangements to its scratch disk, which shows up as elevated disk_bytes on mz_cluster_replica_metrics alongside memory near the limit, rather than an outright OOM. Two common contributors:

  • Compaction state. The same source data can require more memory to rehydrate on one replica than another if their upstream collections are compacted to different degrees; a freshly seeded environment can rehydrate cheaper than a long-running one that has retained more history.
  • Unnecessarily wide objects. An index or materialized view that carries columns no downstream query uses still arranges them, growing the arrangement for no benefit.

Resolution: Project objects down to only the columns downstream queries need, and use Dataflow troubleshooting: Why is Materialize using so much memory? to find which arrangement is largest.

Back to top ↑