Troubleshoot hydration failures

View as Markdown

Hydration rebuilds the in-memory state of the objects on a cluster by reading from Materialize’s storage layer. A cluster is hydrated once every object on it is. A query served by an index that is still hydrating blocks until that index is ready, so a cluster that never finishes hydrating looks like a query that never returns, and a blue/green deployment that waits on it never cuts over. Reading a materialized view without an index goes to its persist shard instead, and waits only when the shard has not yet reached the query’s timestamp.

This guide helps you diagnose a cluster that never reaches a hydrated state. For a hydration that completes but spikes memory, see Memory spikes. To size a cluster from a hydration that did complete, see Optimize cluster size.

Step 1: Find what has not hydrated

mz_internal.mz_hydration_statuses reports one row per object per replica. Filtering on id LIKE 'u%' restricts the result to user objects, leaving out Materialize’s own introspection indexes:

SELECT
    c.name AS cluster_name,
    o.name AS object_name,
    o.type,
    h.replica_id,
    h.hydrated
FROM mz_internal.mz_hydration_statuses AS h
JOIN mz_catalog.mz_objects AS o ON o.id = h.object_id
JOIN mz_catalog.mz_clusters AS c ON c.id = o.cluster_id
WHERE h.hydrated IS NOT TRUE
  AND o.id LIKE 'u%'
ORDER BY c.name, o.name;

 cluster_name | object_name |       type        | replica_id | hydrated
--------------+-------------+-------------------+------------+----------
 batch_jobs   | orders_idx  | index             |            | f
 batch_jobs   | orders_mv   | materialized-view |            | f
(2 rows)

An empty result means every object is hydrated. Otherwise, replica_id tells you which case you are in:

NULL, as in the output above: no replica has reported on the object. Usually the cluster has no replicas, so nothing is hydrating it. Give it one with ALTER CLUSTER ... SET (REPLICATION FACTOR = <int>). A replica that has only just started also reports NULL until it reports in, so re-run the query before acting on a cluster that does have replicas. A cluster left at 0 additionally pins the history of its inputs, so read Step 3 before leaving it that way.
Set: that replica is hydrating the object, or failing to. Continue to Step 2.

Rows are per object per replica, so on a cluster with more than one replica an object appears here while another replica may already be serving it.

Step 2: Check for a rehydration loop

Hydration is memory-intensive, and a replica that runs out of memory while hydrating is killed and restarted, which begins hydration again from the start. If the workload no longer fits the replica’s size, every restart repeats the same out-of-memory kill and the cluster never reaches a hydrated state.

Hydration peaks well above steady state. Steady-state memory is therefore a poor guide to what the next restart will need, and a cluster sized against it can stop fitting as data grows, with no change to the objects on it.

SELECT
    rh.cluster_name,
    rh.replica_name,
    rh.size,
    count(DISTINCT h.occurred_at) FILTER (WHERE h.status = 'offline') AS offline_events,
    max(h.occurred_at) FILTER (WHERE h.reason = 'oom-killed') AS last_oom
FROM mz_internal.mz_cluster_replica_status_history AS h
JOIN mz_internal.mz_cluster_replica_history AS rh ON rh.replica_id = h.replica_id
WHERE h.occurred_at > now() - INTERVAL '1 day'
GROUP BY rh.cluster_name, rh.replica_name, rh.size
HAVING count(DISTINCT h.occurred_at) FILTER (WHERE h.status = 'offline') > 0
ORDER BY offline_events DESC;

 cluster_name | replica_name | size  | offline_events |        last_oom
--------------+--------------+-------+----------------+------------------------
 analytics    | r1           | 25cc  |             12 | 2026-09-23 15:53:41+00
(1 row)

Repeated offline_events over a short window is a rehydration loop. A last_oom confirms memory as the cause. It stays NULL when the orchestrator could not attribute the restart, so treat the restart count as the primary signal.

NOTE: Query the history rather than mz_internal.mz_cluster_replica_statuses, which reports only the current status and misses a restart that happens between polls. Join to mz_internal.mz_cluster_replica_history rather than mz_catalog.mz_cluster_replicas, so that replicas dropped by a resize still resolve to a name. A multi-process replica records one row per process per restart, which is why the count is over distinct occurred_at rather than over rows.

Resolution: size the cluster up with ALTER CLUSTER ... SET (SIZE = '<new_size>'). To choose a size from what the last successful hydration actually required, see Optimize cluster size. To lower the peak instead of raising the size, see Optimize hydration requirements.

Step 3: Check whether an input’s history is pinned

A new object starts reading at an as_of equal to the latest of its inputs' read frontiers, which is the earliest time at which all of them are readable at once.

Materialize compacts historical data whenever possible, to reduce resource consumption. Under normal circumstances, this means that about 1 second of history is available, and so a new object will have to replay only about 1 second of upstream history on top of the current snapshot. However, if compaction has been held back, the new object will need to replay more upstream history. This can take longer, and require more memory.

One pinned input is not enough on its own to cause that. Because the as_of takes the latest of the read frontiers, a single current input keeps it from being dragged back. An object starts far in the past only when every one of its inputs is pinned. A pinned input still retains more unconsolidated updates, which can make reading it more expensive, but it no longer sets where the object starts.

To determine if the input history is pinned, compare each object’s “read frontier” against its “write frontier”, using mz_internal.mz_frontiers:

SELECT
    o.name AS input,
    o.type,
    f.read_frontier::timestamp AS readable_from,
    f.write_frontier::timestamp - f.read_frontier::timestamp AS retained_history
FROM mz_internal.mz_frontiers AS f
JOIN mz_catalog.mz_objects AS o ON o.id = f.object_id
WHERE o.id LIKE 'u%'
  AND f.read_frontier IS NOT NULL
  AND f.write_frontier IS NOT NULL
ORDER BY retained_history DESC
LIMIT 5;

   input      |       type        |      readable_from      | retained_history
--------------+-------------------+-------------------------+------------------
 orders       | table             | 2026-09-15 15:31:02.511 | 192:59:38.363
 shipments    | table             | 2026-09-23 18:30:40     | 00:00:01.874
 shipments_mv | materialized-view | 2026-09-23 18:30:40     | 00:00:01.874
(3 rows)

A healthy object retains about a second of history. Hours or days mean its compaction is being held back.

A single row here is not a diagnosis. By the as_of rule above, pinned inputs only slow an object down when all of its inputs are pinned, so check the whole input set of the object that will not hydrate rather than acting on the worst row. mz_internal.mz_compute_dependencies lists an object’s inputs.

An explicit RETAIN HISTORY window is an expected reason for hours or days of retained history, not a fault. Check mz_internal.mz_history_retention_strategies before treating a large retained_history as a problem.

Find what is holding compaction back

An object holds a read hold on its inputs at a time no greater than its own write frontier, so anything whose write frontier has stopped advancing pins the history of everything it reads. Indexes on a cluster with no replicas are the exception, covered below.

Possible cause: an object cannot finish hydrating

The rehydration loop in Step 2 freezes that object’s write frontier, so it pins its own inputs for as long as the loop runs, and each restart has more history to replay than the last. Resolve it as described in Step 2.

Possible cause: a cluster has a replication factor of `0`

A cluster with REPLICATION FACTOR set to 0 never advances the write frontiers of the objects on it. Materialized views and sinks there go on pinning the history of their inputs for as long as they exist.

An index on such a cluster does not pin its inputs by itself: with no replica to serve reads from it, the controller advances its read hold anyway, and the holds it places on its own inputs follow. An index only pins when something else holds its since back, such as a materialized view or sink on the same cluster that reads from it.

List what such a cluster still carries:

SELECT
    dep.name AS input,
    o.name AS object_name,
    o.type,
    c.name AS cluster_name
FROM mz_internal.mz_compute_dependencies AS d
JOIN mz_catalog.mz_objects AS dep ON dep.id = d.dependency_id
JOIN mz_catalog.mz_objects AS o ON o.id = d.object_id
JOIN mz_catalog.mz_clusters AS c ON c.id = o.cluster_id
WHERE c.replication_factor = 0
ORDER BY c.name, o.name;

 input  | object_name |       type        | cluster_name
--------+-------------+-------------------+--------------
 orders | orders_mv   | materialized-view | batch_jobs
(1 row)

mz_internal.mz_compute_dependencies covers indexes and materialized views but not sinks, so a sink on the cluster pins its inputs without appearing here. Index rows are worth reading against the rule above: an index listed here is pinning only if a materialized view or sink on the same cluster reads from it.

Resolution: drop the materialized views and sinks the cluster carries, along with any index one of them reads from. The read holds belong to the objects rather than to the cluster, so dropping the cluster works only because it takes its objects with it. Setting a cluster’s replication factor to 0 is usually what caused the problem, and never fixes it. An index the cluster carries on its own is not holding anything back and does not need dropping.

Dropping is destructive. Dropping a materialized view discards its persisted output, so recreating it hydrates it again from its inputs. One with dependents needs DROP MATERIALIZED VIEW ... CASCADE, which drops those dependents too and leaves you to recreate them. Capture the definitions with SHOW CREATE MATERIALIZED VIEW or SHOW CREATE INDEX first.

Once the objects are gone, the read frontiers of their inputs advance again and a new deployment can follow within seconds.

Do not restore a replica to a cluster that has sat at 0 for days while it still carries a materialized view or sink. The replica rehydrates that object, and any index it reads from, through the whole retained backlog, which is the out-of-memory case in Step 2. Drop those objects and recreate them instead. A cluster carrying only indexes does not have this problem, because their read holds were forwarded while it was paused and a new replica starts them near the present.