Home  /  Concepts

Clusters

View as Markdown

Overview

Clusters are pools of compute resources (CPU, memory, and scratch disk space) for running your workloads.

Clusters and workloads

The following operations require a cluster in Materialize:

Each session has an active cluster, which you can change with SET CLUSTER.

SET CLUSTER = 'my_transform_cluster';

SELECT and SUBSCRIBE statements run in the session’s active cluster.

Objects that require compute (e.g., indexes, materialized views, sources) are associated with a cluster when they are created, either:

  • the session’s active cluster by default, or

  • the cluster specified by the IN CLUSTER <cluster> clause in the CREATE statement.

Cross-cluster objects

Tables, views, and materialized views are accessible across clusters. That is, you can query or reference them from any cluster.

Cluster-local objects

Indexes are accessible only from their own cluster. Indexed results reside in the memory of the cluster where the index is created, and a cluster’s memory cannot be accessed from another cluster.

For more on indexes and clusters, see Indexes.

Resource isolation

Clusters provide resource isolation. Each cluster provisions dedicated compute resources and can fail independently from other clusters. All workloads on a given cluster compete for access to that cluster’s compute resources.

Workloads on different clusters are strictly isolated from one another. That is, a given workload has access only to the CPU, memory, and scratch disk of the cluster it runs on.

Resource isolation lets you place workloads on separate clusters to prevent them from competing for compute resources: for example, sources in one cluster, materialized views in a second, and indexes that serve queries in a third, as in the recommended three-tier architecture.

Cluster replicas

The replication factor of a cluster determines the number of replicas provisioned for the cluster.

Each replica of a cluster provisions a new pool of compute resources to perform exactly the same work on exactly the same data. That is, replicas are redundant copies of the cluster’s workload, not shards: each replica processes the full workload.

Materialize automatically assigns names to replicas (e.g., r1, r2). You can view information about individual replicas in the Materialize console and the system catalog.

Fault tolerance

Provisioning more than one replica for a cluster improves fault tolerance. Clusters with multiple replicas can tolerate failures of the underlying hardware that cause a replica to become unreachable. As long as one replica of the cluster remains available, the cluster can continue to maintain dataflows and serve queries.

NOTE:

  • For Cloud, each replica incurs cost, calculated as cluster size * replication factor per second. See Usage & billing (Cloud) for more details.

  • Increasing the replication factor does not increase the cluster’s work capacity. Replicas are exact copies of one another: each replica must do exactly the same work as all the other replicas of the cluster (i.e., maintain the same dataflows and process the same queries). To increase the capacity of a cluster, you must increase its size.

Availability guarantees

When provisioning replicas,

  • For clusters sized up to and including 3200cc, Materialize guarantees that all provisioned replicas in a cluster are distributed across the underlying cloud provider’s availability zones.

  • For clusters sized above 3200cc, even distribution of replicas across availability zones cannot be guaranteed.

See also Hydration considerations.

Cluster sizing

When creating a cluster, you must choose its size (e.g., 25cc, 50cc, 100cc), which determines its resource allocation (CPU, memory, and scratch disk space) and cost (for Cloud). The appropriate size for a cluster depends on the resource requirements of your workload. Larger clusters have more compute resources available and can therefore process data faster and handle larger data volumes.

To gauge the performance and utilization of your clusters, use the Environment Overview page in the Materialize Console.

As your workload changes, you can resize a cluster. A resize triggers hydration. During hydration, the cluster keeps serving since Materialize provisions new replicas at the target size and hydrates them before retiring the old ones.

Hydration considerations

Hydration is the reconstruction of an object’s in-memory state by reading from Materialize’s storage layer and existing indexes; hydration does not read from the upstream system.

Depending on the object, hydration (or rehydration) occurs after:

  • An object is created, triggering its hydration.
    • This includes dropping and recreating objects to force re-planning. For example, after dropping an index, you can drop and recreate its dependent objects to force them to re-plan. The recreated objects then hydrate like any newly created object.
  • A cluster replica restarts, such as during Materialize Cloud’s routine maintenance or after an out-of-memory event. Hydration can be memory-intensive and can itself trigger the out-of-memory event. The replica then restarts and rehydrates again, potentially creating a restart-and-rehydrate loop if the replica is undersized.
  • A cluster resize. A cluster resize provisions new replicas at the target size and hydrates them before retiring the old ones. The cluster keeps serving throughout.
  • Adding a replica to a cluster, which hydrates the new replica only. Existing replicas are unaffected and keep serving.

Hydration is per cluster replica: when a hydration trigger occurs, the objects on the affected replicas hydrate. When a replica restarts, every object on it re-hydrates. A resize or an added replica hydrates only the new replicas.

💡 Tip: Hydration primarily impacts memory usage, and its speed scales with cluster size. To handle the temporary compute increases during hydration, you can configure an autoscaling strategy that provisions an extra burst replica at a larger size while the cluster has un-hydrated objects.

For more information, including hydration strategies and the memory usage of hydrating objects, see Hydration.

Best practices

The following provides some general guidelines for clusters. See also Operational guidelines.

Three-tier architecture in production

In production, use a three-tier architecture, if feasible.

Image of the 3-tier architecture: Source cluster(s), Compute/Transform cluster(s), Serving cluster(s)

A three-tier architecture consists of:

Tier Description
Source cluster(s)

A dedicated cluster(s) for sources.

In addition, for upsert sources:

  • Consider separating upsert sources from your other sources. Upsert sources have higher resource requirements (since, for upsert sources, Materialize maintains each key and associated last value for the key as well as to perform deduplication). As such, if possible, use a separate source cluster for upsert sources.

  • Consider using a larger cluster size during snapshotting for upsert sources. Once the snapshotting operation is complete, you can downsize the cluster to align with the steady-state ingestion.
Compute/Transform cluster(s)

A dedicated cluster(s) for compute/transformation:

  • Materialized views to persist, in durable storage, the results that will be served. Results of materialized views are available across all clusters.

    💡 Tip: If you are using stacked views (i.e., views whose definition depends on other views) to reduce SQL complexity, generally, only the topmost view (i.e., the view whose results will be served) should be a materialized view. The underlying views that do not serve results do not need to be materialized.

  • Indexes, only as needed, to make transformation fast (such as possibly indexes on join keys).

    💡 Tip: From the compute/transformation clusters, do not create indexes on the materialized views for the purposes of serving the view results. Instead, use the serving cluster(s) when creating indexes to serve the results.
Serving cluster(s) A dedicated cluster(s) for serving queries, including indexes on the materialized views. Indexes are local to the cluster in which they are created.

Benefits of a three-tier architecture include:

See also Operational guidelines.

Alternatives

Alternatively, if a three-tier architecture is not feasible or unnecessary due to low volume or a non-production setup, a two cluster or a single cluster architecture may suffice.

See Appendix: Alternative cluster architectures for details.

Use production clusters for production workloads only

Use production cluster(s) for production workloads only. That is, avoid using production cluster(s) to run development workloads or non-production tasks.

Back to top ↑