Home  /  Concepts

Clusters

Overview

Clusters are pools of compute resources (CPU, memory, and scratch disk space) for running your workloads.

The following operations require compute resources in Materialize, and so need to be associated with a cluster:

Resource isolation

Clusters provide resource isolation. Each cluster provisions dedicated compute resources and can fail independently from other clusters.

Workloads on different clusters are strictly isolated from one another. That is, a given workload has access only to the CPU, memory, and scratch disk of the cluster that it is running on. All workloads on a given cluster compete for access to that cluster’s compute resources.

Fault tolerance

The replication factor of a cluster determines the number of replicas provisioned for the cluster. Each replica of the cluster provisions a new pool of compute resources to perform exactly the same work on exactly the same data.

Provisioning more than one replica for a cluster improves fault tolerance. Clusters with multiple replicas can tolerate failures of the underlying hardware that cause a replica to become unreachable. As long as one replica of the cluster remains available, the cluster can continue to maintain dataflows and serve queries.

NOTE:

  • Increasing the replication factor does not increase the cluster’s work capacity. Replicas are exact copies of one another: each replica must do exactly the same work as all the other replicas of the cluster(i.e., maintain the same dataflows and process the same queries).

    To increase the capacity of a cluster, you must increase its size.

  • See also Usage.

Materialize automatically assigns names to replicas (e.g., r1, r2). You can view information about individual replicas in the Materialize console and the system catalog.

Cluster sizing

When creating a cluster, you must choose its size (e.g., 25cc, 50cc, 100cc), which determines its resource allocation (CPU, memory, and scratch disk space).

For self-managed Materialize, the cluster sizes are configured with the following default resource allocations:

Size Scale CPU Limit Disk Limit Memory Limit
25cc 1 0.5 7762MiB 3881MiB
50cc 1 1 15525MiB 7762MiB
100cc 1 2 31050MiB 15525MiB
200cc 1 4 62100MiB 31050MiB
300cc 1 6 93150MiB 46575MiB
400cc 1 8 124201MiB 62100MiB
600cc 1 12 186301MiB 93150MiB
800cc 1 16 248402MiB 124201MiB
1200cc 1 24 372603MiB 186301MiB
1600cc 1 31 481280MiB 240640MiB
3200cc 1 62 962560MiB 481280MiB
6400cc 2 62 962560MiB 481280MiB
NOTE: If you have modified the default cluster size configurations, you can query the mz_cluster_replica_sizes system catalog table for the specific resource allocations.

The appropriate size for a cluster depends on the resource requirements of your workload. Larger clusters have more compute resources available and can therefore process data faster and handle larger data volumes.

As your workload changes, you can resize a cluster. Depending on the type of objects in the cluster, this operation might incur downtime. See Resizing downtime for more details.

💡 Tip: To gauge the performance and utilization of your clusters, use the Environment Overview page in the Materialize Console.

Best practices

The following provides some general guidelines for clusters. See also Operational guidelines.

Three-tier architecture in production

In production, use a three-tier architecture, if feasible. A three-tier architecture consists of:

Tier
A dedicated cluster(s) for sources.
A dedicated cluster(s) for compute (e.g., materialized views).
A dedicated cluster(s) for serving queries, including indexes.

Benefits of a three-tier architecture include:

Upsert source consideration

In addition:

  • Consider separating upsert sources from your other sources. Upsert sources have higher resource requirements (since, for upsert sources, Materialize maintains each key and associated last value for the key as well as to perform deduplication). As such, if possible, use a separate source cluster for upsert sources.

  • Consider using a larger cluster size during snapshotting for upsert sources. Once the snapshotting operation is complete, you can downsize the cluster to align with the steady-state ingestion.

Alternatives

Alternatively, if a three-tier architecture is not feasible or unnecessary due to low volume or a non-production setup, a two cluster or a single cluster architecture may suffice.

    If a three-tier architecture is not feasible or unnecessary due to low volume or a non-production setup, a two cluster architecture may suffice. A two-cluster architecture consists of:

    • A dedicated cluster for sources.

    • A dedicated cluster for both compute and serving queries.

    Benefits of a two-cluster architecture include:

    However, with a two-cluster architecture, compute and queries compete for the same cluster resources.

    If a three-tier architecture is not feasible or unnecessary due to low volume or a non-production setup, a single cluster may suffice for your sources, compute objects, and query serving needs.

    Benefits of a single cluster architecture include:

    • Cost effective

    Limitations of a single cluster architecture include:

    • Sources, compute objects, and queries compete for cluster resources.

    • Blue/green deployment is unsupported since sources would need to be dropped and recreated, putting strain on your upstream system during source recreation.

      To support blue/green deployments, use a two-cluster architecture by moving compute objects to a new cluster (i.e., recreating compute objects in a new cluster).

    Use production clusters for production workloads only

    Use production cluster(s) for production workloads only. That is, avoid using production cluster(s) to run development workloads or non-production tasks.

    Consider hydration requirements

    During hydration, materialized views require memory proportional to both the input and output. When estimating required resources, consider both the hydration cost and the steady-state cost.

    Back to top ↑