Home  /  Manage Materialize

Operational guidelines

View as Markdown

The following provides some general guidelines for production.

Clusters

Production clusters for production workloads only

Use production cluster(s) for production workloads only. That is, avoid using production cluster(s) to run development workloads or non-production tasks.

Three-tier architecture

In production, use a three-tier architecture, if feasible.

Image of the 3-tier architecture: Source cluster(s), Compute/Transform cluster(s), Serving cluster(s)

A three-tier architecture consists of:

Tier Description
Source cluster(s)

A dedicated cluster(s) for sources.

In addition, for upsert sources:

  • Consider separating upsert sources from your other sources. Upsert sources have higher resource requirements (since, for upsert sources, Materialize maintains each key and associated last value for the key as well as to perform deduplication). As such, if possible, use a separate source cluster for upsert sources.

  • Consider using a larger cluster size during snapshotting for upsert sources. Once the snapshotting operation is complete, you can downsize the cluster to align with the steady-state ingestion.
Compute/Transform cluster(s)

A dedicated cluster(s) for compute/transformation:

  • Materialized views to persist, in durable storage, the results that will be served. Results of materialized views are available across all clusters.

    💡 Tip: If you are using stacked views (i.e., views whose definition depends on other views) to reduce SQL complexity, generally, only the topmost view (i.e., the view whose results will be served) should be a materialized view. The underlying views that do not serve results do not need to be materialized.

  • Indexes, only as needed, to make transformation fast (such as possibly indexes on join keys).

    💡 Tip: From the compute/transformation clusters, do not create indexes on the materialized views for the purposes of serving the view results. Instead, use the serving cluster(s) when creating indexes to serve the results.
Serving cluster(s) A dedicated cluster(s) for serving queries, including indexes on the materialized views. Indexes are local to the cluster in which they are created.

Benefits of a three-tier architecture include:

Alternatives

If a three-tier architecture is infeasible or unnecessary due to low volume or a non-production setup, a two cluster or a single cluster architecture may suffice.

See Appendix: Alternative cluster architectures for details.

Sources

Scheduling

If possible, schedule creating new sources during off-peak hours to mitigate the impact of snapshotting on both the upstream system and the Materialize cluster.

Separate cluster(s) for sources

In production, if possible, use a dedicated cluster for sources; i.e., avoid putting sources on the same cluster that hosts compute objects, sinks, and/or serves queries.

In addition, for upsert sources:

  • Consider separating upsert sources from your other sources. Upsert sources have higher resource requirements (since, for upsert sources, Materialize maintains each key and associated last value for the key as well as to perform deduplication). As such, if possible, use a separate source cluster for upsert sources.

  • Consider using a larger cluster size during snapshotting for upsert sources. Once the snapshotting operation is complete, you can downsize the cluster to align with the steady-state ingestion.

See also Production cluster architecture.

Sinks

Separate sinks from sources

To allow for blue/green deployment, avoid putting sinks on the same cluster that hosts sources .

See also Cluster architecture.

Snapshotting considerations

For upsert sources, snapshotting is a resource-intensive operation that can require a significant amount of CPU and memory.

Hydration considerations

When sizing a cluster, budget for hydration memory on top of the steady-state cost. The table below summarizes, per object type, when each object hydrates and the memory it uses. For more on hydration, including strategies to reduce its impact, see Hydration.

Object Hydration behavior
Materialized views
  • When: Hydrates on creation and on every replica (re)start or cluster resize.
  • What: Rebuilds the dataflow’s operator state: the arrangements that joins, aggregations, and similar operators keep to update results incrementally. Note: A materialized view’s result lives in durable storage, so it rebuilds only this maintenance state, not the result.
  • Memory Use: Scales with the view’s definition, which it holds at steady state, plus a transient output buffer up to twice the output size: the current output plus a read-back of the previously persisted output. On first creation, since there is no previous output, the buffer is a single output size.
Indexes
  • When: Hydrates on creation and on every replica (re)start or cluster resize.
  • What: Rebuilds the arranged (indexed) data it keeps in memory to serve reads, plus any operator arrangements its dataflow maintains (for joins, aggregations, and similar).
  • Memory Use: Its memory is proportional to the indexed data plus those arrangements, and is held for as long as the index exists.
Kafka upsert sources and associated read-only tables/subsources
  • When: On replica (re)start or cluster resize. These sources do not hydrate on creation; instead, on creation, their indexes are built as part of snapshotting.
  • What: Rebuilds the table’s or subsource’s internal upsert index from storage.
  • Memory Use: The index holds the latest value per key, so its memory scales with the source’s key space. On standard cluster sizes it can spill to disk when the key space exceeds memory.
Append-only Kafka sources and CDC database sources (PostgreSQL, MySQL, SQL Server), and their read-only tables/subsources
  • When: On replica (re)start or cluster resize, marked hydrated as soon as the dataflow starts.
  • What: Effectively nothing. These sources keep no internal index to rebuild and resume from their persisted position, so hydration is a no-op.
  • Memory Use: Negligible, since there is no index to hold.
Webhook sources

Not applicable. A webhook source is not maintained by a dataflow. It receives data pushed over HTTP and writes the data directly to storage, so it does not hydrate.
Sinks

  • When: If created WITH (SNAPSHOT = true) (the default), hydrates:

    • On creation, when the sink first emits its input snapshot.
    • On a replica (re)start, but only if the sink restarted before recording any progress: it then re-reads the whole input snapshot, and any data already written to the external system is discarded, but the memory cost still occurs. An established sink resumes from its recorded progress without re-reading the snapshot.

  • What: Loads a full copy of its input snapshot into the arrangement that feeds the sink before it can emit.

  • Memory Use: Peaks at roughly a full copy of the input snapshot, then decreases as the snapshot is written out. Negligible on a restart of an established sink. At steady state, a sink retains little in memory.
Subscriptions
  • When: On creation and, while it remains active, on every replica (re)start: the dataflow is re-installed on the (re)started replica and the subscription resumes. A subscription that targets a specific replica instead ends with an error when that replica restarts. A subscription ends with its session and is not reported in mz_hydration_statuses.
  • What: Rebuilds the dataflow when it starts.
  • Memory Use: Scales with the dataflow, held while the subscription runs.

Role-based access control (RBAC)

    Cloud

    Follow the principle of least privilege

    Role-based access control in Materialize should follow the principle of least privilege. Grant only the minimum access necessary for users and service accounts to perform their duties.

    Restrict the assignment of Organization Admin role

    An Organization Admin has superuser privileges in the database. Following the principle of least privilege, only assign Organization Admin role to those users who require superuser privileges.

    Restrict the granting of CREATEROLE privilege

    Roles with the CREATEROLE privilege can obtain the privileges of any other role in the system by granting themselves that role. Avoid granting CREATEROLE unnecessarily.

    Use Reusable Roles for Privilege Assignment

    When possible, avoid granting privileges directly to individual user or service account roles (which are named after email addresses or service account user). Instead, create reusable, functional roles (e.g., data_reader, view_manager) with well-defined privileges, and grant these roles to the individual user or service account roles. You can also grant functional roles to other functional roles to compose more complex functional roles.

    See also Manage database roles.

    Audit for unused roles and privileges.

    Audit and remove unused roles periodically.

    See also Show roles in system and Drop a role for more information.

    Self-Managed

    Follow the principle of least privilege

    Role-based access control in Materialize should follow the principle of least privilege. Grant only the minimum access necessary for users and service accounts to perform their duties.

    Restrict the granting of CREATEROLE privilege

    Roles with the CREATEROLE privilege can obtain the privileges of any other role in the system by granting themselves that role. Avoid granting CREATEROLE unnecessarily.

    Use Reusable Roles for Privilege Assignment

    When possible, avoid granting privileges directly to individual user or service account roles. Instead, create reusable, functional roles (e.g., data_reader, view_manager) with well-defined privileges, and grant these roles to the individual user or service account roles. You can also grant functional roles to other functional roles to compose more complex functional roles.

    See also Manage database roles.

    Audit for unused roles and privileges.

    Audit and remove unused roles periodically.

    See also Show roles in system and Drop a role for more information.

    Back to top ↑