Materialize Self-Managed v26.0.0: Upstream Schema Change Support, Cost Efficiency Improvements, and Security Enhancements

We're thrilled to announce Materialize v26.0.0, the newest major release of Self-Managed Materialize. This update includes an enhancement to sources, allowing you to handle upstream schema changes with zero downtime. With v26.0.0, swap is enabled by default, reducing the amount of physical memory (RAM) required to run workloads. Finally, we’ve added support for SASL/SCRAM authentication, giving you a more secure way to connect to Materialize.

If you'd like to get started right away, follow our first-time installation guide or our upgrade guide. But if you're curious about the details, read on!

Materialize sources ingest data continuously from external systems, like PostgreSQL databases, or Kafka topics. In simple terms, you can think about this as ingesting continuous streams of data and the associated schemas. Historically, handling schema changes in database sources has been difficult in Materialize. To incorporate a schema change, you would have had to drop and recreate the source, risking downtime

With the latest release of Materialize, you can handle two of the most important types of upstream schema changes from PostgreSQL sources seamlessly, without any downtime: adding and dropping columns from your upstream database.

To enable this, we’ve made a slight modification to the syntax for creating a source. The code block below shows what it now looks like to create a source which can handle upstream schema changes:

-- First, create a connection, using the same syntax as before:

CREATE CONNECTION pg_connection TO POSTGRES (

  HOST '<host>',

  PORT 5432,

  USER 'materialize',

  PASSWORD SECRET pgpass,

  SSL MODE 'require',

  DATABASE '<database>'

);

-- Next, create a source

CREATE SOURCE IF NOT EXISTS my_source

    FROM POSTGRES CONNECTION pg_connection (PUBLICATION mz_source);

-- Finally, create a table from the source:

CREATE SCHEMA v1;

CREATE TABLE v1.T

    FROM SOURCE my_source(REFERENCE public.T);

In the example above, we’re ingesting data from an upstream table named T, with a single column A into Materialize. If you’ve used Materialize before, you might notice the differences between the new syntax and the legacy syntax! If you’re not ready to use the new syntax yet, don’t worry. The legacy syntax will continue to be supported, and you can migrate over when you are ready.

Of course, as before, you can create a materialized view which reads from the source table:

/* First, create a connection, using the same syntax as before: */

CREATE MATERIALIZED VIEW v1.matview AS

    SELECT SUM(A) from v1.T;

If we make a schema change to T, such as adding a column B, you can incorporate that schema change by creating a new version of the table:

CREATE SCHEMA v2;

CREATE TABLE v2.T

FROM SOURCE my_source(REFERENCE public.T);

CREATE MATERIALIZED VIEW v2.matview AS

    SELECT SUM(A) 

    FROM v2.T

    WHERE B = true;

You might have downstream consumers reading data from v1.matview and v1.source. If you want those consumers to receive the new version, you can atomically switch the v1 and v2 schemas, using the ALTER SCHEMA command:

ALTER SCHEMA v1 SWAP WITH v2;

The command above switches both schema names atomically. As soon as the ALTER command executes, downstream consumers will begin receiving data from your newest source & materialized view.

For a detailed tutorial, and a guide on how to drop columns safely, check out our documentation on how to handle upstream schema changes with zero downtime.

The newest source syntax is available today in private preview. It is supported for PostgreSQL sources, and we plan to add support for SQLServer and MySQL in the near future. Please contact our support team if you would like early access to it.

Materialize now supports excluding specific columns when ingesting data from PostgreSQL sources. This feature allows you to substantially reduce hydration times by ignoring unnecessary columns. You can also use EXCLUDE COLUMNS in conjunction with the latest source syntax to handle dropping columns from an upstream database without any downtime in Materialize.

Following in a similar vein to the previous example: imagine you have an upstream PostgreSQL table T, with columns A and B. You can exclude column A during ingestion:

-- Create the source

CREATE SOURCE IF NOT EXISTS my_source

    FROM POSTGRES CONNECTION pg_connection (PUBLICATION mz_source);

-- Create a table, 'T', in the v1 schema which excludes column 'A'

CREATE SCHEMA v1;

CREATE TABLE v1.T

    FROM SOURCE my_source(REFERENCE public.T) WITH (EXCLUDE COLUMNS (A));

EXCLUDE COLUMNS is supported on the legacy source syntax as well:

CREATE SOURCE mz_source 

FROM POSTGRES CONNECTION pg_conn (PUBLICATION 'mz_source', EXCLUDE COLUMNS (T.A))

FOR ALL TABLES;

WIth this release, EXCLUDE COLUMNS is now supported across all our relational database source types.

Swap allows for infrequently accessed data to be moved from memory to disk. Enabling swap reduces the physical memory (RAM) required to run workloads on Materialize and improves cost efficiency. Since we first introduced swap in our Cloud product, many Materialize users have been able to downsize their clusters while maintaining similar performance levels as before.

With v26.0.0, swap is enabled by default for Self-Managed environments. Simply follow our installation & upgrade guides to use it.

We've introduced SASL/SCRAM-SHA-256 authentication support for Materialize. SASL is a framework for modern authentication protocols, and SCRAM-SHA-256 is a modern authentication mechanism that never sends your password over the network. Instead, both the client and server use cryptographic proofs to verify identity. This means your database connections are now protected by enterprise-grade authentication that never exposes passwords on the wire.

Using SCRAM is particularly useful when setting up a connection pooler such as PgBouncer. Connection poolers like PgBouncer improve performance by reusing a small pool of active database connections across many clients, eliminating the expensive overhead of repeatedly creating and tearing down new connections for each request. With SCRAM, PgBouncer can authenticate users using hashed credentials retrieved from the database rather than requiring plaintext passwords in its configuration file.

To use this feature, follow our guide to enable SASL/SCRAM authentication, and our guide to enable connection pooling.

v26 incorporates numerous smaller improvements:

• COPY TO now supports all S3 compatible destinations, including Google Cloud Storage
• Multi-replica clusters with sources & sinks, which allow you to make zero downtime updates to source cluster sizes
• Support for compression with webhook sources
• Performance optimizations for incoming source data, to reduce CPU utilization
• Several Console bugfixes, including fixing session timeout errors

A heads up: starting with v26.0.0, you will be required to provide a license key to use Self-Managed Materialize.

We continue to offer a complete free Self-Managed Community License, with a limit of 24 GiB of memory and 48 GiB of disk space. There are no time restrictions on Community License usage. You can obtain your Self-Managed Community license key through our website. If your workload requires more resources, contact our team for a Self-Managed Enterprise license.

A license key is required only once when creating a new environment or upgrading an existing environment to v26.0.0. After obtaining your license key, please follow our installation guides, or our upgrade guides, to add it to Materialize.

We’ve historically released updates to Materialize Cloud on a weekly basis. We’re going to start doing the same for our Self-Managed product as well. This means that v26.0.0 is the first of many upcoming releases.

If you’re a user of Materialize Cloud: you already have access to the exciting updates above! If you’re an existing user of Materialize Self-Managed, follow our upgrade guide here. If you’ve never used Materialize before - choose the deployment model which works best for you.