The Streaming Database

Consistency, Scalability, Low Latency: Pick Three

The stream processor at the heart of Materialize proactively computes results with strong consistency and sub-second latency — all in a distributed, cloud-native architecture that allows for unbounded scale.

SQL Writes

INSERT

Databases

Message Brokers

SQL Control Plane

Compute (Clusters)

In-Memory Indexes

Dataflows

In-Memory Indexes

Dataflows

Storage (S3)

Sources

Source Data, Tables,
Materialized Views

Sinks

SQL Reads

SELECT

SUBSCRIBE

Message Brokers

Key Features

Presents as PostgreSQL

Manage and query Materialize using any Postgres driver or tool.

Streaming Inputs

Pull in streams of data from Kafka or stream from Postgres via replication.

Built for JOINs

Multi-way, complex join support across real-time streams - all in standard SQL.

Separate Storage & Compute

Data is stored cheaply, compute scales independently and without limits.

Incremental Compute Engine

Instead of re-computing on every query, results are updated as data changes.

Active Replication

Use replication to increase availability, reduce downtime, scale seamlessly.

Low-Latency Reads

Results can be maintained in memory, making read latency similar to Redis.

Event-Driven Primitives

Sink changes out to Kafka, or subscribe to query updates in standard Postgres.

What Can You Build with Materialize?

Data Engineers and Developers use Materialize as a better engine for cutting-edge data products.

Real-Time Analytics

Use the same ANSI SQL from data warehouses to build real-time views that serve internal and customer-facing dashboards, APIs and apps.

Read More →

Automation and Alerting

Build user-facing notifications, fraud and risk models, and automated services using event-driven SQL primitives in a streaming database.

Read More →

Segmentation and Personalization

Build engaging experiences with customer data aggregations that are always up-to-date: personalization, recommendations, dynamic pricing and more.

Read More →

ML Ops

Power online feature stores with continually updated data, monitor and react to changes in ML effectiveness - all in standard SQL.

Read More →

Same SQL, New Powers

Gain streaming data capabilities without learning a new language or framework.

Incrementally Maintained Views

Write complex SQL transformations as materialized views that efficiently update themselves as inputs change.

Learn More

Sliding Windows

Write queries that filter to a window of time anchored to the present, Materialize will update results as time advances.

Learn More

SQL Alerting

Write alerts as SQL queries with filters and subscribe to new rows as they appear.

Learn More

incremental.sql

1CREATE MATERIALIZED VIEW my_view AS
2  SELECT userid, COUNT(api.id), COUNT(pageviews.id)
3  FROM users
4  JOIN pageviews on users.id = pageviews.userid
5  JOIN api ON users.id = api.userId
6  GROUP BY userid;

$

userID	api_calls	pageviews
VPLaKV	400	20
MN37Mt	60	9
1fT4KY	72	42
sT4QY	10	342

Incrementally Maintained Views

Write complex SQL transformations as materialized views that efficiently update themselves as inputs change.

Learn More

incremental.sql

1CREATE MATERIALIZED VIEW my_view AS
2  SELECT userid, COUNT(api.id), COUNT(pageviews.id)
3  FROM users
4  JOIN pageviews on users.id = pageviews.userid
5  JOIN api ON users.id = api.userId
6  GROUP BY userid;

$

userID	api_calls	pageviews
VPLaKV	400	20
MN37Mt	60	9
1fT4KY	72	42
sT4QY	10	342

Sliding Windows

Write queries that filter to a window of time anchored to the present, Materialize will update results as time advances.

Learn More

sliding.sql

1CREATE MATERIALIZED VIEW my_window AS
2  SELECT date_trunc('minute', received_at),
3  COUNT(*) as order_ct, SUM(amount) as revenue
4  FROM orders
5  WHERE mz_now() < received_at + interval '5 minutes'
6  GROUP BY 1;

$

minute	order_ct	revenue

SQL Alerting

Write alerts as SQL queries with filters and subscribe to new rows as they appear.

Learn More

alerting.sql

1SELECT userID, email, MAX(orders.id) as last_order
2FROM users
3JOIN orders ON orders.userID = users.id
4GROUP BY userId, email
5-- Use a filter to surface users with a high % of fraud
6HAVING SUM(is_fraud) / COUNT(orders.id)::FLOAT > 0.5;

$

userID	email	last_order
REOtIb	a@gmail.com	13/12/2022
Y5KBE8	b@yahoo.com	9/12/2022
Wj7JQ0	c@hotmail.com	13/12/2022
tPCQ0	d@xyz.com	13/11/2022

Why Materialize?

Built from the ground up to solve the hard problems in data.

Strong Consistency Guarantees for Streaming Data

Materialize guarantees strict serializable consistency that is standard in databases, but unprecendented in distributed stream processors.

Consistency in Streaming →

Multi-way and Cross-Stream Joins

Write the same kinds of complex, multi-way SQL joins you would use on a traditional data warehouse. Materialize maintains the joins efficiently in memory and serves real-time results.

Joins in Materialize →

Event-Driven Primitives: Sink and Subscribe

Push updates out of the database without performance limits using two new primitives:

SQL clients can use SUBSCRIBE to get pushed incremental updates to results instead of polling.
Kafka users can create SINKS to push changes to results in a view out to a topic.

Subscribe to changes in a view →

PostgreSQL Up Front, Timely Dataflow Underneath

Materialize expands access to powerful stream processing capabilities of Timely and Differential Dataflow by wrapping them in a familiar and accessible SQL layer that is wire-compatible with Postgres.

Postgres Compatibility Explainer →

PG Wire compatibility Dataflow. Already used in correctness-critical global production deployments by Fortune 100 companies, these battle-tested systems avoid many of the shortcomings of other approaches to stream processing.

Works with Your Existing Data Stack

PostgreSQL

Streaming infrastructure is not required to use Materialize for real-time computation. Connect directly to any Postgres database via CDC and continually ingest data as it changes.

View Postgres Docs →

dbt

Run your existing dbt models on top of streaming data in Materialize, and dbt persists a materialized view. No matter how much or how frequently your data arrives, your model will stay up to date.

View dbt Docs →

Kafka

Connect multiple Kafka topics to Materialize and easily explore, transform, and join streaming datasets - and sink maintained SQL query results downstream to new, enriched Kafka topics.

View Kafka Docs →