Stay up to date with the latest from Materialize. Technical Deep-Dives, announcements, and articles about Materialize, streaming, databases, and the future of data products.

Today, we’re excited to announce a product that we feel is transformational: a persistent, scalable, cloud-native Materialize.

Arjun Narayan

Frank McSherry

Announcing the next generation of Materialize

Join Marta Paes (Head of Developer Experience) for a demo on setting up CI/CD workflows to deploy dbt projects to Materialize.

Marta Paes

CI/CD workflows for dbt+Materialize

Here's a framework for thinking about reducing the time between when raw data is available to transform with dbt, and when it is delivering value to your customers.

Jessica Laughlin

Towards Real-Time dbt

Materialize aims to be usable by anyone who knows SQL, but for those interested in going deeper and understanding the architecture powering Materialize, this post is for you!

Brennan Vincent

The Software Architecture of Materialize

If you are familiar with materialized views and indexes from other databases, this article will help you apply that understanding to Materialize.

Chuck Larrieu Casias

When to Use Indexes and Materialized Views

Let's build (in Python) the Differential Dataflow framework at the heart of Materialize, and explain what it's doing along the way.

Ruchir Khaitan

Building Differential Dataflow from scratch

If you're familiar with data warehouses, this article will help you understand Materialize Clusters in relation to well-known components in Snowflake.

Dennis Hume

Clusters, explained with Data Warehouses

Understand how to optimize joins with indexes and late materialization.

Delta Joins and Late Materialization

Differential Dataflow is capable of incrementally updated iterative computation (recursion) but we haven't yet wired it up to SQL. Let's talk about what recursion could look like in Materialize, and why it's important.

Recursion in Materialize

Get an overview of how streaming databases differ from traditional DBs. What are the tradeoffs? How are they used?

Andy Hattemer

What is a Streaming Database?

Materialize is written in Rust. Why did we make that decision and how has it turned out for the project?

Gus Wynn

Our experience with Rust!

Let's demonstrate the unique features of Materialize by building the core functionality of a customer data platform.

Real-Time Customer Data Platform Views on Materialize

As a streaming database, Materialize is fundamentally different on the inside, but it's compatible with PostgreSQL in a few important ways.

How and why is Materialize compatible with PostgreSQL?

Join Andy Hattemer and Joaquin Colacci from our Developer Experience team for a technical overview on the major updates to Materialize.

Joaquin Colacci

The next generation of {Materialize}, an Overview

Connect a real-time stream of user-centric events from Segment to Materialize, using Kafka as the intermediary.

How to Connect Segment to Materialize

Even in traditional databases, indexes can at different times be the problem and the solution when it comes to scaling. In this article we discuss how indexes change in streaming databases.

Indexes: A Silent Frenemy

Join Anna Glander from the Developer Experience team for a live session and Q&A on using Materialize + dbt to do continuous testing.

Anna Glander

Materialize + dbt: From Streaming Analytics to Continuous Testing

In traditional databases, a SQL query used as a test runs as a point-in-time check. In streaming, the same query can run continually as data changes, creating a SQL-based data monitoring primitive.

Real-time data quality tests using dbt and Materialize

Let's explore a hands-on example where we use dbt (data build tool) to manage and document a streaming analytics workflow from a message broker to Metabase.

Managing streaming analytics pipelines with dbt

The key to Materialize's ability to separate compute from storage and scale horizontally without sacrificing consistency is a concept called virtual time.

Virtual Time: The Secret to Strong Consistency and Scalable Performance in Materialize

What is a Data Application? How do they help our customers? What new challenges do we face when building Data Apps? Here's our perspective.

Let’s talk about Data Apps

Connect headless BI tool Cube.js to the read-side of Materialize to get Rest/GraphQL API's, Authentication, metrics modelling, and more out of the box.

Igor Lukanin

Announcing the Materialize Integration with Cube

The `materialized` binary is stable and performant, the time has come to break it apart into separate services to enable the next phase: unbounded scale in a cloud architecture.

Materialize's unbundled cloud architecture

Let's use Materialize to deliver a feature store that continuously updates dimensions as new data becomes available without compromising on correctness or speed.

Seth Wiesman

Creating a Real-Time Feature Store with Materialize

Join Materialize cofounder and chief scientist Frank McSherry on June 22nd at 2pm ET for a live session and Q&A on the upcoming unbundled architecture of Materialize.

Technical Overview of Materialize's New Unbundled Architecture

Developers have long wished for the ability to subscribe to changes in a SQL query or a view in a database. Materialize has a SUBSCRIBE primitive that makes it possible.

Subscribe to changes in a view with Materialize

Changelog: AWS roles for S3/Kinesis, PostgreSQL source improvements, Schema Registry SSL, SELECT statements in Tail queries, jsonb subscripting, DBeaver support, & Tailscale in cloud.

Nicolle Eagan

What's new in Materialize? Vol. 2

Come hang out with us and build a streaming project alongside folks from all over the world! We’ll help you get the inspiration flowing with a seed project and some ideas for real-time data sources.

Andreas Fuchs

Amy Chen

Patrick Angeles

Materialize + dbt + Redpanda Virtual Hack Day 2022

The PostgreSQL write-ahead log (WAL) was originally created to enable hot standby's and multi-node setups, but it also works great as a source of data for Materialize. This article discusses how and why.

Petros Angelatos

Connecting Materialize directly to PostgreSQL via the Replication stream

In this article, we will talk about one of the ways we approach the testing of the SQL engine of the product at Materialize. We hope to cover other modules and interesting angles in the future.

Philip Stoev

Taming the beast that is a SQL database

Materialize Cloud works with Tailscale, a VPN solution based on the state-of-the-art WireGuard protocol, to help customers connect their Materialize clusters with services on their private networks.

Introducing: Tailscale + Materialize

Changelog: Kafka source metadata, protobuf+schema registry for Redpanda, Time bucketing with date_bin, Metabase integration, cloud metrics and monitoring, and new availability region.

What's new in Materialize? Volume 1

Join us to talk about how Redpanda and Materialize fit together, and how they can be used to build powerful streaming architectures previously only accessible to large enterprise teams.

Redpanda + Materialize

Redpanda is a source-available, Kafka-compatible streaming data framework that works both as an upstream data source for Materialize and downstream data sink. Read on to learn how to start building with Redpanda and Materialize.

Taking streaming analytics further faster with Redpanda + Materialize

Materialize raises another round of funding to help build a cloud-native streaming data warehouse.

Materialize raises a $60M Series C, bringing total funding to over $100M

Change Data Capture (CDC) is finally gaining widespread adoption as a architectural primitive. Why now?

Change Data Capture is having a moment. Why?

Materialize Cloud Enters Open Beta

Release: 0.9

Materialize & Datalot: Real-time Application Development

Release: 0.8

Streaming joins must maintain the pre-join datasets in memory, making them potentially costly operations. Materialize uses shared arrangements to allow multiple join statements to share the same pre-join index.

Maintaining Joins using Few Resources

Differential dataflow uses simple linear operators: `map`, `filter`, `flat_map` and complex: `explode` and temporal filter operators. But, with some thinking, we can generalize them all to a restricted form of join.

Generalizing linear operators in differential dataflow

Debezium and Materialize can be used as powerful tools for joining high-volume streams of data from Kafka and tables from databases.

Join Kafka with a Database using Debezium and Materialize

Build a real-time A/B testing stack with Segment, Kinesis and Materialize.

Real-time A/B test results with Segment, Kinesis, and Materialize

Let's demonstrate how to manage streaming SQL in Materialize with dbt by porting the classic dbt jaffle-shop demo scenario to the world of streaming.

dbt + Materialize demo: Running dbt’s jaffle_shop with Materialize

Release: 0.7

dbt is a tool for managing SQL data transformations. Materialize is a streaming database. When used together, analytics works the way it always should have: Define transforms in SQL, get results in real-time.

Introducing: dbt + Materialize

Subquery optimization is a high-complexity, high-impact task in databases. This post gives a rough map of existing approaches to optimizing subqueries and also describes how Materialize differs from them..

Jamie Brandon

How Materialize and other databases optimize SQL subqueries

Temporal filters give you a powerful SQL primitive for defining time-windowed computations over temporal data.

Temporal Filters: Enabling Windowed Queries in Materialize

Let's build a python application to demonstrate how developers can create real-time, event-driven experiences for their users, powered by Materialize.

Chris Golden

A Simple and Efficient Real Time Application Powered by Materialize's TAIL Command

Slicing up Temporal Aggregates in Materialize

Let's use Conway's Game of Life to illustrate how to write algorithms in differential dataflow.

Life in Differential Dataflow

Release: 0.6

Joins in streaming systems are one of the harder things to do both correctly and efficiently. Let's talk about ways that Materialize maintains them, starting with basic binary joins and working our way up to delta joins.

Joins in Materialize

In principle, it is possible to use Kafka as a database. But in doing so you will confront every hard problem that database management systems have faced for decades.

George Fraser

Kafka is not a Database

Materialize can be used to quickly build scalable backends for real-time apps! In this blog post, we describe two apps that you can try out at home that run on actual, live data.

Andi Wang

Live Maintained Views on Boston Transit to Run at Home

Materialize Raises a Series B

Release: Materialize 0.5

Materialize under the Hood

In today's post we are going to show off Materialize's `LATERAL` join, and how you can use it to implement some pretty neat query patterns in an incremental view maintenance engine!

Lateral Joins and Demand-Driven Queries

Here we set the context for and propose a change data capture protocol: a means of writing down and reading back changes to data.

Change Data Capture (part 1)

Querying materialized views, unlike querying tables or logical views, can reduce query costs by maintaining results in memory that are only updated when necessary. Read on to learn more!

Why Use a Materialized View?

An explanation of our rationale for why Materialize chose not to use RocksDB as its underlying storage engine.

Why not RocksDB for streaming storage?

Robust Reductions in Materialize

Release: Materialize 0.4

Streaming TAIL to the Browser - A One Day Project

Eventual consistency is common for key-value stores, where the trade-off is well understood and manageable. But in a streaming system, eventual consistency creates unboundedly large and systematic errors.

Eventual Consistency isn't for Streaming

Rust for Data-Intensive Computation

How do you build a streaming database from scratch? Here is a roadmap: Start with a streaming framework, build a performant single binary, then break it up into a scalable distributed database platform.

Materialize: Roadmap to Building a Streaming Database on Timely Dataflow

Arjun Narayan introduces the CMU DB group to streaming databases, the problems they solve, and specific architectural decisions in Materialize.

CMU DB Talk: Building Materialize

Release: Materialize 0.3

Frameworks that process unbounded streams of data need to be diligent about not also using unbounded amounts of memory. This post discusses some of the tricks used by Differential Dataflow to manage and limit memory use.

Managing memory with differential dataflow

In-order reliable message delivery is not enough. Showing views over streams of data requires thinking through additional consistency semantics to deliver correct results.

What consistency guarantees should you expect from your streaming data platform?

Upserts in Differential Dataflow

What is a materialized view, where can you use them, and how are they useful?

What is a Materialized View?

Incremental computation systems are used to make frequent compute-intensive tasks faster and more efficient in compilers, front-ends, and IDEs, but what does it look like at the data layer?

Incremental Computation in the Database

Let's review the internal architecture of Materialize, starting with the some context of how it's different than other databases.

Sean Loiselle

What’s inside Materialize? An architecture overview

Taking Materialize for a spin on NYC taxi data

Get a high-level overview of what Streaming SQL means, why it's useful, how it's being used in the real world, and how you can start using it yourself.

What is Streaming SQL?

View Maintenance: A New Approach to Data Processing

Materialize Beta: The Details

Despite substantial progress, data still moves too slowly. The solution is a different paradigm: Streaming. Materialize is a streaming data warehouse built on principles of interoperability and consistency at millisecond latency.

Blog

Announcing the next generation of Materialize

Subscribe

Get Materialize Updates in your inbox:

Follow Along

Our Product

Developer

Data Guides

Company

Join the Materialize Community

Announcing the next generation of Materialize

Subscribe

Get Materialize Updates in your inbox:

Follow Along

Footer

Our Product

Developer

Data Guides

Company

Join the Materialize Community