Blog

With major updates to the streaming replication connection to PostgreSQL, users can now set up Materialize as a drop-in enabler of real-time, incrementally updated, materialized views for their PostgreSQL database.

If you're already familiar with stream processors you may wonder: When is it better to use Materialize vs a Stream Processor? And why?

Materialize maintains an official Terraform Provider you can use to manage your clusters, replicas, connections and secrets as code.

You could just write SQL, get continually updated results on Materialize. But if you want to get more scale, performance, power, here is a gentle introduction to key internals that will help.

Four questions, and their answers, to explain ACID transactions and how they are handled within Materialize.

We spent hours interviewing engineers running change data capture in production, and compiled a list of their planning tips, design patterns to use and footguns to avoid here.

Here's a framework for thinking about reducing the time between when raw data is available to transform with dbt, and when it is delivering value to your customers.

Materialize aims to be usable by anyone who knows SQL, but for those interested in going deeper and understanding the architecture powering Materialize, this post is for you!

If you are familiar with materialized views and indexes from other databases, this article will help you apply that understanding to Materialize.

Let's build (in Python) the Differential Dataflow framework at the heart of Materialize, and explain what it's doing along the way.

If you're familiar with data warehouses, this article will help you understand Materialize Clusters in relation to well-known components in Snowflake.

Understand how to optimize joins with indexes and late materialization.

Differential Dataflow is capable of incrementally updated iterative computation (recursion) but we haven't yet wired it up to SQL. Let's talk about what recursion could look like in Materialize, and why it's important.

Get an overview of how streaming databases differ from traditional DBs. What are the tradeoffs? How are they used?

Materialize is written in Rust. Why did we make that decision and how has it turned out for the project?

Let's demonstrate the unique features of Materialize by building the core functionality of a customer data platform.

As a streaming database, Materialize is fundamentally different on the inside, but it's compatible with PostgreSQL in a few important ways.

Today, we’re excited to announce a product that we feel is transformational: a persistent, scalable, cloud-native Materialize.

Connect a real-time stream of user-centric events from Segment to Materialize, using Kafka as the intermediary.

Even in traditional databases, indexes can at different times be the problem and the solution when it comes to scaling. In this article we discuss how indexes change in streaming databases.

In traditional databases, a SQL query used as a test runs as a point-in-time check. In streaming, the same query can run continually as data changes, creating a SQL-based data monitoring primitive.

Let's explore a hands-on example where we use dbt (data build tool) to manage and document a streaming analytics workflow from a message broker to Metabase.

The key to Materialize's ability to separate compute from storage and scale horizontally without sacrificing consistency is a concept called virtual time.

What is a Data Application? How do they help our customers? What new challenges do we face when building Data Apps? Here's our perspective.

Connect headless BI tool Cube.js to the read-side of Materialize to get Rest/GraphQL API's, Authentication, metrics modelling, and more out of the box.

The `materialized` binary is stable and performant, the time has come to break it apart into separate services to enable the next phase: unbounded scale in a cloud architecture.

Let's use Materialize to deliver a feature store that continuously updates dimensions as new data becomes available without compromising on correctness or speed.

Developers have long wished for the ability to subscribe to changes in a SQL query or a view in a database. Materialize has a SUBSCRIBE primitive that makes it possible.

Changelog: AWS roles for S3/Kinesis, PostgreSQL source improvements, Schema Registry SSL, SELECT statements in Tail queries, jsonb subscripting, DBeaver support, & Tailscale in cloud.

The PostgreSQL write-ahead log (WAL) was originally created to enable hot standby's and multi-node setups, but it also works great as a source of data for Materialize. This article discusses how and why.

In this article, we will talk about one of the ways we approach the testing of the SQL engine of the product at Materialize. We hope to cover other modules and interesting angles in the future.

Materialize Cloud works with Tailscale, a VPN solution based on the state-of-the-art WireGuard protocol, to help customers connect their Materialize clusters with services on their private networks.

Changelog: Kafka source metadata, protobuf+schema registry for Redpanda, Time bucketing with date_bin, Metabase integration, cloud metrics and monitoring, and new availability region.

Redpanda is a source-available, Kafka-compatible streaming data framework that works both as an upstream data source for Materialize and downstream data sink. Read on to learn how to start building with Redpanda and Materialize.

Materialize raises another round of funding to help build a cloud-native streaming data warehouse.

Change Data Capture (CDC) is finally gaining widespread adoption as a architectural primitive. Why now?

Streaming joins must maintain the pre-join datasets in memory, making them potentially costly operations. Materialize uses shared arrangements to allow multiple join statements to share the same pre-join index.

Differential dataflow uses simple linear operators: `map`, `filter`, `flat_map` and complex: `explode` and temporal filter operators. But, with some thinking, we can generalize them all to a restricted form of join.

Debezium and Materialize can be used as powerful tools for joining high-volume streams of data from Kafka and tables from databases.

Build a real-time A/B testing stack with Segment, Kinesis and Materialize.

Let's demonstrate how to manage streaming SQL in Materialize with dbt by porting the classic dbt jaffle-shop demo scenario to the world of streaming.

Subquery optimization is a high-complexity, high-impact task in databases. This post gives a rough map of existing approaches to optimizing subqueries and also describes how Materialize differs from them..

dbt is a tool for managing SQL data transformations. Materialize is a streaming database. When used together, analytics works the way it always should have: Define transforms in SQL, get results in real-time.

Temporal filters give you a powerful SQL primitive for defining time-windowed computations over temporal data.

Let's build a python application to demonstrate how developers can create real-time, event-driven experiences for their users, powered by Materialize.

Let's use Conway's Game of Life to illustrate how to write algorithms in differential dataflow.

Joins in streaming systems are one of the harder things to do both correctly and efficiently. Let's talk about ways that Materialize maintains them, starting with basic binary joins and working our way up to delta joins.

In principle, it is possible to use Kafka as a database. But in doing so you will confront every hard problem that database management systems have faced for decades.

Materialize can be used to quickly build scalable backends for real-time apps! In this blog post, we describe two apps that you can try out at home that run on actual, live data.

In today's post we are going to show off Materialize's `LATERAL` join, and how you can use it to implement some pretty neat query patterns in an incremental view maintenance engine!

Here we set the context for and propose a change data capture protocol: a means of writing down and reading back changes to data.

Querying materialized views, unlike querying tables or logical views, can reduce query costs by maintaining results in memory that are only updated when necessary. Read on to learn more!

An explanation of our rationale for why Materialize chose not to use RocksDB as its underlying storage engine.

Eventual consistency is common for key-value stores, where the trade-off is well understood and manageable. But in a streaming system, eventual consistency creates unboundedly large and systematic errors.

How do you build a streaming database from scratch? Here is a roadmap: Start with a streaming framework, build a performant single binary, then break it up into a scalable distributed database platform.

Arjun Narayan introduces the CMU DB group to streaming databases, the problems they solve, and specific architectural decisions in Materialize.

Frameworks that process unbounded streams of data need to be diligent about not also using unbounded amounts of memory. This post discusses some of the tricks used by Differential Dataflow to manage and limit memory use.

In-order reliable message delivery is not enough. Showing views over streams of data requires thinking through additional consistency semantics to deliver correct results.

Let's review the internal architecture of Materialize, starting with the some context of how it's different than other databases.

Incremental computation systems are used to make frequent compute-intensive tasks faster and more efficient in compilers, front-ends, and IDEs, but what does it look like at the data layer?

What is a materialized view, where can you use them, and how are they useful?

Get a high-level overview of what Streaming SQL means, why it's useful, how it's being used in the real world, and how you can start using it yourself.

Despite substantial progress, data still moves too slowly. The solution is a different paradigm: Streaming. Materialize is a streaming data warehouse built on principles of interoperability and consistency at millisecond latency.