Deep-diveGeneralizing linear operators in differential dataflow

Differential dataflows contain many operators, some of which are very complicated, but many of which are relatively simple. The map operator applies a transformation to each record. The filter operator applies a predicate to each record, and drops records that do not pass it. The flat_map operator applies a function to each record that can […]

Deep-diveJoin Kafka with a Database using Debezium and Materialize

The Problem We need to provide (internal or end-user) access to a view of data that combines a fast-changing stream of events from Kafka with a table from a database (which is also changing). Here are a few real-world examples where this problem comes up: Calculate API usage by joining API logs in Kafka with […]

Use CaseReal-time A/B test results with Segment, Kinesis, and Materialize

Introduction This is meant primarily to demonstrate how the Segment + Kinesis + Materialize stack can create new capabilities around querying, joining, and ultimately materializing real-time views of customer-centric data. In this case, we’re using A/B testing analytics as the data. Why? There’s a set of well-known problems and hard-earned lessons that data-centric organizations go […]

Ecosystemdbt + Materialize demo: Running dbt’s jaffle_shop with Materialize

Introduction We’ve recently partnered with dbt and community members to release a beta Materialize dbt adapter that allows you to transform your streaming data in real time using Materialize as your data warehouse. This tutorial serves as a practical hands-on demonstration of the adapter. In this case, we are using static not streaming data, but […]

ReleaseRelease: 0.7

Materialize 0.7 was released on 08 February 2021 with significant improvements around getting data into Materialize. Key change: Source data from Amazon Web Services S3 S3 sources for Materialize are fully tested but under the experimental flag until 0.8. With S3 sources, you can: Point Materialize at S3 buckets using the same CREATE SOURCE syntax […]

EcosystemIntroducing: dbt + Materialize

Managing data is hard. Managing data pipelines is even harder. The meaning of individual tables or values in your data warehouse gets lost in translation across organizations. Another team’s refactor breaks your team’s pipeline. And, it’s normally very difficult to tell who made what change and when. dbt, data build tool, alleviates these frustrations by […]

Deep-diveHow Materialize and other databases optimize SQL subqueries

Subqueries are a SQL feature that allow writing queries nested inside a scalar expression in an outer query. Using subqueries is often the most natural way to express a given problem, but their use is discouraged because most databases struggle to execute them efficiently. This post gives a rough map of existing approaches to optimizing […]

StreamingStreaming SQL: What is it, why is it useful?

Summary Streaming SQL is about taking the same declarative SQL used to write database queries, and instead running it on streams of fast-changing data. This is useful because: Data is often more valuable when you can act on it quickly The existing tools for deriving real-time insights from streams are too complex. The “declarative” nature […]

Deep-diveTemporal Filters: Enabling Windowed Queries in Materialize

Materialize provides a SQL interface to work with continually changing data. You write SQL queries as if against static data, and then as your data change we keep the results of your queries automatically up to date, in milliseconds. Materialize leans hard into the ideal that SQL is what you know best, and what you […]

Deep-diveA Simple and Efficient Real Time Application Powered by Materialize’s TAIL Command

Within the web development community, there has been a clear shift towards frameworks that implement incremental view maintenance and for good reason. When state is updated incrementally, applications perform better and require fewer resources. Using Materialize, developers and data analysts can adopt the same, event driven techniques in their data processing pipelines, leveraging existing SQL […]

About This Blog

Welcome! On our blog, you’ll hear more about the inner workings of Materialize – what we’ve built, what we plan to build, and how it all works together.

New here? Read these