Materialize v0.8 was released on 9 June 2021 with significant improvements and features, including temporal filters, upserts, PubNub sources, S3 sources, and native Postgres sources. Note that the release is also available in Materialize Cloud. Key Change: Postgres Sources We now support POSTGRES sources. We use Postgres’s native replication protocol so you can stream updates directly […]
Today’s post is on a topic that a lot of folks have asked for, once they dive a bit into Materialize. One of our join implementation strategies uses a surprisingly small amount of additional memory: none. “None” is a surprising amount of memory because streaming joins normally need to maintain their inputs indexed in memory. […]
Differential dataflows contain many operators, some of which are very complicated, but many of which are relatively simple. The map operator applies a transformation to each record. The filter operator applies a predicate to each record, and drops records that do not pass it. The flat_map operator applies a function to each record that can […]
The Problem We need to provide (internal or end-user) access to a view of data that combines a fast-changing stream of events from Kafka with a table from a database (which is also changing). Here are a few real-world examples where this problem comes up: Calculate API usage by joining API logs in Kafka with […]
Introduction This is meant primarily to demonstrate how the Segment + Kinesis + Materialize stack can create new capabilities around querying, joining, and ultimately materializing real-time views of customer-centric data. In this case, we’re using A/B testing analytics as the data. Why? There’s a set of well-known problems and hard-earned lessons that data-centric organizations go […]
Introduction We’ve recently partnered with dbt and community members to release a beta Materialize dbt adapter that allows you to transform your streaming data in real time using Materialize as your data warehouse. This tutorial serves as a practical hands-on demonstration of the adapter. In this case, we are using static not streaming data, but […]
Materialize 0.7 was released on 08 February 2021 with significant improvements around getting data into Materialize. Key change: Source data from Amazon Web Services S3 S3 sources for Materialize are fully tested but under the experimental flag until 0.8. With S3 sources, you can: Point Materialize at S3 buckets using the same CREATE SOURCE syntax […]
Managing data is hard. Managing data pipelines is even harder. The meaning of individual tables or values in your data warehouse gets lost in translation across organizations. Another team’s refactor breaks your team’s pipeline. And, it’s normally very difficult to tell who made what change and when. dbt, data build tool, alleviates these frustrations by […]
Subqueries are a SQL feature that allow writing queries nested inside a scalar expression in an outer query. Using subqueries is often the most natural way to express a given problem, but their use is discouraged because most databases struggle to execute them efficiently. This post gives a rough map of existing approaches to optimizing […]
Summary Streaming SQL is about taking the same declarative SQL used to write database queries, and instead running it on streams of fast-changing data. This is useful because: Data is often more valuable when you can act on it quickly The existing tools for deriving real-time insights from streams are too complex. The “declarative” nature […]