Today’s post is on a topic that a lot of folks have asked for, once they dive a bit into Materialize. One of our join implementation strategies uses a surprisingly small amount of additional memory: none. “None” is a surprising amount of memory because streaming joins normally need to maintain their inputs indexed in memory. […]
Differential dataflows contain many operators, some of which are very complicated, but many of which are relatively simple. The map operator applies a transformation to each record. The filter operator applies a predicate to each record, and drops records that do not pass it. The flat_map operator applies a function to each record that can […]
Materialize provides a SQL interface to work with continually changing data. You write SQL queries as if against static data, and then as your data change we keep the results of your queries automatically up to date, in milliseconds. Materialize leans hard into the ideal that SQL is what you know best, and what you […]
Materialize computes and maintains SQL queries as your underlying data change. This makes it especially well-suited to tracking the current state of various SQL queries and aggregates! But, what if you want to root around in the past? Maybe you want to compare today’s numbers to yesterday‘s numbers. Maybe you want to scrub through the […]
This post is also available at my personal blog. Materialize allows you to maintain declarative, relational SQL queries over continually changing data. One of the most powerful features of SQL queries are joins: the ability to correlate records from multiple collections of data. Joins also happen to be one of the harder things to do […]
Today we will take a bit of a tour of the moving parts that make up Materialize. This tour isn’t meant to be exhaustive, but rather to show off some of the moments where things might be different from what you expect, and to give you a sense for why Materialize is relatively better at […]
In today’s post we are going to show off Materialize’s LATERAL join (courtesy @benesch), and how you can use it to implement some pretty neat query patterns in an incremental view maintenance engine! In particular, in the streaming SQL setting, lateral joins automatically turn your SQL prepared statement queries into what is essentially a streaming, […]
At Materialize we traffic in computation over data that change. As a consequence, it is important to have a way to write down and read back changes to data. An unambiguous, robust, and performant way to write down and read back changes to data. What makes this challenging? Why not just write out a log […]
Materialize is an incremental view maintenance engine, one which takes your SQL queries expressed as views and continually maintains them as your data change. Surely there are a lot of ways one could do this, ranging from the very naïve (just recompute from scratch) to the more sophisticated end of the spectrum (what we do). […]
Streaming systems consume inputs and produce outputs asyncronously: the output of a system at any moment may not reflect all of the inputs seen so far. These systems provide various guarantees about how their outputs relate to their input. Among the weaker (but not unpopular) guarantees is eventual consistency. Informally, eventual consistency means if the […]