Introduction We’ve recently partnered with dbt and community members to release a beta Materialize dbt adapter that allows you to transform your streaming data in real time using Materialize as your data warehouse. This tutorial serves as a practical hands-on demonstration of the adapter. In this case, we are using static not streaming data, but […]
Materialize 0.7 was released on 08 February 2021 with significant improvements around getting data into Materialize. Key change: Source data from Amazon Web Services S3 S3 sources for Materialize are fully tested but under the experimental flag until 0.8. With S3 sources, you can: Point Materialize at S3 buckets using the same CREATE SOURCE syntax […]
Managing data is hard. Managing data pipelines is even harder. The meaning of individual tables or values in your data warehouse gets lost in translation across organizations. Another team’s refactor breaks your team’s pipeline. And, it’s normally very difficult to tell who made what change and when. dbt, data build tool, alleviates these frustrations by […]
Subqueries are a SQL feature that allow writing queries nested inside a scalar expression in an outer query. Using subqueries is often the most natural way to express a given problem, but their use is discouraged because most databases struggle to execute them efficiently. This post gives a rough map of existing approaches to optimizing […]
Summary Streaming SQL is about taking the same declarative SQL used to write database queries, and instead running it on streams of fast-changing data. This is useful because: Data is often more valuable when you can act on it quickly The existing tools for deriving real-time insights from streams are too complex. The “declarative” nature […]
Materialize provides a SQL interface to work with continually changing data. You write SQL queries as if against static data, and then as your data change we keep the results of your queries automatically up to date, in milliseconds. Materialize leans hard into the ideal that SQL is what you know best, and what you […]
Within the web development community, there has been a clear shift towards frameworks that implement incremental view maintenance and for good reason. When state is updated incrementally, applications perform better and require fewer resources. Using Materialize, developers and data analysts can adopt the same, event driven techniques in their data processing pipelines, leveraging existing SQL […]
Materialize computes and maintains SQL queries as your underlying data change. This makes it especially well-suited to tracking the current state of various SQL queries and aggregates! But, what if you want to root around in the past? Maybe you want to compare today’s numbers to yesterday‘s numbers. Maybe you want to scrub through the […]
I’ve been working at Materialize for almost a year now, and I have really enjoyed learning about and using Differential Dataflow (hereafter just Differential) in my day-to-day work. In this post, I’ll introduce Differential and talk through implementing a few common programming problems like list intersection and everyone’s favorite, FizzBuzz, as dataflow programs. Finally, I’ll […]
Materialize 0.6 makes it easier to consume streams and build streaming applications. We’ve also made a number of changes that improve our SQL compatibility. Here’s more details on some noteworthy features we’ve added in this release: What’s changed in Materialize 0.6 Easily listen to streaming changes TAIL is a Materialize-specific command we recently introduced to […]