At our May Onsite the Materialize R&D team held a hackathon. Folks submitted briefs, and then voted with their feet to form teams to work on whatever struck their fancy. As it turns out eight things struck our collective fancy, and over about a day and a half the teams made remarkable progress in building up demos of their ideas!
I’m here to tell you about them.
Caveat: These are not features live in MZ, and some of them may never be. But they were interesting to us, and they may be interesting to you as well. Like us, you can vote with your feet, and reach out if any of them are as fascinating to you as they were to us!
The Hackathon
We asked folks to prepare briefs, just a few sentences, proposing a thing to dive into. We ended up with 20-30 of these, but folks settled on eight that had a team behind each of them. We worked for a day and a half, and then presented what we did back at the larger team.
We’re just going to go straight through the eight projects and unpack each, more briefly than they fairly deserve.
MZ Tutorial Game: Making learning about Materialize fun!
Learning to pilot MZ can be tricky without a great motivating example of what you would do and why. The team has experience with games, and thought that an interactive puzzle against continually changing data might be a fun motivator. Your eyeballs aren’t fast enough to catch the data in the act, but you can use SQL to cast your net and Materialize to continually and consistently check your catch.
Console Echo Mode: Reveal the SQL we use in our own Console.
Materialize’s web console runs SQL against your Materialize instance to power its dashboards. These aren’t secret SQL queries, and they are the sort of thing you might benefit from knowing about, to mirror in your own logic. The team added UI mechanisms to reveal the SQL backing any of the console’s visual elements.
MWASMZ: Web Assembly UDFs in Materialize.
It can be hard to get custom code into a database, and it’s equally hard waiting for the database to implement all the logic that you need.
Web assembly is a great way to provide logic that can be sandboxed (run safely), and a least-common-denominator Materialize could support.
The team added a wasm(data, prog) function to Materialize, and an optimization when prog is static to build and repeatedly re-use a web assembly runtime.
An example using recursion (Collatz conjecture) sped up by 50x using WASM to replace SQL.
Upsert in S3: Back our most expensive source with object storage.
Materialize provides an “upsert” source for events with keys, where you would like the most recent value for each key. This source is expensive, because we have to maintain all keys and values in order to produce the correct retractions when one updates. The team prototyped a cloud storage backend for this source, accessing the data remotely rather than maintaining it in memory.
Formal methods: Verify that Materialize is utterly devoid of errors.
Computer programs do things, usually what you ask them, but are these the things you intend? “Formal methods” is the name for tools and techniques that specify and validate these programs. The team took the Kani Rust verifier for a spin, hoping to verify “uniqueness” and “monotonicity” for our SQL functions. Instead, they found several corner cases (floating point arithmetic) in Materialize, and some issues in Kani itself (issues filed, bugs fixed, world improved).
Endpoints from Views: Build Rest APIs directly from views.
Materialize makes it easy to write SQL to back your live data products.
At the moment, it’s still up to you to figure out how to take the data and serve it up.
The team built a CREATE API command that would automatically set up authenticated Rest endpoints, and is also able to build Prometheus sinks for you.
EXPLAIN ANALYZE: Explain MZ performance as it happens.
Materialize does a LOT of work for you behind the scenes, and it can be a bit of a mystery what is going on, and why.
The compute plane has full insight into what is happening, but it surfaces it using raw data that doesn’t mean much to users.
The team stitched together the higher-level concepts the user speaks (joins, aggregates) with the lower-level implemenation details.
The EXPLAIN ANALYZE command subsequently shipped, and you can now get live data about your ongoing view maintenance dataflows.
S3 consensus: Use S3 preconditions as our consensus mechanism.
Materialize relies on a consensus layer, which for many users is PostgreSQL. This ends up being a performance bottleneck, as well as an additional system to operate for our self-managed users, even though the operations our consensus layer requires are much simpler than arbitrary SQL transactions. The team explored using S3 preconditions for PutObject to build a consensus layer directly against S3. They built a prototype, showed it off, and even exercised it with a bit of Jepsen testing.
Hackathon v2, and next steps
Many of the projects have working prototypes, some have shipped, but all of them have interesting next steps. The hackathon was, to my eyes, an energizing shot in the arm that really called out just how many interesting things there are still to do with Materialize. I’m really excited to do it again, but also eager to make sure we turn the best ideas into great Product features at the same time.
While I would normally invite you to grab Materialize and try all of these out, that’s not how it works in this case. Some of these features are live, but if any of them resonate with you the right thing to do is reach out and say so! We’ll be hard at work evolving them, and the next wave of new ideas, but user signals of excitement are the clearest clarion call.
