Materialize Documentation
Join the Community github/materialize

Kafka Sink Topic Reuse

BETA! This feature is in beta. It may have performance or stability issues and is not subject to our backwards compatibility guarantee. Available since v0.9.0.

By default, Materialize creates new, distinct topics for sinks after each restart. To enable the reuse of the existing topic instead and provide exactly-once processing guarantees, Materialize must be able to do two things:

This allows for exactly-once stream processing, meaning that each incoming event affects the final results only once, even if the stream is disrupted or Materialize is restarted.

Exactly-once stream processing is currently available only for Kafka sources and the views based on them.

When you create a sink, you must:

Additionally, the sink consistency topic cannot be written to by any other process, including another Materialize instance or another sink.

Because this feature is still in beta, we strongly suggest that you start with test data, rather than with production. Please escalate any issues to us.

Example

CREATE SINK quotes_sink
FROM quotes
INTO KAFKA BROKER 'localhost:9092' TOPIC 'quotes-eo-sink'
WITH (reuse_topic=true, consistency_topic='quotes-eo-sink-consistency')
FORMAT AVRO USING
  CONFLUENT SCHEMA REGISTRY 'http://localhost:8081';
Did this info help?
Yes No