Kafka

Materialize provides native connector for Kafka message broker. To ingest data from Kafka, you need to

  1. Create a connection that specifies access and authentication parameters.
  2. Create a source that specifies the format of the data you want to ingest.

Supported versions

The Kafka source supports Kafka 3.2+ and is compatible with most common Kafka hosted services, including all supported versions of the Confluent Platform.

Formats

Materialize can decode incoming bytes of data from several formats:

  • Avro
  • Protobuf
  • CSV
  • Plain text
  • Raw bytes
  • JSON

Envelopes

What Materialize actually does with the data it receives depends on the “envelope” your data provides:

Envelope Action
Append-only Inserts all received data; does not support updates or deletes.
Debezium Treats data as wrapped in a “diff envelope” that indicates whether the record is an insertion, deletion, or update. The Debezium envelope is only supported by sources published to Kafka by [Debezium].

For more information, see CREATE SOURCE: Kafka - Using Debezium.
Upsert Treats data as having a key and a value. New records with non-null value that have the same key as a preexisting record in the dataflow will replace the preexisting record. New records with null value that have the same key as preexisting record will cause the preexisting record to be deleted.

For more information, see CREATE SOURCE: Kafka - Handling upserts.

Integration guides

See also

Back to top ↑