Kafka
Materialize provides native connector for Kafka message broker. To ingest data from Kafka, you need to
- Create a connection that specifies access and authentication parameters.
- Create a source that specifies the format of the data you want to ingest.
Supported versions
The Kafka source supports Kafka 3.2+ and is compatible with most common Kafka hosted services, including all supported versions of the Confluent Platform.
Formats
Materialize can decode incoming bytes of data from several formats:
- Avro
- Protobuf
- CSV
- Plain text
- Raw bytes
- JSON
Envelopes
What Materialize actually does with the data it receives depends on the “envelope” your data provides:
| Envelope | Action |
|---|---|
| Append-only | Inserts all received data; does not support updates or deletes. |
| Debezium | Treats data as wrapped in a “diff envelope” that indicates whether the record is an insertion, deletion, or update. The Debezium envelope is only supported by sources published to Kafka by [Debezium]. For more information, see CREATE SOURCE: Kafka - Using Debezium. |
| Upsert | Treats data as having a key and a value. New records with non-null value that have the same key as a preexisting record in the dataflow will replace the preexisting record. New records with null value that have the same key as preexisting record will cause the preexisting record to be deleted. For more information, see CREATE SOURCE: Kafka - Handling upserts. |