Webhook source: support for batched events

Many SaaS applications support webhooks as a way to push events to other applications (like Materialize) as they occur. To maximize the volume of events that can be sent at once, it’s common for these applications to batch multiple events into a single network request — and we strive to support what’s common!

The webhook source can now automatically unwrangle batched events in the following formats:

JSON arrays

If you’re expecting events batched as a JSON array, use BODY FORMAT JSON ARRAY. Each object will be appended to the source as an individual row.

CREATE SOURCE webhook_json_array IN CLUSTER quickstart FROM WEBHOOK
BODY FORMAT JSON ARRAY;

-- Send multiple events in a JSON array to the webhook source.
HTTP POST TO 'webhook_json_array'
  [
    { 'event_type': 'a' },
    { 'event_type': 'b' },
    { 'event_type': 'c' }
  ]

-- 🎉
SELECT COUNT(*) FROM webhook_json_array;
3

Newline-delimited JSON (aka NDJSON)

If you’re expecting events batched as NDJSON, use BODY FORMAT JSON as you would otherwise! Each object will be appended to the source as an individual row.

CREATE SOURCE webhook_json IN CLUSTER quickstart FROM WEBHOOK
BODY FORMAT JSON;

-- Send multiple events delimited by newlines to the webhook source.
HTTP POST to 'webhook_json'
  { 'event_type': 'foo' }
  { 'event_type': 'bar' }

-- 🎉
SELECT COUNT(*) FROM webhook_json;
2

Besides making webhook ingestion more ergonomic, this change also allows you to take advantage of performance optimizations like temporal filter pushdown.