Materialize vs Databricks for Live Agent Context

Operational databases hold the current state of a business. Databricks's lakehouse, the Delta tables, materialized views, and SQL warehouses that Lakeflow pipelines build and serve, is well suited to historical analysis, large scans, backfills, feature engineering, training data, and other workloads that can tolerate delay.

Agents need something different when they are making operational decisions: current business objects they can retrieve quickly and combine without seeing contradictory versions of the underlying data.

This paper calls those business objects "context products." A context product turns raw operational records into a maintained representation of something an agent needs to understand, such as a customer, account, order, shipment, or approval state. It may depend on one source table or many, across databases and systems. What matters is the business object it presents and the fact that it stays current as source data changes.

Databricks can build and serve context products, and it offers more than one way to do each. A materialized view runs its defining query on a serverless pipeline, on a schedule, on a source-update trigger with a one-minute minimum interval, or inside a continuous pipeline at a trigger interval that defaults to one minute, and writes the result to Delta storage. Agents then read that result through a SQL warehouse, or through a copy that a second pipeline syncs into Lakebase, Databricks's managed Postgres, at a minimum interval of fifteen seconds. Each stage is a separate pipeline with its own schedule, its own compute, and its own view of when "now" is.

Materialize maintains context products on one tier and serves them from another, but the tiers share one logical timeline and the transformation is continuous. It ingests the same operational change streams from PostgreSQL (including Lakebase), MySQL, SQL Server, and Kafka, updates each context product on a transform cluster as changes arrive, and serves indexed reads from dedicated serving clusters, with every read at one consistent logical timestamp. It exposes those context products to agents directly through a built-in MCP server or plain SQL, keeps external search and vector indexes current by pushing precise deltas through its sinks, can push changes to agents, and can write finished context products to Unity Catalog as Iceberg tables so operational and historical workloads share one definition.

A context product is a queryable business object derived from operational data and kept current as that data changes. For example, an account context product might combine the account record with open invoices, recent support activity, and current entitlements. An order context product might combine order state, inventory, payment status, and shipment events. An agent can retrieve one of these objects directly or combine several in a transaction.

A useful context product may come from a single table or from a much larger graph of joins, filters, aggregates, and window functions. The relevant considerations are end-to-end context latency, from a source commit to the moment an agent can read the result; whether related context products agree when the agent reads them; and what resources are required to serve many concurrent agents.

A context layer has two parts. The first is the ontology, or knowledge graph: which objects exist, what they mean, how they join, which one is authoritative. An agent uses this to choose the right tool or write the right query. The second is the objects themselves: the maintained customer, order, or account record the agent actually reads. Databricks and Materialize both provide both parts. They differ in how each part is produced and in how tightly the two are bound together.

Ontology on Databricks. Genie Ontology, in public preview, is "the unified context layer that gives the Genie family of AI experiences a business-aware map of your organization." It combines modeled context (Unity Catalog metric views, domains, and Pages) with inferred context extracted from dashboards, SQL queries, and Genie usage. Databricks's managed MCP servers, also in public preview, expose Genie, Unity Catalog functions, an AI search server, and a Databricks SQL executor to agents. Databricks's documentation recommends the Genie path for analytics: "Genie resolves business terms through Genie Ontology, your governed semantic layer, which produces more accurate answers than an agent writing SQL directly." In both cases the agent's question becomes SQL against tables at query time.

Ontology on Materialize. The ontology pattern in the Materialize documentation is a curated table of join relationships. "Each row describes a single join: the columns in one table that reference columns in another," and an agent queries it through the MCP server's query tool before writing multi-table SQL. Above that sits the semantic-layer layout that the mz-ontology-design agent skill encodes: a shared raw database, a shared core database whose core.public schema is the documented ontology and the only core schema consumers may read, and one database per use case, with core.public.relationships as the relationship registry and COMMENT ON as the documentation contract. Those comments are what the MCP server surfaces to the agent as tool descriptions, and the columns of a view's index are surfaced as its preferred lookup keys.

Objects on Databricks. A materialized view refreshed on a schedule, then, for low-latency reads, copied into Lakebase by a second pipeline on a second schedule.

Objects on Materialize. An incrementally maintained view on a transform cluster, indexed on a serving cluster, updated as source changes arrive.

The practical difference is the freshness of the ontology's underlying context. On Databricks the ontology points at tables that were last refreshed at some earlier moment, and its purpose is to help an agent generate better SQL against them. On Materialize the ontology points directly to live context.

There is no single Databricks component that maintains and serves a context product. The path from an operational commit to an agent's read is a chain of stages, each configured, scheduled, and billed on its own.

Lakeflow Connect provides managed CDC connectors for PostgreSQL, MySQL, SQL Server, and Oracle. The connector has two halves. An ingestion gateway "runs on classic compute, and it runs continuously to capture changes before change logs can be truncated in the source." An ingestion pipeline runs on serverless compute and applies the staged changes to Delta tables on its own schedule. Because the gateway never stops, "you provision and pay for its classic compute even when your ingestion pipeline is idle." A pipeline handles at most 250 tables.

How often the pipeline applies changes is the customer's choice, within limits. "There is no limit on how often you can schedule the ingestion pipeline to run. However, Databricks recommends at least 5 minutes between intervals because it takes some time for the serverless compute to start up." For PostgreSQL and SQL Server, "Databricks does not support running the ingestion pipeline in continuous mode"; only the gateway runs continuously. A newer integrated CDC pipeline, in Beta for SQL Server, MySQL, and Oracle and not available for PostgreSQL, can run continuously on always-on compute and applies changes "typically within a few minutes" in its speed-optimized mode, which is limited to 50 tables.

The context product itself is a materialized view. "Materialized views automatically create and use serverless pipelines to process refresh operations"; a SQL warehouse only coordinates. There are two ways to drive the refresh.

The first is a schedule or trigger on the view. SCHEDULE EVERY accepts only hours, days, and weeks. SCHEDULE CRON accepts a six-field expression with a seconds field. TRIGGER ON UPDATE "automatically refreshes a pipeline when upstream source data changes," throttled with AT MOST EVERY, and "the minimum trigger interval is 1 minute." Each firing is a serverless pipeline update that pays its own startup and planning overhead; "serverless workloads using standard performance mode typically start within four to six minutes after being triggered," and performance-optimized mode starts faster at a higher DBU rate.

The second is to define the view inside a continuous Lakeflow pipeline. "Continuous pipelines require an always-running cluster, which is more expensive but reduces processing latency." The pipelines.trigger.interval property controls how often a table is refreshed; its default is "five seconds for streaming queries" and "one minute for complete queries when all input data is from Delta sources." A materialized view is a complete query. The documentation positions continuous mode for "data updates desired between every 10 seconds and a few minutes."

Whether a refresh is incremental depends on the query and the platform. "Incremental refresh is only available when the compute you are connected to is serverless." Source tables need row tracking enabled, and change data feed is recommended. Joins, filters, GROUP BY, window functions with PARTITION BY, and CTEs can be incrementalized. Recursive CTEs, most non-deterministic functions, and SUM or AVG over floating-point columns fall back to a full recompute; Databricks recommends casting to DECIMAL. A cost model chooses between incremental and full refresh on each run unless a REFRESH POLICY overrides it.

Once the view exists, the customer chooses how agents read it.

A serverless SQL warehouse reads the view directly. Databricks suggests "a cluster for every 10 concurrent queries," with intelligent workload management adding clusters as queues form and a queue limit of 1,000. The warehouse auto-stops after ten minutes by default; the floor is five minutes in the UI and one minute through the API. This path has the fewest hops, but it puts a query planner and an analytical execution engine between the agent and a point lookup.

A Lakebase synced table copies the view into Postgres. Lakebase is "a fully managed Postgres database integrated into the Databricks platform," positioned "as an online feature store for ML models, or as a state store for agents." Synced tables "use managed Lakeflow pipelines to continuously update both the Unity Catalog synced table and the Postgres table with changes from the source table." The sync runs in one of three modes:

Triggered and continuous modes require change data feed on the source. The sync writes at "approximately 150 rows per second per Capacity Unit (CU)" in continuous and triggered modes. A source table can feed up to 20 synced tables, and "each synced table uses up to 16 connections to your Lakebase database." The synced table needs a primary key; rows with null key columns are excluded, and duplicate keys fail the pipeline unless deduplication is configured. Only additive schema changes propagate. And "Databricks strictly recommends running only read queries" against a synced table.

In Materialize, the context product is an incrementally updated materialized view. The recommended production layout uses three tiers of clusters. Source clusters ingest operational change streams from PostgreSQL, MySQL, SQL Server, and Kafka. Transform clusters maintain the materialized views and persist their results to durable storage. Serving clusters hold indexes on those views and answer reads from memory. A single cluster can do all three, but production deployments generally separate serving so that a heavy query or a rehydrating view cannot affect agent reads, and so that each tier scales independently. Unlike the Databricks chain, the tiers share one logical timeline: a read on the serving cluster sees exactly the state the transform cluster has produced, and the transformation runs continuously rather than on a refresh schedule.

Transform clusters are sized for the incoming change rate and the amount of maintained state. Serving clusters are sized for the indexes they hold and the read concurrency they absorb. Indexed reads do not add a per-query compute charge, so increasing the number of readers does not change the bill until the serving cluster needs to grow. In customer deployments, indexed serving clusters have handled thousands of reads per second while the underlying data changes, with tail latencies under 20 milliseconds on index reads.

Agents reach these context products without a custom API layer. Starting in v26.24, Materialize includes a built-in materialize-agent MCP server, in public preview, that runs inside the database. Its get_data_products tool lists materialized views and indexed views; get_data_product_details returns each one's JSON schema, column descriptions, and comments; read_data_product fetches rows; and an optional query tool runs arbitrary SELECT statements against objects the agent role can see. "Comments are surfaced to the agent to help it better understand when and how to use the data products," and when a view is indexed, "the indexed columns are surfaced to the agent as preferred lookup keys, enabling index point-lookups instead of index scans." Materialize recommends a dedicated cluster and schema for agent traffic, a functional role such as mcp_agent with USAGE on only that cluster and schema, and SELECT on only the curated data products, so agent reads are isolated from other workloads and from operational metadata.

The metadata an agent needs to compose products is part of the same layer. The ontology table documents join relationships, and the mz-ontology-design skill (installed with npx skills add MaterializeInc/agent-skills) gives a coding agent the rules for the raw, core, and use-case layering, the admission test for a public semantic object, the grain of entities, events, measurements, and relationship objects, and the COMMENT ON contract that becomes the MCP tool description.

The table compares the serving tier each system would use for agent reads: a Lakebase synced table on Databricks, since it is the path Databricks documents for low-latency application reads, and an indexed serving cluster on Materialize.

Where the customer skips Lakebase and reads the materialized view from a serverless SQL warehouse, the freshness row improves by the sync interval and the correctness row gets worse: each table in a query is read at its own latest Delta version, with no transaction around them, and the read itself goes through query planning and an analytical executor sized in units of roughly ten concurrent queries per cluster.

End-to-end context latency starts when the operational database commits a change. Both systems must ingest that change, update the relevant context product, and serve the result. The difference is the number of stages, the floor at each one, and what happens when a stage cannot meet its target.

In Databricks's best case, with the gateway running, the ingestion pipeline on its shortest practical schedule, the refresh pipeline in continuous mode at a short trigger interval, and the sync in continuous mode, end-to-end latency is several minutes, most of it in the ingestion stage, and every component is billing around the clock. A production configuration where each stage is tuned for cost lands at five to fifteen minutes. In Materialize, end-to-end context latency is about a second at the default one-second source timestamp interval, and that interval can be lowered.

Databricks and Materialize take different approaches to consistency, and the choice determines whether the context an agent assembles describes a single state of the business.

Delta Lake provides snapshot isolation for a single table: "Readers continue to see a consistent snapshot view of the table that the Databricks job started with, even when a table is modified during a job." The unit of that guarantee is the table. A query that joins two materialized views reads each at its own latest committed version, and if their refreshes finished at different moments, the query joins two different points in time. Within a single Lakeflow pipeline update, the tables in that pipeline are computed from inputs read at the start of the update, which aligns them with each other; the guarantee ends when the update does, and it never covered a view maintained by a different pipeline or a different team.

Multi-statement transactions extend the guarantee to several statements but keep the per-table unit: "When you access a table in a transaction, Databricks captures a consistent snapshot of that table at first access." Two tables first touched at different points in the transaction are snapshotted at different moments. Transactions are limited to Unity Catalog managed tables with catalog commits, which became generally available in May 2026, up to 100 tables, on Databricks Runtime 18.0 and above or a SQL warehouse; transactions on managed Iceberg tables are in private preview.

In Lakebase, a Postgres transaction is consistent over the synced tables it reads. But each synced table is maintained by its own pipeline, and nothing coordinates two of them. Two context products can be at different lakehouse versions, and the lakehouse versions themselves came from refreshes that finished at different times. The agent gets a consistent read of copies whose relative age is unknown.

Materialize assigns logical time as changes enter the system and preserves that time through computation. A query that reads several context products sees all of them at one timestamp. This remains true across isolation levels; the isolation choice changes the tradeoff between recency and latency, not whether the query is internally consistent.

Agents write. They open cases, reserve inventory, adjust balances, and then read context to decide what to do next. Where those writes land, and how quickly they come back around as context, shapes what kinds of agents a platform can support.

On Databricks, the write target for agent state is Lakebase. Databricks positions it "as a state store for agents," and it is a full Postgres, so the write itself is immediate. Getting that write back into a context product is not trivial. Lakebase Change Data Feed, in public preview, captures every insert, update, and delete from the write-ahead log into a Unity Catalog Delta history table, "batched and flushed every ~15 seconds"; the materialized view that depends on it refreshes at its trigger interval, no sooner than a minute; the synced table that serves it copies the new version back to Lakebase at the sync interval, no sooner than fifteen seconds. An agent that writes to Lakebase and then reads a context product from Lakebase is reading a copy that has not yet heard about its own write, for two minutes or more.

Lakebase is also closed to outside readers at the replication level. Its compatibility page states that "replicating data to or from a Lakebase database using native Postgres logical replication is not yet available," so the change data feed into Delta is the only way out.

Materialize does not take agent writes; it reads them from wherever they land. Native sources exist for PostgreSQL, MySQL, SQL Server, and Kafka, so an agent writing to any of those operational stores sees the change in its context products within about a second, in the same logical timeline as every other product, and under strict serializable isolation with real-time recency the next read is guaranteed to include it. The practical rule is to keep agent state in a store the context layer can read. Where that store is Postgres, MySQL, or SQL Server, Materialize reads it directly.

Some agent architectures avoid polling to minimize reaction time and save tokens spent asking the same question multiple times. A change in the business, such as a payment failing or an order shipping, should wake the agent, and the agent should then read the rest of its context as of the same moment. That asks two things of the context layer: a way to deliver changes as they happen, and a way to tell the consumer when what it has received is complete.

Databricks has no primitive by which a client subscribes to a changing query result. The available paths are a Structured Streaming job reading a Delta table's change data feed, which delivers insert, update_preimage, update_postimage, and delete rows per commit but runs as a Spark job the customer keeps alive and cannot span a non-additive schema change; a Lakeflow job with a table-update trigger, which "checks for table updates, and when a table is updated, the job is run," watching up to ten tables, with settings for a minimum time between triggers and a wait after the last change but no documented detection latency; or a SQL alert that runs a query on a warehouse on a schedule set in minutes, hours, days, or weeks. Real-Time Mode, the lowest-latency streaming path Databricks offers, cannot read from or write to Delta, so it cannot emit changes from a context product. End to end, a push consumer learns of an operational change after ingestion, the refresh interval, and whichever trigger or poll interval it uses.

Materialize delivers changes with SUBSCRIBE, which streams inserts and deletes to a view, materialized view, source, or ad hoc query over the PostgreSQL wire protocol. Each update carries a logical timestamp, and progress messages report when a timestamp is complete, so a consumer can act on a consistent frontier and distinguish a quiet period from a stall. For many consumers, a Kafka sink emits the same changes with an upsert or Debezium envelope and exactly-once delivery by default.

• Agents often write to an operational system and immediately read context to decide what to do next. A minute of lag can break that feedback loop and make it impossible to build interactive agents with humans in the loop.
• Stale data can cause an incorrect action, not merely an outdated display. A delayed dashboard is inconvenient; a refund approved against an old account state can be wrong.
• Agents combine several business objects at the moment of decision. Those objects need to describe the same point in time.
• An explicit freshness failure lets the application pause, retry, or escalate. A successful query that silently returns old data does not.
• Concurrency comes from many agents issuing small reads, rather than a smaller number of analysts running exploratory queries.
• Prepared context products reduce tokens as well as database work. They replace wide raw payloads, repeated text-to-SQL attempts, and polling with a direct call that returns only the context the agent needs.

A practical routing rule is to use Materialize for named, maintained context products and Databricks for novel analytical questions. If a context product already answers the question, the agent should call it instead of generating SQL against raw tables.

For Materialize reads that combine multiple products, define a composite context product or issue indexed reads inside one read-only transaction. The first statement in a read-only transaction fixes which schemas later statements can reference, so place products that agents read together in the same schema. An unindexed ad hoc join creates a one-time dataflow and has materially lower throughput.

Materialize and Databricks can consume the same operational databases and Kafka change streams. Materialize maintains the current business objects used by agents and operational applications. Databricks retains history and serves analytical and training workloads. Rather than create two parallel pipelines, Materialize can deliver finished context products to Databricks. There are two paths:

• A native Iceberg sink writes exactly-once updates to Iceberg tables registered in Databricks Unity Catalog on AWS, or in Amazon S3 Tables. Each change carries _mz_timestamp and _mz_diff, so Databricks can reconstruct current state or keep the full change history.
• A bulk export to Amazon S3 with COPY TO can be loaded on a schedule.

Either allows batch scoring, feature engineering, and historical analysis to use the same definitions as the live agent path.

The architecture runs the other way too. Where agents keep their state in a Postgres (including Lakebase), MySQL, or SQL Server database, Materialize ingests it directly, so the agent's own writes join its context products within seconds. Databricks keeps the history; Materialize keeps the live objects; neither definition is written twice. Lakebase can hold agent state in this design, with the caveat that until it exposes logical replication, state written there reaches context only through the Databricks chain.

This design avoids maintaining separate transformation logic for live and historical systems. It also removes the sync pipeline between the lakehouse and a serving copy, because the serving tier reads the maintained object directly. Both systems remain active, but they are active for different reasons: Materialize is sized for the rate of change and maintained state, while Databricks runs the pipelines and warehouses required to refresh and serve its copies.

Databricks and Materialize charge for different parts of a context pipeline.

For a context layer that must stay within tens of seconds of the source on Databricks, the following components are running and billing continuously:

• The Lakeflow Connect ingestion gateway, on classic compute, which runs whether or not the ingestion pipeline is.
• The ingestion pipeline, on serverless, at whatever interval keeps bronze current.
• The refresh pipeline for the materialized views, in continuous mode with an always-running cluster, or firing on a per-minute cron with per-run overhead.
• The synced-table pipeline into Lakebase, in continuous mode.
• Lakebase compute, billed in capacity-unit hours under the Database Serverless Compute SKU, sized first for sync write throughput at roughly 150 rows per second per CU and then for reads, with read replicas as needed. An instance whose scale-to-zero has been off for more than 24 hours is billed as Always-on at a lower baseline rate; one serving agents around the clock will be. Storage bills separately in Databricks Storage Units at 15 DSUs per GB-month.
• Or, in place of the last two, a serverless SQL warehouse that agent traffic keeps from auto-stopping, adding clusters in steps of roughly ten concurrent queries.

Materialize's cost changes when source change volume or maintained state requires a larger transform cluster, or when read concurrency requires a larger serving cluster, not per query and not per stage. There is no sync pipeline because there is no serving copy, and there is no per-refresh overhead because there is no refresh.

For the same interactive workload Materialize is typically less expensive, but the architectural differences in freshness and consistency matter more.

Databricks can maintain and serve agent context, particularly when a team values platform consolidation, already keeps its history in the lakehouse, and can tolerate a minute or more between an operational change and the moment an agent can read its consequences. Lakebase is the right Databricks serving option for short, concurrent reads, and it is a capable state store for the agent's own writes.

Materialize is the stronger fit when an agent needs context that reflects operational changes within seconds, when several business objects must agree at the moment they are read, when the application must reject data that is too stale to support an action, or when search and vector indexes must track the business as it changes. A combined architecture lets Materialize serve that live context, reading from the same operational stores the agents write to, while Databricks remains the system for history, training, exploration, and large-scale analytical work.