AI systems require a substantial amount of high-quality data to produce accurate results. The problem is how to produce, publish, and manage that data in a well-governed manner.
Data products are a method for packaging data for enhanced discoverability, management, and governance. As such, they offer several unique advantages when it comes to managing data for AI systems.
In this article, we’ll review what data products are and how they simplify managing high-quality data at scale for AI. We’ll also see how you can use Materialize to simplify developing data products for AI.
What is a data product?
A data product is a data asset that’s developed, packaged, and shipped in a manner analogous to a software release. It combines a polished, high-quality dataset with everything you need to use it, including metadata, business logic, and a semantic layer.
A data product can be any data deliverable, including a table, a set of tables, an API, or a reporting dashboard. These products are developed, not by a centralized data team, but by the team closest to the data.
To qualify as a “data product,” a given data deliverable must adhere to a set of characteristics. These include:
- Discoverable (data consumers can find and use it via self-service methods)
- Addressable (it has a unique, permanent address)
- Understandable (it describes itself with metadata and documentation)
- Trustworthy (communicates its Service Level Objectives and Service Level Indicators)
- Interoperable (can work together with other data products)
How do data products support AI?
Data products have been around for a while. They’re receiving increased attention with the explosion of AI use cases.
Large Language Models (LLMs) work by using probabilistic reasoning based on neural networks to predict the next token in a sequence. These models work better the more high-quality data they have. This is true no matter whether you’re creating your own model, fine-tuning an existing one, or adding domain context using retrieval-augmented generation (RAG).
The data contained within AI systems is often generalized and, typically, outdated by anywhere from a few months to several years. However, an increasing number of use cases—those involving financial data, IoT data, and so on— require data with high data freshness. To deliver reliable results, it’s critical to supply AI systems with operational data - fast, fresh, and correct data that reflects the current state of your business.
At the same time, the rise of AI use cases raises additional concerns about the origins, quality, and overall governance of the underlying data. Defects such as bias and explicit attacks, including data poisoning, can lead to LLMs producing inaccurate or harmful results.
Data products meet these dual demands. Operational data products - data products that deliver data quickly with high consistency - facilitate the rapid delivery, discovery, and use of operational data. Since these operational data products are both discoverable and interoperable, this makes it easy for data consumers to find pre-packaged data and incorporate it into their AI solutions.
The accompanying metadata and documentation also facilitate strong governance, as consumers can easily verify who owns a dataset, its lineage, and the data quality characteristics. Companies can also establish compliance standards for new datasets before approving their publication.
To drive this home, here’s a list of the eight characteristics of a data product and how each one benefits AI:
| Data product characteristic | Benefit for AI solutions |
|---|---|
| Discoverable | Developers can find the data they need for new AI solutions easily without spending days or weeks digging through data silos |
| Addressable | Developers can immediately plug data into an AI solution via its unique address |
| Understandable | Developers can see what a dataset’s fields mean and how they were calculated without outside assistance, speeding time to implementation |
| Trustworthy | Anyone can see what data products go into an AI solution, who owns them, and where the data comes from |
| Natively Accessible | Developers can access the data via SQL, API, BI tools, or whatever access method works best for them |
| Interoperable | Standardized datasets can be implemented easily into AI solutions. Data products from different teams can connect to one another easily. |
| Valuable on its own | AI solution developers can immediately use a dataset without the need to understand |
| Secure | The organization can verify that AI solutions only use data approved for AI use case consumption |
To make a long story short, operational data products make it easier and faster for developers to create new AI solutions by using data products as composable building blocks.
AI data product best practices
How do you create good data products for AI? Here are a few guidelines to follow:
Formalize your data product use cases
Too often in data projects, engineering teams run full steam ahead without adequately understanding the end user’s needs. This results in datasets that go underutilized because they’re hard to use out of the box.
Data products should be standalone datasets that are valuable by themselves. This requires meeting at the start of the process with all relevant data stakeholders - both data producers and consumers - to understand what users need from a given data product.
Decentralize data product management
One challenge with scaling data for AI is that, traditionally, the creation of new datasets has been so complex that it required fielding all new data requests through a central engineering team. Inevitably, that team gets overwhelmed, and work on new data slows to a crawl.
With data products, the team that’s closest to the data for a given problem domain should ideally be the ones who own the associated data product. Organizations can facilitate this by providing self-service tools that help teams spin up the compute, storage, data transformation pipeline infrastructure, and other assets required to create a new data product.
Create data contracts
A data contract is a metadata specification that defines a data product, including its current version, the data it contains, and its service-level agreements (SLAs). Defining data products via a data contract makes it easier to evolve the data product over time without breaking downstream consumers. It gives consumers time to understand and adapt their systems to breaking changes - a removed field, a changed field format, etc. - while keeping their existing solutions operational.
Gather data product metrics
Collecting data product metrics gives you insight into the quality and usage of the product. Metrics can include:
- Uptime vs. downtime
- Number of incidents
- Time to incident resolution
- Usage
- Links to other data products
- Overall quality of the dataset as measured by documentation, statistical analysis, number of data tests, etc.
Creating an operational data architecture for AI
Data products can greatly decrease the time required to bring a new AI solution to market. However, there are a few challenges involved in making operational data products a reality:
Data trustworthiness
As noted above, data for AI solutions increasingly needs to be operational. In other words, it needs to be fast, fresh, and correct. (Think of use cases that, e.g., analyze sensor data from IoT devices installed in equipment on a manufacturing floor.)
Traditional cloud data warehouses typically can’t deliver on all three of these requirements. Operational data products require a streaming architecture that can rapidly ingest and transform data, while also supporting fast and consistent queries. Typically, standing up such architectures requires specialized technical expertise, as well as time and money.
Demands on teams
Many teams are struggling to keep up with the demand for data for AI. From a business standpoint, most are short-staffed and don’t have the resources and skills required to master new and evolving technologies.
This constraint, unfortunately, won’t change any time soon. Teams need technology that helps them fulfill exponentially increasing demands for operational data as headcount grows at a slow, linear pace.
From an architectural perspective, existing line-of-business databases running on MySQL and PostgreSQL are struggling to meet the processing demands required for all of this data.
Materialize for AI data products
Solving these disparate challenges requires an operational data store that can do two things:
- Process complex transformations of operational data without compromising data trustworthiness; and
- Expose datasets as data products to enable rapid AI application development
Materialize is a real-time data integration platform you can use to build operational data products you can trust. Operational data products are operational because they’re fresh, fast, and correct. They’re data products because they’re curated and reusable units of data that are composable into new solutions.
Using Materialize, teams that are closest to their data can create and expose their datasets as operational data products. Because data products are interoperable, they can easily integrate data sets from other teams, creating an operational data mesh.
Materialize doesn’t require any specialized knowledge to use. Using out-of-the-box integrations, developers can sync real-time data sources from upstream OLTP databases, streaming platforms, webhooks, and other systems. They can then easily transform this data using SQL and expose the end result as a standard PostgreSQL-style view.
AI developers can find and access these views using standard SQL. Materialize handles the mechanics of keeping data up-to-date, without requiring developers to learn a new language or technology.
Over time, as the number of operational data products created using Materialize increases, the cost of producing new AI solutions decreases. That reduces the total cost per use case while also accelerating time to market.
Additionally, Materialize is easy to fit into your existing architecture. It resides downstream of your primary data sources, complementing them rather than replacing them. Materialize is serverless and scales automatically to meet demand, making it easy to administer.
Conclusion
Data products can accelerate the time it takes to bring a new AI solution from prototype to production by composing new AI-driven apps from verified, high-quality, and well-governed data sets. Using Materialize, you can create operational data products that are fast, fresh, correct, and composable, enabling you to ship more data in fewer cycles.
To learn more about using Materialize to create operational datasets you can trust, contact us today.
