The Data Product Lifecycle
After introducing the data products last week, I thought we should look at them a little more deeply. It is clear by now that building a data product is not a one-time project. It is an iterative, end-to-end process that treats data with the same long-term commitment we give to consumer software. Unlike traditional datasets that often sit static and forgotten, a data product evolves to meet a business's changing needs.
While different companies might use slightly different models, most successful teams follow a six-stage lifecycle to ensure their data remains high-quality, secure, and valuable.
| Stage | Focus | Key Activities |
|---|---|---|
| 1. Requirements | Problem Definition | Identify stakeholders, define the "why," and set high-level governance. |
| 2. Design | Technical Blueprint | Create data models, schemas, and formal Data Contracts. |
| 3. Development | Implementation | Build MVPs, run production-scale tests, and automate deployment. |
| 4. Operations | Maintenance | Monitor SLAs, track data quality, and manage schema drift. |
| 5. Iteration | Evolution | Collect user feedback and add new features to improve ROI. |
| 6. Deprecation | Decommissioning | Notify users, archive assets, and clean up the data ecosystem. |
Stage 1: Business Requirement Gathering
The biggest mistake teams can make is starting with the data they have instead of the problem they need to solve. Similar to CRISP-DM, this first stage is about asking why the product is required and how it will support a specific decision or outcome.
At this point, you identify your target consumers and how they will access the data: whether through an API, a dashboard, or a SQL query. It is also the best time for an early governance check, where you define the data domain, assign a high-level owner, and determine the privacy classification.
Stage 2: Data Product Design
Once the goal is clear, you move to the technical and governance blueprint. This phase defines the data model, including schemas and semantic definitions that explain what the data actually represents. A critical part of this stage is creating Data Contracts. These are formal agreements between the people producing the data and those using it, setting clear expectations for quality, freshness, and uptime. By defining these rules early, you ensure the product is interoperable and can be easily combined with other assets across the company.
Stage 3: Development and Deployment
Now the product moves from a specification to a reality. At this stage, you build a functional prototype or a Minimum Viable Product (MVP) to gather early user feedback.
You must test the product using real data at a production scale to ensure integrity and usability. Once validated, the product is deployed through automated pipelines and registered in a central catalog. This makes the product discoverable and accessible to anyone authorized to use it.
Stage 4: Monitoring and Operations
A data product is only as good as its last update. This stage focuses on tracking performance metrics, including data freshness, quality, and adherence to agreed-upon Service Level Agreements (SLAs).
Operations teams monitor for any schema drift or unexpected changes in source systems that could break downstream reports. They also manage compliance needs, such as maintaining audit trails and masking sensitive information to keep data secure and trustworthy.
Stage 5: Iteration and Improvement
Business needs are rarely static, so your data products shouldn't be either. At this stage, you use telemetry and direct user feedback to identify user frustrations and ideas for new features.
You might find that a marketing team needs a new field added to a customer profile, or a finance team needs more frequent updates. By continuously iterating, you improve the product’s Return on Investment (ROI) and ensure it remains relevant to the organization's daily workflows.
Stage 6: Deprecation
Every product eventually reaches the end of its useful life. Perhaps a newer, better product has replaced it, or the original business problem no longer exists.
Responsible deprecation is vital to prevent "dark data" from cluttering your systems. This final stage involves notifying all consumers, archiving assets in accordance with legal requirements, and revoking access. By decommissioning stale products properly, you maintain business trust and keep your data ecosystem clean and manageable.
