Data mesh is an approach to data management that helps organizations become more agile and prudent with managing, analyzing, and distributing data.

Four main principles are fundamental in a data mesh. The first principle is domain ownership, and the second is data as a product, both of which were covered in previous blog posts.  

This post will cover the third principle, the self-serve data platform. This platform is the organization’s data conduit, streamlining the creation, deployment, and maintenance by domain owners and allowing for the data product’s accessibility and shareability.

Organizations benefit by reducing product development and ownership costs and empowering the domain teams to optimize the utility of data products. Likewise, the platform ensures consistency of data product attributes like quality and compliance through automated governance policies.

 

What is a Self-Serve Data Platform?

A self-serve platform is a set of technologies and the underlying infrastructure that simplify otherwise complex decentralized data management and sharing. This platform allows organizations to attain key objectives like:

Autonomous team productivity: The ability to enable a team to complete their work with a sense of autonomy and without involving another team.

Exchange of trusted & governed data products: Smooth exchange of data products with a certified level of data quality from the provider to the consumer. The platform allows for consistent quality, security, compliance, and process through governance policy structure and enforcement.

Accelerate time to value: The platform should abstract technical complexity and provide elegant APIs to decrease the knowledge ramp-up required for domain teams and the steps required to deploy data products.

Scalable and flexible sharing: Data sharing is enabled across internal and external consumers, such as a network of partners. To reach this breadth of sharing, designing for secure interoperability and integration with other platforms is required.

Innovation Culture: Free domain teams from data management activities not contributing to innovation, so they can focus on data discovery, exploration, and analysis to uncover valuable insights to be shared.

 

What Problems Does a Self-Serve Platform Address?

Data mesh architecture allows organizations to develop valuable and useful data products. Knowledge experts create these products within business domains across the organization.

However, the actual value and utility of the data products can only be realized through cost-effective and efficient shareability and discoverability across the organization. 

For this reason, a self-serve platform is key to unlocking the actual value of the data mesh. Below are three specific problems the platform addresses. 

Shareability and discoverability: The platform allows domain teams to seamlessly share their data products across the organization. Likewise, knowledge workers can easily access and find useful data products on the platform.

Duplication of efforts from domains: Domain teams are empowered to share and use data products on the platform, which reduces duplication, specifically across cross-functional domain teams.

Costs of operation: The self-serve platform integrates and shares standard capabilities with existing data platforms. Additionally, it reduces cognitive load across domains and allows for the functioning of a general technologist throughout the system.

Product inconsistency and incompatibility: The platform provides domain agnostic infrastructure and services, which includes the execution of policies to ensure consistency and compatibility of each data product.

 

Self-Serve Data Platform Role in Data Mesh

Within a data mesh architecture, the self-serve data platform is designed to optimize and integrate with existing technologies within a data architecture. Integration examples include data storage, processing frameworks, query languages, data catalogs, and pipeline workflow management.

Likewise, in the book Data Mesh: Delivering Data-Driven Value at Scale, author Zhamak Dehghani outlines six platform characteristics. These self-serve platform characteristics optimize the existing platforms and allow for the functionality of the data mesh architecture.

Serves autonomous domain-oriented teams: Enables domain engineering teams in building, sharing, and using data products.

Manages autonomous interoperable data products: The platform works with data products to make them discoverable, usable, trustworthy, and secure for end users throughout the product life cycle.

Integrated platform of operational and analytical capabilities: The platform provides a connected experience for domains, bringing together the operational and analytical constituents of the data architecture.

Designed for a generalist majority: The platform promotes interoperability between different technologies and reduces the need for proprietary languages and experiences designed for specialists. It incentivizes and enables experienced generalist developers. 

Favor decentralized technologies: The platform supports and enables the notion of the decentralized data architecture of a data mesh. Additionally, they provide centralization of tasks that help remove friction from domains and technologies.

Domain agnostic: The platform is designed to enable all domain teams by balancing domain-agnostic capabilities with domain-specific data modeling, processing, and sharing across the
organization.

 

The Challenges of a Self-Serve Data Platform

Below is the primary challenge of the self-serve platform and the mesh principle that addresses the challenge.

Consistent product quality and policy adherence: As with the domain ownership and data as a product principles, the self-serve platform needs to support the optimization of consistent product quality, security, privacy, and legal compliance. This challenge is addressed through the fourth mesh principle of federated computational governance, which provides an operating model to balance domain autonomy with organization-wide interoperability of a data mesh.

 

What’s Next After the Self-Serve Platform?

The self-serve data platform allows for the shareability and discoverability of data products created by business domains. 

The data platform brings the data mesh architecture together through the cost reductions of decentralized data ownership, lessens the complexities of data infrastructures, diminishes the need for technology specialists, and automates quality and data policies through governance.

The fourth principle is the principle of federated governance which ensures system engagement, mitigates domain isolation, administers product standardization, and addresses platform operational needs. We will cover this principle in our next blog post.

Data mesh is a new approach to organizational data management that helps data-driven organizations become more agile and prudent with managing, analyzing, and distributing data.

Four main principles are fundamental in a data mesh system. The first principle is domain ownership which we discussed in our previous blog post.  

The second principle is data as a product of which domain data is developed and shared as a product for internal and external customers.

The following post will cover the data as a product principle and the benefits it presents to an organization.

 

What is Data as a Product?

Data as a product is the concept of leveraging analytical domain sourced data to develop a high utility data product to be shared and used across the organization.

In a decentralized data mesh architecture, domain teams ingest, process, and distribute their sourced data as usable data products. Developing data products at the domain source reduces friction and high costs typical to traditional data management methodologies.

Data as a product is an evolution in thinking in which external product management techniques are applied internally. Internal users of the data product are the customers, and the domain teams are responsible for the product and prioritizing the customer experience.

 

What Problems Does Data as a Product Address?

The value of data within a data-driven organization is uncovered through its usability. For data to become usable, it needs to be converted into a product and then shared within the organization.

Likewise, evolving the thinking around data from an asset to a product promotes an organization where data becomes part of the culture when planning, making decisions and implementing strategies.

As such, data as a product addresses the following data problems within an organization.

Data siloing: Data as a product prevents domains from becoming a funnel for data collection, and instead, domains become a data product team creating products to be shared.

Value: Data as a product increases the value of data by increasing the utility and shareability of the data.

Utility: Data as a product extracts more utility from data as teams innovate around data products, increasing the quality and trustfulness of the data.

 

Data as a Product Role in Data Mesh

Data as a product can be defined through its customer experience. To provide a unified customer experience, the domain teams optimize the usability of the data for the organization.

Various attributes define organizational data usability. Zhamak Dehghani best defines these attributes in her book Data Mesh. 

The following summarizes the eight attributes Dehghani outlines for data usability within the organization:

Discoverable: Data products are developed with discoverability as a priority. Data users can easily search and find data products for their needs.

Addressable: Each data product includes a unique address that allows users to programmatically and consistently access the data product as it changes throughout the product lifecycle.

Understandable: Data products are developed to be easily understood by the user. Users will understand the data’s meaning, how it is presented, and how it has been used within the organization.

Trustworthy: The truthfulness of the data is established in each data product. The user has confidence in the product’s reliability and accurate facts. Ensuring data quality in a data mesh presents unique challenges. Learn how to select the best technology for data quality management.

Natively Accessible: Data products can be read and accessed by various modes of access by different user types.

Interoperable: Data products have consistent standards for exchanging and using data.

Valuable: Data products are developed with stand-alone value for the user. 

Secure: Data products contain standardized security policies.

 

The Challenges of Data as a Product and How They are Addressed

Below are the primary challenges of data products and the mesh principle that addresses the challenge.

Cost of ownership: Domain ownership of data products can increase the cost of data through increased data product management. This cost is mitigated through the self-serve platform principle, which reduces duplicative efforts and increases workload capacity and productivity.

Consistency of value and utility: Consistency of product quality across multiple domains can become a challenge and ultimately reduce the value and trustworthiness of the data. The federated governance principle helps address this through the application of data quality technology, global policies, and accountability across domains and subject matter experts.

 

What’s Next After Data as a Product?

Data as a product optimizes enterprise data by turning analytical data into a product to be shared and used across the organization. By turning data into a product, the organization reduces data distribution friction and costs and prevents data siloing from domain ownership.

To distribute the product, a self-serve platform is needed. Developing a self-serve platform is the next step in a data mesh architecture, and we will cover it in our next blog post.

In our previous post, we discussed the opportunity of developing a data mesh architecture and how it can benefit data-driven enterprise organizations.

In the next four posts of our data mesh series, we will cover the fundamental principles that make up the organizational functionality of a data mesh. This post will cover the first principle, Domain Ownership, and the evolution of decentralized data management.

 

What is Domain Ownership?

Domains represent natural business units in an organization where data is sourced or consumed.

Therefore, domain ownership decentralizes the data ownership and management responsibilities to the business units or domains within an organization, creating data management at the source.

Domain ownership changes the flow of data within the organization. In a traditional architecture, data flows from the source to a centralized system, like a data lake, to be ingested, processed, and served. This centralized system is commonly where new individuals or groups take responsibility for owning the data. In a data mesh, the domain is the source where data is ingested, processed, and served to the organization.

 

What Problems Does Domain Ownership Address?

As enterprise organizations become increasingly data-driven, there is a higher demand for acute knowledge of the data source to quickly process and share data in a valuable and agile way.

Rapidly understanding and processing data is a challenge with traditional centralized architecture where data is partitioned by technology whose data engineers work with multiple sources.

Domain ownership solves these challenges in the following three primary ways.

Knowledge responsibility: Domain ownership allows teams closest to and most knowledgeable of the data to own it, improving the value and truthfulness of the data outputs.

Change and agility: Data modeling is localized to the business units most familiar with the data through domain ownership. This localization allows for more agile modeling and the ability to implement change without centralized coordination.

Scaled data use and sharing: Increasing the number of sources ingesting and processing data means higher data utility and output, leading to increased sharing and consumption.

 

Domain Ownership Role in Data Mesh

Data ingestion and processing: In a decentralized architecture, the domains are responsible for ingesting data for immediate use or storage. Likewise, they are responsible for processing or cleaning data for analysis and insights.

Analytics and data product: Once domains have ingested and processed the source data, they are responsible for analytics and developing a data product to be distributed and shared.

Domain data can be classified into three variations.

Source-aligned domain data: Data facts are generated by business operations. Source-aligned data most closely relates to domain events generated by domain operational systems.

Aggregate domain data: Data bound from multiple domain sources to form a data product.

Consumer aligned domain data: Data transformed for specific use cases across organizational functions.

These variations represent how domains interact with the data and develop subsequent data products.

 

The Challenges of Domain Ownership and How They are Addressed

Domain ownership can have a unique set of challenges that must be addressed. Below are the primary challenges and the mesh principle that addresses the challenge.

Risk of data siloing to where domains collect data and are not incentivized to share. This risk is addressed through the data as a product principle.

Lack of empowerment across domains to share data within the organization. This challenge is addressed through the self-serve platform principle.

The risk of poor engagement and isolation occurs if there is no structure for accountability, domain organization, global operation, and policy. This challenge is addressed through the federated governance principle.

 

What’s Next After Taking Domain Ownership?

A domain ownership approach allows data management at the source, with the source experts overseeing the data. This approach allows for reduced friction and scalability of serving analytical data to organizational data consumers. Domains allow the mesh to work by using domain data products upstream and downstream.

To learn more about the next three principles of data as a product, self-serve platform, and federated governance and how they support the challenges of domain ownership, stay tuned for our upcoming blog post.

Organizationally sourced data is increasingly complex and filled with valuable business intelligence. As organizations become increasingly data-driven, they will need to become more agile and prudent with managing, analyzing, and distributing data.

Implementing a data mesh architecture allows an organization to leverage a decentralized data management system that aligns with the current organizational structure. A data mesh distributes the data burden to experts at the domain data source. Keeping domain experts close to the data and analytics throughout most of the domain data lifecycle is an efficient and cost-conscious strategy for optimizing data management.

In the following post, I will explain the data mesh concept, how it addresses current data architecture problems, and the four principles of a data mesh.

What is a Data Mesh?

Zhamak Dehghani coined the term data mesh in her book Data Mesh: Delivering Data Driven Value at Scale.

Dehghani defines data mesh as a decentralized sociotechnical approach to sharing, accessing, and managing analytical data in complex and large-scale environments within or across organizations. 

If we break down this definition, we get four main themes inherent to data mesh architecture.

Decentralized: Moving from one central management system that serves as the aggregator and distributor of data to multiple domains or business units each managing data they create and source. These decentralized domains serve as the aggregators and distributors.

Sociotechnical: An evolved approach to data management that encompasses the interactions between people, technical architecture, and solutions.

Share & Access: A data platform that can consistently extract utility and value out of data and make it available to users across the organization when needed.

Manage analytical data: Mature management and collaborative governance from multiple stakeholders is needed to provide consistency, security, and quality of business intelligence.

What problems does a data mesh address?

For data-driven enterprises, traditional centralized data architecture can quickly become a bottleneck for organizations that need to quickly ingest, process, and distribute data to add value and utility for business units.   

Outlined below is an overview of how a data mesh solves the common challenges that data-driven organizations have with traditional architecture.

Centralized and siloed management and ownership: A data mesh takes a distributed and decentralized approach to data management at the source. Taking the burden off a central location to ingest, process, and serve data.

Speed and agility: Decentralizing data management to the data source improves efficiency and reduces bottlenecks and bureaucratic red tape by having that data processed and distributed by domain or business unit source experts.

Value and utility: Domain source experts consistently develop and serve data products to a platform based on organizational needs.

The previously mentioned themes and problems a data mesh addresses manifest into four core principles that interplay to make the mesh structure work within an organization.

Data Mesh Principles

Data mesh leverages four principles that have a symbiotic relationship within a data-driven enterprise information management strategy.

Domain Ownership: Domains are business units within an organization; these domains take on ownership and management of data in relation to specific domains. 

Data as a Product: Domain data is developed and shared as a product for internal and external consumers. 

Self-Serve Platform: The platform houses the products developed by the domains, allowing for product shareability and accessibility across domains and functions.

Federated Governance: Governance of the domains, products, and platforms that are decentralized and managed by various stakeholders such as domain representatives, legal, compliance, and security.

The four principles are what make the data mesh architecture function. Likewise, each principle plays off the functionality of the other to address challenges that each one could create. 

Here is how the interplay of the principles ensures optimal functionality within the organization. 

Domain Ownership depends on:

• Data as a Product to prevent data siloing
• Self-Serve Platform to empower domain teams
• Federated Governance to increase engagement and reduce domain isolation

Data as a Product depends on:

• Self-Serve Platform to reduce the cost of ownership
• Federated Governance to get high-order value by interconnecting data products distributed by one or more domain teams

Federated Governance depends on the Self-Serve Platform to enforce a consistent and reliable policy.

While the four main principles provide the framework for the data mesh to function, a shift in organizational thinking and perspective is required to ensure the data mesh adheres to the principles.  

Below are six categories that are essential for a change in how the organization thinks about data management.

Organization perspective: The organization moves from a centralized model to a decentralized data ownership model. In a decentralized model, the business domains are the data owners. 

Architectural perspective: The architecture moves beyond collecting data in lakes and monolithic warehouses. The architecture uses a distributed approach to accessing data products through standardized protocols. 

Technological perspective: Data evolves from being a consequence of code. Data and code become a cohesive and adaptable entity. 

Operational perspective: Data governance takes on a federated approach that relies on a computation system of policies replacing a centralized top-down and often human-administered approach. 

Principal perspective: Data evolves from an asset to a product for internal and external users. 

Infrastructure perspective: Removes the bifurcated approach to data and analytics that contains one system for analytics processing and another for transactional processing of operational applications. The infrastructure evolves to integrate transactional and analytical processing physically or virtually under a unified platform.

Expected Outcome

Data mesh architecture is a solution for data-driven enterprises that consistently rely on data to evolve, improve, make decisions, and manage lines of business. A data mesh can allow an organization to stay agile, extract value and improve the utility of data, and allow for malleability as the organization evolves.