In a data mesh, a self-service data platform enables users to generate value
from data by enabling them to autonomously build, share, and use data products.
To fully realize these benefits, we recommend that your self-service data
platform provide the capabilities described in this document.

This document is part of a series which describes how to implement a data mesh
on Google Cloud. It assumes that you have read and are familiar with the
concepts described in
[Build a modern, distributed Data Mesh with Google Cloud](https://services.google.com/fh/files/misc/build-a-modern-distributed-datamesh-with-google-cloud-whitepaper.pdf)
and [Architecture and functions in a data mesh](https://docs.cloud.google.com/architecture/data-mesh).

The series has the following parts:

- [Architecture and functions in a data mesh](https://docs.cloud.google.com/architecture/data-mesh)
- Design a self-service data platform for a data mesh (this document)
- [Build data products in a data mesh](https://docs.cloud.google.com/architecture/build-data-products-data-mesh)
- [Discover and consume data products in a data mesh](https://docs.cloud.google.com/architecture/discover-consume-data-products-data-mesh)

Data platform teams typically create central self-service data platforms, as
described in this document. This team builds the solutions and components that
domain teams (both data producers and data consumers) can use to both create and
consume data products. Domain teams represent functional parts of a data mesh.
By building these components, the data platform team enables a smooth
development experience and reduces the complexity of building, deploying, and
maintaining data products that are secure and interoperable.

Ultimately, the data platform team should allow domain teams to move faster.
They help increase the efficiency of domain teams by providing those teams with a
limited set of tools that address their needs. In providing these tools,
the data platform team removes the burden of having the domain team build and
source these tools themselves. The tooling choices should be customizable to
different needs and not force an inflexible way of working on the data domain
teams.

The data platform team shouldn't focus on building custom solutions for data
pipeline orchestrators or for continuous integration and continuous deployment
(CI/CD) systems. Solutions such as CI/CD systems are readily available as
managed cloud services, for example, Cloud Build. Using managed cloud
services can reduce operational overheads for the data platform team and let
them focus on the specific needs of the data domain teams as the users of the
platform. With reduced operational overhead, the data platform team can focus
more time on addressing the specific needs of the data domain teams.

## Architecture

The following diagram illustrates the architecture components of a self-service
data platform. The diagram also shows how these components can support teams as
they develop and consume data products across the data mesh.

![Self-service data platform components, as described in the following text.](https://docs.cloud.google.com/static/architecture/images/data-mesh-self-service-data-platform.svg)

As shown in the preceding diagram, the self-service data platform provides the
following:

- **Platform solutions:** These solutions consist of composable
  components for provisioning Google Cloud projects and resources, which users
  select and assemble in different combinations to meet their specific
  requirements. Instead of directly interacting with the components, users of
  the platform can interact with platform solutions to help them to achieve a
  specific goal. Data domain teams should design platform solutions to solve common
  pain-points and friction areas that cause slowdowns in data product
  development and consumption. For example, data domain teams onboarding onto
  the data mesh can use an
  [infrastructure-as-code (IaC)](https://en.wikipedia.org/wiki/Infrastructure_as_code)
  template. Using IaC templates lets them quickly create a set of
  Google Cloud projects with standard Identity and Access Management (IAM)
  permissions, networking, security policies, and relevant Google Cloud
  APIs enabled for data product development. We recommend that each solution is
  accompanied with documentation such as "how to get started" guidance and code
  samples. Data platform solutions and their components must be secure and
  compliant by default.

- **Common services:** These services provide data product
  discoverability, management, sharing, and observability. These services
  facilitate data consumers' trust in data products, and are an effective way
  for data producers to alert data consumers to issues with their data products.

Data platform solutions and common services might include the following:

- IaC templates to set up foundational data product development workspace environments, which include the following:
  - IAM
  - Logging and monitoring
  - Networking
  - Security and compliance guardrails
  - Resource tagging for billing attribution
  - Data product storage, transformation, and publishing
  - Data product registration, cataloging, and metadata tagging
- IaC templates which follow organizational security guardrails and best practices that can be used to deploy Google Cloud resources into existing data product development workspaces.
- Application and data pipeline templates that can be used to bootstrap new projects or used as reference for existing projects. Examples of such templates include the following:
  - Usage of common libraries and frameworks
  - Integration with platform logging, monitoring, and observability tooling
  - Build and test tooling
  - Configuration management
  - Packaging and CI/CD pipelines for deployment
  - Authentication, deployment, and management of credentials
- Common services to provide data product observability and governance which can include the following:
  - Uptime checks to show the overall state of data products.
  - Custom metrics to give helpful indicators about data products.
  - Operational support by the central team such that data consumer teams are alerted of changes in data products they use.
  - Product scorecards to show how data products are performing.
  - A metadata catalog for discovering data products.
  - A centrally defined set of computational policies that can be applied globally across the data mesh.
  - A data marketplace to facilitate data sharing across domain teams.

[Create platform components and solutions using IaC templates](https://docs.cloud.google.com/architecture/design-self-service-data-platform-data-mesh#create_platform_components_and_solutions_using_iac_templates)
discusses the advantages of IaC templates to expose and deploy data products.
[Provide common services](https://docs.cloud.google.com/architecture/design-self-service-data-platform-data-mesh#provide_common_services)
discusses why it's helpful to provide domain teams with common infrastructure
components that have been built and are managed by the data platform team.

## Create platform components and solutions using IaC templates

The goal of data platform teams is to set up self-service data platforms to get
more value from data. To build these platforms, they create and provide domain
teams with vetted, secure, and self-serviceable infrastructure templates. Domain
teams use these templates to deploy their data development and data consumption
environments. IaC templates help data platform teams achieve that goal and
enable scale. Using vetted and trusted IaC templates simplifies the resource
deployment process for domain teams by allowing those teams to reuse existing
CI/CD pipelines. This approach lets domain teams quickly get started and become
productive within the data mesh.

IaC templates can be created using an IaC tool. Although there are multiple IaC
tools, including
[Cloud Config Connector](https://docs.cloud.google.com/config-connector/docs/overview),
[Pulumi](https://www.pulumi.com/),
[Chef](https://www.chef.io/products/chef-infra#:%7E:text=Chef%20Infra%20is%20a%20configuration,system%20state%20automated%20and%20testable.),
and
[Ansible](https://www.ansible.com/integrations/infrastructure),
this document provides examples for
[Terraform](https://www.terraform.io/)-based
IaC tools. Terraform is an open source IaC tool that allows the data platform
team to efficiently create composable platform components and solutions for
Google Cloud resources. Using Terraform, the data platform team writes
code that specifies the chosen end-state and lets the tool figure out how to
achieve that state. This declarative approach lets the data platform team treat
infrastructure resources as immutable artifacts for deployment across
environments. It also helps to reduce the risk of inconsistencies arising
between deployed resources and the declared code in source control (referred to
as *configuration drift*). Configuration drift caused by ad hoc and manual
changes to infrastructure hinders safe and repeatable deployment of IaC
components into production environments.

Common IaC templates for composable platform components include using
[Terraform modules](https://www.terraform.io/language/modules/develop)
for deploying resources such as a BigQuery dataset, Cloud Storage
bucket, or Cloud SQL database. Terraform modules can be combined into
end-to-end solutions for deploying complete Google Cloud projects, including
relevant
[resources](https://www.terraform.io/language/resources)
deployed using the composable
[modules](https://www.terraform.io/language/modules).
Example Terraform modules can be found in the
[Terraform blueprints for Google Cloud](https://learn.hashicorp.com/collections/terraform/gcp-get-started).

Each Terraform module should by default satisfy security guardrails and
compliance policies that your organization uses. These guardrails and policies
can also be expressed as code and be automated using automated compliance
verification tooling such as Google Cloud
[policy validation tool](https://cloud.google.com/docs/terraform/policy-validation).

Your organization should continuously test the platform-provided Terraform
modules, using the same automated compliance guardrails that it uses to promote
changes into production.

To make IaC components and solutions discoverable and consumable for domain
teams that have minimal experience with Terraform, we recommend that you use
services such as
[Service Catalog](https://docs.cloud.google.com/service-catalog/docs/terraform-configuration).
Users who have significant customization requirements should be allowed to
create their own deployment solutions from the same composable Terraform
templates used by existing solutions.

When using Terraform, we recommend that you follow the Google Cloud
best-practices as outlined in
[Best practices for using Terraform](https://cloud.google.com/docs/terraform/best-practices-for-terraform).

To illustrate how Terraform can be used to create platform components, the
following sections discuss examples of how Terraform can be used to expose
consumption interfaces and to consume a data product.

### Expose a consumption interface

A consumption interface for a data product is a set of guarantees on the data
quality and operational parameters provided by the data domain team to enable
other teams to discover and use their data products. Each consumption interface
also includes a product support model and product documentation. A data product
may have different types of consumption interfaces, such as APIs or streams, as
described in
[Build data products in a data mesh](https://docs.cloud.google.com/architecture/build-data-products-data-mesh).
The most common consumption interface might be a BigQuery
authorized dataset, authorized view, or authorized function. This interface
exposes a read-only virtual table, which is expressed as a query into the data
mesh. The interface does not grant reader permissions to directly access the
underlying data.

Google provides an example
[Terraform module for creating authorized views](https://github.com/terraform-google-modules/terraform-google-bigquery/tree/master/modules/authorization)
without granting teams permissions to the underlying authorized datasets. The
following code from this Terraform module grants these IAM
permissions on the `dataset_id` authorized view:

    module "add_authorization" {
      source = "terraform-google-modules/bigquery/google//modules/authorization"
      version = "~> 4.1"

      dataset_id = module.dataset.bigquery_dataset.dataset_id
      project_id = module.dataset.bigquery_dataset.project

      roles = [
        {
          role           = "roles/bigquery.dataEditor"
          group_by_email = "ops@mycompany.com"
        }
      ]

      authorized_views = [
        {
          project_id = "view_project"
          dataset_id = "view_dataset"
          table_id   = "view_id"
        }
      ]
      authorized_datasets = [
        {
          project_id = "auth_dataset_project"
          dataset_id = "auth_dataset"
        }
      ]
    }

If you need to grant users access to multiple views, granting access to each
authorized view can be both time consuming and harder to maintain. Instead of
creating multiple authorized views, you can use an
[authorized dataset](https://docs.cloud.google.com/bigquery/docs/authorized-datasets)
to automatically authorize any views created in the authorized dataset.

### Consume a data product

For most analytics use cases, consumption patterns are determined by the
application that the data is being used in. The main use of a centrally provided
consumption environment is for data exploration before the data is used within
the consuming application. As discussed in
[Discover and consume products in a data mesh](https://docs.cloud.google.com/architecture/discover-consume-data-products-data-mesh), SQL is the most commonly used method for querying data products. For this reason,
the data platform should provide data consumers with a SQL application for
exploration of the data.

Depending on the analytics use case, you may be able to use Terraform to deploy
the consumption environment for data consumers. For example, data science is a
common use case for data consumers. You can use Terraform to deploy
Gemini Enterprise Agent Platform user-managed notebooks to be used as a data science development
environment. From the data science notebooks, data consumers can use their
credentials to sign in to the data mesh to explore data to which they have access
and develop ML models based on this data.

To learn how to use Terraform to deploy and help to secure a notebook
environment on Google Cloud, see
[Build and deploy generative AI and machine learning models in an enterprise](https://docs.cloud.google.com/architecture/blueprints/genai-mlops-blueprint).

## Provide common services

In addition to self-service IaC components and solutions, the data platform
team might also take ownership over building and operating common shared
platform services used by multiple data domain teams. Common examples of shared
platform services include self-hosted third-party software such as business
intelligence visualization tools or a Kafka cluster. In Google Cloud, the
data platform team might choose to manage resources such as Knowledge Catalog
and Cloud Logging sinks on behalf of data domain teams. Managing resources
for the data domain teams lets the data platform team facilitate centralized
policy management and auditing across the organization.

The following sections show how to use Knowledge Catalog for central
management and governance within a data mesh on Google Cloud, and the
implementation of data observability features in a data mesh.

### Knowledge Catalog for data governance

Knowledge Catalog provides a data management platform that helps you to
build independent data domains within a data mesh that spans the organization.
Knowledge Catalog lets you maintain central controls for governing and
monitoring the data across domains.

With Knowledge Catalog an organization can logically organize their data
([supported data sources](https://docs.cloud.google.com/dataplex/docs/best-practices))
and related artifacts such as code, notebooks, and logs, into a
Knowledge Catalog lake that represents a data domain. In the following
diagram, a sales domain uses Knowledge Catalog to organize its assets,
including data quality metrics and logs, into Knowledge Catalog zones.

![Assets organized by Knowledge Catalog.](https://docs.cloud.google.com/static/architecture/images/dataplex.png)

As shown in the preceding diagram, Knowledge Catalog can be used to
manage domain data across the following assets:

- Knowledge Catalog allows data domain teams to consistently manage their data assets in a logical group called a Knowledge Catalog lake. The data domain team can organize their Knowledge Catalog assets within the same Knowledge Catalog lake without physically moving data or storing it into a single storage system. Knowledge Catalog assets can refer to Cloud Storage buckets and BigQuery datasets stored in multiple Google Cloud projects other than the Google Cloud project containing the Knowledge Catalog lake. Knowledge Catalog assets can be structured or unstructured, or be stored in an analytical data lake or data warehouse. In the diagram, there are data lakes for the sales domain, supply chain domain, and product domain.
- Knowledge Catalog zones enable the data domain team to further organize data assets into smaller subgroups within the same Knowledge Catalog lake and add structures that capture key aspects of the subgroup. For example, Knowledge Catalog zones can be used to group associated data assets in a data product. Grouping data assets into a single Knowledge Catalog zone allows data domain teams to manage access policies and data governance policies consistently across the zone as a single data product. In the diagram, there are data zones for offline sales, online sales, supply chain warehouses, and products.

Knowledge Catalog lakes and zones enable an organization to unify
distributed data and organize it based on the business context. This arrangement
forms the foundation for activities such as managing metadata, setting up
governance policies, and monitoring data quality. Such activities allow the
organization to manage its distributed data at scale, such as in a data mesh.

### Data observability

Each data domain should implement its own monitoring and alerting mechanisms,
ideally using a standardized approach. Each domain can apply the monitoring
practices described in
[Concepts in service monitoring](https://docs.cloud.google.com/stackdriver/docs/solutions/slo-monitoring), making the necessary adjustments to the data domains. Observability is a large topic, and is
outside of the scope of this document. This section only addresses patterns which
are useful in data mesh implementations.

For products with multiple data consumers, providing timely information to each
consumer about the status of the product can become an operational burden. Basic
solutions, such as manually managed email distributions, are typically prone to
error. They can be helpful for notifying consumers of planned outages, upcoming
product launches, and deprecations, but they don't provide real-time operational
awareness.

Central services can play an important role in monitoring the health and quality
of the products in the data mesh. Although not a prerequisite for a successful
implementation of the data mesh, implementing observability features can improve
satisfaction of the data producers and consumers, and reduce overall operational
and support costs. The following diagram shows an architecture of data mesh
observability based on
[Cloud Monitoring](https://docs.cloud.google.com/monitoring).

![Data mesh observability.](https://docs.cloud.google.com/static/architecture/images/data-mesh-observability.svg)

The following sections describe the components shown in the diagram, which are
as follows:

- [Uptime checks](https://docs.cloud.google.com/architecture/design-self-service-data-platform-data-mesh#uptime_checks) to show the overall state of data products.
- [Custom metrics](https://docs.cloud.google.com/architecture/design-self-service-data-platform-data-mesh#custom_metrics) to give helpful indicators about data products.
- [Operational support by the central data platform team](https://docs.cloud.google.com/architecture/design-self-service-data-platform-data-mesh#operational_support_by_the_central_data_platform_team) to alert data consumers of changes in the data products that they use.
- [Product scorecards](https://docs.cloud.google.com/architecture/design-self-service-data-platform-data-mesh#product_scorecards) and dashboards to show how data products are performing.

#### Uptime checks

Data products can create simple custom applications that implement
[uptime checks](https://docs.cloud.google.com/monitoring/uptime-checks/introduction). These checks can serve as high-level indicators of the overall state of the product.
For example, if the data product team discovers a sudden drop in data quality of
its product, the team can mark that product unhealthy. Uptime checks that are
close to real time are especially important to data consumers who have derived
products that rely on the constant availability of the data in the upstream data
product. Data producers should build their uptime checks to include checking their
upstream dependencies, thus providing an accurate picture of the health of their
product to their data consumers.

Data consumers can include product uptime checks into their processing. For
example, a composer job that generates a report based on the data provided by a
data product can, as the first step, validate whether the product is in the
"running" state. We recommend that your uptime check application returns a
structured payload in the message body of its HTTP response. This structured
payload should indicate whether there's a problem, the root cause of the problem
in human readable form, and if possible, the estimated time to restore the
service. This structured payload can also provide more fine-grained information
about the state of the product. For example, it can contain the health
information for each of the views in the authorized dataset exposed as a
product.

#### Custom metrics

Data products can have various custom metrics to measure their usefulness. Data
producer teams can publish these custom metrics to their designated
domain-specific Google Cloud projects. To create a
[unified monitoring experience](https://docs.cloud.google.com/monitoring/settings/multiple-projects) across all data products, a central data mesh monitoring project
can be given access to those domain-specific projects.

Each type of data product consumption interface has different metrics to measure
its usefulness. Metrics can also be specific to the business domain. For
example, the metrics for BigQuery tables exposed through views or
through the Storage Read API can be as follows:

- The number of rows.
- Data freshness (expressed as the number of seconds before the measurement time).
- The data quality score.
- The data that's available. This metric can indicate that the data is available for querying. An alternative is to use the uptime checks mentioned earlier in this document.

These metrics can be viewed as service level indicators (SLI) for a particular
product.

For data streams (implemented as Pub/Sub topics), this list can be
[the standard Pub/Sub metrics](https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z#gcp-pubsub), which are available through topics.

#### Operational support by the central data platform team

The central data platform team can expose custom dashboards to display
different levels of details to the data consumers. A simple status dashboard
that lists the products in the data mesh and uptime status for those products
can help answer multiple end-user requests.

The central team can also serve as a notification distribution hub to notify
data consumers about various events in the data products they use. Typically,
this hub is made by creating
[alerting policies](https://docs.cloud.google.com/monitoring/uptime-checks/uptime-alerting-policies). Centralizing this function can reduce the work that must be
done by each data producer team. Creating these policies doesn't require knowledge
of the data domains and should help avoid bottlenecks in data consumption.

An ideal end state for data mesh monitoring is for the data product tag template
to expose the SLIs and service-level objectives
(SLOs) that the product supports when the product becomes available. The central
team can then automatically deploy the corresponding alerting using service
monitoring with the Monitoring API.

#### Product scorecards

As part of the central governance agreement, the
[four functions in a data mesh](https://docs.cloud.google.com/architecture/data-mesh#functions_in_a_data_mesh) can define the criteria to
create scorecards for data products. These scorecards can become an objective
measurement of data product performance.

Many of the variables used to calculate the scorecards are the percentage of
time that data products are meeting their SLO. Useful criteria can be the
percentage of uptime, average data quality scores, and percentage of products
with data freshness that does not fall below a threshold. To calculate these
metrics automatically using
[Prometheus Query Language (PromQL)](https://docs.cloud.google.com/monitoring/promql), the custom metrics and the results of the uptime checks from the central
monitoring project should be sufficient.

## What's next

- Learn more about [BigQuery](https://docs.cloud.google.com/bigquery).
- Read about [Dataplex](https://docs.cloud.google.com/dataplex).
- For more reference architectures, diagrams, and best practices, explore the [Cloud Architecture Center](https://docs.cloud.google.com/architecture).