Set up cross-cloud connection for Workday Data Lake

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

Note: For support during the preview, send an email to [e-mail].

A cross-cloud connection to Workday Data Lake lets you query your Workday data directly within Google Cloud. You can then use Lakehouse to manage access to and analyze your federated data without copying or moving data.

Use cases

Connecting Lakehouse to Workday Data Lake supports several key use cases:

• Unify analytics: Correlate Workday HR and compensation data with Google Cloud data, for example, to provide context for sales and quotas.
• Leverage Google Cloud ecosystem: For example, use Google's agent framework with BigQuery ML and Workday HR data to predict employee retention.
• Stream real-time, no-copy data: Analyze Workday procurement and accounts payable data alongside logistics and inventory data stored in Google Cloud to report supply chain inefficiencies and optimize vendor costs.

Before you begin

1. Review the Lakehouse overview to understand how Lakehouse manages access to data.
2. Read about accessing cross-cloud data to understand how it works.
3. Review the supported catalogs to verify compatibility.
4. Understand how to use regional Secret Manager secrets to authenticate with Workday Data Lake.
5. Check in with your Workday Data Lake administrators to set up authentication as described in this document. Administrators might need to contact Workday Support to enable Data Lake access, which can take time to resolve.
Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Verify that billing is enabled for your Google Cloud project.

Enable the BigLake, Secret Manager APIs.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the APIs

Verify that billing is enabled for your Google Cloud project.

Enable the BigLake, Secret Manager APIs.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the APIs

Required roles

To get the permissions that you need to set up cross-cloud access, ask your administrator to grant you the following IAM roles on your project:

• Manage Lakehouse catalogs: BigLake Admin (roles/biglake.admin)
• Manage secrets: Secret Manager Admin (roles/secretmanager.admin)

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Caution: Ask a Secret Manager administrator to create the secret and grant you access to it. This follows the principle of least privilege by ensuring that you have access to this Lakehouse secret only, rather than broad access to all secrets in the project.

Supported catalog details

This document provides instructions for setting up Lakehouse with Workday Data Lake. To access other catalogs, see Supported catalogs.

Limitations and considerations

Keep the following in mind when accessing a Workday Data Lake:

• Read-only: Federated catalogs in Lakehouse are read-only views of the remote catalog. To create, update, or delete resources, you must use Workday directly.
• Network routing: Connections and queries route securely over the public internet.
• Data freshness: The --refresh-interval flag determines how often Lakehouse synchronizes metadata. The value must be 0s (disabled) or at least 300s (5 minutes). As the number of namespaces and tables in a catalog increases, background metadata refreshes take longer to complete. If the previous refresh overruns its scheduled interval, the system skips the current cycle and resumes at the next scheduled interval.
• Colocation: To avoid connectivity issues and minimize latency and data transfer costs, create the federated catalog and regional secret in the Google Cloud region closest to the region where your Workday instance resides.

General workflow

To access cross-cloud data in Workday Data Lake, follow these general steps:

1. Set up federation: Configure secret-based authentication and create a federated catalog in Lakehouse.
   1. In Workday, create an Integration System User (ISU) and an API Client for Integrations.
   2. Create a secret in Secret Manager with your Workday API credentials.
   3. Create a federated catalog in Lakehouse and grant the catalog service account access to the secret.
2. Verify the connection: Verify that Lakehouse can connect to your remote catalog and synchronize metadata.
3. Query data: Run queries against your federated data using BigQuery or Managed Service for Apache Spark. For more information, see Query remote data.
4. Configure permissions: Use Identity and Access Management (IAM) to manage who can view and query the federated data.

Set up federation

To query your data, you must set up a Lakehouse federated catalog that connects to your remote Workday Data Lake.

Configure authentication

Federation requires authenticating to the remote Workday Data Lake using credentials stored securely in regional Secret Manager secrets.

1. In Workday, complete the following setup:

   1. Create an Integration System User (ISU): Run the Create Integration System User task to create a dedicated account that Lakehouse uses to synchronize resources.
   2. Enable Workday Data Lake access for the ISU: Grant the ISU access to Workday Data Lake. Contact Workday support to enable this access. You cannot perform this step yourself; wait for Workday to configure access in your Workday tenant before continuing.
   3. Register API Client for Integrations: Run the Register API Client for Integrations task.
   4. Save the client ID and secret: Save the OAuth client ID and secret for the following step.
   5. Generate a non-expiring refresh token: In the API Client for Integrations, use Manage Refresh Tokens for Integrations to generate a non-expiring refresh token for the ISU.
   6. Save the refresh token: Save the generated refresh token for the following step.

2. Create a JSON file named credentials.json with the saved data from the previous step:

   {
     "client_id": "CLIENT_ID",
     "client_secret": "CLIENT_SECRET",
     "refresh_token": "REFRESH_TOKEN"
   }

   Replace the following:

   • CLIENT_ID: The OAuth client ID from your Workday API Client for Integrations.
   • CLIENT_SECRET: The OAuth client secret from your Workday API Client for Integrations.
   • REFRESH_TOKEN: The non-expiring refresh token generated for your Workday ISU.

3. Configure the regional endpoint for Secret Manager:

   By default, Secret Manager uses a global endpoint. To avoid connectivity issues and minimize latency and data transfer costs, create your secret and catalog in the same region. To override the default global endpoint with a regional secret, run the following command:

   gcloud config set api_endpoint_overrides/secretmanager https://secretmanager.REGION.rep.googleapis.com/

   Replace the following:

   • REGION: The Google Cloud region where you store your Secret Manager secret. For example, us-east4.

4. Upload the payload to Secret Manager:

   gcloud secrets create WORKDAY_SECRET_NAME \
     --location="REGION" \
     --project="PROJECT_ID" \
     --data-file=credentials.json

5. Securely delete the credentials.json file to prevent credential leakage.

   Replace the following:

   • WORKDAY_SECRET_NAME: A unique name for your Workday secret in Secret Manager, for example, workday-api-credentials or workday-data-lake-secret.
   • REGION: The Google Cloud region where you create the secret, for example, us-east4.
   • PROJECT_ID: Your Google Cloud project ID.

Create a federated catalog

To create a federated catalog using the gcloud CLI, run the following command:

gcloud alpha biglake iceberg catalogs create FEDERATED_CATALOG_NAME \
    --project="PROJECT_ID" \
    --primary-location="REGION" \
    --catalog-type="federated" \
    --federated-catalog-type="workday" \
    --secret-name="projects/PROJECT_ID/locations/REGION/secrets/WORKDAY_SECRET_NAME" \
    --workday-base-url="WORKDAY_BASE_URL" \
    --workday-tenant="WORKDAY_TENANT" \
    --refresh-interval="REFRESH_INTERVAL" \
    --namespace-filters="NAMESPACE_FILTERS"

Replace the following:

• FEDERATED_CATALOG_NAME: A name for your federated catalog in Lakehouse.
• PROJECT_ID: Your Google Cloud project ID.
• REGION: The Lakehouse region where you create the federated catalog, for example, us-east4. To minimize latency and data transfer costs, select the Google Cloud region closest to your Workday instance. This region must be the same region where you stored your secret.
• WORKDAY_SECRET_NAME: The name of your Workday secret in Secret Manager.
• WORKDAY_BASE_URL: The base URL of your Workday instance. For example, impl-services1.wd12.myworkday.com or wd501.myworkday.com.
• WORKDAY_TENANT: The Workday tenant name.
• REFRESH_INTERVAL: Optional: Specifies how often to update the catalog's information. Set this value as a duration, for example, 300s or 5m. Shorter intervals update data more often but can cost more in API calls. Longer intervals can cost less, but the queried data might not reflect your most current dataset. If omitted, the refresh interval defaults to 5 minutes (300s). Setting the value to 0s disables background metadata refresh.
• NAMESPACE_FILTERS: Optional: A comma-separated list of namespaces to federate, for example, finance,hr. If omitted, Lakehouse includes all namespaces.

Complete authentication setup

After you create the catalog, Lakehouse provisions a unique service account for it, identified as biglake-service-account in the resource description.

You must grant this service account the Secret Manager Secret Accessor role (roles/secretmanager.secretAccessor) on the secret that you created earlier. It might take a few minutes for new IAM policies to take effect.

Console

1. In the Google Cloud console, go to Lakehouse.

   [Go to Lakehouse][5]

2. Click the name of the federated catalog that you created for Workday.

3. On the Catalog details page, in the alert banner, click Grant secret permissions.

   Lakehouse grants the roles/secretmanager.secretAccessor role on the secret to the provisioned service account.

gcloud CLI

1. Grant the catalog's service account permission to access the secret:

   gcloud config set api_endpoint_overrides/secretmanager https://secretmanager.REGION.rep.googleapis.com/
   gcloud secrets add-iam-policy-binding WORKDAY_SECRET_NAME \
     --project="PROJECT_ID" \
     --location="REGION" \
     --member="serviceAccount:$(gcloud alpha biglake iceberg catalogs describe FEDERATED_CATALOG_NAME \
         --project="PROJECT_ID" \
         --format='value(biglake-service-account)')" \
         --role="roles/secretmanager.secretAccessor"

2. To verify that the federated catalog service account has access to the secret, run the following command:

   gcloud config set api_endpoint_overrides/secretmanager https://secretmanager.REGION.rep.googleapis.com/
   gcloud secrets get-iam-policy WORKDAY_SECRET_NAME \
        --project="PROJECT_ID" \
        --location="REGION"

   In the output, verify that the biglake-service-account service account has the roles/secretmanager.secretAccessor role.

Replace the following:

• REGION: The Google Cloud region where you store your Secret Manager secret and created the federated catalog, for example, us-east4.
• WORKDAY_SECRET_NAME: The name of your Workday secret in Secret Manager.
• PROJECT_ID: Your Google Cloud project ID.
• FEDERATED_CATALOG_NAME: The name of your federated catalog in Lakehouse.

Verify the connection

Verify that the background metadata refresh completed successfully and synchronized your namespaces and tables.

1. Verify that the refresh status indicates success:

   gcloud alpha biglake iceberg catalogs describe FEDERATED_CATALOG_NAME \
     --project="PROJECT_ID"

2. Confirm that the namespaces are synchronized:

   gcloud alpha biglake iceberg namespaces list \
     --project="PROJECT_ID" \
     --catalog="FEDERATED_CATALOG_NAME"

Replace the following:

• PROJECT_ID: Your Google Cloud project ID.
• FEDERATED_CATALOG_NAME: The name of your federated catalog in Lakehouse.

What's next

• Query remote data.
• Troubleshoot cross-cloud data access.