Troubleshoot cross-cloud Lakehouse for Apache Iceberg issues

This page shows you how to resolve common issues with cross-cloud Lakehouse for Apache Iceberg.

Data and resources are stale or not updating

Cross-cloud Lakehouse federated catalogs synchronize metadata from a remote cloud based on a refresh interval. A catalog's background metadata refresh may take longer the more resources there are (shares, schemas, namespaces, tables). If the previous refresh overruns, the current refresh will be skipped, but the next refresh will be rescheduled at the following interval.

If queried data appears stale, the catalog's background metadata refresh may be overrunning or failing. Remote cloud resources are also subject to the same behavior. If you delete a resource in the remote cloud, then it will only be deleted from Lakehouse on the next successful background metadata refresh.

Check the background metadata refresh status on a federated catalog in the Lakehouse Google Cloud console page, or by running the gcloud CLI:

Go to Lakehouse

gcloud alpha biglake iceberg catalogs describe FEDERATED_CATALOG_NAME \
    --project="PROJECT_ID"

gcloud alpha biglake delta-sharing catalogs describe FEDERATED_CATALOG_NAME \
    --project="PROJECT_ID"

Connectivity or routing issues

If you encounter connectivity or routing issues when you use a private interconnect, verify the following:

  • Verify route propagation: Check the Cloud Router in Google Cloud to verify that it has learned the AWS VPC prefixes. Check your AWS route tables to verify that they have routes back to your Google Cloud VPC.
  • Check ILB health: In the Google Cloud console, go to Network Services > Load balancing. Check whether the backends (NEGs) for your ILB backend service are healthy. If they aren't, verify network connectivity and AWS Security Group rules.
  • Test connectivity from Google Cloud: Launch a test VM instance in the same Google Cloud VPC and subnet as the ILB or Service Directory endpoint and try to connect to the AWS ENI IP addresses on port 443 (for example, by using curl or telnet).
  • Service Directory resolution: Verify that the catalog's service account has permissions to resolve Service Directory endpoints (roles/servicedirectory.viewer, roles/servicedirectory.pscAuthorizedService).
  • Security Groups and firewall rules: Verify that Google Cloud firewall rules and AWS Security Groups allow traffic on TCP port 443 between the relevant IP ranges.