# Query federated catalogs with Spark and Lakehouse runtime catalog

> [!WARNING]
>
> **Preview**
>
>
> This feature is
>
> subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the
> [Service Specific
> Terms](https://docs.cloud.google.com/terms/service-terms#1).
>
> Pre-GA features are available "as is" and might have limited support.
>
> For more information, see the
> [launch stage descriptions](https://cloud.google.com/products/#product-launch-stages).

After you set up cross-cloud data access in Lakehouse runtime catalog, you can query
remote Iceberg tables from Managed Service for Apache Spark without
migrating files or managing separate cloud storage credentials.

Lakehouse runtime catalog handles metadata synchronization and secure data access
through [credential vending](https://docs.cloud.google.com/lakehouse/docs/credential-vending.md.txt). When Spark queries a federated
table, Lakehouse runtime catalog translates the catalog metadata and issues
short-lived, scoped credentials for Spark to read data from Amazon S3 or
Cloud Storage, treating the tables as if they were local to your
Google Cloud environment.

## Supported federated catalog types

Managed Service for Apache Spark supports querying the following federated catalog
types through the Lakehouse runtime catalog Iceberg REST Catalog
endpoint:

| Federated catalog type | Supported storage backends | Setup guide in Lakehouse runtime catalog |
|:---|:---|:---|
| **AWS Glue Data Catalog** | Amazon S3 (`s3://`---default AWS account catalog or Amazon S3 table bucket) | [Connect to AWS Glue](https://docs.cloud.google.com/lakehouse/docs/set-up-cross-cloud-connection-aws-glue.md.txt) |
| **Databricks Unity Catalog** | Amazon S3 (`s3://`) or Cloud Storage (`gs://`) external locations | [Connect to Databricks Unity Catalog](https://docs.cloud.google.com/lakehouse/docs/set-up-cross-cloud-connection-databricks.md.txt) |
| **Snowflake Horizon Catalog** | Amazon S3 (`s3://`) or Cloud Storage (`gs://`) | [Connect to Snowflake Horizon Catalog](https://docs.cloud.google.com/lakehouse/docs/set-up-cross-cloud-connection-snowflake.md.txt) |

To learn about the catalog readiness requirements and underlying Spark parameter
definitions for each storage provider, see
[Query remote data in Lakehouse runtime catalog](https://docs.cloud.google.com/lakehouse/docs/query-remote-data.md.txt) and
[Spark Configuration Parameters](https://docs.cloud.google.com/lakehouse/docs/query-remote-data.md.txt#spark-parameters).

## Before you begin

1. Ensure that you have Iceberg tables in your remote catalog.
2. Set up a cross-cloud connection and federated catalog in Lakehouse runtime catalog for your remote catalog provider:
   - [AWS Glue Data Catalog](https://docs.cloud.google.com/lakehouse/docs/set-up-cross-cloud-connection-aws-glue.md.txt)
   - [Databricks Unity Catalog](https://docs.cloud.google.com/lakehouse/docs/set-up-cross-cloud-connection-databricks.md.txt)
   - [Snowflake Horizon Catalog](https://docs.cloud.google.com/lakehouse/docs/set-up-cross-cloud-connection-snowflake.md.txt)

<br />

#### Required Identity and Access Management (IAM) roles

Certain IAM roles are required to run the examples on this
page. Depending on organization policies, these roles might have already been
granted. To check role grants, see
[Do you need to grant roles?](https://docs.cloud.google.com/managed-spark/docs/concepts/iam/iam.md.txt#do_you_need_to_grant_roles).
For more information about granting roles, see
[Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).

The roles that you need depend on how your workload runs.
Managed Service for Apache Spark clusters and
Managed Service for Apache Spark serverless runtimes 2.3 and earlier run workloads
as a service account. Managed Service for Apache Spark serverless runtimes 3.0 and
later run workloads with your end-user credentials (EUC) by default, so you
grant data access roles to your user account instead of a service account. If
you specify a custom service account for a 3.0 or later runtime, the workload
runs with service account credentials. For more information, see
[Serverless deployments roles](https://docs.cloud.google.com/managed-spark/docs/concepts/iam-serverless.md.txt#s8s-role-requirements).

##### User roles


To get the permissions that
you need to create a Managed Service for Apache Spark cluster, submit a batch workload, or create an interactive session,

ask your administrator to grant you the
following IAM roles:

- Use a cluster or a serverless runtime 2.3 or earlier:
  - [Dataproc Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataproc.md.txt#dataproc.editor) (`roles/dataproc.editor`) on the project
  - [Service Account User](https://docs.cloud.google.com/iam/docs/roles-permissions/iam.md.txt#iam.serviceAccountUser) (`roles/iam.serviceAccountUser`) on the Compute Engine default service account or a custom service account
- Use a serverless runtime 3.0 or later: [Dataproc Serverless Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataproc.md.txt#dataproc.serverlessEditor) (`roles/dataproc.serverlessEditor`) on the project
- Read federated tables with end-user credentials on a serverless runtime 3.0 or later: [BigLake Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/biglake.md.txt#biglake.viewer) (`roles/biglake.viewer`) on the project that contains the federated catalog
- Use service account credentials on a serverless runtime 3.0 or later: [Service Account User](https://docs.cloud.google.com/iam/docs/roles-permissions/iam.md.txt#iam.serviceAccountUser) (`roles/iam.serviceAccountUser`) on the custom service account


##### Service account roles

Grant the following roles to the service account that runs your workload.
Managed Service for Apache Spark serverless runtimes 3.0 and later need these roles
only if the workload runs with service account credentials.


To ensure that the Compute Engine default service account or a custom service account has the necessary
permissions to run Managed Service for Apache Spark jobs and read federated tables,

ask your administrator to grant the
following IAM roles to the Compute Engine default service account or a custom service account:

**Important:** You must grant these roles to the Compute Engine default service account or a custom service account, *not* to your user account. Failure to grant the roles to the correct principal might result in permission errors.

- Run Managed Service for Apache Spark jobs on a cluster or a serverless runtime 2.3 or earlier: [Dataproc Worker](https://docs.cloud.google.com/iam/docs/roles-permissions/dataproc.md.txt#dataproc.worker) (`roles/dataproc.worker`) on the project
- Run Managed Service for Apache Spark jobs on a serverless runtime 3.0 or later:
  - [Dataproc Serverless Node](https://docs.cloud.google.com/iam/docs/roles-permissions/dataproc.md.txt#dataproc.serverlessNode) (`roles/dataproc.serverlessNode`) on the project
  - [Dataproc Resource Manager Node Service Agent](https://docs.cloud.google.com/iam/docs/roles-permissions/dataprocrm.md.txt#dataprocrm.nodeServiceAgent) (`roles/dataprocrm.nodeServiceAgent`) on the project
- Discover and read federated table metadata at runtime: [BigLake Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/biglake.md.txt#biglake.viewer) (`roles/biglake.viewer`) on the project that contains the federated catalog


To ensure that the Managed Service for Apache Spark Service Agent (`service-PROJECT_NUMBER@dataproc-accounts.iam.gserviceaccount.com`) has the necessary
permissions to resolve the `dataproc.lakehouse.catalog.` simplified configuration property when you create a cluster, submit a batch workload, or create an interactive session,

ask your administrator to grant the
[BigLake Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/biglake.md.txt#biglake.viewer) (`roles/biglake.viewer`) IAM role to the Managed Service for Apache Spark Service Agent (`service-PROJECT_NUMBER@dataproc-accounts.iam.gserviceaccount.com`) on the project that contains the federated catalog.


**Important:** You must grant this role to the Managed Service for Apache Spark Service Agent (`service-PROJECT_NUMBER@dataproc-accounts.iam.gserviceaccount.com`), *not* to your user account. Failure to grant the role to the correct principal might result in permission errors.

<br />

#### Configure outbound network connectivity for Amazon S3

When you query federated catalogs backed by Amazon S3,
Managed Service for Apache Spark serverless batch workloads and interactive
sessions, and internal-IP
Managed Service for Apache Spark clusters can't connect to
Amazon S3 endpoints with their default
[network configuration](https://docs.cloud.google.com/managed-spark/docs/concepts/network-serverless.md.txt).
Provision a Cloud Router and Cloud NAT on your workload VPC network to
allow outbound traffic to Amazon S3:

```bash
gcloud compute routers create lakehouse-router \
    --network=NETWORK_NAME \
    --region=REGION

gcloud compute routers nats create lakehouse-nat \
    --router=lakehouse-router \
    --auto-allocate-nat-external-ips \
    --nat-all-subnet-ip-ranges \
    --region=REGION
```

Replace the following:

- `NETWORK_NAME`: the VPC network for your Managed Service for Apache Spark cluster, batch workload, or interactive session, for example, `default`.
- `REGION`: the Google Cloud region for your Managed Service for Apache Spark cluster, batch workload, or interactive session.

## Configure Managed Service for Apache Spark and query federated tables

You can configure a Managed Service for Apache Spark cluster, serverless batch
workload, or interactive session to connect to your federated catalog in
Lakehouse runtime catalog. These Managed Service for Apache Spark resources support the
`dataproc.lakehouse.catalog.CATALOG_NAME` simplified
configuration property (recommended), which reads the federated catalog
definition from Lakehouse runtime catalog and generates the required
Iceberg REST Catalog Spark properties, including the
storage `io-impl` and credential-vending headers (the cluster form of the
property adds the `dataproc:` prefix).

Both simplified configuration and manual configuration use Google authorization
for Iceberg (`GoogleAuthManager`), which requires one of
the following cluster image or serverless runtime versions:

- Managed Service for Apache Spark clusters: 2.2 image versions 2.2.65 and later, or 2.3 image versions 2.3.11 and later.
- Managed Service for Apache Spark serverless batch workloads and interactive sessions: 2.2 runtimes 2.2.60 and later, 2.3 runtimes 2.3.10 and later, or 3.0 runtimes.

For serverless batch workloads and interactive sessions, the required roles
depend on the runtime version. Runtimes 3.0 and later use end-user credentials
by default. For details, see
[Required IAM roles](https://docs.cloud.google.com/managed-spark/docs/guides/query-federated-catalogs.md.txt#required_roles) and
[Serverless deployments roles](https://docs.cloud.google.com/managed-spark/docs/concepts/iam-serverless.md.txt#s8s-role-requirements).

### Cluster

Create a Managed Service for Apache Spark cluster with the Iceberg optional
component using simplified configuration properties (recommended), or by
manually specifying catalog properties.

#### Simplified configuration using properties (recommended)

Create the cluster with the federated catalog property:

```bash
gcloud dataproc clusters create CLUSTER_NAME \
  --project=PROJECT_ID \
  --region=REGION \
  --image-version=IMAGE_VERSION \
  --optional-components=ICEBERG,JUPYTER \
  --enable-component-gateway \
  --properties="dataproc:dataproc.lakehouse.catalog.CATALOG_NAME=projects/PROJECT_ID/catalogs/FEDERATED_CATALOG_NAME,spark:spark.sql.defaultCatalog=CATALOG_NAME"
```

Replace the following:

- `CLUSTER_NAME`: a name for your cluster.
- `PROJECT_ID`: your Google Cloud project ID.
- `REGION`: the Google Cloud region for the cluster, for example, `us-central1`.
- `IMAGE_VERSION`: the Managed Service for Apache Spark image version---for example, `2.3-debian12`. For supported versions, see the requirements earlier in this section.
- `CATALOG_NAME`: a name for the local Spark catalog, for example, `federated_catalog`. It can be the same as `FEDERATED_CATALOG_NAME`.
- `FEDERATED_CATALOG_NAME`: the ID of the federated catalog that you created in [Before you begin](https://docs.cloud.google.com/managed-spark/docs/guides/query-federated-catalogs.md.txt#before-you-begin).

> [!NOTE]
> **Note:** You can pass multiple comma-separated catalogs to the `--properties` flag. Each cluster catalog property requires the `dataproc:` prefix.

#### Querying the Snowflake Horizon Catalog

If you are querying a Snowflake Horizon Catalog, note the
following:

- **Snowflake storage:** Simplified configuration supports
  Snowflake tables stored in Amazon S3. For Snowflake tables
  stored in Cloud Storage, use
  [manual configuration](https://docs.cloud.google.com/managed-spark/docs/guides/query-federated-catalogs.md.txt#configure-cluster-manual) with
  `io-impl` set to
  `org.apache.iceberg.gcp.gcs.GCSFileIO`.

- **Snowflake `STRING` columns:** If you encounter errors
  querying Snowflake `STRING` columns where the table's storage
  serialization policy in Snowflake isn't `COMPATIBLE`, you can
  change the storage serialization policy to `COMPATIBLE` in
  Snowflake or disable Parquet vectorization in Spark by adding the
  following properties to the `--properties` flag:

  ```bash
  spark:spark.sql.iceberg.vectorization.enabled=false,spark:spark.sql.catalog.CATALOG_NAME.table-override.read.parquet.vectorization.enabled=false
      
  ```

In your Jupyter notebook or PySpark application, create the `SparkSession`
without specifying catalog configs:

```python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("APP_NAME").getOrCreate()
```

Replace `APP_NAME` with the name of
your Spark application.

#### Manual configuration

If you don't use the simplified configuration property, create the cluster
as previously described in this section, without passing the
`dataproc:dataproc.lakehouse.catalog.` property to the
`--properties` flag, and then configure your
`SparkSession` manually. For a list of storage provider parameter
values, see
[Spark Configuration Parameters](https://docs.cloud.google.com/lakehouse/docs/query-remote-data.md.txt#spark-parameters).

```python
from pyspark.sql import SparkSession

catalog_name = "CATALOG_NAME"

spark = SparkSession.builder.appName("APP_NAME") \
.config('spark.sql.defaultCatalog', catalog_name) \
.config(f'spark.sql.catalog.{catalog_name}', 'org.apache.iceberg.spark.SparkCatalog') \
.config(f'spark.sql.catalog.{catalog_name}.type', 'rest') \
.config(f'spark.sql.catalog.{catalog_name}.uri', 'https://biglake.googleapis.com/iceberg/v1/restcatalog') \
.config(f'spark.sql.catalog.{catalog_name}.warehouse', 'bl://projects/PROJECT_ID/catalogs/FEDERATED_CATALOG_NAME') \
.config(f'spark.sql.catalog.{catalog_name}.header.x-goog-user-project', 'PROJECT_ID') \
.config(f'spark.sql.catalog.{catalog_name}.rest.auth.type', 'org.apache.iceberg.gcp.auth.GoogleAuthManager') \
.config(f'spark.sql.catalog.{catalog_name}.header.X-Iceberg-Access-Delegation', 'vended-credentials') \
.config(f'spark.sql.catalog.{catalog_name}.io-impl', 'IO_IMPL') \
.config(f'spark.sql.catalog.{catalog_name}.s3.client-factory-impl', 'org.apache.iceberg.aws.s3.DefaultS3FileIOAwsClientFactory') \
.config('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions') \
.getOrCreate()
```

Replace `IO_IMPL` with the
`FileIO` implementation for the storage that contains your
tables:

- Tables in Amazon S3: `org.apache.iceberg.aws.s3.S3FileIO`
- Tables in Cloud Storage: `org.apache.iceberg.gcp.gcs.GCSFileIO`

The `s3.client-factory-impl` property applies only to tables in
Amazon S3. Omit this line for Cloud Storage tables.

### Serverless

Submit a PySpark batch workload to Managed Service for Apache Spark serverless
using simplified configuration properties (recommended), or by manually
specifying catalog properties.

#### Simplified configuration using properties (recommended)

Pass the federated catalog property without the `dataproc:` prefix when you
submit the batch workload:

```bash
gcloud dataproc batches submit pyspark PYSPARK_FILE \
  --project=PROJECT_ID \
  --region=REGION \
  --version=RUNTIME_VERSION \
  --properties="dataproc.lakehouse.catalog.CATALOG_NAME=projects/PROJECT_ID/catalogs/FEDERATED_CATALOG_NAME,spark.sql.defaultCatalog=CATALOG_NAME"
```

Replace the following:

- `PYSPARK_FILE`: the `gs://` Cloud Storage path to your PySpark application file.
- `PROJECT_ID`: your Google Cloud project ID.
- `REGION`: the region for the batch workload.
- `RUNTIME_VERSION`: the Managed Service for Apache Spark serverless runtime version, for example, `3.0`. For supported versions, see the requirements earlier in this section. The [required IAM roles](https://docs.cloud.google.com/managed-spark/docs/guides/query-federated-catalogs.md.txt#required_roles) depend on the runtime version.
- `CATALOG_NAME`: a name for the local Spark catalog, for example, `federated_catalog`. It can be the same as `FEDERATED_CATALOG_NAME`.
- `FEDERATED_CATALOG_NAME`: the ID of the federated catalog that you created in [Before you begin](https://docs.cloud.google.com/managed-spark/docs/guides/query-federated-catalogs.md.txt#before-you-begin).

> [!NOTE]
> **Note:** You can pass multiple comma-separated catalogs to the `--properties` flag.

#### Querying the Snowflake Horizon Catalog

If you are querying a Snowflake Horizon Catalog, note the
following:

- **Snowflake storage:** Simplified configuration supports
  Snowflake tables stored in Amazon S3. For Snowflake tables
  stored in Cloud Storage, use
  [manual configuration](https://docs.cloud.google.com/managed-spark/docs/guides/query-federated-catalogs.md.txt#configure-batch-manual) with
  `io-impl` set to
  `org.apache.iceberg.gcp.gcs.GCSFileIO`.

- **Snowflake `STRING` columns:** If you encounter errors
  querying Snowflake `STRING` columns where the table's storage
  serialization policy in Snowflake isn't `COMPATIBLE`, you can
  change the storage serialization policy to `COMPATIBLE` in
  Snowflake or disable Parquet vectorization in Spark by adding the
  following properties to the `--properties` flag:

  ```bash
  spark.sql.iceberg.vectorization.enabled=false,spark.sql.catalog.CATALOG_NAME.table-override.read.parquet.vectorization.enabled=false
      
  ```

In the PySpark application file, create the `SparkSession` without
specifying catalog configs:

```python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("APP_NAME").getOrCreate()
```

#### Manual configuration

If you don't use the simplified configuration property, specify the catalog
properties with the `--properties` flag of the
`gcloud dataproc batches submit` command. For a list of storage
provider parameter values, see
[Spark Configuration Parameters](https://docs.cloud.google.com/lakehouse/docs/query-remote-data.md.txt#spark-parameters).

```bash
gcloud dataproc batches submit pyspark PYSPARK_FILE \
  --project=PROJECT_ID \
  --region=REGION \
  --version=RUNTIME_VERSION \
  --properties="\
spark.sql.defaultCatalog=CATALOG_NAME,\
spark.sql.catalog.CATALOG_NAME=org.apache.iceberg.spark.SparkCatalog,\
spark.sql.catalog.CATALOG_NAME.type=rest,\
spark.sql.catalog.CATALOG_NAME.uri=https://biglake.googleapis.com/iceberg/v1/restcatalog,\
spark.sql.catalog.CATALOG_NAME.warehouse=bl://projects/PROJECT_ID/catalogs/FEDERATED_CATALOG_NAME,\
spark.sql.catalog.CATALOG_NAME.header.x-goog-user-project=PROJECT_ID,\
spark.sql.catalog.CATALOG_NAME.rest.auth.type=org.apache.iceberg.gcp.auth.GoogleAuthManager,\
spark.sql.catalog.CATALOG_NAME.header.X-Iceberg-Access-Delegation=vended-credentials,\
spark.sql.catalog.CATALOG_NAME.io-impl=IO_IMPL,\
spark.sql.catalog.CATALOG_NAME.s3.client-factory-impl=org.apache.iceberg.aws.s3.DefaultS3FileIOAwsClientFactory,\
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
```

Replace `IO_IMPL` with the
`FileIO` implementation for the storage that contains your
tables:

- Tables in Amazon S3: `org.apache.iceberg.aws.s3.S3FileIO`
- Tables in Cloud Storage: `org.apache.iceberg.gcp.gcs.GCSFileIO`

The `s3.client-factory-impl` property applies only to tables in
Amazon S3. Omit this property for Cloud Storage tables.

### Serverless

Create a Managed Service for Apache Spark serverless interactive session in a
notebook through the
[`ManagedSparkSession`](https://docs.cloud.google.com/managed-spark/docs/guides/spark-connect-client.md.txt)
API using simplified configuration properties (recommended) or by manually
specifying catalog properties.

#### Simplified configuration using properties (recommended)

In a notebook that has the `google-cloud-spark-connect` package installed,
such as a [Google Cloud Data Agent Kit](https://docs.cloud.google.com/data-agent-kit/overview.md.txt) notebook or a
Vertex AI Workbench notebook, pass the federated catalog property
without the `dataproc:` prefix when you create the session:

```python
from google.cloud.managed_spark_connect import ManagedSparkSession
from google.cloud.dataproc_v1 import Session

catalog_name = "CATALOG_NAME"

session_config = Session()
session_config.runtime_config.version = 'RUNTIME_VERSION'

spark = ManagedSparkSession.builder.sessionId("SESSION_ID") \
.sessionConfig(session_config) \
.config(f'dataproc.lakehouse.catalog.{catalog_name}', 'projects/PROJECT_ID/catalogs/FEDERATED_CATALOG_NAME') \
.config('spark.sql.defaultCatalog', catalog_name) \
.getOrCreate()
```

Replace the following:

- `CATALOG_NAME`: a name for the local Spark catalog, for example, `federated_catalog`. It can be the same as `FEDERATED_CATALOG_NAME`.
- `RUNTIME_VERSION`: the Managed Service for Apache Spark serverless runtime version. `google-cloud-spark-connect` requires runtime `3.0` or later. The [required IAM roles](https://docs.cloud.google.com/managed-spark/docs/guides/query-federated-catalogs.md.txt#required_roles) depend on the runtime version.
- `SESSION_ID`: a custom ID for your Spark session, for example, `my-spark-session`.
- `PROJECT_ID`: your Google Cloud project ID.
- `FEDERATED_CATALOG_NAME`: the ID of the federated catalog that you created in [Before you begin](https://docs.cloud.google.com/managed-spark/docs/guides/query-federated-catalogs.md.txt#before-you-begin).

You can also configure a federated catalog in a
[runtime profile (session template)](https://docs.cloud.google.com/managed-spark/docs/guides/create-serverless-sessions-templates.md.txt) by selecting
**Lakehouse runtime catalog** under **Additional configuration** \>
**Metastore configuration** on the **Create a runtime profile** page in the
Google Cloud console, or by setting these properties in a session template using
the gcloud CLI or Dataproc API.

#### Querying the Snowflake Horizon Catalog

If you are querying a Snowflake Horizon Catalog, note the
following:

- **Snowflake storage:** Simplified configuration supports
  Snowflake tables stored in Amazon S3. For Snowflake tables
  stored in Cloud Storage, use
  [manual configuration](https://docs.cloud.google.com/managed-spark/docs/guides/query-federated-catalogs.md.txt#configure-session-manual) with
  `io-impl` set to
  `org.apache.iceberg.gcp.gcs.GCSFileIO`.

- **Snowflake `STRING` columns:** If you encounter errors
  querying Snowflake `STRING` columns where the table's storage
  serialization policy in Snowflake isn't `COMPATIBLE`, you can
  change the storage serialization policy to `COMPATIBLE` in
  Snowflake or disable Parquet vectorization in Spark by adding the
  following `.config()` calls to the session builder:

  ```python
  .config('spark.sql.iceberg.vectorization.enabled', 'false') \
  .config(f'spark.sql.catalog.{catalog_name}.table-override.read.parquet.vectorization.enabled', 'false')
      
  ```

#### Manual configuration

If you don't use the simplified configuration property, specify the catalog
properties when you create the session. For a list of storage provider
parameter values, see
[Spark Configuration Parameters](https://docs.cloud.google.com/lakehouse/docs/query-remote-data.md.txt#spark-parameters).

```python
from google.cloud.managed_spark_connect import ManagedSparkSession
from google.cloud.dataproc_v1 import Session

catalog_name = "CATALOG_NAME"

session_config = Session()
session_config.runtime_config.version = 'RUNTIME_VERSION'

spark = ManagedSparkSession.builder.sessionId("SESSION_ID") \
.sessionConfig(session_config) \
.config('spark.sql.defaultCatalog', catalog_name) \
.config(f'spark.sql.catalog.{catalog_name}', 'org.apache.iceberg.spark.SparkCatalog') \
.config(f'spark.sql.catalog.{catalog_name}.type', 'rest') \
.config(f'spark.sql.catalog.{catalog_name}.uri', 'https://biglake.googleapis.com/iceberg/v1/restcatalog') \
.config(f'spark.sql.catalog.{catalog_name}.warehouse', 'bl://projects/PROJECT_ID/catalogs/FEDERATED_CATALOG_NAME') \
.config(f'spark.sql.catalog.{catalog_name}.header.x-goog-user-project', 'PROJECT_ID') \
.config(f'spark.sql.catalog.{catalog_name}.rest.auth.type', 'org.apache.iceberg.gcp.auth.GoogleAuthManager') \
.config(f'spark.sql.catalog.{catalog_name}.header.X-Iceberg-Access-Delegation', 'vended-credentials') \
.config(f'spark.sql.catalog.{catalog_name}.io-impl', 'IO_IMPL') \
.config(f'spark.sql.catalog.{catalog_name}.s3.client-factory-impl', 'org.apache.iceberg.aws.s3.DefaultS3FileIOAwsClientFactory') \
.config('spark.sql.extensions', 'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions') \
.getOrCreate()
```

Replace `RUNTIME_VERSION` as
described in the simplified configuration example earlier in this tab.

Replace `IO_IMPL` with the
`FileIO` implementation for the storage that contains your
tables:

- Tables in Amazon S3: `org.apache.iceberg.aws.s3.S3FileIO`
- Tables in Cloud Storage: `org.apache.iceberg.gcp.gcs.GCSFileIO`

The `s3.client-factory-impl` property applies only to tables in
Amazon S3. Omit this line for Cloud Storage tables.

## Query federated tables

After you configure your Spark session, you can query tables in your federated
catalog using PySpark or Spark SQL.

> [!NOTE]
> **Note:** The `dataproc.lakehouse.catalog.` property registers the catalog with Spark, but it doesn't set it as the *default* catalog. The previous examples set the serverless `spark.sql.defaultCatalog` or the `spark:spark.sql.defaultCatalog` cluster property, which lets you use unqualified table names. If you omit these properties, you must qualify table names with the catalog name, as shown in the following examples.

1. Read a federated table using the PySpark DataFrame API:

   ```python
   df = spark.table("CATALOG_NAME.NAMESPACE_NAME.TABLE_NAME")
   df.show(10, truncate=False)
   ```

   Replace the following:
   - `CATALOG_NAME`: the local Spark catalog name. If you set `spark.sql.defaultCatalog` to `CATALOG_NAME`, you can omit this prefix.
   - `NAMESPACE_NAME`: the namespace (database or schema) in the federated catalog.
   - `TABLE_NAME`: the table name in the federated catalog.
2. Query the federated table using Spark SQL:

   ```python
   spark.sql("""
       SELECT *
       FROM CATALOG_NAME.NAMESPACE_NAME.TABLE_NAME
       LIMIT 10
   """).show()
   ```

### Join or materialize data across federated and local catalogs

Federated catalogs are read-only. To join remote cross-cloud data with local
tables or materialize federated query results into Cloud Storage, attach
your federated catalog and a read-write catalog in Lakehouse runtime catalog using
comma-separated `dataproc.lakehouse.catalog.` properties:

### Cluster

Managed Service for Apache Spark cluster example:

```bash
--properties="dataproc:dataproc.lakehouse.catalog.FEDERATED_SPARK_CATALOG=projects/PROJECT_ID/catalogs/FEDERATED_CATALOG_NAME,dataproc:dataproc.lakehouse.catalog.LOCAL_SPARK_CATALOG=projects/PROJECT_ID/catalogs/LOCAL_CATALOG_ID,spark:spark.sql.defaultCatalog=LOCAL_SPARK_CATALOG"
```

### Serverless

Managed Service for Apache Spark serverless batch workload example:

```bash
--properties="dataproc.lakehouse.catalog.FEDERATED_SPARK_CATALOG=projects/PROJECT_ID/catalogs/FEDERATED_CATALOG_NAME,dataproc.lakehouse.catalog.LOCAL_SPARK_CATALOG=projects/PROJECT_ID/catalogs/LOCAL_CATALOG_ID,spark.sql.defaultCatalog=LOCAL_SPARK_CATALOG"
```

### Serverless

Managed Service for Apache Spark serverless interactive session example:

```python
from google.cloud.managed_spark_connect import ManagedSparkSession
from google.cloud.dataproc_v1 import Session

session_config = Session()
session_config.runtime_config.version = 'RUNTIME_VERSION'

spark = ManagedSparkSession.builder.sessionId("SESSION_ID") \
  .sessionConfig(session_config) \
  .config('dataproc.lakehouse.catalog.FEDERATED_SPARK_CATALOG', 'projects/PROJECT_ID/catalogs/FEDERATED_CATALOG_NAME') \
  .config('dataproc.lakehouse.catalog.LOCAL_SPARK_CATALOG', 'projects/PROJECT_ID/catalogs/LOCAL_CATALOG_ID') \
  .config('spark.sql.defaultCatalog', 'LOCAL_SPARK_CATALOG') \
  .getOrCreate()
```

Replace `RUNTIME_VERSION` as
described in the interactive session example earlier on this page.

After setting these properties, you can query the federated catalog and write
the results into a table in your local Lakehouse runtime catalog.

The principal that runs your workload is the service account for clusters,
serverless runtimes 2.3 and earlier, and serverless runtimes 3.0 and later with
service account credentials. For serverless runtimes 3.0 and later with
end-user credentials (the default), the principal is your user account.


To ensure that the principal that runs your workload has the necessary
permissions to write table data into your local catalog,

ask your administrator to grant the
following IAM roles to the principal that runs your workload:

- Write table data in credential vending mode: [BigLake Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/biglake.md.txt#biglake.editor) (`roles/biglake.editor`) on the project that contains the local catalog
- Write table data in non-credential vending mode:
  - [BigLake Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/biglake.md.txt#biglake.editor) (`roles/biglake.editor`) on the project that contains the local catalog
  - [Storage Object User](https://docs.cloud.google.com/iam/docs/roles-permissions/storage.md.txt#storage.objectUser) (`roles/storage.objectUser`) on all Cloud Storage buckets associated with the local catalog


The following example creates a table in your local catalog from a federated
table:

```python
spark.sql("""
    CREATE TABLE IF NOT EXISTS LOCAL_SPARK_CATALOG.LOCAL_NAMESPACE.materialized_table
    USING iceberg
    AS SELECT *
    FROM FEDERATED_SPARK_CATALOG.REMOTE_NAMESPACE.REMOTE_TABLE
""")
```

## What's next

- [Create a lakehouse with Spark and Lakehouse runtime catalog](https://docs.cloud.google.com/managed-spark/docs/guides/create-lakehouse.md.txt).
- Learn more about [querying remote data in Lakehouse runtime catalog](https://docs.cloud.google.com/lakehouse/docs/query-remote-data.md.txt).
- Read [About cross-cloud data access](https://docs.cloud.google.com/lakehouse/docs/about-cross-cloud-data-access.md.txt).
- Learn about [credential vending](https://docs.cloud.google.com/lakehouse/docs/credential-vending.md.txt).