The Lakehouse runtime catalog lets you share data across Apache Spark,
Apache Hive, and BigQuery by providing a serverless, unified
metastore. This fully managed metadata layer eliminates the need for separate,
self-hosted Hive metastores for open source workloads.

Optimized for Apache Spark `ExternalCatalog` compatibility, this integration
supports a subset of the Hive Metastore interface. To determine whether your
workloads depend on unsupported features such as transactions, compactions, or
Kerberos, review the [feature comparison](https://docs.cloud.google.com/lakehouse/docs/about-spark-hive-metastore.md.txt#feature-compare-hive) and
[limitations](https://docs.cloud.google.com/lakehouse/docs/spark-hive-limitations.md.txt).

## How Hive integrates with the Lakehouse runtime catalog

The Hive metastore endpoint manages standard Apache Hive and Spark tables
(using Hive SerDes or Spark data sources) rather than Apache Iceberg tables.
To simplify connecting your Spark jobs to this endpoint, Managed Service for Apache Spark
images are preconfigured with the necessary client libraries and
dependencies.

The following sequence describes how Spark connects to the metastore:

1. Apache Spark connects to external metadata catalogs by using the standard Apache Hive `IMetastoreClient` interface.
2. The Lakehouse runtime catalog implements a custom `IMetastoreClient` to provide a fully managed metastore service for your Spark and Hive metadata.
3. Preconfigured Managed Service for Apache Spark runtimes automatically use this custom client to route metadata operations directly to the metastore.

After setup, you can query your Spark-created tables directly in
BigQuery. This integration supports various storage
formats---such as Parquet, ORC, Avro, and Delta Lake---along with specific data type
mappings between Spark and BigQuery, and
[customer-managed encryption keys (CMEK)](https://docs.cloud.google.com/storage/docs/encryption/customer-managed-keys.md.txt) for data stored in
Cloud Storage.

## How the Hive catalog endpoint integrates with Google Cloud services

To understand how borderless Lakehouse manages your data, see how the
Hive catalog architecture integrates with Google Cloud services.
Rather than maintaining self-hosted metastores, your open source workloads
connect to the Lakehouse runtime catalog to manage Hive
database and table definitions, while storing the underlying data files
directly in Cloud Storage warehouse directories.

The following diagram illustrates how compute engines such as Managed Service for Apache Spark
use the Lakehouse runtime catalog to manage table metadata while
reading and writing underlying data files directly in Hive.
![Components of a Lakehouse architecture, including Managed Service for Apache Spark, Cloud Storage, and the Apache Hive catalog.](https://docs.cloud.google.com/static/lakehouse/images/hivecatalog-lakehouse.png) Lakehouse Hive catalog architecture diagram.

## Feature comparison with Hive Metastore

The following table compares entities and operations in Hive Metastore and
Lakehouse:

| Entity or operation | Hive Metastore | Lakehouse runtime catalog |
|---|---|---|
| Catalog | ✅ | ✅ |
| Database (create, delete, update) | ✅ | ✅ |
| Table (create, delete, update) | ✅ | ✅ |
| Partition (add, drop, update) | ✅ | ✅ |
| Table privileges | ✅ | ✅ (through Identity and Access Management (IAM)) |
| Partition privileges | ✅ | ✅ (through IAM) |
| User-defined functions | ✅ | Not supported |
| Bucketing / Skewing columns | ✅ | Not supported |
| Table and partition column stats | ✅ | Not supported |
| Key constraints | ✅ | Not supported |
| Delegation tokens (Kerberos) | ✅ | Not supported |
| Column privileges | ✅ | Not supported |
| Roles | ✅ | Not supported |
| Workload manager resource plans | ✅ | Not supported |
| Transactions and compactions | ✅ | Not supported |

## Migrate from Dataproc Metastore

To upgrade and migrate your metadata from Dataproc Metastore to the
Lakehouse runtime catalog, see [Migrate metadata from
Dataproc Metastore to
Lakehouse](https://docs.cloud.google.com/lakehouse/docs/migrate-dataproc-metastore).

## What's next

- [Use Spark and Hive with the Lakehouse runtime catalog](https://docs.cloud.google.com/lakehouse/docs/use-spark-hive-metastore.md.txt).
- [Set up Spark with Delta tables](https://docs.cloud.google.com/lakehouse/docs/use-spark-delta-tables.md.txt).
- [Use Lakehouse regional endpoints](https://docs.cloud.google.com/lakehouse/docs/regional-endpoints.md.txt).