About the Hive catalog endpoint

The Lakehouse runtime catalog lets you share data across Apache Spark, Apache Hive, and BigQuery by providing a serverless, unified metastore. This fully managed metadata layer eliminates the need for separate, self-hosted Hive metastores for open source workloads.

Optimized for Apache Spark ExternalCatalog compatibility, this integration supports a subset of the Hive Metastore interface. To determine whether your workloads depend on unsupported features such as transactions, compactions, or Kerberos, review the feature comparison and limitations.

How Hive integrates with the Lakehouse runtime catalog

The Hive metastore endpoint manages standard Apache Hive and Spark tables (using Hive SerDes or Spark data sources) rather than Apache Iceberg tables. To simplify connecting your Spark jobs to this endpoint, Managed Service for Apache Spark images are preconfigured with the necessary client libraries and dependencies.

The following sequence describes how Spark connects to the metastore:

  1. Apache Spark connects to external metadata catalogs by using the standard Apache Hive IMetastoreClient interface.
  2. The Lakehouse runtime catalog implements a custom IMetastoreClient to provide a fully managed metastore service for your Spark and Hive metadata.
  3. Preconfigured Managed Service for Apache Spark runtimes automatically use this custom client to route metadata operations directly to the metastore.

After setup, you can query your Spark-created tables directly in BigQuery. This integration supports various storage formats—such as Parquet, ORC, Avro, and Delta Lake—along with specific data type mappings between Spark and BigQuery, and customer-managed encryption keys (CMEK) for data stored in Cloud Storage.

How the Hive catalog endpoint integrates with Google Cloud services

To understand how borderless Lakehouse manages your data, see how the Hive catalog architecture integrates with Google Cloud services. Rather than maintaining self-hosted metastores, your open source workloads connect to the Lakehouse runtime catalog to manage Hive database and table definitions, while storing the underlying data files directly in Cloud Storage warehouse directories.

The following diagram illustrates how compute engines such as Managed Service for Apache Spark use the Lakehouse runtime catalog to manage table metadata while reading and writing underlying data files directly in Hive.

Components of a Lakehouse architecture, including Managed Service for Apache Spark, Cloud Storage, and the Apache Hive catalog.
Lakehouse Hive catalog architecture diagram.

Feature comparison with Hive Metastore

The following table compares entities and operations in Hive Metastore and Lakehouse:

Entity or operation Hive Metastore Lakehouse runtime catalog
Catalog ✅ ✅
Database (create, delete, update) ✅ ✅
Table (create, delete, update) ✅ ✅
Partition (add, drop, update) ✅ ✅
Table privileges ✅ ✅ (through Identity and Access Management (IAM))
Partition privileges ✅ ✅ (through IAM)
User-defined functions ✅ Not supported
Bucketing / Skewing columns ✅ Not supported
Table and partition column stats ✅ Not supported
Key constraints ✅ Not supported
Delegation tokens (Kerberos) ✅ Not supported
Column privileges ✅ Not supported
Roles ✅ Not supported
Workload manager resource plans ✅ Not supported
Transactions and compactions ✅ Not supported

Migrate from Dataproc Metastore

To upgrade and migrate your metadata from Dataproc Metastore to the Lakehouse runtime catalog, see Migrate metadata from Dataproc Metastore to Lakehouse.

What's next