About Hive Catalogs in Lakehouse runtime catalog

The Lakehouse runtime catalog is a serverless, unified metastore for Lakehouse for Apache Iceberg that simplifies managing self-hosted Hive Metastores. This single, fully managed metadata layer eliminates the need for separate metadata stores for open-source workloads. It lets you seamlessly share data across Apache Spark, Apache Hive, and BigQuery.

Optimized for Apache Spark ExternalCatalog compatibility, this integration supports a subset of the Hive Metastore interface. To see if your workloads depend on unsupported features like transactions, compactions, or Kerberos, review the feature comparison and limitations.

How Hive integrates with the Lakehouse runtime catalog

The Hive metastore endpoint manages standard Apache Hive and Spark tables (using Hive SerDes or Spark data sources) rather than Apache Iceberg tables. To simplify connecting your Spark jobs to this endpoint, Managed Service for Apache Spark images are preconfigured with the necessary client libraries and dependencies.

The following sequence describes how Spark connects to the metastore:

  1. Apache Spark connects to external metadata catalogs by using the standard Apache Hive IMetastoreClient interface.
  2. The Lakehouse runtime catalog implements a custom IMetastoreClient to provide a fully managed metastore service for your Spark and Hive metadata.
  3. Preconfigured Managed Service for Apache Spark runtimes automatically use this custom client to route metadata operations directly to the metastore.

After setup, you can query your Spark-created tables directly in BigQuery. This integration supports various storage formats—such as Parquet, ORC, and Avro—along with specific data type mappings between Spark and BigQuery.

Feature comparison with Hive Metastore

The following table compares entities and operations in Hive Metastore and Lakehouse.

Entity or operation Hive Metastore Lakehouse runtime catalog
Catalog ✅ ✅
Database (create, delete, update) ✅ ✅
Table (create, delete, update) ✅ ✅
Partition (add, drop, update) ✅ ✅
User-defined functions ✅ ❌
Bucketing columns ✅ ❌
Skewed columns ✅ ❌
Table column stats ✅ ❌
Partition column stats ✅ ❌
Key constraints ✅ ❌
Primary keys ✅ ❌
Master keys ✅ ❌
Delegation tokens (Kerberos) ✅ ❌
Workload manager resource plans ✅ ❌
Transactions and compactions ✅ ❌
Table privileges ✅ ✅ (through Identity and Access Management (IAM))
Column privileges ✅ ❌
Partition privileges ✅ ✅ (through IAM)
Roles ✅ ❌

What's next