This document describes the different table formats available when building a lakehouse on BigLake and helps you choose the right one for your needs.
When building a lakehouse on BigLake, you can choose from several table formats that offer different levels of management, performance, and interoperability. Your choice depends on where your data originates, which engines you want to use for writing and transformation, and how much control you need over storage and metadata.
Table formats
When building a lakehouse on BigLake, you have the following choices for the format of your tables:
- BigLake Iceberg tables are Iceberg tables that you create from open source engines and store in Cloud Storage. Like all tables that use BigLake metastore, they can be read by open source engines and BigQuery. However, only open source engines can write to it. This option is best if you want your ETL workflow to be managed by open source engines.
- BigLake Iceberg tables in BigQuery are Iceberg tables that you create from BigQuery and store in Cloud Storage. Like all tables that use BigLake metastore, they can be read by open source engines and BigQuery. However, BigQuery is the only engine that can directly write to them. This option is best if you want your extract, transform, and load (ETL) workflow to be fully managed by BigQuery.
- Standard BigQuery tables are fully managed by BigQuery and have the most advanced data analytics and management features. You can still connect these tables to BigLake metastore. This option is best for non- Iceberg tables.
- External tables are tables that are outside of BigLake metastore. The data and metadata of these tables are completely self-managed, where you are fully reliant upon the capabilities of open table formats (such as Iceberg, Apache Hudi, or Delta Lake). BigQuery only has the ability to read from these tables. Choose this option for data and metadata that you want to manage on your own in a third-party catalog.
Use the following chart to compare your table format options:
| External tables | BigLake Iceberg tables | BigLake Iceberg tables in BigQuery | Standard BigQuery tables | |
|---|---|---|---|---|
| Metastore | External or self-hosted metastore | BigLake metastore | BigLake metastore | BigQuery |
| Storage | Cloud Storage / Amazon S3 / Azure | Cloud Storage | Cloud Storage | BigQuery |
| Storage optimization | Customer or third-party managed | Customer or third-party managed | Google managed | Google managed |
| Read / Write |
Open source engines (read/write) BigQuery (read only) |
Open source engines (read/write) BigQuery (read only) |
Open source engines (read only with Iceberg
libraries, read/write interoperability with BigQuery Storage API)
BigQuery (read/write) |
Open source engines (read/write interoperability with
BigQuery Storage API) BigQuery (read/write) |
| Use cases | Staging tables for BigQuery loads, legacy query-only tables | Open lakehouse | Open lakehouse with high-performant, enterprise-grade storage for advanced analytics, streaming, and AI | Enterprise-grade storage for advanced analytics, streaming, and AI |