無邊界湖倉公開資料集

無邊界湖倉會透過 Apache Iceberg REST 目錄提供高品質的公開資料集，並透過Google Cloud 公開資料集計畫向大眾開放。

這些資料集提供唯讀存取權，您可以使用 Apache Spark、Trino、Flink 或 BigQuery 存取這些資料集，並整合到應用程式中。這些資料集的儲存空間費用由 Google 支付，Google 也透過 Lakehouse 將這些資料集提供給大眾存取。您只需要支付資料查詢費用。

這些公開資料集旨在降低 Iceberg 的入門門檻。您不需要管理基礎架構來學習 Iceberg，只需要連線即可。您可以運用這些資料集：

• 透過 Lakehouse 使用 BigQuery，直接以 SQL 查詢這些資料表，並與您的私人資料合併。
• 針對即時 REST 目錄測試 OSS 引擎 (例如 Spark、Trino 或 Flink) 設定。

事前準備

登入 Google Cloud 帳戶。如果您是 Google Cloud新手，歡迎 建立帳戶，親自評估產品在實際工作環境中的成效。新客戶還能獲得價值 $300 美元的免費抵免額，可用於執行、測試及部署工作負載。

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Lakehouse API.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the API

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Lakehouse API.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the API

如要連線至 Apache Spark，您必須具備下列條件：

• 環境中設定的應用程式預設憑證 (ADC)。

公開資料集位置

每個公開資料集都儲存在特定位置，例如 US 或 EU。Lakehouse 公開資料集儲存在 US 多區域位置。查詢公開資料集時，請確保處理位置與資料集位置相容。

使用 Apache Spark 存取公開資料集

由於 Lakehouse 公開資料集是透過 Iceberg REST 目錄提供，因此您可以從 Apache Spark 和其他相容引擎存取這些資料集。您可以使用任何標準 Spark 環境 (例如地端部署、Managed Service for Apache Spark 或其他雲端供應商) 連線至公開資料集。

連線至 Apache Spark

啟動 Spark SQL 工作階段時，請使用下列設定旗標。 這些標記會設定名為 lakehouse-sample 的目錄，指向公開 REST 端點。

spark-sql \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.0,org.apache.iceberg:iceberg-gcp-bundle:1.10.0 \
  --conf spark.hadoop.hive.cli.print.header=true \
  --conf spark.sql.catalog.bqms=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.bqms.type=rest \
  --conf spark.sql.catalog.bqms.uri=https://biglake.googleapis.com/iceberg/v1/restcatalog \
  --conf spark.sql.catalog.bqms.warehouse=gs://biglake-public-nyc-taxi-iceberg \
  --conf spark.sql.catalog.bqms.header.x-goog-user-project=PROJECT_ID \
  --conf spark.sql.catalog.bqms.rest.auth.type=google \
  --conf spark.sql.catalog.bqms.io-impl=org.apache.iceberg.gcp.gcs.GCSFileIO \
  --conf spark.sql.catalog.bqms.header.X-Iceberg-Access-Delegation=vended-credentials \
  --conf spark.sql.defaultCatalog=lakehouse-sample

將 PROJECT_ID 替換為 Google Cloud 專案 ID。

查詢範例

連線完成後，您就能透過 SQL 完整存取資料集。紐約市計程車資料集已推出，並以 Iceberg 資料表的形式建立模型，可展示分區和中繼資料功能。

下列查詢會匯總數百萬筆記錄，找出各乘客數的平均票價和行程距離。本範例說明 Iceberg 如何使用分區修剪功能，有效率地掃描資料檔案，而不需列出目錄：

SELECT
    passenger_count,
    COUNT(1) AS num_trips,
    ROUND(AVG(total_amount), 2) AS avg_fare,
    ROUND(AVG(trip_distance), 2) AS avg_distance
FROM
    lakehouse-sample.public_data.nyc_taxicab
WHERE
    data_file_year = 2021
    AND passenger_count > 0
GROUP BY
    passenger_count
ORDER BY
    num_trips DESC;

Iceberg 最強大的功能之一是「時光旅行」。您可以查詢資料表在過去特定時間的狀態。您可以透過下列查詢稽核變更，方法是比較目前版本與特定快照的資料列數：

-- Compare the row count of the current version vs. a specific snapshot
SELECT
    'Current State' AS version,
    COUNT(*) AS count
FROM lakehouse-sample.public_data.nyc_taxicab
UNION ALL
SELECT
    'Past State' AS version,
    COUNT(*) AS count
FROM lakehouse-sample.public_data.nyc_taxicab VERSION AS OF 2943559336503196801Q;

查詢歷來中繼資料表 (例如 SELECT * FROM lakehouse-sample.public_data.nyc_taxicab.history)，即可找出快照 ID，並回溯查看資料集隨時間成長的過程。

詳情請參閱「搭配 Spark、Iceberg REST 目錄和 Cloud Storage 使用 Lakehouse 執行階段目錄」。

可用的資料集

Lakehouse 提供範例資料表，您可以將這些資料表當做 Apache Iceberg 資料表查詢。

biglake-public-nyc-taxi-iceberg 資料集包含下列 Apache Iceberg 格式的資料表：

名稱	說明
nyc_taxicab	紐約市計程車暨禮車管理局 (TLC) 行程記錄資料。

後續步驟

• 進一步瞭解 Lakehouse 管理的 Apache Iceberg 資料表。
• 進一步瞭解 Lakehouse 執行階段目錄 Iceberg REST 目錄。