无边界湖仓一体公开数据集

无边界 Lakehouse 托管通过 Apache Iceberg REST Catalog 提供的优质公共数据集，并作为 Google Cloud 公共数据集计划的一部分向公众提供这些数据集。

这些数据集可供您以只读方式访问，您可以使用 Apache Spark、Trino、Flink 或 BigQuery 访问这些数据集并将其集成到您的应用中。Google 会支付这些数据集的存储费用，并通过 Lakehouse 提供对数据的公开访问权限。您只需为对数据执行的查询付费。

这些公共数据集旨在降低 Iceberg 的入门门槛。 您无需管理基础架构即可学习 Iceberg，只需连接即可。您可以使用这些数据集来：

• 使用 BigQuery（通过 Lakehouse）直接使用 SQL 查询这些表，并将其与您的私有数据相结合。
• 针对实时 REST 目录测试您的 OSS 引擎（例如 Spark、Trino 或 Flink）配置。

准备工作

登录您的 Google Cloud 账号。如果您是 Google Cloud新手，请 创建一个账号来评估我们的产品在实际场景中的表现。新客户还可获享 $300 赠金，用于运行、测试和部署工作负载。

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Lakehouse API.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the API

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Lakehouse API.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the API

在与 Apache Spark 建立连接之前，您必须具备以下条件：

• 在您的环境中设置应用默认凭证 (ADC)。

公共数据集位置

每个公共数据集都存储在一个特定位置，例如 US 或 EU。Lakehouse 公共数据集存储在 US 多区域位置。查询公共数据集时，请确保处理位置与数据集位置兼容。

使用 Apache Spark 访问公开数据集

由于 Lakehouse 公共数据集是通过 Iceberg REST Catalog 提供的，因此您可以从 Apache Spark 和其他兼容的引擎访问这些数据集。您可以使用任何标准 Spark 环境（例如本地环境、Managed Service for Apache Spark 或其他云供应商）连接到公共数据集。

使用 Apache Spark 进行连接

启动 Spark SQL 会话时，请使用以下配置标志。 这些标志配置了一个名为 lakehouse-sample 的目录，该目录指向公共 REST 端点。

spark-sql \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.0,org.apache.iceberg:iceberg-gcp-bundle:1.10.0 \
  --conf spark.hadoop.hive.cli.print.header=true \
  --conf spark.sql.catalog.bqms=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.bqms.type=rest \
  --conf spark.sql.catalog.bqms.uri=https://biglake.googleapis.com/iceberg/v1/restcatalog \
  --conf spark.sql.catalog.bqms.warehouse=gs://biglake-public-nyc-taxi-iceberg \
  --conf spark.sql.catalog.bqms.header.x-goog-user-project=PROJECT_ID \
  --conf spark.sql.catalog.bqms.rest.auth.type=google \
  --conf spark.sql.catalog.bqms.io-impl=org.apache.iceberg.gcp.gcs.GCSFileIO \
  --conf spark.sql.catalog.bqms.header.X-Iceberg-Access-Delegation=vended-credentials \
  --conf spark.sql.defaultCatalog=lakehouse-sample

请将 PROJECT_ID 替换为您的 Google Cloud 项目 ID。

示例查询

连接后，您将拥有对数据集的完整 SQL 访问权限。纽约市出租车数据集以 Iceberg 表的形式提供，用于演示分区和元数据功能。

以下查询会汇总数百万条记录，以按乘客人数查找平均车费和行程距离。它展示了 Iceberg 如何通过使用分区剪枝高效扫描数据文件，而无需列出目录：

SELECT
    passenger_count,
    COUNT(1) AS num_trips,
    ROUND(AVG(total_amount), 2) AS avg_fare,
    ROUND(AVG(trip_distance), 2) AS avg_distance
FROM
    lakehouse-sample.public_data.nyc_taxicab
WHERE
    data_file_year = 2021
    AND passenger_count > 0
GROUP BY
    passenger_count
ORDER BY
    num_trips DESC;

Iceberg 最强大的功能之一是时间旅行。您可以查询过去特定时间点的表。通过以下查询，您可以比较当前版本的行数与特定快照的行数，从而审核更改：

-- Compare the row count of the current version vs. a specific snapshot
SELECT
    'Current State' AS version,
    COUNT(*) AS count
FROM lakehouse-sample.public_data.nyc_taxicab
UNION ALL
SELECT
    'Past State' AS version,
    COUNT(*) AS count
FROM lakehouse-sample.public_data.nyc_taxicab VERSION AS OF 2943559336503196801Q;

通过查询历史元数据表（例如 SELECT * FROM lakehouse-sample.public_data.nyc_taxicab.history），您可以找到快照 ID，并回溯查看数据集随时间增长的情况。

如需了解详情，请参阅将 Lakehouse 运行时目录与 Spark、Iceberg REST 目录和 Cloud Storage 搭配使用。

可用的数据集

湖仓一体提供了示例表，您可以像查询 Apache Iceberg 表一样查询这些示例表。

biglake-public-nyc-taxi-iceberg 数据集包含以下采用 Apache Iceberg 格式的表：

名称	说明
nyc_taxicab	纽约市出租车和豪华轿车委员会 (TLC) 行程记录数据。

后续步骤

• 详细了解 Lakehouse 管理的 Apache Iceberg 表。
• 详细了解 Lakehouse 运行时目录 Iceberg REST 目录。