Menjelajahi dan mengekstrak data

Arsitektur data multi-tahap mengatur data ke dalam tingkat kualitas yang terus ditingkatkan: perunggu, perak, dan emas. Arsitektur ini menyediakan metodologi terstruktur untuk mengubah data mentah menjadi aset yang bersih dan andal.

Dokumen ini menunjukkan cara membuat tingkat perunggu, tempat Anda menempatkan data dari sistem sumber eksternal. Tingkat ini menyediakan satu sumber tepercaya, memastikan asal-usul data yang lengkap, dan memungkinkan pemrosesan ulang pipeline.

Sebelum memulai

Login ke akun Google Cloud Anda. Jika Anda baru menggunakan Google Cloud, buat akun untuk mengevaluasi performa produk kami dalam skenario dunia nyata. Pelanggan baru juga mendapatkan kredit gratis senilai $300 untuk menjalankan, menguji, dan men-deploy workload.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Compute Engine, BigQuery, and Cloud Storage APIs.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

Enable the APIs

Create a service account:

1. Ensure that you have the Create Service Accounts IAM role (roles/iam.serviceAccountCreator) and the Project IAM Admin role (roles/resourcemanager.projectIamAdmin). Learn how to grant roles.

2. In the Google Cloud console, go to the Create service account page.

   Go to Create service account
3. Select your project.

4. In the Service account name field, enter a name. The Google Cloud console fills in the Service account ID field based on this name.

   In the Service account description field, enter a description. For example, Service account for quickstart.

5. Click Create and continue.

6. Grant the following roles to the service account: Storage Object Admin, Dataproc Worker.

   To grant a role, find the Select a role list, then select the role.

   To grant additional roles, click Add another role and add each additional role.

   Note: The Role field affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later.
7. Click Continue.

8. Click Done to finish creating the service account.

Instal Google Cloud CLI.

Catatan: Jika Anda telah menginstal gcloud CLI sebelumnya, pastikan Anda memiliki versi terbaru dengan menjalankan gcloud components update.

Jika Anda menggunakan penyedia identitas (IdP) eksternal, Anda harus login ke gcloud CLI dengan identitas gabungan Anda terlebih dahulu.

Untuk melakukan inisialisasi gcloud CLI, jalankan perintah berikut:

gcloud init

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Compute Engine, BigQuery, and Cloud Storage APIs.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

Enable the APIs

Create a service account:

1. Ensure that you have the Create Service Accounts IAM role (roles/iam.serviceAccountCreator) and the Project IAM Admin role (roles/resourcemanager.projectIamAdmin). Learn how to grant roles.

2. In the Google Cloud console, go to the Create service account page.

   Go to Create service account
3. Select your project.

4. In the Service account name field, enter a name. The Google Cloud console fills in the Service account ID field based on this name.

   In the Service account description field, enter a description. For example, Service account for quickstart.

5. Click Create and continue.

6. Grant the following roles to the service account: Storage Object Admin, Dataproc Worker.

   To grant a role, find the Select a role list, then select the role.

   To grant additional roles, click Add another role and add each additional role.

   Note: The Role field affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later.
7. Click Continue.

8. Click Done to finish creating the service account.

Instal Google Cloud CLI.

Catatan: Jika Anda telah menginstal gcloud CLI sebelumnya, pastikan Anda memiliki versi terbaru dengan menjalankan gcloud components update.

Jika Anda menggunakan penyedia identitas (IdP) eksternal, Anda harus login ke gcloud CLI dengan identitas gabungan Anda terlebih dahulu.

Untuk melakukan inisialisasi gcloud CLI, jalankan perintah berikut:

gcloud init

Menyimpan data mentah di Cloud Storage

Menyerap data mentah ke lapisan penyimpanan awal di Cloud Storage. Untuk menyimulasikan penyerapan data, ekspor set data publik dari BigQuery ke bucket Cloud Storage. Proses ini meniru sistem eksternal yang mengirimkan file data mentah ke area landing.

1. Buat bucket Cloud Storage untuk berfungsi sebagai data lake. Buat di region yang sama dengan cluster Managed Service for Apache Spark Anda untuk mengoptimalkan performa.

   gsutil mb -l REGION gs://BUCKET_NAME/
   

2. Ekspor tabel bigquery-public-data:samples.shakespeare ke bucket Cloud Storage Anda dalam format CSV.

   bq extract \
       --destination_format CSV \
       "bigquery-public-data:samples.shakespeare" \
       "gs://BUCKET_NAME/raw/shakespeare/shakespeare.csv"
   

   Perintah ini memulai tugas ekspor yang menulis konten tabel ke jalur Cloud Storage yang ditentukan.

Membaca data mentah dengan PySpark

Setelah data berada di Cloud Storage, gunakan tugas PySpark di cluster Managed Service for Apache Spark untuk membaca dan menjelajahinya. Apache Spark berinteraksi dengan Cloud Storage melalui konektor Cloud Storage, yang memungkinkan Anda menggunakan skema URI gs:// untuk membaca dan menulis data.

1. Gunakan skrip PySpark berikut untuk membuat SparkSession, mengonfigurasinya untuk akses Cloud Storage, dan membaca file CSV mentah ke dalam DataFrame.

   from pyspark.sql import SparkSession
   
   # --- Configuration ---
   gcs_bucket = "BUCKET_NAME"
   raw_path = f"gs://{gcs_bucket}/raw/shakespeare/shakespeare.csv"
   # For local development only.
   service_account_key_path = "/path/to/your/service-account-key.json"
   
   # --- Spark Session Initialization ---
   spark = SparkSession.builder \
     .appName("DataprocETL-RawIngestion") \
     .config("spark.jars", "https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop3-latest.jar") \
     .getOrCreate()
   
   # --- Authentication for local development ---
   # This step is not necessary when running on a Dataproc cluster
   # with the service account attached to the cluster VMs.
   spark.conf.set("google.cloud.auth.service.account.json.keyfile", service_account_key_path)
   
   # --- Read Raw Data from Cloud Storage ---
   # Read the raw CSV data into a DataFrame.
   # inferSchema=True scans the data to determine column types.
   raw_df = spark.read.csv(raw_path, header=True, inferSchema=True)
   
   # --- Initial Exploration ---
   print("Raw data count:", raw_df.count())
   print("Schema:")
   raw_df.printSchema()
   print("Sample of raw data:")
   raw_df.show(10, truncate=False)
   
   # --- Stop Spark Session ---
   spark.stop()
   

2. Jalankan skrip sebagai tugas Managed Service for Apache Spark untuk menyerap dan menjelajahi data mentah.

Pola penyerapan data

Paket perunggu dapat mendukung berbagai pola penyerapan di luar upload file batch. Managed Service for Apache Spark adalah mesin serbaguna yang mampu menangani berbagai skenario penyerapan.

Penyerapan streaming

Untuk sumber data berkelanjutan seperti data sensor IoT atau log aplikasi, gunakan pipeline streaming. Anda dapat menggunakan Managed Service for Apache Spark untuk memproses aliran bervolume tinggi dari layanan seperti Pub/Sub atau Apache Kafka dan menempatkan data di tingkat perunggu.

Penyerapan database

Untuk menjaga agar data lake tetap tersinkronisasi dengan database operasional, gunakan Pengambilan Data Perubahan (CDC). Job Spark Managed Service for Apache Spark dapat berlangganan topik Pub/Sub yang menerima peristiwa perubahan, memproses stream, dan menerapkan perubahan ke penyimpanan data mentah di Cloud Storage.

Langkah berikutnya

• Pelajari lebih lanjut Managed Service for Apache Spark.
• Pelajari lebih lanjut konektor Cloud Storage.
• Jelajahi set data publik BigQuery.