Esplora ed estrai i dati

Un'architettura dei dati multifase organizza i dati in livelli di qualità progressivamente raffinati: bronze, silver e gold. Questa architettura fornisce una metodologia strutturata per trasformare i dati non elaborati in asset puliti e affidabili.

Questo documento mostra come creare il livello bronze, in cui vengono inseriti i dati provenienti da sistemi di origine esterni. Questo livello fornisce un'unica fonte attendibile, garantisce la tracciabilità completa dei dati e consente il riprocessamento delle pipeline.

Prima di iniziare

Accedi al tuo account Google Cloud . Se non conosci Google Cloud, crea un account per valutare le prestazioni dei nostri prodotti in scenari reali. I nuovi clienti ricevono anche 300 $di crediti senza costi per l'esecuzione, il test e il deployment dei carichi di lavoro.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Compute Engine, BigQuery, and Cloud Storage APIs, if any are not already enabled.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the APIs

Create a service account:

1. Ensure that you have the Create Service Accounts IAM role (roles/iam.serviceAccountCreator) and the Project IAM Admin role (roles/resourcemanager.projectIamAdmin). Learn how to grant roles.

2. In the Google Cloud console, go to the Create service account page.

   Go to Create service account
3. Select your project.

4. In the Service account name field, enter a name. The Google Cloud console fills in the Service account ID field based on this name.

   In the Service account description field, enter a description. For example, Service account for quickstart.

5. Click Create and continue.

6. Grant the following roles to the service account: Storage Object Admin, Dataproc Worker.

   To grant a role, find the Select a role list, then select the role.

   To grant additional roles, click Add another role and add each additional role.

   Note: The Role field affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later.
7. Click Continue.

8. Click Done to finish creating the service account.

Install the Google Cloud CLI.

Note: If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

To initialize the gcloud CLI, run the following command:

gcloud init

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the Compute Engine, BigQuery, and Cloud Storage APIs, if any are not already enabled.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the APIs

Create a service account:

1. Ensure that you have the Create Service Accounts IAM role (roles/iam.serviceAccountCreator) and the Project IAM Admin role (roles/resourcemanager.projectIamAdmin). Learn how to grant roles.

2. In the Google Cloud console, go to the Create service account page.

   Go to Create service account
3. Select your project.

4. In the Service account name field, enter a name. The Google Cloud console fills in the Service account ID field based on this name.

   In the Service account description field, enter a description. For example, Service account for quickstart.

5. Click Create and continue.

6. Grant the following roles to the service account: Storage Object Admin, Dataproc Worker.

   To grant a role, find the Select a role list, then select the role.

   To grant additional roles, click Add another role and add each additional role.

   Note: The Role field affects which resources the service account can access in your project. You can revoke these roles or grant additional roles later.
7. Click Continue.

8. Click Done to finish creating the service account.

Install the Google Cloud CLI.

Note: If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

To initialize the gcloud CLI, run the following command:

gcloud init

Inserire i dati non elaborati in Cloud Storage

Importa i dati non elaborati in un livello di archiviazione iniziale in Cloud Storage. Per simulare l'importazione dei dati, esporta un set di dati pubblico da BigQuery in un bucket Cloud Storage. Questo processo simula un sistema esterno che invia file di dati non elaborati a una zona di destinazione.

1. Crea un bucket Cloud Storage da utilizzare come data lake. Crealo nella stessa regione del cluster Managed Service for Apache Spark per ottimizzare le prestazioni.

   gsutil mb -l REGION gs://BUCKET_NAME/
   

2. Esporta la tabella bigquery-public-data:samples.shakespeare nel bucket Cloud Storage in formato CSV.

   bq extract \
       --destination_format CSV \
       "bigquery-public-data:samples.shakespeare" \
       "gs://BUCKET_NAME/raw/shakespeare/shakespeare.csv"
   

   Questo comando avvia un job di esportazione che scrive i contenuti della tabella nel percorso Cloud Storage specificato.

Leggi i dati non elaborati con PySpark

Dopo aver inserito i dati in Cloud Storage, utilizza un job PySpark su un cluster Managed Service for Apache Spark per leggerli ed esplorarli. Apache Spark interagisce con Cloud Storage tramite il connettore Cloud Storage, che consente di utilizzare lo schema URI gs:// per leggere e scrivere dati.

1. Utilizza il seguente script PySpark per creare una SparkSession, configurarla per l'accesso a Cloud Storage e leggere il file CSV non elaborato in un DataFrame.

   from pyspark.sql import SparkSession
   
   # --- Configuration ---
   gcs_bucket = "BUCKET_NAME"
   raw_path = f"gs://{gcs_bucket}/raw/shakespeare/shakespeare.csv"
   # For local development only.
   service_account_key_path = "/path/to/your/service-account-key.json"
   
   # --- Spark Session Initialization ---
   spark = SparkSession.builder \
     .appName("DataprocETL-RawIngestion") \
     .config("spark.jars", "https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop3-latest.jar") \
     .getOrCreate()
   
   # --- Authentication for local development ---
   # This step is not necessary when running on a Dataproc cluster
   # with the service account attached to the cluster VMs.
   spark.conf.set("google.cloud.auth.service.account.json.keyfile", service_account_key_path)
   
   # --- Read Raw Data from Cloud Storage ---
   # Read the raw CSV data into a DataFrame.
   # inferSchema=True scans the data to determine column types.
   raw_df = spark.read.csv(raw_path, header=True, inferSchema=True)
   
   # --- Initial Exploration ---
   print("Raw data count:", raw_df.count())
   print("Schema:")
   raw_df.printSchema()
   print("Sample of raw data:")
   raw_df.show(10, truncate=False)
   
   # --- Stop Spark Session ---
   spark.stop()
   

2. Esegui lo script come job Managed Service for Apache Spark per importare ed esplorare i dati non elaborati.

Pattern di acquisizione dei dati

Il livello Bronze può supportare vari pattern di importazione oltre ai caricamenti di file batch. Managed Service for Apache Spark è un motore versatile in grado di gestire diversi scenari di importazione.

Acquisizione di flussi di dati

Per le origini dati continue come i dati dei sensori IoT o i log delle applicazioni, utilizza una pipeline di streaming. Puoi utilizzare Managed Service for Apache Spark per elaborare flussi di volumi elevati da un servizio come Pub/Sub o Apache Kafka e inserire i dati nel livello bronze.

Importazione del database

Per mantenere sincronizzato il data lake con i database operativi, utilizza Change Data Capture (CDC). Un job Managed Service for Apache Spark può sottoscrivere un argomento Pub/Sub che riceve eventi di modifica, elaborare il flusso e applicare le modifiche al datastore non elaborato in Cloud Storage.

Passaggi successivi

• Scopri di più su Managed Service for Apache Spark.
• Scopri di più sul connettore Cloud Storage.
• Esplora i set di dati pubblici di BigQuery.