# Explore and extract data

A multi-stage data architecture organizes data into progressively refined
quality tiers: bronze, silver, and gold. This architecture provides a structured
methodology for transforming raw data into clean, reliable assets.

This document shows you how to create the bronze tier, where you land data from
external source systems. This tier provides a single source of truth, ensures
complete data lineage, and enables the reprocessing of pipelines.

## Before you begin

<br />

## Land raw data in Cloud Storage

Ingest raw data into an initial storage layer in Cloud Storage. To simulate data
ingestion, export a public dataset from BigQuery to a Cloud Storage bucket. This
process mimics an external system delivering raw data files to a landing zone.

1. Create a Cloud Storage bucket to serve as the data lake. Create it in the
   same region as your Managed Service for Apache Spark cluster to optimize performance.

   ```bash
   gsutil mb -l REGION gs://BUCKET_NAME/
   ```
2. Export the `bigquery-public-data:samples.shakespeare` table to your Cloud Storage bucket in CSV format.

   ```bash
   bq extract \
       --destination_format CSV \
       "bigquery-public-data:samples.shakespeare" \
       "gs://BUCKET_NAME/raw/shakespeare/shakespeare.csv"
   ```

   This command starts an export job that writes the table's contents to the
   specified Cloud Storage path.

## Read raw data with PySpark

After landing the data in Cloud Storage, use a PySpark job on a Managed Service for Apache Spark cluster
to read and explore it. Apache Spark interacts with Cloud Storage through the
Cloud Storage connector, which lets you use the `gs://` URI scheme to read and
write data.

1. Use the following PySpark script to create a SparkSession, configure it for
   Cloud Storage access, and read the raw CSV file into a DataFrame.

   ```python
   from pyspark.sql import SparkSession

   # --- Configuration ---
   gcs_bucket = "BUCKET_NAME"
   raw_path = f"gs://{gcs_bucket}/raw/shakespeare/shakespeare.csv"
   # For local development only.
   service_account_key_path = "/path/to/your/service-account-key.json"

   # --- Spark Session Initialization ---
   spark = SparkSession.builder \
     .appName("DataprocETL-RawIngestion") \
     .config("spark.jars", "https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop3-latest.jar") \
     .getOrCreate()

   # --- Authentication for local development ---
   # This step is not necessary when running on a Dataproc cluster
   # with the service account attached to the cluster VMs.
   spark.conf.set("google.cloud.auth.service.account.json.keyfile", service_account_key_path)

   # --- Read Raw Data from Cloud Storage ---
   # Read the raw CSV data into a DataFrame.
   # inferSchema=True scans the data to determine column types.
   raw_df = spark.read.csv(raw_path, header=True, inferSchema=True)

   # --- Initial Exploration ---
   print("Raw data count:", raw_df.count())
   print("Schema:")
   raw_df.printSchema()
   print("Sample of raw data:")
   raw_df.show(10, truncate=False)

   # --- Stop Spark Session ---
   spark.stop()
   ```
2. Run the script as a Managed Service for Apache Spark job to ingest and explore the raw data.

## Data ingestion patterns

The bronze tier can support various ingestion patterns beyond batch file
uploads. Managed Service for Apache Spark is a versatile engine capable of handling diverse ingestion
scenarios.

### Streaming ingestion

For continuous data sources like IoT sensor data or application logs, use a
streaming pipeline. You can use Managed Service for Apache Spark to process high-volume streams from a
service like Pub/Sub or Apache Kafka and land the data in the bronze tier.

### Database ingestion

To keep the data lake synchronized with operational databases, use Change Data
Capture (CDC). A Managed Service for Apache Spark job can subscribe to a Pub/Sub topic that
receives change events, process the stream, and apply the changes to the raw
datastore in Cloud Storage.

## What's next

- Learn more about [Managed Service for Apache Spark](https://cloud.google.com/managed-spark/docs).
- Learn more about the [Cloud Storage connector](https://cloud.google.com/managed-spark/docs/concepts/connectors/cloud-storage).
- Explore [BigQuery public datasets](https://cloud.google.com/bigquery/public-data).