BigQuery connector

You can use a BigQuery connector to enable programmatic read and write access to BigQuery. This is an ideal way to process data that is stored in BigQuery. Command-line access is not exposed. The BigQuery connector is a library that enables Spark and Hadoop applications to process data from BigQuery and write data to BigQuery using its native terminology.

The GoogleCloudDataproc/spark-bigquery-connector is also available for reading data from BigQuery. It takes advantage of the BigQueryStorage API.

Pricing

When using the connector, charges include BigQuery usage fees. The following service-specific charges may also apply:

• Cloud Storage - the connector downloads data into a Cloud Storage bucket before or during job execution. After the job successfully completes, the data is deleted from Cloud Storage. You are charged for this storage according to Cloud Storage pricing. To avoid excess charges, check your Cloud Storage account and remove unneeded temporary files.
• Storage Read API - the connector reads data from tables using the Storage Read API to achieve high throughput.
  • When requests originate from either the Managed Service for Apache Spark serverless or BigQuery stored procedures for Apache Spark Storage Read API streaming read costs are delegated to the connector at no additional charge. Standard network egress charges still apply if data is read across regions.
  • When requests originate from cluster-based Apache Spark workloads (such as the Managed Service for Apache Spark on clusters or Apache Spark running on Google Kubernetes Engine (GKE)), standard Storage Read API pricing applies.

Available connectors

The following BigQuery connectors are available for use in the Hadoop ecosystem:

1. The Spark BigQuery Connector adds a Spark data source, which allows DataFrames to interact directly with BigQuery tables using Spark's read and write operations.
2. The Hive BigQuery Connector adds a Storage Handler, which allows Apache Hive to interact directly with BigQuery tables using HiveQL syntax.

3. The Hadoop BigQuery Connector allows Hadoop mappers and reducers to interact with BigQuery tables using abstracted versions of the InputFormat and OutputFormat classes.

   Caution: The Hadoop BigQuery connector for Hadoop MapReduce is not supported. Recommendation: Use the Spark BigQuery Connector for improved performance. For more information, see Use the BigQuery Connector with Spark.

Use the connectors

For a quick start using the BigQuery connector, see the following examples:

• Spark example
• Java MapReduce example
• Connect Managed Service for Apache Spark cluster to BigQuery

What's next

• Learn more about BigQuery.
• Follow the BigQuery example for Spark.
• Learn more about the Hive BigQuery Connector.
• Follow the BigQuery example for Java MapReduce.