You can use a BigQuery connector to enable programmatic read and write
access to [BigQuery](https://docs.cloud.google.com/bigquery). This is an ideal way to process
data that is stored in BigQuery. Command-line access is not exposed.
The BigQuery connector is a library that enables Spark and Hadoop
applications to process data from BigQuery and write data to
BigQuery using its native terminology.

> [!IMPORTANT]
> The [GoogleCloudDataproc/spark-bigquery-connector](https://github.com/GoogleCloudDataproc/spark-bigquery-connector) is also available for reading data from BigQuery. It takes advantage of the [BigQueryStorage API](https://docs.cloud.google.com/bigquery/docs/reference/storage.md.txt).

## Pricing

When using the connector, charges include [BigQuery usage fees](https://cloud.google.com/bigquery/pricing).
The following service-specific charges may also apply:

- [Cloud Storage](https://docs.cloud.google.com/storage) - the connector downloads data into a Cloud Storage bucket before or during job execution. After the job successfully completes, the data is deleted from Cloud Storage. You are charged for this storage according to [Cloud Storage pricing](https://cloud.google.com/storage/pricing). To avoid excess charges, check your Cloud Storage account and remove unneeded temporary files.
- [Storage Read API](https://docs.cloud.google.com/bigquery/docs/reference/storage.md.txt) - the connector reads data from tables using the Storage Read API to achieve high throughput.
  - When requests originate from either the [Managed Service for Apache Spark serverless](https://docs.cloud.google.com/managed-spark/docs/serverless-overview.md.txt) or [BigQuery stored procedures for Apache Spark](https://docs.cloud.google.com/bigquery/docs/spark-procedures.md.txt) Storage Read API streaming read costs are delegated to the connector at no additional charge. Standard network egress charges still apply if data is read across regions.
  - When requests originate from cluster-based Apache Spark workloads (such as the [Managed Service for Apache Spark on clusters](https://docs.cloud.google.com/managed-spark/docs/concepts/clusters-overview.md.txt) or Apache Spark running on Google Kubernetes Engine (GKE)), standard [Storage Read API pricing](https://docs.cloud.google.com/bigquery/docs/reference/storage.md.txt#pricing) applies.

## Available connectors

The following BigQuery connectors are available for use in
the Hadoop ecosystem:

1. The [Spark BigQuery Connector](https://github.com/GoogleCloudDataproc/spark-bigquery-connector) adds a Spark data source, which allows DataFrames to interact directly with BigQuery tables using Spark's `read` and `write` operations.
2. The [Hive BigQuery Connector](https://github.com/GoogleCloudDataproc/hive-bigquery-connector) adds a Storage Handler, which allows Apache Hive to interact directly with BigQuery tables using HiveQL syntax.
3. The [Hadoop BigQuery Connector](https://github.com/GoogleCloudDataproc/hadoop-connectors)
   allows Hadoop mappers and reducers to interact with BigQuery
   tables using abstracted versions of the
   [InputFormat](http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/InputFormat.html)
   and [OutputFormat](http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/OutputFormat.html)
   classes.

   > [!CAUTION]
   > **Caution:** The Hadoop BigQuery connector for Hadoop MapReduce is not supported. Recommendation: Use the [Spark BigQuery Connector](https://github.com/GoogleCloudDataproc/spark-bigquery-connector) for improved performance. For more information, see [Use the BigQuery Connector with Spark](https://docs.cloud.google.com/managed-spark/docs/tutorials/bigquery-connector-spark-example.md.txt).

## Use the connectors

For a quick start using the BigQuery connector, see the following examples:

- [Spark example](https://docs.cloud.google.com/managed-spark/docs/tutorials/bigquery-connector-spark-example.md.txt)
- [Java MapReduce example](https://docs.cloud.google.com/managed-spark/docs/tutorials/bigquery-connector-mapreduce-example.md.txt)
- [Connect Managed Service for Apache Spark cluster to BigQuery](https://console.cloud.google.com/?walkthrough_id=dataproc--dataproc-bq-spark-connector)

## What's next

- Learn more about [BigQuery](https://docs.cloud.google.com/bigquery).
- Follow the [BigQuery example for Spark](https://docs.cloud.google.com/managed-spark/docs/tutorials/bigquery-connector-spark-example.md.txt).
- Learn more about the [Hive BigQuery Connector](https://docs.cloud.google.com/managed-spark/docs/concepts/connectors/hive-bigquery.md.txt).
- Follow the [BigQuery example for Java MapReduce](https://docs.cloud.google.com/managed-spark/docs/tutorials/bigquery-connector-mapreduce-example.md.txt).