This page describes how to get batch inferences using BigQuery.

## 1. Prepare your inputs

### BigQuery storage input

Your service account must have have appropriate BigQuery
permissions. To grant the service account the **BigQuery User** role,
use the [`gcloud iam service-accounts add-iam-policy-binding`](https://cloud.google.com/sdk/gcloud/reference/iam/service-accounts/add-iam-policy-binding)
command as follows:

```bash
    gcloud projects add-iam-policy-binding PROJECT_ID \
        --member="serviceAccount:SERVICE_ACCOUNT_ID@PROJECT_ID.iam.gserviceaccount.com" \
        --role="roles/bigquery.user"
```

Replace the following values:

- <var translate="no">PROJECT_ID</var>: The ID for the project that your service account was created in.
- <var translate="no">SERVICE_ACCOUNT_ID</var>: The ID for the service account.

A `request` column is required, and must be [valid JSON](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/data-types.md.txt#json_type). This
JSON data represents your input for the model.

The content in the `request` column must match the structure of a
[`GenerateContentRequest`](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.publishers.models/generateContent.md.txt#request-body).

Your input table can have column data types other than
`request`. These columns can have [BigQuery data types](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/data-types.md.txt) except
for the following: array, struct, range, datetime, and
geography. These columns are ignored for content generation but
included in the output table.

> [!IMPORTANT]
> **Important:** Don't use `response` or `status` as column names. These keywords are reserved by the system to provide information about the outcome of the batch inference job.

| Example input (JSON) |
|---|
| { "contents": [ { "role": "user", "parts": [ { "text": "Give me a recipe for banana bread." } ] } ], "system_instruction": { "parts": [ { "text": "You are a chef." } ] } } |

## 2. Submit a batch job

You can create a batch job through the Google Cloud console, the Google Gen AI SDK,
or the REST API.

The job and your table must be in the same region.

> [!NOTE]
> To see an example of using batch inference,
> run the "Intro to Batch Predictions with the Gemini API" notebook in one of the following
> environments:
>
> [![](https://docs.cloud.google.com/static/vertex-ai/images/colab-logo-32px.png)Open in Colab](https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/batch-prediction/intro_batch_prediction.ipynb)
>
>
> \|
>
> [![](https://docs.cloud.google.com/static/vertex-ai/images/colab-enterprise-logo-32px.png)Open in Colab Enterprise](https://console.cloud.google.com/agent-platform/colab/import/https%3A%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fbatch-prediction%2Fintro_batch_prediction.ipynb)
>
>
> \|
>
> [![](https://docs.cloud.google.com/static/vertex-ai/images/vertex-ai-workbench-logo-32px.png)Open
> in Agent Platform Workbench](https://console.cloud.google.com/agent-platform/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/batch-prediction/intro_batch_prediction.ipynb)
>
>
> \|
>
> [![](https://docs.cloud.google.com/static/vertex-ai/images/github-logo-32px.png)View on GitHub](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/batch-prediction/intro_batch_prediction.ipynb)

### Console

1. In the Agent Platform section of the Google Cloud console, go to the **Batch Inference** page.

   <br />

   <br />

   [Go to Batch Inference](https://console.cloud.google.com/agent-platform/batch-predictions)
2. Click **Create**.

### REST

To create a batch inference job, use the
[`projects.locations.batchPredictionJobs.create`](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.batchPredictionJobs/create.md.txt) method.


Before using any of the request data,
make the following replacements:

- <var class="edit" scope="ENDPOINT_PREFIX" translate="no">ENDPOINT_PREFIX</var>: The region of the model resource followed by `-`. For example, `us-central1-`. If using the global endpoint, leave blank. **Note:** The global endpoint isn't supported for batch inference using tuned models.
- <var class="edit" scope="LOCATION" translate="no">LOCATION</var>: A region that supports Gemini models. If using the global endpoint, enter `global`.
- <var class="edit" scope="PROJECT_ID" translate="no">PROJECT_ID</var>: Your [project ID](https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects.md.txt#identifiers).
- <var class="edit" scope="MODEL_PATH" translate="no">MODEL_PATH</var>: The publisher model name, for example, `publishers/google/models/gemini-2.0-flash-001`; or the tuned endpoint name, for example, `projects/PROJECT_ID/locations/LOCATION/models/MODEL_ID`, where <var translate="no">MODEL_ID</var> is the model ID of the tuned model.
- <var class="edit" scope="INPUT_URI" translate="no">INPUT_URI</var>: The BigQuery table where your batch inference input is located such as `bq://myproject.mydataset.input_table`. The dataset must be located in the same region as the batch inference job. Multi-region datasets are not supported.
- <var class="edit" scope="OUTPUT_FORMAT" translate="no">OUTPUT_FORMAT</var>: To output to a BigQuery table, specify `bigquery`. To output to a Cloud Storage bucket, specify `jsonl`.
- <var class="edit" scope="DESTINATION" translate="no">DESTINATION</var>: For BigQuery, specify `bigqueryDestination`. For Cloud Storage, specify `gcsDestination`.
- <var class="edit" scope="OUTPUT_URI_FIELD_NAME" translate="no">OUTPUT_URI_FIELD_NAME</var>: For BigQuery, specify `outputUri`. For Cloud Storage, specify `outputUriPrefix`.
- <var class="edit" scope="OUTPUT_URI" translate="no">OUTPUT_URI</var>: For BigQuery, specify the table location such as `bq://myproject.mydataset.output_result`. The region of the output BigQuery dataset must be the same as the Agent Platform batch inference job. For Cloud Storage, specify the bucket and directory location such as `gs://mybucket/path/to/output`.


HTTP method and URL:

```
POST https://ENDPOINT_PREFIXaiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/batchPredictionJobs
```


Request JSON body:

```
{
  "displayName": "my-bigquery-batch-inference-job",
  "model": "MODEL_PATH",
  "inputConfig": {
    "instancesFormat": "bigquery",
    "bigquerySource":{
      "inputUri" : "INPUT_URI"
    }
  },
  "outputConfig": {
    "predictionsFormat": "OUTPUT_FORMAT",
    "DESTINATION": {
      "OUTPUT_URI_FIELD_NAME": "OUTPUT_URI"
    }
  }
}
```

To send your request, choose one of these options:

#### curl

> [!NOTE]
> **Note:** The following command assumes that you have logged in to the `gcloud` CLI with your user account by running [`gcloud init`](https://docs.cloud.google.com/sdk/gcloud/reference/init) or [`gcloud auth login`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) , or by using [Cloud Shell](https://docs.cloud.google.com/shell/docs.md.txt), which automatically logs you into the `gcloud` CLI . You can check the currently active account by running [`gcloud auth list`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/list.md.txt).


Save the request body in a file named `request.json`,
and execute the following command:

```
curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://ENDPOINT_PREFIXaiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/batchPredictionJobs"
```

#### PowerShell

> [!NOTE]
> **Note:** The following command assumes that you have logged in to the `gcloud` CLI with your user account by running [`gcloud init`](https://docs.cloud.google.com/sdk/gcloud/reference/init) or [`gcloud auth login`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) . You can check the currently active account by running [`gcloud auth list`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/list.md.txt).


Save the request body in a file named `request.json`,
and execute the following command:

```
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://ENDPOINT_PREFIXaiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/batchPredictionJobs" | Select-Object -Expand Content
```

You should receive a JSON response similar to the following.

#### Response

```
{
  "name": "projects/PROJECT_ID/locations/LOCATION/batchPredictionJobs/BATCH_JOB_ID",
  "displayName": "my-bigquery-batch-inference-job",
  "model": "publishers/google/models/gemini-3.5-flash",
  "inputConfig": {
    "instancesFormat": "bigquery",
    "bigquerySource": {
      "inputUri" : "INPUT_URI"
    }
  },
  "outputConfig": {
    "predictionsFormat": "OUTPUT_FORMAT",
    "DESTINATION": {
      "OUTPUT_URI_FIELD_NAME": "OUTPUT_URI"
    }
  },
  "state": "JOB_STATE_PENDING",
  "createTime": "2024-10-16T19:33:59.153782Z",
  "updateTime": "2024-10-16T19:33:59.153782Z",
  "modelVersionId": "1"
}
```
The response includes a unique identifier for the batch job. You can poll for the status of the batch job using the <var translate="no">BATCH_JOB_ID</var>. For more information, see [Monitor the job status](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/capabilities/batch-inference/new-job-from-bigquery.md.txt#monitor). Note: Custom Service account, live progress, CMEK, and VPCSC reports are not supported.

### Python

#### Install

```
pip install --upgrade google-genai
```


To learn more, see the
[SDK reference documentation](https://googleapis.github.io/python-genai/).


Set environment variables to use the Google Gen AI SDK with Vertex AI:

```bash
# Replace the `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` values
# with appropriate values for your project.
export GOOGLE_CLOUD_PROJECT=GOOGLE_CLOUD_PROJECT
export GOOGLE_CLOUD_LOCATION=global
export GOOGLE_GENAI_USE_ENTERPRISE=True
```

<br />

    import time

    from google import genai
    from google.genai.types import CreateBatchJobConfig, HttpOptions, JobState

    client = genai.Client(http_options=HttpOptions(api_version="v1"))

    # TODO(developer): Update and un-comment below line
    # output_uri = f"bq://your-project.your_dataset.your_table"

    job = client.batches.create(
        # To use a tuned model, set the model param to your tuned model using the following format:
        # model="projects/{PROJECT_ID}/locations/{LOCATION}/models/{MODEL_ID}
        model="gemini-3.5-flash",
        src="bq://storage-samples.generative_ai.batch_requests_for_multimodal_input",
        config=CreateBatchJobConfig(dest=output_uri),
    )
    print(f"Job name: {job.name}")
    print(f"Job state: {job.state}")
    # Example response:
    # Job name: projects/.../locations/.../batchPredictionJobs/9876453210000000000
    # Job state: JOB_STATE_PENDING

    # See the documentation: https://googleapis.github.io/python-genai/genai.html#genai.types.BatchJob
    completed_states = {
        JobState.JOB_STATE_SUCCEEDED,
        JobState.JOB_STATE_FAILED,
        JobState.JOB_STATE_CANCELLED,
        JobState.JOB_STATE_PAUSED,
    }

    while job.state not in completed_states:
        time.sleep(30)
        job = client.batches.get(name=job.name)
        print(f"Job state: {job.state}")
    # Example response:
    # Job state: JOB_STATE_PENDING
    # Job state: JOB_STATE_RUNNING
    # Job state: JOB_STATE_RUNNING
    # ...
    # Job state: JOB_STATE_SUCCEEDED

## 3. Monitor the job status and progress

After the job is submitted, you can check the status of your batch job using
API, SDK and Google Cloud console.

### Console

1. Go to the **Batch Inference** page.

   <br />

   <br />

   [Go to Batch Inference](https://console.cloud.google.com/agent-platform/batch-predictions)
2. Select your batch job to monitor its progress.

### REST

To monitor a batch inference job, use the
[`projects.locations.batchPredictionJobs.get`](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.batchPredictionJobs/get.md.txt) method and view the [`CompletionStats`](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.batchPredictionJobs.md.txt#CompletionStats) field in the response.


Before using any of the request data,
make the following replacements:

- <var class="edit" scope="ENDPOINT_PREFIX" translate="no">ENDPOINT_PREFIX</var>: The region of the model resource followed by `-`. For example, `us-central1-`. If using the global endpoint, leave blank.
- <var class="edit" scope="LOCATION" translate="no">LOCATION</var>: A region that supports Gemini models. If using the global endpoint, enter `global`.
- <var class="edit" scope="PROJECT_ID" translate="no">PROJECT_ID</var>: Your project ID.
- <var class="edit" scope="BATCH_JOB_ID" translate="no">BATCH_JOB_ID</var>: Your batch job ID.


HTTP method and URL:

```
GET https://ENDPOINT_PREFIXaiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/batchPredictionJobs/BATCH_JOB_ID
```

To send your request, choose one of these options:

#### curl

> [!NOTE]
> **Note:** The following command assumes that you have logged in to the `gcloud` CLI with your user account by running [`gcloud init`](https://docs.cloud.google.com/sdk/gcloud/reference/init) or [`gcloud auth login`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) , or by using [Cloud Shell](https://docs.cloud.google.com/shell/docs.md.txt), which automatically logs you into the `gcloud` CLI . You can check the currently active account by running [`gcloud auth list`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/list.md.txt).


Execute the following command:

```
curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://ENDPOINT_PREFIXaiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/batchPredictionJobs/BATCH_JOB_ID"
```

#### PowerShell

> [!NOTE]
> **Note:** The following command assumes that you have logged in to the `gcloud` CLI with your user account by running [`gcloud init`](https://docs.cloud.google.com/sdk/gcloud/reference/init) or [`gcloud auth login`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) . You can check the currently active account by running [`gcloud auth list`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/list.md.txt).


Execute the following command:

```
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://ENDPOINT_PREFIXaiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/batchPredictionJobs/BATCH_JOB_ID" | Select-Object -Expand Content
```

You should receive a JSON response similar to the following.

#### Response

```
{
  "name": "projects/PROJECT_ID/locations/LOCATION/batchPredictionJobs/BATCH_JOB_ID",
  "displayName": "my-cloud-storage-batch-prediction-job",
  "model": "publishers/google/models/gemini-3.5-flash",
  ...
  "state": "JOB_STATE_PENDING",
  "completionStats": {
    "successfulCount": "10",
    "failedCount": "1"
  },
  "createTime": "2024-10-16T19:33:59.153782Z",
  "updateTime": "2024-10-16T19:33:59.153782Z",
  "modelVersionId": "1"
}
```

### Python

#### Install

```
pip install --upgrade google-genai
```


To learn more, see the
[SDK reference documentation](https://googleapis.github.io/python-genai/).


Set environment variables to use the Google Gen AI SDK with Vertex AI:

```bash
# Replace the `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` values
# with appropriate values for your project.
export GOOGLE_CLOUD_PROJECT=GOOGLE_CLOUD_PROJECT
export GOOGLE_CLOUD_LOCATION=global
export GOOGLE_GENAI_USE_ENTERPRISE=True
```

<br />

        from google import genai
        from google.genai.types import HttpOptions

        client = genai.Client(http_options=HttpOptions(api_version="v1"))

        # Get the batch job
    # Eg. batch_job_name = "projects/123456789012/locations/.../batchPredictionJobs/1234567890123456789"
        batch_job = client.batches.get(name=batch_job_name)

        print(f"Job state: {batch_job.state}")
        # Example response:
        # Job state: JOB_STATE_PENDING
        # Job state: JOB_STATE_RUNNING
        # Job state: JOB_STATE_SUCCEEDED

The status of the a given batch job can be any of the following:

- `JOB_STATE_PENDING`: Queue for capacity. The job can be in `queue` state up to 72-hour before entering `running` state.
- `JOB_STATE_RUNNING`: The input file was successfully validated and the batch is being run.
- `JOB_STATE_SUCCEEDED`: The batch has been completed and the results are ready
- `JOB_STATE_FAILED`: the input file has failed the validation process, or couldn't be completed within the 24-hour time window after entering `RUNNING` state.
- `JOB_STATE_CANCELLING`: The batch is being cancelled.
- `JOB_STATE_CANCELLED`: The batch was cancelled.

## 4. Retrieve batch output

When a batch inference task completes, the output is stored in the
BigQuery table that you specified in your request.

For succeeded rows, model responses are stored in the `response` column.
Otherwise, error details are stored in the `status` column for further
inspection.

### Output example

**Successful example**

```json
{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": "In a medium bowl, whisk together the flour, baking soda, baking powder."
          }
        ]
      },
      "finishReason": "STOP",
      "safetyRatings": [
        {
          "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
          "probability": "NEGLIGIBLE",
          "probabilityScore": 0.14057204,
          "severity": "HARM_SEVERITY_NEGLIGIBLE",
          "severityScore": 0.14270912
        }
      ]
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 8,
    "candidatesTokenCount": 396,
    "totalTokenCount": 404
  }
}
```

**Failed example**

- Request

  ```json
  {"contents":[{"parts":{"text":"Explain how AI works in a few words."},"role":"tester"}]}
  ```
- Response

  ```json
  Bad Request: {"error": {"code": 400, "message": "Please use a valid role: user, model.", "status": "INVALID_ARGUMENT"}}
  ```

## Resume an incomplete batch job

If a batch inference job fails, is canceled, or
expires, you can resume it by creating a new batch job. The new job only
processes the incomplete or failed requests from the previous job, merging the
new results with those from the previously completed requests. For example, if
the incomplete batch job was 90% complete, the new job processes the remaining
10% of the requests.

To resume a batch job, create a new batch job, supplying the
[Cloud Storage file](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/capabilities/batch-inference/new-job-from-cloud-storage.md.txt)
or [BigQuery table output](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/capabilities/batch-inference/new-job-from-bigquery.md.txt)
as input to the new job.

> [!NOTE]
> **Note:** In rare cases where users modify the output file of the job before resuming, the system resumes from the modified output file. The system doesn't validate whether the output file has been modified.