> [!NOTE]
> **Note:** For the most updated computation-based evaluation features, see [Define your metrics](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/determine-eval.md.txt#computation-metrics).

You can evaluate the performance of foundation models and your tuned generative
AI models on Gemini Enterprise Agent Platform. The models are evaluated using a set of metrics
against an evaluation dataset that you provide. This page explains how
computation-based model evaluation through the evaluation pipeline service
works, how to create and format the evaluation dataset, and how to perform the
evaluation using the Google Cloud console, Agent Platform API, or the
Agent Platform SDK for Python.

## How computation-based model evaluation works

To evaluate the performance of a model, you first create an evaluation dataset
that contains prompt and ground truth pairs. For each pair, the prompt is the
input that you want to evaluate, and the ground truth is the ideal response for
that prompt. During evaluation, the prompt in each pair of the evaluation
dataset is passed to the model to produce an output. The output generated by the
model and the ground truth from the evaluation dataset are used to compute the
evaluation metrics.

The type of metrics used for evaluation depends on the task that you are
evaluating. The following table shows the supported tasks and the metrics used
to evaluate each task:

| Task | Metric |
|---|---|
| Classification | Micro-F1, Macro-F1, Per class F1 |
| Summarization | ROUGE-L |
| Question answering | Exact Match |
| Text generation | BLEU, ROUGE-L |

## Supported models

Model evaluation is supported for the following models:

- **`text-bison`**: Base and tuned versions.

- **Gemini**: All tasks except classification.

## Prepare evaluation dataset

The evaluation dataset that's used for model evaluation includes prompt and
ground truth pairs that align with the task that you want to evaluate. Your
dataset must include a minimum of 1 prompt and ground truth pair and at least
10 pairs for meaningful metrics. The more examples you
give, the more meaningful the results.

### Dataset format

Your evaluation dataset must be in [JSON Lines](https://jsonlines.org/) (JSONL)
format where each line contains a single prompt and ground truth pair specified
in the `input_text` and `output_text` fields, respectively. The `input_text`
field contains the prompt that you want to evaluate, and the `output_text` field
contains the ideal response for the prompt.

The maximum token length for `input_text` is 8,192, and the maximum token length
for `output_text` is 1,024.

## Upload evaluation dataset to Cloud Storage

You can either
[create a new Cloud Storage bucket](https://docs.cloud.google.com/storage/docs/creating-buckets.md.txt#create_a_new_bucket)
or use an existing one to store your dataset file. The bucket must be in the
same region as the model.

After your bucket is ready,
[upload](https://docs.cloud.google.com/storage/docs/creating-buckets.md.txt#create_a_new_bucket)
your dataset file to the bucket.

## Perform model evaluation

You can evaluate models by using the REST API or the Google Cloud console.

#### Permissions required for this task

To perform this task, you must grant [Identity and Access Management (IAM)](https://docs.cloud.google.com/iam/docs.md.txt) roles to each of the following service accounts:

| Service account | Default principal | Description | Roles |
|---|---|---|---|
| Agent Platform Service Agent | `service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com` | The Agent Platform Service Agent is automatically provisioned for your project and granted a predefined role. However, if an org policy modifies the default permissions of the Agent Platform Service Agent, you must manually grant the role to the service agent. | [Vertex AI Service Agent (`roles/aiplatform.serviceAgent`)](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/general/access-control.md.txt#aiplatform.serviceAgent) |
| Agent Platform Pipelines Service Account | `PROJECT_NUMBER-compute@developer.gserviceaccount.com` | The service account that runs the pipeline. The default service account used is the [Compute Engine default service account](https://docs.cloud.google.com/compute/docs/access/service-accounts.md.txt#default_service_account). Optionally, you can use a custom service account instead of the default service account. | - [Agent Platform User (`roles/aiplatform.user`)](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/general/access-control.md.txt#aiplatform.user) - [Storage Object User (`roles/storage.objectUser`)](https://docs.cloud.google.com/storage/docs/access-control/iam-roles.md.txt#standard-roles) |

Depending on your input and output data sources, you may also need to grant the Agent Platform Pipelines
Service Account additional roles:

| Data source | Role | Where to grant the role |
|---|---|---|
| [Standard BigQuery table](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#standard_tables) | [BigQuery Data Editor](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.dataEditor) | Project that runs the pipeline |
| [Standard BigQuery table](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#standard_tables) | [BigQuery Data Viewer](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.dataViewer) | Project that the table belongs to |
| [BigQuery view](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#views) of a [standard BigQuery table](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#standard_tables) | [BigQuery Data Editor](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.dataEditor) | Project that runs the pipeline |
| [BigQuery view](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#views) of a [standard BigQuery table](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#standard_tables) | [BigQuery Data Viewer](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.dataViewer) | Project that the view belongs to |
| [BigQuery view](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#views) of a [standard BigQuery table](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#standard_tables) | [BigQuery Data Viewer](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.dataViewer) | Project that the table belongs to |
| [BigQuery external table](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#external_tables) that has a source Cloud Storage file | [BigQuery Data Editor](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.dataEditor) | Project that runs the pipeline |
| [BigQuery external table](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#external_tables) that has a source Cloud Storage file | [BigQuery Data Viewer](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.dataViewer) | Project that the external table belongs to |
| [BigQuery external table](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#external_tables) that has a source Cloud Storage file | [Storage Object Viewer](https://docs.cloud.google.com/storage/docs/access-control/iam-roles.md.txt) | Project that the source file belongs to |
| [BigQuery view](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#views) of a [BigQuery external table](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#external_tables) that has a source Cloud Storage file | [BigQuery Data Editor](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.dataEditor) | Project that runs the pipeline |
| [BigQuery view](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#views) of a [BigQuery external table](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#external_tables) that has a source Cloud Storage file | [BigQuery Data Viewer](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.dataViewer) | Project that the view belongs to |
| [BigQuery view](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#views) of a [BigQuery external table](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#external_tables) that has a source Cloud Storage file | [BigQuery Data Viewer](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.dataViewer) | Project that the external table belongs to |
| [BigQuery view](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#views) of a [BigQuery external table](https://docs.cloud.google.com/bigquery/docs/tables-intro.md.txt#external_tables) that has a source Cloud Storage file | [Storage Object Viewer](https://docs.cloud.google.com/storage/docs/access-control/iam-roles.md.txt) | Project that the source file belongs to |
| Cloud Storage file | [BigQuery Data Viewer](https://docs.cloud.google.com/bigquery/docs/access-control.md.txt#bigquery.dataViewer) | Project that runs the pipeline |

### REST


To create a model evaluation job, send a `POST` request by using the
[pipelineJobs](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.pipelineJobs/create.md.txt) method.


Before using any of the request data,
make the following replacements:

- <var class="edit" scope="PROJECT_ID" translate="no">PROJECT_ID</var>: The Google Cloud project that runs the pipeline components.
- <var class="edit" scope="PIPELINEJOB_DISPLAYNAME" translate="no">PIPELINEJOB_DISPLAYNAME</var>: A display name for the pipelineJob.
- <var class="edit" scope="LOCATION" translate="no">LOCATION</var>: The region to run the pipeline components. Currently, only `us-central1` is supported.
- <var class="edit" scope="DATASET_URI" translate="no">DATASET_URI</var>: The Cloud Storage URI of your reference dataset. You can specify one or multiple URIs. This parameter supports [wildcards](https://docs.cloud.google.com/storage/docs/wildcards.md.txt). To learn more about this parameter, see [InputConfig](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.batchPredictionJobs.md.txt#InputConfig).
- <var class="edit" scope="OUTPUT_DIR" translate="no">OUTPUT_DIR</var>: The Cloud Storage URI to store evaluation output.
- <var class="edit" scope="MODEL_NAME" translate="no">MODEL_NAME</var>: Specify a publisher model or a tuned model resource as follows:
  - **Publisher model:** `publishers/google/models/MODEL@MODEL_VERSION`

    Example: `publishers/google/models/text-bison@002`
  - **Tuned model:** `projects/PROJECT_NUMBER/locations/LOCATION/models/ENDPOINT_ID`

    Example: `projects/123456789012/locations/us-central1/models/1234567890123456789`

  The evaluation job doesn't impact any existing deployments of the model or their resources.
- <var class="edit" scope="EVALUATION_TASK" translate="no">EVALUATION_TASK</var>: The task that you want to evaluate the model on. The evaluation job computes a set of metrics relevant to that specific task. Acceptable values include the following:
  - `summarization`
  - `question-answering`
  - `text-generation`
  - `classification`
- <var class="edit" scope="INSTANCES_FORMAT" translate="no">INSTANCES_FORMAT</var>: The format of your dataset. Currently, only `jsonl` is supported. To learn more about this parameter, see [InputConfig](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.batchPredictionJobs.md.txt#InputConfig).
- <var class="edit" scope="PREDICTIONS_FORMAT" translate="no">PREDICTIONS_FORMAT</var>: The format of the evaluation output. Currently, only `jsonl` is supported. To learn more about this parameter, see [InputConfig](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.batchPredictionJobs.md.txt#InputConfig).
- <var class="edit" scope="MACHINE_TYPE" translate="no">MACHINE_TYPE</var>: (Optional) The machine type for running the evaluation job. The default value is `e2-highmem-16`. For a list of supported machine types, see [Machine types](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/training/configure-compute.md.txt#machine-types).
- <var class="edit" scope="SERVICE_ACCOUNT" translate="no">SERVICE_ACCOUNT</var>: (Optional) The service account to use for running the evaluation job. To learn how to create a custom service account, see [Configure a service account with granular permissions](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/pipelines/configure-project.md.txt#service-account). If unspecified, the [Gemini Enterprise Agent Platform Custom Code Service Agent](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/general/access-control.md.txt#service-agents) is used.
- <var class="edit" scope="NETWORK" translate="no">NETWORK</var>: (Optional) The fully qualified name of the Compute Engine network to peer the evaluatiuon job to. The format of the network name is `projects/PROJECT_NUMBER/global/networks/NETWORK_NAME`. If you specify this field, you need to have a [VPC
  Network Peering for Gemini Enterprise Agent Platform](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/general/vpc-peering). If left unspecified, the evaluation job is not peered with any network.
- <var class="edit" scope="KEY_NAME" translate="no">KEY_NAME</var>: (Optional) The name of the customer-managed encryption key (CMEK). If configured, resources created by the evaluation job is encrypted using the provided encryption key. The format of the key name is `projects/PROJECT_ID/locations/REGION/keyRings/KEY_RING/cryptoKeys/KEY`. The key needs to be in the same region as the evaluation job.


HTTP method and URL:

```
POST https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/pipelineJobs
```


Request JSON body:

```
{
  "displayName": "PIPELINEJOB_DISPLAYNAME",
  "runtimeConfig": {
    "gcsOutputDirectory": "gs://OUTPUT_DIR",
    "parameterValues": {
      "project": "PROJECT_ID",
      "location": "LOCATION",
      "batch_predict_gcs_source_uris": ["gs://DATASET_URI"],
      "batch_predict_gcs_destination_output_uri": "gs://OUTPUT_DIR",
      "model_name": "MODEL_NAME",
      "evaluation_task": "EVALUATION_TASK",
      "batch_predict_instances_format": "INSTANCES_FORMAT",
      "batch_predict_predictions_format: "PREDICTIONS_FORMAT",
      "machine_type": "MACHINE_TYPE",
      "service_account": "SERVICE_ACCOUNT",
      "network": "NETWORK",
      "encryption_spec_key_name": "KEY_NAME"
    }
  },
  "templateUri": "https://us-kfp.pkg.dev/vertex-evaluation/pipeline-templates/evaluation-llm-text-generation-pipeline/1.0.1"
}
```

To send your request, choose one of these options:

#### curl

> [!NOTE]
> **Note:** The following command assumes that you have logged in to the `gcloud` CLI with your user account by running [`gcloud init`](https://docs.cloud.google.com/sdk/gcloud/reference/init) or [`gcloud auth login`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) , or by using [Cloud Shell](https://docs.cloud.google.com/shell/docs.md.txt), which automatically logs you into the `gcloud` CLI . You can check the currently active account by running [`gcloud auth list`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/list.md.txt).


Save the request body in a file named `request.json`,
and execute the following command:

```
curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/pipelineJobs"
```

#### PowerShell

> [!NOTE]
> **Note:** The following command assumes that you have logged in to the `gcloud` CLI with your user account by running [`gcloud init`](https://docs.cloud.google.com/sdk/gcloud/reference/init) or [`gcloud auth login`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/login.md.txt) . You can check the currently active account by running [`gcloud auth list`](https://docs.cloud.google.com/sdk/gcloud/reference/auth/list.md.txt).


Save the request body in a file named `request.json`,
and execute the following command:

```
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/pipelineJobs" | Select-Object -Expand Content
```

You should receive a JSON response similar to the following. Note that `pipelineSpec`
has been truncated to save space.

#### Response

```
......
.....
 "state": "PIPELINE_STATE_PENDING",
  "labels": {
    "vertex-ai-pipelines-run-billing-id": "1234567890123456789"
  },
  "runtimeConfig": {
    "gcsOutputDirectory": "gs://my-evaluation-bucket/output",
    "parameterValues": {
      "project": "my-project",
      "location": "us-central1",
      "batch_predict_gcs_source_uris": [
        "gs://my-evaluation-bucket/reference-datasets/eval_data.jsonl"
      ],
      "batch_predict_gcs_destination_output_uri": "gs://my-evaluation-bucket/output",
      "model_name": "publishers/google/models/text-bison@002"
    }
  },
  "serviceAccount": "123456789012-compute@developer.gserviceaccount.com",
  "templateUri": "https://us-kfp.pkg.dev/vertex-evaluation/pipeline-templates/evaluation-llm-text-generation-pipeline/1.0.1",
  "templateMetadata": {
    "version": "sha256:d4c0d665533f6b360eb474111aa5e00f000fb8eac298d367e831f3520b21cb1a"
  }
}
```

#### Example curl command

```json
PROJECT_ID=myproject
REGION=us-central1
MODEL_NAME=publishers/google/models/text-bison@002
TEST_DATASET_URI=gs://my-gcs-bucket-uri/dataset.jsonl
OUTPUT_DIR=gs://my-gcs-bucket-uri/output

curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
"https://${REGION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${REGION}/pipelineJobs" -d \
$'{
  "displayName": "evaluation-llm-text-generation-pipeline",
  "runtimeConfig": {
    "gcsOutputDirectory": "'${OUTPUT_DIR}'",
    "parameterValues": {
      "project": "'${PROJECT_ID}'",
      "location": "'${REGION}'",
      "batch_predict_gcs_source_uris": ["'${TEST_DATASET_URI}'"],
      "batch_predict_gcs_destination_output_uri": "'${OUTPUT_DIR}'",
      "model_name": "'${MODEL_NAME}'",
    }
  },
  "templateUri": "https://us-kfp.pkg.dev/vertex-evaluation/pipeline-templates/evaluation-llm-text-generation-pipeline/1.0.1"
}'
```

### Python


Before trying this sample, follow the Python setup instructions in the
[Agent Platform quickstart using
client libraries](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/start/client-libraries).


To authenticate to Agent Platform, set up Application Default Credentials.
For more information, see

[Set up authentication for a local development environment](https://docs.cloud.google.com/docs/authentication/set-up-adc-local-dev-environment.md.txt).

```python
import os

import vertexai
from vertexai.evaluation import EvalResult, EvalTask

# TODO (Developer) Set environment variables
PROJECT_ID = os.getenv("GOOGLE_CLOUD_PROJECT")
LOCATION_ID = os.getenv("LOCATION_ID", "us-central1")
MODEL_NAME = os.getenv("MODEL_NAME", "gemini-2.5-flash")


def evaluate_model() -> EvalResult:
    """Evaluate the performance of a generative AI model."""

    vertexai.init(project=PROJECT_ID, location=LOCATION_ID)

    # Dataset URI containing input prompts and ground truth labels
    dataset_uri = "gs://cloud-samples-data/ai-platform/generative_ai/llm_classification_bp_input_prompts_with_ground_truth.jsonl"

    metric_column_mapping = {"reference": "ground_truth"}

    # Define evaluation task
    eval_task = EvalTask(
        dataset=dataset_uri,
        metrics=["exact_match"],
        experiment="gemini-classification-eval",
        metric_column_mapping=metric_column_mapping,
    )

    # Define a prompt template so the generative Gemini model produces formatted labels
    prompt_template = (
        "Classify the following text into exactly one category from "
        "[nature, news, sports, health, startups]. Return only the category name:\n\n{prompt}"
    )

    # Evaluate using a modern Gemini model
    eval_result = eval_task.evaluate(
        model=MODEL_NAME,
        prompt_template=prompt_template,
    )

    print("=== SUMMARY METRICS ===")
    print(eval_result.summary_metrics)

    print("\n=== METRICS TABLE SAMPLE ===")
    print(eval_result.metrics_table.head())

    return eval_result

```

### Console

To create a model evaluation job by using the Google Cloud console, perform
the following steps:

1. In the Google Cloud console, go to the **Gemini Enterprise Agent Platform Model Registry** page.

   [Go to
   Gemini Enterprise Agent Platform Model Registry](https://console.cloud.google.com/agent-platform/models)
2. Click the name of the model that you want to evaluate.
3. In the **Evaluate** tab, click **Create evaluation** and configure as follows:
   - **Objective**: Select the task that you want to evaluate.
   - **Target column or field** : (Classification only) Enter the target column for prediction. Example: `ground_truth`.
   - **Source path**: Enter or select the URI of your evaluation dataset.
   - **Output format** : Enter the format of the evaluation output. Only `jsonl` is supported.
   - **Cloud Storage path**: Enter or select the URI to store evaluation output.
   - **Class names**: (Classification only) Enter the list of possible class names.
   - **Number of compute nodes**: Enter the number of compute nodes to run the evaluation job.
   - **Machine type**: Select a machine type to use for running the evaluation job.
4. Click **Start evaluation**

<br />

## View evaluation results

You can find the evaluation results in the Cloud Storage output directory
that you specified when creating the evaluation job. The file is named
`evaluation_metrics.json`.

For tuned models, you can also view evaluation results in the Google Cloud console:

1. In the Agent Platform section of the Google Cloud console, go to
   the **Gemini Enterprise Agent Platform Model Registry** page.

   [Go to Gemini Enterprise Agent Platform Model Registry](https://console.cloud.google.com/agent-platform/models)
2. Click the name of the model to view its evaluation metrics.

3. In the **Evaluate** tab, click the name of the evaluation run that you want
   to view.

## What's next

- Learn about [generative AI
  evaluation](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/evaluation-overview).
- Learn how to [tune a foundation
  model](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/tuning).