This sample demonstrates how to evaluate two Generative AI models for their summarization quality using pairwise comparison. The evaluation uses a metric that assesses how well each model summarizes a given text.

## Explore further


For detailed documentation that includes this code sample, see the following:

- [Gen AI evaluation service API](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/models/evaluation)

## Code sample

### Go


Before trying this sample, follow the Go setup instructions in the
[Vertex AI quickstart using
client libraries](https://docs.cloud.google.com/vertex-ai/docs/start/client-libraries).


For more information, see the
[Vertex AI Go API
reference documentation](https://docs.cloud.google.com/go/docs/reference/cloud.google.com/go/aiplatform/latest/apiv1).


To authenticate to Vertex AI, set up Application Default Credentials.
For more information, see

[Set up authentication for a local development environment](https://docs.cloud.google.com/docs/authentication/set-up-adc-local-dev-environment).

    import (
    	context_pkg "context"
    	"fmt"
    	"io"

    	aiplatform "cloud.google.com/go/aiplatform/apiv1beta1"
    	aiplatformpb "cloud.google.com/go/aiplatform/apiv1beta1/aiplatformpb"
    	"google.golang.org/api/option"
    )

    // pairwiseEvaluation lets the judge model to compare the responses of two models and pick the better one
    func pairwiseEvaluation(w io.Writer, projectID, location string) error {
    	// location = "us-central1"
    	ctx := context_pkg.Background()
    	apiEndpoint := fmt.Sprintf("%s-aiplatform.googleapis.com:443", location)
    	client, err := aiplatform.https://docs.cloud.google.com/go/docs/reference/cloud.google.com/go/aiplatform/latest/apiv1beta1.html#cloud_google_com_go_aiplatform_apiv1beta1_EvaluationClient_NewEvaluationClient(ctx, option.WithEndpoint(apiEndpoint))

    	if err != nil {
    		return fmt.Errorf("unable to create aiplatform client: %w", err)
    	}
    	defer client.Close()

    	context := `
    As part of a comprehensive initiative to tackle urban congestion and foster
    sustainable urban living, a major city has revealed ambitious plans for an
    extensive overhaul of its public transportation system. The project aims not
    only to improve the efficiency and reliability of public transit but also to
    reduce the city\'s carbon footprint and promote eco-friendly commuting options.
    City officials anticipate that this strategic investment will enhance
    accessibility for residents and visitors alike, ushering in a new era of
    efficient, environmentally conscious urban transportation.
    `
    	instruction := "Summarize the text such that a five-year-old can understand."
    	baselineResponse := `
    The city wants to make it easier for people to get around without using cars.
    They're going to make the buses and trains better and faster, so people will want to
    use them more. This will help the air be cleaner and make the city a better place to live.
    `
    	candidateResponse := `
    The city is making big changes to how people get around. They want to make the buses and
    trains work better and be easier for everyone to use. This will also help the environment
    by getting people to use less gas. The city thinks these changes will make it easier for
    everyone to get where they need to go.
    `

    	req := aiplatformpb.EvaluateInstancesRequest{
    		Location: fmt.Sprintf("projects/%s/locations/%s", projectID, location),
    		MetricInputs: &aiplatformpb.EvaluateInstancesRequest_PairwiseSummarizationQualityInput{
    			PairwiseSummarizationQualityInput: &aiplatformpb.PairwiseSummarizationQualityInput{
    				MetricSpec: &aiplatformpb.PairwiseSummarizationQualitySpec{},
    				Instance: &aiplatformpb.PairwiseSummarizationQualityInstance{
    					Context:            &context,
    					Instruction:        &instruction,
    					Prediction:         &candidateResponse,
    					BaselinePrediction: &baselineResponse,
    				},
    			},
    		},
    	}

    	resp, err := client.EvaluateInstances(ctx, &req)
    	if err != nil {
    		return fmt.Errorf("evaluateInstances failed: %v", err)
    	}

    	results := resp.GetPairwiseSummarizationQualityResult()
    	fmt.Fprintf(w, "choice: %s\n", results.GetPairwiseChoice())
    	fmt.Fprintf(w, "confidence: %.2f\n", results.GetConfidence())
    	fmt.Fprintf(w, "explanation:\n%s\n", results.GetExplanation())
    	// Example response:
    	// choice: BASELINE
    	// confidence: 0.50
    	// explanation:
    	// BASELINE response is easier to understand. For example, the phrase "..." is easier to understand than "...". Thus, BASELINE response is ...

    	return nil
    }

### Python


Before trying this sample, follow the Python setup instructions in the
[Vertex AI quickstart using
client libraries](https://docs.cloud.google.com/vertex-ai/docs/start/client-libraries).


For more information, see the
[Vertex AI Python API
reference documentation](https://docs.cloud.google.com/python/docs/reference/aiplatform/latest).


To authenticate to Vertex AI, set up Application Default Credentials.
For more information, see

[Set up authentication for a local development environment](https://docs.cloud.google.com/docs/authentication/set-up-adc-local-dev-environment).

    import pandas as pd

    import https://docs.cloud.google.com/python/docs/reference/vertexai/latest
    from vertexai.generative_models import https://docs.cloud.google.com/python/docs/reference/vertexai/latest/vertexai.preview.generative_models.GenerativeModel.html
    from vertexai.evaluation import (
        EvalTask,
        PairwiseMetric,
        MetricPromptTemplateExamples,
    )

    # TODO(developer): Update & uncomment line below
    # PROJECT_ID = "your-project-id"
    https://docs.cloud.google.com/python/docs/reference/vertexai/latest.init(project=PROJECT_ID, location="us-central1")

    prompt = """
    Summarize the text such that a five-year-old can understand.

    # Text

    As part of a comprehensive initiative to tackle urban congestion and foster
    sustainable urban living, a major city has revealed ambitious plans for an
    extensive overhaul of its public transportation system. The project aims not
    only to improve the efficiency and reliability of public transit but also to
    reduce the city\'s carbon footprint and promote eco-friendly commuting options.
    City officials anticipate that this strategic investment will enhance
    accessibility for residents and visitors alike, ushering in a new era of
    efficient, environmentally conscious urban transportation.
    """

    eval_dataset = pd.DataFrame({"prompt": [prompt]})

    # Baseline model for pairwise comparison
    baseline_model = GenerativeModel("gemini-2.0-flash-lite-001")

    # Candidate model for pairwise comparison
    candidate_model = GenerativeModel(
        "gemini-2.0-flash-001", generation_config={"temperature": 0.4}
    )

    prompt_template = MetricPromptTemplateExamples.get_prompt_template(
        "pairwise_summarization_quality"
    )

    summarization_quality_metric = PairwiseMetric(
        metric="pairwise_summarization_quality",
        metric_prompt_template=prompt_template,
        baseline_model=baseline_model,
    )

    eval_task = EvalTask(
        dataset=eval_dataset,
        metrics=[summarization_quality_metric],
        experiment="pairwise-experiment",
    )
    result = eval_task.https://docs.cloud.google.com/python/docs/reference/vertexai/latest/vertexai._genai.evals.Evals.html#vertexai__genai_evals_Evals_evaluate(model=candidate_model)

    baseline_model_response = https://docs.cloud.google.com/python/docs/reference/vertexai/latest/vertexai._genai.types.CheckQueryJobResultDict.html#vertexai__genai_types_CheckQueryJobResultDict_result.metrics_table["baseline_model_response"].iloc[0]
    candidate_model_response = https://docs.cloud.google.com/python/docs/reference/vertexai/latest/vertexai._genai.types.CheckQueryJobResultDict.html#vertexai__genai_types_CheckQueryJobResultDict_result.metrics_table["response"].iloc[0]
    winner_model = https://docs.cloud.google.com/python/docs/reference/vertexai/latest/vertexai._genai.types.CheckQueryJobResultDict.html#vertexai__genai_types_CheckQueryJobResultDict_result.metrics_table[
        "pairwise_summarization_quality/pairwise_choice"
    ].iloc[0]
    explanation = https://docs.cloud.google.com/python/docs/reference/vertexai/latest/vertexai._genai.types.CheckQueryJobResultDict.html#vertexai__genai_types_CheckQueryJobResultDict_result.metrics_table[
        "pairwise_summarization_quality/explanation"
    ].iloc[0]

    print(f"Baseline's story:\n{baseline_model_response}")
    print(f"Candidate's story:\n{candidate_model_response}")
    print(f"Winner: {winner_model}")
    print(f"Explanation: {explanation}")
    # Example response:
    # Baseline's story:
    # A big city wants to make it easier for people to get around without using cars! They're going to make buses and trains ...
    #
    # Candidate's story:
    # A big city wants to make it easier for people to get around without using cars! ... This will help keep the air clean ...
    #
    # Winner: CANDIDATE
    # Explanation: Both responses adhere to the prompt's constraints, are grounded in the provided text, and ... However, Response B ...

## What's next


To search and filter code samples for other Google Cloud products, see the
[Google Cloud sample browser](https://docs.cloud.google.com/docs/samples?product=generativeaionvertexai).