REST Resource: projects.locations.apps.evaluationRuns

Resource: EvaluationRun

An evaluation run represents an all the evaluation results from an evaluation execution.

JSON representation 
{
  "name": string,
  "displayName": string,
  "evaluationResults": [
    string
  ],
  "createTime": string,
  "initiatedBy": string,
  "appVersion": string,
  "appVersionDisplayName": string,
  "changelog": string,
  "evaluations": [
    string
  ],
  "evaluationDataset": string,
  "evaluationType": enum (EvaluationRun.EvaluationType),
  "state": enum (EvaluationRun.EvaluationRunState),
  "progress": {
    object (EvaluationRun.Progress)
  },
  "config": {
    object (EvaluationConfig)
  },
  "error": {
    object (Status)
  },
  "errorInfo": {
    object (EvaluationErrorInfo)
  },
  "evaluationRunSummaries": {
    string: {
      object (EvaluationRun.EvaluationRunSummary)
    },
    ...
  },
  "runCount": integer,
  "personaRunConfigs": [
    {
      object (PersonaRunConfig)
    }
  ],
  "optimizationConfig": {
    object (OptimizationConfig)
  },
  "scheduledEvaluationRun": string,
  "goldenRunMethod": enum (GoldenRunMethod)
}
Fields
name

string

Identifier. The unique identifier of the evaluation run. Format: projects/{project}/locations/{location}/apps/{app}/evaluationRuns/{evaluationRun}
displayName

string

Optional. User-defined display name of the evaluation run. default: " run - ".
evaluationResults[]

string

Output only. The evaluation results that are part of this run. Format: projects/{project}/locations/{location}/apps/{app}/evaluations/{evaluation}/results/{result}
createTime

string (Timestamp format)

Output only. Timestamp when the evaluation run was created.

Uses RFC 3339, where generated output will always be Z-normalized and use 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: "2014-10-02T15:01:23Z", "2014-10-02T15:01:23.045123456Z" or "2014-10-02T15:01:23+05:30".
initiatedBy

string

Output only. The user who initiated the evaluation run.
appVersion

string

Output only. The app version to evaluate. Format: projects/{project}/locations/{location}/apps/{app}/versions/{version}
appVersionDisplayName

string

Output only. The display name of the appVersion that the evaluation ran against.
changelog

string

Output only. The changelog of the app version that the evaluation ran against. This is populated if user runs evaluation on latest/draft.
evaluations[]

string

Output only. The evaluations that are part of this run. The list must contain evaluations of the same type, either all golden or all scenario. This field is mutually exclusive with evaluationDataset. Format: projects/{project}/locations/{location}/apps/{app}/evaluations/{evaluation}
evaluationDataset

string

Output only. The evaluation dataset that this run is associated with. This field is mutually exclusive with evaluations. Format: projects/{project}/locations/{location}/apps/{app}/evaluationDatasets/{evaluationDataset}
evaluationType

enum (EvaluationRun.EvaluationType)

Output only. The type of the evaluations in this run.
state

enum (EvaluationRun.EvaluationRunState)

Output only. The state of the evaluation run.
progress

object (EvaluationRun.Progress)

Output only. The progress of the evaluation run.
config

object (EvaluationConfig)

Output only. The configuration used in the run.
error
(deprecated)

object (Status)

Output only. Deprecated: Use errorInfo instead. Errors encountered during execution.
errorInfo

object (EvaluationErrorInfo)

Output only. Error information for the evaluation run.
evaluationRunSummaries

map (key: string, value: object (EvaluationRun.EvaluationRunSummary))

Output only. Map of evaluation name to EvaluationRunSummary.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.
runCount

integer

Output only. The number of times the evaluations inside the run were run.
personaRunConfigs[]

object (PersonaRunConfig)

Output only. The configuration to use for the run per persona.
optimizationConfig

object (OptimizationConfig)

Optional. Configuration for running the optimization step after the evaluation run. If not set, the optimization step will not be run.
scheduledEvaluationRun

string

Output only. The scheduled evaluation run resource name that created this evaluation run. This field is only set if the evaluation run was created by a scheduled evaluation run. Format: projects/{project}/locations/{location}/apps/{app}/scheduledEvaluationRuns/{scheduledEvaluationRun}
goldenRunMethod

enum (GoldenRunMethod)

Output only. The method used to run the evaluation.

EvaluationRun.EvaluationType

The type of the evaluations in this run.

Enums
EVALUATION_TYPE_UNSPECIFIED Evaluation type is not specified.
GOLDEN Golden evaluation.
SCENARIO Scenario evaluation.

EvaluationRun.EvaluationRunState

The state of the evaluation run.

Enums
EVALUATION_RUN_STATE_UNSPECIFIED Evaluation run state is not specified.
RUNNING Evaluation run is running.
COMPLETED Evaluation run has completed.
ERROR The evaluation run has an error.

EvaluationRun.Progress

The progress of the evaluation run.

JSON representation 
{
  "totalCount": integer,
  "failedCount": integer,
  "errorCount": integer,
  "completedCount": integer,
  "passedCount": integer
}
Fields
totalCount

integer

Output only. Total number of evaluation results in this run.
failedCount

integer

Output only. Number of completed evaluation results with an outcome of FAIL. (EvaluationResult.execution_state is COMPLETED and EvaluationResult.evaluation_status is FAIL).
errorCount

integer

Output only. Number of evaluation results that failed to execute. (EvaluationResult.execution_state is ERROR).
completedCount

integer

Output only. Number of evaluation results that finished successfully. (EvaluationResult.execution_state is COMPLETED).
passedCount

integer

Output only. Number of completed evaluation results with an outcome of PASS. (EvaluationResult.execution_state is COMPLETED and EvaluationResult.evaluation_status is PASS).

EvaluationRun.EvaluationRunSummary

Contains the summary of passed and failed result counts for a specific evaluation in an evaluation run.

JSON representation 
{
  "passedCount": integer,
  "failedCount": integer,
  "errorCount": integer
}
Fields
passedCount

integer

Output only. Number of passed results for the associated Evaluation in this run.
failedCount

integer

Output only. Number of failed results for the associated Evaluation in this run.
errorCount

integer

Output only. Number of error results for the associated Evaluation in this run.

PersonaRunConfig

Configuration for running an evaluation for a specific persona.

JSON representation 
{
  "persona": string,
  "taskCount": integer
}
Fields
persona

string

Optional. The persona to use for the evaluation. Format: projects/{project}/locations/{location}/apps/{app}/evaluationPersonas/{evaluationPersona}
taskCount

integer

Optional. The number of tasks to run for the persona.

OptimizationConfig

Configuration for running the optimization step after the evaluation run.

JSON representation 
{
  "generateLossReport": boolean,
  "assistantSessionName": string,
  "reportSummary": string,
  "shouldSuggestFix": boolean,
  "status": enum (OptimizationConfig.OptimizationStatus),
  "errorMessage": string
}
Fields
generateLossReport

boolean

Optional. Whether to generate a loss report.
assistantSessionName

string

Output only. The assistant session to use for the optimization based on this evaluation run. Format: projects/{project}/locations/{location}/apps/{app}/assistantSessions/{assistantSession}
reportSummary

string

Output only. The summary of the loss report.
shouldSuggestFix

boolean

Output only. Whether to suggest a fix for the losses.
status

enum (OptimizationConfig.OptimizationStatus)

Output only. The status of the optimization run.
errorMessage

string

Output only. The error message if the optimization run failed.

OptimizationConfig.OptimizationStatus

The status of the optimization run.

Enums
OPTIMIZATION_STATUS_UNSPECIFIED Optimization status is not specified.
RUNNING Optimization is running.
COMPLETED Optimization has completed.
ERROR Optimization failed due to an internal error.

Methods

delete

 Deletes an evaluation run.

get

 Gets details of the specified evaluation run.

list

 Lists all evaluation runs in the given app.