Class CodeExecutionMetric (1.144.0) 

CodeExecutionMetric(
    *,
    name: typing.Optional[str] = None,
    customFunction: typing.Optional[str] = None,
    promptTemplate: typing.Optional[str] = None,
    judgeModelSystemInstruction: typing.Optional[str] = None,
    returnRawOutput: typing.Optional[bool] = None,
    parseAndReduceFn: typing.Optional[typing.Callable[[...], typing.Any]] = None,
    aggregateSummaryFn: typing.Optional[typing.Callable[[...], typing.Any]] = None,
    remoteCustomFunction: typing.Optional[str] = None,
    judgeModel: typing.Optional[str] = None,
    judgeModelGenerationConfig: typing.Optional[
        google.genai.types.GenerationConfig
    ] = None,
    judgeModelSamplingCount: typing.Optional[int] = None,
    rubricGroupName: typing.Optional[str] = None,
    metricSpecParameters: typing.Optional[dict[str, typing.Any]] = None,
    metricResourceName: typing.Optional[str] = None,
    **extra_data: typing.Any
)

A metric that executes custom Python code for evaluation.

Methods

CodeExecutionMetric

CodeExecutionMetric(
    *,
    name: typing.Optional[str] = None,
    customFunction: typing.Optional[str] = None,
    promptTemplate: typing.Optional[str] = None,
    judgeModelSystemInstruction: typing.Optional[str] = None,
    returnRawOutput: typing.Optional[bool] = None,
    parseAndReduceFn: typing.Optional[typing.Callable[[...], typing.Any]] = None,
    aggregateSummaryFn: typing.Optional[typing.Callable[[...], typing.Any]] = None,
    remoteCustomFunction: typing.Optional[str] = None,
    judgeModel: typing.Optional[str] = None,
    judgeModelGenerationConfig: typing.Optional[
        google.genai.types.GenerationConfig
    ] = None,
    judgeModelSamplingCount: typing.Optional[int] = None,
    rubricGroupName: typing.Optional[str] = None,
    metricSpecParameters: typing.Optional[dict[str, typing.Any]] = None,
    metricResourceName: typing.Optional[str] = None,
    **extra_data: typing.Any
)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

model_post_init

model_post_init(context: Any, /) -> None

This function is meant to behave like a BaseModel method to initialise private attributes.

It takes context as an argument since that's what pydantic-core passes when calling it.

validate_code

validate_code(value: typing.Optional[str]) -> typing.Optional[str]

API documentation for validate_code method.