public interface CustomCodeExecutionSpecOrBuilder extends MessageOrBuilderImplements
MessageOrBuilderMethods
getEvaluationFunction()
public abstract String getEvaluationFunction()Required. Python function. Expected user to define the following function, e.g.: def evaluate(instance: dict[str, Any]) -> float: Please include this function signature in the code snippet. Instance is the evaluation instance, any fields populated in the instance are available to the function as instance[field_name].
Example:
Example input:
`
instance= EvaluationInstance(
response=EvaluationInstance.InstanceData(text="The answer is 4."),
reference=EvaluationInstance.InstanceData(text="4")
)
Example converted input:
{
'response': {'text': 'The answer is 4.'},
'reference': {'text': '4'}
}
Example python function:
def evaluate(instance: dict[str, Any]) -> float:
if instance'response' == instance'reference':
return 1.0
return 0.0
CustomCodeExecutionSpec is also supported in Batch Evaluation (EvalDataset RPC) and Tuning Evaluation. Each line in the input jsonl file will be converted to dict[str, Any] and passed to the evaluation function.
optional string evaluation_function = 1 [(.google.api.field_behavior) = REQUIRED];
| Returns | |
|---|---|
| Type | Description |
String |
The evaluationFunction. |
getEvaluationFunctionBytes()
public abstract ByteString getEvaluationFunctionBytes()Required. Python function. Expected user to define the following function, e.g.: def evaluate(instance: dict[str, Any]) -> float: Please include this function signature in the code snippet. Instance is the evaluation instance, any fields populated in the instance are available to the function as instance[field_name].
Example:
Example input:
`
instance= EvaluationInstance(
response=EvaluationInstance.InstanceData(text="The answer is 4."),
reference=EvaluationInstance.InstanceData(text="4")
)
Example converted input:
{
'response': {'text': 'The answer is 4.'},
'reference': {'text': '4'}
}
Example python function:
def evaluate(instance: dict[str, Any]) -> float:
if instance'response' == instance'reference':
return 1.0
return 0.0
CustomCodeExecutionSpec is also supported in Batch Evaluation (EvalDataset RPC) and Tuning Evaluation. Each line in the input jsonl file will be converted to dict[str, Any] and passed to the evaluation function.
optional string evaluation_function = 1 [(.google.api.field_behavior) = REQUIRED];
| Returns | |
|---|---|
| Type | Description |
ByteString |
The bytes for evaluationFunction. |
hasEvaluationFunction()
public abstract boolean hasEvaluationFunction()Required. Python function. Expected user to define the following function, e.g.: def evaluate(instance: dict[str, Any]) -> float: Please include this function signature in the code snippet. Instance is the evaluation instance, any fields populated in the instance are available to the function as instance[field_name].
Example:
Example input:
`
instance= EvaluationInstance(
response=EvaluationInstance.InstanceData(text="The answer is 4."),
reference=EvaluationInstance.InstanceData(text="4")
)
Example converted input:
{
'response': {'text': 'The answer is 4.'},
'reference': {'text': '4'}
}
Example python function:
def evaluate(instance: dict[str, Any]) -> float:
if instance'response' == instance'reference':
return 1.0
return 0.0
CustomCodeExecutionSpec is also supported in Batch Evaluation (EvalDataset RPC) and Tuning Evaluation. Each line in the input jsonl file will be converted to dict[str, Any] and passed to the evaluation function.
optional string evaluation_function = 1 [(.google.api.field_behavior) = REQUIRED];
| Returns | |
|---|---|
| Type | Description |
boolean |
Whether the evaluationFunction field is set. |