public static interface EvaluationParserConfig.CustomCodeParserConfigOrBuilder extends MessageOrBuilderImplements
MessageOrBuilderMethods
getParsingFunction()
public abstract String getParsingFunction()Required. Python function for parsing results. The function should be defined within this string.
The function takes a list of strings (LLM responses) and should return either a list of dictionaries (for rubrics) or a single dictionary (for a metric result).
Example function signature: def parse(responses: list[str]) -> list[dict[str, Any]] | dict[str, Any]:
When parsing rubrics, return a list of dictionaries, where each dictionary represents a Rubric. Example for rubrics: [ { "content": {"property": {"description": "The response is factual."}}, "type": "FACTUALITY", "importance": "HIGH" }, { "content": {"property": {"description": "The response is fluent."}}, "type": "FLUENCY", "importance": "MEDIUM" } ]
When parsing critique results, return a dictionary representing a MetricResult. Example for a metric result: { "score": 0.8, "explanation": "The model followed most instructions.", "rubric_verdicts": [...] }
... code for result extraction and aggregation
optional string parsing_function = 1 [(.google.api.field_behavior) = REQUIRED];
| Returns | |
|---|---|
| Type | Description |
String |
The parsingFunction. |
getParsingFunctionBytes()
public abstract ByteString getParsingFunctionBytes()Required. Python function for parsing results. The function should be defined within this string.
The function takes a list of strings (LLM responses) and should return either a list of dictionaries (for rubrics) or a single dictionary (for a metric result).
Example function signature: def parse(responses: list[str]) -> list[dict[str, Any]] | dict[str, Any]:
When parsing rubrics, return a list of dictionaries, where each dictionary represents a Rubric. Example for rubrics: [ { "content": {"property": {"description": "The response is factual."}}, "type": "FACTUALITY", "importance": "HIGH" }, { "content": {"property": {"description": "The response is fluent."}}, "type": "FLUENCY", "importance": "MEDIUM" } ]
When parsing critique results, return a dictionary representing a MetricResult. Example for a metric result: { "score": 0.8, "explanation": "The model followed most instructions.", "rubric_verdicts": [...] }
... code for result extraction and aggregation
optional string parsing_function = 1 [(.google.api.field_behavior) = REQUIRED];
| Returns | |
|---|---|
| Type | Description |
ByteString |
The bytes for parsingFunction. |
hasParsingFunction()
public abstract boolean hasParsingFunction()Required. Python function for parsing results. The function should be defined within this string.
The function takes a list of strings (LLM responses) and should return either a list of dictionaries (for rubrics) or a single dictionary (for a metric result).
Example function signature: def parse(responses: list[str]) -> list[dict[str, Any]] | dict[str, Any]:
When parsing rubrics, return a list of dictionaries, where each dictionary represents a Rubric. Example for rubrics: [ { "content": {"property": {"description": "The response is factual."}}, "type": "FACTUALITY", "importance": "HIGH" }, { "content": {"property": {"description": "The response is fluent."}}, "type": "FLUENCY", "importance": "MEDIUM" } ]
When parsing critique results, return a dictionary representing a MetricResult. Example for a metric result: { "score": 0.8, "explanation": "The model followed most instructions.", "rubric_verdicts": [...] }
... code for result extraction and aggregation
optional string parsing_function = 1 [(.google.api.field_behavior) = REQUIRED];
| Returns | |
|---|---|
| Type | Description |
boolean |
Whether the parsingFunction field is set. |