Content

The base structured datatype containing multi-part content of a message.

A Content includes a role field designating the producer of the Content and a parts field containing multi-part data that contains the content of the message turn.

Fields
role string

Optional. The producer of the content. Must be either 'user' or 'model'.

Useful to set for multi-turn conversations, otherwise can be left blank or unset.

parts[] object (Part)

Required. Ordered Parts that constitute a single message. Parts may have different IANA MIME types.

JSON representation 
{
  "role": string,
  "parts": [
    {
      object (Part)
    }
  ]
}

Part

A datatype containing media that is part of a multi-part Content message.

A Part consists of data which has an associated datatype. A Part can only contain one of the accepted types in Part.data.

A Part must have a fixed IANA MIME type identifying the type and subtype of the media if inlineData or fileData field is filled with raw bytes.

Fields
thought boolean

Optional. Indicates if the part is thought from the model.

thoughtSignature string (bytes format)

Optional. An opaque signature for the thought so it can be reused in subsequent requests.

A base64-encoded string.

data Union type
data can be only one of the following:
text string

Optional. Text part (can be code).

inlineData object (Blob)

Optional. Inlined bytes data.

fileData object (FileData)

Optional. URI based data.

functionCall object (FunctionCall)

Optional. A predicted [FunctionCall] returned from the model that contains a string representing the [FunctionDeclaration.name] with the parameters and their values.

functionResponse object (FunctionResponse)

Optional. The result output of a [FunctionCall] that contains a string representing the [FunctionDeclaration.name] and a structured JSON object containing any output from the function call. It is used as context to the model.

executableCode object (ExecutableCode)

Optional. code generated by the model that is meant to be executed.

codeExecutionResult object (CodeExecutionResult)

Optional. result of executing the [ExecutableCode].

metadata Union type
metadata can be only one of the following:
videoMetadata object (VideoMetadata)

Optional. Video metadata. The metadata should only be specified while the video data is presented in inlineData or fileData.

JSON representation 
{
  "thought": boolean,
  "thoughtSignature": string,

  // data
  "text": string,
  "inlineData": {
    object (Blob)
  },
  "fileData": {
    object (FileData)
  },
  "functionCall": {
    object (FunctionCall)
  },
  "functionResponse": {
    object (FunctionResponse)
  },
  "executableCode": {
    object (ExecutableCode)
  },
  "codeExecutionResult": {
    object (CodeExecutionResult)
  }
  // Union type

  // metadata
  "videoMetadata": {
    object (VideoMetadata)
  }
  // Union type
}

Blob

Content blob.

Fields
mimeType string

Required. The IANA standard MIME type of the source data.

data string (bytes format)

Required. Raw bytes.

A base64-encoded string.

JSON representation 
{
  "mimeType": string,
  "data": string
}

FileData

URI based data.

Fields
mimeType string

Required. The IANA standard MIME type of the source data.

fileUri string

Required. URI.

JSON representation 
{
  "mimeType": string,
  "fileUri": string
}

FunctionCall

A predicted [FunctionCall] returned from the model that contains a string representing the [FunctionDeclaration.name] and a structured JSON object containing the parameters and their values.

Fields
id string

Optional. The unique id of the function call. If populated, the client to execute the functionCall and return the response with the matching id.

name string

Optional. The name of the function to call. Matches [FunctionDeclaration.name].

args object (Struct format)

Optional. The function parameters and values in JSON object format. See [FunctionDeclaration.parameters] for parameter details.

JSON representation 
{
  "id": string,
  "name": string,
  "args": {
    object
  }
}

FunctionResponse

The result output from a [FunctionCall] that contains a string representing the [FunctionDeclaration.name] and a structured JSON object containing any output from the function is used as context to the model. This should contain the result of a [FunctionCall] made based on model prediction.

Fields
id string

Optional. The id of the function call this response is for. Populated by the client to match the corresponding function call id.

name string

Required. The name of the function to call. Matches [FunctionDeclaration.name] and [FunctionCall.name].

response object (Struct format)

Required. The function response in JSON object format. Use "output" key to specify function output and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as function output.

JSON representation 
{
  "id": string,
  "name": string,
  "response": {
    object
  }
}

ExecutableCode

code generated by the model that is meant to be executed, and the result returned to the model.

Generated when using the [CodeExecution] tool, in which the code will be automatically executed, and a corresponding [CodeExecutionResult] will also be generated.

Fields
language enum (Language)

Required. Programming language of the code.

code string

Required. The code to be executed.

JSON representation 
{
  "language": enum (Language),
  "code": string
}

Language

Supported programming languages for the generated code.

Enums
LANGUAGE_UNSPECIFIED Unspecified language. This value should not be used.
PYTHON Python >= 3.10, with numpy and simpy available.

CodeExecutionResult

result of executing the [ExecutableCode].

Only generated when using the [CodeExecution] tool, and always follows a part containing the [ExecutableCode].

Fields
outcome enum (Outcome)

Required. Outcome of the code execution.

output string

Optional. Contains stdout when code execution is successful, stderr or other description otherwise.

JSON representation 
{
  "outcome": enum (Outcome),
  "output": string
}

Outcome

Enumeration of possible outcomes of the code execution.

Enums
OUTCOME_UNSPECIFIED Unspecified status. This value should not be used.
OUTCOME_OK code execution completed successfully.
OUTCOME_FAILED code execution finished but with a failure. stderr should contain the reason.
OUTCOME_DEADLINE_EXCEEDED code execution ran for too long, and was cancelled. There may or may not be a partial output present.

VideoMetadata

metadata describes the input video content.

Fields
startOffset string (Duration format)

Optional. The start offset of the video.

A duration in seconds with up to nine fractional digits, ending with 's'. Example: "3.5s".

endOffset string (Duration format)

Optional. The end offset of the video.

A duration in seconds with up to nine fractional digits, ending with 's'. Example: "3.5s".

fps number

Optional. The frame rate of the video sent to the model. If not specified, the default value will be 1.0. The fps range is (0.0, 24.0].

JSON representation 
{
  "startOffset": string,
  "endOffset": string,
  "fps": number
}