To customize how Agent Platform serves online inferences from your
custom-trained model, you can specify a custom container instead of a [prebuilt
container](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/predictions/pre-built-containers) when you create a
[`Model`](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.models.md.txt)
resource. When you use a custom container, Agent Platform runs a
Docker container of your choice on each inference node.

You might want to use a custom container for any of the following reasons:

- to serve inferences from an ML model trained using a framework that isn't available as a [prebuilt container](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/predictions/pre-built-containers.md.txt)
- to preprocess inference requests or postprocess the inferences generated by your model
- to run an inference server written in a programming language of your choice
- to install dependencies that you want to use to customize inferences

This guide describes how to create a model
that uses a custom container. It
doesn't provide detailed instructions about designing and creating a Docker
container image.

## Prepare a container image

To create a `Model` that uses a custom container, you must provide a
Docker container image as the basis of that container. This container image must
meet the requirements described in [Custom container
requirements](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/predictions/custom-container-requirements).

If you plan to use an existing container image created by a third party that you
trust, you might be able to skip one or both of the following sections.

### Create a container image

Design and build a Docker container image that meets the [container image
requirements](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/predictions/custom-container-requirements#image).

To learn the basics of designing and building a Docker container image, read the
[Docker documentation's
quickstart](https://docs.docker.com/get-started/)

### Push the container image to Artifact Registry

Push your container image to an
[Artifact Registry](https://docs.cloud.google.com/artifact-registry/docs/overview.md.txt) repository.

Learn how to [push a container image to
Artifact Registry](https://docs.cloud.google.com/artifact-registry/docs/docker/pushing-and-pulling).

## Create a `Model`

To create a `Model` that uses a custom container, do one of the following:

- [Import a custom-trained model.](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/model-registry/import-model.md.txt)

- [Create a `TrainingPipeline` that runs a `CustomJob` and imports the resulting
  artifacts as a
  `Model`.](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/training/create-training-pipeline#custom-job-model-upload)

The follow sections show how to configure the API fields related to custom
containers when you create a `Model` in one of these ways.

### Container-related API fields

When you create the `Model`, make sure to configure the [`containerSpec`
field](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.models#Model.FIELDS.container_spec) with
your custom container details, rather than with a [prebuilt
container](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/predictions/pre-built-containers).

You must specify a [`ModelContainerSpec`
message](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.models#modelcontainerspec) in
the `Model.containerSpec` field. Within this message, you can specify the
following subfields:

`imageUri` (required)

:   The Artifact Registry URI of your container image. Specify a digest
    (`IMAGE@sha256:DIGEST`), a tag
    (`IMAGE:TAG`), or neither (defaults to
    `:latest`); don't specify both a tag and a digest in the same URI. For
    details, see [Image URI
    format](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/predictions/custom-container-requirements#image-uri-format).

    If you are using the
    [`gcloud ai models upload`
    command](https://docs.cloud.google.com/sdk/gcloud/reference/ai/models/upload),
    you can use the `--container-image-uri` flag to specify this field.

`command` (optional)

:   An array of an executable and arguments to override the container's
    [`ENTRYPOINT` instruction](https://docs.docker.com/engine/reference/builder/#entrypoint).
    To learn more about how to format this field and how it
    interacts with the `args` field, read the [API reference for
    `ModelContainerSpec`](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.models#modelcontainerspec).

    If you are using the
    `gcloud ai models upload` command,
    you can use the `--container-command` flag to specify this field.

`args` (optional)

:   An array of an executable and arguments to override the container's
    [`CMD`](https://docs.docker.com/engine/reference/builder/#cmd)
    To learn more about how to format this field and how it interacts with the
    `command` field, read the [API reference for
    `ModelContainerSpec`](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.models#modelcontainerspec).

    If you are using the
    `gcloud ai models upload` command,
    you can use the `--container-args` flag to specify this field.

`ports` (optional)

:   An array of ports; Agent Platform sends [liveness checks, health
    checks, and inference
    requests](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/predictions/custom-container-requirements#image) to your
    container on the first port listed, or `8080` by default. Specifying
    additional ports has no effect.

    If you are using the
    `gcloud ai models upload` command,
    you can use the `--container-ports` flag to specify this field.

`env` (optional)

:   An array of environment variables that the container's `ENTRYPOINT` instruction,
    as well as the `command` and `args` fields, can reference. To learn more
    about how other fields can reference these environment variables, read the
    [API reference for
    `ModelContainerSpec`](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.models#modelcontainerspec).

    If you are using the
    `gcloud ai models upload` command,
    you can use the `--container-env-vars` flag to specify this field.

`healthRoute` (optional)

:   The path on your container's HTTP server where you want
    Agent Platform to send [health
    checks](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/predictions/custom-container-requirements#health).

    If you don't specify this field, then when you deploy the `Model` as a
    `DeployedModel` to an `Endpoint` resource it defaults to
    `/v1/endpoints/ENDPOINT/deployedModels/DEPLOYED_MODEL`,
    where <var translate="no">ENDPOINT</var> is replaced by the last segment of the
    `Endpoint`'s [`name`
    field](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.endpoints#Endpoint.FIELDS.name)
    (following `endpoints/`) and <var translate="no">DEPLOYED_MODEL</var> is replaced by the
    `DeployedModel`'s [`id`
    field](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.endpoints#DeployedModel.FIELDS.id).

    If you are using the
    `gcloud ai models upload` command,
    you can use the `--container-health-route` flag to specify this field.

`predictRoute` (optional)

:   The path on your container's HTTP server where you want
    Agent Platform to forward [inference
    requests](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/predictions/custom-container-requirements#inference).

    If you don't specify this field, then when you deploy the `Model` as a
    `DeployedModel` to an `Endpoint` resource it defaults to
    `/v1/endpoints/ENDPOINT/deployedModels/DEPLOYED_MODEL:predict`,
    where <var translate="no">ENDPOINT</var> is replaced by the last segment of the
    `Endpoint`'s [`name`
    field](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.endpoints#Endpoint.FIELDS.name)
    (following `endpoints/`) and <var translate="no">DEPLOYED_MODEL</var> is replaced by the
    `DeployedModel`'s [`id`
    field](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/rest/v1/projects.locations.endpoints#DeployedModel.FIELDS.id).

    If you are using the
    `gcloud ai models upload` command,
    you can use the `--container-predict-route` flag to specify this field.

`invokeRoutePrefix` (optional)

:   Invoke route prefix for the custom container. Setting this field to
    "/\*" enables arbitrary routing for the model. Once deployed, any non-root
    route on the model server will be accessible with invoke HTTP call Such as,
    "/invoke/foo/bar" would be forwarded as "/foo/bar" to the model server.
    This feature is in the public preview stage. To create an invoke enabled
    model, follow directions to
    [use arbitrary custom routes](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/predictions/use-arbitrary-custom-routes.md.txt).

`sharedMemorySizeMb` (optional)

:   The amount of VM memory to reserve in a shared memory volume for the model
    in megabytes.

    Shared memory is an Inter-process communication (IPC) mechanism that allows
    multiple processes to access and manipulate a common block of memory. The
    amount of shared memory needed, if any, is an implementation detail of your
    container and model. Consult your model server documentation for guidelines.

    If you are using the
    `gcloud ai models upload` command,
    you can use the `--container-shared-memory-size-mb` flag to specify this field.

`startupProbe` (optional)

:   Specification for the probe that checks whether the container application
    has started.

    If you are using the
    `gcloud ai models upload` command,
    you can use the `--container-startup-probe-exec, --container-startup-probe-period-seconds, --container-startup-probe-timeout-seconds` flag to specify this field.

`healthProbe` (optional)

:   Specification for the probe that checks whether a container is ready to
    accept traffic.

    If you are using the
    `gcloud ai models upload` command,
    you can use the `--container-health-probe-exec, --container-health-probe-period-seconds, --container-health-probe-timeout-seconds` flag to specify this field.

In addition to the variables that you set in the `Model.containerSpec.env`
field, Agent Platform sets several other variables based on your
configuration. Learn more about
[using these environment variables in these fields and in the container's `ENTRYPOINT` instruction](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/predictions/custom-container-requirements.md.txt#variables).

### Model import examples

The following examples show how to specify container-related API fields when you
import a model.

> [!NOTE]
> **Note:** If you use a `TrainingPipeline` to create a `Model`, you must specify these fields in a different context. You can't use the Google Cloud CLI to create a `TrainingPipeline`. [Read more about creating custom `TrainingPipeline`
> resources](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/training/create-training-pipeline).

### gcloud

The following example uses the [`gcloud ai models upload`
command](https://docs.cloud.google.com/sdk/gcloud/reference/ai/models/upload):

```bash
gcloud ai models upload \
  --region=LOCATION \
  --display-name=MODEL_NAME \
  --container-image-uri=IMAGE_URI \\
--container-command=COMMAND \\
--container-args=ARGS \\
--container-ports=PORTS \\
--container-env-vars=ENV \\
--container-health-route=HEALTH_ROUTE \\
--container-predict-route=PREDICT_ROUTE \\
--container-shared-memory-size-mb=SHARED_MEMORY_SIZE \\
--container-startup-probe-exec=STARTUP_PROBE_EXEC \\
--container-startup-probe-period-seconds=STARTUP_PROBE_PERIOD \\
--container-startup-probe-timeout-seconds=STARTUP_PROBE_TIMEOUT \\
--container-health-probe-exec=HEALTH_PROBE_EXEC \\
--container-health-probe-period-seconds=HEALTH_PROBE_PERIOD \\
--container-health-probe-timeout-seconds=HEALTH_PROBE_TIMEOUT \
  --artifact-uri=PATH_TO_MODEL_ARTIFACT_DIRECTORY
```

The `--container-image-uri` flag is required; all other flags that begin
with `--container-` are optional. To learn about the values for these fields,
see the [preceding section of this guide](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/predictions/use-custom-container.md.txt#fields).

### Java


Before trying this sample, follow the Java setup instructions in the
[Agent Platform quickstart using
client libraries](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/start/client-libraries).


To authenticate to Agent Platform, set up Application Default Credentials.
For more information, see

[Set up authentication for a local development environment](https://docs.cloud.google.com/docs/authentication/set-up-adc-local-dev-environment.md.txt).

```java

import com.google.api.gax.longrunning.OperationFuture;
import com.google.cloud.aiplatform.v1.LocationName;
import com.google.cloud.aiplatform.v1.Model;
import com.google.cloud.aiplatform.v1.ModelContainerSpec;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import com.google.cloud.aiplatform.v1.UploadModelOperationMetadata;
import com.google.cloud.aiplatform.v1.UploadModelResponse;
import java.io.IOException;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;

public class UploadModelSample {
  public static void main(String[] args)
      throws InterruptedException, ExecutionException, TimeoutException, IOException {
    // TODO(developer): Replace these variables before running the sample.
    String project = "YOUR_PROJECT_ID";
    String modelDisplayName = "YOUR_MODEL_DISPLAY_NAME";
    String metadataSchemaUri =
        "gs://google-cloud-aiplatform/schema/trainingjob/definition/custom_task_1.0.0.yaml";
    String imageUri = "YOUR_IMAGE_URI";
    String artifactUri = "gs://your-gcs-bucket/artifact_path";
    uploadModel(project, modelDisplayName, metadataSchemaUri, imageUri, artifactUri);
  }

  static void uploadModel(
      String project,
      String modelDisplayName,
      String metadataSchemaUri,
      String imageUri,
      String artifactUri)
      throws IOException, InterruptedException, ExecutionException, TimeoutException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      LocationName locationName = LocationName.of(project, location);

[ModelContainerSpec](https://docs.cloud.google.com/java/docs/reference/google-cloud-aiplatform/latest/com.google.cloud.aiplatform.v1.ModelContainerSpec.html) modelContainerSpec =
[ModelContainerSpec](https://docs.cloud.google.com/java/docs/reference/google-cloud-aiplatform/latest/com.google.cloud.aiplatform.v1.ModelContainerSpec.html).newBuilder().setImageUri(imageUri).build();

      Model model =
          Model.newBuilder()
              .setDisplayName(modelDisplayName)
              .setMetadataSchemaUri(metadataSchemaUri)
              .setArtifactUri(artifactUri)
              .setContainerSpec(modelContainerSpec)
              .build();

      OperationFuture<UploadModelResponse, UploadModelOperationMetadata> uploadModelResponseFuture =
          modelServiceClient.uploadModelAsync(locationName, model);
      System.out.format(
          "Operation name: %s\n", uploadModelResponseFuture.getInitialFuture().get().getName());
      System.out.println("Waiting for operation to finish...");
      UploadModelResponse uploadModelResponse = uploadModelResponseFuture.get(5, TimeUnit.MINUTES);

      System.out.println("Upload Model Response");
      System.out.format("Model: %s\n", uploadModelResponse.getModel());
    }
  }
}
```

### Node.js


Before trying this sample, follow the Node.js setup instructions in the
[Agent Platform quickstart using
client libraries](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/start/client-libraries).


To authenticate to Agent Platform, set up Application Default Credentials.
For more information, see

[Set up authentication for a local development environment](https://docs.cloud.google.com/docs/authentication/set-up-adc-local-dev-environment.md.txt).

```javascript
/**
 * TODO(developer): Uncomment these variables before running the sample.\
 */

// const modelDisplayName = 'YOUR_MODEL_DISPLAY_NAME';
// const metadataSchemaUri = 'YOUR_METADATA_SCHEMA_URI';
// const imageUri = 'YOUR_IMAGE_URI';
// const artifactUri = 'YOUR_ARTIFACT_URI';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function uploadModel() {
  // Configure the parent resources
  const parent = `projects/${project}/locations/${location}`;
  // Configure the model resources
  const model = {
    displayName: modelDisplayName,
    metadataSchemaUri: '',
    artifactUri: artifactUri,
containerSpec: {
imageUri: imageUri,
command: \[\],
args: \[\],
env: \[\],
ports: \[\],
predictRoute: '',
healthRoute: '',
},
  };
  const request = {
    parent,
    model,
  };

  console.log('PARENT AND MODEL');
  console.log(parent, model);
  // Upload Model request
  const [response] = await modelServiceClient.uploadModel(request);
  console.log(`Long running operation : ${response.name}`);

  // Wait for operation to complete
  await response.promise();
  const result = response.result;

  console.log('Upload model response ');
  console.log(`\tModel : ${result.model}`);
}
uploadModel();
```

### Python

To learn how to install or update the Vertex AI SDK for Python, see [Install the Vertex AI SDK for Python](https://docs.cloud.google.com/vertex-ai/docs/start/use-vertex-ai-python-sdk).

For more information, see the
[Python API reference documentation](https://docs.cloud.google.com/python/docs/reference/aiplatform/latest.md.txt).

    from typing import Dict, Optional, Sequence

    from google.cloud import aiplatform
    from google.cloud.aiplatform import explain


    def upload_model_sample(
        project: str,
        location: str,
        display_name: str,
        serving_container_image_uri: str,
        artifact_uri: Optional[str] = None,
        serving_container_predict_route: Optional[str] = None,
        serving_container_health_route: Optional[str] = None,
        description: Optional[str] = None,
        serving_container_command: Optional[Sequence[str]] = None,
        serving_container_args: Optional[Sequence[str]] = None,
        serving_container_environment_variables: Optional[Dict[str, str]] = None,
        serving_container_ports: Optional[Sequence[int]] = None,
        instance_schema_uri: Optional[str] = None,
        parameters_schema_uri: Optional[str] = None,
        prediction_schema_uri: Optional[str] = None,
        explanation_metadata: Optional[explain.ExplanationMetadata] = None,
        explanation_parameters: Optional[explain.ExplanationParameters] = None,
        sync: bool = True,
    ):

    aiplatform.init(project=project, location=location)

        model = aiplatform.Model.upload(
            display_name=display_name,
            artifact_uri=artifact_uri,
            serving_container_image_uri=serving_container_image_uri,
    serving_container_predict_route=serving_container_predict_route,
    serving_container_health_route=serving_container_health_route,
    instance_schema_uri=instance_schema_uri,
    parameters_schema_uri=parameters_schema_uri,
            prediction_schema_uri=prediction_schema_uri,
            description=description,
            serving_container_command=serving_container_command,
            serving_container_args=serving_container_args,
            serving_container_environment_variables=serving_container_environment_variables,
            serving_container_ports=serving_container_ports,
            explanation_metadata=explanation_metadata,
            explanation_parameters=explanation_parameters,
            sync=sync,
        )

        model.wait()

        print(model.display_name)
        print(model.resource_name)
        return model

<br />

For more context, read the
[Model import guide](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/model-registry/import-model.md.txt).

## Send inference requests

To send an online inference request to your `Model`, follow the instructions at
[Get inferences from a custom trained model](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/predictions/get-online-predictions.md.txt):
this process works the same regardless of whether you use a custom container.

Read about [predict request and response requirements for custom
containers](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/predictions/custom-container-requirements#inference).

## What's next

- To learn about everything to consider when you design a custom container to use with Agent Platform, read [Custom container
  requirements](https://docs.cloud.google.com/gemini-enterprise-agent-platform/machine-learning/predictions/custom-container-requirements).