Manage cluster

If you're interested in Gemini Enterprise Agent Platform training clusters, contact your sales representative for access.

Once a cluster is deployed, you can change or remove it using the following Agent Platform API endpoints:

• Update: Modifies an existing cluster configuration.
• Delete: Permanently removes a cluster and its resources.

Updating a cluster is available only through the Agent Platform API. You can delete a cluster from either the Agent Platform API or the Google Cloud console.

For instructions on listing all clusters or viewing individual clusters, see View clusters.

Authentication

alias gcurl='curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json"'

Update a cluster

UPDATE_PAYLOAD specifies the local path to a JSON file that defines the full ModelDevelopmentCluster you want to update to.

For example, to update the node count of a pool of a CPU-only cluster, use the following JSON payload:

{
  "display_name": "DISPLAY_NAME",
  "network": {
    "network": "projects/PROJECT_ID/global/networks/NETWORK",
    "subnetwork": "projects/PROJECT_ID/regions/REGION/subnetworks/SUBNETWORK"
  },
  "node_pools": [
    {
      "id": "cpu",
      "machine_spec": {
        "machine_type": "n2-standard-8"
      },
      "scaling_spec": {
        "min_node_count": UPDATED_MIN_NODE_COUNT,
        "max_node_count": UPDATED_MAX_NODE_COUNT
      },
      "zone": "ZONE",
      "enable_public_ips": true,
      "boot_disk": {
        "boot_disk_type": "pd-standard",
        "boot_disk_size_gb": 120
      }
    },
    {
      "id": "login",
      "machine_spec": {
        "machine_type": "n2-standard-8",
      },
      "scaling_spec": {
        "min_node_count": 1,
        "max_node_count": 1
      },
         "zone": "ZONE",
         "enable_public_ips": true,
      "boot_disk": {
        "boot_disk_type": "pd-standard",
        "boot_disk_size_gb": 120
      }
    },
  ],
  "orchestrator_spec": {
      "slurm_spec": {
      "home_directory_storage": "projects/PROJECT_ID/locations/ZONE/instances/FILESTORE",
      "partitions": [
         {
          "id": "cpu",
          "node_pool_ids": [
            "cpu"
          ]
        }
      ],
      "login_node_pool_id": "login"
    }
  }
}

gcurl -X PATCH -d @UPDATE_PAYLOAD https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/modelDevelopmentClusters/CLUSTER_ID

Query Parameters

• updateMask (string, optional): A FieldMask that specifies which fields of the Model Development cluster resource to update. Only the fields listed in the updateMask are changed.

  The following fields within the ModelDevelopmentCluster resource can be specified in the updateMask:

  • labels
  • node_pools
  • orchestrator_spec.slurm_spec.partitions
  • orchestrator_spec.slurm_spec.login_node_pool_id
  • orchestrator_spec.slurm_spec.prolog_bash_scripts
  • orchestrator_spec.slurm_spec.epilog_bash_scripts
  • orchestrator_spec.slurm_spec.scheduling
  • orchestrator_spec.slurm_spec.accounting

• updateMode (enum, optional): Specifies the update mode. Possible values are:

  • UPDATE_MODE_UNSPECIFIED: The default value, treated as USER_AND_SERVICE.
  • USER_ONLY: Apply only user-specified field changes from the request. The service won't refresh service-managed fields, like startup, prolog, or epilog scripts. Each node pool also stays on the image it is currently running unless you set node_image yourself, so a USER_ONLY update that omits node_image won't recreate your nodes.
  • USER_AND_SERVICE: Apply user-specified field changes and let the service refresh service-managed fields.
  We recommend using USER_AND_SERVICE to ensure your cluster is up-to-date.

The command below updates both the node pool configuration and the Slurm partitions.

gcurl -X PATCH -d @update-payload.json \
'https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/modelDevelopmentClusters/CLUSTER_ID?updateMask=orchestrator_spec.slurm_spec.partitions,node_pools&updateMode=USER_AND_SERVICE'

Important note on repeated fields

For repeated fields, such as node_pools, prolog_bash_scripts, and epilog_bash_scripts, the API only supports a full replacement operation. The user must provide the entire, expected list of items in the request payload to replace the existing list completely.

Update Slurm scheduling and accounting settings

You can update Slurm scheduling settings, including preemption and accounting setting, on a running cluster. Applying these settings doesn't restart or drain any nodes. The service regenerates slurm.conf and reloads the Slurm controller so queued and running jobs are unaffected.

There is always a risk that nodes may reboot after running jobs complete due to underlying dependency changes. However, that risk is minimized with the USER_ONLY update mode.

The new settings might take a short time to appear in scontrol show config after the operation completes.

The command below turns on partition-priority preemption for an existing cluster.

gcurl -X PATCH -d @update-payload.json \
'https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/modelDevelopmentClusters/CLUSTER_ID?updateMask=orchestrator_spec.slurm_spec.scheduling&updateMode=USER_ONLY'

Where update-payload.json contains, in part:

{
  ...,
  "orchestrator_spec": {
    "slurm_spec": {
      ...,
      "scheduling": {
        "preempt_type": "preempt/partition_prio",
        "preempt_mode": "REQUEUE",
        "preempt_exempt_time": "1:01"
      }
    }
  }
}

Caution: scheduling and accounting are each replaced in full, in the same way as the repeated fields described earlier. Naming one of them in the updateMask replaces the whole object, so any field you leave out reverts to its default rather than keeping its current value. Always send the complete set of settings you want. To turn a setting off, send the object with that field omitted; to clear the whole object, name it in the updateMask and omit it from the payload.

The same caution applies if you call PATCH without an updateMask. In that case the service diffs your entire payload against the current cluster, so a payload that omits scheduling clears any scheduling settings the cluster already has. Pass an updateMask to scope the update to the fields you intend to change.

Node image selection

Because node_pools is replaced in full, a node pool in your update payload that omits node_image resolves to the current default image for its machine type. That default advances over time, so omitting node_image moves the node pool to the latest available image, which is often what you want: it picks up driver, security, and bug fixes.

Caution: Moving a node pool to a different image recreates its nodes. Compute nodes in the pool are drained and replaced, and the login nodes are recreated, which ends active SSH sessions and discards data stored on the login node's boot disk. Because the image is resolved on every USER_AND_SERVICE update, an unrelated change such as editing a label is enough to trigger this once a newer default image is available. To apply your changes without moving any node pool off its current image, use updateMode=USER_ONLY.

To keep a node pool on a specific image across updates, set node_image to that image. If you read the current value from Get in order to send it back, note that a cluster that has never been updated reports an empty node_image.

Post-update behavior and cluster status

A successful request returns a Long Running Operation (LRO). You can then monitor the status of this operation using the following command:

gcurl https://REGION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/operations/OPERATION_ID

Once the update operation completes successfully, most modifications take effect immediately (or within the 15s Slurm sync time) and require no further action. Only changes to the underlying cluster compute infrastructure—including modifications to storage sources and startup scripts—require the resources to be recreated.

Delete a cluster

  gcurl -X DELETE https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/modelDevelopmentClusters/CLUSTER_ID
  

This command returns a Long-Running Operation on success, which you can then monitor using the operations describe command.

gcurl https://REGION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/REGION/operations/OPERATION_ID

What's next

Managing your Gemini Enterprise Agent Platform training cluster lets you optimize its usage, integrate it into automated workflows, and prepare your trained models for deployment.

• Orchestrate your training with Gemini Enterprise Agent Platform Pipelines: Automate the entire lifecycle of your training jobs, from data preparation to model registration, using pipelines that target your managed cluster.
  • Learn about orchestrating jobs on a training cluster
• Monitor and debug your training jobs: Track progress and resource usage, and identify issues with your distributed training jobs running on the cluster.
  • Monitor training jobs on Gemini Enterprise Agent Platform
• Deploy your model for inference: After your models are trained and registered, deploy them to a Gemini Enterprise Agent Platform endpoint to serve online inference requests at scale.
  • Deploy a model to an endpoint
• Optimize costs: Regularly review your cluster utilization and delete the cluster when it's not actively in use to minimize billing for reserved hardware.