gcluster job submit

NAME

gcluster job submit - submits a workload to a cluster.

SYNOPSIS

gcluster job submit [FLAGS]

DESCRIPTION

The gcluster job submit command submits a workload to a cluster. On GKE clusters, Cluster Toolkit deploys workloads as Kubernetes JobSets integrated with Kueue.

FLAGS

  • -i, --image IMAGE: The name of the prebuilt container image to run.
  • -B, --base-image IMAGE: The name of the base image for Crane to build upon. This flag requires the --build-context flag.
  • -b, --build-context DIR: The path to the build context directory for Crane. This flag is required when you use the --base-image flag.
  • -e, --command COMMAND: The command to execute in the container (such as python train.py). This flag is required for standard job submissions, but is omitted when submitting a headless server cluster (--pathways-headless).
  • --compute-type TYPE: The type of compute to request, such as n2-standard-32 or nvidia-l4. This flag is required.
  • -o, --dry-run-out PATH: The path to output the generated Kubernetes manifest instead of applying it.
  • -f, --platform PLATFORM: The target platform for the image build. This flag is used with the --base-image flag.
  • --mount VOLUME: The volumes to mount.
  • --env KEY=VALUE: Custom environment variables to pass to the workload container.
  • -n, --name NAME: The name of the workload to create. This flag is required.
  • -q, --queue QUEUE: The name of the Kueue LocalQueue.

  • --num-nodes N: The number of nodes to use per group or slice.

    Note: The `--num-nodes` flag can't be used with TPU jobs because the
  node count is calculated automatically from the `--topology` value.

  • --num-slices N: The number of independent groups or slices to use.

  • --restarts N: The maximum number of restarts for the JobSet.

  • --gke-ttl-after-finished DURATION: The time to retain the JobSet after it finishes.

  • --grace-period DURATION: The time to wait before forcefully terminating a pod.

  • --gke-disable-parallel-containers: Disables parallel containers for TPU7x on GKE.

  • --placement-policy POLICY: The name of the GKE placement policy to use.

  • --node-constraint KEY=VALUE: Key=value pairs for node labels to target specific nodes.

  • --cpu-affinity AFFINITY: CPU affinity rules.

  • --restart-on-exit-codes CODES: A list of exit codes that don't trigger a job failure.

  • --image-pull-secret SECRET: A comma-separated list of secrets for pulling images.

  • --service-account ACCOUNT: The service account name for the pods.

  • --topology TOPOLOGY: The TPU slice topology.

  • --gke-scheduler SCHEDULER: The Kubernetes Scheduler name.

  • --await-job-completion: Waits for the submitted job to complete.

  • --timeout TIMEOUT: The time to wait for the job.

  • --priority PRIORITY: A priority class name.

  • --verbose: Turns on verbose logging for the workload.

  • --gke-nap-provisioning MODEL: The compute provisioning model for GKE node auto-provisioning, such as on-demand, spot, or reservation.

  • --gke-nap-reservation RESERVATION: The name of the Google Cloud reservation for GKE node auto-provisioning. This field is required when you set the value --gke-nap-provisioning=reservation.

  • --pathways: Generates a manifest for a Pathways job.

  • --pathways-proxy-server-image IMAGE: The image for the Pathways proxy server.

  • --pathways-server-image IMAGE: The image for the Pathways server.

  • --pathways-worker-image IMAGE: The image for the Pathways worker.

  • --pathways-headless: Prevents deploying the user's workload container within the pathways-head job.

  • --pathways-gcs-location LOCATION: The Cloud Storage location to store Pathways artifacts. This flag is required when --pathways is set.

  • --pathways-elastic-slices N: Configures the number of elastic slices.

  • --pathways-max-slice-restarts N: The maximum times the workers in a slice can be restarted.

  • --pathways-proxy-args ARGS: Additional arguments for pathways-proxy.

  • --pathways-server-args ARGS: Additional arguments for pathways-rm.

  • --pathways-worker-args ARGS: Additional arguments for pathways-worker.

  • --pathways-proxy-env KEY=VALUE: Environment variables for the Pathways proxy.

  • --pathways-server-env KEY=VALUE: Environment variables for the Pathways server.

  • --pathways-worker-env KEY=VALUE: Environment variables for the Pathways worker.

  • --pathways-colocated-python-sidecar-image IMAGE: The image for the Python-based sidecar.

  • --pathways-head-np NP: The node pool to use for the Pathways head job.

  • --gke-mtc-enabled: Turns on Multi-Tier Checkpointing (MTC).

  • --gke-mtc-ramdisk-dir DIR: The ramdisk directory path for MTC. This flag is required when the --gke-mtc-enabled flag is set.

  • --gke-custom-templates-path PATH: The path to the local directory containing custom GKE templates.

PERSISTENT FLAGS

These flags are available to all gcluster job commands:

  • -c, --cluster CLUSTER_NAME: The name of the GKE cluster.
  • -l, --location LOCATION: The location, either a region or a zone, of the GKE cluster.
  • -p, --project PROJECT_ID: The Google Cloud project ID.
  • --gke-namespace NAMESPACE: The target GKE namespace for the operation.
  • --skip-prereqs: Skips local environment prerequisite checks, such as gcloud, kubectl, and docker authentication.

EXAMPLES

To submit a workload named train-job to a cluster:

gcluster job submit \
    --cluster=my-cluster \
    --location=us-central1-a \
    --name=train-job \
    --image=gcr.io/my-project/training:v1 \
    --command="python3 train.py"

To submit a multi-node workload:

gcluster job submit \
    --cluster=my-cluster \
    --location=us-central1-a \
    --name=multi-node-job \
    --image=gcr.io/my-project/training:v1 \
    --command="python3 train.py" \
    --compute-type=n2-standard-32 \
    --num-nodes=4