If you're writing your own training code instead of using AutoML},
there are
several ways of doing Vertex AI serverless training to consider. This document provides a
brief overview and comparison of the different ways you can run
serverless training.

## Serverless training resources on Vertex AI

There are three types of Vertex AI resources you can create to
train custom models on Vertex AI:

- [Custom jobs](https://docs.cloud.google.com/vertex-ai/docs/training/create-custom-job)
- [Hyperparameter tuning jobs](https://docs.cloud.google.com/vertex-ai/docs/training/using-hyperparameter-tuning)
- [Training pipelines](https://docs.cloud.google.com/vertex-ai/docs/training/create-training-pipeline)

When you create a *custom job*, you specify settings that Vertex AI
needs to run your training code, including:

- One for single-node training ([`WorkerPoolSpec`](https://docs.cloud.google.com/vertex-ai/docs/reference/rest/v1/CustomJobSpec#workerpoolspec)), or multiple worker pools for distributed training
- Optional settings for configuring job scheduling ([`Scheduling`](https://docs.cloud.google.com/vertex-ai/docs/reference/rest/v1/CustomJobSpec#scheduling)), [setting
  certain environment variables for your training
  code](https://docs.cloud.google.com/vertex-ai/docs/training/code-requirements#environment-variables), [using a custom
  service account](https://docs.cloud.google.com/vertex-ai/docs/general/custom-service-account), and [using VPC Network
  Peering](https://docs.cloud.google.com/vertex-ai/docs/general/vpc-peering)

Within the worker pool(s), you can specify the following settings:

- [Machine types and accelerators](https://docs.cloud.google.com/vertex-ai/docs/training/configure-compute)
- [Configuration of what type of training code the worker pool
  runs](https://docs.cloud.google.com/vertex-ai/docs/training/configure-container-settings): either a Python training application ([`PythonPackageSpec`](https://docs.cloud.google.com/vertex-ai/docs/reference/rest/v1/CustomJobSpec#pythonpackagespec)) or a custom container ([`ContainerSpec`](https://docs.cloud.google.com/vertex-ai/docs/reference/rest/v1/CustomJobSpec#containerspec))

[Hyperparameter tuning jobs](https://docs.cloud.google.com/vertex-ai/docs/training/using-hyperparameter-tuning) have additional settings to configure, such as the
metric. Learn more about
[hyperparameter tuning](https://docs.cloud.google.com/vertex-ai/docs/training/hyperparameter-tuning-overview).

A *training pipeline* orchestrates serverless training
jobs or hyperparameter tuning
jobs with additional steps, such as loading a dataset or uploading the model to
Vertex AI after the training job is successfully completed.

### Serverless training resources

To view existing training pipelines in your project, go to the **Training
Pipelines** page in the **Vertex AI** section of the
Google Cloud console.

[Go to Training pipelines](https://console.cloud.google.com/vertex-ai/training/training-pipelines)

> [!NOTE]
> **Note:** The **Training pipelines** page shows AutoML training pipelines, in addition to serverless training pipelines. You can use the **Model type** column to distinguish between the two.

To view existing custom jobs in your project, go to the **Custom jobs** page.

[Go to Custom jobs](https://console.cloud.google.com/vertex-ai/training/custom-jobs)

To view existing hyperparameter tuning jobs in your project, go to the
**Hyperparameter tuning** page.

[Go to Hyperparameter tuning](https://console.cloud.google.com/vertex-ai/training/hyperparameter-tuning-jobs)

## Prebuilt and custom containers

Before you submit a serverless training job, hyperparameter
tuning job, or a
training pipeline to Vertex AI, you need to create a [Python
training application](https://docs.cloud.google.com/vertex-ai/docs/training/create-python-pre-built-container) or a [custom container](https://docs.cloud.google.com/vertex-ai/docs/training/create-custom-container) to define the training code and
dependencies you want to run on Vertex AI. If you create a Python
training application using TensorFlow, PyTorch, scikit-learn, or XGBoost, you
can use our prebuilt containers to run your code. If you're not sure which of
these options to choose, refer to the
[training code requirements](https://docs.cloud.google.com/vertex-ai/docs/training/code-requirements) to learn more.

## Distributed training

You can configure a serverless training job,
hyperparameter tuning job, or a
training pipeline for distributed training by specifying multiple
worker pools:

- Use your first worker pool to configure your primary replica, and set the replica count to 1.
- Add more worker pools to configure worker replicas, parameter server replicas, or evaluator replicas, if your machine learning framework supports these additional cluster tasks for distributed training.

Learn more about
[using distributed training](https://docs.cloud.google.com/vertex-ai/docs/training/distributed-training).

## What's next

- Learn how to [create a persistent resource](https://docs.cloud.google.com/vertex-ai/docs/training/persistent-resource-create) to run serverless training jobs.
- See [Create serverless training jobs](https://docs.cloud.google.com/vertex-ai/docs/training/create-custom-job) to learn how to create serverless training jobs to run your serverless training applications on Vertex AI.
- See [Create training pipelines](https://docs.cloud.google.com/vertex-ai/docs/training/create-training-pipeline) to learn how to create training pipelines to run serverless training applications on Vertex AI.
- See [Use hyperparameter tuning](https://docs.cloud.google.com/vertex-ai/docs/training/using-hyperparameter-tuning) to learn about Hyperparameter tuning searches.