This document describes how to create a Managed Service for Apache Spark zero-scale cluster.

Managed Service for Apache Spark zero-scale clusters provide a cost-effective way to use
Managed Service for Apache Spark clusters. Unlike
[standard Managed Service for Apache Spark clusters](https://docs.cloud.google.com/managed-spark/docs/guides/create-cluster.md.txt)
that require at least two primary workers, Managed Service for Apache Spark zero-scale clusters
use only [secondary workers](https://docs.cloud.google.com/managed-spark/docs/concepts/compute/secondary-vms.md.txt)
that can be scaled down to zero.

Managed Service for Apache Spark zero-scale clusters are ideal for use as long-running clusters
that experience idle periods, such as a cluster that hosts a Jupiter notebook.
They provide improved resource utilization through the use of zero-scale
autoscaling policies.

## Characteristics and limitations

A Managed Service for Apache Spark zero-scale cluster shares similarities with a standard
cluster, but has the following unique characteristics and limitations:

- Requires image version `2.2.53` or later.
- Supports only secondary workers, not primary workers.
- Includes services such as YARN, but doesn't support the HDFS file system.

  - To use Cloud Storage as the default file system, set the `core:fs.defaultFS` cluster property to a Cloud Storage bucket location (`gs://BUCKET_NAME`).
  - If you disable a component during cluster creation, also disable HDFS.
- Can't be converted to or from a standard cluster.

- Requires an autoscaling policy for `ZERO_SCALE` cluster types.

- Requires selecting
  [flexible VMs](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/flexible-vms.md.txt#how_to_request_flexible_vms)
  as machine type.

- Doesn't support the Oozie component.

- Can't be created from the Google Cloud console.

## Optional: Configure an autoscaling policy

You can configure an autoscaling policy to define secondary working scaling for
a zero-scale cluster. When doing so, note the following:

- Set the cluster type to `ZERO_SCALE`.
- Configure an autoscaling policy to the secondary worker config only.

For more information, see
[Create an autoscaling policy](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/autoscaling.md.txt#create_an_autoscaling_policy).

## Create a Managed Service for Apache Spark zero-scale cluster

Create a zero-scale cluster using the gcloud CLI or
the Dataproc API.

> [!NOTE]
> **Note:** When selecting a machine type for zero-scale clusters, use [flexible VMs](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/flexible-vms.md.txt#how_to_request_flexible_vms).

### gcloud

Run
[`gcloud dataproc clusters create`](https://docs.cloud.google.com/sdk/gcloud/reference/dataproc/clusters/create.md.txt)
command locally in a terminal window or in
[Cloud Shell](https://console.cloud.google.com/?cloudshell=true%22).

```bash
gcloud dataproc clusters create CLUSTER_NAME \
    --region=REGION \
    --cluster-type=zero-scale \
    --autoscaling-policy=AUTOSCALING_POLICY \
    --properties=core:fs.defaultFS=gs://BUCKET_NAME \
    --secondary-worker-machine-types="type=MACHINE_TYPE1[,type=MACHINE_TYPE2...][,rank=RANK]"
    ...other args
```

Replace the following:

- <var translate="no">CLUSTER_NAME</var>: name of the Managed Service for Apache Spark zero-scale cluster.
- <var translate="no">REGION</var>: an [available Compute Engine region](https://docs.cloud.google.com/compute/docs/regions-zones.md.txt#available).
- <var translate="no">AUTOSCALING_POLICY</var>: the ID or resource URI of the autoscaling policy.
- <var translate="no">BUCKET_NAME</var>: name of your Cloud Storage bucket.
- <var translate="no">MACHINE_TYPE</var>: specific Compute Engine machine type, such as `n1-standard-4`, `e2-standard-8`.
- <var translate="no">RANK</var>: defines the priority of a list of machine types.

### REST

Create a zero-scale cluster using a Managed Service for Apache Spark REST API
[cluster.create](https://docs.cloud.google.com/managed-spark/docs/reference/rest/v1/projects.regions.clusters/create.md.txt)
request:

- Set [`ClusterConfig.ClusterType`](https://docs.cloud.google.com/managed-spark/docs/reference/rest/v1/ClusterConfig.md.txt#ClusterType.ENUM_VALUES.ZERO_SCALE) for the `secondaryWorkerConfig` to `ZERO_SCALE`.
- Set the [`AutoscalingConfig.policyUri`](https://docs.cloud.google.com/managed-spark/docs/reference/rest/v1/ClusterConfig.md.txt#AutoscalingConfig.FIELDS.policy_uri) with the `ZERO_SCALE` autoscaling policy ID.
- Add the `core:fs.defaultFS:gs://BUCKET_NAME` [SoftwareConfig.property](https://docs.cloud.google.com/static/managed-spark/docs/reference/rest/v1/ClusterConfig#SoftwareConfig.FIELDS.properties). Replace <var translate="no">BUCKET_NAME</var> with the name of your Cloud Storage bucket.

## What's next

- Learn more about [Managed Service for Apache Spark autoscaling](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/autoscaling.md.txt).