This document shows you how to enable the Managed Service for Apache Spark
Spark performance enhancements to help your Managed Service for Apache Spark jobs
process more data in less time with reduced costs.

Managed Service for Apache Spark performance enhancements include:

- Spark Optimizer enhancements:
  - Optimizer rules written for better Spark plans
  - Improved performance of the Managed Service for Apache Spark BigQuery connector when used in Spark jobs
- Spark Execution enhancements:
  - Spark execution engine improvements

**Other Managed Service for Apache Spark performance improvements:** See Managed Service for Apache Spark
[cluster caching](https://docs.cloud.google.com/managed-spark/docs/concepts/cluster-caching.md.txt),
which helps reduce the amount of time spent accessing data in Cloud Storage.

You can enable Spark performance enhancements on a cluster or on a Spark job:

- Spark performance enhancements [enabled on a cluster](https://docs.cloud.google.com/managed-spark/docs/guides/performance-enhancements.md.txt#enable_enhancements_at_cluster_creation)
  apply, by default, to all Spark jobs run on the cluster, whether submitted to the
  Managed Service for Apache Spark or
  [submitted directly to the cluster](https://docs.cloud.google.com/managed-spark/docs/guides/submit-job.md.txt#submit_a_job_directly_on_your_cluster).

- Spark performance enhancements can also be
  [enabled or disabled on a job](https://docs.cloud.google.com/managed-spark/docs/guides/performance-enhancements.md.txt#enable_or_disable_enhancements_at_job_submission)
  that is submitted to Managed Service for Apache Spark. Spark performance
  enhancements settings applied to a job override any conflicting settings set
  at the cluster level for the specified job only.

## Pricing

Spark performance enhancements don't incur additional charges. Standard
[Managed Service for Apache Spark pricing](https://cloud.google.com/managed-spark/pricing#on_pricing) applies.

## Considerations

Spark performance enhancements adjusts Spark properties, including the following
properties:

- [`spark.sql.shuffle.partitions`](https://docs.cloud.google.com/managed-spark/docs/support/spark-job-tuning.md.txt#configuring_partitions): Spark performance enhancements set this property to `1000` for `2.2` image version clusters. This setting can slow small jobs.
- `spark.dataproc.sql.catalog.file.index.stats.enabled`: This setting can result in driver OOM (Out-Of-Memory) conditions if the Hive partition count is high. Disabling this property can fix the OOM condition.

## Enable enhancements at cluster creation

You can use the Google Cloud console, Google Cloud CLI, and the Dataproc
API to enable Managed Service for Apache Spark performance enhancements
when you create a Managed Service for Apache Spark cluster with image versions
2.0.69+, 2.1.17+, 2.2.0+, and later image releases.

### Console

1. In the Google Cloud console, open the [**Create cluster**](https://console.cloud.google.com/dataproc/clusters/create) page.
2. Click **Additional configuration** to expand that section.
3. Edit **Customization \& Other**.
4. In the **Cluster properties** section, add the following properties:
   - To enable Spark optimization enhancements:
     1. Click **+ Add properties**.
     2. Select **spark** in the **Prefix** list.
     3. Enter `spark.dataproc.enhanced.optimizer.enabled` in the **Key** field and `true` in the **Value** field.
   - To enable Spark execution enhancements:
     1. Click **+ Add properties**.
     2. Select **spark** in the **Prefix** list.
     3. Enter `spark.dataproc.enhanced.execution.enabled` in the **Key** field and `true` in the **Value** field.
5. Complete other cluster fields, then click **Create cluster**.

### gcloud

1. Run the following
   [gcloud dataproc clusters create](https://docs.cloud.google.com/sdk/gcloud/reference/dataproc/clusters/create.md.txt)
   command locally in a terminal window or in
   [Cloud Shell](https://console.cloud.google.com/?cloudshell=true%22).

   ```
   gcloud dataproc clusters create CLUSTER_NAME \
       --project=PROJECT_ID \
       --region=REGION \
       --image-version=IMAGE \
       --properties=PROPERTIES
   ```

   Notes:
   - <var translate="no">CLUSTER_NAME</var>: The cluster name, which must be unique within a project. The name must start with a lowercase letter, and can contain up to 51 lowercase letters, numbers, and hyphens. It cannot end with a hyphen. The name of a deleted cluster can be reused.
   - <var translate="no">PROJECT_ID</var>: The project to associate with the cluster.
   - <var translate="no">REGION</var>: The [Compute Engine region](https://docs.cloud.google.com/compute/docs/regions-zones.md.txt#available) where the cluster will be located, such as `us-central1`.
     - You can add the optional `--zone=ZONE` flag to specify a zone within the specified region, such as `us-central1-a`. If you do not specify a zone, the Managed Service for Apache Spark [autozone placement](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/auto-zone.md.txt) feature selects a zone with the specified region.
   - <var translate="no">IMAGE</var>: The Managed Service for Apache Spark optimizer and execution performance enhancements are available in Managed Service for Apache Spark image versions `2.0.69+` and `2.1.17+` and later releases. If you omit this flag, Managed Service for Apache Spark will select the latest subminor version of the default Managed Service for Apache Spark image for the cluster (see [Default Managed Service for Apache Spark image version](https://docs.cloud.google.com/managed-spark/docs/concepts/versioning/dataproc-version-clusters#default_dataproc_image_version)).
   - <var translate="no">PROPERTIES</var>:

     - To enable Spark optimization enhancements, specify:

     ```bash
     spark:spark.dataproc.enhanced.optimizer.enabled=true
     ```
     - To enable Spark execution enhancements, specify:

     ```bash
     spark:spark.dataproc.enhanced.execution.enabled=true
     ```
     - To enable Spark optimization and execution enhancements, specify:

     ```bash
     spark:spark.dataproc.enhanced.optimizer.enabled=true,spark:spark.dataproc.enhanced.execution.enabled=true
     ```

### API

1. Specify the following
   [`SoftwareConfig.properties`](https://docs.cloud.google.com/static/managed-spark/docs/reference/rest/v1/ClusterConfig#SoftwareConfig.FIELDS.properties)
   as part of a [`clusters.create`](https://docs.cloud.google.com/managed-spark/docs/reference/rest/v1/projects.regions.clusters/create.md.txt) request:

   - To enable Spark optimization enhancements, specify:

   ```bash
   "spark:spark.dataproc.enhanced.optimizer.enabled": "true"
   ```
   - To enable Spark execution enhancements, specify:

   ```bash
   "spark:spark.dataproc.enhanced.execution.enabled": "true"
   ```
   - To enable Spark optimization and execution enhancements, specify:

   ```bash
   "spark:spark.dataproc.enhanced.optimizer.enabled": "true","spark:spark.dataproc.enhanced.execution.enabled": "true"
   ```

<br />

## Enable or disable enhancements at job submission

You can use the Google Cloud console, Google Cloud CLI, and the Dataproc
API to enable or disable Spark performance enhancements on a Spark job submitted
to Managed Service for Apache Spark.

### Console

1. In the Google Cloud console, open the [Jobs](https://console.cloud.google.com/dataproc/jobs) page.
2. On the **Jobs** page, click **Submit job** , then scroll to the job **Properties** section.
   1. To enable Spark optimization enhancements:
      1. Click **+ Add properties** . Add "spark.dataproc.enhanced.optimizer.enabled" in the **Key** field and "true" in **Value** field.
   2. To enable Spark execution enhancements:
      1. Click **+ Add properties**.
      2. Add "spark.dataproc.enhanced.execution.enabled" in the **Key** field and "true" in **Value** field.
3. Complete filling in or confirming the other job submission fields, then click **Submit**.

### gcloud

1. Run the following
   [gcloud dataproc jobs submit](https://docs.cloud.google.com/sdk/gcloud/reference/dataproc/jobs/submit)
   command locally in a terminal window or in
   [Cloud Shell](https://console.cloud.google.com/?cloudshell=true%22).

   ```
   gcloud dataproc jobs submit SPARK_JOB_TYPE \
       --cluster=CLUSTER_NAME \
       --region=REGION \
       --properties=PROPERTIES
   ```

   Notes:
   - <var translate="no">SPARK_JOB_TYPE</var>: Specify `spark`, `pyspark`, `spark-sql` or `spark-r` .
   - <var translate="no">CLUSTER_NAME</var>: The name of the job where the job will run.
   - <var translate="no">REGION</var>: The region where the cluster is located.
   - <var translate="no">PROPERTIES</var>:

     - To enable Spark optimization enhancements, specify:

     ```bash
     spark.dataproc.enhanced.optimizer.enabled=true
     ```
     - To enable Spark execution enhancements, specify:

     ```bash
     spark.dataproc.enhanced.execution.enabled=true
     ```
     - To enable Spark optimization and execution enhancements, specify:

     ```bash
     spark.dataproc.enhanced.optimizer.enabled=true,spark.dataproc.enhanced.execution.enabled=true
     ```

### API

1. Specify the following `properties` for a
   [SparkJob](https://docs.cloud.google.com/managed-spark/docs/reference/rest/v1/SparkJob.md.txt),
   [PySparkJob](https://docs.cloud.google.com/managed-spark/docs/reference/rest/v1/PySparkJob.md.txt),
   [SparkSqlJob](https://docs.cloud.google.com/managed-spark/docs/reference/rest/v1/SparkSqlJob.md.txt), or
   [SparkRJob](https://docs.cloud.google.com/managed-spark/docs/reference/rest/v1/SparkRJob.md.txt)
   as part of a
   [`jobs.submit`](https://docs.cloud.google.com/managed-spark/docs/reference/rest/v1/projects.regions.jobs/submit.md.txt)
   request:

   - To enable Spark optimization enhancements, specify:

   ```bash
   "spark.dataproc.enhanced.optimizer.enabled=true"
   ```
   - To enable Spark execution enhancements, specify:

   ```bash
   "spark.dataproc.enhanced.execution.enabled=true"
   ```
   - To enable Spark optimization and execution enhancements, specify:

   ```bash
   "spark.dataproc.enhanced.execution.enabled=true,spark.dataproc.enhanced.optimizer.enabled=true"
   ```

<br />

> [!IMPORTANT]
> **Note:** At the bottom of the [Create cluster](https://console.cloud.google.com/dataproc/clusters/create) page, click **Equivalent command line** or **Equivalent REST** to have the Google Cloud console construct an equivalent gcloud CLI command or API REST request.