<br />

This document explains common cluster creation error messages and provides
tips on troubleshooting cluster creation issues.

## Common cluster creation error messages

- **User not authorized to act as service account**

  Cause: The principal attempting to create the Managed Service for Apache Spark cluster does not have the necessary permissions to use the specified service account. Managed Service for Apache Spark users are required to have service account `ActAs` permission to deploy Managed Service for Apache Spark resources;
  this permission is included in the Service Account User role (`roles/iam.serviceAccountUser`)
  (see [Managed Service for Apache Spark roles](https://docs.cloud.google.com/managed-spark/docs/concepts/iam/iam.md.txt#dataproc-roles)).

  Solution: Identify the [user or service account](https://docs.cloud.google.com/managed-spark/docs/concepts/iam/principals.md.txt) trying to create the Managed Service for Apache Spark cluster. Grant that principal the Service Account User role (`roles/iam.serviceAccountUser`) on the service account the cluster is configured to use (typically, the [Managed Service for Apache Spark VM service account](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/service-accounts.md.txt#VM_service_account)).
- **Operation timed out:** Only 0 out of 2 minimum required datanodes/node managers running.

  Cause: The controller node is unable to create the cluster because
  it cannot communicate with worker nodes.

  Solution:
  - **Check firewall rule warnings.**
  - Make sure the correct firewall rules are in place. For more information, see [Overview of the default Managed Service for Apache Spark firewall rules](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/network.md.txt#overview_of_the_default_dataproc_firewall_rules).
  - **Perform a [connectivity test](https://console.cloud.google.com/net-intelligence/connectivity/tests/list)** in the Google Cloud console to determine what is blocking communication between the controller and worker nodes.
- **Required `compute.subnetworks.use` permission for `projects/{projectId}/regions/{region}/subnetworks/{subnetwork}`**

  Cause: This error can occur when you attempt to setup a Managed Service for Apache Spark
  cluster using a VPC network in another project and the Managed Service for Apache Spark
  [Service Agent](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/service-accounts.md.txt#service_agent_account)
  service account does not have the necessary permissions on the Shared VPC
  project that is hosting the network.

  Solution: Follow the steps listed in
  [Create a cluster that uses a VPC network in another project](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/network.md.txt#create_a_cluster_that_uses_a_vpc_network_in_another_project).
- **The zone `projects/zones/{zone}` does not have enough resources available to fulfill the request `(resource type:compute)`**

  Cause: The zone being used to create the cluster does not have sufficient resources.

  Solution:
  - **Use the Managed Service for Apache Spark
    [Auto Zone placement](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/auto-zone.md.txt)
    feature** to create the cluster in any of a region's zones with available resources.
  - **Create the cluster in a different zone.**
- **Quota Exceeded errors**

  Insufficient CPUS/CPUS_ALL_REGIONS quota  

  Insufficient 'DISKS_TOTAL_GB' quota  

  Insufficient 'IN_USE_ADDRESSES' quota

  Cause: Your [CPU](https://docs.cloud.google.com/compute/quotas#cpu_quota), [disk](https://docs.cloud.google.com/compute/quotas#disk_quota),
  or [IP address](https://docs.cloud.google.com/compute/quotas#external_ip_addresses) request exceeds your
  available quota.

  Solution: Request additional quota from the
  [Google Cloud console](https://console.cloud.google.com/iam-admin/quotas).
- **Initialization action failed**

  Cause: The initialization action provided during cluster creation failed
  to install.

  Solution:
  - **See [initialization actions considerations and guidelines](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/init-actions.md.txt#important_considerations_and_guidelines).**
  - **Examine the output logs.** The error message should provide a link to the logs in Cloud Storage.
- **Failed to initialize node `CLUSTER-NAME-m`. ... See output in: `<gs://PATH_TO_STARTUP_SCRIPT_OUTPUT>`**

  Cause: Managed Service for Apache Spark cluster controller node failed to be initialized.

  Solution:
  - Review the startup script output logs listed in the error message (`gs://PATH_TO_STARTUP_SCRIPT_OUTPUT`) and verify the cause of the failed node initialization.
  - Causes can include [Managed Service for Apache Spark cluster network configuration](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/network.md.txt) issues and [failed installation of Python package dependencies](https://docs.cloud.google.com/managed-spark/docs/tutorials/python-configuration.md.txt).
  - If the issue is not resolved after you review the startup-script logs, fix any user side issues, then retry with exponential backoff, contact [Google Cloud support](https://docs.cloud.google.com/managed-spark/docs/support/getting-support.md.txt#get_a_google_support_package).
- **Cluster creation failed: IP address space exhausted**

  Cause: IP address space needed to provision the requested cluster nodes is
  unavailable.

  Solution:
  - Create a cluster with fewer worker nodes, but a larger machine type.
  - Create a cluster on a different subnetwork or network.
  - Reduce usage on the network to free IP address space.
  - Wait until sufficient IP space becomes available on the network.
- **Initialization script error message: The repository <var translate="no">REPO_NAME</var> no longer has a Release file**

  Cause: The Debian oldstable backports repository was purged.

  Solution:

  Add the following code before the code that runs `apt-get` in your initialization script.

      oldstable=$(curl -s https://deb.debian.org/debian/dists/oldstable/Release | awk '/^Codename/ {print $2}');
      stable=$(curl -s https://deb.debian.org/debian/dists/stable/Release | awk '/^Codename/ {print $2}');

      matched_files="$(grep -rsil '\-backports' /etc/apt/sources.list*)"
      if [[ -n "$matched_files" ]]; then
        for filename in "$matched_files"; do
          grep -e "$oldstable-backports" -e "$stable-backports" "$filename" || \
            sed -i -e 's/^.*-backports.*$//' "$filename"
        done
      fi

- **Timeout waiting for instance `DATAPROC_CLUSTER_VM_NAME` to report in** or
  **Network is unreachable: `dataproccontrol-REGION.googleapis.com`**

  Cause: These error messages indicate that the networking setup of your
  Managed Service for Apache Spark cluster is incomplete: you may be missing the [route to
  the default internet gateway](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/network#route_requirements)
  or [firewall rules](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/network.md.txt#firewall_requirements).

  Solution:

  To troubleshoot this issue, you can create the following [Connectivity Tests](https://docs.cloud.google.com/network-intelligence-center/docs/connectivity-tests/concepts/overview.md.txt):
  - [Create a Connectivity Test](https://docs.cloud.google.com/network-intelligence-center/docs/connectivity-tests/how-to/running-connectivity-tests.md.txt#testing-between-vms) between two Managed Service for Apache Spark cluster VMs. The outcome of this test will help you understand whether the ingress or egress allow firewall rules of your network apply to the cluster VMs correctly.
  - [Create a Connectivity Test](https://docs.cloud.google.com/network-intelligence-center/docs/connectivity-tests/how-to/running-connectivity-tests.md.txt#vpc-to-non-cloud) between a Managed Service for Apache Spark cluster VM and a current Managed Service for Apache Spark control API IP address. To get a current Managed Service for Apache Spark control API IP address, use the following command:

  ```
  dig dataproccontrol-REGION.googleapis.com A
  ```

  Use any of the IPv4 addresses in the answer section of the output.

  The outcome of the Connectivity Test will help you understand whether the route
  to the default internet gateway and the egress allow firewall are properly
  configured.

  Based on the outcomes of the Connectivity Tests:
  - Add a route to the internet to your cluster VPC network: [`0.0.0.0/0` for IPv4](https://docs.cloud.google.com/vpc/docs/using-routes.md.txt#add-default-route-ipv4) and [`::/0` for IPv6](https://docs.cloud.google.com/vpc/docs/using-routes.md.txt#addingroute-ipv6) with `--next-hop-gateway=default-internet-gateway`.
  - Add [firewall rules for access control](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/network.md.txt#create_an_ingress_firewall_rule).
- **Error due to update**

  Cause: The cluster accepted a job submitted to the Managed Service for Apache Spark
  service, but was unable to scale up or down manually or through autoscaling.
  This error can also be caused by a non-standard cluster configuration.

  Solution:
  - **Cluster reset:** Open a support ticket, include a
    [diagnostic tar file](https://docs.cloud.google.com/managed-spark/docs/support/diagnose-clusters.md.txt),
    and ask for the cluster to be reset to a RUNNING state.

  - **New cluster:** [Recreate the cluster](https://docs.cloud.google.com/managed-spark/docs/guides/recreate-cluster.md.txt)
    with the same configuration. This solution can be faster than a
    support-provided reset.

## Cluster troubleshooting tips

This section provides additional guidance on troubleshooting common issues that
can prevent the creation of Managed Service for Apache Spark clusters.

When a Managed Service for Apache Spark cluster fails to provision, it often produces
a generic error message or reports a `PENDING` or `PROVISIONING` status
before failing. The key to diagnosing and solving cluster failure issues
is to examine cluster logs and assess common failure points.

### Common symptoms

The following are common symptoms associated with cluster creation failures:

- Cluster status remains `PENDING` or `PROVISIONING` for an extended period.
- Cluster transitions to `ERROR` state.
- Generic API errors during cluster creation, such as `Operation timed out`.
- Logged or API response error messages, such as:

  - `RESOURCE_EXHAUSTED`: related to CPU, disk, or IP address quotas
  - `Instance failed to start`
  - `Permission denied`
  - `Unable to connect to service_name.googleapis.com` or `Could not reach required Google APIs`
  - `Connection refused` or `network unreachable`
  - Errors related to initialization actions failing, such as script execution errors and file not found.

### Review cluster logs

An important initial step when diagnosing cluster creation failures is reviewing the
detailed cluster logs available in Cloud Logging.

1. Go to Logs Explorer: Open the [Logs Explorer](https://console.cloud.google.com/logs/viewer) in the Google Cloud console.
2. Filter for Managed Service for Apache Spark clusters:
   - In the **Resource** drop-down, select `Cloud Managed Service for Apache Spark Cluster`.
   - Enter your `cluster_name` and `project_id`. You can also filter by `location` (region).
3. Examine Log Entries:
   - Look for `ERROR` or `WARNING` level messages that occur close to the time of the cluster creation failure.
   - Pay attention to logs from `master-startup`, `worker-startup`, and `agent` components for insights into VM-level or Managed Service for Apache Spark agent issues.
   - For insight into VM boot-time issues, filter logs by `resource.type="gce_instance"`, and look for messages from the instance names associated with your cluster nodes, such as `CLUSTER_NAME-m` or `CLUSTER_NAME-w-0`. Serial console logs can reveal network configuration issues, disk problems, and script failures that occur early in the VM lifecycle.

### Common cluster failure causes and troubleshooting tips

This section outlines common reasons why Managed Service for Apache Spark cluster
creation might fail and provides troubleshooting tips to help troubleshoot cluster failures.

#### Insufficient IAM permissions

The [VM service account](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/service-accounts.md.txt#VM_service_account)
that your Managed Service for Apache Spark cluster uses must have appropriate
IAM roles to provision Compute Engine instances, access Cloud Storage
buckets, write logs, and interact with other Google Cloud services.

- Required Worker role: Verify that the VM service account has the [Managed Service for Apache Spark Worker](https://docs.cloud.google.com/managed-spark/docs/concepts/iam/iam.md.txt#dataproc.worker) role (`roles/dataproc.worker`). This role has the minimum permissions required for Managed Service for Apache Spark to manage cluster resources.
- Data access permissions: If your jobs read from or write to Cloud Storage or BigQuery, the service account needs related roles, such as `Storage Object Viewer`, `Storage Object Creator`, or `Storage Object Admin` for Cloud Storage, or `BigQuery Data Viewer` or `BigQuery Editor` for BigQuery.
- Logging permissions: The service account must have a role with permissions needed to write logs to Cloud Logging, such as the `Logging Writer` role.

Troubleshooting tips:

- Identify service account: Determine the
  [VM service account](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/service-accounts.md.txt#VM_service_account)
  that your cluster is configured to use. If not specified, the default is the
  [Compute Engine default service account](https://docs.cloud.google.com/compute/docs/access/service-accounts.md.txt#default_service_account).

- Verify IAM roles: Go to
  the [IAM \& Admin \> IAM](https://console.cloud.google.com/iam-admin/iam)
  page in the Google Cloud console, find the cluster VM service account,
  and then verify that it has the necessary roles needed for cluster operations.
  Grant any missing roles.

#### Resource quotas exceeded

Managed Service for Apache Spark clusters consume resources from Compute Engine and
other Google Cloud services. Exceeding project or regional quotas can cause
cluster creation failures.

- Common [Managed Service for Apache Spark quotas](https://docs.cloud.google.com/managed-spark/quotas.md.txt) to check:
  - `CPUs` (regional)
  - `DISKS_TOTAL_GB` (regional)
  - `IN_USE_ADDRESSES` (regional for internal IPs, global for external IPs)
  - Dataproc API quotas, such as `ClusterOperationRequestsPerMinutePerProjectPerRegion` To compare Managed Service for Apache Spark quotas with Managed Service for Apache Spark quotas, see [Managed Service for Apache Spark quotas](https://docs.cloud.google.com/managed-spark/quotas.md.txt).

Troubleshooting tips:

- Review quotas: Go to the [IAM \& Admin \> IAM](https://console.cloud.google.com/iam-admin/iam) page in the Google Cloud console. Filter by "Service" for "Compute Engine API" and "Dataproc API."
- Check usage vs. limit: Identify any quotas that are at or near their limits.
- If necessary, request a quota increase.

#### Network configuration issues

[Network configuration](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/network.md.txt)
issues, such as incorrect VPC network, subnet, firewall, or DNS configuration,
are a common cause of cluster creation failures. Cluster instances must be able
to communicate with each other and with Google APIs.

- VPC network and subnet:
  - Verify that the cluster VPC network and subnet exist and are configured correctly.
  - Verify that the subnet has a sufficient range of available IP addresses.
- [Private Google Access](https://docs.cloud.google.com/vpc/docs/private-google-access.md.txt) (PGA): If cluster VMs have internal IP addresses and need to reach Google APIs for Cloud Storage, Cloud Logging, and other operations, verify that [Private Google Access](https://docs.cloud.google.com/vpc/docs/private-google-access.md.txt) is enabled on the subnet. By default, Managed Service for Apache Spark clusters created with 2.2+ image versions provision VMs with internal-only IP addresses with Private Google Access enabled on the cluster regional subnet.
- [Private Service Connect](https://docs.cloud.google.com/managed-spark/docs/networking-private-service-connect.md.txt) (PSC): If you are using Private Service Connect to access Google APIs, verify that necessary [Private Service Connect endpoints](https://docs.cloud.google.com/vpc/docs/about-accessing-google-apis-endpoints.md.txt) are correctly configured for the Google APIs that Managed Service for Apache Spark depends on, such as `dataproc.googleapis.com`, `storage.googleapis.com`, `compute.googleapis.com`, and `logging.googleapis.com`. DNS entries for the APIs must resolve to private IP addresses. Note that using Private Service Connect does not eliminate the need to use VPC peering to communicate with other customer-managed VPC networks.

  > [!IMPORTANT]
  > For detailed Private Service Connect network troubleshooting, see [Managed Service for Apache Spark cluster networking with Private Service Connect](https://docs.cloud.google.com/managed-spark/docs/networking-private-service-connect.md.txt).

- VPC Peering: If your cluster communicates with resources in other VPC networks, such as shared VPC host projects or other customer VPCs, verify that VPC peering is correctly configured and routes are propagating.
- Firewall rules:

  - Default rules: Verify that default firewall rules, such as `allow-internal` or `allow-ssh`, are not overly restrictive.
  - Custom rules: If custom firewall rules are in place, verify that they
    allow needed communication paths:

    - Internal communication within the cluster (between *-m* and *-w* nodes).
    - Outbound traffic from cluster VMs to Google APIs, using either public
      IPs or an internet gateway, Private Google Access, or
      Private Service Connect endpoints.

    - Traffic to any external data sources or services that your jobs depend on.

- DNS resolution: Confirm that cluster instances can correctly resolve DNS
  names for Google APIs and any internal or external services.

Troubleshooting tips:

- Review network configuration: Inspect the VPC network and subnet settings where the cluster is being deployed.
- Check firewall rules: Review firewall rules in the VPC network or shared VPC host project.
- Test connectivity: Launch a temporary Compute Engine VM in the cluster subnet and run the following steps:
  - `ping` or `curl` to external Google API domains, such as `storage.googleapis.com`.
  - `nslookup` to verify DNS resolution to expected IP addresses (Private Google Access or Private Service Connect).
  - Run Google Cloud [connectivity tests](https://docs.cloud.google.com/network-intelligence-center/docs/connectivity-tests/concepts/overview.md.txt) to diagnose paths from a test VM to relevant endpoints.

#### Initialization action failures

Managed Service for Apache Spark initialization actions are scripts that run on cluster VMs
during cluster creation. Errors in these scripts can prevent cluster startup.

Troubleshooting tips:

- Examine logs for initialization action errors: Look for log entries related to `init-actions` or `startup-script` for the cluster instances in Cloud Logging.
- Check script paths and permissions: Verify that initialization action scripts are correctly located in Cloud Storage and that the cluster VM service account has the `Storage Object Viewer` role needed to read Cloud Storage scripts.
- Debug script logic: Test script logic on a separate Compute Engine VM that mimics the cluster environment to identify errors. Add verbose logging to the script.

#### Regional resource availability (stockouts)

Occasionally, a machine type or resource in a region or zone
experiences temporary unavailability (stockout). Typically, this results
in `RESOURCE_EXHAUSTED` errors unrelated to project quota issues.

Troubleshooting tips:

- Try a different zone or region: Attempt to create the cluster in a different zone within the same region, or in a different region.
- Use Auto Zone placement: Use the Managed Service for Apache Spark [Auto Zone placement](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/auto-zone.md.txt) feature to automatically select a zone with capacity.
- Adjust machine type: If using a custom or specialized machine type, try a standard machine type to see if that resolves the issue.

### Contact Cloud Customer Care

If you continue to experience cluster failure issues, contact
[Cloud Customer Care](https://docs.cloud.google.com/support). Describe the cluster failure issue and
troubleshooting steps taken. Additionally, provide the following information:

- [Cluster diagnostic data](https://docs.cloud.google.com/managed-spark/docs/support/diagnose-clusters.md.txt#share_the_archive_file)
- Output from the following command:

  ```
    gcloud dataproc clusters describe CLUSTER_NAME \
        --region=REGION
    
  ```
- Exported logs for the failed cluster.

## Use the `gcpdiag` tool

[`gcpdiag`](https://gcpdiag.dev)
is an open source tool. It is not an officially supported Google Cloud product.
You can use the `gcpdiag` tool to help you identify and fix Google Cloud
project issues. For more information, see the
[gcpdiag project on GitHub](https://github.com/GoogleCloudPlatform/gcpdiag/#gcpdiag---diagnostics-for-google-cloud-platform).
The `gcpdiag` tool helps you discover the following Managed Service for Apache Spark
cluster creation issues by performing the following checks:

- **Stockout errors:** Evaluates Logs Explorer logs to discover stockouts in regions and zones.
- **Insufficient quota:** Checks quota availability in the Managed Service for Apache Spark cluster project.
- **Incomplete network configuration:** Performs network connectivity tests, including checks for necessary firewall rules and external and internal IP configuration. **If the cluster has been deleted, the `gcpdiag` tool cannot
  perform a network connectivity check.**
- **Incorrect cross-project configuration:** Checks for cross-project service accounts and reviews additional roles and organization policies enforcement.
- **Missing shared Virtual Private Cloud network IAM roles:** If the Managed Service for Apache Spark cluster uses a Shared VPC network, checks for the addition of required service account roles.
- **Initialization action failures**: Evaluates Logs Explorer logs to discover initialization action script failures and timeouts.

For a list of `gcpdiag` dataproc steps, including cluster-creation steps, see
[Managed Service for Apache Spark steps](https://gcpdiag.dev/runbook/steps/dataproc/).

### Run the `gcpdiag` command

You can run the `gcpdiag` command from [Cloud Shell](https://docs.cloud.google.com/shell) in the
Google Cloud console or within a [Docker](https://www.docker.com/) container.

## Google Cloud console

1. Complete and then copy the following command.

```console
gcpdiag runbook managed-spark/cluster-creation \
    --parameter project_id=PROJECT_ID \
    --parameter cluster_name=CLUSTER_NAME \
    --parameter OPTIONAL_FLAGS
```
2. Open the Google Cloud console and activate Cloud Shell.
[Open Cloud console](https://console.cloud.google.com/dataproc/clusters?cloudshell=true)
3. Paste the copied command.
4. Run the `gcpdiag` command, which downloads the `gcpdiag` docker image, and then performs diagnostic checks. If applicable, follow the output instructions to fix failed checks.

## Docker

You can [run `gcpdiag` using a wrapper](https://github.com/GoogleCloudPlatform/gcpdiag?tab=readme-ov-file#installation) that starts `gcpdiag` in a
[Docker](https://www.docker.com/) container. Docker or
[Podman](https://podman.io/) must be installed.

1. Copy and run the following command on your local workstation.

   ```console
   curl https://gcpdiag.dev/gcpdiag.sh >gcpdiag && chmod +x gcpdiag
   ```
2. Execute the `gcpdiag` command.

   ```console
   ./gcpdiag runbook managed-spark/cluster-creation \
       --parameter project_id=PROJECT_ID \
       --parameter cluster_name=CLUSTER_NAME \
       --parameter OPTIONAL_FLAGS
   ```
View [available parameters](https://gcpdiag.dev/runbook/diagnostic-trees/managed-spark/cluster-creation/#parameters) for this runbook.

Replace the following:

- <var class="readonly" translate="no">PROJECT_ID</var>: The ID of the project containing the resource
- <var class="readonly" translate="no">CLUSTER_NAME</var>: The name of the target Managed Service for Apache Spark cluster in your project
- <var class="readonly" translate="no">OPTIONAL_PARAMETERS</var>: Add one or more of the following optional parameters. **These parameters are required if
  the cluster has been deleted.**
  - `cluster_uuid`: The UUID of the target Managed Service for Apache Spark cluster in your project
  - `service_account`: The Managed Service for Apache Spark cluster [VM service account](https://docs.cloud.google.com/managed-spark/docs/concepts/configuring-clusters/service-accounts.md.txt#VM-service-account)
  - `subnetwork`: The Managed Service for Apache Spark cluster subnetwork full URI path
  - `internal_ip_only`: True or False
  - `cross_project`: The cross-project ID if the Managed Service for Apache Spark cluster uses a VM service account in another project

Useful flags:

- `--universe-domain`: If applicable, the [Trusted Partner Sovereign Cloud](https://cloud.google.com/blog/products/identity-security/new-sovereign-controls-for-gcp-via-assured-workloads) domain hosting the resource
- `--parameter` or `-p`: Runbook parameters

For a list and description of all `gcpdiag` tool flags, see the
[`gcpdiag` usage instructions](https://github.com/GoogleCloudPlatform/gcpdiag?tab=readme-ov-file#usage).

## What's next

- Learn more about the [Managed Service for Apache Spark monitoring and troubleshooting tools](https://docs.cloud.google.com/managed-spark/docs/support/troubleshoot-monitor).
- Learn how to [diagnose Managed Service for Apache Spark clusters](https://docs.cloud.google.com/managed-spark/docs/support/diagnose-command).
- Refer to the [Managed Service for Apache Spark FAQ](https://docs.cloud.google.com/managed-spark/docs/resources/faq.md.txt) document.