Troubleshoot issues

This page shows you how to resolve issues that you might encounter when using Eventarc Advanced.

CMEK issues

You can use customer-managed encryption keys (CMEK) to protect Eventarc. The keys are created and managed through Cloud Key Management Service (Cloud KMS). The following table describes different CMEK issues and how to resolve them when using Cloud KMS with Eventarc.

Issues that occur when creating or updating Eventarc resources

CMEK issue Error message Description
Disabled key $KEY is not enabled, current state is: DISABLED

The provided Cloud KMS key has been disabled for an Eventarc resource. Events or messages associated with the resource are no longer protected.

Solution:

  1. Display the key used for a resource:
  2. Re-enable the Cloud KMS key.
Exceeded quota Quota exceeded for limit

Your quota limit for Cloud KMS requests has been reached.

Solution:

  • Limit the number of Cloud KMS calls.
  • Increase the quota.
For more information, see Monitor and adjust Cloud KMS quotas.
Mismatched region Key region $REGION must match the resource to be protected

The provided KMS key region is different from the region of the resource.

Solution:

Use a Cloud KMS key from the same region.
Organization policy constraint project/PROJECT_ID violated org policy constraint

Eventarc is integrated with the following two organization policy constraints to help ensure CMEK usage across an organization. Any existing Eventarc resource isn't subject to a policy that is set after the resource is created; however, updating the resource might fail.

  • constraints/gcp.restrictNonCmekServices causes all resource creation requests without a specified Cloud KMS key to fail.

    Solution:

    Specify a Cloud KMS key for the Eventarc resource. For more information, see Require CMEKs for new Eventarc resources.

  • constraints/gcp.restrictCmekCryptoKeyProjects restricts the Cloud KMS keys that you can use to protect an Eventarc resource.

    Solution:

    Use a supported Cloud KMS key from an allowed Eventarc project, folder, or organization. For more information, see Restrict Cloud KMS keys for an Eventarc project.

Issues that occur during event delivery

CMEK issue Error message Description
Disabled key $KEY is not enabled, current state is: DISABLED

The provided Cloud KMS key has been disabled for an Eventarc resource. Events or messages associated with the resource are no longer protected.

Solution:

  1. Display the key used for a resource:
  2. Re-enable the Cloud KMS key.
Exceeded quota Quota exceeded for limit

Your quota limit for Cloud KMS requests has been reached.

Solution:

  • Limit the number of Cloud KMS calls.
  • Increase the quota.
For more information, see Monitor and adjust Cloud KMS quotas.
Permission error Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource $KEY (or it may not exist)

Either the provided Cloud KMS key doesn't exist or the Identity and Access Management (IAM) permission is not properly configured.

Solution:

To resolve issues that you might encounter when using externally managed keys through Cloud External Key Manager (Cloud EKM), see Cloud EKM error reference.

HTTP 400 Bad Request errors

Encountering a generic HTTP 400 Bad Request error for a pipeline often signals a transformation or message binding issue, or an issue related to the targeted endpoint. Make sure to do the following:

  • Look for syntax or logic errors in your CEL expressions. For example, in this document, see Transformation issues.
  • Verify that event messages conform to the expected schema. For example, in this document, see Invalid request format.
  • If the event destination itself is returning the error, check your targeted endpoint.

HTTP 503 Service Unavailable errors

If you encounter an HTTP 503 Service Unavailable error for a pipeline that routes messages to a Google destination using a DNS address—for example, Cloud Run—make sure that Private Google Access is enabled on the subnet used in the network attachment; otherwise, the DNS address can't be resolved.

Permission denied while using Eventarc service agent

If you encounter the following error while trying to create an Eventarc Advanced resource, wait a few minutes (potentially, seven), and then try creating the resource again:

Permission denied while using the Eventarc Service Agent. If you recently started to use
Eventarc, it may take a few minutes before all necessary permissions are propagated to the
Service Agent. Otherwise, verify that it has Eventarc Service Agent role.

A service agent acts as the identity of a given Google Cloud service for a particular project. For more information, see Service agents and view the Identity and Access Management (IAM) permissions for the Eventarc Service Agent role (roles/eventarc.serviceAgent).

If you are still running into the previous error after attempting to create your resource again, complete the following steps to verify that the Eventarc service agent exists in your Google Cloud project and has the necessary role:

  1. In the Google Cloud console, go to the IAM page.

    Go to IAM

  2. On the View by principals tab, select the Include Google-provided role grants checkbox.

  3. In the list of principals, locate the Eventarc service agent, which uses the following format:

    service-PROJECT_NUMBER@gcp-sa-eventarc.iam.gserviceaccount.com

  4. Verify that the service agent has the Eventarc Service Agent role. If the service agent doesn't have the role, then grant the role.

Publishing to Cloud Run job issues

This section describes how to resolve issues that you might encounter when publishing events to a Cloud Run job. For an end-to-end demonstration, see the quickstart.

HTTP 403 Forbidden errors

If you encounter an HTTP 403 Forbidden error while trying to execute a Cloud Run job that overrides the job configuration—for example, by defining a message binding—make sure that you have granted the Cloud Run Developer (roles/run.developer) or Cloud Run Jobs Executor With Overrides (roles/run.jobsExecutorWithOverrides) IAM role to the OAuth service account used by the Eventarc pipeline. For more information, see Override job configuration for a specific execution.

Invalid request format

When defining a message binding, you must use the http_endpoint_message_binding_template key to transform the event into the format expected by the Cloud Run Admin API—for example: {"body": ""}.

If the key is missing or empty, the event message routed to the job won't conform to the expected schema for the Cloud Run job endpoint, and this prevents the event message from successfully triggering the job execution.

Transformation issues

If your event data can't be or isn't transformed as expected when specifying a message binding, make sure that the syntax of your CEL expression is correct.

Incorrect CEL expression

For example, in the following expression, quoted string substitutions—such as \"${message.data}\"—are used to evaluate and insert variable values when CEL extension functions should be used instead:

cel_expression: "{\"body\":{\"overrides\":{\"containerOverrides\":[{\"env\":[{\"name\":\"CLOUD_EVENT_PAYLOAD\",\"value\":\"${message.data}\"},{\"name\":\"CLOUD_EVENT_ATTRIBUTES\",\"value\":\"${message.attributes}\"}]}]}}}"

Additionally, event attributes are top-level fields on the message object and aren't contained within a separate attributes field—such as message.attributes. Each attribute of an incoming CloudEvents message should be accessed using the message.key value, where key is the name of the attribute.

Correct CEL expression

To resolve these issues, append the toJsonString() function directly to the object to pass the message data as a JSON string. To capture the event attributes as a JSON string while excluding the payload, use message.removeFields(["data"]).toJsonString(). For example:

cel_expression: "{\"body\":{\"overrides\":{\"containerOverrides\":[{\"env\":[{\"name\":\"CLOUD_EVENT_PAYLOAD\",\"value\":\"message.data.toJsonString()\"},{\"name\":\"CLOUD_EVENT_ATTRIBUTES\",\"value\":\"message.removeFields(['data']).toJsonString()\"}]}]}}}"

For more information, see Transform received events.

What's next