Troubleshoot system errors in Backup for GKE

Autopilot Standard

This page describes system-related errors you might experience when you use Backup for GKE, things to consider when you back up resources, and steps for troubleshooting the issue.

Error 100010108: Failed to backup PersistentVolumeClaim - Missing Backup Plan CMEK for encrypted disk

Error 100010108 occurs when creating regionally scoped volume backups, and the Backup Plan CMEK is mandated because the GKE cluster references an encrypted disk.

Understanding the Error

When creating regionally scoped volume backups, the Backup Plan CMEK is mandated if the GKE cluster references any encrypted disk. This is a strict requirement for regionally scoped volume backups; if the source disk is encrypted, the snapshot for that disk is only possible if a Backup Plan CMEK is provided. If your Backup Plan lacks this CMEK configuration, the snapshot creation fails, and you will see an error message similar to the following:

failed to create snapshot as disk "pvc-1" is encrypted with CMEK "projects/proj1/locations/us-east1/keyRings/keyring1/cryptoKeys/key1/cryptoKeyVersions/1", but no Backup Plan CMEK is provided. Please add CMEK to the Backup Plan.

Troubleshooting Steps

1. Identify the missing CMEK configuration: The error message confirms that the PVC being backed up is encrypted, but the Backup Plan configuration is missing an encryption key.
2. Add a CMEK to the Backup Plan: Update your Backup Plan configuration to include a Customer-Managed Encryption Key (CMEK). This key will be used to encrypt the backup data, including the regionally scoped volume backups.
3. Ensure proper permissions: Verify that the Backup for GKE service agent ([e-mail]) has the roles/cloudkms.cryptoKeyEncrypterDecrypter role assigned for the newly added CMEK.
4. Retest the Backup Operation: Once the Backup Plan is updated with the CMEK and permissions are granted, retry the backup operation.

Error 100020102: Strict permissive mode - Failed to backup CRD - Unsupported v1beta1 API version

Error 100020102 occurs when an attempt to back up a CustomResourceDefinition that was originally applied as a apiextensions.k8s.io/v1beta1 version fails because it lacks the structural schema that's required in the apiextensions.k8s.io/v1 API version. This error results in the following error message: Strict permissive mode - Failed to backup CRD - Unsupported v1beta1 API Version.

This error occurs because the apiextensions.k8s.io/v1 API version was removed in Google Kubernetes Engine version 1.22. For more information about the API removal for GKE version 1.22, see API removals for GKE v1.22.

Backup operation behavior in non-permissive mode

In non-permissive mode, or in a strict backup plan, the backup operation fails if it encounters a resource that can't be backed up, such as a CustomResourceDefinition that was created with the v1beta1 API. This error occurs because the resource lacks the structural schema that's required by the v1 API. The presence of this CustomResourceDefinition is considered a critical error because it might not restore correctly to a newer cluster.

To resolve this error, use the following instructions:

1. Identify the problematic CustomResourceDefinition by running the kubectl get crd command:

   kubectl get crd CRD_NAME
   

   Replace CRD_NAME with the name of the CustomResourceDefinition from your error message.

2. In the YAML output, confirm whether the CustomResourceDefinition was properly converted from the vbeta1 API to the v1 API by locating the following conditions:

   1. spec.versions: locate the spec.versions condition by looking through each version that's listed under the spec.versions field. If any of the spec.versions are missing the schema.openAIV3Schema field, the CustomResourceDefinition doesn't have a structural schema defined for that version.

   2. status.conditions: locate the status.conditions condition by finding the type:NonStructuralSchema condition. If the status.conditions's status is true, it explicitly confirms that the schema is not structural.

3. Upgrade the CustomResourceDefinition to v1 API version by using the following steps:

   1. Edit the existing CustomResourceDefinition to make it compatible with the v1 standard by adding a structural schema that defines every field and its type within the custom resource. For more information about how to add a structural schema, see Specifying a structural schema.

   2. Apply the compatible v1 manifest to your cluster.

      Note: In some cases, you might need to delete the previously created CustomResourceDefinition that lacks a structural schema and re-apply the v1 version so that the API server correctly handles the resource convention.

4. If the upgrade succeeds, re-attempt the backup operation. Otherwise, use one of the following methods to resolve the issue:

   • Delete the CustomResourceDefinition by running the kubectl delete crd command if the CustomResourceDefinition isn't being used in the cluster.

     Note: You shouldn't attempt to delete a CustomResourceDefinition if it's actively serving an existing custom resource in the cluster. You either need to attempt to upgrade the CustomResourceDefinition to v1 API version again or enable permissive mode.

     kubectl delete crd CRD_NAME
     

     Replace CRD_NAME with the name of the CustomResourceDefinitionyou want to delete.

   • Enable permissive mode on the backup plan, which allows Backup for GKE to skip the resource—including CustomResourceDefinitions in the v1beta1 API version—and continue with the rest of the backup operation. For more information about how to enable permissive mode, see Enable permissive mode on a backup plan.

     Warning: The problematic CustomResourceDefinition and its associated custom resources won't be backed up and won't be included when you restore. To successfully restore such a backup, you need to exclude the resources served by the CustomResourceDefinition from the restore, or create the CustomResourceDefinition in the target cluster before you start the restore.

5. Re-attempt the backup operation. If the operation continues to fail, contact Cloud Customer Care for further assistance.

Error 100040102: Namespace not found

Error 100040102 occurs when an attempt to perform a backup operation fails because a namespace specified in the backup scope can't be found within the cluster. The Backup for GKE agent wasn't able to locate one or more namespaces that were explicitly listed in the selectedNamespaces field of the BackupPlan configuration. Backup for GKE requires all specified namespaces to be present in the cluster at the time the backup operation is initiated. Failure to find the namespace results in the following error message:

Namespace [NAMESPACE_NAME] is not found.

To resolve this issue, use the following instructions:

1. Verify that the namespace has been entered correctly by checking the selectedNamespaces list in your BackupPlan configuration.

2. Confirm that the namespace reported in the error message exists by running the kubectl get namespace command:

   kubectl get namespace NAMESPACE_NAME
   

   Replace NAMESPACE_NAME with the name of the namespace reported in the error message.

   If the namespace doesn't exist, a message appears stating that the namespace wasn't found, for example, Error from server (NotFound): namespaces "[NAMESPACE_NAME]" not found.

3. Correct the BackupPlan. If the namespace was misspelled, update the BackupPlan with the correct namespace name. If the namespace truly no longer exists and does not need to be backed up, remove it from the selectedNamespaces list in the BackupPlan configuration.

4. Re-attempt the backup operation after making the necessary corrections to the BackupPlan and initiate a new backup.

If the operation continues to fail, contact Cloud Customer Care for further assistance.

What's next

• Read the Backup for GKE error codes overview page.