Troubleshoot the admission webhook

This page shows you how to resolve issues with the Config Sync admission webhook. To learn more about the webhook, see Prevent config drift.

Troubleshoot KNV 2009 errors

The following sections help you resolve KNV2009 errors.

Admission webhook connection refused

If you have drift protection enabled, you might receive the following error when reconciler tries to apply a config to the cluster:

KNV2009: Internal error occurred: failed calling webhook "v1.admission-webhook.configsync.gke.io": Post "https://admission-webhook.config-management-system.svc:8676/admission-webhook?timeout=3s": dial tcp 10.92.2.14:8676: connect: connection refused

This error means that the admission webhook isn't ready yet or has become unhealthy. It's usually a transient error that you might see when bootstrapping Config Sync.

If the issue persists, describe the admission webhook Deployment to see if its Pods can be scheduled and are healthy:

kubectl describe deploy admission-webhook -n config-management-system

kubectl get pods -n config-management-system -l app=admission-webhook

A Deployment with healthy Pods has output similar to the following:

Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
...
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
...

Admission webhook request I/O timeout

If you receive an error similar to the following when the reconciler tries to apply a config to the cluster, the admission webhook port 8676 might be blocked by the firewall to the control plane network:

KNV2009: Internal error occurred: failed calling webhook "v1.admission-webhook.configsync.gke.io": Post https://admission-webhook.config-management-system.svc:8676/admission-webhook?timeout=3s: dial tcp 10.1.1.186:8676: i/o timeout

To resolve this issue, add a firewall rule to allow port 8676, which the Config Sync admission webhook uses for drift prevention. Port 8676 needs to be open from the control plane to the nodes because the control plane needs to be able to reach the webhook backend on the cluster nodes.

Admission webhook denied a request

If you receive the following error when you try to apply a change to a field that Config Sync manages, then you might have made a conflicting change:

error: OBJECT could not be patched: admission webhook "v1.admission-webhook.configsync.gke.io"
denied the request: fields managed by Config Sync can not be modified

If you have drift protection enabled, when you declare a field in a config and your repository is synced to a cluster, Config Sync manages that field. Any change you attempt to make to that field is a conflicting change.

For example, if you have a Deployment config in your repository with a label of environment:prod and you try to change that label to environment:dev in your cluster, there would be a conflicting change and you would receive the preceding error message. However, if you add a new label (for example, tier:frontend) to the Deployment, there wouldn't be a conflict.

If you want Config Sync to ignore any changes to an object, you can add the annotation described in Ignoring object mutations.

Failed to delete all resource types

A namespace stuck in the Terminating phase has the following condition:

    message: 'Failed to delete all resource types, 1 remaining: admission webhook
      "v1.admission-webhook.configsync.gke.io" denied the request: system:serviceaccount:kube-system:namespace-controller
      is not authorized to delete managed resource "_configmap_bookstore_cm1"'
    reason: ContentDeletionFailed
    status: "True"
    type: NamespaceDeletionContentFailure

This error happens when you are trying to delete a Namespace object from a root repository, but some objects under the namespace are still actively managed by a namespace reconciler. When a namespace is deleted, the namespace controller, whose service account is system:serviceaccount:kube-system:namespace-controller, would try to delete all the objects in that namespace. However, the Config Sync admission webhook only allows the root or namespace reconciler to delete these objects, and denies the namespace controller to delete these objects.

To work around this issue, delete the Config Sync admission webhook:

kubectl delete deployment.apps/admission-webhook -n config-management-system

The ConfigManagement Operator recreates the Config Sync admission webhook.

If this work around does not work, you might need to reinstall Config Sync.

To avoid running into the error again, remove the namespace repository before removing the namespace.

What's next

  • If you're still experiencing issues, check to see if your problem is a known issue.

  • If you can't find a solution to your problem in the documentation, see Get support for further help, including advice on the following topics: