GKE clusters running version 1.29 or later don't support
Transportation Layer Security (TLS) certificates signed with the [SHA-1
algorithm](https://en.wikipedia.org/wiki/SHA-1). To prevent impact to your
clusters, you need to replace incompatible certificates of
[webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#configure-admission-webhooks-on-the-fly)
and [extension API
server](https://kubernetes.io/docs/tasks/extend-kubernetes/configure-aggregation-layer/#contacting-the-extension-apiserver)
backends with certificates using [compatible signing
algorithms](https://docs.cloud.google.com/kubernetes-engine/docs/deprecations/sha1-1-29#compatible-signing-algorithms) before upgrading your clusters to
version 1.29.

### Impact to clusters of this removal

GKE [pauses automatic
upgrades](https://docs.cloud.google.com/kubernetes-engine/docs/deprecations#how_kubernetes_deprecations_work_with)
when it detects that a cluster is using certificates that are incompatible with
version 1.29. After you replace the certificates with certificates using
[compatible signing algorithms](https://docs.cloud.google.com/kubernetes-engine/docs/deprecations/sha1-1-29.md.txt#compatible-signing-algorithms), or version 1.28
reaches [end of support](https://docs.cloud.google.com/kubernetes-engine/docs/release-schedule.md.txt),
GKE resumes automatic upgrades.

If you don't replace the incompatible certificates before upgrading to 1.29,
you could experience the following issues with your clusters:

- GKE webhook backends that use TLS certificates signed with the SHA-1 algorithm will stop working due to authentication failure. Webhook calls will fail for the Kubernetes control plane communicating with your webhooks with incompatible certificates. Depending on your configuration, especially if you use admission webhooks, failure to contact a webhook might block resource creation on your cluster, such as Pod creation, which can be very disruptive.
- Calls to APIs served by the extension API servers will fail.

## Why Kubernetes is removing this capability

GKE operates open source Kubernetes, which uses the
[kube-apiserver component](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/)
to contact your webhook and extension API server backends using TLS. The
kube-apiserver component is written in the Go programming language.

From Go version 1.18, Go started [rejecting TLS certificates signed with the
SHA-1 algorithm](https://go.dev/doc/go1.18#sha1), but left a debug switch
`x509sha1=1` to enable the old behavior to ease the migration process.
GKE version 1.24 was the first version built using Go version
1.18. GKE builds of Kubernetes had enabled this debug switch
until version 1.29. The switch will be [removed in Go
1.24](https://github.com/golang/go/issues/62048). GKE 1.29 builds
Kubernetes with the switch disabled to prepare for the future Go removal of the
debug switch. After GKE upgrades your clusters to version 1.29,
calls from your cluster's control plane to webhooks or extension API servers in
the cluster that provide a TLS certificate signed with SHA-1 algorithm will
fail.

## Identify affected clusters

GKE monitors your clusters and uses the [Recommender
service](https://docs.cloud.google.com/recommender/docs/overview) to deliver guidance through insights and
recommendations identifying clusters that have webhook or extension API server
backends using TLS certificates signed with the SHA-1 algorithm. Or, you can use
logs to identify calls to affected backends from your cluster.

### How to get insights and recommendations

For clusters running version 1.24 or later, follow the instructions to [view
insights and
recommendations](https://docs.cloud.google.com/kubernetes-engine/docs/deprecations/viewing-deprecation-insights-and-recommendations#view-insights-recs).
You can get insights using the gcloud CLI, or the
Recommender API, filtering with the subtype
`DEPRECATION_K8S_SHA_1_CERTIFICATE`.

### How to get logs

For clusters running 1.24 or later with Cloud Logging
[enabled](https://docs.cloud.google.com/stackdriver/docs/solutions/gke/installing#installing),
GKE provides a
[Cloud Audit Logs log](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/audit-logging.md.txt#viewing_your_projects_admin_activity_log)
to identify calls to affected backends from your cluster. You can use the
following filter to search for the logs:

    logName =~ "projects/.*/logs/cloudaudit.googleapis.com%2Factivity"
    resource.type = "k8s_cluster"
    operation.producer = "k8s.io"
    "insecure-sha1.invalid-cert.kubernetes.io"

The audit logs include the hostname of the affected backend. To learn more about
how to interpret the results, see the next section.

## Interpret the guidance from insights and recommendations

A recommendation includes the hostname of the affected backend, and whether it
is webhook or extension API server. Hostnames that refer to Services in the
cluster follow the format `<service-name>.<namespace>.svc`.

If the affected backend certificate is from a webhook server, the hostname can
either be a Service in the cluster, or a URL. To learn more, see [Contacting the
webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#contacting-the-webhook).

If the affected certificate is from an extension API server, the hostname is a
Service in the cluster. To learn more, see the [Contacting the extension
apiserver](https://kubernetes.io/docs/tasks/extend-kubernetes/configure-aggregation-layer/#contacting-the-extension-apiserver).

After you have identified the affected backend, follow the instructions to
[inspect the certificate of a Service](https://docs.cloud.google.com/kubernetes-engine/docs/deprecations/sha1-1-29.md.txt#inspect-cert-service) or [Inspect the
certificate of a URL backend](https://docs.cloud.google.com/kubernetes-engine/docs/deprecations/sha1-1-29#inspect-url-backend), depending on the type.

If your clusters have not called servers with affected certificates in the last
30 days, you won't see any recommendations.

#### Example recommendations

See the following example list of recommendations:

    RECOMMENDATION_ID                     PRIMARY_IMPACT_CATEGORY  RECOMMENDATION_STATE  LAST_REFRESH_TIME               PRIORITY  RECOMMENDER_SUBTYPE                DESCRIPTION
    26bfcb32-6f2a-407c-874f-8cf55b3af912  RELIABILITY              ACTIVE                2024-02-15T01:09:04.454456273Z  P2        DEPRECATION_K8S_SHA_1_CERTIFICATE  Update the webhook and/or extension API servers that use certificates signed with SHA-1 algorithm to use certificates with compatible signing algorithms prior to upgrading the cluster to version 1.29. [Learn more](https://docs.cloud.google.com/kubernetes-engine/docs/deprecations/sha1-1-29.md.txt#mitigate_the_risk_of_upgrading_to_129).

To get details of the cluster and Service, [describe the
recommendation](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/optimize-with-recommenders#view-insights-recs).
The output for a Service named `example-webhook` in the `default` namespace is
similar to the following:

    associatedInsights:
    - insight: projects/<CLUSTER_PROJECT_NUMBER>/locations/<CLUSTER_LOCATION>/insightTypes/google.container.DiagnosisInsight/insights/d76887a8-9eed-41a0-9459-d49dee43455e
    content:
      overview:
        featureDeprecationRecommendation:
        - featureName: x.509_certificate_signature_algorithm
          featureReplacementValue: algorithm [compatible with GKE v1.29](https://docs.cloud.google.com/kubernetes-engine/docs/deprecations/sha1-1-29.md.txt#compatible-signing-algorithms)
          featureValue: SHA1
          stopServingVersion: '1.29'
          targetType: hostname
          targetValue: example-webhook.default.svc
        targetClusters:
        - clusterId: 3be916a554724c79a2314c8baee3fd57cf1c39df1ad34c3daf291db701b6d541
          clusterUri: //container.googleapis.com/projects/<CLUSTER_PROJECT_NUMBER>/locations/<CLUSTER_LOCATION>/clusters/<CLUSTER_NAME>
    description: Update the webhook and/or extension API servers that use certificates
      signed with SHA-1 algorithm to use certificates with compatible signing algorithms
      prior to upgrading the cluster to version 1.29. [Learn more](https://docs.cloud.google.com/kubernetes-engine/docs/deprecations/sha1-1-29.md.txt#mitigate_the_risk_of_upgrading_to_129).
    etag: '"ad50aac8278951d5"'
    lastRefreshTime: '2024-02-15T01:09:04.454456273Z'
    name: projects/<CLUSTER_PROJECT_NUMBER>/locations/<CLUSTER_LOCATION>/recommenders/google.container.DiagnosisRecommender/recommendations/26bfcb32-6f2a-407c-874f-8cf55b3af912
    primaryImpact:
      category: RELIABILITY
      reliabilityProjection:
        risks:
        - SERVICE_DISRUPTION
    priority: P2
    recommenderSubtype: DEPRECATION_K8S_SHA_1_CERTIFICATE
    stateInfo:
      state: ACTIVE
    targetResources:
    - //container.googleapis.com/projects/<CLUSTER_PROJECT_NUMBER>/locations/<CLUSTER_LOCATION>/clusters/<CLUSTER_NAME>

### Inspect the certificate of a Service

Both webhooks and extension API servers can be backed by Services.

After you have identified relevant backend Services to inspect, use the
following instructions to inspect the certificate of each Service to check which
certificates use the SHA-1 algorithm and need to be updated.

1. Find the selector and target port of the Service:

   ```bash
   kubectl describe service --namespace=NAMESPACE SERVICE_NAME
   ```

   Replace `NAMESPACE` and `SERVICE_NAME` with the values from `targetValue`.

   The output is similar to the following:

       Name: example-service
       Namespace: default
       Labels: run=nginx
       Selector: run=nginx
       Type: ClusterIP
       IP: 172.21.xxx.xxx
       Port: 443
       TargetPort: 444

   This output indicates that `example-service` has the selector `run=nginx` and
   target port `444`.

   > [!NOTE]
   > **Note:** When the `TargetPort` is not included, it is the same value as the `Port` field.

2. Find a Pod matching the selector:

   ```bash
   kubectl get pods --namespace=NAMESPACE --selector=run=nginx
   ```

   The output is similar to the following:

       NAME          READY   STATUS    RESTARTS   AGE
       example-pod   1/1     Running   0          21m

   This output indicates that the matching Pod is `example-pod`.
3. Set up a port forward from your `kubectl` localhost to the Pod:

   > [!NOTE]
   > **Note:** These next commands use port `8888`, however you can pick a different port, as this port is chosen arbitrarily.

   ```bash
   kubectl port-forward --namespace=NAMESPACE pods/example-pod 8888:SERVICE_TARGET_PORT &
   ```

   Replace `SERVICE_TARGET_PORT` with the `TargetPort` value
   from the Service. If `TargetPort` is not included, use the `Port` value.
4. Use `openssl` to show the certificate that the Service uses:

   ```bash
   openssl s_client -connect localhost:8888 </dev/null | openssl x509 -noout -text
   ```

   This example output shows a valid certificate signed with the
   [SHA-256 algorithm](https://en.wikipedia.org/wiki/SHA-2):

       Certificate:
           Data:
               ...
               Signature Algorithm: sha256WithRSAEncryption
       ...
           Signature Algorithm: sha256WithRSAEncryption

   This example output shows an invalid certificate signed with the SHA-1
   algorithm:

       Certificate:
           Data:
               ...
               Signature Algorithm: sha1WithRSAEncryption
       ...
           Signature Algorithm: sha1WithRSAEncryption

   If the output from the certificate is similar, you must update the certificate
   to use a [compatible signing algorithm](https://docs.cloud.google.com/kubernetes-engine/docs/deprecations/sha1-1-29.md.txt#compatible-signing-algorithms). For
   example, if you use the `certificate.k8s.io` API to manage TLS certificates in
   your cluster, you can follow the instructions to [create a certificate signing
   request](https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/#create-a-certificate-signing-request).

#### Clean up the port forward

To clean up the port forward running in the background, find the process and
terminate it.

1. Run the following command to list the running processes:

   ```bash
   jobs
   ```

   See the output to get the ID of the process to terminate:

       [1]+  Running                 kubectl port-forward pods/example-pod 8888:444 &

   This example output indicates that the process ID is `1`.
2. Terminate the process, replacing `PROCESS_ID`:

   ```bash
   kill %PROCESS_ID
   ```

   See the following output:

       [1]+  Terminated              kubectl port-forward pods/example 8888:444

   This example output shows that the process was terminated.

### Inspect the certificate of a URL backend

If the webhook uses a `url` backend, directly connect to the hostname specified
in the URL. For example, if the URL is `https://example.com:123/foo/bar`, use
the following `openssl` command to show the certificate that the backend uses:

```bash
openssl s_client -connect example.com:123 </dev/null | openssl x509 -noout -text
```

This example output shows a valid certificate signed with the
[SHA-256 algorithm](https://en.wikipedia.org/wiki/SHA-2):

    Certificate:
        Data:
            ...
            Signature Algorithm: sha256WithRSAEncryption
    ...
        Signature Algorithm: sha256WithRSAEncryption

This example output shows an invalid certificate signed with the SHA-1
algorithm:

    Certificate:
        Data:
            ...
            Signature Algorithm: sha1WithRSAEncryption
    ...
        Signature Algorithm: sha1WithRSAEncryption

If the output from the certificate is similar, you must update the certificate
to use a [compatible signing algorithm](https://docs.cloud.google.com/kubernetes-engine/docs/deprecations/sha1-1-29.md.txt#compatible-signing-algorithms). For
example, if you use the `certificate.k8s.io` API to manage TLS certificates in
your cluster, you can follow the instructions to [create a certificate signing
request](https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/#create-a-certificate-signing-request).

## Mitigate the risk of upgrading to 1.29

After you have identified affected clusters and their backend Services using
certificates signed with SHA-1 algorithm, you must update the Services to use
certificates with [compatible signing
algorithms](https://docs.cloud.google.com/kubernetes-engine/docs/deprecations/sha1-1-29#compatible-signing-algorithms) prior to upgrading the clusters to
version 1.29.

Affected clusters are automatically detected by GKE and [won't be
auto-upgraded](https://docs.cloud.google.com/kubernetes-engine/docs/deprecations#auto-upgrade-pause) to
version 1.29 until either incompatible certificates are no longer used or
version 1.28 reaches [end of support](https://docs.cloud.google.com/kubernetes-engine/docs/release-schedule.md.txt).
Once 1.28 reaches end of support, the clusters will be auto-upgraded to 1.29.

### Compatible signing algorithms

GKE version 1.29 is compatible with supported algorithms in the
Go x509 package. This includes the following algorithms:

- `SHA256WithRSA`
- `SHA384WithRSA`
- `SHA512WithRSA`
- `ECDSAWithSHA256`
- `ECDSAWithSHA384`
- `ECDSAWithSHA512`
- `SHA256WithRSAPSS`
- `SHA384WithRSAPSS`
- `SHA512WithRSAPSS`
- `PureEd25519`

To find available algorithms, see the [x509.go source
file](https://go.dev/src/crypto/x509/x509.go) and search for
`UnknownSignatureAlgorithm SignatureAlgorithm = iota`. Algorithms that the Go
x509 package supports are listed in the `const` block with this line. To find
unsupported insecure signing algorithms, search for uses of
`InsecureAlgorithmError` in the file.

## Resources

See the following resources for additional information on this change:

- Go 1.18 [release notes](https://go.dev/doc/go1.18#sha1)
- Go 1.24 [x509sha1 debug switch
  removal](https://golang.org/doc/go1.15#commonname)
- Go x509 [supported algorithms](https://go.dev/src/crypto/x509/x509.go)