> [!NOTE]
> **Note:** This document uses a [general availability](https://cloud.google.com/products#product-launch-stages) feature named *GKE control plane authority* that's only available for eligible Google Cloud customers. To request access to this feature, contact your Google Cloud account team.

This page describes how to verify connections made by Google personnel to your
Google Kubernetes Engine (GKE) cluster control plane by correlating
GKE logs with Access Transparency logs.

Access Transparency logs record the actions that Google personnel take when
accessing your content. This guide is intended for security administrators who
want additional verification of the contents of Access Transparency logs and associated
Access Approval approvals by correlating with additional logging
sources from GKE. **This verification is entirely optional and is
not required to secure your control plane.**

Ensure that you're familiar with the following concepts:

- [Access Transparency](https://docs.cloud.google.com/assured-workloads/access-transparency/docs/overview.md.txt)
- [Access Transparency logs](https://docs.cloud.google.com/assured-workloads/access-transparency/docs/reading-logs.md.txt)
- [Access Approval](https://docs.cloud.google.com/assured-workloads/access-approval/docs/overview.md.txt)
- [SSH connections in Compute Engine](https://docs.cloud.google.com/compute/docs/instances/ssh.md.txt)

This page describes one part of a set of optional control plane features in
GKE that lets you perform tasks like verifying your control plane security posture
or configuring encryption and credential signing in the control plane using keys that you manage.
For details, see
[About GKE control plane authority](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-control-plane-authority.md.txt).

By default, Google Cloud applies various security measures to the managed control plane.
This page describes *optional* capabilities that give you more visibility or control over
the GKE control plane.

## About Google access to cluster control plane instances

During troubleshooting sessions or for other justified business reasons, Google
personnel like site reliability engineers and Cloud Customer Care employees might
need *administrative access* to the Compute Engine instances that host
your control plane. Depending on your Customer Care support package and
configuration, Access Transparency provides detailed audit logging for this
administrative access. Access Approval lets you require explicit
approval before any Google personnel can access your resources. To learn more
about administrative access and the tooling that you can use to authorize access
and to record changes, see
[Administrative access for Google employees](https://docs.cloud.google.com/docs/security/overview/whitepaper.md.txt#administrative_access_for_google_employees).

### Control plane access logs

When you enable GKE control plane authority, GKE
generates *control plane access logs* that you can optionally use to
cross-reference the audit logs generated by Access Transparency and by
Access Approval. GKE adds control plane access
logs to the
[`_Default` bucket in Logging](https://docs.cloud.google.com/logging/docs/routing/overview.md.txt#default-bucket)
to record incoming network connections and specific SSH events
in your control plane instances. You must enable
GKE control plane authority in your project to generate control
plane access logs for your clusters.

GKE generates the following access logs for the control plane:

- [Control plane connection logs](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/verify-google-connections-control-plane.md.txt#control-plane-connection-logs)
- [Control plane SSH logs](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/verify-google-connections-control-plane.md.txt#control-plane-ssh-logs)

The volume of control plane connection logs depends on factors like the number
of nodes in the cluster, the number of control plane instances (regional
clusters have more control plane instances than zonal clusters), and how often
your workloads call the Kubernetes API server. The volume of SSH logs is small
and depends on the number of node restarts.

To verify the connections to your control plane, you find the control plane
access logs for your cluster and match those logs to audit logs from
Access Transparency and Access Approval. This lets you confirm
that all of the SSH connections to your control plane instances were as a result
of authorized administrative access by Google personnel. When you enable
GKE control plane authority for your cluster, all SSH access by Google
personnel to your control plane is *non-interactive*, which means that every SSH
connection runs a single command that you authorize.
To see command-level details for administrative access, enroll in Augmented
Access Approval and Augmented Access Transparency. For more information
about eligibility and enrollment, contact your Google Cloud account team.


## Pricing

The following pricing considerations apply:

- Control plane access logs are subject to [Logging pricing](https://cloud.google.com/stackdriver/pricing#logging-cost).
- Access Transparency is included with specific Customer Care subscriptions. For details, see [Access Transparency pricing](https://cloud.google.com/assured-workloads/access-transparency/pricing).

## Before you begin


Before you start, make sure that you have performed the following tasks:

- Enable the Google Kubernetes Engine API.
[Enable Google Kubernetes Engine API](https://console.cloud.google.com/apis/enableflow?apiid=container.googleapis.com)
- To use the Google Cloud CLI for this task, [install](https://docs.cloud.google.com/sdk/docs/install) and then [initialize](https://docs.cloud.google.com/sdk/docs/initialize.md.txt) the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the `gcloud components update` command. Earlier gcloud CLI versions might not support running the commands in this document.

  > [!NOTE]
  > **Note:** For existing gcloud CLI installations, make sure to set the `compute/region` [property](https://docs.cloud.google.com/sdk/docs/properties.md.txt#setting_properties). If you use primarily zonal clusters, set the `compute/zone` instead. By setting a default location, you can avoid errors in the gcloud CLI like the following: `One of [--zone, --region] must be supplied: Please specify location`. You might need to specify the location in certain commands if the location of your cluster differs from the default that you set.

<!-- -->

- Enable the Cloud Logging API.


  <br />

  **Roles required to enable APIs**


  To enable APIs, you need the `serviceusage.services.enable` permission. If you
  created the project, then you likely already have this permission through the
  Owner role (`roles/owner`). Otherwise, you can get this permission through the
  Service Usage Admin role (`roles/serviceusage.serviceUsageAdmin`).
  [Learn how to grant roles](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).

  [Enable the API](https://console.cloud.google.com/apis/enableflow?apiid=logging.googleapis.com)

  <br />

- Enable Access Transparency for your organization. For instructions, see
  [Enabling Access Transparency](https://docs.cloud.google.com/assured-workloads/access-transparency/docs/enable.md.txt).

- Optionally, enable Access Approval for your project and select the
  GKE service. For instructions, see
  [Review and approve access requests using the Google-managed signing key](https://docs.cloud.google.com/assured-workloads/access-approval/docs/review-approve-access-requests-google-keys.md.txt).

- Ensure that your environment is eligible to use
  GKE control plane authority features. To opt-in to these
  features, contact your Google Cloud sales team.

### Requirements

Control plane access logs require GKE version 1.31.1-gke.1846000
or later.

### Required roles and permissions


To get the permissions that
you need to enable log generation and to access and process logs,

ask your administrator to grant you the
following IAM roles:

- Enable control plane connection logging in your cluster: [Kubernetes Engine Cluster Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/container.md.txt#container.clusterAdmin) (`roles/container.clusterAdmin`) on your project
- Access logs and to use Logs Explorer and Observability Analytics: [Logs Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/logging.md.txt#logging.viewer) (`roles/logging.viewer`) on your project
- Enable Access Transparency for the organization: [Access Transparency Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/axt.md.txt#axt.admin) (`roles/axt.admin`) on your organization


For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).


You might also be able to get
the required permissions through [custom
roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) or other [predefined
roles](https://docs.cloud.google.com/iam/docs/roles-overview#predefined).

## Enable GKE cluster control plane access logs

You can enable control plane access log generation for Autopilot mode
and Standard mode clusters by enabling the corresponding logging
component. For more information about control plane log types, see
[View GKE logs](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/view-logs.md.txt#control_plane_logs).

The supported logging component names for control plane access logs are as
follows:

- Control plane SSH logs: `KCP_SSHD`
- Control plane connection logs: `KCP_CONNECTION`

### Enable control plane access logs on a new cluster

The following example creates an Autopilot mode cluster with both types
of control plane access logs enabled. To only enable one type of control plane
access log, omit the corresponding component name from the command.

```bash
gcloud container clusters create-auto CLUSTER_NAME \
    --location=LOCATION \
    --logging=SYSTEM,KCP_SSHD,KCP_CONNECTION
```

Replace the following:

- `CLUSTER_NAME`: the name for the new cluster.
- `LOCATION`: the location to create the cluster in.

To specify logging components when you create a cluster using the
GKE API, in the
[`projects.locations.clusters.create`](https://docs.cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters/create.md.txt)
method, set the corresponding values in the
[`LoggingConfig` object](https://docs.cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters.md.txt#loggingconfig)
of the `Cluster` resource.

### Enable control plane access logs on an existing cluster

> [!CAUTION]
> **Caution:** Existing clusters might have specific logging components already configured. When you update an existing cluster using the Google Cloud CLI or using an API call, you must include the existing logging configuration in your update request.

To update the logging configuration of an existing cluster to enable control
plane access logs, you must do the following:

1. Find the existing logging components that your cluster uses.
2. Identify the corresponding values to specify in the `--logging` flag in the gcloud CLI to keep those logging components enabled.
3. Update your cluster logging configuration to enable the control plane access logs **alongside** your existing logging configuration.

The values that you specify for the `--logging` flag in the
`gcloud container clusters update` command are different from the values that
you see when you describe your cluster.

1. Check the existing logging configuration of the cluster:

   ```bash
   gcloud container clusters describe CLUSTER_NAME \
       --location=LOCATION \
       --flatten=loggingConfig \
       --format='csv[delimiter=",",no-heading](componentConfig.enableComponents)'
   ```

   The output is similar to the following:

       SYSTEM_COMPONENTS,WORKLOADS,APISERVER,SCHEDULER,CONTROLLER_MANAGER

2. Identify the gcloud CLI values for the `--logging` flag that
   correspond to the logging component configuration from the output of the
   preceding step. For a list of the gcloud CLI values that
   correspond to specific logging components, see the
   [Available logs](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-logs.md.txt#available-logs) table.

3. Update the logging configuration with control plane access logs:

   ```bash
   gcloud container clusters update CLUSTER_NAME \
       --location=LOCATION \
       --logging=SYSTEM,EXISTING_LOGS,KCP_ACCESS_LOGS
   ```

   Replace the following:
   - `EXISTING_LOGS`: a comma-separated list of logging components that your cluster already uses. Ensure that you specify the *gcloud CLI values* that correspond to these logging components, taken from the [Available logs](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/about-logs.md.txt#available-logs) table.
   - `KCP_ACCESS_LOGS`: a comma-separated list of the
     control plane access log types to enable for the cluster, as follows:

     - For control plane SSH logs, specify `KCP_SSHD`.
     - For control plane connection logs, specify `KCP_CONNECTION`.

To specify logging components when you update a cluster using the
GKE API, in the
[`projects.locations.clusters.update`](https://docs.cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters/update.md.txt)
method, set the existing **and** new logging component values in the
[`LoggingConfig` object](https://docs.cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters.md.txt#loggingconfig)
of the
[`ClusterUpdate` resource](https://docs.cloud.google.com/kubernetes-engine/docs/reference/rest/v1/ClusterUpdate.md.txt).

#### Example cluster update to enable control plane access logs

Consider a cluster that shows the following logging configuration after running
the `gcloud container clusters describe` command:

    SYSTEM_COMPONENTS,WORKLOADS,APISERVER,SCHEDULER,CONTROLLER_MANAGER

The following cluster update command enables both types of control plane access
logs while retaining the existing log configuration for this example cluster:

```bash
gcloud container clusters update example-cluster \
    --location=us-central1 \
    --logging=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,KCP_SSHD,KCP_CONNECTION
```

## Cross-reference control plane access logs with Access Transparency logs

To verify control plane access for a cluster, get the control plane connection
logs, the control plane SSH logs, and the Access Transparency logs for that
cluster:

1. In the Google Cloud console, open the **Logs Explorer** page.


   [Go to Logs Explorer](https://console.cloud.google.com/logs/query)

   <br />

2. To get all of the logs for a specific cluster, including the control plane
   access logs and Access Transparency logs, run the following query:

   ```sql
   (logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-connection"
   resource.labels.cluster_name="CLUSTER_NAME"
   jsonPayload.connection.dest_port="22")
   OR
   (logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-sshd"
   resource.labels.cluster_name="CLUSTER_NAME")
   OR
   (logName="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Faccess_transparency"
   json_payload.accesses.methodName="GoogleInternal.SSH.Master"
   json_payload.accesses.resourceName="//container.googleapis.com/projects/PROJECT_NUMBER/locations/LOCATION/clusters/CLUSTER_NAME")
   ```

The output should display all of the following types of logs for your cluster:

- Access Transparency log
- Control plane connection log
- SSH logs for each SSH session

### Perform verification checks

Your primary verification check is whether you see all of the log types for any
SSH connections when you run the Logging query from the preceding
section. Every Access Transparency log should have a corresponding control plane
connection log and one or more SSH logs. These logs are for actions that humans
perform in your control plane instances, so the log volume should be small.

You can optionally perform the following additional checks of the log contents:

1. For each control plane SSH log, check whether an Access Transparency log exists in a 15-minute window before the timestamp of the SSH log. This window of time accounts for the final SSH session closure happening several minutes after the initial connection was logged by Access Transparency.
2. For each control plane connection log, check whether an Access Transparency log exists in a 5-minute window before the timestamp of the control plane connection log.
3. If you use Access Approval for your cluster, check whether
   each Access Transparency log has a corresponding `accessApprovals` field.
   Cross-reference this field against
   [Access Approval requests](https://docs.cloud.google.com/assured-workloads/access-approval/docs/approval-request-details.md.txt)
   for your cluster.

   To get Access Approval requests for your project, see
   [View historical Access Approval requests](https://docs.cloud.google.com/assured-workloads/access-approval/docs/view-historical-requests.md.txt).
   Access Approval might be subject to
   [exclusions](https://docs.cloud.google.com/assured-workloads/access-approval/docs/overview.md.txt#exclusions).
4. Optionally,
   [validate the signature](https://docs.cloud.google.com/assured-workloads/access-approval/docs/validate-request-signature.md.txt)
   of the signed Access Approval associated with the
   Access Transparency log.

## Control plane access log details

This section provides details and examples of the control plane access logs that
GKE generates when Google personnel connect to your control plane
instances.

### Control plane connection logs

GKE adds a *control plane connection log* for every new incoming
network connection to a control plane instance. These logs include specific
details like the following:

- Source and destination IP addresses and ports
- Connection direction and protocol

The following example is of a control plane connection log:

```json
{
  insertId: "z1eq8wonio335a5h",
  jsonPayload: {
    instance: {
      vm_name: "gke-dee49f0d6fa34ce3a2ac-f513-d195-vm",
      zone: "us-central1-c"
    },
    cluster: {
      cluster_id: "CLUSTER_ID",
      cluster_urn: "//container.googleapis.com/projects/PROJECT_NUMBER/locations/us-central1-c/clusters/CLUSTER_NAME"
    },
    connection: {
      state: "NEW",
      src_ip: "192.0.2.100",
      src_port: 32774,
      dest_ip: "203.0.113.12",
      dest_port: 22,
      direction: "INGRESS"
      protocol: "TCP"
    },
  }
  logName: "projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-connection",
  receiveTimestamp: "2024-04-11T04:08:01.883070399Z",
  resource: {
    labels: {
      cluster_name: "CLUSTER_NAME",
      location: "us-central1-c",
      project_id: "PROJECT_ID"
    }
    type: "gke_cluster",
  }
  severity: "NOTICE",
  timestamp: "2024-04-11T04:07:59.019330Z"
}
```

The following fields in the log entry are relevant to verifying Google's
actions:

- `cluster.cluster_urn`: The fully-qualified resource identifier of the cluster.
  This identifier has the format
  `//container.googleapis.com/projects/PROJECT_NUMBER/locations/LOCATION/clusters/CLUSTER_NAME`,
  with the following variables:

  - `PROJECT_NUMBER`: the numerical project number of your clusters project.
  - `LOCATION`: the Google Cloud location of your cluster.
  - `CLUSTER_NAME`: the name of your cluster.
- `connection`: Details about the connection attempt. This field has the
  following information:

  - `state`: the state of the connection. For new connections, the value is `NEW`.
  - `src_ip`: the IP address of the connection source.
  - `src_port`: the port number of the connection source.
  - `dest_ip`: the internal IP address of your control plane VM.
  - `dest_port`: the destination port number.
  - `direction`: the connection direction. The value is always `INGRESS`.
  - `protocol`: the IP protocol, like `TCP`.

### Control plane SSH logs

GKE adds control plane SSH logs for events related to SSH
connections to control plane instances. GKE records the following
events:

- SSH key accepted for a user
- Session changed status from 0 to 1, indicating that the user successfully logged in
- SSH session opened
- SSH session closed
- Session changed status from 1 to 0, indicating that the user logged out
- SSH session failed

For example, the following control plane SSH log is for an SSH session being
opened:

```json
{
  insertId: "8llczemdulwbbwpa",
  jsonPayload: {
    instance: {
      vm_name: "gke-06cb920c609941c0a5ce-6840-40e9-vm",
      zone: "us-central1-c"
    },
    cluster: {
      cluster_id: "891e6d12889747748c1ac16ffcc6cb7c0a96450b36864eb680917c119fd801d0",
      cluster_urn: "//container.googleapis.com/projects/PROJECT_NUMBER/locations/us-central1/clusters/CLUSTER_NAME",
    },
    message: "pam_unix(sshd:session): session opened for user REDACTED by (uid=0)",
  },
  logName: "projects/PROJECT_ID/logs/container.googleapis.com%2Fkcp-ssh",
  receiveTimestamp: "2024-04-09T13:21:55.231436462Z"
  resource: {
    type: "gke_cluster",
    labels: {
      cluster_name: "CLUSTER_NAME",
      location: "us-central1",
      project_id: "PROJECT_ID"
    }
  },
  severity: "NOTICE",
  timestamp: "2024-04-09T13:21:50.742246Z"
}
```

The following fields in the log entry are relevant to verifying Google's
actions:

- `cluster.cluster_urn`: The fully-qualified resource identifier of the cluster.
  This identifier has the format
  `//container.googleapis.com/projects/PROJECT_NUMBER/locations/LOCATION/clusters/CLUSTER_NAME`,
  with the following variables:

  - `PROJECT_NUMBER`: the numerical project number of your cluster project.
  - `LOCATION`: the Google Cloud location of your cluster.
  - `CLUSTER_NAME`: the name of your cluster.
- `message`: Details about the SSH connection.

## Disable control plane access logs

1. To view the specific log types that your cluster uses, run the following
   command:

   ```bash
   gcloud container clusters describe CLUSTER_NAME \
       --location=LOCATION \
       --flatten=loggingConfig \
       --format='csv[delimiter=",",no-heading](componentConfig.enableComponents)'
   ```

   The output is similar to the following:

       SYSTEM_COMPONENTS,WORKLOADS,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,KCP_SSHD,KCP_CONNECTION

2. To disable the control plane access logs for a cluster, run the following
   command:

   ```bash
   gcloud container clusters update CLUSTER_NAME \
       --location=LOCATION \
       --logging=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER
   ```

In the `--logging` flag, specify the logging components from the output of the
previous command. This example command disables control plane access logs but
keeps other control plane component logs enabled.

To disable logging components using the GKE API, set the
corresponding values in the `LoggingConfig` object of the `ClusterUpdate` resource in
the
[`projects.locations.clusters.update` method](https://docs.cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters/update.md.txt).

## What's next

- Learn about [control plane security](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/control-plane-security.md.txt).
- Learn about [administrative access for Google employees](https://docs.cloud.google.com/docs/security/overview/whitepaper.md.txt#administrative_access_for_google_employees).
- Learn how to [configure logging and monitoring for GKE](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/config-logging-monitoring).
- Learn how to [configure field-level access to logs](https://docs.cloud.google.com/logging/docs/field-level-acl.md.txt).
- Learn about [logging usage limits](https://docs.cloud.google.com/logging/quotas.md.txt).
- Learn about [Augmented Administrative Access](https://docs.cloud.google.com/assured-workloads/access-approval/docs/augmented/augmented-access-overview).