Use External Secrets Operator

This page describes how to use External Secrets Operator (ESO) to synchronize secrets from Secret Manager to your Google Distributed Cloud connected clusters.

External Secrets Operator is an open-source Kubernetes operator that integrates external secret management systems. The operator reads information from external APIs and automatically injects the values into a Kubernetes secret.

Prerequisites

Before you can use External Secrets Operator, you need to do the following:

  • Create a functional Distributed Cloud connected cluster.
  • Verify that the following APIs are enabled in your Google Cloud project. For information about enabling APIs, see Enabling services:
    • secretmanager.googleapis.com
    • iamcredentials.googleapis.com

  • Ensure that you have the following command line tools installed:

    • The latest version of the Google Cloud CLI, which includes gcloud, the command line tool for interacting with Google Cloud.
    • kubectl

    If you're using Cloud Shell as your shell environment for interacting with Google Cloud, these tools are installed for you.

  • Ensure that you have initialized the gcloud CLI for use with your project.

  • Enable Workload Identity Federation on the cluster. The Workload Identity Pool is automatically available and follows the format PROJECT_ID.svc.id.goog.

  • Install External Secrets Operator on your cluster. For installation instructions, see the External Secrets Operator documentation. We recommend installing the operator in a dedicated namespace, such as external-secrets. Don't install it in system-managed namespaces such as kube-system or gke-system.

Distributed Cloud connected clusters are automatically registered to a Fleet in the project where they are created.

Authentication

External Secrets Operator requires authentication to access Secret Manager. Distributed Cloud connected uses Fleet Workload Identity Federation to allow workloads to authenticate to Google Cloud APIs.

To configure authentication for External Secrets Operator:

  1. Set up the trust relationship between your cluster and Google Cloud by following the instructions in Workload Identity Cluster Authentication.
  2. Ensure that the Google service account that External Secrets Operator uses has the Secret Manager Secret Accessor (roles/secretmanager.secretAccessor) role on the secrets that you want to access.

  3. Configure the External Secrets Operator Pod to use Workload Identity Federation by following the Configure the workload instructions.

    Most customers use Workload Identity Federation for authentication. To configure Workload Identity Federation, you must annotate the Kubernetes ServiceAccount used by the operator with the Google service account. To add this annotation, run the kubectl annotate command:

    kubectl annotate serviceaccount KSA_NAME \
    --namespace OPERATOR_NAMESPACE \
    iam.gke.io/gcp-service-account=GSA_EMAIL

    Alternatively, you can add the annotation to your ServiceAccount YAML:

    apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: GSA_EMAIL
  name: KSA_NAME
  namespace: OPERATOR_NAMESPACE

    Replace the following values:

    • KSA_NAME: the name of the Kubernetes ServiceAccount used by the operator
    • OPERATOR_NAMESPACE: the namespace where you installed the operator
    • GSA_EMAIL: the email address of the Google service account

  4. Provide the credential configuration file to the operator Pod. For more information, see Configure the workload.

Create ESO resources to synchronize secrets

After you configure authentication, you can create External Secrets Operator resources to synchronize secrets.

Create a SecretStore

A SecretStore specifies how to access the external secret management system. You can create SecretStore resources in the same namespace as External Secrets Operator or in application namespaces. For more information, see SecretStore in the Kubernetes documentation.

  1. Create a file named secret-store.yaml with the following content:

    apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: gcp-store
  namespace: NAMESPACE
spec:
  provider:
    gcpsm:
      projectID: PROJECT_ID

    Replace the following values:

    • NAMESPACE: the namespace where you want to create the SecretStore
    • PROJECT_ID: your Google Cloud project ID where secrets are stored

  2. Use the kubectl apply command to apply the manifest:

    kubectl apply -f secret-store.yaml

Create a ClusterSecretStore

A ClusterSecretStore is a cluster-scoped resource that ExternalSecret resources can use in any namespace. For more information, see ClusterSecretStore in the Kubernetes documentation.

  1. Create a file named cluster-secret-store.yaml with the following content:

    apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: gcp-cluster-store
spec:
  provider:
    gcpsm:
      projectID: PROJECT_ID

    Replace PROJECT_ID with your Google Cloud project ID where secrets are stored.

  2. Apply the manifest:

    kubectl apply -f cluster-secret-store.yaml

Create an ExternalSecret

An ExternalSecret declares what data to fetch and where to store it in the cluster. Create ExternalSecret resources in the namespace where the application Pods consume the resulting Kubernetes Secret. For more information, see ExternalSecret in the Kubernetes documentation.

  1. Create a file named external-secret.yaml with the following content:

    apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: EXTERNAL_SECRET
  namespace: NAMESPACE
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: SecretStore
    name: gcp-store
  target:
    name: K8S_SECRET_NAME
    creationPolicy: Owner
  data:
  - secretKey: K8S_SECRET_KEY
    remoteRef:
      key: SECRET_NAME

    Replace the following values:

    • EXTERNAL_SECRET: the name of the ExternalSecret resource.
    • NAMESPACE: the namespace where you created the SecretStore.
    • K8S_SECRET_NAME: the name of the Kubernetes Secret to be created by ESO.
    • K8S_SECRET_KEY: the key in the Kubernetes Secret data.
    • SECRET_NAME: the name of the secret in Google Cloud Secret Manager.

    If you use a ClusterSecretStore, set kind: ClusterSecretStore, and update the name in secretStoreRef.

  2. Apply the manifest:

    kubectl apply -f external-secret.yaml

Synchronize multiple keys from a JSON secret

If your secret in Secret Manager contains a JSON string, you can extract all keys as individual entries in the Kubernetes secret.

  1. Create a file named external-secret-json.yaml with the following content:

    apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: EXTERNAL_SECRET
  namespace: NAMESPACE
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: SecretStore
    name: gcp-store
  target:
    name: K8S_SECRET_NAME
    creationPolicy: Owner
  dataFrom:
  - extract:
      key: SECRET_NAME

  2. Apply the manifest:

    kubectl apply -f external-secret-json.yaml

Each key-value pair in the JSON secret is mapped to a key-value pair in the resulting Kubernetes secret.

Troubleshooting

If your secrets don't synchronize, use the following steps to troubleshoot:

  1. Use the kubectl get command to check the status of the ExternalSecret resource:

    kubectl get externalsecret EXTERNAL_SECRET -n NAMESPACE -o yaml

    Examine the status section for any error messages or failed conditions.

  2. Use the kubectl logs command to check the logs of the External Secrets Operator controller Pod:

    kubectl logs -l app.kubernetes.io/name=external-secrets -n OPERATOR_NAMESPACE

  3. Verify that the Kubernetes ServiceAccount used by the operator is correctly annotated with the Google service account email. For more information, see Workload Identity Federation on the cluster.

  4. Verify that the Google service account has the necessary permissions and that Workload Identity Federation is correctly configured. For more information, see Workload Identity Federation on the cluster.

What's next