This tutorial shows developers and operators who deploy containers to Kubernetes how to use container image digests to identify container images. A container image digest uniquely and immutably identifies a container image.
Deploying containers images by using the image digest provides several benefits compared to using image tags. For more information on image digests, see the accompanying document on using container image digests before you continue this tutorial.
The image argument for containers in a Kubernetes Pod specification accepts
images with digests. This argument applies everywhere you use a Pod
specification, such as in the template section of Deployment, StatefulSet,
DaemonSet, ReplicaSet, CronJob, and Job resources.
To deploy an image by using the digest, you use the image name, followed by
@sha256: and the digest value. The following is an example of a Deployment
resource that uses an image with a digest. A Deployment is a Kubernetes API object that lets you run multiple replicas of Pods that are distributed among the nodes in a cluster..
apiVersion: apps/v1 kind: Deployment metadata: name: echo-deployment spec: selector: matchLabels: app: echo template: metadata: labels: app: echo spec: containers: - name: echoserver image: gcr.io/google-containers/echoserver@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229 ports: - containerPort: 8080
One downside of using image digests is that you don't know the digest value until after you have published your image to a registry. As you build new images, the digest value changes, and you need a way to update your Kubernetes manifests each time you deploy.
This tutorial shows how you can use tools such as
Skaffold,
kpt,
digester,
kustomize,
gke-deploy,
and
ko
to use image digests in your manifests.
Recommendations
This document presents several ways to use image digests in
Kubernetes deployments. The tools described in this document are complementary.
For example, you can use the output of a kpt function with kustomize to
create variants for different environments. Skaffold can
build images using ko
and deploy the images to your Kubernetes clusters using kubectl or kpt.
The reason the tools are complementary is that they perform structured edits based on the Kubernetes resource model (KRM). This model makes the tools pluggable, and you can evolve your use of the tools to create processes and pipelines that help you deploy your apps and services.
To get started, we recommend the approach that works best with your existing tools and processes:
Skaffold can add digests to image references. You enable this function with a small configuration change. Adopting Skaffold provides additional benefits, such as abstracting away how different tools build and deploy container images.
By using the digester tool as a mutating admission webhook in your Kubernetes clusters, you can add digests to all your deployments with minimal impact on your current processes for building and deployment container images. The digester webhook also simplifies adoption of Binary Authorization, because it only requires a label to be added to a namespace.
kpt is a great option if you need a flexible tool to manipulate Kubernetes manifests. The digester tool can be used as a client-side KRM function in a kpt pipeline.
If you already use kustomize to manage Kubernetes manifests across environments, we recommend that you take advantage of its image transformers to deploy images by digest.
kois a great way to build and publish images for Go apps, and it is used by open source projects such as Knative, Tekton, and sigstore.
If you don't use any of the tools described in this document, we recommend that you start with Skaffold and the digester webhook. Skaffold is a common tool used by both developers and release teams, and it integrates with the other tools described in this tutorial. You can take advantage of these integration options as your requirements evolve. The digester Kubernetes webhook complements Skaffold by enabling digest-based deployments for an entire cluster.
Objectives
- Use Skaffold to build and push an image, and to insert the image name and digest in a Kubernetes manifest.
- Use the digester client-side function and mutating admission webhook to add digests to images in Kubernetes Pods and Pod templates.
- Use kpt setters to replace an image tag in a Kubernetes manifest with an image digest.
- Use kustomize to generate a Kubernetes manifest with an image digest.
- Use
gke-deployto resolve an image tag to a digest in a Kubernetes manifest. - Use
koto build and push an image, and to insert the image name and digest in a Kubernetes manifest.
Costs
In this document, you use the following billable components of Google Cloud:
To generate a cost estimate based on your projected usage,
use the pricing calculator.
When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.
Before you begin
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
Roles required to select or create a project
- Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
-
Create a project: To create a project, you need the Project Creator
(
roles/resourcemanager.projectCreator), which contains theresourcemanager.projects.createpermission. Learn how to grant roles.
-
Verify that billing is enabled for your Google Cloud project.
-
Enable the Artifact Registry API.
Roles required to enable APIs
To enable APIs, you need the Service Usage Admin IAM role (
roles/serviceusage.serviceUsageAdmin), which contains theserviceusage.services.enablepermission. Learn how to grant roles. -
In the Google Cloud console, activate Cloud Shell.
In Cloud Shell, set the default project for Google Cloud CLI:
gcloud config set project PROJECT_IDReplace
PROJECT_IDwith your [project ID].Create a container image repository in Artifact Registry:
gcloud artifacts repositories create REPOSITORY \ --location=LOCATION \ --repository-format=dockerReplace the following:
REPOSITORY: the name you want to use for your repository, for instancedigest-tutorial.LOCATION: an Artifact Registry location, for instanceus-central1.
Configure authentication to the Artifact Registry location for the CLI tools used in this tutorial:
gcloud auth configure-docker LOCATION-docker.pkg.dev
Using Skaffold
Skaffold is a command-line tool for continuous development and deployment of applications to Kubernetes clusters.
Use Skaffold to build an image, push the image to Artifact Registry, and
replace the image placeholder value in a Kubernetes manifest template with the
name, tag, and digest of the pushed image:
In Cloud Shell, create and go to a directory to store the files that you create in this section:
mkdir -p ~/container-image-digests-tutorial/skaffold cd ~/container-image-digests-tutorial/skaffoldClone the Skaffold Git repository:
git clone https://github.com/GoogleContainerTools/skaffold.gitGo to the directory of the
getting-startedexample:cd skaffold/examples/getting-startedCheckout the Git tag that matches your version of Skaffold:
git checkout $(skaffold version)View the
skaffold.yamlconfiguration file:cat skaffold.yamlThe file resembles the following:
apiVersion: skaffold/v4beta6 kind: Config build: artifacts: - image: skaffold-example manifests: rawYaml: - k8s-pod.yaml
The
build.artifactssection contains a placeholder image name. Skaffold looks for this placeholder in the input manifest files.The
manifestssection tells Skaffold to read an input manifest from the current directory with the namek8s-pod.yaml.For an overview of all available options, see the
skaffold.yamlreference documentation.View the Kubernetes manifest template:
cat k8s-pod.yamlThe file is the following:
apiVersion: v1 kind: Pod metadata: name: getting-started spec: containers: - name: getting-started image: skaffold-example
The
skaffold-exampleplaceholder value in theimagefield matches the value of theimagefield in theskaffold.yamlfile. Skaffold replaces this placeholder value with the full image name and digest in the rendered output.Build and push the image to Artifact Registry:
skaffold build \ --default-repo=LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY \ --file-output=artifacts.json \ --interactive=false \ --push=true \ --update-check=falseThis command uses the following flags:
- The
--file-outputflag specifies the file where Skaffold saves information about the built image, including the digest value. - The
--pushflag instructs Skaffold to push the built image to the container image registry specified by the--default-repoflag. - The
--interactiveand--update-checkflags are both set tofalse. Set these flags tofalsein non-interactive environments, such as build pipelines, but leave them as their default values (truefor both flags) for local development.
If you use Cloud Deploy to deploy to GKE, use the file from the
--file-outputflag as the value of the--build-artifactsflag when you create a release.- The
Render the expanded Kubernetes manifest with the name, tag, and digest of the container image from the previous step:
skaffold render \ --build-artifacts=artifacts.json \ --digest-source=none \ --interactive=false \ --offline=true \ --output=rendered.yaml \ --update-check=falseThis command uses the following flags:
- The
--build-artifactsflag references the output file from theskaffold buildcommand in the previous step. - The
--digest-source=noneflag means that Skaffold uses the digest value from the file provided in the--build-artifactsflag, instead of resolving the digest from the container image registry. - The
--offline=trueflag means that you can run the command without requiring access to a Kubernetes cluster. - The
--outputflag specifies the output file for the rendered manifest.
- The
View the rendered manifest:
cat rendered.yamlThe output resembles the following:
apiVersion: v1 kind: Pod metadata: name: getting-started spec: containers: - image: LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORY/skaffold-example:TAG@sha256:DIGEST name: getting-started
In this output, you see the following values:
TAG: the tag that Skaffold assigned to the image.DIGEST: the image digest value
Using digester
Digester adds digests to container and init container images in Kubernetes Pod and Pod template specifications. Digester replaces container image references that use tags:
spec:
containers:
- image: gcr.io/google-containers/echoserver:1.10
With references that use the image digest:
spec:
containers:
- image: gcr.io/google-containers/echoserver:1.10@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229
Digester can run either as a mutating admission webhook in a Kubernetes cluster, or as a client-side KRM function with the kpt or kustomize command-line tools.
Using the digester KRM function
In Cloud Shell, create and go to a directory to store the files that you create in this section:
mkdir -p ~/container-image-digests-tutorial/digester-fn cd ~/container-image-digests-tutorial/digester-fnDownload the digester binary:
mkdir -p ${HOME}/bin export PATH=${HOME}/bin:${PATH} DIGESTER_VERSION=$(curl -sL https://api.github.com/repos/google/k8s-digester/releases/latest | jq -r .tag_name) curl -L "https://github.com/google/k8s-digester/releases/download/${DIGESTER_VERSION}/digester_$(uname -s)_$(uname -m)" --output ${HOME}/bin/digester chmod +x ${HOME}/bin/digesterCreate a Kubernetes Pod manifest that references the image
gcr.io/google-containers/echoserverusing the tag1.10:cat << EOF > pod.yaml apiVersion: v1 kind: Pod metadata: name: echo spec: containers: - name: echoserver image: gcr.io/google-containers/echoserver:1.10 ports: - containerPort: 8080 EOFRun the digester KRM function using kpt with the manifests in the current directory (
.):kpt fn eval . --exec digesterWhen you run this command, kpt performs an in-place update of the manifests in the current directory. If you want kpt to show the updated manifest on the console and leave the manifest file unchanged, add the
--output unwrapflag.View the updated manifest:
cat pod.yamlThe file is the following:
apiVersion: v1 kind: Pod metadata: name: echo spec: containers: - name: echoserver image: gcr.io/google-containers/echoserver:1.10@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229 ports: - containerPort: 8080
Using the digester admission webhook
In Cloud Shell, create and go to a directory to store the files that you create in this section:
mkdir -p ~/container-image-digests-tutorial/digester-webhook cd ~/container-image-digests-tutorial/digester-webhookCreate a local Kubernetes cluster using kind:
kind create clusterkind is a command-line tool to run local Kubernetes clusters using Docker.
Deploy the digester webhook:
DIGESTER_VERSION=$(curl -sL https://api.github.com/repos/google/k8s-digester/releases/latest | jq -r .tag_name) kustomize build "https://github.com/google/k8s-digester.git/manifests?ref=${DIGESTER_VERSION}" | kubectl apply -f -Create a Kubernetes namespace called
digester-demoin the kind cluster:kubectl create namespace digester-demoAdd the
digest-resolution: enabledlabel to thedigester-demonamespace:kubectl label namespace digester-demo digest-resolution=enabledThe digester webhook adds digests to Pods in namespaces with this label.
Create a Kubernetes Deployment manifest that references the image
gcr.io/google-containers/echoserverusing the tag1.10:cat << EOF > deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: echo-deployment spec: selector: matchLabels: app: echo template: metadata: labels: app: echo spec: containers: - name: echoserver image: gcr.io/google-containers/echoserver:1.10 ports: - containerPort: 8080 EOFApply the manifest in the
digester-demonamespace:kubectl apply --filename deployment.yaml --namespace digester-demo \ --output jsonpath='{.spec.template.spec.containers[].image}{"\n"}'The
--outputflag instructskubectlto output the image name to the console, followed by a newline character. The output is the following:gcr.io/google-containers/echoserver:1.10@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229This output shows that the digester webhook added the image digest to the Pod template specification in the Deployment resource.
Delete the kind cluster to free up resources in your Cloud Shell session:
kind delete cluster
Using kpt setters
kpt is a command-line tool to manage, manipulate, customize, and apply Kubernetes resource manifests.
You can use the create-setters and apply-setters KRM functions from the
kpt Functions Catalog
to update image digests in your Kubernetes manifests when you build new
images.
In Cloud Shell, create and go to a directory to store the files that you create in this section:
mkdir -p ~/container-image-digests-tutorial/kpt cd ~/container-image-digests-tutorial/kptCreate a kpt package in the current directory:
kpt pkg init --description "Container image digest tutorial"Create a Kubernetes Pod manifest that references the image
gcr.io/google-containers/echoserverusing the tag1.10:cat << EOF > pod.yaml apiVersion: v1 kind: Pod metadata: name: echo spec: containers: - name: echoserver image: gcr.io/google-containers/echoserver:1.10 ports: - containerPort: 8080 EOFUse kpt to create a setter called
echoimagefor the manifest field, where the existing value isgcr.io/google-containers/echoserver:1.10:kpt fn eval . \ --image gcr.io/kpt-fn/create-setters@sha256:0220cc87f29ff9abfa3a3b5643aa50f18d355d5e9dc9e1f518119633ddc4895c \ -- "echoimage=gcr.io/google-containers/echoserver:1.10"View the manifest:
cat pod.yamlThe file is the following:
apiVersion: v1 kind: Pod metadata: name: echo spec: containers: - name: echoserver image: gcr.io/google-containers/echoserver:1.10 # kpt-set: ${echoimage} ports: - containerPort: 8080
Get the digest value of the container image:
DIGEST=$(gcloud container images describe \ gcr.io/google-containers/echoserver:1.10 \ --format='value(image_summary.digest)')Set the new field value:
kpt fn eval . \ --image gcr.io/kpt-fn/apply-setters@sha256:4d4295727183396f0c3c6a75d2560254c2f9041a39e95dc1e5beffeb49cc1a12 \ -- "echoimage=gcr.io/google-containers/echoserver:1.10@$DIGEST"When you run this command, kpt performs an in-place replacement of the
imagefield value in the manifest.View the updated manifest:
cat pod.yamlThe file is the following:
apiVersion: v1 kind: Pod metadata: name: echo spec: containers: - name: echoserver image: gcr.io/google-containers/echoserver:1.10@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229 # kpt-set: ${echoimage} ports: - containerPort: 8080
Using kustomize image transformers
kustomize is a command-line tool that lets you customize Kubernetes manifests by using overlays, patches, and transformers.
You can use the kustomize image transformer to update the image name, tag, and digest in your existing manifest.
The following kustomization.yaml snippet shows you how to configure the image
transformer to use the transformer digest value for images where the Pod
specification image value matches the transformer name value:
images: - name: gcr.io/google-containers/echoserver digest: sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229
To use a kustomize image transformer with an image digest, do the following:
In Cloud Shell, create and go to a directory to store the files that you create in this section:
mkdir -p ~/container-image-digests-tutorial/kustomize cd ~/container-image-digests-tutorial/kustomizeCreate a
kustomization.yamlfile:kustomize initCreate a Kubernetes manifest with a Pod specification that references the image
gcr.io/google-containers/echoserverusing the tag1.10:cat << EOF > pod.yaml apiVersion: v1 kind: Pod metadata: name: echo spec: containers: - name: echoserver image: gcr.io/google-containers/echoserver:1.10 ports: - containerPort: 8080 EOFAdd the manifest as a resource in the
kustomization.yamlfile:kustomize edit add resource pod.yamlUse an image transformer to update the digest of the image:
kustomize edit set image \ gcr.io/google-containers/echoserver@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229View the image transformer in the
kustomization.yamlfile:cat kustomization.yamlThe file is the following:
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - pod.yaml images: - digest: sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229 name: gcr.io/google-containers/echoserver
View the resulting manifest:
kustomize build .The output is the following:
apiVersion: v1 kind: Pod metadata: name: echo spec: containers: - image: gcr.io/google-containers/echoserver@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229 name: echoserver ports: - containerPort: 8080
To run the kustomize transformer and apply the resulting manifest to a Kubernetes cluster in one step, you can use the
kubectl applycommand with the--kustomizeflag:kubectl apply --kustomize .If you want to apply the output later, you can redirect the output of the
kustomize buildcommand to a file.
Using gke-deploy
gke-deploy
is a command-line tool that you use with
Google Kubernetes Engine (GKE).
gke-deploy wraps the kubectl command-line tool and can modify the
resources that you create following Google's recommended practices.
If you use the gke-deploy sub-commands prepare or run, gke-deploy
resolves your image tags to digests and saves the expanded manifests with the
image digests in the file output/expanded/aggregated-resources.yaml by
default.
You can use gke-deploy run to both substitute the image tag for a digest and
apply the expanded manifest to your GKE cluster. Although
this command is convenient, there is a downside: the image tag is substituted at
deployment time. The image associated with the tag might have changed in the
time between when you decided to deploy, and when you deployed, resulting in
deploying an unexpected image. For production deployments, we recommend separate
steps for generating and applying manifests.
To replace an image tag in a Kubernetes deployment manifest with the image digest, do the following:
In Cloud Shell, create and go to a directory to store the files that you create in this section:
mkdir -p ~/container-image-digests-tutorial/gke-deploy cd ~/container-image-digests-tutorial/gke-deployInstall
gke-deploy:go install github.com/GoogleCloudPlatform/cloud-builders/gke-deploy@latestCreate a Kubernetes Deployment manifest that references the image
gcr.io/google-containers/echoserverusing the tag1.10:cat << EOF > deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: echo-deployment spec: selector: matchLabels: app: echo template: metadata: labels: app: echo spec: containers: - name: echoserver image: gcr.io/google-containers/echoserver:1.10 ports: - containerPort: 8080 EOFGenerate an expanded manifest based on the
deployment.yamlmanifest:gke-deploy prepare \ --filename deployment.yaml \ --image gcr.io/google-containers/echoserver:1.10 \ --version 1.10View the expanded manifest:
cat output/expanded/aggregated-resources.yamlThe output is the following:
apiVersion: apps/v1 kind: Deployment metadata: labels: app.kubernetes.io/managed-by: gcp-cloud-build-deploy app.kubernetes.io/version: "1.10" name: echo-deployment namespace: default spec: selector: matchLabels: app: echo template: metadata: labels: app: echo app.kubernetes.io/managed-by: gcp-cloud-build-deploy app.kubernetes.io/version: "1.10" spec: containers: - image: gcr.io/google-containers/echoserver@sha256:cb5c1bddd1b5665e1867a7fa1b5fa843a47ee433bbb75d4293888b71def53229 name: echoserver ports: - containerPort: 8080
In the expanded manifest, the image tag is replaced by the digest.
The
--versionargument you used with thegke-deploycommand sets the value of theapp.kubernetes.io/versionlabel in the deployment and the Pod template metadata of the expanded manifest.To learn how to use
gke-deploywith Cloud Build, see the Cloud Build documentation forgke-deploy.
Using ko
ko
is a command-line tool and library for building
Go
container images and deploying them to Kubernetes clusters. ko builds images
without using the Docker daemon, so you can use it in environments where you
can't install Docker.
The ko sub-command
build
builds images and publishes them to a container image registry or loads them
into your local Docker daemon.
The ko sub-command
resolve
does the following:
- Identifies the images to build by finding placeholders in the
imagefields of the Kubernetes manifests that you provide by using the--filenameargument. - Builds and publishes your images.
- Replaces the
imagevalue placeholders with the names and digests of the images it built. - Prints the expanded manifests.
The ko sub-commands
apply,
create,
and
run
perform the same steps as resolve, and then execute kubectl apply,
create, or run with the expanded manifests.
To build an image from Go source code, and add the digest of the image to a Kubernetes deployment manifest, do the following
In Cloud Shell, create and go to a directory to store the files that you create in this section:
mkdir -p ~/container-image-digests-tutorial/ko cd ~/container-image-digests-tutorial/koDownload
koand add it to yourPATH:mkdir -p ${HOME}/bin export PATH=${HOME}/bin:${PATH} KO_VERSION=$(curl -sL https://api.github.com/repos/ko-build/ko/releases/latest | jq -r .tag_name | cut -c2-) curl -L "https://github.com/ko-build/ko/releases/download/v${KO_VERSION}/ko_${KO_VERSION}_$(uname -s)_$(uname -m).tar.gz" | tar -zxC ${HOME}/bin koCreate a Go app with the module name
example.com/hello-worldin a new directory calledapp:mkdir -p app/cmd/ko-example cd app go mod init example.com/hello-world cat << EOF > cmd/ko-example/main.go package main import "fmt" func main() { fmt.Println("hello world") } EOFDefine the image repository that
kouses to publish images:export KO_DOCKER_REPO=LOCATION-docker.pkg.dev/PROJECT_ID/REPOSITORYThis example uses Artifact Registry, but you can use
kowith a different container image registry.To build and publish an image for your app, do one of the following steps:
Build and publish an image for your app by providing the path to your Go main package:
ko build --base-import-paths ./cmd/ko-exampleThe optional argument
--base-import-pathsmeans thatkouses the short name of the main package directory as the image name.koprints the image name and digest tostdoutin the following format:LOCATION-docker.pkg.dev/PROJECT_ID/ko-example@sha256:DIGESTIn this output,
DIGESTis the image digest value.Use
koto replace a manifest placeholder with the name and digest of the image it builds and publishes:Create a Kubernetes Pod manifest. The manifest uses the placeholder
ko://IMPORT_PATH_OF_YOUR_MAIN_PACKAGEas the value of theimagefield:cat << EOF > ko-pod.yaml apiVersion: v1 kind: Pod metadata: name: ko-example spec: containers: - name: hello-world image: ko://example.com/hello-world/cmd/ko-example EOFBuild and publish an image for your app, and replace the manifest placeholder with the image name and digest:
ko resolve --base-import-paths --filename ko-pod.yamlkoprints the manifest with the image name and digest tostdout:apiVersion: v1 kind: Pod metadata: name: ko-example spec: containers: - name: hello-world image: LOCATION-docker.pkg.dev/PROJECT_ID/ko-example@sha256:DIGEST
In this output,
DIGESTis the image digest value.
Clean up
The easiest way to eliminate billing is to delete the Google Cloud project that you created for the tutorial. Alternatively, you can delete the individual resources.
Delete the project
- In the Google Cloud console, go to the Manage resources page.
- In the project list, select the project that you want to delete, and then click Delete.
- In the dialog, type the project ID, and then click Shut down to delete the project.
Delete the resources
If you want to keep the Google Cloud project that you used in this tutorial, delete the individual resources:
In Cloud Shell, delete the files you created in this tutorial:
cd rm -rf ~/container-image-digests-tutorialDelete the container image repository in Artifact Registry:
gcloud artifacts repositories delete REPOSITORY \ --location=LOCATION --async --quiet
What's next
- Learn more about container image digests.
- Learn more about the digester client-side KRM function and Kubernetes mutating webhook.
- Explore GitOps-style continuous delivery with Cloud Build.