Offer Terraform Kubernetes apps through Cloud Marketplace

Terraform Kubernetes apps in Google Cloud Marketplace let your customers use Terraform to deploy your containerized apps on Google Kubernetes Engine. You can offer Terraform Kubernetes apps publicly to all Cloud Marketplace users, or privately by using Private Offers, and you can offer them for free, or charge for usage based on metrics that you select. If your app is free, your users are only charged for their usage of Google Cloud resources.

Terraform Kubernetes apps that you offer through Cloud Marketplace define their configurations by using Helm charts, which you host in Artifact Registry. After that's set up, you onboard your app to Cloud Marketplace through Producer Portal.

Overview of onboarding a Terraform Kubernetes app

Offering a Terraform Kubernetes app through Cloud Marketplace involves the following steps:

  1. Prepare your Terraform Kubernetes app for Cloud Marketplace. This includes setting up Artifact Registry and pushing your app's images to it.

  2. Add pricing information for your app in Producer Portal.

  3. Configure your app's deployment in Producer Portal, including its Terraform module, Helm chart, and container images.

  4. Publish your product to Cloud Marketplace. As you onboard your product, you submit various parts of your product for review. The Cloud Marketplace team reviews each submission and works with you to resolve any issues before approving it. When all of your submissions are approved, you can publish and launch the product within a few minutes.

Requirements for Terraform Kubernetes apps

All products that you offer through Cloud Marketplace must comply with the Cloud Marketplace listing requirements.

Your Terraform Kubernetes app must meet the following additional requirements:

  • Your app must run on GKE nodes that use x86 processors.
  • All of your app's Helm charts and container images must be uploaded to your Artifact Registry registry, to the same repository and project that you're using for your Terraform Kubernetes app.
  • Your Artifact Registry registry must include a Helm chart.
  • Your Terraform module must be hosted as a ZIP file in a Cloud Storage bucket located in the same project that you're using for your Terraform Kubernetes app.
  • You must enable versioning for the Cloud Storage bucket that you use to host your Terraform module.
  • You must include a user guide, with steps to deploy your app from the command line, configure the app, and use the app. For details, see Requirements for your user guide in this document.
  • To protect users from unstable APIs, use only beta or generally available Kubernetes resources.

Requirements for your app images

Your app images must meet the following requirements:

  • All of your app's images must be tagged with the release track and the current version. For example, if you're releasing version 2.0.5 on the 2.0 release track, all corresponding images must be tagged with 2.0 and 2.0.5. For details, see Organize your releases.

  • All of your app's images must contain the following annotation in their image manifest:

    com.googleapis.cloudmarketplace.product.service.name=services/SERVICE_NAME

    Replace SERVICE_NAME with the name of your service. To find your service name, refer to the table of products on the Overview page in Producer Portal. For more information about annotations, see the Open Container Initiative's documentation of annotations on GitHub.

Requirements for your Artifact Registry repository

Your app's Helm chart and container images must be part of a single Artifact Registry repository.

For example, if your Artifact Registry repository is us-docker.pkg.dev/exampleproject, your app's Helm chart and container images should resemble the following:

  • us-docker.pkg.dev/exampleproject/exampleapp/chart
  • us-docker.pkg.dev/exampleproject/exampleapp/image1
  • us-docker.pkg.dev/exampleproject/exampleapp/image2

Requirements for the Terraform module

Customers use the Terraform module to deploy your product from Cloud Marketplace through Infrastructure Manager. The Terraform module packages your app's Kubernetes configuration.

Use the example module as a starting point, and follow the partner guide to build a Terraform module that's compatible with your app.

Parameters passed to your app

Your Terraform module must declare parameters that need to be collected from customers when they select your app. These parameters are then supplied to the Helm chart container when users deploy the app.

To configure these parameters, your Terraform module must include a schema.yaml file.

To learn how to create a schema.yaml, see the guide to creating a schema on GitHub.

GPU cluster requests

If your app has specific GPU needs, or is GPU intensive, you can specify the type and number of GPUs in the cluster by using your deployer schema. If you specify your GPU needs, you turn off assisted cluster creation.

Your app may request a generic Nvidia GPU, or a specific Nvidia platform.

Requirements for your app's user guide

Your app's user guide must include the following information:

  • Overview: A general overview of your app, covering basic functions and configuration options. This section must also link to your published app on Cloud Marketplace.
  • One-time setup: This includes steps for configuring Helm and installing the Application CustomResourceDefinition (CRD) so that your cluster can manage the Application resource.
  • Installation: This includes the following:
  • Basic Usage: Guides for the following tasks:
    • Modifying usernames and passwords
    • Connecting to an admin console, if applicable
    • Connecting a client tool and running a sample command, if applicable
    • Enabling ingress and installing TLS certs, if applicable
  • Back up and restore: Information about backing up the app state and restoring the app state from a backup.
  • Image updates: Information about updating the app images for patches or minor updates.
  • Scaling: Information about scaling the app, if applicable.
  • Deletion: Guides for deleting the app and cleaning up any resources that might remain, such as PersistentVolumeClaims.

Pricing models for Terraform Kubernetes apps

You can choose one of the following pricing models for your app:

  • Free: Customers only pay for the Google Cloud resources that they use.
  • Usage-based: Customers pay for your app based on measurements that you choose, called metrics, such as the number of containers or storage time. If you choose a usage-based pricing model, your app must measure and report usage information to Google.

After you've published a Terraform Kubernetes app with usage-based pricing, you can update your app's pricing. To update your app's pricing, it must be at least 30 days since you published your app or last updated its pricing.

Usage-based pricing

Google Cloud Marketplace uses Kubernetes pod annotations to track usage and charge customers for their deployments of your Cloud Marketplace apps. Cloud Marketplace tracks billing and metering at the pod level, and aggregates them at the product listing level.

If you deploy multiple pods on the same machine (vCPU, GPU, TPU, or memory), the system assigns the cumulative usage occurring across all of the pods deployed on the machine to each of the pods.

Cloud Marketplace treats different GKE SKUs within each vCPU, TPU, GPU, or memory family of SKUs as identical. For example, if GPU is set up as a metric, the configured price applies regardless of whether your customer uses GKE Autopilot T4 GPU Premium, GKE Autopilot L4GKE Autopilot H100 GPU Premium, GKE Autopilot A100 40 GiB GPU, GKE Autopilot A100 80 GiB GPU, or GKE Autopilot H100 GPU Premium.

We strongly recommend that you integrate business logic to ensure accurate customer billing into your Cloud Marketplace apps. To mitigate the impact of accidental removal of label annotations, we recommend that you design your app to periodically confirm that the label annotations are still present. If an annotation is missing, your app should limit its functionality until the label has been reinstated. For an example of this in Python, see the following example:

pod_name = os.environ.get("HOSTNAME")
namespace = open(
      "/var/run/secrets/kubernetes.io/serviceaccount/namespace"
).read()
# Load in-cluster config
config.load_incluster_config()
v1 = client.CoreV1Api()
# Get the Pod object
pod = v1.read_namespaced_pod(name=pod_name, namespace=namespace)
annotations = pod.metadata.annotations
if annotations:
print("Annotations for this pod:")
for key, value in annotations.items():
      print(f"  {key}: {value}")
else:
print("No annotations found on this pod.")

What's next

Next
Understand how customers see your deployment packages