# AI/ML orchestration on GKE documentation


[Google Kubernetes Engine (GKE)](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/kubernetes-engine-overview.md.txt)
provides a single, unified platform to orchestrate your entire AI/ML lifecycle. It gives you the
power and flexibility to supercharge your training, inference, and agentic workloads, so you can
streamline your infrastructure and start delivering results. GKE's state-of-the-art
orchestration capabilities provide the following:

- **Hardware accelerators**: access and manage the high-powered GPUs and TPUs you need, for both training and inference, at scale.
- **Stack flexibility**: integrate with the distributed computing, data processing, and model serving frameworks you already know and trust.
- **Managed Kubernetes simplicity**: get all the benefits of a managed platform to automate, scale, and enhance the security of your entire AI/ML lifecycle while maintaining flexibility.

<br />

Explore our blogs, tutorials, and best practices to see how GKE can optimize your AI/ML workloads.
For more information about benefits and available features, see the [Introduction to AI/ML workloads on GKE overview](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/machine-learning.md.txt).

## Documentation resources

Find quickstarts and guides, review key references, and get help with common issues.

### Manage AI infrastructure and accelerators

- Concept

  [About accelerator consumption options for AI/ML workloads in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/consumption-option.md.txt)

- Concept

  [About GPUs in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/gpus.md.txt)

- Concept

  [About TPUs in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/tpus.md.txt)

- Best practice

  [About GPU, TPU, and H4D consumption with flex-start provisioning mode](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dws.md.txt)

- Best practice

  [Run cost-optimized Kubernetes applications on GKE](https://docs.cloud.google.com/architecture/best-practices-for-running-cost-effective-kubernetes-applications-on-gke.md.txt)

- Best practice

  [Design storage for AI and ML workloads in Google Cloud](https://docs.cloud.google.com/architecture/ai-ml/storage-for-ai-ml)

- Quickstart

  [Deploy GPU-accelerated Ray for AI workloads on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/add-on/ray-on-gke/quickstarts/ray-gpu-cluster.md.txt)

- Video

  [Introduction to Cloud TPUs for machine learning](https://www.youtube.com/watch?v=cGelTHeLsN4)

- Video

  [Build large-scale machine learning on Cloud TPUs with GKE](https://youtu.be/wtKhG1aTgtY?si=Vk8Y-Hql-l9YPrTD)

### Serve training and inference workloads

- Quickstart

  [Train a model with GPUs on GKE Standard mode](https://docs.cloud.google.com/kubernetes-engine/docs/quickstarts/train-model-gpus-standard.md.txt)

- Tutorial

  [Train a model with GPUs on GKE Autopilot mode](https://docs.cloud.google.com/kubernetes-engine/docs/quickstarts/train-model-gpus-autopilot.md.txt)

- How-to

  [Train large-scale machine learning models on GKE with multi-tier checkpointing](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/training/multi-tier-checkpointing.md.txt)

- Tutorial

  [Optimize GKE resource utilization for mixed AI/ML training and inference workloads](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/mixed-workloads.md.txt)

- Best practice

  [Reference architecture for AI/ML inference on GKE](https://github.com/GoogleCloudPlatform/accelerated-platforms/blob/main/docs/platforms/gke/base/use-cases/inference-ref-arch/README.md)

- Concept

  [About model inference on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/machine-learning/inference.md.txt)

- Best practice

  [Best practices for batch inference on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/best-practices/machine-learning/inference/batch-inference.md.txt)

- How-to

  [Analyze model serving performance and costs with GKE Inference Quickstart](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/inference-quickstart.md.txt)

- Tutorial

  [Discover more tutorials for model inference on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/machine-learning/inference.md.txt#examples)

### Serve agentic workloads

- Concept

  [Plan an architecture for AI agent workloads on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/machine-learning/agent-workloads.md.txt)

- Concept

  [About Agent Substrate on GKE](https://docs.cloud.google.com/kubernetes-engine/ai-ml/about-agent-substrate.md.txt)

- How-to

  [Install Agent Substrate on GKE](https://docs.cloud.google.com/kubernetes-engine/ai-ml/install-overview-substrate.md.txt)

- Best practice

  [Design your infrastructure for Agent Substrate on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/design-substrate-infrastructure.md.txt)

- How-to

  [Install Agent Sandbox on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/how-install-agent-sandbox.md.txt)

- Tutorial

  [Isolate AI code execution with Agent Sandbox on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/agent-sandbox.md.txt)

- Concept

  [Choose storage for AI agentic workloads](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/machine-learning/agent-sandbox-storage.md.txt)

- How-to

  [Manage Agent Sandbox on GKE storage](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/agent-sandbox-storage.md.txt)

- How-to

  [Save and restore Agent Sandbox on GKE environments](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/agent-sandbox-pod-snapshots.md.txt)

- Concept

  [About Agent Identity for GKE](https://docs.cloud.google.com/kubernetes-engine/security/about-agent-identity.md.txt)

## Related resources

Training and tutorials Use cases Code samples Explore self-paced training, use cases, reference architectures, and code samples with examples of how to use and connect Google Cloud services. Reference architecture Training and tutorials

### [Optimize AI and ML workloads with Cloud Storage and GKE](https://docs.cloud.google.com/architecture/optimize-ai-ml-workloads-cloud-storage-fuse.md.txt)


Learn how to use Cloud Storage FUSE to optimize performance for AI and ML workloads on GKE.

AI/ML Inference AI/ML Training Storage

<br />

Reference architecture Training and tutorials

### [Optimize AI and ML workloads with Managed Lustre and GKE](https://docs.cloud.google.com/architecture/optimize-ai-ml-workloads-managed-lustre.md.txt)


Learn how to use Managed Lustre to optimize performance for AI and ML workloads on GKE.

AI/ML Inference AI/ML Training Storage

<br />

Tutorial Training and tutorials

### [Use open source Kata Containers with Agent Sandbox on GKE](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/examples/kata-gke-sandbox)


Learn how to use open source Kata Containers with Agent Sandbox on GKE to isolate untrusted code execution.

Training Tutorial Agentic AI Agent Sandbox

<br />

Tutorial Training and tutorials

### [Deploy an agentic AI application on GKE with the Agent Development Kit (ADK) and a self-hosted LLM](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/agentic-adk-vllm.md.txt)


Learn how to deploy and manage a containerized agentic AI application on GKE, using the Agent Development Kit (ADK) and vLLM for scalable inference with Llama 3.1.

Tutorial AI/ML Inference Agentic AI

<br />

Tutorial Training and tutorials

### [Deploy an agentic AI application on GKE with the Agent Development Kit (ADK) and Agent Platform](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/agentic-adk-vertex.md.txt)


Learn how to deploy and manage a containerized agentic AI application on GKE, using the Agent Development Kit (ADK) and Agent Platform for scalable inference with Gemini 2.0 Flash.

Tutorial AI/ML Inference Agentic AI

<br />

Tutorial Training and tutorials

### [Train a model with PyTorch, Ray, and GKE on GPUs](https://docs.cloud.google.com/kubernetes-engine/docs/add-on/ray-on-gke/tutorials/train-model-ray-pytorch.md.txt)


Learn how to use GKE to train a model with Ray and PyTorch on GPUs.

Tutorial AI/ML Training Ray GPUs

<br />

Tutorial Training and tutorials

### [Train an LLM using JAX, Ray Train, and TPU Trillium on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/distributed-training-tpu.md.txt)


Learn how to train an LLM on GKE using JAX, MaxText, Ray Train, and TPUs.

Tutorial AI/ML Training Ray TPUs

<br />

Tutorial Training and tutorials

### [Multislice and Elastic Training on TPUs using Ray Train on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/multi-slice-training-ray.md.txt)


Learn how to run Multislice and elastic training workloads on TPUs using Ray Train on GKE.

Tutorial AI/ML Training Ray TPUs

<br />

Quickstart Training and tutorials

### [Serve open LLMs on GKE with a pre-configured architecture using Terraform](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/serve-open-models-terraform.md.txt)


Learn how to deploy and serve open LLMs on GKE for inference using a pre-configured architecture with Terraform.

Quickstart AI/ML Inference GPUs

<br />

Tutorial Training and tutorials

### [Serve LLMs like Deepseek-R1 671B or Llama 3.1 405B on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/serve-multihost-gpu.md.txt)


Learn how to serve LLMs, such as DeepSeek-R1 671B and Llama 3.1 405B, on multi-host GPUs in GKE using vLLM and LeaderWorkerSet (LWS).

Tutorial AI/ML Inference GPUs

<br />

Tutorial Training and tutorials

### [Serve Gemma using GPUs on GKE with vLLM](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/serve-gemma-gpu-vllm.md.txt)


Learn how to serve Gemma open models using GPUs on GKE with vLLM.

Tutorial AI/ML Inference GPUs

<br />

Tutorial Training and tutorials

### [Serve an LLM using TPU Trillium on GKE with vLLM](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/serve-vllm-tpu.md.txt)


Learn how to serve an LLM using TPU Trillium on GKE with vLLM.

Tutorial AI/ML Inference TPUs

<br />

Tutorial Training and tutorials

### [Serve an LLM with multi-cluster Ray Serve and Inference Gateway](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/serve-multi-cluster-ray-inference-gateway.md.txt)


Learn how to configure multi-cluster model serving with Ray Serve and GKE Inference Gateway.

Tutorial AI/ML Inference Ray

<br />

Tutorial Training and tutorials

### [Serve open source models using TPUs on GKE with Optimum TPU](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/optimum-tpu.md.txt)


Learn how to deploy LLMs using Tensor Processing Units (TPUs) on GKE with the Optimum TPU serving framework from Hugging Face.

Tutorial AI/ML Inference TPU

<br />

Tutorial Training and tutorials

### [Serve LLMs on GKE with a cost-optimized and high-availability GPU provisioning strategy](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/dws-flex-start-inference.md.txt)


Learn how to optimize costs for LLM-serving workloads on GKE using DWS Flex-start.

Cost optimization GPU DWS

<br />

Training Training and tutorials

### [Serving Large Language Models with KubeRay on TPUs](https://www.youtube.com/watch?v=RK_u6cfPnnw)


Learn how to serve large language models (LLMs) with KubeRay on TPUs, and how this can help improve the performance of your models.

Video Ray TPUs

<br />

Training Training and tutorials

### [Accelerate AI/ML data loading with Hyperdisk ML](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/hyperdisk-ml.md.txt)


Learn how to how to simplify and accelerate the loading of AI/ML model weights on GKE using Hyperdisk ML.

Tutorial AI/ML Data Loading

<br />

Training Training and tutorials

### [Best practices for optimizing LLM inference with GPUs on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/best-practices/machine-learning/inference/llm-optimization.md.txt)


Learn best practices for optimizing LLM inference performance with GPUs on GKE using the vLLM and Text Generation Inference (TGI) serving frameworks.

Tutorial AI/ML Inference GPUs

<br />

Training Training and tutorials

### [Manage the GPU Stack with the NVIDIA GPU Operator on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/gpu-operator.md.txt)


Learn when to use the NVIDIA GPU operator and how to enable the NVIDIA GPU Operator on GKE.

Tutorial GPUs

<br />

Training Training and tutorials

### [Configure autoscaling for LLM workloads on TPUs](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/autoscaling-tpu.md.txt)


Learn how to set up your autoscaling infrastructure by using the GKE Horizontal Pod Autoscaler (HPA) to deploy the Gemma LLM using single-host JetStream.

Tutorial TPUs

<br />

Training Training and tutorials

### [Deploy a Ray Serve application with a Stable Diffusion model on GKE with TPUs](https://docs.cloud.google.com/kubernetes-engine/docs/add-on/ray-on-gke/tutorials/deploy-ray-serve-stable-diffusion-tpu.md.txt)


Learn how to deploy and serve a Stable Diffusion model on GKE using TPUs, Ray Serve, and the Ray Operator add-on.

Tutorial AI/ML Inference Ray TPUs

<br />

Training Training and tutorials

### [Configure autoscaling for LLM workloads on GPUs with GKE](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/machine-learning/inference/autoscaling.md.txt)


Learn how to set up your autoscaling infrastructure by using the GKE Horizontal Pod Autoscaler (HPA) to deploy the Gemma LLM with the Hugging Face Text Generation Interface (TGI) serving framework.

Tutorial GPUs

<br />

Training Training and tutorials

### [Train Llama2 with Megatron-LM on A3 Mega virtual machines](https://docs.cloud.google.com/kubernetes-engine/docs/quickstarts/training-megatron-llama-workload.md.txt)


Learn how to run a container-based, Megatron-LM PyTorch workload on A3 Mega.

Tutorial AI/ML Training GPUs

<br />

Training Training and tutorials

### [Deploy GPU workloads in Autopilot](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/autopilot-gpus.md.txt)


Learn how to request hardware accelerators (GPUs) in your GKE Autopilot workloads.

Tutorial GPUs

<br />

Training Training and tutorials

### [Serve an LLM with multiple GPUs in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/serve-multiple-gpu.md.txt)


Learn how to serve Llama 2 70B or Falcon 40B using multiple NVIDIA L4 GPUs with GKE.

Tutorial AI/ML Inference GPUs

<br />

Training Training and tutorials

### [Getting started with Ray on GKE](https://cloud.google.com/blog/products/containers-kubernetes/use-ray-on-kubernetes-with-kuberay)


Learn how to easily start using Ray on GKE by running a workload on a Ray cluster.

Tutorial Ray

<br />

Training Training and tutorials

### [Serve an LLM on L4 GPUs with Ray](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/serve-llm-l4-ray.md.txt)


Learn how to serve Falcon 7b, Llama2 7b, Falcon 40b, or Llama2 70b using the Ray framework in GKE.

Tutorial AI/ML Inference Ray GPUs

<br />

Training Training and tutorials

### [Orchestrate TPU Multislice workloads using JobSet and Kueue](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/tpu-multislice-kueue.md.txt)


Learn how to orchestrate a Jax workload on multiple TPU slices on GKE by using JobSet and Kueue.

Tutorial TPUs

<br />

Training Training and tutorials

### [Monitoring GPU workloads on GKE with NVIDIA Data Center GPU Manager (DCGM)](https://cloud.google.com/blog/products/containers-kubernetes/monitoring-gpu-workloads-on-gke-with-nvidia-data-center-gpu-manager)


Learn how to observe GPU workloads on GKE with NVIDIA Data Center GPU Manager (DCGM).

Tutorial AI/ML Observability GPUs

<br />

Training Training and tutorials

### [Quickstart: Train a model with GPUs on GKE Standard clusters](https://docs.cloud.google.com/kubernetes-engine/docs/quickstarts/train-model-gpus-standard.md.txt)


This quickstart shows you how to deploy a training model with GPUs in GKE and store the predictions in Cloud Storage.

Tutorial AI/ML Training GPUs

<br />

Training Training and tutorials

### [Running large-scale machine learning on GKE](https://youtu.be/CxemKHmRtsc?si=O_pZvitsaPJ2SzUf)


This video shows how GKE helps solve common challenges of training large AI models at scale, and the best practices for training and serving large-scale machine learning models on GKE.

Video AI/ML Training AI/ML Inference

<br />

Tutorial Training and tutorials

### [TensorFlow on GKE Autopilot with GPU acceleration](https://cloud.google.com/blog/products/containers-kubernetes/automate-tensorflow-ml-workloads-on-cloud-with-autopilot)


This blog post is a step-by-step guide to the creation, execution, and teardown of a Tensorflow-enabled Jupiter notebook.

Blog AI/ML Training AI ML Inference GPUs

<br />

Training Training and tutorials

### [Implement a Job queuing system with quota sharing between namespaces on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/kueue-cohort.md.txt)


This tutorial uses Kueue to show you how to implement a Job queueing system, and configure workload resource and quota sharing between different namespaces on GKE.

Tutorial AI/ML Batch

<br />

Training Training and tutorials

### [Build a RAG chatbot with GKE and Cloud Storage](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/build-rag-chatbot.md.txt)


This tutorial shows you how to integrate a Large Language Model application based on retrieval-augmented generation with PDF files that you upload to a Cloud Storage bucket.

Tutorial AI/ML Data Loading

<br />

Training Training and tutorials

### [Analyze data on GKE using BigQuery, Cloud Run, and Gemma](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/gke-bq-integration.md.txt)


This tutorial shows you how to analyze big datasets on GKE by leveraging BigQuery for data storage and processing, Cloud Run for request handling, and a Gemma LLM for data analysis and predictions.

Tutorial AI/ML Data Loading

<br />

Use case Use cases

### [Distributed data preprocessing with GKE and Ray: Scaling for the enterprise](https://cloud.google.com/blog/products/ai-machine-learning/preprocessing-large-datasets-with-ray-and-gke)


Learn how to leverage GKE and Ray to efficiently preprocess large datasets for machine learning.

MLOps Training Ray

<br />

Use case Use cases

### [Data loading best practices for AI/ML inference on GKE](https://cloud.google.com/blog/products/containers-kubernetes/improve-data-loading-times-for-ml-inference-apps-on-gke)


Learn how to speed up data loading times for your machine learning applications on Google Kubernetes Engine.

Inference Hyperdisk ML Cloud Storage FUSE

<br />

Use case Use cases

### [Save on GPUs: Smarter autoscaling for your GKE inferencing workloads](https://cloud.google.com/blog/products/containers-kubernetes/tuning-the-gke-hpa-to-run-inference-on-gpus)


Learn how to optimize your GPU inference costs by fine-tuning GKE's Horizontal Pod Autoscaler for maximum efficiency.

Inference GPU HPA

<br />

Use case Use cases

### [Efficiently serve optimized AI models with NVIDIA NIM microservices on GKE](https://cloud.google.com/blog/products/containers-kubernetes/nvidia-nims-are-available-on-gke)


Learn how to deploy cutting-edge NVIDIA NIM microservices on GKE with ease and accelerate your AI workloads.

AI NVIDIA NIM

<br />

Use case Use cases

### [Accelerate Ray in production with new Ray Operator on GKE](https://cloud.google.com/blog/products/ai-machine-learning/accelerate-ray-in-production-with-ray-operator-on-gke)


Learn how Ray Operator on GKE simplifies your AI/ML production deployments, boosting performance and scalability.

AI TPU Ray

<br />

Use case Use cases

### [Maximize your LLM serving throughput for GPUs on GKE --- a practical guide](https://cloud.google.com/blog/products/ai-machine-learning/selecting-gpus-for-llm-serving-on-gke)


Learn how to maximize large language model (LLM) serving throughput for GPUs on GKE, including infrastructure decisions and model server optimizations.

LLM GPU NVIDIA

<br />

Use case Use cases

### [Best practices for running batch workloads on GKE](https://docs.cloud.google.com/kubernetes-engine/docs/best-practices/batch-platform-on-gke.md.txt)


Learn how to build and optimize batch processing platforms on GKE

Batch Performance Cost optimization

<br />

Use case Use cases

### [High performance AI/ML storage through Local SSD support on GKE](https://cloud.google.com/blog/products/containers-kubernetes/high-performance-aiml-storage-through-local-ssd-support-on-gke)


Learn how to use Local SSDs to provide high-performance AI/ML storage on GKE.

AI NVMe Local SSD

<br />

Use case Use cases

### [Machine learning with JAX on Kubernetes with NVIDIA GPUs](https://cloud.google.com/blog/products/containers-kubernetes/machine-learning-with-jax-on-kubernetes-with-nvidia-gpus)


Learn how to run JAX multi-GPU, multi-node applications on GKE with NVIDIA GPUs.

GPUs JAX ML

<br />

Use case Use cases

### [LiveX AI reduces customer support costs with AI agents trained and served on GKE and NVIDIA AI](https://cloud.google.com/blog/products/containers-kubernetes/livex-ai-build-ai-agents-on-gke-infrastructure)


How LiveX AI uses GKE to build AI agents that enhance customer satisfaction and reduce costs.

GenAI NVIDIA GPU

<br />

Use case Use cases

### [Infrastructure for a RAG-capable generative AI application using GKE and Cloud SQL](https://docs.cloud.google.com/architecture/rag-capable-gen-ai-app-using-gke.md.txt)


Reference architecture for running a generative AI application with retrieval-augmented
generation (RAG) using GKE, Cloud SQL, Ray, Hugging Face, and LangChain.

GenAI RAG Ray

<br />

Use case Use cases

### [Reference architecture for a batch processing platform on GKE](https://github.com/ai-on-gke/batch-reference-architecture/blob/main/README.md)


Reference architecture for a batch processing platform on GKE in Standard mode using Kueue to manage resoure quotas.

AI Kueue Batch

<br />

Use case Use cases

### [Innovating in patent search: How IPRally leverages AI with GKE and Ray](https://cloud.google.com/blog/products/containers-kubernetes/iprally-builds-ai-based-patent-search-platform-on-gke-and-ray)


How IPRally uses GKE and Ray to build a scalable, efficient ML platform for faster patent searches with better accuracy.

AI Ray GPU

<br />

Use case Use cases

### [Performance deep dive of Gemma on Google Cloud](https://cloud.google.com/blog/products/ai-machine-learning/performance-deepdive-of-gemma-on-google-cloud)


Leverage Gemma on Cloud GPUs and Cloud TPUs for inference and training efficiency on GKE.

AI Gemma Performance

<br />

Use case Use cases

### [Gemma on GKE deep dive: New innovations to serve open generative AI models](https://cloud.google.com/blog/products/containers-kubernetes/serving-gemma-on-google-kubernetes-engine-deep-dive)


Use best-in-class Gemma open models to build portable, customizable AI applications and deploy them on GKE.

AI Gemma Performance

<br />

Use case Use cases

### [Advanced scheduling for AI/ML with Ray and Kueue](https://cloud.google.com/blog/products/containers-kubernetes/using-kuberay-and-kueue-to-orchestrate-ray-applications-in-gke)


Orchestrate Ray applications in GKE with KubeRay and Kueue.

Kueue Ray KubeRay

<br />

Use case Use cases

### [How to secure Ray on Google Kubernetes Engine](https://cloud.google.com/blog/products/containers-kubernetes/securing-ray-to-run-on-google-kubernetes-engine)


Apply security insights and hardening techniques for training AI/ML workloads using Ray on GKE.

AI Ray Security

<br />

Use case Use cases

### [Design storage for AI and ML workloads in Google Cloud](https://docs.cloud.google.com/ai-hypercomputer/docs/storage.md.txt)


Select the best combination of storage options for AI and ML workloads on Google Cloud.

AI ML Storage

<br />

Use case Use cases

### [Automatic driver installation simplifies using NVIDIA GPUs in GKE](https://cloud.google.com/blog/products/containers-kubernetes/gke-can-now-automatically-install-nvidia-gpu-drivers)


Automatically install Nvidia GPU drivers in GKE.

GPU NVIDIA Installation

<br />

Use case Use cases

### [Accelerate your generative AI journey with NVIDIA NeMo framework on GKEE](https://cloud.google.com/blog/products/compute/gke-and-nvidia-nemo-framework-to-train-generative-ai-models)


Train generative AI models using GKE and NVIDIA NeMo framework.

GenAI NVIDIA NeMo

<br />

Use case Use cases

### [Why GKE for your Ray AI workloads?](https://cloud.google.com/blog/products/containers-kubernetes/the-benefits-of-using-gke-for-running-ray-ai-workloads)


Improve scalability, cost-efficiency, fault tolerance, isolation, and portability by using GKE for Ray workloads.

AI Ray Scale

<br />

Use case Use cases

### [Simplifying MLOps using Weights \& Biases with Google Kubernetes Engine](https://cloud.google.com/blog/products/containers-kubernetes/running-weights-and-biases-launch-ml-platform-on-gke)


Simplify the model development and deployment process using Weights \& Biases with GKE.

Cost optimization TPUs GPUs

<br />

Use case Use cases

### [Running AI on fully managed GKE, now with new compute options, pricing and resource reservations](https://cloud.google.com/blog/products/containers-kubernetes/new-features-to-run-ai-more-efficiently-on-fully-managed-gke)


Gain improved GPU support, performance, and lower pricing for AI/ML workloads with GKE Autopilot.

GPU Autopilot Performance

<br />

Use case Use cases

### [How SEEN scaled output 89x and reduced GPU costs by 66% using GKE](https://cloud.google.com/blog/products/containers-kubernetes/startup-scales-personalized-video-output-with-gke)


Startup scales personalized video output with GKE.

GPU Scale Containers

<br />

Use case Use cases

### [How Spotify is unleashing ML Innovation with Ray and GKE](https://engineering.atspotify.com/2023/02/unleashing-ml-innovation-at-spotify-with-ray/)


How Ray is transforming ML development at Spotify.

ML Ray Containers

<br />

Use case Use cases

### [How Ordaōs Bio takes advantage of generative AI on GKE](https://youtu.be/JzxlB7U9Y14?si=2E8OwOXCGiPey7q8)


Ordaōs Bio, one of the leading AI accelerators for biomedical research and discovery, is finding solutions to novel immunotherapies in oncology and chronic inflammatory disease.

Performance TPU Cost optimization

<br />

Use case Use cases

### [GKE from a growing startup powered by ML](https://youtu.be/EVD5sx4O73k?si=LX_aci_WJFZupA2N)


How Moloco, a Silicon Valley startup, harnessed the power of GKE and Tensor Flow Enterprise to supercharge its machine learning (ML) infrastructure.

ML Scale Cost optimization

<br />

Use case Use cases

### [Improving launch time of Stable Diffusion on GKE by 4x](https://cloud.google.com/blog/products/containers-kubernetes/stable-diffusion-containers-on-gke)


Learn how to improve the launch time of Stable Diffusion on GKE.

Performance Scaling PD

<br />

Code sample Code Samples

### [Google Kubernetes Engine (GKE) Samples](https://github.com/GoogleCloudPlatform/kubernetes-engine-samples/tree/main/ai-ml)


View sample applications used in official GKE product tutorials.


Code sample Code Samples

### [GKE AI Labs Samples](https://gke-ai-labs.dev/)


View experimental samples for leveraging GKE to accelerate your AI/ML initiatives.


Code sample Code Samples

### [GKE Accelerated Platforms](https://github.com/GoogleCloudPlatform/accelerated-platforms/tree/main/docs/platforms/gke/base)


View reference architectures and solutions for deploying accelerated workloads on GKE.


## Related videos