This document provides a reference architecture that you can use to design the infrastructure to run a generative artificial intelligence (AI) application with retrieval-augmented generation (RAG). The intended audience for this document includes developers and administrators of generative AI applications and cloud architects. The document assumes a basic understanding of AI, machine learning (ML), and large language model (LLM) concepts. This document doesn't provide guidance about how to design and develop a generative AI application.
Architecture
The following diagram shows a high-level view of an architecture for a RAG-capable generative AI application in Google Cloud:
The architecture contains the following interconnected components:
- Data ingestion subsystem
- The data ingestion subsystem prepares and processes the external data that's used to enable the RAG capability. It interacts with the other subsystems in the architecture through the database layer.
- Serving subsystem
- The serving subsystem handles the request-response flow between the generative AI application and its users. It interacts with the data ingestion subsystem through the database layer.
- Quality evaluation subsystem
- The quality evaluation subsystem evaluates the quality of responses that the serving subsystem generates. It interacts with the serving subsystem directly and with the data ingestion subsystem through the database layer.
- Databases
All the subsystems in the architecture interact with the databases. The databases store the following data:
- Prompts
- Vectorized embeddings of the data that's used for RAG
- Configurations of the serverless functions in the data ingestion and quality evaluation subsystems
The following diagram shows a detailed view of the architecture:
The following sections provide detailed descriptions of the components and data flow within each subsystem of the architecture.
Data ingestion subsystem
The data ingestion subsystem ingests data from external sources such as files, databases, and streaming services. The uploaded data includes prompts for quality evaluation. The data ingestion subsystem provides the RAG capability in the architecture. The following diagram shows details of the data ingestion subsystem in the architecture:
The following are the steps in the data-ingestion flow:
- Data is uploaded to a Cloud Storage bucket. The data sources might be applications, databases, or streaming services.
- When data is uploaded to Cloud Storage, a message is published to a Pub/Sub topic.
- Pub/Sub triggers a Cloud Run function to process the uploaded data.
- The function is started by using configuration data that's stored in an AlloyDB for PostgreSQL database.
- The function parses the raw data, formats it as required, and then divides it into chunks.
The function uses an appropriate embedding model on Gemini Enterprise Agent Platform to create vectorized embeddings of the ingested data.
The function stores the embeddings in an AlloyDB database that has the
pgvector-compatiblevectorextension enabled. As described in the following section, when the serving subsystem processes user requests, it uses the embeddings in the vector database to retrieve relevant domain-specific data.
Serving subsystem
The serving subsystem handles the request-response flow between the generative AI application and its users. The following diagram shows details of the serving subsystem in the architecture:
The following are the steps in the request-response flow in the serving subsystem:
- Users submit requests to the generative AI application through a frontend (for example, a chatbot or mobile app).
The generative AI application converts the natural-language request to embeddings.
The application completes the retrieval part of the RAG approach:
- The application performs a semantic search for the embedding in the AlloyDB vector store that's maintained by the data ingestion subsystem. Semantic search helps find embeddings based on the intent of a prompt rather than its textual content.
- The application combines the original request with the raw data that's retrieved based on the matching embedding to create a contextualized prompt.
The application sends the contextualized prompt to an LLM inference stack that runs on Agent Platform.
The LLM inference stack uses a generative AI LLM, which can be a foundation LLM or a custom LLM, and generates a response that's constrained to the provided context.
- The application can store logs of the request-response activity in Cloud Logging. You can view and use the logs for monitoring using Cloud Monitoring. Google doesn't access or use log data.
- The application loads the responses to BigQuery for offline analytics.
The application screens the responses by using responsible AI filters.
The application sends the screened responses to users through the frontend.
Quality evaluation subsystem
The following diagram shows details of the quality evaluation subsystem in the architecture:
When the quality evaluation subsystem receives a request, it does the following:
- Pub/Sub triggers a Cloud Run function.
- The function is started by using configuration data that's stored in an AlloyDB database.
- The function pulls evaluation prompts from an AlloyDB database. The prompts were previously uploaded to the database by the data ingestion subsystem.
The function uses the evaluation prompts to assess the quality of the responses that the serving subsystem generates.
The output of this evaluation consists of evaluation scores for metrics like factual accuracy and relevance.
The function loads the evaluation scores and the prompts and responses that were evaluated to BigQuery for future analysis.
Products used
The following is a summary of all the Google Cloud products that the preceding architecture uses:
- Gemini Enterprise Agent Platform: A comprehensive platform that lets you build, scale, govern, and optimize enterprise‑grade AI agents.
- Cloud Run functions: A serverless compute platform that lets you run single-purpose functions directly in Google Cloud.
- BigQuery: An enterprise data warehouse that helps you manage and analyze your data with built-in features like machine learning, geospatial analysis, and business intelligence.
- Cloud Storage: A low-cost, no-limit object store for diverse data types. Data can be accessed from within and outside Google Cloud, and it's replicated across locations for redundancy.
- AlloyDB for PostgreSQL: A fully managed, PostgreSQL-compatible database service that's designed for your most demanding workloads, including hybrid transactional and analytical processing.
- Pub/Sub: An asynchronous and scalable messaging service that decouples services that produce messages from services that process those messages.
- Cloud Logging: A real-time log management system with storage, search, analysis, and alerting.
- Cloud Monitoring: A service that provides visibility into the performance, availability, and health of your applications and infrastructure.
Use cases
RAG is an effective technique to improve the quality of output that's generated from an LLM. This section provides examples of use cases for which you can use RAG-capable generative AI applications.
Personalized product recommendations
An online shopping site might use an LLM-powered chatbot to assist customers with finding products or getting shopping-related help. The questions from a user can be augmented by using historical data about the user's buying behavior and website interaction patterns. The data might include user reviews and feedback that's stored in an unstructured datastore or search-related metrics that are stored in a web analytics data warehouse. The augmented question can then be processed by the LLM to generate personalized responses that the user might find more appealing and compelling.
Clinical assistance systems
Clinicians in hospitals need to review and synthesize patient health information to make informed care decisions. A generative AI application that uses a medical foundation model like MedGemma can help clinicians summarize clinical notes and review medical context. The responses that the application generates can be grounded in historical patient records by contextualizing prompts with data from the hospital's electronic health record (EHR) database or from an external knowledge base like PubMed.
Efficient legal research
Generative AI-powered legal research lets lawyers quickly query large volumes of statutes and case laws to identify relevant legal precedents or summarize complex legal concepts. The output of such research can be enhanced by augmenting a lawyer's prompts with data that's retrieved from the law firm's proprietary corpus of contracts, past legal communication, and internal case records. This design approach helps generate responses that are relevant to the legal domain that the lawyer specializes in.
Design alternatives
This section presents alternative design approaches that you can consider for your RAG-capable generative AI application in Google Cloud.
Managed RAG framework
To orchestrate context-augmented LLM workflows without building a custom ingestion and retrieval pipeline, you can use RAG Engine on Gemini Enterprise Agent Platform. RAG Engine manages data ingestion, chunking, embedding generation, indexing, and retrieval for RAG workloads.
Fully managed vector search
If you need an architecture that uses a fully managed vector search product, you can use Agent Platform and Vector Search, which provides optimized serving infrastructure for very large-scale vector search. For more information, see RAG infrastructure for generative AI using Agent Platform and Vector Search.
Open-source tools and models
If you want to rapidly build and deploy RAG-capable generative AI applications by using open source tools and models Ray, Hugging Face, and LangChain, see RAG infrastructure for generative AI using GKE and Cloud SQL.
Design considerations
This section provides guidance to help you develop a RAG-capable generative AI architecture in Google Cloud that meets your specific requirements for security and compliance, reliability, cost, and performance. The guidance in this section isn't exhaustive. Depending on the specific requirements of your generative AI application and the Google Cloud products and features that you use, you might need to consider additional design factors and trade-offs.
Security and compliance
This section describes factors that you should consider when you design and build a RAG-capable generative AI application in Google Cloud that meets your security and compliance requirements.
Agent Platform security considerations
To help ensure that your Agent Platform configurations and usage meet your security requirements, consider the following design factors and recommendations:
Security controls: Agent Platform supports Google Cloud security controls that you can use to meet your requirements for data residency, data encryption, network security, and access transparency. For more information, see Security controls for machine learning services and Security controls for Generative AI.
Prompt and response screening: To screen RAG prompts and model responses for prompt injection, jailbreak attempts, harmful content, and sensitive data leaks, use Model Armor templates and filters.
Cloud Run functions data encryption
To protect your data by using a key that you control, you can use customer-managed encryption keys (CMEKs).
AlloyDB security considerations
To help ensure that your AlloyDB configuration meets your security requirements, consider the following recommendations:
- Data encryption: By default, data that's stored in AlloyDB is encrypted using Google-owned and Google-managed encryption keys. If you need to use encryption keys that you control and manage, you can use CMEKs. For more information, see About CMEK.
- Network control: To mitigate the risk of data exfiltration from AlloyDB databases, you can create a service perimeter by using VPC Service Controls.
- Connection security: By default, an AlloyDB instance accepts only connections that use SSL. To further secure connections to your AlloyDB databases, you can use the AlloyDB Auth Proxy connector. The Auth Proxy connector provides Identity and Access Management (IAM)-based connection authorization and uses a TLS 1.3 connection with a 256-bit AES cipher to verify client and server identities and encrypt data traffic. For more information, see About the AlloyDB Auth Proxy. For connections that are created by using Java, Python, Go, or Node.js, use the appropriate Language Connector instead of the Auth Proxy connector.
- Data residency: AlloyDB helps you meet data residency requirements. Data is stored or replicated within the regions that you specify.
BigQuery security considerations
To help ensure that your BigQuery configuration meets your security requirements, consider the following recommendations:
- Data governance: BigQuery provides many features that you can use to control access to data, protect sensitive data, and help ensure data accuracy and consistency. For more information, see Introduction to data governance in BigQuery.
- Data residency: BigQuery helps you meet data residency requirements. Data is stored within the region that you specify.
Cloud Storage security considerations
To help ensure that your Cloud Storage configuration meets your security requirements, consider the following recommendations:
- Data encryption: By default, the data that's stored in Cloud Storage is encrypted using Google-owned and Google-managed encryption keys. If required, you can use CMEKs or your own keys that you manage by using an external management method like customer-supplied encryption keys (CSEKs). For more information, see Data encryption options.
- Access control: Cloud Storage supports two methods for granting users access to your buckets and objects: IAM and access control lists (ACLs). In most cases, use IAM, which lets you grant permissions at the bucket and project levels. For more information, see Overview of access control.
- Data protection: The data that you load into the data ingestion subsystem through Cloud Storage might include sensitive data. To protect such data, you can use Sensitive Data Protection to discover, classify, and de-identify the data. For more information, see Using Sensitive Data Protection with Cloud Storage.
- Data residency: Cloud Storage helps you meet data residency requirements. Data is stored or replicated within the regions that you specify.
Pub/Sub security considerations
To help ensure that your Pub/Sub configuration meets your security requirements, consider the following recommendations:
- Data encryption: By default, Pub/Sub encrypts all messages, both at rest and in transit, by using Google-owned and Google-managed encryption keys. Pub/Sub supports the use of CMEKs for message encryption at the application layer. For more information, see Configuring message encryption.
- Data residency: If you have data residency requirements, to help ensure that message data is stored in specific locations, you can configure message storage policies.
Cloud Logging security considerations
To help ensure that your Cloud Logging configuration meets your security requirements, consider the following design factors and recommendations:
- Administrative-activity audit: Admin Activity audit logs are enabled by default for all the Google Cloud services that are used in this reference architecture. These logs record API calls or other actions that modify the configuration or metadata of Google Cloud resources.
- Data-access audit: Data Access audit logs are enabled by default for BigQuery. For the other services that are used in this architecture, you can enable Data Access audit logs. The logs let you track API calls that read the configuration or metadata of resources or user requests to create, modify, or read user-provided resource data.
- Data residency: To help meet data residency requirements, you can configure Cloud Logging to store log data in the region that you specify. For more information, see Regionalize your logs.
For security principles and recommendations that are specific to AI and ML workloads, see AI and ML perspective: Security in the Well-Architected Framework.
Reliability
This section describes design factors that you should consider to build and operate reliable infrastructure for a RAG-capable generative AI application in Google Cloud.
Cloud Run functions reliability considerations
To help ensure that your Cloud Run functions implementation meets your reliability requirements, consider the following design factors and recommendations:
- Zone redundancy: Cloud Run functions run across multiple zones within a region to provide built-in resilience against zone outages.
- Failure handling: To handle duplicate events or transient failures safely, write idempotent functions and configure event-driven function retries. To isolate messages that repeatedly fail processing, configure a Pub/Sub dead-letter topic.
AlloyDB reliability considerations
To help ensure that your AlloyDB configuration meets your reliability requirements, consider the following design factors and recommendations:
- High availability: By default, AlloyDB clusters provide high availability (HA) with automatic failover. The primary instance has redundant nodes that are located in two different zones within a region. This redundancy helps ensure that the clusters are resilient against zone outages.
- Disaster recovery: To plan for recovery from region outages, you can use cross-region replication.
BigQuery reliability considerations
Data that you load into BigQuery is stored synchronously in two zones within the region that you specify. This redundancy helps ensure that your data isn't lost when a zone outage occurs. For more information about reliability features in BigQuery, see Understand reliability.
Cloud Storage reliability considerations
You can create Cloud Storage buckets in one of the following location types: regional, dual-region, or multi-region. Data that's stored in regional buckets is replicated synchronously across multiple zones within a region. For higher availability, you can use dual-region or multi-region buckets, where data is replicated asynchronously across regions.
Pub/Sub reliability considerations
To help ensure that your Pub/Sub configuration meets your reliability requirements, consider the following recommendations:
- Rate control: To manage transient spikes in message traffic, you can configure flow control in the publisher settings.
- Failure handling: To handle failed publishes, adjust the retry-request variables as necessary. For more information, see Retry requests.
For more information about region-specific considerations, see Geography and regions.
For reliability principles and recommendations that are specific to AI and ML workloads, see AI and ML perspective: Reliability in the Well-Architected Framework.
Cost optimization
This section provides guidance to help you optimize the cost of setting up and operating a RAG-capable generative AI application in Google Cloud.
Agent Platform context caching
To reduce input token costs when repeated RAG requests share large system instructions or retrieved context, use context caching.
AlloyDB cost considerations
To help optimize the cost of AlloyDB, consider the following recommendations:
- Instance availability: By default, a primary instance of an AlloyDB cluster is highly available (HA). The instance has an active node and a standby node. If the active node fails, AlloyDB fails over to the standby node automatically. If you don't need HA for the databases, then you can reduce cost by making the cluster's primary instance a basic instance. A basic instance isn't resilient against zone outages, and it has longer downtime during maintenance operations. For more information, see Reduce costs using basic instances.
- Rate optimization: If you can predict the CPU and memory requirements of your AlloyDB instance, then you can save money by getting discounts for committed usage. For more information, see AlloyDB committed use discounts.
BigQuery cost considerations
BigQuery lets you estimate the cost of queries before you run them. To optimize query costs, you need to optimize storage and query computation. For more information, see Estimate and control costs.
Cloud Storage cost considerations
For the Cloud Storage bucket that you use to load data into the data ingestion subsystem, choose an appropriate storage class based on the data-retention and access-frequency requirements of your workloads. For example, to control storage costs, you can choose the Standard storage class and use Object Lifecycle Management. This approach lets you automatically downgrade objects to a lower-cost storage class or delete objects based on conditions that you set.
Cloud Logging cost considerations
To control the cost of storing logs, consider the following recommendations:
- Reduce the volume of logs by excluding or filtering unnecessary log entries. For more information, see Log sink filters.
- Reduce the period for which log entries are retained. For more information, see Configure custom retention.
For cost optimization principles and recommendations that are specific to AI and ML workloads, see AI and ML perspective: Cost optimization in the Well-Architected Framework.
Performance
This section describes the factors that you should consider when you design and build a RAG-capable generative AI application in Google Cloud that meets your performance requirements.
Cloud Run functions latency
To reduce cold-start latency after periods of inactivity, configure a minimum number of instances.
AlloyDB performance considerations
To help ensure that your AlloyDB configuration meets your performance requirements, consider the following recommendations:
- Vector indexing and columnar engine: To accelerate vector similarity queries in RAG workloads, create an Approximate Nearest Neighbor (ANN) index such as ScaNN or HNSW. For more information, see Choose an index strategy. To further speed up vector scans and filtered queries, see Accelerate vector search with the columnar engine.
- Query performance analysis: To help you analyze and improve query performance of the databases, AlloyDB provides a Query Insights tool. You can use this tool to monitor performance and trace the source of a problematic query. For more information, see About query insights.
- Instance monitoring: To get an overview of the status and performance of your databases and to view detailed metrics such as peak connections and maximum replication lag, you can use the System Insights dashboard. For more information, see Monitor an instance using the AlloyDB System Insights dashboard.
- Read scaling: To reduce the load on your primary AlloyDB instance and to scale out the capacity to handle read requests, you can add read pool instances to the cluster. For more information, see What is an AlloyDB instance?.
BigQuery performance considerations
To help ensure that your BigQuery usage meets your performance requirements, consider the following recommendations:
- Query performance insights: BigQuery provides a query execution graph that you can use to analyze query performance and get performance insights for issues like slot contention and insufficient shuffle quota. For more information, see Get query performance insights.
- Query computation optimization: After you address the issues that you identify through query performance insights, you can further optimize queries by using techniques like reducing the volume of input and output data. For more information, see Optimize query computation.
Cloud Storage performance considerations
To upload large files, you can use a method called parallel composite uploads. With this strategy, the large file is split into chunks. The chunks are uploaded to Cloud Storage in parallel, and then the data is recomposed in the cloud. When network bandwidth and disk speed aren't limiting factors, parallel composite uploads can be faster than regular upload operations. However, this strategy has some limitations and cost implications. For more information, see Parallel composite uploads.
For performance optimization principles and recommendations that are specific to AI and ML workloads, see AI and ML perspective: Performance optimization in the Well-Architected Framework.
Deployment
To get started and experiment with building infrastructure on Google Cloud for RAG-capable generative AI applications, use the sample code that's available in GitHub. This code deploys a Python-based chat application on Cloud Run and uses a fully managed Cloud SQL database for vector search.
What's next
- AlloyDB Agentic RAG Application with MCP Toolbox
- RAG infrastructure for generative AI using Agent Platform and Vector Search
- RAG infrastructure for generative AI using GKE and Cloud SQL
- RAG infrastructure for generative AI using Gemini Enterprise and Agent Platform
- GraphRAG infrastructure for generative AI using Agent Platform and Spanner Graph
- For an overview of architectural principles and recommendations that are specific to AI and ML workloads in Google Cloud, see the AI and ML perspective in the Well-Architected Framework.
- For more reference architectures, diagrams, and best practices, explore the Cloud Architecture Center.
Contributors
Author: Kumar Dhanagopal | Cross-Product Solution Developer
Other contributors:
- Andrew Brook | Engineering Director
- Anna Berenberg | Engineering Fellow
- Assaf Namer | Principal Cloud Security Architect
- Balachandar Krishnamoorthy | Principal Software Engineer
- Daniel Lees | Cloud Security Architect
- Derek Downey | Developer Relations Engineer
- Eran Lewis | Senior Product Manager
- Geoffrey Anderson | Product Manager
- Gleb Otochkin | Cloud Advocate, Databases
- Hamsa Buvaraghan | AI Product Manager
- Irina Sigler | Product Manager
- Jack Wotherspoon | Developer Advocate
- Jason Davenport | Developer Advocate
- Jordan Totten | Customer Engineer
- Julia Wiesinger | Product Manager
- Kara Greenfield | Customer Engineer
- Kurtis Van Gent | Staff Software Engineer
- Per Jacobsson | Software Engineer
- Pranav Nambiar | Director
- Richard Hendricks | Architecture Center Staff
- Safiuddin Khaja | Cloud Engineer
- Sandy Ghai | Group Product Manager
- Vladimir Vuskovic | Product Management Director
- Steren Giannini | Group Product Manager
- Wietse Venema | Developer Relations Engineer