This tutorial guides you through the process of creating a retrieval-augmented generation (RAG) pipeline based on parsed PDF content.
PDF files, such as financial documents, can be challenging to use in RAG pipelines because of their complex structure and mix of text, figures, and tables. This tutorial shows you how to use the [`AI.CHUNK_DOCUMENT` function](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-chunk-document.md.txt) to build a RAG pipeline based on key information extracted from a PDF file. ## Objectives This tutorial covers the following tasks:
- Create a Cloud Storage bucket and upload a sample PDF file. - Use the [`AI.CHUNK_DOCUMENT` function](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-chunk-document.md.txt) to parse the PDF contents into chunks and then write that content to a BigQuery table. - Generate embeddings from the chunked PDF content, and then write those embeddings to a BigQuery table. Embeddings are numerical representations of the PDF content that enable you to perform semantic search and retrieval on the PDF content. - Use the [`VECTOR_SEARCH` function](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/search_functions.md.txt#vector_search) on the embeddings to identify semantically similar PDF content. - Perform retrieval-augmented generation (RAG) by using the [`AI.GENERATE` function](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate.md.txt) to generate text, using vector search results to augment the prompt input and improve results. ## Costs In this document, you use the following billable components of Google Cloud: - [BigQuery](https://cloud.google.com/bigquery/pricing): You incur costs for the data that you process in BigQuery. - [Gemini Enterprise Agent Platform](https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing): You incur costs for calls to Agent Platform models. - [Document AI](https://cloud.google.com/document-ai/pricing): You incur costs for calls to the Document AI API. - [Cloud Storage](https://cloud.google.com/storage/pricing): You incur costs for object storage in Cloud Storage. To generate a cost estimate based on your projected usage, use the [pricing calculator](https://docs.cloud.google.com/products/calculator). New Google Cloud users might be eligible for a [free trial](https://docs.cloud.google.com/free).
When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see [Clean up](https://docs.cloud.google.com/bigquery/docs/rag-pipeline-pdf.md.txt#clean-up). ## Before you begin ### Console 1. Make sure that you have the following role or roles on the project: **Storage Admin** , **Document AI Editor** , **BigQuery Admin** , **Project IAM Admin** #### Check for the roles 1. In the Google Cloud console, go to the **IAM** page. [Go to IAM](https://console.cloud.google.com/projectselector/iam-admin/iam?supportedpurview=project) 2. Select the project. 3. In the **Principal** column, find all rows that identify you or a group that you're included in. To learn which groups you're included in, contact your administrator. 4. For all rows that specify or include you, check the **Role** column to see whether the list of roles includes the required roles. #### Grant the roles 1. In the Google Cloud console, go to the **IAM** page. [Go to IAM](https://console.cloud.google.com/projectselector/iam-admin/iam?supportedpurview=project) 2. Select the project. 3. Click **Grant access**. 4. In the **New principals** field, enter your user identifier. This is typically the email address for a Google Account. 5. Click **Select a role**, then search for the role. 6. To grant additional roles, click **Add another role** and add each additional role. 7. Click **Save**.
### gcloud 1. Grant roles to your user account. Run the following command once for each of the following IAM roles: ` roles/storage.admin, roles/documentai.editor, roles/bigquery.admin, roles/resourcemanager.projectIamAdmin ` ```bash gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE ``` Replace the following: - `PROJECT_ID`: Your project ID. - `USER_IDENTIFIER`: The identifier for your user account. For example, `myemail@example.com`. - `ROLE`: The IAM role that you grant to your user account.
## Create a dataset To create a BigQuery dataset, select one of the following options: ### Console 1. In the Google Cloud console, go to the **BigQuery** page. [Go to BigQuery](https://console.cloud.google.com/bigquery) 2. In the left pane, click **Explorer**: ![Highlighted button for the Explorer pane.](https://docs.cloud.google.com/static/bigquery/images/explorer-tab.png) If you don't see the left pane, click **Expand left pane** to open the pane. 3. In **Explorer** , expand your project, and then click **Datasets**. 4. On the **Datasets** page, click **Create dataset**. 5. In the **Create dataset** pane, do the following: - For **Dataset ID** , enter `bqml_tutorial`. - For **Data location** , select **US**. Leave the remaining default settings as they are. 6. Click **Create dataset**. ### bq To create a new dataset, use the [`bq mk --dataset` command](https://docs.cloud.google.com/bigquery/docs/reference/bq-cli-reference.md.txt#mk-dataset). 1. Create a dataset named `bqml_tutorial` with the data location set to `US`: ``` bq mk --dataset \ --location=US \ --description "BigQuery ML tutorial dataset." \ bqml_tutorial ``` 2. Confirm that the dataset was created: ```bash bq ls ``` ### API Call the [`datasets.insert`](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/datasets/insert.md.txt) method with a defined [dataset resource](https://docs.cloud.google.com/bigquery/docs/reference/rest/v2/datasets.md.txt):
```json { "datasetReference": { "datasetId": "bqml_tutorial" } } ``` ## Upload the sample PDF to Cloud Storage To upload the sample PDF to Cloud Storage, follow these steps: 1. Download the `scf23.pdf` sample PDF by going to and clicking download . 2. [Create a Cloud Storage bucket](https://docs.cloud.google.com/storage/docs/creating-buckets.md.txt). 3. [Upload](https://docs.cloud.google.com/storage/docs/uploading-objects.md.txt) the `scf23.pdf` file to the bucket. ## Parse the PDF file into chunks Use the document processor with the `AI.CHUNK_DOCUMENT` function to parse the PDF file into chunks, and then write that content to a table: 1. In the Google Cloud console, go to the **BigQuery** page. [Go to BigQuery](https://console.cloud.google.com/bigquery) 2. In the query editor, run the following statement: ```googlesql CREATE or REPLACE TABLE bqml_tutorial.chunked_pdf AS ( SELECT * FROM AI.CHUNK_DOCUMENT('gs://BUCKET_NAME/scf23.pdf', chun>k_size = 250) ); ```
3. In the query editor, run the following query to view a subset of the parsed PDF content: ```googlesql SELECT chunk_id, start_page, end_page, content FROM `bqml_tutorial.chunked_pdf` ORDER BY chunk_id LIMIT 3; ``` The output is similar to the following: ``` +---+---+---+---+ | chunk_id | start_page | end_page | content | +---+---+---+---+ | 1 | 1 | 1 | OF NORS. • BOARD OF GOVERNO FEDERAL | | | | | RESERVE SYSTEM RESEARCH & ANALYSIS | | 2 | 1 | 7 | # Changes in U.S. Family Finances from | | | | | 2019 to 2022 | | | | | | | | | | Evidence from the Survey of Consumer... | | 3 | 3 | 5 | # Changes in U.S. Family Finances from | | | | | 2019 to 2022 | | | | | | | | | | ## Contents... | +---+---+---+---+ ``` ## Generate embeddings Generate embeddings for the parsed PDF content and then write them to a table: 1. In the Google Cloud console, go to the **BigQuery** page. [Go to BigQuery](https://console.cloud.google.com/bigquery) 2. In the query editor, run the following statement: ```googlesql CREATE OR REPLACE TABLE `bqml_tutorial.embeddings` AS ( SELECT *, AI.EMBED(content, endpoint => 'text-embedding-005').result AS embedding FROM bqml_tutorial.chunked_pdf ); ``` ## Run a vector search Run a vector search against the parsed PDF content. The following query takes text input, creates an embedding for that input using the `AI.EMBED` function, and then uses the `VECTOR_SEARCH` function to match the input embedding with the most similar PDF content embeddings. The results are the top ten PDF chunks that are most related to changes in family net worth. 1. Go to the **BigQuery** page. [Go to BigQuery](https://console.cloud.google.com/bigquery) 2. In the query editor, run the following SQL statement: ```googlesql SELECT distance, base.chunk_id AS chunk_id, base.start_page AS start_page, base.end_page AS end_page, base.content FROM VECTOR_SEARCH( TABLE `bqml_tutorial.embeddings`, 'embedding', query_value => AI.EMBED( 'Did the typical family net worth increase? If so, by how much?', endpoint => 'text-embedding-005').result, top_k => 3, OPTIONS => '{"fraction_lists_to_search": 0.01}') ORDER BY distance DESC; ``` The output is similar to the following: ``` +---+---+---+---+---+ | distance | chunk_id | start_page | end_page | content | +---+---+---+---+---+ | 0.608587 | 25 | 17 | 18 | # Net Worth | | | | | | | | | | | | 18 Between the first quarter of | | | | | | 2019 and the first quarter of... | +---+---+---+---+---+ | 0.606387 | 29 | 19 | 21 | ## Net Worth | | | | | | | | | | | | ## Net Worth by Family | | | | | | Characteristics... | +---+---+---+---+---+ | 0.599438 | 23 | 17 | 21 | # Net Worth | | | | | | | | | | | | The net improvements in... | +---+---+---+---+---+ ``` ## Generate text augmented by vector search results Perform a vector search on the embeddings to identify semantically similar PDF content, and then use the `AI.GENERATE` function with the vector search results to augment the prompt input and improve the text generation results. In this case, the query uses information from the PDF chunks to answer a question about the change in family net worth over the past decade. 1. In the Google Cloud console, go to the **BigQuery** page. [Go to BigQuery](https://console.cloud.google.com/bigquery) 2. In the query editor, run the following statement: ```googlesql SELECT AI.GENERATE( CONCAT('Did the typical family net worth change? How does this compare the SCF survey a decade earlier? Be concise and use the following context:', STRING_AGG(FORMAT("context: %s", base.content), ',\n') ) ).result AS response FROM VECTOR_SEARCH( TABLE `bqml_tutorial.embeddings`, 'embedding', query_value => AI.EMBED( 'Did the typical family net worth increase? If so, by how much?', endpoint => 'text-embedding-005').result, top_k => 3, OPTIONS => '{"fraction_lists_to_search": 0.01}') ``` The output is similar to the following: ``` +---+ | response | +---+ | Yes, the typical family net worth changed significantly. Between 2019 | | and 2022, real median net worth surged by 37% to $192,900. | | | | This growth contrasts sharply with the period a decade earlier. Between | | the 2010 and 2013 SCF surveys, real median net worth actually decreased | | by 2%. | +---+ ``` ## Clean up To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources. ### Delete the project > [!CAUTION] > **Caution** : Deleting a project has the following effects: > > - **Everything in the project is deleted.** If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project. > - **Custom project IDs are lost.** When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an `appspot.com` URL, delete selected resources inside the project instead of deleting the whole project. > > > If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects > can help you avoid exceeding project quota limits. 1. Delete a Google Cloud project: ``` gcloud projects delete PROJECT_ID ```
## What's next - Learn more about [embeddings and vector search](https://docs.cloud.google.com/bigquery/docs/vector-search-intro.md.txt). - Learn more about performing [semantic search and RAG](https://docs.cloud.google.com/bigquery/docs/vector-index-text-search-tutorial.md.txt).