Working with list operations

Some services return potentially large lists of items, such as rows or resource descriptions. To keep CPU and memory usage under control, services return these resources in pages: smaller subsets of the items with a continuation token to request the next subset.

Iterating over items by page can be tedious. The client libraries provide adapters to convert the pages into asynchronous iterators. This guide shows you how to work with these adapters.

Prerequisites

This guide uses the Secret Manager service to demonstrate list operations. The concepts included in this guide also apply to other services.

To follow this guide, you'll need to to have enabled the Secret Manager service, logged in, and ensured that your account has the necessary permissions. For guidance on how to fulfill these prerequisites, follow the service quickstart.

For complete setup instructions for the Rust libraries, see Setting up your development environment.

Dependencies

Add the Secret Manager library to your Cargo.toml file:

cargo add google-cloud-secretmanager-v1

Iterating list methods

To help iterate the items in a list method, the APIs return an implementation of the ItemPaginator trait. Introduce the trait into scope using a use declaration:

use google_cloud_gax::paginator::ItemPaginator as _;

To iterate over the items, use the by_item method.

let mut list = client
    .list_secrets()
    .set_parent(format!("projects/{project_id}"))
    .by_item();
while let Some(secret) = list.next().await {
    let secret = secret?;
    println!("  secret={}", secret.name)
}

In rare cases, pages might contain extra information that you need access to, or you may need to checkpoint your progress across processes. If needed, you can iterate over full pages instead of individual items.

1. Introduce Paginator into scope using a use declaration:

   use google_cloud_gax::paginator::Paginator as _;

2. Iterate over the pages using by_page:

   let mut list = client
       .list_secrets()
       .set_parent(format!("projects/{project_id}"))
       .by_page();
   while let Some(page) = list.next().await {
       let page = page?;
       println!("  next_page_token={}", page.next_page_token);
       page.secrets
           .into_iter()
           .for_each(|secret| println!("    secret={}", secret.name));
   }

Working with futures::Stream

You may want to use these APIs in the larger Rust ecosystem of asynchronous streams, such as tokio::Stream. This can be achieved as follows:

1. Enable the unstable-streams feature in the google_cloud_gax crate. The name of this feature serves as notice that these APIs are unstable. You should only use them if you are prepared to deal with any breaks that result from incompatible changes to the futures::Stream trait.

   cargo add google-cloud-gax --features unstable-stream
   

2. The following examples also use the futures::stream::StreamExt trait, which you enable by adding the futures crate.

   cargo add futures
   

3. Add the required use declarations:

   use futures::stream::StreamExt;
   use google_cloud_gax::paginator::ItemPaginator as _;

4. Use the into_stream function to convert ItemPaginator into a futures::Stream.

   let list = client
       .list_secrets()
       .set_parent(format!("projects/{project_id}"))
       .by_item()
       .into_stream();
   list.map(|secret| -> gax::Result<()> {
       println!("  secret={}", secret?.name);
       Ok(())
   })
   .fold(Ok(()), async |acc, result| -> gax::Result<()> {
       acc.and(result)
   })
   .await?;

Similarly, you can use the into_stream function to convert Paginator into a futures::Stream.

let list = client
    .list_secrets()
    .set_parent(format!("projects/{project_id}"))
    .by_page()
    .into_stream();
list.enumerate()
    .map(|(index, page)| -> gax::Result<()> {
        println!("page={}, next_page_token={}", index, page?.next_page_token);
        Ok(())
    })
    .fold(Ok(()), async |acc, result| -> gax::Result<()> {
        acc.and(result)
    })
    .await?;

Resuming list methods by setting next page token

In some cases, such as an interrupted list operation, you can set the next page token to resume paginating from a specific page.

let page = client
    .list_secrets()
    .set_parent(format!("projects/{project_id}"))
    .send()
    .await;
let page = page?;
let mut next_page_token = page.next_page_token.clone();
page.secrets
    .into_iter()
    .for_each(|secret| println!("    secret={}", secret.name));

while !next_page_token.is_empty() {
    println!("  next_page_token={next_page_token}");

    let page = client
        .list_secrets()
        .set_parent(format!("projects/{project_id}"))
        .set_page_token(next_page_token)
        .send()
        .await;
    let page = page?;
    next_page_token = page.next_page_token.clone();

    page.secrets
        .into_iter()
        .for_each(|secret| println!("    secret={}", secret.name));
}

When to use the paginator helpers

The Google Cloud Client Libraries for Rust provide an adapter to convert the list RPCs as defined by AIP-4233 into types implementing ItemPaginator and Paginator if the Google API List method follows the pagination guideline defined by AIP-158. In brief, this guideline requires that each call to a List method returns a page of resource items (for example, secrets) along with a token that you can pass to the List method to retrieve the next page.

Most Google Cloud services follow these guidelines. In the cases in which they don't, you must implement your own adapter to iterate over the results.