장기 실행 작업 테스트 작성

Rust용 Google Cloud 클라이언트 라이브러리는 장기 실행 작업(LRO)과의 상호작용을 간소화하는 도우미를 제공합니다. 단위 테스트에서 LRO의 동작을 시뮬레이션하려면 이러한 도우미가 추상화하는 기본 세부정보를 이해해야 합니다. 이 가이드에서는 오류 시나리오를 포함하여 자동 및 수동 LRO 폴링 모두에 대한 단위 테스트를 작성하는 방법을 보여줍니다.

기본 요건

이 가이드에서는 사용자가 다음 가이드를 숙지했다고 가정하고 작성되었습니다.

자동 폴링 테스트

애플리케이션 코드가 lro::Poller::until_done()(자동 폴링)를 기다린다고 가정합니다.

// An example application function that automatically polls.
//
// It starts an LRO, awaits the result, and processes it.
pub async fn my_automatic_poller(
    client: &speech::client::Speech,
    project_id: &str,
) -> Result<Option<wkt::Duration>> {
    use google_cloud_lro::Poller;
    client
        .batch_recognize()
        .set_recognizer(format!(
            "projects/{project_id}/locations/global/recognizers/_"
        ))
        .poller()
        .until_done()
        .await
        .map(|r| r.total_billed_duration)
}

애플리케이션은 LRO의 최종 결과만 사용하므로 중간 폴링 결과를 처리하는 방법을 테스트할 필요가 없습니다. 테스트는 모의에서 직접 LRO의 최종 결과를 반환할 수 있습니다.

longrunning::model::Operation 만들기

호출에서 다음 응답을 반환한다고 가정해 보겠습니다.

fn expected_duration() -> Option<wkt::Duration> {
    Some(wkt::Duration::clamp(100, 0))
}

fn expected_response() -> BatchRecognizeResponse {
    BatchRecognizeResponse::new().set_or_clear_total_billed_duration(expected_duration())
}

스텁은 BatchRecognizeResponse가 아닌 longrunning::model::Operation를 반환합니다. 예상 응답을 Operation::result에 패킹합니다.

fn make_finished_operation(response: &BatchRecognizeResponse) -> Result<Response<Operation>> {
    let any = wkt::Any::from_msg(response).expect("test message should succeed");
    let operation = Operation::new()
        .set_done(true)
        .set_result(OperationResult::Response(any.into()));
    Ok(Response::from(operation))
}

done 필드를 true로 설정하여 Poller에 작업이 완료되었음을 알리고 폴링 루프를 종료합니다.

.set_done(true)

테스트 작성

  1. speech::stub::Speech 특성을 구현하는 모의 구조체를 정의합니다.

    mockall::mock! {
        #[derive(Debug)]
        Speech {}
        impl speech::stub::Speech for Speech {
            async fn batch_recognize(&self, req: BatchRecognizeRequest, _options: gax::options::RequestOptions) -> Result<Response<Operation>>;
        }
    }

  2. 테스트에서 모의 객체를 만들고 이에 대한 기대치를 설정합니다.

    let mut mock = MockSpeech::new();
    mock.expect_batch_recognize()
        .return_once(|_, _| make_finished_operation(&expected_response()));

  3. 모의에서 클라이언트를 만들고 함수를 호출하고 응답을 확인합니다.

    // Create a client, implemented by our mock.
    let client = speech::client::Speech::from_stub(mock);
    
    // Call our function which automatically polls.
    let billed_duration = my_automatic_poller(&client, "my-project").await?;
    
    // Verify the final result of the LRO.
    assert_eq!(billed_duration, expected_duration());

중간 메타데이터를 사용한 수동 폴링 테스트

애플리케이션 코드가 수동으로 폴링하고 부분 업데이트를 처리한다고 가정해 보겠습니다.

pub struct BatchRecognizeResult {
    pub progress_updates: Vec<i32>,
    pub billed_duration: Result<Option<wkt::Duration>>,
}

// An example application function that manually polls.
//
// It starts an LRO. It consolidates the polling results, whether full or
// partial.
//
// In this case, it is the `BatchRecognize` RPC. If we get a partial update,
// we extract the `progress_percent` field. If we get a final result, we
// extract the `total_billed_duration` field.
pub async fn my_manual_poller(
    client: &speech::client::Speech,
    project_id: &str,
) -> BatchRecognizeResult {
    use google_cloud_lro::{Poller, PollingResult};
    let mut progress_updates = Vec::new();
    let mut poller = client
        .batch_recognize()
        .set_recognizer(format!(
            "projects/{project_id}/locations/global/recognizers/_"
        ))
        .poller();
    while let Some(p) = poller.poll().await {
        match p {
            PollingResult::Completed(r) => {
                let billed_duration = r.map(|r| r.total_billed_duration);
                return BatchRecognizeResult {
                    progress_updates,
                    billed_duration,
                };
            }
            PollingResult::InProgress(m) => {
                if let Some(metadata) = m {
                    // This is a silly application. Your application likely
                    // performs some task immediately with the partial
                    // update, instead of storing it for after the operation
                    // has completed.
                    progress_updates.push(metadata.progress_percent);
                }
            }
            PollingResult::PollingError(e) => {
                return BatchRecognizeResult {
                    progress_updates,
                    billed_duration: Err(e),
                };
            }
        }
        tokio::time::sleep(std::time::Duration::from_millis(500)).await;
    }

    // We can only get here if `poll()` returns `None`, but it only returns
    // `None` after it returned `PollingResult::Completed`. Therefore this
    // is never reached.
    unreachable!("loop should exit via the `Completed` branch.");
}

모의에서 진행 중인 작업을 반환하여 중간 메타데이터를 수신할 때 애플리케이션이 어떻게 작동하는지 시뮬레이션할 수 있습니다.

longrunning::model::Operation 만들기

BatchRecognize RPC는 부분 결과를 speech::model::OperationMetadata로 반환합니다. 이 메타데이터를 반환된 longrunning::model::Operation의 Operation::metadata 필드에 패킹합니다.

fn make_partial_operation(progress: i32) -> Result<Response<Operation>> {
    let metadata = OperationMetadata::new().set_progress_percent(progress);
    let any = wkt::Any::from_msg(&metadata).expect("test message should succeed");
    let operation = Operation::new().set_metadata(any);
    Ok(Response::from(operation))
}

테스트 작성

  1. speech::stub::Speech 특성을 구현하고 get_operation()를 재정의하는 모의 구조체를 정의합니다.

    mockall::mock! {
        #[derive(Debug)]
        Speech {}
        impl speech::stub::Speech for Speech {
            async fn batch_recognize(&self, req: BatchRecognizeRequest, _options: gax::options::RequestOptions) -> Result<Response<Operation>>;
            async fn get_operation(&self, req: GetOperationRequest, _options: gax::options::RequestOptions) -> Result<Response<Operation>>;
        }
    }

  2. 테스트에서 모의 객체를 만들고 부분 결과(25%, 50%, 75%)를 반환한 후 최종 결과를 반환하도록 기대치를 설정합니다.

    let mut seq = mockall::Sequence::new();
    let mut mock = MockSpeech::new();
    mock.expect_batch_recognize()
        .once()
        .in_sequence(&mut seq)
        .returning(|_, _| make_partial_operation(25));
    mock.expect_get_operation()
        .once()
        .in_sequence(&mut seq)
        .returning(|_, _| make_partial_operation(50));
    mock.expect_get_operation()
        .once()
        .in_sequence(&mut seq)
        .returning(|_, _| make_partial_operation(75));
    mock.expect_get_operation()
        .once()
        .in_sequence(&mut seq)
        .returning(|_, _| make_finished_operation(&expected_response()));

    기대치가 구성되는 방식에 관한 다음 세부정보를 참고하세요.

    • 초기 RPC와 후속 RPC: 첫 번째 기대치는 batch_recognize()에 설정되고 이후의 모든 기대치는 get_operation()에 설정됩니다. 초기 BatchRecognize RPC는 서버에서 LRO를 시작하고 name 필드에 작업 식별자를 반환합니다(간결성을 위해 테스트 코드에서 생략됨). 그 이후로 클라이언트 라이브러리는 GetOperation RPC를 사용하여 해당 LRO의 상태를 폴링합니다.
    • 호출 순서: 기대치는 Sequence에 등록됩니다. 이렇게 하면 mockall가 호출 순서를 확인하고 각 폴링과 일치하는 expect_get_operation 기대치를 결정할 수 있습니다.
  3. 모의에서 클라이언트를 만들고 함수를 호출하고 응답을 확인합니다.

    // Create a client, implemented by our mock.
    let client = speech::client::Speech::from_stub(mock);
    
    // Call our function which manually polls.
    let result = my_manual_poller(&client, "my-project").await;
    
    // Verify the partial metadata updates, and the final result.
    assert_eq!(result.progress_updates, [25, 50, 75]);
    assert_eq!(result.billed_duration?, expected_duration());

오류 시뮬레이션

LRO 중에 여러 지점에서 오류가 발생할 수 있습니다.

애플리케이션에서 자동 폴링을 사용하는 경우 모든 오류 소스는 동일합니다. until_done()는 발생 위치에 관계없이 Result에서 오류를 반환합니다. LRO를 시작하는 오류를 시뮬레이션하면 가장 간단한 테스트가 생성됩니다.

LRO 시작 오류 시뮬레이션

LRO를 시작할 때 오류를 시뮬레이션하려면 초기 요청이 오류와 함께 실패하도록 구성합니다.

mock.expect_batch_recognize().return_once(|_, _| {
    use gax::error::Error;
    use gax::error::rpc::{Code, Status};
    let status = Status::default()
        .set_code(Code::Aborted)
        .set_message("Resource exhausted");
    Err(Error::service(status))
});

수동 폴링의 경우 PollingResult::Completed 브랜치는 LRO 시작에서 오류를 수신하여 폴링 루프를 종료합니다.

PollingResult::Completed(r) => {

오류를 일으키는 LRO 시뮬레이션

중간 메타데이터를 반환한 후 실패하는 LRO를 시뮬레이션하려면 최종 longrunning::model::Operation에서 오류를 반환합니다.

fn make_failed_operation(status: rpc::model::Status) -> Result<Response<Operation>> {
    let operation = Operation::new()
        .set_done(true)
        .set_result(OperationResult::Error(status.into()));
    Ok(Response::from(operation))
}

get_operation에서 실패한 Operation를 반환하도록 기대치를 설정합니다.

mock.expect_get_operation()
    .once()
    .in_sequence(&mut seq)
    .returning(|_, _| {
        // This is a common error for `Create*` RPCs, which are often
        // LROs. It is less applicable to `BatchRecognize` in practice.
        let status = rpc::model::Status::default()
            .set_code(gax::error::rpc::Code::AlreadyExists as i32)
            .set_message("resource already exists");
        make_failed_operation(status)
    });

수동 폴링에서 PollingResult::Completed 브랜치는 오류 결과를 수신하여 폴링 루프를 종료합니다.

PollingResult::Completed(r) => {

폴링 오류 시뮬레이션

폴링 정책이 소진되면 폴링 루프가 종료될 수도 있습니다. 이 경우 클라이언트 라이브러리는 LRO가 완료되었는지 확실하게 확인할 수 없습니다.

애플리케이션에 폴링 오류를 처리하는 커스텀 로직이 있는 경우 get_operation 예상에서 직접 오류를 반환하여 테스트할 수 있습니다.

mock.expect_get_operation()
    .once()
    .in_sequence(&mut seq)
    .returning(|_, _| {
        use gax::error::Error;
        use gax::error::rpc::{Code, Status};
        let status = Status::default()
            .set_code(Code::Aborted)
            .set_message("Operation was aborted");
        Err(Error::service(status))
    });

수동 폴링에서 PollingResult::PollingError 브랜치는 시뮬레이션된 폴링 오류를 수신합니다.

PollingResult::PollingError(e) => {

테스트 코드 작성 완료

다음 섹션에서는 이 가이드에 설명된 테스트의 전체 코드를 보여줍니다.

자동 폴링: 전체 테스트

use gax::Result;
use gax::response::Response;
use google_cloud_gax as gax;
use google_cloud_longrunning as longrunning;
use google_cloud_speech_v2 as speech;
use google_cloud_wkt as wkt;
use longrunning::model::Operation;
use longrunning::model::operation::Result as OperationResult;
use speech::model::{BatchRecognizeRequest, BatchRecognizeResponse};

// Example application code that is under test
mod my_application {
    use super::*;

    // An example application function that automatically polls.
    //
    // It starts an LRO, awaits the result, and processes it.
    pub async fn my_automatic_poller(
        client: &speech::client::Speech,
        project_id: &str,
    ) -> Result<Option<wkt::Duration>> {
        use google_cloud_lro::Poller;
        client
            .batch_recognize()
            .set_recognizer(format!(
                "projects/{project_id}/locations/global/recognizers/_"
            ))
            .poller()
            .until_done()
            .await
            .map(|r| r.total_billed_duration)
    }
}

#[cfg(test)]
mod tests {
    use super::my_application::*;
    use super::*;

    mockall::mock! {
        #[derive(Debug)]
        Speech {}
        impl speech::stub::Speech for Speech {
            async fn batch_recognize(&self, req: BatchRecognizeRequest, _options: gax::options::RequestOptions) -> Result<Response<Operation>>;
        }
    }

    fn expected_duration() -> Option<wkt::Duration> {
        Some(wkt::Duration::clamp(100, 0))
    }

    fn expected_response() -> BatchRecognizeResponse {
        BatchRecognizeResponse::new().set_or_clear_total_billed_duration(expected_duration())
    }

    fn make_finished_operation(response: &BatchRecognizeResponse) -> Result<Response<Operation>> {
        let any = wkt::Any::from_msg(response).expect("test message should succeed");
        let operation = Operation::new()
            .set_done(true)
            .set_result(OperationResult::Response(any.into()));
        Ok(Response::from(operation))
    }

    #[tokio::test]
    async fn automatic_polling() -> Result<()> {
        // Create a mock, and set expectations on it.
        let mut mock = MockSpeech::new();
        mock.expect_batch_recognize()
            .return_once(|_, _| make_finished_operation(&expected_response()));

        // Create a client, implemented by our mock.
        let client = speech::client::Speech::from_stub(mock);

        // Call our function which automatically polls.
        let billed_duration = my_automatic_poller(&client, "my-project").await?;

        // Verify the final result of the LRO.
        assert_eq!(billed_duration, expected_duration());

        Ok(())
    }
}

중간 메타데이터를 사용한 수동 폴링: 전체 테스트

use gax::Result;
use gax::response::Response;
use google_cloud_gax as gax;
use google_cloud_longrunning as longrunning;
use google_cloud_speech_v2 as speech;
use google_cloud_wkt as wkt;
use longrunning::model::operation::Result as OperationResult;
use longrunning::model::{GetOperationRequest, Operation};
use speech::model::{BatchRecognizeRequest, BatchRecognizeResponse, OperationMetadata};

// Example application code that is under test
mod my_application {
    use super::*;

    pub struct BatchRecognizeResult {
        pub progress_updates: Vec<i32>,
        pub billed_duration: Result<Option<wkt::Duration>>,
    }

    // An example application function that manually polls.
    //
    // It starts an LRO. It consolidates the polling results, whether full or
    // partial.
    //
    // In this case, it is the `BatchRecognize` RPC. If we get a partial update,
    // we extract the `progress_percent` field. If we get a final result, we
    // extract the `total_billed_duration` field.
    pub async fn my_manual_poller(
        client: &speech::client::Speech,
        project_id: &str,
    ) -> BatchRecognizeResult {
        use google_cloud_lro::{Poller, PollingResult};
        let mut progress_updates = Vec::new();
        let mut poller = client
            .batch_recognize()
            .set_recognizer(format!(
                "projects/{project_id}/locations/global/recognizers/_"
            ))
            .poller();
        while let Some(p) = poller.poll().await {
            match p {
                PollingResult::Completed(r) => {
                    let billed_duration = r.map(|r| r.total_billed_duration);
                    return BatchRecognizeResult {
                        progress_updates,
                        billed_duration,
                    };
                }
                PollingResult::InProgress(m) => {
                    if let Some(metadata) = m {
                        // This is a silly application. Your application likely
                        // performs some task immediately with the partial
                        // update, instead of storing it for after the operation
                        // has completed.
                        progress_updates.push(metadata.progress_percent);
                    }
                }
                PollingResult::PollingError(e) => {
                    return BatchRecognizeResult {
                        progress_updates,
                        billed_duration: Err(e),
                    };
                }
            }
            tokio::time::sleep(std::time::Duration::from_millis(500)).await;
        }

        // We can only get here if `poll()` returns `None`, but it only returns
        // `None` after it returned `PollingResult::Completed`. Therefore this
        // is never reached.
        unreachable!("loop should exit via the `Completed` branch.");
    }
}

#[cfg(test)]
mod tests {
    use super::my_application::*;
    use super::*;

    mockall::mock! {
        #[derive(Debug)]
        Speech {}
        impl speech::stub::Speech for Speech {
            async fn batch_recognize(&self, req: BatchRecognizeRequest, _options: gax::options::RequestOptions) -> Result<Response<Operation>>;
            async fn get_operation(&self, req: GetOperationRequest, _options: gax::options::RequestOptions) -> Result<Response<Operation>>;
        }
    }

    fn expected_duration() -> Option<wkt::Duration> {
        Some(wkt::Duration::clamp(100, 0))
    }

    fn expected_response() -> BatchRecognizeResponse {
        BatchRecognizeResponse::new().set_or_clear_total_billed_duration(expected_duration())
    }

    fn make_finished_operation(
        response: &BatchRecognizeResponse,
    ) -> Result<gax::response::Response<Operation>> {
        let any = wkt::Any::from_msg(response).expect("test message should succeed");
        let operation = Operation::new()
            .set_done(true)
            .set_result(OperationResult::Response(any.into()));
        Ok(Response::from(operation))
    }

    fn make_partial_operation(progress: i32) -> Result<Response<Operation>> {
        let metadata = OperationMetadata::new().set_progress_percent(progress);
        let any = wkt::Any::from_msg(&metadata).expect("test message should succeed");
        let operation = Operation::new().set_metadata(any);
        Ok(Response::from(operation))
    }

    #[tokio::test]
    async fn manual_polling_with_metadata() -> Result<()> {
        let mut seq = mockall::Sequence::new();
        let mut mock = MockSpeech::new();
        mock.expect_batch_recognize()
            .once()
            .in_sequence(&mut seq)
            .returning(|_, _| make_partial_operation(25));
        mock.expect_get_operation()
            .once()
            .in_sequence(&mut seq)
            .returning(|_, _| make_partial_operation(50));
        mock.expect_get_operation()
            .once()
            .in_sequence(&mut seq)
            .returning(|_, _| make_partial_operation(75));
        mock.expect_get_operation()
            .once()
            .in_sequence(&mut seq)
            .returning(|_, _| make_finished_operation(&expected_response()));

        // Create a client, implemented by our mock.
        let client = speech::client::Speech::from_stub(mock);

        // Call our function which manually polls.
        let result = my_manual_poller(&client, "my-project").await;

        // Verify the partial metadata updates, and the final result.
        assert_eq!(result.progress_updates, [25, 50, 75]);
        assert_eq!(result.billed_duration?, expected_duration());

        Ok(())
    }
}

오류 시뮬레이션: 전체 테스트

use gax::Result;
use gax::response::Response;
use google_cloud_gax as gax;
use google_cloud_longrunning as longrunning;
use google_cloud_rpc as rpc;
use google_cloud_speech_v2 as speech;
use google_cloud_wkt as wkt;
use longrunning::model::operation::Result as OperationResult;
use longrunning::model::{GetOperationRequest, Operation};
use speech::model::{BatchRecognizeRequest, OperationMetadata};

// Example application code that is under test
mod my_application {
    use super::*;

    pub struct BatchRecognizeResult {
        pub progress_updates: Vec<i32>,
        pub billed_duration: Result<Option<wkt::Duration>>,
    }

    // An example application function that manually polls.
    //
    // It starts an LRO. It consolidates the polling results, whether full or
    // partial.
    //
    // In this case, it is the `BatchRecognize` RPC. If we get a partial update,
    // we extract the `progress_percent` field. If we get a final result, we
    // extract the `total_billed_duration` field.
    pub async fn my_manual_poller(
        client: &speech::client::Speech,
        project_id: &str,
    ) -> BatchRecognizeResult {
        use google_cloud_lro::{Poller, PollingResult};
        let mut progress_updates = Vec::new();
        let mut poller = client
            .batch_recognize()
            .set_recognizer(format!(
                "projects/{project_id}/locations/global/recognizers/_"
            ))
            .poller();
        while let Some(p) = poller.poll().await {
            match p {
                PollingResult::Completed(r) => {
                    let billed_duration = r.map(|r| r.total_billed_duration);
                    return BatchRecognizeResult {
                        progress_updates,
                        billed_duration,
                    };
                }
                PollingResult::InProgress(m) => {
                    if let Some(metadata) = m {
                        // This is a silly application. Your application likely
                        // performs some task immediately with the partial
                        // update, instead of storing it for after the operation
                        // has completed.
                        progress_updates.push(metadata.progress_percent);
                    }
                }
                PollingResult::PollingError(e) => {
                    return BatchRecognizeResult {
                        progress_updates,
                        billed_duration: Err(e),
                    };
                }
            }
            tokio::time::sleep(std::time::Duration::from_millis(500)).await;
        }

        // We can only get here if `poll()` returns `None`, but it only returns
        // `None` after it returned `PollingResult::Completed`. Therefore this
        // is never reached.
        unreachable!("loop should exit via the `Completed` branch.");
    }
}

#[cfg(test)]
mod tests {
    use super::my_application::*;
    use super::*;

    mockall::mock! {
        #[derive(Debug)]
        Speech {}
        impl speech::stub::Speech for Speech {
            async fn batch_recognize(&self, req: BatchRecognizeRequest, _options: gax::options::RequestOptions) -> Result<Response<Operation>>;
            async fn get_operation(&self, req: GetOperationRequest, _options: gax::options::RequestOptions) -> Result<Response<Operation>>;
        }
    }

    fn make_partial_operation(progress: i32) -> Result<Response<Operation>> {
        let metadata = OperationMetadata::new().set_progress_percent(progress);
        let any = wkt::Any::from_msg(&metadata).expect("test message should succeed");
        let operation = Operation::new().set_metadata(any);
        Ok(Response::from(operation))
    }

    fn make_failed_operation(status: rpc::model::Status) -> Result<Response<Operation>> {
        let operation = Operation::new()
            .set_done(true)
            .set_result(OperationResult::Error(status.into()));
        Ok(Response::from(operation))
    }

    #[tokio::test]
    async fn error_starting_lro() -> Result<()> {
        let mut mock = MockSpeech::new();
        mock.expect_batch_recognize().return_once(|_, _| {
            use gax::error::Error;
            use gax::error::rpc::{Code, Status};
            let status = Status::default()
                .set_code(Code::Aborted)
                .set_message("Resource exhausted");
            Err(Error::service(status))
        });

        // Create a client, implemented by our mock.
        let client = speech::client::Speech::from_stub(mock);

        // Call our function which manually polls.
        let result = my_manual_poller(&client, "my-project").await;

        // Verify the the final result.
        assert!(
            result.billed_duration.is_err(),
            "{:?}",
            result.billed_duration
        );

        Ok(())
    }

    #[tokio::test]
    async fn lro_ending_in_error() -> Result<()> {
        let mut seq = mockall::Sequence::new();
        let mut mock = MockSpeech::new();
        mock.expect_batch_recognize()
            .once()
            .in_sequence(&mut seq)
            .returning(|_, _| make_partial_operation(25));
        mock.expect_get_operation()
            .once()
            .in_sequence(&mut seq)
            .returning(|_, _| make_partial_operation(50));
        mock.expect_get_operation()
            .once()
            .in_sequence(&mut seq)
            .returning(|_, _| make_partial_operation(75));
        mock.expect_get_operation()
            .once()
            .in_sequence(&mut seq)
            .returning(|_, _| {
                // This is a common error for `Create*` RPCs, which are often
                // LROs. It is less applicable to `BatchRecognize` in practice.
                let status = rpc::model::Status::default()
                    .set_code(gax::error::rpc::Code::AlreadyExists as i32)
                    .set_message("resource already exists");
                make_failed_operation(status)
            });

        // Create a client, implemented by our mock.
        let client = speech::client::Speech::from_stub(mock);

        // Call our function which manually polls.
        let result = my_manual_poller(&client, "my-project").await;

        // Verify the partial metadata updates, and the final result.
        assert_eq!(result.progress_updates, [25, 50, 75]);
        assert!(
            result.billed_duration.is_err(),
            "{:?}",
            result.billed_duration
        );

        Ok(())
    }

    #[tokio::test]
    async fn polling_loop_error() -> Result<()> {
        let mut seq = mockall::Sequence::new();
        let mut mock = MockSpeech::new();
        mock.expect_batch_recognize()
            .once()
            .in_sequence(&mut seq)
            .returning(|_, _| make_partial_operation(25));
        mock.expect_get_operation()
            .once()
            .in_sequence(&mut seq)
            .returning(|_, _| {
                use gax::error::Error;
                use gax::error::rpc::{Code, Status};
                let status = Status::default()
                    .set_code(Code::Aborted)
                    .set_message("Operation was aborted");
                Err(Error::service(status))
            });

        // Create a client, implemented by our mock.
        let client = speech::client::Speech::from_stub(mock);

        // Call our function which manually polls.
        let result = my_manual_poller(&client, "my-project").await;

        // Verify the partial metadata updates, and the final result.
        assert_eq!(result.progress_updates, [25]);
        assert!(
            result.billed_duration.is_err(),
            "{:?}",
            result.billed_duration
        );

        Ok(())
    }
}