> [!NOTE]
> **Note:** Automated restore verification is generally available (GA) with an allowlist. To use this feature, request access through your [Technical Account Manager (TAM)](https://cloud.google.com/tam) or [create a support case with Cloud Customer Care](https://docs.cloud.google.com/support/docs/customer-care-procedures.md.txt#create_a_support_case).

Automated restore verification validates backup recoverability by scheduling
restores and running validation scripts on temporary instances. This feature
helps you verify that backups can be successfully restored, validate recovery
time objectives (RTO), and satisfy data governance requirements.

## Overview of automated restore verification

To validate backup recoverability for Compute Engine instances, Persistent Disk
and Google Cloud Hyperdisk volumes, and Cloud SQL instances, you define how to
restore and validate your data by using *restore templates* . Each restore
template specifies restore parameters and optional pre-restore, post-restore,
and verification scripts that run as Cloud Run jobs. A restore
template maps to a specific *data source*, which represents a backed-up
resource in a backup vault, rather than to a single backup. While a template is
tied to one data source, a single data source can have multiple templates for
different testing scenarios.

To automate testing, you create *restore verification plans* that define the
execution schedule and lifecycle policies, such as how long temporary test
resources are retained. You link a verification plan to a restore template by
using a *restore verification plan association*. A single plan can manage
multiple associations, enabling complex testing schedules.

When an automated run or an on-demand test triggers, a
*restore template execution* is created. By default, this process orchestrates
the restoration of the latest backup from the data source to a temporary test
instance.

### Architecture and isolation

To enforce security isolation, automated restore verification lets you separate
responsibilities across distinct project roles:

- **Source resource project:** Contains the active production resources being
  backed up.

- **Backup vault project:** Stores the immutable backups and data sources in a
  backup vault, along with your restore templates. The restore template and the
  backup vault containing the data source must reside in the same project.

- **Restore orchestration project:** Serves as the scheduling control plane,
  storing your restore verification plans and restore verification plan
  associations. These resources can reside in a different project from the
  restore template and backup vault.

- **Target resource project:** Acts as the sandbox where temporary test
  resources are provisioned during verification executions, and can reside in a
  separate project.

- **Cloud Run jobs project:** Hosts optional pre-restore,
  post-restore, and verification Cloud Run jobs, which can also
  reside in a separate project.

### Regional requirements

The restore verification plan, restore verification plan association, restore
template, provisioned target resource, and Cloud Run jobs must
all reside in the same Google Cloud region. However, to support cross-region
disaster recovery topologies, the backup vault containing the data source can
reside in a different region. The backup vault and data source must still
reside in the same project as the restore template.

## Limitations

Automated restore verification has the following limitations:

- **Supported workloads:** Automated restore verification supports only the
  following workload resource types:

  - Compute Engine instances (`compute.googleapis.com/Instance`)
  - Zonal Persistent Disk and Hyperdisk volumes (`compute.googleapis.com/Disk`)
  - Regional Persistent Disk and Hyperdisk volumes (`compute.googleapis.com/RegionDisk`)
  - Cloud SQL instances (`sqladmin.googleapis.com/Instance`)
- **Encryption key downgrade not supported:** Restoring a backup encrypted with
  [customer-managed encryption keys (CMEK)](https://docs.cloud.google.com/backup-disaster-recovery/docs/concepts/cmek.md.txt)
  to a target instance or disk encrypted with
  Google-owned and Google-managed encryption keys (GMEK) isn't supported for
  Compute Engine instance and disk (Persistent Disk and
  Hyperdisk) workloads.

- **Selective disk restore not supported:**
  [Selective disk restores](https://docs.cloud.google.com/backup-disaster-recovery/docs/cloud-console/compute/disk-restore.md.txt#restore-disk-from-compute-instance-backup)
  (restoring individual disks from a Compute Engine instance backup)
  aren't supported for Compute Engine instance and disk workloads.

- **Management interfaces:** Creating, updating, deleting, and triggering
  restore templates, restore verification plans, plan associations, and template
  executions through the Google Cloud console or Terraform aren't supported.

## Before you begin

Before you begin, complete the following prerequisites:

- **Project allowlist:** Automated restore verification is available only for
  projects on an allowlist. To add your project to the allowlist, request
  access through your [TAM](https://cloud.google.com/tam) or
  [create a support case with Customer Care](https://docs.cloud.google.com/support/docs/customer-care-procedures.md.txt#create_a_support_case).

- **Backup and DR service enabled:** The Backup and DR API
  must be enabled in the source resource, backup vault, restore
  orchestration, and target resource projects.

- **Backup vault configuration:** A functional backup vault must be active
  and assigned to protect the primary workload resources.

- **Active workload backups:** Active, valid backup snapshots must exist for the
  target Compute Engine, Persistent Disk, Hyperdisk,
  or Cloud SQL resources.

- **Cloud Run hook scripts (Optional):** If you intend to run
  pre-restore, post-restore, or verification hooks, build and deploy your
  script containers as Cloud Run jobs in the same region as
  your restore template.

## Get required roles


To get the permissions that
you need to set up and manage automated restore verification,

ask your administrator to grant you the
following IAM roles on your project:

- Full administration: [Backup and DR Admin](https://docs.cloud.google.com/iam/docs/roles-permissions/backupdr.md.txt#backupdr.admin) (`roles/backupdr.admin`)
- Create and manage backups and restores: [Backup and DR User V2](https://docs.cloud.google.com/iam/docs/roles-permissions/backupdr.md.txt#backupdr.userv2) (`roles/backupdr.userv2`)
- Run verification and restores: [Backup and DR Restore User](https://docs.cloud.google.com/iam/docs/roles-permissions/backupdr.md.txt#backupdr.restoreUser) (`roles/backupdr.restoreUser`)
- View resources only: [Backup and DR Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/backupdr.md.txt#backupdr.viewer) (`roles/backupdr.viewer`)


For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).


You might also be able to get
the required permissions through [custom
roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) or other [predefined
roles](https://docs.cloud.google.com/iam/docs/roles-overview#predefined).

#### Granular permissions for custom roles

To use custom roles instead of predefined roles, include the granular
permissions listed in the following table in your custom roles:

| Target resource | Action | Required permission |
|---|---|---|
| **Restore Template** | Create | `backupdr.restoreTemplates.createForResourceType` `backupdr.bvdataSources.get` and `backupdr.bvdataSources.useReadOnlyForResourceType` (on the data source) `backupdr.compute.restoreFromBackupVault` (on the target project for Compute Engine, Persistent Disk, and Hyperdisk) `backupdr.restoreTargets.useForCloudSqlInstance` (on the target project for Cloud SQL) `run.jobs.get` and `run.jobs.runWithOverrides` (on any configured Cloud Run jobs) |
| **Restore Template** | Update | `backupdr.restoreTemplates.updateForResourceType` `backupdr.compute.restoreFromBackupVault` (on the target project for Compute Engine, Persistent Disk, and Hyperdisk) `backupdr.restoreTargets.useForCloudSqlInstance` (on the target project for Cloud SQL) `run.jobs.get` and `run.jobs.runWithOverrides` (on any configured Cloud Run jobs) |
| **Restore Template** | Delete | `backupdr.restoreTemplates.deleteForResourceType` |
| **Restore Template** | Get | `backupdr.restoreTemplates.getForResourceType` |
| **Restore Template** | List | `backupdr.restoreTemplates.list` |
| **Restore Template** | Run | `backupdr.restoreTemplates.runForResourceType` |
| **Restore Verification Plan** | Create | `backupdr.restoreVerificationPlans.create` |
| **Restore Verification Plan** | Delete | `backupdr.restoreVerificationPlans.delete` |
| **Restore Verification Plan** | List | `backupdr.restoreVerificationPlans.list` |
| **Restore Verification Plan** | Get | `backupdr.restoreVerificationPlans.get` |
| **Restore Verification Plan Association** | Create | `backupdr.restoreVerificationPlanAssociations.createForResourceType` (on the plans project) `backupdr.restoreTemplates.useForResourceType` (on the template project) |
| **Restore Verification Plan Association** | Delete | `backupdr.restoreVerificationPlanAssociations.deleteForResourceType` |
| **Restore Verification Plan Association** | Get | `backupdr.restoreVerificationPlanAssociations.getForResourceType` |
| **Restore Verification Plan Association** | List | `backupdr.restoreVerificationPlanAssociations.list` |
| **Restore Verification Plan Association** | Trigger | `backupdr.restoreVerificationPlanAssociations.triggerRestoreVerificationForResourceType` |
| **Restore Template Execution** | Get | `backupdr.restoreTemplateExecutions.getForResourceType` |
| **Restore Template Execution** | List | `backupdr.restoreTemplateExecutions.list` |
| **Restore Template Execution** | Trigger cleanup | `backupdr.restoreTemplateExecutions.triggerRestoreCleanupForResourceType` |

### Grant roles to service agents

To let verification runs execute without failing with `PERMISSION_DENIED`
errors, grant the following roles to the Backup and DR service agents:

- **Backup vault service agent**
  (`service-VAULT_PROJECT_NUMBER@gcp-sa-backupdr-pr.iam.gserviceaccount.com`),
  where `VAULT_PROJECT_NUMBER` is the project number of the
  backup vault project:

  - On the **target resource project**, grant the workload-specific role that
    lets the service agent create and delete temporary test resources:

    - **Compute Engine instances** : Compute Editor (`roles/compute.editor`)
    - **Persistent Disk and Hyperdisk volumes** : Compute Editor (`roles/compute.editor`)
    - **Cloud SQL instances** : Cloud SQL Admin (`roles/cloudsql.admin`)
  - On the **Cloud Run jobs project** (if you configure
    pre-restore, post-restore, or verification jobs), grant the
    Cloud Run Developer (`roles/run.developer`) role.

- **Plans project service agent**
  (`service-PLANS_PROJECT_NUMBER@gcp-sa-backupdr-pr.iam.gserviceaccount.com`),
  where `PLANS_PROJECT_NUMBER` is the project number of the
  restore orchestration project:

  - If the restore verification plan is in a different project from the restore template, grant the Backup and DR Service Agent (`roles/backupdr.serviceAgent`) role on the **restore template project** (the backup vault project).

To grant these roles to the service agents, use the Google Cloud console or
the [gcloud CLI](https://docs.cloud.google.com/sdk/gcloud.md.txt):

### Console

1. In the Google Cloud console, go to the **IAM** page:

   [Go to IAM](https://console.cloud.google.com/iam-admin/iam)
2. Select the **target resource project**.

3. Click **Grant access**.

4. In the **New principals** field, enter
   `service-VAULT_PROJECT_NUMBER@gcp-sa-backupdr-pr.iam.gserviceaccount.com`,
   where `VAULT_PROJECT_NUMBER` is the project number of
   the backup vault project.

5. In the **Select a role** menu, select the role for your target workload:

   - For Compute Engine instances or Persistent Disk and Hyperdisk volumes, select **Compute Editor** (`roles/compute.editor`).
   - For Cloud SQL instances, select **Cloud SQL Admin** (`roles/cloudsql.admin`).
6. Click **Save**.

7. If you configure pre-restore, post-restore, or verification
   Cloud Run jobs, select the **Cloud Run jobs
   project** , click **Grant access** , enter
   `service-VAULT_PROJECT_NUMBER@gcp-sa-backupdr-pr.iam.gserviceaccount.com`
   in the **New principals** field, select the **Cloud Run
   Developer** (`roles/run.developer`) role, and then click **Save**.

8. If the restore verification plan is in a different project from the
   restore template, select the **backup vault project** (the restore
   template project), click **Grant access** , enter
   `service-PLANS_PROJECT_NUMBER@gcp-sa-backupdr-pr.iam.gserviceaccount.com`
   in the **New principals** field, select the **Backup and DR
   Service Agent** (`roles/backupdr.serviceAgent`) role, and then click
   **Save**.

### gcloud

1. On the target resource project, grant the workload-specific role
   (`roles/compute.editor` for Compute Engine and disk workloads, or
   `roles/cloudsql.admin` for Cloud SQL workloads) to the backup
   vault service agent:

   ```bash
   gcloud projects add-iam-policy-binding TARGET_PROJECT_ID \
       --member="serviceAccount:service-VAULT_PROJECT_NUMBER@gcp-sa-backupdr-pr.iam.gserviceaccount.com" \
       --role="TARGET_WORKLOAD_ROLE"
   ```
2. If you configure pre-restore, post-restore, or verification
   Cloud Run jobs, grant the Cloud Run
   Developer (`roles/run.developer`) role to the backup vault service agent
   on the Cloud Run jobs project:

   ```bash
   gcloud projects add-iam-policy-binding CLOUD_RUN_PROJECT_ID \
       --member="serviceAccount:service-VAULT_PROJECT_NUMBER@gcp-sa-backupdr-pr.iam.gserviceaccount.com" \
       --role="roles/run.developer"
   ```
3. If the restore verification plan is in a different project from the
   restore template, grant the Backup and DR Service Agent
   (`roles/backupdr.serviceAgent`) role to the plans project service agent
   on the restore template project (the backup vault project):

   ```bash
   gcloud projects add-iam-policy-binding VAULT_PROJECT_ID \
       --member="serviceAccount:service-PLANS_PROJECT_NUMBER@gcp-sa-backupdr-pr.iam.gserviceaccount.com" \
       --role="roles/backupdr.serviceAgent"
   ```

   Replace the following:
   - `TARGET_PROJECT_ID`: the project ID of the target
     resource project.

   - `VAULT_PROJECT_NUMBER`: the project number of the
     backup vault project.

   - `TARGET_WORKLOAD_ROLE`: the role for the target
     workload (`roles/compute.editor` for Compute Engine instances
     and Persistent Disk or Hyperdisk volumes, or
     `roles/cloudsql.admin` for Cloud SQL instances).

   - `CLOUD_RUN_PROJECT_ID`: the project ID of the
     Cloud Run jobs project.

   - `VAULT_PROJECT_ID`: the project ID of the restore
     template project (the backup vault project).

   - `PLANS_PROJECT_NUMBER`: the project number of the
     restore orchestration project.

## Create a restore template

A restore template defines the parameters for the restore operation, including
the target project and optional custom hooks. You can link a restore template to
a restore verification plan for scheduled runs, or run the template directly for
on-demand testing.

> [!NOTE]
> **Note:** Creating, updating, deleting, and triggering restore templates, restore verification plans, plan associations, and template executions aren't supported in the Google Cloud console or Terraform. To run these commands from the Google Cloud console, click **Activate Cloud Shell** in the toolbar to open Cloud Shell.

Create a restore template by using the gcloud CLI:

```bash
gcloud backup-dr restore-templates create RESTORE_TEMPLATE_ID \
    --project=VAULT_PROJECT_ID \
    --location=LOCATION \
    --resource-type="RESOURCE_TYPE" \
    --data-source="projects/VAULT_PROJECT_ID/locations/VAULT_LOCATION/backupVaults/VAULT_ID/dataSources/DATA_SOURCE_ID" \
    --restore-properties='RESTORE_PROPERTIES_JSON' \
    --description="Restore template for automated verification" \
    --pre-restore-cloud-run-job="projects/CLOUD_RUN_PROJECT_ID/locations/LOCATION/jobs/PRE_JOB" \
    --pre-restore-timeout="300s" \
    --post-restore-cloud-run-job="projects/CLOUD_RUN_PROJECT_ID/locations/LOCATION/jobs/POST_JOB" \
    --post-restore-timeout="600s" \
    --verification-cloud-run-job="projects/CLOUD_RUN_PROJECT_ID/locations/LOCATION/jobs/VERIFY_JOB" \
    --verification-timeout="300s"
```

Replace the following:

- `RESTORE_TEMPLATE_ID`: the ID of the restore template to
  create.

- `VAULT_PROJECT_ID`: the project ID of the backup vault
  project.

- `LOCATION`: the region where the restore template and plans
  reside.

- `RESOURCE_TYPE`: the target resource type. Specify one of the
  following supported values:

  - `compute.googleapis.com/Instance` (Compute Engine instance)
  - `compute.googleapis.com/Disk` (zonal Persistent Disk or Hyperdisk volume)
  - `compute.googleapis.com/RegionDisk` (regional Persistent Disk or Hyperdisk volume)
  - `sqladmin.googleapis.com/Instance` (Cloud SQL instance)
- `VAULT_LOCATION`: the region of the backup vault.

- `VAULT_ID`: the ID of the backup vault.

- `DATA_SOURCE_ID`: the ID of the data source.

- `RESTORE_PROPERTIES_JSON`: a JSON object containing the
  workload-specific restore parameters. For the required schema and examples for
  each workload type, see
  [Configure `--restore-properties` by workload type](https://docs.cloud.google.com/backup-disaster-recovery/docs/restore-data/automated-restore-verification.md.txt#restore-properties-by-workload).

- `CLOUD_RUN_PROJECT_ID`: the project ID of the
  Cloud Run jobs project.

- `PRE_JOB`: the name of the optional pre-restore
  Cloud Run job.

- `POST_JOB`: the name of the optional post-restore
  Cloud Run job.

- `VERIFY_JOB`: the name of the optional verification
  Cloud Run job.

### Configure `--restore-properties` by workload type

The `--restore-properties` flag accepts a JSON object that matches the REST API
request structure for the target workload type. Passing the target project,
location (zone or region), and target resource or instance name is mandatory for
all workload types.

In the target resource or instance name field, include the `{EXECUTION_ID}`
dynamic placeholder (for example, `"name": "test-restore-vm-{EXECUTION_ID}"`).
The `{EXECUTION_ID}` placeholder dynamically appends the unique execution ID so
that repeated or overlapping verification runs don't fail due to target resource
name collisions. This placeholder is especially important for
Cloud SQL, where deleted instance names can't be reused right away.

The following table lists the REST API request schema, mandatory fields, and an
example `--restore-properties` JSON payload for each supported workload type:

| Workload resource type | REST API request schema | Mandatory fields | Example `--restore-properties` JSON |
|---|---|---|---|
| **Compute Engine instance** (`compute.googleapis.com/Instance`) | [`instances.insert`](https://docs.cloud.google.com/compute/docs/reference/rest/v1/instances/insert.md.txt) | `project`, `zone`, and `resource.name` | {"project": "TARGET_PROJECT_ID", "zone": "TARGET_ZONE", "resource": {"name": "test-restore-vm-{EXECUTION_ID}", "machineType": "zones/TARGET_ZONE/machineTypes/e2-medium"}} |
| **Zonal disk** (`compute.googleapis.com/Disk`) | [`disks.insert`](https://docs.cloud.google.com/compute/docs/reference/rest/v1/disks/insert.md.txt) | `project`, `zone`, and `resource.name` | {"project": "TARGET_PROJECT_ID", "zone": "TARGET_ZONE", "resource": {"name": "test-restore-disk-{EXECUTION_ID}", "type": "projects/TARGET_PROJECT_ID/zones/TARGET_ZONE/diskTypes/pd-balanced"}} |
| **Regional disk** (`compute.googleapis.com/RegionDisk`) | [`regionDisks.insert`](https://docs.cloud.google.com/compute/docs/reference/rest/v1/regionDisks/insert.md.txt) | `project`, `region`, and `resource.name` | {"project": "TARGET_PROJECT_ID", "region": "TARGET_REGION", "resource": {"name": "test-restore-regdisk-{EXECUTION_ID}", "replicaZones": ["projects/TARGET_PROJECT_ID/zones/REPLICA_ZONE_1", "projects/TARGET_PROJECT_ID/zones/REPLICA_ZONE_2"]}} |
| **Cloud SQL instance** (`sqladmin.googleapis.com/Instance`) | [`instances.restoreBackup`](https://docs.cloud.google.com/sql/docs/mysql/admin-api/rest/v1/instances/restoreBackup.md.txt) | `project`, `instance`, and `body.restoreInstanceSettings.region` | {"project": "TARGET_PROJECT_ID", "instance": "test-restore-sql-{EXECUTION_ID}", "body": {"restoreInstanceSettings": {"region": "TARGET_REGION", "settings": {"tier": "db-custom-2-7680"}}}} |

## Set up restore verification schedules

After you create a restore template, follow these steps to schedule automated
verification runs by using the gcloud CLI:

1. Create a restore verification plan to define the schedule and lifecycle
   policy for automated verification runs:

   ```bash
   gcloud backup-dr restore-verification-plans create PLAN_ID \
       --project=ORCHESTRATION_PROJECT_ID \
       --location=LOCATION \
       --schedule="0 1 * * *" \
       --time-zone="America/New_York" \
       --success-cleanup-delay="1h" \
       --failure-cleanup-delay="24h" \
       --description="Daily restore verification plan"
   ```

   Replace the following:
   - `PLAN_ID`: the ID of the restore verification plan to
     create.

   - `ORCHESTRATION_PROJECT_ID`: the project ID of the restore
     orchestration project.

   - `LOCATION`: the region where the plan is created.

   The `--schedule` flag requires fixed integer values for both the minute
   (`0-59`) and hour (`0-23`) fields---that is, at most one scheduled run per
   day. Wildcards (`*` or `*/N`), ranges, or lists in the minute or hour fields
   are rejected.
2. Create a restore verification plan association to link the restore
   verification plan to a specific restore template:

   ```bash
   gcloud backup-dr restore-verification-plan-associations create ASSOCIATION_ID \
       --project=ORCHESTRATION_PROJECT_ID \
       --location=LOCATION \
       --restore-verification-plan=PLAN_ID \
       --resource-type="RESOURCE_TYPE" \
       --restore-template="projects/VAULT_PROJECT_ID/locations/LOCATION/restoreTemplates/RESTORE_TEMPLATE_ID"
   ```

   Replace the following:
   - `ASSOCIATION_ID`: the ID of the restore verification plan
     association to create.

   - `ORCHESTRATION_PROJECT_ID`: the project ID of the restore
     orchestration project.

   - `LOCATION`: the region where the association is created.

   - `PLAN_ID`: the ID of the restore verification plan.

   - `RESOURCE_TYPE`: the target resource type matching the
     restore template (`compute.googleapis.com/Instance`,
     `compute.googleapis.com/Disk`, `compute.googleapis.com/RegionDisk`, or
     `sqladmin.googleapis.com/Instance`).

   - `VAULT_PROJECT_ID`: the project ID of the backup vault
     project.

   - `RESTORE_TEMPLATE_ID`: the ID of the restore template.

## Run on-demand restore verification tests

Perform an ad hoc verification run to validate a specific recovery point on
demand:

1. Trigger on-demand restore verification to validate a specific recovery point
   by using a plan association:

   ```bash
   gcloud backup-dr restore-verification-plan-associations trigger-restore-verification ASSOCIATION_ID \
       --project=ORCHESTRATION_PROJECT_ID \
       --location=LOCATION \
       --restore-verification-plan=PLAN_ID
   ```

   Replace the following:
   - `ASSOCIATION_ID`: the ID of the restore verification plan
     association.

   - `ORCHESTRATION_PROJECT_ID`: the project ID of the restore
     orchestration project.

   - `LOCATION`: the region of the association.

   - `PLAN_ID`: the ID of the restore verification plan.

2. Run a restore template directly to perform an ad hoc execution without a
   verification plan. The `restore-templates run` command creates an on-demand
   execution without cleanup (`WITHOUT_CLEANUP`). Passing the `--backup` flag is
   optional; if you omit `--backup`, the command restores the latest backup from
   the data source configured in the restore template:

   ```bash
   gcloud backup-dr restore-templates run RESTORE_TEMPLATE_ID \
       --project=VAULT_PROJECT_ID \
       --location=LOCATION \
       --backup="projects/VAULT_PROJECT_ID/locations/VAULT_LOCATION/backupVaults/VAULT_ID/dataSources/DATA_SOURCE_ID/backups/BACKUP_ID"
   ```

   Replace the following:
   - `RESTORE_TEMPLATE_ID`: the ID of the restore template.

   - `VAULT_PROJECT_ID`: the project ID of the backup vault
     project.

   - `LOCATION`: the region of the template.

   - `VAULT_LOCATION`: the region of the backup vault.

   - `VAULT_ID`: the ID of the backup vault.

   - `DATA_SOURCE_ID`: the ID of the data source.

   - `BACKUP_ID` (Optional): the ID of a specific backup to
     restore for verification. Omit the `--backup` flag to use the latest backup
     from the data source configured in the restore template.

3. Trigger on-demand cleanup to delete temporary test instances manually. The
   `trigger-restore-cleanup` command applies only to verification runs
   triggered by using a restore verification plan association, such as
   scheduled runs or on-demand runs from step 1. Use this command to clean up
   resources early during the cleanup delay window or to retry a failed
   cleanup. Calling the `trigger-restore-cleanup` command on an execution
   created by the `restore-templates run` command fails with a
   `400 INVALID_ARGUMENT` (`RESTORE_TEMPLATE_EXECUTION_CLEANUP_NOT_SUPPORTED`)
   error:

   ```bash
   gcloud backup-dr restore-template-executions trigger-restore-cleanup EXECUTION_ID \
       --project=VAULT_PROJECT_ID \
       --location=LOCATION \
       --restore-template=RESTORE_TEMPLATE_ID
   ```

   Replace the following:
   - `EXECUTION_ID`: the ID of the restore execution.

   - `VAULT_PROJECT_ID`: the project ID of the backup vault
     project.

   - `LOCATION`: the region of the execution.

   - `RESTORE_TEMPLATE_ID`: the ID of the restore template.

## Monitor, alert, and report on verification runs

You can monitor the status of your verification runs, set up alerts, and
generate reports in the Google Cloud console.

1. Monitor the status of your verification jobs centrally by using the
   Google Cloud console. Viewing the **Restore verification** report in the
   Google Cloud console requires your project to be on the allowlist and
   requires [Observability Analytics](https://docs.cloud.google.com/logging/docs/buckets.md.txt#upgrade-bucket) to
   be enabled on the `_Default` log bucket in the project:

   1. In the Google Cloud console, go to the **Reports dashboard** page:

      [Go to Reports dashboard](https://console.cloud.google.com/backupdr/report-dashboard)
   2. Select the **Restore verification** tab:

      - If your project isn't on the allowlist, a banner appears stating that
        the feature is available only for projects on the allowlist. To add
        your project to the allowlist, request access through your
        [TAM](https://cloud.google.com/tam),
        [create a support case with Customer Care](https://docs.cloud.google.com/support/docs/customer-care-procedures.md.txt#create_a_support_case),
        or click **Support** in the banner.

      - If your project is on the allowlist and the `_Default` log bucket
        isn't upgraded yet, follow the on-screen prompt to upgrade the bucket
        to use Observability Analytics.

   3. Select a time range (`1 hour`, `12 hours`, `1 day`, `7 days`,
      `14 days`, `30 days`, or **Custom**) to view the summary widgets:

      - **Overall status distribution:** Shows the total count and status
        distribution of restore verification runs.

      - **Failure status summary:** Lists the **Failure count** for each
        verification phase (**Preparing failures** , **Pre-restore failures** ,
        **Restore failures** , **Post-restore failures** , **Verification
        failures** , and **Cleanup failures**).

   4. In the **Restore verification** jobs table, use the **Filter** field to
      filter runs by property name or value, such as **Overall status**
      (**Succeeded** , **Failed** , **Running** , or **Skipped** ), **Resource
      name** , **Resource type** , **Job category** , or **Restore verification
      plan** . To view the vaulted backup details for a resource, click its
      name in the **Resource name** column.

   5. Inspect the six status columns for each run:

      - **Overall status:** Shows the overall execution outcome, which fails
        if any phase encounters an error. Click the status in the **Overall
        status** column to open the **Restore verification job details** side
        panel, which displays run metadata (**Resource name** , **Resource
        type** , **Job category** , **Backup ID** , **Restore verification
        plan** , and **Vault** ) and a phase breakdown table showing the
        **Phase** , **Status** , **Start time** , **End time** , **Duration** ,
        **Error code** , **Error type** , and **Error message** for all
        execution phases (**Preparing** , **Executing pre-restore** ,
        **Restoring** , **Executing post-restore** , **Verifying** , **Waiting
        for cleanup** , and **Cleaning up** ). Click **Close** to close the side
        panel.

      - **Pre-restore scripts:** Indicates the status of scripts run before
        resource restoration, or **Skipped** if not configured.

      - **Restore status:** Indicates the status of restoring resource data
        from the backup to the target environment.

      - **Post-restore scripts:** Indicates the status of scripts run after
        resource restoration, or **Skipped** if not configured.

      - **Verification scripts:** Indicates the status of validation scripts
        run on the restored resource, or **Skipped** if not configured.

      - **Cleanup status:** Indicates the status of deleting the temporary
        restored test resource.

2. Set up long-term retention for your verification logs by using
   BigQuery:

   1. In the Google Cloud console, go to the **Log Router** page:

      [Go to Log Router](https://console.cloud.google.com/logs/router)
   2. Click **Create sink** to configure log export from Cloud Logging to a
      **BigQuery dataset** for retention beyond 30 days.

   3. Connect your BigQuery verification log dataset to
      **Data Studio** to generate immutable audit reports.

## Manage verification plans and test configurations

You can manage your verification plans, plan associations, and restore
templates, including modifying or deleting them:

- You can update restore templates in place as needed.

- Restore verification plans and restore verification plan associations are
  immutable after you create them. To change their settings, you must delete and
  recreate them.

### Modify a template

Update a restore template configuration by using the following command:

```bash
gcloud backup-dr restore-templates update RESTORE_TEMPLATE_ID \
    --project=VAULT_PROJECT_ID \
    --location=LOCATION \
    --description="Updated restore template configuration" \
    --restore-properties='{"project": "TARGET_PROJECT_ID", "zone": "TARGET_ZONE", "resource": {"name": "test-restore-vm-v2-{EXECUTION_ID}", "machineType": "zones/TARGET_ZONE/machineTypes/e2-standard-4"}}' \
    --verification-cloud-run-job="projects/CLOUD_RUN_PROJECT_ID/locations/LOCATION/jobs/NEW_VERIFY_JOB" \
    --verification-timeout="600s"
```

Replace the following:

- `RESTORE_TEMPLATE_ID`: the ID of the restore template to update.
- `VAULT_PROJECT_ID`: the project ID of the backup vault project.
- `LOCATION`: the region of the template.
- `TARGET_PROJECT_ID`: the project ID of the target resource project.
- `TARGET_ZONE`: the zone for the target resource.
- `CLOUD_RUN_PROJECT_ID`: the project ID of the Cloud Run jobs project.
- `NEW_VERIFY_JOB`: the name of the new verification Cloud Run job.

## Delete or detach verification resources

Delete or detach verification resources when they are no longer needed.

### Detach a restore verification plan association

Detach a restore verification plan association from a template by using the
following command:

```bash
gcloud backup-dr restore-verification-plan-associations delete ASSOCIATION_ID \
    --project=ORCHESTRATION_PROJECT_ID \
    --location=LOCATION \
    --restore-verification-plan=PLAN_ID
```

Replace the following:

- `ASSOCIATION_ID`: the ID of the restore verification plan
  association to delete.

- `ORCHESTRATION_PROJECT_ID`: the project ID of the restore
  orchestration project.

- `LOCATION`: the region of the association.

- `PLAN_ID`: the ID of the restore verification plan.

### Delete a restore verification plan

Delete a restore verification plan by using the following command. The
`--force` flag also deletes any child restore verification plan associations
under the plan:

```bash
gcloud backup-dr restore-verification-plans delete PLAN_ID \
    --project=ORCHESTRATION_PROJECT_ID \
    --location=LOCATION \
    --force
```

Replace the following:

- `PLAN_ID`: the ID of the restore verification plan to
  delete.

- `ORCHESTRATION_PROJECT_ID`: the project ID of the restore
  orchestration project.

- `LOCATION`: the region of the plan.

### Delete a restore template

Delete a restore template by using the following command. The `--force` flag
also deletes child execution records under the template, unless an execution is
in the `RUNNING` state. An execution remains in the `RUNNING` state while
waiting in its cleanup delay window. In this case, the `trigger-restore-cleanup`
command must complete before you can delete the template:

```bash
gcloud backup-dr restore-templates delete RESTORE_TEMPLATE_ID \
    --project=VAULT_PROJECT_ID \
    --location=LOCATION \
    --force
```

Replace the following:

- `RESTORE_TEMPLATE_ID`: the ID of the restore template to
  delete.

- `VAULT_PROJECT_ID`: the project ID of the backup vault
  project.

- `LOCATION`: the region of the template.

## What's next

- [Learn about backup vaults](https://docs.cloud.google.com/backup-disaster-recovery/docs/concepts/backup-vault.md.txt)
- [Overview of restore operations](https://docs.cloud.google.com/backup-disaster-recovery/docs/restore-data/restore-overview.md.txt)
- [Monitor jobs in the Google Cloud console](https://docs.cloud.google.com/backup-disaster-recovery/docs/monitor-reports/monitor-jobs-console.md.txt)
- [Manage backups in the Google Cloud console](https://docs.cloud.google.com/backup-disaster-recovery/docs/cloud-console/bv-backup-manage.md.txt)