Set up an experiment

This page explains how to create an experiment. An experiment is an instance of a fault injection experiment against your target resources, which should fail because of the fault.

To set up a new experiment, you'll create an experiment template, which defines the fault to be injected and the target resources.

Set up a new experiment template for Failover Cloud SQL

This fault lets you failover from a primary to a replica instance of HA Cloud SQL.

Before proceeding, you must have the following:

  • Permission to view Cloud SQL instances in the project.
  • Fault Injection Testing API enabled for the project.
  • The roles/faulttesting.operator role.

To create a new experiment template, do the following:

  1. Open the Google Cloud console.
  2. Navigate to the Fault Injection Testing Overview page.
  3. Select Create experiment template.
    • Alternatively, from the Fault Injection Testing Overview page, click Experiment templates on the left navigation bar, and click Create Template.
  4. In the Fault Type section, select the Failover Cloud SQL option.
  5. In the Target section, use the filters and select a single instance. NOTE: For Cloud SQL, when an experiment is stopped, the database will continue to run in the secondary zone and it will not automatically revert to running in the initial primary zone. If you want to fail back the instance, you can do so manually or by running another experiment targeting the newly failed-over instance.
  6. The duration field in the Duration section is non-editable, and set to 3 minutes.
  7. In the Template name section, provide a clear and descriptive name for the template. Optionally provide details in the description.
  8. In the Verify section:
    • See a recap of the settings you selected.
    • Wait for the automated dry run to finish.
    • After the dry run is complete, the list of resources that will receive the fault injection appears. If there are errors with the targeted resources, they will also appear here.
  9. Create the experiment template. A confirmation message indicating that the template has been created appears, and the new template shows up in the list of templates.

Set up a new experiment template for Degrade Application Traffic fault

This fault lets you selectively degrade L7 traffic through a specified application load balancer.

Before proceeding, you must have the following:

  • Permission to view Load Balancer Forwarding rules and Backend Services in the project – compute.viewer, compute.networkViewer or viewer role.
  • The roles/faulttesting.operator role.

You can run test commands in the Google Cloud CLI. To get to the Google Cloud CLI, install the Google Cloud SDK, then run gcloud init in your terminal to authenticate. Access the web-based console at https://console.cloud.google.com/. For command-line access, download the SDK for your OS (Windows, Linux, macOS), and run gcloud init to sign in and configure your project. Example test commands include:

To create a new experiment template, do the following:

  1. Open the Google Cloud console.
  2. Navigate to the Fault Injection Testing Overview page.
  3. Select Create experiment template.
    • Alternatively, from the Fault Injection Testing Overview page, click Experiment templates on the left navigation bar, and click Create Template.
  4. In the Fault Type section, select the Degrade application traffic option.
  5. In the Target section, use the filters and select either XLB (External Load Balancer) or ILB (Internal Load Balancer).
    • (Mandatory) Select the Load balancer frontend service that is part of the forwarding rule you want to target.
  6. In the Fault Parameters section, set the following:
    • Type of load balancer fault - choose from the following:
      • HTTP delay injection - insert a delay into the selected forwarding rule.
      • Delay percentage - percentage of traffic to impact with a delay. The remaining traffic will pass as normal.
      • Latency duration - how much latency to add to this delayed traffic.
    • HTTP error injection - send HTTP error responses for the selected forwarding rule. Specify the following parameters:
      • Error code - the error response code you expect to see from your load balancer, ideally a 5xx response code.
      • Error percentage - percentage of traffic that will send the specified error response code. The remaining traffic will receive normal response codes.
    • Experiment duration - determines how long the fault condition should last.
  7. In the Template name section, provide a clear and descriptive name for the template. Optionally provide details in the description.
  8. In the Verify section:
    1. See a recap of the settings you selected.
    2. Wait for the automated dry run to finish.
    3. After the dry run is complete, the list of resources that will receive the fault injection appears. If there are errors with the targeted resources, they will also appear here.
  9. Create the experiment template. A confirmation message indicating that the template has been created appears, and the new template shows up in the list of templates.

What's next

Run an experiment using the experiment template you created.