<br />

<button value="Before v004.009">Before v004.009</button> <button value="v004.009 - v004.010">v004.009 - v004.010</button> <button value="v004.011" default="">v004.011 and later</button> This page is valid for the engine versions within the following major engine version groupings. To view the page for other engine versions, use the selector at the top of this page.

<br />

- [aml-retail.default.v004.011](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/private/engine-versions/aml-retail.default.v004#aml-retail.default.v004.011)
- [aml-commercial.default.v004.011](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/private/engine-versions/aml-commercial.default.v004#aml-commercial.default.v004.011)

In summary, here are the changes in this page versus v004.010:

- Added `partyCountExceedingThresholdPerPeriodHint` as an alternative performance target.
- Simplified the recall metric by removing the lookback buffer, allowing for more transparent evaluation across all specified backtest periods.
- The `ObservedRecallValues` output now includes a new field `partyCountExceedingThresholdPerPeriod`.

## Overview

Backtest results provide you with a summary of model performance in a specified
timeframe. They are generated by predicting on all customers within a backtest
period and evaluating model performance against available risk events.

Backtest results can be used to measure model performance on a separate time
range from that used in training, or also over time to check for performance
degradation.

## How to backtest

To create a
[BacktestResult resource](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/reference/rest/v1/projects.locations.instances.backtestResults.md.txt),
see
[Create and manage backtest results](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/create-and-manage-backtest-results.md.txt).

In particular, you need to select the following:

- **The data to use for backtesting:**

  Specify a dataset and an end time within the date range of the dataset.

  Training uses labels and features based on complete calendar months up to,
  but not including, the month of the selected end time. For more information,
  see
  [Dataset time ranges](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/overview-model-preparation.md.txt#dataset-time-ranges).
- **The backtest periods:**

  Specify how many months of labeled data to use for backtesting (that is, the
  number of [backtest periods](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/evaluate-model.md.txt#backtest-periods)).
- **The performance target:**

  Specify how to set the operating point for evaluation using the [PerformanceTarget](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/reference/rest/v1/projects.locations.instances.backtestResults.md.txt#PerformanceTarget) field. You must provide exactly one of the following:
  - `partyInvestigationsPerPeriodHint`: Specify the approximate number of investigations performed per month. This option uses steady-state approximations to estimate investigation volume.
  - `partyCountExceedingThresholdPerPeriodHint`: Specify the average monthly number of unique parties that should score at or above the risk threshold. This option provides a more direct and transparent way to set the operating point.

  Backtesting evaluates the AML AI model at a range of volumes centered around the hint you specify. For more information, see [Backtest output](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/evaluate-model.md.txt?ev_major_version=v004.011#backtest-output).
- **A model created using a [consistent dataset](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/overview-model-preparation.md.txt#dataset-consistency):**

  See
  [Create a model](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/create-and-manage-models.md.txt#create-model).

### Backtest periods

The
[`backtestPeriods`](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/reference/rest/v1/projects.locations.instances.backtestResults.md.txt#BacktestResult.FIELDS.backtest_periods)
field specifies how many consecutive calendar months to use features and labels
from in the performance evaluation of this model.

The following apply to backtest data:

- The months used in evaluation are the most recent complete calendar months prior to the specified `endTime`. For example, if `endTime` is `2023-04-15T23:21:00Z` and `backtestPeriods` is `5`, then the labels from the following months are used: 2023-03, 2023-02, 2023-01, 2022-12, and 2022-11.
- You should use the most recent available data for backtesting when evaluating a model in preparation for production use.
- Backtest periods must be set to `3` or greater.
- Starting with engine version **v004.011**, all specified backtest periods
  are used for performance evaluation, as the simplified recall metric no
  longer requires a lookback buffer for steady-state approximation.

- Avoid using overlapping months for training and backtesting as this risks
  overfitting. Make sure backtest and training end times are at least
  *`backtestPeriods`* apart. That is,

  (backtest results end time month) \>= (model end time month) +
  `backtestPeriods`

Optionally, you can also
[create prediction results](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/create-and-manage-prediction-results.md.txt)
for a model and conduct your own party-level analyses of model performance.

## Backtest output

The
[backtest results metadata](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/create-and-manage-backtest-results.md.txt#export-metadata)
contains the following metrics. In particular, these metrics show you the
following:

- How the model performs compared to labels from a separate time period and
  for a variety of different operating points (investigation volumes) or [risk score](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/reference/schemas/aml-output-data-model.md.txt#risk-scores) thresholds.


  **Note** : Recall metrics only show the share of historical [`AML_EXIT`](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/reference/lifecycle-risk-case.md.txt) events that would be alerted with a given model, and can be used to decide to proceed to live testing. Given that not all customers alerted by the model have previously been investigated, real-life performance is expected to be significantly better.

  <br />

- Measurements which can be used to assess [dataset consistency](https://docs.cloud.google.com/financial-services/anti-money-laundering/docs/overview-model-preparation.md.txt#dataset-consistency) (for example, by comparing the
  missingness values of feature families from different operations)

| Metric name | Metric description | Example metric value |
|---|---|---|
| ObservedRecallValues | Recall metric measured on the dataset specified for backtesting. The API includes 20 of these measurements, at different operating points, evenly distributed from 0 (not included) until 2 \* `partyInvestigationsPerPeriodHint` or `partyCountExceedingThresholdPerPeriodHint` depending on which of these two fields was used as input to create BacktestResults. The API adds a final recall measurement at the specified hint. <br /> Alongside the recall value, we also provide the numerator and denominator as `partiesCount` and `identifiedPartiesCount` respectively. We also provide the number of parties exceeding threshold as `partyCountExceedingThresholdPerPeriod`. | ``` { "recallValues": [ { "partyInvestigationsPerPeriod": 5000, "partyCountExceedingThresholdPerPeriod": 10000, "recallValue": 0.80, "partiesCount": 60, "identifiedPartiesCount": 48, "scoreThreshold": 0.42, }, ... ... { "partyInvestigationsPerPeriod": 8000, "partyCountExceedingThresholdPerPeriod": 16000, "recallValue": 0.85, "partiesCount": 60, "identifiedPartiesCount": 51, "scoreThreshold": 0.30, }, ], } ``` |
| ObservedRecallValuesPerTypology | Recall metric on a risk typology level measured on the dataset specified for backtesting. The measurements follow the same approach as `ObservedRecallValues`. | ``` { "recallValuesPerTypology": [ { "partyInvestigationsPerPeriod": 5000, "partyCountExceedingThresholdPerPeriod": 10000, "riskTypology": "risk_typology_id_1", "recallValue": 0.80, "partiesCount": 60, "identifiedPartiesCount": 48, "scoreThreshold": 0.42, }, { "partyInvestigationsPerPeriod": 8000, "partyCountExceedingThresholdPerPeriod": 16000, "riskTypology": "risk_typology_id_1", "recallValue": 0.90, "partiesCount": 60, "identifiedPartiesCount": 54, "scoreThreshold": 0.30, }, ... ... { "partyInvestigationsPerPeriod": 8000, "partyCountExceedingThresholdPerPeriod": 16000, "riskTypology": "risk_typology_id_2", "recallValue": 0.75, "partiesCount": 4 "identifiedPartiesCount": 3, "scoreThreshold": 0.30, }, ], } ``` |
| Missingness | Share of missing values across all features in each feature family. Ideally, all AML AI feature families should have a Missingness near to 0. Exceptions may occur where the data underlying those feature families is unavailable for integration. A significant change in this value for any feature family between tuning, training, evaluation, and prediction can indicate inconsistency in the datasets used. | ``` { "featureFamilies": [ { "featureFamily": "unusual_wire_credit_activity", "missingnessValue": 0.00, }, ... ... { "featureFamily": "party_supplementary_data_id_3", "missingnessValue": 0.45, }, ], } ``` |
| Skew | Metrics showing skew between training and prediction or backtest datasets. Family skew indicates changes in the distribution of feature values within a feature family, weighted by importance of the feature within that family. Max skew indicates the maximum skew of any feature within that family. Skew values range from 0, representing no significant change in the distribution of values of features in the family, to 1 for the most significant change. A large value for either family skew or max skew indicates a significant change in the structure of your data in a way that may impact model performance. Family skew takes the value -1 when no features in the family are used by the model. For large skew values, you should do one of the following: - Investigate changes in the data used by that feature family (see model governance support materials) and fix any input data issues - Retrain a model on more recent data You should set thresholds for acting on family and max skew values based on observing the natural variation in skew metrics over several months. | ``` { "featureFamilies": [ { "featureFamily": "unusual_wire_credit_activity", "familySkewValue": 0.10, "maxSkewValue": 0.14, }, ... ... { "featureFamily": "party_supplementary_data_id_3", "familySkewValue": 0.11, "maxSkewValue": 0.11, }, ], } ``` |