You can specify whether a pipeline task must be rerun if it fails, by
configuring the retries for that task. You can set the number of attempts to
rerun the task on failure and the delay between subsequent retries.

Use the following code sample to configure the failure policy of a pipeline task
named `train_op` by using the
[`set_retry`](https://kubeflow-pipelines.readthedocs.io/page/source/dsl.html#kfp.dsl.PipelineTask.set_retry)
method in the Kubeflow Pipelines SDK:

```python
from kfp import dsl

@dsl.pipeline(name='custom-container-pipeline')
def pipeline():
  generate = generate_op()
  train = (
    train_op(
      training_data=generate.outputs['training_data'],
      test_data=generate.outputs['test_data'],
      config_file=generate.outputs['config_file'])
    .set_retry(
      num_retries=NUMBER_OF_RETRIES,
      backoff_duration='BACKOFF_DURATION',
      backoff_factor=BACKOFF_FACTOR,
      backoff_max_duration='BACKOFF_MAX_DURATION'
    )
```

Replace the following:

- <var translate="no">NUMBER_OF_RETRIES</var>: The number of times to retry the task upon failure.

- <var translate="no">BACKOFF_DURATION</var>: Optional. The duration of time wait after the task
  fails before retrying. If you don't set this parameter, the duration
  is set to `0s`, by default.

- <var translate="no">BACKOFF_FACTOR</var>: Optional. The factor by which the backoff duration
  is multiplied for each subsequent retry. If you don't set this parameter, the
  backoff factor is set to `2.0`, by default.

- <var translate="no">BACKOFF_MAX_DURATION</var>: Optional. The maximum backoff duration between subsequent retries.
  If you don't set this parameter, the maximum duration is set to `3600s`, by default.

> [!CAUTION]
> **Caution:** You can't pass output parameters from other pipeline tasks or pipeline input parameters as parameter values for the set_retry method. These values must be available when you compile the pipeline.