Stream lifecycle

Each stream in Datastream progresses through specific states as you create, start, pause, resume, modify, or delete it. Understanding how your actions and runtime errors affect stream states helps you manage data replication and recover from failures.

Stream states and transitions

The following diagram shows the states and transitions in the lifecycle of a stream. In this diagram, the upside-down trapezoid boxes represent actions that you perform with the stream, and the rectangle boxes represent the resulting stream states:

Datastream user flow diagram

First, you create a stream. Datastream uses this stream to transfer data from a source into a destination. After the stream is created, the state of the stream is Not started.

After creating the stream, you start it. The status of the stream changes to Starting. This signifies that the stream is in the process of being started and validated.

  • If the stream is valid, then the status of the stream changes from Starting to Running.
  • If the stream isn't valid, then the status of the stream changes from Starting back to Not started. You can then rectify the problems of the stream. For example, if errors are associated with the connectivity information of the stream, then modify the stream definition. After resolving all problems of the stream, start it again, and the status changes to Starting, and then to Running.

You can pause a stream. When a stream is paused, Datastream doesn't pull any new data from the source into the destination. However, some data may continue to be written to the destination because in flight data continues to flow.

Therefore, after you pause a stream, the status of the stream changes from Running to Draining. Draining a stream is the process of emptying the stream so that it doesn't contain any data. After the stream is drained of all data, then the status of the stream changes from Draining to Paused.

When you resume a paused stream, the status of the stream changes from Paused to Running.

There are two actions that you can perform at any time during the lifecycle of the stream:

Stream errors

A running stream can encounter errors which might cause the stream to change its state to Failed or Failed permanently:

  • A Failed status means that the stream encountered a recoverable error. This means that the stream is either still active, or continuously attempting to run.
  • A Failed permanently stream is a stream that ran into an unrecoverable error which prevents it from continuing to run. Such errors might cause data loss.

You can address the issues for a Failed stream, and the stream automatically resumes. Its state changes from Failed back to Running. For more information, see Troubleshoot a stream.

You can manually recover a Failed permanently stream. For more information, see Recover a stream.

What's next