Use eager watermark advancement

In a Dataflow streaming pipeline, delayed or out-of-order records can stall your pipeline's watermark. These types of records result from instances where time discrepancies occur, such as events from offline devices or clients with clock skew. A stalled watermark delays your on-time windowed results and event-time timers, and can cause sudden spikes in worker utilization when the delayed data finally clears.

You can prevent delayed records from holding back your pipeline by enabling eager watermark advancement for your Streaming Engine jobs. When you enable this feature, Dataflow continues to process delayed records, but those records don't stall the watermark, allowing your windows and timers to run on schedule.

Enabling eager watermark advancement helps you achieve the following:

  • Predictable latency: Output windowed aggregations on time and run event-time timers on schedule without waiting for delayed records to finish processing.
  • Smoother worker utilization: Close windows and run timers steadily as on-time data advances the watermark, avoiding worker utilization spikes from backlogged windows closing all at once.

Overview

In a streaming pipeline, the watermark is a threshold indicating that Dataflow has tracked that the data up to this point has been processed. Windowed aggregations output on-time results, and event-time timers run only after the watermark passes the end of the window or the timer timestamp.

By default, the watermark is based on the minimum unprocessed event timestamp, and it never moves backward. When a record arrives with an event timestamp behind the current watermark, the watermark stalls until that record finishes processing. Because downstream watermarks can't advance past upstream watermarks, a stalled upstream watermark stalls every stage after it. As a delayed record moves through your pipeline, each stage's watermark stays stalled until the record finishes processing in that stage. Even when your pipeline is otherwise healthy, this cascade delays windowed results and event-time timers for on-time data across all downstream stages. When the delayed records finally clear, watermarks jump forward rapidly, often triggering a burst of window closures that spikes worker utilization.

Eager watermark advancement changes this behavior for Streaming Engine jobs by decoupling delayed records from watermark progression. When a record arrives with an event timestamp behind the current watermark, Dataflow processes the record without holding back the stage's watermark. The watermark continues to advance based on your on-time data, while delayed records are processed and output according to your pipeline's allowed lateness configuration.

When to use eager watermark advancement

Eager watermark advancement is effective in the following scenarios where watermark progress is stalled by records that arrive behind the current watermark:

  • Bursts of delayed data: Scenarios where mobile devices, IoT sensors, or network equipment go offline for hours or days, buffer events locally, and output a burst of historical data upon reconnecting.
  • Out-of-order event times in Pub/Sub: Pub/Sub topics where messages arrive with event timestamps older than the source watermark, or sparsely populated topics where an idle subscription's watermark advances to wall-clock time between messages.
  • Kafka partition recovery: Kafka sources configured with event time or CreateTime, where a partition's watermark advances while the partition is idle, and the partition then receives a backlog of older records.
  • Custom past timestamps: User transforms that output records with historical timestamps by using outputWithTimestamp.

Late data with windowed aggregations

Without eager watermark advancement, a stalled watermark keeps windows open. If the watermark stays stalled long enough, delayed records that arrive later can be counted as on time, which provides some unintended protection against late data. With eager watermark advancement, late records no longer stall the watermark, so more records are correctly classified as late. Late records within the allowed lateness are output in late panes, and late records after it are left out of the window's results.

When eager watermark advancement doesn't apply

Eager watermark advancement doesn't resolve watermark lag caused by slow processing of on-time data. If transforms take a long time to process on-time records due to compute-heavy operations, slow external RPCs, or worker CPU and memory bottlenecks, the watermark is held back by those slow elements. Eager watermark advancement only decouples data that is already behind the watermark. It doesn't accelerate slow on-time processing.

Support and limitations

All Apache Beam SDK versions that run on Streaming Engine (Java, Python, and Go) support eager watermark advancement. No SDK upgrade is required.

Eager watermark advancement has the following requirements and limitations:

  • Streaming Engine required: The pipeline must use Streaming Engine.
  • Streaming jobs only: You can only enable eager watermark advancement for streaming pipelines.
  • Allowed lateness: To handle late records in windowed aggregations or stateful processing, set allowed lateness to cover your expected delays before you enable eager watermark advancement. Records later than the allowed lateness are dropped. For details, see Late data.
  • Bounded sources: If all sources in a streaming job are bounded, the job doesn't finish on its own when the input ends. To stop the job, drain it.
  • New or replacement jobs only: You can't change eager watermark advancement on an actively running job. To enable or disable the feature for an existing pipeline, you must deploy a replacement job by using the pipeline update option.

Before you begin

Before you enable eager watermark advancement, do the following:

  1. Configure allowed lateness to cover expected delays.
  2. Check that your pipeline's trigger, accumulation mode, and PaneInfo handling work with late panes.

After enabling eager watermark advancement, monitor the droppedDueToLateness counter under Custom counters in the Job info panel of the Google Cloud console to verify that dropped late data remains at an acceptable rate for your use case.

Enable eager watermark advancement

To enable eager watermark advancement, set the enable_eager_watermark_advancement service option when you launch a pipeline:

Java

--dataflowServiceOptions=enable_eager_watermark_advancement

Python

--dataflow_service_options=enable_eager_watermark_advancement

Go

--dataflow_service_options=enable_eager_watermark_advancement

gcloud

When you run a template using the gcloud dataflow jobs run command or a Flex Template using the gcloud dataflow flex-template run command, include the --enable-streaming-engine and --additional-experiments flags:

gcloud dataflow flex-template run JOB_NAME \
    --template-file-gcs-location=TEMPLATE_LOCATION \
    --region=REGION \
    --enable-streaming-engine \
    --additional-experiments=enable_eager_watermark_advancement

Replace the following values:

  • JOB_NAME: the name of your Dataflow job
  • TEMPLATE_LOCATION: the Cloud Storage path to your Flex Template specification file
  • REGION: the region where the job runs

Terraform

Set additional_experiments on the google_dataflow_flex_template_job or google_dataflow_job resource:

resource "google_dataflow_flex_template_job" "pipeline" {
  # ...
  enable_streaming_engine = true
  additional_experiments  = ["enable_eager_watermark_advancement"]
}

REST

Use the additionalExperiments field in the RuntimeEnvironment object, or in the FlexTemplateRuntimeEnvironment object when launching a Flex Template:

{
  "additionalExperiments": ["enable_eager_watermark_advancement"]
}

Console

  1. In the Google Cloud console, go to the Dataflow Jobs page.

    Go to Jobs

  2. Click Create job from template.

  3. Select a template from the Dataflow template menu.

  4. Expand Optional parameters.

  5. In the Additional experiments field, enter enable_eager_watermark_advancement.

  6. Click Run job.

Verify that eager watermark advancement is enabled

When a job starts or is updated with eager watermark advancement enabled, the job logs contain the following message:

Eager watermark advancement is enabled for this job.

When an update turns eager watermark advancement on or off, the job logs contain a message like the following:

Eager watermark advancement setting changed during a job update. Old setting: false, New setting: true

You can also check the Pipeline options section of the Job info panel. The enable_eager_watermark_advancement option appears in the dataflowServiceOptions row, or in the experiments row if you set it as an experiment.

Disable eager watermark advancement

You can't disable eager watermark advancement on an actively running job. To disable the feature for an existing pipeline, launch a replacement job by using the --update pipeline option and omit enable_eager_watermark_advancement from the replacement options. For more information, see Update a streaming pipeline. Because the watermark never moves backward, the replacement job's watermark continues from the previous job's watermark and resumes default advancement behavior.

Monitor eager watermark advancement

For jobs that have eager watermark advancement enabled, Dataflow reports the following per-stage metrics to Cloud Monitoring. Use these metrics together with the standard dataflow.googleapis.com/job/per_stage_data_watermark_age (data freshness) metric to evaluate delayed data and watermark latency:

  • dataflow.googleapis.com/job/per_stage_data_watermark_age_with_late_data: what the stage's data watermark age would be if eager watermark advancement were disabled. The difference between this metric and per_stage_data_watermark_age is the watermark latency that eager watermark advancement removes.
  • dataflow.googleapis.com/job/per_stage_oldest_pending_element_age: the age of the oldest pending element at or upstream of the stage, including delayed records. For pipelines with windowed aggregations, the difference between this metric and per_stage_data_watermark_age approximates how far the oldest delayed record is behind the watermark. If that difference approaches your allowed lateness, delayed records are at risk of being dropped. To verify whether any late records were dropped, check the droppedDueToLateness counter under Custom counters in the Google Cloud console (see Before you begin).

Both metrics are reported in seconds and include a stage label. For more information, see Use Monitoring.

What's next