Install or update the Apache Beam SDK

This page shows you how to install or update the Apache Beam SDK so that you can run your pipelines on the Dataflow service.

Install the Apache Beam SDK

The Apache Beam SDK is an open source programming model for data pipelines. You define these pipelines with an Apache Beam program and can choose a runner, such as Dataflow, to execute your pipeline.

Java

The latest released version for the Apache Beam SDK for Java is 2.76.0. See the release announcement for information about the changes included in the release.

To get the Apache Beam SDK for Java using Maven, use one of the released artifacts from the Maven Central Repository.

Add dependencies and dependency management tools to your pom.xml file for the SDK artifact. For details, see Manage pipeline dependencies in Dataflow.

For more information about Apache Beam SDK for Java dependencies, see Apache Beam SDK for Java dependencies and Managing Beam dependencies in Java in the Apache Beam documentation.

Python

The latest released version for the Apache Beam SDK for Python is 2.76.0. See the release announcement for information about the changes included in the release.

To obtain the Apache Beam SDK for Python, use one of the released packages from the Python Package Index.

Install Python wheel by running the following command:

pip install wheel

Install the latest version of the Apache Beam SDK for Python by running the following command from a virtual environment:

pip install 'apache-beam[gcp]'

Depending on the connection, the installation might take some time.

Go

The latest released version for the Apache Beam SDK for Go is 2.76.0. See the release announcement for information about the changes included in the release.

To install the latest version of the Apache Beam SDK for Go, run the the following command:

go get -u github.com/apache/beam/sdks/v2/go/pkg/beam

Set up your development environment

For information about setting up your Google Cloud project and development environment to use Dataflow, follow one of the tutorials:

Source code and examples

The Apache Beam source code is available in the Apache Beam repository on GitHub.

Java

Code samples are available in the Apache Beam Examples directory on GitHub.

Python

Code samples are available in the Apache Beam Examples directory on GitHub.

Go

Code samples are available in the Apache Beam Examples directory on GitHub.

Find the Dataflow SDK version

Installation details depend on your development environment. If you're using Maven, you can have multiple versions of the Dataflow SDK "installed," in one or more local Maven repositories.

Java

To find out what version of the Dataflow SDK that a given pipeline is running, you can look at the console output when running with DataflowPipelineRunner or BlockingDataflowPipelineRunner. The console will contain a message like the following, which contains the Dataflow SDK version information:

Python

To find out what version of the Dataflow SDK that a given pipeline is running, you can look at the console output when running with DataflowRunner. The console will contain a message like the following, which contains the Dataflow SDK version information:

Go

To find out what version of the Dataflow SDK that a given pipeline is running, you can look at the console output when running with DataflowRunner. The console will contain a message like the following, which contains the Dataflow SDK version information:

  INFO: Executing pipeline on the Dataflow Service, ...
  Dataflow SDK version: <version>

Update the Apache Beam SDK

When you upgrade your pipeline to a later Apache Beam SDK version, update your project dependency configurations and check for potential breaking changes between versions.

Check release notes and breaking changes

Before upgrading, check the Apache Beam release notes for changes, bug fixes, and potential breaking changes.

If you're upgrading across multiple minor versions, review the release notes for all intermediate releases. For example, if you're upgrading from version 2.50.0 to 2.60.0, review the release notes for versions 2.51.0 through 2.60.0. Breaking changes, deprecations, and configuration updates are cumulative across releases.

For known issues and connector-specific notices that affect Dataflow, see the Details column in the SDK version support status table.

Update SDK dependencies

To upgrade the SDK version in your project, update your build configuration or environment for the language your pipeline uses. Replace VERSION with your target Apache Beam SDK version (for example, 2.76.0).

Java

To upgrade the Apache Beam SDK for Java, update the SDK version in your build configuration file, such as pom.xml for Maven or build.gradle for Gradle.

Maven

In your pom.xml file, update the beam.version property or update the <version> tag in your Apache Beam dependencies:

<properties>
  <beam.version>VERSION</beam.version>
</properties>

<dependencies>
  <dependency>
    <groupId>org.apache.beam</groupId>
    <artifactId>beam-sdks-java-core</artifactId>
    <version>${beam.version}</version>
  </dependency>
  <dependency>
    <groupId>org.apache.beam</groupId>
    <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
    <version>${beam.version}</version>
  </dependency>
</dependencies>

Gradle

In your build.gradle file, update the Apache Beam dependency declarations:

dependencies {
  implementation 'org.apache.beam:beam-sdks-java-core:VERSION'
  implementation 'org.apache.beam:beam-runners-google-cloud-dataflow-java:VERSION'
}

For more information, see Manage Java pipeline dependencies.

Python

To upgrade the Apache Beam SDK for Python in your local or virtual environment, use pip:

pip install --upgrade 'apache-beam[gcp]==VERSION'

In your requirements.txt or setup.py file, update the pinned version:

apache-beam[gcp]==VERSION

Go

To upgrade the Apache Beam SDK for Go, use go get to retrieve the target version and update your go.mod file:

go get -u github.com/apache/beam/sdks/v2@vVERSION
go mod tidy

For more information, see Manage Go pipeline dependencies.

Deploy upgraded pipelines

After you update the SDK version in your code, choose a deployment strategy based on your pipeline type:

  • Streaming pipelines: For streaming pipelines, you can deploy upgraded code by using in-place job replacement, running parallel pipelines, or stopping and restarting the job. For detailed instructions, compatibility rules, and deployment strategies, see Update an existing pipeline and Upgrade a streaming pipeline.
  • Batch pipelines: Batch pipelines don't support in-flight updates or in-place replacement jobs. Let the existing job run to completion, or cancel the running job, and then launch a new job with the upgraded SDK version.

Best practices for CI/CD and updating deployments

To minimize operational risk when rolling out SDK upgrades across automated build and deployment pipelines, follow these best practices:

  • Pin dependencies: Always pin exact SDK versions such as apache-beam[gcp]==VERSION in Python, or exact version strings in Maven and Gradle. Avoid dynamic ranges (such as >= or latest) to prevent untested minor upgrades from occurring during automated CI/CD builds.
  • Validate in staging environments: Deploy and test your upgraded pipeline in a dedicated staging or pre-production environment before promoting the changes to production. Validate pipeline throughput, metric stability, and downstream outputs.
  • Use a test or parallel pipelines: For mission-critical streaming applications, run the upgraded pipeline in parallel with the existing pipeline. Verify data parity and output correctness between both jobs before you drain the older pipeline.

What's next