Datastream overview

Datastream is a serverless change data capture (CDC) and replication service that lets you synchronize data across heterogeneous databases and applications with minimal latency. Because Datastream is serverless, it scales automatically without requiring you to provision or manage resources.

You can replicate data into BigQuery for near real-time analytics or write change event streams into Cloud Storage for event-driven architectures. Integrating with Pub/Sub and Dataflow templates lets you load data into destinations such as Cloud SQL and Spanner. Datastream also integrates with Knowledge Catalog for centralized metadata and lineage management across Google Cloud.

Datastream supports database sources including MySQL, Oracle, PostgreSQL, AlloyDB for PostgreSQL, SQL Server, MongoDB, and Spanner. You can also stream data from Salesforce and from Preview sources such as Salesforce Marketing Cloud, Microsoft Dataverse, ServiceNow, and Workday. Private connectivity options, status reporting, and schema change handling support reliable replication for migrations and hybrid-cloud deployments.

Use cases

The streaming capabilities of Datastream enable a variety of use cases:

  • Replicating and synchronizing data across your organization with minimal latency

    You can synchronize data across heterogeneous databases and applications reliably, with low latency, and with minimal impact to the performance of your source. Unlock the power of data streams for analytics, database replication, cloud migration, and event-driven architectures across hybrid environments.

  • Scale up or down with a serverless architecture seamlessly

    Get up and running fast with a serverless and easy-to-use service that scales seamlessly as your data volumes shift. Focus on deriving up-to-date insights from your data and responding to high-priority issues, instead of managing infrastructure, performance tuning, or resource provisioning.

  • Integrate with the Google Cloud data integration suite

    Connect data across your organization with the Google Cloud data integration suite of products. Integrate Datastream with Dataflow job templates to read data from a Cloud Storage bucket and load it into a variety of destinations, such as BigQuery, Spanner, and Cloud SQL.

Experience elements

There are three main elements in Datastream:

  • Private connectivity configurations enable Datastream to communicate with a data source over a private network (internally within Google Cloud, or with external sources connected over VPN or Interconnect). This communication happens through a Virtual Private Cloud (VPC) peering connection.
  • Connection profiles represent connectivity information to both a source and a destination. This information will be used by a stream.
  • Streams use the information in the connection profiles to transfer CDC and backfill data from the source to the destination.

What's next