Datastream organizes replicated data into a three-level hierarchy of
streams, objects, and events. A **stream** connects a data source to a
destination, while an **object** represents a specific source entity in that
stream, such as a database table. An **event** captures a single data change
from an object, such as a database insert.

Each event includes data and metadata that identify its source stream and
object. You can use this metadata to track change origins, remove duplicate
records in your destination, and maintain ordering across downstream consumers.

## About events

Each event consists of three types of data:

- **Event data**: This represents the change to the data itself from the object originating from the stream source. Every event contains the entirety of the row that changed.
- **Generic metadata**: This metadata appears on every event generated by Datastream which is used for actions, such as removing duplicate data in the destination.
- **Source-specific metadata**: This metadata appears on every event generated by a specific stream source. This metadata varies by source.

### Event data

Event data is the payload of every change from a given object originating from a stream source.

Events are in either the Avro or JSON format.

When working with the Avro format, for each column, the event contains the column index and value. Using the column index, the column name and unified type can be retrieved from the schema in the Avro header.

> [!NOTE]
> **Note:** When Datastream converts data to the Avro format, special characters in column names are replaced with underscores. This can lead to duplicate column names and result in your streams failing. For more information about naming conventions in Avro, see the [Apache Avro documentation](https://avro.apache.org/docs/1.11.1/specification/#names).

When working with the JSON format, for each column, the event contains the column name and value.

Event metadata can be used to collect information about the event's origin, as well as to remove duplicate data in the destination and order events by the downstream consumer.

The following tables list and describe the fields and data types for generic and source-specific event metadata.

### Generic metadata

This metadata is consistent across streams of all types.

| Field | Avro type | JSON type | Description |
|---|---|---|---|
| `stream_name` | string | string | The unique stream name as defined at creation time. |
| `read_method` | string | string | Indicates if the data was read from the source using a change data capture (CDC) method, as part of historical backfill, or as part of a supplementation task that's created when a transaction is rolled back during CDC replication. Possible values include: - `cdc` - `backfill` - `oracle-cdc-logminer` - `oracle-backfill` - `oracle-supplementation` - `mysql-cdc-binlog` - `mysql-backfill-incremental` - `mysql-backfill-fulldump` - `postgres-cdc-wal` - `postgresql-backfill` - `sqlserver-cdc` - `sqlserver-backfill` - `salesforce-cdc` - `salesforce-backfill` |
| `object` | string | string | The name used to group different types of events, typically the name of the table or object in the source. |
| `schema_key` | string | string | A unique identifier for the unified schema of the event. |
| `uuid` | string | string | A unique identifier for the event generated by Datastream. |
| `read_timestamp` | [timestamp-millis](https://avro.apache.org/docs/++version++/specification/#timestamps) | string | The timestamp (UTC) when the record was read by Datastream (the epoch timestamp in milliseconds). |
| `source_timestamp` | [timestamp-millis](https://avro.apache.org/docs/++version++/specification/#timestamps) | string | The timestamp (UTC) when the record changed on the source (the epoch timestamp in milliseconds). |
| `sort_keys` | `{"type": "array", "items": ["string", "long"]}` | array | An array of values that can be used to sort events in the order in which they occurred. |

### Source-specific metadata

This metadata is associated with CDC and backfill events from a source database. To view this metadata, select a source from the drop-down menu that follows.
All sources MySQL Oracle PostgreSQL SQL Server Spanner Salesforce MongoDB

| Source | Field | Avro type | JSON type | Description |
|---|---|---|---|---|
| MySQL | `log_file` | string | string | The log file that Datastream pulls events from in CDC replication when using binlog-based replication. |
| MySQL | `log_position` | long | long | The log position (offset) in the MySQL binary log when using binlog-based replication. |
| MySQL | `binlog_file` | string | string | The binary log file that Datastream pulls events from in CDC replication when using GTID-based replication. |
| MySQL | `binlog_position` | long | long | The position (offset) in the MySQL binary log when using GTID-based replication. |
| MySQL | `gtid` | string | string | The global transaction identifier (GTID) that uniquely identifies the transaction for the event when using GTID-based replication. |
| MySQL | `datastream_master_server_uuid` | string | string | The UUID of the MySQL source server instance from which Datastream reads events when using GTID-based replication. |
| MySQL | `datastream_master_server_id` | long | long | The server ID of the MySQL source server instance from which Datastream reads events when using GTID-based replication. |
| MySQL | `primary_keys` | string array | string array | The list of (one or more) column names that make up the tables primary key. If the table doesn't have a primary key, then this field is empty. |
| MySQL | `is_deleted` | boolean | boolean | - A `true` value indicates that the row was deleted in the source. - A `false` value signifies that the row wasn't deleted. |
| MySQL | `database` | string | string | The database associated with the event. |
| MySQL | `table` | string | string | The table associated with the event. |
| MySQL | `change_type` | string | string | The type of change (`INSERT`, `UPDATE-INSERT`, `UPDATE-DELETE`, and `DELETE`) that the event represents. For MySQL row-based replication, each event for a row with a primary key change contains two images: a "before" image whose columns are matched against when searching for the row to be updated, and an "after" image containing the changes. For example, for the `UPDATE-DELETE` change, `UPDATE-DELETE` is the "before" image and `UPDATE-INSERT` is the "after" image. Learn more about [MySQL row-based replication](https://dev.mysql.com/doc/refman/5.6/en/replication-options-binary-log.html#sysvar_binlog_row_image:). |
| Oracle | `log_file` | string | string | The log file that Datastream pulls events from in CDC replication. |
| Oracle | `scn` | long | long | The log position (offset) in the Oracle transaction log. |
| Oracle | `row_id` | string | string | Oracle's [row_id](https://docs.oracle.com/cd/B19306_01/server.102/b14200/pseudocolumns008.htm). |
| Oracle | `is_deleted` | boolean | boolean | - A `true` value indicates that the row was deleted in the source. - A `false` value signifies that the row wasn't deleted. |
| Oracle | `database` | string | string | The database associated with the event. |
| Oracle | `schema` | string | string | The schema associated with the table from the event. |
| Oracle | `table` | string | string | The table associated with the event. |
| Oracle | `change_type` | string | string | The type of change (`INSERT`, `UPDATE-INSERT`, `UPDATE-DELETE`, and `DELETE`) that the event represents. For Oracle row-based replication, each event for a row with a primary key change contains two images: a "before" image whose columns are matched against when searching for the row to be updated, and an "after" image containing the changes. For example, for the `UPDATE-DELETE` change, `UPDATE-DELETE` is the "before" image and `UPDATE-INSERT` is the "after" image. Learn more about [Oracle row-based replication](https://docs.oracle.com/cd/B19306_01/server.102/b14215/logminer.htm). |
| Oracle | `tx_id` | string | string | The transaction ID to which the event belongs. |
| Oracle | `rs_id` | string | string | The record set ID. The coupling of `rs_id` and `ssn` uniquely identifies a row in `V$LOGMNR_CONTENTS`. `rs_id` uniquely identifies the redo record that generated the row. |
| Oracle | `ssn` | long | long | A SQL sequence number. This number is used with `rs_id` and uniquely identifies a row in `V$LOGMNR_CONTENTS`. |
| PostgreSQL | `schema` | string | string | The schema associated with the table from the event. |
| PostgreSQL | `table` | string | string | The table associated with the event. |
| PostgreSQL | `is_deleted` | boolean | boolean | - A `true` value indicates that the row was deleted in the source. - A `false` value signifies that the row wasn't deleted. |
| PostgreSQL | `change_type` | string | string | The type of change (`INSERT`, `UPDATE`, `DELETE`) that the event represents. For PostgreSQL row-based replication, each event for a row with a primary key change contains two images: a "before" image whose columns are matched against when searching for the row to be updated, and an "after" image containing the changes. For example, for the `UPDATE-DELETE` change, `UPDATE-DELETE` is the "before" image and `UPDATE-INSERT` is the "after" image. Learn more about [PostgreSQL row-based replication](https://www.postgresql.org/docs/current/logical-replication-row-filter.html#LOGICAL-REPLICATION-ROW-FILTER-TRANSFORMATIONS). |
| PostgreSQL | `tx_id` | string | string | The transaction ID to which the event belongs. |
| PostgreSQL | `lsn` | string | string | The log sequence number for the current entry. |
| PostgreSQL | `primary_keys` | string array | string array | The list of (one or more) column names that make up the tables primary key. If the table doesn't have a primary key, then this field is empty. |
| SQL Server | `table` | string | string | The table associated with the event. |
| SQL Server | `database` | long | long | The database associated with the event. |
| SQL Server | `schema` | string array | string array | The schema associated with the table from the event. |
| SQL Server | `is_deleted` | boolean | boolean | - A `true` value indicates that the row was deleted in the source. - A `false` value signifies that the row wasn't deleted. |
| SQL Server | `lsn` | string | string | The log sequence number for the event. |
| SQL Server | `tx_id` | string | string | The transaction ID to which the event belongs. |
| SQL Server | `physical_location` | integer array | integer array | The physical location of the log record described by three integers: file ID, page ID and slot ID of the record. |
| SQL Server | `replication_index` | string array | string array | The list of column names of an index that can uniquely identify a row in the table. |
| SQL Server | `change_type` | string | string | The type of change (`INSERT`, `UPDATE`, `DELETE`) that the event represents. |
| Spanner | `commit_timestamp` | [timestamp-micros](https://avro.apache.org/docs/++version++/specification/#timestamps) | string | The commit timestamp of the event. |
| Spanner | `snapshot` | boolean | boolean | Whether the event is a backfill snapshot event. |
| Spanner | `project_id` | string | string | The Spanner project identifier. |
| Spanner | `instance_id` | string | string | The Spanner instance identifier. |
| Spanner | `database_id` | string | string | The Spanner database identifier. |
| Spanner | `schema` | string | string | The schema associated with the table from the event. Empty if the table isn't in a named schema. |
| Spanner | `table` | string | string | The Spanner table. |
| Spanner | `change_stream_name` | string | string | The Spanner change stream. |
| Spanner | `server_transaction_id` | string | string | The transaction identifier associated with the event. |
| Spanner | `record_sequence` | long or string | long or string | The record sequence associated with the event. For change streams created with the `IMMUTABLE_KEY_RANGE` partition mode, the value is a numeric sequence number of type `long`. For change streams created with the `MUTABLE_KEY_RANGE` partition mode, the value is the `record_sequence` string from the data change record, such as `1cac8f3f38eeaf53-00000000`. |
| Spanner | `mod_number` | long | long | The modification number associated with the event. |
| Spanner | `transaction_tag` | string | string | The tag for the associated transaction. |
| Spanner | `number_of_records_in_transaction` | long | long | The number of records in the associated transaction. |
| Spanner | `value_capture_type` | string | string | The value capture type of the change stream. Possible values are `NEW_ROW` and `NEW_ROW_AND_OLD_VALUES`. |
| Spanner | `change_type` | string | string | The type of change (`INSERT`, `UPDATE`, or `DELETE`) that the event represents. |
| Spanner | `primary_keys` | string array | string array | The list of primary key columns for the table. |
| Spanner | `is_deleted` | boolean | boolean | Whether the change represents a `DELETE` event. |
| Salesforce | `object_name` | string | string | The name of the Salesforce object associated with the event. |
| Salesforce | `domain` | string | string | The name of the domain associated with the event. |
| Salesforce | `is_deleted` | boolean | boolean | - A `true` value indicates that the row was deleted in the source. - A `false` value signifies that the row wasn't deleted. |
| Salesforce | `change_type` | string | string | The type of change (`INSERT`, `UPDATE`, `DELETE`) that the event represents. |
| Salesforce | `primary_keys` | string array | string array | The list of column names that make up the tables primary key. If the table doesn't have a primary key, then this field is empty. |
| MongoDB | `database` | string | string | The database associated with the event. |
| MongoDB | `collection` | string | string | The collection associated with the event. Collections are analogous to tables in relational databases. |
| MongoDB | `change_type` | string | string | The type of change (`CREATE`, `UPDATE`, and `DELETE`) that the event represents. |
| MongoDB | `is_deleted` | boolean | boolean | - A `true` value indicates that the row was deleted in the source. - A `false` value signifies that the row wasn't deleted. |
| MongoDB | `primary_keys` | string array | string array | The `_id` field which acts as the primary key for each document in a collection. |

### Example of an event flow

This flow illustrates the events generated by three consecutive operations:
`INSERT`, `UPDATE`, and `DELETE`, on a single row in a `SAMPLE` table for a source database.

> [!NOTE]
> **Note:** This sample event flow is for an Oracle database, but it's similar for a MySQL or PostgreSQL database.

| TIME | THIS_IS_MY_PK (int) | FIELD1 (nchar nullable) | FIELD2 (nchar non-null)\> |
|---|---|---|---|
| 0 | 1231535353 | foo | TLV |
| 1 | 1231535353 | NULL | TLV |

#### INSERT (T0)

The message payload consists of the entirety of the new row.

    {
      "stream_name": "projects/myProj/locations/myLoc/streams/Oracle-to-Source",
      "read_method": "oracle-cdc-logminer",
      "object": "SAMPLE.TBL",
      "uuid": "d7989206-380f-0e81-8056-240501101100",
      "read_timestamp": "2019-11-07T07:37:16.808Z",
      "source_timestamp": "2019-11-07T02:15:39",  
      "source_metadata": {
        "log_file": ""
        "scn": 15869116216871,
        "row_id": "AAAPwRAALAAMzMBABD",
        "is_deleted": false,
        "database": "DB1",
        "schema": "ROOT",
        "table": "SAMPLE"
        "change_type": "INSERT",
        "tx_id": 
        "rs_id": "0x0073c9.000a4e4c.01d0",
        "ssn": 67,
      },
      "payload": {
        "THIS_IS_MY_PK": "1231535353",
        "FIELD1": "foo",
        "FIELD2": "TLV",
      }
    }

#### UPDATE (T1)

The message payload consists of the entirety of the new row. It doesn't include previous values.

    {
      "stream_name": "projects/myProj/locations/myLoc/streams/Oracle-to-Source",
      "read_method": "oracle-cdc-logminer",
      "object": "SAMPLE.TBL",
      "uuid": "e6067366-1efc-0a10-a084-0d8701101101",
      "read_timestamp": "2019-11-07T07:37:18.808Z",
      "source_timestamp": "2019-11-07T02:17:39",  
      "source_metadata": {
        "log_file": 
        "scn": 15869150473224,
        "row_id": "AAAGYPAATAAPIC5AAB",
        "is_deleted": false,
        "database":
        "schema": "ROOT",
        "table": "SAMPLE"
        "change_type": "UPDATE",
        "tx_id":
        "rs_id": "0x006cf4.00056b26.0010",
        "ssn": 0,
      },
      "payload": {
        "THIS_IS_MY_PK": "1231535353",
        "FIELD1": null,
        "FIELD2": "TLV",
      }
    }

#### DELETE (T2)

The message payload consists of the entirety of the new row.

    {
      "stream_name": "projects/myProj/locations/myLoc/streams/Oracle-to-Source",
      "read_method": "oracle-cdc-logminer",
      "object": "SAMPLE.TBL",
      "uuid": "c504f4bc-0ffc-4a1a-84df-6aba382fa651",
      "read_timestamp": "2019-11-07T07:37:20.808Z",
      "source_timestamp": "2019-11-07T02:19:39",
      "source_metadata": {
        "log_file": 
        "scn": 158691504732555,
        "row_id": "AAAGYPAATAAPIC5AAC",
        "is_deleted": true,
        "database":
        "schema": "ROOT",
        "table": "SAMPLE"
        "change_type": "DELETE",
        "tx_id":
        "rs_id": "0x006cf4.00056b26.0011",
        "ssn": 0,
      },
      "payload": {
        "THIS_IS_MY_PK": "1231535353",
        "FIELD1": null,
        "FIELD2": "TLV",
      }
    }

## Ordering and consistency

This section covers how Datastream handles ordering and
consistency.

### Ordering

Datastream doesn't guarantee in-order event delivery, but each
event contains the full row of data, the timestamp of when the data was written
to the source, and ordering metadata.

Events that are part of an initial backfill have the
[`read_method`](https://docs.cloud.google.com/datastream/docs/events-and-streams#genericmetadata) field set to `backfill` or ending with
`-backfill`, `-fulldump`, or `-incremental`. Backfill events are generated out
of order by design and can be consumed in any order.

Events that are part of ongoing change data capture (CDC) replication have the
`read_method` field set to `cdc` or a source-specific CDC value, such as
`mysql-cdc-binlog`, `oracle-cdc-logminer`, `postgres-cdc-wal`, `sqlserver-cdc`,
or `salesforce-cdc`. How you order ongoing replication events across all
sources depends on your destination:

| Destination | Description |
|---|---|
| BigQuery | How events are ordered in BigQuery depends on the [write mode](https://docs.cloud.google.com/datastream/docs/configure-bigquery-destination#configure-write-mode) that you configure for your stream: - **Merge write mode**: BigQuery merges out-of-order events in the correct sequence automatically using the event metadata and an internal change sequence number (CSN). Tables without a primary key are written in append-only mode. - **Append-only write mode** , including [Apache Iceberg tables](https://docs.cloud.google.com/datastream/docs/destination-blmt): Datastream appends change events to the destination table and populates the [`datastream_metadata`](https://docs.cloud.google.com/datastream/docs/destination-bigquery#table-metadata) `STRUCT` column. To determine the order in which events occurred at the source, sort the events using the `datastream_metadata.change_sequence_number` field or the `datastream_metadata.sort_keys` array field. |
| Cloud Storage | In Cloud Storage, events from the same time period can span more than one file. To determine the order in which events occurred at the source, sort the events using the `https://docs.cloud.google.com/datastream/docs/events-and-streams#genericmetadata` array field in the generic metadata. |

### Consistency

Datastream ensures that the data from the source database is delivered to the destination at least once. No event is missed, but there's a possibility of duplicate events in the stream. The window for duplicate events should be on the order of minutes, and the universally unique identifier (UUID) of the event in the [event metadata](https://docs.cloud.google.com/datastream/docs/events-and-streams#genericmetadata) can be used to detect duplicates.

However, don't rely solely on the UUID to deduplicate events downstream. If a pipeline interruption occurs, Datastream re-reads the source log from the last saved checkpoint to ensure delivery. Because Datastream dynamically generates UUIDs when it reads an event, this type of retry results in fresh UUIDs for the exact same source data. Additionally, during bulk operations in the source database, multiple distinct rows might share the same change sequence number (CSN).

To reliably deduplicate data, partition the events by the source table primary keys, and group them using the change sequence number (CSN) (the `change_sequence_number` field). This ensures that retries of the same event are correctly collapsed while preserving legitimately distinct row updates from bulk transactions.

When database log files contain uncommitted transactions, if any transactions are rolled back, then the database reflects this in the log files as "reverse" data manipulation language (DML) operations. For example, a rolled-back `INSERT` operation will have a corresponding `DELETE` operation. Datastream reads these operations from the log files.

## About streams

Every stream has metadata that describes both the stream and the source from which it pulls data. This metadata includes information such as the stream name, the source and destination connection profiles.

To view the complete definition of the Stream object, see the [API Reference](https://docs.cloud.google.com/datastream/docs/reference/rest/v1/projects.locations.streams) documentation.

### Stream state and status

A stream can be in one of the following states:

- `Not started`
- `Starting`
- `Running`
- `Draining`
- `Paused`
- `Failed`
- `Failed permanently`

> [!NOTE]
> For more information about these states, including how they're used in the lifecycle of a stream, see [Stream lifecycle](https://docs.cloud.google.com/datastream/docs/stream-states-and-actions).

You can use the logs to find additional status information, such as the tables
backfilling or number of rows processed. You can also use the
[`FetchStreamErrors`](https://docs.cloud.google.com/datastream/docs/manage-streams#fetchstreamerrors) API to
retrieve errors.

## Object metadata available using the discover API

The discover API returns objects which represent the structure of the objects
defined in the data source or destination that's represented by the connection
profile. Each object has metadata on the object itself, as well as for every
field of data that it pulls. This metadata is available using the [discover
API](https://docs.cloud.google.com/datastream/docs/reference/rest/v1/projects.locations.connectionProfiles/discover).

## What's next

- To learn more about streams, see [Stream lifecycle](https://docs.cloud.google.com/datastream/docs/stream-states-and-actions).
- To learn how to create a stream, see [Create a stream](https://docs.cloud.google.com/datastream/docs/create-a-stream).