You can use Datastream to monitor the data and events that a stream processes and configure alerting policies to track replication health. In Datastream, an event represents a single change in the source, such as a new row added to a database table. To monitor stream errors, see Troubleshoot a stream.
Datastream displays stream performance in graphs for throughput, unsupported events, data freshness, system latency, and total latency. Throughput shows the rate at which a stream transfers data, in megabytes, or processes events. Unsupported events tracks the number of source events that Datastream can't process.
Data freshness measures the time between when data is written to the source
and when Datastream reads the oldest event being processed. This
metric is 0 when there are no new events to read, and it reflects activity
spikes only after Datastream starts reading those events.
System latency tracks the time between when Datastream reads
an event and writes it to the destination. Total latency represents the
end-to-end delay from when data is written to the source to when the
corresponding events reach the destination.
Monitor data or events processed
Go to the Streams page in the Google Cloud Console.
Click the stream that you want to monitor.
Optional. Scroll until the Data freshness graph appears. For this graph, click the Create alerting policy link to create an alerting policy for it. An alerting policy describes a set of conditions that you want to monitor for the graph.
After clicking the link, the Create alerting policy page appears in Cloud Monitoring. On this page, you can define the alerting policy for the graph. This includes specifying which criteria will trigger the policy, who'll be notified if the criteria are met, and how they'll be notified.
For more information about alerting policies, including how to create them, see Managing metric-based alerting policies.
Click the MONITORING tab.
Scroll until the graph appears that represents the data or events that Datastream is monitoring.
Optionally, if you're looking at the Throughput graph, then select (bytes/sec) to see the amount of data that Datastream transfers from the source to the destination, or (event/sec) to see how many events are associated with the data being transferred.
To view how much data or how many events Datastream processed over a period of hours or days, click 1 hour, 6 hours, 12 hours, 1 day, 2 days, 4 days, 7 days, 14 days, or 30 days.
Or, to set a custom range, select the Custom menu, and then use the Calendar picker to specify a start date and time and end date and time for the amount of data or the number of events that you want to view.
Datastream updates the graph in the pane to reflect your selection and displays the interval that you specified.
For example, if today is October 31, 2021, and you click 30 days, the graph will show the amount of data or the number of events Datastream processed since October 01, 2021.
Hold the pointer over the line in the graph to display how much data or how many events Datastream processed for a particular date and time.
What's next
- To learn more about streams, see Stream lifecycle.
- To learn how to view information about your stream, see View a stream.
- To learn how to modify a stream, see Modify a stream.