Monitor workload framework metrics

This document explains how to collect and view Megascale XLA (MXLA), Orbax, and PyGrain (Grain) metrics for Cloud TPU workloads running on Google Kubernetes Engine (GKE), and how to view analyzers built on top of these metrics in the ML Diagnostics platform. By monitoring these metrics, you can identify and troubleshoot performance bottlenecks across multislice communication, checkpointing, and data loading.

Each set of metrics targets a distinct component of your workload to help you diagnose issues at different stages of your pipeline:

  • Megascale XLA (MXLA) metrics: Measure cross-slice Data Center Network (DCN) transfer latencies, device-to-host (D2H) and host-to-device (H2D) transfer times, compute latency, and end-to-end collective operations in multislice TPU workloads. Use these metrics to identify stragglers and hardware bottlenecks.
  • Orbax metrics: Track asynchronous checkpoint saves and restore durations, data volume, and write throughput from the Orbax checkpointing library. Use these metrics to detect checkpointing delays, stalls, or excessive training-thread blocking time.
  • PyGrain (Grain) metrics: Measure dataset iteration latency, data source read times, total bytes read, and prefetch buffer readiness from the PyGrain data-loading library. Use these metrics to determine whether your workload is input-bound due to I/O or data-loading bottlenecks.

You can access these metrics through a Prometheus port that you read directly, the TPU Monitoring Library, the tpu-info CLI, or Cloud Monitoring.

The ML Diagnostics platform also includes workload monitoring analyzers that automatically analyzes these metrics when a workload experiences performance degradation. These analyzers determine whether a drop in TPU duty cycle is caused by MXLA network transfer latency, Orbax asynchronous checkpointing, or PyGrain data loading.

Megascale metrics

Megascale XLA (MXLA) is a communication runtime and compiler extension that coordinates collective operations across multiple TPU slices over the Data Center Network (DCN). While Inter-Chip Interconnect (ICI) links handle high-speed communication between TPU chips within a single slice, MXLA manages data transfers between slices, host-side reductions, and data movement between host memory and TPU high bandwidth memory (HBM).

When you run multislice workloads on TPUs, you can collect MXLA metrics to troubleshoot cross-slice network and host-processing bottlenecks. GKE collects these metrics by default as GKE system metrics, and you can view them in Cloud Monitoring. The metrics are also automatically available in the ML Diagnostics system metrics tab, where they are aggregated automatically for the ML run workload context.

The following table lists the available Megascale metrics:

Metric name GKE metric name Description
MXLA DCN transfer latency container/multislice/network/dcn_transfer_latencies

Measures round-trip Data Center Network (DCN) transfer latency for each chunk of data. Outliers can correlate with faulty network hardware.

The metric has the following properties:

  • Metric type: CUMULATIVE
  • Value type: DISTRIBUTION
  • Unit: microseconds (us)
MXLA inbound transfer latency container/multislice/network/dcn_inbound_transfer_latencies

Measures one-way inbound DCN transfer latency for each chunk of data. Outliers can correlate with faulty network hardware.

The metric has the following properties:

  • Metric type: CUMULATIVE
  • Value type: DISTRIBUTION
  • Unit: microseconds (us)
MXLA compute latency container/multislice/accelerator/compute_latencies

Measures the host compute latency required to complete a reduction operation. Outliers can indicate CPU or memory issues, such as dual inline memory module (DIMM) faults.

The metric has the following properties:

  • Metric type: CUMULATIVE
  • Value type: DISTRIBUTION
  • Unit: microseconds (us)
MXLA e2e collective latency container/multislice/network/collective_end_to_end_latencies

Measures end-to-end collective latency over the DCN, from when the host initiates the operation to when all peers receive the output. Includes host-side data reduction and sending output to the TPU. Results are strings that detail the buffer size, operation type, and mean, P50, P90, P95, and P99.9 latencies.

The metric has the following properties:

  • Metric type: CUMULATIVE
  • Value type: DISTRIBUTION
  • Unit: microseconds (us)
MXLA d2h latency container/multislice/accelerator/device_to_host_transfer_latencies

Measures one-way device-to-host (D2H) transfer latency for each chunk of data. Outliers can correlate with faulty network hardware.

The metric has the following properties:

  • Metric type: CUMULATIVE
  • Value type: DISTRIBUTION
  • Unit: microseconds (us)
MXLA h2d latency container/multislice/accelerator/host_to_device_transfer_latencies

Measures one-way host-to-device (H2D) transfer latency for each chunk of data. Outliers can correlate with faulty network hardware.

The metric has the following properties:

  • Metric type: CUMULATIVE
  • Value type: DISTRIBUTION
  • Unit: microseconds (us)

Orbax metrics

Orbax is a distributed checkpointing library for JAX workloads that saves and restores model weights, optimizer states, and dataset iterators across single-host and multi-host environments. To minimize training downtime, Orbax supports asynchronous checkpointing, which copies states from TPU memory to host memory during the synchronous phase and then writes the checkpoint to storage in a background thread while training continues.

If your workload uses Orbax for checkpointing, you can collect Orbax metrics to troubleshoot checkpointing delays, slow write throughput, or stalls. Orbax exports these metrics on a Prometheus port, which you can read directly or send to Cloud Monitoring by using Google Cloud Managed Service for Prometheus. To enable these metrics in GKE, Cloud Monitoring, or ML Diagnostics, see Access Orbax and PyGrain metrics.

The following table lists the available Orbax metrics:

Prometheus metric name Description
jax_checkpoint_write_async_total_duration_secs

Total time for the entire asynchronous checkpoint write pipeline.

The metric has the following properties:

  • Metric path: /jax/checkpoint/write/async/total_duration_secs
  • Metric type: Histogram
  • Unit: seconds (s)
jax_orbax_write_gbytes

Total gigabytes written during a checkpoint save operation.

The metric has the following properties:

  • Metric path: /jax/orbax/write/gbytes
  • Metric type: Histogram
  • Unit: gigabytes (GBy)
jax_orbax_write_gbytes_per_sec

Write throughput in gigabytes per second (GBps) across both the synchronous and asynchronous phases of a checkpoint save operation.

The metric has the following properties:

  • Metric path: /jax/orbax/write/gbytes_per_sec
  • Metric type: Histogram
  • Unit: gigabytes per second (GBps)
jax_checkpoint_write_async_blocking_duration_secs

Time spent in the blocking phase of an asynchronous checkpoint save before handing off to the background thread.

The metric has the following properties:

  • Metric path: /jax/checkpoint/write/async/blocking_duration_secs
  • Metric type: Histogram
  • Unit: seconds (s)
jax_orbax_read_total_duration_secs

Total end-to-end duration of a checkpoint restore operation.

The metric has the following properties:

  • Metric path: /jax/orbax/read/total_duration_secs
  • Metric type: Histogram
  • Unit: seconds (s)
jax_orbax_read_blocking_duration_secs

Duration of the synchronous (blocking) preparation phase of a checkpoint restore operation.

The metric has the following properties:

  • Metric path: /jax/orbax/read/blocking_duration_secs
  • Metric type: Histogram
  • Unit: seconds (s)
jax_orbax_write_async_start_total

Count of asynchronous checkpoint write operations started.

The metric has the following properties:

  • Metric path: /jax/orbax/write/async/start
  • Metric type: Counter
  • Unit: count (1)

PyGrain metrics

PyGrain (Grain) is a Python library for reading, preprocessing, and batching input data for machine learning training and evaluation workloads. PyGrain provides deterministic, reproducible data pipelines that prefetch and transform records on the host CPU so that TPU accelerators aren't stalled waiting for input batches.

If your workload uses PyGrain for data loading, you can collect PyGrain metrics to troubleshoot input-pipeline bottlenecks, slow storage reads, or depleted prefetch buffers. PyGrain exports these metrics on a Prometheus port, which you can read directly or send to Cloud Monitoring by using Google Cloud Managed Service for Prometheus. To enable these metrics in GKE, Cloud Monitoring, or ML Diagnostics, see Access Orbax and PyGrain metrics.

The following table lists the available PyGrain metrics:

Prometheus metric name Description
grain_python_dataset_next_duration_ns

Time spent in the next method call for the dataset.

The metric has the following properties:

  • Metric path: /grain/python/dataset/next_duration_ns
  • Metric type: Histogram
  • Unit: nanoseconds (ns)
grain_python_dataset_prefetch_buffer_ready_count

Number of ready elements in the prefetch buffer.

The metric has the following properties:

  • Metric path: /grain/python/dataset/prefetch_buffer_ready_count
  • Metric type: Histogram
  • Unit: count (1)
grain_python_data_sources_bytes_read_total

Total bytes read from data sources.

The metric has the following properties:

  • Metric path: /grain/python/data_sources/bytes_read
  • Metric type: Counter
  • Unit: bytes (By)
grain_python_dataset_source_read_time_ns

Time spent reading from the data source.

The metric has the following properties:

  • Metric path: /grain/python/dataset/source_read_time_ns
  • Metric type: Histogram
  • Unit: nanoseconds (ns)
grain_python_data_loader_iterator_get_next

Time spent in the next method call for the DataLoaderIterator class.

The metric has the following properties:

  • Metric path: /grain/python/data_loader/iterator_get_next
  • Metric type: Histogram
  • Unit: nanoseconds (ns)

Prerequisites

Before you collect Megascale, Orbax, or PyGrain metrics or use the associated ML Diagnostics analyzers, make sure that your environment meets the requirements in the following sections.

Megascale requirements

To collect Megascale metrics and view them in Cloud Monitoring, meet the following version requirements:

  • Install LibTPU version 0.40.0 or later.
  • To view these metrics in Cloud Monitoring, install GKE version 1.36.4-gke.1247000 or later.

Orbax and PyGrain requirements

To export Orbax and PyGrain metrics from your workload, meet the following package and container requirements:

  • Python packages:
    • Install Orbax (orbax-checkpoint) version 0.12.1 or later.
    • Install PyGrain (grain) version 0.2.18 or later.
    • Install the prometheus-client Python package in your container image. The orbax-checkpoint package installs prometheus-client automatically as a dependency, but grain doesn't. If your workload uses only PyGrain, install the package by running pip install prometheus-client.
  • Workload container configuration:
    • Set the following environment variables:
      • ENABLE_ORBAX_PROMETHEUS_TELEMETRY="true" (enables Orbax metrics)
      • ENABLE_PYGRAIN_PROMETHEUS_TELEMETRY="true" (enables PyGrain metrics)
    • Expose TCP port 9431 on the workload container.

The following example container specification sets the telemetry environment variables and exposes TCP port 9431:

containers:
- name: WORKLOAD_CONTAINER
  env:
  - name: ENABLE_ORBAX_PROMETHEUS_TELEMETRY
    value: "true"
  - name: ENABLE_PYGRAIN_PROMETHEUS_TELEMETRY
    value: "true"
  ports:
  - name: framework-metrics
    containerPort: 9431
    protocol: TCP

Replace WORKLOAD_CONTAINER with the name of your workload container.

ML Diagnostics analyzer requirements

To use the ML Diagnostics analyzers for Megascale, Orbax, and PyGrain metrics, install GKE version 1.36.4-gke.1247000 or later.

Access Megascale metrics

When your cluster runs a supported GKE version and LibTPU, GKE automatically collects Megascale metrics so that you can view them in Cloud Monitoring. You can also deploy a Prometheus collector of your choice to read Megascale metrics directly from the MXLA runtime's Prometheus port.

Access Orbax and PyGrain metrics

When you enable Prometheus telemetry on your workload container, Orbax and PyGrain exports metrics on TCP port 9431. You can access these metrics in either of the following ways:

  • Read the Prometheus endpoint directly: Deploy a Prometheus collector of your choice to read Orbax and PyGrain metrics from port 9431 on the workload container.
  • Send metrics to Cloud Monitoring: Use Google Cloud Managed Service for Prometheus to read port 9431 on each node and send the metrics to Cloud Monitoring.

View metrics with Google Cloud Managed Service for Prometheus

To send Orbax and PyGrain metrics from port 9431 to Cloud Monitoring, deploy a Google Cloud Managed Service for Prometheus PodMonitoring custom resource in your GKE cluster. The PodMonitoring resource instructs the Google Cloud Managed Service for Prometheus collector on each node to read port 9431 and send the metrics to Cloud Monitoring. For more information, see Configure a PodMonitoring resource.

The prometheus.googleapis.com endpoint and the Google Cloud Managed Service for Prometheus collector are enabled by default in GKE clusters. Using Google Cloud Managed Service for Prometheus incurs additional charges.

Enable managed collection

Managed collection for Google Cloud Managed Service for Prometheus is enabled by default in GKE Standard clusters running GKE version 1.27 or later. For more information, see Enable managed collection.

If managed collection is disabled on your cluster, enable it:

gcloud container clusters update CLUSTER_NAME \
  --enable-managed-prometheus \
  --region=REGION

Replace the following:

  • CLUSTER_NAME: the name of your GKE cluster.
  • REGION: the region where your cluster is located.

If you modified the default permissions on the node service account, make sure that it has the Monitoring Metric Writer (roles/monitoring.metricWriter) role.

Configure the workload Pod template

To prepare your workload Pods for scraping, update the Pod template in your JobSet, Job, or Deployment manifest:

  1. Add a Pod label, such as ml-framework-metrics: "enabled", so that the PodMonitoring resource can select the workload Pods.
  2. Set the ENABLE_ORBAX_PROMETHEUS_TELEMETRY and ENABLE_PYGRAIN_PROMETHEUS_TELEMETRY environment variables to "true".
  3. Expose TCP port 9431 and assign it a port name, such as framework-metrics.

The following example Pod template configures these fields:

template:
  metadata:
    labels:
      ml-framework-metrics: "enabled"     # used by the PodMonitoring selector
  spec:
    containers:
    - name: WORKLOAD_CONTAINER
      env:
      - name: ENABLE_ORBAX_PROMETHEUS_TELEMETRY
        value: "true"
      - name: ENABLE_PYGRAIN_PROMETHEUS_TELEMETRY
        value: "true"
      ports:
      - name: framework-metrics
        containerPort: 9431
        protocol: TCP

Replace WORKLOAD_CONTAINER with the name of your workload container.

Verify the metrics endpoint

Before you deploy the PodMonitoring resource, verify that your workload container exports metrics on port 9431. Forward port 9431 from a running workload Pod and query the /metrics endpoint:

kubectl -n NAMESPACE port-forward pod/WORKLOAD_POD 9431:9431
curl -s localhost:9431/metrics | grep -E '^(jax_orbax|jax_checkpoint|grain_python)'

Replace the following:

  • NAMESPACE: the Kubernetes namespace where your workload runs.
  • WORKLOAD_POD: the name of a running workload Pod.

Orbax initializes each metric the first time it records a value, so Orbax metrics appear only after the first checkpoint save or restore operation.

Deploy the PodMonitoring resource

To configure Google Cloud Managed Service for Prometheus to read your workload Pods, complete the following steps:

  1. Save the following manifest as orbax-grain-podmonitoring.yaml:

    apiVersion: monitoring.googleapis.com/v1
    kind: PodMonitoring
    metadata:
      name: orbax-grain-metrics
      namespace: WORKLOAD_NAMESPACE  # must match the workload namespace
    spec:
      selector:
        matchLabels:
          ml-framework-metrics: "enabled"
      endpoints:
      - port: framework-metrics             # the named containerPort (9431)
        path: /metrics
        interval: 30s                        # a longer interval reduces sample volume and cost
        metricRelabeling:                    # optional: keep only Orbax and PyGrain series to control cost
        - action: keep
          sourceLabels: [__name__]
          regex: (jax_orbax_|jax_checkpoint_|grain_python_).+
      targetLabels:                          # optional: copy the JobSet name onto every series
        fromPod:
        - from: jobset.sigs.k8s.io/jobset-name
          to: jobset_name
    

    Replace WORKLOAD_NAMESPACE with the Kubernetes namespace where your workload runs.

  2. Apply the manifest and confirm that the PodMonitoring resource is created:

    kubectl apply -f orbax-grain-podmonitoring.yaml
    kubectl get podmonitoring -A
    

A PodMonitoring resource reads Pods only in its own namespace. If your workloads run across multiple namespaces, use the same specification with kind: ClusterPodMonitoring and omit the metadata.namespace field. A ClusterPodMonitoring resource discovers matching Pods across all namespaces.

Optional: Verify that Google Cloud Managed Service for Prometheus is scraping the Pods

To verify that Google Cloud Managed Service for Prometheus is actively scraping your workload Pods, complete the following steps:

  1. In the gmp-public namespace, edit the OperatorConfig resource and set features.targetStatus.enabled to true.
  2. Inspect the PodMonitoring resource and confirm that the targets report Health: up:

    kubectl -n NAMESPACE describe podmonitoring/orbax-grain-metrics
    

    Replace NAMESPACE with the Kubernetes namespace where your workload runs.

  3. After you finish troubleshooting, set features.targetStatus.enabled back to false in the OperatorConfig resource. Enabling target status is recommended only when you are investigating active issues.

Query the metrics in Cloud Monitoring

To query the metrics sent to Cloud Monitoring, complete the following steps:

  1. In the Google Cloud console, go to Monitoring > Metrics Explorer.
  2. Select PromQL, and then enter the following query to confirm that your targets are reporting:

    up{namespace="NAMESPACE"}
    

    Replace NAMESPACE with your workload's Kubernetes namespace.

In the menu-based metric picker, metrics are stored using the format prometheus.googleapis.com/METRIC_NAME/TYPE, where TYPE is counter, histogram, or gauge. Each library constructs its Prometheus metric name from its metric path by replacing / with _. Counters append a _total suffix, and histograms are exported as _bucket, _sum, and _count time series.

You can use the following example PromQL queries to analyze Orbax and PyGrain metrics:

# Asynchronous checkpoint saves started per Pod
sum by (pod) (increase(jax_orbax_write_async_start_total[1h]))

# Average time training is blocked by asynchronous checkpointing
sum(rate(jax_checkpoint_write_async_blocking_duration_secs_sum[10m]))
  / sum(rate(jax_checkpoint_write_async_blocking_duration_secs_count[10m]))

# PyGrain P99 dataset iterator next() latency (nanoseconds)
histogram_quantile(0.99, sum by (le) (rate(grain_python_dataset_next_duration_ns_bucket[5m])))

# PyGrain bytes read per second, by data source
sum by (source) (rate(grain_python_data_sources_bytes_read_total[5m]))

# Median prefetch-buffer readiness (low values can indicate an I/O bottleneck)
histogram_quantile(0.5, sum by (le) (rate(grain_python_dataset_prefetch_buffer_ready_count_bucket[5m])))

Cost considerations

Google Cloud Managed Service for Prometheus billing is based on the number of ingested samples. To reduce costs, configure a longer scraping interval and use metricRelabeling to keep only the time series that you need. You can view the ingestion volume for each metric on the Metrics Management page in Cloud Monitoring.

View analyzer outputs in ML Diagnostics

ML Diagnostics workload monitoring and workload-contextualized system metrics are automatically enabled for GKE TPU JobSet and Job workloads running on GKE version 1.36.4-gke.1247000 and later. Workload monitoring tracks the aggregated TPU duty cycle for your workload. In the event of a 15% or greater drop in duty cycle (performance degradation), ML Diagnostics runs analyzers across the infrastructure, orchestrator, and framework layers.

If your workload runs across multiple slices (using the MXLA layer) or uses the PyGrain or Orbax libraries, ML Diagnostics runs the following framework analyzers to determine whether the performance degradation was caused by network transfer latency, data loading, or asynchronous checkpointing:

ML Diagnostics analyzer reports are provided at no additional charge, and generating analyzer results doesn't require configuring Google Cloud Managed Service for Prometheus for Orbax and PyGrain metrics. In the System metrics tab of the ML Diagnostics Google Cloud console, Megascale metrics appear automatically when you use supported LibTPU and GKE versions.

You can view analyzer outputs in either of the following ways:

  • Google Cloud console: In the ML Diagnostics Google Cloud console (see View machine learning runs), open the Monitoring overview tab, find the performance degradation event, and click View details to inspect the analyzer report.
  • Google Cloud CLI or REST API: Use the monitored-events commands to list and describe events and their analyzer reports. For more information, see Get started with the ML Diagnostics CLI.