Monitor performance using performance counters in Mainframe Connector

This guide describes how to enable and collect performance counter traces in Mainframe Connector.

Overview

Performance counters capture high-resolution execution metrics and thread-level state data during Mainframe Connector operations, such as dataset transcoding, Cloud Storage uploads, and BigQuery data loading. You can use counter traces to diagnose performance bottlenecks, memory or queue backpressure, thread blocking, network or I/O constraints, and mainframe system resource usage.

Key metrics captured

Performance counter traces include high-frequency samples of internal metrics such as:

  • Worker state tracking: Time spent in active, queue_blocked, io_blocked, compressing, and hashing states.
  • I/O and throughput: Read and write bytes, transfer throughput, and chunk transcode rates.
  • Queue dynamics: Buffer and worker queue sizes across execution pipeline stages.
  • System metrics: Low-level operation timings correlated to nanosecond-precision timestamps.

Performance counter traces collect execution metadata and timing statistics only. They don't collect dataset contents, customer records, or personally identifiable information (PII).

Enable performance counters

Performance counters are disabled by default. You can enable them by setting the MFC_PERF_FILE environment variable to the target output path before running Mainframe Connector jobs.

Shell environments

To enable performance counters in a shell environment inside Linux or z/OS UNIX System Services, export MFC_PERF_FILE before invoking Mainframe Connector:

export MFC_PERF_FILE="/tmp/mfc_perf.perf"
copy binary file:/path/to/file gs://bucket/path/to/object

To disable performance counter collection, unset the environment variable:

unset MFC_PERF_FILE

Mainframe JCL environment

To enable performance counters for a JCL job running Mainframe Connector, define MFC_PERF_FILE in your Mainframe Connector procedure or job's procedure PARM. When you provide your own PARM, always keep the default PARM (/I+ com.google.bqsh.Bqsh) too.

The following example shows how to enable performance counters for a JCL job:

//STEP01 EXEC BQSH,PARMDD=MFCPERF
//INFILE DD DSN=HLQ.DATA.FILENAME,DISP=SHR
//COPYBOOK DD DSN=HLQ.COPYBOOK.FILENAME,DISP=SHR
//MFCPERF DD *
ENVAR("MFC_PERF_FILE=/tmp/mfc_perf.perf")/I+
com.google.bqsh.Bqsh
/*
//STDIN DD *
BUCKET=my-target-bucket-1234
gsutil cp --replace gs://$BUCKET/tablename.orc
/*

Configure performance counters

You can configure the sampling frequency and maximum file size of performance counter logs using the following environment variables:

Environment variable Description Default value
MFC_PERF_FILE The path where performance counter metrics will be written. If unset or empty, performance counter collection is disabled. Disabled
MFC_PREF_COLLECTION_INTERVAL_MS The sampling frequency in milliseconds. Specifies how often metrics are flushed to the output file. 100 (0.1 seconds)
MFC_MAX_PERF_MAX_FILE_SIZE_BYTES The maximum size limit for the output file in bytes. Once reached, writing stops automatically to prevent filling up disk storage. 67108864 (64 MiB)

Send counter files to Google Support for analysis

The output file generated by the performance counters feature (for example, /tmp/mfc_performance.perf) is formatted as a binary dataset containing structured metric schemas and measurement records. It is not human-readable plain text and cannot be evaluated directly with standard log viewers.

The output file requires internal Google analysis tooling and specialized metric visualizers to decode state transitions and queue depth graphs. To analyze the performance data, submit the generated file to Google Cloud Support or your Technical Account Manager.

When you open a support case regarding performance issues, slow transcoding throughput, or unexpected job duration, follow these steps:

  1. Set MFC_PERF_FILE to a valid path (for example, /tmp/mfc_perf.perf).
  2. Reproduce the issue by running the job execution under performance monitoring.
  3. Retrieve the generated .perf file from the specified path.
  4. Attach the .perf file along with your job logs to your Google Cloud Support case for evaluation by Google engineers.