Smart sharding in Developer Device Platform Device Run

Smart sharding in Developer Device Platform Device Run distributes Android instrumentation tests and iOS XCTests across parallel shards using historical test execution timings stored in Cloud Storage. Unlike uniform sharding, which divides test cases by count regardless of how long each test runs, smart sharding balances shards by expected duration so that parallel shards finish around the same target time.

This document explains how the smart sharding algorithm calculates shard counts and assigns tests, describes the schema of the YAML timing record stored in your Cloud Storage bucket, and answers common questions about Android Test Orchestrator overhead and the 30-day rolling data window. For command-line instructions using the Google Cloud CLI, see Android instrumentation and iOS XCTest instructions.

How the smart sharding algorithm works

Sharding is not equivalent to how many tests can run concurrently. It's possible that available devices are less than the shard limit at the time of their request. The actual concurrent number of devices a user can be allocated may be lower than the number of shards they specified.

When you submit a session with --sharding-option=smart in the gcloud CLI or configure smartSharding in the REST API, Developer Device Platform runs a six-stage workflow across session creation, test execution, and completion:

  1. Read historical timing data and match the device context. Before scheduling shards, the Device Run service agent reads your smart sharding YAML file from Cloud Storage. It searches the records list for a shardingContext that matches the target job's deviceModel and osVersion (case-insensitive). If no exact match exists for that device and OS combination, Developer Device Platform falls back to the first available records entry in the file. If the YAML file doesn't exist yet, the session proceeds as a cold start with empty timing history.
  2. Filter samples to the 30-day rolling window and compute expected durations. For the selected context record, Developer Device Platform retains only timing samples whose recordTime falls within the past 30 days and computes the arithmetic mean duration for each test case. If Android Test Orchestrator is enabled with --orchestrator-version, Developer Device Platform also computes the 30-day mean of the ANDROID_TEST_ORCHESTRATOR overhead samples and adds that mean overhead to every test case's expected duration.
  3. Discover runnable tests and assign fallback estimates. On the worker host, Developer Device Platform statically inspects your test package—reading DEX files in the test APK on Android, or compiled .xctest symbols and .xctestrun filters on iOS—applies any --test-targets filters, and excludes ignored or suppressed tests (such as @Ignore or @Suppress). Each runnable test unit receives an expected duration based on three tiers:
    • Recorded test case: uses its 30-day historical mean duration (plus mean orchestrator overhead when Android Test Orchestrator is enabled). On Android, parameterized JUnit classes (such as classes using @RunWith(Parameterized.class), JUnitParamsRunner, or TestParameterInjector) are scheduled as a single class-level unit whose expected duration is the sum of all recorded method durations for that class.
    • Unrecorded test case in an existing suite: uses the mean expected duration of all recorded non-zero test cases in the matched record.
    • Cold start (no recorded history): uses a default estimate of 30 seconds per test unit.
  4. Calculate the shard count. Developer Device Platform sums the expected durations of all runnable test units and divides by your configured --smart-sharding-target-duration (valid from 2m to 1h), rounding up to the nearest whole shard:

    optimal_shards = max(1, ceil(total_expected_duration / target_duration))
    

    Next, Developer Device Platform sets the final shard count between 1 and the lowest of the following limits:

    • The total number of runnable test units (preventing empty shards).
    • Your --smart-sharding-max-shard-count value, if set greater than 0.
    • The platform limit (20 shards for physical devices or 200 shards for Android virtual devices).
  5. Assign tests using longest-processing-time-first bin packing. Developer Device Platform sorts all runnable test units in descending order of expected duration (slowest to fastest) and initializes a min-heap of empty shards ordered by accumulated duration. Taking one test unit at a time, it assigns the slowest remaining test to whichever shard has the smallest accumulated duration and updates that shard's total. Scheduling slow "long-pole" tests first isolates them onto separate shards, while shorter tests fill the remaining gaps to keep total shard durations balanced.

  6. Record passing timings and update Cloud Storage. After all shards finish, Developer Device Platform collects the measured startTime to endTime durations of passing test cases, prunes any samples older than 30 days across all device contexts in the YAML file, removes any context records that have no remaining valid samples, and writes the updated YAML file back to Cloud Storage using generation-matched optimistic concurrency retries.

Smart sharding YAML file structure

When you run a session with the gcloud CLI, --smart-sharding-record-name specifies the base filename (excluding the YAML extension) of the timing record stored in your Google Cloud project's Cloud Storage bucket:

gs://BUCKET_NAME/automation/smart-sharding/RECORD_NAME.yaml

Replace the following placeholders:

  • BUCKET_NAME: the Cloud Storage bucket name specified with --bucket-name, or <var>PROJECT_ID</var>-devicerun by default (where PROJECT_ID is your Google Cloud project ID).
  • RECORD_NAME: the record identifier passed to --smart-sharding-record-name without the YAML file extension (for example, checkout-suite maps to automation/smart-sharding/checkout-suite.yaml).

Example YAML timing record

The following example shows a smart sharding YAML file (checkout-suite.yaml) containing timing records for two device contexts—an Android device running with Android Test Orchestrator and an iOS device:

records:
- shardingContext:
    deviceModel: shiba
    osVersion: '34'
  testCaseTimingRecords:
  - testCaseName: com.example.shop.CheckoutTest#testCompletePurchase
    timingSamples:
    - duration: 42.180s
      recordTime: '2026-11-08T14:20:11Z'
    - duration: 44.050s
      recordTime: '2026-11-09T16:45:03Z'
  - testCaseName: com.example.shop.CartTest#testApplyPromoCode
    timingSamples:
    - duration: 8.600s
      recordTime: '2026-11-08T14:20:11Z'
    - duration: 9.120s
      recordTime: '2026-11-09T16:45:03Z'
  conditionalOverheads:
  - condition: ANDROID_TEST_ORCHESTRATOR
    timingSamples:
    - duration: 2.850s
      recordTime: '2026-11-08T14:20:11Z'
    - duration: 3.150s
      recordTime: '2026-11-09T16:45:03Z'
- shardingContext:
    deviceModel: iphonese3
    osVersion: '18.4'
  testCaseTimingRecords:
  - testCaseName: ShopTests.CheckoutTests#testApplePaySheet
    timingSamples:
    - duration: 19.340s
      recordTime: '2026-11-09T17:10:29Z'

YAML field reference

The smart sharding YAML file serializes the SuiteTiming schema using camelCase field names. The following table describes each field in the file:

Field Type Description
metadata Map (string to string) Optional suite-level key-value metadata preserved across aggregation updates.
records List of objects Per-device timing entries (SuiteTimingWithContext), partitioned by hardware model and OS version.
records[].shardingContext Object Hardware and OS dimensions that scope a set of timing measurements.
records[].shardingContext.deviceModel string Model identifier of the device where timings were recorded (for example, shiba on Android or iphonese3 on iOS).
records[].shardingContext.osVersion string OS or SDK version of the device (for example, Android API level '34' or iOS version '18.4').
records[].testCaseTimingRecords List of objects Historical execution durations for individual passing test cases in this context.
records[].testCaseTimingRecords[].testCaseName string Fully qualified test identifier in package.ClassName#methodName format (or package.ClassName if method metadata is absent).
records[].testCaseTimingRecords[].timingSamples List of objects Rolling 30-day list of measured durations for the test case, appended after passing runs.
timingSamples[].duration Duration string Measured duration formatted in seconds with optional fractional precision (for example, 10s or 42.180s).
timingSamples[].recordTime Timestamp string RFC 3339 UTC timestamp indicating when Developer Device Platform recorded the sample (for example, '2026-11-09T16:45:03Z').
records[].conditionalOverheads List of objects Optional execution overheads measured under specific runner modes, scoped to shardingContext. Omitted when no conditional overheads apply.
records[].conditionalOverheads[].condition string Identifier for the execution condition. Supported value: ANDROID_TEST_ORCHESTRATOR.
records[].conditionalOverheads[].timingSamples List of objects Rolling 30-day list of per-transition overhead samples recorded from qualifying shards.

Understand ANDROID_TEST_ORCHESTRATOR overhead timings

When you run Android instrumentation tests with --orchestrator-version, your smart sharding YAML file includes a conditionalOverheads entry with condition: ANDROID_TEST_ORCHESTRATOR. Unlike testCaseTimingRecords, these overhead samples aren't attached to an individual test method or a numbered shard.

What one overhead duration sample represents

When Android Test Orchestrator is enabled, it runs each test method in its own isolated Instrumentation process. Individual test startTime and endTime timestamps capture only the time spent inside each test method—they don't include the inter-test gap spent stopping the previous app process, optionally clearing package data (clearPackageData), and starting a fresh Instrumentation process for the next test.

To measure that inter-test transition cost, Developer Device Platform inspects each passing shard that runs two or more test cases (N > 1) and calculates:

  1. Total shard execution span: the elapsed wall-clock time from the earliest test startTime in the shard to the latest test endTime in the shard.
  2. Total test case duration: the sum of (endTime - startTime) across all N test cases in that shard.
  3. Average per-transition overhead: because N sequential tests have N - 1 transitions between tests, Developer Device Platform subtracts the summed test durations from the shard span and divides by N - 1:

    overhead_sample = max(0, total_span - total_test_duration) / (N - 1)
    

Each passing multi-test shard in your job contributes one TimingSample to ANDROID_TEST_ORCHESTRATOR. For example, if a session runs across three shards and each shard passes with at least two test cases, that session appends three overhead samples with the session's recordTime. Shards that run only a single test case (N = 1) have zero inter-test transitions and don't emit an overhead sample.

Why overhead isn't linked to a test case or shard

Developer Device Platform stores ANDROID_TEST_ORCHESTRATOR samples at the shardingContext (deviceModel and osVersion) level rather than per test or per shard for two reasons:

  • Overhead depends on the device and OS, not test logic. Process startup, binder initialization, and package-data clearing take roughly the same time regardless of which JUnit method runs next, whereas shard groupings change dynamically whenever test timings or shard counts change.
  • Separating overhead preserves clean baseline test timings. Keeping process-isolation overhead out of testCaseTimingRecords means you can enable or turn off --orchestrator-version across runs that share the same YAML record without distorting the underlying per-test execution durations.

How Developer Device Platform uses overhead during shard assignment

When --orchestrator-version is set on a new session, Developer Device Platform computes the 30-day average of valid ANDROID_TEST_ORCHESTRATOR samples (avg_overhead) and adds it to every test case's 30-day average duration before calculating shard counts or packing shards:

expected_test_duration = avg_test_duration + avg_overhead

If --orchestrator-version isn't set on the session, avg_overhead is 0s and any existing ANDROID_TEST_ORCHESTRATOR entries in the YAML file are ignored for that run.

Adding avg_overhead to every test case affects sharding in three ways:

  • Accurate shard count for suites with many fast tests. Consider a suite of 60 unit-style instrumentation tests that each run in 1s, with a 3s average orchestrator transition overhead and a 2m (120s) target shard duration. Without overhead, the total expected duration looks like 60s, so all 60 tests pack into 1 shard that actually takes nearly four minutes to finish (60s of test execution plus 59 * 3s = 177s of orchestrator transitions). By adding the 3s overhead to each test (4s expected duration per test, or 240s total), Developer Device Platform allocates 2 shards (ceil(240s / 120s)) so each shard finishes near the two-minute target.
  • Balanced shard packing when test counts differ. During longest-processing-time-first bin packing, a shard assigned one 40s UI test incurs 43s of expected cost (40s + 3s), whereas a shard assigned 10 4s tests incurs 70s of expected cost (10 * (4s + 3s)). Including per-test overhead prevents shards with many short tests from silently accumulating minutes of uncounted process-restart latency.
  • Realistic estimates for newly added tests. Because unrecorded tests in an existing suite inherit the mean expected duration of known tests—which already includes avg_overhead—new tests automatically account for orchestrator overhead on their very first run.

Understand the 30-day data window and initial run behavior

Smart sharding doesn't require 30 days of historical data before it begins optimizing your test runs. Even a single recorded sample activates duration-based shard calculation and bin packing on the very next session. Instead, 30 days is the rolling retention and lookback window that Developer Device Platform uses when reading and updating the YAML timing record.

Why smart sharding uses a 30-day rolling window

Every time Developer Device Platform reads or updates a smart sharding YAML file, it filters out any TimingSample whose recordTime is more than 30 days old. This 30-day window balances two engineering goals:

  • Adapting to codebase changes and bounding file size. App features, UI flows, and test implementations evolve over time. Automatically evicting samples older than 30 days keeps obsolete timings—for example, timings from before a screen redesign or network-layer refactor—from skewing current shard assignments. Evicting expired entries (and dropping device contexts that no longer have valid samples) also keeps the YAML file compact and well under the 32 MB limit.
  • Smoothing run-to-run variance while supporting weekly suites. A single test run can experience transient noise from garbage collection, background system services, or emulator scheduling. Averaging samples across a 30-day window smooths out one-off spikes while retaining enough history that nightly, weekly, or sprint-cadence continuous integration pipelines keep warm timing records between runs.

What to expect in weeks one and two

If you enable smart sharding on an existing test suite today, execution times follow a rapid warm-up curve rather than waiting weeks to improve:

  1. First run (cold start). Because the YAML file doesn't exist in Cloud Storage yet, Developer Device Platform assigns every runnable test unit the 30-second default estimate. It calculates the shard count as ceil(num_tests * 30s / target_duration) (capped by your maximum shard count and platform limits) and distributes tests across shards—similar to uniform sharding. On this first run:
    • If your tests average much less than 30 seconds, Run 1 might allocate more shards than needed (up to --smart-sharding-max-shard-count).
    • If your suite contains several multi-minute UI tests, two slow tests might land in the same shard by chance, making that shard the bottleneck for wall-clock completion time.
    • When Run 1 finishes, Developer Device Platform creates the YAML file in Cloud Storage with the measured durations of your passing tests.
  2. Second run (immediately after Run 1 completes). Smart sharding is already active. Developer Device Platform reads the durations recorded by Run 1, recalculates the shard count from your suite's real execution time, and isolates your slowest tests onto separate shards first. You typically see the primary reduction in wall-clock session duration and shard-count right-sizing starting on Run 2—even on the same day you enable smart sharding.
  3. Remainder of week one and week two. Over the next several runs, shard balance continues to tighten in three specific cases:
    • Smoothing single-run variance. As each passing run appends another TimingSample, transient fluctuations from Run 1 average out, reducing straggler shards that finish slightly past --smart-sharding-target-duration.
    • Filling in tests from initially failed runs. To prevent fast crashes or test timeouts from corrupting historical averages, Developer Device Platform records timings only for passing tests (and, on Android, only from shards whose overall suite status is PASSED). Any test that didn't record a timing on Run 1 uses the suite's mean duration on Run 2, and then switches to its own measured timing once it passes.
    • Adding new tests or device models. Any new test cases committed during weeks one and two automatically use the suite's mean expected duration instead of reverting to the 30-second default. If you add a second --device model to the session using the same record name, its first run uses the existing device's timings as an initial fallback and then records a dedicated shardingContext entry for subsequent runs.

Frequently asked questions

The following questions cover common operational scenarios when running smart sharding in CI/CD pipelines.

What's next