Troubleshoot pipeline failures

Linux

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

For information about access to this release, see the access request page.

If your Image Builder pipeline fails during the customization phase, review the log files, configure the interactive debug flag on your worker VM, or analyze intermediate debug images to identify compile-time or script errors.

Before you begin

• If you haven't already, set up authentication. Authentication verifies your identity for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine by selecting one of the following options:

  1. Install the Google Cloud CLI. After installation, initialize the Google Cloud CLI by running the following command:

     gcloud init

     If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

     Note: If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update.
  2. Set a default region and zone.

Review logging channels

Image Builder writes logs to multiple locations to help you diagnose failures at different levels of the stack:

• Cloud Build orchestrator logs: Displays the high-level builder steps in the Cloud Build Console. If a step fails, a link to the corresponding Cloud Storage workdir directory is printed.
• Customization VM agent logs: The customization agent captures the standard output (stdout) and standard error (stderr) of your customization steps. The orchestrator polls these records and streams them directly into the Cloud Build trace logs. If the logs are truncated or you need to inspect them in full, go to the folder matching your build ID in the _GCS_WORKDIR bucket.
• Serial console logs: If the guest OS fails to boot, the customization agent can't start. In these cases, look at the serial console logs. The orchestrator streams the worker VM's serial port output directly to your Cloud Storage workdir bucket. A link to the serial log file is printed at the end of the failing Cloud Build trace.
• Guest OS journal logs: For detailed system-level troubleshooting on Linux, the customization agent uploads a full system journal log copy to your Cloud Storage workdir bucket at the end of the run.

Resolve script failures using the debug flag

By default, if a customization step fails, Image Builder terminates execution and deletes the worker VM immediately to control costs.

To debug scripts interactively, you can enable the debug flag. This keeps the VM active, allowing you to connect directly to the active guest system by using SSH.

Enable the debug flag

In your imagebuilder.yaml file, set debug: true under the infrastructureConfig block:

infrastructureConfig:
  machineType: e2-standard-4
  zone: us-central1-a
  # Enable interactive debugging
  debug: true
  instanceDurationHours: 1.5

Trigger the build and wait for failure

Submit your build using standard methods. If the pipeline encounters a script error, the following occurs:

1. The orchestrator pauses the build execution.
2. Image Builder does not delete the worker VM. It remains running in your project.
3. The VM remains active for the time specified in the instanceDurationHours property before automatic termination.

Connect to the worker VM to debug

Look at the Cloud Build console output to retrieve the name of the active worker VM instance. Open a terminal and connect to the worker VM by using gcloud compute ssh:

gcloud compute ssh WORKER_VM_NAME \
    --zone=BUILD_ZONE \
    --project=PROJECT_ID

Once connected, you can run commands in real time, inspect target files, analyze running processes, review installer history logs, and test script modifications directly on the disk.

Analyze intermediate debug images

As an alternative to keeping the worker VM active with the debug flag, you can analyze the disk state by using the auto-generated debug image:

1. Whenever a build fails with debug: false (the default), the orchestrator exports the partially customized boot disk as a Compute Engine image.
2. The debug image is named using the pattern image-builder-debug-image-BUILD_ID and is assigned to the image-builder-debug-images image family.

3. To inspect the file system, create a temporary VM instance using this debug image as the boot disk:

   gcloud compute instances create temp-debug-vm \
       --image-family=image-builder-debug-images \
       --zone=ZONE \
       --project=PROJECT_ID
   

4. Connect to the temp-debug-vm instance to inspect the disk and isolate why the customization failed.