Enable workload and system log collection for VMs

Log collection for Linux and Windows virtual machines (VMs) in Google Distributed Cloud (GDC) air-gapped lets you centralize workload and system logs in Loki. After directing guest output to the console, you can query and visualize entries in Grafana to monitor and troubleshoot your instances.

This document is for developers in platform administrator or application operator groups that collect and view logs from VMs in a Distributed Cloud environment. For more information, see Audiences for GDC air-gapped documentation.

Before you begin

To get the permissions that you need to view logs in Grafana, ask your IAM Project Admin to grant you the Project Grafana Viewer (project-grafana-viewer) role.

Prerequisites for version 1.14 and earlier

Starting with version 1.15, GDC automatically ingests all messages written to the virtual machine (VM) console output into Loki and makes them available for viewing in Grafana by default. This streamlined process removes the need for manual resource configuration to capture basic console output. If you are using version 1.15 or later, you can skip the following steps and move on to the next section.

In contrast, for version 1.14 and all earlier releases, log collection isn't automatic. To ingest VM console logs into the centralized logging platform, create and configure a LoggingTarget resource within your project namespace. This resource defines the specific pod and container targets—such as the guest-console-log container within the virt-launcher pods—that the logging pipeline scrapes for data.

If you are using version 1.14 or earlier, complete these steps to set up the prerequisites:

  1. To get the permissions that you need to manage LoggingTarget resources and list the VMs in your project, ask your IAM Project Admin to grant you the following project-level IAM roles:

    • Logging Target Creator (loggingtarget-creator): to create LoggingTarget resources.
    • Logging Target Editor (loggingtarget-editor): to modify or delete LoggingTarget resources.
    • Virtual Machine Project Viewer (project-vm-viewer) or Virtual Machine Project Admin (project-vm-admin): to list the VMs in your project.

  2. Set some environment variables:

    # ID of the project that contains the targeted VMs
PROJECT_ID=PROJECT_ID

    Replace PROJECT_ID with the ID of the project that contains the targeted VMs.

  3. Create a LoggingTarget resource:

    kubectl --kubeconfig MANAGEMENT_API_SERVER apply -f - <<EOF
apiVersion: logging.gdc.goog/v1
kind: LoggingTarget
metadata:
  # Arbitrary name for your logging target
  name: my-vm-logger
  # Project namespace that contains the targeted VMs
  namespace: ${PROJECT_ID:?}
spec:
  selector:
    # The pod name prefixes to collect logs from.
    # The logging platform scrapes all pods with names
    # that start with the specified prefixes.
    # The values must contain only lowercase letters, numbers, and hyphens.
    # The relationship between different list elements
    # is an 'OR' relationship.
    matchPodNames:
    - virt-launcher
    # Match the container named "guest-console-log",
    # which handles console logs inside the kubevirt pod.
    matchContainerNames:
    - guest-console-log
  # Specify a service name to use as a label.
  # For user workloads consider this field as a workload name.
  # Required.
  serviceName: my-vm-logger-service
EOF

    Replace MANAGEMENT_API_SERVER with the path to the kubeconfig file of the management API server. If you don't have this file, sign in and generate it.

    This example uses virt-launcher as the pod name prefix selector, which targets every VM in your project. To target a specific VM, add the VM's name to the prefix, for example, virt-launcher-my-vm. To find the VM names, list the VMs in your project:

    kubectl --kubeconfig MANAGEMENT_API_SERVER -n ${PROJECT_ID:?} \
  get virtualmachines.virtualmachine.gdc.goog

See Collect logs from your Kubernetes workflows for more details on how to customize LoggingTarget resources.

Publish workload logs

Configure the guest operating system of each VM to send its logs to the VM console. Follow the steps for your operating system.

Publish workload logs from a Linux VM

This section outlines the process for configuring a Linux-based virtual machine (VM) to publish workload logs to the GDC logging infrastructure. These steps apply to Ubuntu 22.04 guest operating systems. For other distributions, the steps might differ.

Publish manual logs

To generate a log manually, write to /dev/console:

echo "This is a manual log from a Linux VM written on $(date)" | sudo tee /dev/console

Publish application logs

To publish logs from your VM-based application to Loki, configure the application to direct its output to the VM console (/dev/console).

Example with Nginx

This section demonstrates how to configure an application, using Nginx as an example, to direct its logs to the VM console (/dev/console) so the GDC logging pipeline can collect them.

  1. Ensure the Nginx user can write to the console:

    sudo usermod -a -G tty www-data

  2. Restart Nginx:

    sudo systemctl restart nginx

  3. Edit the main Nginx configuration file:

    sudo nano /etc/nginx/nginx.conf

  4. Inside the http { ... } block, find any existing access_log and error_log directives. Comment them out or replace them. Add the following lines to send logs to /dev/console:

    http {
    # ... other http settings ...

    # Send access logs to the VM's console
    # Comment out any existing access_log file paths
    # access_log /var/log/nginx/access.log;
    access_log /dev/console;

    # Send error logs to the VM's console
    # Comment out any existing error_log file paths
    # error_log /var/log/nginx/error.log;
    error_log /dev/console;

    # ... include server blocks, etc. ...
}

  5. Test the Nginx configuration:

    sudo nginx -t

    You will see output indicating the syntax is okay.

  6. Reload Nginx:

    sudo systemctl reload nginx

Forward system logs

The following steps configure the VM's internal system logging daemon (rsyslog) to forward all system logs to the VM console, where the logging agent can pick them up.

  1. Create a configuration file to instruct rsyslog to direct all system messages to the console:

    echo '*.*     action(type="omfile" file="/dev/console")' | sudo tee /etc/rsyslog.d/40-console.conf

  2. Allow the syslog user to write to the console:

    sudo usermod -a -G tty syslog

  3. Restart the rsyslog service:

    sudo systemctl restart rsyslog

  4. At this point, rsyslog forwards all system logs to Loki. You can also generate a log manually by using the logger command, for example:

    logger "This is a message generated with logger on $(date)"

Publish workload logs from a Windows VM

This section outlines the process for configuring a Windows-based VM to publish workload logs to the GDC logging infrastructure. These steps apply to Windows Server 2019 guest operating systems. For other distributions, the steps might differ.

Publish manual logs

To generate a log manually, write to the serial console (COM1) from a PowerShell terminal:

try {
    $port = New-Object System.IO.Ports.SerialPort COM1
    if (-not $port.IsOpen) {
        $port.Open()
    }
    $message = "MANUAL LOG TEST on $(Get-Date): Hello from PowerShell to the serial console!"
    $port.WriteLine($message)
    Write-Host "Message sent to COM1: $message"
    $port.Close()
} catch {
    Write-Error "Error writing to serial port COM1: $($_.Exception.Message)"
}

Forward system logs

Unlike Linux, Windows doesn't have a built-in configuration to redirect system events directly to a serial console file stream. To achieve this, you must create a background process (Scheduled Task) that monitors the Windows Event Log and forwards new entries to the COM1 serial port.

  1. Create a PowerShell script named C:\LogForwarder.ps1. This script opens the serial port and continuously checks for new entries in the "System" event log:

    $portName = "COM1"
try {
    $port = New-Object System.IO.Ports.SerialPort $portName
    # Configure port settings if necessary, defaults usually work for virt-launcher
    if (-not $port.IsOpen) { $port.Open() }

    # Get the current latest log index to start monitoring from now
    $lastIndex = (Get-EventLog -LogName System -Newest 1).Index

    Write-Host "Starting System Log Forwarder on $portName..."

    while ($true) {
        # Retrieve logs created after the last checked index
        $newLogs = Get-EventLog -LogName System -Newest 20 | Where-Object { $_.Index -gt $lastIndex } | Sort-Object Index

        foreach ($log in $newLogs) {
            # Format the log message for Loki/Grafana readability
            $message = "SYSTEM LOG: [$($log.EntryType)] Source: $($log.Source) - ID: $($log.InstanceId) - Message: $($log.Message)"

            # Write to Serial Port
            if ($port.IsOpen) {
                $port.WriteLine($message)
            }

            # Update the index tracker
            $lastIndex = $log.Index
        }
        # Wait before polling again to reduce CPU usage
        Start-Sleep -Seconds 5
    }
} catch {
    Write-Error "Log Forwarder Error: $($_.Exception.Message)"
}

  2. To ensure log forwarding starts automatically when the VM boots (and restarts if it fails), set up a Scheduled Task running as the SYSTEM account. Run the following command in an Administrator PowerShell terminal:

    $action = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-ExecutionPolicy Bypass -File C:\LogForwarder.ps1"

$trigger = New-ScheduledTaskTrigger -AtStartup

$principal = New-ScheduledTaskPrincipal -UserId "NT AUTHORITY\SYSTEM" -LogonType ServiceAccount

$settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -RestartCount 3 -RestartInterval (New-TimeSpan -Minutes 1)

Register-ScheduledTask -Action $action -Trigger $trigger -Principal $principal -Settings $settings -TaskName "SystemLogForwarder"

  3. You can either reboot the VM to test the trigger, or manually start the task immediately:

    Start-ScheduledTask -TaskName "SystemLogForwarder"

    At this point, the system writes Windows System events to COM1, and the logging pipeline ingests them.

Visualize logs

Follow these steps to visualize logs in Grafana:

  1. Navigate to the Explore tab in Grafana.
  2. Select Operational logs as your data source.

  3. In the query-builder interface, add a label filter using one of the following options:

    • Filter by pod name: set the label to pod, the operator to =~ (regular expression match), and the value to virt-launcher-VM_NAME-.*, where VM_NAME is the name of your VM. VM pod names consist of the virt-launcher- prefix, the VM name, and a random suffix, for example, virt-launcher-my-linux-vm-tfc6b. The regular expression matches any suffix, which you can't see in advance.

    • Filter by service name: set the label to service_name, the operator to =, and the value to one of the following:

      • In version 1.15 and later: vm-system-virt-launcher, the service name of the LoggingTarget resource that GDC creates automatically.
      • In version 1.14 and earlier: the serviceName value of your LoggingTarget resource, for example, my-vm-logger-service from the previous example.

    Alternatively, enter the equivalent query directly in the text field, for example, {pod=~"virt-launcher-my-linux-vm-.*"} or {service_name="vm-system-virt-launcher"}.

  4. Select a time range and click Run query. The logs appear at the bottom of the page.