Log collection for Linux and Windows virtual machines (VMs) in
Google Distributed Cloud (GDC) air-gapped lets you centralize workload and system logs in
Loki. After directing guest output to the console, you can query and visualize
entries in Grafana to monitor and troubleshoot your instances.

This document is for developers in platform administrator or application
operator groups that collect and view logs from VMs in a Distributed Cloud
environment. For more information, see
[Audiences for GDC air-gapped documentation](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/resources/audiences.md.txt).

## Before you begin

To get the permissions that you need to view logs in Grafana, ask your IAM
Project Admin to grant you the Project Grafana Viewer
(`project-grafana-viewer`) role.

### Prerequisites for version 1.14 and earlier

Starting with version 1.15, GDC automatically ingests
all messages written to the virtual machine (VM) console output into Loki and
makes them available for viewing in Grafana by default. This streamlined
process removes the need for manual resource configuration to capture basic
console output. **If you are using version 1.15 or later, you can skip the
following steps and move on to the next section.**

In contrast, for version 1.14 and all earlier releases, log collection isn't
automatic. To ingest VM console logs into the centralized logging platform,
create and configure a `LoggingTarget` resource within your project namespace.
This resource defines the specific pod and container targets---such as the
`guest-console-log` container within the `virt-launcher` pods---that the logging
pipeline scrapes for data.

If you are using version 1.14 or earlier, complete these steps to set up the
prerequisites:

1. To get the permissions that you need to manage `LoggingTarget` resources
   and list the VMs in your project, ask your IAM Project Admin to grant you
   the following project-level IAM roles:

   - Logging Target Creator (`loggingtarget-creator`): to create `LoggingTarget` resources.
   - Logging Target Editor (`loggingtarget-editor`): to modify or delete `LoggingTarget` resources.
   - Virtual Machine Project Viewer (`project-vm-viewer`) or Virtual Machine Project Admin (`project-vm-admin`): to list the VMs in your project.
2. Set some environment variables:

   ```bash
   # ID of the project that contains the targeted VMs
   PROJECT_ID=PROJECT_ID
   ```

   Replace `PROJECT_ID` with the ID of the project
   that contains the targeted VMs.
3. Create a `LoggingTarget` resource:

   ```bash
   kubectl --kubeconfig MANAGEMENT_API_SERVER apply -f - <<EOF
   apiVersion: logging.gdc.goog/v1
   kind: LoggingTarget
   metadata:
     # Arbitrary name for your logging target
     name: my-vm-logger
     # Project namespace that contains the targeted VMs
     namespace: ${PROJECT_ID:?}
   spec:
     selector:
       # The pod name prefixes to collect logs from.
       # The logging platform scrapes all pods with names
       # that start with the specified prefixes.
       # The values must contain only lowercase letters, numbers, and hyphens.
       # The relationship between different list elements
       # is an 'OR' relationship.
       matchPodNames:
       - virt-launcher
       # Match the container named "guest-console-log",
       # which handles console logs inside the kubevirt pod.
       matchContainerNames:
       - guest-console-log
     # Specify a service name to use as a label.
     # For user workloads consider this field as a workload name.
     # Required.
     serviceName: my-vm-logger-service
   EOF
   ```

   Replace `MANAGEMENT_API_SERVER` with the path to the
   kubeconfig file of the management API server. If you don't have this file,
   [sign in and generate it](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/application/ao-user/iam/sign-in.md.txt#zonal-resources-kubeconfig).

   This example uses `virt-launcher` as the pod name prefix selector, which
   targets every VM in your project. To target a specific VM, add the VM's
   name to the prefix, for example, `virt-launcher-my-vm`. To find the VM
   names, list the VMs in your project:

   ```bash
   kubectl --kubeconfig MANAGEMENT_API_SERVER -n ${PROJECT_ID:?} \
     get virtualmachines.virtualmachine.gdc.goog
   ```

See
[Collect logs from your Kubernetes workflows](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/collect-logs.md.txt)
for more details on how to customize `LoggingTarget` resources.

## Publish workload logs

Configure the guest operating system of each VM to send its logs to the VM
console. Follow the steps for your operating system.

### Publish workload logs from a Linux VM

This section outlines the process for configuring a Linux-based virtual machine
(VM) to publish workload logs to the GDC logging
infrastructure. These steps apply to Ubuntu 22.04 guest operating systems. For
other distributions, the steps might differ.

#### Publish manual logs

To generate a log manually, write to `/dev/console`:

```bash
echo "This is a manual log from a Linux VM written on $(date)" | sudo tee /dev/console
```

#### Publish application logs

To publish logs from your VM-based application to Loki, configure the
application to direct its output to the VM console (`/dev/console`).

##### Example with Nginx

This section demonstrates how to configure an application, using Nginx as an
example, to direct its logs to the VM console (`/dev/console`) so the
GDC logging pipeline can collect them.

1. Ensure the Nginx user can write to the console:

   ```bash
   sudo usermod -a -G tty www-data
   ```
2. Restart Nginx:

   ```bash
   sudo systemctl restart nginx
   ```
3. Edit the main Nginx configuration file:

   ```bash
   sudo nano /etc/nginx/nginx.conf
   ```
4. Inside the `http { ... }` block, find any existing `access_log` and
   `error_log` directives. Comment them out or replace them. Add the following
   lines to send logs to `/dev/console`:

   ```nginx
   http {
       # ... other http settings ...

       # Send access logs to the VM's console
       # Comment out any existing access_log file paths
       # access_log /var/log/nginx/access.log;
       access_log /dev/console;

       # Send error logs to the VM's console
       # Comment out any existing error_log file paths
       # error_log /var/log/nginx/error.log;
       error_log /dev/console;

       # ... include server blocks, etc. ...
   }
   ```
5. Test the Nginx configuration:

   ```bash
   sudo nginx -t
   ```

   You will see output indicating the syntax is okay.
6. Reload Nginx:

   ```bash
   sudo systemctl reload nginx
   ```

#### Forward system logs

The following steps configure the VM's internal system logging daemon
(`rsyslog`) to forward all system logs to the VM console, where the logging
agent can pick them up.

1. Create a configuration file to instruct `rsyslog` to direct all system
   messages to the console:

   ```bash
   echo '*.*     action(type="omfile" file="/dev/console")' | sudo tee /etc/rsyslog.d/40-console.conf
   ```
2. Allow the `syslog` user to write to the console:

   ```bash
   sudo usermod -a -G tty syslog
   ```
3. Restart the `rsyslog` service:

   ```bash
   sudo systemctl restart rsyslog
   ```
4. At this point, `rsyslog` forwards all system logs to Loki. You can also
   generate a log manually by using the `logger` command, for example:

   ```bash
   logger "This is a message generated with logger on $(date)"
   ```

### Publish workload logs from a Windows VM

This section outlines the process for configuring a Windows-based VM to publish
workload logs to the GDC logging infrastructure. These
steps apply to Windows Server 2019 guest operating systems. For other
distributions, the steps might differ.

#### Publish manual logs

To generate a log manually, write to the serial console (`COM1`) from a
PowerShell terminal:

```powershell
try {
    $port = New-Object System.IO.Ports.SerialPort COM1
    if (-not $port.IsOpen) {
        $port.Open()
    }
    $message = "MANUAL LOG TEST on $(Get-Date): Hello from PowerShell to the serial console!"
    $port.WriteLine($message)
    Write-Host "Message sent to COM1: $message"
    $port.Close()
} catch {
    Write-Error "Error writing to serial port COM1: $($_.Exception.Message)"
}
```

#### Forward system logs

Unlike Linux, Windows doesn't have a built-in configuration to redirect system
events directly to a serial console file stream. To achieve this, you must
create a background process (Scheduled Task) that monitors the Windows Event
Log and forwards new entries to the `COM1` serial port.

1. Create a PowerShell script named `C:\LogForwarder.ps1`. This script opens
   the serial port and continuously checks for new entries in the "System"
   event log:

   ```powershell
   $portName = "COM1"
   try {
       $port = New-Object System.IO.Ports.SerialPort $portName
       # Configure port settings if necessary, defaults usually work for virt-launcher
       if (-not $port.IsOpen) { $port.Open() }

       # Get the current latest log index to start monitoring from now
       $lastIndex = (Get-EventLog -LogName System -Newest 1).Index

       Write-Host "Starting System Log Forwarder on $portName..."

       while ($true) {
           # Retrieve logs created after the last checked index
           $newLogs = Get-EventLog -LogName System -Newest 20 | Where-Object { $_.Index -gt $lastIndex } | Sort-Object Index

           foreach ($log in $newLogs) {
               # Format the log message for Loki/Grafana readability
               $message = "SYSTEM LOG: [$($log.EntryType)] Source: $($log.Source) - ID: $($log.InstanceId) - Message: $($log.Message)"

               # Write to Serial Port
               if ($port.IsOpen) {
                   $port.WriteLine($message)
               }

               # Update the index tracker
               $lastIndex = $log.Index
           }
           # Wait before polling again to reduce CPU usage
           Start-Sleep -Seconds 5
       }
   } catch {
       Write-Error "Log Forwarder Error: $($_.Exception.Message)"
   }
   ```
2. To ensure log forwarding starts automatically when the VM boots (and
   restarts if it fails), set up a Scheduled Task running as the `SYSTEM`
   account. Run the following command in an Administrator PowerShell terminal:

   ```powershell
   $action = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-ExecutionPolicy Bypass -File C:\LogForwarder.ps1"

   $trigger = New-ScheduledTaskTrigger -AtStartup

   $principal = New-ScheduledTaskPrincipal -UserId "NT AUTHORITY\SYSTEM" -LogonType ServiceAccount

   $settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -RestartCount 3 -RestartInterval (New-TimeSpan -Minutes 1)

   Register-ScheduledTask -Action $action -Trigger $trigger -Principal $principal -Settings $settings -TaskName "SystemLogForwarder"
   ```
3. You can either reboot the VM to test the trigger, or manually start the task
   immediately:

   ```powershell
   Start-ScheduledTask -TaskName "SystemLogForwarder"
   ```

   At this point, the system writes Windows System events to `COM1`, and the
   logging pipeline ingests them.

## Visualize logs

Follow these steps to visualize logs in Grafana:

1. Navigate to the **Explore** tab in Grafana.
2. Select **Operational logs** as your data source.
3. In the query-builder interface, add a label filter using one of the
   following options:

   - Filter by pod name: set the label to `pod`, the operator to `=~` (regular expression match), and the value to `virt-launcher-VM_NAME-.*`, where `VM_NAME` is the name of your VM. VM pod names consist of the `virt-launcher-` prefix, the VM name, and a random suffix, for example, `virt-launcher-my-linux-vm-tfc6b`. The regular expression matches any suffix, which you can't see in advance.
   - Filter by service name: set the label to `service_name`, the operator to
     `=`, and the value to one of the following:

     - In version 1.15 and later: `vm-system-virt-launcher`, the service name of the `LoggingTarget` resource that GDC creates automatically.
     - In version 1.14 and earlier: the `serviceName` value of your `LoggingTarget` resource, for example, `my-vm-logger-service` from the previous example.

   Alternatively, enter the equivalent query directly in the text field, for
   example, `{pod=~"virt-launcher-my-linux-vm-.*"}` or
   `{service_name="vm-system-virt-launcher"}`.
4. Select a time range and click **Run query**. The logs appear at the bottom
   of the page.