Log collection in Google Distributed Cloud (GDC) air-gapped lets you aggregate operational
events from your Kubernetes workflows for monitoring and troubleshooting. Deploy
a `LoggingTarget` custom resource in the management API server to pull, index,
and label project logs from Kubernetes worker node virtual machines (VMs). This
pipeline doesn't support log collection from standalone user VMs.

After configuring log collection, you can
[query and view logs](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/query-and-view-logs.md.txt) in
Grafana or through the Log Query API. For instrumentation recommendations, see
the [Kubernetes logging
guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md).

## Before you begin

To get the permissions that you need to manage `LoggingTarget` custom
resources, ask your IAM Org Admin or IAM Project Admin to grant you one
of the associated `LoggingTarget` roles.

Depending on the level of access and permissions you need, you might obtain
creator, editor, or viewer roles for this resource in an organization or a
project. For more information, see [Prepare IAM permissions](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/obs-iam-permissions.md.txt).

## Configure the `LoggingTarget` custom resource

The `LoggingTarget` custom resource instructs the logging pipeline to collect
logs within your GDC project. You can specify the
targets for which you are collecting logs, a log parser, access levels, and any
additional settings.

> [!IMPORTANT]
> **Important:** You must deploy the `LoggingTarget` custom resource into the project namespace of the Management API server from where you want to collect logs.

This resource defines the following configurations:

- **Targets**: The criteria for selecting the pods and containers from which you want to collect logs. You can specify cluster names, pod name prefixes, and container name prefixes.
- **Log analyzer**: A predefined parser to interpret log entries, map log outputs to labels, and extract relevant fields.
- **Service identification**: The service name applied as a label to the log entries for easier identification and filtering.
- **Authorization**: The access level for log entries, controlling who can view
  the collected logs.

  > [!NOTE]
  > **Note:** By default, all operational log entries are saved for the tenant ID of the project namespace. To overwrite this behavior, provide a `logAccessLevel` value in the `LoggingTarget` custom resource.

By configuring these parameters within the `LoggingTarget` custom resource, you
can precisely control the collection and organization of logs.

Follow these steps to collect operational logs:

1. Determine the GDC project from which you want to collect logs.
2. In your `LoggingTarget` configuration, specify the pods for collecting your
   operational logs, the project namespace, and any additional settings.

   The following YAML file shows an example of a `LoggingTarget` configuration
   in `my-project-namespace`, where the pod name prefix to collect logs from
   is `my-pod-prefix`, the access level for log entries is granted to
   Application Operators (`ao`), the service name is `my-service-name`, and the
   log format is JSON:

   ```yaml
   # Configures a log scraping job
   apiVersion: logging.gdc.goog/v1
   kind: LoggingTarget
   metadata:
     # Choose a namespace that matches the namespace of the workload pods.
     namespace: my-project-namespace
     name: my-logging-target
   spec:
     selector:
       matchPodNames:
         - my-pod-prefix
     parser: json
     logAccessLevel: ao
     serviceName: my-service-name
   ```

   Refer to the
   [complete `LoggingTarget` specification](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/platform/pa-user/collect-logs.md.txt#loggingtarget-specification) and
   the [API reference documentation](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/apis/service/logging/v1/logging-v1.md.txt#loggingtarget)
   for additional fields and options.
3. Apply the `LoggingTarget` configuration to the Management API server within
   the same namespace as your target pods:

   ```bash
   kubectl --kubeconfig KUBECONFIG_PATH apply -f LOGGING_TARGET_NAME.yaml
   ```

   Replace the following:
   - `KUBECONFIG_PATH`: the path to the kubeconfig file for the Management API server.
   - `LOGGING_TARGET_NAME`: the name of the `LoggingTarget` definition file, such as `my-logging-target`.

   The logging pipeline begins collecting logs from the additional components
   of your project.

You can query your collected logs using the Grafana user interface or the Log
Query API. For more information, see
[Query and view logs](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/application/ao-user/query-and-view-logs).

If you use Grafana to query your logs, you can use the built-in color coding
feature for different log levels. For more information on setting log-level
values, see <https://grafana.com/docs/grafana/latest/explore/logs-integration/>.

## Complete `LoggingTarget` specification

The following YAML file shows an example for the complete specification of the
`LoggingTarget` custom resource. For more information and a complete description
of fields, see the
[API reference documentation](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/gdcag/apis/service/logging/v1/logging-v1.md.txt#loggingtarget).

```yaml
# Configures a log scraping job
apiVersion: logging.gdc.goog/v1
kind: LoggingTarget
metadata:
  # Choose a namespace that matches the namespace of the workload pods.
  namespace: PROJECT_NAMESPACE
  name: LOGGING_TARGET_NAME
spec:
  # Choose a matching pattern that identifies the pods for this job.
  # Optional.
  # Relationship between different selectors: 'AND'
  selector:
    # The clusters to collect logs from.
    # The default configuration is to collect logs from all clusters.
    # The relationship between different clusters is an 'OR' relationship.
    # Optional
    matchClusters:
    - my-cluster
    - another-cluster

    # The pod name prefixes to collect logs from.
    # The logging platform scrapes all pods with names
    # that start with the specified prefixes.
    # The values must contain '[a-z0-9-]' characters only.
    # The relationship between different list elements is an 'OR' relationship.
    # Optional.
    matchPodNames:
      - my-pod-prefix
      - another-pod-prefix

    # The container name prefixes to collect logs from.
    # The logging platform scrapes all containers with names
    # that start with the specified prefixes.
    # The values must contain '[a-z0-9-]' characters only.
    # The relationship between different list elements is an 'OR' relationship.
    # Optional.
    matchContainerNames:
      - my-container-prefix
      - another-container-prefix

  # Choose the predefined parser for log entries.
  # Use parsers to map the log output to labels and extract fields.
  # Specify the log format.
  # Optional.
  # Options: klog_text, klog_json, klogr, gdch_json, json
  parser: klog_text

  # Specify an access level for log entries.
  # The default value is 'ao'.
  # Optional.
  # Options: ao, pa, io
  logAccessLevel: ao

  # Specify a service name to be applied as a label.
  # For user workloads consider this field as a workload name.
  # Required.
  serviceName: my-service-name

  # The additional static fields to apply to log entries.
  # The field is a key-value pair, where the field name is the key and
  # the field value is the value.
  # Optional.
  additionalFields:
    app: workload2
    key: value
```

Replace the following:

- `PROJECT_NAMESPACE`: your project namespace.
- `LOGGING_TARGET_NAME`: the name of the `LoggingTarget` definition file.