Log collection in Google Distributed Cloud (GDC) air-gapped appliance lets you aggregate operational
events from your workflows for monitoring and troubleshooting. Deploy a
`LoggingTarget` custom resource to your project namespace in the management API
server to pull, index, and label logs from your service components.

After configuring log collection, you can
[query and view logs](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/platform/pa-user/query-and-view-logs) in the
Grafana user interface. For instrumentation recommendations, see the [Kubernetes
logging
guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md).

## Before you begin

To get the permissions that you need to manage `LoggingTarget` custom resources, ask your Organization IAM Admin or Project IAM Admin to grant you one
of the associated `LoggingTarget` roles.

Depending on the level of access and permissions you need, you might obtain creator, editor, or viewer roles for this resource in an organization or a
project. For more information, see [Prepare IAM permissions](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/platform/pa-user/obs-iam-permissions).

- **Application Operators** : To get the permissions you need to manage `LoggingTarget` custom resources in a project in the management API server, ask your Project IAM Admin to grant you one of the following roles in your project namespace:

  - **LoggingTarget Creator** : creates `LoggingTarget` custom resources. Request the LoggingTarget Creator (`loggingtarget-creator`) role.
  - **LoggingTarget Editor** : edits or modifies `LoggingTarget` custom resources. Request the LoggingTarget Editor (`loggingtarget-editor`) role.
  - **LoggingTarget Viewer** : views `LoggingTarget` custom resources. Request the LoggingTarget Viewer (`loggingtarget-viewer`) role.
- **Platform Administrators** : To get the permissions you need to manage `LoggingTarget` custom resources in the `platform-obs` namespace in the management API server, ask your Organization IAM Admin to grant you one of the following cluster roles in the `platform-obs` namespace:

  - **LoggingTarget PA Creator** : creates `LoggingTarget` custom resources. Request the LoggingTarget PA Creator (`loggingtarget-pa-creator`) cluster role.
  - **LoggingTarget PA Editor** : edits or modifies `LoggingTarget` custom resources. Request the LoggingTarget PA Editor (`loggingtarget-pa-editor`) cluster role.
  - **LoggingTarget PA Viewer** : views `LoggingTarget` custom resources. Request the LoggingTarget PA Viewer (`loggingtarget-pa-viewer`) cluster role.

## Collect operational logs

Operational logs record conditions, changes, and actions as you manage ongoing operations in applications and services on GDC. Deploy the `LoggingTarget` custom resource to the Management API server to configure the system logging pipeline for collecting operational logs from specific services at the project level.

Complete the following steps to collect operational logs from a service:

1. Configure the `LoggingTarget` custom resource in the Management API server, specifying the selected pods for collecting your operational logs, the project namespace, and any additional settings. To learn what the `LoggingTarget` must look like, see [Configure the `LoggingTarget` custom resource](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/platform/pa-user/collect-logs#loggingtarget).
2. Deploy the `LoggingTarget` custom resource to your project namespace in the Management API server:

       kubectl --kubeconfig MANAGEMENT_API_SERVER apply -f LOGGING_TARGET_NAME

   Replace the following:
   - `MANAGEMENT_API_SERVER`: the zonal Management API server's kubeconfig path.
   - `LOGGING_TARGET_NAME`: the name of the `LoggingTarget` custom resource, such as `my-service-logging-target.yaml`.

   The pipeline starts collecting logs from the additional components of your project.
3. Query your logs from the Grafana instance of your project.
   For more information, see [Query and view logs](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/admin/query-and-view-logs).

Use the built-in color coding feature of Grafana for different log levels of the service. For more information on setting log-level values, see <https://grafana.com/docs/grafana/latest/explore/logs-integration/>.

## Configure the `LoggingTarget` custom resource

The `LoggingTarget` custom resource instructs the logging pipeline to collect
logs from specific services within your GDC project. You
can specify the targets for which you are collecting logs, a log parser, access
levels, and any additional settings.

> [!IMPORTANT]
> **Important:** You must deploy the `LoggingTarget` custom resource into the project namespace of the Management API server from where you want to collect logs.

This resource defines the following configurations:

- **Targets**: The criteria for selecting the pods and containers from which you want to collect logs. You can specify cluster names, pod name prefixes, and container name prefixes.
- **Log analyzer**: A predefined parser to interpret log entries, map log outputs to labels, and extract relevant fields.
- **Service identification**: The service name applied as a label to the log entries for easier identification and filtering.
- **Authorization**: The access level for log entries, controlling who can view
  the collected logs.

  > [!NOTE]
  > **Note:** By default, all operational log entries are saved for the tenant ID of the project namespace. To overwrite this behavior, provide a `logAccessLevel` value in the `LoggingTarget` custom resource.

By configuring these parameters within the `LoggingTarget` custom resource, you
can precisely control the collection and organization of logs from your
services.

Follow these steps to collect operational logs from a service:

1. Determine the GDC project from which you want to collect logs.
2. In your `LoggingTarget` configuration, specify the pods for collecting your
   operational logs, the project namespace, and any additional settings.

   The following YAML file shows an example of a `LoggingTarget` configuration
   in `my-project-namespace`, where the pod name prefix to collect logs from
   is `my-pod-prefix`, the access level for log entries is granted to
   Application Operators (`ao`), the service name is `my-service-name`, and the
   log format is JSON:

       # Configures a log scraping job
       apiVersion: logging.gdc.goog/v1
       kind: LoggingTarget
       metadata:
         # Choose a namespace that matches the namespace of the workload pods.
         namespace: my-project-namespace
         name: my-service-logging-target
       spec:
         selector:
           matchPodNames:
             - my-pod-prefix
         parser: json
         logAccessLevel: ao
         serviceName: my-service-name

   Refer to the
   [complete `LoggingTarget` specification](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/platform/pa-user/collect-logs#loggingtarget-specification) and
   the [API reference documentation](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/apis/service/logging/v1/logging-v1#loggingtarget)
   for additional fields and options.
3. Apply the `LoggingTarget` configuration to the Management API server within
   the same namespace as your target pods:

       kubectl --kubeconfig KUBECONFIG_PATH apply -f LOGGING_TARGET_NAME.yaml

   Replace the following:
   - `KUBECONFIG_PATH`: the path to the kubeconfig file for the Management API server.
   - `LOGGING_TARGET_NAME`: the name of the `LoggingTarget` definition file, such as `my-service-logging-target`.

   The logging pipeline begins collecting logs from the additional components
   of your project.

You can query your collected logs using the Grafana user interface or the Log
Query API. For more information, see
[Query and view logs](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/application/ao-user/query-and-view-logs).

If you use Grafana to query your logs, you can use the built-in color coding
feature for different log levels of the service. For more information on setting
log-level values, see
<https://grafana.com/docs/grafana/latest/explore/logs-integration/>.

## Complete `LoggingTarget` specification

The following YAML file shows an example for the complete specification of the
`LoggingTarget` custom resource. For more information and a complete description
of fields, see the
[API reference documentation](https://docs.cloud.google.com/distributed-cloud/hosted/docs/latest/appliance/apis/service/logging/v1/logging-v1#loggingtarget).

    # Configures a log scraping job
    apiVersion: logging.gdc.goog/v1
    kind: LoggingTarget
    metadata:
      # Choose a namespace that matches the namespace of the workload pods.
      namespace: PROJECT_NAMESPACE
      name: LOGGING_TARGET_NAME
    spec:
      # Choose a matching pattern that identifies the pods for this job.
      # Optional.
      # Relationship between different selectors: 'AND'
      selector:
        # The clusters to collect logs from.
        # The default configuration is to collect logs from all clusters.
        # The relationship between different clusters is an 'OR' relationship.
        # Optional
        matchClusters:
        - my-cluster
        - another-cluster

        # The pod name prefixes to collect logs from.
        # The logging platform scrapes all pods with names
        # that start with the specified prefixes.
        # The values must contain '[a-z0-9-]' characters only.
        # The relationship between different list elements is an 'OR' relationship.
        # Optional.
        matchPodNames:
          - my-pod-prefix
          - another-pod-prefix

        # The container name prefixes to collect logs from.
        # The logging platform scrapes all containers with names
        # that start with the specified prefixes.
        # The values must contain '[a-z0-9-]' characters only.
        # The relationship between different list elements is an 'OR' relationship.
        # Optional.
        matchContainerNames:
          - my-container-prefix
          - another-container-prefix

      # Choose the predefined parser for log entries.
      # Use parsers to map the log output to labels and extract fields.
      # Specify the log format.
      # Optional.
      # Options: klog_text, klog_json, klogr, gdch_json, json
      parser: klog_text

      # Specify an access level for log entries.
      # The default value is 'ao'.
      # Optional.
      # Options: ao, pa, io
      logAccessLevel: ao

      # Specify a service name to be applied as a label.
      # For user workloads consider this field as a workload name.
      # Required.
      serviceName: my-service-name

      # The additional static fields to apply to log entries.
      # The field is a key-value pair, where the field name is the key and
      # the field value is the value.
      # Optional.
      additionalFields:
        app: workload2
        key: value

Replace the following:

- `PROJECT_NAMESPACE`: your project namespace.
- `LOGGING_TARGET_NAME`: the name of the `LoggingTarget` definition file.