Create metrics from logs

LoggingRule custom resources in Google Distributed Cloud (GDC) air-gapped appliance let you generate time series metrics from existing log streams. Recording rules defined within a LoggingRule custom resource precompute frequently used or computationally expensive LogQL expressions and save the results as new metrics.

Storing precomputed results eliminates redundant calculations each time an expression is evaluated. This approach reduces query latency for complex dashboards and alerts that repeatedly evaluate the same log data.

Before you begin

To get the permissions that you need to manage LoggingRule custom resources, ask your Organization IAM Admin or Project IAM Admin to grant you one of the associated LoggingRule roles.

Depending on the level of access and permissions you need, you might obtain creator, editor, or viewer roles for this resource in an organization or a project. For more information, see Prepare IAM permissions.

Create recording rules

To create a recording rule, define a name for the recorded log and a valid expression that produces a numeric value. This expression, when evaluated, generates the new metric. Deploy the LoggingRule custom resource in your project namespace on the Management API server to activate the recording rule.

For more information about recording rules, see https://grafana.com/docs/loki/latest/rules/.

Important: Before creating recording rules, ensure you have a deployed project with active workloads and configured logging targets. In this way, the rules have data to operate on. For more information, see Collect logs.

Follow these steps to create recording rules in your project namespace:

1. Determine the GDC project from which you want to create recording rules.

2. Create a YAML file defining the LoggingRule custom resource.

   The complete LoggingRule specification shows an example for log-based recording rules. For more information, see the API reference documentation.

3. Replace the following values in the YAML file according to your needs:

   Field name	Description
   namespace	The project namespace.
   name	The name for the rule configuration.
   source	The log source for the recording rule. Valid options are operational and audit.
   interval	The duration of the rule evaluation interval in seconds.
   limit	Optional. The maximum number of alerts. Set to 0 for unlimited alerts.
   recordRules	The definitions for calculating new metrics.
   recordRules.record	The record name for the new metric. The value must be a valid metric name that defines the time series where the results are stored.
   recordRules.expr	A LogQL expression for the log rule, which must evaluate to a numeric value.
   recordRules.labels	Optional. The key-value pairs of labels to add to or overwrite the new metric.

4. Save the YAML file.

5. Apply the LoggingRule configuration to the Management API server within the same namespace as your recording rules:

   kubectl --kubeconfig KUBECONFIG_PATH apply -f LOGGING_RULE_NAME.yaml
   

   Replace the following:

   • KUBECONFIG_PATH: the path to the kubeconfig file for the Management API server.
   • LOGGING_RULE_NAME: the name of the LoggingRule definition file.

Complete LoggingRule specification

A LoggingRule custom resource contains recording rules that describe the conditions to create new metrics based on existing logs for observability.

The following YAML file shows a template for the LoggingRule custom resource. For more information, see the API reference documentation.

# Configures either an alert or a target record for precomputation.
apiVersion: logging.gdc.goog/v1
kind: LoggingRule
metadata:
  # Choose a namespace that matches the project namespace.
  # The alert or record is produced in the same namespace.
  namespace: PROJECT_NAMESPACE
  name: LOGGING_RULE_NAME
spec:
  # Choose the log source to base alerts on (operational or audit logs).
  # Optional.
  # Valid options: 'operational' and 'audit'
  # Default value: 'operational'
  source: operational

  # Rule evaluation interval.
  interval: 60s

  # Configure the limit for the number of alerts.
  # A value of '0' means no limit.
  # Optional.
  # Default value: '0'
  limit: 0

  # Configure recording rules to generate new metrics based on existing logs.
  # Recording rules precompute expressions that are frequently needed or computationally expensive.
  # Results are saved as a new set of time series.
  recordRules:
    # Define the time series where you want to write the recording rule.
    # The value must be a valid metric name.
  - record: MyMetricsName

    # Define the LogQL expression to evaluate for this rule.
    expr: rate({service_name="bob-service"} [1m])

    # Define labels to add or overwrite.
    # Map of key-value pairs.
    # Optional.
    labels:
      verb: read

Replace the following:

• PROJECT_NAMESPACE: your project namespace.
• LOGGING_RULE_NAME: the name of the LoggingRule definition file.