Process-health filters

You can count the processes running on your virtual machines (VMs) that match specific criteria by using process-health filters in Cloud Monitoring. Use these filters in alerting policies and charts through the Cloud Monitoring API or direct filter mode in the Google Cloud console.

To monitor resource consumption such as CPU utilization for specific processes, see Monitor processes on VMs. Process-health filters share syntax with standard resource and metric filters; for general syntax rules, see Monitoring filters.

Before you begin

If you aren't familiar with metrics, time series, and monitored resources, see Metrics, time series, and resources.

Processes that are counted

Monitoring counts processes by applying a regular expression to the command line that invoked the process. If a process doesn't have a command-line field available, then that process isn't counted.

One way to determine whether a process can be counted is to view the output of the Linux ps command:

    ps aux | grep nfs
    USER      PID  %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root      1598  0.0  0.0      0     0 ?        S<   Oct25   0:00 [nfsd4]
    root      1639  0.0  0.0      0     0 ?        S    Oct25   2:33 [nfsd]
    root      1640  0.0  0.0      0     0 ?        S    Oct25   2:36 [nfsd]

When the entry in the COMMAND column is wrapped with square brackets—for example, [nfsd]—the command-line information for the process isn't available and therefore the process isn't counted.

Process-health filter structure

A process-health filter identifies which processes to count and one or more resources whose processes are to be counted. For example, the following JSON describes an alerting policy that sends a notification if the number of processes is less than 30 on any Compute Engine VM instance:

     {
        "displayName": "Count all processes",
        "conditionThreshold": {
          "aggregations": [],
          "comparison": "COMPARISON_LT",
          "duration": "0s",
          "filter": "select_process_count(\"*\") resource.type=\"gce_instance\"",
          "thresholdValue": 30,
          "trigger": {
            "count": 1
          }
        }
     }

In this example, the value of the filter statement is a string with two clauses. The first clause, select_process_count(\"*\"), specifies that all processes are counted. The second clause, resource.type=\"gce_instance\", identifies that Compute Engine VMs are to be monitored.

If you use the Google Cloud console, then use direct filter mode to enter the value of a Monitoring filter. However, be sure to remove any escapes that protect a substring. For example, to count all processes for Compute Engine VMs, enter the following:

    select_process_count("*") resource.type="gce_instance"

For information about how to access direct filter mode when using Metrics Explorer or when creating alerting policies or charts on dashboards, see the following documents:

Resource identifier

A process-health filter must set the resource.type field to specify the VMs whose processes are counted. The value of this filter must be one of the following:

  • gce_instance
  • aws_ec2_instance

If you only specify the resource.type field, then processes on all VMs are counted:

  • To select a single VM instance, add a metric.labels.instance_name filter object.
  • To select a group of VMs, add a group.id filter object.

For more information on the resource.type field, see Monitoring filters.

Process identifier

A process-health filter must call the function select_process_count. The arguments of this function identify the processes to be counted.

There are three filter objects that you can specify in a call to select_process_count:

  • command_line (or metric.labels.command_line): This filter applies to the command line used to start the process. Command lines are truncated after 1024 characters, so text in a command line beyond that limit can't be matched against.

  • command (or metric.labels.command): This filter applies to the command line used to start the process. Commands are truncated after 1024 characters, so text in a command beyond that limit can't be matched against.

  • user (or metric.labels.user): This filter applies to the user that started the process.

You can either use positional arguments or named arguments in the call to select_process_count. If you use named arguments, then you must specify the filter object, an equals statement, =, and a value. If you use positional arguments, then you only specify the value. A case-sensitive string test determines whether a process is a match to the filter.

The value of a filter object can be any of the following:

  • string (exact match)
  • * (wildcard)
  • has_substring(string)
  • starts_with(string)
  • ends_with(string)
  • monitoring.regex.full_match(string)

If you specify multiple filters, then the following rules apply:

  • command_line is joined to command by a logical OR. A process is counted when it matches either filter.
  • user is joined to command_line (command) by a logical AND. A process is a match only when it matches the user filter and the command_line (command) filter.
  • If you apply all filters, then a process is counted when it matches the user filter and when it matches the command_line or command filter.

Named arguments

To use named arguments, specify the filter name, an equals statement, =, and then the filter value. You can specify named arguments in any order.

For example, the following matches all processes started by root when the command line includes the string nginx:

     select_process_count("command_line=has_substring(\"nginx\")","user=root")

This example uses a regular expression match on the command line:

     select_process_count("command_line=monitoring.regex.full_match(\".*nginx.*\")","user=starts_with(\"root\")")

This example counts all processes whose command line is /bin/bash:

     select_process_count("command=/bin/bash")

This example counts all processes started by the user www whose command line starts with /bin/bash:

     select_process_count("user=www", "command_line=starts_with(\"/bin/bash \")")

Positional arguments

To use positional arguments, you supply only the filter value. The following rules apply to positional arguments:

  • If a single argument is provided, then that argument is interpreted as a command-line filter object:
        select_process_count("*")
        select_process_count("/sbin/init")
        select_process_count("starts_with(\"/bin/bash -c\")")
        select_process_count("ends_with(\"--alsologtostderr\")")
        select_process_count("monitoring.regex.full_match(\".*nginx.*\")")
  • If two arguments are provided, then the first argument is interpreted as a command-line filter and the second is a user filter. A process is counted when it matches both filter objects:
        select_process_count("/sbin/init", "root")