Learn how to monitor the health, uptime, and upgrades of your Google Distributed Cloud connected workloads. For more information about configuring monitoring and about the available metrics, see Logs and metrics.
Monitor workload health and uptime
You are responsible for monitoring the health and uptime of your workloads (containers and virtual machines) running on Distributed Cloud connected.
Container workloads
To monitor the health and uptime of the containerized workloads that you deploy, we recommend using the Prometheus metrics solution. You can configure Prometheus to scrape metrics from your Pods and Services. Google-managed system services are monitored and managed automatically by Google. For more information about configuring Prometheus, see Collect metrics with Prometheus.
Use the following key metrics to monitor for container workloads. If you use Prometheus, you're responsible for deploying and managing Kube State Metrics to expose these specific metrics for your workloads:
kube_pod_status_phase: Monitor for Pods inFailedorUnknownphases.kube_pod_container_status_waiting_reason: Identify Pods stuck inCrashLoopBackOfforImagePullBackOff.container_cpu_usage_seconds_totalandcontainer_memory_working_set_bytes(from cAdvisor): Monitor resource consumption to prevent out-of-memory (OOM) issues.
Virtual machine workloads
In Distributed Cloud connected, virtual machines (VMs) run inside
standard Kubernetes Pods, typically prefixed with virt-launcher-. Monitor VM
health and status primarily by checking the VirtualMachine custom resource
status or by monitoring application metrics from within the VM.
Use the underlying virt-launcher Pod metrics as secondary indicators for the
following details:
- Resource usage: Monitor CPU and memory consumption of launcher Pods to ensure that VMs have sufficient resources.
- Pod phase: Track launcher Pod phases to identify VMs that fail to start or crash.
To adjust the resources assigned to a VM, modify the VM custom resource
specification. Don't modify the underlying virt-launcher Pod YAMLs directly,
because these are managed by the platform. Any direct changes will be
overwritten or lead to reconciliation errors.
To troubleshoot VMs stuck in a pending state, see Troubleshoot virtual machines stuck in Pending state.
Monitor Distributed Cloud connected upgrades
Monitor the progress and status of Distributed Cloud connected
software upgrades using Cloud Monitoring metrics and gcloud commands.
Upgrades are scheduled and run by Google. Monitoring is strictly for visibility
and planning workload migration.
For detailed instructions about how to check if an upgrade is in progress, monitor its status, and troubleshoot failed upgrades, see Troubleshoot software upgrades.
Multi-cluster monitoring
If you manage many Distributed Cloud connected clusters, we recommend using Cloud Monitoring to aggregate and filter metrics.
Use resource labels: Metrics exported to Cloud Monitoring include labels that identify the source. The labels available depend on the resource type:
For cluster and container metrics, such as
k8s_container, key labels include the following values:project_id: The Google Cloud project hosting the cluster.location: The Google Cloud region where the cluster is registered.cluster_name: The name of the cluster.
For hardware metrics, such as
edgecontainer.googleapis.com/machine, key labels include the following values:resource_container: The Google Cloud project associated with the hardware.location: The Google Cloud region where the Distributed Cloud connected zone is registered.machine_id: The identifier of the machine.
Hardware metrics don't include a
cluster_namelabel directly.
Create custom dashboards: Build dashboards that display key health indicators across your entire fleet. Key health indicators include connectivity, VM status, and resource usage. To show data from multiple clusters in a single chart, use wildcards or group-by operations.
Set up multi-cluster alerts: Configure alerting policies that apply to multiple clusters. For example, you can create an alert that triggers if any machine in any cluster loses connectivity.
Alerting strategy
Monitoring and maintaining Distributed Cloud connected is a shared responsibility. This section explains what Google alerts on and how you can configure your own alerting policies.
What Google alerts on
Google continuously monitors the underlying infrastructure. It takes action and notifies you if necessary for the following situations:
- Hardware failures: Power supply failures, fan failures, overheating, or disk failures, such as a disk reaching its spare threshold or end of viable life.
- Connectivity issues: Complete loss of connection between the Distributed Cloud connected zone and Google Cloud.
- Control plane health: Failures in the Kubernetes control plane or Google-managed system services.
- Upgrade failures: Automated upgrade processes that get stuck or fail.
Configure alerts
Configure your own alerting policies in Cloud Monitoring or in Prometheus for issues affecting your workloads or local environment.
To configure alerts in Cloud Monitoring, use the Google Cloud console or the Cloud Monitoring API. For detailed instructions, see Create alerting policies. You can create alerting policies based on the metrics documented in Logs and metrics.
If you use Prometheus to collect metrics, define standard Prometheus alerting rules in your Prometheus configuration.
You can set up alerts for the following issues:
| Issue | Description | Monitoring system | Details |
|---|---|---|---|
| Workload downtime | Pods or VMs failing to start or crash-looping | Prometheus | For container workloads, create alerts on kube_pod_status_phase to detect Pods in Failed or Unknown phases. You can also alert on kube_pod_container_status_waiting_reason when it equals CrashLoopBackOff or ImagePullBackOff. |
| Cloud Monitoring | For VM workloads running in Pods, set up alerts using standard Kubernetes container metrics in Cloud Monitoring to track the status of virt-launcher Pods. |
||
| Workload resource exhaustion | Disk usage approaching capacity, or high memory or CPU usage on workloads | Prometheus | For container workloads, set threshold alerts on container_cpu_usage_seconds_total and container_memory_working_set_bytes (from cAdvisor) to identify Pods approaching their resource limits. |
| Cloud Monitoring | For machine storage, monitor the machine disk utilization metric edgecontainer.googleapis.com/machine/disk/utilization to detect when node storage is approaching capacity. |
||
| Local network issues | Network interface drops on machines | Cloud Monitoring | Monitor whether the machine network up metric edgecontainer.googleapis.com/machine/network/up is false.
|
| Loss of internet connectivity | Cloud Monitoring | Monitor whether the network connectivity metric edgecontainer.googleapis.com/machine/network/connectivity is false.
|
|
| Machine restarts | Unexpected machine reboots or shutdowns | Cloud Monitoring | Configure alerts with edgecontainer.googleapis.com/machine/uptime and edgecontainer.googleapis.com/machine/restart_count metrics. For more information, see Troubleshoot machine restarts. |
Distributed Cloud connected hardware is managed by Google. You don't have SSH access to the machines or control over the host operating system. If alerts based on machine-level metrics fire, such as network connectivity or restarts, your action items are limited to checking physical power, cabling, and local network and firewall configurations, or escalating the issue to Google Support.