Configure alert policy with a dashboard in GCP

In Google Cloud Monitoring, alert policies and dashboards are typically managed separately.

There isn’t a direct concept of “assigning” an alert policy to a dashboard. Instead, dashboards are used to visualize metrics, while alert policies are used to monitor these metrics and trigger alerts based on certain conditions.

However, you can use Terraform to manage both resources in a way that aligns your monitoring infrastructure. Here’s how you can handle alert policies and dashboards using Terraform for Google Cloud Platform (GCP):

Steps to Manage Dashboards and Alert Policies Using Terraform

1. Create a Dashboard

First, define a Google Cloud Monitoring dashboard using the google_monitoring_dashboard resource. Here’s an example:

resource "google_monitoring_dashboard" "example_dashboard" {
  dashboard_json = <<EOF
{
  "displayName": "Example Dashboard",
  "gridLayout": {
    "columns": 2,
    "widgets": [
      {
        "title": "Example Widget",
        "xyChart": {
          "dataSets": [
            {
              "timeSeriesQuery": {
                "timeSeriesFilter": {
                  "filter": "metric.type=\"compute.googleapis.com/instance/cpu/utilization\"",
                  "aggregation": {
                    "alignmentPeriod": "60s",
                    "perSeriesAligner": "ALIGN_MEAN"
                  }
                }
              }
            }
          ]
        }
      }
    ]
  }
}
EOF
}

This JSON configuration specifies a widget that displays CPU utilization for a Compute Engine instance.

2. Define an Alert Policy

Then, create an alert policy using the google_monitoring_alert_policy resource. Here’s an example:

resource "google_monitoring_alert_policy" "example_alert_policy" {
  display_name = "CPU Utilization Alert"
  conditions {
    display_name = "VM Instance - CPU Utilization"
    condition_threshold {
      filter          = "metric.type=\"compute.googleapis.com/instance/cpu/utilization\""
      comparison      = "COMPARISON_GT"
      threshold_value = 0.8
      duration        = "60s"
      aggregations {
        alignment_period    = "60s"
        per_series_aligner  = "ALIGN_MEAN"
      }
    }
  }

  notification_channels = [google_monitoring_notification_channel.example_channel.name]
}

This alert policy triggers when the CPU utilization exceeds a threshold value.

3. Optional: Create Notification Channels

You may also need to define notification channels if you want to receive alerts:

resource "google_monitoring_notification_channel" "example_channel" {
  display_name = "Example Channel"
  type         = "email"
  labels = {
    email_address = "your-email@example.com"
  }
}

Linkage

Logical Association: While there’s no direct “assignment” of an alert policy to a dashboard, metrics displayed on a dashboard can be the same ones monitored by an alert policy. Maintaining consistent metric types and filters across your alert policies and dashboards helps ensure that your monitoring visualizations and alerts are aligned.

Documentation: Consider documenting configurations where specific dashboards visualize the metrics monitored by certain alert policies. This can help teams understand the context of both the visual and alerting aspects.

Conclusion

In Terraform, dashboards are used for visualization, and alert policies are used for monitoring and notifying based on conditions. By configuring both properly, you create a cohesive monitoring ecosystem that visualizes and reacts to the same data.