Set Up Monitoring Notifications

There are several predefined alerts in the monitoring dashboards. When an alert changes state, it sends out notifications. In order to receive alerts from Grafana, you need to configure a notification channel.

Grafana supports various notification types (email, Slack, Webhooks, etc.). Please check out Grafana's documentation for more notification type supports.

This document covers setting up common notification channels such as: email, Slack, PagerDuty, and Webhooks.

Before setting up a notification, make sure your monitoring containers are up and running.

Email

1. Configure SMTP Settings in the monitoring-compose.yml file.

   services:
     ...
     grafana:
     ...
       environment:
         GF_SECURITY_ADMIN_PASSWORD: ${GF_SECURITY_ADMIN_PASSWORD:?Missing env GF_SECURITY_ADMIN_PASSWORD for Grafana admin password}
         WA_PROMETHEUS_ENDPOINT: ${WA_PROMETHEUS_ENDPOINT:-http://prometheus:9090}
         GF_SERVER_ROOT_URL: your-grafana-public-endpoint #(ex. https://foo.bar.com:3000)
         GF_SMTP_ENABLED: 1
         GF_SMTP_HOST: your-smtp-host #(ex. smtp.gmail.com:465)
         GF_SMTP_USER: your-smtp-auth-user #(ex. you@email.com)
         GF_SMTP_PASSWORD: your-smtp-auth-password
         GF_SMTP_SKIP_VERIFY: 1
       depends_on:
         - 'prometheus'
   

2. Restart Grafana.

   docker-compose -f monitoring-compose.yml up -d

3. Access Notification channels in the Alerting panel.
   

   Notification channels
4. Enter your email address and test your setup by clicking the Send Test button.
   

   Send Test
   If it's configured correctly, you should receive an email like this:
   

   Grafana email
   If not, check your Grafana container logs to debug.
5. Click Save to save your setup. After that, you will receive alerts from Grafana. See Grafana's email notification documentation for more details.

Slack

1. Create a Slack App by entering a name (e.g., alert) and choosing a workspace to install your app.
2. Click the Incoming Webhooks button to add this feature to your app.
   

   Incoming Webhooks
3. Click the Activate Incoming Webhooks toggle to switch it on, the click Add New Webhook to Workspace.
   

   Add New Webhook to Workspace
4. Pick a channel that the app will post to, and click the Authorize button to authorize your app.
5. Copy the Webhook URL (e.g., https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX) to set up the Slack notification in Grafana. Test your setup by clicking the Send Test button. If it's configure correctly, you should receive a message on configured channel like this:
   

   Slack notification
   If not, check your Grafana container logs to debug.
6. Click Save to save your setup. After that, you will receive alerts from Grafana. See Grafana's Slack notification documentation for more details.

PagerDuty

1. Add a new service (e.g., https://XXXX.pagerduty.com/services/new), and select Use our API directly as the Integration Type.
   

   Add a Service
2. Copy the Integration Key (e.g., de7439f367354513b9e1ba4042f0e3b8) to set up a PagerDuty notification in Grafana. Test your set up by clicking the Send Test button. If it's configured correctly, you should receive an alert on the PagerDuty Alerts Page like this:
   

   PagerDuty notification
   If not, please check your Grafana container logs to debug.
3. Click Save to save your setup. After that, you will receive alerts from Grafana. See Grafana's PagerDuty notification documentation for more details.

Webhooks

You could also set up a Webhook server to receive alerts.

1. Set up Webhook notifications in Grafana. Test your set up by clicking the Send Test button. If it's configured correctly, you should receive a JSON payload like this:

   {
       "evalMatches": [
           {
               "metric": "High value",
               "tags": null,
               "value": 100
           },
           {
               "metric": "Higher Value",
               "tags": null,
               "value": 200
           }
       ],
       "message": "Someone is testing the alert notification within grafana.",
       "ruleId": 0,
       "ruleName": "Test notification",
       "ruleUrl": "http://localhost:3000/",
       "state": "alerting",
       "title": "[Alerting] Test notification"
   }
   

   If not, check your Grafana container logs to debug.
2. Click Save to save your setup. After that, you will receive alerts from Grafana. See Grafana's Webhook notification documentation for more details.