Configure notifications

Create a new notification

Open the notifications page via the link in the bottom half of hamburger menu in the SUSE Observability UI. This opens an overview of all notifications that are already configured including their status.

You can check if the desired notification already exists. If not create a new one with the "Add new notification" button.

Configure when to notify

Notification configuration showing the silencing option and scope filters

Configure the notification:

  1. Name - Choose a name that is short but still describes what the intent is of this notification. It is for your own reference in the notifications overview.

  2. Status - The notification can be disabled temporarily in case it is not yet needed, turns out to be too noisy etc.

  3. Notify when - A critical health state always triggers a notification, but optionally also deviating states can be included. Silencing changes this behavior as described in Silencing and notifications.

  4. Scope - The example includes all monitors and resources. Use the filters in the scope section to limit this selection.

Silencing and notifications

Silencing mutes a monitor result without changing its health state. By default, a muted result does not trigger health notifications. If a notification is already open, silencing closes it. For webhooks, the close event contains reason: "HealthStateSilenced" and the note in silencingNote. Other monitor results continue to trigger notifications when they match the configuration.

Under Notify when, enable The silencing rules change to receive notifications about silencing. With this option enabled, silencing opens a notification or updates an existing notification with the silencing reason. For webhooks, the open event contains state: "MUTED" and the note in reason, including when the monitor result is CLEAR. See Silencing payloads for examples. The scope filters still apply. Changes to the silencing reason update the notification.

When silencing ends, the current health state determines what happens next. A critical result, or a deviating result included in the configuration, opens or updates a health notification. Otherwise, an open silencing notification closes. These rules also apply to silencing from external health sources.

Scope

There are 4 possible scope filters. By default a notification will be sent for each critical (and optionally deviating) health state. The filters are used to limit this scope. A health state will only result in a notification when it matches all filters.

  • Monitors: Select one or more specific monitors. Notifications will only be sent for health states of the selected monitors.

  • Monitor tags: Select one or more monitor tags. Notifications will only be sent for health states of monitors that have at least one of the selected tags.

  • Component types: Select one or more component types. Notifications will only be sent for health states of components of the selected types.

  • Component tags: Select one or more component tags. Notifications will only be sent for health states of components that have at least one matching value for each distinct key in the selected tags.

Example component tags

Say the selected tags are [ k8s-scope:prod-us/checkout, k8s-scope:prod-emea/checkout, team:blue ]. The keys in this case are k8s-scope and team.

Then these components will match:

  • a component with tags [ k8s-scope:prod-emea/checkout, team:blue ]

  • a component with tags [ k8s-scope:prod-us/checkout, team:blue ]

While these components will not match:

  • a component with tags [ k8s-scope:prod-emea/checkout ]

  • a component with tags [ k8s-scope:prod-us/checkout, team:green ]

  • a component with tags [ k8s-scope:prod-emea/carts, team:blue ]

Where to send notifications?

SUSE Observability can send notifications to different external systems via channels. Supported channels are:

  • Slack - Send notifications to Slack

  • Teams - Send notifications to Microsoft Teams

  • Webhook - Send notifications to a webhook, the webhook endpoint can translate the SUSE Observability payload into any custom third-party API needed

  • Opsgenie - Send notifications to OpsGenie

  • E-mail - Send notifications via e-mail

In general, SUSE® Observability sends two types of messages for notifications:

  1. An open message when an unmuted health state is Critical, or Deviating if included in the configuration. When The silencing rules change is enabled, muted results also send open messages, including CLEAR results. For webhooks, these silencing events contain state: "MUTED" and the note in reason. Updates send another open message for the same notification.

  2. A close message when the result no longer qualifies, or when the component disappears or the monitor is removed. With The silencing rules change disabled, muting an existing notification also sends a close message. For webhooks, this event contains reason: "HealthStateSilenced" and the note in silencingNote.