Alertmanager is the tool that dispatches Prometheus alerts. It is a core component of a Prometheus monitoring stack.
One year after the previous release, Alertmanager 0.22.0 has been released.
During this timeframe, we have merged a lot of new features, which I will showcase in this blog post.
New date picker in silence form
The experience creating a new silence is now greatly improved with the addition of a new date picker. To view the date picker, simply click on the calendar icon next to the date fields.
Note: in Alertmanager, all dates are in UTC.
Negative matchers
Alerts are identified by labels in Alertmanager. In order to decide where to send an alert, matchers are used. Matchers are rules that can match labels with fixed values and regexes.
The matchers are used in the routing, in the silences, and for the inhibition of alerts.
This release adds a revamp of the alertmanager matchers. New matchers can be defined inline, in a PromQL-Like syntax, such as:
route:
receiver: team-X-mail
routes:
- matchers: [ '{severity="critical",environment="prod"}' ]
receiver: team-X-pager
There is also a short form:
route:
receiver: team-X-mail
routes:
- matchers: [ severity=critical, environment=prod ]
receiver: team-X-pager
With this new matchers, it is also possible to have negative matchers:
route:
receiver: team-X-pager
routes:
- matchers: [ '{severity!="critical",environment!="prod"}' ]
receiver: team-X-mail
This provides new ways of writing your alertmanager configuration, which can make them shorter or more readable.
amtool, the alertmanager command line tool, also knows the new matchers. In
this snippet, we silence all the alerts which do not have a label
“environment” equal to “production”:
amtool silence add --alertmanager.url=http://127.0.0.1:9093 \
-c 'Silencing everything but production.' \
'environment!=prod'
The reference documentation provides more details.
Time-based muting
Another great feature is the addition of time-based muting in the routes. It means that alertmanager can decide whether to send notifications based on defined time windows.
route:
receiver: team-X-pager
routes:
- matchers: [ '{severity!="critical",environment!="prod"}' ]
receiver: team-X-mail
mute_time_intervals: [out-of-business-hours]
mute_time_intervals:
- name: out-of-business-hours
time_intervals:
# Mute on Saturdays and Sundays, all day.
- weekdays: ['Saturday','Sunday']
# Mute in the morning and in the evening, any day.
- times:
- start_time: '00:00'
end_time: '08:00'
- start_time: '18:00'
end_time: '24:00'
This configuration means that team-X-mail will only receive emails during weekdays, between 08:00 and 18:00 UTC.
Other enhancements
Alertmanager now supports TLS and basic authentication, like the Node Exporter.
The same flags and configuration format are used for both projects, providing consistency in the way you can secure your Prometheus stack.
Conclusion
This alertmanager release is packed - and there is more to come. Notably, we have lifted the moratorium on new integrations, and we expect new notification integrations to be implemented over the next few months.
You can read the full changelog to learn about all the other bug fixes and enhancements.
Useful links
Are you having trouble with setting up and scaling Prometheus? Our team of observability experts can support the operations of your Prometheus environments.
