Hosted graphite alerting service using..
- Modified Graphite-beacon.
- Alertmanager from the Prometheus project.
- Consul from Hashicorp.
- Consul template.
- Docker.
- Some tooling (python based scripts).
Modified graphite-beacon:
We use a modified version of open source graphite-beacon. This is the alert generator. The modification was done to introduce the notion of 'service'. Every alert config has a service key which is passed along every alert that is generated by that instance of the graphite beacon. The implementation meant a simple change to the existing http handler to enable it to http post the alert in Alertmanager compatible format. The service key provides us the necessary name spacing which is critical in a hosted environment. We build docker images of this for every service.
Furthermore, we introduced support for a 'team' label within an alert config by means of a small modification int the graphite-beacon code. This was needed for a big service consisting of more than one teams. It enabled us to use the 'team' label in Alertmanager to route the alert to the respective team's notification channel.
The following is an example of a test-service service. The configs in the bold letters are the additions made to the open source graphite-beacon.
We use a modified version of open source graphite-beacon. This is the alert generator. The modification was done to introduce the notion of 'service'. Every alert config has a service key which is passed along every alert that is generated by that instance of the graphite beacon. The implementation meant a simple change to the existing http handler to enable it to http post the alert in Alertmanager compatible format. The service key provides us the necessary name spacing which is critical in a hosted environment. We build docker images of this for every service.
Furthermore, we introduced support for a 'team' label within an alert config by means of a small modification int the graphite-beacon code. This was needed for a big service consisting of more than one teams. It enabled us to use the 'team' label in Alertmanager to route the alert to the respective team's notification channel.
The following is an example of a test-service service. The configs in the bold letters are the additions made to the open source graphite-beacon.
Alertmanager:
We use the stock Alertmanager from the Prometheus project. This is a common piece of infrastructure. The alert generators hits the client interface of the Alertmanager. It primarily uses the service and the severity alert labels to route alerts appropriately per the configuration. The following is an example config snippet.
We use the stock Alertmanager from the Prometheus project. This is a common piece of infrastructure. The alert generators hits the client interface of the Alertmanager. It primarily uses the service and the severity alert labels to route alerts appropriately per the configuration. The following is an example config snippet.
Consul:
We use the consul key store to store the alert configuration. Every alert generator docker instance has its own config key which it watches for changes.The host running the docker instances runs the consul agent in client mode.
We use the consul key store to store the alert configuration. Every alert generator docker instance has its own config key which it watches for changes.The host running the docker instances runs the consul agent in client mode.
Consul template:
Consul template (version 0.16) to exec the alert generator within a docker instance. This lets us watch the right consul key (alert config) in daemon mode reloading/restarting the alert generator on new config.
Consul template (version 0.16) to exec the alert generator within a docker instance. This lets us watch the right consul key (alert config) in daemon mode reloading/restarting the alert generator on new config.
Docker:
We run docker instances of the alert generators as well as the alert manager. Only one Alertmanager docker container is run while every service has its own alert generator container. The following are the configs for a typical graphite-beacon docker configs
ubuntu@ubuntu ~/docker/graphite-beacon/test-service$ ls -al
total 64
drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 24 05:35 .
drwxrwxr-x 6 ubuntu ubuntu 4096 Nov 4 05:18 ..
-rw-rw-r-- 1 ubuntu ubuntu 50 Oct 24 05:35 config_test-service.tmpl
-rw-rw-r-- 1 ubuntu ubuntu 1090 Oct 24 05:35 Dockerfile
-rw-rw-r-- 1 ubuntu ubuntu 46058 Oct 24 05:35 python-graphite-beacon_0.25.5_all.deb
ubuntu@ubuntu ~$ cat config_test-service.tmpl
{{key "/monitoring/configs/alerts/test-service"}}
ubuntu@ubuntu ~$ cat Dockerfile
FROM ubuntu:14.04
MAINTAINER Samit Pal (samit04@gmail.com)
RUN apt-get update && apt-get install --force-yes -y python-pip
RUN apt-get update && apt-get install --force-yes -y python-pkg-resources zip wget
RUN pip install 'tornado==4.1.0'
RUN pip install 'funcparserlib==0.3.6'
ADD python-graphite-beacon_0.25.5_all.deb /python-graphite-beacon_0.25.5_all.deb
RUN dpkg -i /python-graphite-beacon_0.25.5_all.deb
#Setup Consul Template Files
RUN mkdir /etc/consul-templates
ADD config_test-service.tmpl /etc/consul-templates/config_test-service.tmpl
# Get consul-template. We need version 0.16 and above.
ENV consul_template_version 0.16.0
RUN wget https://releases.hashicorp.com/consul-template/${consul_template_version}/consul-template_${consul_template_version}_linux_amd64.zip
RUN unzip consul-template_${consul_template_version}_linux_amd64.zip
RUN cp consul-template /usr/local/bin/
CMD consul-template -consul localhost:8500 -template="/etc/consul-templates/config_test-service.tmpl:/tmp/test-service.json" -exec "graphite-beacon --config=/tmp/test-service.json --logging=debug"
We run docker instances of the alert generators as well as the alert manager. Only one Alertmanager docker container is run while every service has its own alert generator container. The following are the configs for a typical graphite-beacon docker configs
ubuntu@ubuntu ~/docker/graphite-beacon/test-service$ ls -al
total 64
drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 24 05:35 .
drwxrwxr-x 6 ubuntu ubuntu 4096 Nov 4 05:18 ..
-rw-rw-r-- 1 ubuntu ubuntu 50 Oct 24 05:35 config_test-service.tmpl
-rw-rw-r-- 1 ubuntu ubuntu 1090 Oct 24 05:35 Dockerfile
-rw-rw-r-- 1 ubuntu ubuntu 46058 Oct 24 05:35 python-graphite-beacon_0.25.5_all.deb
ubuntu@ubuntu ~$ cat config_test-service.tmpl
{{key "/monitoring/configs/alerts/test-service"}}
ubuntu@ubuntu ~$ cat Dockerfile
FROM ubuntu:14.04
MAINTAINER Samit Pal (samit04@gmail.com)
RUN apt-get update && apt-get install --force-yes -y python-pip
RUN apt-get update && apt-get install --force-yes -y python-pkg-resources zip wget
RUN pip install 'tornado==4.1.0'
RUN pip install 'funcparserlib==0.3.6'
ADD python-graphite-beacon_0.25.5_all.deb /python-graphite-beacon_0.25.5_all.deb
RUN dpkg -i /python-graphite-beacon_0.25.5_all.deb
#Setup Consul Template Files
RUN mkdir /etc/consul-templates
ADD config_test-service.tmpl /etc/consul-templates/config_test-service.tmpl
# Get consul-template. We need version 0.16 and above.
ENV consul_template_version 0.16.0
RUN wget https://releases.hashicorp.com/consul-template/${consul_template_version}/consul-template_${consul_template_version}_linux_amd64.zip
RUN unzip consul-template_${consul_template_version}_linux_amd64.zip
RUN cp consul-template /usr/local/bin/
CMD consul-template -consul localhost:8500 -template="/etc/consul-templates/config_test-service.tmpl:/tmp/test-service.json" -exec "graphite-beacon --config=/tmp/test-service.json --logging=debug"
Push scripts:
We developed some config push scripts in python. These enable us to push the config to consul in an user friendly manner. Apart from pushing the configs to consul it also does some basic sanity checks on the configs. In an ideal case the configs are checked out from version control following which the push scripts are run over the same.
We developed some config push scripts in python. These enable us to push the config to consul in an user friendly manner. Apart from pushing the configs to consul it also does some basic sanity checks on the configs. In an ideal case the configs are checked out from version control following which the push scripts are run over the same.
