README.md 3.47 KB
Newer Older
David Ocana's avatar
David Ocana committed
1
2
# prometheus-app-monitoring

Charles Short's avatar
Charles Short committed
3
4
5
6
## Summary

This describes how to use the pre-installed Magnum kube-prometheus stack to create a new prometheus instance and alert manager cluster to monitor an example application (mysql) and send alerts to Slack.

Charles Short's avatar
Charles Short committed
7
8
You will end up with two namespaces with the following resources - 

Charles Short's avatar
Charles Short committed
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
 - Production `Namespace`
    - MySQL server `Deployment`
    - MySQL `Service`
    - MySQL `PersistentVolumeClaim`
    - MySQL prometheus exporter `Deployment`
    - MySQL exporter `Service`
    - Prometheus `ServiceMonitor`
 - Monitoring `Namespace`
    - Prometheus `ServiceAccount`
    - `Prometheus` server
    - Prometheus `Service`
    - `PrometheusRule`
    - `AlertManager` cluster
    - AlertManger `Service`
    - Alert Manager `Secret` for config
Charles Short's avatar
Charles Short committed
24

Charles Short's avatar
Charles Short committed
25
26
## Workflow

Charles Short's avatar
Charles Short committed
27
 - Create a `monitoring` namespace to run prometheus and alertmanager, and a `production` namespace for mysql. Note that the namespaces are given labels for reference later on.
Charles Short's avatar
Charles Short committed
28
29
30
31
32
33
34

`kubectl create -f kube-prometheus/mysql/namespace.yaml`

 - Create the mysql application in the `production` namespace

`kubectl create -f kube-prometheus/mysql/mysql.yaml`

Charles Short's avatar
Charles Short committed
35
 - Create the mysql prometheus exporter in the `production` namespace and `kind: ServiceMonitor` to point Prometheus at.
Charles Short's avatar
Charles Short committed
36
37
38

`kubectl create -f kube-prometheus/mysql/mysql-exporter.yaml`

Charles Short's avatar
Charles Short committed
39
 - Create the prometheus service account for required roles in the `monitoring` namespace
Charles Short's avatar
Charles Short committed
40
41
42

`kubectl create -f kube-prometheus/prometheus/serviceaccount.yaml`

Charles Short's avatar
Charles Short committed
43
 - Create the new prometheus server in the `monitoring` namespace with included alerting spec for when we add alertmanager later on
Charles Short's avatar
Charles Short committed
44
45
46

`kubectl create -f kube-prometheus/prometheus/prometheus.yaml`

Charles Short's avatar
Charles Short committed
47
 - Create the service to expose prometheus in the `monitoring` namespace
Charles Short's avatar
Charles Short committed
48
49
50
51
52
53
54
55
56
  
`kubectl create -f kube-prometheus/prometheus/promexpose.yaml`

 - Check you can reach the service by running the command on your local machine and browsing to `http://127.0.0.1:9090/targets`

`kubectl --namespace monitoring port-forward svc/prometheus 9090`

 - Edit [/kube-prometheus/alertmanager/alertmanagersecret.yaml](/kube-prometheus/alertmanager/alertmanagersecret.yaml) to change the `slack_api_url` and `channel` 

Charles Short's avatar
Charles Short committed
57
 - Create the alertmanager secret for the config in the `monitoring` namespace
Charles Short's avatar
Charles Short committed
58
59
60

`kubectl create -f kube-prometheus/alertmanager/alertmanagersecret.yaml`

Charles Short's avatar
Charles Short committed
61
 - Create the alertmanager cluster in the `monitoring` namespace (you can change the replica count if you wish, default is 1)
Charles Short's avatar
Charles Short committed
62
63
64

`kubectl create -f kube-prometheus/alertmanager/amcluster.yaml`

Charles Short's avatar
Charles Short committed
65
 - Create the service to expose alertmanager in the `monitoring` namespace
Charles Short's avatar
Charles Short committed
66
67
68
69
70
71
72
73
74
75
76

`kubectl create -f kube-prometheus/alertmanager/amexpose.yaml`

 - Check you can reach the service by running the command on your local machine and browsing to `http://127.0.0.1:9093`

`kubectl --namespace monitoring port-forward svc/amservice 9093`
 
 - Create a rule to trigger a test alert

`kubectl create -f kube-prometheus/alertmanager/amrule.yaml`

Charles Short's avatar
Charles Short committed
77
Now wait a couple of minutes and the alert will appear in prometheus and alertmanager, Slack will be notified 
Charles Short's avatar
Charles Short committed
78
79
80
81


### Grafana

Charles Short's avatar
Charles Short committed
82
Use the preinstalled Grafana instance to visualise the prometheus metrics by adding a new datasource
Charles Short's avatar
Charles Short committed
83
84
85
86
87

[http://docs.embassy.ebi.ac.uk/userguide/Embassyv4_Magnum.html#monitoring](http://docs.embassy.ebi.ac.uk/userguide/Embassyv4_Magnum.html#monitoring)

Log in and navigate to Data Sources and add a new one. You will need a url defining the namespace as the prometheus service will be in a different namespace

Charles Short's avatar
Charles Short committed
88
`http://prometheus.monitoring.svc.cluster.local:9090/`
Charles Short's avatar
Charles Short committed
89