# prometheus-app-monitoring

## Summary

This describes how to use the pre-installed Magnum kube-prometheus stack to create a new prometheus instance and alert manager cluster to monitor an example application (mysql) and send alerts to Slack.

You will end up with two namespaces with the following resources - 

 - Production `Namespace`
    - MySQL server `Deployment`
    - MySQL `Service`
    - MySQL `PersistentVolumeClaim`
    - MySQL prometheus exporter `Deployment`
    - MySQL exporter `Service`
    - Prometheus `ServiceMonitor`
 - Monitoring `Namespace`
    - Prometheus `ServiceAccount`
    - `Prometheus` server
    - Prometheus `Service`
    - `PrometheusRule`
    - `AlertManager` cluster
    - AlertManger `Service`
    - Alert Manager `Secret` for config

## Workflow

 - Create a `monitoring` namespace to run prometheus and alertmanager, and a `production` namespace for mysql. Note that the namespaces are given labels for reference later on.

`kubectl create -f kube-prometheus/mysql/namespace.yaml`

 - Create the mysql application in the `production` namespace

`kubectl create -f kube-prometheus/mysql/mysql.yaml`

 - Create the mysql prometheus exporter in the `production` namespace and `kind: ServiceMonitor` to point Prometheus at.

`kubectl create -f kube-prometheus/mysql/mysql-exporter.yaml`

 - Create the prometheus service account for required roles in the `monitoring` namespace

`kubectl create -f kube-prometheus/prometheus/serviceaccount.yaml`

 - Create the new prometheus server in the `monitoring` namespace with included alerting spec for when we add alertmanager later on

`kubectl create -f kube-prometheus/prometheus/prometheus.yaml`

 - Create the service to expose prometheus in the `monitoring` namespace
  
`kubectl create -f kube-prometheus/prometheus/promexpose.yaml`

 - Check you can reach the service by running the command on your local machine and browsing to `http://127.0.0.1:9090/targets`

`kubectl --namespace monitoring port-forward svc/prometheus 9090`

 - Edit [/kube-prometheus/alertmanager/alertmanagersecret.yaml](/kube-prometheus/alertmanager/alertmanagersecret.yaml) to change the `slack_api_url` and `channel` 

 - Create the alertmanager secret for the config in the `monitoring` namespace

`kubectl create -f kube-prometheus/alertmanager/alertmanagersecret.yaml`

 - Create the alertmanager cluster in the `monitoring` namespace (you can change the replica count if you wish, default is 1)

`kubectl create -f kube-prometheus/alertmanager/amcluster.yaml`

 - Create the service to expose alertmanager in the `monitoring` namespace

`kubectl create -f kube-prometheus/alertmanager/amexpose.yaml`

 - Check you can reach the service by running the command on your local machine and browsing to `http://127.0.0.1:9093`

`kubectl --namespace monitoring port-forward svc/amservice 9093`
 
 - Create a rule to trigger a test alert

`kubectl create -f kube-prometheus/alertmanager/amrule.yaml`

Now wait a couple of minutes and the alert will appear in prometheus and alertmanager, Slack will be notified 


### Grafana

Use the preinstalled Grafana instance to visualise the prometheus metrics by adding a new datasource

[http://docs.embassy.ebi.ac.uk/userguide/Embassyv4_Magnum.html#monitoring](http://docs.embassy.ebi.ac.uk/userguide/Embassyv4_Magnum.html#monitoring)

Log in and navigate to Data Sources and add a new one. You will need a url defining the namespace as the prometheus service will be in a different namespace

`http://prometheus.monitoring.svc.cluster.local:9090/`