How To Setup Prometheus and Grafana Monitoring on Kubernetes | by Guillaume Vincent | Mar, 2022

Tourist viewer so they can see the city from afar
Photo by Matthijs van Schuppen on Unsplash

Timeline

2012 — built at SoundCloud 2013 — use in SoundCloud's production May 2016 — CNCF incubated Prometheus July 2016 — Prometheus 1.0 released Nov 2017 — Prometheus 2.0 released Aug 2018 — CNCF graduated Prometheus production-ready
The Prometheus project timeline

Concepts

Example of a metric exposes to Prometheus over HTTP
Example of a metric exposes to Prometheus over HTTP

Architecture

The Prometheus architecture from https://prometheus.io/
  • Prometheus server scraping and stores time-series data. It also provides a user interface to query the metrics.
  • Client libraries for instrumenting application code.
  • Pushgateway supports metrics collection from short-lived jobs
  • Exporters for services that do not instrument Prometheus metrics directly.
  • Alertmanager handles real-time alerts based on triggers

Installing Helm

$ brew install helm

Creating the dedicated monitoring namespace

$ kubectl create namespace monitoring

Installing kube-prometheus-stack with Helm

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 
$ helm repo update
$ helm upgrade --namespace monitoring --install kube-stack-prometheus prometheus-community/kube-prometheus-stack --set prometheus-node-exporter.hostRootFsMount.enabled=false
$ kubectl get -n monitoring crds                                                           
NAME CREATED AT
alertmanagerconfigs.monitoring.coreos.com 2022-03-15T10:54:41Z
alertmanagers.monitoring.coreos.com 2022-03-15T10:54:42Z
podmonitors.monitoring.coreos.com 2022-03-15T10:54:42Z
probes.monitoring.coreos.com 2022-03-15T10:54:42Z
prometheuses.monitoring.coreos.com 2022-03-15T10:54:42Z
prometheusrules.monitoring.coreos.com 2022-03-15T10:54:42Z
servicemonitors.monitoring.coreos.com 2022-03-15T10:54:42Z
thanosrulers.monitoring.coreos.com 2022-03-15T10:54:42Z
$ kubectl get pods -n monitoring                                                                    
NAME READY STATUS RESTARTS AGE
alertmanager-kube-stack-prometheus-kube-alertmanager-0 2/2 Running 0 2m36s
kube-stack-prometheus-grafana-6994bd6c69-h6s9z 3/3 Running 0 13h
kube-stack-prometheus-kube-operator-86667b5cdf-cqndt 1/1 Running 0 13h
kube-stack-prometheus-kube-state-metrics-fc9878699-dpgh6 1/1 Running 0 13h
kube-stack-prometheus-prometheus-node-exporter-vrjsl 1/1 Running 0 13h
prometheus-kube-stack-prometheus-kube-prometheus-0 2/2 Running 0 13h
$ kubectl port-forward --namespace monitoring svc/kube-stack-prometheus-kube-prometheus 9090:9090
Dashboard in the Prometheus web UI showing time-series metrics
Dashboard in the Prometheus web UI showing time-series metrics
The metric endpoints discovered by Prometheus are visible in Status>Targets menu” class=”cf kk kl” src=”https://miro.medium.com/max/1400/0*0oN5q_IE5N100Fq-.png” width=”700″ height=”183″/></div>
</div><figcaption class=The metric endpoints discovered by Prometheus are visible in Status>Targets menu
Few target endpoints discovered by Prometheus server
Few target endpoints discovered by Prometheus server
$ kubectl get secret --namespace monitoring kube-stack-prometheus-grafana -o jsonpath='{.data.admin-user}' | base64 -d 
$ kubectl get secret --namespace monitoring kube-stack-prometheus-grafana -o jsonpath='{.data.admin-password}' | base64 -d
$ kubectl port-forward --namespace monitoring svc/kube-stack-prometheus-grafana 8080:80
The Grafana login interface
The Grafana login interface
List of the provisioned Grafana dashboards
List of the provisioned Grafana dashboards
A Grafana dashboard showing Kubernetes pod compute resources

Leave a Comment