Monitoring apps with Prometheus and Grafana on an OVHcloud Managed Kubernetes Service

Find out how to monitor and visualize metrics with Prometheus and Grafana on an OVHcloud Managed Kubernetes Service

Last updated March 16, 2022.

Objective

In this tutorial we will show you how to monitor your applications/workloads on an OVHcloud Managed Kubernetes cluster.

Before you begin

This tutorial presupposes that you already have a working OVHcloud Managed Kubernetes cluster, and some basic knowledge of how to operate it. If you want to know more on those topics, please look at the OVHcloud Managed Kubernetes Service Quickstart guide.

You also need to have Helm installed on your workstation and your cluster. Please refer to the How to install Helm on OVHcloud Managed Kubernetes Service tutorial.

Instructions

In this guide you will:

  • install Prometheus operator (it will install Prometheus & Grafana)
  • query Prometheus
  • visualize metrics on Grafana

Prometheus operator

The Prometheus Operator provides Kubernetes native deployment and management of Prometheus and related monitoring components.

Prometheus Architecture

The purpose of this project is to simplify and automate the configuration of a Prometheus-based monitoring stack for Kubernetes clusters. The Prometheus operator also deploys Grafana and its dashboards, to visualize our metrics in a user-friendly way.

If you are interested into the operator, feel free to read the Prometheus operator official documentation.

Installing the Prometheus operator

For this tutorial we are using the Prometheus Operator Helm chart found on the Prometheus Community repository.

Add the Prometheus Helm repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

This will add the Prometheus repository and update all of your repositories:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...
...Successfully got an update from the "prometheus-community" chart repository
...
Update Complete. ⎈Happy Helming!⎈

You need to modify some settings. To do this, inspect the chart to retrieve these values ​​in a file:

helm inspect values prometheus-community/kube-prometheus-stack > /tmp/kube-prometheus-stack.values

Open the /tmp/kube-prometheus-stack.values file in your favorite editor.

Then search into it for adminPassword and replace it with the password you want to use for Grafana.

By default, the Grafana password should be like this:

  adminPassword: prom-operator

Copy and save the Grafana admin password, you will use it later in this tutorial.

You are now ready to install Prometheus and Grafana.

helm install prometheus-community/kube-prometheus-stack \
--create-namespace --namespace prometheus \
--generate-name \
--values /tmp/kube-prometheus-stack.values \
--set prometheus.service.type=LoadBalancer \
--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
--set grafana.service.type=LoadBalancer

As you can see, a new prometheus namespace will be created and we specified that we want to deploy a LoadBalancer in order to access externally to Prometheus and Grafana easily.

You should have a behavior like this:

$ helm install prometheus-community/kube-prometheus-stack \
--create-namespace --namespace prometheus \
--generate-name \
--values /tmp/kube-prometheus-stack.values \
--set prometheus.service.type=LoadBalancer \
--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
--set grafana.service.type=LoadBalancer
NAME: kube-prometheus-stack-1647417678
LAST DEPLOYED: Wed Mar 16 09:01:23 2022
NAMESPACE: prometheus
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace prometheus get pods -l "release=kube-prometheus-stack-1647417678"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

You can also verify by checking the Pods in the new prometheus namespace:

$ kubectl get pods -n prometheus
NAME                                                              READY   STATUS    RESTARTS   AGE
alertmanager-kube-prometheus-stack-1647-alertmanager-0            2/2     Running   0          14m
kube-prometheus-stack-1647-operator-57b69b9667-d92sc              1/1     Running   0          14m
kube-prometheus-stack-1647417678-grafana-695574b948-znqsd         3/3     Running   0          14m
kube-prometheus-stack-1647417678-kube-state-metrics-6d4bc49pqlc   1/1     Running   0          14m
kube-prometheus-stack-1647417678-prometheus-node-exporter-gxphr   1/1     Running   0          14m
kube-prometheus-stack-1647417678-prometheus-node-exporter-jdtr6   1/1     Running   0          14m
kube-prometheus-stack-1647417678-prometheus-node-exporter-xkqt2   1/1     Running   0          14m
prometheus-kube-prometheus-stack-1647-prometheus-0                2/2     Running   0          14m

You can also check that Prometheus and Grafana have an external IP:

$ kubectl get svc -n prometheus
NAME                                                        TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE
alertmanager-operated                                       ClusterIP      None                      9093/TCP,9094/TCP,9094/UDP   15m
kube-prometheus-stack-1647-alertmanager                     ClusterIP      10.3.14.89                9093/TCP                     15m
kube-prometheus-stack-1647-operator                         ClusterIP      10.3.176.201              443/TCP                      15m
kube-prometheus-stack-1647-prometheus                       LoadBalancer   10.3.105.242   51.210.210.213   9090:31819/TCP               15m
kube-prometheus-stack-1647417678-grafana                    LoadBalancer   10.3.28.111    51.210.253.47    80:32481/TCP                 15m
kube-prometheus-stack-1647417678-kube-state-metrics         ClusterIP      10.3.235.254              8080/TCP                     15m
kube-prometheus-stack-1647417678-prometheus-node-exporter   ClusterIP      10.3.237.94               9100/TCP                     15m
prometheus-operated                                         ClusterIP      None                      9090/TCP                     14m

If it's not the case, please wait until the Load Balancers are correctly created.

Visualize the metrics

Now you can retrieve Prometheus and Grafana URLs with the following commands:

export PROMETHEUS_URL=$(kubectl get svc -n prometheus -l app=kube-prometheus-stack-prometheus -o jsonpath='{.items[].status.loadBalancer.ingress[].ip}')

echo Prometheus URL: http://$PROMETHEUS_URL:9090

export GRAFANA_URL=$(kubectl get svc -n prometheus -l app.kubernetes.io/name=grafana -o jsonpath='{.items[].status.loadBalancer.ingress[].ip}')

echo Grafana URL: http://$GRAFANA_URL

You should obtain the following result:

$ export PROMETHEUS_URL=$(kubectl get svc -n prometheus -l app=kube-prometheus-stack-prometheus -o jsonpath='{.items[].status.loadBalancer.ingress[].ip}')

$ echo Prometheus URL: http://$PROMETHEUS_URL:9090
Prometheus URL: http://152.228.168.191:9090

$ export GRAFANA_URL=$(kubectl get svc -n prometheus -l app.kubernetes.io/name=grafana -o jsonpath='{.items[].status.loadBalancer.ingress[].ip}')

$ echo Grafana URL: http://$GRAFANA_URL
Grafana URL: http://51.178.69.167

Open your browser and go to the Prometheus interface.

Without doing anything on your side, several built-in metrics are already available. You can test them by typing sum(kube_pod_owner{job="kube-state-metrics"}) by (namespace) in the search bar.

Click on the Execute button to determine if the Kubernetes metrics are visible:

Prometheus

Thanks to the PromQL query you can list the number of pods per namespace in your Kubernetes cluster.

Kubernetes components emit metrics in Prometheus format. You can find more information in the Kubernetes official documentation.

You can also go to the Grafana interface. Open your browser and point to http://$GRAFANA_URL value using the credentials below:

  • Login: admin
  • Password: prom-operator (by default)

Grafana

Grafana Home Page

This time again, thanks to the prometheus operator and without doing anything on your side, several dashboards are available. You can show them by clicking on the Dashboards icon, in the Grafana sidebar, then on browse:

Grafana Grafana

For example, you can click on the General/Kubernetes/Compute Resources/Cluster dashboard to visualize the metrics of your Kubernetes cluster:

Grafana

Cleanup

First, execute the helm list command in every namespace (with -A option) in your Kubernetes cluster to see what you've installed.

$ helm list -A
NAME                                NAMESPACE       REVISION    UPDATED                                 STATUS      CHART                           APP VERSION
kube-prometheus-stack-1647417678    prometheus      1           2022-03-16 09:01:23.979906 +0100 CET    deployed    kube-prometheus-stack-33.2.1    0.54.1

Now, you can delete what you've installed in this tutorial, thanks to helm uninstall commands:

helm uninstall kube-prometheus-stack-1647417678 -n prometheus

You should have a behavior like this:

$ helm uninstall kube-prometheus-stack-1647417678 -n prometheus
release "kube-prometheus-stack-1647417678" uninstalled

Go further

Prometheus and Grafana are very powerful monitoring tools, but also have alerting systems. Don't hesitate to dig in, to create alerts for example.

To learn more about using your Kubernetes cluster the practical way, we invite you to look at our OVHcloud Managed Kubernetes documentation.

Join our community of users.


Did you find this guide useful?

Please feel free to give any suggestions in order to improve this documentation.

Whether your feedback is about images, content, or structure, please share it, so that we can improve it together.

Your support requests will not be processed via this form. To do this, please use the "Create a ticket" form.

Thank you. Your feedback has been received.


These guides might also interest you...

OVHcloud Community

Access your community space. Ask questions, search for information, post content, and interact with other OVHcloud Community members.

Discuss with the OVHcloud community

In accordance with the 2006/112/CE Directive, modified on 01/01/2015, prices exclude VAT. VAT may vary according to the customer's country of residence.