How to Configure Prometheus and Grafana in Multi Node Kubernetes

In this article, We will configure the Prometheus and Grafana in Multi-Node Kubernetes Cluster

Prometheus:

It is widely used for monitoring containerized environments, including Kubernetes. Prometheus collects metrics from various sources, stores them in a time-series database, and provides a powerful query language for analyzing and visualizing the collected data.

Grafana

Grafana is an open-source data visualization and monitoring tool that allows you to create rich dashboards and graphs for visualizing data from various sources, including time-series databases like Prometheus. Grafana provides a user-friendly interface to query, analyze, and display metrics and logs in real time. It offers a wide range of visualization options, and alerting capabilities, and supports integration with various data sources.

Prerequisites:

Kubernetes Cluster
NFS Server (Optional for Lower Environments)
Helm

Installation:

Step 1: [Identify default NFS Storage Class]

For the Production Cluster, We must be required to configure the persistent volume storage class and for development purposes, we can disable the volume in the Prometheus helm chart.

arun@Aruns-MacBook-Air Documents % kubectl get nodes
NAME       STATUS   ROLES           AGE   VERSION
master     Ready    control-plane   55d   v1.26.3
worker01   Ready    <none>          55d   v1.26.3
worker02   Ready    <none>          55d   v1.26.3

arun@Aruns-MacBook-Air Documents % kubectl get pods -n monitoring
No resources found in monitoring namespace.

arun@Aruns-MacBook-Air Documents % kubectl get sc
NAME                   PROVISIONER                                   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-client (default)   k8s-sigs.io/nfs-subdir-external-provisioner   Delete          Immediate           false                  94m

Step 2: [Add Prometheus and Grafana Helm Repo]

Using Helm, We added the Prometheus and Grafana helm repositories.

arun@Aruns-MacBook-Air Documents % helm repo list
Error: no repositories to show
arun@Aruns-MacBook-Air Documents %

arun@Aruns-MacBook-Air Documents % helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories


arun@Aruns-MacBook-Air Documents % helm repo add grafana https://grafana.github.io/helm-charts
"grafana" has been added to your repositories


arun@Aruns-MacBook-Air Documents % helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
arun@Aruns-MacBook-Air Documents %

Step 3: [Install Prometheus]

Here we installed Prometheus using the helm chart in the monitoring namespace.

arun@Aruns-MacBook-Air Documents % kubectl create ns monitoring
namespace/monitoring created
arun@Aruns-MacBook-Air Documents %

arun@Aruns-MacBook-Air Documents % helm install prometheus prometheus-community/prometheus -n monitoring
NAME: prometheus
LAST DEPLOYED: Thu May 25 22:15:32 2023
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.monitoring.svc.cluster.local

Step 4: [Install Grafana]

Here we installed Grafana using the helm chart in the monitoring namespace.

arun@Aruns-MacBook-Air Documents % helm install grafana grafana/grafana -n monitoring
NAME: grafana
LAST DEPLOYED: Thu May 25 23:10:46 2023
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Step 5: [Configure NodePort to Prometheus and Grafana Services]

Configure NodePort/Ingress for exposing to outside cluster.

arun@Aruns-MacBook-Air Documents % kubectl patch -n monitoring service prometheus-server -p '{"spec": {"type": "NodePort"}}'
service/prometheus-server patched

arun@Aruns-MacBook-Air Documents % kubectl patch -n monitoring service grafana -p '{"spec": {"type": "NodePort"}}'
service/grafana patched

Prometheus and Grafana Dashboard: