Monitor of Kubernetes Using Prometheus Grafana

The Prometheus community is updating so fast that some of the documentation written before is a bit out of date. Recently, I started to focus on observability again, and make up for some knowledge points in operation and maintenance.

1. Explanation of terms

• Grafana

A visualization tool that provides various visualization panels and supports various data sources, including Prometheus, OpenTSDB, MySQL, etc.

• Prometheus

A time series database, mainly used to collect, store, and provide query data to the public.

• Exporter

A program used to expose service monitoring metrics, providing API interface to Prometheus to pull monitoring data.

• PromQL

Prometheus’ built-in data query language, which provides support for rich querying, aggregation, and logical computing capabilities for time series data.

2. Install Prometheus

• Add Helm source

1
2

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

• Install Prometheus

1

helm install prometheus -n monitor prometheus-community/prometheus --create-namespace

If prometheus-node-exporter is not working, the default port 9100 is probably occupied. You can change the default port by editing DaemonSet with the following command.

1

kubectl -n monitor edit ds prometheus-node-exporter

• Uninstall

1

helm uninstall prometheus -n monitor

3. Installing Grafana

• Add the Helm source

1
2

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

1

helm -n monitor install grafana grafana/grafana

It is worth noting that with this installation method, Grafana data is stored under the /var/lib/grafana path of the Pod, and the associated configuration will be lost if you restart Grafana. For production environments, you need to mount the storage volume. If a StorageClass is available, you can use the following parameters for persistent storage.

1

helm -n monitor install grafana grafana/grafana  --set persistence.storageClassName="YOUR_STORAGECLASS_NAME" --set persistence.enabled="true"

• Get the login password for the admin account

1
2
3

kubectl -n monitor get secret grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

JIsmMsWaN8rF5ryS7rVohHyFWKzyahR7u0OJsiJL

• Modify the access method of the Grafana service to NodePort

1

kubectl -n monitor patch svc grafana -p '{"spec": {"type": "NodePort"}}'

1
2
3
4

kubectl -n monitor get svc grafana

NAME                 TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
grafana   NodePort   10.233.6.118   <none>        80:31892/TCP   117s

This is used to access Grafana via the host IP + 31892.

• Uninstall

1

helm -n monitor uninstall grafana

4. Configuring Use

4.1 Adding data sources

In the left navigation bar, find [Data Sources].

Fill in the Prometheus access address http://prometheus-server.monitor.svc and you are done.

4.2 Adding a template

Find the Import button in the left navigation bar [+], I chose the panel with id 10856 to import. You can also go to the Grafana official website and select the appropriate panel to import, https://grafana.com/grafana/dashboards.

As shown below, here you need to select the data source added above.

4.3 View Panel

4.4 grafana.ini configuration

Some of Grafana’s configurations are implemented by modifying grafana.ini, here are just a few that I use.

1. edit the configuration file

   1	kubectl -n monitor edit cm grafana

2. Add configuration

   • Allow anonymous access

   1 2 3

   grafana.ini: | [auth.anonymous] enabled = true

   • Allow Iframe embedding

   1 2 3

   grafana.ini: | [security] allow_embedding = true

3. Restart Grafana to take effect

   1	kubectl -n monitor rollout restart deployment grafana