I have a small Dell machine (Optiplex 5050) running Openwrt virtualized with Proxmox, and I noticed a while ago that my home network was sometimes lagging, and then I looked at the monitoring on Proxmox and found that the openwrt virtual machine was running at 100% CPU during this time, so no wonder it was very laggy.

I also have a small Dell machine (Optiplex 5080), which is also virtualized on Proxmox, and the two Dell’s add up to a total of 5 virtual machines, so it’s time to get on a monitor.

The community for Proxmox monitoring generally have Prometheus and influxdb two ways, I use here is Prometheus.

prometheus-pve-exporter

Like the Node exporter, a Proxmox exporter is also required, which can be used with the prometheus-pve-exporter project. The prometheus-pve-exporter calls Proxmox’s API to collect metrics, and then Prometheus then grabs the metrics data from the prometheus-pve-exporter and finally displays it through Grafana.

The prometheus-pve-exporter can be configured using either a configuration file or environment variables. Here I passed in PVE_USER/PVE_PASSWORD/PVE_VERIFY_SSL to authenticate when calling the Proxmox API. Since this is a monitoring request, it is recommended to create a new read-only user on Proxmox instead of using root; creating a new user on Proxmox is easy and can be done on the page, but authorizing the user can only be done from the command line.

As follows, grant PVEAuditor privileges to user monitor@pve.

1

pveum aclmod / -user monitor@pve -role PVEAuditor

Note that the format of PVE_USER is user@pve or user@pam, the difference between the two is that pve is a proxmox user, for example, I have a cluster of 2 Proxmoxes here, create a pve user to access 2 Proxmox machines; pam is a linux user, the username password may be different for different Proxmox machines (Proxmox is actually a Linux machine too).

PVE_VERIFY_SSL needs to be set to false, because it is a self-signed certificate.

prometheus-pve-exporter can be started directly or containerized. Because I want to use Prometheus for monitoring later, I deployed prometheus-pve-exporter to kubernetes and created a svc. Deployment is written similar to the following.

1

docker run --name prometheus-pve-exporter -d -p 127.0.0.1:9221:9221 -v /path/to/pve.yml:/etc/pve.yml prompve/prometheus-pve-exporter

The created svc.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: pve-exporter
  name: pve-exporter
  namespace: default
spec:
  ports:
  - name: pve-exporter-0
    port: 9221
    protocol: TCP
    targetPort: 9221
  selector:
    app: pvc-exporter-5050
  type: LoadBalancer

After prometheus-pve-exporter is started, metrics can be requested via http.

1

http://{service loadbalancer ip}:9221/pve?target=1.2.3.4

A prometheus-pve-exporter can request multiple Proxmox(targets).

Prometheus

I am using Prometheus Operator here, so just configure ServiceMonitor. endpoints configure 2 Proxmox.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: pve-exporter
  name: pve-exporter
spec:
  endpoints:
  - interval: 30s
    params:
      target:
      - 1.1.1.1
    path: /pve
    port: pve-exporter-0
    scheme: http
  - interval: 30s
    params:
      target:
      - 1.1.1.2
    path: /pve
    port: pve-exporter-0
    scheme: http
  jobLabel: k8s-app
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      k8s-app: pve-exporter

Note that namespaceSelector and selector.matchLabels and the pve-exporter created above should match up.

After the ServiceMonitor is created, Prometheus Operator will automatically refresh the Prometheus configuration, after which the pve monitoring information (e.g. pve_up) can be viewed on Prometheus.

Grafana

Once the Proxmox data is collected in Prometheus, it is ready to be presented via Grafana. Here I am using Proxmox via Prometheus by Pietro Saccardi.

At this point, it is relatively easy to display the status of all Proxmox VMs in a unified way.