prometheus installed with kube-prometheus will only monitor default kube-system monitoring (kube-prometheus creates its own ns), but if you want to add other namespaces, you need to do something else.

1. monitor endpoint resources in other namespaces

What you need to do

  1. Create a role in the new namespace to get the monitoring information.
  2. bind the created role to the prometheus-k8s sa in the monitoring namespace.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

kubectl create ns test

namespace=test

cat <<EOF | kubectl apply -f - 
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: prometheus-k8s
  namespace: ${namespace}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: prometheus-k8s
subjects:
- kind: ServiceAccount
  name: prometheus-k8s
  namespace: monitoring

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: prometheus-k8s
  namespace: ${namespace}
rules:
- apiGroups:
  - ""
  resources:
  - services
  - endpoints
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
EOF

2. Monitor serviceMonitor resources in other namespaces

serviceMonitorNamespaceSelector Match the namespace tag, or only match the resources in its own namespace if not specified.

serviceMonitorSelector serviceMonitor’s tag matching, if not specified, only matches resources in its own namespace.

Modify the Prometheus resource configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
...
# 增加 ns 匹配的标签
  serviceMonitorNamespaceSelector:
    matchLabels:
      serviceMonitor: prometheus
      
# 或者 增加下面的匹配，用来匹配 serviceMonitor
  serviceMonitorSelector:
    matchLabels:
      serviceMonitor: prometheus

Adding namespace tags.

1
2
3

for ns in default kube-system monitoring test; do 
  kubectl patch ns $ns --patch '{"metadata":{"labels":{"serviceMonitor": "prometheus" } } }'
done

Add a tag for serviceMonitor.

1

kubectl patch -n test servicemonitor demo-app --patch '{"metadata":{"labels":{"serviceMonitor":"prometheus"}}}' --type=merge

Testing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

cat <<EOF | kubectl-test apply -f - 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-demo-app
  namespace: test
  labels:
    app: ingress-demo-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ingress-demo-app
  template:
    metadata:
      labels:
        app: ingress-demo-app
        namespace: test
    spec:
      containers:
      - name: whoami
        image: traefik/whoami:v1.6.1
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: ingress-demo-app
  namespace: test
  labels:
    app: ingress-demo-app
spec:
  type: ClusterIP
  selector:
    app: ingress-demo-app
  ports:
    - name: http
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-demo-app
  namespace: test
  labels:
    app: ingress-demo-app
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: app.demo.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: ingress-demo-app
            port:
              number: 80
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    name: ingress-demo-app
  name: ingress-demo-app
  namespace: test
spec:
  endpoints:
  - port: http
    path: /health
    interval: 5s
  selector:
    matchLabels:
      app: ingress-demo-app
EOF