2019-06-09

使用 Prometheus Operator 监控 Kubernetes Etcd

描述：

上次已经配置了 Prometheus Operator 监控 Kubernetes，但是并没有监控 ETCD 的信息，所以这里我们配置一下监控 Etcd 信息。

系统环境：

Prometheus Operator版本： 0.29 (Prometheus Operator更名为Kube-prometheus，且版本变为：0.1.0)
Kubernetes 版本： 1.14.0

一、查看 Etcd 信息

获取 Etcd Pod 名称

1
$ kubectl get pods -n kube-system | grep etcd
2
etcd-k8s-master-2-11                      1/1     Running   9          55d

查看 Etcd 描述信息

1
$ kubectl describe pod etcd-k8s-master-2-11 -n kube-system
2

3
......
4
Containers:
5
    Command:
6
      etcd
7
      --advertise-client-urls=https://192.168.2.11:2379
8
      --cert-file=/etc/kubernetes/pki/etcd/server.crt
9
      --client-cert-auth=true
10
      --data-dir=/var/lib/etcd
11
      --initial-advertise-peer-urls=https://192.168.2.11:2380
12
      --initial-cluster=k8s-master-2-11=https://192.168.2.11:2380
13
      --key-file=/etc/kubernetes/pki/etcd/server.key
14
      --listen-client-urls=https://127.0.0.1:2379,https://192.168.2.11:2379
15
      --listen-peer-urls=https://192.168.2.11:2380
16
      --name=k8s-master-2-11
17
      --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
18
      --peer-client-cert-auth=true
19
      --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
20
      --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
21
      --snapshot-count=10000
22
      --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
23
......

可以看到 ETCD 的证书文件在 Kubernetes Master 节点的 “/etc/kubernetes/pki/etcd/” 文件夹下。

二、将证书存入 Kubernetes

利用 kubectl 命令将三个证书文件存入 Kubernetes 的 Secret 资源下。

1
$ kubectl create secret generic etcd-certs --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.crt --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.key --from-file=/etc/kubernetes/pki/etcd/ca.crt -n monitoring

查看刚刚创建的资源

1
$ kubectl get secret etcd-certs -n monitoring
2

3
NAME         TYPE     DATA   AGE
4
etcd-certs   Opaque   3      1m

三、将证书挂入 Prometheus

编译 prometheus 资源

1
$ kubectl edit prometheus k8s -n monitoring

将证书挂入

1
apiVersion: monitoring.coreos.com/v1
2
kind: Prometheus
3
metadata:
4
  annotations:
5
    kubectl.kubernetes.io/last-applied-configuration: |
6
      {"apiVersion":"monitoring.coreos.com/v1","kind":"Prometheus","metadata":{"annotations":{},"labels":{"prometheus":"k8s"},"name":"k8s","namespace":"monitoring"},"spec":{"alerting":{"alertmanagers":[{"name":"alertmanager-main","namespace":"monitoring","port":"web"}]},"baseImage":"quay-mirror.qiniu.com/prometheus/prometheus","nodeSelector":{"beta.kubernetes.io/os":"linux"},"replicas":2,"resources":{"requests":{"memory":"400Mi"}},"ruleSelector":{"matchLabels":{"prometheus":"k8s","role":"alert-rules"}},"securityContext":{"fsGroup":2000,"runAsNonRoot":true,"runAsUser":1000},"serviceAccountName":"prometheus-k8s","serviceMonitorNamespaceSelector":{},"serviceMonitorSelector":{},"storage":{"volumeClaimTemplate":{"spec":{"resources":{"requests":{"storage":"8Gi"}},"storageClassName":"fast"}}},"version":"v2.7.2"}}
7
  creationTimestamp: "2019-06-07T22:15:37Z"
8
  generation: 5
9
  labels:
10
    prometheus: k8s
11
  name: k8s
12
  namespace: monitoring
13
  resourceVersion: "2128109"
14
  selfLink: /apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheuses/k8s
15
  uid: c6daa0a1-8971-11e9-bc01-000c29d98697
16
spec:
17
  alerting:
18
    alertmanagers:
19
    - name: alertmanager-main
20
      namespace: monitoring
21
      port: web
22
  baseImage: quay-mirror.qiniu.com/prometheus/prometheus
23
  nodeSelector:
24
    beta.kubernetes.io/os: linux
25
  replicas: 2
26
  resources:
27
    requests:
28
      memory: 400Mi
29
  ruleSelector:
30
    matchLabels:
31
      prometheus: k8s
32
      role: alert-rules
33
  secrets:                  #------新增证书配置，将etcd证书挂入
34
  - etcd-certs

更新完成后就可以在 Prometheus Pod 中看到上面挂入的 etcd 证书，我们可以进入 Pod 中查看：

1
$ kubectl exec -it prometheus-k8s-0 /bin/sh -n monitoring
2

3
/prometheus $ ls /etc/prometheus/secrets/etcd-certs/
4
ca.crt                  healthcheck-client.crt  healthcheck-client.key

四、创建 Etcd Service & Endpoints

因为 ETCD 是独立于集群之外的，所以我们需要创建一个 Endpoints 将其代理到 Kubernetes 集群，然后创建一个 Service 绑定 Endpoints，然后 Kubernetes 集群的应用就可以访问 ETCD 了。

etcd-service.yaml

1
apiVersion: v1
2
kind: Service
3
metadata:
4
  name: etcd-k8s
5
  namespace: kube-system
6
  labels:
7
    k8s-app: etcd
8
spec:
9
  type: ClusterIP
10
  clusterIP: None       #设置为None，不分配Service IP
11
  ports:
12
  - name: port
13
    port: 2379
14
    protocol: TCP
15
---
16
apiVersion: v1
17
kind: Endpoints
18
metadata:
19
  name: etcd-k8s
20
  namespace: kube-system
21
  labels:
22
    k8s-app: etcd
23
subsets:
24
- addresses:
25
  - ip: 192.168.2.11    #Etcd 所在节点的IP
26
  ports:
27
  - port: 2379          #Etcd 端口号

创建 Service & Endpoints

1
$ kubectl apply -f etcd-service.yaml

五、创建 ServiceMonitor

创建 Prometheus 监控资源，配置用于监控 Etcd 参数。

etcd-monitor.yaml

1
apiVersion: monitoring.coreos.com/v1
2
kind: ServiceMonitor
3
metadata:
4
  name: etcd-k8s
5
  namespace: monitoring
6
  labels:
7
    k8s-app: etcd-k8s
8
spec:
9
  jobLabel: k8s-app
10
  endpoints:
11
  - port: port
12
    interval: 30s
13
    scheme: https
14
    tlsConfig:
15
      caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
16
      certFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.crt
17
      keyFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.key
18
      insecureSkipVerify: true
19
  selector:
20
    matchLabels:
21
      k8s-app: etcd
22
  namespaceSelector:
23
    matchNames:
24
    - kube-system