当前位置: 代码迷 >> 综合 >> prometheus 配置热更新以及更改副本数高可用
  详细解决方案

prometheus 配置热更新以及更改副本数高可用

热度:57   发布时间:2023-09-30 11:54:35.0

prometheus热更新


为了每次修改配置文件可以热加载prometheus,也就是不停止prometheus,就可以使配置生效,修改了prometheus-cfg.yaml文件中prometheus的配置,通过kubectl apply -f prometheus-cfg.yaml和kubectl apply -f prometheus-deploy.yaml更新资源清单文件,想要使配置生效可用如下热加载命令:

curl -X POST http://10.244.1.66:9090/-/reload

10.244.1.66是prometheus的pod的ip地址,如何查看prometheus的pod ip,可用如下命令:

kubectl get pods -n monitor-sa -o wide | grep prometheus

显示如下, 10.244.1.7就是prometheus的ip

[root@master prometheus]# vim prometheus-cfg.yaml 
[root@master prometheus]# kubectl apply -f prometheus-cfg.yaml 
configmap/prometheus-config configured[root@master ~]# kubectl get pod -o wide -n monitor
NAME                                 READY   STATUS    RESTARTS   AGE   IP              NODE     NOMINATED NODE   READINESS GATES
grafana-757fcd5f7c-mgbjg             1/1     Running   9          19d   10.233.96.104   node2    <none>           <none>
node-exporter-bl92s                  1/1     Running   11         21d   192.168.100.7   node2    <none>           <none>
node-exporter-dgzlt                  1/1     Running   11         21d   192.168.100.6   node1    <none>           <none>
node-exporter-lrt46                  1/1     Running   11         21d   192.168.100.5   master   <none>           <none>
prometheus-server-7fb65555b9-bbh4j   1/1     Running   1          10h   10.233.90.79    node1    <none>           <none>[root@master ~]# curl -X POST http://10.233.90.79:9090/-/reload
You have new mail in /var/spool/mail/root[root@master ~]# kubectl logs prometheus-server-7fb65555b9-bbh4j -n monitor -f
level=info ts=2021-11-01T13:01:24.926892426Z caller=main.go:588 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2021-11-01T13:01:24.929509257Z caller=kubernetes.go:191 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2021-11-01T13:01:24.930321253Z caller=kubernetes.go:191 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2021-11-01T13:01:24.931003091Z caller=kubernetes.go:191 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"
level=info ts=2021-11-01T13:01:24.93168627Z caller=kubernetes.go:191 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"

 热加载速度比较慢,可以暴力重启prometheus,如修改上面的prometheus-cfg.yaml文件之后,可执行如下强制删除:

kubectl delete -f prometheus-cfg.yaml
kubectl delete -f prometheus-deploy.yaml

 然后再通过apply更新:

kubectl apply -f prometheus-cfg.yaml
kubectl apply -f prometheus-deploy.yaml

注意:

线上最好热加载,暴力删除可能造成监控数据的丢失

更改副本数


[root@master prometheus]# kubectl get pod -n monitor
NAME                                 READY   STATUS    RESTARTS   AGE
grafana-757fcd5f7c-mgbjg             1/1     Running   11         20d
node-exporter-bl92s                  1/1     Running   13         22d
node-exporter-dgzlt                  1/1     Running   13         22d
node-exporter-lrt46                  1/1     Running   13         22d
prometheus-server-7fb65555b9-bbh4j   1/1     Running   3          36h
prometheus-server-7fb65555b9-jd62q   1/1     Running   0          26s[root@master prometheus]# kubectl get ep -n monitor
NAME         ENDPOINTS                              AGE
grafana      10.233.96.123:3000                     20d
prometheus   10.233.90.93:9090,10.233.96.126:9090   36h

这两个pod不管访问哪个,访问到的数据都是一样的,这样就可以保证普罗米修斯的高可用了。

[root@master prometheus]# kubectl get pvc -n monitor
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
grafana      Bound    pvc-828870bc-2a40-4332-ab5a-9d62eb2c6112   5Gi        RWX            managed-nfs-storage   20d
prometheus   Bound    pvc-13175f30-cad6-4bc3-ad48-3ee6d170db3e   5Gi        RWX            managed-nfs-storage   19d

然后你更新配置之后,可以删除两个pod当作的其中一个,这样还是可以访问的,因为做了高可用,另外一个还可以提供服务。 这样就可以一个一个pod删除达到更新配置的效果。(中间要保证新起来的pod处于ready状态再将老的pod给删除掉)