K8S prometheus operator监控工作原理介绍_云虚拟主机

1. prometheus operator介绍

介绍文章：

2016年年末，coreos引入了operator 模式，并发布了prometheus operator 作为operator模式的工作示例。prometheus operator自动创建和管理prometheus监控实例。

prometheus operator的任务是使得在kubernetes运行prometheus仅可能容易，同时保留可配置性以及使kubernetes配置原生。

prometheus operator使我们的生活更容易——部署和维护。

2. 它如何工作

为了理解这个问题，我们首先需要了解prometheus operator得工作原理。

prometheus operator架构图.

我们成功部署 prometheus operator后可以看到一个新的crds(custom resource defination):

prometheus，定义一个期望的prometheus deployment。
servicemonitor，声明式指定应该如何监控服务组；operator根据定义自动创建prometheusscrape配置。
alertmanager，定义期望的alertmanager deployment。

当服务新版本更新时，将会常见一个新pod。prometheus监控k8s api，因此当它检测到这种变化时，它将为这个新服务(pod)创建一组新的配置。

3. servicemonitor

prometheus operator使用一个crd，叫做 servicemonitor 将配置抽象到目标。
下面是个servicemonitor的示例：

apiversion: monitoring.coreos.com/v1alpha1
kind: servicemonitor
metadata:
  name: frontend
  labels:
    tier: frontend
spec:
  selector:
    matchlabels:
      tier: frontend
  endpoints:
  - port: web            # 指定exporter端口，这里指定的是endpoint的名称
    interval: 10s        # 刷新间隔时间

这仅仅是定义一组服务应该如何被监控。现在我们需要定义一个包含了该servicemonitor的prometheus实例到其配置：

apiversion: monitoring.coreos.com/v1alpha1
kind: prometheus
metadata:
  name: prometheus-frontend
  labels:
    prometheus: frontend
spec:
  version: v1.3.0
  #定义应包括标签为“tier=frontend”的所有servicemonitor 到服务器的配置中
  servicemonitors:
  - selector:
      matchlabels:
        tier: frontend

现在prometheus将会监控每个带有tier: frontend label的服务。

4. helm安装

先决条件：

部署了helm

准备好动手操作：

 helm repo add coreos https://s3-eu-west-1.amazonaws.com/coreos-charts/stable/
 helm install coreos/prometheus-operator --name prometheus-operator --namespace monitoring

到目前为止，我们已经在我们的集群中安装了prometheus operator的tpr。
现在我们来部署prometheus，alertmanager和grafana。

tip: 当我使用一个庞大的helm charts时，我更倾向于创建一个独立的value.yaml文件将包含我所有自定义的变更。这么做使我和同事为后期的变化和修改更容易。

helm install coreos/kube-prometheus --name kube-prometheus   \
       -f my_changes/prometheus.yaml                           \
       -f my_changes/grafana.yaml                              \
       -f my_changes/alertmanager.yaml

检查一切是否运行正常

 kubectl -n monitoring get po
name                                                   ready     status    restarts   age
alertmanager-kube-prometheus-0                         2/2       running   0          1h
kube-prometheus-exporter-kube-state-68dbb4f7c9-tr6rp   2/2       running   0          1h
kube-prometheus-exporter-node-bqcj4                    1/1       running   0          1h
kube-prometheus-exporter-node-jmcq2                    1/1       running   0          1h
kube-prometheus-exporter-node-qnzsn                    1/1       running   0          1h
kube-prometheus-exporter-node-v4wn8                    1/1       running   0          1h
kube-prometheus-exporter-node-x5226                    1/1       running   0          1h
kube-prometheus-exporter-node-z996c                    1/1       running   0          1h
kube-prometheus-grafana-54c96ffc77-tjl6g               2/2       running   0          1h
prometheus-kube-prometheus-0                           2/2       running   0          1h
prometheus-operator-1591343780-5vb5q                   1/1       running   0          1h

访问下prometheus ui看一下targets页面：

 kubectl -n monitoring port-forward prometheus-kube-prometheus-0 9090
forwarding from 127.0.0.1:9090 -> 9090

浏览器展示如下：

5. yaml文件安装

此安装方法本人亲测有效，用到的yaml文件都打包好了。解压之后直接kubectl apply即可用。会自动监控当前集群的所有node节点和pod。只需更改yaml文件中需要用到的镜像。我这里都推到了公司公网harbor仓库。部分镜像已经打成tar包。直接docker load -i即可用。

kube-state.tar.gz

webhook-dingtalk.tar.gz

prometheus-adapter.tar.gz

5.1 安装

#软件包集成了node exporter alertmanager grafana prometheus ingress 所有服务的配置，只需解压到k8s master中。
[root@lecode-k8s-master monitor]# ll
total 1820
-rw-r--r-- 1 root root     875 mar 11  2022 alertmanager-alertmanager.yaml
-rw-r--r-- 1 root root     515 mar 11  2022 alertmanager-poddisruptionbudget.yaml
-rw-r--r-- 1 root root    4337 mar 11  2022 alertmanager-prometheusrule.yaml
-rw-r--r-- 1 root root    1483 mar 14  2022 alertmanager-secret.yaml
-rw-r--r-- 1 root root     301 mar 11  2022 alertmanager-serviceaccount.yaml
-rw-r--r-- 1 root root     540 mar 11  2022 alertmanager-servicemonitor.yaml
-rw-r--r-- 1 root root     614 mar 11  2022 alertmanager-service.yaml
drwxr-x--- 2 root root    4096 oct 25 13:49 backsvc #这里是grafana的service配置。nodeport模式。用于外部访问。选择使用
-rw-r--r-- 1 root root     278 mar 11  2022 blackbox-exporter-clusterrolebinding.yaml
-rw-r--r-- 1 root root     287 mar 11  2022 blackbox-exporter-clusterrole.yaml
-rw-r--r-- 1 root root    1392 mar 11  2022 blackbox-exporter-configuration.yaml
-rw-r--r-- 1 root root    3081 mar 11  2022 blackbox-exporter-deployment.yaml
-rw-r--r-- 1 root root      96 mar 11  2022 blackbox-exporter-serviceaccount.yaml
-rw-r--r-- 1 root root     680 mar 11  2022 blackbox-exporter-servicemonitor.yaml
-rw-r--r-- 1 root root     540 mar 11  2022 blackbox-exporter-service.yaml
-rw-r--r-- 1 root root    2521 oct 25 13:36 dingtalk-dep.yaml
-rw-r--r-- 1 root root     721 mar 11  2022 grafana-dashboarddatasources.yaml
-rw-r--r-- 1 root root 1448347 mar 11  2022 grafana-dashboarddefinitions.yaml
-rw-r--r-- 1 root root     625 mar 11  2022 grafana-dashboardsources.yaml
-rw-r--r-- 1 root root    8098 mar 11  2022 grafana-deployment.yaml
-rw-r--r-- 1 root root      86 mar 11  2022 grafana-serviceaccount.yaml
-rw-r--r-- 1 root root     398 mar 11  2022 grafana-servicemonitor.yaml
-rw-r--r-- 1 root root     468 mar 30  2022 grafana-service.yaml
drwxr-xr-x 2 root root    4096 oct 25 13:32 ingress #这里ingress资源也是可以直接用，可以把prometheus和grafana服务暴露在外部。
-rw-r--r-- 1 root root    2639 mar 14  2022 kube-prometheus-prometheusrule.yaml
-rw-r--r-- 1 root root    3380 mar 14  2022 kube-prometheus-prometheusrule.yamlbak
-rw-r--r-- 1 root root   63531 mar 11  2022 kubernetes-prometheusrule.yaml
-rw-r--r-- 1 root root    6912 mar 11  2022 kubernetes-servicemonitorapiserver.yaml
-rw-r--r-- 1 root root     425 mar 11  2022 kubernetes-servicemonitorcoredns.yaml
-rw-r--r-- 1 root root    6431 mar 11  2022 kubernetes-servicemonitorkubecontrollermanager.yaml
-rw-r--r-- 1 root root    7629 mar 11  2022 kubernetes-servicemonitorkubelet.yaml
-rw-r--r-- 1 root root     530 mar 11  2022 kubernetes-servicemonitorkubescheduler.yaml
-rw-r--r-- 1 root root     464 mar 11  2022 kube-state-metrics-clusterrolebinding.yaml
-rw-r--r-- 1 root root    1712 mar 11  2022 kube-state-metrics-clusterrole.yaml
-rw-r--r-- 1 root root    2934 oct 25 13:40 kube-state-metrics-deployment.yaml
-rw-r--r-- 1 root root    3082 mar 11  2022 kube-state-metrics-prometheusrule.yaml
-rw-r--r-- 1 root root     280 mar 11  2022 kube-state-metrics-serviceaccount.yaml
-rw-r--r-- 1 root root    1011 mar 11  2022 kube-state-metrics-servicemonitor.yaml
-rw-r--r-- 1 root root     580 mar 11  2022 kube-state-metrics-service.yaml
-rw-r--r-- 1 root root     444 mar 11  2022 node-exporter-clusterrolebinding.yaml
-rw-r--r-- 1 root root     461 mar 11  2022 node-exporter-clusterrole.yaml
-rw-r--r-- 1 root root    3047 mar 11  2022 node-exporter-daemonset.yaml
-rw-r--r-- 1 root root   14356 apr 11  2022 node-exporter-prometheusrule.yaml
-rw-r--r-- 1 root root     270 mar 11  2022 node-exporter-serviceaccount.yaml
-rw-r--r-- 1 root root     850 mar 11  2022 node-exporter-servicemonitor.yaml
-rw-r--r-- 1 root root     492 mar 11  2022 node-exporter-service.yaml
-rw-r--r-- 1 root root     482 mar 11  2022 prometheus-adapter-apiservice.yaml
-rw-r--r-- 1 root root     576 mar 11  2022 prometheus-adapter-clusterroleaggregatedmetricsreader.yaml
-rw-r--r-- 1 root root     494 mar 11  2022 prometheus-adapter-clusterrolebindingdelegator.yaml
-rw-r--r-- 1 root root     471 mar 11  2022 prometheus-adapter-clusterrolebinding.yaml
-rw-r--r-- 1 root root     378 mar 11  2022 prometheus-adapter-clusterroleserverresources.yaml
-rw-r--r-- 1 root root     409 mar 11  2022 prometheus-adapter-clusterrole.yaml
-rw-r--r-- 1 root root    2204 mar 11  2022 prometheus-adapter-configmap.yaml
-rw-r--r-- 1 root root    2530 oct 25 13:39 prometheus-adapter-deployment.yaml
-rw-r--r-- 1 root root     506 mar 11  2022 prometheus-adapter-poddisruptionbudget.yaml
-rw-r--r-- 1 root root     515 mar 11  2022 prometheus-adapter-rolebindingauthreader.yaml
-rw-r--r-- 1 root root     287 mar 11  2022 prometheus-adapter-serviceaccount.yaml
-rw-r--r-- 1 root root     677 mar 11  2022 prometheus-adapter-servicemonitor.yaml
-rw-r--r-- 1 root root     501 mar 11  2022 prometheus-adapter-service.yaml
-rw-r--r-- 1 root root     447 mar 11  2022 prometheus-clusterrolebinding.yaml
-rw-r--r-- 1 root root     394 mar 11  2022 prometheus-clusterrole.yaml
-rw-r--r-- 1 root root    5000 mar 11  2022 prometheus-operator-prometheusrule.yaml
-rw-r--r-- 1 root root     715 mar 11  2022 prometheus-operator-servicemonitor.yaml
-rw-r--r-- 1 root root     499 mar 11  2022 prometheus-poddisruptionbudget.yaml
-rw-r--r-- 1 root root   14021 mar 11  2022 prometheus-prometheusrule.yaml
-rw-r--r-- 1 root root    1184 mar 11  2022 prometheus-prometheus.yaml
-rw-r--r-- 1 root root     471 mar 11  2022 prometheus-rolebindingconfig.yaml
-rw-r--r-- 1 root root    1547 mar 11  2022 prometheus-rolebindingspecificnamespaces.yaml
-rw-r--r-- 1 root root     366 mar 11  2022 prometheus-roleconfig.yaml
-rw-r--r-- 1 root root    2047 mar 11  2022 prometheus-rolespecificnamespaces.yaml
-rw-r--r-- 1 root root     271 mar 11  2022 prometheus-serviceaccount.yaml
-rw-r--r-- 1 root root     531 mar 11  2022 prometheus-servicemonitor.yaml
-rw-r--r-- 1 root root     558 mar 11  2022 prometheus-service.yaml
drw-r--r-- 2 root root    4096 oct 24 12:31 setup
#先apply setup目录中的yaml文件。然后apply一级目录下的yaml文件。backsvc中的grafana的service资源清单。根据情况调整为nodeport或clusterip。k8s集群会自动在每台k8s节点部署node-exporter并收集数据。登录grafana后初始账号密码为admin admin。添加dashboard即可监控k8s集群
[root@lecode-k8s-master monitor]# cd setup/
[root@lecode-k8s-master setup]#   kubectl apply -f .
[root@lecode-k8s-master setup]# cd ..
[root@lecode-k8s-master monitor]# kubectl apply -f .
[root@lecode-k8s-master monitor]# kubectl get po -n monitoring 
name                                   ready   status    restarts   age
alertmanager-main-0                    2/2     running   0          74m
alertmanager-main-1                    2/2     running   0          74m
alertmanager-main-2                    2/2     running   0          74m
blackbox-exporter-6798fb5bb4-d9m7m     3/3     running   0          74m
grafana-64668d8465-x7x9z               1/1     running   0          74m
kube-state-metrics-569d89897b-hlqxj    3/3     running   0          57m
node-exporter-6vqxg                    2/2     running   0          74m
node-exporter-7dxh6                    2/2     running   0          74m
node-exporter-9j5xk                    2/2     running   0          74m
node-exporter-ftrmn                    2/2     running   0          74m
node-exporter-qszkn                    2/2     running   0          74m
node-exporter-wjkgj                    2/2     running   0          74m
prometheus-adapter-5dd78c75c6-h2jf7    1/1     running   0          58m
prometheus-adapter-5dd78c75c6-qpwzv    1/1     running   0          58m
prometheus-k8s-0                       2/2     running   0          74m
prometheus-k8s-1                       2/2     running   0          74m
prometheus-operator-75d9b475d9-mmzgs   2/2     running   0          80m
webhook-dingtalk-6ffc94b49-z9z6l       1/1     running   0          61m
[root@lecode-k8s-master backsvc]# kubectl get svc -n monitoring 
name                    type        cluster-ip      external-ip   port(s)                      age
alertmanager-main       nodeport    10.98.35.93     <none>        9093:30093/tcp               72m
alertmanager-operated   clusterip   none            <none>        9093/tcp,9094/tcp,9094/udp   72m
blackbox-exporter       clusterip   10.109.10.110   <none>        9115/tcp,19115/tcp           72m
grafana                 nodeport    10.110.48.214   <none>        3000:30300/tcp               72m
kube-state-metrics      clusterip   none            <none>        8443/tcp,9443/tcp            72m
node-exporter           clusterip   none            <none>        9100/tcp                     72m
prometheus-adapter      clusterip   10.97.23.176    <none>        443/tcp                      72m
prometheus-k8s          clusterip   10.100.92.254   <none>        9090/tcp                     72m
prometheus-operated     clusterip   none            <none>        9090/tcp                     72m
prometheus-operator     clusterip   none            <none>        8443/tcp                     78m
webhook-dingtalk        clusterip   10.100.131.63   <none>        80/tcp                       72m

5.2 访问服务

暴露服务三种方法：用service资源的nodeport模式，或者用k8s的ingress暴露服务或者本地nginx代理。本地的nginx代理模式

这里我grafana用的是nodeport模式。prometheus用的是nginx代理。

附上nginx配置文件

[root@lecode-k8s-master setup]# cat /usr/local/nginx/conf/4-layer-conf.d/lecode-prometheus-operator.conf 
#代理prometheus内置dashboard ui
upstream prometheus-dashboard {
    server 10.100.92.254:9090; #这里ip为prometheus-k8s svc资源的ip
}
server {
    listen  9090;
    proxy_pass prometheus-dashboard;
}
#代理grafana
upstream grafana {
    server 10.1.82.89:3000; #这里ip为grafana svc资源的ip
}
server {
    listen  3000;
    proxy_pass grafana;
}

访问prometheus targets

5.3 接入grafana

访问grafana（默认密码是admin admin）

去grafana官网下载对应dashboard 地址：https://grafana.com/grafana/dashboards/

6. 监控集群外的服务

6.1 exporter安装

在对应服务的本地安装对应的exporter用于收集数据（这里以mysql为例）

#下载对应服务的exporter  
#插件下载地址：https://www.modb.pro/db/216588
#插件下载地址：https://prometheus.io/download/
#下载完成后解压mysqld_exporter-0.13.0.linux-amd64.tar.gz
#配置mysql-exporter
在root路径下，创建.my.cnf文件。内容如下:
[root@lecode-test-001 ~]# cat /root/.my.cnf 
[client]
user=mysql_monitor
password=mysql@123
#创建mysql 用户并授权
create user ‘mysql_monitor'@‘localhost' identified by ‘mysql@123' with max_user_connections 3;
grant process, replication client, select on . to ‘mysql_monitor'@‘localhost';
flush privileges;
exit
#启动mysqld_exporter
[root@lecode-test-001 mysql-exporter]# nohup mysqld_exporter &
#找到对应的端口
[root@lecode-test-001 mysql-exporter]# tail -f nohup.out 
level=info ts=2022-10-25t09:26:54.464z caller=mysqld_exporter.go:277 msg="starting msqyld_exporter" version="(version=0.13.0, branch=head, revision=ad2847c7fa67b9debafccd5a08bacb12fc9031f1)"
level=info ts=2022-10-25t09:26:54.464z caller=mysqld_exporter.go:278 msg="build context" (gogo1.16.4,userroot@e2043849cb1f,date20210531-07:30:16)=(missing)
level=info ts=2022-10-25t09:26:54.464z caller=mysqld_exporter.go:293 msg="scraper enabled" scraper=global_status
level=info ts=2022-10-25t09:26:54.464z caller=mysqld_exporter.go:293 msg="scraper enabled" scraper=global_variables
level=info ts=2022-10-25t09:26:54.464z caller=mysqld_exporter.go:293 msg="scraper enabled" scraper=slave_status
level=info ts=2022-10-25t09:26:54.464z caller=mysqld_exporter.go:293 msg="scraper enabled" scraper=info_schema.innodb_cmp
level=info ts=2022-10-25t09:26:54.464z caller=mysqld_exporter.go:293 msg="scraper enabled" scraper=info_schema.innodb_cmpmem
level=info ts=2022-10-25t09:26:54.464z caller=mysqld_exporter.go:293 msg="scraper enabled" scraper=info_schema.query_response_time
level=info ts=2022-10-25t09:26:54.464z caller=mysqld_exporter.go:303 msg="listening on address" address=:9104 #这是exporter的端口
level=info ts=2022-10-25t09:26:54.464z caller=tls_config.go:191 msg="tls is disabled." http2=false
#检查端口
[root@lecode-test-001 mysql-exporter]# ss -lntup |grep 9104
tcp    listen     0      128      :::9104                 :::*                   users:(("mysqld_exporter",pid=26115,fd=3))

6.2 k8s配置

创建endpoint资源关联对应服务主机的exporter端口。绑定service资源，通过servicemonitor资源添加prometheus targets，

1）官方格式

 kubectl -n monitoring get prometheus kube-prometheus -o yaml
apiversion: monitoring.coreos.com/v1
kind: prometheus
metadata:
  labels:
    app: prometheus
    chart: prometheus-0.0.14
    heritage: tiller
    prometheus: kube-prometheus
    release: kube-prometheus
  name: kube-prometheus
  namespace: monitoring
spec:
  ...
  baseimage: quay.io/prometheus/prometheus
  servicemonitorselector:
    matchlabels:
      prometheus: kube-prometheus 
#接下来就是按照格式创建对应的servicemonitor资源

通过ep资源把外部服务关联到k8s内部，绑定对应的svc资源。在由servicemonitor绑定对应的svc资源把数据提交给prometheus，servicemonitor通过标签选择器关联service，而service只需要通过端口关联ep，这里的标签和端口一定要注意一致.

2) 创建资源

[root@lecode-k8s-master monitor]# cat mysql.yaml 
apiversion: v1
kind: endpoints
metadata:
  name: mysql-test
  namespace: monitoring
subsets:
- addresses:
  - ip: 192.168.1.17  # ip为安装应用服务器的ip
  ports:
  - name: mysql
    port: 9104 #export的端口
    protocol: tcp
---
apiversion: v1
kind: service
metadata:
  labels:
    app.kubernetes.io/component: mysql
    app.kubernetes.io/name: mysql
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: mysql-test
  namespace: monitoring
spec:
  clusterip: none
  clusterips:
  - none
  ports:
  - name: mysql
    port: 9104
    protocol: tcp
  sessionaffinity: none
  type: clusterip
---
apiversion: monitoring.coreos.com/v1
kind: servicemonitor
metadata:
  labels:
    app.kubernetes.io/component: mysql
    app.kubernetes.io/name: mysql
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: mysql-test
  namespace: monitoring
spec:
  endpoints:
  - bearertokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: mysql
    tlsconfig:
      insecureskipverify: true
  selector:
    matchlabels:
      app.kubernetes.io/component: mysql
      app.kubernetes.io/name: mysql
      app.kubernetes.io/part-of: kube-prometheus
#创建        
[root@lecode-k8s-master monitor]# kubectl apply -f mysql.yaml 
endpoints/mysql-test created
service/mysql-test created
servicemonitor.monitoring.coreos.com/mysql-test created
#检查        
[root@lecode-k8s-master monitor]# kubectl get -f  mysql.yaml 
name                   endpoints           age
endpoints/mysql-test   192.168.1.17:9104   10m
name                 type        cluster-ip   external-ip   port(s)    age
service/mysql-test   clusterip   none         <none>        9104/tcp   10m
name                                              age
servicemonitor.monitoring.coreos.com/mysql-test   10m

6.3 接入grafana

7 监控集群外的服务-redis

7.1 exporter安装

#部署redis-exporter
#插件下载地址：https://www.modb.pro/db/216588
[root@lecode-test-001 ~]# tar xf redis_exporter-v1.3.2.linux-amd64.tar.gz 
[root@lecode-test-001 ~]# ll
drwxr-xr-x  2 root root     4096 nov  6  2019 redis_exporter-v1.3.2.linux-amd64
-rw-r--r--  1 root root  3376155 oct 27 10:26 redis_exporter-v1.3.2.linux-amd64.tar.gz
[root@lecode-test-001 ~]# mv redis_exporter-v1.3.2.linux-amd64 redis_exporter
[root@lecode-test-001 ~]# cd redis_exporter
[root@lecode-test-001 redis_exporter]# ll
total 8488
-rw-r--r-- 1 root root    1063 nov  6  2019 license
-rw-r--r-- 1 root root   10284 nov  6  2019 readme.md
-rwxr-xr-x 1 root root 8675328 nov  6  2019 redis_exporter
[root@lecode-test-001 redis_exporter]# nohup ./redis_exporter -redis.addr  192.168.1.17:6379  -redis.password  'redislecodetest@shuli123' & 
[1] 4564
[root@lecode-test-001 redis_exporter]# nohup: ignoring input and appending output to â€˜nohup.outâ€™
[root@lecode-test-001 redis_exporter]# tail -f nohup.out 
time="2022-10-27t10:26:48+08:00" level=info msg="redis metrics exporter v1.3.2    build date: 2019-11-06-02:25:20    sha1: 175a69f33e8267e0a0ba47caab488db5e83a592e    go: go1.13.4    goos: linux    goarch: amd64"
time="2022-10-27t10:26:48+08:00" level=info msg="providing metrics at :9121/metrics"
#端口为9121

7.2 k8s配置

#创建redis-servicemonitor资源
[root@lecode-k8s-master monitor]# cat redis.yaml 
apiversion: v1
kind: endpoints
metadata:
  name: redis-test
  namespace: monitoring
subsets:
- addresses:
  - ip: 192.168.1.17 # ip为安装应用服务器的ip
  ports:
  - name: redis
    port: 9121 #exporter端口
    protocol: tcp
---
apiversion: v1
kind: service
metadata:
  labels:
    app.kubernetes.io/component: redis
    app.kubernetes.io/name: redis
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: redis-test
  namespace: monitoring
spec:
  clusterip: none
  clusterips:
  - none
  ports:
  - name: redis
    port: 9121
    protocol: tcp
  sessionaffinity: none
  type: clusterip
---
apiversion: monitoring.coreos.com/v1
kind: servicemonitor
metadata:
  labels:
    app.kubernetes.io/component: redis
    app.kubernetes.io/name: redis
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: redis-test
  namespace: monitoring
spec:
  endpoints:
  - bearertokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: redis
    tlsconfig:
      insecureskipverify: true
  selector:
    matchlabels:
      app.kubernetes.io/component: redis
      app.kubernetes.io/name: redis
      app.kubernetes.io/part-of: kube-prometheus
#创建资源
[root@lecode-k8s-master monitor]# kubectl apply -f redis.yaml 
endpoints/redis-test created
service/redis-test created
servicemonitor.monitoring.coreos.com/redis-test created
[root@lecode-k8s-master monitor]#  kubectl get ep,svc,servicemonitor -n monitoring  |grep redis
endpoints/redis-test              192.168.1.17:9121                                                  6m2s
service/redis-test              clusterip   none            <none>        9121/tcp                     6m2s
servicemonitor.monitoring.coreos.com/redis-test                6m2s

7.3 prometheus检查

7.4 接入grafana

dashboard id： 11835

8. 监控集群外的服务-kafka

8.1 安装exporter

#插件下载地址：https://www.modb.pro/db/216588
[root@lecode-test-001 ~]# tar xf kafka_exporter-1.4.2.linux-amd64.tar.gz 
[root@lecode-test-001 ~]# ll
drwxrwxr-x  2 2000 2000     4096 sep 16  2021 kafka_exporter-1.4.2.linux-amd64
-rw-r--r--  1 root root  8499720 oct 27 15:30 kafka_exporter-1.4.2.linux-amd64.tar.gz
[root@lecode-test-001 ~]# mv kafka_exporter-1.4.2.linux-amd64 kafka_exporter
[root@lecode-test-001 ~]# cd kafka_exporter
[root@lecode-test-001 kafka_exporter]# ll
total 17676
-rwxr-xr-x 1 2000 2000 18086208 sep 16  2021 kafka_exporter
-rw-rw-r-- 1 2000 2000    11357 sep 16  2021 license
[root@lecode-test-001 kafka_exporter]# nohup ./kafka_exporter --kafka.server=192.168.1.17:9092 &
[1] 20777
[root@lecode-test-001 kafka_exporter]# nohup: ignoring input and appending output to ‘nohup.out'
[root@lecode-test-001 kafka_exporter]# tail -f nohup.out 
i1027 15:32:38.904075   20777 kafka_exporter.go:769] starting kafka_exporter (version=1.4.2, branch=head, revision=0d5d4ac4ba63948748cc2c53b35ed95c310cd6f2)
i1027 15:32:38.905515   20777 kafka_exporter.go:929] listening on http :9308
#exporter端口为9308

8.2 k8s配置

[root@lecode-k8s-master monitor]# cat kafka.yaml 
apiversion: v1
kind: endpoints
metadata:
  name: kafka-test
  namespace: monitoring
subsets:
- addresses:
  - ip: 192.168.1.17  # ip为安装应用服务器的ip
  ports:
  - name: kafka
    port: 9308 #export的端口
    protocol: tcp
---
apiversion: v1
kind: service
metadata:
  labels:
    app.kubernetes.io/component: kafka
    app.kubernetes.io/name: kafka
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: kafka-test
  namespace: monitoring
spec:
  clusterip: none
  clusterips:
  - none
  ports:
  - name: kafka
    port: 9308
    protocol: tcp
  sessionaffinity: none
  type: clusterip
---
apiversion: monitoring.coreos.com/v1
kind: servicemonitor
metadata:
  labels:
    app.kubernetes.io/component: kafka
    app.kubernetes.io/name: kafka
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: kafka-test
  namespace: monitoring
spec:
  endpoints:
  - bearertokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: kafka
    tlsconfig:
      insecureskipverify: true
  selector:
    matchlabels:
      app.kubernetes.io/component: kafka
      app.kubernetes.io/name: kafka
      app.kubernetes.io/part-of: kube-prometheus
#创建
[root@lecode-k8s-master monitor]# kubectl apply -f kafka.yaml 
endpoints/kafka-test created
service/kafka-test created
servicemonitor.monitoring.coreos.com/kafka-test created
[root@lecode-k8s-master monitor]# kubectl get -f kafka.yaml 
name                   endpoints           age
endpoints/kafka-test   192.168.1.17:9308   8m49s
name                 type        cluster-ip   external-ip   port(s)    age
service/kafka-test   clusterip   none         <none>        9308/tcp   8m49s
name                                              age
servicemonitor.monitoring.coreos.com/kafka-test   8m48s

8.3 prometheus检查

8.4 接入grafana

dashboard id：7589

9. 监控集群外的服务-zookeeper

9.1 安装exporter

exporter下载地址：https://github.com/carlpett/zookeeper_exporter/releases/download/v1.0.2/zookeeper_exporter
[root@lecode-test-001 zookeeper_exporter]# nohup ./zookeeper_exporter -zookeeper 192.168.1.17:2181 -bind-addr :9143 &
[2] 8310
[root@lecode-test-001 zookeeper_exporter]# nohup: ignoring input and appending output to ‘nohup.out'
[root@lecode-test-001 zookeeper_exporter]# tail -f nohup.out 
time="2022-10-27t15:58:27+08:00" level=info msg="zookeeper_exporter, version v1.0.2 (branch: head, revision: d6e929223f6b3bf5ff25dd0340e8194cbd4d04fc)\n  build user:       @bd731f434d23\n  build date:       2018-05-01t20:40:14+0000\n  go version:       go1.10.1"
time="2022-10-27t15:58:27+08:00" level=info msg="starting zookeeper_exporter"
time="2022-10-27t15:58:27+08:00" level=info msg="starting metric http endpoint on :9143"
#exporter端口为9143

9.2 k8s配置

[root@lecode-k8s-master monitor]# cat zookeeper.yaml 
apiversion: v1
kind: endpoints
metadata:
  name: zookeeper-test
  namespace: monitoring
subsets:
- addresses:
  - ip: 192.168.1.17
  ports:
  - name: zookeeper
    port: 9143
    protocol: tcp
---
apiversion: v1
kind: service
metadata:
  labels:
    app.kubernetes.io/component: zookeeper
    app.kubernetes.io/name: zookeeper
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: zookeeper-test
  namespace: monitoring
spec:
  clusterip: none
  clusterips:
  - none
  ports:
  - name: zookeeper
    port: 9143
    protocol: tcp
  sessionaffinity: none
  type: clusterip
---
apiversion: monitoring.coreos.com/v1
kind: servicemonitor
metadata:
  labels:
    app.kubernetes.io/component: zookeeper
    app.kubernetes.io/name: zookeeper
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: zookeeper-test
  namespace: monitoring
spec:
  endpoints:
  - bearertokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: zookeeper
    tlsconfig:
      insecureskipverify: true
  selector:
    matchlabels:
      app.kubernetes.io/component: zookeeper
      app.kubernetes.io/name: zookeeper
      app.kubernetes.io/part-of: kube-prometheus
#创建
[root@lecode-k8s-master monitor]# kubectl apply -f zookeeper.yaml 
endpoints/zookeeper-test created
service/zookeeper-test created
servicemonitor.monitoring.coreos.com/zookeeper-test created
[root@lecode-k8s-master monitor]# kubectl get -f  zookeeper.yaml 
name                       endpoints           age
endpoints/zookeeper-test   192.168.1.17:9143   9m55s
name                     type        cluster-ip   external-ip   port(s)    age
service/zookeeper-test   clusterip   none         <none>        9143/tcp   9m55s
name                                                  age
servicemonitor.monitoring.coreos.com/zookeeper-test   9m55s

9.3 prometheus检查

9.4 接入grafana

dashboard id：15026

以上就是k8s prometheus operator监控工作原理介绍的详细内容，更多关于k8s prometheus operator监控的资料请关注代码网其它相关文章！

K8S prometheus operator监控工作原理介绍

2024年05月19日 • 云虚拟主机 •我要评论