Prometheus-operator对RocketMQ的监控

共计 6693 个字符，预计需要花费 17 分钟才能阅读完成。

找了一圈官方文档，发现官方没有提供rocketmq-exporter的镜像，那就只好自行制作镜像了

多阶段构建exporter镜像

[root@k8s-master rocketmq-exporter]# cat >dockerfile<<-'EOF'
FROM slpcat/maven:3.6-oraclejdk-8 AS build
MAINTAINER 若虚 <slpcat@qq.com>

# Install required packages
RUN \
    apt-get update && apt-get upgrade -y && apt-get install -y git

# install from source
RUN \
    cd /opt && \
    git clone https://github.com/apache/rocketmq-exporter.git && \
    cd rocketmq-exporter

WORKDIR /opt/rocketmq-exporter

RUN \
    mvn clean install -Dmaven.test.skip=true

FROM slpcat/oraclejdk:8-bullseye

RUN \
    mkdir -p /opt/rocketmq-exporter

COPY --from=build /opt/rocketmq-exporter/target/rocketmq-exporter-0.0.2-SNAPSHOT-exec.jar /opt/rocketmq-exporter/rocketmq-exporter-0.0.2-SNAPSHOT.jar

WORKDIR /opt/rocketmq-exporter

CMD ["sh", "-c", "java $JAVA_OPTS -jar rocketmq-exporter-0.0.2-SNAPSHOT.jar"]
EOF

[root@k8s-master rocketmq-exporter]# docker build . -t rocketmq-exporter:0.0.1 

部署rocketmq-exporter

创建deploy

看了下配置，说docker制作镜像中的mvn编译时，需要提前将application.yaml中的nameserver地址写上，不然默认就是获取localhost:9876，如果这个镜像不是和我的rocketmq再同一个deploy中部署，那就会启动报错。既然为jar启动的，那我们可以在运行时添加参数，从而覆盖application.yaml配置

[root@k8s-master rocketmq-exporter]# cat >rocketmq-exporter.yaml<<-'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rocketmq-exporter
  labels:
    app: rocketmq-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rocketmq-exporter
  template:
    metadata:
      labels:
        app: rocketmq-exporter
    spec:
      containers:
      - name: rocketmq-exporter
        image: rocketmq-exporter:0.0.1
        command:
          [
            "sh",
            "-c",
            "java $JAVA_OPTS -jar rocketmq-exporter-0.0.2-SNAPSHOT.jar --rocketmq.config.namesrvAddr=mqnamesrv:9876",
          ]
        ports:
        - name: monitor-port
          containerPort: 5557
          protocol: TCP
EOF
[root@k8s-master rocketmq-exporter]# kubectl apply -f rocketmq-exporter.yaml

创建svc

[root@k8s-master rocketmq-exporter]# cat >rocketmq-exporter-svc.yaml<<-'EOF'
apiVersion: v1
kind: Service
metadata:
  name: rocketmq-exporter-svc
  namespace: default
  labels:
    app: rocketmq-exporter-svc
spec:
  type: ClusterIP
  ports:
  - name: monitor-port
    targetPort: 5557
    port: 5557
    protocol: TCP
  selector:
   app: rocketmq-exporter
EOF

[root@k8s-master rocketmq-exporter]# kubectl apply -f rocketmq-exporter-svc.yaml

测试监控数据获取

# 查看当前资源
[root@k8s-master rocketmq-exporter]# kubectl get pod
NAME                                 READY   STATUS    RESTARTS   AGE
mqbroker-0                           1/1     Running   0          3h43m
mqnamesrv-0                          1/1     Running   0          3h43m
mqui-0                               1/1     Running   0          3h43m
rocketmq-exporter-57d66c8995-dgvnt   1/1     Running   0          75m

# 获取svc地址
[root@k8s-master rocketmq-exporter]# kubectl get svc
NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
kubernetes              ClusterIP   10.96.0.1      <none>        443/TCP          31d
mqnamesrv               NodePort    10.96.42.220   <none>        9876:31816/TCP   3h41m
mqui                    NodePort    10.96.76.12    <none>        8080:30685/TCP   3h40m
rocketmq-exporter-svc   ClusterIP   10.96.64.158   <none>        5557/TCP         72m

[root@k8s-master rocketmq-exporter]# curl 10.96.64.158:5557/metrics
rocketmq_producer_count{cluster="DefaultCluster",broker="mqbroker-0.mqbroker.default.svc.cluster.local",group="CLIENT_INNER_PRODUCER",} 2.0
rocketmq_producer_offset{cluster="DefaultCluster",broker="mqbroker-0.mqbroker.default.svc.cluster.local",topic="mqbroker-0.mqbroker.default.svc.cluster.local",} 0.0
rocketmq_producer_offset{cluster="DefaultCluster",broker="mqbroker-0.mqbroker.default.svc.cluster.local",topic="OFFSET_MOVED_EVENT",} 0.0
rocketmq_producer_offset{cluster="DefaultCluster",broker="mqbroker-0.mqbroker.default.svc.cluster.local",topic="DefaultCluster_REPLY_TOPIC",} 0.0
rocketmq_producer_offset{cluster="DefaultCluster",broker="mqbroker-0.mqbroker.default.svc.cluster.local",topic="SCHEDULE_TOPIC_XXXX",} 0.0
rocketmq_producer_offset{cluster="DefaultCluster",broker="mqbroker-0.mqbroker.default.svc.cluster.local",topic="SELF_TEST_TOPIC",} 0.0
rocketmq_producer_offset{cluster="DefaultCluster",broker="mqbroker-0.mqbroker.default.svc.cluster.local",topic="DefaultCluster",} 0.0
rocketmq_producer_offset{cluster="DefaultCluster",broker="mqbroker-0.mqbroker.default.svc.cluster.local",topic="RMQ_SYS_TRANS_HALF_TOPIC",} 0.0
rocketmq_producer_offset{cluster="DefaultCluster",broker="mqbroker-0.mqbroker.default.svc.cluster.local",topic="BenchmarkTest",} 0.0
rocketmq_producer_offset{cluster="DefaultCluster",broker="mqbroker-0.mqbroker.default.svc.cluster.local",topic="TBW102",} 0.0
rocketmq_topic_retry_offset{cluster="DefaultCluster",broker="mqbroker-0.mqbroker.default.svc.cluster.local",topic="%RETRY%TOOLS_CONSUMER",} 0.0

配置prometheus

创建servicemonitor

[root@k8s-master rocketmq-exporter]# cat >rocketmq-servicemonitor.yaml<<-'EOF'
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: rocketmq-exporter
  namespace: monitoring
  labels:
    app: rocketmq-exporter
spec:
  jobLabel: rocketmq-exporter
  endpoints:
  - port: monitor-port
    interval: 30s
    scheme: http
  selector:
    matchLabels:
      app: rocketmq-exporter-svc
  namespaceSelector:
    matchNames:
    - default
EOF

[root@k8s-master rocketmq-exporter]# kubectl apply -f rocketmq-servicemonitor.yaml

查看promethes target

grafan面板配置

可以导入以下两个ID面板：

• 14612
• 10477

注意创建的mq可能没有什么数据，指标会很少，可以自行登录mqbroker中执行tools工具生成数据

# 测试生产消息
# 再tools.sh中添加变量： export NAMESRV_ADDR=mqnamesrv:9876
[root@k8s-master rocketmq-exporter]# kubectl exec -it mqbroker-0 -- /bin/bash
[rocketmq@mqbroker-0 bin]$ bash tools.sh  org.apache.rocketmq.example.quickstart.Producer
。。。。
SendResult [sendStatus=SEND_OK, msgId=7F00000102A13A71F4DD01977E7403E5, offsetMsgId=0A64EBCD00002A9F0000000000176A50, messageQueue=MessageQueue [topic=TopicTest, brokerName=mqbroker-0.mqbroker.default.svc.cluster.local, queueId=1], queueOffset=1999]
SendResult [sendStatus=SEND_OK, msgId=7F00000102A13A71F4DD01977E7603E6, offsetMsgId=0A64EBCD00002A9F0000000000176B10, messageQueue=MessageQueue [topic=TopicTest, brokerName=mqbroker-0.mqbroker.default.svc.cluster.local, queueId=2], queueOffset=1999]
SendResult [sendStatus=SEND_OK, msgId=7F00000102A13A71F4DD01977E7803E7, offsetMsgId=0A64EBCD00002A9F0000000000176BD0, messageQueue=MessageQueue [topic=TopicTest, brokerName=mqbroker-0.mqbroker.default.svc.cluster.local, queueId=3], queueOffset=1999]
07:25:05.543 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.100.235.205:10911] result: true
07:25:05.550 [NettyClientSelector_1] INFO RocketmqRemoting - closeChannel: close the connection to remote address[10.96.42.220:9876] result: true

# 测试消费消息
[rocketmq@mqbroker-0 bin]$ bash tools.sh  org.apache.rocketmq.example.quickstart.Consumer
。。。。。
ConsumeMessageThread_please_rename_unique_group_name_4_8 Receive New Messages: [MessageExt [brokerName=mqbroker-0.mqbroker.default.svc.cluster.local, queueId=0, storeSize=192, queueOffset=1835, sysFlag=0, bornTimestamp=1667287503968, bornHost=/10.100.235.205:33586, storeTimestamp=1667287503969, storeHost=/10.100.235.205:10911, msgId=0A64EBCD00002A9F0000000000157D90, commitLogOffset=1408400, bodyCRC=1743584292, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message{topic='TopicTest', flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=2000, CONSUME_START_TIME=1667287559965, UNIQ_KEY=7F00000102A13A71F4DD019778600154, CLUSTER=DefaultCluster, TAGS=TagA}, body=[72, 101, 108, 108, 111, 32, 82, 111, 99, 107, 101, 116, 77, 81, 32, 51, 52, 48], transactionId='null'}]]