Prometheus 监控进程

Process-exporter

　　process-exporter可以用来检测所选进程的存活状态

下载process-exporter

　　下载地址：https://github.com/ncabatoff/process-exporter/releases/tag/v0.4.0　　

安装部署process-exporter

tar -zxvf process-exporter-0.4.0.linux-amd64.tar.gz -C /usr/local/process-exporter

　　编写配置文件

配置文件根据变量名匹配到配置文件

• {{.Comm}} 包含原始可执行文件的基本名称，即 /proc//stat

• {{.ExeBase}} 包含可执行文件的基本名称

• {{.ExeFull}} 包含可执行文件的标准路径

• {{.Username}} 包含有效用户的用户名

cat process-exporter.yaml
process_names:
  
  - name: "{{.Matches}}"
    cmdline:
    - 'dbbakup'
  
  - name: "{{.Matches}}"
    cmdline:
    - 'mysql'
  
  - name: "{{.Matches}}"
    cmdline:
    - 'pushgateway'

注 cmdline: 所选进程的唯一标识，ps -ef 可以查询到。如果改进程不存在，则不会有该进程的数据采集到。

　　编写启动脚本

cat /usr/lib/systemd/system/process_exporter.service
[Unit]
Description=Prometheus exporter for processors metrics, written in Go with pluggable metric collectors.
Documentation=https://github.com/ncabatoff/process-exporter
After=network.target
  
[Service]
Type=simple
User=root
WorkingDirectory=/usr/local/process-exporter
ExecStart=/usr/local/process-exporter/process-exporter -config.path=/usr/local/process-exporter/process-exporter.yaml
Restart=on-failure
  
[Install]
WantedBy=multi-user.target

启动process-exporter

systemctl daemon-reload
systemctl start process_exporter
systemctl enable process_exporter

验证

curl localhost:9256/metrics

修改Prometheus配置文件

增加
 
 - job_name: 'process'
    static_configs:
    - targets: ['172.16.8.187:9256']

　　重启Prometheus

添加告警规则

cat process.yml 
groups:
- name: process
  rules:
  - alert: backup-mysql-Down
    expr: absent(namedprocess_namegroup_states{groupname="map[:mysql]"})
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: backup slave mysql docker process Down (instance {{ $labels.instance }})
      description: "backup slave mysql process is down\n LABELS = {{ $labels }}"
 
 
  - alert: pushgateway-Down
    expr: absent(namedprocess_namegroup_states{groupname="map[:pushgateway]"})
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: pushgateway process Down (instance {{ $labels.instance }})
      description: "pushgateway process is down\n LABELS = {{ $labels }}"
 
  - alert: dbback-process-Down
    expr: absent(namedprocess_namegroup_states{groupname="map[:dbbakup]"})
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: dbbackup docker process Down (instance {{ $labels.instance }})
      description: "dbbackup docker process is down\n LABELS = {{ $labels }}"