157 lines
8.0 KiB
Markdown
157 lines
8.0 KiB
Markdown
# IO 高负载
|
||
|
||
系统如果出现 IO WAIT 高，说明 IO 设备的速度跟不上 CPU 的处理速度，CPU 需要在那里干等，这里的等待实际也占用了 CPU 时间，导致系统负载升高，可能就会影响业务进程的处理速度，导致业务超时。
|
||
|
||
## 如何判断 ？
|
||
|
||
使用 `top` 命令看下当前负载：
|
||
|
||
```text
|
||
top - 19:42:06 up 23:59, 2 users, load average: 34.64, 35.80, 35.76
|
||
Tasks: 679 total, 1 running, 678 sleeping, 0 stopped, 0 zombie
|
||
Cpu(s): 15.6%us, 1.7%sy, 0.0%ni, 74.7%id, 7.9%wa, 0.0%hi, 0.1%si, 0.0%st
|
||
Mem: 32865032k total, 30989168k used, 1875864k free, 370748k buffers
|
||
Swap: 8388604k total, 5440k used, 8383164k free, 7982424k cached
|
||
|
||
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
|
||
9783 mysql 20 0 17.3g 16g 8104 S 186.9 52.3 3752:33 mysqld
|
||
5700 nginx 20 0 1330m 66m 9496 S 8.9 0.2 0:20.82 php-fpm
|
||
6424 nginx 20 0 1330m 65m 8372 S 8.3 0.2 0:04.97 php-fpm
|
||
```
|
||
|
||
`%wa` (wait) 表示 IO WAIT 的 cpu 占用，默认看到的是所有核的平均值，要看每个核的 `%wa` 值需要按下 "1":
|
||
|
||
```text
|
||
top - 19:42:08 up 23:59, 2 users, load average: 34.64, 35.80, 35.76
|
||
Tasks: 679 total, 1 running, 678 sleeping, 0 stopped, 0 zombie
|
||
Cpu0 : 29.5%us, 3.7%sy, 0.0%ni, 48.7%id, 17.9%wa, 0.0%hi, 0.1%si, 0.0%st
|
||
Cpu1 : 29.3%us, 3.7%sy, 0.0%ni, 48.9%id, 17.9%wa, 0.0%hi, 0.1%si, 0.0%st
|
||
Cpu2 : 26.1%us, 3.1%sy, 0.0%ni, 64.4%id, 6.0%wa, 0.0%hi, 0.3%si, 0.0%st
|
||
Cpu3 : 25.9%us, 3.1%sy, 0.0%ni, 65.5%id, 5.4%wa, 0.0%hi, 0.1%si, 0.0%st
|
||
Cpu4 : 24.9%us, 3.0%sy, 0.0%ni, 66.8%id, 5.0%wa, 0.0%hi, 0.3%si, 0.0%st
|
||
Cpu5 : 24.9%us, 2.9%sy, 0.0%ni, 67.0%id, 4.8%wa, 0.0%hi, 0.3%si, 0.0%st
|
||
Cpu6 : 24.2%us, 2.7%sy, 0.0%ni, 68.3%id, 4.5%wa, 0.0%hi, 0.3%si, 0.0%st
|
||
Cpu7 : 24.3%us, 2.6%sy, 0.0%ni, 68.5%id, 4.2%wa, 0.0%hi, 0.3%si, 0.0%st
|
||
Cpu8 : 23.8%us, 2.6%sy, 0.0%ni, 69.2%id, 4.1%wa, 0.0%hi, 0.3%si, 0.0%st
|
||
Cpu9 : 23.9%us, 2.5%sy, 0.0%ni, 69.3%id, 4.0%wa, 0.0%hi, 0.3%si, 0.0%st
|
||
Cpu10 : 23.3%us, 2.4%sy, 0.0%ni, 68.7%id, 5.6%wa, 0.0%hi, 0.0%si, 0.0%st
|
||
Cpu11 : 23.3%us, 2.4%sy, 0.0%ni, 69.2%id, 5.1%wa, 0.0%hi, 0.0%si, 0.0%st
|
||
Cpu12 : 21.8%us, 2.4%sy, 0.0%ni, 60.2%id, 15.5%wa, 0.0%hi, 0.0%si, 0.0%st
|
||
Cpu13 : 21.9%us, 2.4%sy, 0.0%ni, 60.6%id, 15.2%wa, 0.0%hi, 0.0%si, 0.0%st
|
||
Cpu14 : 21.4%us, 2.3%sy, 0.0%ni, 72.6%id, 3.7%wa, 0.0%hi, 0.0%si, 0.0%st
|
||
Cpu15 : 21.5%us, 2.2%sy, 0.0%ni, 73.2%id, 3.1%wa, 0.0%hi, 0.0%si, 0.0%st
|
||
Cpu16 : 21.2%us, 2.2%sy, 0.0%ni, 73.6%id, 3.0%wa, 0.0%hi, 0.0%si, 0.0%st
|
||
Cpu17 : 21.2%us, 2.1%sy, 0.0%ni, 73.8%id, 2.8%wa, 0.0%hi, 0.0%si, 0.0%st
|
||
Cpu18 : 20.9%us, 2.1%sy, 0.0%ni, 74.1%id, 2.9%wa, 0.0%hi, 0.0%si, 0.0%st
|
||
Cpu19 : 21.0%us, 2.1%sy, 0.0%ni, 74.4%id, 2.5%wa, 0.0%hi, 0.0%si, 0.0%st
|
||
Cpu20 : 20.7%us, 2.0%sy, 0.0%ni, 73.8%id, 3.4%wa, 0.0%hi, 0.0%si, 0.0%st
|
||
Cpu21 : 20.8%us, 2.0%sy, 0.0%ni, 73.9%id, 3.2%wa, 0.0%hi, 0.0%si, 0.0%st
|
||
Cpu22 : 20.8%us, 2.0%sy, 0.0%ni, 74.4%id, 2.8%wa, 0.0%hi, 0.0%si, 0.0%st
|
||
Cpu23 : 20.8%us, 1.9%sy, 0.0%ni, 74.4%id, 2.8%wa, 0.0%hi, 0.0%si, 0.0%st
|
||
Mem: 32865032k total, 30209248k used, 2655784k free, 370748k buffers
|
||
Swap: 8388604k total, 5440k used, 8383164k free, 7986552k cached
|
||
```
|
||
|
||
`wa` 通常是 0%，如果经常在 1% 之上，说明存储设备的速度已经太慢，无法跟上 cpu 的处理速度。
|
||
|
||
## 如何排查 ？
|
||
|
||
### 使用 iostat 检查设备是否 hang 住
|
||
|
||
```bash
|
||
iostat -xhd 2
|
||
```
|
||
|
||
如果有 100% 的 `%util` 的设备，说明该设备基本 hang 住了
|
||
|
||
|
||
|
||
### 观察高 IO 的磁盘读写情况
|
||
|
||
```bash
|
||
# 捕获 %util 超过 90 时 vdb 盘的读写指标，每秒检查一次
|
||
while true; do iostat -xhd | grep -A1 vdb | grep -v vdb | awk '{if ($NF > 90){print $0}}'; sleep 1s; done
|
||
```
|
||
|
||
如果读写流量或 IOPS 不高，但 `%util` 不高，通常是磁盘本身有问题了，需要检查下磁盘。 在云上托管的 k8s 集群通常就使用的云厂商的云盘(比如腾讯云CBS)，可以拿到磁盘 ID 反馈下。
|
||
|
||
如果读写流量或 IOPS 高，继续下面的步骤排查出哪些进程导致的 IO 高负载。
|
||
|
||
### 查看哪些进程占住磁盘
|
||
|
||
```bash
|
||
fuser -v -m /dev/vdb
|
||
```
|
||
|
||
### 查找 D 状态的进程
|
||
|
||
D 状态 (Disk Sleep) 表示进程正在等待 IO，不可中断，正常情况下不会保持太久，如果进程长时间处于 D 状态，通常是设备故障
|
||
|
||
```bash
|
||
ps -eo pid,ppid,stat,command
|
||
|
||
## 捕获 D 状态的进程
|
||
while true; do ps -eo pid,ppid,stat,command | awk '{if ($3 ~ /D/) {print $0}}'; sleep 0.5s; done
|
||
```
|
||
|
||
### 观察高 IO 进程
|
||
|
||
```bash
|
||
iotop -oP
|
||
# 展示 I/O 统计，每秒更新一次
|
||
pidstat -d 1
|
||
# 只看某个进程
|
||
pidstat -d 1 -p 3394470
|
||
```
|
||
|
||
## 使用 pidstat 统计
|
||
|
||
```bash
|
||
timeout 10 pidstat -dl 3 > io.txt
|
||
cat io.txt | awk '{if ($6>2000||$5>2000)print $0}'
|
||
```
|
||
|
||
### 使用 ebpf 抓高 IOPS 进程
|
||
|
||
安装 bcc-tools:
|
||
```bash
|
||
yum install -y bcc-tools
|
||
```
|
||
|
||
分析:
|
||
```bash
|
||
$ cd /usr/share/bcc/tools
|
||
$ ./biosnoop 5 > io.txt
|
||
$ cat io.txt | awk '{print $3,$2,$4,$5}' | sort | uniq -c | sort -rn | head -10
|
||
6850 3356537 containerd vdb R
|
||
1294 3926934 containerd vdb R
|
||
864 1670 xfsaild/vdb vdb W
|
||
578 3953662 kworker/u180:1 vda W
|
||
496 3540267 logsys_cfg_cli vdb R
|
||
459 1670 xfsaild/vdb vdb R
|
||
354 3285936 php-fpm vdb R
|
||
340 3285934 php-fpm vdb R
|
||
292 2952592 sap1001 vdb R
|
||
273 324710 python vdb R
|
||
$ pstree -apnhs 3356537
|
||
systemd,1 --switched-root --system --deserialize 22
|
||
└─containerd,3895
|
||
└─{containerd},3356537
|
||
$ timeout 10 strace -fp 3895 > strace.txt 2>&1
|
||
# vdb 的 IOPS 高，vdb 挂载到了 /data 目录，这里过滤下 "/data"
|
||
$ grep "/data" strace.txt | tail -10
|
||
[pid 19562] newfstatat(AT_FDCWD, "/data/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/6974/fs/data/log/monitor/snaps/20211010/ps-2338.log", {st_mode=S_IFREG|0644, st_size=6509, ...}, AT_SYMLINK_NOFOLLOW) = 0
|
||
[pid 19562] newfstatat(AT_FDCWD, "/data/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/6974/fs/data/log/monitor/snaps/20211010/ps-2339.log", {st_mode=S_IFREG|0644, st_size=6402, ...}, AT_SYMLINK_NOFOLLOW) = 0
|
||
[pid 19562] newfstatat(AT_FDCWD, "/data/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/6974/fs/data/log/monitor/snaps/20211010/ps-2340.log", {st_mode=S_IFREG|0644, st_size=6509, ...}, AT_SYMLINK_NOFOLLOW) = 0
|
||
[pid 19562] newfstatat(AT_FDCWD, "/data/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/6974/fs/data/log/monitor/snaps/20211010/ps-2341.log", {st_mode=S_IFREG|0644, st_size=6509, ...}, AT_SYMLINK_NOFOLLOW) = 0
|
||
[pid 19562] newfstatat(AT_FDCWD, "/data/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/6974/fs/data/log/monitor/snaps/20211010/ps-2342.log", {st_mode=S_IFREG|0644, st_size=6970, ...}, AT_SYMLINK_NOFOLLOW) = 0
|
||
[pid 19562] newfstatat(AT_FDCWD, "/data/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/6974/fs/data/log/monitor/snaps/20211010/ps-2343.log", {st_mode=S_IFREG|0644, st_size=6509, ...}, AT_SYMLINK_NOFOLLOW) = 0
|
||
[pid 19562] newfstatat(AT_FDCWD, "/data/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/6974/fs/data/log/monitor/snaps/20211010/ps-2344.log", {st_mode=S_IFREG|0644, st_size=6402, ...}, AT_SYMLINK_NOFOLLOW) = 0
|
||
[pid 19562] newfstatat(AT_FDCWD, "/data/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/6974/fs/data/log/monitor/snaps/20211010/ps-2345.log", <unfinished ...>
|
||
[pid 19562] newfstatat(AT_FDCWD, "/data/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/6974/fs/data/log/monitor/snaps/20211010/ps-2346.log", {st_mode=S_IFREG|0644, st_size=7756, ...}, AT_SYMLINK_NOFOLLOW) = 0
|
||
[pid 19562] newfstatat(AT_FDCWD, "/data/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/6974/fs/data/log/monitor/snaps/20211010/ps-2347.log", Process 3895 detached
|
||
$ grep "/data" strace.txt > data.txt
|
||
# 合并且排序，自行用脚本分析下哪些文件操作多
|
||
$ cat data.txt | awk -F '"' '{print $2}' | sort | uniq -c | sort -n > data-sorted.txt
|
||
```
|