k8s孤儿Pod

1,539次阅读

共计 3121 个字符，预计需要花费 8 分钟才能阅读完成。

今天的云挺饱满的，可惜下起了太阳雨……..

k8s孤儿Pod

最近博主在清理pod时，看到kubelet日志一直在刷一种日志

2023 08 03101:41:27.670582591Z W0803 01:41:27 6/0427 3388387 kubelet getters.go:297] Path "/var/lib/kubelet/pods/937d53d9-e49d-4cd4-975c-56b252a826ae/volumes" does not exist 
2023 08 03T01:41:27.670588854Z W0803 01:41:27.670448 3388387 kubelet getters.go:297] Path "/var/lib/kubelet/pods/ab5c894a-aa6c-490e-970a-15f419ae4dd9/volumes" does not exist 
2023 08 03T01:41:27.6705930947 W0803 01:41:27.670456 3388387 kubelet getters.go:297] Path "/var/ib/kubelet/pods/ab5c894a-aa6c-490e -970a-15f419ae4dd9/volumes" does not exist 
2023 08 03T01:41:27.670597180Z W0803 01:41:27.670477 3388387 kubelet getters.go:297] Path "/var/lib/kubelet/pods/dbf4672d-00cf-4bc7-9bdf-6fe8e5e82fb4/volumes" does not exist 
2023 08 03T01:41:27.670601213Z W0803 01:41:27.670485 3388387 kubelet getters.go:297] Path "/ar/ib/kubelet/pods/dbf4672d-00cf-4bc7-9bdf-6fe8e5e82fb4/volumes" does not exist There were a total of 3 errors similar to this. Turn up verbosity to see them.
2023 08 03T01:41:27.6706053287 W0803 01:41:27.670497 3388387 kubelet volumes.go:154] orphaned pod "937d53d9-e49d-4cd4-975c-56b252a826ae" found, but volume subpaths are still present on disk:

通过日志中的Pod uid去寻找pod，已无法找到，说明pod已删除，但是目前看还有资源未释放掉，尝试手动删除

[root@XXXXX pods]# rm -rf /var/lib/kubelet/pods/035b275f-ffb2-43debaa6-64ee73e55b35
rm: cannot remove '/var/lib/kubelet/pods/035b275f-ffb2-43debaa6-64ee73e55b35/volume -subpaths/multus-cfg/kube-multus/2': Device or resource busy

grep以下mount信息看看

[root@XXXXX pods]# mount -l | grep '035b275f-ffb2-43debaa6-64ee73e55b35'
/dev/mapper/rootvg-var on /var/lib/kubelet/pods/035b275f-ffb2-43debaa6-64ee73e55b35/volume subpaths/multus-cfg/kube-multus/2 type xis (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)

看来得先umount后再删掉

[root@XXXXX pods]# umount /var/lib/kubelet/pods/035b275f-ffb2-43debaa6-64ee73e55b35
[root@XXXXX pods]# mount -l | grep 035b275f-ffb2-43debaa6-64ee73e55b35
[root@XXXXX pods]# rm -rf umount /var/lib/kubelet/pods/035b275f-ffb2-43debaa6-64ee73e55b35

此时再看kubelet日志已少一条该类型报错，那剩余的类似错误依照这个方式处理即可

google了一番，说是kubelet再回收orphaned pod时，遇到pod内还挂载volumePath或subPath就会报这个错

// cleanupOrphanedPodDirs removes the volumes of pods that should not be
// running and that have no containers running.  Note that we roll up logs here since it runs in the main loop.
func (kl *Kubelet) cleanupOrphanedPodDirs(pods []*v1.Pod, runningPods []*kubecontainer.Pod) error {
        // If there are still volume directories, do not delete directory
        volumePaths, err := kl.getPodVolumePathListFromDisk(uid)
        if err != nil {
            orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but error %v occurred during reading volume dir from disk", uid, err))
            continue
        }
        if len(volumePaths) > 0 {
            orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but volume paths are still present on disk", uid))
            continue
        }

        // If there are any volume-subpaths, do not cleanup directories
        volumeSubpathExists, err := kl.podVolumeSubpathsDirExists(uid)
        if err != nil {
            orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but error %v occurred during reading of volume-subpaths dir from disk", uid, err))
            continue
        }
        if volumeSubpathExists {
            orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("orphaned pod %q found, but volume subpaths are still present on disk", uid))
            continue
        }
}

如果是这样，那为什么pod清理时，为什么会无法释放挂载点？这下不得不了解下pod的释放流程了。。。

正文完