docker中的文件系统

2,196次阅读
没有评论

共计 17093 个字符，预计需要花费 43 分钟才能阅读完成。

docker中的文件系统

重新温习下docker中的文件系统，因为最近有好多pod被evicted，查了kubelet日志是因磁盘满导致

联合文件系统

linux中的联合文件系统UnionFS允许合并一个或多个文件系统目录中的内容，同时保持内容物理隔离。此处主要以OverlayFS展开描述联合挂载

查看当前docker的存储驱动

xadocker@xadocker-virtual-machine:~/Desktop$ docker info --format '{{.Driver}}'   
overlay2

此时查看目录/var/lib/docker/overlay2只有一个目录l(字母l)

xadocker@xadocker-virtual-machine:~/Desktop$ sudo ls -l /var/lib/docker/overlay2
total 4
drwx------ 2 root root 4096 8月  20 12:56 l
xadocker@xadocker-virtual-machine:~/Desktop$ sudo ls -l /var/lib/docker/overlay2/l
total 0

拉个镜像下来看看变化

xadocker@xadocker-virtual-machine:~/Desktop$ sudo docker pull nginx:1.14.2
1.14.2: Pulling from library/nginx
27833a3ba0a5: Pull complete 
0f23e58bd0b7: Pull complete 
8ca774778e85: Pull complete 
Digest: sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d
Status: Downloaded newer image for nginx:1.14.2
docker.io/library/nginx:1.14.2

查看该镜像的layers信息

root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker inspect nginx:1.14.2 | jq -r '.[] | .RootFS'
{
  "Type": "layers",
  "Layers": [
    "sha256:5dacd731af1b0386ead06c8b1feff9f65d9e0bdfec032d2cd0bc03690698feda",
    "sha256:b8f18c3b860b067be09836beadd676a0aa1e784ec28cf730986859b4146c344a",
    "sha256:82ae01d5004e2143b642b1a008624e7521c73ab18e5776a22f18a172b9dbec80"
  ]
}

可以看到layers有三层，而此时的/var/lib/docker/overlay2目录存在以下内容

root@xadocker-virtual-machine:/var/lib/docker/overlay2# tree -L 2
.
├── 0ee4b77a1e8f6c374e4974182bf850036680b3bccecbc05d4d206373cd943a33
│   ├── committed
│   ├── diff
│   └── link
├── 43c54c31282405e68e577d9edcfc572e695cd71aabc6193123e9676da14970d9
│   ├── committed
│   ├── diff
│   ├── link
│   ├── lower
│   └── work
├── 441adc17b9c104687e6d9ea8224b8da74489542a303ae64c78e9e9243194777e
│   ├── diff
│   ├── link
│   ├── lower
│   └── work
└── l
    ├── GHJM6TNXCEPVQCFQSKRJ7YHQXH -> ../43c54c31282405e68e577d9edcfc572e695cd71aabc6193123e9676da14970d9/diff
    ├── OU3ZI5MET46OMFNFWPEEHCW47X -> ../0ee4b77a1e8f6c374e4974182bf850036680b3bccecbc05d4d206373cd943a33/diff
    └── U4BVMECOSCTSWIUQ5WWS4TY7MN -> ../441adc17b9c104687e6d9ea8224b8da74489542a303ae64c78e9e9243194777e/diff

其中没有lower的则为镜像的基层，因为lower记录着其父级层信息内容

root@xadocker-virtual-machine:/var/lib/docker/overlay2# cat 43c54c31282405e68e577d9edcfc572e695cd71aabc6193123e9676da14970d9/lower 
l/OU3ZI5MET46OMFNFWPEEHCW47X

而diff内容则是代表当前层的内容

# 基层diff目录
root@xadocker-virtual-machine:/var/lib/docker/overlay2# ls 0ee4b77a1e8f6c374e4974182bf850036680b3bccecbc05d4d206373cd943a33/diff/
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

# 非基层diff目录
root@xadocker-virtual-machine:/var/lib/docker/overlay2# ls 43c54c31282405e68e577d9edcfc572e695cd71aabc6193123e9676da14970d9/diff/
etc  lib  tmp  usr  var

此时我们看下镜像的GraphDriver信息

root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker inspect nginx:1.14.2 | jq -r '.[] | .GraphDriver'
{
  "Data": {
    "LowerDir": "/var/lib/docker/overlay2/43c54c31282405e68e577d9edcfc572e695cd71aabc6193123e9676da14970d9/diff:/var/lib/docker/overlay2/0ee4b77a1e8f6c374e4974182bf850036680b3bccecbc05d4d206373cd943a33/diff",
    "MergedDir": "/var/lib/docker/overlay2/441adc17b9c104687e6d9ea8224b8da74489542a303ae64c78e9e9243194777e/merged",
    "UpperDir": "/var/lib/docker/overlay2/441adc17b9c104687e6d9ea8224b8da74489542a303ae64c78e9e9243194777e/diff",
    "WorkDir": "/var/lib/docker/overlay2/441adc17b9c104687e6d9ea8224b8da74489542a303ae64c78e9e9243194777e/work"
  },
  "Name": "overlay2"
}

从上面信息可以看到有以下4个目录

  • LowerDir：包含镜像层的除当前层（最上层）diff
  • UpperDir：指向当前镜像当前层（最上层）diff
  • WorkDir：官方说供OverlayFS内部使用
  • MergedDir：包含所有层的联合挂载内容

但是从前面的目录信息来看，并没有发现有MergedDir，因为该层是容器运行后的对LowerDir和UpperDir的联合挂载点，目前没启动容器所以没有

root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker create  --name test-nginx nginx:1.14.2 
a1b587358364ee26e11b52e30bcc9febc89c89612cb6ded972c5d6a6f0a16384
root@xadocker-virtual-machine:/var/lib/docker/overlay2# tree -L 2
.
├── 0ee4b77a1e8f6c374e4974182bf850036680b3bccecbc05d4d206373cd943a33
│   ├── committed
│   ├── diff
│   └── link
├── 43c54c31282405e68e577d9edcfc572e695cd71aabc6193123e9676da14970d9
│   ├── committed
│   ├── diff
│   ├── link
│   ├── lower
│   └── work
├── 441adc17b9c104687e6d9ea8224b8da74489542a303ae64c78e9e9243194777e
│   ├── committed
│   ├── diff
│   ├── link
│   ├── lower
│   └── work
├── 541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada
│   ├── diff
│   ├── link
│   ├── lower
│   └── work
├── 541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada-init
│   ├── committed
│   ├── diff
│   ├── link
│   ├── lower
│   └── work
└── l
    ├── A42HGWMUAQXXX7PJCXEY4BA46H -> ../541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada-init/diff
    ├── GHJM6TNXCEPVQCFQSKRJ7YHQXH -> ../43c54c31282405e68e577d9edcfc572e695cd71aabc6193123e9676da14970d9/diff
    ├── HOHSUFEKJXNVVCJSZWJSAYRDS2 -> ../541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada/diff
    ├── OU3ZI5MET46OMFNFWPEEHCW47X -> ../0ee4b77a1e8f6c374e4974182bf850036680b3bccecbc05d4d206373cd943a33/diff
    └── U4BVMECOSCTSWIUQ5WWS4TY7MN -> ../441adc17b9c104687e6d9ea8224b8da74489542a303ae64c78e9e9243194777e/diff

20 directories, 13 files


root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker inspect test-nginx | jq -r '.[]| .GraphDriver'
{
  "Data": {
    "LowerDir": "/var/lib/docker/overlay2/541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada-init/diff:/var/lib/docker/overlay2/441adc17b9c104687e6d9ea8224b8da74489542a303ae64c78e9e9243194777e/diff:/var/lib/docker/overlay2/43c54c31282405e68e577d9edcfc572e695cd71aabc6193123e9676da14970d9/diff:/var/lib/docker/overlay2/0ee4b77a1e8f6c374e4974182bf850036680b3bccecbc05d4d206373cd943a33/diff",
    "MergedDir": "/var/lib/docker/overlay2/541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada/merged",
    "UpperDir": "/var/lib/docker/overlay2/541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada/diff",
    "WorkDir": "/var/lib/docker/overlay2/541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada/work"
  },
  "Name": "overlay2"
}

root@xadocker-virtual-machine:/var/lib/docker/overlay2# ll 541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada-init/diff/etc/
total 8
drwxr-xr-x 2 root root 4096 8月  20 13:52 ./
drwxr-xr-x 4 root root 4096 8月  20 13:52 ../
-rwxr-xr-x 1 root root    0 8月  20 13:52 hostname*
-rwxr-xr-x 1 root root    0 8月  20 13:52 hosts*
lrwxrwxrwx 1 root root   12 8月  20 13:52 mtab -> /proc/mounts
-rwxr-xr-x 1 root root    0 8月  20 13:52 resolv.conf*

# hostname、hosts、resolv.conf都为空的内容
root@xadocker-virtual-machine:/var/lib/docker/overlay2# cat 541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada-init/diff/etc/hostname 
root@xadocker-virtual-machine:/var/lib/docker/overlay2# cat 541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada-init/diff/etc/hosts
root@xadocker-virtual-machine:/var/lib/docker/overlay2# cat 541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada-init/diff/etc/resolv.conf

从上面可以看到create容器时会生成两个目录，其中一个带init后缀，搜索一通后说该层只是用来初始化一些配置的如hostname,dns解析等，当我们用命令去配置hostname和dns解析时就是在该层去操作配置，其实就是引入中间init层来做初始化配置的修改

root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker create --name nginx-test2-xad --hostname nginx-test2-hostname --add-host nginx.xadocker.cn:192.168.44.168 --dns 223.5.5.5 nginx:1.14.2

但是通过上面命令创建该容器后但其init目录里的文件信息还是空的，inspect后发现hosts/hostname/resolv.conf却在/var/lib/docker/container内，暂未发现这层的关系，先埋个坑咯~

此时再创建一个容器并查看信息

root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker create  --name test-nginx-2 nginx:1.14.2 
4ca89f3e1b0ace54dcde604eadbf5a430fd5d02e64f1d6f056f261b90ba5c349

root@xadocker-virtual-machine:/var/lib/docker/overlay2# ls -lrt
total 32
drwx--x--- 3 root root 4096 8月  20 12:58 0ee4b77a1e8f6c374e4974182bf850036680b3bccecbc05d4d206373cd943a33
drwx--x--- 4 root root 4096 8月  20 12:58 43c54c31282405e68e577d9edcfc572e695cd71aabc6193123e9676da14970d9
drwx--x--- 4 root root 4096 8月  20 13:52 441adc17b9c104687e6d9ea8224b8da74489542a303ae64c78e9e9243194777e
drwx--x--- 4 root root 4096 8月  20 13:52 541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada-init
drwx--x--- 4 root root 4096 8月  20 13:52 541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada
drwx------ 2 root root 4096 8月  20 14:07 l
drwx--x--- 4 root root 4096 8月  20 14:07 3a757b7e83a308c49c22c2b493f40ed393ae015fc6b6fd7404ff74026a1e03d4-init
drwx--x--- 4 root root 4096 8月  20 14:07 3a757b7e83a308c49c22c2b493f40ed393ae015fc6b6fd7404ff74026a1e03d4


root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker inspect test-nginx-2 | jq -r '.[]| .GraphDriver'
{
  "Data": {
    "LowerDir": "/var/lib/docker/overlay2/3a757b7e83a308c49c22c2b493f40ed393ae015fc6b6fd7404ff74026a1e03d4-init/diff:/var/lib/docker/overlay2/441adc17b9c104687e6d9ea8224b8da74489542a303ae64c78e9e9243194777e/diff:/var/lib/docker/overlay2/43c54c31282405e68e577d9edcfc572e695cd71aabc6193123e9676da14970d9/diff:/var/lib/docker/overlay2/0ee4b77a1e8f6c374e4974182bf850036680b3bccecbc05d4d206373cd943a33/diff",
    "MergedDir": "/var/lib/docker/overlay2/3a757b7e83a308c49c22c2b493f40ed393ae015fc6b6fd7404ff74026a1e03d4/merged",
    "UpperDir": "/var/lib/docker/overlay2/3a757b7e83a308c49c22c2b493f40ed393ae015fc6b6fd7404ff74026a1e03d4/diff",
    "WorkDir": "/var/lib/docker/overlay2/3a757b7e83a308c49c22c2b493f40ed393ae015fc6b6fd7404ff74026a1e03d4/work"
  },
  "Name": "overlay2"
}

可以看到此时create的容器test-nginx和test-nginx-2是复用了镜像层，这样可以实现节约磁盘的使用，现在尝试将test-nginx启动看看

root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker ps -a
CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS    PORTS     NAMES
4ca89f3e1b0a   nginx:1.14.2   "nginx -g 'daemon of…"   4 minutes ago    Created             test-nginx-2
a1b587358364   nginx:1.14.2   "nginx -g 'daemon of…"   19 minutes ago   Created             test-nginx
root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker start test-nginx
test-nginx
root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker ps -a
CONTAINER ID   IMAGE          COMMAND                  CREATED          STATUS        PORTS     NAMES
4ca89f3e1b0a   nginx:1.14.2   "nginx -g 'daemon of…"   4 minutes ago    Created                 test-nginx-2
a1b587358364   nginx:1.14.2   "nginx -g 'daemon of…"   19 minutes ago   Up 1 second   80/tcp    test-nginx

此时再查看该容器目录就会有一个merged目录了，该目录内容和我们进入到容器里的看到的内容一致

root@xadocker-virtual-machine:/var/lib/docker/overlay2# tree 541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada -L 1
541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada
├── diff
├── link
├── lower
├── merged
└── work

3 directories, 2 files

# 查看挂载信息
root@xadocker-virtual-machine:/var/lib/docker/overlay2# mount | grep over
overlay on /var/lib/docker/overlay2/541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/A42HGWMUAQXXX7PJCXEY4BA46H:/var/lib/docker/overlay2/l/U4BVMECOSCTSWIUQ5WWS4TY7MN:/var/lib/docker/overlay2/l/GHJM6TNXCEPVQCFQSKRJ7YHQXH:/var/lib/docker/overlay2/l/OU3ZI5MET46OMFNFWPEEHCW47X,upperdir=/var/lib/docker/overlay2/541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada/diff,workdir=/var/lib/docker/overlay2/541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada/work)


# 该目录和我们进入到容器里查看的目录
root@xadocker-virtual-machine:/var/lib/docker/overlay2# ls 541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada/merged/
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker exec -it test-nginx ls
bin  boot  dev	etc  home  lib	lib64  media  mnt  opt	proc  root  run  sbin  srv  sys  tmp  usr  var

root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker exec -it test-nginx cat /etc/hostname
a1b587358364
root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker exec -it test-nginx cat /etc/hosts
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.17.0.2	a1b587358364
root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker exec -it test-nginx cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 192.168.44.2
search localdomain

但是当我们停止容器时，merged目录就会消失，并不代表着内容丢失，因为diff内会记录着该层的变化，merged只是联合挂载后内容

root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker exec -it test-nginx bash
root@a1b587358364:/# ls /tmp/
root@a1b587358364:/# echo hello >> /tmp/test.txt
root@a1b587358364:/# exit
exit

root@xadocker-virtual-machine:/var/lib/docker/overlay2# docker stop test-nginx
test-nginx
root@xadocker-virtual-machine:/var/lib/docker/overlay2# ls 541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada
diff link lower work
root@xadocker-virtual-machine:/var/lib/docker/overlay2# cat 541c581d71c636420217bcc4b7a699a445c924de01fe7adf4dc40e68a1806ada/diff/tmp/test.txt 
hello

容器是如何读写overlay2的呢？

docker中的文件系统

读取文件

考虑容器打开文件以通过覆盖进行读取访问的三种场景。

  • 该文件不存在于容器层中：如果容器打开一个文件进行读取访问，并且该文件尚不存在于容器 ( upperdir) 中，则会从映像 ( lowerdir) 中读取该文件。这会产生非常小的性能开销
  • 文件只存在于容器层：如果容器打开一个文件进行读访问，并且该文件存在于容器（upperdir）中而不是镜像（lowerdir）中，则直接从容器中读取
  • 文件同时存在于容器层和镜像层：如果容器打开一个文件进行读访问，并且该文件存在于镜像层和容器层，则读取该文件在容器层的版本。容器层（upperdir）中的文件掩盖了镜像层中的同名文件lowerdir

修改文件或目录

  • 首次写入文件：容器首次写入现有文件时，该文件在容器中不存在 ( upperdir)。驱动程序 overlay2执行copy_up操作以将文件从映像 ( lowerdir) 复制到容器 ( upperdir)。然后，容器将更改写入容器层中文件的新副本。然而，OverlayFS 在文件级别而不是块级别工作。这意味着所有 OverlayFS copy_up 操作都会复制整个文件，即使文件非常大并且只修改了其中的一小部分。这会对容器写入性能产生显着影响。然而，有两件事值得注意：
    • copy_up 操作仅在第一次写入给定文件时发生。对同一文件的后续写入将针对已复制到容器的文件副本进行操作
    • OverlayFS 适用于多个层。这意味着在具有多层的图像中搜索文件时，性能可能会受到影响
  • 删除文件和目录：
    • 当删除容器内的文件时，会在容器中创建一个空白upperdir文件 。图像层中的文件版本lowerdir不会被删除（因为它lowerdir是只读的）。但是，whiteout 文件会阻止容器使用它。
    • 当删除容器内的目录时，会在容器内创建一个不透明目录upperdir。这与whiteout文件的工作方式相同，并且有效地防止目录被访问，即使它仍然存在于图像中( lowerdir)
  • 重命名目录：rename只有源路径和目标路径都在顶层时才允许调用目录。否则，它将返回EXDEV错误（“不允许跨设备链接”）。您的应用程序需要设计为处理EXDEV和回退到“复制和取消链接”策略

统计POD中容器的增量文件大小

从上面的一些简单说明可以了解到upperdir即为容器的增量内容，所以我们可以分别统计下k8s pod中看看谁占的多

[root@master-192-168-31-100 workdir]# cat get-docker-upperdir.sh 
#!/bin/bash
docker inspect `docker ps -q` | jq -r '.[] | [.Config.Labels["io.kubernetes.pod.namespace"],.Config.Labels["io.kubernetes.pod.name"],.Config.Labels["io.kubernetes.container.name"], .GraphDriver.Data.UpperDir] | @tsv' | \
while read line
do
    echo -n $line" " 
    echo $line | awk '{print $4}' | xargs du -sk | awk '{print $1}' 
done


# 执行
[root@master-192-168-31-100 workdir]# bash get-docker-upperdir.sh | column -t
tigera-operator   tigera-operator-66f6445597-7h2q9               tigera-operator          /var/lib/docker/overlay2/721b82301cd351ad27a8d0c00ee32dd3ebdbcfc2cfe0ef695b11902356e650e5/diff  0
calico-apiserver  calico-apiserver-868f4c547b-t9tkw              calico-apiserver         /var/lib/docker/overlay2/8178c649017efb057411f174ba7ba395f37cc4025269c64a2f8347ea6a061062/diff  0
calico-apiserver  calico-apiserver-868f4c547b-t9tkw              POD                      /var/lib/docker/overlay2/639f27b12febc2b1652db14174128ecea6daaa08ccc843bb65970d0e65eb1cb8/diff  0
kube-system       coredns-7d75679df-prvsb                        coredns                  /var/lib/docker/overlay2/b324209e3fdaaabb20df39f9ff2502c1b7c1926c51b942240ae4cecb684f88c7/diff  0
calico-system     calico-kube-controllers-77b99b58ff-jfkk7       calico-kube-controllers  /var/lib/docker/overlay2/954ff02124ea07d5d4134e34bd84cd26f168e49ac3875f161aa08861a35f128f/diff  4

其实到这里我们才拿到POD容器里的upperdir增量内容，但还有一部分存储是在POD里的emptydir或者hostpath内，这种类型的volume也是霍霍宿主机硬盘的帮凶

对于emptydir，会在pod的目录中生成并挂在到容器里，pod目录在/var/lib/kubelet/pods/中对应的POD ID里

[root@master-192-168-31-100 workdir]# ll /var/lib/kubelet/pods/
总用量 0
drwxr-x--- 5 root root 71 8月  25 22:49 087f6929-223f-4605-b6ef-5afa8719fc78
drwxr-x--- 5 root root 71 8月  25 23:19 182e216a-3608-45c3-8813-b2fd71b99d68
drwxr-x--- 5 root root 71 8月  25 22:49 225b47362d283e421361cb0d30e95081
drwxr-x--- 5 root root 71 8月  25 23:20 3df101f9-636f-4df4-bd59-4c77811babe3
drwxr-x--- 5 root root 71 8月  25 23:20 4e8507c3-6f07-4e49-83bb-9c97292cdb77

# 一个demo测试样例
[root@master-192-168-31-100 workdir]# cat nginx-test.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-container
          image: nginx:latest
          ports:
            - containerPort: 80
          volumeMounts:
            - name: my-emptydir
              mountPath: /data   # 这里指定挂载路径
      volumes:
        - name: my-emptydir
          emptyDir: {}  # 这里配置 EmptyDir 卷

在emptydir中创建测试文件测试下

root@master-192-168-31-100 workdir]# kubectl exec -it my-deployment-87fcddd69-rf9qm -- bash
root@my-deployment-87fcddd69-rf9qm:/data# dd if=/dev/zero of=/data/test.db bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.743424 s, 1.4 GB/s

定位到该pod目录内可以看到一个empty-dir后缀的目录

[root@node-192-168-31-102 ~]# tree /var/lib/kubelet/pods/aae19680-6f64-4c56-9725-9494bf8ac8ee/volumes/kubernetes.io~empty-dir/
/var/lib/kubelet/pods/aae19680-6f64-4c56-9725-9494bf8ac8ee/volumes/kubernetes.io~empty-dir/
└── my-emptydir
    └── test.db

1 directory, 1 file

但是我们去这个容器的upperdir中可以看到emptydir的挂载点对应的data目录，但是其内容为空，因为容器挂载时会在容器里创建这个名字的目录（所以upperdir里有），但是一旦容器启动后，就被volume覆盖遮住了

[root@node-192-168-31-102 ~]# tree /var/lib/docker/overlay2/be6b4cebf42bdd14c525f446706d39298fdaa8476e723d3acf7527a11737db61/diff/data/
/var/lib/docker/overlay2/be6b4cebf42bdd14c525f446706d39298fdaa8476e723d3acf7527a11737db61/diff/data/

0 directories, 0 files

# 查看对应的目录id
[root@master-192-168-31-100 workdir]# kubectl exec -it my-deployment-87fcddd69-rf9qm -- ls -lid /data
202980633 drwxrwxrwx 2 root root 21 Aug 26 16:25 /data

[root@node-192-168-31-102 ~]# ll -di  /var/lib/docker/overlay2/be6b4cebf42bdd14c525f446706d39298fdaa8476e723d3acf7527a11737db61/diff/data/
68731756 drwxr-xr-x 2 root root 6 8月  27 00:22 /var/lib/docker/overlay2/be6b4cebf42bdd14c525f446706d39298fdaa8476e723d3acf7527a11737db61/diff/data/
[root@node-192-168-31-102 ~]# ll -di /var/lib/kubelet/pods/aae19680-6f64-4c56-9725-9494bf8ac8ee/volumes/kubernetes.io~empty-dir/my-emptydir/
202980633 drwxrwxrwx 2 root root 21 8月  27 00:25 /var/lib/kubelet/pods/aae19680-6f64-4c56-9725-9494bf8ac8ee/volumes/kubernetes.io~empty-dir/my-emptydir/

看来只能获取pod目录里的emptydir了，先获取POD ID，在去定位emptydir就够了，这里就略了。而对于hostpath类型，则不会在pod目录里创建，而是自行定义的路径了，不解析pod的spec的情况下，好像只能从docker的mounts里获取了，略

正文完
docker k8s
2023-08-21
 1
xadocker
版权声明：本站原创文章，由 xadocker 2023-08-21发表，共计17093字。
转载说明：除特殊说明外本站文章皆由CC-4.0协议发布，转载请注明出处。
使用prometheus-adapter实现自定义指标HPA
k8s Deployment
持续分析kubelet
k8s 中的dns
评论（没有评论）