k8s-Statefulset 资源 - SRE回忆录

共计 9555 个字符，预计需要花费 24 分钟才能阅读完成。

StatefulSet是一种有状态的资源集合，管理所有有状态的服务，常用于Mysql/MongoDB集群等。StatefulSet所管理的Pod有如下特点：

具有固定网络标识的Pod名称和hostname，(statefulset名称)−(序号)，序号：[0，副本数-1)
Pod顺序启停
稳定的存储，通过VolumeClaimTemplate为每个Pod创建一一对应的PV，删除或减少副本数，不会移除相关pv。
其对应的Service为headless service（解析该名称将会放回对应所有的Pod的Endpoint列表）

创建StatefulSet 测试资源

# 创建一个demo
[root@node1 pratic]# cat >statusful-nginx.yaml<<EOF
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 5
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
      annotations:
        volume.beta.kubernetes.io/storage-class: "nfsserver"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 100Mi
EOF

Pod启停有序性

观察StatusfulSet Pod创建顺序

web-0 pod处于Running和Ready状态后web-1 Pod才会被启动，这就是有状态集的启动有序性

[root@node1 pratic]# kubectl apply -f statusful-nginx.yaml
[root@node1 pratic]# kubectl get pods -w
NAME    READY   STATUS              RESTARTS   AGE
web-0   1/1     Running             0          32s
web-1   1/1     Running             0          18s
web-2   0/1     ContainerCreating   0          8s
web-2   1/1     Running             0          16s
web-3   0/1     Pending             0          0s
web-3   0/1     Pending             0          0s
web-3   0/1     Pending             0          2s
web-3   0/1     ContainerCreating   0          2s
web-3   0/1     ContainerCreating   0          3s
web-3   1/1     Running             0          12s
web-4   0/1     Pending             0          0s
web-4   0/1     Pending             0          0s
web-4   0/1     Pending             0          2s
web-4   0/1     ContainerCreating   0          2s
web-4   0/1     ContainerCreating   0          3s
web-4   1/1     Running             0          9s

Pod网络标志固定

测试每个pod的hostname


[root@node1 pratic]# for i in {0..4}; do kubectl exec "web-$i" -- sh -c 'hostname'; done
web-0
web-1
web-2
web-3
web-4

测试Pod的DNS地址

# 创建一个静态pod busybox
[root@node1 pratic]# kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm
If you don't see a command prompt, try pressing enter.
/ # nslookup web-0.nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-0.nginx
Address 1: 10.100.166.172 web-0.nginx.default.svc.cluster.local

/ # nslookup web-1.nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-1.nginx
Address 1: 10.100.166.173 web-1.nginx.default.svc.cluster.local

/ # nslookup web-2.nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-2.nginx
Address 1: 10.100.166.174 web-2.nginx.default.svc.cluster.local

测试固定的网络标识

1.先删除创建的Pod

# 先删除创建的demo pod，再删除的同时开启另一个终端查看demo pod资源创建情况
[root@node1 pratic]# kubectl delete pod -l app=nginx
pod "web-0" deleted
pod "web-1" deleted
pod "web-2" deleted
pod "web-3" deleted
pod "web-4" deleted

# 另一终端查看资源创建情况
[root@node1 pratic]# kubectl get pod -w -l app=nginx
NAME  web-0   1/1     Terminating         0          5m2s
web-1   1/1     Terminating         0          4m52s
web-2   1/1     Terminating         0          4m45s
web-3   1/1     Terminating         0          4m37s
web-4   1/1     Terminating         0          4m27s
web-1   0/1     Terminating         0          4m54s
web-4   0/1     Terminating         0          4m29s
web-2   0/1     Terminating         0          4m47s
web-3   0/1     Terminating         0          4m39s
web-0   0/1     Terminating         0          5m4s
web-3   0/1     Terminating         0          4m45s
web-3   0/1     Terminating         0          4m45s
web-0   0/1     Terminating         0          5m10s
web-0   0/1     Terminating         0          5m10s
web-1   0/1     Terminating         0          5m
web-0   0/1     Pending             0          0s
web-1   0/1     Terminating         0          5m
web-0   0/1     Pending             0          0s
web-2   0/1     Terminating         0          4m53s
web-2   0/1     Terminating         0          4m53s
web-4   0/1     Terminating         0          4m35s
web-4   0/1     Terminating         0          4m35s
web-0   0/1     ContainerCreating   0          1s
web-0   0/1     ContainerCreating   0          2s
web-0   1/1     Running             0          8s
web-1   0/1     Pending             0          0s
web-1   0/1     Pending             0          0s
web-1   0/1     ContainerCreating   0          0s
web-1   0/1     ContainerCreating   0          1s
web-1   1/1     Running             0          9s
web-2   0/1     Pending             0          0s
web-2   0/1     Pending             0          0s
web-2   0/1     ContainerCreating   0          0s
web-2   0/1     ContainerCreating   0          2s
web-2   1/1     Running             0          11s
web-3   0/1     Pending             0          0s
web-3   0/1     Pending             0          0s
web-3   0/1     ContainerCreating   0          0s
web-3   0/1     ContainerCreating   0          1s
web-3   1/1     Running             0          7s
web-4   0/1     Pending             0          0s
web-4   0/1     Pending             0          0s
web-4   0/1     ContainerCreating   0          0s
web-4   0/1     ContainerCreating   0          1s
web-4   1/1     Running             0          9s

2.测试此时重建的pod的网络标志

# 查看重建pod后的hostname
[root@node1 pratic]# for i in {0..4}; do kubectl exec web-$i -- sh -c 'hostname'; done
web-0
web-1
web-2
web-3
web-4

查看重建pod后的dns记录
/ # nslookup web-0.nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-0.nginx
Address 1: 10.100.166.188 10-100-166-188.nginx-service.default.svc.cluster.local
/ # nslookup web-1.nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-1.nginx
Address 1: 10.100.166.189 web-1.nginx.default.svc.cluster.local
/ # nslookup web-2.nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-2.nginx
Address 1: 10.100.166.190 10-100-166-190.nginx-service.default.svc.cluster.local

# 获取到
/ # nslookup nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      nginx
Address 1: 10.100.166.190 web-2.nginx.default.svc.cluster.local
Address 2: 10.100.166.189 10-100-166-189.nginx-service.default.svc.cluster.local
Address 3: 10.100.166.188 10-100-166-188.nginx-service.default.svc.cluster.local
Address 4: 10.100.166.134 10-100-166-134.nginx-service.default.svc.cluster.local
Address 5: 10.100.166.136 10-100-166-136.nginx-service.default.svc.cluster.local

从上面测试可以看出Pod 的序号、主机名、SRV 条目和记录名称没有改变，但和 Pod 相关联的 IP 地址可能发生了改变。所以在应用中使用StatefulSet时应该使用pod的SRV记录。

稳定的存储

查看测试资源创建的pvc/pv，发现其分别为每个pod创建了一个pvc，且是一一根据序号对应

# 查看pvc
[root@node1 pratic]# kubectl get pvc -l app=nginx
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0   Bound    pvc-a08aa8a7-cd1d-4584-b55b-80cd1b217e66   100Mi      RWO            nfsserver      33m
www-web-1   Bound    pvc-5707ad69-9e7c-4423-9144-e3e2f000b9ed   100Mi      RWO            nfsserver      33m
www-web-2   Bound    pvc-975520a6-de1f-4e26-b7d8-65e3353866ab   100Mi      RWO            nfsserver      33m
www-web-3   Bound    pvc-4e7b5b31-a334-4503-8b82-935d2a561cc0   100Mi      RWO            nfsserver      33m
www-web-4   Bound    pvc-39787c7a-2ae3-44ca-8821-d2bad2a4a75c   100Mi      RWO            nfsserver      33m

# 查看pvc
[root@node1 pratic]# kubectl get pv | grep www-web
pvc-39787c7a-2ae3-44ca-8821-d2bad2a4a75c   100Mi      RWO            Delete           Bound         default/www-web-4                            nfsserver                               34m
pvc-4e7b5b31-a334-4503-8b82-935d2a561cc0   100Mi      RWO            Delete           Bound         default/www-web-3                            nfsserver                               34m
pvc-5707ad69-9e7c-4423-9144-e3e2f000b9ed   100Mi      RWO            Delete           Bound         default/www-web-1                            nfsserver                               35m
pvc-975520a6-de1f-4e26-b7d8-65e3353866ab   100Mi      RWO            Delete           Bound         default/www-web-2                            nfsserver                               35m
pvc-a08aa8a7-cd1d-4584-b55b-80cd1b217e66   100Mi      RWO            Delete           Bound         default/www-web-0                            nfsserver                               35m

测试pvc和pod的对应关系

1.将每个Pod hostname写入各自PV中

[root@node1 pratic]# for i in {0..4}; do kubectl exec "web-$i" -- sh -c 'echo "$(hostname)" > /usr/share/nginx/html/index.html'; done
[root@node1 pratic]# for i in {0..4}; do kubectl exec -i -t "web-$i" -- curl http://localhost/; done
web-0
web-1
web-2
web-3
web-4

2.删除所有Pod

[root@node1 pratic]# kubectl delete pod -l app=nginx
pod "web-0" deleted
pod "web-1" deleted
pod "web-2" deleted
pod "web-3" deleted
pod "web-4" deleted

3.查看pv/pvc

# 五个 PersistentVolumeClaims 和五个 PersistentVolumes 仍然存在
[root@node1 pratic]# kubectl get pvc -l app=nginx
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0   Bound    pvc-a08aa8a7-cd1d-4584-b55b-80cd1b217e66   100Mi      RWO            nfsserver      33m
www-web-1   Bound    pvc-5707ad69-9e7c-4423-9144-e3e2f000b9ed   100Mi      RWO            nfsserver      33m
www-web-2   Bound    pvc-975520a6-de1f-4e26-b7d8-65e3353866ab   100Mi      RWO            nfsserver      33m
www-web-3   Bound    pvc-4e7b5b31-a334-4503-8b82-935d2a561cc0   100Mi      RWO            nfsserver      33m
www-web-4   Bound    pvc-39787c7a-2ae3-44ca-8821-d2bad2a4a75c   100Mi      RWO            nfsserver      33m
[root@node1 pratic]# kubectl get pv | grep www-web
pvc-39787c7a-2ae3-44ca-8821-d2bad2a4a75c   100Mi      RWO            Delete           Bound         default/www-web-4                            nfsserver                               34m
pvc-4e7b5b31-a334-4503-8b82-935d2a561cc0   100Mi      RWO            Delete           Bound         default/www-web-3                            nfsserver                               34m
pvc-5707ad69-9e7c-4423-9144-e3e2f000b9ed   100Mi      RWO            Delete           Bound         default/www-web-1                            nfsserver                               35m
pvc-975520a6-de1f-4e26-b7d8-65e3353866ab   100Mi      RWO            Delete           Bound         default/www-web-2                            nfsserver                               35m
pvc-a08aa8a7-cd1d-4584-b55b-80cd1b217e66   100Mi      RWO            Delete           Bound         default/www-web-0                            nfsserver                               35m

4.验证每个Pod和PV bond关系，虽然所有Pod重新创建和调度，但是他们PersistentVolumeClaim 相关联的 PersistentVolume 被重新挂载到了各自的 volumeMount 上。

[root@node1 pratic]# for i in {0..4}; do kubectl exec -i -t "web-$i" -- curl http://localhost/; done
web-0
web-1
web-2
web-3
web-4

StatefulSet扩缩容

扩容

[root@node1 pratic]# kubectl scale sts web --replicas=8
statefulset.apps/web scaled

# 安顺序扩容
[root@node1 pratic]# kubectl get pod -w -l app=nginx
......
web-5   0/1     Pending             0          0s
web-5   0/1     Pending             0          0s
web-5   0/1     Pending             0          1s
web-5   0/1     ContainerCreating   0          1s
web-5   0/1     ContainerCreating   0          2s
web-5   1/1     Running             0          8s
web-6   0/1     Pending             0          0s
web-6   0/1     Pending             0          0s
web-6   0/1     Pending             0          2s
web-6   0/1     ContainerCreating   0          2s
web-6   0/1     ContainerCreating   0          3s
web-6   1/1     Running             0          13s
web-7   0/1     Pending             0          0s
web-7   0/1     Pending             0          0s
web-7   0/1     Pending             0          2s
web-7   0/1     ContainerCreating   0          2s
web-7   0/1     ContainerCreating   0          4s
web-7   1/1     Running             0          11s

缩容

[root@node1 pratic]# kubectl scale sts web --replicas=5
statefulset.apps/web scaled

# 按倒序缩容
[root@node1 pratic]# kubectl get pod -w -l app=nginx
......
web-7   1/1     Terminating         0          2m50s
web-7   0/1     Terminating         0          2m52s
web-7   0/1     Terminating         0          2m53s
web-7   0/1     Terminating         0          2m53s
web-6   1/1     Terminating         0          3m6s
web-6   0/1     Terminating         0          3m7s
web-6   0/1     Terminating         0          3m8s
web-6   0/1     Terminating         0          3m8s
web-5   1/1     Terminating         0          3m16s
web-5   0/1     Terminating         0          3m17s
web-5   0/1     Terminating         0          3m25s
web-5   0/1     Terminating         0          3m25s

podManagementPolicy

有时候StatefulSet的启停有序性并不是必要的，只需要网络标志和存储固定即可，所有可以使用podManagementPolicy字段控制

[root@node1 pratic]# cat >statusfulset-nginx2.yaml<<EOF
apiVersion: v1
kind: Service
metadata:
  name: nginx2
  labels:
    app: nginx2
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx2
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web2
spec:
  serviceName: "nginx2"
  podManagementPolicy: "Parallel"
  replicas: 4
  selector:
    matchLabels:
      app: nginx2
  template:
    metadata:
      labels:
        app: nginx2
    spec:
      containers:
      - name: nginx2
        image: nginx:latest
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www2
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www2
      annotations:
        volume.beta.kubernetes.io/storage-class: "nfsserver"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 100Mi
EOF

测试查看创建Pod的顺序

[root@node1 pratic]# kubectl apply -f statusfulset-nginx2.yaml 
service/nginx2 created
statefulset.apps/web2 created

# 不再按顺序等待pod状态为running和ready，而是同时创建
[root@node1 pratic]# kubectl get pod -w -l app=nginx2
NAME     READY   STATUS              RESTARTS   AGE
web2-0   0/1     ContainerCreating   0          4s
web2-1   0/1     ContainerCreating   0          4s
web2-2   0/1     ContainerCreating   0          4s
web2-3   0/1     ContainerCreating   0          4s
web2-1   1/1     Running             0          8s
web2-3   1/1     Running             0          13s
web2-0   1/1     Running             0          19s
web2-2   1/1     Running             0          25s