共计 7121 个字符,预计需要花费 18 分钟才能阅读完成。
![部署多套ingress-controller 部署多套ingress-controller](https://www.xadocker.cn/wp-content/uploads/2022/05/79d8af581db41e3e25f1b12b36d627c4.png)
在k8s集群中一般用ingress来暴露服务到外部访问,如果项目过多且均需要被外部访问,单ingress-controller就负重前行扛大旗,所以我们需要对入口做流量的分流,避免相互影响,减少故障半径
nginx-ingress
查看官方文档,想要部署多套nginx-ingress则需要配置–ingress-class参数,默认为nginx
######## 略
args:
- /nginx-ingress-controller
- --publish-service=ingress-nginx/ingress-nginx-controller
- --election-id=ingress-controller-leader
# 多套nginx-ingress时,需要将此处配置不同值
- --ingress-class=nginx
- --configmap=ingress-nginx/ingress-nginx-controller
######## 略
配置该参数后,则在声明ingress资源时,需要指定ingress就可以使用以下方式
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp
spec:
ingressClassName: ingress1
######## 略
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp
spec:
ingressClassName: ingress2
######## 略
部署多套nginx-ingress
此处我们使用helm部署nginx-ingress
为两套nginx-ingress分开namespace部署
[root@k8s-master ingress]# kubectl create ns ingress-frontend
namespace/ingress-frontend created
[root@k8s-master ingress]# kubectl create ns ingress-backend
namespace/ingress-backend created
# 配置ingress仓库
[root@k8s-master ingress]# helm repo add ingress https://kubernetes.github.io/ingress-nginx
[root@k8s-master ingress]# helm repo update
部署frontend专用ingress
[root@k8s-master ingress]# helm install ingress-frontend ingress/ingress-nginx --set controller.ingressClass=ingress-frontend --set controller.service.type=NodePort --set controller.service.httpPort.nodePort=30010 --set controller.stats.enabled=true --set controller.metrics.enabled=true --namespace ingress-frontend --version 3.41.0
部署backend专用ingress
[root@k8s-master ingress]# helm install ingress-backend ingress/ingress-nginx --set controller.ingressClass=ingress-backend --set controller.service.type=NodePort --set controller.service.httpPort.nodePort=30020 --set controller.stats.enabled=true --set controller.metrics.enabled=true --namespace ingress-backend --version 3.41.0
查看ingress-controller运行状态
[root@k8s-master ingress-prj]# kubectl get pod -n ingress-frontend
NAME READY STATUS RESTARTS AGE
ingress-frontend-ingress-nginx-controller-676cdbf784-7fzzq 1/1 Running 0 14m
[root@k8s-master ingress-prj]# kubectl get pods -n ingress-backend
NAME READY STATUS RESTARTS AGE
ingress-backend-ingress-nginx-controller-654dbc499b-9mqzd 1/1 Running 0 15m
测试使用不同nginx-ingress
部署demo服务
[root@k8s-master ingress-prj]# cat podinfo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
containers:
- name: api
image: stefanprodan/podinfo
ports:
- containerPort: 9898
---
apiVersion: v1
kind: Service
metadata:
name: api
spec:
ports:
- port: 80
targetPort: 9898
selector:
app: api
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
spec:
selector:
matchLabels:
app: frontend
template:
metadata:
labels:
app: frontend
spec:
containers:
- name: frontend
image: stefanprodan/podinfo
ports:
- containerPort: 9898
---
apiVersion: v1
kind: Service
metadata:
name: frontend
spec:
ports:
- port: 80
targetPort: 9898
selector:
app: frontend
type: LoadBalancer
创建ingress资源
前端服务ingress资源
[root@k8s-master ingress-prj]# cat ingress-frontend.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: frontend
spec:
ingressClassName: ingress-frontend
rules:
- host: "example.com"
http:
paths:
- backend:
serviceName: frontend
servicePort: 80
path: /
pathType: Prefix
[root@k8s-master ingress-prj]# kubectl apply -f ingress-frontend.yaml
ingress.extensions/frontend created
后端服务ingress资源
[root@k8s-master ingress-prj]# cat ingress-backend.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: backend
spec:
ingressClassName: ingress-backend
rules:
- host: "api.example.com"
http:
paths:
- backend:
serviceName: api
servicePort: 80
path: /
pathType: Prefix
[root@k8s-master ingress-prj]# kubectl apply -f ingress-backend.yaml
ingress.extensions/backend created
查看ingress资源
[root@k8s-master ingress-prj]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
backend ingress-backend api.example.com 80 116s
frontend ingress-frontend example.com 80 7m21s
测试各自的流量入口
测试后端ingress入口
[root@k8s-master ~]# kubectl run -ti --rm=true busybox --image=busybox
# 测试后端ingress入口
/ # wget --header="Host: api.example.com" -qO- ingress-backend-ingress-nginx-controller.ingress-backend
{
"hostname": "api-69f5868bb9-9tsqg",
"version": "6.3.0",
"revision": "e2e85a960447a56a1fa45747d2275abd28c13870",
"color": "#34577c",
"logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
"message": "greetings from podinfo v6.3.0",
"goos": "linux",
"goarch": "amd64",
"runtime": "go1.19.4",
"num_goroutine": "6",
"num_cpu": "8"
}/ #
# 测试后端ingress入口访问前端服务域名
/ # wget --header="Host: example.com" -qO- ingress-backend-ingress-nginx-controller.ingress-backend
wget: server returned error: HTTP/1.1 404 Not Found
测试前端服务ingress入口
/ # wget --header="Host: example.com" --header="User-Agent: Mozilla" -qO- ingress-frontend-ingress-nginx-controller.ingress-frontend
<!DOCTYPE html>
<html>
<head>
<title>frontend-85d76795d9-xzr2b</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
<link rel="shortcut icon" type="image/png" href="https://kubernetes.io/images/favicon.png">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400,500,700,900" rel="stylesheet">
<link href="https://fonts.googleapis.com/css?family=Material+Icons" rel="stylesheet">
<link href="https://cdn.jsdelivr.net/npm/vuetify@2.x/dist/vuetify.min.css" rel="stylesheet">
<style>
[v-cloak] {
display: none;
}
.v-application .v-parallax {
height: 100vh !important;
}
</style>
</head>
<body>
<div id="app" v-cloak>
<v-app dark>
<v-content>
<section>
<v-parallax id="parallax-hero" :style="cuddleStyle" src="https://upload.wikimedia.org/wikipedia/commons/c/ca/1x1.png">
<v-layout
column
######## 略
# 测试前端ingress入口访问后端服务域名
/ # wget --header="Host: example.com" --header="User-Agent: Mozilla" -qO- ingress-frontend-ingress-nginx-controller.ingress-backend
wget: server returned error: HTTP/1.1 404 Not Found
通过指定ingressClassName: ingress-frontend
从而实现不同ingress-controller入口,所以后续集群的入口可以按如下部署结构优化:
- ingress-nginx使用hostport部署
- ingress-nginx使用daemonset+nodeselector选择器部署
- 按不同组织或项目划分多个ingress-controller
- 每套ingress-controller都配有内外网slb
![部署多套ingress-controller 部署多套ingress-controller](https://www.xadocker.cn/wp-content/uploads/image-19.png)
prometheus监控ingress-nginx
ingress-nginx开启metrics功能
前面helm部署时已配置metrics开始,所以对应创建出来的pod和svc会有一个10254端口,该端口就是用来暴露ingress-nginx-controller服务的监控信息
[root@k8s-master ingress-prj]# kubectl get svc -n ingress-frontend
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-frontend-ingress-nginx-controller NodePort 10.96.89.103 <none> 80:30482/TCP,443:31314/TCP 154m
ingress-frontend-ingress-nginx-controller-admission ClusterIP 10.96.124.62 <none> 443/TCP 154m
ingress-frontend-ingress-nginx-controller-metrics ClusterIP 10.96.250.201 <none> 10254/TCP 154m
创建servicemonitor抓取ingress-nginx-controller
[root@k8s-master ingress-prj]# cat servicemonitor-ingress-frontend.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: nginx-ingress-frontend
namespace: monitoring
labels:
app.kubernetes.io/component: controller
spec:
jobLabel: app.kubernetes.io/component
endpoints:
- port: metrics
interval: 5s
selector:
matchLabels:
app.kubernetes.io/component: controller
namespaceSelector:
matchNames:
- ingress-frontend
[root@k8s-master ingress-prj]# cat servicemonitor-ingress-backend.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: nginx-ingress-backend
namespace: monitoring
labels:
app.kubernetes.io/component: controller
spec:
jobLabel: app.kubernetes.io/component
endpoints:
- port: metrics
interval: 5s
selector:
matchLabels:
app.kubernetes.io/component: controller
namespaceSelector:
matchNames:
- ingress-backend
查看targets
![部署多套ingress-controller 部署多套ingress-controller](https://www.xadocker.cn/wp-content/uploads/image-21.png)
查看grafana
![部署多套ingress-controller 部署多套ingress-controller](https://www.xadocker.cn/wp-content/uploads/image-20.png)
正文完