一文读懂Ingress-Nginx的实践_缓存

❤️ 摘要: ingress-nginx 是 kubernetes 中用于管理 http 和 https 流量的强大工具。通过配置 ingress 资源，用户可以灵活地路由流量到后端服务，增强了集群的可访问性和可扩展性。本文将深入探讨 ingress-nginx 的工作原理、基本配置及最佳实践，并通过实际案例来验证其功能。

1 概念

1.1 什么是ingress？

ingress是kubernetes的一种 api 对象，用于管理集群内服务的外部访问。ingress 可以提供从集群外部到集群内service的 http 和 https 路由，并可以基于域名、路径等规则将请求转发到集群内的service上。

下面是 ingress 的一个简单示例，可将外部的流量导流到k8s集群的同一 service，再由service将流量发送到后端的pod：

1.1.1 主要功能：

http/https 路由: ingress 允许外部 http 和 https 请求根据预定义的路由规则访问集群内部的服务。路由规则可以基于主机名、url 路径等进行配置。
反向代理: ingress 控制器通常会充当反向代理，将外部请求路由到集群内部的服务。ingress 不会公开任意端口或协议，通常使用service.type=nodeport或service.type=loadbalancer类型的服务。
域名支持: 通过 ingress，可以为集群中的服务配置域名，从而提供更友好的访问方式。例如，可以将 foo.example.com 指向集群内的 foo-service 服务。
负载均衡: ingress 控制器可以将流量分配给多个服务实例，实现负载均衡，从而提高服务的可用性和性能。在云环境中，ingress 通常会与云提供商的负载均衡器集成，以确保流量能够从外部访问集群。
tls/ssl 支持: ingress 支持 https 协议，允许为服务配置 tls 证书，以确保传输的安全性。tls 证书可以使用 kubernetes 的 secret 对象进行管理。
虚拟主机: ingress 可以配置虚拟主机，使得多个服务可以通过不同的域名或子域名进行访问。

1.2 ingress的组件

ingress 控制器: 为了使 ingress 资源正常工作，集群必须运行一个 ingress 控制器。类似kubernetes集群的kube-apiserver服务，负责管理和执行 ingress 资源定义的路由规则。常见的 ingress 控制器包括 nginx ingress controller、traefik、haproxy 等，还有公有云厂商的ingress 控制器包括azure的aks application gateway、阿里云的alibaba cloud mse ingress。
ingress 对象: 用户自定义的 kubernetes 对象，描述了外部流量访问集群内service的规则集。

1.3 什么是ingress-nginx

ingress nginx是 kubernetes 中最常用的 ingress 控制器之一，由 nginx 官方提供支持（备注：nginx和k8s官方各自维护了一套 nginx ingress controller）。

nginx ingress controller 是 nginx 和 nginx plus 的ingress controller实现，可以对websocket、grpc、tcp 和 udp 应用程序进行负载平衡。它支持标准ingress功能，例如基于内容的路由和 tls/ssl 终止。

1.4 ingress-nginx优点和限制

优点	限制
经过市场广泛使用，社区支持活跃。	需要对 nginx 配置有一定的理解，以便充分利用其功能。
与 nginx 的成熟生态系统兼容，提供稳定和高性能的 http/https 处理。	在非常复杂的路由场景中，配置可能变得繁琐。
丰富的配置选项和灵活的路由机制。

1.5 版本兼容性矩阵

ingress-nginx 项目支持的版本意味着官方已经完成了 e2e 测试，并且它们正在通过列出的版本。 ingress-nginx 版本可能适用于旧版本，但该项目不做出这种保证。

支持	ingress-nginx版本	k8s支持版本	alpine 版本	nginx版本	helm chart 版本
🔄	v1.11.2	1.30, 1.29, 1.28, 1.27, 1.26	3.20.0	1.25.5	4.11.2
🔄	v1.11.1	1.30, 1.29, 1.28, 1.27, 1.26	3.20.0	1.25.5	4.11.1
🔄	v1.11.0	1.30, 1.29, 1.28, 1.27, 1.26	3.20.0	1.25.5	4.11.0
🔄	v1.10.4	1.30, 1.29, 1.28, 1.27, 1.26	3.20.0	1.25.5	4.10.4
🔄	v1.10.3	1.30, 1.29, 1.28, 1.27, 1.26	3.20.0	1.25.5	4.10.3
🔄	v1.10.2	1.30, 1.29, 1.28, 1.27, 1.26	3.20.0	1.25.5	4.10.2
🔄	v1.10.1	1.30, 1.29, 1.28, 1.27, 1.26	3.19.1	1.25.3	4.10.1
🔄	v1.10.0	1.29, 1.28, 1.27, 1.26	3.19.1	1.25.3	4.10.0

2 实践： ingress nginx部署

本文k8s环境是v1.29.7, ingress-nginx是v1.11.1

⚠️ 注意：提前将ingress-nginx的镜像下载并上传到私有仓库

[root@k8s-master1 ingress-nginx]# grep -n -r image: ingress-nginx.yaml
447:        image:  registry.k8s.io/ingress-nginx/controller:v1.11.1 @sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a
548:        image:  registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1 @sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366

2.1 使用helm部署ingress-nginx

官方推荐helm方式部署，如果环境没有helm需要先下载： helm下载地址

2.1.1 安装和配置helm

安装helm

tar -zxvf helm-v3.15.3-linux-amd64.tar.gz --strip-components 1 -c /usr/local/bin linux-amd64/helm

添加ingress-nginx的repo

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

更新repo信息

helm repo update

搜索ingress-nginx的chart

[root@k8s-master1 ingress-nginx]# helm search repo ingress-nginx --version 4.11.1
name                            chart version   app version     description                      
ingress-nginx/ingress-nginx     4.11.1          1.11.1          ingress controller for kubernetes using nginx a...

❔ 说明： --version 4.11.1：指定chart版本搜索

下载ingress-nginx的chart

mkdir helm && cd helm
helm pull ingress-nginx/ingress-nginx --version 4.11.1

解压charts

# 解压chart
tar zxvf ingress-nginx-4.11.1.tgz

# 进入chart目录
cd ingress-nginx

2.1.2 配置和创建ingress-nginx

调整values.yaml的参数

镜像仓库和镜像

  21   image:
  22     ## keep false as default for now!
  23     chroot: false
  24     #注释原仓库地址
  25     #registry: registry.k8s.io
  26     #替换成阿里云仓库地址
  27     registry: registry.cn-hangzhou.aliyuncs.com
  28     image: ingress-nginx-steven/controller
  32     tag: "v1.11.1"
  33     # 注释sha256校验码
  34     #digest: sha256:e6439a12b52076965928e83b7b56aae6731231677b01e81818bce7fa5c60161a

 ---
 805       image:
 806         # 注释原仓库地址
 807         #registry: registry.k8s.io
 808         registry: registry.cn-hangzhou.aliyuncs.com
 809         image: ingress-nginx-steven/kube-webhook-certgen
 813         tag: v1.4.1
 814         # 注释sha256校验码
 815         #digest: sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366

使用本地网络和dns

# 启用主机网络 
103   hostnetwork: true
 ...
# 启用主机dns服务
78   dnspolicy: clusterfirstwithhostnet

修改ingress模式和指定port

 # 默认是loadbalancer
 484     #type: loadbalancer
 485     type: nodeport
 ...
 540     appprotocol: true
 541     nodeports:
 542       # -- node port allocated for the external http listener. if left empty, the service co     ntroller allocates one from the configured node port range.
 543       http: "30080"
 544       # -- node port allocated for the external https listener. if left empty, the service c     ontroller allocates one from the configured node port range.
 545       https: "30443"

创建前，执行以下命令检查配置是否修改成功：

helm install ingress-nginx . --dry-run=client --namespace=ingress-nginx

创建ingress-nignx

helm install ingress-nginx . --namespace=ingress-nginx --create-namespace

查看是否创建成功

[root@k8s-master1 ingress-nginx]# helm list --namespace ingress-nginx
name            namespace       revision        updated                                 status  chart                    app version
ingress-nginx   ingress-nginx   1               2024-08-27 21:44:34.137344711 +0800 cst deployedingress-nginx-4.11.1     1.11.1

卸载ingress-nginx

[root@k8s-master1 ingress-nginx]# helm uninstall ingress-nginx -n ingress-nginx
release "ingress-nginx" uninstalled

2.2 使用yaml文件部署ingress-nginx

1) 参考官方yaml文件

2) 也可以参考以下配置

apiversion: v1
kind: namespace
metadata:
  labels:
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  name: ingress-nginx
---
apiversion: v1
automountserviceaccounttoken: true
kind: serviceaccount
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx
  namespace: ingress-nginx
---
apiversion: v1
automountserviceaccounttoken: true
kind: serviceaccount
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx-admission
  namespace: ingress-nginx
---
apiversion: rbac.authorization.k8s.io/v1
kind: role
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx
  namespace: ingress-nginx
rules:
- apigroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
- apigroups:
  - ""
  resources:
  - configmaps
  - pods
  - secrets
  - endpoints
  verbs:
  - get
  - list
  - watch
- apigroups:
  - ""
  resources:
  - services
  verbs:
  - get
  - list
  - watch
- apigroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- apigroups:
  - networking.k8s.io
  resources:
  - ingresses/status
  verbs:
  - update
- apigroups:
  - networking.k8s.io
  resources:
  - ingressclasses
  verbs:
  - get
  - list
  - watch
- apigroups:
  - coordination.k8s.io
  resourcenames:
  - ingress-nginx-leader
  resources:
  - leases
  verbs:
  - get
  - update
- apigroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
- apigroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
- apigroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch
  - get
---
apiversion: rbac.authorization.k8s.io/v1
kind: role
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx-admission
  namespace: ingress-nginx
rules:
- apigroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - create
---
apiversion: rbac.authorization.k8s.io/v1
kind: clusterrole
metadata:
  labels:
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx
rules:
- apigroups:
  - ""
  resources:
  - configmaps
  - endpoints
  - nodes
  - pods
  - secrets
  - namespaces
  verbs:
  - list
  - watch
- apigroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - list
  - watch
- apigroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
- apigroups:
  - ""
  resources:
  - services
  verbs:
  - get
  - list
  - watch
- apigroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- apigroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
- apigroups:
  - networking.k8s.io
  resources:
  - ingresses/status
  verbs:
  - update
- apigroups:
  - networking.k8s.io
  resources:
  - ingressclasses
  verbs:
  - get
  - list
  - watch
- apigroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch
  - get
---
apiversion: rbac.authorization.k8s.io/v1
kind: clusterrole
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx-admission
rules:
- apigroups:
  - admissionregistration.k8s.io
  resources:
  - validatingwebhookconfigurations
  verbs:
  - get
  - update
---
apiversion: rbac.authorization.k8s.io/v1
kind: rolebinding
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx
  namespace: ingress-nginx
roleref:
  apigroup: rbac.authorization.k8s.io
  kind: role
  name: ingress-nginx
subjects:
- kind: serviceaccount
  name: ingress-nginx
  namespace: ingress-nginx
---
apiversion: rbac.authorization.k8s.io/v1
kind: rolebinding
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx-admission
  namespace: ingress-nginx
roleref:
  apigroup: rbac.authorization.k8s.io
  kind: role
  name: ingress-nginx-admission
subjects:
- kind: serviceaccount
  name: ingress-nginx-admission
  namespace: ingress-nginx
---
apiversion: rbac.authorization.k8s.io/v1
kind: clusterrolebinding
metadata:
  labels:
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx
roleref:
  apigroup: rbac.authorization.k8s.io
  kind: clusterrole
  name: ingress-nginx
subjects:
- kind: serviceaccount
  name: ingress-nginx
  namespace: ingress-nginx
---
apiversion: rbac.authorization.k8s.io/v1
kind: clusterrolebinding
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx-admission
roleref:
  apigroup: rbac.authorization.k8s.io
  kind: clusterrole
  name: ingress-nginx-admission
subjects:
- kind: serviceaccount
  name: ingress-nginx-admission
  namespace: ingress-nginx
---
apiversion: v1
data:
  allow-snippet-annotations: "false"
kind: configmap
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
---
apiversion: v1
kind: service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  externaltrafficpolicy: local
  ipfamilies:
  - ipv4
  ipfamilypolicy: singlestack
  ports:
  - appprotocol: http
    name: http
    port: 80
    protocol: tcp
    targetport: http
  - appprotocol: https
    name: https
    port: 443
    protocol: tcp
    targetport: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: loadbalancer
---
apiversion: v1
kind: service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx-controller-admission
  namespace: ingress-nginx
spec:
  ports:
  - appprotocol: https
    name: https-webhook
    port: 443
    targetport: webhook
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: clusterip
---
apiversion: apps/v1
kind: deployment
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  minreadyseconds: 0
  revisionhistorylimit: 10
  selector:
    matchlabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
  strategy:
    rollingupdate:
      maxunavailable: 1
    type: rollingupdate
  template:
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.11.1
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --publish-service=$(pod_namespace)/ingress-nginx-controller
        - --election-id=ingress-nginx-leader
        - --controller-class=k8s.io/ingress-nginx
        - --ingress-class=nginx
        - --configmap=$(pod_namespace)/ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        - --enable-metrics=false
        env:
        - name: pod_name
          valuefrom:
            fieldref:
              fieldpath: metadata.name
        - name: pod_namespace
          valuefrom:
            fieldref:
              fieldpath: metadata.namespace
        - name: ld_preload
          value: /usr/local/lib/libmimalloc.so
        image: registry.cn-hangzhou.aliyuncs.com/ingress-nginx-steven/controller:v1.11.1
        imagepullpolicy: ifnotpresent
        lifecycle:
          prestop:
            exec:
              command:
              - /wait-shutdown
        livenessprobe:
          failurethreshold: 5
          httpget:
            path: /healthz
            port: 10254
            scheme: http
          initialdelayseconds: 10
          periodseconds: 10
          successthreshold: 1
          timeoutseconds: 1
        name: controller
        ports:
        - containerport: 80
          name: http
          protocol: tcp
        - containerport: 443
          name: https
          protocol: tcp
        - containerport: 8443
          name: webhook
          protocol: tcp
        readinessprobe:
          failurethreshold: 3
          httpget:
            path: /healthz
            port: 10254
            scheme: http
          initialdelayseconds: 10
          periodseconds: 10
          successthreshold: 1
          timeoutseconds: 1
        resources:
          requests:
            cpu: 100m
            memory: 90mi
        securitycontext:
          allowprivilegeescalation: false
          capabilities:
            add:
            - net_bind_service
            drop:
            - all
          readonlyrootfilesystem: false
          runasnonroot: true
          runasuser: 101
          seccompprofile:
            type: runtimedefault
        volumemounts:
        - mountpath: /usr/local/certificates/
          name: webhook-cert
          readonly: true
      dnspolicy: clusterfirst
      nodeselector:
        kubernetes.io/os: linux
      serviceaccountname: ingress-nginx
      terminationgraceperiodseconds: 300
      volumes:
      - name: webhook-cert
        secret:
          secretname: ingress-nginx-admission
---
apiversion: batch/v1
kind: job
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx-admission-create
  namespace: ingress-nginx
spec:
  template:
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.11.1
      name: ingress-nginx-admission-create
    spec:
      containers:
      - args:
        - create
        - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(pod_namespace).svc
        - --namespace=$(pod_namespace)
        - --secret-name=ingress-nginx-admission
        env:
        - name: pod_namespace
          valuefrom:
            fieldref:
              fieldpath: metadata.namespace
        image: registry.cn-hangzhou.aliyuncs.com/ingress-nginx-steven/kube-webhook-certgen:v1.4.1
        imagepullpolicy: ifnotpresent
        name: create
        securitycontext:
          allowprivilegeescalation: false
          capabilities:
            drop:
            - all
          readonlyrootfilesystem: true
          runasnonroot: true
          runasuser: 65532
          seccompprofile:
            type: runtimedefault
      nodeselector:
        kubernetes.io/os: linux
      restartpolicy: onfailure
      serviceaccountname: ingress-nginx-admission
---
apiversion: batch/v1
kind: job
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx-admission-patch
  namespace: ingress-nginx
spec:
  template:
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.11.1
      name: ingress-nginx-admission-patch
    spec:
      containers:
      - args:
        - patch
        - --webhook-name=ingress-nginx-admission
        - --namespace=$(pod_namespace)
        - --patch-mutating=false
        - --secret-name=ingress-nginx-admission
        - --patch-failure-policy=fail
        env:
        - name: pod_namespace
          valuefrom:
            fieldref:
              fieldpath: metadata.namespace
        image: registry.cn-hangzhou.aliyuncs.com/ingress-nginx-steven/kube-webhook-certgen:v1.4.1
        imagepullpolicy: ifnotpresent
        name: patch
        securitycontext:
          allowprivilegeescalation: false
          capabilities:
            drop:
            - all
          readonlyrootfilesystem: true
          runasnonroot: true
          runasuser: 65532
          seccompprofile:
            type: runtimedefault
      nodeselector:
        kubernetes.io/os: linux
      restartpolicy: onfailure
      serviceaccountname: ingress-nginx-admission
---
apiversion: networking.k8s.io/v1
kind: ingressclass
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: nginx
spec:
  controller: k8s.io/ingress-nginx
---
apiversion: admissionregistration.k8s.io/v1
kind: validatingwebhookconfiguration
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.1
  name: ingress-nginx-admission
webhooks:
- admissionreviewversions:
  - v1
  clientconfig:
    service:
      name: ingress-nginx-controller-admission
      namespace: ingress-nginx
      path: /networking/v1/ingresses
  failurepolicy: fail
  matchpolicy: equivalent
  name: validate.nginx.ingress.kubernetes.io
  rules:
  - apigroups:
    - networking.k8s.io
    apiversions:
    - v1
    operations:
    - create
    - update
    resources:
    - ingresses
  sideeffects: none
 官方yaml文件

根据官方yaml，修改镜像地址：

[root@k8s-master1 ingress-nginx]# grep -n  image: ingress-nginx.yaml
448:        image:  harbor.zx/hcie/controller:v1.11.1 
549:        image:  harbor.zx/hcie/kube-webhook-certgen:v1.4.1 
602:        image:  harbor.zx/hcie/kube-webhook-certgen:v1.4.1

修改ingress的端口：

352   ports:
353   - appprotocol: http
354     name: http
355     port: 80
356     protocol: tcp
357     targetport: http
358      nodeport: 30080 
359   - appprotocol: https
360     name: https
361     port: 443
362     protocol: tcp
363     targetport: https
364      nodeport: 30443

修改ingress的代理方式：

369   #type: loadbalancer  # 默认是负载均衡，如果是云环境建议使用默认参数
370   type: nodeport

部署ingress-nginx：

kubectl apply -f ingress-nginx.yaml

部署后输出结果：

namespace/ingress-nginx created
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
configmap/ingress-nginx-controller created
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created

2.3 部署后查看ingress状态

部署后，执行以下命令：

kubectl get svc,pod -n ingress-nginx -owide

查看pod和svc的状态：

name                                         type        cluster-ip       external-ip   port(s)                      age   selector
service/ingress-nginx-controller              nodeport     10.245.187.253   <none>         80:30080/tcp,443:30443/tcp    18m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-admission   clusterip   10.245.245.14    <none>        443/tcp                      18m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

name                                            ready   status      restarts   age   ip              node          nominated node   readiness gates
pod/ingress-nginx-admission-create-h6s42        0/1     completed   0          18m   172.16.126.2    k8s-worker2   <none>           <none>
pod/ingress-nginx-admission-patch-79kv5         0/1     completed   0          18m   172.16.126.3    k8s-worker2   <none>           <none>
pod/ingress-nginx-controller-746fcdfcdd-6gt4n   1/1     running     0          18m   172.16.194.85    k8s-worker1    <none>           <none>

❔ 说明：

nodeport: service的类型是在ingress-nginx.yaml中自定义。
80:30080/tcp, 443:30443/tcp: 这两个端口也是在ingress-nginx.yaml中自定义，后面访问使用这两个端口。
ingress-nginx-controller的pod为ingress-nginx-controller-746fcdfcdd-6gt4n运行在k8s-worker1节点上。后续访问ingress代理的服务，都要用k8s-worker1节点的ip做dns解析。

执行以下命令，查看ingressclass：

[root@k8s-master1 ingress-nginx]# kubectl get ingressclass
name    controller             parameters   age
nginx   k8s.io/ingress-nginx   <none>       25m

❔ 说明:

ingress类型：nginx是ingress-nginx默认创建的；
当创建ingress对象时，需要指定ingressclass字段(.spec.ingressclassname)
可以对ingressclass添加ingressclass.kubernetes.io/is-default-class 注解，这样 ingressclass 会被视为默认的 ingress 类。当某个 ingressclass 资源将此注解设置为 true 时，没有指定类的新 ingress 资源将被分配到此默认类。

2.4 创建实例测试 ingress

2.4.1 部署ingress对象

最后，我们打算使用1个3副本的deployment组成nginx-service，然后创建一个ingress，用于代理nginx的服务。yaml配置如下：

# 部署3个nginx实例
apiversion: apps/v1
kind: deployment
metadata:
  name: nginx-test
spec:
  replicas: 3
  selector:
    matchlabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: registry.cn-hangzhou.aliyuncs.com/hcie/nginx:1.26.1-alpine
        ports:
        - containerport: 80
---
# 部署nginx的service
apiversion: v1
kind: service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx-pod
  type: clusterip
  ports:
  - port: 80
    targetport: 80
---
# 创建ingress
apiversion: networking.k8s.io/v1
kind: ingress
metadata:
  name: ingress-http
spec:
  # 上面的ingressclass名称
  ingressclassname: "nginx"
  rules:
  # 域名，稍后要通过这个域名访问服务
  - host: nginx.steven.com
    http:
      paths:
      # 路径，访问域名时后面添加的路径
      - path: /
        pathtype: prefix
        backend:
          # 要代理的服务和服务的端口
          service:
            name: nginx-service
            port: 
              number: 80

部署deployment、service、ingress

kubectl apply -f ingress-test.yaml

执行以下命令，查看部署结果：

[root@k8s-master1 ingress-nginx]# kubectl get deployment,svc,ingress
name                         ready   up-to-date   available   age
deployment.apps/nginx-test   3/3     3            3           19s

name                    type        cluster-ip       external-ip   port(s)   age
service/nginx-service   clusterip   10.245.127.127   <none>        80/tcp    19s

name                                     class   hosts            address          ports   age
ingress.networking.k8s.io/ingress-http   nginx   nginx.test.com   10.245.187.253   80      19s

查看下ingress-http详细信息：

[root@k8s-master1 ingress-nginx]# kubectl describe ingress ingress-http
name:             ingress-http
labels:           <none>
namespace:        default
address:          10.245.187.253
ingress class:    nginx
default backend:  <default>
rules:
  host            path  backends
  ----            ----  --------
  nginx.test.com
                  /   nginx-service:80 (<none>)
annotations:      <none>
events:
  type    reason  age                    from                      message
  ----    ------  ----                   ----                      -------
  normal  sync    2m49s (x2 over 2m56s)  nginx-ingress-controller  scheduled for sync

查看ingress-controller的日志信息：

[root@k8s-master1 ingress-nginx]# kubectl -n ingress-nginx logs -f ingress-nginx-controller-746fcdfcdd-6gt4n
-------------------------------------------------------------------------------
nginx ingress controller
  release:       v1.11.1
  build:         7c44f992012555ff7f4e47c08d7c542ca9b4b1f7
  repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.5

-------------------------------------------------------------------------------

w0826 03:58:30.736251       7 client_config.go:659] neither --kubeconfig nor --master was specified.  using the inclusterconfig.  this might not work.
i0826 03:58:30.736660       7 main.go:205] "creating api client" host="https://10.245.0.1:443"
i0826 03:58:37.887271       7 main.go:248] "running in kubernetes cluster" major="1" minor="29" git="v1.29.7" state="clean" commit="4e4a18878ce330fefda1dc46acca88ba355e9ce7" platform="linux/amd64"
i0826 03:58:38.432286       7 main.go:101] "ssl fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
i0826 03:58:38.490727       7 ssl.go:535] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
i0826 03:58:38.516493       7 nginx.go:271] "starting nginx ingress controller"
i0826 03:58:38.538981       7 event.go:377] event(v1.objectreference{kind:"configmap", namespace:"ingress-nginx", name:"ingress-nginx-controller", uid:"0dba2870-a96b-4b5e-a908-b985139e5d63", apiversion:"v1", resourceversion:"624593", fieldpath:""}): type: 'normal' reason: 'create' configmap ingress-nginx/ingress-nginx-controller
i0826 03:58:39.719275       7 nginx.go:317] "starting nginx process"
i0826 03:58:39.720802       7 leaderelection.go:250] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader...
i0826 03:58:39.723329       7 nginx.go:337] "starting validation webhook" address=":8443" certpath="/usr/local/certificates/cert" keypath="/usr/local/certificates/key"
i0826 03:58:39.724636       7 controller.go:193] "configuration changes detected, backend reload required"
i0826 03:58:39.748482       7 leaderelection.go:260] successfully acquired lease ingress-nginx/ingress-nginx-leader
i0826 03:58:39.748953       7 status.go:85] "new leader elected" identity="ingress-nginx-controller-746fcdfcdd-6gt4n"
 i0826 03:58:39.826280       7 controller.go:213] "backend successfully reloaded"
i0826 03:58:39.826390       7 controller.go:224] "initial sync, sleeping for 1 second" 
i0826 03:58:39.826978       7 event.go:377] event(v1.objectreference{kind:"pod", namespace:"ingress-nginx", name:"ingress-nginx-controller-746fcdfcdd-6gt4n", uid:"68964da5-0da2-4ca7-84f3-742e3d1646ed", apiversion:"v1", resourceversion:"624720", fieldpath:""}): type: 'normal' reason: 'reload' nginx reload triggered due to a change in configuration

从最后几条日志来看，我们创建的ingress已经与ingress-nginx成功关联上了，下面可以进行访问测试了。

2.4.2 访问测试

2.4.2.1 在hosts文件里添加解析记录：

❤️ 推荐：如果安装了火绒，可以使用内置安全工具，方便修改。

添加记录：

192.168.3.44 nginx.steven.com

访问测试：

2.4.3 pod负载均衡测试

修改nginx的index.html

[root@k8s-master1 ingress-nginx]# kubectl exec -it nginx-test-6fc95f9f89-dsqkg -- sh
/ # echo "nginx1"> /usr/share/nginx/html/index.html
/ # exit
[root@k8s-master1 ingress-nginx]# kubectl exec -it nginx-test-6fc95f9f89-h7khh -- sh
/ # echo "nginx2"> /usr/share/nginx/html/index.html
/ # exit
[root@k8s-master1 ingress-nginx]# kubectl exec -it nginx-test-6fc95f9f89-z2v6l -- sh
/ # echo "nginx3"> /usr/share/nginx/html/index.html
/ # exit

再次访问测试，默认是轮询方式，负载均衡主要是service的作用。

3 daemonset方式部署ingress-nginx

daemonset 确保 ingress controller 的副本在集群中的每个节点上运行。这么做的原因如下：

它确保入口控制器保持服务流量高可用，即使某个节点发生故障。
它允许 ingress controller 在集群中的所有节点之间均匀分配流量，从而提高性能。
它允许 ingress controller 绑定到主机的网络命名空间，这对于某些功能（例如外部 dns）是必需的。

3.1 配置daemonset

3.1.1 helm部署方式

# 切换到chart目录
cd helm/ingress-nginx
vim values.yaml

修改ingress-nginx-controller的kind字段

 222   #kind: deployment
 223   kind: daemonset

更新helm配置

helm upgrade ingress-nginx . -n ingress-nginx

查看部署结果

[root@k8s-master1 ingress-nginx]# kubectl get pod,svc -owide -n ingress-nginx
name                                 ready   status    restarts   age    ip          node          nominated node   readiness gates
pod/ingress-nginx-controller-2bljr   1/1     running   0          61s    10.10.0.1   k8s-master1   <none>           <none>
pod/ingress-nginx-controller-2s59q   1/1     running   0          3m1s   10.10.0.4   k8s-worker1   <none>           <none>
pod/ingress-nginx-controller-dppwt   1/1     running   0          61s    10.10.0.5   k8s-worker2   <none>           <none>
pod/ingress-nginx-controller-gzshs   1/1     running   0          61s    10.10.0.3   k8s-master3   <none>           <none>
pod/ingress-nginx-controller-hbj9d   0/1     running   0          61s    10.10.0.2   k8s-master2   <none>           <none>

name                                         type        cluster-ip       external-ip   port(s)                      age    selector
service/ingress-nginx-controller             nodeport    10.245.133.125   <none>        80:30080/tcp,443:30443/tcp   3m2s   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-admission   clusterip   10.245.236.128   <none>        443/tcp                      3m2s   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

看到每个节点都部署了一个ingress-controller的pod

3.1.2 yaml文件部署方式

修改这行即可

396 #kind: deployment
397 kind: daemonset

验证方法如4.2节，这里就不重复叙述了。

4 结论

ingress-nginx 是 kubernetes 中强大而灵活的流量管理工具，能够有效地路由外部请求到内部服务。通过本文的讲解与实践示例，您应该能够快速上手并在自己的集群中实现 ingress-nginx 的配置与管理。继续探索 ingress-nginx 的更多高级功能，将使您的应用架构更加高效和安全。