Write by lyc at 2022-4-8
ingress-nginx 官网
Nginx Ingress 高并发实践
k8s的 Nginx Ingress 调优

ingress-nginx 性能调优

1.ingress-nginx 内核参数调优

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
....
# Source: ingress-nginx/templates/controller-deployment.yaml
....

      initContainers:
      - name: setsysctl
        image: busybox
        securityContext:
          privileged: true
        command:
        - sh
        - -c
        - |
          sysctl -w net.core.somaxconn=65535
          sysctl -w net.ipv4.ip_local_port_range="1024 65535"
          sysctl -w net.ipv4.tcp_max_tw_buckets=55000
          sysctl -w net.ipv4.tcp_tw_reuse=1
          sysctl -w fs.file-max=1048576
          sysctl -w net.ipv4.tcp_fin_timeout=15
          sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
...

2.ingress-nginx Global 全局参数调优

根据业务场景来调节。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  allow-snippet-annotations: 'true'
  # Nginx 日志
  disable-access-log: "true"
  log-format-escape-json: "true"
  log-format-upstream: '{"time_local": "$time_local","http_host": "$http_host","remote_addr": "$remote_addr","remote_port": "$remote_port","remote_user": "$remote_user","request": "$request","status": "$status","request_time": "$request_time","request_body": "$request_body","body_bytes_sent": "$body_bytes_sent","http_referer": "$http_referer","upstream_addr": "$upstream_addr","upstream_response_time": "$upstream_response_time","http_x_forwarded_for": "$http_x_forwarded_for","scheme": "$scheme","gzip_ratio": "$gzip_ratio","http_user_agent": "$http_user_agent"}'
  # 性能
  keep-alive-requests: "10000"
  upstream-keepalive-connections: "1000"
  max-worker-connections: "65536"
  # 反向代理
  proxy-body-size: "10m"
  proxy-connect-timeout: "10"
  proxy-send-timeout: "30"
  proxy-read-timeout: "30"
  # 开启压缩
  enable-brotli: "true"

兼容支持旧版本 TLS（可选）

由官方文档 ingress configmap ssl-protocols 得知，ingress-nginx 默认只支持 TLSv1.3 TLSv1.2，部分老的业务需要向下兼容，所以我们需要开启 TLSv1.3 TLSv1.2 TLSv1.1 TLSv1 的支持。

使用 SSL Configuration Generator 配置文件生成器，输入 ingress-nginx 内的 nginx版本、openssl版本，得到 ssl-ciphers 串。

在 YAML 下的 ConfigMap 下加入如下配置：

1
2
3
# TLS版本
ssl-protocols: "TLSv1.3 TLSv1.2 TLSv1.1 TLSv1"
ssl-ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA"

使用以下 https 在线测评站点来检测结果：

3.ingress-nginx http级参数调优（可选）

ingress.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ${K8S_APP_LABEL}
  labels:
    pro: ${K8S_PRO_LABEL}
    app: ${K8S_APP_LABEL}
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/enable-access-log: "true"
    nginx.ingress.kubernetes.io/configuration-snippet: |
       access_log /var/log/nginx/${K8S_INGRESS_HOST}.access.log upstreaminfo;
       error_log  /var/log/nginx/${K8S_INGRESS_HOST}.error.log;
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - ${K8S_INGRESS_HOST}
    secretName: ${K8S_INGRESS_TLS}
  rules:
  - host: ${K8S_INGRESS_HOST}
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: ${K8S_APP_LABEL}
            port:
              number: 80

ingress.yaml（proxy_body_size=100m）

适用于 100m 以内大文件下载的反向代理。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ${K8S_APP_LABEL}
  labels:
    pro: ${K8S_PRO_LABEL}
    app: ${K8S_APP_LABEL}
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/enable-access-log: "true"
    nginx.ingress.kubernetes.io/configuration-snippet: |
       access_log /var/log/nginx/${K8S_INGRESS_HOST}.access.log upstreaminfo;
       error_log  /var/log/nginx/${K8S_INGRESS_HOST}.error.log;
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
    nginx.ingress.kubernetes.io/client-body-buffer-size: "128k"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "75s"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "75s"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "75s"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "64k"
    nginx.ingress.kubernetes.io/proxy-buffers-number: "32"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - ${K8S_INGRESS_HOST}
    secretName: ${K8S_INGRESS_TLS}
  rules:
  - host: ${K8S_INGRESS_HOST}
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: ${K8S_APP_LABEL}
            port:
              number: 80