我要评论nginx-controller是什么呢?
- 它是一个能调度nginx的一个kubernetes operator,它能监听用户创建,更新,删除nginxconf对象,来调度本地的nginx实现配置的动态更新。如添加新的代理(http,https,tcp,udp),缓存(浏览器缓存,本地缓存),ssl证书(配置本身,configmap,secret),更新,删除等
- 它使用nginx本身的配置文件(如nginx.conf)来作为配置参数,使nginx配置更透明,易于维护
安装之前准备
- 你得有一个k8s环境
- 创建crd
如果你的版本是1.29以下的版本,请删掉crd里面所有有关x-kubernetes-validations的部分
crd.yaml
apiversion: apiextensions.k8s.io/v1
kind: customresourcedefinition
metadata:
name: nginxconfs.stable.lhstack.com
spec:
names:
kind: nginxconf
plural: nginxconfs
singular: nginxconf
listkind: nginxconflist
shortnames:
- ncf
group: stable.lhstack.com
scope: namespaced
versions:
- name: v1
served: true
storage: true
schema:
openapiv3schema:
type: object
description: "nginx 对应http/stream组中include哪一项引入的配置"
x-kubernetes-validations:
- rule: "has(self.spec) && has(self.spec.config)"
message: "spec.config参数为必填项"
- rule: "(self.spec.configtype == 'custom' && size(self.spec.customconfigpath) > 0) || (has(self.spec.configtype) && self.spec.configtype != 'custom') || !has(self.spec.configtype)"
message: "spec.configtype是custom时，spec.customconfigpath参数为必填项"
properties:
spec:
type: object
required:
- config
properties:
additions:
type: object
description: "附加configmap,secret,文本内容到指定路径文件中,使用场景: 如tls证书"
properties:
values:
type: array
description: "将items.value中的内容输出到容器指定路径"
items:
type: object
x-kubernetes-validations:
- rule: "size(self.value) != 0 && size(self.path) != 0"
message: "values.value,values.value参数为必填项"
properties:
value:
type: string
description: "要输出到文件的内容"
path:
type: string
description: "输出目标路径"
secrets:
type: array
description: "将secret中的内容输出到容器指定路径"
items:
type: object
x-kubernetes-validations:
- rule: "(has(self.name) && has(self.path)) || (has(self.name) && has(self.items))"
message: "(secrets.path,secrets.name)或者(secrets.items,secrets.name)参数为必填项"
- rule: "(has(self.path) && !has(self.items)) || (!has(self.path) && has(self.items))"
message: "secrets.path和secrets.items参数不能并存,只能二选一"
properties:
path:
type: string
description: "输出目标路径,同items参数不能并存,此路径必须是一个目录,不存在即创建目录(多级目录会同时创建)"
name:
type: string
description: "secret名称"
namespace:
type: string
description: "secret所在命名空间"
items:
type: array
description: "secret中每一项,同path参数不能并存"
items:
type: object
x-kubernetes-validations:
- rule: "size(self.key) != 0 && size(self.path) != 0"
message: "items.key和items.path不能为空"
properties:
key:
type: string
description: "secret项中的key"
path:
type: string
description: "secret中key的value值需要输出到的目标文件路径,此路径必须是一个文件地址,不存在即创建文件(多级目录会同时创建目录)"
configmaps:
type: array
description: "将configmap中的内容输出到容器指定路径"
items:
type: object
x-kubernetes-validations:
- rule: "(has(self.name) && has(self.path)) || (has(self.name) && has(self.items))"
message: "(configmaps.path,configmaps.name)或者(configmaps.items,configmaps.name)参数为必填项"
- rule: "(has(self.path) && !has(self.items)) || (!has(self.path) && has(self.items))"
message: "configmaps.path和configmaps.items参数不能并存,只能二选一"
properties:
path:
type: string
description: "输出目标路径,同items参数不能并存,此路径必须是一个目录,不存在即创建目录(多级目录会同时创建)"
name:
type: string
description: "configmap名称"
namespace:
type: string
description: "configmap所在命名空间"
items:
type: array
description: "configmap中每一项,同path参数不能并存"
items:
type: object
x-kubernetes-validations:
- rule: "size(self.key) != 0 && size(self.path) != 0"
message: "items.key和items.path不能为空"
properties:
key:
type: string
description: "configmap项中的key"
path:
type: string
description: "configmap中key的value值需要输出到的目标文件路径,此路径必须是一个文件地址,不存在即创建文件(多级目录会同时创建目录)"
customconfigpath:
type: string
description: "当configtype=custom时才生效,定义配置写入到指定目录下面"
configtype:
description: "配置类型,可选值 http,stream,custom,default: http"
enum:
- http
- stream
- custom
type: string
config:
type: string
description: |
配置内容:
server {
listen 80;
listen [::]:80;
server_name localhost;
#access_log /var/log/nginx/host.access.log main;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
# proxy the php scripts to apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the php scripts to fastcgi server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param script_filename /scripts$fastcgi_script_name;
# include fastcgi_params;
#}
# deny access to .htaccess files, if apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
---- 执行命令,创建crd
kubectl apply -f crd.yaml
- 然后就可以使用
kubectl explain nginxconf查看对应的文档了，如:
安装nginx-controller
这里我使用deployment来部署nginx-controller
deployment.yaml
apiversion: v1
kind: namespace
metadata:
name: ingress
---
apiversion: v1
kind: serviceaccount
metadata:
name: nginx-controller
namespace: ingress
---
apiversion: rbac.authorization.k8s.io/v1
kind: clusterrolebinding
metadata:
name: nginx-controller
namespace: ingress
subjects:
- kind: serviceaccount
name: nginx-controller
namespace: ingress
roleref:
apigroup: rbac.authorization.k8s.io
kind: clusterrole
name: cluster-admin
---
apiversion: apps/v1
kind: deployment
metadata:
name: nginx-controller
namespace: ingress
spec:
replicas: 2
selector:
matchlabels:
app: ingress
template:
metadata:
labels:
app: ingress
spec:
serviceaccountname: nginx-controller
containers:
- name: controller
image: lhstack/nginx-controller:latest
imagepullpolicy: ifnotpresent
ports:
- containerport: 80
name: "http"
protocol: "tcp"
- containerport: 443
name: "https"
protocol: "tcp"
readinessprobe:
httpget:
port: 9099
path: /readyz
successthreshold: 1
failurethreshold: 3
timeoutseconds: 3 #请求超时
periodseconds: 30 #每隔30秒检查一次
initialdelayseconds: 5 #5秒之后开始检测
livenessprobe:
httpget:
port: 9099
path: /healthz
successthreshold: 1
failurethreshold: 3
timeoutseconds: 3 #请求超时
periodseconds: 60 #每隔60秒检查一次
initialdelayseconds: 5 #5秒之后开始检测
env:
- name: kube_namespace
value: "ingress" # 这里用命名空间隔离配置,意味着只有ingress命名空间下的nginxconf才会生效,如果不设置或者为空,就会监听所有命名空间下的配置
resources:
requests:
memory: 32mi
cpu: 10m
limits:
memory: 64mi
cpu: 10m
---
apiversion: v1
kind: service
metadata:
name: ingress
namespace: ingress
spec:
selector:
app: ingress
type: nodeport
clusterip: 10.43.80.80 #这里固定一下ip,方便使用dns指向指定ip,前提是需要安装dns服务
ports:
- port: 80
name: http
protocol: tcp
nodeport: 30080
- port: 443
name: https
protocol: tcp
nodeport: 30443然后就可以通过命令看到启动了两个容器
由于我本地的环境和默认的不太一样,所以ip,type可能不相同,但是影响不大,我相信你们也能理解
添加一个http代理
这里代理百度试试，由于我已经安装了dns服务，就可以通过域名直接访问
baidu-nginx-conf.yaml
apiversion: stable.lhstack.com/v1
kind: nginxconf
metadata:
name: baidu-web
namespace: default #我本地配置监听的是default命名空间
spec:
config: |
server {
server_name baidu.lhstack.com;
listen 80;
gzip on;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
gzip_min_length 1000;
gzip_comp_level 6;
gzip_proxied any;
gzip_vary on;
location / {
proxy_pass https://www.baidu.com;
proxy_http_version 1.1;
}
}可以看到创建成功了，日志也已经检测到了
现在访问浏览器试试
增加tls证书支持
生成证书
这里我使用cfssl+cfssljson生成证书
- 编写ca-config.json
{
"signing": {
"default": {
"expiry": "876000h"
},
"profiles": {
"lhstack": {
"expiry": "876000h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}- 编写ca-csr.json
{
"cn": "lhstack",
"key#34;: {
"algo": "ecdsa",
"size": 256
},
"names": [
{
"c": "cn",
"st": "chengdu",
"l": "chengdu",
"o":"nginxconf",
"ou":"lhstack"
}
]
}- 生成ca证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
- 编写server-crs.json配置
{
"cn": "lhstack.com",
"hosts":[
"*.lhstack.com"
],
"key": {
"algo": "ecdsa",
"size": 256
},
"names": [
{
"c": "cn",
"st": "chengdu",
"l": "chengdu",
"o":"nginxconf",
"ou":"lhstack"
}
]
}- 生成服务端证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=lhstack server-csr.json | cfssljson -bare server
浏览器导入ca证书
google浏览器 设置->隐私和安全->安全->管理证书->受信任的根证书颁发机构->导入->浏览->将文件类型选择为所有类型->选择ca.pem->导入即可
为代理服务添加证书
使用value输出证书
- 编写配置
apiversion: stable.lhstack.com/v1
kind: nginxconf
metadata:
name: baidu-web
namespace: default #我本地配置监听的是default命名空间
spec:
additions:
values:
- path: /opt/tls/baidu/tls.key
value: | #复制server-key.pem的内容
-----begin ec private key-----
mhccaqeeiat5gx3jgiezs/ummtkabnuazxzvjpm1g2huyre1aageoaogccqgsm49
awehouqdqgaeqosrm0qvmx/yt1wy6idp1mteqkncfdpn2hwlr8wtk8urfszdtec4
tvq5qheqxpadlnxbldx8e88ii/1l7mcgmg==
-----end ec private key-----
- path: /opt/tls/baidu/tls.crt
value: | #复制server.pem的内容
-----begin certificate-----
miicdjccahugawibagiubrgevdgozilz0gbxhywnapstwn4wcgyikozizj0eawiw
atelmakga1uebhmcq04xedaobgnvbagtb0nozw5nrhuxedaobgnvbactb0nozw5n
rhuxejaqbgnvbaotcw5naw54q29uzjeqma4ga1uecxmhbghzdgfjazeqma4ga1ue
axmhbghzdgfjazagfw0yndaymtkwnje0mdbaga8ymti0mdeynja2mtqwmfowbtel
makga1uebhmcq04xedaobgnvbagtb0nozw5nrhuxedaobgnvbactb0nozw5nrhux
ejaqbgnvbaotcw5naw54q29uzjeqma4ga1uecxmhbghzdgfjazeumbiga1ueaxml
bghzdgfjay5jb20wwtatbgcqhkjopqibbggqhkjopqmbbwncaaso5jezrbuxf/jp
vzjqionwzmsoqdx8m83afythzc0rxst+xl214li1wrmqf6pekaowdduuphwtzwgj
/uvuzycao4gamigxma4ga1uddweb/wqeawifodadbgnvhsuefjaubggrbgefbqcd
aqyikwybbquhawiwdaydvr0taqh/baiwadadbgnvhq4efgqunxnd1vf52cfi4jnf
mjdwjinrfkgwhwydvr0jbbgwfoauwt+onec6wltc+imeynhbxlrh7oawgaydvr0r
bbewd4inki5sahn0ywnrlmnvbtakbggqhkjopqqdagnjadbgaieahbqbwhu/9f6d
6e7s48ltk2gv4jhvfk27qpv1+e7gbdociqcyrc8+igo7ejlpzifiookkdfgsz5cz
hucqbhbgfgl0bg==
-----end certificate-----
config: |
server {
listen 80;
#配置80端口永久重定向443
server_name baidu.lhstack.com;
rewrite ^(.*)$ https://${server_name}$1 permanent;
}
server {
server_name baidu.lhstack.com;
listen 443 ssl http2;
client_max_body_size 50m;
ssl_certificate /opt/tls/baidu/tls.crt;
ssl_certificate_key /opt/tls/baidu/tls.key;
ssl_session_timeout 5m;
ssl_ciphers ecdhe-rsa-aes128-gcm-sha256:ecdhe:ecdh:aes:high:!null:!anull:!md5:!adh:!rc4;
ssl_protocols tlsv1 tlsv1.1 tlsv1.2 tlsv1.3;
ssl_prefer_server_ciphers on;
gzip on;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
gzip_min_length 1000;
gzip_comp_level 6;
gzip_proxied any;
gzip_vary on;
location / {
proxy_pass https://www.baidu.com;
proxy_http_version 1.1;
}
}更新配置
kubectl apply -f baidu-nginx-conf.yaml
可以看到，更新成功了
浏览器访问，可以看到，这里就变成https了
使用configmap输出证书删除之前的配置
kubectl delete -f baidu-nginx-conf.yaml
浏览器也不可访问,变成了我默认的服务
证书文件也清理干净了
- 编写配置
apiversion: v1
kind: configmap
metadata:
name: baidu-nginx-conf
namespace: kube-system
data:
tls.key: |
-----begin ec private key-----
mhccaqeeiat5gx3jgiezs/ummtkabnuazxzvjpm1g2huyre1aageoaogccqgsm49
awehouqdqgaeqosrm0qvmx/yt1wy6idp1mteqkncfdpn2hwlr8wtk8urfszdtec4
tvq5qheqxpadlnxbldx8e88ii/1l7mcgmg==
-----end ec private key-----
tls.crt: |
-----begin certificate-----
miicdjccahugawibagiubrgevdgozilz0gbxhywnapstwn4wcgyikozizj0eawiw
atelmakga1uebhmcq04xedaobgnvbagtb0nozw5nrhuxedaobgnvbactb0nozw5n
rhuxejaqbgnvbaotcw5naw54q29uzjeqma4ga1uecxmhbghzdgfjazeqma4ga1ue
axmhbghzdgfjazagfw0yndaymtkwnje0mdbaga8ymti0mdeynja2mtqwmfowbtel
makga1uebhmcq04xedaobgnvbagtb0nozw5nrhuxedaobgnvbactb0nozw5nrhux
ejaqbgnvbaotcw5naw54q29uzjeqma4ga1uecxmhbghzdgfjazeumbiga1ueaxml
bghzdgfjay5jb20wwtatbgcqhkjopqibbggqhkjopqmbbwncaaso5jezrbuxf/jp
vzjqionwzmsoqdx8m83afythzc0rxst+xl214li1wrmqf6pekaowdduuphwtzwgj
/uvuzycao4gamigxma4ga1uddweb/wqeawifodadbgnvhsuefjaubggrbgefbqcd
aqyikwybbquhawiwdaydvr0taqh/baiwadadbgnvhq4efgqunxnd1vf52cfi4jnf
mjdwjinrfkgwhwydvr0jbbgwfoauwt+onec6wltc+imeynhbxlrh7oawgaydvr0r
bbewd4inki5sahn0ywnrlmnvbtakbggqhkjopqqdagnjadbgaieahbqbwhu/9f6d
6e7s48ltk2gv4jhvfk27qpv1+e7gbdociqcyrc8+igo7ejlpzifiookkdfgsz5cz
hucqbhbgfgl0bg==
-----end certificate-----
---
apiversion: stable.lhstack.com/v1
kind: nginxconf
metadata:
name: baidu-web
namespace: default #我本地配置监听的是default命名空间
spec:
additions:
configmaps:
- name: baidu-nginx-conf
namespace: kube-system #不填默认使用default作为命名空间
path: /opt/tls/baidu
config: |
server {
listen 80;
#配置80端口永久重定向443
server_name baidu.lhstack.com;
rewrite ^(.*)$ https://${server_name}$1 permanent;
}
server {
server_name baidu.lhstack.com;
listen 443 ssl http2;
client_max_body_size 50m;
ssl_certificate /opt/tls/baidu/tls.crt;
ssl_certificate_key /opt/tls/baidu/tls.key;
ssl_session_timeout 5m;
ssl_ciphers ecdhe-rsa-aes128-gcm-sha256:ecdhe:ecdh:aes:high:!null:!anull:!md5:!adh:!rc4;
ssl_protocols tlsv1 tlsv1.1 tlsv1.2 tlsv1.3;
ssl_prefer_server_ciphers on;
gzip on;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
gzip_min_length 1000;
gzip_comp_level 6;
gzip_proxied any;
gzip_vary on;
location / {
proxy_pass https://www.baidu.com;
proxy_http_version 1.1;
}
}
- 更新配置
kubectl apply -f baidu-nginx-conf.yaml
浏览器也可以访问了
使用secret输出证书删除之前的配置
kubectl delete -f baidu-nginx-conf.yaml
证书文件也被清理干净
- 编写配置
apiversion: v1
kind: secret
metadata:
name: baidu-nginx-conf
namespace: kube-system
type: kubernetes/tls #data要求为base64格式
data:
tls.key: ls0tls1crudjtibfqybquklwqvrfietfws0tls0tck1iy0nbuuvfsufunwdym2pnsuvauy91bw10a0fitnvhelhavmpwbtfnmmh1wvjlmufbr2vvqw9hq0nxr1nnndkkqxdfsg9vuurrz0ffcu9tuk0wuvznwc95vdfxwtzprhaxbvrfcutuy2zeue4yafdmujh3des4vxjmc1pkdgvdnap0vne1cwhlcvhwqursblhiber4oeu4oeljlzfmn21jz21npt0kls0tls1ftkqgrumgufjjvkfursblrvktls0tlq==
tls.crt: ls0tls1crudjtibdrvjusuzjq0furs0tls0tck1jsunkakndqwh1z0f3sujbz0lvynjnzvzkz096suxamgdceeh5v25bchnuv240d0nnwullb1pjemowruf3sxckyvrfte1ba0dbmvvfqmhnq1ewnhhfrefpqmdovkjbz1rcme5vwlc1bljivxhfrefpqmdovkjby1rcme5vwlc1bgpssfv4rwpbuujntlzcqw9uq1c1bmfxntrrmjl1wmpfuu1bnedbmvvfq3hnsgjhahpkr0zqyxpfuu1bnedbmvvfckf4tuhir2h6zedgamf6qwdgdzb5tkrbeu1ua3doakuwturcyudbohlnvekwturfeu5qqtjnvff3tuzvd2juruwktufrr0exvuvcae1duta0eevequ9cz05wqkfnveiwtm9avzvuukhveevequ9cz05wqkfjveiwtm9avzvuukhveapfakfrqmdovkjbb1rdvzvuyvc1nfeyoxvaakvrtue0r0exvuvdee1iykdoemrhrmphekvvtujjr0exvuvbee1mcmjhahpkr0zqyxk1amiymhdxvefuqmdjcwhrak9quulcqmdncwhrak9quu1cqndoq0fbu281skv6ukjvegyvslakvlpqculpbldatvnvcwr4oe04m2fgwxriekmwcnhtdct4bdixnexpmvdybxfgnnbla0fpv2rkdvvqshduendnagovvxz1wnldyw80r2fnsudytue0r0exvwred0vcl3drruf3suzvrefkqmdovkhtvuvgakfvqmdnckjnruzcuwneckfrwulld1lcqlfvsef3sxdeqvlevliwvefrsc9cqul3qurbzejntlziutrfrmdrvw54tmqxvmy1mkngstrkbmykbupkv0pjtljma2d3shdzrfzsmgpcqmd3rm9bvxd0k29uzum2d0xuqytptuvzbmhieexysddpqxdhqvlevliwugpcqkv3rdrjtktpnxnhse4wwvdockxttnziveflqmdncwhrak9quvfeqwdoskfeqkdbauvbaejryldids85rjzkcjzfn3m0ogx0azjhdjrkahzmazi3uvbwmstln0dczg9dsvfdevjjoctjz083zwpscfpjzklpb0trzeznu1o1q1oksfvduwjoymdgz2wwqmc9pqotls0tluvorcbdrvjusuzjq0furs0tls0t
---
apiversion: stable.lhstack.com/v1
kind: nginxconf
metadata:
name: baidu-web
namespace: default #我本地配置监听的是default命名空间
spec:
additions:
secrets:
- name: baidu-nginx-conf
namespace: kube-system #不填默认使用default作为命名空间
path: /opt/tls/baidu
config: |
server {
listen 80;
#配置80端口永久重定向443
server_name baidu.lhstack.com;
rewrite ^(.*)$ https://${server_name}$1 permanent;
}
server {
server_name baidu.lhstack.com;
listen 443 ssl http2;
client_max_body_size 50m;
ssl_certificate /opt/tls/baidu/tls.crt;
ssl_certificate_key /opt/tls/baidu/tls.key;
ssl_session_timeout 5m;
ssl_ciphers ecdhe-rsa-aes128-gcm-sha256:ecdhe:ecdh:aes:high:!null:!anull:!md5:!adh:!rc4;
ssl_protocols tlsv1 tlsv1.1 tlsv1.2 tlsv1.3;
ssl_prefer_server_ciphers on;
gzip on;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
gzip_min_length 1000;
gzip_comp_level 6;
gzip_proxied any;
gzip_vary on;
location / {
proxy_pass https://www.baidu.com;
proxy_http_version 1.1;
}
}
- 更新配置
kubectl apply -f baidu-nginx-conf.yaml
浏览器也能正常访问
如果证书过期了，需要更新证书内容怎么办?
如果使用的configmap,secret保存的证书,那么就需要手动更新configmap,secret,然后使用以下命令:
触发所有nginxconf更新事件
这个指令会触发所有nginxconf事件,然后走更新流程,就会拉取最新的configmap,secret内容输出到指定路径
kubectl annotate nginxconf --all -a --overwrite updated=$(date +%s)
触发一个nginxconf更新事件
如果已知需要更新的nginxconf配置,则使用此命令触发更新事件即可
kubectl annotate -n {命名空间} nginxconf {nginxconf的名称} --overwrite update=$(date +%s)如果使用的value输出,则不需要,因为value本身是nginxconf对象中的某一个参数,当发生修改时,kubernetes会判定为nginxconf发生了改变,自然而然就会触发更新事件
由于部分图片违规，所以对域名做了遮掩处理，相关域名通过配置可查看
总结
以上为个人经验，希望能给大家一个参考，也希望大家多多支持代码网。
发表评论