k8s http/https nginx ingress (by quqi99)

作者：张华 发表于：2019-11-05
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明 (https://zhhuabj.blog.csdn.net)

问题

• L7 LB, 为了实现http到https的重定向, 需要将L4 LB换成L7 LB, 这样真实的client ip放在了x-forwarded-for header里, 客户真实的scheme放在了x-forwarded-proto里. 所以在L7 LB模式中, 并没有将tcp连接透传给nginx-ingress. NOTE: 这块可以使用"curl -H ‘X-Forwarded-Proto: https’ -H ‘X-Forwsarded-Host: newhost’ 192.168.99.135:8000"模拟代替.
• nginx-ingress, 默认的nginx-ingress里使用(proxy_set_header X-Forwarded-Proto $pass_access_scheme;)与(proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;)获取的是L7 LB的IP作为client ip, 以及将L7
• LB访问的scheme作为scheme. 这里应该改从header里取(http_x_forwarded_proto). backend, 如果backend利用了x-forwarded-*这些header来构造redirect url的话, 如果没有获取到x-forwarded-proto时, 它如果要继续转发(Http Redirect)的话, 可以就会出现url问题.

Client与nginx-ingress之间可以配置https，此时nginx-ignress与backends仍然可以配置http, ssl termination, ssl passthrough三种模式。若Client前面还有L7 LB那就是四种模式：

• client --(HTTP)–> ingress-controller --(HTTP)–> backend # http lb
• client --(HTTPs)–> ingress-controller --(HTTP)–> backend # ssl termination http
• client --(HTTPs)–> ingress-controller --(HTTPS)–> backend # ssl termination https (nginx.ingress.kubernetes.io/backend-protocol: “HTTPS” or nginx.ingress.kubernetes.io/secure-backends: “true”)
• client --(HTTPs)–> ingress-controller --(HTTPs)–> backend # ssl passthrough https (–enable-ssl-passthrough and nginx.ingress.kubernetes.io/ssl-passthrough: “true”)
• client --(HTTPs)–> L7 LB --(use-forwarded-headers=true, X-Forwarded-Proto=https)–> ingress-controller --(HTTPs)–> backend # ssl termination with F5 - https://bugs.launchpad.net/bugs/1842286

http ingress

client --(HTTP)–> ingress-controller --(HTTP)–> backend # http lb

./generate-bundle.sh --docker
juju deploy ./b/kubernetes.yaml --overlay ./b/o/etcd.yaml --overlay ./b/o/etcd-easyrsa.yaml --overlay ./b/o/easyrsa.yaml --overlay ./b/o/k8s-etcd.yaml --overlay ./b/o/k8s-lb.yaml --overlay ./b/o/k8s-cni-flannel.yaml --overlay ./b/o/k8s-easyrsa.yaml --overlay ./b/o/k8s-docker.yaml --channel=stable
juju trust openstack-integrator
juju scp kubernetes-master/0:config ~/.kube/config
juju run-action etcd/0 health --wait
juju scp kubernetes-master/0:config ~/.kube/config
kubectl get pods --all-namespaceskubectl get services --namespace=ingress-nginx-kubernetes-worker default-http-backend-kubernetes-worker -o yaml
kubectl get daemonset --namespace=ingress-nginx-kubernetes-worker nginx-ingress-controller-kubernetes-worker -o yamlcontainers:- args:- /nginx-ingress-controller- --configmap=$(POD_NAMESPACE)/nginx-configuration- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services- --udp-services-configmap=$(POD_NAMESPACE)/udp-services- --annotations-prefix=nginx.ingress.kubernetes.io- --enable-ssl-chain-completion=False- --enable-ssl-passthrough=False#deploy http ingress
kubectl run nginx --image=nginx --port 80
kubectl expose deployment nginx --port=80 --target-port=80 --type=NodePort
cat <<EOF >ingress-test.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:name: ingress-service
spec:rules:- host: "quqi.com"http:paths:- backend:serviceName: nginxservicePort: 80
EOF
kubectl apply -f ./ingress-test.yaml
kubectl describe ingresses ingress-service
kubectl run -i --rm --tty client --image=tutum/curl
$ kubectl describe service nginx
IP:                       10.152.183.250
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  31463/TCP
Endpoints:                10.1.60.3:80
$ juju status |grep worker/0
kubernetes-worker/0*      active    idle   4        10.5.0.53       80/tcp,443/tcp  Kubernetes worker running.#curl http(s)://<worker-ip>:NodePort>/ -H 'Host: quqi.com'
curl http://10.5.0.53:31463/ -H 'Host: quqi.com'
curl http://10.5.0.53:31463/ -H 'Host: quqi.com' -I -L

ssl termination http

client --(HTTPs)–> ingress-controller --(HTTP)–> backend # ssl termination http
ssl termination可参见：https://www.howtoing.com/how-to-set-up-nginx-load-balancing-with-ssl-termination/
在k8s中， 如果nginx-ingress前面还有LB如F5的话，nginx-ingress并不是和client直接相连的，client传给F5的链接里的https标记会最终设置成X-Forwarded-Proto=https, 所以nginx-ingress这端需要在configmap中设置use-forwarded-headers=true(kubectl -n ingress-nginx-kubernetes-worker edit cm nginx-configura
tion), If true, nginx-ingress passes the incoming X-Forwarded-* (eg: X-Forwarded-Proto) headers to upstreams. Use this option when nginx is behind another L7 proxy / LB that is setting these headers.

openssl genrsa -passout pass:password -out ca.key
openssl req -x509 -passin pass:password -new -nodes -key ca.key -days 3650 -out ca.crt -subj "/C=CN/ST=BJ/O=STS/CN=ssltest.quqi.com"
openssl genrsa -passout pass:password -out server.key
openssl req -new -key server.key -out server.csr -subj "/C=CN/ST=BJ/O=STS/CN=ssltest.quqi.com"
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcr