当前位置： 代码迷 >> 综合 >> OpenStack SSL (by joshua)
  详细解决方案

OpenStack SSL (by joshua)

热度：89   发布时间：2023-12-13 09:08:09.0

版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明 (作者：张华 发表于：2018-02-07)

OpenStack Charm如何支持SSL

OpenStack Charm需为每个OpenStack服务配置证书（/var/lib/keystone/juju_ssl/, /usr/local/share/ca-certificates/)与https endpoint，同时也会生成/etc/apache2/ssl/keystone配置。

  1. 一种方式有证书就直接指定证书. 注意：因要避免SNI问题所以必须使用hostname而不是ip来创建证书，这样除了配置证书还得配置hostname
juju config <openstack-charm> os-admin-hostname='X.xxx.com' os-public-hostname='X.xxx.com' os-internal-hostname='X.xxx.com' ssl_ca='$cat ~/ssl_ca.crt' ssl_cert='$cat ~/ssl_cert.crt' ssl_key=''$cat ~/ssl_key.key'
juju config keystone ssl_ca | base64 --decode > keystone_CA.crt
we can see that the .crt file is expired:
# cat keystone_CA.crt | openssl x509 -noout -enddate
notAfter=Dec 20 16:04:05 2019 GMT
  1. 第二种方式是由charm来自动生成证书。如果用户没有指定ssl_cert与ssl_key，那么is_cert_provided_in_config=False，charm将自动生成证书。
def is_cert_provided_in_config():cert = config('ssl_cert')key = config('ssl_key')return bool(cert and key)

具体命令如下，也可参考：https://paste.ubuntu.com/26533565/

NOTE:  vboth use-https=true and https-serveice-endpoints=true are only used self-signed cert - https://review.opendev.org/#/c/555028/ 
# deprecation - https://review.openstack.org/#/c/560915/
# now we can use：xxx.sh -s xenial -r queens --create-model --name ssl:stsstack --num-compute 1 --openstack-dashboard --ssl --nova-console --runjuju config keystone use-https=true
juju config keystone https-service-endpoints=true
tree /var/lib/keystone/juju_ssl/
grep -r 'ssl' /etc/keystone/
grep -r 'ssl' /etc/nova/  #on nova node
select * from endpoint;
juju run --unit keystone/0 "relation-ids identity-service"
juju run --unit keystone/0 "relation-list -r identity-service:13"
juju run --unit keystone/0 "relation-get -r identity-service:13 - nova-cloud-controller/0"export OS_REGION_NAME=RegionOne
export OS_PROJECT_NAME=admin
export OS_PASSWORD=openstack
export OS_AUTH_URL=https://10.5.0.28:5000/v2.0
export OS_USERNAME=admin
openstack --insecure endpoint list
openstack --insecure server list
openstack --insecure --debug server list
openstack --insecure --debug volume list

OpenStack HA Charm如何支持SSL

如果如nova-cloud-controller服务前又运行了下列命令添加了HA服务时呢？

juju add-unit nova-cloud-controller -n 2  
juju deploy hacluster ncc-hacluster --series xenial  
juju add-relation nova-cloud-controller ncc-hacluster  
juju config nova-cloud-controller vip=10.5.104.1

corosync将通过下列配置将VIP=10.5.104.1创建在juju-2f3cc6-mitaka-10上。

root@juju-2f3cc6-mitaka-10:~# crm status
Last updated: Wed Feb  7 04:18:31 2018		Last change: Wed Feb  7 04:09:33 2018 by hacluster via crmd on juju-2f3cc6-mitaka-10
Stack: corosync
Current DC: juju-2f3cc6-mitaka-6 (version 1.1.14-70404b0) - partition with quorum
3 nodes and 4 resources configured
Online: [ juju-2f3cc6-mitaka-10 juju-2f3cc6-mitaka-11 juju-2f3cc6-mitaka-6 ]
Full list of resources:Resource Group: grp_nova_vipsres_nova_ens3_vip	(ocf::heartbeat:IPaddr2):	Started juju-2f3cc6-mitaka-10Clone Set: cl_nova_haproxy [res_nova_haproxy]Started: [ juju-2f3cc6-mitaka-10 juju-2f3cc6-mitaka-11 juju-2f3cc6-mitaka-6 ]root@juju-2f3cc6-mitaka-10:~# ip addr show ens3 |grep 10.5.104.1inet 10.5.104.1/16 brd 10.5.255.255 scope global secondary ens3

同时，三个HA节点上的haproxy的配置(/etc/haproxy/haproxy.cfg)将创建代理。

frontend tcp-in_nova-api-os-computebind *:8774bind :::8774acl net_10.5.0.15 dst 10.5.0.15/255.255.0.0use_backend nova-api-os-compute_10.5.0.15 if net_10.5.0.15default_backend nova-api-os-compute_10.5.0.15
backend nova-api-os-compute_10.5.0.15balance leastconnserver nova-cloud-controller-1 10.5.0.15:8764 checkserver nova-cloud-controller-0 10.5.0.37:8764 checkserver nova-cloud-controller-2 10.5.0.52:8764 check

最重要的是，三个HA节点上Apache都将设置代理“RequestHeader set X-Forwarded-Proto “https””，完整的配置如下:

root@juju-2f3cc6-mitaka-10:~# cat /etc/apache2/sites-available/openstack_https_frontend.conf 
Listen 8764
<VirtualHost 10.5.0.15:8764>ServerName 10.5.104.1SSLEngine onSSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2SSLCipherSuite HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUMSSLCertificateFile /etc/apache2/ssl/nova/cert_10.5.104.1# See LP 1484489 - this is to support <= 2.4.7 and >= 2.4.8SSLCertificateChainFile /etc/apache2/ssl/nova/cert_10.5.104.1SSLCertificateKeyFile /etc/apache2/ssl/nova/key_10.5.104.1ProxyPass / http://localhost:8754/ProxyPassReverse / http://localhost:8754/ProxyPreserveHost onRequestHeader set X-Forwarded-Proto "https"
</VirtualHost>
<Proxy *>Order deny,allowAllow from all
</Proxy>
<Location />Order allow,denyAllow from all
</Location>

OpenStack Horizon和Apache代理之间如何支持SSL

openstack horizon使用了Django框架，Django中有如下代码（https://github.com/django/django/blob/2.0.2/django/http/request.py#L191），

    @propertydef scheme(self):if settings.SECURE_PROXY_SSL_HEADER:try:header, value = settings.SECURE_PROXY_SSL_HEADERexcept ValueError:raise ImproperlyConfigured('The SECURE_PROXY_SSL_HEADER setting must be a tuple containing two values.')if self.META.get(header) == value:return 'https'return self._get_scheme()

它会根据django中是否配置了SECURE_PROXY_SSL_HEADER，然后比较和header中传过来的SECURE_PROXY_SSL_HEADER(self.META.get(header))是否相等, 若相等才返回https给前端代理。所以这说明需要做两件事情：

  1. 一是在django的conf/global_settings.py模板中配置： SECURE_PROXY_SSL_HEADER = (‘HTTP_X_FORWARDED_PROTO’, ‘https’)
  2. 二是前端代理apache中的配置传过来名为X-Forwarded-Proto的header: RequestHeader set X-Forwarded-Proto “https”
    更多见： https://design.canonical.com/2015/08/django-behind-a-proxy-fixing-absolute-urls/

OpenStack其他工程和Apache代理之间如何支持SSL

至于其它openstack工程如nova则需要使用olso.middleware中的http_proxy_to_wsgi.py(https://github.com/openstack/oslo.middleware/blob/stable/ocata/oslo_middleware/http_proxy_to_wsgi.py)用于解析前端代理apache传过来的X-Forwarded-Proto

            # World before RFC7239forwarded_proto = req.environ.get("HTTP_X_FORWARDED_PROTO")if forwarded_proto:req.environ['wsgi.url_scheme'] = forwarded_proto

nova-api中再根据wsgi.url_scheme（https://github.com/openstack/nova/blob/stable/ocata/nova/api/openstack/urlmap.py）决定是使用80还是ssl的443
注意: 感觉是nova-api再转发或调用其他服务时如果没X-Forwarded-Proto的话, 之前的https就变成http了.

            if environ['wsgi.url_scheme'] == 'http':port = '80'else:port = '443'

配置http_proxy_to_wsgi的方法是在/etc/nova/api-paste.ini中添加http_proxy_to_wsgi（ eg: https://bugs.launchpad.net/keystone/+bug/1590608)

[composite:openstack_compute_api_v21]
use = call:nova.api.auth:pipeline_factory_v21
noauth2 = cors http_proxy_to_wsgi compute_req_id faultwrap sizelimit noauth2 osapi_compute_app_v21
keystone = cors http_proxy_to_wsgi compute_req_id faultwrap sizelimit authtoken keystonecontext osapi_compute_app_v21
[filter:http_proxy_to_wsgi]
paste.filter_factory = oslo_middleware.http_proxy_to_wsgi:HTTPProxyToWSGI.factory

附录 - 原生OpenStack配置SSL的方法

1, Create ssl certifate$ sudo keystone-manage ssl_setup --keystone-user keystone --keystone-group keystone
$ chown -R keystone:keystone /etc/keystone/ssl
$ sudo ls /etc/keystone/ssl/certs
01.pem	ca.pem	index.txt  index.txt.attr  index.txt.old  keystone.pem	openssl.conf  req.pem  serial  serial.old
$ sudo ls /etc/keystone/ssl/private
cakey.pem  keystonekey.pemopenssl genrsa -out /etc/keystone/ssl/private/cakey.pem 1024
openssl req -new -x509 -extensions v3_ca -key /etc/keystone/ssl/private/cakey.pem -out /etc/keystone/ssl/certs/ca.pem -days 3650 -config /etc/keystone/ssl/certs/openssl.conf -subj /C=US/ST=Unset/L=Unset/O=Unset/CN=localhost
openssl genrsa -out /etc/keystone/ssl/private/keystonekey.pem 1024
openssl req -key /etc/keystone/ssl/private/keystonekey.pem -new -out /etc/keystone/ssl/certs/req.pem -config /etc/keystone/ssl/certs/openssl.conf -subj /C=US/ST=Unset/L=Unset/O=Unset/CN=localhost
openssl ca -batch -out /etc/keystone/ssl/certs/keystone.pem -config /etc/keystone/ssl/certs/openssl.conf -days 3650d -cert /etc/keystone/ssl/certs/ca.pem -keyfile /etc/keystone/ssl/private/cakey.pem -infiles /etc/keystone/ssl/certs/req.pem2, Keystone ssl configuration# vi /etc/keystone/keystone.conf
[signing]
certfile = /var/lib/keystone/juju_ssl/pki/certs/signing_cert.pem
keyfile = /var/lib/keystone/juju_ssl/pki/privates/signing_key.pem
ca_certs = /var/lib/keystone/juju_ssl/pki/certs/ca.pem
ca_key = /var/lib/keystone/juju_ssl/pki/certs/ca_key.pemexport OS_AUTH_URL=https://{keystoneHost}:5000/v2.0# create new https endpoint
keystone endpoint-create
--service keystone --region RegionOne --publicurl
https://{keystoneHost}:5000/v2.0 --internalurl
https://{keystoneHost}:35357/v2.0 --adminurl
https://{keystoneHost}:35357/v2.0# delete old http endpoint
keystone –-insecure endpoint-delete {old endpoint id}3, Nova ssl configuration# configure nova to use https to connect keystone
# vi /etc/nova/api-paste.ini
auth_uri = https://10.1.0.92:5000/v2.0
auth_protocol = https
insecure = Trueservice openstack-nova-api restart
service openstack-nova-compute restart
service openstack-nova-scheduler restart
service openstack-nova-cert restart
service openstack-nova-conductor restart
service openstack-nova-consoleauth restart# vi /etc/nova/nova.conf [DEFAULT]
enabled_ssl_apis=osapi_compute
ssl_cert_file=/etc/nova/ssl/keystone.pem
ssl_key_file=/etc/nova/ssl/keystonekey.pem# test it, at this time other component has not be configured to use https, so use --insecure
nova --insecure hypervisor-list# copy ssl certifate for nova
# mkdir /etc/nova/ssl
# cp /etc/keystone/ssl/certs/keystone.pem /etc/nova/ssl/
# cp /etc/keystone/ssl/private/keystonekey.pem /etc/nova/ssl/
# chown -R nova:nova /etc/nova/ssl/# create new https endpoint for nova
# keystone --insecure
endpoint-create --service nova --region RegionOne --publicurl
"https://{novaHost}:8774/v2/%(tenant_id)s" --internalurl
"https://{novaHost}:8774/v2/%(tenant_id)s" --adminurl
"https://{novaHost}:8774/v2/%(tenant_id)s"

20200831更新

目前自动产生证书的代码已经被删除（https://review.openstack.org/#/c/560915/），所以只能自定义证书．可以通过下列命令快速搭建ssl + horizon + novnc环境.

./generate-bundle.sh -s xenial -r queens --create-model --name ssl:stsstack --num-compute 1 --openstack-dashboard --ssl --nova-console
#modify ./b/ssl/o/nova-console.yaml for the error 'relation ["nova-cloud-controller" "memcached"] is defined more than once'
./generate-bundle.sh --name ssl:stsstack --replay --run

但是novarc有点问题，source之后只能运行

nova --insecure hypervisor-list

要想去掉–insecure，还需要：

export OS_CACERT=/home/xxx/openstack/ssl/openstack-ssl/results/cacert.pem

但依然有点问题，唯独horizon没有添加ssl支持，继续：

juju config openstack-dashboard enforce-ssl=true
ssl_results=/home/ubuntu/stsstack-bundles/openstack/ssl/openstack-ssl/results
juju config openstack-dashboard ssl_ca=`base64 ${ssl_results}/cacert.pem| tr -d '\n'`
juju config openstack-dashboard ssl_cert=`base64 ${ssl_results}/servercert.pem| tr -d '\n'`
juju config openstack-dashboard ssl_key=`base64 ${ssl_results}/serverkey.pem| tr -d '\n'`

最后开内网v皮恩之后，访问：

sudo sshuttle --python=/usr/bin/python2.7 -r ubuntu@bastion 10.5.0.0/16 -D
https://10.5.2.147  #make sure not use proxy, admin_domain/admin/openstack

20210127更新 - ssl & vault

./generate-bundle.sh -s xenial -r queens --create-model --name ssl:stsstack --num-compute 1 --openstack-dashboard --nova-console --vault
./generate-bundle.sh --name ssl:stsstack --replay --run
./tools/vault-unseal-and-authorise.sh
juju config openstack-dashboard enforce-ssl=true
juju add-relation openstack-dashboard:certificates vault:certificates
juju config nova-cloud-controller console-access-protocol=novnc
./configure
source novarc
openstack --insecure endpoint list
openstack endpoint list
./tools/instance_launch.sh 1 bionic
#add 'socks-proxy 127.0.0.1 7078' into openxxx.conf then start it by 'openxxx --config xxx.conf --daemon'
#sudo sshuttle --python=/usr/bin/python2.7 -r ubuntu@bastion 10.5.0.0/16 10.0.0.0/16 -l 0.0.0.0:0 -D
#access horizon (NOTE: don't proxy it in SwitchOmega), and don't run sshuttle in bastion as well
#https://10.5.2.248  admin_domain/admin/openstack
nova get-vnc-console bionic-032738 novnc
nmap --script ssl-enum-ciphers -p 6080 10.5.1.161nova-cloud-controller/0
#https://bugs.launchpad.net/ubuntu/+source/python-eventlet/+bug/1904988
echo 'deb http://archive.ubuntu.com/ubuntu/ xenial-proposed restricted main multiverse universe' |sudo tee -a /etc/apt/sources.list
sudo apt update
sudo apt-cache policy python-eventlet
sudo apt-get install python-eventlet/xenial-proposed
sudo systemctl restart nova-novncproxyubuntu@zhhuabj-bastion:~/stsstack-bundles/openstack$ nmap --script ssl-enum-ciphers -p 6080 10.5.1.161 |grep -i tlsv
|   TLSv1.0: 
|   TLSv1.1: 
|   TLSv1.2:

备份网址

https://zhhuabj.github.io/2018/02/07/OpenStack-SSL/

  相关解决方案

代码迷 - 您身边的软件异常，代码异常，编程异常助手（发现，解决，分享）
Copyright © 2009-2013 DAIMAMI.COM. All rights reserved.