作者：张华 发表于：2015-10-01
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明
( http://blog.csdn.net/quqi99)
VIP: 10.0.1.6
FIP: 192.168.101.4
VM1: 10.0.1.3
VM2: 10.0.1.4
1，安装
使用devstack安装时添加 ENABLED_SERVICES+=,q-fwaas 即可。
2, 配置文件
a, /etc/neutron/neutron.conf
[DEFAULT]
service_plugins = neutron.services.l3_router.l3_router_plugin.L3RouterPlugin,neutron_lbaas.services.loadbalancer.plugin.LoadBalancerPlugin,neutron_vpnaas.services.vpn.plugin.VPNDriverPlugin,neutron_fwaas.services.firewall.fwaas_plugin.FirewallPlugin
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
b, /etc/neutron/neutron_lbaas.conf
[service_providers]
service_provider=LOADBALANCER:Haproxy:neutron_lbaas.services.loadbalancer.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default
ubuntu@joshua-devstack:~$ neutron net-list
+--------------------------------------+---------+-------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+-------------------------------------------------------+
| e88e2c63-e86d-4cba-a49f-0487c9153227 | public | a820b11c-f8f4-4023-8944-39e6fbb517bf 192.168.101.0/24 |
| fd8a17e0-eb10-45e6-a84c-9d87810ef6e0 | private | 3d013961-10fa-4705-9c3f-ae9d5c373e7a 10.0.1.0/24 |
+--------------------------------------+---------+-------------------------------------------------------+
3, lbaas配置
neutron lb-pool-create --lb-method ROUND_ROBIN --name mypool --protocol HTTP --subnet-id private-subnet
neutron lb-vip-create --name myvip --protocol-port 80 --protocol HTTP --subnet-id private-subnet mypool
neutron floatingip-create public
neutron floatingip-associate ca119ad1-501c-46e7-b064-aefbea8d356a 566ef461-c435-4b4e-9479-705e2a58b10a
ubuntu@joshua-devstack:~$ sudo ip netns exec qlbaas-74b31af8-c15b-469c-88e8-667598ecc12b ip addr show tap566ef461-c4
24: tap566ef461-c4: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether fa:16:3e:e7:1d:25 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.6/24 brd 10.0.1.255 scope global tap566ef461-c4
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fee7:1d25/64 scope link
valid_lft forever preferred_lft forever
ubuntu@joshua-devstack:~$ sudo ip netns exec qrouter-25d7d6ae-047c-4bca-bf96-664794aa84b2 ip addr show
12: qr-839a5881-9e: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether fa:16:3e:59:1a:72 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.1/24 brd 10.0.1.255 scope global qr-839a5881-9e
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe59:1a72/64 scope link
valid_lft forever preferred_lft forever
13: qg-addee699-0a: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether fa:16:3e:3f:eb:1b brd ff:ff:ff:ff:ff:ff
inet 192.168.101.3/24 brd 192.168.101.255 scope global qg-addee699-0a
valid_lft forever preferred_lft forever
inet 192.168.101.4/32 brd 192.168.101.4 scope global qg-addee699-0a
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe3f:eb1b/64 scope link
valid_lft forever preferred_lft forever
ubuntu@joshua-devstack:~$ ps -ef|grep haproxy
nobody 9438 1 0 06:36 ? 00:00:00 haproxy -f /opt/stack/data/neutron/lbaas/74b31af8-c15b-469c-88e8-667598ecc12b/conf -p /opt/stack/data/neutron/lbaas/74b31af8-c15b-469c-88e8-667598ecc12b/pid
ubuntu@joshua-devstack:~$ cat /opt/stack/data/neutron/lbaas/74b31af8-c15b-469c-88e8-667598ecc12b/conf
global
daemon
user nobody
group nogroup
log /dev/log local0
log /dev/log local1 notice
stats socket /opt/stack/data/neutron/lbaas/74b31af8-c15b-469c-88e8-667598ecc12b/sock mode 0666 level user
defaults
log global
retries 3
option redispatch
timeout connect 5000
timeout client 50000
timeout server 50000
frontend a352b6fa-6eeb-41de-9fe6-256c1fe8e36a
option tcplog
bind 10.0.1.6:80
mode http
default_backend 74b31af8-c15b-469c-88e8-667598ecc12b
option forwardfor
backend 74b31af8-c15b-469c-88e8-667598ecc12b
mode http
balance roundrobin
option forwardfor
server 05f6d6de-951c-4423-bb4d-acc7dbccec2c 10.1.1.3:80 weight 1
server beb74c1b-5fb8-4153-935d-e295892de314 10.1.1.4:80 weight 1
这种配置实际有一个mtu方面的问题，client走router到vip-xxx接口的MTU是1500，当使用gre隧道时虚机的mtu可能设置为1400， haproxy会创建两个tcp连接，一个接收客户端连接，一个连backend vm， 所以我们应该使用 bind 10.0.1.6:80 mss 1360 来通知客户端也使用mss=1360。这个bug见： https://bugs.launchpad.net/neutron/+bug/1376446?comments=all
4, 发生了什么
配置一个LB实例后，会在l3-agent节点上创建一个qlbaas-XXX名空间，里面是VIP，由于没有为VIP设置路由，所以VIP的网段与虚机网段一致（这一点与opencontrail不同，opencontrail是服务实例找两个随机的计算节点上部署active与passive两个haproxy实例，如果vip network与vm network相同的话，这两个计算节点上都会有相同的VIP，虽然是局部隔离的，主动发消息由于带了该计算节点的MAC地址所以回来的包能找到地址，但是这样从FIP主动找VIP包却是不知道该往哪个计算节点的VIP转包的）。
5, 测试
在两个计算节点上运行如下脚本充当WEB服务器：
MYIP=$(ifconfig eth0|grep 'inet addr'|awk -F: '{print $2}'| awk '{print $1}')
while true; do echo -e "HTTP/1.0 200 OK\r\n\r\nWelcome to $MYIP" | sudo nc -l -p 80 ; done
然后执行：
wget -O - <VIP>
wget -O - <FIP>
6, 在GRE模式下的MTU影响
外网IP (192.168.101.1)设置在br-ex网桥上, qrouter-xxx名空间里的qg-接口上的IP(192.168.101.3)与floating IP (192.168.101.4)插在br-ex网桥上。
lbaas-xxx名空间里的上的VIP(10.0.1.6)的tap设置与qrouter-xxx名空间上的qr-接口上的网关IP(10.0.1.1)插在br-int上。
上面的接口在一台机器上不受mtu的影响，但另外两个虚机(10.0.1.3, 10.0.1.4)可能在另外的台机器上，和网络节点通过br-int与br-phy两个网桥相连。由于MTU的影响，虚机的MTU可设置为1400.
附录一 ： LBaaS v2
neutron security-group-create lbaas
neutron security-group-rule-create --direction ingress --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0 lbaas
neutron security-group-rule-create --direction ingress --protocol tcp --port-range-min 443 --port-range-max 443 --remote-ip-prefix 0.0.0.0/0 lbaas
neutron security-group-rule-create --direction ingress --protocol icmp lbaas
neutron port-update --security-group lbaas <vip_port_id>
neutron lbaas-listener-create --name test-lb-http --loadbalancer test-lb --protocol HTTP --protocol-port 80
neutron lbaas-pool-create --name test-lb-pool-http --lb-algorithm ROUND_ROBIN --listener test-lb-http --protocol HTTP
neutron lbaas-member-create --subnet private_subnet --address 192.168.21.12 --protocol-port 80 test-lb-pool-http
neutron lbaas-member-create --subnet private_subnet --address 192.168.21.14 --protocol-port 80 test-lb-pool-http
neutron lbaas-healthmonitor-create --delay 5 --max-retries 2 --timeout 10 --type HTTP --pool test-lb-pool-http
neutron lbaas-listener-create --name test-lb-https --loadbalancer test-lb --protocol HTTPS --protocol-port 443
neutron lbaas-pool-create --name test-lb-pool-https --lb-algorithm LEAST_CONNECTIONS --listener test-lb-https --protocol HTTPS
neutron lbaas-member-create --subnet private_subnet --address 192.168.21.12 --protocol-port 443 test-lb-pool-https
neutron lbaas-member-create --subnet private_subnet --address 192.168.21.14 --protocol-port 443 test-lb-pool-https
neutron lbaas-healthmonitor-create --delay 5 --max-retries 2 --timeout 10 --type HTTPS --pool test-lb-pool-https
neutron floatingip-create ext_net
#neutron floatingip-associate $(neutron floatingip-list |grep 10.5.150.5 |awk '{print $2}') $(neutron port-list |grep '192.168.21.11' |awk '{print $2}')
neutron floatingip-associate FLOATINGIP_ID LOAD_BALANCER_PORT_ID
neutron quota-update --tenant-id TENANT_UUID --loadbalancer 25
neutron quota-update --tenant-id TENANT_UUID --pool 50
neutron lbaas-loadbalancer-stats test-lb
调试方法： echo 'show stat;show table' | socat stdio /var/lib/neutron/lbaas/v2/xxx/haproxy_stats.sock
apparmor rule需添加flags=(attach_disconnected) ： https://review.openstack.org/#/c/568228/