作者：张华 发表于：2021-03-04
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明
前一篇基础是：Play with OVN - https://blog.csdn.net/quqi99/article/details/103194137
这一篇将主要讲openstack如何来使用ovn的。
测试环境
下面将搭建一下类似的测试环境。
./generate-bundle.sh --name ovn --series bionic --release ussuri --ovn --vault --create-model --run
juju add-unit nova-compute
./configure
source novarcneutron net-create private2 --provider:network_type geneve --provider:segmentation_id 1012
neutron subnet-create --gateway 192.168.22.1 private2 192.168.22.0/24 --enable_dhcp=True --name private_subnet2
ROUTER_ID=$(neutron router-list |grep ' provider-router ' |awk '{print $2}')
SUBNET_ID=$(neutron subnet-list |grep '192.168.22.0/24' |awk '{print $2}')
neutron router-interface-add $ROUTER_ID $SUBNET_ID
nova hypervisor-list
openstack server create --wait --image bionic --flavor m1.small --key-name testkey --nic net-id=$(openstack net show private -f value -c id) --availability-zone=nova:juju-c40d4b-ovn-13.cloud.sts i1
./tools/float_all.sh
./tools/sec_groups.sh
openstack server create --wait --image cirros2 --flavor m1.small --key-name testkey --nic net-id=$(openstack net show private -f value -c id) --availability-zone=nova:juju-c40d4b-ovn-6.cloud.sts i2
openstack server create --wait --image cirros2 --flavor m1.small --key-name testkey --nic net-id=$(openstack net show private2 -f value -c id) --availability-zone=nova:juju-c40d4b-ovn-6.cloud.sts i3$ nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------------------+
| 82c89129-0335-4e33-b117-be940a7020d4 | i1 | ACTIVE | - | Running | private=192.168.21.161, 10.5.150.115 |
| 74641e74-3401-44f9-8d7d-bef3ea0fdb92 | i2 | ACTIVE | - | Running | private=192.168.21.3 |
| 37f7c5c0-844c-4d8d-ad95-aa29b7418dc0 | i3 | ACTIVE | - | Running | private2=192.168.22.47 |
+--------------------------------------+------+--------+------------+-------------+--------------------------------------+
OVN Northbound DB与Neutron的概念映射
Neutron中有Network, Subnet, Router, Port的概念，OVN Northbound DB中也有对应的逻辑概念: Switch=Neutron Subnet, Port=Neutron Port, Distributed Router=Neutron DVR Router, Gateway Router=Neutron Centralized L3, Port=Neutron Port
举例查看OVN Northbound DB中和L3 NAT相关的数据
#run in compute node
# ovs-vsctl get open . external_ids
{hostname=juju-c40d4b-ovn-6.cloud.sts, ovn-bridge-mappings="physnet1:br-data", ovn-cms-options=enable-chassis-as-gw, ovn-encap-ip="10.5.0.178", ovn-encap-type=geneve, ovn-remote="ssl:10.5.2.178:6642,ssl:10.5.1.220:6642,ssl:10.5.1.157:6642", rundir="/var/run/openvswitch", system-id=juju-c40d4b-ovn-6.cloud.sts}
export SB=$(sudo ovs-vsctl get open . external_ids:ovn-remote | sed -e 's/\"//g')
export NB=$(sudo ovs-vsctl get open . external_ids:ovn-remote | sed -e 's/\"//g' | sed -e 's/6642/6641/g')#从所有计算节点上的ovn-controller中所出用于中心化l3的那个(搜索enable-chassis-as-gw得知是juju-c40d4b-ovn-6.cloud.sts(uuid=f8004279-14d2-48fd-8b6a-f025706fa8a8)
#run in ovnnb_db master and ovnsb_db master
juju ssh ovn-central/1 -- sudo -s
root@juju-c40d4b-ovn-8:~# ovn-sbctl list chassis
_uuid : add1028c-e19c-4f23-8795-a0c64f16fdcd
encaps : [7417dd02-d2d5-45bb-88b9-3dafe07c92c6]
external_ids : {datapath-type=system, iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", neutron-metadata-proxy-networks="d0382b73-eb07-4314-a803-b957662f618c", "neutron:liveness_check_at"="2021-03-04T10:34:37.869388+00:00", "neutron:metadata_liveness_check_at"="2021-03-04T10:34:38.163481+00:00", "neutron:ovn-metadata-id"="4849b5a6-3134-4ca2-9fea-38c24aef6121", "neutron:ovn-metadata-sb-cfg"="578", ovn-bridge-mappings="", ovn-chassis-mac-mappings="", ovn-cms-options=""}
hostname : juju-c40d4b-ovn-13.cloud.sts
name : juju-c40d4b-ovn-13.cloud.sts
nb_cfg : 578
transport_zones : []
vtep_logical_switches: []
_uuid : f8004279-14d2-48fd-8b6a-f025706fa8a8
encaps : [50fde256-2d20-4e4d-aa6e-c7838edee407]
external_ids : {datapath-type=system, iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", neutron-metadata-proxy-networks="d0382b73-eb07-4314-a803-b957662f618c,f1b85533-3f78-4f44-9785-996e725bb3bf", "neutron:liveness_check_at"="2021-03-04T10:34:37.322922+00:00", "neutron:metadata_liveness_check_at"="2021-03-04T10:34:37.602977+00:00", "neutron:ovn-metadata-id"="9dd62f6d-c41b-42e2-b424-7d0bbf0902ea", "neutron:ovn-metadata-sb-cfg"="578", ovn-bridge-mappings="physnet1:br-data", ovn-chassis-mac-mappings="", ovn-cms-options=enable-chassis-as-gw}
hostname : juju-c40d4b-ovn-6.cloud.sts
name : juju-c40d4b-ovn-6.cloud.sts
nb_cfg : 578
transport_zones : []
vtep_logical_switches: []#或者直接找到它l3 ovn-controller
root@juju-c40d4b-ovn-8:~# ovn-nbctl list Gateway_Chassis
_uuid : 5de561d4-77e5-467d-8b69-ec064e949d8c
chassis_name : juju-c40d4b-ovn-6.cloud.sts
external_ids : {}
name : lrp-304cfe5a-d25c-41aa-bfbe-9ba60c7248c2_juju-c40d4b-ovn-6.cloud.sts
options : {}
priority : 1#找到neutron router的external_gateway_info和routerid=1a4bf8d2-c885-4af8-8b9c-061c7b27fa69
$ openstack router show provider-router --fit-width |grep external_gateway_info
| external_gateway_info | {"network_id": "1d7749fd-90c9-4f31-ada4-50f1845ca32e", "external_fixed_ips": [{"subnet_id": "4009f18b-eb09-4b74-a0ac-ce29537838a3", "ip_address": "10.5.152.46"}], "enable_snat": true} #找到与此neutron router对应的ovn router
root@juju-c40d4b-ovn-8:~# ovn-nbctl find Logical_Router name=neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69
_uuid : 89a12d3b-28ad-466b-97d6-971c669aee44
enabled : true
external_ids : {"neutron:availability_zone_hints"="", "neutron:gw_port_id"="304cfe5a-d25c-41aa-bfbe-9ba60c7248c2", "neutron:revision_number"="5", "neutron:router_name"=provider-router}
load_balancer : []
name : neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69
nat : [6196ba5f-568b-486e-b7d6-add825d2f8f9, b1e5878e-95f0-45f7-b3a2-232b550be281, cb82abf5-55b5-4731-aa24-b993ac4621d9]
options : {}
policies : []
ports : [3aa3c7ce-ec0f-4d3e-9a3b-15ae7f750622, f964314d-ac73-4dda-a940-9c6910850b34, fb90ec4c-ac5e-4415-96e8-28ea18e53205]
static_routes : [cd814869-e8a0-4dde-9730-362d5d83a1d0]#从OVN NAT northbound表中验证SNAT
root@juju-c40d4b-ovn-8:~# ovn-nbctl find NAT type=snat
_uuid : b1e5878e-95f0-45f7-b3a2-232b550be281
external_ids : {}
external_ip : "10.5.152.46"
external_mac : []
logical_ip : "192.168.22.0/24"
logical_port : []
options : {}
type : snat
_uuid : 6196ba5f-568b-486e-b7d6-add825d2f8f9
external_ids : {}
external_ip : "10.5.152.46"
external_mac : []
logical_ip : "192.168.21.0/24"
logical_port : []
options : {}
type : snatroot@juju-c40d4b-ovn-8:~# ovn-nbctl find NAT type=dnat_and_snat
_uuid : cb82abf5-55b5-4731-aa24-b993ac4621d9
external_ids : {"neutron:fip_external_mac"="fa:16:3e:b6:11:c5", "neutron:fip_id"="9bf3a29c-e7fe-4dca-8c59-a8809ae87db9", "neutron:fip_port_id"="13d0a59c-e25d-48f5-af68-ca18dbbf139d", "neutron:revision_number"="2", "neutron:router_name"=neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69}
external_ip : "10.5.150.115"
external_mac : []
logical_ip : "192.168.21.161"
logical_port : "13d0a59c-e25d-48f5-af68-ca18dbbf139d"
options : {}
type : dnat_and_snatroot@juju-c40d4b-ovn-8:~# ovn-nbctl lr-nat-list neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69
TYPE EXTERNAL_IP LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT
dnat_and_snat 10.5.150.115 192.168.21.161
snat 10.5.152.46 192.168.21.0/24
snat 10.5.152.46 192.168.22.0/24
OVN Southbound DB - L2 Logical Flow (同网段大二层)
$ openstack port list
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+
| 0a2c4125-791c-4824-837a-1a940e78673a | | fa:16:3e:2d:ce:27 | ip_address='192.168.21.2', subnet_id='a1607cfe-3fa8-40a1-891e-776801f90342' | DOWN |
| 13d0a59c-e25d-48f5-af68-ca18dbbf139d | | fa:16:3e:54:36:ad | ip_address='192.168.21.161', subnet_id='a1607cfe-3fa8-40a1-891e-776801f90342' | ACTIVE |
| 1ba6b6f4-140b-4bb7-a0fd-6d880cda47ff | | fa:16:3e:d2:1d:ec | ip_address='192.168.22.47', subnet_id='450d6660-b862-4908-b501-4c8533211b23' | ACTIVE |
| 304cfe5a-d25c-41aa-bfbe-9ba60c7248c2 | | fa:16:3e:50:aa:2a | ip_address='10.5.152.46', subnet_id='4009f18b-eb09-4b74-a0ac-ce29537838a3' | ACTIVE |
| 3999ba6e-7f80-499b-8dd8-fa87d0f4a63e | | fa:16:3e:76:aa:1e | ip_address='192.168.22.2', subnet_id='450d6660-b862-4908-b501-4c8533211b23' | DOWN |
| 7a9ae9c2-3dd5-498b-9c4d-09a101fc3120 | | fa:16:3e:b0:67:15 | | DOWN |
| abe38147-7909-4708-ad02-d478e62e7ff1 | | fa:16:3e:22:d6:67 | ip_address='192.168.21.1', subnet_id='a1607cfe-3fa8-40a1-891e-776801f90342' | ACTIVE |
| b236113a-86e2-4d69-8de8-f1086cc17a7b | | fa:16:3e:93:b1:62 | ip_address='192.168.22.1', subnet_id='450d6660-b862-4908-b501-4c8533211b23' | ACTIVE |
| cd9fefdb-00f0-4efd-950b-84ba32788571 | | fa:16:3e:78:2f:34 | ip_address='192.168.21.3', subnet_id='a1607cfe-3fa8-40a1-891e-776801f90342' | ACTIVE |
| eb40953f-28f3-46f4-a28a-50786d1090b5 | | fa:16:3e:b6:11:c5 | ip_address='10.5.150.115', subnet_id='4009f18b-eb09-4b74-a0ac-ce29537838a3' | N/A |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+
对于vm1=i1(192.168.21.161)访问vm3=i3(192.168.21.3)
$ openstack port list |grep -E '192.168.21.3|192.168.21.161'
| 13d0a59c-e25d-48f5-af68-ca18dbbf139d | | fa:16:3e:54:36:ad | ip_address='192.168.21.161', subnet_id='a1607cfe-3fa8-40a1-891e-776801f90342' | ACTIVE |
| cd9fefdb-00f0-4efd-950b-84ba32788571 | | fa:16:3e:78:2f:34 | ip_address='192.168.21.3', subnet_id='a1607cfe-3fa8-40a1-891e-776801f90342' | ACTIVE |
1, port security只是对从虚机进来的包检查它的IP与MAC是否对应
root@juju-c40d4b-ovn-8:~# ovn-sbctl lflow-list |grep inport |grep 13d0a59c-e25d-48f5-af68-ca18dbbf139d |grep -E 'table=0'table=0 (ls_in_port_sec_l2 ), priority=50 , match=(inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == {fa:16:3e:54:36:ad}), action=(next;)
2,
接下来的看这篇文章 -
OpenStack SDN With OVN (Part 2) - Network Engineering Analysis
https://networkop.co.uk/blog/2016/12/10/ovn-part2/
OVN Southbound DB - L3 Logical Flow(南北)
见 － https://networkop.co.uk/blog/2016/12/10/ovn-part2/
OVN Southbound DB - ovn controller Logical Flow (不同网段东西)
通过ovn-trace调试OVN Sourthbound DB逻辑流
ovn-trace能用来帮助调试或者理解上面的ovn南向逻辑流
root@juju-c40d4b-ovn-8:~# ovn-sbctl lflow-list |grep -i datapath
Datapath: "neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69" aka "provider-router" (587cfb5a-2797-405f-83f0-385211e2ad78) Pipeline: ingress
Datapath: "neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69" aka "provider-router" (587cfb5a-2797-405f-83f0-385211e2ad78) Pipeline: egress
Datapath: "neutron-1d7749fd-90c9-4f31-ada4-50f1845ca32e" aka "ext_net" (74359f20-4009-4c7e-afd8-3f3dd2423b77) Pipeline: ingress
Datapath: "neutron-1d7749fd-90c9-4f31-ada4-50f1845ca32e" aka "ext_net" (74359f20-4009-4c7e-afd8-3f3dd2423b77) Pipeline: egress
Datapath: "neutron-b2f023b1-4a05-4443-b334-cf47e90a1567" aka "private" (d0382b73-eb07-4314-a803-b957662f618c) Pipeline: ingress
Datapath: "neutron-b2f023b1-4a05-4443-b334-cf47e90a1567" aka "private" (d0382b73-eb07-4314-a803-b957662f618c) Pipeline: egress
Datapath: "neutron-1c537bdd-5633-4263-a364-b14cecd4e92d" aka "private2" (f1b85533-3f78-4f44-9785-996e725bb3bf) Pipeline: ingress
Datapath: "neutron-1c537bdd-5633-4263-a364-b14cecd4e92d" aka "private2" (f1b85533-3f78-4f44-9785-996e725bb3bf) Pipeline: egress
root@juju-c40d4b-ovn-8:~# ovn-nbctl show c6cc6d66-91af-4613-87aa-cbd770d8040d
switch c6cc6d66-91af-4613-87aa-cbd770d8040d (neutron-b2f023b1-4a05-4443-b334-cf47e90a1567) (aka private)port abe38147-7909-4708-ad02-d478e62e7ff1type: routerrouter-port: lrp-abe38147-7909-4708-ad02-d478e62e7ff1port 13d0a59c-e25d-48f5-af68-ca18dbbf139daddresses: ["fa:16:3e:54:36:ad 192.168.21.161"]port 0a2c4125-791c-4824-837a-1a940e78673atype: localportaddresses: ["fa:16:3e:2d:ce:27 192.168.21.2"]port cd9fefdb-00f0-4efd-950b-84ba32788571addresses: ["fa:16:3e:78:2f:34 192.168.21.3"]
例如上面的192.168.21.161如何访问192.168.21.3的.
root@juju-c40d4b-ovn-8:~# ovn-trace --minimal neutron-b2f023b1-4a05-4443-b334-cf47e90a1567 'inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == fa:16:3e:54:36:ad && eth.dst == fa:16:3e:78:2f:34'
# reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:54:36:ad,dl_dst=fa:16:3e:78:2f:34,dl_type=0x0000
output("cd9fef");root@juju-c40d4b-ovn-8:~# ovn-trace neutron-b2f023b1-4a05-4443-b334-cf47e90a1567 'inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == fa:16:3e:54:36:ad && eth.dst == fa:16:3e:78:2f:34'
# reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:54:36:ad,dl_dst=fa:16:3e:78:2f:34,dl_type=0x0000ingress(dp="private", inport="13d0a5")
--------------------------------------0. ls_in_port_sec_l2 (ovn-northd.c:4516): inport == "13d0a5" && eth.src == {fa:16:3e:54:36:ad}, priority 50, uuid 620c23f4next;
19. ls_in_l2_lkup (ovn-northd.c:6779): eth.dst == fa:16:3e:78:2f:34, priority 50, uuid cdf396b9outport = "cd9fef";output;egress(dp="private", inport="13d0a5", outport="cd9fef")
-------------------------------------------------------9. ls_out_port_sec_l2 (ovn-northd.c:4582): outport == "cd9fef" && eth.dst == {fa:16:3e:78:2f:34}, priority 50, uuid b48eb374output;/* output to "cd9fef", type "" */
root@juju-c40d4b-ovn-8:~# ovn-trace --summary neutron-b2f023b1-4a05-4443-b334-cf47e90a1567 'inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == fa:16:3e:54:36:ad && eth.dst == fa:16:3e:78:2f:34'
# reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:54:36:ad,dl_dst=fa:16:3e:78:2f:34,dl_type=0x0000
ingress(dp="private", inport="13d0a5") {next;outport = "cd9fef";output;egress(dp="private", inport="13d0a5", outport="cd9fef") {output;/* output to "cd9fef", type "" */;};
};
再例如如何访问8.8.8.8
root@juju-c40d4b-ovn-8:~# ovn-trace --detail neutron-b2f023b1-4a05-4443-b334-cf47e90a1567 'inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == fa:16:3e:54:36:ad && ip4.dst == 8.8.8.8'
# ip,reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:54:36:ad,dl_dst=00:00:00:00:00:00,nw_src=0.0.0.0,nw_dst=8.8.8.8,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0ingress(dp="private", inport="13d0a5")
--------------------------------------0. ls_in_port_sec_l2 (ovn-northd.c:4516): inport == "13d0a5" && eth.src == {fa:16:3e:54:36:ad}, priority 50, uuid 620c23f4next;1. ls_in_port_sec_ip (ovn-northd.c:4225): inport == "13d0a5" && eth.src == fa:16:3e:54:36:ad && ip, priority 80, uuid 47d5e0e8drop;计算节点上的openflow flow
上面的都是OVN南向DB的逻辑流，直接到计算节点上的openflow通过“ovs-ofctl dump-flows br-int”查看，可通过“ovs-appctl ofproto/trace”来调试(ovn Logical Flow流过ovn-trace调试)，见：https://blog.russellbryant.net/2016/11/11/ovn-logical-flows-and-ovn-trace/
如：
sudo ovs-appctl ofproto/trace br-int in_port=6,arp,arp_spa=192.168.21.7,dl_src=fa:16:3e:c4:58:9c
可使用ovs-stat snap工具来方便生成ovs-appctl辅助命令，见： https://blog.csdn.net/quqi99/article/details/111831695
实操 - 虚机访问外部UDP服务调试流表
找到了centralized l3为juju-c40d4b-ovn-6.cloud.sts，
root@juju-c40d4b-ovn-6:~# sudo ovs-appctl dpif/show
system@ovs-system: hit:243081 missed:5389br-data:br-data 65534/2: (internal)ens8 1/3: (system)patch-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7-to-br-int 2/none: (patch: peer=patch-br-int-to-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7)br-int:br-int 65534/1: (internal)ovn-juju-c-0 2/4: (geneve: csum=true, key=flow, remote_ip=10.5.0.191)patch-br-int-to-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7 1/none: (patch: peer=patch-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7-to-br-int)tap1ba6b6f4-14 5/7: (system)tapcd9fefdb-00 3/5: (system)tapd0382b73-e0 4/6: (system)tapf1b85533-30 6/8: (system)root@juju-c40d4b-ovn-6:~# ovs-vsctl show
2849984e-7c3c-4390-b6f0-2cb47c757ca0Manager "ptcp:6640:127.0.0.1"is_connected: trueBridge br-intfail_mode: securedatapath_type: systemPort tapcd9fefdb-00Interface tapcd9fefdb-00Port tap1ba6b6f4-14Interface tap1ba6b6f4-14Port tapd0382b73-e0Interface tapd0382b73-e0Port tapf1b85533-30Interface tapf1b85533-30Port br-intInterface br-inttype: internalPort ovn-juju-c-0Interface ovn-juju-c-0type: geneveoptions: {csum="true", key=flow, remote_ip="10.5.0.191"}Port patch-br-int-to-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7Interface patch-br-int-to-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7type: patchoptions: {peer=patch-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7-to-br-int}Bridge br-datafail_mode: standalonedatapath_type: systemPort patch-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7-to-br-intInterface patch-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7-to-br-inttype: patchoptions: {peer=patch-br-int-to-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7}Port br-dataInterface br-datatype: internalPort ens8Interface ens8type: systemovs_version: "2.13.1"当ssh into i1，在l3上会看到：
tcp 6 426771 ESTABLISHED src=10.5.0.8 dst=10.5.150.115 sport=39746 dport=22 src=192.168.21.161 dst=10.5.0.8 sport=22 dport=39746 [ASSURED] mark=0 zone=1 use=1
tcp 6 431995 ESTABLISHED src=10.5.0.8 dst=10.5.150.115 sport=39904 dport=22 src=192.168.21.161 dst=10.5.0.8 sport=22 dport=39904 [ASSURED] mark=0 zone=1 use=1当从i1上运行"nc -uvz 10.5.0.2 53", 在L3上会看到
root@juju-c40d4b-ovn-6:~# conntrack -L |grep 192.168.21.161
conntrack v1.4.4 (conntrack-tools): 52 flow entries have been shown.
udp 17 25 src=192.168.21.161 dst=10.5.0.2 sport=55185 dport=53 [UNREPLIED] src=10.5.0.2 dst=10.5.150.115 sport=53 dport=55185 mark=0 zone=2 use=1
udp 17 6 src=192.168.21.161 dst=10.5.0.2 sport=36199 dport=53 [UNREPLIED] src=10.5.0.2 dst=10.5.150.115 sport=53 dport=36199 mark=0 zone=2 use=1在SouthBound DB master上看到了ovn逻辑流是：
ubuntu@zhhuabj-bastion:~/stsstack-bundles/openstack$ juju ssh ovn-central/2 -- sudo -s
root@juju-c40d4b-ovn-9:~# ovn-trace --detail neutron-b2f023b1-4a05-4443-b334-cf47e90a1567 'inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == fa:16:3e:54:36:ad && ip4.dst == 10.5.0.2'
# ip,reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:54:36:ad,dl_dst=00:00:00:00:00:00,nw_src=0.0.0.0,nw_dst=10.5.0.2,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0
ingress(dp="private", inport="13d0a5")
--------------------------------------0. ls_in_port_sec_l2 (ovn-northd.c:4516): inport == "13d0a5" && eth.src == {fa:16:3e:54:36:ad}, priority 50, uuid 620c23f4next;1. ls_in_port_sec_ip (ovn-northd.c:4225): inport == "13d0a5" && eth.src == fa:16:3e:54:36:ad && ip, priority 80, uuid 47d5e0e8在L3上调试openflow
sudo snap install ovs-stat
sudo snap connect ovs-stat:openvswitch
sudo snap connect ovs-stat:network-control
#sudo snap connect ovs-stat:removable-media
#ovs-stat -p /tmp/results --tree ./sosreport-015 --openstack #don't use sudo
ovs-stat -p /tmp/results --tree --openstack
sudo ls /tmp/snap.ovs-stat/tmp/results
ovs-stat -p /tmp/results --host juju-c40d4b-ovn-6 --query ""
root@juju-c40d4b-ovn-6:~# ovs-stat -p /tmp/results --host juju-c40d4b-ovn-6 --query "ofproto-trace.port tapf1b85533-30"
[arp]
no source ips found - skipping
[icmp]
no source ips found - skipping
[dhcp]
sudo ovs-appctl ofproto/trace br-int udp,in_port=6,dl_src=12:6b:c5:a6:f5:47,dl_dst=ff:ff:ff:ff:ff:ff,nw_src=0.0.0.0,nw_dst=255.255.255.255,udp_src=68,udp_dst=67
[vm-to-vm]
sudo ovs-appctl ofproto/trace br-int in_port=6,tcp,dl_src=12:6b:c5:a6:f5:47,dl_dst=MAC_OF_REMOTE_INSTANCE
sudo ovs-appctl ofproto/trace br-int in_port=6,dl_vlan=,dl_src=12:6b:c5:a6:f5:47,dl_dst=MAC_OF_REMOTE_INSTANCE在L3节点上看到的openflow flow是：
root@juju-c40d4b-ovn-6:~# ovs-ofctl dump-flows br-int |grep 192.168.21.161cookie=0xe51693a7, duration=86438.109s, table=14, n_packets=22223, n_bytes=1532774, idle_age=33, hard_age=65534, priority=100,ip,reg14=0x1,metadata=0x2,nw_dst=10.5.150.115 actions=ct(commit,table=15,zone=NXM_NX_REG11[0..15],nat(dst=192.168.21.161))cookie=0x1043afc5, duration=86542.428s, table=21, n_packets=1, n_bytes=42, idle_age=65534, hard_age=65534, priority=50,arp,metadata=0x3,arp_tpa=192.168.21.161,arp_op=1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:54:36:ad,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],load:0xfa163e5436ad->NXM_NX_ARP_SHA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xc0a815a1->NXM_OF_ARP_SPA[],move:NXM_NX_REG14[]->NXM_NX_REG15[],load:0x1->NXM_NX_REG10[0],resubmit(,32)cookie=0xd56bfbc4, duration=86438.109s, table=40, n_packets=53762, n_bytes=13079808, idle_age=33, hard_age=65534, priority=100,ip,reg15=0x1,metadata=0x2,nw_src=192.168.21.161 actions=ct(table=41,zone=NXM_NX_REG11[0..15],nat)cookie=0xf8bdd8ec, duration=86438.109s, table=41, n_packets=16596, n_bytes=1295012, idle_age=34, hard_age=65534, priority=161,ip,reg15=0x1,metadata=0x2,nw_src=192.168.21.161 actions=ct(commit,table=42,zone=NXM_NX_REG12[0..15],nat(src=10.5.150.115))cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-new+est-rpl+trk,ct_label=0/0x1,ip,metadata=0x3,nw_src=192.168.21.161 actions=conjunction(9,1/2)cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-new+est-rpl+trk,ct_label=0x1/0x1,ip,metadata=0x4,nw_src=192.168.21.161 actions=conjunction(3,1/2)cookie=0x0, duration=85847.306s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-new+est-rpl+trk,ct_label=0x1/0x1,ip,metadata=0x3,nw_src=192.168.21.161 actions=conjunction(7,1/2)cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-new+est-rpl+trk,ct_label=0/0x1,ip,metadata=0x4,nw_src=192.168.21.161 actions=conjunction(5,1/2)cookie=0x0, duration=85847.306s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=+new-est+trk,ip,metadata=0x3,nw_src=192.168.21.161 actions=conjunction(6,1/2)cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=+new-est+trk,ip,metadata=0x4,nw_src=192.168.21.161 actions=conjunction(2,1/2)cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-trk,ip,metadata=0x4,nw_src=192.168.21.161 actions=conjunction(4,1/2)cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-trk,ip,metadata=0x3,nw_src=192.168.21.161 actions=conjunction(8,1/2)
附录 - NB与SB表
root@juju-c40d4b-ovn-8:~# ovn-nbctl show
switch 38364b40-19f0-473f-8878-da50b652cc67 (neutron-1d7749fd-90c9-4f31-ada4-50f1845ca32e) (aka ext_net)port 7a9ae9c2-3dd5-498b-9c4d-09a101fc3120type: localportaddresses: ["fa:16:3e:b0:67:15"]port 304cfe5a-d25c-41aa-bfbe-9ba60c7248c2type: routerrouter-port: lrp-304cfe5a-d25c-41aa-bfbe-9ba60c7248c2port provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7type: localnetaddresses: ["unknown"]
switch c6cc6d66-91af-4613-87aa-cbd770d8040d (neutron-b2f023b1-4a05-4443-b334-cf47e90a1567) (aka private)port abe38147-7909-4708-ad02-d478e62e7ff1type: routerrouter-port: lrp-abe38147-7909-4708-ad02-d478e62e7ff1port 13d0a59c-e25d-48f5-af68-ca18dbbf139daddresses: ["fa:16:3e:54:36:ad 192.168.21.161"]port 0a2c4125-791c-4824-837a-1a940e78673atype: localportaddresses: ["fa:16:3e:2d:ce:27 192.168.21.2"]port cd9fefdb-00f0-4efd-950b-84ba32788571addresses: ["fa:16:3e:78:2f:34 192.168.21.3"]
switch c53d2b6f-6721-4b97-bb8e-c62df9bd952b (neutron-1c537bdd-5633-4263-a364-b14cecd4e92d) (aka private2)port 1ba6b6f4-140b-4bb7-a0fd-6d880cda47ffaddresses: ["fa:16:3e:d2:1d:ec 192.168.22.47"]port b236113a-86e2-4d69-8de8-f1086cc17a7btype: routerrouter-port: lrp-b236113a-86e2-4d69-8de8-f1086cc17a7bport 3999ba6e-7f80-499b-8dd8-fa87d0f4a63etype: localportaddresses: ["fa:16:3e:76:aa:1e 192.168.22.2"]
router 89a12d3b-28ad-466b-97d6-971c669aee44 (neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69) (aka provider-router)port lrp-abe38147-7909-4708-ad02-d478e62e7ff1mac: "fa:16:3e:22:d6:67"networks: ["192.168.21.1/24"]port lrp-b236113a-86e2-4d69-8de8-f1086cc17a7bmac: "fa:16:3e:93:b1:62"networks: ["192.168.22.1/24"]port lrp-304cfe5a-d25c-41aa-bfbe-9ba60c7248c2mac: "fa:16:3e:50:aa:2a"networks: ["10.5.152.46/16"]gateway chassis: [juju-c40d4b-ovn-6.cloud.sts]nat 6196ba5f-568b-486e-b7d6-add825d2f8f9external ip: "10.5.152.46"logical ip: "192.168.21.0/24"type: "snat"nat b1e5878e-95f0-45f7-b3a2-232b550be281external ip: "10.5.152.46"logical ip: "192.168.22.0/24"nat cb82abf5-55b5-4731-aa24-b993ac4621d9external ip: "10.5.150.115"logical ip: "192.168.21.161"type: "dnat_and_snat"root@juju-c40d4b-ovn-8:~# ovn-sbctl show
Chassis juju-c40d4b-ovn-13.cloud.stshostname: juju-c40d4b-ovn-13.cloud.stsEncap geneveip: "10.5.0.191"options: {csum="true"}Port_Binding "13d0a59c-e25d-48f5-af68-ca18dbbf139d"
Chassis juju-c40d4b-ovn-6.cloud.stshostname: juju-c40d4b-ovn-6.cloud.stsEncap geneveip: "10.5.0.178"options: {csum="true"}Port_Binding cr-lrp-304cfe5a-d25c-41aa-bfbe-9ba60c7248c2Port_Binding "cd9fefdb-00f0-4efd-950b-84ba32788571"Port_Binding "1ba6b6f4-140b-4bb7-a0fd-6d880cda47ff"
OVN Southbound DB CLI
#https://blog.csdn.net/zhengmx100/article/details/75426710
# https://docs.openstack.org/networking-ovn/ocata/refarch/refarch.html
ovn-nbctl list Logical_Switch
ovn-nbctl list Logical_Switch_Port
ovn-nbctl list ACL
ovn-nbctl list Address_Set
ovn-nbctl list Logical_Router
ovn-nbctl list Logical_Router_Port
ovn-sbctl list Chassis
ovn-sbctl list Encap
ovn-nbctl list Address_Set
ovn-sbctl lflow-list
ovn-sbctl list Multicast_Group
ovn-sbctl list Datapath_Binding
ovn-sbctl list Port_Binding
ovn-sbctl list MAC_Binding
ovn-nbctl list Gateway_Chassis
ovn-nbctl list dhcp_optionsovn-nbctl show
ovn-sbctl show# ovn-sbctl show
Chassis juju-e28e17-ovn2-8.cloud.stshostname: juju-e28e17-ovn2-8.cloud.stsEncap geneveip: "10.5.2.173"options: {csum="true"}Port_Binding "bdbe0b57-eb1e-41d5-a06d-b2fd3050fa4d"Port_Binding cr-lrp-a3076445-eb22-411b-ad68-2672c0abcaa3#从它找到Port_Binding对应的chassis
# ovn-sbctl list Port_Binding cr-lrp-a3076445-eb22-411b-ad68-2672c0abcaa3
_uuid : c689e404-2534-40b7-92b0-0a0cb306c458
chassis : 3ffd5c8b-86c1-498d-9128-84289a1e832a
datapath : 2ec13388-171d-4719-9b6f-39a755a68f60
encap : []
external_ids : {}
gateway_chassis : []
ha_chassis_group : beeb76fd-aa54-43f6-aeed-2c1d71acf2b4
logical_port : cr-lrp-a3076445-eb22-411b-ad68-2672c0abcaa3
mac : ["fa:16:3e:1c:06:c6 10.5.151.201/16"]
nat_addresses : []
options : {distributed-port=lrp-a3076445-eb22-411b-ad68-2672c0abcaa3}
parent_port : []
tag : []
tunnel_key : 2
type : chassisredirect
virtual_parent : []# 再从chassis找到encaps
# ovn-sbctl list Chassis
_uuid : 3ffd5c8b-86c1-498d-9128-84289a1e832a
encaps : [0406d6d8-b1d2-45b4-8461-22b661bcb7dd]
external_ids : {datapath-type=system, iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", neutron-metadata-proxy-networks="e2ed122c-9442-4116-9793-5c2323f6de29", "neutron:liveness_check_at"="2021-04-01T10:09:44.354244+00:00", "neutron:metadata_liveness_check_at"="2021-04-01T10:09:45.089358+00:00", "neutron:ovn-metadata-id"="6e1038ee-52f4-4bdf-afb9-0e220696176c", "neutron:ovn-metadata-sb-cfg"="571", ovn-bridge-mappings="physnet1:br-data", ovn-chassis-mac-mappings="", ovn-cms-options=enable-chassis-as-gw}
hostname : juju-e28e17-ovn2-8.cloud.sts
name : juju-e28e17-ovn2-8.cloud.sts
nb_cfg : 571
transport_zones : []
vtep_logical_switches: []#从Encap找到tunnel IP
# ovn-sbctl list Encap
_uuid : 0406d6d8-b1d2-45b4-8461-22b661bcb7dd
chassis_name : juju-e28e17-ovn2-8.cloud.sts
ip : "10.5.2.173"
options : {csum="true"}
type : geneve# ovn-sbctl list Datapath_Binding
_uuid : e2ed122c-9442-4116-9793-5c2323f6de29
external_ids : {logical-switch="eb7511fa-6746-4981-9ddc-6e1c73e9a7ee", name=neutron-b3187a23-b54e-4cc0-bc6d-4caabdb02b0c, name2=private}
tunnel_key : 3
_uuid : 2ec13388-171d-4719-9b6f-39a755a68f60
external_ids : {logical-router="8bbdfe13-4266-432e-a1b9-aae2dacf6ebd", name=neutron-0bb8206c-96e5-4fd8-a6ad-66e591620496, name2=provider-router}
tunnel_key : 2
_uuid : a00e41b6-d46b-4ce4-a0f9-ddfcca7c241c
external_ids : {logical-switch="375943d6-4b12-44c2-907d-ca59e8a3915d", name=neutron-bbc0362f-44ac-4bab-accf-613c68c6fb66, name2=ext_net}ovn-sbctl lflow-list |grep icmp
如何找到dhcp在哪个host上呢？
用OVN实现OpenStack原理以流表分析
1, 每个Node(Chassis）上创建br-data2ovs-vsctl --may-exist add-br br-data2
#ovs-vsctl --may-exist add-port br-data2 eth2
ovs-vsctl set open . external-ids:ovn-bridge-mappings=physnet1:br-data,physnet2:br-data22, 东西向的两个Subnet(Switch)和vRouter(Logical Router)ovn-nbctl ls-add sw0
ovn-nbctl lsp-add sw0 sw0-port1
ovn-nbctl lsp-set-addresses sw0-port1 "00:00:01:00:00:03 10.0.0.3"ovn-nbctl ls-add sw1
ovn-nbctl lsp-add sw1 sw1-port1
ovn-nbctl lsp-set-addresses sw1-port1 "00:00:02:00:00:03 20.0.0.3"ovn-nbctl lr-add lr0
# Connect sw0 to lr0
ovn-nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24
ovn-nbctl lsp-add sw0 sw0-lr0
ovn-nbctl lsp-set-type sw0-lr0 router
ovn-nbctl lsp-set-addresses sw0-lr0 router
ovn-nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0
# Connect sw1 to lr0
ovn-nbctl lrp-add lr0 lr0-sw1 00:00:00:00:ff:02 20.0.0.1/24
ovn-nbctl lsp-add sw1 sw1-lr0
ovn-nbctl lsp-set-type sw1-lr0 router
ovn-nbctl lsp-set-addresses sw1-lr0 router
ovn-nbctl lsp-set-options sw1-lr0 router-port=lr0-sw1
ovn-nbctl show3, 南北向流量，br-ex(public)，并在vRouter(lr0)中设置外部IP（即将lr0与public关联）这里network_name为physnet2, 且localnet port定义在switch上意味着运行在Gateway Chassis (Centralized L3)上的ovn-controller将在br-int与br-data2之前创建patch portovn-nbctl ls-add public
# Create a localnet port
ovn-nbctl lsp-add public ln-public
ovn-nbctl lsp-set-type ln-public localnet
ovn-nbctl lsp-set-addresses ln-public unknown
ovn-nbctl lsp-set-options ln-public network_name=physnet2# 这个vRouter中的外部接口(lr0-public)也应该部署在L3上，它在将流量经patch port转给br-ex前需SNAT
# 并且当有人要访问外部地址172.168.0.200时,Gateway Chassis也需响应ARP reply
ovn-nbctl lrp-add lr0 lr0-public 00:00:20:20:12:13 172.168.0.200/24
ovn-nbctl lsp-add public public-lr0
ovn-nbctl lsp-set-type public-lr0 router
ovn-nbctl lsp-set-addresses public-lr0 router
ovn-nbctl lsp-set-options public-lr0 router-port=lr0-public有两个方法schedule Gateway router port(lr0-public):
a, non_HA, 只有single Gateway Chassis, eg: schedule到ovn-controllerovn-nbctl set logical_router_port lr0-public options:redirect-chassis=juju-e28e17-ovn2-8.cloud.sts# ovn-sbctl show |grep lr0-publicPort_Binding cr-lr0-public
b, HA, 有多个Gateway Chassis, 一个挂了，另外高优先级的能起来，类似于VRRPovn-nbctl lrp-set-gateway-chassis lr0-public juju-e28e17-ovn2-8.cloud.sts 20ovn-nbctl lrp-set-gateway-chassis lr0-public controller-1 15ovn-nbctl lrp-set-gateway-chassis lr0-public controller-2 10#ovn-nbctl lrp-del-gateway-chassis lr0-public controller-1"ovn-nbctl list gateway_chassis"能看到有多个Gateway Chassis"ovn-nbctl list logical_router_port lr0-public"也能看到多个Gateway Chassis对于这个HA模式，OVN使用BFD(Bidirectional Forwarding Detection)协议，它配置BFD在tunnel ports上,当一个Gateway Chassis上的Distributed gateway port挂了的话，其他的Gateway Chassis上都能检测到它的type是chassisredirect# ovn-sbctl list Port_Binding cr-lr0-public |grep typetype : chassisredirect4, 南北向出虚机流量的流表如下：
a, vrouter上的内部网关(lr0-sw0)与外部网关(lr0-public)将有下列流表(逻辑流)：table=0 (lr_in_admission ), priority=50 , match=(eth.dst == 00:00:00:00:ff:01 && inport == "lr0-sw0"), action=(next;)table=7 (lr_in_ip_routing ), priority=49 , match=(ip4.dst == 172.168.0.0/24), action=(ip.ttl--; reg0 = ip4.dst; reg1 = 172.168.0.200; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;)table=9 (lr_in_gw_redirect ), priority=50 , match=(outport == "lr0-public"), action=(outport = "cr-lr0-public"; next;)b, 假如 cr-lr0-public被schedule在controller-0上，包应该走tunnel porttable=32, priority=100,reg15=0x4,metadata=0x3 actions=load:0x3->NXM_NX_TUN_ID[0..23],set_field:0x4->tun_metadata0,move:NXM_NX_REG14[0..14]->NXM_NX_TUN_METADATA0[16..30],output:ovn-cont-0c, controller-0 chassis收到tunnel port的包，并把它转到lr0
table=0, priority=100,in_port="ovn-comp-0" actions=move:NXM_NX_TUN_ID[0..23]->OXM_OF_METADATA[0..23],move:NXM_NX_TUN_METADATA0[16..30]->NXM_NX_REG14[0..14],move:NXM_NX_TUN_METADATA0[0..15]->NXM_NX_REG15[0..15],resubmit(,33)d, 接着是SNAT
table=1 (lr_out_snat ), priority=25 , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.200);)e, 发到vRouter的外网网关lr0-public port
table=3 (lr_out_delivery ), priority=100 , match=(outport == "lr0-public"), action=(output;)f, 包将localnet port(patch port)进provider bridge(br-ex)并最终到达目的地5, 南北向进虚机流量的流表如下：在controller-0 chassis上：a, 物理网关收到包，走br-ex，再经local net port(patch port)进vRouter(lr0)table=0,priority=100,in_port="patch-br-int-to",dl_vlan=0 actions=strip_vlan,load:0x1->NXM_NX_REG13[],load:0x7->NXM_NX_REG11[],load:0x8->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8)b, DNAT(UnSNAT): 172.168.0.200 -> 10.0.0.3table=0 (lr_in_admission ), priority=50 , match=(eth.dst == 00:00:20:20:12:13 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(next;)table=3 (lr_in_unsnat ), priority=100 , match=(ip && ip4.dst == 172.168.0.200 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)c, 10.0.0.3属于Switch sw0, vRouter(lr0)需将流量导到lr0-sw0table=7 (lr_in_ip_routing ), priority=49 , match=(ip4.dst == 10.0.0.0/24), action=(ip.ttl--; reg0 = ip4.dst; reg1 = 10.0.0.1; eth.src = 00:00:00:00:ff:01; outport = "lr0-sw0"; flags.loopback = 1; next;)e, The ingress pipeline of sw0 is run and the packet is sent to compute-0 via the tunnel port because OVN knows that sw0-port1 resides on compute-0.在compute-0 Chassic上：f, compute-0 receives the traffic on the tunnel port and sends the traffic to the egress pipeline of logical switch sw0, In the egress pipeline, the packet is delivered to sw0-port1.
从ovnmeta ns中ping同台节点虚机的流表
例如tapbdbe0b57-eb/fe:16:3e:ec:f5:9a对应的是虚机，而tape2ed122c-90是ovnmeta ns里的tap
ip netns exec ovnmeta-e2ed122c-9442-4116-9793-5c2323f6de29 ip addr show
tape2ed122c-91@if12 -> tape2ed122c-90@if2# ovs-vsctl -- --columns=name,ofport list Interface tape2ed122c-90
name : tape2ed122c-90
ofport : 3ovs-vsctl -- --columns=name,ofport list Interface tapbdbe0b57-eb
name : tapbdbe0b57-eb
ofport : 2# ovs-ofctl -O OpenFlow13 dump-flows br-int |grep 'in_port=3'cookie=0x3384108f, duration=38141.438s, table=0, n_packets=0, n_bytes=0, priority=100,in_port=3 actions=set_field:0x9->reg13,set_field:0x1->reg11,set_field:0x6->reg12,set_field:0x3->metadata,set_field:0x1->reg14,resubmit(,8)Flow: in_port=3,vlan_tci=0x0000,dl_src=fe:16:3e:ec:f5:9a,dl_dst=12:d6:31:fe:59:a8,dl_type=0x0000bridge("br-int")
----------------- in_port=3, priority 100, cookie 0x3384108fset_field:0x9->reg13set_field:0x1->reg11set_field:0x6->reg12set_field:0x3->metadataset_field:0x1->reg14resubmit(,8)- reg14=0x1,metadata=0x3, priority 50, cookie 0x3c9c4889resubmit(,9)- metadata=0x3, priority 0, cookie 0x6475678dresubmit(,10)- metadata=0x3, priority 0, cookie 0x9593deb1resubmit(,11)- metadata=0x3, priority 0, cookie 0x4db625a0resubmit(,12)- metadata=0x3, priority 0, cookie 0x395e746eresubmit(,13)- metadata=0x3, priority 0, cookie 0x57defbe5resubmit(,14)- metadata=0x3, priority 0, cookie 0x32c7898eresubmit(,15)- metadata=0x3, priority 0, cookie 0xf06ee7a1resubmit(,16)- metadata=0x3, priority 0, cookie 0x351d116cresubmit(,17)- metadata=0x3, priority 0, cookie 0x3f2d1d9eresubmit(,18)- metadata=0x3, priority 0, cookie 0x42601c1aresubmit(,19)- metadata=0x3, priority 0, cookie 0x2b6f92a0resubmit(,20)- metadata=0x3, priority 0, cookie 0xabefbd0dresubmit(,21)- metadata=0x3, priority 0, cookie 0x4854b450resubmit(,22)- metadata=0x3, priority 0, cookie 0xf346b65dresubmit(,23)- metadata=0x3, priority 0, cookie 0xb2d4f8c9resubmit(,24)- metadata=0x3, priority 0, cookie 0x49d3f6beresubmit(,25)- metadata=0x3, priority 0, cookie 0x5bd4e34bresubmit(,26)- metadata=0x3, priority 0, cookie 0x8e0cea9resubmit(,27)- No match.drop
Final flow: reg11=0x1,reg12=0x6,reg13=0x9,reg14=0x1,metadata=0x3,in_port=3,vlan_tci=0x0000,dl_src=fe:16:3e:ec:f5:9a,dl_dst=12:d6:31:fe:59:a8,dl_type=0x0000
Megaflow: recirc_id=0,ct_state=-new-est-rel-rpl-inv-trk,ct_label=0/0x1,eth,in_port=3,dl_src=fe:16:3e:ec:f5:9a,dl_dst=12:d6:31:fe:59:a8,dl_type=0x0000
Datapath actions: drop
从ovnmeta ns中ping同台节点虚机的逻辑流表
https://blog.csdn.net/zhengmx100/article/details/78140948
流表是单个机器的(流表由table组成，table包含flow,flow又有priority, match, action)，逻辑流表是多台机器编排的(逻辑流表下发到每台机器的ovn-controller，而ovn-controller知道根据当前的物理环境即本地端口如何到达其他机器而转换成流表）。
例如创建一个logical switch sw0, 再创建两个port。sw0将有两个pipeline (ingress pipeline与egress pipline), 两个port port在相同主机与不同相同主机所走的pipeline不一样，如下：
逻辑流表就是类似于neutron的port等这些高层概念，它的port只是网络中的一个元素并不限定在某台节点上。可以使用"ovn-sbctl lflow-list"命令来查看完整的logical flow。理解logical flow最好的方式就是使用ovn-trace命令。ovn-trace能够让你看到OVN对一个包是怎么处理的。
ovn-trace DATAPATH MICROFLOW
例如，还是用ovnmeta ns(192.168.21.2/12:d6:31:fe:59:a8)中来ping VM(192.168.21.18/fe:16:3e:ec:f5:9a)
switch eb7511fa-6746-4981-9ddc-6e1c73e9a7ee (neutron-b3187a23-b54e-4cc0-bc6d-4caabdb02b0c) (aka private)port bdbe0b57-eb1e-41d5-a06d-b2fd3050fa4daddresses: ["fa:16:3e:ec:f5:9a 192.168.21.18"]port a6194e3b-6de5-4334-beb7-a898760488d0type: localportaddresses: ["fa:16:3e:de:68:05 192.168.21.2"]...# ovn-trace --summary private 'inport == "a6194e3b-6de5-4334-beb7-a898760488d0" && eth.src == 12:d6:31:fe:59:a8 && eth.dst == fe:16:3e:ec:f5:9a'
# reg14=0x1,vlan_tci=0x0000,dl_src=12:d6:31:fe:59:a8,dl_dst=fe:16:3e:ec:f5:9a,dl_type=0x0000
ingress(dp="private", inport="a6194e") {next;
};
root@juju-e28e17-ovn2-10:/home/ubuntu# ovn-trace --detail private 'inport == "a6194e3b-6de5-4334-beb7-a898760488d0" && eth.src == 12:d6:31:fe:59:a8 && eth.dst == fe:16:3e:ec:f5:9a'
# reg14=0x1,vlan_tci=0x0000,dl_src=12:d6:31:fe:59:a8,dl_dst=fe:16:3e:ec:f5:9a,dl_type=0x0000ingress(dp="private", inport="a6194e")
--------------------------------------0. ls_in_port_sec_l2 (ovn-northd.c:4514): inport == "a6194e", priority 50, uuid 3c9c4889next;
19. ls_in_l2_lkup: no match (implicit drop)
20211228更新 - VM ping GW丢包
VM(10.10.30.20, hosted in ecs4), GW chassis(ecs12)
在计算节点上抓包并分析, 从VM上得ping GW
openstack port list --server <vm>
tcpdump -enli tap<first-11-chars> -p `hostname`_<vm-1-tap>.pcap
tshark -r ecs4_xx.pcap ip.src==10.10.30.20 and icmp
在GW chassis(the node with the highest priority)上抓包
sudo ovn-nbctl lrp-list neutron-<router-uuid>
sudo ovn-nbctl lrp-get-gateway-chassis lrp-<ovn-port-uuid>
tcpdump -enli bond1 "(icmp or arp)" -w `hostname`_bond1.pcap
tshark -r ecs4_xx.pcap ip.src==10.10.30.20 and icmp
确实看到了intermittent pings
$ tshark -r ecs4_37552ee4-38.pcap ip.src==10.10.30.20 and icmp
254 74.664491 10.10.30.20 → 10.10.30.1 ICMP 98 Echo (ping) request id=0x17ab, seq=99/25344, ttl=64
267 75.679441 10.10.30.20 → 10.10.30.1 ICMP 98 Echo (ping) request id=0x17ab, seq=100/25600, ttl=64
268 75.679799 10.10.30.1 → 10.10.30.20 ICMP 98 Echo (ping) reply id=0x17ab, seq=100/25600, ttl=254 (request in 267)
ecs4上的sosreport看到了下列3种error：
$ grep -r 'deferred action limit reached' var/log/kern.log |tail -n1
Nov 8 13:14:30 ecs4 kernel: [9964180.307470] openvswitch: ovs-system: deferred action limit reached, drop recirc action2021-11-10T00:00:31.476Z|147680|poll_loop|INFO|wakeup due to [POLLIN] on fd 3 (10.10.5.180:42162<->10.10.5.166:6642) at lib/stream-ssl.c:832 (101% CPU usage)
2021-11-10T00:01:07.194Z|147681|timeval|WARN|Unreasonably long 1110ms poll interval (1095ms user, 12ms system)
2021-11-10T00:01:07.194Z|147682|timeval|WARN|faults: 17299 minor, 0 major
2021-11-10T00:01:07.194Z|147683|coverage|INFO|Dropped 5 log messages in last 74 seconds (most recently, 35 seconds ago) due to excessive rate$ var/log/ovn/ovn-controller.log.1.gz:2021-11-10T08:21:40.110Z|154925|ovsdb_idl|WARN|transaction error: {"details":"Transaction causes multiple rows in \"MAC_Binding\" table to have identical values (lrp-fbf33f64-0cce-497d-a261-2d3d88e20b80 and \"::\") for index on columns \"logical_port\" and \"ip\". First row, with UUID 4e63d47d-791b-4cc1-ab3c-3d3ac29b5439, existed in the database before this transaction and was not modified by the transaction. Second row, with UUID 6d6281a0-a16e-4fbc-b8b2-da59038f22d5, was inserted by this transaction.","error":"constraint violation"}$ sudo ovn-nbctl show| egrep "^router |lrp-fbf33f64-0cce-497d-a261-2d3d88e20b80"| grep "port lrp" -B1
router 4307456d-3f8b-412c-a784-812f3e73fbfc (neutron-dbf7c13b-751a-41da-b504-09576617213e) (aka ansible-int)
port lrp-fbf33f64-0cce-497d-a261-2d3d88e20b80
$ sudo ovn-nbctl show 4307456d-3f8b-412c-a784-812f3e73fbfc
Possible bugs
[1] https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686
[2] https://bugs.launchpad.net/charm-neutron-api/+bug/1921986
Reference
[1] https://networkop.co.uk/blog/2016/12/10/ovn-part2/
[2] https://numans.blog/2018/11/30/how-to-create-an-open-virtual-network-distributed-gateway-router/