| Message ID | 20240919221205.370244-1-roberto.acosta@luizalabs.com |
|---|---|
| State | New |
| Headers | show |
| Series | [ovs-dev,v8] northd: Fix logical router load-balancer nat rules when using DGP. | expand |
| Context | Check | Description |
|---|---|---|
| ovsrobot/apply-robot | success | apply and check: success |
| ovsrobot/github-robot-_Build_and_Test | success | github build: passed |
| ovsrobot/github-robot-_ovn-kubernetes | fail | github build: failed |
On Thu, Sep 19, 2024 at 6:12 PM Roberto Bartzen Acosta via dev <ovs-dev@openvswitch.org> wrote: > > This commit fixes the build_distr_lrouter_nat_flows_for_lb function to > include a DNAT flow entry for each DGP in use. Since we have added support > to create multiple gateway ports per logical router, it's necessary to > include in the LR NAT rules pipeline a specific entry for each attached DGP. > Otherwise, the inbound traffic will only be redirected when the incoming LRP > matches the chassis_resident field. > > Additionally, this patch includes the ability to use load-balancer with DGPs > attached to multiple chassis. We can have each of the DGPs associated with a > different chassis, and in this case the DNAT rules added by default will not > be enough to guarantee outgoing traffic. > > To solve the multiple chassis for DGPs problem, this patch include a new > config options to be configured in the load-balancer. If the use_stateless_nat > is set to true, the logical router that references this load-balancer will use > Stateless NAT rules when the logical router has multiple DGPs. After applying > this patch and setting the use_stateless_nat option, the inbound and/or > outbound traffic can pass through any chassis where the DGP resides without > having problems with CT state. > > Reported-at: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2054322 > Fixes: 15348b7b806f ("ovn-northd: Multiple distributed gateway port support.") > > Signed-off-by: Roberto Bartzen Acosta <roberto.acosta@luizalabs.com> Hi Roberto, Thanks for the patch. I tested this patch using the test example in multinode.at. The test case adds the below load balancer [root@ovn-central ~]# ovn-nbctl lb-list UUID LB PROTO VIP IPs f3e29869-3bb5-4df0-960a-171106f5913a lb0 tcp 172.16.0.100:9000 10.0.0.3:80,10.0.0.4:80 And the below logical flows are generated by this patch -------- [root@ovn-central ~]# ovn-sbctl dump-flows lr0 | grep 172.16.0.100 table=6 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst == 172.16.0.100), action=(ct_dnat;) table=8 (lr_in_dnat ), priority=120 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && is_chassis_resident("cr-lr0-public-p1")), action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80,10.0.0.4:80);) table=8 (lr_in_dnat ), priority=120 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && is_chassis_resident("cr-lr0-public-p2")), action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80,10.0.0.4:80);) table=3 (lr_out_snat ), priority=160 , match=(ip4 && ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && tcp.src == 80)) && (inport == "lr0-public-p1" || outport == "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), action=(ip4.src=172.16.0.100; tcp.src=9000; next;) table=3 (lr_out_snat ), priority=160 , match=(ip4 && ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && tcp.src == 80)) && (inport == "lr0-public-p2" || outport == "lr0-public-p2") && is_chassis_resident("cr-lr0-public-p2") && tcp), action=(ip4.src=172.16.0.100; tcp.src=9000; next;) -------------- I fail to understand the reason for modifying the ip4.dst before calling ct_lb_mark. Can you please explain why ? Because the action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;) will first modify the ip4.dst to 10.0.0.3 and then to 10.0.0.4 and then the ct_lb_mark will actually do the conntrack with NAT to either 10.0.0.3 or 10.0.0.4. Is it because you want the conntrack entry to not have 172.16.0.100 ? Also I don't understand why this patch adds the logical flows in "lr_out_snat" stage ? Using the system multinode test as an example, the below fails (which is a regression) --- root@ovn-chassis-3 ~]# ip netns exec publicp1 nc -vz 10.0.0.3 80 ---- In the above test, publicp1 with IP 20.0.0.3 when it tries to connect to one if the backends directly (without the LB VIP), it fails. It fails because of the logical flows in "lr_out_snat". Looks to me the solution proposed here is incomplete. Also please note that in our CI we run the multinode tests periodically once a day using the v0.1 of the ovn-fake-multinode and the tests you added will fail. This needs to be fixed and until we move to the latest version of ovn-fake-multinode. Thanks Numan > --- > northd/en-lr-stateful.c | 12 - > northd/northd.c | 116 ++++++-- > ovn-nb.xml | 10 + > tests/multinode-macros.at | 40 +++ > tests/multinode.at | 556 ++++++++++++++++++++++++++++++++++++++ > tests/ovn-northd.at | 320 ++++++++++++++++++++++ > 6 files changed, 1017 insertions(+), 37 deletions(-) > > diff --git a/northd/en-lr-stateful.c b/northd/en-lr-stateful.c > index baf1bd2f8..f09691af6 100644 > --- a/northd/en-lr-stateful.c > +++ b/northd/en-lr-stateful.c > @@ -516,18 +516,6 @@ lr_stateful_record_create(struct lr_stateful_table *table, > > table->array[od->index] = lr_stateful_rec; > > - /* Load balancers are not supported (yet) if a logical router has multiple > - * distributed gateway port. Log a warning. */ > - if (lr_stateful_rec->has_lb_vip && lr_has_multiple_gw_ports(od)) { > - static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1); > - VLOG_WARN_RL(&rl, "Load-balancers are configured on logical " > - "router %s, which has %"PRIuSIZE" distributed " > - "gateway ports. Load-balancer is not supported " > - "yet when there is more than one distributed " > - "gateway port on the router.", > - od->nbr->name, od->n_l3dgw_ports); > - } > - > return lr_stateful_rec; > } > > diff --git a/northd/northd.c b/northd/northd.c > index a267cd5f8..bbe97acf8 100644 > --- a/northd/northd.c > +++ b/northd/northd.c > @@ -11807,31 +11807,30 @@ static void > build_distr_lrouter_nat_flows_for_lb(struct lrouter_nat_lb_flows_ctx *ctx, > enum lrouter_nat_lb_flow_type type, > struct ovn_datapath *od, > - struct lflow_ref *lflow_ref) > + struct lflow_ref *lflow_ref, > + struct ovn_port *dgp, > + bool stateless_nat) > { > - struct ovn_port *dgp = od->l3dgw_ports[0]; > - > - const char *undnat_action; > - > - switch (type) { > - case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > - undnat_action = "flags.force_snat_for_lb = 1; next;"; > - break; > - case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > - undnat_action = "flags.skip_snat_for_lb = 1; next;"; > - break; > - case LROUTER_NAT_LB_FLOW_NORMAL: > - case LROUTER_NAT_LB_FLOW_MAX: > - undnat_action = lrouter_use_common_zone(od) > - ? "ct_dnat_in_czone;" > - : "ct_dnat;"; > - break; > - } > + struct ds dnat_action = DS_EMPTY_INITIALIZER; > > /* Store the match lengths, so we can reuse the ds buffer. */ > size_t new_match_len = ctx->new_match->length; > size_t undnat_match_len = ctx->undnat_match->length; > > + /* dnat_action: Add the LB backend IPs as a destination action of the > + * lr_in_dnat NAT rule with cumulative effect because any > + * backend dst IP used in the action list will redirect the > + * packet to the ct_lb pipeline. > + */ > + if (stateless_nat) { > + for (size_t i = 0; i < ctx->lb_vip->n_backends; i++) { > + struct ovn_lb_backend *backend = &ctx->lb_vip->backends[i]; > + bool ipv6 = !IN6_IS_ADDR_V4MAPPED(&backend->ip); > + ds_put_format(&dnat_action, "%s.dst=%s;", ipv6 ? "ip6" : "ip4", > + backend->ip_str); > + } > + } > + ds_put_format(&dnat_action, "%s", ctx->new_action[type]); > > const char *meter = NULL; > > @@ -11841,20 +11840,46 @@ build_distr_lrouter_nat_flows_for_lb(struct lrouter_nat_lb_flows_ctx *ctx, > > if (ctx->lb_vip->n_backends || !ctx->lb_vip->empty_backend_rej) { > ds_put_format(ctx->new_match, " && is_chassis_resident(%s)", > - od->l3dgw_ports[0]->cr_port->json_key); > + dgp->cr_port->json_key); > } > > ovn_lflow_add_with_hint__(ctx->lflows, od, S_ROUTER_IN_DNAT, ctx->prio, > - ds_cstr(ctx->new_match), ctx->new_action[type], > + ds_cstr(ctx->new_match), ds_cstr(&dnat_action), > NULL, meter, &ctx->lb->nlb->header_, > lflow_ref); > > ds_truncate(ctx->new_match, new_match_len); > > + ds_destroy(&dnat_action); > if (!ctx->lb_vip->n_backends) { > return; > } > > + struct ds undnat_action = DS_EMPTY_INITIALIZER; > + struct ds snat_action = DS_EMPTY_INITIALIZER; > + > + switch (type) { > + case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > + ds_put_format(&undnat_action, "flags.force_snat_for_lb = 1; next;"); > + break; > + case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > + ds_put_format(&undnat_action, "flags.skip_snat_for_lb = 1; next;"); > + break; > + case LROUTER_NAT_LB_FLOW_NORMAL: > + case LROUTER_NAT_LB_FLOW_MAX: > + ds_put_format(&undnat_action, "%s", > + lrouter_use_common_zone(od) ? "ct_dnat_in_czone;" > + : "ct_dnat;"); > + break; > + } > + > + /* undnat_action: Remove the ct action from the lr_out_undenat NAT rule. > + */ > + if (stateless_nat) { > + ds_clear(&undnat_action); > + ds_put_format(&undnat_action, "next;"); > + } > + > /* We need to centralize the LB traffic to properly perform > * the undnat stage. > */ > @@ -11873,11 +11898,41 @@ build_distr_lrouter_nat_flows_for_lb(struct lrouter_nat_lb_flows_ctx *ctx, > ds_put_format(ctx->undnat_match, ") && (inport == %s || outport == %s)" > " && is_chassis_resident(%s)", dgp->json_key, dgp->json_key, > dgp->cr_port->json_key); > + /* Use the LB protocol as matching criteria for out undnat and snat when > + * creating LBs with stateless NAT. */ > + if (stateless_nat) { > + ds_put_format(ctx->undnat_match, " && %s", ctx->lb->proto); > + } > ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_UNDNAT, 120, > - ds_cstr(ctx->undnat_match), undnat_action, > - &ctx->lb->nlb->header_, > + ds_cstr(ctx->undnat_match), > + ds_cstr(&undnat_action), &ctx->lb->nlb->header_, > lflow_ref); > + > + /* snat_action: Add a new lr_out_snat rule with the LB VIP as source IP > + * action to perform the NAT stateless pipeline completely. > + */ > + if (stateless_nat) { > + if (ctx->lb_vip->port_str) { > + ds_put_format(&snat_action, "%s.src=%s; %s.src=%s; next;", > + ctx->lb_vip->address_family == AF_INET6 ? > + "ip6" : "ip4", > + ctx->lb_vip->vip_str, ctx->lb->proto, > + ctx->lb_vip->port_str); > + } else { > + ds_put_format(&snat_action, "%s.src=%s; next;", > + ctx->lb_vip->address_family == AF_INET6 ? > + "ip6" : "ip4", > + ctx->lb_vip->vip_str); > + } > + ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_SNAT, 160, > + ds_cstr(ctx->undnat_match), > + ds_cstr(&snat_action), &ctx->lb->nlb->header_, > + lflow_ref); > + } > + > ds_truncate(ctx->undnat_match, undnat_match_len); > + ds_destroy(&undnat_action); > + ds_destroy(&snat_action); > } > > static void > @@ -12022,6 +12077,8 @@ build_lrouter_nat_flows_for_lb( > * lflow generation for them. > */ > size_t index; > + bool use_stateless_nat = smap_get_bool(&lb->nlb->options, > + "use_stateless_nat", false); > BITMAP_FOR_EACH_1 (index, bitmap_len, lb_dps->nb_lr_map) { > struct ovn_datapath *od = lr_datapaths->array[index]; > enum lrouter_nat_lb_flow_type type; > @@ -12043,8 +12100,17 @@ build_lrouter_nat_flows_for_lb( > if (!od->n_l3dgw_ports) { > bitmap_set1(gw_dp_bitmap[type], index); > } else { > - build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > - lb_dps->lflow_ref); > + /* Create stateless LB NAT rules when using multiple DGPs and > + * use_stateless_nat is true. > + */ > + bool stateless_nat = (od->n_l3dgw_ports > 1) > + ? use_stateless_nat : false; > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > + struct ovn_port *dgp = od->l3dgw_ports[i]; > + build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > + lb_dps->lflow_ref, dgp, > + stateless_nat); > + } > } > > if (lb->affinity_timeout) { > diff --git a/ovn-nb.xml b/ovn-nb.xml > index 2836f58f5..ad03c6214 100644 > --- a/ovn-nb.xml > +++ b/ovn-nb.xml > @@ -2302,6 +2302,16 @@ or > local anymore by the ovn-controller. This option is set to > <code>false</code> by default. > </column> > + > + <column name="options" key="use_stateless_nat" > + type='{"type": "boolean"}'> > + If the load balancer is configured with <code>use_stateless_nat</code> > + option to <code>true</code>, the logical router that references this > + load balancer will use Stateless NAT rules when the logical router > + has multiple distributed gateway ports(DGP). Otherwise, the outbound > + traffic may be dropped in scenarios where we have different chassis > + for each DGP. This option is set to <code>false</code> by default. > + </column> > </group> > </table> > > diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at > index 757917626..2f69433fc 100644 > --- a/tests/multinode-macros.at > +++ b/tests/multinode-macros.at > @@ -40,6 +40,27 @@ m4_define([M_START_TCPDUMP], > ] > ) > > +# M_EXEC([fake_node], [command]) > +# > +# Execute 'command' in 'fakenode' > +m4_define([M_EXEC], > + [podman exec $1 $2]) > + > +# M_CHECK_EXEC([fake_node], [command], other_params...) > +# > +# Wrapper for AT_CHECK that executes 'command' inside 'fake_node''s'. > +# 'other_params' as passed as they are to AT_CHECK. > +m4_define([M_CHECK_EXEC], > + [ AT_CHECK([M_EXEC([$1], [$2])], m4_shift(m4_shift(m4_shift($@)))) ] > +) > + > +# M_FORMAT_CT([ip-addr]) > +# > +# Strip content from the piped input which would differ from test to test > +# and limit the output to the rows containing 'ip-addr'. > +# > +m4_define([M_FORMAT_CT], > + [[grep -F "dst=$1," | sed -e 's/id=[0-9]*/id=<cleared>/g' -e 's/state=[0-9_A-Z]*/state=<cleared>/g' | sort | uniq | sed -e 's/zone=[[0-9]]*/zone=<cleared>/' -e 's/mark=[[0-9]]*/mark=<cleared>/' ]]) > > OVS_START_SHELL_HELPERS > > @@ -76,6 +97,25 @@ multinode_nbctl () { > m_as ovn-central ovn-nbctl "$@" > } > > +check_fake_multinode_setup_by_nodes() { > + check m_as ovn-central ovn-nbctl --wait=sb sync > + for c in $1 > + do > + AT_CHECK([m_as $c ovn-appctl -t ovn-controller version], [0], [ignore]) > + done > +} > + > +cleanup_multinode_resources_by_nodes() { > + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db > + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd > + check m_as ovn-central ovn-nbctl --wait=sb sync > + for c in $1 > + do > + m_as $c ovs-vsctl del-br br-int > + m_as $c ip --all netns delete > + done > +} > + > # m_count_rows TABLE [CONDITION...] > # > # Prints the number of rows in TABLE (that satisfy CONDITION). > diff --git a/tests/multinode.at b/tests/multinode.at > index a0eb8fc67..b1beb4d97 100644 > --- a/tests/multinode.at > +++ b/tests/multinode.at > @@ -1591,3 +1591,559 @@ AT_CHECK([cat ch1_eth2.tcpdump], [0], [dnl > ]) > > AT_CLEANUP > + > +AT_SETUP([ovn multinode load-balancer with multiple DGPs and multiple chassis]) > + > +# Check that ovn-fake-multinode setup is up and running - requires additional nodes > +check_fake_multinode_setup_by_nodes 'ovn-chassis-1 ovn-chassis-2 ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > + > +# Delete the multinode NB and OVS resources before starting the test. > +cleanup_multinode_resources_by_nodes 'ovn-chassis-1 ovn-chassis-2 ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > + > +# Network topology > +# > +# publicp1 (ovn-chassis-3) (20.0.0.3/24) > +# | > +# overlay > +# | > +# DGP public1 (ovn-gw-1) (20.0.0.1/24) > +# | > +# | > +# | > +# lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) 10.0.0.3/24 > +# | | > +# | + --- sw0p2 (ovn-chassis-2) 10.0.0.4/24 > +# | > +# DGP public2 (ovn-gw-2) (30.0.0.1/24) > +# | > +# overlay > +# | > +# publicp2 (ovn-chassis-4) (30.0.0.3/24) > + > +# Delete already used ovs-ports > +m_as ovn-chassis-1 ovs-vsctl del-port br-int sw0p1-p > +m_as ovn-chassis-2 ovs-vsctl del-port br-int sw0p2-p > +m_as ovn-chassis-1 ip link del sw0p1-p > +m_as ovn-chassis-2 ip link del sw0p2-p > +m_as ovn-chassis-3 ovs-vsctl del-port br-int publicp1-p > +m_as ovn-chassis-4 ovs-vsctl del-port br-int publicp2-p > +m_as ovn-chassis-3 ip link del publicp1-p > +m_as ovn-chassis-4 ip link del publicp2-p > + > +# Create East-West switch for LB backends > +check multinode_nbctl ls-add sw0 > +check multinode_nbctl lsp-add sw0 sw0-port1 > +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 10.0.0.3 1000::3" > +check multinode_nbctl lsp-add sw0 sw0-port2 > +check multinode_nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 10.0.0.4 1000::4" > + > +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 50:54:00:00:00:03 1400 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 50:54:00:00:00:04 1400 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a > + > +m_wait_for_ports_up > + > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 10.0.0.4 | FORMAT_PING], \ > +[0], [dnl > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > +]) > + > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 10.0.0.3 | FORMAT_PING], \ > +[0], [dnl > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > +]) > + > +# Create a logical router and attach to sw0 > +check multinode_nbctl lr-add lr0 > +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 1000::a/64 > +check multinode_nbctl lsp-add sw0 sw0-lr0 > +check multinode_nbctl lsp-set-type sw0-lr0 router > +check multinode_nbctl lsp-set-addresses sw0-lr0 router > +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 > + > +# create external connection for N/S traffic using multiple DGPs > +check multinode_nbctl ls-add public > + > +# DGP public1 > +check multinode_nbctl lsp-add public ln-public-1 > +check multinode_nbctl lsp-set-type ln-public-1 localnet > +check multinode_nbctl lsp-set-addresses ln-public-1 unknown > +check multinode_nbctl lsp-set-options ln-public-1 network_name=public1 > + > +# DGP public2 > +# create exteranl connection for N/S traffic > +check multinode_nbctl lsp-add public ln-public-2 > +check multinode_nbctl lsp-set-type ln-public-2 localnet > +check multinode_nbctl lsp-set-addresses ln-public-2 unknown > +check multinode_nbctl lsp-set-options ln-public-2 network_name=public2 > + > +# Attach DGP public1 to GW-1 and chassis-3 (overlay connectivity) > +m_as ovn-gw-1 ovs-vsctl set open . external-ids:ovn-bridge-mappings=public1:br-ex > +m_as ovn-chassis-3 ovs-vsctl set open . external-ids:ovn-bridge-mappings=public1:br-ex > + > +# Attach DGP public2 to GW-2 and chassis-4 (overlay connectivity) > +m_as ovn-gw-2 ovs-vsctl set open . external-ids:ovn-bridge-mappings=public2:br-ex > +m_as ovn-chassis-4 ovs-vsctl set open . external-ids:ovn-bridge-mappings=public2:br-ex > + > +# Create the external LR0 port to the DGP public1 > +check multinode_nbctl lsp-add public public-port1 > +check multinode_nbctl lsp-set-addresses public-port1 "40:54:00:00:00:03 20.0.0.3 2000::3" > + > +check multinode_nbctl lrp-add lr0 lr0-public-p1 00:00:00:00:ff:02 20.0.0.1/24 2000::a/64 > +check multinode_nbctl lsp-add public public-lr0-p1 > +check multinode_nbctl lsp-set-type public-lr0-p1 router > +check multinode_nbctl lsp-set-addresses public-lr0-p1 router > +check multinode_nbctl lsp-set-options public-lr0-p1 router-port=lr0-public-p1 > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p1 ovn-gw-1 10 > + > +# Create a VM on ovn-chassis-3 in the same public1 overlay > +m_as ovn-chassis-3 /data/create_fake_vm.sh public-port1 publicp1 40:54:00:00:00:03 1400 20.0.0.3 24 20.0.0.1 2000::4/64 2000::a > + > +m_wait_for_ports_up public-port1 > + > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [ping -q -c 3 -i 0.3 -w 2 20.0.0.1 | FORMAT_PING], \ > +[0], [dnl > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > +]) > + > +# Create the external LR0 port to the DGP public2 > +check multinode_nbctl lsp-add public public-port2 > +check multinode_nbctl lsp-set-addresses public-port2 "60:54:00:00:00:03 30.0.0.3 3000::3" > + > +check multinode_nbctl lrp-add lr0 lr0-public-p2 00:00:00:00:ff:03 30.0.0.1/24 3000::a/64 > +check multinode_nbctl lsp-add public public-lr0-p2 > +check multinode_nbctl lsp-set-type public-lr0-p2 router > +check multinode_nbctl lsp-set-addresses public-lr0-p2 router > +check multinode_nbctl lsp-set-options public-lr0-p2 router-port=lr0-public-p2 > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p2 ovn-gw-2 10 > + > +# Create a VM on ovn-chassis-4 in the same public2 overlay > +m_as ovn-chassis-4 /data/create_fake_vm.sh public-port2 publicp2 60:54:00:00:00:03 1400 30.0.0.3 24 30.0.0.1 3000::4/64 3000::a > + > +m_wait_for_ports_up public-port2 > + > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [ping -q -c 3 -i 0.3 -w 2 30.0.0.1 | FORMAT_PING], \ > +[0], [dnl > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > +]) > + > +# Add a default route for multiple DGPs - using ECMP > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 20.0.0.3 > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 30.0.0.3 > + > +# Add SNAT rules using gateway-port > +check multinode_nbctl --gateway-port lr0-public-p1 lr-nat-add lr0 snat 20.0.0.1 10.0.0.0/24 > +check multinode_nbctl --gateway-port lr0-public-p2 lr-nat-add lr0 snat 30.0.0.1 10.0.0.0/24 > + > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 20.0.0.3 | FORMAT_PING], \ > +[0], [dnl > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > +]) > + > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 30.0.0.3 | FORMAT_PING], \ > +[0], [dnl > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > +]) > + > +# create LB > +check multinode_nbctl lb-add lb0 "172.16.0.100:80" "10.0.0.3:80,10.0.0.4:80" > +check multinode_nbctl lr-lb-add lr0 lb0 > +check multinode_nbctl ls-lb-add sw0 lb0 > + > +# Set use_stateless_nat to true > +check multinode_nbctl set load_balancer lb0 options:use_stateless_nat=true > + > +# Start backend http services > +M_NS_DAEMONIZE([ovn-chassis-1], [sw0p1], [$PYTHON -m http.server --bind 10.0.0.3 80 >/dev/null 2>&1], [http1.pid]) > +M_NS_DAEMONIZE([ovn-chassis-2], [sw0p2], [$PYTHON -m http.server --bind 10.0.0.4 80 >/dev/null 2>&1], [http2.pid]) > + > +# wait for http server be ready > +sleep 2 > + > +# Flush conntrack entries for easier output parsing of next test. > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > + > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v 172.16.0.100:80 --retry 3 --max-time 1 --local-port 59002 2> curl.out']) > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [sh -c 'cat curl.out | grep -i -e connect | grep -v 'Server:''], \ > +[0], [dnl > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > +* Closing connection > +]) > + > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v 172.16.0.100:80 --retry 3 --max-time 1 --local-port 59003 2> curl.out']) > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [sh -c 'cat curl.out | grep -i -e connect | grep -v 'Server:''], \ > +[0], [dnl > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > +* Closing connection > +]) > + > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > + > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v 172.16.0.100:80 --retry 3 --max-time 1 --local-port 59001']) > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | M_FORMAT_CT(20.0.0.3) | \ > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v 172.16.0.100:80 --retry 3 --max-time 1 --local-port 59000']) > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | M_FORMAT_CT(30.0.0.3) | \ > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +# create a big file on web servers for download > +M_NS_EXEC([ovn-chassis-1], [sw0p1], [dd bs=512 count=200000 if=/dev/urandom of=download_file]) > +M_NS_EXEC([ovn-chassis-2], [sw0p2], [dd bs=512 count=200000 if=/dev/urandom of=download_file]) > + > +# Flush conntrack entries for easier output parsing of next test. > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > + > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59004 2>curl.out']) > + > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') > + > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat curl.out | \ > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > +* Closing connection > +]) > + > +# Check if we have only one backend for the same connection - orig + dest ports > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +# Check if gw-2 is empty to ensure that the traffic only come from/to the originator chassis via DGP public1 > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > +0 > +]) > + > +# Check the backend IP from ct entries on gw-1 (DGP public1) > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" -c) > + > +if [[ $backend_check -gt 0 ]]; then > +# Backend resides on ovn-chassis-1 > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > +grep tcp], [0], [dnl > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59004,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +# Ensure that the traffic only come from ovn-chassis-1 > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +else > +# Backend resides on ovn-chassis-2 > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > +grep tcp], [0], [dnl > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59004,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +# Ensure that the traffic only come from ovn-chassis-2 > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +fi > + > +# Flush conntrack entries for easier output parsing of next test. > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > + > +# Check the flows again for a new source port > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59005 2>curl.out']) > + > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') > + > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat curl.out | \ > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > +* Closing connection > +]) > + > +# Check if we have only one backend for the same connection - orig + dest ports > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +# Check if gw-2 is empty to ensure that the traffic only come from/to the originator chassis via DGP public1 > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > +0 > +]) > + > +# Check the backend IP from ct entries on gw-1 (DGP public1) > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" -c) > + > +if [[ $backend_check -gt 0 ]]; then > +# Backend resides on ovn-chassis-1 > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > +grep tcp], [0], [dnl > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59005,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +# Ensure that the traffic only come from ovn-chassis-1 > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +else > +# Backend resides on ovn-chassis-2 > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > +grep tcp], [0], [dnl > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59005,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +# Ensure that the traffic only come from ovn-chassis-2 > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +fi > + > +# Flush conntrack entries for easier output parsing of next test. > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > + > +# Start a new test using the second DGP as origin (public2) > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59006 2>curl.out']) > + > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') > + > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat curl.out | \ > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > +* Closing connection > +]) > + > +# Check if we have only one backend for the same connection - orig + dest ports > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +# Check if gw-1 is empty to ensure that the traffic only come from/to the originator chassis via DGP public2 > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > +0 > +]) > + > +# Check the backend IP from ct entries on gw-2 (DGP public2) > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" -c) > + > +if [[ $backend_check -gt 0 ]]; then > +# Backend resides on ovn-chassis-1 > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > +grep tcp], [0], [dnl > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59006,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +# Ensure that the traffic only come from ovn-chassis-1 > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +else > +# Backend resides on ovn-chassis-2 > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > +grep tcp], [0], [dnl > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59006,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +# Ensure that the traffic only come from ovn-chassis-2 > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +fi > + > +# Flush conntrack entries for easier output parsing of next test. > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > + > +# Check the flows again for a new source port using the second DGP as origin (public2) > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59007 2>curl.out']) > + > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') > + > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat curl.out | \ > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > +* Closing connection > +]) > + > +# Check if we have only one backend for the same connection - orig + dest ports > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +# Check if gw-1 is empty to ensure that the traffic only come from/to the originator chassis via DGP public2 > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > +0 > +]) > + > +# Check the backend IP from ct entries on gw-1 (DGP public1) > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" -c) > + > +if [[ $backend_check -gt 0 ]]; then > +# Backend resides on ovn-chassis-1 > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > +grep tcp], [0], [dnl > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59007,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +# Ensure that the traffic only come from ovn-chassis-1 > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +else > +# Backend resides on ovn-chassis-2 > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > +grep tcp], [0], [dnl > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59007,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +# Ensure that the traffic only come from ovn-chassis-2 > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl > +0 > +]) > +fi > + > +# Check multiple requests coming from DGP's public1 and public2 > + > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep -v 'Server:'], [0], [dnl > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > +200 OK > +* Closing connection > +]) > + > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep -v 'Server:'], [0], [dnl > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > +200 OK > +* Closing connection > +]) > + > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep -v 'Server:'], [0], [dnl > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > +200 OK > +* Closing connection > +]) > + > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep -v 'Server:'], [0], [dnl > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > +200 OK > +* Closing connection > +]) > + > +# Remove the LB and change the VIP port - different from the backend ports > +check multinode_nbctl lb-del lb0 > + > +# create LB again > +check multinode_nbctl lb-add lb0 "172.16.0.100:9000" "10.0.0.3:80,10.0.0.4:80" > +check multinode_nbctl lr-lb-add lr0 lb0 > +check multinode_nbctl ls-lb-add sw0 lb0 > + > +# Set use_stateless_nat to true > +check multinode_nbctl set load_balancer lb0 options:use_stateless_nat=true > + > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > + > +# Check end-to-end request using a new port for VIP > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port 59008 2>curl.out']) > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | M_FORMAT_CT(20.0.0.3) | \ > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep -v 'Server:'], [0], [dnl > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > +200 OK > +* Closing connection > +]) > + > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > + > +# Check end-to-end request using a new port for VIP > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port 59008 2>curl.out']) > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | M_FORMAT_CT(30.0.0.3) | \ > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > +]) > + > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep -v 'Server:'], [0], [dnl > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > +200 OK > +* Closing connection > +]) > + > +AT_CLEANUP > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > index dcc3dbbc3..9e7a2f225 100644 > --- a/tests/ovn-northd.at > +++ b/tests/ovn-northd.at > @@ -13864,3 +13864,323 @@ check_no_redirect > > AT_CLEANUP > ]) > + > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + NAT Stateless)]) > +ovn_start > + > +check ovn-nbctl ls-add public > +check ovn-nbctl lr-add lr1 > + > +# lr1 DGP ts1 > +check ovn-nbctl ls-add ts1 > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 172.16.10.1/24 > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > + > +# lr1 DGP ts2 > +check ovn-nbctl ls-add ts2 > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 172.16.20.1/24 > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > + > +# lr1 DGP public > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 173.16.0.1/16 > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > + > +check ovn-nbctl ls-add s1 > +# s1 - lr1 > +check ovn-nbctl lsp-add s1 s1_lr1 > +check ovn-nbctl lsp-set-type s1_lr1 router > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 172.16.0.1" > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > + > +# s1 - backend vm1 > +check ovn-nbctl lsp-add s1 vm1 > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 172.16.0.101" > + > +# s1 - backend vm2 > +check ovn-nbctl lsp-add s1 vm2 > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 172.16.0.102" > + > +# s1 - backend vm3 > +check ovn-nbctl lsp-add s1 vm3 > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 172.16.0.103" > + > +# Add the lr1 DGP ts1 to the public switch > +check ovn-nbctl lsp-add public public_lr1_ts1 > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 nat-addresses=router > + > +# Add the lr1 DGP ts2 to the public switch > +check ovn-nbctl lsp-add public public_lr1_ts2 > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 nat-addresses=router > + > +# Add the lr1 DGP public to the public switch > +check ovn-nbctl lsp-add public public_lr1 > +check ovn-nbctl lsp-set-type public_lr1 router > +check ovn-nbctl lsp-set-addresses public_lr1 router > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public nat-addresses=router > + > +# Create the Load Balancer lb1 > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" "172.16.0.103,172.16.0.102,172.16.0.101" > + > +# Set use_stateless_nat to true > +check ovn-nbctl --wait=sb set load_balancer lb1 options:use_stateless_nat=true > + > +# Associate load balancer to s1 > +check ovn-nbctl ls-lb-add s1 lb1 > +check ovn-nbctl --wait=sb sync > + > +ovn-sbctl dump-flows s1 > s1flows > +AT_CAPTURE_FILE([s1flows]) > + > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep "30.0.0.1"], [0], [dnl > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > +]) > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep "30.0.0.1"], [0], [dnl > + table=??(ls_in_lb ), priority=110 , match=(ct.new && ip4.dst == 30.0.0.1), action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > +]) > + > +# Associate load balancer to lr1 with DGP > +check ovn-nbctl lr-lb-add lr1 lb1 > +check ovn-nbctl --wait=sb sync > + > +ovn-sbctl dump-flows lr1 > lr1flows > +AT_CAPTURE_FILE([lr1flows]) > + > +# Check stateless NAT rules for load balancer with multiple DGP > +# 1. Check if the backend IPs are in the ipX.dst action > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep "30.0.0.1"], [0], [dnl > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 30.0.0.1 && is_chassis_resident("cr-lr1-ts1")), action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 30.0.0.1 && is_chassis_resident("cr-lr1-ts2")), action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 30.0.0.1 && is_chassis_resident("cr-lr1_public")), action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > +]) > + > +# 2. Check if the DGP ports are in the match with action next > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], [dnl > + table=??(lr_out_undnat ), priority=0 , match=(1), action=(next;) > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), action=(next;) > +]) > + > +# 3. Check if the VIP IP is in the ipX.src action > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > + table=??(lr_out_snat ), priority=0 , match=(1), action=(next;) > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), action=(next;) > + table=??(lr_out_snat ), priority=160 , match=(ip4 && ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(ip4.src=30.0.0.1; next;) > + table=??(lr_out_snat ), priority=160 , match=(ip4 && ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(ip4.src=30.0.0.1; next;) > + table=??(lr_out_snat ), priority=160 , match=(ip4 && ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), action=(ip4.src=30.0.0.1; next;) > +]) > + > +AT_CLEANUP > +]) > + > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + NAT Stateless) - IPv6]) > +ovn_start > + > +check ovn-nbctl ls-add public > +check ovn-nbctl lr-add lr1 > + > +# lr1 DGP ts1 > +check ovn-nbctl ls-add ts1 > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 2001:db8:aaaa:1::1/64 > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > + > +# lr1 DGP ts2 > +check ovn-nbctl ls-add ts2 > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 2001:db8:aaaa:2::1/64 > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > + > +# lr1 DGP public > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 2001:db8:bbbb::1/64 > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 2001:db8:aaaa:3::1/64 > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > + > +check ovn-nbctl ls-add s1 > +# s1 - lr1 > +check ovn-nbctl lsp-add s1 s1_lr1 > +check ovn-nbctl lsp-set-type s1_lr1 router > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 2001:db8:aaaa:3::1" > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > + > +# s1 - backend vm1 > +check ovn-nbctl lsp-add s1 vm1 > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 2001:db8:aaaa:3::101" > + > +# s1 - backend vm2 > +check ovn-nbctl lsp-add s1 vm2 > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 2001:db8:aaaa:3::102" > + > +# s1 - backend vm3 > +check ovn-nbctl lsp-add s1 vm3 > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 2001:db8:aaaa:3::103" > + > +# Add the lr1 DGP ts1 to the public switch > +check ovn-nbctl lsp-add public public_lr1_ts1 > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 nat-addresses=router > + > +# Add the lr1 DGP ts2 to the public switch > +check ovn-nbctl lsp-add public public_lr1_ts2 > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 nat-addresses=router > + > +# Add the lr1 DGP public to the public switch > +check ovn-nbctl lsp-add public public_lr1 > +check ovn-nbctl lsp-set-type public_lr1 router > +check ovn-nbctl lsp-set-addresses public_lr1 router > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public nat-addresses=router > + > +# Create the Load Balancer lb1 > +check ovn-nbctl --wait=sb lb-add lb1 "2001:db8:cccc::1" "2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101" > + > +# Set use_stateless_nat to true > +check ovn-nbctl --wait=sb set load_balancer lb1 options:use_stateless_nat=true > + > +# Associate load balancer to s1 > +check ovn-nbctl ls-lb-add s1 lb1 > +check ovn-nbctl --wait=sb sync > + > +ovn-sbctl dump-flows s1 > s1flows > +AT_CAPTURE_FILE([s1flows]) > + > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep "2001:db8:cccc::1"], [0], [dnl > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 && ip6.dst == 2001:db8:cccc::1), action=(xxreg1 = 2001:db8:cccc::1; ct_lb_mark;) > +]) > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep "2001:db8:cccc::1"], [0], [dnl > + table=??(ls_in_lb ), priority=110 , match=(ct.new && ip6.dst == 2001:db8:cccc::1), action=(ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > +]) > + > +# Associate load balancer to lr1 with DGP > +check ovn-nbctl lr-lb-add lr1 lb1 > +check ovn-nbctl --wait=sb sync > + > +ovn-sbctl dump-flows lr1 > lr1flows > +AT_CAPTURE_FILE([lr1flows]) > + > +# Check stateless NAT rules for load balancer with multiple DGP > +# 1. Check if the backend IPs are in the ipX.dst action > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep "2001:db8:cccc::1"], [0], [dnl > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && is_chassis_resident("cr-lr1-ts1")), action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && is_chassis_resident("cr-lr1-ts2")), action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && is_chassis_resident("cr-lr1_public")), action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > +]) > + > +# 2. Check if the DGP ports are in the match with action next > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], [dnl > + table=??(lr_out_undnat ), priority=0 , match=(1), action=(next;) > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || outport == "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), action=(next;) > +]) > + > +# 3. Check if the VIP IP is in the ipX.src action > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > + table=??(lr_out_snat ), priority=0 , match=(1), action=(next;) > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), action=(next;) > + table=??(lr_out_snat ), priority=160 , match=(ip6 && ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(ip6.src=2001:db8:cccc::1; next;) > + table=??(lr_out_snat ), priority=160 , match=(ip6 && ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(ip6.src=2001:db8:cccc::1; next;) > + table=??(lr_out_snat ), priority=160 , match=(ip6 && ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || outport == "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), action=(ip6.src=2001:db8:cccc::1; next;) > +]) > + > +AT_CLEANUP > +]) > + > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > +AT_SETUP([Load balancer with Distributed Gateway Ports (DGP)]) > +ovn_start > + > +check ovn-nbctl ls-add public > +check ovn-nbctl lr-add lr1 > + > +# lr1 DGP ts1 > +check ovn-nbctl ls-add ts1 > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 172.16.10.1/24 > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-1 > + > +# lr1 DGP ts2 > +check ovn-nbctl ls-add ts2 > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 172.16.20.1/24 > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-1 > + > +# lr1 DGP public > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 173.16.0.1/16 > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > + > +check ovn-nbctl ls-add s1 > +# s1 - lr1 > +check ovn-nbctl lsp-add s1 s1_lr1 > +check ovn-nbctl lsp-set-type s1_lr1 router > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 172.16.0.1" > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > + > +# s1 - backend vm1 > +check ovn-nbctl lsp-add s1 vm1 > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 172.16.0.101" > + > +# s1 - backend vm2 > +check ovn-nbctl lsp-add s1 vm2 > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 172.16.0.102" > + > +# s1 - backend vm3 > +check ovn-nbctl lsp-add s1 vm3 > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 172.16.0.103" > + > +# Add the lr1 DGP ts1 to the public switch > +check ovn-nbctl lsp-add public public_lr1_ts1 > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 nat-addresses=router > + > +# Add the lr1 DGP ts2 to the public switch > +check ovn-nbctl lsp-add public public_lr1_ts2 > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 nat-addresses=router > + > +# Add the lr1 DGP public to the public switch > +check ovn-nbctl lsp-add public public_lr1 > +check ovn-nbctl lsp-set-type public_lr1 router > +check ovn-nbctl lsp-set-addresses public_lr1 router > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public nat-addresses=router > + > +# Create the Load Balancer lb1 > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" "172.16.0.103,172.16.0.102,172.16.0.101" > + > +# Associate load balancer to s1 > +check ovn-nbctl ls-lb-add s1 lb1 > +check ovn-nbctl --wait=sb sync > + > +ovn-sbctl dump-flows s1 > s1flows > +AT_CAPTURE_FILE([s1flows]) > + > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep "30.0.0.1"], [0], [dnl > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > +]) > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep "30.0.0.1"], [0], [dnl > + table=??(ls_in_lb ), priority=110 , match=(ct.new && ip4.dst == 30.0.0.1), action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > +]) > + > +# Associate load balancer to lr1 with DGP > +check ovn-nbctl lr-lb-add lr1 lb1 > +check ovn-nbctl --wait=sb sync > + > +ovn-sbctl dump-flows lr1 > lr1flows > +AT_CAPTURE_FILE([lr1flows]) > + > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep "30.0.0.1"], [0], [dnl > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 30.0.0.1 && is_chassis_resident("cr-lr1-ts1")), action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 30.0.0.1 && is_chassis_resident("cr-lr1-ts2")), action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 30.0.0.1 && is_chassis_resident("cr-lr1_public")), action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > +]) > + > +AT_CLEANUP > +]) > -- > 2.34.1 > > > -- > > > > > _'Esta mensagem é direcionada apenas para os endereços constantes no > cabeçalho inicial. Se você não está listado nos endereços constantes no > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão > imediatamente anuladas e proibidas'._ > > > * **'Apesar do Magazine Luiza tomar > todas as precauções razoáveis para assegurar que nenhum vírus esteja > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos'.* > > > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev >
Hi Numan, Thanks for your feedback and review. Em qua., 25 de set. de 2024 às 19:50, Numan Siddique <numans@ovn.org> escreveu: > On Thu, Sep 19, 2024 at 6:12 PM Roberto Bartzen Acosta via dev > <ovs-dev@openvswitch.org> wrote: > > > > This commit fixes the build_distr_lrouter_nat_flows_for_lb function to > > include a DNAT flow entry for each DGP in use. Since we have added > support > > to create multiple gateway ports per logical router, it's necessary to > > include in the LR NAT rules pipeline a specific entry for each attached > DGP. > > Otherwise, the inbound traffic will only be redirected when the incoming > LRP > > matches the chassis_resident field. > > > > Additionally, this patch includes the ability to use load-balancer with > DGPs > > attached to multiple chassis. We can have each of the DGPs associated > with a > > different chassis, and in this case the DNAT rules added by default will > not > > be enough to guarantee outgoing traffic. > > > > To solve the multiple chassis for DGPs problem, this patch include a new > > config options to be configured in the load-balancer. If the > use_stateless_nat > > is set to true, the logical router that references this load-balancer > will use > > Stateless NAT rules when the logical router has multiple DGPs. After > applying > > this patch and setting the use_stateless_nat option, the inbound and/or > > outbound traffic can pass through any chassis where the DGP resides > without > > having problems with CT state. > > > > Reported-at: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2054322 > > Fixes: 15348b7b806f ("ovn-northd: Multiple distributed gateway port > support.") > > > > Signed-off-by: Roberto Bartzen Acosta <roberto.acosta@luizalabs.com> > > Hi Roberto, > > Thanks for the patch. I tested this patch using the test example in > multinode.at. > > The test case adds the below load balancer > > [root@ovn-central ~]# ovn-nbctl lb-list > UUID LB PROTO > VIP IPs > f3e29869-3bb5-4df0-960a-171106f5913a lb0 tcp > 172.16.0.100:9000 10.0.0.3:80,10.0.0.4:80 > > And the below logical flows are generated by this patch > > -------- > [root@ovn-central ~]# ovn-sbctl dump-flows lr0 | grep 172.16.0.100 > table=6 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst > == 172.16.0.100), action=(ct_dnat;) > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > is_chassis_resident("cr-lr0-public-p1")), > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80, > 10.0.0.4:80);) > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > is_chassis_resident("cr-lr0-public-p2")), > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80, > 10.0.0.4:80);) > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > tcp.src == 80)) && (inport == "lr0-public-p1" || outport == > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > tcp.src == 80)) && (inport == "lr0-public-p2" || outport == > "lr0-public-p2") && is_chassis_resident("cr-lr0-public-p2") && tcp), > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > -------------- > > > I fail to understand the reason for modifying the ip4.dst before > calling ct_lb_mark. Can you please explain why ? Because the > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;) will first modify the > ip4.dst to 10.0.0.3 and > then to 10.0.0.4 and then the ct_lb_mark will actually do the > conntrack with NAT to either 10.0.0.3 or 10.0.0.4. > > Is it because you want the conntrack entry to not have 172.16.0.100 ? > The only reason I included this ip4.dst action in the DNAT rule is because it's required to accept packets coming from a chassis that doesn't have previously created conntrack entries. The main feature introduced in this patch is to allow the administrator to have multiple DGPs attached to different chassis (is_chassis_resident...). So, my implementation was based on the normal behavior when using stateless NAT for external addresses, where we need to add the ipx.dst in lr_in_dnat for traffic to be received on the chassis (put the DGP port as chassis_resident match, as is the case with stateless NAT [1] with DGP[2]). The question is, if we only have the ct_lb_mark, packets that pass through the chassis and are already part of an active flow in another chassis (same IPs and Ports) will be dropped because there is no correspondence in the backend. So only packets with the NEW flag will be accepted and sent to the backend (at least for TCP traffic). If we only have the ip4.dst action, this will always perform the dnat for the same backend, without balancing. Therefore, the combination of the two actions allows the packet to always be received (regardless of whether conntrack is active for it), and ct_lb_mark will take care of balancing for different backends. If we had conntrack sync between different chassis this would not be necessary, as the ct_lb_mark action could always be executed without dropping packets due to lack of correspondence in the conntrack table. [1] https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15737 [2] https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15726 > > Also I don't understand why this patch adds the logical flows in > "lr_out_snat" stage ? > The flow for lr_out_snat is necessary for the correct functioning of stateless NAT for the same reason explained previously. I mean, if the outgoing packet is redirected to a chassis that doesn't have an active conntrack entry, it will not be NATed by ct_lb action because it doesn't refer to a valid flow (use case with ecmp). So it is necessary to create a stateless SNAT rule (similar to this [3]) with a lower priority than the other router pipeline entries, in this case, if the packet is not SNATed by ct_lb (conntrack missed) it will be SNATed by stateless NAT rule. [3] https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15884 > > Using the system multinode test as an example, the below fails > (which is a regression) > > --- > root@ovn-chassis-3 ~]# ip netns exec publicp1 nc -vz 10.0.0.3 80 > ---- > > In the above test, publicp1 with IP 20.0.0.3 when it tries to connect > to one if the backends directly (without the LB VIP), it fails. > It fails because of the logical flows in "lr_out_snat". > > > Looks to me the solution proposed here is incomplete. > > Also please note that in our CI we run the multinode tests > periodically once a day using the v0.1 of the ovn-fake-multinode > and the tests you added will fail. This needs to be fixed and until > we move to the latest version of ovn-fake-multinode. > I imagine that the test you are doing is using the same port as the LB backend (TCP 80 in this case). So, the stateless lr_out_snat flow will force the output to be SNATed because this port is in use by the backend. Traffic to/from other ports will work without problems and will follow the normal programmed flows (e.g. ICMP). This is necessary to ensure the egress traffic because the DGPs are distributed across multiple chassis. Also, this setup is being validated in the test ovn-fake-multinode testcase (ICMP from the backends chassis use the router's default SNAT and not the LB's). I didn't understand the regression you mentioned because this was programmed to be stateless and it's traffic that uses the same ports as the LB backend, could you explain better? Thanks, Roberto > Thanks > Numan > > > > --- > > northd/en-lr-stateful.c | 12 - > > northd/northd.c | 116 ++++++-- > > ovn-nb.xml | 10 + > > tests/multinode-macros.at | 40 +++ > > tests/multinode.at | 556 ++++++++++++++++++++++++++++++++++++++ > > tests/ovn-northd.at | 320 ++++++++++++++++++++++ > > 6 files changed, 1017 insertions(+), 37 deletions(-) > > > > diff --git a/northd/en-lr-stateful.c b/northd/en-lr-stateful.c > > index baf1bd2f8..f09691af6 100644 > > --- a/northd/en-lr-stateful.c > > +++ b/northd/en-lr-stateful.c > > @@ -516,18 +516,6 @@ lr_stateful_record_create(struct lr_stateful_table > *table, > > > > table->array[od->index] = lr_stateful_rec; > > > > - /* Load balancers are not supported (yet) if a logical router has > multiple > > - * distributed gateway port. Log a warning. */ > > - if (lr_stateful_rec->has_lb_vip && lr_has_multiple_gw_ports(od)) { > > - static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1); > > - VLOG_WARN_RL(&rl, "Load-balancers are configured on logical " > > - "router %s, which has %"PRIuSIZE" distributed " > > - "gateway ports. Load-balancer is not supported " > > - "yet when there is more than one distributed " > > - "gateway port on the router.", > > - od->nbr->name, od->n_l3dgw_ports); > > - } > > - > > return lr_stateful_rec; > > } > > > > diff --git a/northd/northd.c b/northd/northd.c > > index a267cd5f8..bbe97acf8 100644 > > --- a/northd/northd.c > > +++ b/northd/northd.c > > @@ -11807,31 +11807,30 @@ static void > > build_distr_lrouter_nat_flows_for_lb(struct lrouter_nat_lb_flows_ctx > *ctx, > > enum lrouter_nat_lb_flow_type type, > > struct ovn_datapath *od, > > - struct lflow_ref *lflow_ref) > > + struct lflow_ref *lflow_ref, > > + struct ovn_port *dgp, > > + bool stateless_nat) > > { > > - struct ovn_port *dgp = od->l3dgw_ports[0]; > > - > > - const char *undnat_action; > > - > > - switch (type) { > > - case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > - undnat_action = "flags.force_snat_for_lb = 1; next;"; > > - break; > > - case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > - undnat_action = "flags.skip_snat_for_lb = 1; next;"; > > - break; > > - case LROUTER_NAT_LB_FLOW_NORMAL: > > - case LROUTER_NAT_LB_FLOW_MAX: > > - undnat_action = lrouter_use_common_zone(od) > > - ? "ct_dnat_in_czone;" > > - : "ct_dnat;"; > > - break; > > - } > > + struct ds dnat_action = DS_EMPTY_INITIALIZER; > > > > /* Store the match lengths, so we can reuse the ds buffer. */ > > size_t new_match_len = ctx->new_match->length; > > size_t undnat_match_len = ctx->undnat_match->length; > > > > + /* dnat_action: Add the LB backend IPs as a destination action of > the > > + * lr_in_dnat NAT rule with cumulative effect because > any > > + * backend dst IP used in the action list will > redirect the > > + * packet to the ct_lb pipeline. > > + */ > > + if (stateless_nat) { > > + for (size_t i = 0; i < ctx->lb_vip->n_backends; i++) { > > + struct ovn_lb_backend *backend = &ctx->lb_vip->backends[i]; > > + bool ipv6 = !IN6_IS_ADDR_V4MAPPED(&backend->ip); > > + ds_put_format(&dnat_action, "%s.dst=%s;", ipv6 ? "ip6" : > "ip4", > > + backend->ip_str); > > + } > > + } > > + ds_put_format(&dnat_action, "%s", ctx->new_action[type]); > > > > const char *meter = NULL; > > > > @@ -11841,20 +11840,46 @@ build_distr_lrouter_nat_flows_for_lb(struct > lrouter_nat_lb_flows_ctx *ctx, > > > > if (ctx->lb_vip->n_backends || !ctx->lb_vip->empty_backend_rej) { > > ds_put_format(ctx->new_match, " && is_chassis_resident(%s)", > > - od->l3dgw_ports[0]->cr_port->json_key); > > + dgp->cr_port->json_key); > > } > > > > ovn_lflow_add_with_hint__(ctx->lflows, od, S_ROUTER_IN_DNAT, > ctx->prio, > > - ds_cstr(ctx->new_match), > ctx->new_action[type], > > + ds_cstr(ctx->new_match), > ds_cstr(&dnat_action), > > NULL, meter, &ctx->lb->nlb->header_, > > lflow_ref); > > > > ds_truncate(ctx->new_match, new_match_len); > > > > + ds_destroy(&dnat_action); > > if (!ctx->lb_vip->n_backends) { > > return; > > } > > > > + struct ds undnat_action = DS_EMPTY_INITIALIZER; > > + struct ds snat_action = DS_EMPTY_INITIALIZER; > > + > > + switch (type) { > > + case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > + ds_put_format(&undnat_action, "flags.force_snat_for_lb = 1; > next;"); > > + break; > > + case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > + ds_put_format(&undnat_action, "flags.skip_snat_for_lb = 1; > next;"); > > + break; > > + case LROUTER_NAT_LB_FLOW_NORMAL: > > + case LROUTER_NAT_LB_FLOW_MAX: > > + ds_put_format(&undnat_action, "%s", > > + lrouter_use_common_zone(od) ? "ct_dnat_in_czone;" > > + : "ct_dnat;"); > > + break; > > + } > > + > > + /* undnat_action: Remove the ct action from the lr_out_undenat NAT > rule. > > + */ > > + if (stateless_nat) { > > + ds_clear(&undnat_action); > > + ds_put_format(&undnat_action, "next;"); > > + } > > + > > /* We need to centralize the LB traffic to properly perform > > * the undnat stage. > > */ > > @@ -11873,11 +11898,41 @@ build_distr_lrouter_nat_flows_for_lb(struct > lrouter_nat_lb_flows_ctx *ctx, > > ds_put_format(ctx->undnat_match, ") && (inport == %s || outport == > %s)" > > " && is_chassis_resident(%s)", dgp->json_key, > dgp->json_key, > > dgp->cr_port->json_key); > > + /* Use the LB protocol as matching criteria for out undnat and snat > when > > + * creating LBs with stateless NAT. */ > > + if (stateless_nat) { > > + ds_put_format(ctx->undnat_match, " && %s", ctx->lb->proto); > > + } > > ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_UNDNAT, 120, > > - ds_cstr(ctx->undnat_match), undnat_action, > > - &ctx->lb->nlb->header_, > > + ds_cstr(ctx->undnat_match), > > + ds_cstr(&undnat_action), > &ctx->lb->nlb->header_, > > lflow_ref); > > + > > + /* snat_action: Add a new lr_out_snat rule with the LB VIP as > source IP > > + * action to perform the NAT stateless pipeline > completely. > > + */ > > + if (stateless_nat) { > > + if (ctx->lb_vip->port_str) { > > + ds_put_format(&snat_action, "%s.src=%s; %s.src=%s; next;", > > + ctx->lb_vip->address_family == AF_INET6 ? > > + "ip6" : "ip4", > > + ctx->lb_vip->vip_str, ctx->lb->proto, > > + ctx->lb_vip->port_str); > > + } else { > > + ds_put_format(&snat_action, "%s.src=%s; next;", > > + ctx->lb_vip->address_family == AF_INET6 ? > > + "ip6" : "ip4", > > + ctx->lb_vip->vip_str); > > + } > > + ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_SNAT, 160, > > + ds_cstr(ctx->undnat_match), > > + ds_cstr(&snat_action), > &ctx->lb->nlb->header_, > > + lflow_ref); > > + } > > + > > ds_truncate(ctx->undnat_match, undnat_match_len); > > + ds_destroy(&undnat_action); > > + ds_destroy(&snat_action); > > } > > > > static void > > @@ -12022,6 +12077,8 @@ build_lrouter_nat_flows_for_lb( > > * lflow generation for them. > > */ > > size_t index; > > + bool use_stateless_nat = smap_get_bool(&lb->nlb->options, > > + "use_stateless_nat", false); > > BITMAP_FOR_EACH_1 (index, bitmap_len, lb_dps->nb_lr_map) { > > struct ovn_datapath *od = lr_datapaths->array[index]; > > enum lrouter_nat_lb_flow_type type; > > @@ -12043,8 +12100,17 @@ build_lrouter_nat_flows_for_lb( > > if (!od->n_l3dgw_ports) { > > bitmap_set1(gw_dp_bitmap[type], index); > > } else { > > - build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > - lb_dps->lflow_ref); > > + /* Create stateless LB NAT rules when using multiple DGPs > and > > + * use_stateless_nat is true. > > + */ > > + bool stateless_nat = (od->n_l3dgw_ports > 1) > > + ? use_stateless_nat : false; > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > > + struct ovn_port *dgp = od->l3dgw_ports[i]; > > + build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > + lb_dps->lflow_ref, > dgp, > > + stateless_nat); > > + } > > } > > > > if (lb->affinity_timeout) { > > diff --git a/ovn-nb.xml b/ovn-nb.xml > > index 2836f58f5..ad03c6214 100644 > > --- a/ovn-nb.xml > > +++ b/ovn-nb.xml > > @@ -2302,6 +2302,16 @@ or > > local anymore by the ovn-controller. This option is set to > > <code>false</code> by default. > > </column> > > + > > + <column name="options" key="use_stateless_nat" > > + type='{"type": "boolean"}'> > > + If the load balancer is configured with > <code>use_stateless_nat</code> > > + option to <code>true</code>, the logical router that references > this > > + load balancer will use Stateless NAT rules when the logical > router > > + has multiple distributed gateway ports(DGP). Otherwise, the > outbound > > + traffic may be dropped in scenarios where we have different > chassis > > + for each DGP. This option is set to <code>false</code> by > default. > > + </column> > > </group> > > </table> > > > > diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at > > index 757917626..2f69433fc 100644 > > --- a/tests/multinode-macros.at > > +++ b/tests/multinode-macros.at > > @@ -40,6 +40,27 @@ m4_define([M_START_TCPDUMP], > > ] > > ) > > > > +# M_EXEC([fake_node], [command]) > > +# > > +# Execute 'command' in 'fakenode' > > +m4_define([M_EXEC], > > + [podman exec $1 $2]) > > + > > +# M_CHECK_EXEC([fake_node], [command], other_params...) > > +# > > +# Wrapper for AT_CHECK that executes 'command' inside 'fake_node''s'. > > +# 'other_params' as passed as they are to AT_CHECK. > > +m4_define([M_CHECK_EXEC], > > + [ AT_CHECK([M_EXEC([$1], [$2])], m4_shift(m4_shift(m4_shift($@)))) ] > > +) > > + > > +# M_FORMAT_CT([ip-addr]) > > +# > > +# Strip content from the piped input which would differ from test to > test > > +# and limit the output to the rows containing 'ip-addr'. > > +# > > +m4_define([M_FORMAT_CT], > > + [[grep -F "dst=$1," | sed -e 's/id=[0-9]*/id=<cleared>/g' -e > 's/state=[0-9_A-Z]*/state=<cleared>/g' | sort | uniq | sed -e > 's/zone=[[0-9]]*/zone=<cleared>/' -e 's/mark=[[0-9]]*/mark=<cleared>/' ]]) > > > > OVS_START_SHELL_HELPERS > > > > @@ -76,6 +97,25 @@ multinode_nbctl () { > > m_as ovn-central ovn-nbctl "$@" > > } > > > > +check_fake_multinode_setup_by_nodes() { > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > + for c in $1 > > + do > > + AT_CHECK([m_as $c ovn-appctl -t ovn-controller version], [0], > [ignore]) > > + done > > +} > > + > > +cleanup_multinode_resources_by_nodes() { > > + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db > > + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > + for c in $1 > > + do > > + m_as $c ovs-vsctl del-br br-int > > + m_as $c ip --all netns delete > > + done > > +} > > + > > # m_count_rows TABLE [CONDITION...] > > # > > # Prints the number of rows in TABLE (that satisfy CONDITION). > > diff --git a/tests/multinode.at b/tests/multinode.at > > index a0eb8fc67..b1beb4d97 100644 > > --- a/tests/multinode.at > > +++ b/tests/multinode.at > > @@ -1591,3 +1591,559 @@ AT_CHECK([cat ch1_eth2.tcpdump], [0], [dnl > > ]) > > > > AT_CLEANUP > > + > > +AT_SETUP([ovn multinode load-balancer with multiple DGPs and multiple > chassis]) > > + > > +# Check that ovn-fake-multinode setup is up and running - requires > additional nodes > > +check_fake_multinode_setup_by_nodes 'ovn-chassis-1 ovn-chassis-2 > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > + > > +# Delete the multinode NB and OVS resources before starting the test. > > +cleanup_multinode_resources_by_nodes 'ovn-chassis-1 ovn-chassis-2 > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > + > > +# Network topology > > +# > > +# publicp1 (ovn-chassis-3) (20.0.0.3/24) > > +# | > > +# overlay > > +# | > > +# DGP public1 (ovn-gw-1) (20.0.0.1/24) > > +# | > > +# | > > +# | > > +# lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) 10.0.0.3/24 > > +# | | > > +# | + --- sw0p2 (ovn-chassis-2) 10.0.0.4/24 > > +# | > > +# DGP public2 (ovn-gw-2) (30.0.0.1/24) > > +# | > > +# overlay > > +# | > > +# publicp2 (ovn-chassis-4) (30.0.0.3/24) > > + > > +# Delete already used ovs-ports > > +m_as ovn-chassis-1 ovs-vsctl del-port br-int sw0p1-p > > +m_as ovn-chassis-2 ovs-vsctl del-port br-int sw0p2-p > > +m_as ovn-chassis-1 ip link del sw0p1-p > > +m_as ovn-chassis-2 ip link del sw0p2-p > > +m_as ovn-chassis-3 ovs-vsctl del-port br-int publicp1-p > > +m_as ovn-chassis-4 ovs-vsctl del-port br-int publicp2-p > > +m_as ovn-chassis-3 ip link del publicp1-p > > +m_as ovn-chassis-4 ip link del publicp2-p > > + > > +# Create East-West switch for LB backends > > +check multinode_nbctl ls-add sw0 > > +check multinode_nbctl lsp-add sw0 sw0-port1 > > +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 > 10.0.0.3 1000::3" > > +check multinode_nbctl lsp-add sw0 sw0-port2 > > +check multinode_nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 > 10.0.0.4 1000::4" > > + > > +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 > 50:54:00:00:00:03 1400 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a > > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 > 50:54:00:00:00:04 1400 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a > > + > > +m_wait_for_ports_up > > + > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > 10.0.0.4 | FORMAT_PING], \ > > +[0], [dnl > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > +]) > > + > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 > 10.0.0.3 | FORMAT_PING], \ > > +[0], [dnl > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > +]) > > + > > +# Create a logical router and attach to sw0 > > +check multinode_nbctl lr-add lr0 > > +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 > 1000::a/64 > > +check multinode_nbctl lsp-add sw0 sw0-lr0 > > +check multinode_nbctl lsp-set-type sw0-lr0 router > > +check multinode_nbctl lsp-set-addresses sw0-lr0 router > > +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 > > + > > +# create external connection for N/S traffic using multiple DGPs > > +check multinode_nbctl ls-add public > > + > > +# DGP public1 > > +check multinode_nbctl lsp-add public ln-public-1 > > +check multinode_nbctl lsp-set-type ln-public-1 localnet > > +check multinode_nbctl lsp-set-addresses ln-public-1 unknown > > +check multinode_nbctl lsp-set-options ln-public-1 network_name=public1 > > + > > +# DGP public2 > > +# create exteranl connection for N/S traffic > > +check multinode_nbctl lsp-add public ln-public-2 > > +check multinode_nbctl lsp-set-type ln-public-2 localnet > > +check multinode_nbctl lsp-set-addresses ln-public-2 unknown > > +check multinode_nbctl lsp-set-options ln-public-2 network_name=public2 > > + > > +# Attach DGP public1 to GW-1 and chassis-3 (overlay connectivity) > > +m_as ovn-gw-1 ovs-vsctl set open . > external-ids:ovn-bridge-mappings=public1:br-ex > > +m_as ovn-chassis-3 ovs-vsctl set open . > external-ids:ovn-bridge-mappings=public1:br-ex > > + > > +# Attach DGP public2 to GW-2 and chassis-4 (overlay connectivity) > > +m_as ovn-gw-2 ovs-vsctl set open . > external-ids:ovn-bridge-mappings=public2:br-ex > > +m_as ovn-chassis-4 ovs-vsctl set open . > external-ids:ovn-bridge-mappings=public2:br-ex > > + > > +# Create the external LR0 port to the DGP public1 > > +check multinode_nbctl lsp-add public public-port1 > > +check multinode_nbctl lsp-set-addresses public-port1 "40:54:00:00:00:03 > 20.0.0.3 2000::3" > > + > > +check multinode_nbctl lrp-add lr0 lr0-public-p1 00:00:00:00:ff:02 > 20.0.0.1/24 2000::a/64 > > +check multinode_nbctl lsp-add public public-lr0-p1 > > +check multinode_nbctl lsp-set-type public-lr0-p1 router > > +check multinode_nbctl lsp-set-addresses public-lr0-p1 router > > +check multinode_nbctl lsp-set-options public-lr0-p1 > router-port=lr0-public-p1 > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p1 ovn-gw-1 10 > > + > > +# Create a VM on ovn-chassis-3 in the same public1 overlay > > +m_as ovn-chassis-3 /data/create_fake_vm.sh public-port1 publicp1 > 40:54:00:00:00:03 1400 20.0.0.3 24 20.0.0.1 2000::4/64 2000::a > > + > > +m_wait_for_ports_up public-port1 > > + > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [ping -q -c 3 -i 0.3 -w 2 > 20.0.0.1 | FORMAT_PING], \ > > +[0], [dnl > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > +]) > > + > > +# Create the external LR0 port to the DGP public2 > > +check multinode_nbctl lsp-add public public-port2 > > +check multinode_nbctl lsp-set-addresses public-port2 "60:54:00:00:00:03 > 30.0.0.3 3000::3" > > + > > +check multinode_nbctl lrp-add lr0 lr0-public-p2 00:00:00:00:ff:03 > 30.0.0.1/24 3000::a/64 > > +check multinode_nbctl lsp-add public public-lr0-p2 > > +check multinode_nbctl lsp-set-type public-lr0-p2 router > > +check multinode_nbctl lsp-set-addresses public-lr0-p2 router > > +check multinode_nbctl lsp-set-options public-lr0-p2 > router-port=lr0-public-p2 > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p2 ovn-gw-2 10 > > + > > +# Create a VM on ovn-chassis-4 in the same public2 overlay > > +m_as ovn-chassis-4 /data/create_fake_vm.sh public-port2 publicp2 > 60:54:00:00:00:03 1400 30.0.0.3 24 30.0.0.1 3000::4/64 3000::a > > + > > +m_wait_for_ports_up public-port2 > > + > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [ping -q -c 3 -i 0.3 -w 2 > 30.0.0.1 | FORMAT_PING], \ > > +[0], [dnl > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > +]) > > + > > +# Add a default route for multiple DGPs - using ECMP > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 20.0.0.3 > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 30.0.0.3 > > + > > +# Add SNAT rules using gateway-port > > +check multinode_nbctl --gateway-port lr0-public-p1 lr-nat-add lr0 snat > 20.0.0.1 10.0.0.0/24 > > +check multinode_nbctl --gateway-port lr0-public-p2 lr-nat-add lr0 snat > 30.0.0.1 10.0.0.0/24 > > + > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > 20.0.0.3 | FORMAT_PING], \ > > +[0], [dnl > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > +]) > > + > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 > 30.0.0.3 | FORMAT_PING], \ > > +[0], [dnl > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > +]) > > + > > +# create LB > > +check multinode_nbctl lb-add lb0 "172.16.0.100:80" "10.0.0.3:80, > 10.0.0.4:80" > > +check multinode_nbctl lr-lb-add lr0 lb0 > > +check multinode_nbctl ls-lb-add sw0 lb0 > > + > > +# Set use_stateless_nat to true > > +check multinode_nbctl set load_balancer lb0 > options:use_stateless_nat=true > > + > > +# Start backend http services > > +M_NS_DAEMONIZE([ovn-chassis-1], [sw0p1], [$PYTHON -m http.server --bind > 10.0.0.3 80 >/dev/null 2>&1], [http1.pid]) > > +M_NS_DAEMONIZE([ovn-chassis-2], [sw0p2], [$PYTHON -m http.server --bind > 10.0.0.4 80 >/dev/null 2>&1], [http2.pid]) > > + > > +# wait for http server be ready > > +sleep 2 > > + > > +# Flush conntrack entries for easier output parsing of next test. > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > + > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v 172.16.0.100:80 > --retry 3 --max-time 1 --local-port 59002 2> curl.out']) > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [sh -c 'cat curl.out | > grep -i -e connect | grep -v 'Server:''], \ > > +[0], [dnl > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > +* Closing connection > > +]) > > + > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v 172.16.0.100:80 > --retry 3 --max-time 1 --local-port 59003 2> curl.out']) > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [sh -c 'cat curl.out | > grep -i -e connect | grep -v 'Server:''], \ > > +[0], [dnl > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > +* Closing connection > > +]) > > + > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > + > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v 172.16.0.100:80 > --retry 3 --max-time 1 --local-port 59001']) > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | > M_FORMAT_CT(20.0.0.3) | \ > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > [dnl > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v 172.16.0.100:80 > --retry 3 --max-time 1 --local-port 59000']) > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | > M_FORMAT_CT(30.0.0.3) | \ > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > [dnl > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +# create a big file on web servers for download > > +M_NS_EXEC([ovn-chassis-1], [sw0p1], [dd bs=512 count=200000 > if=/dev/urandom of=download_file]) > > +M_NS_EXEC([ovn-chassis-2], [sw0p2], [dd bs=512 count=200000 > if=/dev/urandom of=download_file]) > > + > > +# Flush conntrack entries for easier output parsing of next test. > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > + > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59004 > 2>curl.out']) > > + > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > ':a;N;$!ba;s/\n/\\n/g') > > + > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > curl.out | \ > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > +* Closing connection > > +]) > > + > > +# Check if we have only one backend for the same connection - orig + > dest ports > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > [dnl > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +# Check if gw-2 is empty to ensure that the traffic only come from/to > the originator chassis via DGP public1 > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > +0 > > +]) > > + > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" -c) > > + > > +if [[ $backend_check -gt 0 ]]; then > > +# Backend resides on ovn-chassis-1 > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > +grep tcp], [0], [dnl > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59004,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +# Ensure that the traffic only come from ovn-chassis-1 > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" -c], > [1], [dnl > > +0 > > +]) > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" > -c], [1], [dnl > > +0 > > +]) > > +else > > +# Backend resides on ovn-chassis-2 > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > +grep tcp], [0], [dnl > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59004,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +# Ensure that the traffic only come from ovn-chassis-2 > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" -c], > [1], [dnl > > +0 > > +]) > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" > -c], [1], [dnl > > +0 > > +]) > > +fi > > + > > +# Flush conntrack entries for easier output parsing of next test. > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > + > > +# Check the flows again for a new source port > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59005 > 2>curl.out']) > > + > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > ':a;N;$!ba;s/\n/\\n/g') > > + > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > curl.out | \ > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > +* Closing connection > > +]) > > + > > +# Check if we have only one backend for the same connection - orig + > dest ports > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > [dnl > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +# Check if gw-2 is empty to ensure that the traffic only come from/to > the originator chassis via DGP public1 > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > +0 > > +]) > > + > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" -c) > > + > > +if [[ $backend_check -gt 0 ]]; then > > +# Backend resides on ovn-chassis-1 > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > +grep tcp], [0], [dnl > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59005,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +# Ensure that the traffic only come from ovn-chassis-1 > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" -c], > [1], [dnl > > +0 > > +]) > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" > -c], [1], [dnl > > +0 > > +]) > > +else > > +# Backend resides on ovn-chassis-2 > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > +grep tcp], [0], [dnl > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59005,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +# Ensure that the traffic only come from ovn-chassis-2 > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" -c], > [1], [dnl > > +0 > > +]) > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" > -c], [1], [dnl > > +0 > > +]) > > +fi > > + > > +# Flush conntrack entries for easier output parsing of next test. > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > + > > +# Start a new test using the second DGP as origin (public2) > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59006 > 2>curl.out']) > > + > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > ':a;N;$!ba;s/\n/\\n/g') > > + > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > curl.out | \ > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > +* Closing connection > > +]) > > + > > +# Check if we have only one backend for the same connection - orig + > dest ports > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > [dnl > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +# Check if gw-1 is empty to ensure that the traffic only come from/to > the originator chassis via DGP public2 > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > +0 > > +]) > > + > > +# Check the backend IP from ct entries on gw-2 (DGP public2) > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" -c) > > + > > +if [[ $backend_check -gt 0 ]]; then > > +# Backend resides on ovn-chassis-1 > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > +grep tcp], [0], [dnl > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59006,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +# Ensure that the traffic only come from ovn-chassis-1 > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" -c], > [1], [dnl > > +0 > > +]) > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" > -c], [1], [dnl > > +0 > > +]) > > +else > > +# Backend resides on ovn-chassis-2 > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > +grep tcp], [0], [dnl > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59006,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +# Ensure that the traffic only come from ovn-chassis-2 > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" -c], > [1], [dnl > > +0 > > +]) > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" > -c], [1], [dnl > > +0 > > +]) > > +fi > > + > > +# Flush conntrack entries for easier output parsing of next test. > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > + > > +# Check the flows again for a new source port using the second DGP as > origin (public2) > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59007 > 2>curl.out']) > > + > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > ':a;N;$!ba;s/\n/\\n/g') > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > ':a;N;$!ba;s/\n/\\n/g') > > + > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > curl.out | \ > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > +* Closing connection > > +]) > > + > > +# Check if we have only one backend for the same connection - orig + > dest ports > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > [dnl > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +# Check if gw-1 is empty to ensure that the traffic only come from/to > the originator chassis via DGP public2 > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > +0 > > +]) > > + > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" -c) > > + > > +if [[ $backend_check -gt 0 ]]; then > > +# Backend resides on ovn-chassis-1 > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > +grep tcp], [0], [dnl > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59007,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +# Ensure that the traffic only come from ovn-chassis-1 > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" -c], > [1], [dnl > > +0 > > +]) > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" > -c], [1], [dnl > > +0 > > +]) > > +else > > +# Backend resides on ovn-chassis-2 > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > +grep tcp], [0], [dnl > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59007,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +# Ensure that the traffic only come from ovn-chassis-2 > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" -c], > [1], [dnl > > +0 > > +]) > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" > -c], [1], [dnl > > +0 > > +]) > > +fi > > + > > +# Check multiple requests coming from DGP's public1 and public2 > > + > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > -v 'Server:'], [0], [dnl > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > +200 OK > > +* Closing connection > > +]) > > + > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > -v 'Server:'], [0], [dnl > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > +200 OK > > +* Closing connection > > +]) > > + > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > -v 'Server:'], [0], [dnl > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > +200 OK > > +* Closing connection > > +]) > > + > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > -v 'Server:'], [0], [dnl > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > +200 OK > > +* Closing connection > > +]) > > + > > +# Remove the LB and change the VIP port - different from the backend > ports > > +check multinode_nbctl lb-del lb0 > > + > > +# create LB again > > +check multinode_nbctl lb-add lb0 "172.16.0.100:9000" "10.0.0.3:80, > 10.0.0.4:80" > > +check multinode_nbctl lr-lb-add lr0 lb0 > > +check multinode_nbctl ls-lb-add sw0 lb0 > > + > > +# Set use_stateless_nat to true > > +check multinode_nbctl set load_balancer lb0 > options:use_stateless_nat=true > > + > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > + > > +# Check end-to-end request using a new port for VIP > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port 59008 > 2>curl.out']) > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | > M_FORMAT_CT(20.0.0.3) | \ > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > [dnl > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > -v 'Server:'], [0], [dnl > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > +200 OK > > +* Closing connection > > +]) > > + > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > + > > +# Check end-to-end request using a new port for VIP > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port 59008 > 2>curl.out']) > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | > M_FORMAT_CT(30.0.0.3) | \ > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > [dnl > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > +]) > > + > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > -v 'Server:'], [0], [dnl > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > +200 OK > > +* Closing connection > > +]) > > + > > +AT_CLEANUP > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > > index dcc3dbbc3..9e7a2f225 100644 > > --- a/tests/ovn-northd.at > > +++ b/tests/ovn-northd.at > > @@ -13864,3 +13864,323 @@ check_no_redirect > > > > AT_CLEANUP > > ]) > > + > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + NAT > Stateless)]) > > +ovn_start > > + > > +check ovn-nbctl ls-add public > > +check ovn-nbctl lr-add lr1 > > + > > +# lr1 DGP ts1 > > +check ovn-nbctl ls-add ts1 > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 172.16.10.1/24 > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > + > > +# lr1 DGP ts2 > > +check ovn-nbctl ls-add ts2 > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 172.16.20.1/24 > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > + > > +# lr1 DGP public > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 173.16.0.1/16 > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > + > > +check ovn-nbctl ls-add s1 > > +# s1 - lr1 > > +check ovn-nbctl lsp-add s1 s1_lr1 > > +check ovn-nbctl lsp-set-type s1_lr1 router > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 172.16.0.1" > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > + > > +# s1 - backend vm1 > > +check ovn-nbctl lsp-add s1 vm1 > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 172.16.0.101" > > + > > +# s1 - backend vm2 > > +check ovn-nbctl lsp-add s1 vm2 > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 172.16.0.102" > > + > > +# s1 - backend vm3 > > +check ovn-nbctl lsp-add s1 vm3 > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 172.16.0.103" > > + > > +# Add the lr1 DGP ts1 to the public switch > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > nat-addresses=router > > + > > +# Add the lr1 DGP ts2 to the public switch > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > nat-addresses=router > > + > > +# Add the lr1 DGP public to the public switch > > +check ovn-nbctl lsp-add public public_lr1 > > +check ovn-nbctl lsp-set-type public_lr1 router > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > nat-addresses=router > > + > > +# Create the Load Balancer lb1 > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > "172.16.0.103,172.16.0.102,172.16.0.101" > > + > > +# Set use_stateless_nat to true > > +check ovn-nbctl --wait=sb set load_balancer lb1 > options:use_stateless_nat=true > > + > > +# Associate load balancer to s1 > > +check ovn-nbctl ls-lb-add s1 lb1 > > +check ovn-nbctl --wait=sb sync > > + > > +ovn-sbctl dump-flows s1 > s1flows > > +AT_CAPTURE_FILE([s1flows]) > > + > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep > "30.0.0.1"], [0], [dnl > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > +]) > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > "30.0.0.1"], [0], [dnl > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > ip4.dst == 30.0.0.1), > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > +]) > > + > > +# Associate load balancer to lr1 with DGP > > +check ovn-nbctl lr-lb-add lr1 lb1 > > +check ovn-nbctl --wait=sb sync > > + > > +ovn-sbctl dump-flows lr1 > lr1flows > > +AT_CAPTURE_FILE([lr1flows]) > > + > > +# Check stateless NAT rules for load balancer with multiple DGP > > +# 1. Check if the backend IPs are in the ipX.dst action > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > "30.0.0.1"], [0], [dnl > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > is_chassis_resident("cr-lr1-ts1")), > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > is_chassis_resident("cr-lr1-ts2")), > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > is_chassis_resident("cr-lr1_public")), > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > +]) > > + > > +# 2. Check if the DGP ports are in the match with action next > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], [dnl > > + table=??(lr_out_undnat ), priority=0 , match=(1), > action=(next;) > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") && > is_chassis_resident("cr-lr1_public") && tcp), action=(next;) > > +]) > > + > > +# 3. Check if the VIP IP is in the ipX.src action > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > > + table=??(lr_out_snat ), priority=0 , match=(1), > action=(next;) > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > is_chassis_resident("cr-lr1-ts1") && tcp), action=(ip4.src=30.0.0.1; next;) > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > is_chassis_resident("cr-lr1-ts2") && tcp), action=(ip4.src=30.0.0.1; next;) > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") && > is_chassis_resident("cr-lr1_public") && tcp), action=(ip4.src=30.0.0.1; > next;) > > +]) > > + > > +AT_CLEANUP > > +]) > > + > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + NAT > Stateless) - IPv6]) > > +ovn_start > > + > > +check ovn-nbctl ls-add public > > +check ovn-nbctl lr-add lr1 > > + > > +# lr1 DGP ts1 > > +check ovn-nbctl ls-add ts1 > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > 2001:db8:aaaa:1::1/64 > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > + > > +# lr1 DGP ts2 > > +check ovn-nbctl ls-add ts2 > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > 2001:db8:aaaa:2::1/64 > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > + > > +# lr1 DGP public > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > 2001:db8:bbbb::1/64 > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > 2001:db8:aaaa:3::1/64 > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > + > > +check ovn-nbctl ls-add s1 > > +# s1 - lr1 > > +check ovn-nbctl lsp-add s1 s1_lr1 > > +check ovn-nbctl lsp-set-type s1_lr1 router > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > 2001:db8:aaaa:3::1" > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > + > > +# s1 - backend vm1 > > +check ovn-nbctl lsp-add s1 vm1 > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > 2001:db8:aaaa:3::101" > > + > > +# s1 - backend vm2 > > +check ovn-nbctl lsp-add s1 vm2 > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > 2001:db8:aaaa:3::102" > > + > > +# s1 - backend vm3 > > +check ovn-nbctl lsp-add s1 vm3 > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > 2001:db8:aaaa:3::103" > > + > > +# Add the lr1 DGP ts1 to the public switch > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > nat-addresses=router > > + > > +# Add the lr1 DGP ts2 to the public switch > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > nat-addresses=router > > + > > +# Add the lr1 DGP public to the public switch > > +check ovn-nbctl lsp-add public public_lr1 > > +check ovn-nbctl lsp-set-type public_lr1 router > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > nat-addresses=router > > + > > +# Create the Load Balancer lb1 > > +check ovn-nbctl --wait=sb lb-add lb1 "2001:db8:cccc::1" > "2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101" > > + > > +# Set use_stateless_nat to true > > +check ovn-nbctl --wait=sb set load_balancer lb1 > options:use_stateless_nat=true > > + > > +# Associate load balancer to s1 > > +check ovn-nbctl ls-lb-add s1 lb1 > > +check ovn-nbctl --wait=sb sync > > + > > +ovn-sbctl dump-flows s1 > s1flows > > +AT_CAPTURE_FILE([s1flows]) > > + > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep > "2001:db8:cccc::1"], [0], [dnl > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 > && ip6.dst == 2001:db8:cccc::1), action=(xxreg1 = 2001:db8:cccc::1; > ct_lb_mark;) > > +]) > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > "2001:db8:cccc::1"], [0], [dnl > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > ip6.dst == 2001:db8:cccc::1), > action=(ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > +]) > > + > > +# Associate load balancer to lr1 with DGP > > +check ovn-nbctl lr-lb-add lr1 lb1 > > +check ovn-nbctl --wait=sb sync > > + > > +ovn-sbctl dump-flows lr1 > lr1flows > > +AT_CAPTURE_FILE([lr1flows]) > > + > > +# Check stateless NAT rules for load balancer with multiple DGP > > +# 1. Check if the backend IPs are in the ipX.dst action > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > "2001:db8:cccc::1"], [0], [dnl > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > is_chassis_resident("cr-lr1-ts1")), > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > is_chassis_resident("cr-lr1-ts2")), > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > is_chassis_resident("cr-lr1_public")), > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > +]) > > + > > +# 2. Check if the DGP ports are in the match with action next > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], [dnl > > + table=??(lr_out_undnat ), priority=0 , match=(1), > action=(next;) > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport == > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport == > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || outport == > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > action=(next;) > > +]) > > + > > +# 3. Check if the VIP IP is in the ipX.src action > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > > + table=??(lr_out_snat ), priority=0 , match=(1), > action=(next;) > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > action=(next;) > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport == > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), > action=(ip6.src=2001:db8:cccc::1; next;) > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport == > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), > action=(ip6.src=2001:db8:cccc::1; next;) > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || outport == > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > action=(ip6.src=2001:db8:cccc::1; next;) > > +]) > > + > > +AT_CLEANUP > > +]) > > + > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > +AT_SETUP([Load balancer with Distributed Gateway Ports (DGP)]) > > +ovn_start > > + > > +check ovn-nbctl ls-add public > > +check ovn-nbctl lr-add lr1 > > + > > +# lr1 DGP ts1 > > +check ovn-nbctl ls-add ts1 > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 172.16.10.1/24 > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-1 > > + > > +# lr1 DGP ts2 > > +check ovn-nbctl ls-add ts2 > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 172.16.20.1/24 > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-1 > > + > > +# lr1 DGP public > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 173.16.0.1/16 > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > + > > +check ovn-nbctl ls-add s1 > > +# s1 - lr1 > > +check ovn-nbctl lsp-add s1 s1_lr1 > > +check ovn-nbctl lsp-set-type s1_lr1 router > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 172.16.0.1" > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > + > > +# s1 - backend vm1 > > +check ovn-nbctl lsp-add s1 vm1 > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 172.16.0.101" > > + > > +# s1 - backend vm2 > > +check ovn-nbctl lsp-add s1 vm2 > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 172.16.0.102" > > + > > +# s1 - backend vm3 > > +check ovn-nbctl lsp-add s1 vm3 > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 172.16.0.103" > > + > > +# Add the lr1 DGP ts1 to the public switch > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > nat-addresses=router > > + > > +# Add the lr1 DGP ts2 to the public switch > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > nat-addresses=router > > + > > +# Add the lr1 DGP public to the public switch > > +check ovn-nbctl lsp-add public public_lr1 > > +check ovn-nbctl lsp-set-type public_lr1 router > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > nat-addresses=router > > + > > +# Create the Load Balancer lb1 > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > "172.16.0.103,172.16.0.102,172.16.0.101" > > + > > +# Associate load balancer to s1 > > +check ovn-nbctl ls-lb-add s1 lb1 > > +check ovn-nbctl --wait=sb sync > > + > > +ovn-sbctl dump-flows s1 > s1flows > > +AT_CAPTURE_FILE([s1flows]) > > + > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep > "30.0.0.1"], [0], [dnl > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > +]) > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > "30.0.0.1"], [0], [dnl > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > ip4.dst == 30.0.0.1), > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > +]) > > + > > +# Associate load balancer to lr1 with DGP > > +check ovn-nbctl lr-lb-add lr1 lb1 > > +check ovn-nbctl --wait=sb sync > > + > > +ovn-sbctl dump-flows lr1 > lr1flows > > +AT_CAPTURE_FILE([lr1flows]) > > + > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > "30.0.0.1"], [0], [dnl > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > is_chassis_resident("cr-lr1-ts1")), > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > is_chassis_resident("cr-lr1-ts2")), > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > is_chassis_resident("cr-lr1_public")), > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > +]) > > + > > +AT_CLEANUP > > +]) > > -- > > 2.34.1 > > > > > > -- > > > > > > > > > > _'Esta mensagem é direcionada apenas para os endereços constantes no > > cabeçalho inicial. Se você não está listado nos endereços constantes no > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas > estão > > imediatamente anuladas e proibidas'._ > > > > > > * **'Apesar do Magazine Luiza tomar > > todas as precauções razoáveis para assegurar que nenhum vírus esteja > > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade > por > > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos'.* > > > > > > > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > >
On Thu, Sep 26, 2024 at 10:55 AM Roberto Bartzen Acosta via dev <ovs-dev@openvswitch.org> wrote: > > Hi Numan, > > Thanks for your feedback and review. > > Em qua., 25 de set. de 2024 às 19:50, Numan Siddique <numans@ovn.org> > escreveu: > > > On Thu, Sep 19, 2024 at 6:12 PM Roberto Bartzen Acosta via dev > > <ovs-dev@openvswitch.org> wrote: > > > > > > This commit fixes the build_distr_lrouter_nat_flows_for_lb function to > > > include a DNAT flow entry for each DGP in use. Since we have added > > support > > > to create multiple gateway ports per logical router, it's necessary to > > > include in the LR NAT rules pipeline a specific entry for each attached > > DGP. > > > Otherwise, the inbound traffic will only be redirected when the incoming > > LRP > > > matches the chassis_resident field. > > > > > > Additionally, this patch includes the ability to use load-balancer with > > DGPs > > > attached to multiple chassis. We can have each of the DGPs associated > > with a > > > different chassis, and in this case the DNAT rules added by default will > > not > > > be enough to guarantee outgoing traffic. > > > > > > To solve the multiple chassis for DGPs problem, this patch include a new > > > config options to be configured in the load-balancer. If the > > use_stateless_nat > > > is set to true, the logical router that references this load-balancer > > will use > > > Stateless NAT rules when the logical router has multiple DGPs. After > > applying > > > this patch and setting the use_stateless_nat option, the inbound and/or > > > outbound traffic can pass through any chassis where the DGP resides > > without > > > having problems with CT state. > > > > > > Reported-at: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2054322 > > > Fixes: 15348b7b806f ("ovn-northd: Multiple distributed gateway port > > support.") > > > > > > Signed-off-by: Roberto Bartzen Acosta <roberto.acosta@luizalabs.com> > > > > Hi Roberto, > > > > Thanks for the patch. I tested this patch using the test example in > > multinode.at. > > > > The test case adds the below load balancer > > > > [root@ovn-central ~]# ovn-nbctl lb-list > > UUID LB PROTO > > VIP IPs > > f3e29869-3bb5-4df0-960a-171106f5913a lb0 tcp > > 172.16.0.100:9000 10.0.0.3:80,10.0.0.4:80 > > > > And the below logical flows are generated by this patch > > > > -------- > > [root@ovn-central ~]# ovn-sbctl dump-flows lr0 | grep 172.16.0.100 > > table=6 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst > > == 172.16.0.100), action=(ct_dnat;) > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > is_chassis_resident("cr-lr0-public-p1")), > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80, > > 10.0.0.4:80);) > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > is_chassis_resident("cr-lr0-public-p2")), > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80, > > 10.0.0.4:80);) > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > tcp.src == 80)) && (inport == "lr0-public-p1" || outport == > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > tcp.src == 80)) && (inport == "lr0-public-p2" || outport == > > "lr0-public-p2") && is_chassis_resident("cr-lr0-public-p2") && tcp), > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > -------------- > > > > > > I fail to understand the reason for modifying the ip4.dst before > > calling ct_lb_mark. Can you please explain why ? Because the > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;) will first modify the > > ip4.dst to 10.0.0.3 and > > then to 10.0.0.4 and then the ct_lb_mark will actually do the > > conntrack with NAT to either 10.0.0.3 or 10.0.0.4. > > > > Is it because you want the conntrack entry to not have 172.16.0.100 ? > > > > The only reason I included this ip4.dst action in the DNAT rule is because > it's required to accept packets coming from a chassis that doesn't have > previously created conntrack entries. The main feature introduced in this > patch is to allow the administrator to have multiple DGPs attached to > different chassis (is_chassis_resident...). So, my implementation was based > on the normal behavior when using stateless NAT for external addresses, > where we need to add the ipx.dst in lr_in_dnat for traffic to be received > on the chassis (put the DGP port as chassis_resident match, as is the case > with stateless NAT [1] with DGP[2]). > > The question is, if we only have the ct_lb_mark, packets that pass through > the chassis and are already part of an active flow in another chassis (same > IPs and Ports) will be dropped because there is no correspondence in the > backend. So only packets with the NEW flag will be accepted and sent to the > backend (at least for TCP traffic). If we only have the ip4.dst action, > this will always perform the dnat for the same backend, without balancing. > Therefore, the combination of the two actions allows the packet to always > be received (regardless of whether conntrack is active for it), and > ct_lb_mark will take care of balancing for different backends. > > If we had conntrack sync between different chassis this would not be > necessary, as the ct_lb_mark action could always be executed without > dropping packets due to lack of correspondence in the conntrack table. I still need to understand it properly. I'll get back to you after reading your reply thoroughly. But what's the point of doing multiple ip4.dst modifications for the same packet ? i.e - action=(ip4.dst = 10.0.0.3; ip4.dst = 10.0.0.4.....) The 2nd ip4.dst action overwrites and the packet's ip4.dst will be 10.0.0.4 before ct_lb_mark. > > [1] > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15737 > [2] > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15726 > > > > > > Also I don't understand why this patch adds the logical flows in > > "lr_out_snat" stage ? > > > > The flow for lr_out_snat is necessary for the correct functioning of > stateless NAT for the same reason explained previously. I mean, if the > outgoing packet is redirected to a chassis that doesn't have an active > conntrack entry, it will not be NATed by ct_lb action because it doesn't > refer to a valid flow (use case with ecmp). > > So it is necessary to create a stateless SNAT rule (similar to this [3]) > with a lower priority than the other router pipeline entries, in this case, > if the packet is not SNATed by ct_lb (conntrack missed) it will be SNATed > by stateless NAT rule. > > [3] > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15884 > > > > > > > Using the system multinode test as an example, the below fails > > (which is a regression) > > > > --- > > root@ovn-chassis-3 ~]# ip netns exec publicp1 nc -vz 10.0.0.3 80 > > ---- > > > > In the above test, publicp1 with IP 20.0.0.3 when it tries to connect > > to one if the backends directly (without the LB VIP), it fails. > > It fails because of the logical flows in "lr_out_snat". > > > > > > Looks to me the solution proposed here is incomplete. > > > > Also please note that in our CI we run the multinode tests > > periodically once a day using the v0.1 of the ovn-fake-multinode > > and the tests you added will fail. This needs to be fixed and until > > we move to the latest version of ovn-fake-multinode. > > > > I imagine that the test you are doing is using the same port as the LB > backend (TCP 80 in this case). So, the stateless lr_out_snat flow will > force the output to be SNATed because this port is in use by the backend. > Traffic to/from other ports will work without problems and will follow the > normal programmed flows (e.g. ICMP). > > This is necessary to ensure the egress traffic because the DGPs are > distributed across multiple chassis. Also, this setup is being validated in > the test ovn-fake-multinode testcase (ICMP from the backends chassis use > the router's default SNAT and not the LB's). I didn't understand the > regression you mentioned because this was programmed to be stateless and > it's traffic that uses the same ports as the LB backend, could you explain > better? > To reproduce the issue I mentioned, first run the multinode test you have added. Don't destroy the resources. And then login to ovn-chassis-3 container and run the command #ip netns exec publicp1 curl -v 10.0.0.3:80 So instead of curling to the LB VIP (which is 172.16.0.100:80), curl directly to the backends. Without your patch it works and with your patch it doesn't. If you run tcpdump you'd see the below 16:51:24.526980 40:54:00:00:00:03 > 00:00:00:00:ff:02, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 44276, offset 0, flags [DF], proto TCP (6), length 60) 20.0.0.3.46978 > 10.0.0.3.http: Flags [S], cksum 0x1e34 (incorrect -> 0x4b96), seq 1989771329, win 65280, options [mss 1360,sackOK,TS val 1753008285 ecr 0,nop,wscale 7], length 0 16:51:24.528559 00:00:00:00:ff:02 > 40:54:00:00:00:03, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60) 172.16.0.100.http > 20.0.0.3.46978: Flags [S.], cksum 0xc0a5 (incorrect -> 0x2094), seq 1088931300, ack 1989771330, win 64704, options [mss 1360,sackOK,TS val 3729696164 ecr 1753008285,nop,wscale 7], length 0 16:51:24.528571 40:54:00:00:00:03 > 00:00:00:00:ff:02, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40) The request is sent to 10.0.0.3 but the reply is received from 172.16.0.100 which is wrong. Thanks Numan > Thanks, > Roberto > > > > Thanks > > Numan > > > > > > > --- > > > northd/en-lr-stateful.c | 12 - > > > northd/northd.c | 116 ++++++-- > > > ovn-nb.xml | 10 + > > > tests/multinode-macros.at | 40 +++ > > > tests/multinode.at | 556 ++++++++++++++++++++++++++++++++++++++ > > > tests/ovn-northd.at | 320 ++++++++++++++++++++++ > > > 6 files changed, 1017 insertions(+), 37 deletions(-) > > > > > > diff --git a/northd/en-lr-stateful.c b/northd/en-lr-stateful.c > > > index baf1bd2f8..f09691af6 100644 > > > --- a/northd/en-lr-stateful.c > > > +++ b/northd/en-lr-stateful.c > > > @@ -516,18 +516,6 @@ lr_stateful_record_create(struct lr_stateful_table > > *table, > > > > > > table->array[od->index] = lr_stateful_rec; > > > > > > - /* Load balancers are not supported (yet) if a logical router has > > multiple > > > - * distributed gateway port. Log a warning. */ > > > - if (lr_stateful_rec->has_lb_vip && lr_has_multiple_gw_ports(od)) { > > > - static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1); > > > - VLOG_WARN_RL(&rl, "Load-balancers are configured on logical " > > > - "router %s, which has %"PRIuSIZE" distributed " > > > - "gateway ports. Load-balancer is not supported " > > > - "yet when there is more than one distributed " > > > - "gateway port on the router.", > > > - od->nbr->name, od->n_l3dgw_ports); > > > - } > > > - > > > return lr_stateful_rec; > > > } > > > > > > diff --git a/northd/northd.c b/northd/northd.c > > > index a267cd5f8..bbe97acf8 100644 > > > --- a/northd/northd.c > > > +++ b/northd/northd.c > > > @@ -11807,31 +11807,30 @@ static void > > > build_distr_lrouter_nat_flows_for_lb(struct lrouter_nat_lb_flows_ctx > > *ctx, > > > enum lrouter_nat_lb_flow_type type, > > > struct ovn_datapath *od, > > > - struct lflow_ref *lflow_ref) > > > + struct lflow_ref *lflow_ref, > > > + struct ovn_port *dgp, > > > + bool stateless_nat) > > > { > > > - struct ovn_port *dgp = od->l3dgw_ports[0]; > > > - > > > - const char *undnat_action; > > > - > > > - switch (type) { > > > - case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > - undnat_action = "flags.force_snat_for_lb = 1; next;"; > > > - break; > > > - case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > - undnat_action = "flags.skip_snat_for_lb = 1; next;"; > > > - break; > > > - case LROUTER_NAT_LB_FLOW_NORMAL: > > > - case LROUTER_NAT_LB_FLOW_MAX: > > > - undnat_action = lrouter_use_common_zone(od) > > > - ? "ct_dnat_in_czone;" > > > - : "ct_dnat;"; > > > - break; > > > - } > > > + struct ds dnat_action = DS_EMPTY_INITIALIZER; > > > > > > /* Store the match lengths, so we can reuse the ds buffer. */ > > > size_t new_match_len = ctx->new_match->length; > > > size_t undnat_match_len = ctx->undnat_match->length; > > > > > > + /* dnat_action: Add the LB backend IPs as a destination action of > > the > > > + * lr_in_dnat NAT rule with cumulative effect because > > any > > > + * backend dst IP used in the action list will > > redirect the > > > + * packet to the ct_lb pipeline. > > > + */ > > > + if (stateless_nat) { > > > + for (size_t i = 0; i < ctx->lb_vip->n_backends; i++) { > > > + struct ovn_lb_backend *backend = &ctx->lb_vip->backends[i]; > > > + bool ipv6 = !IN6_IS_ADDR_V4MAPPED(&backend->ip); > > > + ds_put_format(&dnat_action, "%s.dst=%s;", ipv6 ? "ip6" : > > "ip4", > > > + backend->ip_str); > > > + } > > > + } > > > + ds_put_format(&dnat_action, "%s", ctx->new_action[type]); > > > > > > const char *meter = NULL; > > > > > > @@ -11841,20 +11840,46 @@ build_distr_lrouter_nat_flows_for_lb(struct > > lrouter_nat_lb_flows_ctx *ctx, > > > > > > if (ctx->lb_vip->n_backends || !ctx->lb_vip->empty_backend_rej) { > > > ds_put_format(ctx->new_match, " && is_chassis_resident(%s)", > > > - od->l3dgw_ports[0]->cr_port->json_key); > > > + dgp->cr_port->json_key); > > > } > > > > > > ovn_lflow_add_with_hint__(ctx->lflows, od, S_ROUTER_IN_DNAT, > > ctx->prio, > > > - ds_cstr(ctx->new_match), > > ctx->new_action[type], > > > + ds_cstr(ctx->new_match), > > ds_cstr(&dnat_action), > > > NULL, meter, &ctx->lb->nlb->header_, > > > lflow_ref); > > > > > > ds_truncate(ctx->new_match, new_match_len); > > > > > > + ds_destroy(&dnat_action); > > > if (!ctx->lb_vip->n_backends) { > > > return; > > > } > > > > > > + struct ds undnat_action = DS_EMPTY_INITIALIZER; > > > + struct ds snat_action = DS_EMPTY_INITIALIZER; > > > + > > > + switch (type) { > > > + case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > + ds_put_format(&undnat_action, "flags.force_snat_for_lb = 1; > > next;"); > > > + break; > > > + case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > + ds_put_format(&undnat_action, "flags.skip_snat_for_lb = 1; > > next;"); > > > + break; > > > + case LROUTER_NAT_LB_FLOW_NORMAL: > > > + case LROUTER_NAT_LB_FLOW_MAX: > > > + ds_put_format(&undnat_action, "%s", > > > + lrouter_use_common_zone(od) ? "ct_dnat_in_czone;" > > > + : "ct_dnat;"); > > > + break; > > > + } > > > + > > > + /* undnat_action: Remove the ct action from the lr_out_undenat NAT > > rule. > > > + */ > > > + if (stateless_nat) { > > > + ds_clear(&undnat_action); > > > + ds_put_format(&undnat_action, "next;"); > > > + } > > > + > > > /* We need to centralize the LB traffic to properly perform > > > * the undnat stage. > > > */ > > > @@ -11873,11 +11898,41 @@ build_distr_lrouter_nat_flows_for_lb(struct > > lrouter_nat_lb_flows_ctx *ctx, > > > ds_put_format(ctx->undnat_match, ") && (inport == %s || outport == > > %s)" > > > " && is_chassis_resident(%s)", dgp->json_key, > > dgp->json_key, > > > dgp->cr_port->json_key); > > > + /* Use the LB protocol as matching criteria for out undnat and snat > > when > > > + * creating LBs with stateless NAT. */ > > > + if (stateless_nat) { > > > + ds_put_format(ctx->undnat_match, " && %s", ctx->lb->proto); > > > + } > > > ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_UNDNAT, 120, > > > - ds_cstr(ctx->undnat_match), undnat_action, > > > - &ctx->lb->nlb->header_, > > > + ds_cstr(ctx->undnat_match), > > > + ds_cstr(&undnat_action), > > &ctx->lb->nlb->header_, > > > lflow_ref); > > > + > > > + /* snat_action: Add a new lr_out_snat rule with the LB VIP as > > source IP > > > + * action to perform the NAT stateless pipeline > > completely. > > > + */ > > > + if (stateless_nat) { > > > + if (ctx->lb_vip->port_str) { > > > + ds_put_format(&snat_action, "%s.src=%s; %s.src=%s; next;", > > > + ctx->lb_vip->address_family == AF_INET6 ? > > > + "ip6" : "ip4", > > > + ctx->lb_vip->vip_str, ctx->lb->proto, > > > + ctx->lb_vip->port_str); > > > + } else { > > > + ds_put_format(&snat_action, "%s.src=%s; next;", > > > + ctx->lb_vip->address_family == AF_INET6 ? > > > + "ip6" : "ip4", > > > + ctx->lb_vip->vip_str); > > > + } > > > + ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_SNAT, 160, > > > + ds_cstr(ctx->undnat_match), > > > + ds_cstr(&snat_action), > > &ctx->lb->nlb->header_, > > > + lflow_ref); > > > + } > > > + > > > ds_truncate(ctx->undnat_match, undnat_match_len); > > > + ds_destroy(&undnat_action); > > > + ds_destroy(&snat_action); > > > } > > > > > > static void > > > @@ -12022,6 +12077,8 @@ build_lrouter_nat_flows_for_lb( > > > * lflow generation for them. > > > */ > > > size_t index; > > > + bool use_stateless_nat = smap_get_bool(&lb->nlb->options, > > > + "use_stateless_nat", false); > > > BITMAP_FOR_EACH_1 (index, bitmap_len, lb_dps->nb_lr_map) { > > > struct ovn_datapath *od = lr_datapaths->array[index]; > > > enum lrouter_nat_lb_flow_type type; > > > @@ -12043,8 +12100,17 @@ build_lrouter_nat_flows_for_lb( > > > if (!od->n_l3dgw_ports) { > > > bitmap_set1(gw_dp_bitmap[type], index); > > > } else { > > > - build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > > - lb_dps->lflow_ref); > > > + /* Create stateless LB NAT rules when using multiple DGPs > > and > > > + * use_stateless_nat is true. > > > + */ > > > + bool stateless_nat = (od->n_l3dgw_ports > 1) > > > + ? use_stateless_nat : false; > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > > > + struct ovn_port *dgp = od->l3dgw_ports[i]; > > > + build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > > + lb_dps->lflow_ref, > > dgp, > > > + stateless_nat); > > > + } > > > } > > > > > > if (lb->affinity_timeout) { > > > diff --git a/ovn-nb.xml b/ovn-nb.xml > > > index 2836f58f5..ad03c6214 100644 > > > --- a/ovn-nb.xml > > > +++ b/ovn-nb.xml > > > @@ -2302,6 +2302,16 @@ or > > > local anymore by the ovn-controller. This option is set to > > > <code>false</code> by default. > > > </column> > > > + > > > + <column name="options" key="use_stateless_nat" > > > + type='{"type": "boolean"}'> > > > + If the load balancer is configured with > > <code>use_stateless_nat</code> > > > + option to <code>true</code>, the logical router that references > > this > > > + load balancer will use Stateless NAT rules when the logical > > router > > > + has multiple distributed gateway ports(DGP). Otherwise, the > > outbound > > > + traffic may be dropped in scenarios where we have different > > chassis > > > + for each DGP. This option is set to <code>false</code> by > > default. > > > + </column> > > > </group> > > > </table> > > > > > > diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at > > > index 757917626..2f69433fc 100644 > > > --- a/tests/multinode-macros.at > > > +++ b/tests/multinode-macros.at > > > @@ -40,6 +40,27 @@ m4_define([M_START_TCPDUMP], > > > ] > > > ) > > > > > > +# M_EXEC([fake_node], [command]) > > > +# > > > +# Execute 'command' in 'fakenode' > > > +m4_define([M_EXEC], > > > + [podman exec $1 $2]) > > > + > > > +# M_CHECK_EXEC([fake_node], [command], other_params...) > > > +# > > > +# Wrapper for AT_CHECK that executes 'command' inside 'fake_node''s'. > > > +# 'other_params' as passed as they are to AT_CHECK. > > > +m4_define([M_CHECK_EXEC], > > > + [ AT_CHECK([M_EXEC([$1], [$2])], m4_shift(m4_shift(m4_shift($@)))) ] > > > +) > > > + > > > +# M_FORMAT_CT([ip-addr]) > > > +# > > > +# Strip content from the piped input which would differ from test to > > test > > > +# and limit the output to the rows containing 'ip-addr'. > > > +# > > > +m4_define([M_FORMAT_CT], > > > + [[grep -F "dst=$1," | sed -e 's/id=[0-9]*/id=<cleared>/g' -e > > 's/state=[0-9_A-Z]*/state=<cleared>/g' | sort | uniq | sed -e > > 's/zone=[[0-9]]*/zone=<cleared>/' -e 's/mark=[[0-9]]*/mark=<cleared>/' ]]) > > > > > > OVS_START_SHELL_HELPERS > > > > > > @@ -76,6 +97,25 @@ multinode_nbctl () { > > > m_as ovn-central ovn-nbctl "$@" > > > } > > > > > > +check_fake_multinode_setup_by_nodes() { > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > + for c in $1 > > > + do > > > + AT_CHECK([m_as $c ovn-appctl -t ovn-controller version], [0], > > [ignore]) > > > + done > > > +} > > > + > > > +cleanup_multinode_resources_by_nodes() { > > > + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db > > > + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > + for c in $1 > > > + do > > > + m_as $c ovs-vsctl del-br br-int > > > + m_as $c ip --all netns delete > > > + done > > > +} > > > + > > > # m_count_rows TABLE [CONDITION...] > > > # > > > # Prints the number of rows in TABLE (that satisfy CONDITION). > > > diff --git a/tests/multinode.at b/tests/multinode.at > > > index a0eb8fc67..b1beb4d97 100644 > > > --- a/tests/multinode.at > > > +++ b/tests/multinode.at > > > @@ -1591,3 +1591,559 @@ AT_CHECK([cat ch1_eth2.tcpdump], [0], [dnl > > > ]) > > > > > > AT_CLEANUP > > > + > > > +AT_SETUP([ovn multinode load-balancer with multiple DGPs and multiple > > chassis]) > > > + > > > +# Check that ovn-fake-multinode setup is up and running - requires > > additional nodes > > > +check_fake_multinode_setup_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > + > > > +# Delete the multinode NB and OVS resources before starting the test. > > > +cleanup_multinode_resources_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > + > > > +# Network topology > > > +# > > > +# publicp1 (ovn-chassis-3) (20.0.0.3/24) > > > +# | > > > +# overlay > > > +# | > > > +# DGP public1 (ovn-gw-1) (20.0.0.1/24) > > > +# | > > > +# | > > > +# | > > > +# lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) 10.0.0.3/24 > > > +# | | > > > +# | + --- sw0p2 (ovn-chassis-2) 10.0.0.4/24 > > > +# | > > > +# DGP public2 (ovn-gw-2) (30.0.0.1/24) > > > +# | > > > +# overlay > > > +# | > > > +# publicp2 (ovn-chassis-4) (30.0.0.3/24) > > > + > > > +# Delete already used ovs-ports > > > +m_as ovn-chassis-1 ovs-vsctl del-port br-int sw0p1-p > > > +m_as ovn-chassis-2 ovs-vsctl del-port br-int sw0p2-p > > > +m_as ovn-chassis-1 ip link del sw0p1-p > > > +m_as ovn-chassis-2 ip link del sw0p2-p > > > +m_as ovn-chassis-3 ovs-vsctl del-port br-int publicp1-p > > > +m_as ovn-chassis-4 ovs-vsctl del-port br-int publicp2-p > > > +m_as ovn-chassis-3 ip link del publicp1-p > > > +m_as ovn-chassis-4 ip link del publicp2-p > > > + > > > +# Create East-West switch for LB backends > > > +check multinode_nbctl ls-add sw0 > > > +check multinode_nbctl lsp-add sw0 sw0-port1 > > > +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 > > 10.0.0.3 1000::3" > > > +check multinode_nbctl lsp-add sw0 sw0-port2 > > > +check multinode_nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 > > 10.0.0.4 1000::4" > > > + > > > +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 > > 50:54:00:00:00:03 1400 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a > > > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 > > 50:54:00:00:00:04 1400 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a > > > + > > > +m_wait_for_ports_up > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > > 10.0.0.4 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 > > 10.0.0.3 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +# Create a logical router and attach to sw0 > > > +check multinode_nbctl lr-add lr0 > > > +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 > > 1000::a/64 > > > +check multinode_nbctl lsp-add sw0 sw0-lr0 > > > +check multinode_nbctl lsp-set-type sw0-lr0 router > > > +check multinode_nbctl lsp-set-addresses sw0-lr0 router > > > +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 > > > + > > > +# create external connection for N/S traffic using multiple DGPs > > > +check multinode_nbctl ls-add public > > > + > > > +# DGP public1 > > > +check multinode_nbctl lsp-add public ln-public-1 > > > +check multinode_nbctl lsp-set-type ln-public-1 localnet > > > +check multinode_nbctl lsp-set-addresses ln-public-1 unknown > > > +check multinode_nbctl lsp-set-options ln-public-1 network_name=public1 > > > + > > > +# DGP public2 > > > +# create exteranl connection for N/S traffic > > > +check multinode_nbctl lsp-add public ln-public-2 > > > +check multinode_nbctl lsp-set-type ln-public-2 localnet > > > +check multinode_nbctl lsp-set-addresses ln-public-2 unknown > > > +check multinode_nbctl lsp-set-options ln-public-2 network_name=public2 > > > + > > > +# Attach DGP public1 to GW-1 and chassis-3 (overlay connectivity) > > > +m_as ovn-gw-1 ovs-vsctl set open . > > external-ids:ovn-bridge-mappings=public1:br-ex > > > +m_as ovn-chassis-3 ovs-vsctl set open . > > external-ids:ovn-bridge-mappings=public1:br-ex > > > + > > > +# Attach DGP public2 to GW-2 and chassis-4 (overlay connectivity) > > > +m_as ovn-gw-2 ovs-vsctl set open . > > external-ids:ovn-bridge-mappings=public2:br-ex > > > +m_as ovn-chassis-4 ovs-vsctl set open . > > external-ids:ovn-bridge-mappings=public2:br-ex > > > + > > > +# Create the external LR0 port to the DGP public1 > > > +check multinode_nbctl lsp-add public public-port1 > > > +check multinode_nbctl lsp-set-addresses public-port1 "40:54:00:00:00:03 > > 20.0.0.3 2000::3" > > > + > > > +check multinode_nbctl lrp-add lr0 lr0-public-p1 00:00:00:00:ff:02 > > 20.0.0.1/24 2000::a/64 > > > +check multinode_nbctl lsp-add public public-lr0-p1 > > > +check multinode_nbctl lsp-set-type public-lr0-p1 router > > > +check multinode_nbctl lsp-set-addresses public-lr0-p1 router > > > +check multinode_nbctl lsp-set-options public-lr0-p1 > > router-port=lr0-public-p1 > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p1 ovn-gw-1 10 > > > + > > > +# Create a VM on ovn-chassis-3 in the same public1 overlay > > > +m_as ovn-chassis-3 /data/create_fake_vm.sh public-port1 publicp1 > > 40:54:00:00:00:03 1400 20.0.0.3 24 20.0.0.1 2000::4/64 2000::a > > > + > > > +m_wait_for_ports_up public-port1 > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [ping -q -c 3 -i 0.3 -w 2 > > 20.0.0.1 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +# Create the external LR0 port to the DGP public2 > > > +check multinode_nbctl lsp-add public public-port2 > > > +check multinode_nbctl lsp-set-addresses public-port2 "60:54:00:00:00:03 > > 30.0.0.3 3000::3" > > > + > > > +check multinode_nbctl lrp-add lr0 lr0-public-p2 00:00:00:00:ff:03 > > 30.0.0.1/24 3000::a/64 > > > +check multinode_nbctl lsp-add public public-lr0-p2 > > > +check multinode_nbctl lsp-set-type public-lr0-p2 router > > > +check multinode_nbctl lsp-set-addresses public-lr0-p2 router > > > +check multinode_nbctl lsp-set-options public-lr0-p2 > > router-port=lr0-public-p2 > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p2 ovn-gw-2 10 > > > + > > > +# Create a VM on ovn-chassis-4 in the same public2 overlay > > > +m_as ovn-chassis-4 /data/create_fake_vm.sh public-port2 publicp2 > > 60:54:00:00:00:03 1400 30.0.0.3 24 30.0.0.1 3000::4/64 3000::a > > > + > > > +m_wait_for_ports_up public-port2 > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [ping -q -c 3 -i 0.3 -w 2 > > 30.0.0.1 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +# Add a default route for multiple DGPs - using ECMP > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 20.0.0.3 > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 30.0.0.3 > > > + > > > +# Add SNAT rules using gateway-port > > > +check multinode_nbctl --gateway-port lr0-public-p1 lr-nat-add lr0 snat > > 20.0.0.1 10.0.0.0/24 > > > +check multinode_nbctl --gateway-port lr0-public-p2 lr-nat-add lr0 snat > > 30.0.0.1 10.0.0.0/24 > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > > 20.0.0.3 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 > > 30.0.0.3 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +# create LB > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:80" "10.0.0.3:80, > > 10.0.0.4:80" > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > + > > > +# Set use_stateless_nat to true > > > +check multinode_nbctl set load_balancer lb0 > > options:use_stateless_nat=true > > > + > > > +# Start backend http services > > > +M_NS_DAEMONIZE([ovn-chassis-1], [sw0p1], [$PYTHON -m http.server --bind > > 10.0.0.3 80 >/dev/null 2>&1], [http1.pid]) > > > +M_NS_DAEMONIZE([ovn-chassis-2], [sw0p2], [$PYTHON -m http.server --bind > > 10.0.0.4 80 >/dev/null 2>&1], [http2.pid]) > > > + > > > +# wait for http server be ready > > > +sleep 2 > > > + > > > +# Flush conntrack entries for easier output parsing of next test. > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v 172.16.0.100:80 > > --retry 3 --max-time 1 --local-port 59002 2> curl.out']) > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [sh -c 'cat curl.out | > > grep -i -e connect | grep -v 'Server:''], \ > > > +[0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v 172.16.0.100:80 > > --retry 3 --max-time 1 --local-port 59003 2> curl.out']) > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [sh -c 'cat curl.out | > > grep -i -e connect | grep -v 'Server:''], \ > > > +[0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v 172.16.0.100:80 > > --retry 3 --max-time 1 --local-port 59001']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | > > M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v 172.16.0.100:80 > > --retry 3 --max-time 1 --local-port 59000']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | > > M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# create a big file on web servers for download > > > +M_NS_EXEC([ovn-chassis-1], [sw0p1], [dd bs=512 count=200000 > > if=/dev/urandom of=download_file]) > > > +M_NS_EXEC([ovn-chassis-2], [sw0p2], [dd bs=512 count=200000 > > if=/dev/urandom of=download_file]) > > > + > > > +# Flush conntrack entries for easier output parsing of next test. > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59004 > > 2>curl.out']) > > > + > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > > curl.out | \ > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +# Check if we have only one backend for the same connection - orig + > > dest ports > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Check if gw-2 is empty to ensure that the traffic only come from/to > > the originator chassis via DGP public1 > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > +0 > > > +]) > > > + > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > + > > > +if [[ $backend_check -gt 0 ]]; then > > > +# Backend resides on ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59004,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +else > > > +# Backend resides on ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59004,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +fi > > > + > > > +# Flush conntrack entries for easier output parsing of next test. > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +# Check the flows again for a new source port > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59005 > > 2>curl.out']) > > > + > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > > curl.out | \ > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +# Check if we have only one backend for the same connection - orig + > > dest ports > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Check if gw-2 is empty to ensure that the traffic only come from/to > > the originator chassis via DGP public1 > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > +0 > > > +]) > > > + > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > + > > > +if [[ $backend_check -gt 0 ]]; then > > > +# Backend resides on ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59005,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +else > > > +# Backend resides on ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59005,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +fi > > > + > > > +# Flush conntrack entries for easier output parsing of next test. > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +# Start a new test using the second DGP as origin (public2) > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59006 > > 2>curl.out']) > > > + > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > > curl.out | \ > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +# Check if we have only one backend for the same connection - orig + > > dest ports > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Check if gw-1 is empty to ensure that the traffic only come from/to > > the originator chassis via DGP public2 > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > +0 > > > +]) > > > + > > > +# Check the backend IP from ct entries on gw-2 (DGP public2) > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > + > > > +if [[ $backend_check -gt 0 ]]; then > > > +# Backend resides on ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59006,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +else > > > +# Backend resides on ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59006,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +fi > > > + > > > +# Flush conntrack entries for easier output parsing of next test. > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +# Check the flows again for a new source port using the second DGP as > > origin (public2) > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59007 > > 2>curl.out']) > > > + > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > > curl.out | \ > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +# Check if we have only one backend for the same connection - orig + > > dest ports > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Check if gw-1 is empty to ensure that the traffic only come from/to > > the originator chassis via DGP public2 > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > +0 > > > +]) > > > + > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > + > > > +if [[ $backend_check -gt 0 ]]; then > > > +# Backend resides on ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59007,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +else > > > +# Backend resides on ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59007,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +fi > > > + > > > +# Check multiple requests coming from DGP's public1 and public2 > > > + > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +# Remove the LB and change the VIP port - different from the backend > > ports > > > +check multinode_nbctl lb-del lb0 > > > + > > > +# create LB again > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:9000" "10.0.0.3:80, > > 10.0.0.4:80" > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > + > > > +# Set use_stateless_nat to true > > > +check multinode_nbctl set load_balancer lb0 > > options:use_stateless_nat=true > > > + > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +# Check end-to-end request using a new port for VIP > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port 59008 > > 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | > > M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +# Check end-to-end request using a new port for VIP > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port 59008 > > 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | > > M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +AT_CLEANUP > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > > > index dcc3dbbc3..9e7a2f225 100644 > > > --- a/tests/ovn-northd.at > > > +++ b/tests/ovn-northd.at > > > @@ -13864,3 +13864,323 @@ check_no_redirect > > > > > > AT_CLEANUP > > > ]) > > > + > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + NAT > > Stateless)]) > > > +ovn_start > > > + > > > +check ovn-nbctl ls-add public > > > +check ovn-nbctl lr-add lr1 > > > + > > > +# lr1 DGP ts1 > > > +check ovn-nbctl ls-add ts1 > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 172.16.10.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > + > > > +# lr1 DGP ts2 > > > +check ovn-nbctl ls-add ts2 > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 172.16.20.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > + > > > +# lr1 DGP public > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 173.16.0.1/16 > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > + > > > +check ovn-nbctl ls-add s1 > > > +# s1 - lr1 > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 172.16.0.1" > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > + > > > +# s1 - backend vm1 > > > +check ovn-nbctl lsp-add s1 vm1 > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 172.16.0.101" > > > + > > > +# s1 - backend vm2 > > > +check ovn-nbctl lsp-add s1 vm2 > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 172.16.0.102" > > > + > > > +# s1 - backend vm3 > > > +check ovn-nbctl lsp-add s1 vm3 > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 172.16.0.103" > > > + > > > +# Add the lr1 DGP ts1 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP ts2 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP public to the public switch > > > +check ovn-nbctl lsp-add public public_lr1 > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > nat-addresses=router > > > + > > > +# Create the Load Balancer lb1 > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > + > > > +# Set use_stateless_nat to true > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > options:use_stateless_nat=true > > > + > > > +# Associate load balancer to s1 > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows s1 > s1flows > > > +AT_CAPTURE_FILE([s1flows]) > > > + > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > +]) > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > ip4.dst == 30.0.0.1), > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > +]) > > > + > > > +# Associate load balancer to lr1 with DGP > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > +AT_CAPTURE_FILE([lr1flows]) > > > + > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1-ts1")), > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1-ts2")), > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1_public")), > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > +]) > > > + > > > +# 2. Check if the DGP ports are in the match with action next > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], [dnl > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") && > > is_chassis_resident("cr-lr1_public") && tcp), action=(next;) > > > +]) > > > + > > > +# 3. Check if the VIP IP is in the ipX.src action > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > action=(next;) > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > action=(next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(ip4.src=30.0.0.1; next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(ip4.src=30.0.0.1; next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") && > > is_chassis_resident("cr-lr1_public") && tcp), action=(ip4.src=30.0.0.1; > > next;) > > > +]) > > > + > > > +AT_CLEANUP > > > +]) > > > + > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + NAT > > Stateless) - IPv6]) > > > +ovn_start > > > + > > > +check ovn-nbctl ls-add public > > > +check ovn-nbctl lr-add lr1 > > > + > > > +# lr1 DGP ts1 > > > +check ovn-nbctl ls-add ts1 > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > > 2001:db8:aaaa:1::1/64 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > + > > > +# lr1 DGP ts2 > > > +check ovn-nbctl ls-add ts2 > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > > 2001:db8:aaaa:2::1/64 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > + > > > +# lr1 DGP public > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > > 2001:db8:bbbb::1/64 > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > > 2001:db8:aaaa:3::1/64 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > + > > > +check ovn-nbctl ls-add s1 > > > +# s1 - lr1 > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > > 2001:db8:aaaa:3::1" > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > + > > > +# s1 - backend vm1 > > > +check ovn-nbctl lsp-add s1 vm1 > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > > 2001:db8:aaaa:3::101" > > > + > > > +# s1 - backend vm2 > > > +check ovn-nbctl lsp-add s1 vm2 > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > > 2001:db8:aaaa:3::102" > > > + > > > +# s1 - backend vm3 > > > +check ovn-nbctl lsp-add s1 vm3 > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > > 2001:db8:aaaa:3::103" > > > + > > > +# Add the lr1 DGP ts1 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP ts2 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP public to the public switch > > > +check ovn-nbctl lsp-add public public_lr1 > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > nat-addresses=router > > > + > > > +# Create the Load Balancer lb1 > > > +check ovn-nbctl --wait=sb lb-add lb1 "2001:db8:cccc::1" > > "2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101" > > > + > > > +# Set use_stateless_nat to true > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > options:use_stateless_nat=true > > > + > > > +# Associate load balancer to s1 > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows s1 > s1flows > > > +AT_CAPTURE_FILE([s1flows]) > > > + > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep > > "2001:db8:cccc::1"], [0], [dnl > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 > > && ip6.dst == 2001:db8:cccc::1), action=(xxreg1 = 2001:db8:cccc::1; > > ct_lb_mark;) > > > +]) > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > "2001:db8:cccc::1"], [0], [dnl > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > ip6.dst == 2001:db8:cccc::1), > > action=(ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > +]) > > > + > > > +# Associate load balancer to lr1 with DGP > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > +AT_CAPTURE_FILE([lr1flows]) > > > + > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > "2001:db8:cccc::1"], [0], [dnl > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > is_chassis_resident("cr-lr1-ts1")), > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > is_chassis_resident("cr-lr1-ts2")), > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > is_chassis_resident("cr-lr1_public")), > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > +]) > > > + > > > +# 2. Check if the DGP ports are in the match with action next > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], [dnl > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport == > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport == > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || outport == > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > action=(next;) > > > +]) > > > + > > > +# 3. Check if the VIP IP is in the ipX.src action > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > action=(next;) > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > action=(next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport == > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), > > action=(ip6.src=2001:db8:cccc::1; next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport == > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), > > action=(ip6.src=2001:db8:cccc::1; next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || outport == > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > action=(ip6.src=2001:db8:cccc::1; next;) > > > +]) > > > + > > > +AT_CLEANUP > > > +]) > > > + > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (DGP)]) > > > +ovn_start > > > + > > > +check ovn-nbctl ls-add public > > > +check ovn-nbctl lr-add lr1 > > > + > > > +# lr1 DGP ts1 > > > +check ovn-nbctl ls-add ts1 > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 172.16.10.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-1 > > > + > > > +# lr1 DGP ts2 > > > +check ovn-nbctl ls-add ts2 > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 172.16.20.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-1 > > > + > > > +# lr1 DGP public > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 173.16.0.1/16 > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > + > > > +check ovn-nbctl ls-add s1 > > > +# s1 - lr1 > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 172.16.0.1" > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > + > > > +# s1 - backend vm1 > > > +check ovn-nbctl lsp-add s1 vm1 > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 172.16.0.101" > > > + > > > +# s1 - backend vm2 > > > +check ovn-nbctl lsp-add s1 vm2 > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 172.16.0.102" > > > + > > > +# s1 - backend vm3 > > > +check ovn-nbctl lsp-add s1 vm3 > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 172.16.0.103" > > > + > > > +# Add the lr1 DGP ts1 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP ts2 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP public to the public switch > > > +check ovn-nbctl lsp-add public public_lr1 > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > nat-addresses=router > > > + > > > +# Create the Load Balancer lb1 > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > + > > > +# Associate load balancer to s1 > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows s1 > s1flows > > > +AT_CAPTURE_FILE([s1flows]) > > > + > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > +]) > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > ip4.dst == 30.0.0.1), > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > +]) > > > + > > > +# Associate load balancer to lr1 with DGP > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > +AT_CAPTURE_FILE([lr1flows]) > > > + > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1-ts1")), > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1-ts2")), > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1_public")), > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > +]) > > > + > > > +AT_CLEANUP > > > +]) > > > -- > > > 2.34.1 > > > > > > > > > -- > > > > > > > > > > > > > > > _'Esta mensagem é direcionada apenas para os endereços constantes no > > > cabeçalho inicial. Se você não está listado nos endereços constantes no > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas > > estão > > > imediatamente anuladas e proibidas'._ > > > > > > > > > * **'Apesar do Magazine Luiza tomar > > > todas as precauções razoáveis para assegurar que nenhum vírus esteja > > > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade > > por > > > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos'.* > > > > > > > > > > > > _______________________________________________ > > > dev mailing list > > > dev@openvswitch.org > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > > -- > > > > > _‘Esta mensagem é direcionada apenas para os endereços constantes no > cabeçalho inicial. Se você não está listado nos endereços constantes no > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão > imediatamente anuladas e proibidas’._ > > > * **‘Apesar do Magazine Luiza tomar > todas as precauções razoáveis para assegurar que nenhum vírus esteja > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* > > > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
On Thu, Sep 26, 2024 at 12:56 PM Numan Siddique <numans@ovn.org> wrote: > > On Thu, Sep 26, 2024 at 10:55 AM Roberto Bartzen Acosta via dev > <ovs-dev@openvswitch.org> wrote: > > > > Hi Numan, > > > > Thanks for your feedback and review. > > > > Em qua., 25 de set. de 2024 às 19:50, Numan Siddique <numans@ovn.org> > > escreveu: > > > > > On Thu, Sep 19, 2024 at 6:12 PM Roberto Bartzen Acosta via dev > > > <ovs-dev@openvswitch.org> wrote: > > > > > > > > This commit fixes the build_distr_lrouter_nat_flows_for_lb function to > > > > include a DNAT flow entry for each DGP in use. Since we have added > > > support > > > > to create multiple gateway ports per logical router, it's necessary to > > > > include in the LR NAT rules pipeline a specific entry for each attached > > > DGP. > > > > Otherwise, the inbound traffic will only be redirected when the incoming > > > LRP > > > > matches the chassis_resident field. > > > > > > > > Additionally, this patch includes the ability to use load-balancer with > > > DGPs > > > > attached to multiple chassis. We can have each of the DGPs associated > > > with a > > > > different chassis, and in this case the DNAT rules added by default will > > > not > > > > be enough to guarantee outgoing traffic. > > > > > > > > To solve the multiple chassis for DGPs problem, this patch include a new > > > > config options to be configured in the load-balancer. If the > > > use_stateless_nat > > > > is set to true, the logical router that references this load-balancer > > > will use > > > > Stateless NAT rules when the logical router has multiple DGPs. After > > > applying > > > > this patch and setting the use_stateless_nat option, the inbound and/or > > > > outbound traffic can pass through any chassis where the DGP resides > > > without > > > > having problems with CT state. > > > > > > > > Reported-at: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2054322 > > > > Fixes: 15348b7b806f ("ovn-northd: Multiple distributed gateway port > > > support.") > > > > > > > > Signed-off-by: Roberto Bartzen Acosta <roberto.acosta@luizalabs.com> > > > > > > Hi Roberto, > > > > > > Thanks for the patch. I tested this patch using the test example in > > > multinode.at. > > > > > > The test case adds the below load balancer > > > > > > [root@ovn-central ~]# ovn-nbctl lb-list > > > UUID LB PROTO > > > VIP IPs > > > f3e29869-3bb5-4df0-960a-171106f5913a lb0 tcp > > > 172.16.0.100:9000 10.0.0.3:80,10.0.0.4:80 > > > > > > And the below logical flows are generated by this patch > > > > > > -------- > > > [root@ovn-central ~]# ovn-sbctl dump-flows lr0 | grep 172.16.0.100 > > > table=6 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst > > > == 172.16.0.100), action=(ct_dnat;) > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > > is_chassis_resident("cr-lr0-public-p1")), > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80, > > > 10.0.0.4:80);) > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > > is_chassis_resident("cr-lr0-public-p2")), > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80, > > > 10.0.0.4:80);) > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > > tcp.src == 80)) && (inport == "lr0-public-p1" || outport == > > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > > tcp.src == 80)) && (inport == "lr0-public-p2" || outport == > > > "lr0-public-p2") && is_chassis_resident("cr-lr0-public-p2") && tcp), > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > > -------------- > > > > > > > > > I fail to understand the reason for modifying the ip4.dst before > > > calling ct_lb_mark. Can you please explain why ? Because the > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;) will first modify the > > > ip4.dst to 10.0.0.3 and > > > then to 10.0.0.4 and then the ct_lb_mark will actually do the > > > conntrack with NAT to either 10.0.0.3 or 10.0.0.4. > > > > > > Is it because you want the conntrack entry to not have 172.16.0.100 ? > > > > > > > The only reason I included this ip4.dst action in the DNAT rule is because > > it's required to accept packets coming from a chassis that doesn't have > > previously created conntrack entries. The main feature introduced in this > > patch is to allow the administrator to have multiple DGPs attached to > > different chassis (is_chassis_resident...). So, my implementation was based > > on the normal behavior when using stateless NAT for external addresses, > > where we need to add the ipx.dst in lr_in_dnat for traffic to be received > > on the chassis (put the DGP port as chassis_resident match, as is the case > > with stateless NAT [1] with DGP[2]). > > > > The question is, if we only have the ct_lb_mark, packets that pass through > > the chassis and are already part of an active flow in another chassis (same > > IPs and Ports) will be dropped because there is no correspondence in the > > backend. So only packets with the NEW flag will be accepted and sent to the > > backend (at least for TCP traffic). If we only have the ip4.dst action, > > this will always perform the dnat for the same backend, without balancing. > > Therefore, the combination of the two actions allows the packet to always > > be received (regardless of whether conntrack is active for it), and > > ct_lb_mark will take care of balancing for different backends. > > > > If we had conntrack sync between different chassis this would not be > > necessary, as the ct_lb_mark action could always be executed without > > dropping packets due to lack of correspondence in the conntrack table. > > I still need to understand it properly. I'll get back to you after > reading your reply thoroughly. > > But what's the point of doing multiple ip4.dst modifications for the > same packet ? > > i.e - action=(ip4.dst = 10.0.0.3; ip4.dst = 10.0.0.4.....) > > The 2nd ip4.dst action overwrites and the packet's ip4.dst will be > 10.0.0.4 before ct_lb_mark. > > > > > > [1] > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15737 > > [2] > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15726 > > > > > > > > > > Also I don't understand why this patch adds the logical flows in > > > "lr_out_snat" stage ? > > > > > > > The flow for lr_out_snat is necessary for the correct functioning of > > stateless NAT for the same reason explained previously. I mean, if the > > outgoing packet is redirected to a chassis that doesn't have an active > > conntrack entry, it will not be NATed by ct_lb action because it doesn't > > refer to a valid flow (use case with ecmp). > > > > So it is necessary to create a stateless SNAT rule (similar to this [3]) > > with a lower priority than the other router pipeline entries, in this case, > > if the packet is not SNATed by ct_lb (conntrack missed) it will be SNATed > > by stateless NAT rule. > > > > [3] > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15884 > > > > > > > > > > > > Using the system multinode test as an example, the below fails > > > (which is a regression) > > > > > > --- > > > root@ovn-chassis-3 ~]# ip netns exec publicp1 nc -vz 10.0.0.3 80 > > > ---- > > > > > > In the above test, publicp1 with IP 20.0.0.3 when it tries to connect > > > to one if the backends directly (without the LB VIP), it fails. > > > It fails because of the logical flows in "lr_out_snat". > > > > > > > > > Looks to me the solution proposed here is incomplete. > > > > > > Also please note that in our CI we run the multinode tests > > > periodically once a day using the v0.1 of the ovn-fake-multinode > > > and the tests you added will fail. This needs to be fixed and until > > > we move to the latest version of ovn-fake-multinode. > > > > > > > I imagine that the test you are doing is using the same port as the LB > > backend (TCP 80 in this case). So, the stateless lr_out_snat flow will > > force the output to be SNATed because this port is in use by the backend. > > Traffic to/from other ports will work without problems and will follow the > > normal programmed flows (e.g. ICMP). > > > > This is necessary to ensure the egress traffic because the DGPs are > > distributed across multiple chassis. Also, this setup is being validated in > > the test ovn-fake-multinode testcase (ICMP from the backends chassis use > > the router's default SNAT and not the LB's). I didn't understand the > > regression you mentioned because this was programmed to be stateless and > > it's traffic that uses the same ports as the LB backend, could you explain > > better? > > > > To reproduce the issue I mentioned, first run the multinode test you > have added. > Don't destroy the resources. And then login to ovn-chassis-3 container and run > the command > > #ip netns exec publicp1 curl -v 10.0.0.3:80 > > So instead of curling to the LB VIP (which is 172.16.0.100:80), curl > directly to the backends. Without your patch it works > and with your patch it doesn't. > > If you run tcpdump you'd see the below > > 16:51:24.526980 40:54:00:00:00:03 > 00:00:00:00:ff:02, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 64, id 44276, offset 0, flags [DF], > proto TCP (6), length 60) > 20.0.0.3.46978 > 10.0.0.3.http: Flags [S], cksum 0x1e34 (incorrect > -> 0x4b96), seq 1989771329, win 65280, options [mss 1360,sackOK,TS val > 1753008285 ecr 0,nop,wscale 7], length 0 > 16:51:24.528559 00:00:00:00:ff:02 > 40:54:00:00:00:03, ethertype IPv4 > (0x0800), length 74: (tos 0x0, ttl 63, id 0, offset 0, flags [DF], > proto TCP (6), length 60) > 172.16.0.100.http > 20.0.0.3.46978: Flags [S.], cksum 0xc0a5 > (incorrect -> 0x2094), seq 1088931300, ack 1989771330, win 64704, > options [mss 1360,sackOK,TS val 3729696164 ecr 1753008285,nop,wscale > 7], length 0 > 16:51:24.528571 40:54:00:00:00:03 > 00:00:00:00:ff:02, ethertype IPv4 > (0x0800), length 54: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], > proto TCP (6), length 40) > > The request is sent to 10.0.0.3 but the reply is received from > 172.16.0.100 which is wrong. Also If I modify the LB VIP tcp port to 9000 [root@ovn-central ~]# ovn-nbctl lb-list UUID LB PROTO VIP IPs f3e29869-3bb5-4df0-960a-171106f5913a lb0 tcp 172.16.0.100:9000 10.0.0.3:80,10.0.0.4:80 When a request is sent to 10.0.0.3:80, the reply is received from 172.16.0.100:9000. Thanks Numan > > Thanks > Numan > > > > Thanks, > > Roberto > > > > > > > Thanks > > > Numan > > > > > > > > > > --- > > > > northd/en-lr-stateful.c | 12 - > > > > northd/northd.c | 116 ++++++-- > > > > ovn-nb.xml | 10 + > > > > tests/multinode-macros.at | 40 +++ > > > > tests/multinode.at | 556 ++++++++++++++++++++++++++++++++++++++ > > > > tests/ovn-northd.at | 320 ++++++++++++++++++++++ > > > > 6 files changed, 1017 insertions(+), 37 deletions(-) > > > > > > > > diff --git a/northd/en-lr-stateful.c b/northd/en-lr-stateful.c > > > > index baf1bd2f8..f09691af6 100644 > > > > --- a/northd/en-lr-stateful.c > > > > +++ b/northd/en-lr-stateful.c > > > > @@ -516,18 +516,6 @@ lr_stateful_record_create(struct lr_stateful_table > > > *table, > > > > > > > > table->array[od->index] = lr_stateful_rec; > > > > > > > > - /* Load balancers are not supported (yet) if a logical router has > > > multiple > > > > - * distributed gateway port. Log a warning. */ > > > > - if (lr_stateful_rec->has_lb_vip && lr_has_multiple_gw_ports(od)) { > > > > - static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1); > > > > - VLOG_WARN_RL(&rl, "Load-balancers are configured on logical " > > > > - "router %s, which has %"PRIuSIZE" distributed " > > > > - "gateway ports. Load-balancer is not supported " > > > > - "yet when there is more than one distributed " > > > > - "gateway port on the router.", > > > > - od->nbr->name, od->n_l3dgw_ports); > > > > - } > > > > - > > > > return lr_stateful_rec; > > > > } > > > > > > > > diff --git a/northd/northd.c b/northd/northd.c > > > > index a267cd5f8..bbe97acf8 100644 > > > > --- a/northd/northd.c > > > > +++ b/northd/northd.c > > > > @@ -11807,31 +11807,30 @@ static void > > > > build_distr_lrouter_nat_flows_for_lb(struct lrouter_nat_lb_flows_ctx > > > *ctx, > > > > enum lrouter_nat_lb_flow_type type, > > > > struct ovn_datapath *od, > > > > - struct lflow_ref *lflow_ref) > > > > + struct lflow_ref *lflow_ref, > > > > + struct ovn_port *dgp, > > > > + bool stateless_nat) > > > > { > > > > - struct ovn_port *dgp = od->l3dgw_ports[0]; > > > > - > > > > - const char *undnat_action; > > > > - > > > > - switch (type) { > > > > - case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > > - undnat_action = "flags.force_snat_for_lb = 1; next;"; > > > > - break; > > > > - case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > > - undnat_action = "flags.skip_snat_for_lb = 1; next;"; > > > > - break; > > > > - case LROUTER_NAT_LB_FLOW_NORMAL: > > > > - case LROUTER_NAT_LB_FLOW_MAX: > > > > - undnat_action = lrouter_use_common_zone(od) > > > > - ? "ct_dnat_in_czone;" > > > > - : "ct_dnat;"; > > > > - break; > > > > - } > > > > + struct ds dnat_action = DS_EMPTY_INITIALIZER; > > > > > > > > /* Store the match lengths, so we can reuse the ds buffer. */ > > > > size_t new_match_len = ctx->new_match->length; > > > > size_t undnat_match_len = ctx->undnat_match->length; > > > > > > > > + /* dnat_action: Add the LB backend IPs as a destination action of > > > the > > > > + * lr_in_dnat NAT rule with cumulative effect because > > > any > > > > + * backend dst IP used in the action list will > > > redirect the > > > > + * packet to the ct_lb pipeline. > > > > + */ > > > > + if (stateless_nat) { > > > > + for (size_t i = 0; i < ctx->lb_vip->n_backends; i++) { > > > > + struct ovn_lb_backend *backend = &ctx->lb_vip->backends[i]; > > > > + bool ipv6 = !IN6_IS_ADDR_V4MAPPED(&backend->ip); > > > > + ds_put_format(&dnat_action, "%s.dst=%s;", ipv6 ? "ip6" : > > > "ip4", > > > > + backend->ip_str); > > > > + } > > > > + } > > > > + ds_put_format(&dnat_action, "%s", ctx->new_action[type]); > > > > > > > > const char *meter = NULL; > > > > > > > > @@ -11841,20 +11840,46 @@ build_distr_lrouter_nat_flows_for_lb(struct > > > lrouter_nat_lb_flows_ctx *ctx, > > > > > > > > if (ctx->lb_vip->n_backends || !ctx->lb_vip->empty_backend_rej) { > > > > ds_put_format(ctx->new_match, " && is_chassis_resident(%s)", > > > > - od->l3dgw_ports[0]->cr_port->json_key); > > > > + dgp->cr_port->json_key); > > > > } > > > > > > > > ovn_lflow_add_with_hint__(ctx->lflows, od, S_ROUTER_IN_DNAT, > > > ctx->prio, > > > > - ds_cstr(ctx->new_match), > > > ctx->new_action[type], > > > > + ds_cstr(ctx->new_match), > > > ds_cstr(&dnat_action), > > > > NULL, meter, &ctx->lb->nlb->header_, > > > > lflow_ref); > > > > > > > > ds_truncate(ctx->new_match, new_match_len); > > > > > > > > + ds_destroy(&dnat_action); > > > > if (!ctx->lb_vip->n_backends) { > > > > return; > > > > } > > > > > > > > + struct ds undnat_action = DS_EMPTY_INITIALIZER; > > > > + struct ds snat_action = DS_EMPTY_INITIALIZER; > > > > + > > > > + switch (type) { > > > > + case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > > + ds_put_format(&undnat_action, "flags.force_snat_for_lb = 1; > > > next;"); > > > > + break; > > > > + case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > > + ds_put_format(&undnat_action, "flags.skip_snat_for_lb = 1; > > > next;"); > > > > + break; > > > > + case LROUTER_NAT_LB_FLOW_NORMAL: > > > > + case LROUTER_NAT_LB_FLOW_MAX: > > > > + ds_put_format(&undnat_action, "%s", > > > > + lrouter_use_common_zone(od) ? "ct_dnat_in_czone;" > > > > + : "ct_dnat;"); > > > > + break; > > > > + } > > > > + > > > > + /* undnat_action: Remove the ct action from the lr_out_undenat NAT > > > rule. > > > > + */ > > > > + if (stateless_nat) { > > > > + ds_clear(&undnat_action); > > > > + ds_put_format(&undnat_action, "next;"); > > > > + } > > > > + > > > > /* We need to centralize the LB traffic to properly perform > > > > * the undnat stage. > > > > */ > > > > @@ -11873,11 +11898,41 @@ build_distr_lrouter_nat_flows_for_lb(struct > > > lrouter_nat_lb_flows_ctx *ctx, > > > > ds_put_format(ctx->undnat_match, ") && (inport == %s || outport == > > > %s)" > > > > " && is_chassis_resident(%s)", dgp->json_key, > > > dgp->json_key, > > > > dgp->cr_port->json_key); > > > > + /* Use the LB protocol as matching criteria for out undnat and snat > > > when > > > > + * creating LBs with stateless NAT. */ > > > > + if (stateless_nat) { > > > > + ds_put_format(ctx->undnat_match, " && %s", ctx->lb->proto); > > > > + } > > > > ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_UNDNAT, 120, > > > > - ds_cstr(ctx->undnat_match), undnat_action, > > > > - &ctx->lb->nlb->header_, > > > > + ds_cstr(ctx->undnat_match), > > > > + ds_cstr(&undnat_action), > > > &ctx->lb->nlb->header_, > > > > lflow_ref); > > > > + > > > > + /* snat_action: Add a new lr_out_snat rule with the LB VIP as > > > source IP > > > > + * action to perform the NAT stateless pipeline > > > completely. > > > > + */ > > > > + if (stateless_nat) { > > > > + if (ctx->lb_vip->port_str) { > > > > + ds_put_format(&snat_action, "%s.src=%s; %s.src=%s; next;", > > > > + ctx->lb_vip->address_family == AF_INET6 ? > > > > + "ip6" : "ip4", > > > > + ctx->lb_vip->vip_str, ctx->lb->proto, > > > > + ctx->lb_vip->port_str); > > > > + } else { > > > > + ds_put_format(&snat_action, "%s.src=%s; next;", > > > > + ctx->lb_vip->address_family == AF_INET6 ? > > > > + "ip6" : "ip4", > > > > + ctx->lb_vip->vip_str); > > > > + } > > > > + ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_SNAT, 160, > > > > + ds_cstr(ctx->undnat_match), > > > > + ds_cstr(&snat_action), > > > &ctx->lb->nlb->header_, > > > > + lflow_ref); > > > > + } > > > > + > > > > ds_truncate(ctx->undnat_match, undnat_match_len); > > > > + ds_destroy(&undnat_action); > > > > + ds_destroy(&snat_action); > > > > } > > > > > > > > static void > > > > @@ -12022,6 +12077,8 @@ build_lrouter_nat_flows_for_lb( > > > > * lflow generation for them. > > > > */ > > > > size_t index; > > > > + bool use_stateless_nat = smap_get_bool(&lb->nlb->options, > > > > + "use_stateless_nat", false); > > > > BITMAP_FOR_EACH_1 (index, bitmap_len, lb_dps->nb_lr_map) { > > > > struct ovn_datapath *od = lr_datapaths->array[index]; > > > > enum lrouter_nat_lb_flow_type type; > > > > @@ -12043,8 +12100,17 @@ build_lrouter_nat_flows_for_lb( > > > > if (!od->n_l3dgw_ports) { > > > > bitmap_set1(gw_dp_bitmap[type], index); > > > > } else { > > > > - build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > > > - lb_dps->lflow_ref); > > > > + /* Create stateless LB NAT rules when using multiple DGPs > > > and > > > > + * use_stateless_nat is true. > > > > + */ > > > > + bool stateless_nat = (od->n_l3dgw_ports > 1) > > > > + ? use_stateless_nat : false; > > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > > > > + struct ovn_port *dgp = od->l3dgw_ports[i]; > > > > + build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > > > + lb_dps->lflow_ref, > > > dgp, > > > > + stateless_nat); > > > > + } > > > > } > > > > > > > > if (lb->affinity_timeout) { > > > > diff --git a/ovn-nb.xml b/ovn-nb.xml > > > > index 2836f58f5..ad03c6214 100644 > > > > --- a/ovn-nb.xml > > > > +++ b/ovn-nb.xml > > > > @@ -2302,6 +2302,16 @@ or > > > > local anymore by the ovn-controller. This option is set to > > > > <code>false</code> by default. > > > > </column> > > > > + > > > > + <column name="options" key="use_stateless_nat" > > > > + type='{"type": "boolean"}'> > > > > + If the load balancer is configured with > > > <code>use_stateless_nat</code> > > > > + option to <code>true</code>, the logical router that references > > > this > > > > + load balancer will use Stateless NAT rules when the logical > > > router > > > > + has multiple distributed gateway ports(DGP). Otherwise, the > > > outbound > > > > + traffic may be dropped in scenarios where we have different > > > chassis > > > > + for each DGP. This option is set to <code>false</code> by > > > default. > > > > + </column> > > > > </group> > > > > </table> > > > > > > > > diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at > > > > index 757917626..2f69433fc 100644 > > > > --- a/tests/multinode-macros.at > > > > +++ b/tests/multinode-macros.at > > > > @@ -40,6 +40,27 @@ m4_define([M_START_TCPDUMP], > > > > ] > > > > ) > > > > > > > > +# M_EXEC([fake_node], [command]) > > > > +# > > > > +# Execute 'command' in 'fakenode' > > > > +m4_define([M_EXEC], > > > > + [podman exec $1 $2]) > > > > + > > > > +# M_CHECK_EXEC([fake_node], [command], other_params...) > > > > +# > > > > +# Wrapper for AT_CHECK that executes 'command' inside 'fake_node''s'. > > > > +# 'other_params' as passed as they are to AT_CHECK. > > > > +m4_define([M_CHECK_EXEC], > > > > + [ AT_CHECK([M_EXEC([$1], [$2])], m4_shift(m4_shift(m4_shift($@)))) ] > > > > +) > > > > + > > > > +# M_FORMAT_CT([ip-addr]) > > > > +# > > > > +# Strip content from the piped input which would differ from test to > > > test > > > > +# and limit the output to the rows containing 'ip-addr'. > > > > +# > > > > +m4_define([M_FORMAT_CT], > > > > + [[grep -F "dst=$1," | sed -e 's/id=[0-9]*/id=<cleared>/g' -e > > > 's/state=[0-9_A-Z]*/state=<cleared>/g' | sort | uniq | sed -e > > > 's/zone=[[0-9]]*/zone=<cleared>/' -e 's/mark=[[0-9]]*/mark=<cleared>/' ]]) > > > > > > > > OVS_START_SHELL_HELPERS > > > > > > > > @@ -76,6 +97,25 @@ multinode_nbctl () { > > > > m_as ovn-central ovn-nbctl "$@" > > > > } > > > > > > > > +check_fake_multinode_setup_by_nodes() { > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > > + for c in $1 > > > > + do > > > > + AT_CHECK([m_as $c ovn-appctl -t ovn-controller version], [0], > > > [ignore]) > > > > + done > > > > +} > > > > + > > > > +cleanup_multinode_resources_by_nodes() { > > > > + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db > > > > + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > > + for c in $1 > > > > + do > > > > + m_as $c ovs-vsctl del-br br-int > > > > + m_as $c ip --all netns delete > > > > + done > > > > +} > > > > + > > > > # m_count_rows TABLE [CONDITION...] > > > > # > > > > # Prints the number of rows in TABLE (that satisfy CONDITION). > > > > diff --git a/tests/multinode.at b/tests/multinode.at > > > > index a0eb8fc67..b1beb4d97 100644 > > > > --- a/tests/multinode.at > > > > +++ b/tests/multinode.at > > > > @@ -1591,3 +1591,559 @@ AT_CHECK([cat ch1_eth2.tcpdump], [0], [dnl > > > > ]) > > > > > > > > AT_CLEANUP > > > > + > > > > +AT_SETUP([ovn multinode load-balancer with multiple DGPs and multiple > > > chassis]) > > > > + > > > > +# Check that ovn-fake-multinode setup is up and running - requires > > > additional nodes > > > > +check_fake_multinode_setup_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > > + > > > > +# Delete the multinode NB and OVS resources before starting the test. > > > > +cleanup_multinode_resources_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > > + > > > > +# Network topology > > > > +# > > > > +# publicp1 (ovn-chassis-3) (20.0.0.3/24) > > > > +# | > > > > +# overlay > > > > +# | > > > > +# DGP public1 (ovn-gw-1) (20.0.0.1/24) > > > > +# | > > > > +# | > > > > +# | > > > > +# lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) 10.0.0.3/24 > > > > +# | | > > > > +# | + --- sw0p2 (ovn-chassis-2) 10.0.0.4/24 > > > > +# | > > > > +# DGP public2 (ovn-gw-2) (30.0.0.1/24) > > > > +# | > > > > +# overlay > > > > +# | > > > > +# publicp2 (ovn-chassis-4) (30.0.0.3/24) > > > > + > > > > +# Delete already used ovs-ports > > > > +m_as ovn-chassis-1 ovs-vsctl del-port br-int sw0p1-p > > > > +m_as ovn-chassis-2 ovs-vsctl del-port br-int sw0p2-p > > > > +m_as ovn-chassis-1 ip link del sw0p1-p > > > > +m_as ovn-chassis-2 ip link del sw0p2-p > > > > +m_as ovn-chassis-3 ovs-vsctl del-port br-int publicp1-p > > > > +m_as ovn-chassis-4 ovs-vsctl del-port br-int publicp2-p > > > > +m_as ovn-chassis-3 ip link del publicp1-p > > > > +m_as ovn-chassis-4 ip link del publicp2-p > > > > + > > > > +# Create East-West switch for LB backends > > > > +check multinode_nbctl ls-add sw0 > > > > +check multinode_nbctl lsp-add sw0 sw0-port1 > > > > +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 > > > 10.0.0.3 1000::3" > > > > +check multinode_nbctl lsp-add sw0 sw0-port2 > > > > +check multinode_nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 > > > 10.0.0.4 1000::4" > > > > + > > > > +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 > > > 50:54:00:00:00:03 1400 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a > > > > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 > > > 50:54:00:00:00:04 1400 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a > > > > + > > > > +m_wait_for_ports_up > > > > + > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > > > 10.0.0.4 | FORMAT_PING], \ > > > > +[0], [dnl > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > +]) > > > > + > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 > > > 10.0.0.3 | FORMAT_PING], \ > > > > +[0], [dnl > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > +]) > > > > + > > > > +# Create a logical router and attach to sw0 > > > > +check multinode_nbctl lr-add lr0 > > > > +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 > > > 1000::a/64 > > > > +check multinode_nbctl lsp-add sw0 sw0-lr0 > > > > +check multinode_nbctl lsp-set-type sw0-lr0 router > > > > +check multinode_nbctl lsp-set-addresses sw0-lr0 router > > > > +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 > > > > + > > > > +# create external connection for N/S traffic using multiple DGPs > > > > +check multinode_nbctl ls-add public > > > > + > > > > +# DGP public1 > > > > +check multinode_nbctl lsp-add public ln-public-1 > > > > +check multinode_nbctl lsp-set-type ln-public-1 localnet > > > > +check multinode_nbctl lsp-set-addresses ln-public-1 unknown > > > > +check multinode_nbctl lsp-set-options ln-public-1 network_name=public1 > > > > + > > > > +# DGP public2 > > > > +# create exteranl connection for N/S traffic > > > > +check multinode_nbctl lsp-add public ln-public-2 > > > > +check multinode_nbctl lsp-set-type ln-public-2 localnet > > > > +check multinode_nbctl lsp-set-addresses ln-public-2 unknown > > > > +check multinode_nbctl lsp-set-options ln-public-2 network_name=public2 > > > > + > > > > +# Attach DGP public1 to GW-1 and chassis-3 (overlay connectivity) > > > > +m_as ovn-gw-1 ovs-vsctl set open . > > > external-ids:ovn-bridge-mappings=public1:br-ex > > > > +m_as ovn-chassis-3 ovs-vsctl set open . > > > external-ids:ovn-bridge-mappings=public1:br-ex > > > > + > > > > +# Attach DGP public2 to GW-2 and chassis-4 (overlay connectivity) > > > > +m_as ovn-gw-2 ovs-vsctl set open . > > > external-ids:ovn-bridge-mappings=public2:br-ex > > > > +m_as ovn-chassis-4 ovs-vsctl set open . > > > external-ids:ovn-bridge-mappings=public2:br-ex > > > > + > > > > +# Create the external LR0 port to the DGP public1 > > > > +check multinode_nbctl lsp-add public public-port1 > > > > +check multinode_nbctl lsp-set-addresses public-port1 "40:54:00:00:00:03 > > > 20.0.0.3 2000::3" > > > > + > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p1 00:00:00:00:ff:02 > > > 20.0.0.1/24 2000::a/64 > > > > +check multinode_nbctl lsp-add public public-lr0-p1 > > > > +check multinode_nbctl lsp-set-type public-lr0-p1 router > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p1 router > > > > +check multinode_nbctl lsp-set-options public-lr0-p1 > > > router-port=lr0-public-p1 > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p1 ovn-gw-1 10 > > > > + > > > > +# Create a VM on ovn-chassis-3 in the same public1 overlay > > > > +m_as ovn-chassis-3 /data/create_fake_vm.sh public-port1 publicp1 > > > 40:54:00:00:00:03 1400 20.0.0.3 24 20.0.0.1 2000::4/64 2000::a > > > > + > > > > +m_wait_for_ports_up public-port1 > > > > + > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [ping -q -c 3 -i 0.3 -w 2 > > > 20.0.0.1 | FORMAT_PING], \ > > > > +[0], [dnl > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > +]) > > > > + > > > > +# Create the external LR0 port to the DGP public2 > > > > +check multinode_nbctl lsp-add public public-port2 > > > > +check multinode_nbctl lsp-set-addresses public-port2 "60:54:00:00:00:03 > > > 30.0.0.3 3000::3" > > > > + > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p2 00:00:00:00:ff:03 > > > 30.0.0.1/24 3000::a/64 > > > > +check multinode_nbctl lsp-add public public-lr0-p2 > > > > +check multinode_nbctl lsp-set-type public-lr0-p2 router > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p2 router > > > > +check multinode_nbctl lsp-set-options public-lr0-p2 > > > router-port=lr0-public-p2 > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p2 ovn-gw-2 10 > > > > + > > > > +# Create a VM on ovn-chassis-4 in the same public2 overlay > > > > +m_as ovn-chassis-4 /data/create_fake_vm.sh public-port2 publicp2 > > > 60:54:00:00:00:03 1400 30.0.0.3 24 30.0.0.1 3000::4/64 3000::a > > > > + > > > > +m_wait_for_ports_up public-port2 > > > > + > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [ping -q -c 3 -i 0.3 -w 2 > > > 30.0.0.1 | FORMAT_PING], \ > > > > +[0], [dnl > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > +]) > > > > + > > > > +# Add a default route for multiple DGPs - using ECMP > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 20.0.0.3 > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 30.0.0.3 > > > > + > > > > +# Add SNAT rules using gateway-port > > > > +check multinode_nbctl --gateway-port lr0-public-p1 lr-nat-add lr0 snat > > > 20.0.0.1 10.0.0.0/24 > > > > +check multinode_nbctl --gateway-port lr0-public-p2 lr-nat-add lr0 snat > > > 30.0.0.1 10.0.0.0/24 > > > > + > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > > > 20.0.0.3 | FORMAT_PING], \ > > > > +[0], [dnl > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > +]) > > > > + > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 > > > 30.0.0.3 | FORMAT_PING], \ > > > > +[0], [dnl > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > +]) > > > > + > > > > +# create LB > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:80" "10.0.0.3:80, > > > 10.0.0.4:80" > > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > > + > > > > +# Set use_stateless_nat to true > > > > +check multinode_nbctl set load_balancer lb0 > > > options:use_stateless_nat=true > > > > + > > > > +# Start backend http services > > > > +M_NS_DAEMONIZE([ovn-chassis-1], [sw0p1], [$PYTHON -m http.server --bind > > > 10.0.0.3 80 >/dev/null 2>&1], [http1.pid]) > > > > +M_NS_DAEMONIZE([ovn-chassis-2], [sw0p2], [$PYTHON -m http.server --bind > > > 10.0.0.4 80 >/dev/null 2>&1], [http2.pid]) > > > > + > > > > +# wait for http server be ready > > > > +sleep 2 > > > > + > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v 172.16.0.100:80 > > > --retry 3 --max-time 1 --local-port 59002 2> curl.out']) > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [sh -c 'cat curl.out | > > > grep -i -e connect | grep -v 'Server:''], \ > > > > +[0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +* Closing connection > > > > +]) > > > > + > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v 172.16.0.100:80 > > > --retry 3 --max-time 1 --local-port 59003 2> curl.out']) > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [sh -c 'cat curl.out | > > > grep -i -e connect | grep -v 'Server:''], \ > > > > +[0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +* Closing connection > > > > +]) > > > > + > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v 172.16.0.100:80 > > > --retry 3 --max-time 1 --local-port 59001']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | > > > M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > > [dnl > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v 172.16.0.100:80 > > > --retry 3 --max-time 1 --local-port 59000']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | > > > M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > > [dnl > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# create a big file on web servers for download > > > > +M_NS_EXEC([ovn-chassis-1], [sw0p1], [dd bs=512 count=200000 > > > if=/dev/urandom of=download_file]) > > > > +M_NS_EXEC([ovn-chassis-2], [sw0p2], [dd bs=512 count=200000 > > > if=/dev/urandom of=download_file]) > > > > + > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59004 > > > 2>curl.out']) > > > > + > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > + > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > > > curl.out | \ > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +* Closing connection > > > > +]) > > > > + > > > > +# Check if we have only one backend for the same connection - orig + > > > dest ports > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > > [dnl > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Check if gw-2 is empty to ensure that the traffic only come from/to > > > the originator chassis via DGP public1 > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > > +0 > > > > +]) > > > > + > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > > + > > > > +if [[ $backend_check -gt 0 ]]; then > > > > +# Backend resides on ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59004,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +else > > > > +# Backend resides on ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59004,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +fi > > > > + > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +# Check the flows again for a new source port > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59005 > > > 2>curl.out']) > > > > + > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > + > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > > > curl.out | \ > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +* Closing connection > > > > +]) > > > > + > > > > +# Check if we have only one backend for the same connection - orig + > > > dest ports > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > > [dnl > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Check if gw-2 is empty to ensure that the traffic only come from/to > > > the originator chassis via DGP public1 > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > > +0 > > > > +]) > > > > + > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > > + > > > > +if [[ $backend_check -gt 0 ]]; then > > > > +# Backend resides on ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59005,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +else > > > > +# Backend resides on ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59005,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +fi > > > > + > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +# Start a new test using the second DGP as origin (public2) > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59006 > > > 2>curl.out']) > > > > + > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > + > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > > > curl.out | \ > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +* Closing connection > > > > +]) > > > > + > > > > +# Check if we have only one backend for the same connection - orig + > > > dest ports > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > > [dnl > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Check if gw-1 is empty to ensure that the traffic only come from/to > > > the originator chassis via DGP public2 > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > > +0 > > > > +]) > > > > + > > > > +# Check the backend IP from ct entries on gw-2 (DGP public2) > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > > + > > > > +if [[ $backend_check -gt 0 ]]; then > > > > +# Backend resides on ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59006,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +else > > > > +# Backend resides on ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59006,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +fi > > > > + > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +# Check the flows again for a new source port using the second DGP as > > > origin (public2) > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59007 > > > 2>curl.out']) > > > > + > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > + > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > > > curl.out | \ > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +* Closing connection > > > > +]) > > > > + > > > > +# Check if we have only one backend for the same connection - orig + > > > dest ports > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > > [dnl > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Check if gw-1 is empty to ensure that the traffic only come from/to > > > the originator chassis via DGP public2 > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > > +0 > > > > +]) > > > > + > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > > + > > > > +if [[ $backend_check -gt 0 ]]; then > > > > +# Backend resides on ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59007,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +else > > > > +# Backend resides on ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59007,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +fi > > > > + > > > > +# Check multiple requests coming from DGP's public1 and public2 > > > > + > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > > -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +200 OK > > > > +* Closing connection > > > > +]) > > > > + > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > > -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +200 OK > > > > +* Closing connection > > > > +]) > > > > + > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > > -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +200 OK > > > > +* Closing connection > > > > +]) > > > > + > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > > -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +200 OK > > > > +* Closing connection > > > > +]) > > > > + > > > > +# Remove the LB and change the VIP port - different from the backend > > > ports > > > > +check multinode_nbctl lb-del lb0 > > > > + > > > > +# create LB again > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:9000" "10.0.0.3:80, > > > 10.0.0.4:80" > > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > > + > > > > +# Set use_stateless_nat to true > > > > +check multinode_nbctl set load_balancer lb0 > > > options:use_stateless_nat=true > > > > + > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +# Check end-to-end request using a new port for VIP > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port 59008 > > > 2>curl.out']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | > > > M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > > [dnl > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > > -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > > +200 OK > > > > +* Closing connection > > > > +]) > > > > + > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +# Check end-to-end request using a new port for VIP > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port 59008 > > > 2>curl.out']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | > > > M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > > [dnl > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > > -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > > +200 OK > > > > +* Closing connection > > > > +]) > > > > + > > > > +AT_CLEANUP > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > > > > index dcc3dbbc3..9e7a2f225 100644 > > > > --- a/tests/ovn-northd.at > > > > +++ b/tests/ovn-northd.at > > > > @@ -13864,3 +13864,323 @@ check_no_redirect > > > > > > > > AT_CLEANUP > > > > ]) > > > > + > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + NAT > > > Stateless)]) > > > > +ovn_start > > > > + > > > > +check ovn-nbctl ls-add public > > > > +check ovn-nbctl lr-add lr1 > > > > + > > > > +# lr1 DGP ts1 > > > > +check ovn-nbctl ls-add ts1 > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 172.16.10.1/24 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > > + > > > > +# lr1 DGP ts2 > > > > +check ovn-nbctl ls-add ts2 > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 172.16.20.1/24 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > > + > > > > +# lr1 DGP public > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 173.16.0.1/16 > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > + > > > > +check ovn-nbctl ls-add s1 > > > > +# s1 - lr1 > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 172.16.0.1" > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > + > > > > +# s1 - backend vm1 > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 172.16.0.101" > > > > + > > > > +# s1 - backend vm2 > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 172.16.0.102" > > > > + > > > > +# s1 - backend vm3 > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 172.16.0.103" > > > > + > > > > +# Add the lr1 DGP ts1 to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > > nat-addresses=router > > > > + > > > > +# Add the lr1 DGP ts2 to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > > nat-addresses=router > > > > + > > > > +# Add the lr1 DGP public to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > > nat-addresses=router > > > > + > > > > +# Create the Load Balancer lb1 > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > > + > > > > +# Set use_stateless_nat to true > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > > options:use_stateless_nat=true > > > > + > > > > +# Associate load balancer to s1 > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > +check ovn-nbctl --wait=sb sync > > > > + > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > +AT_CAPTURE_FILE([s1flows]) > > > > + > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep > > > "30.0.0.1"], [0], [dnl > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > > +]) > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > "30.0.0.1"], [0], [dnl > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > > ip4.dst == 30.0.0.1), > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > +]) > > > > + > > > > +# Associate load balancer to lr1 with DGP > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > +check ovn-nbctl --wait=sb sync > > > > + > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > + > > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > "30.0.0.1"], [0], [dnl > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > is_chassis_resident("cr-lr1-ts1")), > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > is_chassis_resident("cr-lr1-ts2")), > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > is_chassis_resident("cr-lr1_public")), > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > +]) > > > > + > > > > +# 2. Check if the DGP ports are in the match with action next > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], [dnl > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > > action=(next;) > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") && > > > is_chassis_resident("cr-lr1_public") && tcp), action=(next;) > > > > +]) > > > > + > > > > +# 3. Check if the VIP IP is in the ipX.src action > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > > action=(next;) > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > > action=(next;) > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(ip4.src=30.0.0.1; next;) > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(ip4.src=30.0.0.1; next;) > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") && > > > is_chassis_resident("cr-lr1_public") && tcp), action=(ip4.src=30.0.0.1; > > > next;) > > > > +]) > > > > + > > > > +AT_CLEANUP > > > > +]) > > > > + > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + NAT > > > Stateless) - IPv6]) > > > > +ovn_start > > > > + > > > > +check ovn-nbctl ls-add public > > > > +check ovn-nbctl lr-add lr1 > > > > + > > > > +# lr1 DGP ts1 > > > > +check ovn-nbctl ls-add ts1 > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > > > 2001:db8:aaaa:1::1/64 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > > + > > > > +# lr1 DGP ts2 > > > > +check ovn-nbctl ls-add ts2 > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > > > 2001:db8:aaaa:2::1/64 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > > + > > > > +# lr1 DGP public > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > > > 2001:db8:bbbb::1/64 > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > > > 2001:db8:aaaa:3::1/64 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > + > > > > +check ovn-nbctl ls-add s1 > > > > +# s1 - lr1 > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > > > 2001:db8:aaaa:3::1" > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > + > > > > +# s1 - backend vm1 > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > > > 2001:db8:aaaa:3::101" > > > > + > > > > +# s1 - backend vm2 > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > > > 2001:db8:aaaa:3::102" > > > > + > > > > +# s1 - backend vm3 > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > > > 2001:db8:aaaa:3::103" > > > > + > > > > +# Add the lr1 DGP ts1 to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > > nat-addresses=router > > > > + > > > > +# Add the lr1 DGP ts2 to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > > nat-addresses=router > > > > + > > > > +# Add the lr1 DGP public to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > > nat-addresses=router > > > > + > > > > +# Create the Load Balancer lb1 > > > > +check ovn-nbctl --wait=sb lb-add lb1 "2001:db8:cccc::1" > > > "2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101" > > > > + > > > > +# Set use_stateless_nat to true > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > > options:use_stateless_nat=true > > > > + > > > > +# Associate load balancer to s1 > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > +check ovn-nbctl --wait=sb sync > > > > + > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > +AT_CAPTURE_FILE([s1flows]) > > > > + > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep > > > "2001:db8:cccc::1"], [0], [dnl > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 > > > && ip6.dst == 2001:db8:cccc::1), action=(xxreg1 = 2001:db8:cccc::1; > > > ct_lb_mark;) > > > > +]) > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > "2001:db8:cccc::1"], [0], [dnl > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > > ip6.dst == 2001:db8:cccc::1), > > > action=(ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > +]) > > > > + > > > > +# Associate load balancer to lr1 with DGP > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > +check ovn-nbctl --wait=sb sync > > > > + > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > + > > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > "2001:db8:cccc::1"], [0], [dnl > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > is_chassis_resident("cr-lr1-ts1")), > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > is_chassis_resident("cr-lr1-ts2")), > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > is_chassis_resident("cr-lr1_public")), > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > +]) > > > > + > > > > +# 2. Check if the DGP ports are in the match with action next > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], [dnl > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > > action=(next;) > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport == > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport == > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || outport == > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > > action=(next;) > > > > +]) > > > > + > > > > +# 3. Check if the VIP IP is in the ipX.src action > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > > action=(next;) > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > > action=(next;) > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport == > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport == > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || outport == > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > +]) > > > > + > > > > +AT_CLEANUP > > > > +]) > > > > + > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (DGP)]) > > > > +ovn_start > > > > + > > > > +check ovn-nbctl ls-add public > > > > +check ovn-nbctl lr-add lr1 > > > > + > > > > +# lr1 DGP ts1 > > > > +check ovn-nbctl ls-add ts1 > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 172.16.10.1/24 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-1 > > > > + > > > > +# lr1 DGP ts2 > > > > +check ovn-nbctl ls-add ts2 > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 172.16.20.1/24 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-1 > > > > + > > > > +# lr1 DGP public > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 173.16.0.1/16 > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > + > > > > +check ovn-nbctl ls-add s1 > > > > +# s1 - lr1 > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 172.16.0.1" > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > + > > > > +# s1 - backend vm1 > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 172.16.0.101" > > > > + > > > > +# s1 - backend vm2 > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 172.16.0.102" > > > > + > > > > +# s1 - backend vm3 > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 172.16.0.103" > > > > + > > > > +# Add the lr1 DGP ts1 to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > > nat-addresses=router > > > > + > > > > +# Add the lr1 DGP ts2 to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > > nat-addresses=router > > > > + > > > > +# Add the lr1 DGP public to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > > nat-addresses=router > > > > + > > > > +# Create the Load Balancer lb1 > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > > + > > > > +# Associate load balancer to s1 > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > +check ovn-nbctl --wait=sb sync > > > > + > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > +AT_CAPTURE_FILE([s1flows]) > > > > + > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep > > > "30.0.0.1"], [0], [dnl > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > > +]) > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > "30.0.0.1"], [0], [dnl > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > > ip4.dst == 30.0.0.1), > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > +]) > > > > + > > > > +# Associate load balancer to lr1 with DGP > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > +check ovn-nbctl --wait=sb sync > > > > + > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > + > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > "30.0.0.1"], [0], [dnl > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > is_chassis_resident("cr-lr1-ts1")), > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > is_chassis_resident("cr-lr1-ts2")), > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > is_chassis_resident("cr-lr1_public")), > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > +]) > > > > + > > > > +AT_CLEANUP > > > > +]) > > > > -- > > > > 2.34.1 > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > _'Esta mensagem é direcionada apenas para os endereços constantes no > > > > cabeçalho inicial. Se você não está listado nos endereços constantes no > > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas > > > estão > > > > imediatamente anuladas e proibidas'._ > > > > > > > > > > > > * **'Apesar do Magazine Luiza tomar > > > > todas as precauções razoáveis para assegurar que nenhum vírus esteja > > > > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade > > > por > > > > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos'.* > > > > > > > > > > > > > > > > _______________________________________________ > > > > dev mailing list > > > > dev@openvswitch.org > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > > > > > > -- > > > > > > > > > > _‘Esta mensagem é direcionada apenas para os endereços constantes no > > cabeçalho inicial. Se você não está listado nos endereços constantes no > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão > > imediatamente anuladas e proibidas’._ > > > > > > * **‘Apesar do Magazine Luiza tomar > > todas as precauções razoáveis para assegurar que nenhum vírus esteja > > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por > > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* > > > > > > > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Hi Numan, Em qui., 26 de set. de 2024 às 13:58, Numan Siddique <numans@ovn.org> escreveu: > On Thu, Sep 26, 2024 at 12:56 PM Numan Siddique <numans@ovn.org> wrote: > > > > On Thu, Sep 26, 2024 at 10:55 AM Roberto Bartzen Acosta via dev > > <ovs-dev@openvswitch.org> wrote: > > > > > > Hi Numan, > > > > > > Thanks for your feedback and review. > > > > > > Em qua., 25 de set. de 2024 às 19:50, Numan Siddique <numans@ovn.org> > > > escreveu: > > > > > > > On Thu, Sep 19, 2024 at 6:12 PM Roberto Bartzen Acosta via dev > > > > <ovs-dev@openvswitch.org> wrote: > > > > > > > > > > This commit fixes the build_distr_lrouter_nat_flows_for_lb > function to > > > > > include a DNAT flow entry for each DGP in use. Since we have added > > > > support > > > > > to create multiple gateway ports per logical router, it's > necessary to > > > > > include in the LR NAT rules pipeline a specific entry for each > attached > > > > DGP. > > > > > Otherwise, the inbound traffic will only be redirected when the > incoming > > > > LRP > > > > > matches the chassis_resident field. > > > > > > > > > > Additionally, this patch includes the ability to use load-balancer > with > > > > DGPs > > > > > attached to multiple chassis. We can have each of the DGPs > associated > > > > with a > > > > > different chassis, and in this case the DNAT rules added by > default will > > > > not > > > > > be enough to guarantee outgoing traffic. > > > > > > > > > > To solve the multiple chassis for DGPs problem, this patch include > a new > > > > > config options to be configured in the load-balancer. If the > > > > use_stateless_nat > > > > > is set to true, the logical router that references this > load-balancer > > > > will use > > > > > Stateless NAT rules when the logical router has multiple DGPs. > After > > > > applying > > > > > this patch and setting the use_stateless_nat option, the inbound > and/or > > > > > outbound traffic can pass through any chassis where the DGP resides > > > > without > > > > > having problems with CT state. > > > > > > > > > > Reported-at: > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2054322 > > > > > Fixes: 15348b7b806f ("ovn-northd: Multiple distributed gateway port > > > > support.") > > > > > > > > > > Signed-off-by: Roberto Bartzen Acosta < > roberto.acosta@luizalabs.com> > > > > > > > > Hi Roberto, > > > > > > > > Thanks for the patch. I tested this patch using the test example in > > > > multinode.at. > > > > > > > > The test case adds the below load balancer > > > > > > > > [root@ovn-central ~]# ovn-nbctl lb-list > > > > UUID LB PROTO > > > > VIP IPs > > > > f3e29869-3bb5-4df0-960a-171106f5913a lb0 tcp > > > > 172.16.0.100:9000 10.0.0.3:80,10.0.0.4:80 > > > > > > > > And the below logical flows are generated by this patch > > > > > > > > -------- > > > > [root@ovn-central ~]# ovn-sbctl dump-flows lr0 | grep 172.16.0.100 > > > > table=6 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst > > > > == 172.16.0.100), action=(ct_dnat;) > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 > && > > > > is_chassis_resident("cr-lr0-public-p1")), > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > 10.0.0.3:80, > > > > 10.0.0.4:80);) > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 > && > > > > is_chassis_resident("cr-lr0-public-p2")), > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > 10.0.0.3:80, > > > > 10.0.0.4:80);) > > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > > > tcp.src == 80)) && (inport == "lr0-public-p1" || outport == > > > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), > > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > > > tcp.src == 80)) && (inport == "lr0-public-p2" || outport == > > > > "lr0-public-p2") && is_chassis_resident("cr-lr0-public-p2") && tcp), > > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > > > -------------- > > > > > > > > > > > > I fail to understand the reason for modifying the ip4.dst before > > > > calling ct_lb_mark. Can you please explain why ? Because the > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;) will first modify the > > > > ip4.dst to 10.0.0.3 and > > > > then to 10.0.0.4 and then the ct_lb_mark will actually do the > > > > conntrack with NAT to either 10.0.0.3 or 10.0.0.4. > > > > > > > > Is it because you want the conntrack entry to not have 172.16.0.100 ? > > > > > > > > > > The only reason I included this ip4.dst action in the DNAT rule is > because > > > it's required to accept packets coming from a chassis that doesn't have > > > previously created conntrack entries. The main feature introduced in > this > > > patch is to allow the administrator to have multiple DGPs attached to > > > different chassis (is_chassis_resident...). So, my implementation was > based > > > on the normal behavior when using stateless NAT for external addresses, > > > where we need to add the ipx.dst in lr_in_dnat for traffic to be > received > > > on the chassis (put the DGP port as chassis_resident match, as is the > case > > > with stateless NAT [1] with DGP[2]). > > > > > > The question is, if we only have the ct_lb_mark, packets that pass > through > > > the chassis and are already part of an active flow in another chassis > (same > > > IPs and Ports) will be dropped because there is no correspondence in > the > > > backend. So only packets with the NEW flag will be accepted and sent > to the > > > backend (at least for TCP traffic). If we only have the ip4.dst action, > > > this will always perform the dnat for the same backend, without > balancing. > > > Therefore, the combination of the two actions allows the packet to > always > > > be received (regardless of whether conntrack is active for it), and > > > ct_lb_mark will take care of balancing for different backends. > > > > > > If we had conntrack sync between different chassis this would not be > > > necessary, as the ct_lb_mark action could always be executed without > > > dropping packets due to lack of correspondence in the conntrack table. > > > > I still need to understand it properly. I'll get back to you after > > reading your reply thoroughly. > > > > But what's the point of doing multiple ip4.dst modifications for the > > same packet ? > > > > i.e - action=(ip4.dst = 10.0.0.3; ip4.dst = 10.0.0.4.....) > Oh, sure. This is more of a design question when creating the flow. I added the list of all backends to keep the logic of the backends added in ct_lb_mark, but in practice, we could just have the first one in the list and it would be enough. What do you think? a list to be equal to ct_lb_mark or the first element of the backends list? > > > > The 2nd ip4.dst action overwrites and the packet's ip4.dst will be > > 10.0.0.4 before ct_lb_mark. > > > > > > > > > > [1] > > > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15737 > > > [2] > > > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15726 > > > > > > > > > > > > > > Also I don't understand why this patch adds the logical flows in > > > > "lr_out_snat" stage ? > > > > > > > > > > The flow for lr_out_snat is necessary for the correct functioning of > > > stateless NAT for the same reason explained previously. I mean, if the > > > outgoing packet is redirected to a chassis that doesn't have an active > > > conntrack entry, it will not be NATed by ct_lb action because it > doesn't > > > refer to a valid flow (use case with ecmp). > > > > > > So it is necessary to create a stateless SNAT rule (similar to this > [3]) > > > with a lower priority than the other router pipeline entries, in this > case, > > > if the packet is not SNATed by ct_lb (conntrack missed) it will be > SNATed > > > by stateless NAT rule. > > > > > > [3] > > > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15884 > > > > > > > > > > > > > > > > > Using the system multinode test as an example, the below fails > > > > (which is a regression) > > > > > > > > --- > > > > root@ovn-chassis-3 ~]# ip netns exec publicp1 nc -vz 10.0.0.3 80 > > > > ---- > > > > > > > > In the above test, publicp1 with IP 20.0.0.3 when it tries to > connect > > > > to one if the backends directly (without the LB VIP), it fails. > > > > It fails because of the logical flows in "lr_out_snat". > > > > > > > > > > > > Looks to me the solution proposed here is incomplete. > > > > > > > > Also please note that in our CI we run the multinode tests > > > > periodically once a day using the v0.1 of the ovn-fake-multinode > > > > and the tests you added will fail. This needs to be fixed and until > > > > we move to the latest version of ovn-fake-multinode. > > > > > > > > > > I imagine that the test you are doing is using the same port as the LB > > > backend (TCP 80 in this case). So, the stateless lr_out_snat flow will > > > force the output to be SNATed because this port is in use by the > backend. > > > Traffic to/from other ports will work without problems and will follow > the > > > normal programmed flows (e.g. ICMP). > > > > > > This is necessary to ensure the egress traffic because the DGPs are > > > distributed across multiple chassis. Also, this setup is being > validated in > > > the test ovn-fake-multinode testcase (ICMP from the backends chassis > use > > > the router's default SNAT and not the LB's). I didn't understand the > > > regression you mentioned because this was programmed to be stateless > and > > > it's traffic that uses the same ports as the LB backend, could you > explain > > > better? > > > > > > > To reproduce the issue I mentioned, first run the multinode test you > > have added. > > Don't destroy the resources. And then login to ovn-chassis-3 container > and run > > the command > > > > #ip netns exec publicp1 curl -v 10.0.0.3:80 > > > > So instead of curling to the LB VIP (which is 172.16.0.100:80), curl > > directly to the backends. Without your patch it works > > and with your patch it doesn't. > > > > If you run tcpdump you'd see the below > > > > 16:51:24.526980 40:54:00:00:00:03 > 00:00:00:00:ff:02, ethertype IPv4 > > (0x0800), length 74: (tos 0x0, ttl 64, id 44276, offset 0, flags [DF], > > proto TCP (6), length 60) > > 20.0.0.3.46978 > 10.0.0.3.http: Flags [S], cksum 0x1e34 (incorrect > > -> 0x4b96), seq 1989771329, win 65280, options [mss 1360,sackOK,TS val > > 1753008285 ecr 0,nop,wscale 7], length 0 > > 16:51:24.528559 00:00:00:00:ff:02 > 40:54:00:00:00:03, ethertype IPv4 > > (0x0800), length 74: (tos 0x0, ttl 63, id 0, offset 0, flags [DF], > > proto TCP (6), length 60) > > 172.16.0.100.http > 20.0.0.3.46978: Flags [S.], cksum 0xc0a5 > > (incorrect -> 0x2094), seq 1088931300, ack 1989771330, win 64704, > > options [mss 1360,sackOK,TS val 3729696164 ecr 1753008285,nop,wscale > > 7], length 0 > > 16:51:24.528571 40:54:00:00:00:03 > 00:00:00:00:ff:02, ethertype IPv4 > > (0x0800), length 54: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], > > proto TCP (6), length 40) > > > > The request is sent to 10.0.0.3 but the reply is received from > > 172.16.0.100 which is wrong. > > Also If I modify the LB VIP tcp port to 9000 > > [root@ovn-central ~]# ovn-nbctl lb-list > UUID LB PROTO > VIP IPs > f3e29869-3bb5-4df0-960a-171106f5913a lb0 tcp > 172.16.0.100:9000 10.0.0.3:80,10.0.0.4:80 > > When a request is sent to 10.0.0.3:80, the reply is received from > 172.16.0.100:9000. > I got it, thanks for clarifying this. When you use TCP port 80 on LB, you get the response with LB SNATed address (172.16.0.100:80). So, when you change LB TCP port to 9000, you get the response with the same LB SNATed logic (172.16.0.100:9000). I think the logic is correct because Stateless NAT will SNATed the outgoing whenever the corresponding backends and ports/protocols are matched. The case you are trying to validate is out of the scope of Stateless NAT, I guess. Do you have any suggestions for a different approach for the SNAT case? I don't see how to guarantee the return in cases where there is no active conntrack entry without using lr_out_snat for that. Thanks for your review, I really appreciate it. Roberto > > Thanks > Numan > > > > > > Thanks > > Numan > > > > > > > Thanks, > > > Roberto > > > > > > > > > > Thanks > > > > Numan > > > > > > > > > > > > > --- > > > > > northd/en-lr-stateful.c | 12 - > > > > > northd/northd.c | 116 ++++++-- > > > > > ovn-nb.xml | 10 + > > > > > tests/multinode-macros.at | 40 +++ > > > > > tests/multinode.at | 556 > ++++++++++++++++++++++++++++++++++++++ > > > > > tests/ovn-northd.at | 320 ++++++++++++++++++++++ > > > > > 6 files changed, 1017 insertions(+), 37 deletions(-) > > > > > > > > > > diff --git a/northd/en-lr-stateful.c b/northd/en-lr-stateful.c > > > > > index baf1bd2f8..f09691af6 100644 > > > > > --- a/northd/en-lr-stateful.c > > > > > +++ b/northd/en-lr-stateful.c > > > > > @@ -516,18 +516,6 @@ lr_stateful_record_create(struct > lr_stateful_table > > > > *table, > > > > > > > > > > table->array[od->index] = lr_stateful_rec; > > > > > > > > > > - /* Load balancers are not supported (yet) if a logical router > has > > > > multiple > > > > > - * distributed gateway port. Log a warning. */ > > > > > - if (lr_stateful_rec->has_lb_vip && > lr_has_multiple_gw_ports(od)) { > > > > > - static struct vlog_rate_limit rl = > VLOG_RATE_LIMIT_INIT(1, 1); > > > > > - VLOG_WARN_RL(&rl, "Load-balancers are configured on > logical " > > > > > - "router %s, which has %"PRIuSIZE" > distributed " > > > > > - "gateway ports. Load-balancer is not > supported " > > > > > - "yet when there is more than one distributed > " > > > > > - "gateway port on the router.", > > > > > - od->nbr->name, od->n_l3dgw_ports); > > > > > - } > > > > > - > > > > > return lr_stateful_rec; > > > > > } > > > > > > > > > > diff --git a/northd/northd.c b/northd/northd.c > > > > > index a267cd5f8..bbe97acf8 100644 > > > > > --- a/northd/northd.c > > > > > +++ b/northd/northd.c > > > > > @@ -11807,31 +11807,30 @@ static void > > > > > build_distr_lrouter_nat_flows_for_lb(struct > lrouter_nat_lb_flows_ctx > > > > *ctx, > > > > > enum > lrouter_nat_lb_flow_type type, > > > > > struct ovn_datapath *od, > > > > > - struct lflow_ref *lflow_ref) > > > > > + struct lflow_ref *lflow_ref, > > > > > + struct ovn_port *dgp, > > > > > + bool stateless_nat) > > > > > { > > > > > - struct ovn_port *dgp = od->l3dgw_ports[0]; > > > > > - > > > > > - const char *undnat_action; > > > > > - > > > > > - switch (type) { > > > > > - case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > > > - undnat_action = "flags.force_snat_for_lb = 1; next;"; > > > > > - break; > > > > > - case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > > > - undnat_action = "flags.skip_snat_for_lb = 1; next;"; > > > > > - break; > > > > > - case LROUTER_NAT_LB_FLOW_NORMAL: > > > > > - case LROUTER_NAT_LB_FLOW_MAX: > > > > > - undnat_action = lrouter_use_common_zone(od) > > > > > - ? "ct_dnat_in_czone;" > > > > > - : "ct_dnat;"; > > > > > - break; > > > > > - } > > > > > + struct ds dnat_action = DS_EMPTY_INITIALIZER; > > > > > > > > > > /* Store the match lengths, so we can reuse the ds buffer. */ > > > > > size_t new_match_len = ctx->new_match->length; > > > > > size_t undnat_match_len = ctx->undnat_match->length; > > > > > > > > > > + /* dnat_action: Add the LB backend IPs as a destination > action of > > > > the > > > > > + * lr_in_dnat NAT rule with cumulative effect > because > > > > any > > > > > + * backend dst IP used in the action list will > > > > redirect the > > > > > + * packet to the ct_lb pipeline. > > > > > + */ > > > > > + if (stateless_nat) { > > > > > + for (size_t i = 0; i < ctx->lb_vip->n_backends; i++) { > > > > > + struct ovn_lb_backend *backend = > &ctx->lb_vip->backends[i]; > > > > > + bool ipv6 = !IN6_IS_ADDR_V4MAPPED(&backend->ip); > > > > > + ds_put_format(&dnat_action, "%s.dst=%s;", ipv6 ? > "ip6" : > > > > "ip4", > > > > > + backend->ip_str); > > > > > + } > > > > > + } > > > > > + ds_put_format(&dnat_action, "%s", ctx->new_action[type]); > > > > > > > > > > const char *meter = NULL; > > > > > > > > > > @@ -11841,20 +11840,46 @@ > build_distr_lrouter_nat_flows_for_lb(struct > > > > lrouter_nat_lb_flows_ctx *ctx, > > > > > > > > > > if (ctx->lb_vip->n_backends || > !ctx->lb_vip->empty_backend_rej) { > > > > > ds_put_format(ctx->new_match, " && > is_chassis_resident(%s)", > > > > > - od->l3dgw_ports[0]->cr_port->json_key); > > > > > + dgp->cr_port->json_key); > > > > > } > > > > > > > > > > ovn_lflow_add_with_hint__(ctx->lflows, od, S_ROUTER_IN_DNAT, > > > > ctx->prio, > > > > > - ds_cstr(ctx->new_match), > > > > ctx->new_action[type], > > > > > + ds_cstr(ctx->new_match), > > > > ds_cstr(&dnat_action), > > > > > NULL, meter, &ctx->lb->nlb->header_, > > > > > lflow_ref); > > > > > > > > > > ds_truncate(ctx->new_match, new_match_len); > > > > > > > > > > + ds_destroy(&dnat_action); > > > > > if (!ctx->lb_vip->n_backends) { > > > > > return; > > > > > } > > > > > > > > > > + struct ds undnat_action = DS_EMPTY_INITIALIZER; > > > > > + struct ds snat_action = DS_EMPTY_INITIALIZER; > > > > > + > > > > > + switch (type) { > > > > > + case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > > > + ds_put_format(&undnat_action, "flags.force_snat_for_lb = > 1; > > > > next;"); > > > > > + break; > > > > > + case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > > > + ds_put_format(&undnat_action, "flags.skip_snat_for_lb = 1; > > > > next;"); > > > > > + break; > > > > > + case LROUTER_NAT_LB_FLOW_NORMAL: > > > > > + case LROUTER_NAT_LB_FLOW_MAX: > > > > > + ds_put_format(&undnat_action, "%s", > > > > > + lrouter_use_common_zone(od) ? > "ct_dnat_in_czone;" > > > > > + : "ct_dnat;"); > > > > > + break; > > > > > + } > > > > > + > > > > > + /* undnat_action: Remove the ct action from the > lr_out_undenat NAT > > > > rule. > > > > > + */ > > > > > + if (stateless_nat) { > > > > > + ds_clear(&undnat_action); > > > > > + ds_put_format(&undnat_action, "next;"); > > > > > + } > > > > > + > > > > > /* We need to centralize the LB traffic to properly perform > > > > > * the undnat stage. > > > > > */ > > > > > @@ -11873,11 +11898,41 @@ > build_distr_lrouter_nat_flows_for_lb(struct > > > > lrouter_nat_lb_flows_ctx *ctx, > > > > > ds_put_format(ctx->undnat_match, ") && (inport == %s || > outport == > > > > %s)" > > > > > " && is_chassis_resident(%s)", dgp->json_key, > > > > dgp->json_key, > > > > > dgp->cr_port->json_key); > > > > > + /* Use the LB protocol as matching criteria for out undnat > and snat > > > > when > > > > > + * creating LBs with stateless NAT. */ > > > > > + if (stateless_nat) { > > > > > + ds_put_format(ctx->undnat_match, " && %s", > ctx->lb->proto); > > > > > + } > > > > > ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_UNDNAT, > 120, > > > > > - ds_cstr(ctx->undnat_match), > undnat_action, > > > > > - &ctx->lb->nlb->header_, > > > > > + ds_cstr(ctx->undnat_match), > > > > > + ds_cstr(&undnat_action), > > > > &ctx->lb->nlb->header_, > > > > > lflow_ref); > > > > > + > > > > > + /* snat_action: Add a new lr_out_snat rule with the LB VIP as > > > > source IP > > > > > + * action to perform the NAT stateless pipeline > > > > completely. > > > > > + */ > > > > > + if (stateless_nat) { > > > > > + if (ctx->lb_vip->port_str) { > > > > > + ds_put_format(&snat_action, "%s.src=%s; %s.src=%s; > next;", > > > > > + ctx->lb_vip->address_family == AF_INET6 > ? > > > > > + "ip6" : "ip4", > > > > > + ctx->lb_vip->vip_str, ctx->lb->proto, > > > > > + ctx->lb_vip->port_str); > > > > > + } else { > > > > > + ds_put_format(&snat_action, "%s.src=%s; next;", > > > > > + ctx->lb_vip->address_family == AF_INET6 > ? > > > > > + "ip6" : "ip4", > > > > > + ctx->lb_vip->vip_str); > > > > > + } > > > > > + ovn_lflow_add_with_hint(ctx->lflows, od, > S_ROUTER_OUT_SNAT, 160, > > > > > + ds_cstr(ctx->undnat_match), > > > > > + ds_cstr(&snat_action), > > > > &ctx->lb->nlb->header_, > > > > > + lflow_ref); > > > > > + } > > > > > + > > > > > ds_truncate(ctx->undnat_match, undnat_match_len); > > > > > + ds_destroy(&undnat_action); > > > > > + ds_destroy(&snat_action); > > > > > } > > > > > > > > > > static void > > > > > @@ -12022,6 +12077,8 @@ build_lrouter_nat_flows_for_lb( > > > > > * lflow generation for them. > > > > > */ > > > > > size_t index; > > > > > + bool use_stateless_nat = smap_get_bool(&lb->nlb->options, > > > > > + "use_stateless_nat", > false); > > > > > BITMAP_FOR_EACH_1 (index, bitmap_len, lb_dps->nb_lr_map) { > > > > > struct ovn_datapath *od = lr_datapaths->array[index]; > > > > > enum lrouter_nat_lb_flow_type type; > > > > > @@ -12043,8 +12100,17 @@ build_lrouter_nat_flows_for_lb( > > > > > if (!od->n_l3dgw_ports) { > > > > > bitmap_set1(gw_dp_bitmap[type], index); > > > > > } else { > > > > > - build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > > > > - > lb_dps->lflow_ref); > > > > > + /* Create stateless LB NAT rules when using multiple > DGPs > > > > and > > > > > + * use_stateless_nat is true. > > > > > + */ > > > > > + bool stateless_nat = (od->n_l3dgw_ports > 1) > > > > > + ? use_stateless_nat : false; > > > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > > > > > + struct ovn_port *dgp = od->l3dgw_ports[i]; > > > > > + build_distr_lrouter_nat_flows_for_lb(&ctx, type, > od, > > > > > + > lb_dps->lflow_ref, > > > > dgp, > > > > > + > stateless_nat); > > > > > + } > > > > > } > > > > > > > > > > if (lb->affinity_timeout) { > > > > > diff --git a/ovn-nb.xml b/ovn-nb.xml > > > > > index 2836f58f5..ad03c6214 100644 > > > > > --- a/ovn-nb.xml > > > > > +++ b/ovn-nb.xml > > > > > @@ -2302,6 +2302,16 @@ or > > > > > local anymore by the ovn-controller. This option is set to > > > > > <code>false</code> by default. > > > > > </column> > > > > > + > > > > > + <column name="options" key="use_stateless_nat" > > > > > + type='{"type": "boolean"}'> > > > > > + If the load balancer is configured with > > > > <code>use_stateless_nat</code> > > > > > + option to <code>true</code>, the logical router that > references > > > > this > > > > > + load balancer will use Stateless NAT rules when the > logical > > > > router > > > > > + has multiple distributed gateway ports(DGP). Otherwise, > the > > > > outbound > > > > > + traffic may be dropped in scenarios where we have > different > > > > chassis > > > > > + for each DGP. This option is set to <code>false</code> by > > > > default. > > > > > + </column> > > > > > </group> > > > > > </table> > > > > > > > > > > diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at > > > > > index 757917626..2f69433fc 100644 > > > > > --- a/tests/multinode-macros.at > > > > > +++ b/tests/multinode-macros.at > > > > > @@ -40,6 +40,27 @@ m4_define([M_START_TCPDUMP], > > > > > ] > > > > > ) > > > > > > > > > > +# M_EXEC([fake_node], [command]) > > > > > +# > > > > > +# Execute 'command' in 'fakenode' > > > > > +m4_define([M_EXEC], > > > > > + [podman exec $1 $2]) > > > > > + > > > > > +# M_CHECK_EXEC([fake_node], [command], other_params...) > > > > > +# > > > > > +# Wrapper for AT_CHECK that executes 'command' inside > 'fake_node''s'. > > > > > +# 'other_params' as passed as they are to AT_CHECK. > > > > > +m4_define([M_CHECK_EXEC], > > > > > + [ AT_CHECK([M_EXEC([$1], [$2])], > m4_shift(m4_shift(m4_shift($@)))) ] > > > > > +) > > > > > + > > > > > +# M_FORMAT_CT([ip-addr]) > > > > > +# > > > > > +# Strip content from the piped input which would differ from test > to > > > > test > > > > > +# and limit the output to the rows containing 'ip-addr'. > > > > > +# > > > > > +m4_define([M_FORMAT_CT], > > > > > + [[grep -F "dst=$1," | sed -e 's/id=[0-9]*/id=<cleared>/g' -e > > > > 's/state=[0-9_A-Z]*/state=<cleared>/g' | sort | uniq | sed -e > > > > 's/zone=[[0-9]]*/zone=<cleared>/' -e > 's/mark=[[0-9]]*/mark=<cleared>/' ]]) > > > > > > > > > > OVS_START_SHELL_HELPERS > > > > > > > > > > @@ -76,6 +97,25 @@ multinode_nbctl () { > > > > > m_as ovn-central ovn-nbctl "$@" > > > > > } > > > > > > > > > > +check_fake_multinode_setup_by_nodes() { > > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > > > + for c in $1 > > > > > + do > > > > > + AT_CHECK([m_as $c ovn-appctl -t ovn-controller version], > [0], > > > > [ignore]) > > > > > + done > > > > > +} > > > > > + > > > > > +cleanup_multinode_resources_by_nodes() { > > > > > + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db > > > > > + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd > > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > > > + for c in $1 > > > > > + do > > > > > + m_as $c ovs-vsctl del-br br-int > > > > > + m_as $c ip --all netns delete > > > > > + done > > > > > +} > > > > > + > > > > > # m_count_rows TABLE [CONDITION...] > > > > > # > > > > > # Prints the number of rows in TABLE (that satisfy CONDITION). > > > > > diff --git a/tests/multinode.at b/tests/multinode.at > > > > > index a0eb8fc67..b1beb4d97 100644 > > > > > --- a/tests/multinode.at > > > > > +++ b/tests/multinode.at > > > > > @@ -1591,3 +1591,559 @@ AT_CHECK([cat ch1_eth2.tcpdump], [0], [dnl > > > > > ]) > > > > > > > > > > AT_CLEANUP > > > > > + > > > > > +AT_SETUP([ovn multinode load-balancer with multiple DGPs and > multiple > > > > chassis]) > > > > > + > > > > > +# Check that ovn-fake-multinode setup is up and running - requires > > > > additional nodes > > > > > +check_fake_multinode_setup_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > > > + > > > > > +# Delete the multinode NB and OVS resources before starting the > test. > > > > > +cleanup_multinode_resources_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > > > + > > > > > +# Network topology > > > > > +# > > > > > +# publicp1 (ovn-chassis-3) (20.0.0.3/24) > > > > > +# | > > > > > +# overlay > > > > > +# | > > > > > +# DGP public1 (ovn-gw-1) (20.0.0.1/24) > > > > > +# | > > > > > +# | > > > > > +# | > > > > > +# lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) > 10.0.0.3/24 > > > > > +# | | > > > > > +# | + --- sw0p2 (ovn-chassis-2) > 10.0.0.4/24 > > > > > +# | > > > > > +# DGP public2 (ovn-gw-2) (30.0.0.1/24) > > > > > +# | > > > > > +# overlay > > > > > +# | > > > > > +# publicp2 (ovn-chassis-4) (30.0.0.3/24) > > > > > + > > > > > +# Delete already used ovs-ports > > > > > +m_as ovn-chassis-1 ovs-vsctl del-port br-int sw0p1-p > > > > > +m_as ovn-chassis-2 ovs-vsctl del-port br-int sw0p2-p > > > > > +m_as ovn-chassis-1 ip link del sw0p1-p > > > > > +m_as ovn-chassis-2 ip link del sw0p2-p > > > > > +m_as ovn-chassis-3 ovs-vsctl del-port br-int publicp1-p > > > > > +m_as ovn-chassis-4 ovs-vsctl del-port br-int publicp2-p > > > > > +m_as ovn-chassis-3 ip link del publicp1-p > > > > > +m_as ovn-chassis-4 ip link del publicp2-p > > > > > + > > > > > +# Create East-West switch for LB backends > > > > > +check multinode_nbctl ls-add sw0 > > > > > +check multinode_nbctl lsp-add sw0 sw0-port1 > > > > > +check multinode_nbctl lsp-set-addresses sw0-port1 > "50:54:00:00:00:03 > > > > 10.0.0.3 1000::3" > > > > > +check multinode_nbctl lsp-add sw0 sw0-port2 > > > > > +check multinode_nbctl lsp-set-addresses sw0-port2 > "50:54:00:00:00:04 > > > > 10.0.0.4 1000::4" > > > > > + > > > > > +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 > > > > 50:54:00:00:00:03 1400 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a > > > > > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 > > > > 50:54:00:00:00:04 1400 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a > > > > > + > > > > > +m_wait_for_ports_up > > > > > + > > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w > 2 > > > > 10.0.0.4 | FORMAT_PING], \ > > > > > +[0], [dnl > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > +]) > > > > > + > > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w > 2 > > > > 10.0.0.3 | FORMAT_PING], \ > > > > > +[0], [dnl > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > +]) > > > > > + > > > > > +# Create a logical router and attach to sw0 > > > > > +check multinode_nbctl lr-add lr0 > > > > > +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 > 10.0.0.1/24 > > > > 1000::a/64 > > > > > +check multinode_nbctl lsp-add sw0 sw0-lr0 > > > > > +check multinode_nbctl lsp-set-type sw0-lr0 router > > > > > +check multinode_nbctl lsp-set-addresses sw0-lr0 router > > > > > +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 > > > > > + > > > > > +# create external connection for N/S traffic using multiple DGPs > > > > > +check multinode_nbctl ls-add public > > > > > + > > > > > +# DGP public1 > > > > > +check multinode_nbctl lsp-add public ln-public-1 > > > > > +check multinode_nbctl lsp-set-type ln-public-1 localnet > > > > > +check multinode_nbctl lsp-set-addresses ln-public-1 unknown > > > > > +check multinode_nbctl lsp-set-options ln-public-1 > network_name=public1 > > > > > + > > > > > +# DGP public2 > > > > > +# create exteranl connection for N/S traffic > > > > > +check multinode_nbctl lsp-add public ln-public-2 > > > > > +check multinode_nbctl lsp-set-type ln-public-2 localnet > > > > > +check multinode_nbctl lsp-set-addresses ln-public-2 unknown > > > > > +check multinode_nbctl lsp-set-options ln-public-2 > network_name=public2 > > > > > + > > > > > +# Attach DGP public1 to GW-1 and chassis-3 (overlay connectivity) > > > > > +m_as ovn-gw-1 ovs-vsctl set open . > > > > external-ids:ovn-bridge-mappings=public1:br-ex > > > > > +m_as ovn-chassis-3 ovs-vsctl set open . > > > > external-ids:ovn-bridge-mappings=public1:br-ex > > > > > + > > > > > +# Attach DGP public2 to GW-2 and chassis-4 (overlay connectivity) > > > > > +m_as ovn-gw-2 ovs-vsctl set open . > > > > external-ids:ovn-bridge-mappings=public2:br-ex > > > > > +m_as ovn-chassis-4 ovs-vsctl set open . > > > > external-ids:ovn-bridge-mappings=public2:br-ex > > > > > + > > > > > +# Create the external LR0 port to the DGP public1 > > > > > +check multinode_nbctl lsp-add public public-port1 > > > > > +check multinode_nbctl lsp-set-addresses public-port1 > "40:54:00:00:00:03 > > > > 20.0.0.3 2000::3" > > > > > + > > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p1 00:00:00:00:ff:02 > > > > 20.0.0.1/24 2000::a/64 > > > > > +check multinode_nbctl lsp-add public public-lr0-p1 > > > > > +check multinode_nbctl lsp-set-type public-lr0-p1 router > > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p1 router > > > > > +check multinode_nbctl lsp-set-options public-lr0-p1 > > > > router-port=lr0-public-p1 > > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p1 > ovn-gw-1 10 > > > > > + > > > > > +# Create a VM on ovn-chassis-3 in the same public1 overlay > > > > > +m_as ovn-chassis-3 /data/create_fake_vm.sh public-port1 publicp1 > > > > 40:54:00:00:00:03 1400 20.0.0.3 24 20.0.0.1 2000::4/64 2000::a > > > > > + > > > > > +m_wait_for_ports_up public-port1 > > > > > + > > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [ping -q -c 3 -i 0.3 > -w 2 > > > > 20.0.0.1 | FORMAT_PING], \ > > > > > +[0], [dnl > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > +]) > > > > > + > > > > > +# Create the external LR0 port to the DGP public2 > > > > > +check multinode_nbctl lsp-add public public-port2 > > > > > +check multinode_nbctl lsp-set-addresses public-port2 > "60:54:00:00:00:03 > > > > 30.0.0.3 3000::3" > > > > > + > > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p2 00:00:00:00:ff:03 > > > > 30.0.0.1/24 3000::a/64 > > > > > +check multinode_nbctl lsp-add public public-lr0-p2 > > > > > +check multinode_nbctl lsp-set-type public-lr0-p2 router > > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p2 router > > > > > +check multinode_nbctl lsp-set-options public-lr0-p2 > > > > router-port=lr0-public-p2 > > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p2 > ovn-gw-2 10 > > > > > + > > > > > +# Create a VM on ovn-chassis-4 in the same public2 overlay > > > > > +m_as ovn-chassis-4 /data/create_fake_vm.sh public-port2 publicp2 > > > > 60:54:00:00:00:03 1400 30.0.0.3 24 30.0.0.1 3000::4/64 3000::a > > > > > + > > > > > +m_wait_for_ports_up public-port2 > > > > > + > > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [ping -q -c 3 -i 0.3 > -w 2 > > > > 30.0.0.1 | FORMAT_PING], \ > > > > > +[0], [dnl > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > +]) > > > > > + > > > > > +# Add a default route for multiple DGPs - using ECMP > > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 > 20.0.0.3 > > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 > 30.0.0.3 > > > > > + > > > > > +# Add SNAT rules using gateway-port > > > > > +check multinode_nbctl --gateway-port lr0-public-p1 lr-nat-add lr0 > snat > > > > 20.0.0.1 10.0.0.0/24 > > > > > +check multinode_nbctl --gateway-port lr0-public-p2 lr-nat-add lr0 > snat > > > > 30.0.0.1 10.0.0.0/24 > > > > > + > > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w > 2 > > > > 20.0.0.3 | FORMAT_PING], \ > > > > > +[0], [dnl > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > +]) > > > > > + > > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w > 2 > > > > 30.0.0.3 | FORMAT_PING], \ > > > > > +[0], [dnl > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > +]) > > > > > + > > > > > +# create LB > > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:80" "10.0.0.3:80, > > > > 10.0.0.4:80" > > > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > > > + > > > > > +# Set use_stateless_nat to true > > > > > +check multinode_nbctl set load_balancer lb0 > > > > options:use_stateless_nat=true > > > > > + > > > > > +# Start backend http services > > > > > +M_NS_DAEMONIZE([ovn-chassis-1], [sw0p1], [$PYTHON -m http.server > --bind > > > > 10.0.0.3 80 >/dev/null 2>&1], [http1.pid]) > > > > > +M_NS_DAEMONIZE([ovn-chassis-2], [sw0p2], [$PYTHON -m http.server > --bind > > > > 10.0.0.4 80 >/dev/null 2>&1], [http2.pid]) > > > > > + > > > > > +# wait for http server be ready > > > > > +sleep 2 > > > > > + > > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v > 172.16.0.100:80 > > > > --retry 3 --max-time 1 --local-port 59002 2> curl.out']) > > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [sh -c 'cat curl.out > | > > > > grep -i -e connect | grep -v 'Server:''], \ > > > > > +[0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v > 172.16.0.100:80 > > > > --retry 3 --max-time 1 --local-port 59003 2> curl.out']) > > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [sh -c 'cat curl.out > | > > > > grep -i -e connect | grep -v 'Server:''], \ > > > > > +[0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v > 172.16.0.100:80 > > > > --retry 3 --max-time 1 --local-port 59001']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl > dpctl/dump-conntrack | > > > > M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > > [dnl > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v > 172.16.0.100:80 > > > > --retry 3 --max-time 1 --local-port 59000']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl > dpctl/dump-conntrack | > > > > M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > > [dnl > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# create a big file on web servers for download > > > > > +M_NS_EXEC([ovn-chassis-1], [sw0p1], [dd bs=512 count=200000 > > > > if=/dev/urandom of=download_file]) > > > > > +M_NS_EXEC([ovn-chassis-2], [sw0p2], [dd bs=512 count=200000 > > > > if=/dev/urandom of=download_file]) > > > > > + > > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > 59004 > > > > 2>curl.out']) > > > > > + > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack > | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack > | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > + > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > > > > curl.out | \ > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +# Check if we have only one backend for the same connection - > orig + > > > > dest ports > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > > [dnl > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Check if gw-2 is empty to ensure that the traffic only come > from/to > > > > the originator chassis via DGP public1 > > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > + > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep > "dport=80" -c) > > > > > + > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > +# Backend resides on ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59004,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep > "dport=80" -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep > "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +else > > > > > +# Backend resides on ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59004,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep > "dport=80" -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep > "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +fi > > > > > + > > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +# Check the flows again for a new source port > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > 59005 > > > > 2>curl.out']) > > > > > + > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack > | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack > | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > + > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > > > > curl.out | \ > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +# Check if we have only one backend for the same connection - > orig + > > > > dest ports > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > > [dnl > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Check if gw-2 is empty to ensure that the traffic only come > from/to > > > > the originator chassis via DGP public1 > > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > + > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep > "dport=80" -c) > > > > > + > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > +# Backend resides on ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59005,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep > "dport=80" -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep > "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +else > > > > > +# Backend resides on ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59005,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep > "dport=80" -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep > "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +fi > > > > > + > > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +# Start a new test using the second DGP as origin (public2) > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > 59006 > > > > 2>curl.out']) > > > > > + > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack > | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack > | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > + > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > > > > curl.out | \ > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +# Check if we have only one backend for the same connection - > orig + > > > > dest ports > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > > [dnl > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Check if gw-1 is empty to ensure that the traffic only come > from/to > > > > the originator chassis via DGP public2 > > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > + > > > > > +# Check the backend IP from ct entries on gw-2 (DGP public2) > > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep > "dport=80" -c) > > > > > + > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > +# Backend resides on ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59006,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep > "dport=80" -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep > "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +else > > > > > +# Backend resides on ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59006,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep > "dport=80" -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep > "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +fi > > > > > + > > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +# Check the flows again for a new source port using the second > DGP as > > > > origin (public2) > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > 59007 > > > > 2>curl.out']) > > > > > + > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack > | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack > | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > + > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > > > > curl.out | \ > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +# Check if we have only one backend for the same connection - > orig + > > > > dest ports > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > > [dnl > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Check if gw-1 is empty to ensure that the traffic only come > from/to > > > > the originator chassis via DGP public2 > > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > + > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep > "dport=80" -c) > > > > > + > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > +# Backend resides on ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59007,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep > "dport=80" -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep > "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +else > > > > > +# Backend resides on ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59007,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep > "dport=80" -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep > "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +fi > > > > > + > > > > > +# Check multiple requests coming from DGP's public1 and public2 > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > grep > > > > -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +200 OK > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > grep > > > > -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +200 OK > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > grep > > > > -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +200 OK > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > grep > > > > -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +200 OK > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +# Remove the LB and change the VIP port - different from the > backend > > > > ports > > > > > +check multinode_nbctl lb-del lb0 > > > > > + > > > > > +# create LB again > > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:9000" "10.0.0.3:80 > , > > > > 10.0.0.4:80" > > > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > > > + > > > > > +# Set use_stateless_nat to true > > > > > +check multinode_nbctl set load_balancer lb0 > > > > options:use_stateless_nat=true > > > > > + > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +# Check end-to-end request using a new port for VIP > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port > 59008 > > > > 2>curl.out']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl > dpctl/dump-conntrack | > > > > M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > > [dnl > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > grep > > > > -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > > > +200 OK > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +# Check end-to-end request using a new port for VIP > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port > 59008 > > > > 2>curl.out']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl > dpctl/dump-conntrack | > > > > M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > > [dnl > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > grep > > > > -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > > > +200 OK > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +AT_CLEANUP > > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > > > > > index dcc3dbbc3..9e7a2f225 100644 > > > > > --- a/tests/ovn-northd.at > > > > > +++ b/tests/ovn-northd.at > > > > > @@ -13864,3 +13864,323 @@ check_no_redirect > > > > > > > > > > AT_CLEANUP > > > > > ]) > > > > > + > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP > + NAT > > > > Stateless)]) > > > > > +ovn_start > > > > > + > > > > > +check ovn-nbctl ls-add public > > > > > +check ovn-nbctl lr-add lr1 > > > > > + > > > > > +# lr1 DGP ts1 > > > > > +check ovn-nbctl ls-add ts1 > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > 172.16.10.1/24 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > > > + > > > > > +# lr1 DGP ts2 > > > > > +check ovn-nbctl ls-add ts2 > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > 172.16.20.1/24 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > > > + > > > > > +# lr1 DGP public > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > 173.16.0.1/16 > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > 172.16.0.1/24 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > > + > > > > > +check ovn-nbctl ls-add s1 > > > > > +# s1 - lr1 > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > 172.16.0.1" > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > > + > > > > > +# s1 - backend vm1 > > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > 172.16.0.101" > > > > > + > > > > > +# s1 - backend vm2 > > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > 172.16.0.102" > > > > > + > > > > > +# s1 - backend vm3 > > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > 172.16.0.103" > > > > > + > > > > > +# Add the lr1 DGP ts1 to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > > > nat-addresses=router > > > > > + > > > > > +# Add the lr1 DGP ts2 to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > > > nat-addresses=router > > > > > + > > > > > +# Add the lr1 DGP public to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > > > nat-addresses=router > > > > > + > > > > > +# Create the Load Balancer lb1 > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > > > + > > > > > +# Set use_stateless_nat to true > > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > > > options:use_stateless_nat=true > > > > > + > > > > > +# Associate load balancer to s1 > > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > > +check ovn-nbctl --wait=sb sync > > > > > + > > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > > +AT_CAPTURE_FILE([s1flows]) > > > > > + > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | > grep > > > > "30.0.0.1"], [0], [dnl > > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] > == 1 > > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > > > +]) > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > > "30.0.0.1"], [0], [dnl > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > > > ip4.dst == 30.0.0.1), > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > +]) > > > > > + > > > > > +# Associate load balancer to lr1 with DGP > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > > +check ovn-nbctl --wait=sb sync > > > > > + > > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > > + > > > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > > "30.0.0.1"], [0], [dnl > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > is_chassis_resident("cr-lr1-ts1")), > > > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > is_chassis_resident("cr-lr1-ts2")), > > > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > is_chassis_resident("cr-lr1_public")), > > > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > +]) > > > > > + > > > > > +# 2. Check if the DGP ports are in the match with action next > > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], > [dnl > > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > > > action=(next;) > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src > == > > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src > == > > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src > == > > > > 172.16.0.101)) && (inport == "lr1_public" || outport == > "lr1_public") && > > > > is_chassis_resident("cr-lr1_public") && tcp), action=(next;) > > > > > +]) > > > > > + > > > > > +# 3. Check if the VIP IP is in the ipX.src action > > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], > [dnl > > > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > > > action=(next;) > > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > > > action=(next;) > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src > == > > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(ip4.src=30.0.0.1; > next;) > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src > == > > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(ip4.src=30.0.0.1; > next;) > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src > == > > > > 172.16.0.101)) && (inport == "lr1_public" || outport == > "lr1_public") && > > > > is_chassis_resident("cr-lr1_public") && tcp), > action=(ip4.src=30.0.0.1; > > > > next;) > > > > > +]) > > > > > + > > > > > +AT_CLEANUP > > > > > +]) > > > > > + > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP > + NAT > > > > Stateless) - IPv6]) > > > > > +ovn_start > > > > > + > > > > > +check ovn-nbctl ls-add public > > > > > +check ovn-nbctl lr-add lr1 > > > > > + > > > > > +# lr1 DGP ts1 > > > > > +check ovn-nbctl ls-add ts1 > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > > > > 2001:db8:aaaa:1::1/64 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > > > + > > > > > +# lr1 DGP ts2 > > > > > +check ovn-nbctl ls-add ts2 > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > > > > 2001:db8:aaaa:2::1/64 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > > > + > > > > > +# lr1 DGP public > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > > > > 2001:db8:bbbb::1/64 > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > > > > 2001:db8:aaaa:3::1/64 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > > + > > > > > +check ovn-nbctl ls-add s1 > > > > > +# s1 - lr1 > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > > > > 2001:db8:aaaa:3::1" > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > > + > > > > > +# s1 - backend vm1 > > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > > > > 2001:db8:aaaa:3::101" > > > > > + > > > > > +# s1 - backend vm2 > > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > > > > 2001:db8:aaaa:3::102" > > > > > + > > > > > +# s1 - backend vm3 > > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > > > > 2001:db8:aaaa:3::103" > > > > > + > > > > > +# Add the lr1 DGP ts1 to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > > > nat-addresses=router > > > > > + > > > > > +# Add the lr1 DGP ts2 to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > > > nat-addresses=router > > > > > + > > > > > +# Add the lr1 DGP public to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > > > nat-addresses=router > > > > > + > > > > > +# Create the Load Balancer lb1 > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "2001:db8:cccc::1" > > > > "2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101" > > > > > + > > > > > +# Set use_stateless_nat to true > > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > > > options:use_stateless_nat=true > > > > > + > > > > > +# Associate load balancer to s1 > > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > > +check ovn-nbctl --wait=sb sync > > > > > + > > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > > +AT_CAPTURE_FILE([s1flows]) > > > > > + > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | > grep > > > > "2001:db8:cccc::1"], [0], [dnl > > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] > == 1 > > > > && ip6.dst == 2001:db8:cccc::1), action=(xxreg1 = 2001:db8:cccc::1; > > > > ct_lb_mark;) > > > > > +]) > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > > "2001:db8:cccc::1"], [0], [dnl > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > > > ip6.dst == 2001:db8:cccc::1), > > > > > action=(ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > +]) > > > > > + > > > > > +# Associate load balancer to lr1 with DGP > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > > +check ovn-nbctl --wait=sb sync > > > > > + > > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > > + > > > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > > "2001:db8:cccc::1"], [0], [dnl > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > > is_chassis_resident("cr-lr1-ts1")), > > > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > > is_chassis_resident("cr-lr1-ts2")), > > > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > > is_chassis_resident("cr-lr1_public")), > > > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > +]) > > > > > + > > > > > +# 2. Check if the DGP ports are in the match with action next > > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], > [dnl > > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > > > action=(next;) > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > 2001:db8:aaaa:3::102) || > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || > outport == > > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), > action=(next;) > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > 2001:db8:aaaa:3::102) || > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || > outport == > > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), > action=(next;) > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > 2001:db8:aaaa:3::102) || > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || > outport == > > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > > > action=(next;) > > > > > +]) > > > > > + > > > > > +# 3. Check if the VIP IP is in the ipX.src action > > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], > [dnl > > > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > > > action=(next;) > > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > > > action=(next;) > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > 2001:db8:aaaa:3::102) || > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || > outport == > > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), > > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > 2001:db8:aaaa:3::102) || > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || > outport == > > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), > > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > 2001:db8:aaaa:3::102) || > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || > outport == > > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > > +]) > > > > > + > > > > > +AT_CLEANUP > > > > > +]) > > > > > + > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (DGP)]) > > > > > +ovn_start > > > > > + > > > > > +check ovn-nbctl ls-add public > > > > > +check ovn-nbctl lr-add lr1 > > > > > + > > > > > +# lr1 DGP ts1 > > > > > +check ovn-nbctl ls-add ts1 > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > 172.16.10.1/24 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-1 > > > > > + > > > > > +# lr1 DGP ts2 > > > > > +check ovn-nbctl ls-add ts2 > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > 172.16.20.1/24 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-1 > > > > > + > > > > > +# lr1 DGP public > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > 173.16.0.1/16 > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > 172.16.0.1/24 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > > + > > > > > +check ovn-nbctl ls-add s1 > > > > > +# s1 - lr1 > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > 172.16.0.1" > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > > + > > > > > +# s1 - backend vm1 > > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > 172.16.0.101" > > > > > + > > > > > +# s1 - backend vm2 > > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > 172.16.0.102" > > > > > + > > > > > +# s1 - backend vm3 > > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > 172.16.0.103" > > > > > + > > > > > +# Add the lr1 DGP ts1 to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > > > nat-addresses=router > > > > > + > > > > > +# Add the lr1 DGP ts2 to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > > > nat-addresses=router > > > > > + > > > > > +# Add the lr1 DGP public to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > > > nat-addresses=router > > > > > + > > > > > +# Create the Load Balancer lb1 > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > > > + > > > > > +# Associate load balancer to s1 > > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > > +check ovn-nbctl --wait=sb sync > > > > > + > > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > > +AT_CAPTURE_FILE([s1flows]) > > > > > + > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | > grep > > > > "30.0.0.1"], [0], [dnl > > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] > == 1 > > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > > > +]) > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > > "30.0.0.1"], [0], [dnl > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > > > ip4.dst == 30.0.0.1), > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > +]) > > > > > + > > > > > +# Associate load balancer to lr1 with DGP > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > > +check ovn-nbctl --wait=sb sync > > > > > + > > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > > + > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > > "30.0.0.1"], [0], [dnl > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > is_chassis_resident("cr-lr1-ts1")), > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > is_chassis_resident("cr-lr1-ts2")), > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > is_chassis_resident("cr-lr1_public")), > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > +]) > > > > > + > > > > > +AT_CLEANUP > > > > > +]) > > > > > -- > > > > > 2.34.1 > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > _'Esta mensagem é direcionada apenas para os endereços constantes > no > > > > > cabeçalho inicial. Se você não está listado nos endereços > constantes no > > > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo > dessa > > > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações > citadas > > > > estão > > > > > imediatamente anuladas e proibidas'._ > > > > > > > > > > > > > > > * **'Apesar do Magazine Luiza tomar > > > > > todas as precauções razoáveis para assegurar que nenhum vírus > esteja > > > > > presente nesse e-mail, a empresa não poderá aceitar a > responsabilidade > > > > por > > > > > quaisquer perdas ou danos causados por esse e-mail ou por seus > anexos'.* > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > dev mailing list > > > > > dev@openvswitch.org > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > _‘Esta mensagem é direcionada apenas para os endereços constantes no > > > cabeçalho inicial. Se você não está listado nos endereços constantes no > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas > estão > > > imediatamente anuladas e proibidas’._ > > > > > > > > > * **‘Apesar do Magazine Luiza tomar > > > todas as precauções razoáveis para assegurar que nenhum vírus esteja > > > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade > por > > > quaisquer perdas ou danos causados por esse e-mail ou por seus > anexos’.* > > > > > > > > > > > > _______________________________________________ > > > dev mailing list > > > dev@openvswitch.org > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev >
On Thu, Sep 26, 2024 at 10:55 AM Roberto Bartzen Acosta via dev <ovs-dev@openvswitch.org> wrote: > > Hi Numan, > > Thanks for your feedback and review. > > Em qua., 25 de set. de 2024 às 19:50, Numan Siddique <numans@ovn.org> > escreveu: > > > On Thu, Sep 19, 2024 at 6:12 PM Roberto Bartzen Acosta via dev > > <ovs-dev@openvswitch.org> wrote: > > > > > > This commit fixes the build_distr_lrouter_nat_flows_for_lb function to > > > include a DNAT flow entry for each DGP in use. Since we have added > > support > > > to create multiple gateway ports per logical router, it's necessary to > > > include in the LR NAT rules pipeline a specific entry for each attached > > DGP. > > > Otherwise, the inbound traffic will only be redirected when the incoming > > LRP > > > matches the chassis_resident field. > > > > > > Additionally, this patch includes the ability to use load-balancer with > > DGPs > > > attached to multiple chassis. We can have each of the DGPs associated > > with a > > > different chassis, and in this case the DNAT rules added by default will > > not > > > be enough to guarantee outgoing traffic. > > > > > > To solve the multiple chassis for DGPs problem, this patch include a new > > > config options to be configured in the load-balancer. If the > > use_stateless_nat > > > is set to true, the logical router that references this load-balancer > > will use > > > Stateless NAT rules when the logical router has multiple DGPs. After > > applying > > > this patch and setting the use_stateless_nat option, the inbound and/or > > > outbound traffic can pass through any chassis where the DGP resides > > without > > > having problems with CT state. > > > > > > Reported-at: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2054322 > > > Fixes: 15348b7b806f ("ovn-northd: Multiple distributed gateway port > > support.") > > > > > > Signed-off-by: Roberto Bartzen Acosta <roberto.acosta@luizalabs.com> > > > > Hi Roberto, > > > > Thanks for the patch. I tested this patch using the test example in > > multinode.at. > > > > The test case adds the below load balancer > > > > [root@ovn-central ~]# ovn-nbctl lb-list > > UUID LB PROTO > > VIP IPs > > f3e29869-3bb5-4df0-960a-171106f5913a lb0 tcp > > 172.16.0.100:9000 10.0.0.3:80,10.0.0.4:80 > > > > And the below logical flows are generated by this patch > > > > -------- > > [root@ovn-central ~]# ovn-sbctl dump-flows lr0 | grep 172.16.0.100 > > table=6 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst > > == 172.16.0.100), action=(ct_dnat;) > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > is_chassis_resident("cr-lr0-public-p1")), > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80, > > 10.0.0.4:80);) > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > is_chassis_resident("cr-lr0-public-p2")), > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80, > > 10.0.0.4:80);) > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > tcp.src == 80)) && (inport == "lr0-public-p1" || outport == > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > tcp.src == 80)) && (inport == "lr0-public-p2" || outport == > > "lr0-public-p2") && is_chassis_resident("cr-lr0-public-p2") && tcp), > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > -------------- > > > > > > I fail to understand the reason for modifying the ip4.dst before > > calling ct_lb_mark. Can you please explain why ? Because the > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;) will first modify the > > ip4.dst to 10.0.0.3 and > > then to 10.0.0.4 and then the ct_lb_mark will actually do the > > conntrack with NAT to either 10.0.0.3 or 10.0.0.4. > > > > Is it because you want the conntrack entry to not have 172.16.0.100 ? > > > > The only reason I included this ip4.dst action in the DNAT rule is because > it's required to accept packets coming from a chassis that doesn't have > previously created conntrack entries. The main feature introduced in this > patch is to allow the administrator to have multiple DGPs attached to > different chassis (is_chassis_resident...). So, my implementation was based > on the normal behavior when using stateless NAT for external addresses, > where we need to add the ipx.dst in lr_in_dnat for traffic to be received > on the chassis (put the DGP port as chassis_resident match, as is the case > with stateless NAT [1] with DGP[2]). > > The question is, if we only have the ct_lb_mark, packets that pass through > the chassis and are already part of an active flow in another chassis (same > IPs and Ports) will be dropped because there is no correspondence in the > backend. So only packets with the NEW flag will be accepted and sent to the > backend (at least for TCP traffic). If we only have the ip4.dst action, > this will always perform the dnat for the same backend, without balancing. > Therefore, the combination of the two actions allows the packet to always > be received (regardless of whether conntrack is active for it), and > ct_lb_mark will take care of balancing for different backends. > > If we had conntrack sync between different chassis this would not be > necessary, as the ct_lb_mark action could always be executed without > dropping packets due to lack of correspondence in the conntrack table. > > [1] > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15737 > [2] > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15726 > > I'm sorry, but it's not 100% clear to me. I know that you've already explained to Mark in the older version of this patch. Can you please explain with an example ? Let's take the below topology you've added in the mutlinode test as example # Network topology # # publicp1 (ovn-chassis-3) (20.0.0.3/24) # | # overlay # | # DGP public1 (ovn-gw-1) (20.0.0.1/24) # | # | # | # lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) 10.0.0.3/24 # | | # | + --- sw0p2 (ovn-chassis-2) 10.0.0.4/24 # | # DGP public2 (ovn-gw-2) (30.0.0.1/24) # | # overlay # | # publicp2 (ovn-chassis-4) (30.0.0.3/24) load balancer is configured on lr0 -> ovn-nbctl lb-add lb0 "172.16.0.100:80" "10.0.0.3:80,10.0.0.4:80" and it is attached to both lr0 and sw0. Scenario 1: publicp1 with IP 20.0.0.3 sends TCP traffic to VIP 172.16.0.100. I think this is what will happen - The packet (ip4.src = 20.0.0.3 , ip4.dst = 172.16.0.100, tcp.dst = 80) from ovn-chassis-3 will be sent out via the localnet bridge (br-ex) and the packet is received on ovn-gw-1 (as DGP public1 is resident on it) via the localnet bridge and the packet first enters public logical switch pipeline and then the lr0 router pipeline. - In the router's lr_in_dnat state, the packet will be load balanced to one of the backends using ct_lb_mark. Lets say 10.0.0.3 is chosen - The packet from router pipeline lr0 enters sw0 switch pipeline and then the packet is tunneled to ovn-chassis-1 and delivered to sw0p1. - The reply packet (ip4.src = 10.0.0.3, ip4.dst = 20.0.0.3, tcp.src = 80) will enter the router pipeline and since 20.0.0.0/24 is handled by DGP public1, the packet is tunnelled to ovn-gw-1. - In the router pipeline of ovn-gw-1, the packet is undnatted from ip4.src 10.0.0.3 to 172.16.0.100 and the packet is sent out via the localnet bridge. - ovn-chassis-3 receives the packet via the localnet bridge and into br-int and finally to publicp1. In this scenario the load balancing is handled and conntrack entries are created in ovn-gw-1. And there is no need to add flows in "lr_out_snat" for stateless NAT or set ip4.dst to one or all of backend IPs before "ct_lb_mark" in lr_in_dnat stage. Scenario 2: publicp2 with IP 30.0.0.3 sends TCP traffic to VIP 172.16.0.100. This is similar to scenario 1. Except that load balancing happens in ovn-gw-2. since DGP public2 is on this chassis. Scenario 3: An external entity with IP 20.0.0.50 sends TCP traffic to VIP 172.16.0.100. - This scenario is similar to the first one. The packet from this external entity is received on ovn-gw-1 via the localnet bridge. Rest all is the same. Scenario 4: sw0p2 on ovn-chassis-2 sends TCP traffic to VIP 172.16.0.100. - Since sw0 is attached with load balancer lb0, load balancing happens in the source chassis - ovn-chassis-2 itself and depending on the backend chosen, the packet is tunnelled to ovn-chassis-1 (if 10.0.0.3 is chosen) or delivered directly to sw0p2 (if i0.0.0.4 is chosen). Scenario 5: An external entity with IP 40.0.0.40 sends TCP traffic to VIP 172.16.0.100 and there are 2 ECMP routes configured 172.16.0.100 via 20.0.0.1 172.16.0.100 via 30.0.0.1 In this case if the packet uses the route via 20.0.0.1 it will be received on ovn-gw-1 via the localnet bridge br-ex. - With your patch, ip4.dst is modified to the last backend in the list and then ct_lb_mark chosen as one of the backends - And then it is tunnelled to the destination chassis. Is this the scenario 5 you're trying to address with the stateless NAT ? How would the reply packet work ? The reply packet from backend 10.0.0.3 can use either of the paths ? i.e ovn-gw-1 or ovn-gw-2 ? Thanks Numan > > > > Also I don't understand why this patch adds the logical flows in > > "lr_out_snat" stage ? > > > > The flow for lr_out_snat is necessary for the correct functioning of > stateless NAT for the same reason explained previously. I mean, if the > outgoing packet is redirected to a chassis that doesn't have an active > conntrack entry, it will not be NATed by ct_lb action because it doesn't > refer to a valid flow (use case with ecmp). > > So it is necessary to create a stateless SNAT rule (similar to this [3]) > with a lower priority than the other router pipeline entries, in this case, > if the packet is not SNATed by ct_lb (conntrack missed) it will be SNATed > by stateless NAT rule. > > [3] > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15884 > > > > > > > Using the system multinode test as an example, the below fails > > (which is a regression) > > > > --- > > root@ovn-chassis-3 ~]# ip netns exec publicp1 nc -vz 10.0.0.3 80 > > ---- > > > > In the above test, publicp1 with IP 20.0.0.3 when it tries to connect > > to one if the backends directly (without the LB VIP), it fails. > > It fails because of the logical flows in "lr_out_snat". > > > > > > Looks to me the solution proposed here is incomplete. > > > > Also please note that in our CI we run the multinode tests > > periodically once a day using the v0.1 of the ovn-fake-multinode > > and the tests you added will fail. This needs to be fixed and until > > we move to the latest version of ovn-fake-multinode. > > > > I imagine that the test you are doing is using the same port as the LB > backend (TCP 80 in this case). So, the stateless lr_out_snat flow will > force the output to be SNATed because this port is in use by the backend. > Traffic to/from other ports will work without problems and will follow the > normal programmed flows (e.g. ICMP). > > This is necessary to ensure the egress traffic because the DGPs are > distributed across multiple chassis. Also, this setup is being validated in > the test ovn-fake-multinode testcase (ICMP from the backends chassis use > the router's default SNAT and not the LB's). I didn't understand the > regression you mentioned because this was programmed to be stateless and > it's traffic that uses the same ports as the LB backend, could you explain > better? > > Thanks, > Roberto > > > > Thanks > > Numan > > > > > > > --- > > > northd/en-lr-stateful.c | 12 - > > > northd/northd.c | 116 ++++++-- > > > ovn-nb.xml | 10 + > > > tests/multinode-macros.at | 40 +++ > > > tests/multinode.at | 556 ++++++++++++++++++++++++++++++++++++++ > > > tests/ovn-northd.at | 320 ++++++++++++++++++++++ > > > 6 files changed, 1017 insertions(+), 37 deletions(-) > > > > > > diff --git a/northd/en-lr-stateful.c b/northd/en-lr-stateful.c > > > index baf1bd2f8..f09691af6 100644 > > > --- a/northd/en-lr-stateful.c > > > +++ b/northd/en-lr-stateful.c > > > @@ -516,18 +516,6 @@ lr_stateful_record_create(struct lr_stateful_table > > *table, > > > > > > table->array[od->index] = lr_stateful_rec; > > > > > > - /* Load balancers are not supported (yet) if a logical router has > > multiple > > > - * distributed gateway port. Log a warning. */ > > > - if (lr_stateful_rec->has_lb_vip && lr_has_multiple_gw_ports(od)) { > > > - static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1); > > > - VLOG_WARN_RL(&rl, "Load-balancers are configured on logical " > > > - "router %s, which has %"PRIuSIZE" distributed " > > > - "gateway ports. Load-balancer is not supported " > > > - "yet when there is more than one distributed " > > > - "gateway port on the router.", > > > - od->nbr->name, od->n_l3dgw_ports); > > > - } > > > - > > > return lr_stateful_rec; > > > } > > > > > > diff --git a/northd/northd.c b/northd/northd.c > > > index a267cd5f8..bbe97acf8 100644 > > > --- a/northd/northd.c > > > +++ b/northd/northd.c > > > @@ -11807,31 +11807,30 @@ static void > > > build_distr_lrouter_nat_flows_for_lb(struct lrouter_nat_lb_flows_ctx > > *ctx, > > > enum lrouter_nat_lb_flow_type type, > > > struct ovn_datapath *od, > > > - struct lflow_ref *lflow_ref) > > > + struct lflow_ref *lflow_ref, > > > + struct ovn_port *dgp, > > > + bool stateless_nat) > > > { > > > - struct ovn_port *dgp = od->l3dgw_ports[0]; > > > - > > > - const char *undnat_action; > > > - > > > - switch (type) { > > > - case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > - undnat_action = "flags.force_snat_for_lb = 1; next;"; > > > - break; > > > - case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > - undnat_action = "flags.skip_snat_for_lb = 1; next;"; > > > - break; > > > - case LROUTER_NAT_LB_FLOW_NORMAL: > > > - case LROUTER_NAT_LB_FLOW_MAX: > > > - undnat_action = lrouter_use_common_zone(od) > > > - ? "ct_dnat_in_czone;" > > > - : "ct_dnat;"; > > > - break; > > > - } > > > + struct ds dnat_action = DS_EMPTY_INITIALIZER; > > > > > > /* Store the match lengths, so we can reuse the ds buffer. */ > > > size_t new_match_len = ctx->new_match->length; > > > size_t undnat_match_len = ctx->undnat_match->length; > > > > > > + /* dnat_action: Add the LB backend IPs as a destination action of > > the > > > + * lr_in_dnat NAT rule with cumulative effect because > > any > > > + * backend dst IP used in the action list will > > redirect the > > > + * packet to the ct_lb pipeline. > > > + */ > > > + if (stateless_nat) { > > > + for (size_t i = 0; i < ctx->lb_vip->n_backends; i++) { > > > + struct ovn_lb_backend *backend = &ctx->lb_vip->backends[i]; > > > + bool ipv6 = !IN6_IS_ADDR_V4MAPPED(&backend->ip); > > > + ds_put_format(&dnat_action, "%s.dst=%s;", ipv6 ? "ip6" : > > "ip4", > > > + backend->ip_str); > > > + } > > > + } > > > + ds_put_format(&dnat_action, "%s", ctx->new_action[type]); > > > > > > const char *meter = NULL; > > > > > > @@ -11841,20 +11840,46 @@ build_distr_lrouter_nat_flows_for_lb(struct > > lrouter_nat_lb_flows_ctx *ctx, > > > > > > if (ctx->lb_vip->n_backends || !ctx->lb_vip->empty_backend_rej) { > > > ds_put_format(ctx->new_match, " && is_chassis_resident(%s)", > > > - od->l3dgw_ports[0]->cr_port->json_key); > > > + dgp->cr_port->json_key); > > > } > > > > > > ovn_lflow_add_with_hint__(ctx->lflows, od, S_ROUTER_IN_DNAT, > > ctx->prio, > > > - ds_cstr(ctx->new_match), > > ctx->new_action[type], > > > + ds_cstr(ctx->new_match), > > ds_cstr(&dnat_action), > > > NULL, meter, &ctx->lb->nlb->header_, > > > lflow_ref); > > > > > > ds_truncate(ctx->new_match, new_match_len); > > > > > > + ds_destroy(&dnat_action); > > > if (!ctx->lb_vip->n_backends) { > > > return; > > > } > > > > > > + struct ds undnat_action = DS_EMPTY_INITIALIZER; > > > + struct ds snat_action = DS_EMPTY_INITIALIZER; > > > + > > > + switch (type) { > > > + case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > + ds_put_format(&undnat_action, "flags.force_snat_for_lb = 1; > > next;"); > > > + break; > > > + case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > + ds_put_format(&undnat_action, "flags.skip_snat_for_lb = 1; > > next;"); > > > + break; > > > + case LROUTER_NAT_LB_FLOW_NORMAL: > > > + case LROUTER_NAT_LB_FLOW_MAX: > > > + ds_put_format(&undnat_action, "%s", > > > + lrouter_use_common_zone(od) ? "ct_dnat_in_czone;" > > > + : "ct_dnat;"); > > > + break; > > > + } > > > + > > > + /* undnat_action: Remove the ct action from the lr_out_undenat NAT > > rule. > > > + */ > > > + if (stateless_nat) { > > > + ds_clear(&undnat_action); > > > + ds_put_format(&undnat_action, "next;"); > > > + } > > > + > > > /* We need to centralize the LB traffic to properly perform > > > * the undnat stage. > > > */ > > > @@ -11873,11 +11898,41 @@ build_distr_lrouter_nat_flows_for_lb(struct > > lrouter_nat_lb_flows_ctx *ctx, > > > ds_put_format(ctx->undnat_match, ") && (inport == %s || outport == > > %s)" > > > " && is_chassis_resident(%s)", dgp->json_key, > > dgp->json_key, > > > dgp->cr_port->json_key); > > > + /* Use the LB protocol as matching criteria for out undnat and snat > > when > > > + * creating LBs with stateless NAT. */ > > > + if (stateless_nat) { > > > + ds_put_format(ctx->undnat_match, " && %s", ctx->lb->proto); > > > + } > > > ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_UNDNAT, 120, > > > - ds_cstr(ctx->undnat_match), undnat_action, > > > - &ctx->lb->nlb->header_, > > > + ds_cstr(ctx->undnat_match), > > > + ds_cstr(&undnat_action), > > &ctx->lb->nlb->header_, > > > lflow_ref); > > > + > > > + /* snat_action: Add a new lr_out_snat rule with the LB VIP as > > source IP > > > + * action to perform the NAT stateless pipeline > > completely. > > > + */ > > > + if (stateless_nat) { > > > + if (ctx->lb_vip->port_str) { > > > + ds_put_format(&snat_action, "%s.src=%s; %s.src=%s; next;", > > > + ctx->lb_vip->address_family == AF_INET6 ? > > > + "ip6" : "ip4", > > > + ctx->lb_vip->vip_str, ctx->lb->proto, > > > + ctx->lb_vip->port_str); > > > + } else { > > > + ds_put_format(&snat_action, "%s.src=%s; next;", > > > + ctx->lb_vip->address_family == AF_INET6 ? > > > + "ip6" : "ip4", > > > + ctx->lb_vip->vip_str); > > > + } > > > + ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_SNAT, 160, > > > + ds_cstr(ctx->undnat_match), > > > + ds_cstr(&snat_action), > > &ctx->lb->nlb->header_, > > > + lflow_ref); > > > + } > > > + > > > ds_truncate(ctx->undnat_match, undnat_match_len); > > > + ds_destroy(&undnat_action); > > > + ds_destroy(&snat_action); > > > } > > > > > > static void > > > @@ -12022,6 +12077,8 @@ build_lrouter_nat_flows_for_lb( > > > * lflow generation for them. > > > */ > > > size_t index; > > > + bool use_stateless_nat = smap_get_bool(&lb->nlb->options, > > > + "use_stateless_nat", false); > > > BITMAP_FOR_EACH_1 (index, bitmap_len, lb_dps->nb_lr_map) { > > > struct ovn_datapath *od = lr_datapaths->array[index]; > > > enum lrouter_nat_lb_flow_type type; > > > @@ -12043,8 +12100,17 @@ build_lrouter_nat_flows_for_lb( > > > if (!od->n_l3dgw_ports) { > > > bitmap_set1(gw_dp_bitmap[type], index); > > > } else { > > > - build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > > - lb_dps->lflow_ref); > > > + /* Create stateless LB NAT rules when using multiple DGPs > > and > > > + * use_stateless_nat is true. > > > + */ > > > + bool stateless_nat = (od->n_l3dgw_ports > 1) > > > + ? use_stateless_nat : false; > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > > > + struct ovn_port *dgp = od->l3dgw_ports[i]; > > > + build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > > + lb_dps->lflow_ref, > > dgp, > > > + stateless_nat); > > > + } > > > } > > > > > > if (lb->affinity_timeout) { > > > diff --git a/ovn-nb.xml b/ovn-nb.xml > > > index 2836f58f5..ad03c6214 100644 > > > --- a/ovn-nb.xml > > > +++ b/ovn-nb.xml > > > @@ -2302,6 +2302,16 @@ or > > > local anymore by the ovn-controller. This option is set to > > > <code>false</code> by default. > > > </column> > > > + > > > + <column name="options" key="use_stateless_nat" > > > + type='{"type": "boolean"}'> > > > + If the load balancer is configured with > > <code>use_stateless_nat</code> > > > + option to <code>true</code>, the logical router that references > > this > > > + load balancer will use Stateless NAT rules when the logical > > router > > > + has multiple distributed gateway ports(DGP). Otherwise, the > > outbound > > > + traffic may be dropped in scenarios where we have different > > chassis > > > + for each DGP. This option is set to <code>false</code> by > > default. > > > + </column> > > > </group> > > > </table> > > > > > > diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at > > > index 757917626..2f69433fc 100644 > > > --- a/tests/multinode-macros.at > > > +++ b/tests/multinode-macros.at > > > @@ -40,6 +40,27 @@ m4_define([M_START_TCPDUMP], > > > ] > > > ) > > > > > > +# M_EXEC([fake_node], [command]) > > > +# > > > +# Execute 'command' in 'fakenode' > > > +m4_define([M_EXEC], > > > + [podman exec $1 $2]) > > > + > > > +# M_CHECK_EXEC([fake_node], [command], other_params...) > > > +# > > > +# Wrapper for AT_CHECK that executes 'command' inside 'fake_node''s'. > > > +# 'other_params' as passed as they are to AT_CHECK. > > > +m4_define([M_CHECK_EXEC], > > > + [ AT_CHECK([M_EXEC([$1], [$2])], m4_shift(m4_shift(m4_shift($@)))) ] > > > +) > > > + > > > +# M_FORMAT_CT([ip-addr]) > > > +# > > > +# Strip content from the piped input which would differ from test to > > test > > > +# and limit the output to the rows containing 'ip-addr'. > > > +# > > > +m4_define([M_FORMAT_CT], > > > + [[grep -F "dst=$1," | sed -e 's/id=[0-9]*/id=<cleared>/g' -e > > 's/state=[0-9_A-Z]*/state=<cleared>/g' | sort | uniq | sed -e > > 's/zone=[[0-9]]*/zone=<cleared>/' -e 's/mark=[[0-9]]*/mark=<cleared>/' ]]) > > > > > > OVS_START_SHELL_HELPERS > > > > > > @@ -76,6 +97,25 @@ multinode_nbctl () { > > > m_as ovn-central ovn-nbctl "$@" > > > } > > > > > > +check_fake_multinode_setup_by_nodes() { > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > + for c in $1 > > > + do > > > + AT_CHECK([m_as $c ovn-appctl -t ovn-controller version], [0], > > [ignore]) > > > + done > > > +} > > > + > > > +cleanup_multinode_resources_by_nodes() { > > > + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db > > > + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > + for c in $1 > > > + do > > > + m_as $c ovs-vsctl del-br br-int > > > + m_as $c ip --all netns delete > > > + done > > > +} > > > + > > > # m_count_rows TABLE [CONDITION...] > > > # > > > # Prints the number of rows in TABLE (that satisfy CONDITION). > > > diff --git a/tests/multinode.at b/tests/multinode.at > > > index a0eb8fc67..b1beb4d97 100644 > > > --- a/tests/multinode.at > > > +++ b/tests/multinode.at > > > @@ -1591,3 +1591,559 @@ AT_CHECK([cat ch1_eth2.tcpdump], [0], [dnl > > > ]) > > > > > > AT_CLEANUP > > > + > > > +AT_SETUP([ovn multinode load-balancer with multiple DGPs and multiple > > chassis]) > > > + > > > +# Check that ovn-fake-multinode setup is up and running - requires > > additional nodes > > > +check_fake_multinode_setup_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > + > > > +# Delete the multinode NB and OVS resources before starting the test. > > > +cleanup_multinode_resources_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > + > > > +# Network topology > > > +# > > > +# publicp1 (ovn-chassis-3) (20.0.0.3/24) > > > +# | > > > +# overlay > > > +# | > > > +# DGP public1 (ovn-gw-1) (20.0.0.1/24) > > > +# | > > > +# | > > > +# | > > > +# lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) 10.0.0.3/24 > > > +# | | > > > +# | + --- sw0p2 (ovn-chassis-2) 10.0.0.4/24 > > > +# | > > > +# DGP public2 (ovn-gw-2) (30.0.0.1/24) > > > +# | > > > +# overlay > > > +# | > > > +# publicp2 (ovn-chassis-4) (30.0.0.3/24) > > > + > > > +# Delete already used ovs-ports > > > +m_as ovn-chassis-1 ovs-vsctl del-port br-int sw0p1-p > > > +m_as ovn-chassis-2 ovs-vsctl del-port br-int sw0p2-p > > > +m_as ovn-chassis-1 ip link del sw0p1-p > > > +m_as ovn-chassis-2 ip link del sw0p2-p > > > +m_as ovn-chassis-3 ovs-vsctl del-port br-int publicp1-p > > > +m_as ovn-chassis-4 ovs-vsctl del-port br-int publicp2-p > > > +m_as ovn-chassis-3 ip link del publicp1-p > > > +m_as ovn-chassis-4 ip link del publicp2-p > > > + > > > +# Create East-West switch for LB backends > > > +check multinode_nbctl ls-add sw0 > > > +check multinode_nbctl lsp-add sw0 sw0-port1 > > > +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 > > 10.0.0.3 1000::3" > > > +check multinode_nbctl lsp-add sw0 sw0-port2 > > > +check multinode_nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 > > 10.0.0.4 1000::4" > > > + > > > +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 > > 50:54:00:00:00:03 1400 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a > > > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 > > 50:54:00:00:00:04 1400 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a > > > + > > > +m_wait_for_ports_up > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > > 10.0.0.4 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 > > 10.0.0.3 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +# Create a logical router and attach to sw0 > > > +check multinode_nbctl lr-add lr0 > > > +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 > > 1000::a/64 > > > +check multinode_nbctl lsp-add sw0 sw0-lr0 > > > +check multinode_nbctl lsp-set-type sw0-lr0 router > > > +check multinode_nbctl lsp-set-addresses sw0-lr0 router > > > +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 > > > + > > > +# create external connection for N/S traffic using multiple DGPs > > > +check multinode_nbctl ls-add public > > > + > > > +# DGP public1 > > > +check multinode_nbctl lsp-add public ln-public-1 > > > +check multinode_nbctl lsp-set-type ln-public-1 localnet > > > +check multinode_nbctl lsp-set-addresses ln-public-1 unknown > > > +check multinode_nbctl lsp-set-options ln-public-1 network_name=public1 > > > + > > > +# DGP public2 > > > +# create exteranl connection for N/S traffic > > > +check multinode_nbctl lsp-add public ln-public-2 > > > +check multinode_nbctl lsp-set-type ln-public-2 localnet > > > +check multinode_nbctl lsp-set-addresses ln-public-2 unknown > > > +check multinode_nbctl lsp-set-options ln-public-2 network_name=public2 > > > + > > > +# Attach DGP public1 to GW-1 and chassis-3 (overlay connectivity) > > > +m_as ovn-gw-1 ovs-vsctl set open . > > external-ids:ovn-bridge-mappings=public1:br-ex > > > +m_as ovn-chassis-3 ovs-vsctl set open . > > external-ids:ovn-bridge-mappings=public1:br-ex > > > + > > > +# Attach DGP public2 to GW-2 and chassis-4 (overlay connectivity) > > > +m_as ovn-gw-2 ovs-vsctl set open . > > external-ids:ovn-bridge-mappings=public2:br-ex > > > +m_as ovn-chassis-4 ovs-vsctl set open . > > external-ids:ovn-bridge-mappings=public2:br-ex > > > + > > > +# Create the external LR0 port to the DGP public1 > > > +check multinode_nbctl lsp-add public public-port1 > > > +check multinode_nbctl lsp-set-addresses public-port1 "40:54:00:00:00:03 > > 20.0.0.3 2000::3" > > > + > > > +check multinode_nbctl lrp-add lr0 lr0-public-p1 00:00:00:00:ff:02 > > 20.0.0.1/24 2000::a/64 > > > +check multinode_nbctl lsp-add public public-lr0-p1 > > > +check multinode_nbctl lsp-set-type public-lr0-p1 router > > > +check multinode_nbctl lsp-set-addresses public-lr0-p1 router > > > +check multinode_nbctl lsp-set-options public-lr0-p1 > > router-port=lr0-public-p1 > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p1 ovn-gw-1 10 > > > + > > > +# Create a VM on ovn-chassis-3 in the same public1 overlay > > > +m_as ovn-chassis-3 /data/create_fake_vm.sh public-port1 publicp1 > > 40:54:00:00:00:03 1400 20.0.0.3 24 20.0.0.1 2000::4/64 2000::a > > > + > > > +m_wait_for_ports_up public-port1 > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [ping -q -c 3 -i 0.3 -w 2 > > 20.0.0.1 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +# Create the external LR0 port to the DGP public2 > > > +check multinode_nbctl lsp-add public public-port2 > > > +check multinode_nbctl lsp-set-addresses public-port2 "60:54:00:00:00:03 > > 30.0.0.3 3000::3" > > > + > > > +check multinode_nbctl lrp-add lr0 lr0-public-p2 00:00:00:00:ff:03 > > 30.0.0.1/24 3000::a/64 > > > +check multinode_nbctl lsp-add public public-lr0-p2 > > > +check multinode_nbctl lsp-set-type public-lr0-p2 router > > > +check multinode_nbctl lsp-set-addresses public-lr0-p2 router > > > +check multinode_nbctl lsp-set-options public-lr0-p2 > > router-port=lr0-public-p2 > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p2 ovn-gw-2 10 > > > + > > > +# Create a VM on ovn-chassis-4 in the same public2 overlay > > > +m_as ovn-chassis-4 /data/create_fake_vm.sh public-port2 publicp2 > > 60:54:00:00:00:03 1400 30.0.0.3 24 30.0.0.1 3000::4/64 3000::a > > > + > > > +m_wait_for_ports_up public-port2 > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [ping -q -c 3 -i 0.3 -w 2 > > 30.0.0.1 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +# Add a default route for multiple DGPs - using ECMP > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 20.0.0.3 > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 30.0.0.3 > > > + > > > +# Add SNAT rules using gateway-port > > > +check multinode_nbctl --gateway-port lr0-public-p1 lr-nat-add lr0 snat > > 20.0.0.1 10.0.0.0/24 > > > +check multinode_nbctl --gateway-port lr0-public-p2 lr-nat-add lr0 snat > > 30.0.0.1 10.0.0.0/24 > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > > 20.0.0.3 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 > > 30.0.0.3 | FORMAT_PING], \ > > > +[0], [dnl > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > +]) > > > + > > > +# create LB > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:80" "10.0.0.3:80, > > 10.0.0.4:80" > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > + > > > +# Set use_stateless_nat to true > > > +check multinode_nbctl set load_balancer lb0 > > options:use_stateless_nat=true > > > + > > > +# Start backend http services > > > +M_NS_DAEMONIZE([ovn-chassis-1], [sw0p1], [$PYTHON -m http.server --bind > > 10.0.0.3 80 >/dev/null 2>&1], [http1.pid]) > > > +M_NS_DAEMONIZE([ovn-chassis-2], [sw0p2], [$PYTHON -m http.server --bind > > 10.0.0.4 80 >/dev/null 2>&1], [http2.pid]) > > > + > > > +# wait for http server be ready > > > +sleep 2 > > > + > > > +# Flush conntrack entries for easier output parsing of next test. > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v 172.16.0.100:80 > > --retry 3 --max-time 1 --local-port 59002 2> curl.out']) > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [sh -c 'cat curl.out | > > grep -i -e connect | grep -v 'Server:''], \ > > > +[0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v 172.16.0.100:80 > > --retry 3 --max-time 1 --local-port 59003 2> curl.out']) > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [sh -c 'cat curl.out | > > grep -i -e connect | grep -v 'Server:''], \ > > > +[0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v 172.16.0.100:80 > > --retry 3 --max-time 1 --local-port 59001']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | > > M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v 172.16.0.100:80 > > --retry 3 --max-time 1 --local-port 59000']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | > > M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# create a big file on web servers for download > > > +M_NS_EXEC([ovn-chassis-1], [sw0p1], [dd bs=512 count=200000 > > if=/dev/urandom of=download_file]) > > > +M_NS_EXEC([ovn-chassis-2], [sw0p2], [dd bs=512 count=200000 > > if=/dev/urandom of=download_file]) > > > + > > > +# Flush conntrack entries for easier output parsing of next test. > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59004 > > 2>curl.out']) > > > + > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > > curl.out | \ > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +# Check if we have only one backend for the same connection - orig + > > dest ports > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Check if gw-2 is empty to ensure that the traffic only come from/to > > the originator chassis via DGP public1 > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > +0 > > > +]) > > > + > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > + > > > +if [[ $backend_check -gt 0 ]]; then > > > +# Backend resides on ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59004,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +else > > > +# Backend resides on ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59004,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +fi > > > + > > > +# Flush conntrack entries for easier output parsing of next test. > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +# Check the flows again for a new source port > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59005 > > 2>curl.out']) > > > + > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > > curl.out | \ > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +# Check if we have only one backend for the same connection - orig + > > dest ports > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Check if gw-2 is empty to ensure that the traffic only come from/to > > the originator chassis via DGP public1 > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > +0 > > > +]) > > > + > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > + > > > +if [[ $backend_check -gt 0 ]]; then > > > +# Backend resides on ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59005,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +else > > > +# Backend resides on ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59005,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +fi > > > + > > > +# Flush conntrack entries for easier output parsing of next test. > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +# Start a new test using the second DGP as origin (public2) > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59006 > > 2>curl.out']) > > > + > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > > curl.out | \ > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +# Check if we have only one backend for the same connection - orig + > > dest ports > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Check if gw-1 is empty to ensure that the traffic only come from/to > > the originator chassis via DGP public2 > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > +0 > > > +]) > > > + > > > +# Check the backend IP from ct entries on gw-2 (DGP public2) > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > + > > > +if [[ $backend_check -gt 0 ]]; then > > > +# Backend resides on ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59006,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +else > > > +# Backend resides on ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59006,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +fi > > > + > > > +# Flush conntrack entries for easier output parsing of next test. > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +# Check the flows again for a new source port using the second DGP as > > origin (public2) > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59007 > > 2>curl.out']) > > > + > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > ':a;N;$!ba;s/\n/\\n/g') > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > > curl.out | \ > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +* Closing connection > > > +]) > > > + > > > +# Check if we have only one backend for the same connection - orig + > > dest ports > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Check if gw-1 is empty to ensure that the traffic only come from/to > > the originator chassis via DGP public2 > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > +0 > > > +]) > > > + > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" -c) > > > + > > > +if [[ $backend_check -gt 0 ]]; then > > > +# Backend resides on ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59007,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +else > > > +# Backend resides on ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp], [0], [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59007,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" -c], > > [1], [dnl > > > +0 > > > +]) > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" > > -c], [1], [dnl > > > +0 > > > +]) > > > +fi > > > + > > > +# Check multiple requests coming from DGP's public1 and public2 > > > + > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +# Remove the LB and change the VIP port - different from the backend > > ports > > > +check multinode_nbctl lb-del lb0 > > > + > > > +# create LB again > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:9000" "10.0.0.3:80, > > 10.0.0.4:80" > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > + > > > +# Set use_stateless_nat to true > > > +check multinode_nbctl set load_balancer lb0 > > options:use_stateless_nat=true > > > + > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +# Check end-to-end request using a new port for VIP > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port 59008 > > 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | > > M_FORMAT_CT(20.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > + > > > +# Check end-to-end request using a new port for VIP > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port 59008 > > 2>curl.out']) > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | > > M_FORMAT_CT(30.0.0.3) | \ > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], > > [dnl > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > +]) > > > + > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep > > -v 'Server:'], [0], [dnl > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > +200 OK > > > +* Closing connection > > > +]) > > > + > > > +AT_CLEANUP > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > > > index dcc3dbbc3..9e7a2f225 100644 > > > --- a/tests/ovn-northd.at > > > +++ b/tests/ovn-northd.at > > > @@ -13864,3 +13864,323 @@ check_no_redirect > > > > > > AT_CLEANUP > > > ]) > > > + > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + NAT > > Stateless)]) > > > +ovn_start > > > + > > > +check ovn-nbctl ls-add public > > > +check ovn-nbctl lr-add lr1 > > > + > > > +# lr1 DGP ts1 > > > +check ovn-nbctl ls-add ts1 > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 172.16.10.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > + > > > +# lr1 DGP ts2 > > > +check ovn-nbctl ls-add ts2 > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 172.16.20.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > + > > > +# lr1 DGP public > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 173.16.0.1/16 > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > + > > > +check ovn-nbctl ls-add s1 > > > +# s1 - lr1 > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 172.16.0.1" > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > + > > > +# s1 - backend vm1 > > > +check ovn-nbctl lsp-add s1 vm1 > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 172.16.0.101" > > > + > > > +# s1 - backend vm2 > > > +check ovn-nbctl lsp-add s1 vm2 > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 172.16.0.102" > > > + > > > +# s1 - backend vm3 > > > +check ovn-nbctl lsp-add s1 vm3 > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 172.16.0.103" > > > + > > > +# Add the lr1 DGP ts1 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP ts2 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP public to the public switch > > > +check ovn-nbctl lsp-add public public_lr1 > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > nat-addresses=router > > > + > > > +# Create the Load Balancer lb1 > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > + > > > +# Set use_stateless_nat to true > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > options:use_stateless_nat=true > > > + > > > +# Associate load balancer to s1 > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows s1 > s1flows > > > +AT_CAPTURE_FILE([s1flows]) > > > + > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > +]) > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > ip4.dst == 30.0.0.1), > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > +]) > > > + > > > +# Associate load balancer to lr1 with DGP > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > +AT_CAPTURE_FILE([lr1flows]) > > > + > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1-ts1")), > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1-ts2")), > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1_public")), > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > +]) > > > + > > > +# 2. Check if the DGP ports are in the match with action next > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], [dnl > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") && > > is_chassis_resident("cr-lr1_public") && tcp), action=(next;) > > > +]) > > > + > > > +# 3. Check if the VIP IP is in the ipX.src action > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > action=(next;) > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > action=(next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(ip4.src=30.0.0.1; next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(ip4.src=30.0.0.1; next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") && > > is_chassis_resident("cr-lr1_public") && tcp), action=(ip4.src=30.0.0.1; > > next;) > > > +]) > > > + > > > +AT_CLEANUP > > > +]) > > > + > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + NAT > > Stateless) - IPv6]) > > > +ovn_start > > > + > > > +check ovn-nbctl ls-add public > > > +check ovn-nbctl lr-add lr1 > > > + > > > +# lr1 DGP ts1 > > > +check ovn-nbctl ls-add ts1 > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > > 2001:db8:aaaa:1::1/64 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > + > > > +# lr1 DGP ts2 > > > +check ovn-nbctl ls-add ts2 > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > > 2001:db8:aaaa:2::1/64 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > + > > > +# lr1 DGP public > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > > 2001:db8:bbbb::1/64 > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > > 2001:db8:aaaa:3::1/64 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > + > > > +check ovn-nbctl ls-add s1 > > > +# s1 - lr1 > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > > 2001:db8:aaaa:3::1" > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > + > > > +# s1 - backend vm1 > > > +check ovn-nbctl lsp-add s1 vm1 > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > > 2001:db8:aaaa:3::101" > > > + > > > +# s1 - backend vm2 > > > +check ovn-nbctl lsp-add s1 vm2 > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > > 2001:db8:aaaa:3::102" > > > + > > > +# s1 - backend vm3 > > > +check ovn-nbctl lsp-add s1 vm3 > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > > 2001:db8:aaaa:3::103" > > > + > > > +# Add the lr1 DGP ts1 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP ts2 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP public to the public switch > > > +check ovn-nbctl lsp-add public public_lr1 > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > nat-addresses=router > > > + > > > +# Create the Load Balancer lb1 > > > +check ovn-nbctl --wait=sb lb-add lb1 "2001:db8:cccc::1" > > "2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101" > > > + > > > +# Set use_stateless_nat to true > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > options:use_stateless_nat=true > > > + > > > +# Associate load balancer to s1 > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows s1 > s1flows > > > +AT_CAPTURE_FILE([s1flows]) > > > + > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep > > "2001:db8:cccc::1"], [0], [dnl > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 > > && ip6.dst == 2001:db8:cccc::1), action=(xxreg1 = 2001:db8:cccc::1; > > ct_lb_mark;) > > > +]) > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > "2001:db8:cccc::1"], [0], [dnl > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > ip6.dst == 2001:db8:cccc::1), > > action=(ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > +]) > > > + > > > +# Associate load balancer to lr1 with DGP > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > +AT_CAPTURE_FILE([lr1flows]) > > > + > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > "2001:db8:cccc::1"], [0], [dnl > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > is_chassis_resident("cr-lr1-ts1")), > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > is_chassis_resident("cr-lr1-ts2")), > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > is_chassis_resident("cr-lr1_public")), > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > +]) > > > + > > > +# 2. Check if the DGP ports are in the match with action next > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], [dnl > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport == > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport == > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || outport == > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > action=(next;) > > > +]) > > > + > > > +# 3. Check if the VIP IP is in the ipX.src action > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > action=(next;) > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > action=(next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport == > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), > > action=(ip6.src=2001:db8:cccc::1; next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport == > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), > > action=(ip6.src=2001:db8:cccc::1; next;) > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || outport == > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > action=(ip6.src=2001:db8:cccc::1; next;) > > > +]) > > > + > > > +AT_CLEANUP > > > +]) > > > + > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (DGP)]) > > > +ovn_start > > > + > > > +check ovn-nbctl ls-add public > > > +check ovn-nbctl lr-add lr1 > > > + > > > +# lr1 DGP ts1 > > > +check ovn-nbctl ls-add ts1 > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 172.16.10.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-1 > > > + > > > +# lr1 DGP ts2 > > > +check ovn-nbctl ls-add ts2 > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 172.16.20.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-1 > > > + > > > +# lr1 DGP public > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 173.16.0.1/16 > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > + > > > +check ovn-nbctl ls-add s1 > > > +# s1 - lr1 > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 172.16.0.1" > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > + > > > +# s1 - backend vm1 > > > +check ovn-nbctl lsp-add s1 vm1 > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 172.16.0.101" > > > + > > > +# s1 - backend vm2 > > > +check ovn-nbctl lsp-add s1 vm2 > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 172.16.0.102" > > > + > > > +# s1 - backend vm3 > > > +check ovn-nbctl lsp-add s1 vm3 > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 172.16.0.103" > > > + > > > +# Add the lr1 DGP ts1 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP ts2 to the public switch > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > nat-addresses=router > > > + > > > +# Add the lr1 DGP public to the public switch > > > +check ovn-nbctl lsp-add public public_lr1 > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > nat-addresses=router > > > + > > > +# Create the Load Balancer lb1 > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > + > > > +# Associate load balancer to s1 > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows s1 > s1flows > > > +AT_CAPTURE_FILE([s1flows]) > > > + > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > +]) > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > ip4.dst == 30.0.0.1), > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > +]) > > > + > > > +# Associate load balancer to lr1 with DGP > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > +check ovn-nbctl --wait=sb sync > > > + > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > +AT_CAPTURE_FILE([lr1flows]) > > > + > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > "30.0.0.1"], [0], [dnl > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1-ts1")), > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1-ts2")), > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > is_chassis_resident("cr-lr1_public")), > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > +]) > > > + > > > +AT_CLEANUP > > > +]) > > > -- > > > 2.34.1 > > > > > > > > > -- > > > > > > > > > > > > > > > _'Esta mensagem é direcionada apenas para os endereços constantes no > > > cabeçalho inicial. Se você não está listado nos endereços constantes no > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas > > estão > > > imediatamente anuladas e proibidas'._ > > > > > > > > > * **'Apesar do Magazine Luiza tomar > > > todas as precauções razoáveis para assegurar que nenhum vírus esteja > > > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade > > por > > > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos'.* > > > > > > > > > > > > _______________________________________________ > > > dev mailing list > > > dev@openvswitch.org > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > > -- > > > > > _‘Esta mensagem é direcionada apenas para os endereços constantes no > cabeçalho inicial. Se você não está listado nos endereços constantes no > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão > imediatamente anuladas e proibidas’._ > > > * **‘Apesar do Magazine Luiza tomar > todas as precauções razoáveis para assegurar que nenhum vírus esteja > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* > > > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Hi Numan, Em sex., 27 de set. de 2024 às 12:47, Numan Siddique <numans@ovn.org> escreveu: > On Thu, Sep 26, 2024 at 10:55 AM Roberto Bartzen Acosta via dev > <ovs-dev@openvswitch.org> wrote: > > > > Hi Numan, > > > > Thanks for your feedback and review. > > > > Em qua., 25 de set. de 2024 às 19:50, Numan Siddique <numans@ovn.org> > > escreveu: > > > > > On Thu, Sep 19, 2024 at 6:12 PM Roberto Bartzen Acosta via dev > > > <ovs-dev@openvswitch.org> wrote: > > > > > > > > This commit fixes the build_distr_lrouter_nat_flows_for_lb function > to > > > > include a DNAT flow entry for each DGP in use. Since we have added > > > support > > > > to create multiple gateway ports per logical router, it's necessary > to > > > > include in the LR NAT rules pipeline a specific entry for each > attached > > > DGP. > > > > Otherwise, the inbound traffic will only be redirected when the > incoming > > > LRP > > > > matches the chassis_resident field. > > > > > > > > Additionally, this patch includes the ability to use load-balancer > with > > > DGPs > > > > attached to multiple chassis. We can have each of the DGPs associated > > > with a > > > > different chassis, and in this case the DNAT rules added by default > will > > > not > > > > be enough to guarantee outgoing traffic. > > > > > > > > To solve the multiple chassis for DGPs problem, this patch include a > new > > > > config options to be configured in the load-balancer. If the > > > use_stateless_nat > > > > is set to true, the logical router that references this load-balancer > > > will use > > > > Stateless NAT rules when the logical router has multiple DGPs. After > > > applying > > > > this patch and setting the use_stateless_nat option, the inbound > and/or > > > > outbound traffic can pass through any chassis where the DGP resides > > > without > > > > having problems with CT state. > > > > > > > > Reported-at: > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2054322 > > > > Fixes: 15348b7b806f ("ovn-northd: Multiple distributed gateway port > > > support.") > > > > > > > > Signed-off-by: Roberto Bartzen Acosta <roberto.acosta@luizalabs.com> > > > > > > Hi Roberto, > > > > > > Thanks for the patch. I tested this patch using the test example in > > > multinode.at. > > > > > > The test case adds the below load balancer > > > > > > [root@ovn-central ~]# ovn-nbctl lb-list > > > UUID LB PROTO > > > VIP IPs > > > f3e29869-3bb5-4df0-960a-171106f5913a lb0 tcp > > > 172.16.0.100:9000 10.0.0.3:80,10.0.0.4:80 > > > > > > And the below logical flows are generated by this patch > > > > > > -------- > > > [root@ovn-central ~]# ovn-sbctl dump-flows lr0 | grep 172.16.0.100 > > > table=6 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst > > > == 172.16.0.100), action=(ct_dnat;) > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > > is_chassis_resident("cr-lr0-public-p1")), > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > 10.0.0.3:80, > > > 10.0.0.4:80);) > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > > is_chassis_resident("cr-lr0-public-p2")), > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > 10.0.0.3:80, > > > 10.0.0.4:80);) > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > > tcp.src == 80)) && (inport == "lr0-public-p1" || outport == > > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > > tcp.src == 80)) && (inport == "lr0-public-p2" || outport == > > > "lr0-public-p2") && is_chassis_resident("cr-lr0-public-p2") && tcp), > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > > -------------- > > > > > > > > > I fail to understand the reason for modifying the ip4.dst before > > > calling ct_lb_mark. Can you please explain why ? Because the > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;) will first modify the > > > ip4.dst to 10.0.0.3 and > > > then to 10.0.0.4 and then the ct_lb_mark will actually do the > > > conntrack with NAT to either 10.0.0.3 or 10.0.0.4. > > > > > > Is it because you want the conntrack entry to not have 172.16.0.100 ? > > > > > > > > The only reason I included this ip4.dst action in the DNAT rule is > because > > it's required to accept packets coming from a chassis that doesn't have > > previously created conntrack entries. The main feature introduced in this > > patch is to allow the administrator to have multiple DGPs attached to > > different chassis (is_chassis_resident...). So, my implementation was > based > > on the normal behavior when using stateless NAT for external addresses, > > where we need to add the ipx.dst in lr_in_dnat for traffic to be received > > on the chassis (put the DGP port as chassis_resident match, as is the > case > > with stateless NAT [1] with DGP[2]). > > > > The question is, if we only have the ct_lb_mark, packets that pass > through > > the chassis and are already part of an active flow in another chassis > (same > > IPs and Ports) will be dropped because there is no correspondence in the > > backend. So only packets with the NEW flag will be accepted and sent to > the > > backend (at least for TCP traffic). If we only have the ip4.dst action, > > this will always perform the dnat for the same backend, without > balancing. > > Therefore, the combination of the two actions allows the packet to always > > be received (regardless of whether conntrack is active for it), and > > ct_lb_mark will take care of balancing for different backends. > > > > If we had conntrack sync between different chassis this would not be > > necessary, as the ct_lb_mark action could always be executed without > > dropping packets due to lack of correspondence in the conntrack table. > > > > [1] > > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15737 > > [2] > > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15726 > > > > > > I'm sorry, but it's not 100% clear to me. I know that you've already > explained to Mark in the older version of this patch. > Can you please explain with an example ? > > Let's take the below topology you've added in the mutlinode test as example > > # Network topology > # > # publicp1 (ovn-chassis-3) (20.0.0.3/24) > # | > # overlay > # | > # DGP public1 (ovn-gw-1) (20.0.0.1/24) > # | > # | > # | > # lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) 10.0.0.3/24 > # | | > # | + --- sw0p2 (ovn-chassis-2) 10.0.0.4/24 > # | > # DGP public2 (ovn-gw-2) (30.0.0.1/24) > # | > # overlay > # | > # publicp2 (ovn-chassis-4) (30.0.0.3/24) > > > load balancer is configured on lr0 -> ovn-nbctl lb-add lb0 > "172.16.0.100:80" "10.0.0.3:80,10.0.0.4:80" > and it is attached to both lr0 and sw0. > > Scenario 1: > > publicp1 with IP 20.0.0.3 sends TCP traffic to VIP 172.16.0.100. I > think this is what will happen > > - The packet (ip4.src = 20.0.0.3 , ip4.dst = 172.16.0.100, tcp.dst > = 80) from ovn-chassis-3 will be sent out via the localnet bridge > (br-ex) > and the packet is received on ovn-gw-1 (as DGP public1 is resident > on it) via the localnet bridge and the > packet first enters public logical switch pipeline and then the > lr0 router pipeline. > > - In the router's lr_in_dnat state, the packet will be load > balanced to one of the backends using ct_lb_mark. Lets say 10.0.0.3 > is chosen > - The packet from router pipeline lr0 enters sw0 switch pipeline and > then the packet is tunneled to ovn-chassis-1 and delivered to sw0p1. > - The reply packet (ip4.src = 10.0.0.3, ip4.dst = 20.0.0.3, tcp.src = > 80) will enter the router pipeline and since 20.0.0.0/24 is handled by > DGP public1, > the packet is tunnelled to ovn-gw-1. > - In the router pipeline of ovn-gw-1, the packet is undnatted from > ip4.src 10.0.0.3 to 172.16.0.100 and the packet is sent out via the > localnet bridge. > - ovn-chassis-3 receives the packet via the localnet bridge and into > br-int and finally to publicp1. > > > In this scenario the load balancing is handled and conntrack entries > are created in ovn-gw-1. And there is no need to add flows in > "lr_out_snat" for stateless NAT > or set ip4.dst to one or all of backend IPs before "ct_lb_mark" in > lr_in_dnat stage. > > Scenario 2: > publicp2 with IP 30.0.0.3 sends TCP traffic to VIP 172.16.0.100. > > This is similar to scenario 1. Except that load balancing happens in > ovn-gw-2. > since DGP public2 is on this chassis. > > > Scenario 3: > > An external entity with IP 20.0.0.50 sends TCP traffic to VIP > 172.16.0.100. > > - This scenario is similar to the first one. The packet from this > external entity is received on ovn-gw-1 via the localnet bridge. > Rest all is the same. > > Scenario 4: > > sw0p2 on ovn-chassis-2 sends TCP traffic to VIP 172.16.0.100. > > - Since sw0 is attached with load balancer lb0, load balancing > happens in the source chassis - ovn-chassis-2 itself and depending on > the backend chosen, > the packet is tunnelled to ovn-chassis-1 (if 10.0.0.3 is chosen) > or delivered directly to sw0p2 (if i0.0.0.4 is chosen). > > > Scenario 5: > > An external entity with IP 40.0.0.40 sends TCP traffic to VIP > 172.16.0.100 and there are 2 ECMP routes configured > 172.16.0.100 via 20.0.0.1 > 172.16.0.100 via 30.0.0.1 > > In this case if the packet uses the route via 20.0.0.1 it will be > received on ovn-gw-1 via the localnet bridge br-ex. > > - With your patch, ip4.dst is modified to the last backend in the > list and then ct_lb_mark chosen as one of the backends > - And then it is tunnelled to the destination chassis. > > > Is this the scenario 5 you're trying to address with the stateless > NAT ? How would the reply packet work ? > The reply packet from backend 10.0.0.3 can use either of the paths ? > i.e ovn-gw-1 or ovn-gw-2 ? > Yes, this scenario would be the main goal of this patch. Let me explain with an example related to the OVN interconnect (one of the use cases of this patch). In the context of ovn-ic the backend 10.0.0.3 could reply packets using either of the paths (ovn-gw-1 or ovn-gw-2). LB VIP 172.16.0.100 | vm (40.0.0.40) - LR-EXT - lrp1 - chassis1 - Transit switch TS1 - ovn-gw-1 - lr0-public-p1 - lr0 - sw0 - VM 10.0.0.3 (ovn-chassis-1) - lrp2 - chassis2 - Transit switch TS2 - ovn-gw-2 - lr0-public-p2 - - VM 10.0.0.4 (ovn-chassis-2) Let's take an example of the ovn-ic in the above topology: Assumptions: A) logical router LR-EXT: route to 172.16.0.100/32 via ECMP (lrp1/chassis1 and lrp2/chassis2) B) logical router lr0: route to 40.0.0.0/24 via ECMP (lr0-public-p1/ovn-gw-1 and lr0-public-p2/ovn-gw-2) 1 - VM 40.0.0.40 send a request to the LB 172.16.0.100:80 2 - LR-EXT chooses one of the possible outgoing routes via ECMP - e.g. lrp1 / chassis1 / TS1 3 - The lr0 receives the ingoing traffic through the port lr0-public-p1 / ovn-gw-1 because the TS1 is used to start the traffic 4 - with this patch we added lr_in_dnat rules for all DGPs (with or without the stateless NAT config flag) with the stateless NAT config flag True: ...is_chassis_resident("cr-lr0-public-p1")), action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80 ,10.0.0.4:80);) ...is_chassis_resident("cr-lr0-public-p2")), action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80 ,10.0.0.4:80);) 5 - The ip4.dst will perform the action cumulatively, basically the last ip4.dst action applied will be to change dst to 10.0.0.4. 5.1 - The ct_lb_mark will check the conntrack and validate if the traffic refers to a new one (TCP SYNC), and then, it executes the final action provided by ct_lb_mark and select one of the available backends. e.g 10.0.0.3 without the stateless NAT config flag: ...is_chassis_resident("cr-lr0-public-p1")), action=(ct_lb_mark(backends= 10.0.0.3:80,10.0.0.4:80);) ...is_chassis_resident("cr-lr0-public-p2")), action=(ct_lb_mark(backends= 10.0.0.3:80,10.0.0.4:80);) 5 - same as the step 5.1 (ct_lb_mark check only). 6 - VM 10.0.0.3 receives the new traffic and sends a reply (TCP SYN+ACK) 7 - lr0 receives the response packet (from VM 10.0.0.3) in the routing pipeline and route via one of the available paths (ECMP). For example: lr0-public-p2 / ovn-gw-2 / TS2 So, we have a different outgoing path of the original one. 8 - ovn-gw-2 receives (SYN+ACK). We have no conntrack entries on ovn-gw-2 to match and perform the LB SNAT action! The ct_lb_mark previously creates the conntrack entry on chassis ovn-gw-1 (when receives TCP SYN). with the stateless NAT config flag True: The outgoing packet fail to match the lb conntrack but the packet is SNATed by the lr_out_snat rule with a priority lower then cl_lb_mark, executing after cl_lb_mark in the pipeline. 9 - The SYN+ACK packet crosses the TS2 and is delivered to VM 40.0.0.40 via lrp2 (SRC = 172.16.0.100:80) lr_out_snat will match and SNATed: match=(ip4 && ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && tcp.src == 80)) && (inport == "lr0-public-p1" || outport == "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), action=(ip4.src=172.16.0.100; tcp.src=80; next;) without the stateless NAT config flag: The outgoing will be returned to the VM 40.0.0.4 with the LR SNAT for public-p2 port as ip4.src! So, basically this will break the TCP handshake!!! 9 - The SYN+ACK packet cross the TS2 and is delivered to the VM 40.0.0.40 via lrp2 (SRC = 30.0.0.1:80) Moving forward on the happy path (stateless NAT is true): 10 - VM 40.0.0.40 sends an ACK to complete the TCP handshake. 11 - 2 - LR-EXT chooses one of the possible outgoing routes via ECMP Let's assume that LR-EXT correctly execute the ecmp algorithm (per flow basis) and forwards to the same initial path because we're using the same flow (src/dst IPs and src/dts TCP ports still the same). e.g. still using the lrp1 / chassis1 / TS1 12 - The lr0 receive the ingoing traffic through the port lr0-public-p1 / ovn-gw-1 / TS1 With this patch we have 2 possible behaviours: 12.1 - Without Stateless NAT config flag table=1 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && tcp.src == 80)) && (inport == "lr0-public-p1" || outport == "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1")), action=(ct_dnat_in_czone;) table=8 (lr_in_dnat ), priority=120 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && is_chassis_resident("cr-lr0-public-p1")), action=(ct_lb_mark(backends= 10.0.0.3:80,10.0.0.4:80);) openflow: br-int (ovn-gw-1) cookie=0x9381c20e, duration=2361.675s, table=15, n_packets=2, n_bytes=148, idle_age=1564, priority=120,ct_state=+new-rel+trk,tcp,metadata=0x4,nw_dst=172.16.0.100,tp_dst=80 actions=group:3 ovs-appctl dpctl/dump-conntrack (ovn-gw-1) tcp,orig=(src=40.0.0.40,dst=172.16.0.100,sport=35274,dport=80),reply=(src=10.0.0.3,dst=40.0.0.40,sport=80,dport=35274),zone=8,mark=2,protoinfo=(state=TIME_WAIT) At this point, the ACK packet will be discarded because we didn't perform the SYN+ACK return on this chassis (ovn-gw -1). So, without the previously established conntrack to match the ACK packet we broke the TCP handshake. Remember the flow: -> SYN: ovn-gw-1 <- SYN+ACK: ovn-gw-2 -> ACK: ovn-gw-1 12.2 - With stateless NAT config flag is True table=1 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && tcp.src == 80)) && (inport == "lr0-public-p1" || outport == "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1")), action=(next;) table=8 (lr_in_dnat ), priority=120 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && is_chassis_resident("cr-lr0-public-p1")), action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80 ,10.0.0.4:80);) openflow: br-int (ovn-gw-1) cookie=0xab99f240, duration=1086.022s, table=15, n_packets=610, n_bytes=48979, idle_age=825, priority=120,ct_state=+new-rel+trk,tcp,metadata=0x2,nw_dst=172.16.0.100,tp_dst=80 actions=mod_nw_dst:10.0.0.3,mod_nw_dst:10.0.0.4,group:1 ovs-appctl dpctl/dump-conntrack (ovn-gw-1) tcp,orig=(src=40.0.0.40,dst=10.0.0.3,sport=54632,dport=80),reply=(src=10.0.0.3,dst=40.0.0.40,sport=80,dport=54632),zone=7,mark=2,protoinfo=(state=SYN_SENT) This is the reason to use stateless NAT rules. Unlike the case where we don't have the SYN_SENT, now the ACK packet will be accepted and forwarded after performing the mod_nw_dst action. How does this work? We already have the previous flow created by the first packet (SYN), so regardless of the IP.dst modified in the action, the packet will be forwarded to the same backend that corresponds to the conntrack match that is in SYN_SENT state. That's why I created the action with the two modifiers (ip4.dst + ct_lb_mark). Using only the ct_lb_mark action creates a strong dependency on the connection state match in the conntrack entry (which we don't have in these cases). However, only using ip4.dst doesn't create traffic balancing as it will always send to the same backend (last in the ip4.dst). I hope this has helped clarify the design decisions when creating/modifying flows (lr_in_dnat/lr_out_snat/lr_out_undnat). > > Thanks > Numan > > > > > > > > Also I don't understand why this patch adds the logical flows in > > > "lr_out_snat" stage ? > > > > > > > The flow for lr_out_snat is necessary for the correct functioning of > > stateless NAT for the same reason explained previously. I mean, if the > > outgoing packet is redirected to a chassis that doesn't have an active > > conntrack entry, it will not be NATed by ct_lb action because it doesn't > > refer to a valid flow (use case with ecmp). > > > > So it is necessary to create a stateless SNAT rule (similar to this [3]) > > with a lower priority than the other router pipeline entries, in this > case, > > if the packet is not SNATed by ct_lb (conntrack missed) it will be SNATed > > by stateless NAT rule. > > > > [3] > > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15884 > > > > > > > > > > > > Using the system multinode test as an example, the below fails > > > (which is a regression) > > > > > > --- > > > root@ovn-chassis-3 ~]# ip netns exec publicp1 nc -vz 10.0.0.3 80 > > > ---- > > > > > > In the above test, publicp1 with IP 20.0.0.3 when it tries to connect > > > to one if the backends directly (without the LB VIP), it fails. > > > It fails because of the logical flows in "lr_out_snat". > > > > > > > > > Looks to me the solution proposed here is incomplete. > > > > > > Also please note that in our CI we run the multinode tests > > > periodically once a day using the v0.1 of the ovn-fake-multinode > > > and the tests you added will fail. This needs to be fixed and until > > > we move to the latest version of ovn-fake-multinode. > > > > > > > I imagine that the test you are doing is using the same port as the LB > > backend (TCP 80 in this case). So, the stateless lr_out_snat flow will > > force the output to be SNATed because this port is in use by the backend. > > Traffic to/from other ports will work without problems and will follow > the > > normal programmed flows (e.g. ICMP). > > > > This is necessary to ensure the egress traffic because the DGPs are > > distributed across multiple chassis. Also, this setup is being validated > in > > the test ovn-fake-multinode testcase (ICMP from the backends chassis use > > the router's default SNAT and not the LB's). I didn't understand the > > regression you mentioned because this was programmed to be stateless and > > it's traffic that uses the same ports as the LB backend, could you > explain > > better? > > > > Thanks, > > Roberto > > > > > > > Thanks > > > Numan > > > > > > > > > > --- > > > > northd/en-lr-stateful.c | 12 - > > > > northd/northd.c | 116 ++++++-- > > > > ovn-nb.xml | 10 + > > > > tests/multinode-macros.at | 40 +++ > > > > tests/multinode.at | 556 > ++++++++++++++++++++++++++++++++++++++ > > > > tests/ovn-northd.at | 320 ++++++++++++++++++++++ > > > > 6 files changed, 1017 insertions(+), 37 deletions(-) > > > > > > > > diff --git a/northd/en-lr-stateful.c b/northd/en-lr-stateful.c > > > > index baf1bd2f8..f09691af6 100644 > > > > --- a/northd/en-lr-stateful.c > > > > +++ b/northd/en-lr-stateful.c > > > > @@ -516,18 +516,6 @@ lr_stateful_record_create(struct > lr_stateful_table > > > *table, > > > > > > > > table->array[od->index] = lr_stateful_rec; > > > > > > > > - /* Load balancers are not supported (yet) if a logical router > has > > > multiple > > > > - * distributed gateway port. Log a warning. */ > > > > - if (lr_stateful_rec->has_lb_vip && > lr_has_multiple_gw_ports(od)) { > > > > - static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, > 1); > > > > - VLOG_WARN_RL(&rl, "Load-balancers are configured on logical > " > > > > - "router %s, which has %"PRIuSIZE" distributed " > > > > - "gateway ports. Load-balancer is not supported > " > > > > - "yet when there is more than one distributed " > > > > - "gateway port on the router.", > > > > - od->nbr->name, od->n_l3dgw_ports); > > > > - } > > > > - > > > > return lr_stateful_rec; > > > > } > > > > > > > > diff --git a/northd/northd.c b/northd/northd.c > > > > index a267cd5f8..bbe97acf8 100644 > > > > --- a/northd/northd.c > > > > +++ b/northd/northd.c > > > > @@ -11807,31 +11807,30 @@ static void > > > > build_distr_lrouter_nat_flows_for_lb(struct lrouter_nat_lb_flows_ctx > > > *ctx, > > > > enum lrouter_nat_lb_flow_type > type, > > > > struct ovn_datapath *od, > > > > - struct lflow_ref *lflow_ref) > > > > + struct lflow_ref *lflow_ref, > > > > + struct ovn_port *dgp, > > > > + bool stateless_nat) > > > > { > > > > - struct ovn_port *dgp = od->l3dgw_ports[0]; > > > > - > > > > - const char *undnat_action; > > > > - > > > > - switch (type) { > > > > - case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > > - undnat_action = "flags.force_snat_for_lb = 1; next;"; > > > > - break; > > > > - case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > > - undnat_action = "flags.skip_snat_for_lb = 1; next;"; > > > > - break; > > > > - case LROUTER_NAT_LB_FLOW_NORMAL: > > > > - case LROUTER_NAT_LB_FLOW_MAX: > > > > - undnat_action = lrouter_use_common_zone(od) > > > > - ? "ct_dnat_in_czone;" > > > > - : "ct_dnat;"; > > > > - break; > > > > - } > > > > + struct ds dnat_action = DS_EMPTY_INITIALIZER; > > > > > > > > /* Store the match lengths, so we can reuse the ds buffer. */ > > > > size_t new_match_len = ctx->new_match->length; > > > > size_t undnat_match_len = ctx->undnat_match->length; > > > > > > > > + /* dnat_action: Add the LB backend IPs as a destination action > of > > > the > > > > + * lr_in_dnat NAT rule with cumulative effect > because > > > any > > > > + * backend dst IP used in the action list will > > > redirect the > > > > + * packet to the ct_lb pipeline. > > > > + */ > > > > + if (stateless_nat) { > > > > + for (size_t i = 0; i < ctx->lb_vip->n_backends; i++) { > > > > + struct ovn_lb_backend *backend = > &ctx->lb_vip->backends[i]; > > > > + bool ipv6 = !IN6_IS_ADDR_V4MAPPED(&backend->ip); > > > > + ds_put_format(&dnat_action, "%s.dst=%s;", ipv6 ? "ip6" : > > > "ip4", > > > > + backend->ip_str); > > > > + } > > > > + } > > > > + ds_put_format(&dnat_action, "%s", ctx->new_action[type]); > > > > > > > > const char *meter = NULL; > > > > > > > > @@ -11841,20 +11840,46 @@ build_distr_lrouter_nat_flows_for_lb(struct > > > lrouter_nat_lb_flows_ctx *ctx, > > > > > > > > if (ctx->lb_vip->n_backends || !ctx->lb_vip->empty_backend_rej) > { > > > > ds_put_format(ctx->new_match, " && is_chassis_resident(%s)", > > > > - od->l3dgw_ports[0]->cr_port->json_key); > > > > + dgp->cr_port->json_key); > > > > } > > > > > > > > ovn_lflow_add_with_hint__(ctx->lflows, od, S_ROUTER_IN_DNAT, > > > ctx->prio, > > > > - ds_cstr(ctx->new_match), > > > ctx->new_action[type], > > > > + ds_cstr(ctx->new_match), > > > ds_cstr(&dnat_action), > > > > NULL, meter, &ctx->lb->nlb->header_, > > > > lflow_ref); > > > > > > > > ds_truncate(ctx->new_match, new_match_len); > > > > > > > > + ds_destroy(&dnat_action); > > > > if (!ctx->lb_vip->n_backends) { > > > > return; > > > > } > > > > > > > > + struct ds undnat_action = DS_EMPTY_INITIALIZER; > > > > + struct ds snat_action = DS_EMPTY_INITIALIZER; > > > > + > > > > + switch (type) { > > > > + case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > > + ds_put_format(&undnat_action, "flags.force_snat_for_lb = 1; > > > next;"); > > > > + break; > > > > + case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > > + ds_put_format(&undnat_action, "flags.skip_snat_for_lb = 1; > > > next;"); > > > > + break; > > > > + case LROUTER_NAT_LB_FLOW_NORMAL: > > > > + case LROUTER_NAT_LB_FLOW_MAX: > > > > + ds_put_format(&undnat_action, "%s", > > > > + lrouter_use_common_zone(od) ? > "ct_dnat_in_czone;" > > > > + : "ct_dnat;"); > > > > + break; > > > > + } > > > > + > > > > + /* undnat_action: Remove the ct action from the lr_out_undenat > NAT > > > rule. > > > > + */ > > > > + if (stateless_nat) { > > > > + ds_clear(&undnat_action); > > > > + ds_put_format(&undnat_action, "next;"); > > > > + } > > > > + > > > > /* We need to centralize the LB traffic to properly perform > > > > * the undnat stage. > > > > */ > > > > @@ -11873,11 +11898,41 @@ build_distr_lrouter_nat_flows_for_lb(struct > > > lrouter_nat_lb_flows_ctx *ctx, > > > > ds_put_format(ctx->undnat_match, ") && (inport == %s || outport > == > > > %s)" > > > > " && is_chassis_resident(%s)", dgp->json_key, > > > dgp->json_key, > > > > dgp->cr_port->json_key); > > > > + /* Use the LB protocol as matching criteria for out undnat and > snat > > > when > > > > + * creating LBs with stateless NAT. */ > > > > + if (stateless_nat) { > > > > + ds_put_format(ctx->undnat_match, " && %s", ctx->lb->proto); > > > > + } > > > > ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_UNDNAT, > 120, > > > > - ds_cstr(ctx->undnat_match), > undnat_action, > > > > - &ctx->lb->nlb->header_, > > > > + ds_cstr(ctx->undnat_match), > > > > + ds_cstr(&undnat_action), > > > &ctx->lb->nlb->header_, > > > > lflow_ref); > > > > + > > > > + /* snat_action: Add a new lr_out_snat rule with the LB VIP as > > > source IP > > > > + * action to perform the NAT stateless pipeline > > > completely. > > > > + */ > > > > + if (stateless_nat) { > > > > + if (ctx->lb_vip->port_str) { > > > > + ds_put_format(&snat_action, "%s.src=%s; %s.src=%s; > next;", > > > > + ctx->lb_vip->address_family == AF_INET6 ? > > > > + "ip6" : "ip4", > > > > + ctx->lb_vip->vip_str, ctx->lb->proto, > > > > + ctx->lb_vip->port_str); > > > > + } else { > > > > + ds_put_format(&snat_action, "%s.src=%s; next;", > > > > + ctx->lb_vip->address_family == AF_INET6 ? > > > > + "ip6" : "ip4", > > > > + ctx->lb_vip->vip_str); > > > > + } > > > > + ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_SNAT, > 160, > > > > + ds_cstr(ctx->undnat_match), > > > > + ds_cstr(&snat_action), > > > &ctx->lb->nlb->header_, > > > > + lflow_ref); > > > > + } > > > > + > > > > ds_truncate(ctx->undnat_match, undnat_match_len); > > > > + ds_destroy(&undnat_action); > > > > + ds_destroy(&snat_action); > > > > } > > > > > > > > static void > > > > @@ -12022,6 +12077,8 @@ build_lrouter_nat_flows_for_lb( > > > > * lflow generation for them. > > > > */ > > > > size_t index; > > > > + bool use_stateless_nat = smap_get_bool(&lb->nlb->options, > > > > + "use_stateless_nat", > false); > > > > BITMAP_FOR_EACH_1 (index, bitmap_len, lb_dps->nb_lr_map) { > > > > struct ovn_datapath *od = lr_datapaths->array[index]; > > > > enum lrouter_nat_lb_flow_type type; > > > > @@ -12043,8 +12100,17 @@ build_lrouter_nat_flows_for_lb( > > > > if (!od->n_l3dgw_ports) { > > > > bitmap_set1(gw_dp_bitmap[type], index); > > > > } else { > > > > - build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > > > - lb_dps->lflow_ref); > > > > + /* Create stateless LB NAT rules when using multiple > DGPs > > > and > > > > + * use_stateless_nat is true. > > > > + */ > > > > + bool stateless_nat = (od->n_l3dgw_ports > 1) > > > > + ? use_stateless_nat : false; > > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > > > > + struct ovn_port *dgp = od->l3dgw_ports[i]; > > > > + build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > > > + > lb_dps->lflow_ref, > > > dgp, > > > > + stateless_nat); > > > > + } > > > > } > > > > > > > > if (lb->affinity_timeout) { > > > > diff --git a/ovn-nb.xml b/ovn-nb.xml > > > > index 2836f58f5..ad03c6214 100644 > > > > --- a/ovn-nb.xml > > > > +++ b/ovn-nb.xml > > > > @@ -2302,6 +2302,16 @@ or > > > > local anymore by the ovn-controller. This option is set to > > > > <code>false</code> by default. > > > > </column> > > > > + > > > > + <column name="options" key="use_stateless_nat" > > > > + type='{"type": "boolean"}'> > > > > + If the load balancer is configured with > > > <code>use_stateless_nat</code> > > > > + option to <code>true</code>, the logical router that > references > > > this > > > > + load balancer will use Stateless NAT rules when the logical > > > router > > > > + has multiple distributed gateway ports(DGP). Otherwise, the > > > outbound > > > > + traffic may be dropped in scenarios where we have different > > > chassis > > > > + for each DGP. This option is set to <code>false</code> by > > > default. > > > > + </column> > > > > </group> > > > > </table> > > > > > > > > diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at > > > > index 757917626..2f69433fc 100644 > > > > --- a/tests/multinode-macros.at > > > > +++ b/tests/multinode-macros.at > > > > @@ -40,6 +40,27 @@ m4_define([M_START_TCPDUMP], > > > > ] > > > > ) > > > > > > > > +# M_EXEC([fake_node], [command]) > > > > +# > > > > +# Execute 'command' in 'fakenode' > > > > +m4_define([M_EXEC], > > > > + [podman exec $1 $2]) > > > > + > > > > +# M_CHECK_EXEC([fake_node], [command], other_params...) > > > > +# > > > > +# Wrapper for AT_CHECK that executes 'command' inside > 'fake_node''s'. > > > > +# 'other_params' as passed as they are to AT_CHECK. > > > > +m4_define([M_CHECK_EXEC], > > > > + [ AT_CHECK([M_EXEC([$1], [$2])], > m4_shift(m4_shift(m4_shift($@)))) ] > > > > +) > > > > + > > > > +# M_FORMAT_CT([ip-addr]) > > > > +# > > > > +# Strip content from the piped input which would differ from test to > > > test > > > > +# and limit the output to the rows containing 'ip-addr'. > > > > +# > > > > +m4_define([M_FORMAT_CT], > > > > + [[grep -F "dst=$1," | sed -e 's/id=[0-9]*/id=<cleared>/g' -e > > > 's/state=[0-9_A-Z]*/state=<cleared>/g' | sort | uniq | sed -e > > > 's/zone=[[0-9]]*/zone=<cleared>/' -e 's/mark=[[0-9]]*/mark=<cleared>/' > ]]) > > > > > > > > OVS_START_SHELL_HELPERS > > > > > > > > @@ -76,6 +97,25 @@ multinode_nbctl () { > > > > m_as ovn-central ovn-nbctl "$@" > > > > } > > > > > > > > +check_fake_multinode_setup_by_nodes() { > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > > + for c in $1 > > > > + do > > > > + AT_CHECK([m_as $c ovn-appctl -t ovn-controller version], > [0], > > > [ignore]) > > > > + done > > > > +} > > > > + > > > > +cleanup_multinode_resources_by_nodes() { > > > > + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db > > > > + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > > + for c in $1 > > > > + do > > > > + m_as $c ovs-vsctl del-br br-int > > > > + m_as $c ip --all netns delete > > > > + done > > > > +} > > > > + > > > > # m_count_rows TABLE [CONDITION...] > > > > # > > > > # Prints the number of rows in TABLE (that satisfy CONDITION). > > > > diff --git a/tests/multinode.at b/tests/multinode.at > > > > index a0eb8fc67..b1beb4d97 100644 > > > > --- a/tests/multinode.at > > > > +++ b/tests/multinode.at > > > > @@ -1591,3 +1591,559 @@ AT_CHECK([cat ch1_eth2.tcpdump], [0], [dnl > > > > ]) > > > > > > > > AT_CLEANUP > > > > + > > > > +AT_SETUP([ovn multinode load-balancer with multiple DGPs and > multiple > > > chassis]) > > > > + > > > > +# Check that ovn-fake-multinode setup is up and running - requires > > > additional nodes > > > > +check_fake_multinode_setup_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > > + > > > > +# Delete the multinode NB and OVS resources before starting the > test. > > > > +cleanup_multinode_resources_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > > + > > > > +# Network topology > > > > +# > > > > +# publicp1 (ovn-chassis-3) (20.0.0.3/24) > > > > +# | > > > > +# overlay > > > > +# | > > > > +# DGP public1 (ovn-gw-1) (20.0.0.1/24) > > > > +# | > > > > +# | > > > > +# | > > > > +# lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) > 10.0.0.3/24 > > > > +# | | > > > > +# | + --- sw0p2 (ovn-chassis-2) > 10.0.0.4/24 > > > > +# | > > > > +# DGP public2 (ovn-gw-2) (30.0.0.1/24) > > > > +# | > > > > +# overlay > > > > +# | > > > > +# publicp2 (ovn-chassis-4) (30.0.0.3/24) > > > > + > > > > +# Delete already used ovs-ports > > > > +m_as ovn-chassis-1 ovs-vsctl del-port br-int sw0p1-p > > > > +m_as ovn-chassis-2 ovs-vsctl del-port br-int sw0p2-p > > > > +m_as ovn-chassis-1 ip link del sw0p1-p > > > > +m_as ovn-chassis-2 ip link del sw0p2-p > > > > +m_as ovn-chassis-3 ovs-vsctl del-port br-int publicp1-p > > > > +m_as ovn-chassis-4 ovs-vsctl del-port br-int publicp2-p > > > > +m_as ovn-chassis-3 ip link del publicp1-p > > > > +m_as ovn-chassis-4 ip link del publicp2-p > > > > + > > > > +# Create East-West switch for LB backends > > > > +check multinode_nbctl ls-add sw0 > > > > +check multinode_nbctl lsp-add sw0 sw0-port1 > > > > +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 > > > 10.0.0.3 1000::3" > > > > +check multinode_nbctl lsp-add sw0 sw0-port2 > > > > +check multinode_nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 > > > 10.0.0.4 1000::4" > > > > + > > > > +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 > > > 50:54:00:00:00:03 1400 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a > > > > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 > > > 50:54:00:00:00:04 1400 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a > > > > + > > > > +m_wait_for_ports_up > > > > + > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > > > 10.0.0.4 | FORMAT_PING], \ > > > > +[0], [dnl > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > +]) > > > > + > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 > > > 10.0.0.3 | FORMAT_PING], \ > > > > +[0], [dnl > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > +]) > > > > + > > > > +# Create a logical router and attach to sw0 > > > > +check multinode_nbctl lr-add lr0 > > > > +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 > 10.0.0.1/24 > > > 1000::a/64 > > > > +check multinode_nbctl lsp-add sw0 sw0-lr0 > > > > +check multinode_nbctl lsp-set-type sw0-lr0 router > > > > +check multinode_nbctl lsp-set-addresses sw0-lr0 router > > > > +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 > > > > + > > > > +# create external connection for N/S traffic using multiple DGPs > > > > +check multinode_nbctl ls-add public > > > > + > > > > +# DGP public1 > > > > +check multinode_nbctl lsp-add public ln-public-1 > > > > +check multinode_nbctl lsp-set-type ln-public-1 localnet > > > > +check multinode_nbctl lsp-set-addresses ln-public-1 unknown > > > > +check multinode_nbctl lsp-set-options ln-public-1 > network_name=public1 > > > > + > > > > +# DGP public2 > > > > +# create exteranl connection for N/S traffic > > > > +check multinode_nbctl lsp-add public ln-public-2 > > > > +check multinode_nbctl lsp-set-type ln-public-2 localnet > > > > +check multinode_nbctl lsp-set-addresses ln-public-2 unknown > > > > +check multinode_nbctl lsp-set-options ln-public-2 > network_name=public2 > > > > + > > > > +# Attach DGP public1 to GW-1 and chassis-3 (overlay connectivity) > > > > +m_as ovn-gw-1 ovs-vsctl set open . > > > external-ids:ovn-bridge-mappings=public1:br-ex > > > > +m_as ovn-chassis-3 ovs-vsctl set open . > > > external-ids:ovn-bridge-mappings=public1:br-ex > > > > + > > > > +# Attach DGP public2 to GW-2 and chassis-4 (overlay connectivity) > > > > +m_as ovn-gw-2 ovs-vsctl set open . > > > external-ids:ovn-bridge-mappings=public2:br-ex > > > > +m_as ovn-chassis-4 ovs-vsctl set open . > > > external-ids:ovn-bridge-mappings=public2:br-ex > > > > + > > > > +# Create the external LR0 port to the DGP public1 > > > > +check multinode_nbctl lsp-add public public-port1 > > > > +check multinode_nbctl lsp-set-addresses public-port1 > "40:54:00:00:00:03 > > > 20.0.0.3 2000::3" > > > > + > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p1 00:00:00:00:ff:02 > > > 20.0.0.1/24 2000::a/64 > > > > +check multinode_nbctl lsp-add public public-lr0-p1 > > > > +check multinode_nbctl lsp-set-type public-lr0-p1 router > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p1 router > > > > +check multinode_nbctl lsp-set-options public-lr0-p1 > > > router-port=lr0-public-p1 > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p1 > ovn-gw-1 10 > > > > + > > > > +# Create a VM on ovn-chassis-3 in the same public1 overlay > > > > +m_as ovn-chassis-3 /data/create_fake_vm.sh public-port1 publicp1 > > > 40:54:00:00:00:03 1400 20.0.0.3 24 20.0.0.1 2000::4/64 2000::a > > > > + > > > > +m_wait_for_ports_up public-port1 > > > > + > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [ping -q -c 3 -i 0.3 > -w 2 > > > 20.0.0.1 | FORMAT_PING], \ > > > > +[0], [dnl > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > +]) > > > > + > > > > +# Create the external LR0 port to the DGP public2 > > > > +check multinode_nbctl lsp-add public public-port2 > > > > +check multinode_nbctl lsp-set-addresses public-port2 > "60:54:00:00:00:03 > > > 30.0.0.3 3000::3" > > > > + > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p2 00:00:00:00:ff:03 > > > 30.0.0.1/24 3000::a/64 > > > > +check multinode_nbctl lsp-add public public-lr0-p2 > > > > +check multinode_nbctl lsp-set-type public-lr0-p2 router > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p2 router > > > > +check multinode_nbctl lsp-set-options public-lr0-p2 > > > router-port=lr0-public-p2 > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p2 > ovn-gw-2 10 > > > > + > > > > +# Create a VM on ovn-chassis-4 in the same public2 overlay > > > > +m_as ovn-chassis-4 /data/create_fake_vm.sh public-port2 publicp2 > > > 60:54:00:00:00:03 1400 30.0.0.3 24 30.0.0.1 3000::4/64 3000::a > > > > + > > > > +m_wait_for_ports_up public-port2 > > > > + > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [ping -q -c 3 -i 0.3 > -w 2 > > > 30.0.0.1 | FORMAT_PING], \ > > > > +[0], [dnl > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > +]) > > > > + > > > > +# Add a default route for multiple DGPs - using ECMP > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 > 20.0.0.3 > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 > 30.0.0.3 > > > > + > > > > +# Add SNAT rules using gateway-port > > > > +check multinode_nbctl --gateway-port lr0-public-p1 lr-nat-add lr0 > snat > > > 20.0.0.1 10.0.0.0/24 > > > > +check multinode_nbctl --gateway-port lr0-public-p2 lr-nat-add lr0 > snat > > > 30.0.0.1 10.0.0.0/24 > > > > + > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > > > 20.0.0.3 | FORMAT_PING], \ > > > > +[0], [dnl > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > +]) > > > > + > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 > > > 30.0.0.3 | FORMAT_PING], \ > > > > +[0], [dnl > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > +]) > > > > + > > > > +# create LB > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:80" "10.0.0.3:80, > > > 10.0.0.4:80" > > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > > + > > > > +# Set use_stateless_nat to true > > > > +check multinode_nbctl set load_balancer lb0 > > > options:use_stateless_nat=true > > > > + > > > > +# Start backend http services > > > > +M_NS_DAEMONIZE([ovn-chassis-1], [sw0p1], [$PYTHON -m http.server > --bind > > > 10.0.0.3 80 >/dev/null 2>&1], [http1.pid]) > > > > +M_NS_DAEMONIZE([ovn-chassis-2], [sw0p2], [$PYTHON -m http.server > --bind > > > 10.0.0.4 80 >/dev/null 2>&1], [http2.pid]) > > > > + > > > > +# wait for http server be ready > > > > +sleep 2 > > > > + > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v > 172.16.0.100:80 > > > --retry 3 --max-time 1 --local-port 59002 2> curl.out']) > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [sh -c 'cat curl.out | > > > grep -i -e connect | grep -v 'Server:''], \ > > > > +[0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +* Closing connection > > > > +]) > > > > + > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v > 172.16.0.100:80 > > > --retry 3 --max-time 1 --local-port 59003 2> curl.out']) > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [sh -c 'cat curl.out | > > > grep -i -e connect | grep -v 'Server:''], \ > > > > +[0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +* Closing connection > > > > +]) > > > > + > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v > 172.16.0.100:80 > > > --retry 3 --max-time 1 --local-port 59001']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | > > > M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > [dnl > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v > 172.16.0.100:80 > > > --retry 3 --max-time 1 --local-port 59000']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | > > > M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > [dnl > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# create a big file on web servers for download > > > > +M_NS_EXEC([ovn-chassis-1], [sw0p1], [dd bs=512 count=200000 > > > if=/dev/urandom of=download_file]) > > > > +M_NS_EXEC([ovn-chassis-2], [sw0p2], [dd bs=512 count=200000 > > > if=/dev/urandom of=download_file]) > > > > + > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > 59004 > > > 2>curl.out']) > > > > + > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | > sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | > sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > + > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > > > curl.out | \ > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +* Closing connection > > > > +]) > > > > + > > > > +# Check if we have only one backend for the same connection - orig + > > > dest ports > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > [dnl > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Check if gw-2 is empty to ensure that the traffic only come > from/to > > > the originator chassis via DGP public1 > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > > +0 > > > > +]) > > > > + > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" > -c) > > > > + > > > > +if [[ $backend_check -gt 0 ]]; then > > > > +# Backend resides on ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59004,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" > -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +else > > > > +# Backend resides on ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59004,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" > -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +fi > > > > + > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +# Check the flows again for a new source port > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > 59005 > > > 2>curl.out']) > > > > + > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | > sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | > sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > + > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > > > curl.out | \ > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +* Closing connection > > > > +]) > > > > + > > > > +# Check if we have only one backend for the same connection - orig + > > > dest ports > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > [dnl > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Check if gw-2 is empty to ensure that the traffic only come > from/to > > > the originator chassis via DGP public1 > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > > +0 > > > > +]) > > > > + > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" > -c) > > > > + > > > > +if [[ $backend_check -gt 0 ]]; then > > > > +# Backend resides on ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59005,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" > -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +else > > > > +# Backend resides on ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59005,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" > -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +fi > > > > + > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +# Start a new test using the second DGP as origin (public2) > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > 59006 > > > 2>curl.out']) > > > > + > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | > sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | > sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > + > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > > > curl.out | \ > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +* Closing connection > > > > +]) > > > > + > > > > +# Check if we have only one backend for the same connection - orig + > > > dest ports > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > [dnl > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Check if gw-1 is empty to ensure that the traffic only come > from/to > > > the originator chassis via DGP public2 > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > > +0 > > > > +]) > > > > + > > > > +# Check the backend IP from ct entries on gw-2 (DGP public2) > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" > -c) > > > > + > > > > +if [[ $backend_check -gt 0 ]]; then > > > > +# Backend resides on ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59006,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" > -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +else > > > > +# Backend resides on ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59006,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" > -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +fi > > > > + > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +# Check the flows again for a new source port using the second DGP > as > > > origin (public2) > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > 59007 > > > 2>curl.out']) > > > > + > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | > sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | > sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > ':a;N;$!ba;s/\n/\\n/g') > > > > + > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > > > curl.out | \ > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +* Closing connection > > > > +]) > > > > + > > > > +# Check if we have only one backend for the same connection - orig + > > > dest ports > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > [dnl > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Check if gw-1 is empty to ensure that the traffic only come > from/to > > > the originator chassis via DGP public2 > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > > +0 > > > > +]) > > > > + > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" > -c) > > > > + > > > > +if [[ $backend_check -gt 0 ]]; then > > > > +# Backend resides on ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59007,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" > -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +else > > > > +# Backend resides on ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp], [0], [dnl > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59007,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" > -c], > > > [1], [dnl > > > > +0 > > > > +]) > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" > > > -c], [1], [dnl > > > > +0 > > > > +]) > > > > +fi > > > > + > > > > +# Check multiple requests coming from DGP's public1 and public2 > > > > + > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > grep > > > -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +200 OK > > > > +* Closing connection > > > > +]) > > > > + > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > grep > > > -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +200 OK > > > > +* Closing connection > > > > +]) > > > > + > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > grep > > > -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +200 OK > > > > +* Closing connection > > > > +]) > > > > + > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > grep > > > -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > +200 OK > > > > +* Closing connection > > > > +]) > > > > + > > > > +# Remove the LB and change the VIP port - different from the backend > > > ports > > > > +check multinode_nbctl lb-del lb0 > > > > + > > > > +# create LB again > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:9000" "10.0.0.3:80, > > > 10.0.0.4:80" > > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > > + > > > > +# Set use_stateless_nat to true > > > > +check multinode_nbctl set load_balancer lb0 > > > options:use_stateless_nat=true > > > > + > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +# Check end-to-end request using a new port for VIP > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port > 59008 > > > 2>curl.out']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | > > > M_FORMAT_CT(20.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > [dnl > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > grep > > > -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > > +200 OK > > > > +* Closing connection > > > > +]) > > > > + > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > + > > > > +# Check end-to-end request using a new port for VIP > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port > 59008 > > > 2>curl.out']) > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | > > > M_FORMAT_CT(30.0.0.3) | \ > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > [0], > > > [dnl > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > +]) > > > > + > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > grep > > > -v 'Server:'], [0], [dnl > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > > +200 OK > > > > +* Closing connection > > > > +]) > > > > + > > > > +AT_CLEANUP > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > > > > index dcc3dbbc3..9e7a2f225 100644 > > > > --- a/tests/ovn-northd.at > > > > +++ b/tests/ovn-northd.at > > > > @@ -13864,3 +13864,323 @@ check_no_redirect > > > > > > > > AT_CLEANUP > > > > ]) > > > > + > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + > NAT > > > Stateless)]) > > > > +ovn_start > > > > + > > > > +check ovn-nbctl ls-add public > > > > +check ovn-nbctl lr-add lr1 > > > > + > > > > +# lr1 DGP ts1 > > > > +check ovn-nbctl ls-add ts1 > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > 172.16.10.1/24 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > > + > > > > +# lr1 DGP ts2 > > > > +check ovn-nbctl ls-add ts2 > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > 172.16.20.1/24 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > > + > > > > +# lr1 DGP public > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > 173.16.0.1/16 > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > + > > > > +check ovn-nbctl ls-add s1 > > > > +# s1 - lr1 > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > 172.16.0.1" > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > + > > > > +# s1 - backend vm1 > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > 172.16.0.101" > > > > + > > > > +# s1 - backend vm2 > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > 172.16.0.102" > > > > + > > > > +# s1 - backend vm3 > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > 172.16.0.103" > > > > + > > > > +# Add the lr1 DGP ts1 to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > > nat-addresses=router > > > > + > > > > +# Add the lr1 DGP ts2 to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > > nat-addresses=router > > > > + > > > > +# Add the lr1 DGP public to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > > nat-addresses=router > > > > + > > > > +# Create the Load Balancer lb1 > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > > + > > > > +# Set use_stateless_nat to true > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > > options:use_stateless_nat=true > > > > + > > > > +# Associate load balancer to s1 > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > +check ovn-nbctl --wait=sb sync > > > > + > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > +AT_CAPTURE_FILE([s1flows]) > > > > + > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | > grep > > > "30.0.0.1"], [0], [dnl > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] > == 1 > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > > +]) > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > "30.0.0.1"], [0], [dnl > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > > ip4.dst == 30.0.0.1), > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > +]) > > > > + > > > > +# Associate load balancer to lr1 with DGP > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > +check ovn-nbctl --wait=sb sync > > > > + > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > + > > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > "30.0.0.1"], [0], [dnl > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > is_chassis_resident("cr-lr1-ts1")), > > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > is_chassis_resident("cr-lr1-ts2")), > > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > is_chassis_resident("cr-lr1_public")), > > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > +]) > > > > + > > > > +# 2. Check if the DGP ports are in the match with action next > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], > [dnl > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > > action=(next;) > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") > && > > > is_chassis_resident("cr-lr1_public") && tcp), action=(next;) > > > > +]) > > > > + > > > > +# 3. Check if the VIP IP is in the ipX.src action > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > > action=(next;) > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > > action=(next;) > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(ip4.src=30.0.0.1; > next;) > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(ip4.src=30.0.0.1; > next;) > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") > && > > > is_chassis_resident("cr-lr1_public") && tcp), action=(ip4.src=30.0.0.1; > > > next;) > > > > +]) > > > > + > > > > +AT_CLEANUP > > > > +]) > > > > + > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + > NAT > > > Stateless) - IPv6]) > > > > +ovn_start > > > > + > > > > +check ovn-nbctl ls-add public > > > > +check ovn-nbctl lr-add lr1 > > > > + > > > > +# lr1 DGP ts1 > > > > +check ovn-nbctl ls-add ts1 > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > > > 2001:db8:aaaa:1::1/64 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > > + > > > > +# lr1 DGP ts2 > > > > +check ovn-nbctl ls-add ts2 > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > > > 2001:db8:aaaa:2::1/64 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > > + > > > > +# lr1 DGP public > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > > > 2001:db8:bbbb::1/64 > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > > > 2001:db8:aaaa:3::1/64 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > + > > > > +check ovn-nbctl ls-add s1 > > > > +# s1 - lr1 > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > > > 2001:db8:aaaa:3::1" > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > + > > > > +# s1 - backend vm1 > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > > > 2001:db8:aaaa:3::101" > > > > + > > > > +# s1 - backend vm2 > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > > > 2001:db8:aaaa:3::102" > > > > + > > > > +# s1 - backend vm3 > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > > > 2001:db8:aaaa:3::103" > > > > + > > > > +# Add the lr1 DGP ts1 to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > > nat-addresses=router > > > > + > > > > +# Add the lr1 DGP ts2 to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > > nat-addresses=router > > > > + > > > > +# Add the lr1 DGP public to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > > nat-addresses=router > > > > + > > > > +# Create the Load Balancer lb1 > > > > +check ovn-nbctl --wait=sb lb-add lb1 "2001:db8:cccc::1" > > > "2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101" > > > > + > > > > +# Set use_stateless_nat to true > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > > options:use_stateless_nat=true > > > > + > > > > +# Associate load balancer to s1 > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > +check ovn-nbctl --wait=sb sync > > > > + > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > +AT_CAPTURE_FILE([s1flows]) > > > > + > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | > grep > > > "2001:db8:cccc::1"], [0], [dnl > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] > == 1 > > > && ip6.dst == 2001:db8:cccc::1), action=(xxreg1 = 2001:db8:cccc::1; > > > ct_lb_mark;) > > > > +]) > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > "2001:db8:cccc::1"], [0], [dnl > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > > ip6.dst == 2001:db8:cccc::1), > > > > action=(ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > +]) > > > > + > > > > +# Associate load balancer to lr1 with DGP > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > +check ovn-nbctl --wait=sb sync > > > > + > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > + > > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > "2001:db8:cccc::1"], [0], [dnl > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > is_chassis_resident("cr-lr1-ts1")), > > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > is_chassis_resident("cr-lr1-ts2")), > > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > is_chassis_resident("cr-lr1_public")), > > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > +]) > > > > + > > > > +# 2. Check if the DGP ports are in the match with action next > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], > [dnl > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > > action=(next;) > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > 2001:db8:aaaa:3::102) || > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport > == > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > 2001:db8:aaaa:3::102) || > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport > == > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > 2001:db8:aaaa:3::102) || > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || > outport == > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > > action=(next;) > > > > +]) > > > > + > > > > +# 3. Check if the VIP IP is in the ipX.src action > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > > action=(next;) > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > > action=(next;) > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > 2001:db8:aaaa:3::102) || > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport > == > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > 2001:db8:aaaa:3::102) || > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport > == > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > 2001:db8:aaaa:3::102) || > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || > outport == > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > +]) > > > > + > > > > +AT_CLEANUP > > > > +]) > > > > + > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (DGP)]) > > > > +ovn_start > > > > + > > > > +check ovn-nbctl ls-add public > > > > +check ovn-nbctl lr-add lr1 > > > > + > > > > +# lr1 DGP ts1 > > > > +check ovn-nbctl ls-add ts1 > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > 172.16.10.1/24 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-1 > > > > + > > > > +# lr1 DGP ts2 > > > > +check ovn-nbctl ls-add ts2 > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > 172.16.20.1/24 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-1 > > > > + > > > > +# lr1 DGP public > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > 173.16.0.1/16 > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > + > > > > +check ovn-nbctl ls-add s1 > > > > +# s1 - lr1 > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > 172.16.0.1" > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > + > > > > +# s1 - backend vm1 > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > 172.16.0.101" > > > > + > > > > +# s1 - backend vm2 > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > 172.16.0.102" > > > > + > > > > +# s1 - backend vm3 > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > 172.16.0.103" > > > > + > > > > +# Add the lr1 DGP ts1 to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > > nat-addresses=router > > > > + > > > > +# Add the lr1 DGP ts2 to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > > nat-addresses=router > > > > + > > > > +# Add the lr1 DGP public to the public switch > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > > nat-addresses=router > > > > + > > > > +# Create the Load Balancer lb1 > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > > + > > > > +# Associate load balancer to s1 > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > +check ovn-nbctl --wait=sb sync > > > > + > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > +AT_CAPTURE_FILE([s1flows]) > > > > + > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | > grep > > > "30.0.0.1"], [0], [dnl > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] > == 1 > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > > +]) > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > "30.0.0.1"], [0], [dnl > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > > ip4.dst == 30.0.0.1), > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > +]) > > > > + > > > > +# Associate load balancer to lr1 with DGP > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > +check ovn-nbctl --wait=sb sync > > > > + > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > + > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > "30.0.0.1"], [0], [dnl > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > is_chassis_resident("cr-lr1-ts1")), > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > is_chassis_resident("cr-lr1-ts2")), > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > is_chassis_resident("cr-lr1_public")), > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > +]) > > > > + > > > > +AT_CLEANUP > > > > +]) > > > > -- > > > > 2.34.1 > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > _'Esta mensagem é direcionada apenas para os endereços constantes no > > > > cabeçalho inicial. Se você não está listado nos endereços constantes > no > > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo > dessa > > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas > > > estão > > > > imediatamente anuladas e proibidas'._ > > > > > > > > > > > > * **'Apesar do Magazine Luiza tomar > > > > todas as precauções razoáveis para assegurar que nenhum vírus esteja > > > > presente nesse e-mail, a empresa não poderá aceitar a > responsabilidade > > > por > > > > quaisquer perdas ou danos causados por esse e-mail ou por seus > anexos'.* > > > > > > > > > > > > > > > > _______________________________________________ > > > > dev mailing list > > > > dev@openvswitch.org > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > > > > > > -- > > > > > > > > > > _‘Esta mensagem é direcionada apenas para os endereços constantes no > > cabeçalho inicial. Se você não está listado nos endereços constantes no > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas > estão > > imediatamente anuladas e proibidas’._ > > > > > > * **‘Apesar do Magazine Luiza tomar > > todas as precauções razoáveis para assegurar que nenhum vírus esteja > > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade > por > > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* > > > > > > > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev >
On Mon, Sep 30, 2024 at 1:57 PM Roberto Bartzen Acosta via dev <ovs-dev@openvswitch.org> wrote: > > Hi Numan, > > Em sex., 27 de set. de 2024 às 12:47, Numan Siddique <numans@ovn.org> > escreveu: > > > On Thu, Sep 26, 2024 at 10:55 AM Roberto Bartzen Acosta via dev > > <ovs-dev@openvswitch.org> wrote: > > > > > > Hi Numan, > > > > > > Thanks for your feedback and review. > > > > > > Em qua., 25 de set. de 2024 às 19:50, Numan Siddique <numans@ovn.org> > > > escreveu: > > > > > > > On Thu, Sep 19, 2024 at 6:12 PM Roberto Bartzen Acosta via dev > > > > <ovs-dev@openvswitch.org> wrote: > > > > > > > > > > This commit fixes the build_distr_lrouter_nat_flows_for_lb function > > to > > > > > include a DNAT flow entry for each DGP in use. Since we have added > > > > support > > > > > to create multiple gateway ports per logical router, it's necessary > > to > > > > > include in the LR NAT rules pipeline a specific entry for each > > attached > > > > DGP. > > > > > Otherwise, the inbound traffic will only be redirected when the > > incoming > > > > LRP > > > > > matches the chassis_resident field. > > > > > > > > > > Additionally, this patch includes the ability to use load-balancer > > with > > > > DGPs > > > > > attached to multiple chassis. We can have each of the DGPs associated > > > > with a > > > > > different chassis, and in this case the DNAT rules added by default > > will > > > > not > > > > > be enough to guarantee outgoing traffic. > > > > > > > > > > To solve the multiple chassis for DGPs problem, this patch include a > > new > > > > > config options to be configured in the load-balancer. If the > > > > use_stateless_nat > > > > > is set to true, the logical router that references this load-balancer > > > > will use > > > > > Stateless NAT rules when the logical router has multiple DGPs. After > > > > applying > > > > > this patch and setting the use_stateless_nat option, the inbound > > and/or > > > > > outbound traffic can pass through any chassis where the DGP resides > > > > without > > > > > having problems with CT state. > > > > > > > > > > Reported-at: > > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2054322 > > > > > Fixes: 15348b7b806f ("ovn-northd: Multiple distributed gateway port > > > > support.") > > > > > > > > > > Signed-off-by: Roberto Bartzen Acosta <roberto.acosta@luizalabs.com> > > > > > > > > Hi Roberto, > > > > > > > > Thanks for the patch. I tested this patch using the test example in > > > > multinode.at. > > > > > > > > The test case adds the below load balancer > > > > > > > > [root@ovn-central ~]# ovn-nbctl lb-list > > > > UUID LB PROTO > > > > VIP IPs > > > > f3e29869-3bb5-4df0-960a-171106f5913a lb0 tcp > > > > 172.16.0.100:9000 10.0.0.3:80,10.0.0.4:80 > > > > > > > > And the below logical flows are generated by this patch > > > > > > > > -------- > > > > [root@ovn-central ~]# ovn-sbctl dump-flows lr0 | grep 172.16.0.100 > > > > table=6 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst > > > > == 172.16.0.100), action=(ct_dnat;) > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > > > is_chassis_resident("cr-lr0-public-p1")), > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > > 10.0.0.3:80, > > > > 10.0.0.4:80);) > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > > > is_chassis_resident("cr-lr0-public-p2")), > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > > 10.0.0.3:80, > > > > 10.0.0.4:80);) > > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > > > tcp.src == 80)) && (inport == "lr0-public-p1" || outport == > > > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), > > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > > > tcp.src == 80)) && (inport == "lr0-public-p2" || outport == > > > > "lr0-public-p2") && is_chassis_resident("cr-lr0-public-p2") && tcp), > > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > > > -------------- > > > > > > > > > > > > I fail to understand the reason for modifying the ip4.dst before > > > > calling ct_lb_mark. Can you please explain why ? Because the > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;) will first modify the > > > > ip4.dst to 10.0.0.3 and > > > > then to 10.0.0.4 and then the ct_lb_mark will actually do the > > > > conntrack with NAT to either 10.0.0.3 or 10.0.0.4. > > > > > > > > Is it because you want the conntrack entry to not have 172.16.0.100 ? > > > > > > > > > > > > The only reason I included this ip4.dst action in the DNAT rule is > > because > > > it's required to accept packets coming from a chassis that doesn't have > > > previously created conntrack entries. The main feature introduced in this > > > patch is to allow the administrator to have multiple DGPs attached to > > > different chassis (is_chassis_resident...). So, my implementation was > > based > > > on the normal behavior when using stateless NAT for external addresses, > > > where we need to add the ipx.dst in lr_in_dnat for traffic to be received > > > on the chassis (put the DGP port as chassis_resident match, as is the > > case > > > with stateless NAT [1] with DGP[2]). > > > > > > The question is, if we only have the ct_lb_mark, packets that pass > > through > > > the chassis and are already part of an active flow in another chassis > > (same > > > IPs and Ports) will be dropped because there is no correspondence in the > > > backend. So only packets with the NEW flag will be accepted and sent to > > the > > > backend (at least for TCP traffic). If we only have the ip4.dst action, > > > this will always perform the dnat for the same backend, without > > balancing. > > > Therefore, the combination of the two actions allows the packet to always > > > be received (regardless of whether conntrack is active for it), and > > > ct_lb_mark will take care of balancing for different backends. > > > > > > If we had conntrack sync between different chassis this would not be > > > necessary, as the ct_lb_mark action could always be executed without > > > dropping packets due to lack of correspondence in the conntrack table. > > > > > > [1] > > > > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15737 > > > [2] > > > > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15726 > > > > > > > > > > I'm sorry, but it's not 100% clear to me. I know that you've already > > explained to Mark in the older version of this patch. > > Can you please explain with an example ? > > > > Let's take the below topology you've added in the mutlinode test as example > > > > # Network topology > > # > > # publicp1 (ovn-chassis-3) (20.0.0.3/24) > > # | > > # overlay > > # | > > # DGP public1 (ovn-gw-1) (20.0.0.1/24) > > # | > > # | > > # | > > # lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) 10.0.0.3/24 > > # | | > > # | + --- sw0p2 (ovn-chassis-2) 10.0.0.4/24 > > # | > > # DGP public2 (ovn-gw-2) (30.0.0.1/24) > > # | > > # overlay > > # | > > # publicp2 (ovn-chassis-4) (30.0.0.3/24) > > > > > > load balancer is configured on lr0 -> ovn-nbctl lb-add lb0 > > "172.16.0.100:80" "10.0.0.3:80,10.0.0.4:80" > > and it is attached to both lr0 and sw0. > > > > Scenario 1: > > > > publicp1 with IP 20.0.0.3 sends TCP traffic to VIP 172.16.0.100. I > > think this is what will happen > > > > - The packet (ip4.src = 20.0.0.3 , ip4.dst = 172.16.0.100, tcp.dst > > = 80) from ovn-chassis-3 will be sent out via the localnet bridge > > (br-ex) > > and the packet is received on ovn-gw-1 (as DGP public1 is resident > > on it) via the localnet bridge and the > > packet first enters public logical switch pipeline and then the > > lr0 router pipeline. > > > > - In the router's lr_in_dnat state, the packet will be load > > balanced to one of the backends using ct_lb_mark. Lets say 10.0.0.3 > > is chosen > > - The packet from router pipeline lr0 enters sw0 switch pipeline and > > then the packet is tunneled to ovn-chassis-1 and delivered to sw0p1. > > - The reply packet (ip4.src = 10.0.0.3, ip4.dst = 20.0.0.3, tcp.src = > > 80) will enter the router pipeline and since 20.0.0.0/24 is handled by > > DGP public1, > > the packet is tunnelled to ovn-gw-1. > > - In the router pipeline of ovn-gw-1, the packet is undnatted from > > ip4.src 10.0.0.3 to 172.16.0.100 and the packet is sent out via the > > localnet bridge. > > - ovn-chassis-3 receives the packet via the localnet bridge and into > > br-int and finally to publicp1. > > > > > > In this scenario the load balancing is handled and conntrack entries > > are created in ovn-gw-1. And there is no need to add flows in > > "lr_out_snat" for stateless NAT > > or set ip4.dst to one or all of backend IPs before "ct_lb_mark" in > > lr_in_dnat stage. > > > > Scenario 2: > > publicp2 with IP 30.0.0.3 sends TCP traffic to VIP 172.16.0.100. > > > > This is similar to scenario 1. Except that load balancing happens in > > ovn-gw-2. > > since DGP public2 is on this chassis. > > > > > > Scenario 3: > > > > An external entity with IP 20.0.0.50 sends TCP traffic to VIP > > 172.16.0.100. > > > > - This scenario is similar to the first one. The packet from this > > external entity is received on ovn-gw-1 via the localnet bridge. > > Rest all is the same. > > > > Scenario 4: > > > > sw0p2 on ovn-chassis-2 sends TCP traffic to VIP 172.16.0.100. > > > > - Since sw0 is attached with load balancer lb0, load balancing > > happens in the source chassis - ovn-chassis-2 itself and depending on > > the backend chosen, > > the packet is tunnelled to ovn-chassis-1 (if 10.0.0.3 is chosen) > > or delivered directly to sw0p2 (if i0.0.0.4 is chosen). > > > > > > Scenario 5: > > > > An external entity with IP 40.0.0.40 sends TCP traffic to VIP > > 172.16.0.100 and there are 2 ECMP routes configured > > 172.16.0.100 via 20.0.0.1 > > 172.16.0.100 via 30.0.0.1 > > > > In this case if the packet uses the route via 20.0.0.1 it will be > > received on ovn-gw-1 via the localnet bridge br-ex. > > > > - With your patch, ip4.dst is modified to the last backend in the > > list and then ct_lb_mark chosen as one of the backends > > - And then it is tunnelled to the destination chassis. > > > > > > Is this the scenario 5 you're trying to address with the stateless > > NAT ? How would the reply packet work ? > > The reply packet from backend 10.0.0.3 can use either of the paths ? > > i.e ovn-gw-1 or ovn-gw-2 ? > > > > > Yes, this scenario would be the main goal of this patch. > Let me explain with an example related to the OVN interconnect (one of the > use cases of this patch). In the context of ovn-ic the backend 10.0.0.3 > could reply packets using either of the paths (ovn-gw-1 or ovn-gw-2). > > > > LB VIP 172.16.0.100 > > | > vm (40.0.0.40) - LR-EXT - lrp1 - chassis1 - Transit switch TS1 - ovn-gw-1 > - lr0-public-p1 - lr0 - sw0 - VM 10.0.0.3 (ovn-chassis-1) > - lrp2 - chassis2 - Transit switch > TS2 - ovn-gw-2 - lr0-public-p2 - - VM 10.0.0.4 > (ovn-chassis-2) > > > Let's take an example of the ovn-ic in the above topology: > > Assumptions: > A) logical router LR-EXT: route to 172.16.0.100/32 via ECMP (lrp1/chassis1 > and lrp2/chassis2) > B) logical router lr0: route to 40.0.0.0/24 via ECMP > (lr0-public-p1/ovn-gw-1 and lr0-public-p2/ovn-gw-2) > > > 1 - VM 40.0.0.40 send a request to the LB 172.16.0.100:80 > 2 - LR-EXT chooses one of the possible outgoing routes via ECMP - e.g. lrp1 > / chassis1 / TS1 > 3 - The lr0 receives the ingoing traffic through the port lr0-public-p1 / > ovn-gw-1 because the TS1 is used to start the traffic > 4 - with this patch we added lr_in_dnat rules for all DGPs (with or without > the stateless NAT config flag) > > with the stateless NAT config flag True: > ...is_chassis_resident("cr-lr0-public-p1")), > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80 > ,10.0.0.4:80);) > ...is_chassis_resident("cr-lr0-public-p2")), > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80 > ,10.0.0.4:80);) > 5 - The ip4.dst will perform the action cumulatively, basically the last > ip4.dst action applied will be to change dst to 10.0.0.4. > 5.1 - The ct_lb_mark will check the conntrack and validate if the traffic > refers to a new one (TCP SYNC), and then, it executes the final action > provided by ct_lb_mark and select one of the available backends. e.g > 10.0.0.3 > > without the stateless NAT config flag: > ...is_chassis_resident("cr-lr0-public-p1")), action=(ct_lb_mark(backends= > 10.0.0.3:80,10.0.0.4:80);) > ...is_chassis_resident("cr-lr0-public-p2")), action=(ct_lb_mark(backends= > 10.0.0.3:80,10.0.0.4:80);) > > 5 - same as the step 5.1 (ct_lb_mark check only). > > 6 - VM 10.0.0.3 receives the new traffic and sends a reply (TCP SYN+ACK) > > 7 - lr0 receives the response packet (from VM 10.0.0.3) in the routing > pipeline and route via one of the available paths (ECMP). > For example: lr0-public-p2 / ovn-gw-2 / TS2 > So, we have a different outgoing path of the original one. > > 8 - ovn-gw-2 receives (SYN+ACK). We have no conntrack entries on ovn-gw-2 > to match and perform the LB SNAT action! The ct_lb_mark previously creates > the conntrack entry on chassis ovn-gw-1 (when receives TCP SYN). > > with the stateless NAT config flag True: > The outgoing packet fail to match the lb conntrack but the packet is > SNATed by the lr_out_snat rule with a priority lower then cl_lb_mark, > executing after cl_lb_mark in the pipeline. > > 9 - The SYN+ACK packet crosses the TS2 and is delivered to VM 40.0.0.40 via > lrp2 (SRC = 172.16.0.100:80) > > lr_out_snat will match and SNATed: > match=(ip4 && ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == > 10.0.0.4 && tcp.src == 80)) && (inport == "lr0-public-p1" || outport == > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), > action=(ip4.src=172.16.0.100; tcp.src=80; next;) > > without the stateless NAT config flag: > The outgoing will be returned to the VM 40.0.0.4 with the LR SNAT for > public-p2 port as ip4.src! So, basically this will break the TCP > handshake!!! > > 9 - The SYN+ACK packet cross the TS2 and is delivered to the VM 40.0.0.40 > via lrp2 (SRC = 30.0.0.1:80) > > Moving forward on the happy path (stateless NAT is true): > > 10 - VM 40.0.0.40 sends an ACK to complete the TCP handshake. > > 11 - 2 - LR-EXT chooses one of the possible outgoing routes via ECMP > Let's assume that LR-EXT correctly execute the ecmp algorithm (per flow > basis) and forwards to the same initial path because we're using the same > flow (src/dst IPs and src/dts TCP ports still the same). > e.g. still using the lrp1 / chassis1 / TS1 > > 12 - The lr0 receive the ingoing traffic through the port lr0-public-p1 / > ovn-gw-1 / TS1 > > With this patch we have 2 possible behaviours: > > 12.1 - Without Stateless NAT config flag > > table=1 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src == > 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && tcp.src == 80)) && > (inport == "lr0-public-p1" || outport == > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1")), > action=(ct_dnat_in_czone;) > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && !ct.rel > && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > is_chassis_resident("cr-lr0-public-p1")), action=(ct_lb_mark(backends= > 10.0.0.3:80,10.0.0.4:80);) > > openflow: br-int (ovn-gw-1) > cookie=0x9381c20e, duration=2361.675s, table=15, n_packets=2, n_bytes=148, > idle_age=1564, > priority=120,ct_state=+new-rel+trk,tcp,metadata=0x4,nw_dst=172.16.0.100,tp_dst=80 > actions=group:3 > > ovs-appctl dpctl/dump-conntrack (ovn-gw-1) > tcp,orig=(src=40.0.0.40,dst=172.16.0.100,sport=35274,dport=80),reply=(src=10.0.0.3,dst=40.0.0.40,sport=80,dport=35274),zone=8,mark=2,protoinfo=(state=TIME_WAIT) > > At this point, the ACK packet will be discarded because we didn't perform > the SYN+ACK return on this chassis (ovn-gw -1). So, without the previously > established conntrack to match the ACK packet we broke the TCP handshake. > > Remember the flow: > -> SYN: ovn-gw-1 > <- SYN+ACK: ovn-gw-2 > -> ACK: ovn-gw-1 > > 12.2 - With stateless NAT config flag is True > > table=1 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src == > 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && tcp.src == 80)) && > (inport == "lr0-public-p1" || outport == > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1")), action=(next;) > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && !ct.rel > && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > is_chassis_resident("cr-lr0-public-p1")), > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80 > ,10.0.0.4:80);) > > openflow: br-int (ovn-gw-1) > cookie=0xab99f240, duration=1086.022s, table=15, n_packets=610, > n_bytes=48979, idle_age=825, > priority=120,ct_state=+new-rel+trk,tcp,metadata=0x2,nw_dst=172.16.0.100,tp_dst=80 > actions=mod_nw_dst:10.0.0.3,mod_nw_dst:10.0.0.4,group:1 > > ovs-appctl dpctl/dump-conntrack (ovn-gw-1) > tcp,orig=(src=40.0.0.40,dst=10.0.0.3,sport=54632,dport=80),reply=(src=10.0.0.3,dst=40.0.0.40,sport=80,dport=54632),zone=7,mark=2,protoinfo=(state=SYN_SENT) > > This is the reason to use stateless NAT rules. Unlike the case where we > don't have the SYN_SENT, now the ACK packet will be accepted and forwarded > after performing the mod_nw_dst action. > > How does this work? We already have the previous flow created by the first > packet (SYN), so regardless of the IP.dst modified in the action, the > packet will be forwarded to the same backend that corresponds to the > conntrack match that is in SYN_SENT state. That's why I created the action > with the two modifiers (ip4.dst + ct_lb_mark). Using only the ct_lb_mark > action creates a strong dependency on the connection state match in the > conntrack entry (which we don't have in these cases). However, only using > ip4.dst doesn't create traffic balancing as it will always send to the same > backend (last in the ip4.dst). > > > I hope this has helped clarify the design decisions when creating/modifying > flows (lr_in_dnat/lr_out_snat/lr_out_undnat). > > Thanks for the detailed explanation. Its clear to me now. I'd suggest the following 1. In lr_in_dnat stage, choose either the first backend or the last backend to modify the ip4.dst before ct_lb_mark. I've no strong preference. Either would do. 2. Please document or add comments in northd.c (and in ovn-nb.xml)so that we don't lose the context later. 3. Please run the ovn-fake-multinode tests in your github CI repo and make sure it passes. I think you can trigger the test in your repo in the Actions tab. 4. In the setup you described above, is it possible to add the (below) route in the vm (40.0.0.40) deployment so that 40.0.0.0.40 can talk directly to 10.0.0.3 and 10.0.0.4 instead of the LB VIP and send the tcp traffic to 10.0.0.3 and see if it works ? If not, then please document the same i.e with the option stateless_nat in the LB, the backends cannot be directly communicated. logical router LR-EXT: route to 10.0.0.0/24 via ECMP (lrp1/chassis1 and lrp2/chassis2) Thanks Numan > > > > > > Thanks > > Numan > > > > > > > > > > > > Also I don't understand why this patch adds the logical flows in > > > > "lr_out_snat" stage ? > > > > > > > > > > The flow for lr_out_snat is necessary for the correct functioning of > > > stateless NAT for the same reason explained previously. I mean, if the > > > outgoing packet is redirected to a chassis that doesn't have an active > > > conntrack entry, it will not be NATed by ct_lb action because it doesn't > > > refer to a valid flow (use case with ecmp). > > > > > > So it is necessary to create a stateless SNAT rule (similar to this [3]) > > > with a lower priority than the other router pipeline entries, in this > > case, > > > if the packet is not SNATed by ct_lb (conntrack missed) it will be SNATed > > > by stateless NAT rule. > > > > > > [3] > > > > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15884 > > > > > > > > > > > > > > > > > Using the system multinode test as an example, the below fails > > > > (which is a regression) > > > > > > > > --- > > > > root@ovn-chassis-3 ~]# ip netns exec publicp1 nc -vz 10.0.0.3 80 > > > > ---- > > > > > > > > In the above test, publicp1 with IP 20.0.0.3 when it tries to connect > > > > to one if the backends directly (without the LB VIP), it fails. > > > > It fails because of the logical flows in "lr_out_snat". > > > > > > > > > > > > Looks to me the solution proposed here is incomplete. > > > > > > > > Also please note that in our CI we run the multinode tests > > > > periodically once a day using the v0.1 of the ovn-fake-multinode > > > > and the tests you added will fail. This needs to be fixed and until > > > > we move to the latest version of ovn-fake-multinode. > > > > > > > > > > I imagine that the test you are doing is using the same port as the LB > > > backend (TCP 80 in this case). So, the stateless lr_out_snat flow will > > > force the output to be SNATed because this port is in use by the backend. > > > Traffic to/from other ports will work without problems and will follow > > the > > > normal programmed flows (e.g. ICMP). > > > > > > This is necessary to ensure the egress traffic because the DGPs are > > > distributed across multiple chassis. Also, this setup is being validated > > in > > > the test ovn-fake-multinode testcase (ICMP from the backends chassis use > > > the router's default SNAT and not the LB's). I didn't understand the > > > regression you mentioned because this was programmed to be stateless and > > > it's traffic that uses the same ports as the LB backend, could you > > explain > > > better? > > > > > > Thanks, > > > Roberto > > > > > > > > > > Thanks > > > > Numan > > > > > > > > > > > > > --- > > > > > northd/en-lr-stateful.c | 12 - > > > > > northd/northd.c | 116 ++++++-- > > > > > ovn-nb.xml | 10 + > > > > > tests/multinode-macros.at | 40 +++ > > > > > tests/multinode.at | 556 > > ++++++++++++++++++++++++++++++++++++++ > > > > > tests/ovn-northd.at | 320 ++++++++++++++++++++++ > > > > > 6 files changed, 1017 insertions(+), 37 deletions(-) > > > > > > > > > > diff --git a/northd/en-lr-stateful.c b/northd/en-lr-stateful.c > > > > > index baf1bd2f8..f09691af6 100644 > > > > > --- a/northd/en-lr-stateful.c > > > > > +++ b/northd/en-lr-stateful.c > > > > > @@ -516,18 +516,6 @@ lr_stateful_record_create(struct > > lr_stateful_table > > > > *table, > > > > > > > > > > table->array[od->index] = lr_stateful_rec; > > > > > > > > > > - /* Load balancers are not supported (yet) if a logical router > > has > > > > multiple > > > > > - * distributed gateway port. Log a warning. */ > > > > > - if (lr_stateful_rec->has_lb_vip && > > lr_has_multiple_gw_ports(od)) { > > > > > - static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, > > 1); > > > > > - VLOG_WARN_RL(&rl, "Load-balancers are configured on logical > > " > > > > > - "router %s, which has %"PRIuSIZE" distributed " > > > > > - "gateway ports. Load-balancer is not supported > > " > > > > > - "yet when there is more than one distributed " > > > > > - "gateway port on the router.", > > > > > - od->nbr->name, od->n_l3dgw_ports); > > > > > - } > > > > > - > > > > > return lr_stateful_rec; > > > > > } > > > > > > > > > > diff --git a/northd/northd.c b/northd/northd.c > > > > > index a267cd5f8..bbe97acf8 100644 > > > > > --- a/northd/northd.c > > > > > +++ b/northd/northd.c > > > > > @@ -11807,31 +11807,30 @@ static void > > > > > build_distr_lrouter_nat_flows_for_lb(struct lrouter_nat_lb_flows_ctx > > > > *ctx, > > > > > enum lrouter_nat_lb_flow_type > > type, > > > > > struct ovn_datapath *od, > > > > > - struct lflow_ref *lflow_ref) > > > > > + struct lflow_ref *lflow_ref, > > > > > + struct ovn_port *dgp, > > > > > + bool stateless_nat) > > > > > { > > > > > - struct ovn_port *dgp = od->l3dgw_ports[0]; > > > > > - > > > > > - const char *undnat_action; > > > > > - > > > > > - switch (type) { > > > > > - case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > > > - undnat_action = "flags.force_snat_for_lb = 1; next;"; > > > > > - break; > > > > > - case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > > > - undnat_action = "flags.skip_snat_for_lb = 1; next;"; > > > > > - break; > > > > > - case LROUTER_NAT_LB_FLOW_NORMAL: > > > > > - case LROUTER_NAT_LB_FLOW_MAX: > > > > > - undnat_action = lrouter_use_common_zone(od) > > > > > - ? "ct_dnat_in_czone;" > > > > > - : "ct_dnat;"; > > > > > - break; > > > > > - } > > > > > + struct ds dnat_action = DS_EMPTY_INITIALIZER; > > > > > > > > > > /* Store the match lengths, so we can reuse the ds buffer. */ > > > > > size_t new_match_len = ctx->new_match->length; > > > > > size_t undnat_match_len = ctx->undnat_match->length; > > > > > > > > > > + /* dnat_action: Add the LB backend IPs as a destination action > > of > > > > the > > > > > + * lr_in_dnat NAT rule with cumulative effect > > because > > > > any > > > > > + * backend dst IP used in the action list will > > > > redirect the > > > > > + * packet to the ct_lb pipeline. > > > > > + */ > > > > > + if (stateless_nat) { > > > > > + for (size_t i = 0; i < ctx->lb_vip->n_backends; i++) { > > > > > + struct ovn_lb_backend *backend = > > &ctx->lb_vip->backends[i]; > > > > > + bool ipv6 = !IN6_IS_ADDR_V4MAPPED(&backend->ip); > > > > > + ds_put_format(&dnat_action, "%s.dst=%s;", ipv6 ? "ip6" : > > > > "ip4", > > > > > + backend->ip_str); > > > > > + } > > > > > + } > > > > > + ds_put_format(&dnat_action, "%s", ctx->new_action[type]); > > > > > > > > > > const char *meter = NULL; > > > > > > > > > > @@ -11841,20 +11840,46 @@ build_distr_lrouter_nat_flows_for_lb(struct > > > > lrouter_nat_lb_flows_ctx *ctx, > > > > > > > > > > if (ctx->lb_vip->n_backends || !ctx->lb_vip->empty_backend_rej) > > { > > > > > ds_put_format(ctx->new_match, " && is_chassis_resident(%s)", > > > > > - od->l3dgw_ports[0]->cr_port->json_key); > > > > > + dgp->cr_port->json_key); > > > > > } > > > > > > > > > > ovn_lflow_add_with_hint__(ctx->lflows, od, S_ROUTER_IN_DNAT, > > > > ctx->prio, > > > > > - ds_cstr(ctx->new_match), > > > > ctx->new_action[type], > > > > > + ds_cstr(ctx->new_match), > > > > ds_cstr(&dnat_action), > > > > > NULL, meter, &ctx->lb->nlb->header_, > > > > > lflow_ref); > > > > > > > > > > ds_truncate(ctx->new_match, new_match_len); > > > > > > > > > > + ds_destroy(&dnat_action); > > > > > if (!ctx->lb_vip->n_backends) { > > > > > return; > > > > > } > > > > > > > > > > + struct ds undnat_action = DS_EMPTY_INITIALIZER; > > > > > + struct ds snat_action = DS_EMPTY_INITIALIZER; > > > > > + > > > > > + switch (type) { > > > > > + case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > > > + ds_put_format(&undnat_action, "flags.force_snat_for_lb = 1; > > > > next;"); > > > > > + break; > > > > > + case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > > > + ds_put_format(&undnat_action, "flags.skip_snat_for_lb = 1; > > > > next;"); > > > > > + break; > > > > > + case LROUTER_NAT_LB_FLOW_NORMAL: > > > > > + case LROUTER_NAT_LB_FLOW_MAX: > > > > > + ds_put_format(&undnat_action, "%s", > > > > > + lrouter_use_common_zone(od) ? > > "ct_dnat_in_czone;" > > > > > + : "ct_dnat;"); > > > > > + break; > > > > > + } > > > > > + > > > > > + /* undnat_action: Remove the ct action from the lr_out_undenat > > NAT > > > > rule. > > > > > + */ > > > > > + if (stateless_nat) { > > > > > + ds_clear(&undnat_action); > > > > > + ds_put_format(&undnat_action, "next;"); > > > > > + } > > > > > + > > > > > /* We need to centralize the LB traffic to properly perform > > > > > * the undnat stage. > > > > > */ > > > > > @@ -11873,11 +11898,41 @@ build_distr_lrouter_nat_flows_for_lb(struct > > > > lrouter_nat_lb_flows_ctx *ctx, > > > > > ds_put_format(ctx->undnat_match, ") && (inport == %s || outport > > == > > > > %s)" > > > > > " && is_chassis_resident(%s)", dgp->json_key, > > > > dgp->json_key, > > > > > dgp->cr_port->json_key); > > > > > + /* Use the LB protocol as matching criteria for out undnat and > > snat > > > > when > > > > > + * creating LBs with stateless NAT. */ > > > > > + if (stateless_nat) { > > > > > + ds_put_format(ctx->undnat_match, " && %s", ctx->lb->proto); > > > > > + } > > > > > ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_UNDNAT, > > 120, > > > > > - ds_cstr(ctx->undnat_match), > > undnat_action, > > > > > - &ctx->lb->nlb->header_, > > > > > + ds_cstr(ctx->undnat_match), > > > > > + ds_cstr(&undnat_action), > > > > &ctx->lb->nlb->header_, > > > > > lflow_ref); > > > > > + > > > > > + /* snat_action: Add a new lr_out_snat rule with the LB VIP as > > > > source IP > > > > > + * action to perform the NAT stateless pipeline > > > > completely. > > > > > + */ > > > > > + if (stateless_nat) { > > > > > + if (ctx->lb_vip->port_str) { > > > > > + ds_put_format(&snat_action, "%s.src=%s; %s.src=%s; > > next;", > > > > > + ctx->lb_vip->address_family == AF_INET6 ? > > > > > + "ip6" : "ip4", > > > > > + ctx->lb_vip->vip_str, ctx->lb->proto, > > > > > + ctx->lb_vip->port_str); > > > > > + } else { > > > > > + ds_put_format(&snat_action, "%s.src=%s; next;", > > > > > + ctx->lb_vip->address_family == AF_INET6 ? > > > > > + "ip6" : "ip4", > > > > > + ctx->lb_vip->vip_str); > > > > > + } > > > > > + ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_SNAT, > > 160, > > > > > + ds_cstr(ctx->undnat_match), > > > > > + ds_cstr(&snat_action), > > > > &ctx->lb->nlb->header_, > > > > > + lflow_ref); > > > > > + } > > > > > + > > > > > ds_truncate(ctx->undnat_match, undnat_match_len); > > > > > + ds_destroy(&undnat_action); > > > > > + ds_destroy(&snat_action); > > > > > } > > > > > > > > > > static void > > > > > @@ -12022,6 +12077,8 @@ build_lrouter_nat_flows_for_lb( > > > > > * lflow generation for them. > > > > > */ > > > > > size_t index; > > > > > + bool use_stateless_nat = smap_get_bool(&lb->nlb->options, > > > > > + "use_stateless_nat", > > false); > > > > > BITMAP_FOR_EACH_1 (index, bitmap_len, lb_dps->nb_lr_map) { > > > > > struct ovn_datapath *od = lr_datapaths->array[index]; > > > > > enum lrouter_nat_lb_flow_type type; > > > > > @@ -12043,8 +12100,17 @@ build_lrouter_nat_flows_for_lb( > > > > > if (!od->n_l3dgw_ports) { > > > > > bitmap_set1(gw_dp_bitmap[type], index); > > > > > } else { > > > > > - build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > > > > - lb_dps->lflow_ref); > > > > > + /* Create stateless LB NAT rules when using multiple > > DGPs > > > > and > > > > > + * use_stateless_nat is true. > > > > > + */ > > > > > + bool stateless_nat = (od->n_l3dgw_ports > 1) > > > > > + ? use_stateless_nat : false; > > > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > > > > > + struct ovn_port *dgp = od->l3dgw_ports[i]; > > > > > + build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > > > > + > > lb_dps->lflow_ref, > > > > dgp, > > > > > + stateless_nat); > > > > > + } > > > > > } > > > > > > > > > > if (lb->affinity_timeout) { > > > > > diff --git a/ovn-nb.xml b/ovn-nb.xml > > > > > index 2836f58f5..ad03c6214 100644 > > > > > --- a/ovn-nb.xml > > > > > +++ b/ovn-nb.xml > > > > > @@ -2302,6 +2302,16 @@ or > > > > > local anymore by the ovn-controller. This option is set to > > > > > <code>false</code> by default. > > > > > </column> > > > > > + > > > > > + <column name="options" key="use_stateless_nat" > > > > > + type='{"type": "boolean"}'> > > > > > + If the load balancer is configured with > > > > <code>use_stateless_nat</code> > > > > > + option to <code>true</code>, the logical router that > > references > > > > this > > > > > + load balancer will use Stateless NAT rules when the logical > > > > router > > > > > + has multiple distributed gateway ports(DGP). Otherwise, the > > > > outbound > > > > > + traffic may be dropped in scenarios where we have different > > > > chassis > > > > > + for each DGP. This option is set to <code>false</code> by > > > > default. > > > > > + </column> > > > > > </group> > > > > > </table> > > > > > > > > > > diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at > > > > > index 757917626..2f69433fc 100644 > > > > > --- a/tests/multinode-macros.at > > > > > +++ b/tests/multinode-macros.at > > > > > @@ -40,6 +40,27 @@ m4_define([M_START_TCPDUMP], > > > > > ] > > > > > ) > > > > > > > > > > +# M_EXEC([fake_node], [command]) > > > > > +# > > > > > +# Execute 'command' in 'fakenode' > > > > > +m4_define([M_EXEC], > > > > > + [podman exec $1 $2]) > > > > > + > > > > > +# M_CHECK_EXEC([fake_node], [command], other_params...) > > > > > +# > > > > > +# Wrapper for AT_CHECK that executes 'command' inside > > 'fake_node''s'. > > > > > +# 'other_params' as passed as they are to AT_CHECK. > > > > > +m4_define([M_CHECK_EXEC], > > > > > + [ AT_CHECK([M_EXEC([$1], [$2])], > > m4_shift(m4_shift(m4_shift($@)))) ] > > > > > +) > > > > > + > > > > > +# M_FORMAT_CT([ip-addr]) > > > > > +# > > > > > +# Strip content from the piped input which would differ from test to > > > > test > > > > > +# and limit the output to the rows containing 'ip-addr'. > > > > > +# > > > > > +m4_define([M_FORMAT_CT], > > > > > + [[grep -F "dst=$1," | sed -e 's/id=[0-9]*/id=<cleared>/g' -e > > > > 's/state=[0-9_A-Z]*/state=<cleared>/g' | sort | uniq | sed -e > > > > 's/zone=[[0-9]]*/zone=<cleared>/' -e 's/mark=[[0-9]]*/mark=<cleared>/' > > ]]) > > > > > > > > > > OVS_START_SHELL_HELPERS > > > > > > > > > > @@ -76,6 +97,25 @@ multinode_nbctl () { > > > > > m_as ovn-central ovn-nbctl "$@" > > > > > } > > > > > > > > > > +check_fake_multinode_setup_by_nodes() { > > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > > > + for c in $1 > > > > > + do > > > > > + AT_CHECK([m_as $c ovn-appctl -t ovn-controller version], > > [0], > > > > [ignore]) > > > > > + done > > > > > +} > > > > > + > > > > > +cleanup_multinode_resources_by_nodes() { > > > > > + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db > > > > > + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd > > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > > > + for c in $1 > > > > > + do > > > > > + m_as $c ovs-vsctl del-br br-int > > > > > + m_as $c ip --all netns delete > > > > > + done > > > > > +} > > > > > + > > > > > # m_count_rows TABLE [CONDITION...] > > > > > # > > > > > # Prints the number of rows in TABLE (that satisfy CONDITION). > > > > > diff --git a/tests/multinode.at b/tests/multinode.at > > > > > index a0eb8fc67..b1beb4d97 100644 > > > > > --- a/tests/multinode.at > > > > > +++ b/tests/multinode.at > > > > > @@ -1591,3 +1591,559 @@ AT_CHECK([cat ch1_eth2.tcpdump], [0], [dnl > > > > > ]) > > > > > > > > > > AT_CLEANUP > > > > > + > > > > > +AT_SETUP([ovn multinode load-balancer with multiple DGPs and > > multiple > > > > chassis]) > > > > > + > > > > > +# Check that ovn-fake-multinode setup is up and running - requires > > > > additional nodes > > > > > +check_fake_multinode_setup_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > > > + > > > > > +# Delete the multinode NB and OVS resources before starting the > > test. > > > > > +cleanup_multinode_resources_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > > > + > > > > > +# Network topology > > > > > +# > > > > > +# publicp1 (ovn-chassis-3) (20.0.0.3/24) > > > > > +# | > > > > > +# overlay > > > > > +# | > > > > > +# DGP public1 (ovn-gw-1) (20.0.0.1/24) > > > > > +# | > > > > > +# | > > > > > +# | > > > > > +# lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) > > 10.0.0.3/24 > > > > > +# | | > > > > > +# | + --- sw0p2 (ovn-chassis-2) > > 10.0.0.4/24 > > > > > +# | > > > > > +# DGP public2 (ovn-gw-2) (30.0.0.1/24) > > > > > +# | > > > > > +# overlay > > > > > +# | > > > > > +# publicp2 (ovn-chassis-4) (30.0.0.3/24) > > > > > + > > > > > +# Delete already used ovs-ports > > > > > +m_as ovn-chassis-1 ovs-vsctl del-port br-int sw0p1-p > > > > > +m_as ovn-chassis-2 ovs-vsctl del-port br-int sw0p2-p > > > > > +m_as ovn-chassis-1 ip link del sw0p1-p > > > > > +m_as ovn-chassis-2 ip link del sw0p2-p > > > > > +m_as ovn-chassis-3 ovs-vsctl del-port br-int publicp1-p > > > > > +m_as ovn-chassis-4 ovs-vsctl del-port br-int publicp2-p > > > > > +m_as ovn-chassis-3 ip link del publicp1-p > > > > > +m_as ovn-chassis-4 ip link del publicp2-p > > > > > + > > > > > +# Create East-West switch for LB backends > > > > > +check multinode_nbctl ls-add sw0 > > > > > +check multinode_nbctl lsp-add sw0 sw0-port1 > > > > > +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 > > > > 10.0.0.3 1000::3" > > > > > +check multinode_nbctl lsp-add sw0 sw0-port2 > > > > > +check multinode_nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 > > > > 10.0.0.4 1000::4" > > > > > + > > > > > +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 > > > > 50:54:00:00:00:03 1400 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a > > > > > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 > > > > 50:54:00:00:00:04 1400 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a > > > > > + > > > > > +m_wait_for_ports_up > > > > > + > > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > > > > 10.0.0.4 | FORMAT_PING], \ > > > > > +[0], [dnl > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > +]) > > > > > + > > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 > > > > 10.0.0.3 | FORMAT_PING], \ > > > > > +[0], [dnl > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > +]) > > > > > + > > > > > +# Create a logical router and attach to sw0 > > > > > +check multinode_nbctl lr-add lr0 > > > > > +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 > > 10.0.0.1/24 > > > > 1000::a/64 > > > > > +check multinode_nbctl lsp-add sw0 sw0-lr0 > > > > > +check multinode_nbctl lsp-set-type sw0-lr0 router > > > > > +check multinode_nbctl lsp-set-addresses sw0-lr0 router > > > > > +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 > > > > > + > > > > > +# create external connection for N/S traffic using multiple DGPs > > > > > +check multinode_nbctl ls-add public > > > > > + > > > > > +# DGP public1 > > > > > +check multinode_nbctl lsp-add public ln-public-1 > > > > > +check multinode_nbctl lsp-set-type ln-public-1 localnet > > > > > +check multinode_nbctl lsp-set-addresses ln-public-1 unknown > > > > > +check multinode_nbctl lsp-set-options ln-public-1 > > network_name=public1 > > > > > + > > > > > +# DGP public2 > > > > > +# create exteranl connection for N/S traffic > > > > > +check multinode_nbctl lsp-add public ln-public-2 > > > > > +check multinode_nbctl lsp-set-type ln-public-2 localnet > > > > > +check multinode_nbctl lsp-set-addresses ln-public-2 unknown > > > > > +check multinode_nbctl lsp-set-options ln-public-2 > > network_name=public2 > > > > > + > > > > > +# Attach DGP public1 to GW-1 and chassis-3 (overlay connectivity) > > > > > +m_as ovn-gw-1 ovs-vsctl set open . > > > > external-ids:ovn-bridge-mappings=public1:br-ex > > > > > +m_as ovn-chassis-3 ovs-vsctl set open . > > > > external-ids:ovn-bridge-mappings=public1:br-ex > > > > > + > > > > > +# Attach DGP public2 to GW-2 and chassis-4 (overlay connectivity) > > > > > +m_as ovn-gw-2 ovs-vsctl set open . > > > > external-ids:ovn-bridge-mappings=public2:br-ex > > > > > +m_as ovn-chassis-4 ovs-vsctl set open . > > > > external-ids:ovn-bridge-mappings=public2:br-ex > > > > > + > > > > > +# Create the external LR0 port to the DGP public1 > > > > > +check multinode_nbctl lsp-add public public-port1 > > > > > +check multinode_nbctl lsp-set-addresses public-port1 > > "40:54:00:00:00:03 > > > > 20.0.0.3 2000::3" > > > > > + > > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p1 00:00:00:00:ff:02 > > > > 20.0.0.1/24 2000::a/64 > > > > > +check multinode_nbctl lsp-add public public-lr0-p1 > > > > > +check multinode_nbctl lsp-set-type public-lr0-p1 router > > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p1 router > > > > > +check multinode_nbctl lsp-set-options public-lr0-p1 > > > > router-port=lr0-public-p1 > > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p1 > > ovn-gw-1 10 > > > > > + > > > > > +# Create a VM on ovn-chassis-3 in the same public1 overlay > > > > > +m_as ovn-chassis-3 /data/create_fake_vm.sh public-port1 publicp1 > > > > 40:54:00:00:00:03 1400 20.0.0.3 24 20.0.0.1 2000::4/64 2000::a > > > > > + > > > > > +m_wait_for_ports_up public-port1 > > > > > + > > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [ping -q -c 3 -i 0.3 > > -w 2 > > > > 20.0.0.1 | FORMAT_PING], \ > > > > > +[0], [dnl > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > +]) > > > > > + > > > > > +# Create the external LR0 port to the DGP public2 > > > > > +check multinode_nbctl lsp-add public public-port2 > > > > > +check multinode_nbctl lsp-set-addresses public-port2 > > "60:54:00:00:00:03 > > > > 30.0.0.3 3000::3" > > > > > + > > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p2 00:00:00:00:ff:03 > > > > 30.0.0.1/24 3000::a/64 > > > > > +check multinode_nbctl lsp-add public public-lr0-p2 > > > > > +check multinode_nbctl lsp-set-type public-lr0-p2 router > > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p2 router > > > > > +check multinode_nbctl lsp-set-options public-lr0-p2 > > > > router-port=lr0-public-p2 > > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p2 > > ovn-gw-2 10 > > > > > + > > > > > +# Create a VM on ovn-chassis-4 in the same public2 overlay > > > > > +m_as ovn-chassis-4 /data/create_fake_vm.sh public-port2 publicp2 > > > > 60:54:00:00:00:03 1400 30.0.0.3 24 30.0.0.1 3000::4/64 3000::a > > > > > + > > > > > +m_wait_for_ports_up public-port2 > > > > > + > > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [ping -q -c 3 -i 0.3 > > -w 2 > > > > 30.0.0.1 | FORMAT_PING], \ > > > > > +[0], [dnl > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > +]) > > > > > + > > > > > +# Add a default route for multiple DGPs - using ECMP > > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 > > 20.0.0.3 > > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 > > 30.0.0.3 > > > > > + > > > > > +# Add SNAT rules using gateway-port > > > > > +check multinode_nbctl --gateway-port lr0-public-p1 lr-nat-add lr0 > > snat > > > > 20.0.0.1 10.0.0.0/24 > > > > > +check multinode_nbctl --gateway-port lr0-public-p2 lr-nat-add lr0 > > snat > > > > 30.0.0.1 10.0.0.0/24 > > > > > + > > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 > > > > 20.0.0.3 | FORMAT_PING], \ > > > > > +[0], [dnl > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > +]) > > > > > + > > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 > > > > 30.0.0.3 | FORMAT_PING], \ > > > > > +[0], [dnl > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > +]) > > > > > + > > > > > +# create LB > > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:80" "10.0.0.3:80, > > > > 10.0.0.4:80" > > > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > > > + > > > > > +# Set use_stateless_nat to true > > > > > +check multinode_nbctl set load_balancer lb0 > > > > options:use_stateless_nat=true > > > > > + > > > > > +# Start backend http services > > > > > +M_NS_DAEMONIZE([ovn-chassis-1], [sw0p1], [$PYTHON -m http.server > > --bind > > > > 10.0.0.3 80 >/dev/null 2>&1], [http1.pid]) > > > > > +M_NS_DAEMONIZE([ovn-chassis-2], [sw0p2], [$PYTHON -m http.server > > --bind > > > > 10.0.0.4 80 >/dev/null 2>&1], [http2.pid]) > > > > > + > > > > > +# wait for http server be ready > > > > > +sleep 2 > > > > > + > > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v > > 172.16.0.100:80 > > > > --retry 3 --max-time 1 --local-port 59002 2> curl.out']) > > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [sh -c 'cat curl.out | > > > > grep -i -e connect | grep -v 'Server:''], \ > > > > > +[0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v > > 172.16.0.100:80 > > > > --retry 3 --max-time 1 --local-port 59003 2> curl.out']) > > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [sh -c 'cat curl.out | > > > > grep -i -e connect | grep -v 'Server:''], \ > > > > > +[0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v > > 172.16.0.100:80 > > > > --retry 3 --max-time 1 --local-port 59001']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | > > > > M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > [0], > > > > [dnl > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v > > 172.16.0.100:80 > > > > --retry 3 --max-time 1 --local-port 59000']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | > > > > M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > [0], > > > > [dnl > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# create a big file on web servers for download > > > > > +M_NS_EXEC([ovn-chassis-1], [sw0p1], [dd bs=512 count=200000 > > > > if=/dev/urandom of=download_file]) > > > > > +M_NS_EXEC([ovn-chassis-2], [sw0p2], [dd bs=512 count=200000 > > > > if=/dev/urandom of=download_file]) > > > > > + > > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > > 59004 > > > > 2>curl.out']) > > > > > + > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | > > sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | > > sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > + > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > > > > curl.out | \ > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +# Check if we have only one backend for the same connection - orig + > > > > dest ports > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > [0], > > > > [dnl > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Check if gw-2 is empty to ensure that the traffic only come > > from/to > > > > the originator chassis via DGP public1 > > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > + > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" > > -c) > > > > > + > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > +# Backend resides on ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59004,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" > > -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +else > > > > > +# Backend resides on ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59004,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" > > -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +fi > > > > > + > > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +# Check the flows again for a new source port > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > > 59005 > > > > 2>curl.out']) > > > > > + > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | > > sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | > > sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > + > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat > > > > curl.out | \ > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +# Check if we have only one backend for the same connection - orig + > > > > dest ports > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > [0], > > > > [dnl > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Check if gw-2 is empty to ensure that the traffic only come > > from/to > > > > the originator chassis via DGP public1 > > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > + > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" > > -c) > > > > > + > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > +# Backend resides on ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59005,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" > > -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +else > > > > > +# Backend resides on ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59005,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" > > -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +fi > > > > > + > > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +# Start a new test using the second DGP as origin (public2) > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > > 59006 > > > > 2>curl.out']) > > > > > + > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | > > sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | > > sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > + > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > > > > curl.out | \ > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +# Check if we have only one backend for the same connection - orig + > > > > dest ports > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > [0], > > > > [dnl > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Check if gw-1 is empty to ensure that the traffic only come > > from/to > > > > the originator chassis via DGP public2 > > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > + > > > > > +# Check the backend IP from ct entries on gw-2 (DGP public2) > > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" > > -c) > > > > > + > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > +# Backend resides on ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59006,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" > > -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +else > > > > > +# Backend resides on ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59006,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" > > -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +fi > > > > > + > > > > > +# Flush conntrack entries for easier output parsing of next test. > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +# Check the flows again for a new source port using the second DGP > > as > > > > origin (public2) > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > > 59007 > > > > 2>curl.out']) > > > > > + > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | > > sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | > > sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > + > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat > > > > curl.out | \ > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +# Check if we have only one backend for the same connection - orig + > > > > dest ports > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > [0], > > > > [dnl > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Check if gw-1 is empty to ensure that the traffic only come > > from/to > > > > the originator chassis via DGP public2 > > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > + > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" > > -c) > > > > > + > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > +# Backend resides on ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59007,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" > > -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +else > > > > > +# Backend resides on ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59007,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" > > -c], > > > > [1], [dnl > > > > > +0 > > > > > +]) > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" > > > > -c], [1], [dnl > > > > > +0 > > > > > +]) > > > > > +fi > > > > > + > > > > > +# Check multiple requests coming from DGP's public1 and public2 > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > > grep > > > > -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +200 OK > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > > grep > > > > -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +200 OK > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > > grep > > > > -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +200 OK > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > > grep > > > > -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > +200 OK > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +# Remove the LB and change the VIP port - different from the backend > > > > ports > > > > > +check multinode_nbctl lb-del lb0 > > > > > + > > > > > +# create LB again > > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:9000" "10.0.0.3:80, > > > > 10.0.0.4:80" > > > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > > > + > > > > > +# Set use_stateless_nat to true > > > > > +check multinode_nbctl set load_balancer lb0 > > > > options:use_stateless_nat=true > > > > > + > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +# Check end-to-end request using a new port for VIP > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port > > 59008 > > > > 2>curl.out']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | > > > > M_FORMAT_CT(20.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > [0], > > > > [dnl > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > > grep > > > > -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > > > +200 OK > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > + > > > > > +# Check end-to-end request using a new port for VIP > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port > > 59008 > > > > 2>curl.out']) > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | > > > > M_FORMAT_CT(30.0.0.3) | \ > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > [0], > > > > [dnl > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > +]) > > > > > + > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | > > grep > > > > -v 'Server:'], [0], [dnl > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > > > +200 OK > > > > > +* Closing connection > > > > > +]) > > > > > + > > > > > +AT_CLEANUP > > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > > > > > index dcc3dbbc3..9e7a2f225 100644 > > > > > --- a/tests/ovn-northd.at > > > > > +++ b/tests/ovn-northd.at > > > > > @@ -13864,3 +13864,323 @@ check_no_redirect > > > > > > > > > > AT_CLEANUP > > > > > ]) > > > > > + > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + > > NAT > > > > Stateless)]) > > > > > +ovn_start > > > > > + > > > > > +check ovn-nbctl ls-add public > > > > > +check ovn-nbctl lr-add lr1 > > > > > + > > > > > +# lr1 DGP ts1 > > > > > +check ovn-nbctl ls-add ts1 > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > > 172.16.10.1/24 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > > > + > > > > > +# lr1 DGP ts2 > > > > > +check ovn-nbctl ls-add ts2 > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > > 172.16.20.1/24 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > > > + > > > > > +# lr1 DGP public > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > > 173.16.0.1/16 > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > > + > > > > > +check ovn-nbctl ls-add s1 > > > > > +# s1 - lr1 > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > > 172.16.0.1" > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > > + > > > > > +# s1 - backend vm1 > > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > > 172.16.0.101" > > > > > + > > > > > +# s1 - backend vm2 > > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > > 172.16.0.102" > > > > > + > > > > > +# s1 - backend vm3 > > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > > 172.16.0.103" > > > > > + > > > > > +# Add the lr1 DGP ts1 to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > > > nat-addresses=router > > > > > + > > > > > +# Add the lr1 DGP ts2 to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > > > nat-addresses=router > > > > > + > > > > > +# Add the lr1 DGP public to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > > > nat-addresses=router > > > > > + > > > > > +# Create the Load Balancer lb1 > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > > > + > > > > > +# Set use_stateless_nat to true > > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > > > options:use_stateless_nat=true > > > > > + > > > > > +# Associate load balancer to s1 > > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > > +check ovn-nbctl --wait=sb sync > > > > > + > > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > > +AT_CAPTURE_FILE([s1flows]) > > > > > + > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | > > grep > > > > "30.0.0.1"], [0], [dnl > > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] > > == 1 > > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > > > +]) > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > > "30.0.0.1"], [0], [dnl > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > > > ip4.dst == 30.0.0.1), > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > +]) > > > > > + > > > > > +# Associate load balancer to lr1 with DGP > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > > +check ovn-nbctl --wait=sb sync > > > > > + > > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > > + > > > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > > "30.0.0.1"], [0], [dnl > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > is_chassis_resident("cr-lr1-ts1")), > > > > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > is_chassis_resident("cr-lr1-ts2")), > > > > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > is_chassis_resident("cr-lr1_public")), > > > > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > +]) > > > > > + > > > > > +# 2. Check if the DGP ports are in the match with action next > > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], > > [dnl > > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > > > action=(next;) > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") > > && > > > > is_chassis_resident("cr-lr1_public") && tcp), action=(next;) > > > > > +]) > > > > > + > > > > > +# 3. Check if the VIP IP is in the ipX.src action > > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > > > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > > > action=(next;) > > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > > > action=(next;) > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(ip4.src=30.0.0.1; > > next;) > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(ip4.src=30.0.0.1; > > next;) > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == > > > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") > > && > > > > is_chassis_resident("cr-lr1_public") && tcp), action=(ip4.src=30.0.0.1; > > > > next;) > > > > > +]) > > > > > + > > > > > +AT_CLEANUP > > > > > +]) > > > > > + > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + > > NAT > > > > Stateless) - IPv6]) > > > > > +ovn_start > > > > > + > > > > > +check ovn-nbctl ls-add public > > > > > +check ovn-nbctl lr-add lr1 > > > > > + > > > > > +# lr1 DGP ts1 > > > > > +check ovn-nbctl ls-add ts1 > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > > > > 2001:db8:aaaa:1::1/64 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > > > + > > > > > +# lr1 DGP ts2 > > > > > +check ovn-nbctl ls-add ts2 > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > > > > 2001:db8:aaaa:2::1/64 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > > > + > > > > > +# lr1 DGP public > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > > > > 2001:db8:bbbb::1/64 > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > > > > 2001:db8:aaaa:3::1/64 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > > + > > > > > +check ovn-nbctl ls-add s1 > > > > > +# s1 - lr1 > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > > > > 2001:db8:aaaa:3::1" > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > > + > > > > > +# s1 - backend vm1 > > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > > > > 2001:db8:aaaa:3::101" > > > > > + > > > > > +# s1 - backend vm2 > > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > > > > 2001:db8:aaaa:3::102" > > > > > + > > > > > +# s1 - backend vm3 > > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > > > > 2001:db8:aaaa:3::103" > > > > > + > > > > > +# Add the lr1 DGP ts1 to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > > > nat-addresses=router > > > > > + > > > > > +# Add the lr1 DGP ts2 to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > > > nat-addresses=router > > > > > + > > > > > +# Add the lr1 DGP public to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > > > nat-addresses=router > > > > > + > > > > > +# Create the Load Balancer lb1 > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "2001:db8:cccc::1" > > > > "2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101" > > > > > + > > > > > +# Set use_stateless_nat to true > > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > > > options:use_stateless_nat=true > > > > > + > > > > > +# Associate load balancer to s1 > > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > > +check ovn-nbctl --wait=sb sync > > > > > + > > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > > +AT_CAPTURE_FILE([s1flows]) > > > > > + > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | > > grep > > > > "2001:db8:cccc::1"], [0], [dnl > > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] > > == 1 > > > > && ip6.dst == 2001:db8:cccc::1), action=(xxreg1 = 2001:db8:cccc::1; > > > > ct_lb_mark;) > > > > > +]) > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > > "2001:db8:cccc::1"], [0], [dnl > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > > > ip6.dst == 2001:db8:cccc::1), > > > > > > action=(ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > +]) > > > > > + > > > > > +# Associate load balancer to lr1 with DGP > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > > +check ovn-nbctl --wait=sb sync > > > > > + > > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > > + > > > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > > "2001:db8:cccc::1"], [0], [dnl > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > > is_chassis_resident("cr-lr1-ts1")), > > > > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > > is_chassis_resident("cr-lr1-ts2")), > > > > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > > is_chassis_resident("cr-lr1_public")), > > > > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > +]) > > > > > + > > > > > +# 2. Check if the DGP ports are in the match with action next > > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], > > [dnl > > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > > > action=(next;) > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > 2001:db8:aaaa:3::102) || > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport > > == > > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > 2001:db8:aaaa:3::102) || > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport > > == > > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > 2001:db8:aaaa:3::102) || > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || > > outport == > > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > > > action=(next;) > > > > > +]) > > > > > + > > > > > +# 3. Check if the VIP IP is in the ipX.src action > > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl > > > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > > > action=(next;) > > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > > > action=(next;) > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > 2001:db8:aaaa:3::102) || > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport > > == > > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), > > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > 2001:db8:aaaa:3::102) || > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport > > == > > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), > > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > 2001:db8:aaaa:3::102) || > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || > > outport == > > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > > +]) > > > > > + > > > > > +AT_CLEANUP > > > > > +]) > > > > > + > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (DGP)]) > > > > > +ovn_start > > > > > + > > > > > +check ovn-nbctl ls-add public > > > > > +check ovn-nbctl lr-add lr1 > > > > > + > > > > > +# lr1 DGP ts1 > > > > > +check ovn-nbctl ls-add ts1 > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > > 172.16.10.1/24 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-1 > > > > > + > > > > > +# lr1 DGP ts2 > > > > > +check ovn-nbctl ls-add ts2 > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > > 172.16.20.1/24 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-1 > > > > > + > > > > > +# lr1 DGP public > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > > 173.16.0.1/16 > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > > + > > > > > +check ovn-nbctl ls-add s1 > > > > > +# s1 - lr1 > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > > 172.16.0.1" > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > > + > > > > > +# s1 - backend vm1 > > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > > 172.16.0.101" > > > > > + > > > > > +# s1 - backend vm2 > > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > > 172.16.0.102" > > > > > + > > > > > +# s1 - backend vm3 > > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > > 172.16.0.103" > > > > > + > > > > > +# Add the lr1 DGP ts1 to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 > > > > nat-addresses=router > > > > > + > > > > > +# Add the lr1 DGP ts2 to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 > > > > nat-addresses=router > > > > > + > > > > > +# Add the lr1 DGP public to the public switch > > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public > > > > nat-addresses=router > > > > > + > > > > > +# Create the Load Balancer lb1 > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > > > + > > > > > +# Associate load balancer to s1 > > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > > +check ovn-nbctl --wait=sb sync > > > > > + > > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > > +AT_CAPTURE_FILE([s1flows]) > > > > > + > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | > > grep > > > > "30.0.0.1"], [0], [dnl > > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] > > == 1 > > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > > > +]) > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > > "30.0.0.1"], [0], [dnl > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && > > > > ip4.dst == 30.0.0.1), > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > +]) > > > > > + > > > > > +# Associate load balancer to lr1 with DGP > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > > +check ovn-nbctl --wait=sb sync > > > > > + > > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > > + > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > > "30.0.0.1"], [0], [dnl > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > is_chassis_resident("cr-lr1-ts1")), > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > is_chassis_resident("cr-lr1-ts2")), > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > is_chassis_resident("cr-lr1_public")), > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > +]) > > > > > + > > > > > +AT_CLEANUP > > > > > +]) > > > > > -- > > > > > 2.34.1 > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > _'Esta mensagem é direcionada apenas para os endereços constantes no > > > > > cabeçalho inicial. Se você não está listado nos endereços constantes > > no > > > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo > > dessa > > > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas > > > > estão > > > > > imediatamente anuladas e proibidas'._ > > > > > > > > > > > > > > > * **'Apesar do Magazine Luiza tomar > > > > > todas as precauções razoáveis para assegurar que nenhum vírus esteja > > > > > presente nesse e-mail, a empresa não poderá aceitar a > > responsabilidade > > > > por > > > > > quaisquer perdas ou danos causados por esse e-mail ou por seus > > anexos'.* > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > dev mailing list > > > > > dev@openvswitch.org > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > _‘Esta mensagem é direcionada apenas para os endereços constantes no > > > cabeçalho inicial. Se você não está listado nos endereços constantes no > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas > > estão > > > imediatamente anuladas e proibidas’._ > > > > > > > > > * **‘Apesar do Magazine Luiza tomar > > > todas as precauções razoáveis para assegurar que nenhum vírus esteja > > > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade > > por > > > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* > > > > > > > > > > > > _______________________________________________ > > > dev mailing list > > > dev@openvswitch.org > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > -- > > > > > _‘Esta mensagem é direcionada apenas para os endereços constantes no > cabeçalho inicial. Se você não está listado nos endereços constantes no > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão > imediatamente anuladas e proibidas’._ > > > * **‘Apesar do Magazine Luiza tomar > todas as precauções razoáveis para assegurar que nenhum vírus esteja > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* > > > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Thanks Numan! Em seg., 7 de out. de 2024 às 11:34, Numan Siddique <nusiddiq@redhat.com> escreveu: > On Mon, Sep 30, 2024 at 1:57 PM Roberto Bartzen Acosta via dev > <ovs-dev@openvswitch.org> wrote: > > > > Hi Numan, > > > > Em sex., 27 de set. de 2024 às 12:47, Numan Siddique <numans@ovn.org> > > escreveu: > > > > > On Thu, Sep 26, 2024 at 10:55 AM Roberto Bartzen Acosta via dev > > > <ovs-dev@openvswitch.org> wrote: > > > > > > > > Hi Numan, > > > > > > > > Thanks for your feedback and review. > > > > > > > > Em qua., 25 de set. de 2024 às 19:50, Numan Siddique <numans@ovn.org > > > > > > escreveu: > > > > > > > > > On Thu, Sep 19, 2024 at 6:12 PM Roberto Bartzen Acosta via dev > > > > > <ovs-dev@openvswitch.org> wrote: > > > > > > > > > > > > This commit fixes the build_distr_lrouter_nat_flows_for_lb > function > > > to > > > > > > include a DNAT flow entry for each DGP in use. Since we have > added > > > > > support > > > > > > to create multiple gateway ports per logical router, it's > necessary > > > to > > > > > > include in the LR NAT rules pipeline a specific entry for each > > > attached > > > > > DGP. > > > > > > Otherwise, the inbound traffic will only be redirected when the > > > incoming > > > > > LRP > > > > > > matches the chassis_resident field. > > > > > > > > > > > > Additionally, this patch includes the ability to use > load-balancer > > > with > > > > > DGPs > > > > > > attached to multiple chassis. We can have each of the DGPs > associated > > > > > with a > > > > > > different chassis, and in this case the DNAT rules added by > default > > > will > > > > > not > > > > > > be enough to guarantee outgoing traffic. > > > > > > > > > > > > To solve the multiple chassis for DGPs problem, this patch > include a > > > new > > > > > > config options to be configured in the load-balancer. If the > > > > > use_stateless_nat > > > > > > is set to true, the logical router that references this > load-balancer > > > > > will use > > > > > > Stateless NAT rules when the logical router has multiple DGPs. > After > > > > > applying > > > > > > this patch and setting the use_stateless_nat option, the inbound > > > and/or > > > > > > outbound traffic can pass through any chassis where the DGP > resides > > > > > without > > > > > > having problems with CT state. > > > > > > > > > > > > Reported-at: > > > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2054322 > > > > > > Fixes: 15348b7b806f ("ovn-northd: Multiple distributed gateway > port > > > > > support.") > > > > > > > > > > > > Signed-off-by: Roberto Bartzen Acosta < > roberto.acosta@luizalabs.com> > > > > > > > > > > Hi Roberto, > > > > > > > > > > Thanks for the patch. I tested this patch using the test example > in > > > > > multinode.at. > > > > > > > > > > The test case adds the below load balancer > > > > > > > > > > [root@ovn-central ~]# ovn-nbctl lb-list > > > > > UUID LB PROTO > > > > > VIP IPs > > > > > f3e29869-3bb5-4df0-960a-171106f5913a lb0 tcp > > > > > 172.16.0.100:9000 10.0.0.3:80,10.0.0.4:80 > > > > > > > > > > And the below logical flows are generated by this patch > > > > > > > > > > -------- > > > > > [root@ovn-central ~]# ovn-sbctl dump-flows lr0 | grep 172.16.0.100 > > > > > table=6 (lr_in_defrag ), priority=100 , match=(ip && > ip4.dst > > > > > == 172.16.0.100), action=(ct_dnat;) > > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == > 9000 && > > > > > is_chassis_resident("cr-lr0-public-p1")), > > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > > > 10.0.0.3:80, > > > > > 10.0.0.4:80);) > > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && > > > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == > 9000 && > > > > > is_chassis_resident("cr-lr0-public-p2")), > > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > > > 10.0.0.3:80, > > > > > 10.0.0.4:80);) > > > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > > > > tcp.src == 80)) && (inport == "lr0-public-p1" || outport == > > > > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && > tcp), > > > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && > > > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && > > > > > tcp.src == 80)) && (inport == "lr0-public-p2" || outport == > > > > > "lr0-public-p2") && is_chassis_resident("cr-lr0-public-p2") && > tcp), > > > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) > > > > > -------------- > > > > > > > > > > > > > > > I fail to understand the reason for modifying the ip4.dst before > > > > > calling ct_lb_mark. Can you please explain why ? Because the > > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;) will first modify the > > > > > ip4.dst to 10.0.0.3 and > > > > > then to 10.0.0.4 and then the ct_lb_mark will actually do the > > > > > conntrack with NAT to either 10.0.0.3 or 10.0.0.4. > > > > > > > > > > Is it because you want the conntrack entry to not have > 172.16.0.100 ? > > > > > > > > > > > > > > > > The only reason I included this ip4.dst action in the DNAT rule is > > > because > > > > it's required to accept packets coming from a chassis that doesn't > have > > > > previously created conntrack entries. The main feature introduced in > this > > > > patch is to allow the administrator to have multiple DGPs attached to > > > > different chassis (is_chassis_resident...). So, my implementation was > > > based > > > > on the normal behavior when using stateless NAT for external > addresses, > > > > where we need to add the ipx.dst in lr_in_dnat for traffic to be > received > > > > on the chassis (put the DGP port as chassis_resident match, as is the > > > case > > > > with stateless NAT [1] with DGP[2]). > > > > > > > > The question is, if we only have the ct_lb_mark, packets that pass > > > through > > > > the chassis and are already part of an active flow in another chassis > > > (same > > > > IPs and Ports) will be dropped because there is no correspondence in > the > > > > backend. So only packets with the NEW flag will be accepted and sent > to > > > the > > > > backend (at least for TCP traffic). If we only have the ip4.dst > action, > > > > this will always perform the dnat for the same backend, without > > > balancing. > > > > Therefore, the combination of the two actions allows the packet to > always > > > > be received (regardless of whether conntrack is active for it), and > > > > ct_lb_mark will take care of balancing for different backends. > > > > > > > > If we had conntrack sync between different chassis this would not be > > > > necessary, as the ct_lb_mark action could always be executed without > > > > dropping packets due to lack of correspondence in the conntrack > table. > > > > > > > > [1] > > > > > > > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15737 > > > > [2] > > > > > > > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15726 > > > > > > > > > > > > > > I'm sorry, but it's not 100% clear to me. I know that you've already > > > explained to Mark in the older version of this patch. > > > Can you please explain with an example ? > > > > > > Let's take the below topology you've added in the mutlinode test as > example > > > > > > # Network topology > > > # > > > # publicp1 (ovn-chassis-3) (20.0.0.3/24) > > > # | > > > # overlay > > > # | > > > # DGP public1 (ovn-gw-1) (20.0.0.1/24) > > > # | > > > # | > > > # | > > > # lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) 10.0.0.3/24 > > > # | | > > > # | + --- sw0p2 (ovn-chassis-2) 10.0.0.4/24 > > > # | > > > # DGP public2 (ovn-gw-2) (30.0.0.1/24) > > > # | > > > # overlay > > > # | > > > # publicp2 (ovn-chassis-4) (30.0.0.3/24) > > > > > > > > > load balancer is configured on lr0 -> ovn-nbctl lb-add lb0 > > > "172.16.0.100:80" "10.0.0.3:80,10.0.0.4:80" > > > and it is attached to both lr0 and sw0. > > > > > > Scenario 1: > > > > > > publicp1 with IP 20.0.0.3 sends TCP traffic to VIP 172.16.0.100. I > > > think this is what will happen > > > > > > - The packet (ip4.src = 20.0.0.3 , ip4.dst = 172.16.0.100, tcp.dst > > > = 80) from ovn-chassis-3 will be sent out via the localnet bridge > > > (br-ex) > > > and the packet is received on ovn-gw-1 (as DGP public1 is resident > > > on it) via the localnet bridge and the > > > packet first enters public logical switch pipeline and then the > > > lr0 router pipeline. > > > > > > - In the router's lr_in_dnat state, the packet will be load > > > balanced to one of the backends using ct_lb_mark. Lets say 10.0.0.3 > > > is chosen > > > - The packet from router pipeline lr0 enters sw0 switch pipeline and > > > then the packet is tunneled to ovn-chassis-1 and delivered to sw0p1. > > > - The reply packet (ip4.src = 10.0.0.3, ip4.dst = 20.0.0.3, tcp.src = > > > 80) will enter the router pipeline and since 20.0.0.0/24 is handled by > > > DGP public1, > > > the packet is tunnelled to ovn-gw-1. > > > - In the router pipeline of ovn-gw-1, the packet is undnatted from > > > ip4.src 10.0.0.3 to 172.16.0.100 and the packet is sent out via the > > > localnet bridge. > > > - ovn-chassis-3 receives the packet via the localnet bridge and into > > > br-int and finally to publicp1. > > > > > > > > > In this scenario the load balancing is handled and conntrack entries > > > are created in ovn-gw-1. And there is no need to add flows in > > > "lr_out_snat" for stateless NAT > > > or set ip4.dst to one or all of backend IPs before "ct_lb_mark" in > > > lr_in_dnat stage. > > > > > > Scenario 2: > > > publicp2 with IP 30.0.0.3 sends TCP traffic to VIP 172.16.0.100. > > > > > > This is similar to scenario 1. Except that load balancing happens in > > > ovn-gw-2. > > > since DGP public2 is on this chassis. > > > > > > > > > Scenario 3: > > > > > > An external entity with IP 20.0.0.50 sends TCP traffic to VIP > > > 172.16.0.100. > > > > > > - This scenario is similar to the first one. The packet from this > > > external entity is received on ovn-gw-1 via the localnet bridge. > > > Rest all is the same. > > > > > > Scenario 4: > > > > > > sw0p2 on ovn-chassis-2 sends TCP traffic to VIP 172.16.0.100. > > > > > > - Since sw0 is attached with load balancer lb0, load balancing > > > happens in the source chassis - ovn-chassis-2 itself and depending on > > > the backend chosen, > > > the packet is tunnelled to ovn-chassis-1 (if 10.0.0.3 is chosen) > > > or delivered directly to sw0p2 (if i0.0.0.4 is chosen). > > > > > > > > > Scenario 5: > > > > > > An external entity with IP 40.0.0.40 sends TCP traffic to VIP > > > 172.16.0.100 and there are 2 ECMP routes configured > > > 172.16.0.100 via 20.0.0.1 > > > 172.16.0.100 via 30.0.0.1 > > > > > > In this case if the packet uses the route via 20.0.0.1 it will be > > > received on ovn-gw-1 via the localnet bridge br-ex. > > > > > > - With your patch, ip4.dst is modified to the last backend in the > > > list and then ct_lb_mark chosen as one of the backends > > > - And then it is tunnelled to the destination chassis. > > > > > > > > > Is this the scenario 5 you're trying to address with the stateless > > > NAT ? How would the reply packet work ? > > > The reply packet from backend 10.0.0.3 can use either of the paths ? > > > i.e ovn-gw-1 or ovn-gw-2 ? > > > > > > > > > Yes, this scenario would be the main goal of this patch. > > Let me explain with an example related to the OVN interconnect (one of > the > > use cases of this patch). In the context of ovn-ic the backend 10.0.0.3 > > could reply packets using either of the paths (ovn-gw-1 or ovn-gw-2). > > > > > > > > LB VIP 172.16.0.100 > > > > | > > vm (40.0.0.40) - LR-EXT - lrp1 - chassis1 - Transit switch TS1 - > ovn-gw-1 > > - lr0-public-p1 - lr0 - sw0 - VM 10.0.0.3 (ovn-chassis-1) > > - lrp2 - chassis2 - Transit > switch > > TS2 - ovn-gw-2 - lr0-public-p2 - - VM 10.0.0.4 > > (ovn-chassis-2) > > > > > > Let's take an example of the ovn-ic in the above topology: > > > > Assumptions: > > A) logical router LR-EXT: route to 172.16.0.100/32 via ECMP > (lrp1/chassis1 > > and lrp2/chassis2) > > B) logical router lr0: route to 40.0.0.0/24 via ECMP > > (lr0-public-p1/ovn-gw-1 and lr0-public-p2/ovn-gw-2) > > > > > > 1 - VM 40.0.0.40 send a request to the LB 172.16.0.100:80 > > 2 - LR-EXT chooses one of the possible outgoing routes via ECMP - e.g. > lrp1 > > / chassis1 / TS1 > > 3 - The lr0 receives the ingoing traffic through the port lr0-public-p1 / > > ovn-gw-1 because the TS1 is used to start the traffic > > 4 - with this patch we added lr_in_dnat rules for all DGPs (with or > without > > the stateless NAT config flag) > > > > with the stateless NAT config flag True: > > ...is_chassis_resident("cr-lr0-public-p1")), > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > 10.0.0.3:80 > > ,10.0.0.4:80);) > > ...is_chassis_resident("cr-lr0-public-p2")), > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > 10.0.0.3:80 > > ,10.0.0.4:80);) > > 5 - The ip4.dst will perform the action cumulatively, basically the last > > ip4.dst action applied will be to change dst to 10.0.0.4. > > 5.1 - The ct_lb_mark will check the conntrack and validate if the traffic > > refers to a new one (TCP SYNC), and then, it executes the final action > > provided by ct_lb_mark and select one of the available backends. e.g > > 10.0.0.3 > > > > without the stateless NAT config flag: > > ...is_chassis_resident("cr-lr0-public-p1")), > action=(ct_lb_mark(backends= > > 10.0.0.3:80,10.0.0.4:80);) > > ...is_chassis_resident("cr-lr0-public-p2")), > action=(ct_lb_mark(backends= > > 10.0.0.3:80,10.0.0.4:80);) > > > > 5 - same as the step 5.1 (ct_lb_mark check only). > > > > 6 - VM 10.0.0.3 receives the new traffic and sends a reply (TCP SYN+ACK) > > > > 7 - lr0 receives the response packet (from VM 10.0.0.3) in the routing > > pipeline and route via one of the available paths (ECMP). > > For example: lr0-public-p2 / ovn-gw-2 / TS2 > > So, we have a different outgoing path of the original one. > > > > 8 - ovn-gw-2 receives (SYN+ACK). We have no conntrack entries on ovn-gw-2 > > to match and perform the LB SNAT action! The ct_lb_mark previously > creates > > the conntrack entry on chassis ovn-gw-1 (when receives TCP SYN). > > > > with the stateless NAT config flag True: > > The outgoing packet fail to match the lb conntrack but the packet is > > SNATed by the lr_out_snat rule with a priority lower then cl_lb_mark, > > executing after cl_lb_mark in the pipeline. > > > > 9 - The SYN+ACK packet crosses the TS2 and is delivered to VM 40.0.0.40 > via > > lrp2 (SRC = 172.16.0.100:80) > > > > lr_out_snat will match and SNATed: > > match=(ip4 && ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == > > 10.0.0.4 && tcp.src == 80)) && (inport == "lr0-public-p1" || outport == > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), > > action=(ip4.src=172.16.0.100; tcp.src=80; next;) > > > > without the stateless NAT config flag: > > The outgoing will be returned to the VM 40.0.0.4 with the LR SNAT for > > public-p2 port as ip4.src! So, basically this will break the TCP > > handshake!!! > > > > 9 - The SYN+ACK packet cross the TS2 and is delivered to the VM 40.0.0.40 > > via lrp2 (SRC = 30.0.0.1:80) > > > > Moving forward on the happy path (stateless NAT is true): > > > > 10 - VM 40.0.0.40 sends an ACK to complete the TCP handshake. > > > > 11 - 2 - LR-EXT chooses one of the possible outgoing routes via ECMP > > Let's assume that LR-EXT correctly execute the ecmp algorithm (per flow > > basis) and forwards to the same initial path because we're using the same > > flow (src/dst IPs and src/dts TCP ports still the same). > > e.g. still using the lrp1 / chassis1 / TS1 > > > > 12 - The lr0 receive the ingoing traffic through the port lr0-public-p1 / > > ovn-gw-1 / TS1 > > > > With this patch we have 2 possible behaviours: > > > > 12.1 - Without Stateless NAT config flag > > > > table=1 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src > == > > 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && tcp.src == 80)) && > > (inport == "lr0-public-p1" || outport == > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1")), > > action=(ct_dnat_in_czone;) > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && !ct.rel > > && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > is_chassis_resident("cr-lr0-public-p1")), action=(ct_lb_mark(backends= > > 10.0.0.3:80,10.0.0.4:80);) > > > > openflow: br-int (ovn-gw-1) > > cookie=0x9381c20e, duration=2361.675s, table=15, n_packets=2, > n_bytes=148, > > idle_age=1564, > > > priority=120,ct_state=+new-rel+trk,tcp,metadata=0x4,nw_dst=172.16.0.100,tp_dst=80 > > actions=group:3 > > > > ovs-appctl dpctl/dump-conntrack (ovn-gw-1) > > > tcp,orig=(src=40.0.0.40,dst=172.16.0.100,sport=35274,dport=80),reply=(src=10.0.0.3,dst=40.0.0.40,sport=80,dport=35274),zone=8,mark=2,protoinfo=(state=TIME_WAIT) > > > > At this point, the ACK packet will be discarded because we didn't perform > > the SYN+ACK return on this chassis (ovn-gw -1). So, without the > previously > > established conntrack to match the ACK packet we broke the TCP handshake. > > > > Remember the flow: > > -> SYN: ovn-gw-1 > > <- SYN+ACK: ovn-gw-2 > > -> ACK: ovn-gw-1 > > > > 12.2 - With stateless NAT config flag is True > > > > table=1 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src > == > > 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && tcp.src == 80)) && > > (inport == "lr0-public-p1" || outport == > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1")), > action=(next;) > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && !ct.rel > > && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && > > is_chassis_resident("cr-lr0-public-p1")), > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= > 10.0.0.3:80 > > ,10.0.0.4:80);) > > > > openflow: br-int (ovn-gw-1) > > cookie=0xab99f240, duration=1086.022s, table=15, n_packets=610, > > n_bytes=48979, idle_age=825, > > > priority=120,ct_state=+new-rel+trk,tcp,metadata=0x2,nw_dst=172.16.0.100,tp_dst=80 > > actions=mod_nw_dst:10.0.0.3,mod_nw_dst:10.0.0.4,group:1 > > > > ovs-appctl dpctl/dump-conntrack (ovn-gw-1) > > > tcp,orig=(src=40.0.0.40,dst=10.0.0.3,sport=54632,dport=80),reply=(src=10.0.0.3,dst=40.0.0.40,sport=80,dport=54632),zone=7,mark=2,protoinfo=(state=SYN_SENT) > > > > This is the reason to use stateless NAT rules. Unlike the case where we > > don't have the SYN_SENT, now the ACK packet will be accepted and > forwarded > > after performing the mod_nw_dst action. > > > > How does this work? We already have the previous flow created by the > first > > packet (SYN), so regardless of the IP.dst modified in the action, the > > packet will be forwarded to the same backend that corresponds to the > > conntrack match that is in SYN_SENT state. That's why I created the > action > > with the two modifiers (ip4.dst + ct_lb_mark). Using only the ct_lb_mark > > action creates a strong dependency on the connection state match in the > > conntrack entry (which we don't have in these cases). However, only using > > ip4.dst doesn't create traffic balancing as it will always send to the > same > > backend (last in the ip4.dst). > > > > > > I hope this has helped clarify the design decisions when > creating/modifying > > flows (lr_in_dnat/lr_out_snat/lr_out_undnat). > > > > > > Thanks for the detailed explanation. Its clear to me now. > > I'd suggest the following > > 1. In lr_in_dnat stage, choose either the first backend or the last > backend to modify the ip4.dst before ct_lb_mark. > I've no strong preference. Either would do. > > 2. Please document or add comments in northd.c (and in ovn-nb.xml)so > that we don't lose the context later. > > 3. Please run the ovn-fake-multinode tests in your github CI repo and > make sure it passes. I think you can trigger the test in your repo > in the Actions tab. > > 4. In the setup you described above, is it possible to add the > (below) route in the vm (40.0.0.40) deployment so that 40.0.0.0.40 > can > talk directly to 10.0.0.3 and 10.0.0.4 instead of the LB VIP and > send the tcp traffic to 10.0.0.3 and see if it works ? > If not, then please document the same i.e with the option > stateless_nat in the LB, the backends cannot be directly > communicated. > > logical router LR-EXT: route to 10.0.0.0/24 via ECMP (lrp1/chassis1 > and lrp2/chassis2) > > I'll do your suggestions. Just a quick question: how can I customize/parametrize the ovn-fake-multinode elements to run on CI? I mean to enable 2 more ovn-chassis needed to run the related tests. I've enabled it locally in the ./ovn_cluster.sh file but how do I change this for the fake-multinode CI actions? Is it per job or is this global? Regarding your suggestion 4, I imagine it makes more sense to integrate/create a new test with the ovn-interconnect setup + LB + multiple DGPs, which allows us to have multiple paths on routers on both sides (addressing the case discussed above) Thanks, Roberto > Thanks > Numan > > > > > > > > > > > > Thanks > > > Numan > > > > > > > > > > > > > > > > Also I don't understand why this patch adds the logical flows in > > > > > "lr_out_snat" stage ? > > > > > > > > > > > > > The flow for lr_out_snat is necessary for the correct functioning of > > > > stateless NAT for the same reason explained previously. I mean, if > the > > > > outgoing packet is redirected to a chassis that doesn't have an > active > > > > conntrack entry, it will not be NATed by ct_lb action because it > doesn't > > > > refer to a valid flow (use case with ecmp). > > > > > > > > So it is necessary to create a stateless SNAT rule (similar to this > [3]) > > > > with a lower priority than the other router pipeline entries, in this > > > case, > > > > if the packet is not SNATed by ct_lb (conntrack missed) it will be > SNATed > > > > by stateless NAT rule. > > > > > > > > [3] > > > > > > > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15884 > > > > > > > > > > > > > > > > > > > > > > Using the system multinode test as an example, the below fails > > > > > (which is a regression) > > > > > > > > > > --- > > > > > root@ovn-chassis-3 ~]# ip netns exec publicp1 nc -vz 10.0.0.3 80 > > > > > ---- > > > > > > > > > > In the above test, publicp1 with IP 20.0.0.3 when it tries to > connect > > > > > to one if the backends directly (without the LB VIP), it fails. > > > > > It fails because of the logical flows in "lr_out_snat". > > > > > > > > > > > > > > > Looks to me the solution proposed here is incomplete. > > > > > > > > > > Also please note that in our CI we run the multinode tests > > > > > periodically once a day using the v0.1 of the ovn-fake-multinode > > > > > and the tests you added will fail. This needs to be fixed and > until > > > > > we move to the latest version of ovn-fake-multinode. > > > > > > > > > > > > > I imagine that the test you are doing is using the same port as the > LB > > > > backend (TCP 80 in this case). So, the stateless lr_out_snat flow > will > > > > force the output to be SNATed because this port is in use by the > backend. > > > > Traffic to/from other ports will work without problems and will > follow > > > the > > > > normal programmed flows (e.g. ICMP). > > > > > > > > This is necessary to ensure the egress traffic because the DGPs are > > > > distributed across multiple chassis. Also, this setup is being > validated > > > in > > > > the test ovn-fake-multinode testcase (ICMP from the backends chassis > use > > > > the router's default SNAT and not the LB's). I didn't understand the > > > > regression you mentioned because this was programmed to be stateless > and > > > > it's traffic that uses the same ports as the LB backend, could you > > > explain > > > > better? > > > > > > > > Thanks, > > > > Roberto > > > > > > > > > > > > > Thanks > > > > > Numan > > > > > > > > > > > > > > > > --- > > > > > > northd/en-lr-stateful.c | 12 - > > > > > > northd/northd.c | 116 ++++++-- > > > > > > ovn-nb.xml | 10 + > > > > > > tests/multinode-macros.at | 40 +++ > > > > > > tests/multinode.at | 556 > > > ++++++++++++++++++++++++++++++++++++++ > > > > > > tests/ovn-northd.at | 320 ++++++++++++++++++++++ > > > > > > 6 files changed, 1017 insertions(+), 37 deletions(-) > > > > > > > > > > > > diff --git a/northd/en-lr-stateful.c b/northd/en-lr-stateful.c > > > > > > index baf1bd2f8..f09691af6 100644 > > > > > > --- a/northd/en-lr-stateful.c > > > > > > +++ b/northd/en-lr-stateful.c > > > > > > @@ -516,18 +516,6 @@ lr_stateful_record_create(struct > > > lr_stateful_table > > > > > *table, > > > > > > > > > > > > table->array[od->index] = lr_stateful_rec; > > > > > > > > > > > > - /* Load balancers are not supported (yet) if a logical > router > > > has > > > > > multiple > > > > > > - * distributed gateway port. Log a warning. */ > > > > > > - if (lr_stateful_rec->has_lb_vip && > > > lr_has_multiple_gw_ports(od)) { > > > > > > - static struct vlog_rate_limit rl = > VLOG_RATE_LIMIT_INIT(1, > > > 1); > > > > > > - VLOG_WARN_RL(&rl, "Load-balancers are configured on > logical > > > " > > > > > > - "router %s, which has %"PRIuSIZE" > distributed " > > > > > > - "gateway ports. Load-balancer is not > supported > > > " > > > > > > - "yet when there is more than one > distributed " > > > > > > - "gateway port on the router.", > > > > > > - od->nbr->name, od->n_l3dgw_ports); > > > > > > - } > > > > > > - > > > > > > return lr_stateful_rec; > > > > > > } > > > > > > > > > > > > diff --git a/northd/northd.c b/northd/northd.c > > > > > > index a267cd5f8..bbe97acf8 100644 > > > > > > --- a/northd/northd.c > > > > > > +++ b/northd/northd.c > > > > > > @@ -11807,31 +11807,30 @@ static void > > > > > > build_distr_lrouter_nat_flows_for_lb(struct > lrouter_nat_lb_flows_ctx > > > > > *ctx, > > > > > > enum > lrouter_nat_lb_flow_type > > > type, > > > > > > struct ovn_datapath *od, > > > > > > - struct lflow_ref > *lflow_ref) > > > > > > + struct lflow_ref > *lflow_ref, > > > > > > + struct ovn_port *dgp, > > > > > > + bool stateless_nat) > > > > > > { > > > > > > - struct ovn_port *dgp = od->l3dgw_ports[0]; > > > > > > - > > > > > > - const char *undnat_action; > > > > > > - > > > > > > - switch (type) { > > > > > > - case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > > > > - undnat_action = "flags.force_snat_for_lb = 1; next;"; > > > > > > - break; > > > > > > - case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > > > > - undnat_action = "flags.skip_snat_for_lb = 1; next;"; > > > > > > - break; > > > > > > - case LROUTER_NAT_LB_FLOW_NORMAL: > > > > > > - case LROUTER_NAT_LB_FLOW_MAX: > > > > > > - undnat_action = lrouter_use_common_zone(od) > > > > > > - ? "ct_dnat_in_czone;" > > > > > > - : "ct_dnat;"; > > > > > > - break; > > > > > > - } > > > > > > + struct ds dnat_action = DS_EMPTY_INITIALIZER; > > > > > > > > > > > > /* Store the match lengths, so we can reuse the ds buffer. > */ > > > > > > size_t new_match_len = ctx->new_match->length; > > > > > > size_t undnat_match_len = ctx->undnat_match->length; > > > > > > > > > > > > + /* dnat_action: Add the LB backend IPs as a destination > action > > > of > > > > > the > > > > > > + * lr_in_dnat NAT rule with cumulative effect > > > because > > > > > any > > > > > > + * backend dst IP used in the action list will > > > > > redirect the > > > > > > + * packet to the ct_lb pipeline. > > > > > > + */ > > > > > > + if (stateless_nat) { > > > > > > + for (size_t i = 0; i < ctx->lb_vip->n_backends; i++) { > > > > > > + struct ovn_lb_backend *backend = > > > &ctx->lb_vip->backends[i]; > > > > > > + bool ipv6 = !IN6_IS_ADDR_V4MAPPED(&backend->ip); > > > > > > + ds_put_format(&dnat_action, "%s.dst=%s;", ipv6 ? > "ip6" : > > > > > "ip4", > > > > > > + backend->ip_str); > > > > > > + } > > > > > > + } > > > > > > + ds_put_format(&dnat_action, "%s", ctx->new_action[type]); > > > > > > > > > > > > const char *meter = NULL; > > > > > > > > > > > > @@ -11841,20 +11840,46 @@ > build_distr_lrouter_nat_flows_for_lb(struct > > > > > lrouter_nat_lb_flows_ctx *ctx, > > > > > > > > > > > > if (ctx->lb_vip->n_backends || > !ctx->lb_vip->empty_backend_rej) > > > { > > > > > > ds_put_format(ctx->new_match, " && > is_chassis_resident(%s)", > > > > > > - od->l3dgw_ports[0]->cr_port->json_key); > > > > > > + dgp->cr_port->json_key); > > > > > > } > > > > > > > > > > > > ovn_lflow_add_with_hint__(ctx->lflows, od, S_ROUTER_IN_DNAT, > > > > > ctx->prio, > > > > > > - ds_cstr(ctx->new_match), > > > > > ctx->new_action[type], > > > > > > + ds_cstr(ctx->new_match), > > > > > ds_cstr(&dnat_action), > > > > > > NULL, meter, > &ctx->lb->nlb->header_, > > > > > > lflow_ref); > > > > > > > > > > > > ds_truncate(ctx->new_match, new_match_len); > > > > > > > > > > > > + ds_destroy(&dnat_action); > > > > > > if (!ctx->lb_vip->n_backends) { > > > > > > return; > > > > > > } > > > > > > > > > > > > + struct ds undnat_action = DS_EMPTY_INITIALIZER; > > > > > > + struct ds snat_action = DS_EMPTY_INITIALIZER; > > > > > > + > > > > > > + switch (type) { > > > > > > + case LROUTER_NAT_LB_FLOW_FORCE_SNAT: > > > > > > + ds_put_format(&undnat_action, "flags.force_snat_for_lb > = 1; > > > > > next;"); > > > > > > + break; > > > > > > + case LROUTER_NAT_LB_FLOW_SKIP_SNAT: > > > > > > + ds_put_format(&undnat_action, "flags.skip_snat_for_lb = > 1; > > > > > next;"); > > > > > > + break; > > > > > > + case LROUTER_NAT_LB_FLOW_NORMAL: > > > > > > + case LROUTER_NAT_LB_FLOW_MAX: > > > > > > + ds_put_format(&undnat_action, "%s", > > > > > > + lrouter_use_common_zone(od) ? > > > "ct_dnat_in_czone;" > > > > > > + : "ct_dnat;"); > > > > > > + break; > > > > > > + } > > > > > > + > > > > > > + /* undnat_action: Remove the ct action from the > lr_out_undenat > > > NAT > > > > > rule. > > > > > > + */ > > > > > > + if (stateless_nat) { > > > > > > + ds_clear(&undnat_action); > > > > > > + ds_put_format(&undnat_action, "next;"); > > > > > > + } > > > > > > + > > > > > > /* We need to centralize the LB traffic to properly perform > > > > > > * the undnat stage. > > > > > > */ > > > > > > @@ -11873,11 +11898,41 @@ > build_distr_lrouter_nat_flows_for_lb(struct > > > > > lrouter_nat_lb_flows_ctx *ctx, > > > > > > ds_put_format(ctx->undnat_match, ") && (inport == %s || > outport > > > == > > > > > %s)" > > > > > > " && is_chassis_resident(%s)", dgp->json_key, > > > > > dgp->json_key, > > > > > > dgp->cr_port->json_key); > > > > > > + /* Use the LB protocol as matching criteria for out undnat > and > > > snat > > > > > when > > > > > > + * creating LBs with stateless NAT. */ > > > > > > + if (stateless_nat) { > > > > > > + ds_put_format(ctx->undnat_match, " && %s", > ctx->lb->proto); > > > > > > + } > > > > > > ovn_lflow_add_with_hint(ctx->lflows, od, > S_ROUTER_OUT_UNDNAT, > > > 120, > > > > > > - ds_cstr(ctx->undnat_match), > > > undnat_action, > > > > > > - &ctx->lb->nlb->header_, > > > > > > + ds_cstr(ctx->undnat_match), > > > > > > + ds_cstr(&undnat_action), > > > > > &ctx->lb->nlb->header_, > > > > > > lflow_ref); > > > > > > + > > > > > > + /* snat_action: Add a new lr_out_snat rule with the LB VIP > as > > > > > source IP > > > > > > + * action to perform the NAT stateless pipeline > > > > > completely. > > > > > > + */ > > > > > > + if (stateless_nat) { > > > > > > + if (ctx->lb_vip->port_str) { > > > > > > + ds_put_format(&snat_action, "%s.src=%s; %s.src=%s; > > > next;", > > > > > > + ctx->lb_vip->address_family == > AF_INET6 ? > > > > > > + "ip6" : "ip4", > > > > > > + ctx->lb_vip->vip_str, ctx->lb->proto, > > > > > > + ctx->lb_vip->port_str); > > > > > > + } else { > > > > > > + ds_put_format(&snat_action, "%s.src=%s; next;", > > > > > > + ctx->lb_vip->address_family == > AF_INET6 ? > > > > > > + "ip6" : "ip4", > > > > > > + ctx->lb_vip->vip_str); > > > > > > + } > > > > > > + ovn_lflow_add_with_hint(ctx->lflows, od, > S_ROUTER_OUT_SNAT, > > > 160, > > > > > > + ds_cstr(ctx->undnat_match), > > > > > > + ds_cstr(&snat_action), > > > > > &ctx->lb->nlb->header_, > > > > > > + lflow_ref); > > > > > > + } > > > > > > + > > > > > > ds_truncate(ctx->undnat_match, undnat_match_len); > > > > > > + ds_destroy(&undnat_action); > > > > > > + ds_destroy(&snat_action); > > > > > > } > > > > > > > > > > > > static void > > > > > > @@ -12022,6 +12077,8 @@ build_lrouter_nat_flows_for_lb( > > > > > > * lflow generation for them. > > > > > > */ > > > > > > size_t index; > > > > > > + bool use_stateless_nat = smap_get_bool(&lb->nlb->options, > > > > > > + "use_stateless_nat", > > > false); > > > > > > BITMAP_FOR_EACH_1 (index, bitmap_len, lb_dps->nb_lr_map) { > > > > > > struct ovn_datapath *od = lr_datapaths->array[index]; > > > > > > enum lrouter_nat_lb_flow_type type; > > > > > > @@ -12043,8 +12100,17 @@ build_lrouter_nat_flows_for_lb( > > > > > > if (!od->n_l3dgw_ports) { > > > > > > bitmap_set1(gw_dp_bitmap[type], index); > > > > > > } else { > > > > > > - build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, > > > > > > - > lb_dps->lflow_ref); > > > > > > + /* Create stateless LB NAT rules when using multiple > > > DGPs > > > > > and > > > > > > + * use_stateless_nat is true. > > > > > > + */ > > > > > > + bool stateless_nat = (od->n_l3dgw_ports > 1) > > > > > > + ? use_stateless_nat : false; > > > > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { > > > > > > + struct ovn_port *dgp = od->l3dgw_ports[i]; > > > > > > + build_distr_lrouter_nat_flows_for_lb(&ctx, > type, od, > > > > > > + > > > lb_dps->lflow_ref, > > > > > dgp, > > > > > > + > stateless_nat); > > > > > > + } > > > > > > } > > > > > > > > > > > > if (lb->affinity_timeout) { > > > > > > diff --git a/ovn-nb.xml b/ovn-nb.xml > > > > > > index 2836f58f5..ad03c6214 100644 > > > > > > --- a/ovn-nb.xml > > > > > > +++ b/ovn-nb.xml > > > > > > @@ -2302,6 +2302,16 @@ or > > > > > > local anymore by the ovn-controller. This option is set > to > > > > > > <code>false</code> by default. > > > > > > </column> > > > > > > + > > > > > > + <column name="options" key="use_stateless_nat" > > > > > > + type='{"type": "boolean"}'> > > > > > > + If the load balancer is configured with > > > > > <code>use_stateless_nat</code> > > > > > > + option to <code>true</code>, the logical router that > > > references > > > > > this > > > > > > + load balancer will use Stateless NAT rules when the > logical > > > > > router > > > > > > + has multiple distributed gateway ports(DGP). Otherwise, > the > > > > > outbound > > > > > > + traffic may be dropped in scenarios where we have > different > > > > > chassis > > > > > > + for each DGP. This option is set to <code>false</code> > by > > > > > default. > > > > > > + </column> > > > > > > </group> > > > > > > </table> > > > > > > > > > > > > diff --git a/tests/multinode-macros.at b/tests/ > multinode-macros.at > > > > > > index 757917626..2f69433fc 100644 > > > > > > --- a/tests/multinode-macros.at > > > > > > +++ b/tests/multinode-macros.at > > > > > > @@ -40,6 +40,27 @@ m4_define([M_START_TCPDUMP], > > > > > > ] > > > > > > ) > > > > > > > > > > > > +# M_EXEC([fake_node], [command]) > > > > > > +# > > > > > > +# Execute 'command' in 'fakenode' > > > > > > +m4_define([M_EXEC], > > > > > > + [podman exec $1 $2]) > > > > > > + > > > > > > +# M_CHECK_EXEC([fake_node], [command], other_params...) > > > > > > +# > > > > > > +# Wrapper for AT_CHECK that executes 'command' inside > > > 'fake_node''s'. > > > > > > +# 'other_params' as passed as they are to AT_CHECK. > > > > > > +m4_define([M_CHECK_EXEC], > > > > > > + [ AT_CHECK([M_EXEC([$1], [$2])], > > > m4_shift(m4_shift(m4_shift($@)))) ] > > > > > > +) > > > > > > + > > > > > > +# M_FORMAT_CT([ip-addr]) > > > > > > +# > > > > > > +# Strip content from the piped input which would differ from > test to > > > > > test > > > > > > +# and limit the output to the rows containing 'ip-addr'. > > > > > > +# > > > > > > +m4_define([M_FORMAT_CT], > > > > > > + [[grep -F "dst=$1," | sed -e 's/id=[0-9]*/id=<cleared>/g' -e > > > > > 's/state=[0-9_A-Z]*/state=<cleared>/g' | sort | uniq | sed -e > > > > > 's/zone=[[0-9]]*/zone=<cleared>/' -e > 's/mark=[[0-9]]*/mark=<cleared>/' > > > ]]) > > > > > > > > > > > > OVS_START_SHELL_HELPERS > > > > > > > > > > > > @@ -76,6 +97,25 @@ multinode_nbctl () { > > > > > > m_as ovn-central ovn-nbctl "$@" > > > > > > } > > > > > > > > > > > > +check_fake_multinode_setup_by_nodes() { > > > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > > > > + for c in $1 > > > > > > + do > > > > > > + AT_CHECK([m_as $c ovn-appctl -t ovn-controller version], > > > [0], > > > > > [ignore]) > > > > > > + done > > > > > > +} > > > > > > + > > > > > > +cleanup_multinode_resources_by_nodes() { > > > > > > + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db > > > > > > + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl > restart_northd > > > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync > > > > > > + for c in $1 > > > > > > + do > > > > > > + m_as $c ovs-vsctl del-br br-int > > > > > > + m_as $c ip --all netns delete > > > > > > + done > > > > > > +} > > > > > > + > > > > > > # m_count_rows TABLE [CONDITION...] > > > > > > # > > > > > > # Prints the number of rows in TABLE (that satisfy CONDITION). > > > > > > diff --git a/tests/multinode.at b/tests/multinode.at > > > > > > index a0eb8fc67..b1beb4d97 100644 > > > > > > --- a/tests/multinode.at > > > > > > +++ b/tests/multinode.at > > > > > > @@ -1591,3 +1591,559 @@ AT_CHECK([cat ch1_eth2.tcpdump], [0], > [dnl > > > > > > ]) > > > > > > > > > > > > AT_CLEANUP > > > > > > + > > > > > > +AT_SETUP([ovn multinode load-balancer with multiple DGPs and > > > multiple > > > > > chassis]) > > > > > > + > > > > > > +# Check that ovn-fake-multinode setup is up and running - > requires > > > > > additional nodes > > > > > > +check_fake_multinode_setup_by_nodes 'ovn-chassis-1 ovn-chassis-2 > > > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > > > > + > > > > > > +# Delete the multinode NB and OVS resources before starting the > > > test. > > > > > > +cleanup_multinode_resources_by_nodes 'ovn-chassis-1 > ovn-chassis-2 > > > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' > > > > > > + > > > > > > +# Network topology > > > > > > +# > > > > > > +# publicp1 (ovn-chassis-3) (20.0.0.3/24) > > > > > > +# | > > > > > > +# overlay > > > > > > +# | > > > > > > +# DGP public1 (ovn-gw-1) (20.0.0.1/24) > > > > > > +# | > > > > > > +# | > > > > > > +# | > > > > > > +# lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) > > > 10.0.0.3/24 > > > > > > +# | | > > > > > > +# | + --- sw0p2 (ovn-chassis-2) > > > 10.0.0.4/24 > > > > > > +# | > > > > > > +# DGP public2 (ovn-gw-2) (30.0.0.1/24) > > > > > > +# | > > > > > > +# overlay > > > > > > +# | > > > > > > +# publicp2 (ovn-chassis-4) (30.0.0.3/24) > > > > > > + > > > > > > +# Delete already used ovs-ports > > > > > > +m_as ovn-chassis-1 ovs-vsctl del-port br-int sw0p1-p > > > > > > +m_as ovn-chassis-2 ovs-vsctl del-port br-int sw0p2-p > > > > > > +m_as ovn-chassis-1 ip link del sw0p1-p > > > > > > +m_as ovn-chassis-2 ip link del sw0p2-p > > > > > > +m_as ovn-chassis-3 ovs-vsctl del-port br-int publicp1-p > > > > > > +m_as ovn-chassis-4 ovs-vsctl del-port br-int publicp2-p > > > > > > +m_as ovn-chassis-3 ip link del publicp1-p > > > > > > +m_as ovn-chassis-4 ip link del publicp2-p > > > > > > + > > > > > > +# Create East-West switch for LB backends > > > > > > +check multinode_nbctl ls-add sw0 > > > > > > +check multinode_nbctl lsp-add sw0 sw0-port1 > > > > > > +check multinode_nbctl lsp-set-addresses sw0-port1 > "50:54:00:00:00:03 > > > > > 10.0.0.3 1000::3" > > > > > > +check multinode_nbctl lsp-add sw0 sw0-port2 > > > > > > +check multinode_nbctl lsp-set-addresses sw0-port2 > "50:54:00:00:00:04 > > > > > 10.0.0.4 1000::4" > > > > > > + > > > > > > +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 > > > > > 50:54:00:00:00:03 1400 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a > > > > > > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 > > > > > 50:54:00:00:00:04 1400 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a > > > > > > + > > > > > > +m_wait_for_ports_up > > > > > > + > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 > -w 2 > > > > > 10.0.0.4 | FORMAT_PING], \ > > > > > > +[0], [dnl > > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > > +]) > > > > > > + > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 > -w 2 > > > > > 10.0.0.3 | FORMAT_PING], \ > > > > > > +[0], [dnl > > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > > +]) > > > > > > + > > > > > > +# Create a logical router and attach to sw0 > > > > > > +check multinode_nbctl lr-add lr0 > > > > > > +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 > > > 10.0.0.1/24 > > > > > 1000::a/64 > > > > > > +check multinode_nbctl lsp-add sw0 sw0-lr0 > > > > > > +check multinode_nbctl lsp-set-type sw0-lr0 router > > > > > > +check multinode_nbctl lsp-set-addresses sw0-lr0 router > > > > > > +check multinode_nbctl lsp-set-options sw0-lr0 > router-port=lr0-sw0 > > > > > > + > > > > > > +# create external connection for N/S traffic using multiple DGPs > > > > > > +check multinode_nbctl ls-add public > > > > > > + > > > > > > +# DGP public1 > > > > > > +check multinode_nbctl lsp-add public ln-public-1 > > > > > > +check multinode_nbctl lsp-set-type ln-public-1 localnet > > > > > > +check multinode_nbctl lsp-set-addresses ln-public-1 unknown > > > > > > +check multinode_nbctl lsp-set-options ln-public-1 > > > network_name=public1 > > > > > > + > > > > > > +# DGP public2 > > > > > > +# create exteranl connection for N/S traffic > > > > > > +check multinode_nbctl lsp-add public ln-public-2 > > > > > > +check multinode_nbctl lsp-set-type ln-public-2 localnet > > > > > > +check multinode_nbctl lsp-set-addresses ln-public-2 unknown > > > > > > +check multinode_nbctl lsp-set-options ln-public-2 > > > network_name=public2 > > > > > > + > > > > > > +# Attach DGP public1 to GW-1 and chassis-3 (overlay > connectivity) > > > > > > +m_as ovn-gw-1 ovs-vsctl set open . > > > > > external-ids:ovn-bridge-mappings=public1:br-ex > > > > > > +m_as ovn-chassis-3 ovs-vsctl set open . > > > > > external-ids:ovn-bridge-mappings=public1:br-ex > > > > > > + > > > > > > +# Attach DGP public2 to GW-2 and chassis-4 (overlay > connectivity) > > > > > > +m_as ovn-gw-2 ovs-vsctl set open . > > > > > external-ids:ovn-bridge-mappings=public2:br-ex > > > > > > +m_as ovn-chassis-4 ovs-vsctl set open . > > > > > external-ids:ovn-bridge-mappings=public2:br-ex > > > > > > + > > > > > > +# Create the external LR0 port to the DGP public1 > > > > > > +check multinode_nbctl lsp-add public public-port1 > > > > > > +check multinode_nbctl lsp-set-addresses public-port1 > > > "40:54:00:00:00:03 > > > > > 20.0.0.3 2000::3" > > > > > > + > > > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p1 > 00:00:00:00:ff:02 > > > > > 20.0.0.1/24 2000::a/64 > > > > > > +check multinode_nbctl lsp-add public public-lr0-p1 > > > > > > +check multinode_nbctl lsp-set-type public-lr0-p1 router > > > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p1 router > > > > > > +check multinode_nbctl lsp-set-options public-lr0-p1 > > > > > router-port=lr0-public-p1 > > > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p1 > > > ovn-gw-1 10 > > > > > > + > > > > > > +# Create a VM on ovn-chassis-3 in the same public1 overlay > > > > > > +m_as ovn-chassis-3 /data/create_fake_vm.sh public-port1 publicp1 > > > > > 40:54:00:00:00:03 1400 20.0.0.3 24 20.0.0.1 2000::4/64 2000::a > > > > > > + > > > > > > +m_wait_for_ports_up public-port1 > > > > > > + > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [ping -q -c 3 -i > 0.3 > > > -w 2 > > > > > 20.0.0.1 | FORMAT_PING], \ > > > > > > +[0], [dnl > > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > > +]) > > > > > > + > > > > > > +# Create the external LR0 port to the DGP public2 > > > > > > +check multinode_nbctl lsp-add public public-port2 > > > > > > +check multinode_nbctl lsp-set-addresses public-port2 > > > "60:54:00:00:00:03 > > > > > 30.0.0.3 3000::3" > > > > > > + > > > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p2 > 00:00:00:00:ff:03 > > > > > 30.0.0.1/24 3000::a/64 > > > > > > +check multinode_nbctl lsp-add public public-lr0-p2 > > > > > > +check multinode_nbctl lsp-set-type public-lr0-p2 router > > > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p2 router > > > > > > +check multinode_nbctl lsp-set-options public-lr0-p2 > > > > > router-port=lr0-public-p2 > > > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p2 > > > ovn-gw-2 10 > > > > > > + > > > > > > +# Create a VM on ovn-chassis-4 in the same public2 overlay > > > > > > +m_as ovn-chassis-4 /data/create_fake_vm.sh public-port2 publicp2 > > > > > 60:54:00:00:00:03 1400 30.0.0.3 24 30.0.0.1 3000::4/64 3000::a > > > > > > + > > > > > > +m_wait_for_ports_up public-port2 > > > > > > + > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [ping -q -c 3 -i > 0.3 > > > -w 2 > > > > > 30.0.0.1 | FORMAT_PING], \ > > > > > > +[0], [dnl > > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > > +]) > > > > > > + > > > > > > +# Add a default route for multiple DGPs - using ECMP > > > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 > > > 20.0.0.3 > > > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 > > > 30.0.0.3 > > > > > > + > > > > > > +# Add SNAT rules using gateway-port > > > > > > +check multinode_nbctl --gateway-port lr0-public-p1 lr-nat-add > lr0 > > > snat > > > > > 20.0.0.1 10.0.0.0/24 > > > > > > +check multinode_nbctl --gateway-port lr0-public-p2 lr-nat-add > lr0 > > > snat > > > > > 30.0.0.1 10.0.0.0/24 > > > > > > + > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 > -w 2 > > > > > 20.0.0.3 | FORMAT_PING], \ > > > > > > +[0], [dnl > > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > > +]) > > > > > > + > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 > -w 2 > > > > > 30.0.0.3 | FORMAT_PING], \ > > > > > > +[0], [dnl > > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms > > > > > > +]) > > > > > > + > > > > > > +# create LB > > > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:80" "10.0.0.3:80 > , > > > > > 10.0.0.4:80" > > > > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > > > > + > > > > > > +# Set use_stateless_nat to true > > > > > > +check multinode_nbctl set load_balancer lb0 > > > > > options:use_stateless_nat=true > > > > > > + > > > > > > +# Start backend http services > > > > > > +M_NS_DAEMONIZE([ovn-chassis-1], [sw0p1], [$PYTHON -m http.server > > > --bind > > > > > 10.0.0.3 80 >/dev/null 2>&1], [http1.pid]) > > > > > > +M_NS_DAEMONIZE([ovn-chassis-2], [sw0p2], [$PYTHON -m http.server > > > --bind > > > > > 10.0.0.4 80 >/dev/null 2>&1], [http2.pid]) > > > > > > + > > > > > > +# wait for http server be ready > > > > > > +sleep 2 > > > > > > + > > > > > > +# Flush conntrack entries for easier output parsing of next > test. > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v > > > 172.16.0.100:80 > > > > > --retry 3 --max-time 1 --local-port 59002 2> curl.out']) > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [sh -c 'cat > curl.out | > > > > > grep -i -e connect | grep -v 'Server:''], \ > > > > > > +[0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v > > > 172.16.0.100:80 > > > > > --retry 3 --max-time 1 --local-port 59003 2> curl.out']) > > > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [sh -c 'cat > curl.out | > > > > > grep -i -e connect | grep -v 'Server:''], \ > > > > > > +[0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v > > > 172.16.0.100:80 > > > > > --retry 3 --max-time 1 --local-port 59001']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl > dpctl/dump-conntrack | > > > > > M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v > > > 172.16.0.100:80 > > > > > --retry 3 --max-time 1 --local-port 59000']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl > dpctl/dump-conntrack | > > > > > M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# create a big file on web servers for download > > > > > > +M_NS_EXEC([ovn-chassis-1], [sw0p1], [dd bs=512 count=200000 > > > > > if=/dev/urandom of=download_file]) > > > > > > +M_NS_EXEC([ovn-chassis-2], [sw0p2], [dd bs=512 count=200000 > > > > > if=/dev/urandom of=download_file]) > > > > > > + > > > > > > +# Flush conntrack entries for easier output parsing of next > test. > > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > > > 59004 > > > > > 2>curl.out']) > > > > > > + > > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > + > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 > cat > > > > > curl.out | \ > > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +# Check if we have only one backend for the same connection - > orig + > > > > > dest ports > > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Check if gw-2 is empty to ensure that the traffic only come > > > from/to > > > > > the originator chassis via DGP public1 > > > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > + > > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep > "dport=80" > > > -c) > > > > > > + > > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > > +# Backend resides on ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59004,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +else > > > > > > +# Backend resides on ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59004,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +fi > > > > > > + > > > > > > +# Flush conntrack entries for easier output parsing of next > test. > > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +# Check the flows again for a new source port > > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > > > 59005 > > > > > 2>curl.out']) > > > > > > + > > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > + > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 > cat > > > > > curl.out | \ > > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +# Check if we have only one backend for the same connection - > orig + > > > > > dest ports > > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Check if gw-2 is empty to ensure that the traffic only come > > > from/to > > > > > the originator chassis via DGP public1 > > > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > + > > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep > "dport=80" > > > -c) > > > > > > + > > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > > +# Backend resides on ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59005,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +else > > > > > > +# Backend resides on ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59005,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +fi > > > > > > + > > > > > > +# Flush conntrack entries for easier output parsing of next > test. > > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +# Start a new test using the second DGP as origin (public2) > > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > > > 59006 > > > > > 2>curl.out']) > > > > > > + > > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > + > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 > cat > > > > > curl.out | \ > > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +# Check if we have only one backend for the same connection - > orig + > > > > > dest ports > > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Check if gw-1 is empty to ensure that the traffic only come > > > from/to > > > > > the originator chassis via DGP public2 > > > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > + > > > > > > +# Check the backend IP from ct entries on gw-2 (DGP public2) > > > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep > "dport=80" > > > -c) > > > > > > + > > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > > +# Backend resides on ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59006,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +else > > > > > > +# Backend resides on ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59006,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +fi > > > > > > + > > > > > > +# Flush conntrack entries for easier output parsing of next > test. > > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +# Check the flows again for a new source port using the second > DGP > > > as > > > > > origin (public2) > > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port > > > 59007 > > > > > 2>curl.out']) > > > > > > + > > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl > dpctl/dump-conntrack | > > > sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed > > > > > ':a;N;$!ba;s/\n/\\n/g') > > > > > > + > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 > cat > > > > > curl.out | \ > > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +# Check if we have only one backend for the same connection - > orig + > > > > > dest ports > > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Check if gw-1 is empty to ensure that the traffic only come > > > from/to > > > > > the originator chassis via DGP public2 > > > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > + > > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) > > > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep > "dport=80" > > > -c) > > > > > > + > > > > > > +if [[ $backend_check -gt 0 ]]; then > > > > > > +# Backend resides on ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59007,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-1 > > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +else > > > > > > +# Backend resides on ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp], [0], [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59007,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +# Ensure that the traffic only come from ovn-chassis-2 > > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep > "dport=80" > > > -c], > > > > > [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep > "dport=80" > > > > > -c], [1], [dnl > > > > > > +0 > > > > > > +]) > > > > > > +fi > > > > > > + > > > > > > +# Check multiple requests coming from DGP's public1 and public2 > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" > | > > > grep > > > > > -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +200 OK > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" > | > > > grep > > > > > -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +200 OK > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ > > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" > | > > > grep > > > > > -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +200 OK > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" > | > > > grep > > > > > -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 > > > > > > +200 OK > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +# Remove the LB and change the VIP port - different from the > backend > > > > > ports > > > > > > +check multinode_nbctl lb-del lb0 > > > > > > + > > > > > > +# create LB again > > > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:9000" " > 10.0.0.3:80, > > > > > 10.0.0.4:80" > > > > > > +check multinode_nbctl lr-lb-add lr0 lb0 > > > > > > +check multinode_nbctl ls-lb-add sw0 lb0 > > > > > > + > > > > > > +# Set use_stateless_nat to true > > > > > > +check multinode_nbctl set load_balancer lb0 > > > > > options:use_stateless_nat=true > > > > > > + > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +# Check end-to-end request using a new port for VIP > > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O > > > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 > --local-port > > > 59008 > > > > > 2>curl.out']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl > dpctl/dump-conntrack | > > > > > M_FORMAT_CT(20.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" > | > > > grep > > > > > -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > > > > +200 OK > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack > > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack > > > > > > + > > > > > > +# Check end-to-end request using a new port for VIP > > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O > > > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 > --local-port > > > 59008 > > > > > 2>curl.out']) > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl > dpctl/dump-conntrack | > > > > > M_FORMAT_CT(30.0.0.3) | \ > > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], > > > [0], > > > > > [dnl > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) > > > > > > > > > > > > > > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) > > > > > > +]) > > > > > > + > > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ > > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" > | > > > grep > > > > > -v 'Server:'], [0], [dnl > > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 > > > > > > +200 OK > > > > > > +* Closing connection > > > > > > +]) > > > > > > + > > > > > > +AT_CLEANUP > > > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > > > > > > index dcc3dbbc3..9e7a2f225 100644 > > > > > > --- a/tests/ovn-northd.at > > > > > > +++ b/tests/ovn-northd.at > > > > > > @@ -13864,3 +13864,323 @@ check_no_redirect > > > > > > > > > > > > AT_CLEANUP > > > > > > ]) > > > > > > + > > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + > DGP + > > > NAT > > > > > Stateless)]) > > > > > > +ovn_start > > > > > > + > > > > > > +check ovn-nbctl ls-add public > > > > > > +check ovn-nbctl lr-add lr1 > > > > > > + > > > > > > +# lr1 DGP ts1 > > > > > > +check ovn-nbctl ls-add ts1 > > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > > > 172.16.10.1/24 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > > > > + > > > > > > +# lr1 DGP ts2 > > > > > > +check ovn-nbctl ls-add ts2 > > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > > > 172.16.20.1/24 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > > > > + > > > > > > +# lr1 DGP public > > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > > > 173.16.0.1/16 > > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > 172.16.0.1/24 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > > > + > > > > > > +check ovn-nbctl ls-add s1 > > > > > > +# s1 - lr1 > > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > > > 172.16.0.1" > > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > > > + > > > > > > +# s1 - backend vm1 > > > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > > > 172.16.0.101" > > > > > > + > > > > > > +# s1 - backend vm2 > > > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > > > 172.16.0.102" > > > > > > + > > > > > > +# s1 - backend vm3 > > > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > > > 172.16.0.103" > > > > > > + > > > > > > +# Add the lr1 DGP ts1 to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 > router-port=lr1-ts1 > > > > > nat-addresses=router > > > > > > + > > > > > > +# Add the lr1 DGP ts2 to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 > router-port=lr1-ts2 > > > > > nat-addresses=router > > > > > > + > > > > > > +# Add the lr1 DGP public to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1 > router-port=lr1_public > > > > > nat-addresses=router > > > > > > + > > > > > > +# Create the Load Balancer lb1 > > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > > > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > > > > + > > > > > > +# Set use_stateless_nat to true > > > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > > > > options:use_stateless_nat=true > > > > > > + > > > > > > +# Associate load balancer to s1 > > > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > > > +check ovn-nbctl --wait=sb sync > > > > > > + > > > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > > > +AT_CAPTURE_FILE([s1flows]) > > > > > > + > > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | > > > grep > > > > > "30.0.0.1"], [0], [dnl > > > > > > + table=??(ls_in_pre_stateful ), priority=120 , > match=(reg0[[2]] > > > == 1 > > > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > > > > +]) > > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > > > "30.0.0.1"], [0], [dnl > > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new > && > > > > > ip4.dst == 30.0.0.1), > > > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > +]) > > > > > > + > > > > > > +# Associate load balancer to lr1 with DGP > > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > > > +check ovn-nbctl --wait=sb sync > > > > > > + > > > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > > > + > > > > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > > > "30.0.0.1"], [0], [dnl > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > > is_chassis_resident("cr-lr1-ts1")), > > > > > > > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > > is_chassis_resident("cr-lr1-ts2")), > > > > > > > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > > is_chassis_resident("cr-lr1_public")), > > > > > > > > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > +]) > > > > > > + > > > > > > +# 2. Check if the DGP ports are in the match with action next > > > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], > [0], > > > [dnl > > > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > > > > action=(next;) > > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || > (ip4.src == > > > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > > > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) > > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || > (ip4.src == > > > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > > > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) > > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && > > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || > (ip4.src == > > > > > 172.16.0.101)) && (inport == "lr1_public" || outport == > "lr1_public") > > > && > > > > > is_chassis_resident("cr-lr1_public") && tcp), action=(next;) > > > > > > +]) > > > > > > + > > > > > > +# 3. Check if the VIP IP is in the ipX.src action > > > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], > [dnl > > > > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > > > > action=(next;) > > > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > > > > action=(next;) > > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || > (ip4.src == > > > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && > > > > > is_chassis_resident("cr-lr1-ts1") && tcp), > action=(ip4.src=30.0.0.1; > > > next;) > > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || > (ip4.src == > > > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && > > > > > is_chassis_resident("cr-lr1-ts2") && tcp), > action=(ip4.src=30.0.0.1; > > > next;) > > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && > > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || > (ip4.src == > > > > > 172.16.0.101)) && (inport == "lr1_public" || outport == > "lr1_public") > > > && > > > > > is_chassis_resident("cr-lr1_public") && tcp), > action=(ip4.src=30.0.0.1; > > > > > next;) > > > > > > +]) > > > > > > + > > > > > > +AT_CLEANUP > > > > > > +]) > > > > > > + > > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + > DGP + > > > NAT > > > > > Stateless) - IPv6]) > > > > > > +ovn_start > > > > > > + > > > > > > +check ovn-nbctl ls-add public > > > > > > +check ovn-nbctl lr-add lr1 > > > > > > + > > > > > > +# lr1 DGP ts1 > > > > > > +check ovn-nbctl ls-add ts1 > > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > > > > > 2001:db8:aaaa:1::1/64 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 > > > > > > + > > > > > > +# lr1 DGP ts2 > > > > > > +check ovn-nbctl ls-add ts2 > > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > > > > > 2001:db8:aaaa:2::1/64 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 > > > > > > + > > > > > > +# lr1 DGP public > > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > > > > > 2001:db8:bbbb::1/64 > > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > > > > > 2001:db8:aaaa:3::1/64 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > > > + > > > > > > +check ovn-nbctl ls-add s1 > > > > > > +# s1 - lr1 > > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > > > > > 2001:db8:aaaa:3::1" > > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > > > + > > > > > > +# s1 - backend vm1 > > > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > > > > > 2001:db8:aaaa:3::101" > > > > > > + > > > > > > +# s1 - backend vm2 > > > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > > > > > 2001:db8:aaaa:3::102" > > > > > > + > > > > > > +# s1 - backend vm3 > > > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > > > > > 2001:db8:aaaa:3::103" > > > > > > + > > > > > > +# Add the lr1 DGP ts1 to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 > router-port=lr1-ts1 > > > > > nat-addresses=router > > > > > > + > > > > > > +# Add the lr1 DGP ts2 to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 > router-port=lr1-ts2 > > > > > nat-addresses=router > > > > > > + > > > > > > +# Add the lr1 DGP public to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1 > router-port=lr1_public > > > > > nat-addresses=router > > > > > > + > > > > > > +# Create the Load Balancer lb1 > > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "2001:db8:cccc::1" > > > > > "2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101" > > > > > > + > > > > > > +# Set use_stateless_nat to true > > > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 > > > > > options:use_stateless_nat=true > > > > > > + > > > > > > +# Associate load balancer to s1 > > > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > > > +check ovn-nbctl --wait=sb sync > > > > > > + > > > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > > > +AT_CAPTURE_FILE([s1flows]) > > > > > > + > > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | > > > grep > > > > > "2001:db8:cccc::1"], [0], [dnl > > > > > > + table=??(ls_in_pre_stateful ), priority=120 , > match=(reg0[[2]] > > > == 1 > > > > > && ip6.dst == 2001:db8:cccc::1), action=(xxreg1 = 2001:db8:cccc::1; > > > > > ct_lb_mark;) > > > > > > +]) > > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > > > "2001:db8:cccc::1"], [0], [dnl > > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new > && > > > > > ip6.dst == 2001:db8:cccc::1), > > > > > > > > > action=(ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > > +]) > > > > > > + > > > > > > +# Associate load balancer to lr1 with DGP > > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > > > +check ovn-nbctl --wait=sb sync > > > > > > + > > > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > > > + > > > > > > +# Check stateless NAT rules for load balancer with multiple DGP > > > > > > +# 1. Check if the backend IPs are in the ipX.dst action > > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > > > "2001:db8:cccc::1"], [0], [dnl > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > > > is_chassis_resident("cr-lr1-ts1")), > > > > > > > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > > > is_chassis_resident("cr-lr1-ts2")), > > > > > > > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && > > > > > is_chassis_resident("cr-lr1_public")), > > > > > > > > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) > > > > > > +]) > > > > > > + > > > > > > +# 2. Check if the DGP ports are in the match with action next > > > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], > [0], > > > [dnl > > > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), > > > > > action=(next;) > > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > > 2001:db8:aaaa:3::102) || > > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || > outport > > > == > > > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), > action=(next;) > > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > > 2001:db8:aaaa:3::102) || > > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || > outport > > > == > > > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), > action=(next;) > > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && > > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > > 2001:db8:aaaa:3::102) || > > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || > > > outport == > > > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > > > > action=(next;) > > > > > > +]) > > > > > > + > > > > > > +# 3. Check if the VIP IP is in the ipX.src action > > > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], > [dnl > > > > > > + table=??(lr_out_snat ), priority=0 , match=(1), > > > > > action=(next;) > > > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), > > > > > action=(next;) > > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > > 2001:db8:aaaa:3::102) || > > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || > outport > > > == > > > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), > > > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > > 2001:db8:aaaa:3::102) || > > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || > outport > > > == > > > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), > > > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && > > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == > > > 2001:db8:aaaa:3::102) || > > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || > > > outport == > > > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), > > > > > action=(ip6.src=2001:db8:cccc::1; next;) > > > > > > +]) > > > > > > + > > > > > > +AT_CLEANUP > > > > > > +]) > > > > > > + > > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ > > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (DGP)]) > > > > > > +ovn_start > > > > > > + > > > > > > +check ovn-nbctl ls-add public > > > > > > +check ovn-nbctl lr-add lr1 > > > > > > + > > > > > > +# lr1 DGP ts1 > > > > > > +check ovn-nbctl ls-add ts1 > > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 > > > 172.16.10.1/24 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-1 > > > > > > + > > > > > > +# lr1 DGP ts2 > > > > > > +check ovn-nbctl ls-add ts2 > > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 > > > 172.16.20.1/24 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-1 > > > > > > + > > > > > > +# lr1 DGP public > > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 > > > 173.16.0.1/16 > > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 > 172.16.0.1/24 > > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 > > > > > > + > > > > > > +check ovn-nbctl ls-add s1 > > > > > > +# s1 - lr1 > > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 > > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router > > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 > > > 172.16.0.1" > > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 > > > > > > + > > > > > > +# s1 - backend vm1 > > > > > > +check ovn-nbctl lsp-add s1 vm1 > > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 > > > 172.16.0.101" > > > > > > + > > > > > > +# s1 - backend vm2 > > > > > > +check ovn-nbctl lsp-add s1 vm2 > > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 > > > 172.16.0.102" > > > > > > + > > > > > > +# s1 - backend vm3 > > > > > > +check ovn-nbctl lsp-add s1 vm3 > > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 > > > 172.16.0.103" > > > > > > + > > > > > > +# Add the lr1 DGP ts1 to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 > > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 > router-port=lr1-ts1 > > > > > nat-addresses=router > > > > > > + > > > > > > +# Add the lr1 DGP ts2 to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 > > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 > router-port=lr1-ts2 > > > > > nat-addresses=router > > > > > > + > > > > > > +# Add the lr1 DGP public to the public switch > > > > > > +check ovn-nbctl lsp-add public public_lr1 > > > > > > +check ovn-nbctl lsp-set-type public_lr1 router > > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router > > > > > > +check ovn-nbctl lsp-set-options public_lr1 > router-port=lr1_public > > > > > nat-addresses=router > > > > > > + > > > > > > +# Create the Load Balancer lb1 > > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" > > > > > "172.16.0.103,172.16.0.102,172.16.0.101" > > > > > > + > > > > > > +# Associate load balancer to s1 > > > > > > +check ovn-nbctl ls-lb-add s1 lb1 > > > > > > +check ovn-nbctl --wait=sb sync > > > > > > + > > > > > > +ovn-sbctl dump-flows s1 > s1flows > > > > > > +AT_CAPTURE_FILE([s1flows]) > > > > > > + > > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | > > > grep > > > > > "30.0.0.1"], [0], [dnl > > > > > > + table=??(ls_in_pre_stateful ), priority=120 , > match=(reg0[[2]] > > > == 1 > > > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) > > > > > > +]) > > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep > > > > > "30.0.0.1"], [0], [dnl > > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new > && > > > > > ip4.dst == 30.0.0.1), > > > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > +]) > > > > > > + > > > > > > +# Associate load balancer to lr1 with DGP > > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 > > > > > > +check ovn-nbctl --wait=sb sync > > > > > > + > > > > > > +ovn-sbctl dump-flows lr1 > lr1flows > > > > > > +AT_CAPTURE_FILE([lr1flows]) > > > > > > + > > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep > > > > > "30.0.0.1"], [0], [dnl > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > > is_chassis_resident("cr-lr1-ts1")), > > > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > > is_chassis_resident("cr-lr1-ts2")), > > > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new > && > > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && > > > > > is_chassis_resident("cr-lr1_public")), > > > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) > > > > > > +]) > > > > > > + > > > > > > +AT_CLEANUP > > > > > > +]) > > > > > > -- > > > > > > 2.34.1 > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _'Esta mensagem é direcionada apenas para os endereços > constantes no > > > > > > cabeçalho inicial. Se você não está listado nos endereços > constantes > > > no > > > > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo > > > dessa > > > > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações > citadas > > > > > estão > > > > > > imediatamente anuladas e proibidas'._ > > > > > > > > > > > > > > > > > > * **'Apesar do Magazine Luiza tomar > > > > > > todas as precauções razoáveis para assegurar que nenhum vírus > esteja > > > > > > presente nesse e-mail, a empresa não poderá aceitar a > > > responsabilidade > > > > > por > > > > > > quaisquer perdas ou danos causados por esse e-mail ou por seus > > > anexos'.* > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > dev mailing list > > > > > > dev@openvswitch.org > > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > _‘Esta mensagem é direcionada apenas para os endereços constantes no > > > > cabeçalho inicial. Se você não está listado nos endereços constantes > no > > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo > dessa > > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas > > > estão > > > > imediatamente anuladas e proibidas’._ > > > > > > > > > > > > * **‘Apesar do Magazine Luiza tomar > > > > todas as precauções razoáveis para assegurar que nenhum vírus esteja > > > > presente nesse e-mail, a empresa não poderá aceitar a > responsabilidade > > > por > > > > quaisquer perdas ou danos causados por esse e-mail ou por seus > anexos’.* > > > > > > > > > > > > > > > > _______________________________________________ > > > > dev mailing list > > > > dev@openvswitch.org > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > > -- > > > > > > > > > > _‘Esta mensagem é direcionada apenas para os endereços constantes no > > cabeçalho inicial. Se você não está listado nos endereços constantes no > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas > estão > > imediatamente anuladas e proibidas’._ > > > > > > * **‘Apesar do Magazine Luiza tomar > > todas as precauções razoáveis para assegurar que nenhum vírus esteja > > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade > por > > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* > > > > > > > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > >
On Mon, Oct 7, 2024 at 11:01 AM Roberto Bartzen Acosta <roberto.acosta@luizalabs.com> wrote: > > Thanks Numan! > > Em seg., 7 de out. de 2024 às 11:34, Numan Siddique <nusiddiq@redhat.com> escreveu: >> >> On Mon, Sep 30, 2024 at 1:57 PM Roberto Bartzen Acosta via dev >> <ovs-dev@openvswitch.org> wrote: >> > >> > Hi Numan, >> > >> > Em sex., 27 de set. de 2024 às 12:47, Numan Siddique <numans@ovn.org> >> > escreveu: >> > >> > > On Thu, Sep 26, 2024 at 10:55 AM Roberto Bartzen Acosta via dev >> > > <ovs-dev@openvswitch.org> wrote: >> > > > >> > > > Hi Numan, >> > > > >> > > > Thanks for your feedback and review. >> > > > >> > > > Em qua., 25 de set. de 2024 às 19:50, Numan Siddique <numans@ovn.org> >> > > > escreveu: >> > > > >> > > > > On Thu, Sep 19, 2024 at 6:12 PM Roberto Bartzen Acosta via dev >> > > > > <ovs-dev@openvswitch.org> wrote: >> > > > > > >> > > > > > This commit fixes the build_distr_lrouter_nat_flows_for_lb function >> > > to >> > > > > > include a DNAT flow entry for each DGP in use. Since we have added >> > > > > support >> > > > > > to create multiple gateway ports per logical router, it's necessary >> > > to >> > > > > > include in the LR NAT rules pipeline a specific entry for each >> > > attached >> > > > > DGP. >> > > > > > Otherwise, the inbound traffic will only be redirected when the >> > > incoming >> > > > > LRP >> > > > > > matches the chassis_resident field. >> > > > > > >> > > > > > Additionally, this patch includes the ability to use load-balancer >> > > with >> > > > > DGPs >> > > > > > attached to multiple chassis. We can have each of the DGPs associated >> > > > > with a >> > > > > > different chassis, and in this case the DNAT rules added by default >> > > will >> > > > > not >> > > > > > be enough to guarantee outgoing traffic. >> > > > > > >> > > > > > To solve the multiple chassis for DGPs problem, this patch include a >> > > new >> > > > > > config options to be configured in the load-balancer. If the >> > > > > use_stateless_nat >> > > > > > is set to true, the logical router that references this load-balancer >> > > > > will use >> > > > > > Stateless NAT rules when the logical router has multiple DGPs. After >> > > > > applying >> > > > > > this patch and setting the use_stateless_nat option, the inbound >> > > and/or >> > > > > > outbound traffic can pass through any chassis where the DGP resides >> > > > > without >> > > > > > having problems with CT state. >> > > > > > >> > > > > > Reported-at: >> > > https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2054322 >> > > > > > Fixes: 15348b7b806f ("ovn-northd: Multiple distributed gateway port >> > > > > support.") >> > > > > > >> > > > > > Signed-off-by: Roberto Bartzen Acosta <roberto.acosta@luizalabs.com> >> > > > > >> > > > > Hi Roberto, >> > > > > >> > > > > Thanks for the patch. I tested this patch using the test example in >> > > > > multinode.at. >> > > > > >> > > > > The test case adds the below load balancer >> > > > > >> > > > > [root@ovn-central ~]# ovn-nbctl lb-list >> > > > > UUID LB PROTO >> > > > > VIP IPs >> > > > > f3e29869-3bb5-4df0-960a-171106f5913a lb0 tcp >> > > > > 172.16.0.100:9000 10.0.0.3:80,10.0.0.4:80 >> > > > > >> > > > > And the below logical flows are generated by this patch >> > > > > >> > > > > -------- >> > > > > [root@ovn-central ~]# ovn-sbctl dump-flows lr0 | grep 172.16.0.100 >> > > > > table=6 (lr_in_defrag ), priority=100 , match=(ip && ip4.dst >> > > > > == 172.16.0.100), action=(ct_dnat;) >> > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && >> > > > > is_chassis_resident("cr-lr0-public-p1")), >> > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= >> > > 10.0.0.3:80, >> > > > > 10.0.0.4:80);) >> > > > > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && >> > > > > is_chassis_resident("cr-lr0-public-p2")), >> > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends= >> > > 10.0.0.3:80, >> > > > > 10.0.0.4:80);) >> > > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && >> > > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && >> > > > > tcp.src == 80)) && (inport == "lr0-public-p1" || outport == >> > > > > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), >> > > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) >> > > > > table=3 (lr_out_snat ), priority=160 , match=(ip4 && >> > > > > ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && >> > > > > tcp.src == 80)) && (inport == "lr0-public-p2" || outport == >> > > > > "lr0-public-p2") && is_chassis_resident("cr-lr0-public-p2") && tcp), >> > > > > action=(ip4.src=172.16.0.100; tcp.src=9000; next;) >> > > > > -------------- >> > > > > >> > > > > >> > > > > I fail to understand the reason for modifying the ip4.dst before >> > > > > calling ct_lb_mark. Can you please explain why ? Because the >> > > > > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;) will first modify the >> > > > > ip4.dst to 10.0.0.3 and >> > > > > then to 10.0.0.4 and then the ct_lb_mark will actually do the >> > > > > conntrack with NAT to either 10.0.0.3 or 10.0.0.4. >> > > > > >> > > > > Is it because you want the conntrack entry to not have 172.16.0.100 ? >> > > > > >> > > > >> > > >> > > > The only reason I included this ip4.dst action in the DNAT rule is >> > > because >> > > > it's required to accept packets coming from a chassis that doesn't have >> > > > previously created conntrack entries. The main feature introduced in this >> > > > patch is to allow the administrator to have multiple DGPs attached to >> > > > different chassis (is_chassis_resident...). So, my implementation was >> > > based >> > > > on the normal behavior when using stateless NAT for external addresses, >> > > > where we need to add the ipx.dst in lr_in_dnat for traffic to be received >> > > > on the chassis (put the DGP port as chassis_resident match, as is the >> > > case >> > > > with stateless NAT [1] with DGP[2]). >> > > > >> > > > The question is, if we only have the ct_lb_mark, packets that pass >> > > through >> > > > the chassis and are already part of an active flow in another chassis >> > > (same >> > > > IPs and Ports) will be dropped because there is no correspondence in the >> > > > backend. So only packets with the NEW flag will be accepted and sent to >> > > the >> > > > backend (at least for TCP traffic). If we only have the ip4.dst action, >> > > > this will always perform the dnat for the same backend, without >> > > balancing. >> > > > Therefore, the combination of the two actions allows the packet to always >> > > > be received (regardless of whether conntrack is active for it), and >> > > > ct_lb_mark will take care of balancing for different backends. >> > > > >> > > > If we had conntrack sync between different chassis this would not be >> > > > necessary, as the ct_lb_mark action could always be executed without >> > > > dropping packets due to lack of correspondence in the conntrack table. >> > > > >> > > > [1] >> > > > >> > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15737 >> > > > [2] >> > > > >> > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15726 >> > > > >> > > > >> > > >> > > I'm sorry, but it's not 100% clear to me. I know that you've already >> > > explained to Mark in the older version of this patch. >> > > Can you please explain with an example ? >> > > >> > > Let's take the below topology you've added in the mutlinode test as example >> > > >> > > # Network topology >> > > # >> > > # publicp1 (ovn-chassis-3) (20.0.0.3/24) >> > > # | >> > > # overlay >> > > # | >> > > # DGP public1 (ovn-gw-1) (20.0.0.1/24) >> > > # | >> > > # | >> > > # | >> > > # lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) 10.0.0.3/24 >> > > # | | >> > > # | + --- sw0p2 (ovn-chassis-2) 10.0.0.4/24 >> > > # | >> > > # DGP public2 (ovn-gw-2) (30.0.0.1/24) >> > > # | >> > > # overlay >> > > # | >> > > # publicp2 (ovn-chassis-4) (30.0.0.3/24) >> > > >> > > >> > > load balancer is configured on lr0 -> ovn-nbctl lb-add lb0 >> > > "172.16.0.100:80" "10.0.0.3:80,10.0.0.4:80" >> > > and it is attached to both lr0 and sw0. >> > > >> > > Scenario 1: >> > > >> > > publicp1 with IP 20.0.0.3 sends TCP traffic to VIP 172.16.0.100. I >> > > think this is what will happen >> > > >> > > - The packet (ip4.src = 20.0.0.3 , ip4.dst = 172.16.0.100, tcp.dst >> > > = 80) from ovn-chassis-3 will be sent out via the localnet bridge >> > > (br-ex) >> > > and the packet is received on ovn-gw-1 (as DGP public1 is resident >> > > on it) via the localnet bridge and the >> > > packet first enters public logical switch pipeline and then the >> > > lr0 router pipeline. >> > > >> > > - In the router's lr_in_dnat state, the packet will be load >> > > balanced to one of the backends using ct_lb_mark. Lets say 10.0.0.3 >> > > is chosen >> > > - The packet from router pipeline lr0 enters sw0 switch pipeline and >> > > then the packet is tunneled to ovn-chassis-1 and delivered to sw0p1. >> > > - The reply packet (ip4.src = 10.0.0.3, ip4.dst = 20.0.0.3, tcp.src = >> > > 80) will enter the router pipeline and since 20.0.0.0/24 is handled by >> > > DGP public1, >> > > the packet is tunnelled to ovn-gw-1. >> > > - In the router pipeline of ovn-gw-1, the packet is undnatted from >> > > ip4.src 10.0.0.3 to 172.16.0.100 and the packet is sent out via the >> > > localnet bridge. >> > > - ovn-chassis-3 receives the packet via the localnet bridge and into >> > > br-int and finally to publicp1. >> > > >> > > >> > > In this scenario the load balancing is handled and conntrack entries >> > > are created in ovn-gw-1. And there is no need to add flows in >> > > "lr_out_snat" for stateless NAT >> > > or set ip4.dst to one or all of backend IPs before "ct_lb_mark" in >> > > lr_in_dnat stage. >> > > >> > > Scenario 2: >> > > publicp2 with IP 30.0.0.3 sends TCP traffic to VIP 172.16.0.100. >> > > >> > > This is similar to scenario 1. Except that load balancing happens in >> > > ovn-gw-2. >> > > since DGP public2 is on this chassis. >> > > >> > > >> > > Scenario 3: >> > > >> > > An external entity with IP 20.0.0.50 sends TCP traffic to VIP >> > > 172.16.0.100. >> > > >> > > - This scenario is similar to the first one. The packet from this >> > > external entity is received on ovn-gw-1 via the localnet bridge. >> > > Rest all is the same. >> > > >> > > Scenario 4: >> > > >> > > sw0p2 on ovn-chassis-2 sends TCP traffic to VIP 172.16.0.100. >> > > >> > > - Since sw0 is attached with load balancer lb0, load balancing >> > > happens in the source chassis - ovn-chassis-2 itself and depending on >> > > the backend chosen, >> > > the packet is tunnelled to ovn-chassis-1 (if 10.0.0.3 is chosen) >> > > or delivered directly to sw0p2 (if i0.0.0.4 is chosen). >> > > >> > > >> > > Scenario 5: >> > > >> > > An external entity with IP 40.0.0.40 sends TCP traffic to VIP >> > > 172.16.0.100 and there are 2 ECMP routes configured >> > > 172.16.0.100 via 20.0.0.1 >> > > 172.16.0.100 via 30.0.0.1 >> > > >> > > In this case if the packet uses the route via 20.0.0.1 it will be >> > > received on ovn-gw-1 via the localnet bridge br-ex. >> > > >> > > - With your patch, ip4.dst is modified to the last backend in the >> > > list and then ct_lb_mark chosen as one of the backends >> > > - And then it is tunnelled to the destination chassis. >> > > >> > > >> > > Is this the scenario 5 you're trying to address with the stateless >> > > NAT ? How would the reply packet work ? >> > > The reply packet from backend 10.0.0.3 can use either of the paths ? >> > > i.e ovn-gw-1 or ovn-gw-2 ? >> > > >> > >> > >> > Yes, this scenario would be the main goal of this patch. >> > Let me explain with an example related to the OVN interconnect (one of the >> > use cases of this patch). In the context of ovn-ic the backend 10.0.0.3 >> > could reply packets using either of the paths (ovn-gw-1 or ovn-gw-2). >> > >> > >> > >> > LB VIP 172.16.0.100 >> > >> > | >> > vm (40.0.0.40) - LR-EXT - lrp1 - chassis1 - Transit switch TS1 - ovn-gw-1 >> > - lr0-public-p1 - lr0 - sw0 - VM 10.0.0.3 (ovn-chassis-1) >> > - lrp2 - chassis2 - Transit switch >> > TS2 - ovn-gw-2 - lr0-public-p2 - - VM 10.0.0.4 >> > (ovn-chassis-2) >> > >> > >> > Let's take an example of the ovn-ic in the above topology: >> > >> > Assumptions: >> > A) logical router LR-EXT: route to 172.16.0.100/32 via ECMP (lrp1/chassis1 >> > and lrp2/chassis2) >> > B) logical router lr0: route to 40.0.0.0/24 via ECMP >> > (lr0-public-p1/ovn-gw-1 and lr0-public-p2/ovn-gw-2) >> > >> > >> > 1 - VM 40.0.0.40 send a request to the LB 172.16.0.100:80 >> > 2 - LR-EXT chooses one of the possible outgoing routes via ECMP - e.g. lrp1 >> > / chassis1 / TS1 >> > 3 - The lr0 receives the ingoing traffic through the port lr0-public-p1 / >> > ovn-gw-1 because the TS1 is used to start the traffic >> > 4 - with this patch we added lr_in_dnat rules for all DGPs (with or without >> > the stateless NAT config flag) >> > >> > with the stateless NAT config flag True: >> > ...is_chassis_resident("cr-lr0-public-p1")), >> > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80 >> > ,10.0.0.4:80);) >> > ...is_chassis_resident("cr-lr0-public-p2")), >> > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80 >> > ,10.0.0.4:80);) >> > 5 - The ip4.dst will perform the action cumulatively, basically the last >> > ip4.dst action applied will be to change dst to 10.0.0.4. >> > 5.1 - The ct_lb_mark will check the conntrack and validate if the traffic >> > refers to a new one (TCP SYNC), and then, it executes the final action >> > provided by ct_lb_mark and select one of the available backends. e.g >> > 10.0.0.3 >> > >> > without the stateless NAT config flag: >> > ...is_chassis_resident("cr-lr0-public-p1")), action=(ct_lb_mark(backends= >> > 10.0.0.3:80,10.0.0.4:80);) >> > ...is_chassis_resident("cr-lr0-public-p2")), action=(ct_lb_mark(backends= >> > 10.0.0.3:80,10.0.0.4:80);) >> > >> > 5 - same as the step 5.1 (ct_lb_mark check only). >> > >> > 6 - VM 10.0.0.3 receives the new traffic and sends a reply (TCP SYN+ACK) >> > >> > 7 - lr0 receives the response packet (from VM 10.0.0.3) in the routing >> > pipeline and route via one of the available paths (ECMP). >> > For example: lr0-public-p2 / ovn-gw-2 / TS2 >> > So, we have a different outgoing path of the original one. >> > >> > 8 - ovn-gw-2 receives (SYN+ACK). We have no conntrack entries on ovn-gw-2 >> > to match and perform the LB SNAT action! The ct_lb_mark previously creates >> > the conntrack entry on chassis ovn-gw-1 (when receives TCP SYN). >> > >> > with the stateless NAT config flag True: >> > The outgoing packet fail to match the lb conntrack but the packet is >> > SNATed by the lr_out_snat rule with a priority lower then cl_lb_mark, >> > executing after cl_lb_mark in the pipeline. >> > >> > 9 - The SYN+ACK packet crosses the TS2 and is delivered to VM 40.0.0.40 via >> > lrp2 (SRC = 172.16.0.100:80) >> > >> > lr_out_snat will match and SNATed: >> > match=(ip4 && ((ip4.src == 10.0.0.3 && tcp.src == 80) || (ip4.src == >> > 10.0.0.4 && tcp.src == 80)) && (inport == "lr0-public-p1" || outport == >> > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1") && tcp), >> > action=(ip4.src=172.16.0.100; tcp.src=80; next;) >> > >> > without the stateless NAT config flag: >> > The outgoing will be returned to the VM 40.0.0.4 with the LR SNAT for >> > public-p2 port as ip4.src! So, basically this will break the TCP >> > handshake!!! >> > >> > 9 - The SYN+ACK packet cross the TS2 and is delivered to the VM 40.0.0.40 >> > via lrp2 (SRC = 30.0.0.1:80) >> > >> > Moving forward on the happy path (stateless NAT is true): >> > >> > 10 - VM 40.0.0.40 sends an ACK to complete the TCP handshake. >> > >> > 11 - 2 - LR-EXT chooses one of the possible outgoing routes via ECMP >> > Let's assume that LR-EXT correctly execute the ecmp algorithm (per flow >> > basis) and forwards to the same initial path because we're using the same >> > flow (src/dst IPs and src/dts TCP ports still the same). >> > e.g. still using the lrp1 / chassis1 / TS1 >> > >> > 12 - The lr0 receive the ingoing traffic through the port lr0-public-p1 / >> > ovn-gw-1 / TS1 >> > >> > With this patch we have 2 possible behaviours: >> > >> > 12.1 - Without Stateless NAT config flag >> > >> > table=1 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src == >> > 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && tcp.src == 80)) && >> > (inport == "lr0-public-p1" || outport == >> > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1")), >> > action=(ct_dnat_in_czone;) >> > >> > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && !ct.rel >> > && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && >> > is_chassis_resident("cr-lr0-public-p1")), action=(ct_lb_mark(backends= >> > 10.0.0.3:80,10.0.0.4:80);) >> > >> > openflow: br-int (ovn-gw-1) >> > cookie=0x9381c20e, duration=2361.675s, table=15, n_packets=2, n_bytes=148, >> > idle_age=1564, >> > priority=120,ct_state=+new-rel+trk,tcp,metadata=0x4,nw_dst=172.16.0.100,tp_dst=80 >> > actions=group:3 >> > >> > ovs-appctl dpctl/dump-conntrack (ovn-gw-1) >> > tcp,orig=(src=40.0.0.40,dst=172.16.0.100,sport=35274,dport=80),reply=(src=10.0.0.3,dst=40.0.0.40,sport=80,dport=35274),zone=8,mark=2,protoinfo=(state=TIME_WAIT) >> > >> > At this point, the ACK packet will be discarded because we didn't perform >> > the SYN+ACK return on this chassis (ovn-gw -1). So, without the previously >> > established conntrack to match the ACK packet we broke the TCP handshake. >> > >> > Remember the flow: >> > -> SYN: ovn-gw-1 >> > <- SYN+ACK: ovn-gw-2 >> > -> ACK: ovn-gw-1 >> > >> > 12.2 - With stateless NAT config flag is True >> > >> > table=1 (lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src == >> > 10.0.0.3 && tcp.src == 80) || (ip4.src == 10.0.0.4 && tcp.src == 80)) && >> > (inport == "lr0-public-p1" || outport == >> > "lr0-public-p1") && is_chassis_resident("cr-lr0-public-p1")), action=(next;) >> > >> > table=8 (lr_in_dnat ), priority=120 , match=(ct.new && !ct.rel >> > && ip4 && ip4.dst == 172.16.0.100 && tcp && tcp.dst == 9000 && >> > is_chassis_resident("cr-lr0-public-p1")), >> > action=(ip4.dst=10.0.0.3;ip4.dst=10.0.0.4;ct_lb_mark(backends=10.0.0.3:80 >> > ,10.0.0.4:80);) >> > >> > openflow: br-int (ovn-gw-1) >> > cookie=0xab99f240, duration=1086.022s, table=15, n_packets=610, >> > n_bytes=48979, idle_age=825, >> > priority=120,ct_state=+new-rel+trk,tcp,metadata=0x2,nw_dst=172.16.0.100,tp_dst=80 >> > actions=mod_nw_dst:10.0.0.3,mod_nw_dst:10.0.0.4,group:1 >> > >> > ovs-appctl dpctl/dump-conntrack (ovn-gw-1) >> > tcp,orig=(src=40.0.0.40,dst=10.0.0.3,sport=54632,dport=80),reply=(src=10.0.0.3,dst=40.0.0.40,sport=80,dport=54632),zone=7,mark=2,protoinfo=(state=SYN_SENT) >> > >> > This is the reason to use stateless NAT rules. Unlike the case where we >> > don't have the SYN_SENT, now the ACK packet will be accepted and forwarded >> > after performing the mod_nw_dst action. >> > >> > How does this work? We already have the previous flow created by the first >> > packet (SYN), so regardless of the IP.dst modified in the action, the >> > packet will be forwarded to the same backend that corresponds to the >> > conntrack match that is in SYN_SENT state. That's why I created the action >> > with the two modifiers (ip4.dst + ct_lb_mark). Using only the ct_lb_mark >> > action creates a strong dependency on the connection state match in the >> > conntrack entry (which we don't have in these cases). However, only using >> > ip4.dst doesn't create traffic balancing as it will always send to the same >> > backend (last in the ip4.dst). >> > >> > >> > I hope this has helped clarify the design decisions when creating/modifying >> > flows (lr_in_dnat/lr_out_snat/lr_out_undnat). >> > >> > >> >> Thanks for the detailed explanation. Its clear to me now. >> >> I'd suggest the following >> >> 1. In lr_in_dnat stage, choose either the first backend or the last >> backend to modify the ip4.dst before ct_lb_mark. >> I've no strong preference. Either would do. >> >> 2. Please document or add comments in northd.c (and in ovn-nb.xml)so >> that we don't lose the context later. >> >> 3. Please run the ovn-fake-multinode tests in your github CI repo and >> make sure it passes. I think you can trigger the test in your repo >> in the Actions tab. >> >> 4. In the setup you described above, is it possible to add the >> (below) route in the vm (40.0.0.40) deployment so that 40.0.0.0.40 >> can >> talk directly to 10.0.0.3 and 10.0.0.4 instead of the LB VIP and >> send the tcp traffic to 10.0.0.3 and see if it works ? >> If not, then please document the same i.e with the option >> stateless_nat in the LB, the backends cannot be directly >> communicated. >> >> logical router LR-EXT: route to 10.0.0.0/24 via ECMP (lrp1/chassis1 >> and lrp2/chassis2) >> > > I'll do your suggestions. Just a quick question: how can I customize/parametrize the ovn-fake-multinode elements to run on CI? I mean to enable 2 more ovn-chassis needed to run the related tests. I've enabled it locally in the ./ovn_cluster.sh file but how do I change this for the fake-multinode CI actions? Is it per job or is this global? Its global. Take a look here - https://github.com/ovn-org/ovn/blob/main/.github/workflows/ovn-fake-multinode-tests.yml#L152 You can customize here to enbable 2 additional chassis. > > Regarding your suggestion 4, I imagine it makes more sense to integrate/create a new test with the ovn-interconnect setup + LB + multiple DGPs, which allows us to have multiple paths on routers on both sides (addressing the case discussed above) Sounds good to me. Numan > > Thanks, > Roberto > >> >> Thanks >> Numan >> >> >> > >> > >> > > >> > > Thanks >> > > Numan >> > > >> > > >> > > > > >> > > > > Also I don't understand why this patch adds the logical flows in >> > > > > "lr_out_snat" stage ? >> > > > > >> > > > >> > > > The flow for lr_out_snat is necessary for the correct functioning of >> > > > stateless NAT for the same reason explained previously. I mean, if the >> > > > outgoing packet is redirected to a chassis that doesn't have an active >> > > > conntrack entry, it will not be NATed by ct_lb action because it doesn't >> > > > refer to a valid flow (use case with ecmp). >> > > > >> > > > So it is necessary to create a stateless SNAT rule (similar to this [3]) >> > > > with a lower priority than the other router pipeline entries, in this >> > > case, >> > > > if the packet is not SNATed by ct_lb (conntrack missed) it will be SNATed >> > > > by stateless NAT rule. >> > > > >> > > > [3] >> > > > >> > > https://github.com/ovn-org/ovn/blob/b93e9a5e6f3aa3cb3e2065bd8e0aa0b6fc1fd19a/northd/northd.c#L15884 >> > > > >> > > > >> > > > >> > > > > >> > > > > Using the system multinode test as an example, the below fails >> > > > > (which is a regression) >> > > > > >> > > > > --- >> > > > > root@ovn-chassis-3 ~]# ip netns exec publicp1 nc -vz 10.0.0.3 80 >> > > > > ---- >> > > > > >> > > > > In the above test, publicp1 with IP 20.0.0.3 when it tries to connect >> > > > > to one if the backends directly (without the LB VIP), it fails. >> > > > > It fails because of the logical flows in "lr_out_snat". >> > > > > >> > > > > >> > > > > Looks to me the solution proposed here is incomplete. >> > > > > >> > > > > Also please note that in our CI we run the multinode tests >> > > > > periodically once a day using the v0.1 of the ovn-fake-multinode >> > > > > and the tests you added will fail. This needs to be fixed and until >> > > > > we move to the latest version of ovn-fake-multinode. >> > > > > >> > > > >> > > > I imagine that the test you are doing is using the same port as the LB >> > > > backend (TCP 80 in this case). So, the stateless lr_out_snat flow will >> > > > force the output to be SNATed because this port is in use by the backend. >> > > > Traffic to/from other ports will work without problems and will follow >> > > the >> > > > normal programmed flows (e.g. ICMP). >> > > > >> > > > This is necessary to ensure the egress traffic because the DGPs are >> > > > distributed across multiple chassis. Also, this setup is being validated >> > > in >> > > > the test ovn-fake-multinode testcase (ICMP from the backends chassis use >> > > > the router's default SNAT and not the LB's). I didn't understand the >> > > > regression you mentioned because this was programmed to be stateless and >> > > > it's traffic that uses the same ports as the LB backend, could you >> > > explain >> > > > better? >> > > > >> > > > Thanks, >> > > > Roberto >> > > > >> > > > >> > > > > Thanks >> > > > > Numan >> > > > > >> > > > > >> > > > > > --- >> > > > > > northd/en-lr-stateful.c | 12 - >> > > > > > northd/northd.c | 116 ++++++-- >> > > > > > ovn-nb.xml | 10 + >> > > > > > tests/multinode-macros.at | 40 +++ >> > > > > > tests/multinode.at | 556 >> > > ++++++++++++++++++++++++++++++++++++++ >> > > > > > tests/ovn-northd.at | 320 ++++++++++++++++++++++ >> > > > > > 6 files changed, 1017 insertions(+), 37 deletions(-) >> > > > > > >> > > > > > diff --git a/northd/en-lr-stateful.c b/northd/en-lr-stateful.c >> > > > > > index baf1bd2f8..f09691af6 100644 >> > > > > > --- a/northd/en-lr-stateful.c >> > > > > > +++ b/northd/en-lr-stateful.c >> > > > > > @@ -516,18 +516,6 @@ lr_stateful_record_create(struct >> > > lr_stateful_table >> > > > > *table, >> > > > > > >> > > > > > table->array[od->index] = lr_stateful_rec; >> > > > > > >> > > > > > - /* Load balancers are not supported (yet) if a logical router >> > > has >> > > > > multiple >> > > > > > - * distributed gateway port. Log a warning. */ >> > > > > > - if (lr_stateful_rec->has_lb_vip && >> > > lr_has_multiple_gw_ports(od)) { >> > > > > > - static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, >> > > 1); >> > > > > > - VLOG_WARN_RL(&rl, "Load-balancers are configured on logical >> > > " >> > > > > > - "router %s, which has %"PRIuSIZE" distributed " >> > > > > > - "gateway ports. Load-balancer is not supported >> > > " >> > > > > > - "yet when there is more than one distributed " >> > > > > > - "gateway port on the router.", >> > > > > > - od->nbr->name, od->n_l3dgw_ports); >> > > > > > - } >> > > > > > - >> > > > > > return lr_stateful_rec; >> > > > > > } >> > > > > > >> > > > > > diff --git a/northd/northd.c b/northd/northd.c >> > > > > > index a267cd5f8..bbe97acf8 100644 >> > > > > > --- a/northd/northd.c >> > > > > > +++ b/northd/northd.c >> > > > > > @@ -11807,31 +11807,30 @@ static void >> > > > > > build_distr_lrouter_nat_flows_for_lb(struct lrouter_nat_lb_flows_ctx >> > > > > *ctx, >> > > > > > enum lrouter_nat_lb_flow_type >> > > type, >> > > > > > struct ovn_datapath *od, >> > > > > > - struct lflow_ref *lflow_ref) >> > > > > > + struct lflow_ref *lflow_ref, >> > > > > > + struct ovn_port *dgp, >> > > > > > + bool stateless_nat) >> > > > > > { >> > > > > > - struct ovn_port *dgp = od->l3dgw_ports[0]; >> > > > > > - >> > > > > > - const char *undnat_action; >> > > > > > - >> > > > > > - switch (type) { >> > > > > > - case LROUTER_NAT_LB_FLOW_FORCE_SNAT: >> > > > > > - undnat_action = "flags.force_snat_for_lb = 1; next;"; >> > > > > > - break; >> > > > > > - case LROUTER_NAT_LB_FLOW_SKIP_SNAT: >> > > > > > - undnat_action = "flags.skip_snat_for_lb = 1; next;"; >> > > > > > - break; >> > > > > > - case LROUTER_NAT_LB_FLOW_NORMAL: >> > > > > > - case LROUTER_NAT_LB_FLOW_MAX: >> > > > > > - undnat_action = lrouter_use_common_zone(od) >> > > > > > - ? "ct_dnat_in_czone;" >> > > > > > - : "ct_dnat;"; >> > > > > > - break; >> > > > > > - } >> > > > > > + struct ds dnat_action = DS_EMPTY_INITIALIZER; >> > > > > > >> > > > > > /* Store the match lengths, so we can reuse the ds buffer. */ >> > > > > > size_t new_match_len = ctx->new_match->length; >> > > > > > size_t undnat_match_len = ctx->undnat_match->length; >> > > > > > >> > > > > > + /* dnat_action: Add the LB backend IPs as a destination action >> > > of >> > > > > the >> > > > > > + * lr_in_dnat NAT rule with cumulative effect >> > > because >> > > > > any >> > > > > > + * backend dst IP used in the action list will >> > > > > redirect the >> > > > > > + * packet to the ct_lb pipeline. >> > > > > > + */ >> > > > > > + if (stateless_nat) { >> > > > > > + for (size_t i = 0; i < ctx->lb_vip->n_backends; i++) { >> > > > > > + struct ovn_lb_backend *backend = >> > > &ctx->lb_vip->backends[i]; >> > > > > > + bool ipv6 = !IN6_IS_ADDR_V4MAPPED(&backend->ip); >> > > > > > + ds_put_format(&dnat_action, "%s.dst=%s;", ipv6 ? "ip6" : >> > > > > "ip4", >> > > > > > + backend->ip_str); >> > > > > > + } >> > > > > > + } >> > > > > > + ds_put_format(&dnat_action, "%s", ctx->new_action[type]); >> > > > > > >> > > > > > const char *meter = NULL; >> > > > > > >> > > > > > @@ -11841,20 +11840,46 @@ build_distr_lrouter_nat_flows_for_lb(struct >> > > > > lrouter_nat_lb_flows_ctx *ctx, >> > > > > > >> > > > > > if (ctx->lb_vip->n_backends || !ctx->lb_vip->empty_backend_rej) >> > > { >> > > > > > ds_put_format(ctx->new_match, " && is_chassis_resident(%s)", >> > > > > > - od->l3dgw_ports[0]->cr_port->json_key); >> > > > > > + dgp->cr_port->json_key); >> > > > > > } >> > > > > > >> > > > > > ovn_lflow_add_with_hint__(ctx->lflows, od, S_ROUTER_IN_DNAT, >> > > > > ctx->prio, >> > > > > > - ds_cstr(ctx->new_match), >> > > > > ctx->new_action[type], >> > > > > > + ds_cstr(ctx->new_match), >> > > > > ds_cstr(&dnat_action), >> > > > > > NULL, meter, &ctx->lb->nlb->header_, >> > > > > > lflow_ref); >> > > > > > >> > > > > > ds_truncate(ctx->new_match, new_match_len); >> > > > > > >> > > > > > + ds_destroy(&dnat_action); >> > > > > > if (!ctx->lb_vip->n_backends) { >> > > > > > return; >> > > > > > } >> > > > > > >> > > > > > + struct ds undnat_action = DS_EMPTY_INITIALIZER; >> > > > > > + struct ds snat_action = DS_EMPTY_INITIALIZER; >> > > > > > + >> > > > > > + switch (type) { >> > > > > > + case LROUTER_NAT_LB_FLOW_FORCE_SNAT: >> > > > > > + ds_put_format(&undnat_action, "flags.force_snat_for_lb = 1; >> > > > > next;"); >> > > > > > + break; >> > > > > > + case LROUTER_NAT_LB_FLOW_SKIP_SNAT: >> > > > > > + ds_put_format(&undnat_action, "flags.skip_snat_for_lb = 1; >> > > > > next;"); >> > > > > > + break; >> > > > > > + case LROUTER_NAT_LB_FLOW_NORMAL: >> > > > > > + case LROUTER_NAT_LB_FLOW_MAX: >> > > > > > + ds_put_format(&undnat_action, "%s", >> > > > > > + lrouter_use_common_zone(od) ? >> > > "ct_dnat_in_czone;" >> > > > > > + : "ct_dnat;"); >> > > > > > + break; >> > > > > > + } >> > > > > > + >> > > > > > + /* undnat_action: Remove the ct action from the lr_out_undenat >> > > NAT >> > > > > rule. >> > > > > > + */ >> > > > > > + if (stateless_nat) { >> > > > > > + ds_clear(&undnat_action); >> > > > > > + ds_put_format(&undnat_action, "next;"); >> > > > > > + } >> > > > > > + >> > > > > > /* We need to centralize the LB traffic to properly perform >> > > > > > * the undnat stage. >> > > > > > */ >> > > > > > @@ -11873,11 +11898,41 @@ build_distr_lrouter_nat_flows_for_lb(struct >> > > > > lrouter_nat_lb_flows_ctx *ctx, >> > > > > > ds_put_format(ctx->undnat_match, ") && (inport == %s || outport >> > > == >> > > > > %s)" >> > > > > > " && is_chassis_resident(%s)", dgp->json_key, >> > > > > dgp->json_key, >> > > > > > dgp->cr_port->json_key); >> > > > > > + /* Use the LB protocol as matching criteria for out undnat and >> > > snat >> > > > > when >> > > > > > + * creating LBs with stateless NAT. */ >> > > > > > + if (stateless_nat) { >> > > > > > + ds_put_format(ctx->undnat_match, " && %s", ctx->lb->proto); >> > > > > > + } >> > > > > > ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_UNDNAT, >> > > 120, >> > > > > > - ds_cstr(ctx->undnat_match), >> > > undnat_action, >> > > > > > - &ctx->lb->nlb->header_, >> > > > > > + ds_cstr(ctx->undnat_match), >> > > > > > + ds_cstr(&undnat_action), >> > > > > &ctx->lb->nlb->header_, >> > > > > > lflow_ref); >> > > > > > + >> > > > > > + /* snat_action: Add a new lr_out_snat rule with the LB VIP as >> > > > > source IP >> > > > > > + * action to perform the NAT stateless pipeline >> > > > > completely. >> > > > > > + */ >> > > > > > + if (stateless_nat) { >> > > > > > + if (ctx->lb_vip->port_str) { >> > > > > > + ds_put_format(&snat_action, "%s.src=%s; %s.src=%s; >> > > next;", >> > > > > > + ctx->lb_vip->address_family == AF_INET6 ? >> > > > > > + "ip6" : "ip4", >> > > > > > + ctx->lb_vip->vip_str, ctx->lb->proto, >> > > > > > + ctx->lb_vip->port_str); >> > > > > > + } else { >> > > > > > + ds_put_format(&snat_action, "%s.src=%s; next;", >> > > > > > + ctx->lb_vip->address_family == AF_INET6 ? >> > > > > > + "ip6" : "ip4", >> > > > > > + ctx->lb_vip->vip_str); >> > > > > > + } >> > > > > > + ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_SNAT, >> > > 160, >> > > > > > + ds_cstr(ctx->undnat_match), >> > > > > > + ds_cstr(&snat_action), >> > > > > &ctx->lb->nlb->header_, >> > > > > > + lflow_ref); >> > > > > > + } >> > > > > > + >> > > > > > ds_truncate(ctx->undnat_match, undnat_match_len); >> > > > > > + ds_destroy(&undnat_action); >> > > > > > + ds_destroy(&snat_action); >> > > > > > } >> > > > > > >> > > > > > static void >> > > > > > @@ -12022,6 +12077,8 @@ build_lrouter_nat_flows_for_lb( >> > > > > > * lflow generation for them. >> > > > > > */ >> > > > > > size_t index; >> > > > > > + bool use_stateless_nat = smap_get_bool(&lb->nlb->options, >> > > > > > + "use_stateless_nat", >> > > false); >> > > > > > BITMAP_FOR_EACH_1 (index, bitmap_len, lb_dps->nb_lr_map) { >> > > > > > struct ovn_datapath *od = lr_datapaths->array[index]; >> > > > > > enum lrouter_nat_lb_flow_type type; >> > > > > > @@ -12043,8 +12100,17 @@ build_lrouter_nat_flows_for_lb( >> > > > > > if (!od->n_l3dgw_ports) { >> > > > > > bitmap_set1(gw_dp_bitmap[type], index); >> > > > > > } else { >> > > > > > - build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, >> > > > > > - lb_dps->lflow_ref); >> > > > > > + /* Create stateless LB NAT rules when using multiple >> > > DGPs >> > > > > and >> > > > > > + * use_stateless_nat is true. >> > > > > > + */ >> > > > > > + bool stateless_nat = (od->n_l3dgw_ports > 1) >> > > > > > + ? use_stateless_nat : false; >> > > > > > + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { >> > > > > > + struct ovn_port *dgp = od->l3dgw_ports[i]; >> > > > > > + build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, >> > > > > > + >> > > lb_dps->lflow_ref, >> > > > > dgp, >> > > > > > + stateless_nat); >> > > > > > + } >> > > > > > } >> > > > > > >> > > > > > if (lb->affinity_timeout) { >> > > > > > diff --git a/ovn-nb.xml b/ovn-nb.xml >> > > > > > index 2836f58f5..ad03c6214 100644 >> > > > > > --- a/ovn-nb.xml >> > > > > > +++ b/ovn-nb.xml >> > > > > > @@ -2302,6 +2302,16 @@ or >> > > > > > local anymore by the ovn-controller. This option is set to >> > > > > > <code>false</code> by default. >> > > > > > </column> >> > > > > > + >> > > > > > + <column name="options" key="use_stateless_nat" >> > > > > > + type='{"type": "boolean"}'> >> > > > > > + If the load balancer is configured with >> > > > > <code>use_stateless_nat</code> >> > > > > > + option to <code>true</code>, the logical router that >> > > references >> > > > > this >> > > > > > + load balancer will use Stateless NAT rules when the logical >> > > > > router >> > > > > > + has multiple distributed gateway ports(DGP). Otherwise, the >> > > > > outbound >> > > > > > + traffic may be dropped in scenarios where we have different >> > > > > chassis >> > > > > > + for each DGP. This option is set to <code>false</code> by >> > > > > default. >> > > > > > + </column> >> > > > > > </group> >> > > > > > </table> >> > > > > > >> > > > > > diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at >> > > > > > index 757917626..2f69433fc 100644 >> > > > > > --- a/tests/multinode-macros.at >> > > > > > +++ b/tests/multinode-macros.at >> > > > > > @@ -40,6 +40,27 @@ m4_define([M_START_TCPDUMP], >> > > > > > ] >> > > > > > ) >> > > > > > >> > > > > > +# M_EXEC([fake_node], [command]) >> > > > > > +# >> > > > > > +# Execute 'command' in 'fakenode' >> > > > > > +m4_define([M_EXEC], >> > > > > > + [podman exec $1 $2]) >> > > > > > + >> > > > > > +# M_CHECK_EXEC([fake_node], [command], other_params...) >> > > > > > +# >> > > > > > +# Wrapper for AT_CHECK that executes 'command' inside >> > > 'fake_node''s'. >> > > > > > +# 'other_params' as passed as they are to AT_CHECK. >> > > > > > +m4_define([M_CHECK_EXEC], >> > > > > > + [ AT_CHECK([M_EXEC([$1], [$2])], >> > > m4_shift(m4_shift(m4_shift($@)))) ] >> > > > > > +) >> > > > > > + >> > > > > > +# M_FORMAT_CT([ip-addr]) >> > > > > > +# >> > > > > > +# Strip content from the piped input which would differ from test to >> > > > > test >> > > > > > +# and limit the output to the rows containing 'ip-addr'. >> > > > > > +# >> > > > > > +m4_define([M_FORMAT_CT], >> > > > > > + [[grep -F "dst=$1," | sed -e 's/id=[0-9]*/id=<cleared>/g' -e >> > > > > 's/state=[0-9_A-Z]*/state=<cleared>/g' | sort | uniq | sed -e >> > > > > 's/zone=[[0-9]]*/zone=<cleared>/' -e 's/mark=[[0-9]]*/mark=<cleared>/' >> > > ]]) >> > > > > > >> > > > > > OVS_START_SHELL_HELPERS >> > > > > > >> > > > > > @@ -76,6 +97,25 @@ multinode_nbctl () { >> > > > > > m_as ovn-central ovn-nbctl "$@" >> > > > > > } >> > > > > > >> > > > > > +check_fake_multinode_setup_by_nodes() { >> > > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync >> > > > > > + for c in $1 >> > > > > > + do >> > > > > > + AT_CHECK([m_as $c ovn-appctl -t ovn-controller version], >> > > [0], >> > > > > [ignore]) >> > > > > > + done >> > > > > > +} >> > > > > > + >> > > > > > +cleanup_multinode_resources_by_nodes() { >> > > > > > + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db >> > > > > > + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd >> > > > > > + check m_as ovn-central ovn-nbctl --wait=sb sync >> > > > > > + for c in $1 >> > > > > > + do >> > > > > > + m_as $c ovs-vsctl del-br br-int >> > > > > > + m_as $c ip --all netns delete >> > > > > > + done >> > > > > > +} >> > > > > > + >> > > > > > # m_count_rows TABLE [CONDITION...] >> > > > > > # >> > > > > > # Prints the number of rows in TABLE (that satisfy CONDITION). >> > > > > > diff --git a/tests/multinode.at b/tests/multinode.at >> > > > > > index a0eb8fc67..b1beb4d97 100644 >> > > > > > --- a/tests/multinode.at >> > > > > > +++ b/tests/multinode.at >> > > > > > @@ -1591,3 +1591,559 @@ AT_CHECK([cat ch1_eth2.tcpdump], [0], [dnl >> > > > > > ]) >> > > > > > >> > > > > > AT_CLEANUP >> > > > > > + >> > > > > > +AT_SETUP([ovn multinode load-balancer with multiple DGPs and >> > > multiple >> > > > > chassis]) >> > > > > > + >> > > > > > +# Check that ovn-fake-multinode setup is up and running - requires >> > > > > additional nodes >> > > > > > +check_fake_multinode_setup_by_nodes 'ovn-chassis-1 ovn-chassis-2 >> > > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' >> > > > > > + >> > > > > > +# Delete the multinode NB and OVS resources before starting the >> > > test. >> > > > > > +cleanup_multinode_resources_by_nodes 'ovn-chassis-1 ovn-chassis-2 >> > > > > ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' >> > > > > > + >> > > > > > +# Network topology >> > > > > > +# >> > > > > > +# publicp1 (ovn-chassis-3) (20.0.0.3/24) >> > > > > > +# | >> > > > > > +# overlay >> > > > > > +# | >> > > > > > +# DGP public1 (ovn-gw-1) (20.0.0.1/24) >> > > > > > +# | >> > > > > > +# | >> > > > > > +# | >> > > > > > +# lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) >> > > 10.0.0.3/24 >> > > > > > +# | | >> > > > > > +# | + --- sw0p2 (ovn-chassis-2) >> > > 10.0.0.4/24 >> > > > > > +# | >> > > > > > +# DGP public2 (ovn-gw-2) (30.0.0.1/24) >> > > > > > +# | >> > > > > > +# overlay >> > > > > > +# | >> > > > > > +# publicp2 (ovn-chassis-4) (30.0.0.3/24) >> > > > > > + >> > > > > > +# Delete already used ovs-ports >> > > > > > +m_as ovn-chassis-1 ovs-vsctl del-port br-int sw0p1-p >> > > > > > +m_as ovn-chassis-2 ovs-vsctl del-port br-int sw0p2-p >> > > > > > +m_as ovn-chassis-1 ip link del sw0p1-p >> > > > > > +m_as ovn-chassis-2 ip link del sw0p2-p >> > > > > > +m_as ovn-chassis-3 ovs-vsctl del-port br-int publicp1-p >> > > > > > +m_as ovn-chassis-4 ovs-vsctl del-port br-int publicp2-p >> > > > > > +m_as ovn-chassis-3 ip link del publicp1-p >> > > > > > +m_as ovn-chassis-4 ip link del publicp2-p >> > > > > > + >> > > > > > +# Create East-West switch for LB backends >> > > > > > +check multinode_nbctl ls-add sw0 >> > > > > > +check multinode_nbctl lsp-add sw0 sw0-port1 >> > > > > > +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 >> > > > > 10.0.0.3 1000::3" >> > > > > > +check multinode_nbctl lsp-add sw0 sw0-port2 >> > > > > > +check multinode_nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 >> > > > > 10.0.0.4 1000::4" >> > > > > > + >> > > > > > +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 >> > > > > 50:54:00:00:00:03 1400 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a >> > > > > > +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 >> > > > > 50:54:00:00:00:04 1400 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a >> > > > > > + >> > > > > > +m_wait_for_ports_up >> > > > > > + >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 >> > > > > 10.0.0.4 | FORMAT_PING], \ >> > > > > > +[0], [dnl >> > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> > > > > > +]) >> > > > > > + >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 >> > > > > 10.0.0.3 | FORMAT_PING], \ >> > > > > > +[0], [dnl >> > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> > > > > > +]) >> > > > > > + >> > > > > > +# Create a logical router and attach to sw0 >> > > > > > +check multinode_nbctl lr-add lr0 >> > > > > > +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 >> > > 10.0.0.1/24 >> > > > > 1000::a/64 >> > > > > > +check multinode_nbctl lsp-add sw0 sw0-lr0 >> > > > > > +check multinode_nbctl lsp-set-type sw0-lr0 router >> > > > > > +check multinode_nbctl lsp-set-addresses sw0-lr0 router >> > > > > > +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 >> > > > > > + >> > > > > > +# create external connection for N/S traffic using multiple DGPs >> > > > > > +check multinode_nbctl ls-add public >> > > > > > + >> > > > > > +# DGP public1 >> > > > > > +check multinode_nbctl lsp-add public ln-public-1 >> > > > > > +check multinode_nbctl lsp-set-type ln-public-1 localnet >> > > > > > +check multinode_nbctl lsp-set-addresses ln-public-1 unknown >> > > > > > +check multinode_nbctl lsp-set-options ln-public-1 >> > > network_name=public1 >> > > > > > + >> > > > > > +# DGP public2 >> > > > > > +# create exteranl connection for N/S traffic >> > > > > > +check multinode_nbctl lsp-add public ln-public-2 >> > > > > > +check multinode_nbctl lsp-set-type ln-public-2 localnet >> > > > > > +check multinode_nbctl lsp-set-addresses ln-public-2 unknown >> > > > > > +check multinode_nbctl lsp-set-options ln-public-2 >> > > network_name=public2 >> > > > > > + >> > > > > > +# Attach DGP public1 to GW-1 and chassis-3 (overlay connectivity) >> > > > > > +m_as ovn-gw-1 ovs-vsctl set open . >> > > > > external-ids:ovn-bridge-mappings=public1:br-ex >> > > > > > +m_as ovn-chassis-3 ovs-vsctl set open . >> > > > > external-ids:ovn-bridge-mappings=public1:br-ex >> > > > > > + >> > > > > > +# Attach DGP public2 to GW-2 and chassis-4 (overlay connectivity) >> > > > > > +m_as ovn-gw-2 ovs-vsctl set open . >> > > > > external-ids:ovn-bridge-mappings=public2:br-ex >> > > > > > +m_as ovn-chassis-4 ovs-vsctl set open . >> > > > > external-ids:ovn-bridge-mappings=public2:br-ex >> > > > > > + >> > > > > > +# Create the external LR0 port to the DGP public1 >> > > > > > +check multinode_nbctl lsp-add public public-port1 >> > > > > > +check multinode_nbctl lsp-set-addresses public-port1 >> > > "40:54:00:00:00:03 >> > > > > 20.0.0.3 2000::3" >> > > > > > + >> > > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p1 00:00:00:00:ff:02 >> > > > > 20.0.0.1/24 2000::a/64 >> > > > > > +check multinode_nbctl lsp-add public public-lr0-p1 >> > > > > > +check multinode_nbctl lsp-set-type public-lr0-p1 router >> > > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p1 router >> > > > > > +check multinode_nbctl lsp-set-options public-lr0-p1 >> > > > > router-port=lr0-public-p1 >> > > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p1 >> > > ovn-gw-1 10 >> > > > > > + >> > > > > > +# Create a VM on ovn-chassis-3 in the same public1 overlay >> > > > > > +m_as ovn-chassis-3 /data/create_fake_vm.sh public-port1 publicp1 >> > > > > 40:54:00:00:00:03 1400 20.0.0.3 24 20.0.0.1 2000::4/64 2000::a >> > > > > > + >> > > > > > +m_wait_for_ports_up public-port1 >> > > > > > + >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [ping -q -c 3 -i 0.3 >> > > -w 2 >> > > > > 20.0.0.1 | FORMAT_PING], \ >> > > > > > +[0], [dnl >> > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> > > > > > +]) >> > > > > > + >> > > > > > +# Create the external LR0 port to the DGP public2 >> > > > > > +check multinode_nbctl lsp-add public public-port2 >> > > > > > +check multinode_nbctl lsp-set-addresses public-port2 >> > > "60:54:00:00:00:03 >> > > > > 30.0.0.3 3000::3" >> > > > > > + >> > > > > > +check multinode_nbctl lrp-add lr0 lr0-public-p2 00:00:00:00:ff:03 >> > > > > 30.0.0.1/24 3000::a/64 >> > > > > > +check multinode_nbctl lsp-add public public-lr0-p2 >> > > > > > +check multinode_nbctl lsp-set-type public-lr0-p2 router >> > > > > > +check multinode_nbctl lsp-set-addresses public-lr0-p2 router >> > > > > > +check multinode_nbctl lsp-set-options public-lr0-p2 >> > > > > router-port=lr0-public-p2 >> > > > > > +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p2 >> > > ovn-gw-2 10 >> > > > > > + >> > > > > > +# Create a VM on ovn-chassis-4 in the same public2 overlay >> > > > > > +m_as ovn-chassis-4 /data/create_fake_vm.sh public-port2 publicp2 >> > > > > 60:54:00:00:00:03 1400 30.0.0.3 24 30.0.0.1 3000::4/64 3000::a >> > > > > > + >> > > > > > +m_wait_for_ports_up public-port2 >> > > > > > + >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [ping -q -c 3 -i 0.3 >> > > -w 2 >> > > > > 30.0.0.1 | FORMAT_PING], \ >> > > > > > +[0], [dnl >> > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> > > > > > +]) >> > > > > > + >> > > > > > +# Add a default route for multiple DGPs - using ECMP >> > > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 >> > > 20.0.0.3 >> > > > > > +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 >> > > 30.0.0.3 >> > > > > > + >> > > > > > +# Add SNAT rules using gateway-port >> > > > > > +check multinode_nbctl --gateway-port lr0-public-p1 lr-nat-add lr0 >> > > snat >> > > > > 20.0.0.1 10.0.0.0/24 >> > > > > > +check multinode_nbctl --gateway-port lr0-public-p2 lr-nat-add lr0 >> > > snat >> > > > > 30.0.0.1 10.0.0.0/24 >> > > > > > + >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 >> > > > > 20.0.0.3 | FORMAT_PING], \ >> > > > > > +[0], [dnl >> > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> > > > > > +]) >> > > > > > + >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 >> > > > > 30.0.0.3 | FORMAT_PING], \ >> > > > > > +[0], [dnl >> > > > > > +3 packets transmitted, 3 received, 0% packet loss, time 0ms >> > > > > > +]) >> > > > > > + >> > > > > > +# create LB >> > > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:80" "10.0.0.3:80, >> > > > > 10.0.0.4:80" >> > > > > > +check multinode_nbctl lr-lb-add lr0 lb0 >> > > > > > +check multinode_nbctl ls-lb-add sw0 lb0 >> > > > > > + >> > > > > > +# Set use_stateless_nat to true >> > > > > > +check multinode_nbctl set load_balancer lb0 >> > > > > options:use_stateless_nat=true >> > > > > > + >> > > > > > +# Start backend http services >> > > > > > +M_NS_DAEMONIZE([ovn-chassis-1], [sw0p1], [$PYTHON -m http.server >> > > --bind >> > > > > 10.0.0.3 80 >/dev/null 2>&1], [http1.pid]) >> > > > > > +M_NS_DAEMONIZE([ovn-chassis-2], [sw0p2], [$PYTHON -m http.server >> > > --bind >> > > > > 10.0.0.4 80 >/dev/null 2>&1], [http2.pid]) >> > > > > > + >> > > > > > +# wait for http server be ready >> > > > > > +sleep 2 >> > > > > > + >> > > > > > +# Flush conntrack entries for easier output parsing of next test. >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v >> > > 172.16.0.100:80 >> > > > > --retry 3 --max-time 1 --local-port 59002 2> curl.out']) >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [sh -c 'cat curl.out | >> > > > > grep -i -e connect | grep -v 'Server:''], \ >> > > > > > +[0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v >> > > 172.16.0.100:80 >> > > > > --retry 3 --max-time 1 --local-port 59003 2> curl.out']) >> > > > > > +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [sh -c 'cat curl.out | >> > > > > grep -i -e connect | grep -v 'Server:''], \ >> > > > > > +[0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v >> > > 172.16.0.100:80 >> > > > > --retry 3 --max-time 1 --local-port 59001']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | >> > > > > M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v >> > > 172.16.0.100:80 >> > > > > --retry 3 --max-time 1 --local-port 59000']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | >> > > > > M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# create a big file on web servers for download >> > > > > > +M_NS_EXEC([ovn-chassis-1], [sw0p1], [dd bs=512 count=200000 >> > > > > if=/dev/urandom of=download_file]) >> > > > > > +M_NS_EXEC([ovn-chassis-2], [sw0p2], [dd bs=512 count=200000 >> > > > > if=/dev/urandom of=download_file]) >> > > > > > + >> > > > > > +# Flush conntrack entries for easier output parsing of next test. >> > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port >> > > 59004 >> > > > > 2>curl.out']) >> > > > > > + >> > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > + >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat >> > > > > curl.out | \ >> > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if we have only one backend for the same connection - orig + >> > > > > dest ports >> > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if gw-2 is empty to ensure that the traffic only come >> > > from/to >> > > > > the originator chassis via DGP public1 >> > > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > + >> > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) >> > > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" >> > > -c) >> > > > > > + >> > > > > > +if [[ $backend_check -gt 0 ]]; then >> > > > > > +# Backend resides on ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59004,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +else >> > > > > > +# Backend resides on ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59004,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +fi >> > > > > > + >> > > > > > +# Flush conntrack entries for easier output parsing of next test. >> > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +# Check the flows again for a new source port >> > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port >> > > 59005 >> > > > > 2>curl.out']) >> > > > > > + >> > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > + >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat >> > > > > curl.out | \ >> > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if we have only one backend for the same connection - orig + >> > > > > dest ports >> > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if gw-2 is empty to ensure that the traffic only come >> > > from/to >> > > > > the originator chassis via DGP public1 >> > > > > > +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > + >> > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) >> > > > > > +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" >> > > -c) >> > > > > > + >> > > > > > +if [[ $backend_check -gt 0 ]]; then >> > > > > > +# Backend resides on ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59005,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +else >> > > > > > +# Backend resides on ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59005,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +fi >> > > > > > + >> > > > > > +# Flush conntrack entries for easier output parsing of next test. >> > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +# Start a new test using the second DGP as origin (public2) >> > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port >> > > 59006 >> > > > > 2>curl.out']) >> > > > > > + >> > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > + >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat >> > > > > curl.out | \ >> > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if we have only one backend for the same connection - orig + >> > > > > dest ports >> > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if gw-1 is empty to ensure that the traffic only come >> > > from/to >> > > > > the originator chassis via DGP public2 >> > > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > + >> > > > > > +# Check the backend IP from ct entries on gw-2 (DGP public2) >> > > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" >> > > -c) >> > > > > > + >> > > > > > +if [[ $backend_check -gt 0 ]]; then >> > > > > > +# Backend resides on ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59006,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +else >> > > > > > +# Backend resides on ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59006,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +fi >> > > > > > + >> > > > > > +# Flush conntrack entries for easier output parsing of next test. >> > > > > > +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +# Check the flows again for a new source port using the second DGP >> > > as >> > > > > origin (public2) >> > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port >> > > 59007 >> > > > > 2>curl.out']) >> > > > > > + >> > > > > > +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | >> > > sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed >> > > > > ':a;N;$!ba;s/\n/\\n/g') >> > > > > > + >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat >> > > > > curl.out | \ >> > > > > > +grep -i -e connect | grep -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if we have only one backend for the same connection - orig + >> > > > > dest ports >> > > > > > +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Check if gw-1 is empty to ensure that the traffic only come >> > > from/to >> > > > > the originator chassis via DGP public2 >> > > > > > +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > + >> > > > > > +# Check the backend IP from ct entries on gw-1 (DGP public1) >> > > > > > +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" >> > > -c) >> > > > > > + >> > > > > > +if [[ $backend_check -gt 0 ]]; then >> > > > > > +# Backend resides on ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59007,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-1 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +else >> > > > > > +# Backend resides on ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp], [0], [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59007,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +# Ensure that the traffic only come from ovn-chassis-2 >> > > > > > +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" >> > > -c], >> > > > > [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" >> > > > > -c], [1], [dnl >> > > > > > +0 >> > > > > > +]) >> > > > > > +fi >> > > > > > + >> > > > > > +# Check multiple requests coming from DGP's public1 and public2 >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ >> > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | >> > > grep >> > > > > -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +200 OK >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ >> > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | >> > > grep >> > > > > -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +200 OK >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ >> > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | >> > > grep >> > > > > -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +200 OK >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O >> > > > > 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ >> > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | >> > > grep >> > > > > -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 80 >> > > > > > +200 OK >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +# Remove the LB and change the VIP port - different from the backend >> > > > > ports >> > > > > > +check multinode_nbctl lb-del lb0 >> > > > > > + >> > > > > > +# create LB again >> > > > > > +check multinode_nbctl lb-add lb0 "172.16.0.100:9000" "10.0.0.3:80, >> > > > > 10.0.0.4:80" >> > > > > > +check multinode_nbctl lr-lb-add lr0 lb0 >> > > > > > +check multinode_nbctl ls-lb-add sw0 lb0 >> > > > > > + >> > > > > > +# Set use_stateless_nat to true >> > > > > > +check multinode_nbctl set load_balancer lb0 >> > > > > options:use_stateless_nat=true >> > > > > > + >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +# Check end-to-end request using a new port for VIP >> > > > > > +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O >> > > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port >> > > 59008 >> > > > > 2>curl.out']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | >> > > > > M_FORMAT_CT(20.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ >> > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | >> > > grep >> > > > > -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 >> > > > > > +200 OK >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack >> > > > > > +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack >> > > > > > + >> > > > > > +# Check end-to-end request using a new port for VIP >> > > > > > +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O >> > > > > 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port >> > > 59008 >> > > > > 2>curl.out']) >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | >> > > > > M_FORMAT_CT(30.0.0.3) | \ >> > > > > > +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], >> > > [0], >> > > > > [dnl >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) >> > > > > > >> > > > > >> > > +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) >> > > > > > +]) >> > > > > > + >> > > > > > +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ >> > > > > > +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | >> > > grep >> > > > > -v 'Server:'], [0], [dnl >> > > > > > +* Connected to 172.16.0.100 (172.16.0.100) port 9000 >> > > > > > +200 OK >> > > > > > +* Closing connection >> > > > > > +]) >> > > > > > + >> > > > > > +AT_CLEANUP >> > > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at >> > > > > > index dcc3dbbc3..9e7a2f225 100644 >> > > > > > --- a/tests/ovn-northd.at >> > > > > > +++ b/tests/ovn-northd.at >> > > > > > @@ -13864,3 +13864,323 @@ check_no_redirect >> > > > > > >> > > > > > AT_CLEANUP >> > > > > > ]) >> > > > > > + >> > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ >> > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + >> > > NAT >> > > > > Stateless)]) >> > > > > > +ovn_start >> > > > > > + >> > > > > > +check ovn-nbctl ls-add public >> > > > > > +check ovn-nbctl lr-add lr1 >> > > > > > + >> > > > > > +# lr1 DGP ts1 >> > > > > > +check ovn-nbctl ls-add ts1 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 >> > > 172.16.10.1/24 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 >> > > > > > + >> > > > > > +# lr1 DGP ts2 >> > > > > > +check ovn-nbctl ls-add ts2 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 >> > > 172.16.20.1/24 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 >> > > > > > + >> > > > > > +# lr1 DGP public >> > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 >> > > 173.16.0.1/16 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 >> > > > > > + >> > > > > > +check ovn-nbctl ls-add s1 >> > > > > > +# s1 - lr1 >> > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 >> > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router >> > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 >> > > 172.16.0.1" >> > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 >> > > > > > + >> > > > > > +# s1 - backend vm1 >> > > > > > +check ovn-nbctl lsp-add s1 vm1 >> > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 >> > > 172.16.0.101" >> > > > > > + >> > > > > > +# s1 - backend vm2 >> > > > > > +check ovn-nbctl lsp-add s1 vm2 >> > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 >> > > 172.16.0.102" >> > > > > > + >> > > > > > +# s1 - backend vm3 >> > > > > > +check ovn-nbctl lsp-add s1 vm3 >> > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 >> > > 172.16.0.103" >> > > > > > + >> > > > > > +# Add the lr1 DGP ts1 to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Add the lr1 DGP ts2 to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Add the lr1 DGP public to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Create the Load Balancer lb1 >> > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" >> > > > > "172.16.0.103,172.16.0.102,172.16.0.101" >> > > > > > + >> > > > > > +# Set use_stateless_nat to true >> > > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 >> > > > > options:use_stateless_nat=true >> > > > > > + >> > > > > > +# Associate load balancer to s1 >> > > > > > +check ovn-nbctl ls-lb-add s1 lb1 >> > > > > > +check ovn-nbctl --wait=sb sync >> > > > > > + >> > > > > > +ovn-sbctl dump-flows s1 > s1flows >> > > > > > +AT_CAPTURE_FILE([s1flows]) >> > > > > > + >> > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | >> > > grep >> > > > > "30.0.0.1"], [0], [dnl >> > > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] >> > > == 1 >> > > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) >> > > > > > +]) >> > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep >> > > > > "30.0.0.1"], [0], [dnl >> > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && >> > > > > ip4.dst == 30.0.0.1), >> > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > +]) >> > > > > > + >> > > > > > +# Associate load balancer to lr1 with DGP >> > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 >> > > > > > +check ovn-nbctl --wait=sb sync >> > > > > > + >> > > > > > +ovn-sbctl dump-flows lr1 > lr1flows >> > > > > > +AT_CAPTURE_FILE([lr1flows]) >> > > > > > + >> > > > > > +# Check stateless NAT rules for load balancer with multiple DGP >> > > > > > +# 1. Check if the backend IPs are in the ipX.dst action >> > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep >> > > > > "30.0.0.1"], [0], [dnl >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && >> > > > > is_chassis_resident("cr-lr1-ts1")), >> > > > > >> > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && >> > > > > is_chassis_resident("cr-lr1-ts2")), >> > > > > >> > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && >> > > > > is_chassis_resident("cr-lr1_public")), >> > > > > >> > > action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > +]) >> > > > > > + >> > > > > > +# 2. Check if the DGP ports are in the match with action next >> > > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], >> > > [dnl >> > > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), >> > > > > action=(next;) >> > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && >> > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == >> > > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && >> > > > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) >> > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && >> > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == >> > > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && >> > > > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) >> > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip4 && >> > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == >> > > > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") >> > > && >> > > > > is_chassis_resident("cr-lr1_public") && tcp), action=(next;) >> > > > > > +]) >> > > > > > + >> > > > > > +# 3. Check if the VIP IP is in the ipX.src action >> > > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl >> > > > > > + table=??(lr_out_snat ), priority=0 , match=(1), >> > > > > action=(next;) >> > > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), >> > > > > action=(next;) >> > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && >> > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == >> > > > > 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && >> > > > > is_chassis_resident("cr-lr1-ts1") && tcp), action=(ip4.src=30.0.0.1; >> > > next;) >> > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && >> > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == >> > > > > 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && >> > > > > is_chassis_resident("cr-lr1-ts2") && tcp), action=(ip4.src=30.0.0.1; >> > > next;) >> > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip4 && >> > > > > ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == >> > > > > 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") >> > > && >> > > > > is_chassis_resident("cr-lr1_public") && tcp), action=(ip4.src=30.0.0.1; >> > > > > next;) >> > > > > > +]) >> > > > > > + >> > > > > > +AT_CLEANUP >> > > > > > +]) >> > > > > > + >> > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ >> > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + >> > > NAT >> > > > > Stateless) - IPv6]) >> > > > > > +ovn_start >> > > > > > + >> > > > > > +check ovn-nbctl ls-add public >> > > > > > +check ovn-nbctl lr-add lr1 >> > > > > > + >> > > > > > +# lr1 DGP ts1 >> > > > > > +check ovn-nbctl ls-add ts1 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 >> > > > > 2001:db8:aaaa:1::1/64 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 >> > > > > > + >> > > > > > +# lr1 DGP ts2 >> > > > > > +check ovn-nbctl ls-add ts2 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 >> > > > > 2001:db8:aaaa:2::1/64 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 >> > > > > > + >> > > > > > +# lr1 DGP public >> > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 >> > > > > 2001:db8:bbbb::1/64 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 >> > > > > 2001:db8:aaaa:3::1/64 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 >> > > > > > + >> > > > > > +check ovn-nbctl ls-add s1 >> > > > > > +# s1 - lr1 >> > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 >> > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router >> > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 >> > > > > 2001:db8:aaaa:3::1" >> > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 >> > > > > > + >> > > > > > +# s1 - backend vm1 >> > > > > > +check ovn-nbctl lsp-add s1 vm1 >> > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 >> > > > > 2001:db8:aaaa:3::101" >> > > > > > + >> > > > > > +# s1 - backend vm2 >> > > > > > +check ovn-nbctl lsp-add s1 vm2 >> > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 >> > > > > 2001:db8:aaaa:3::102" >> > > > > > + >> > > > > > +# s1 - backend vm3 >> > > > > > +check ovn-nbctl lsp-add s1 vm3 >> > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 >> > > > > 2001:db8:aaaa:3::103" >> > > > > > + >> > > > > > +# Add the lr1 DGP ts1 to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Add the lr1 DGP ts2 to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Add the lr1 DGP public to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Create the Load Balancer lb1 >> > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "2001:db8:cccc::1" >> > > > > "2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101" >> > > > > > + >> > > > > > +# Set use_stateless_nat to true >> > > > > > +check ovn-nbctl --wait=sb set load_balancer lb1 >> > > > > options:use_stateless_nat=true >> > > > > > + >> > > > > > +# Associate load balancer to s1 >> > > > > > +check ovn-nbctl ls-lb-add s1 lb1 >> > > > > > +check ovn-nbctl --wait=sb sync >> > > > > > + >> > > > > > +ovn-sbctl dump-flows s1 > s1flows >> > > > > > +AT_CAPTURE_FILE([s1flows]) >> > > > > > + >> > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | >> > > grep >> > > > > "2001:db8:cccc::1"], [0], [dnl >> > > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] >> > > == 1 >> > > > > && ip6.dst == 2001:db8:cccc::1), action=(xxreg1 = 2001:db8:cccc::1; >> > > > > ct_lb_mark;) >> > > > > > +]) >> > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep >> > > > > "2001:db8:cccc::1"], [0], [dnl >> > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && >> > > > > ip6.dst == 2001:db8:cccc::1), >> > > > > >> > > action=(ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) >> > > > > > +]) >> > > > > > + >> > > > > > +# Associate load balancer to lr1 with DGP >> > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 >> > > > > > +check ovn-nbctl --wait=sb sync >> > > > > > + >> > > > > > +ovn-sbctl dump-flows lr1 > lr1flows >> > > > > > +AT_CAPTURE_FILE([lr1flows]) >> > > > > > + >> > > > > > +# Check stateless NAT rules for load balancer with multiple DGP >> > > > > > +# 1. Check if the backend IPs are in the ipX.dst action >> > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep >> > > > > "2001:db8:cccc::1"], [0], [dnl >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && >> > > > > is_chassis_resident("cr-lr1-ts1")), >> > > > > >> > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && >> > > > > is_chassis_resident("cr-lr1-ts2")), >> > > > > >> > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && >> > > > > is_chassis_resident("cr-lr1_public")), >> > > > > >> > > action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) >> > > > > > +]) >> > > > > > + >> > > > > > +# 2. Check if the DGP ports are in the match with action next >> > > > > > +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], >> > > [dnl >> > > > > > + table=??(lr_out_undnat ), priority=0 , match=(1), >> > > > > action=(next;) >> > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && >> > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == >> > > 2001:db8:aaaa:3::102) || >> > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport >> > > == >> > > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) >> > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && >> > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == >> > > 2001:db8:aaaa:3::102) || >> > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport >> > > == >> > > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) >> > > > > > + table=??(lr_out_undnat ), priority=120 , match=(ip6 && >> > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == >> > > 2001:db8:aaaa:3::102) || >> > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || >> > > outport == >> > > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), >> > > > > action=(next;) >> > > > > > +]) >> > > > > > + >> > > > > > +# 3. Check if the VIP IP is in the ipX.src action >> > > > > > +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl >> > > > > > + table=??(lr_out_snat ), priority=0 , match=(1), >> > > > > action=(next;) >> > > > > > + table=??(lr_out_snat ), priority=120 , match=(nd_ns), >> > > > > action=(next;) >> > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && >> > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == >> > > 2001:db8:aaaa:3::102) || >> > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport >> > > == >> > > > > "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), >> > > > > action=(ip6.src=2001:db8:cccc::1; next;) >> > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && >> > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == >> > > 2001:db8:aaaa:3::102) || >> > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport >> > > == >> > > > > "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), >> > > > > action=(ip6.src=2001:db8:cccc::1; next;) >> > > > > > + table=??(lr_out_snat ), priority=160 , match=(ip6 && >> > > > > ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == >> > > 2001:db8:aaaa:3::102) || >> > > > > (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || >> > > outport == >> > > > > "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), >> > > > > action=(ip6.src=2001:db8:cccc::1; next;) >> > > > > > +]) >> > > > > > + >> > > > > > +AT_CLEANUP >> > > > > > +]) >> > > > > > + >> > > > > > +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ >> > > > > > +AT_SETUP([Load balancer with Distributed Gateway Ports (DGP)]) >> > > > > > +ovn_start >> > > > > > + >> > > > > > +check ovn-nbctl ls-add public >> > > > > > +check ovn-nbctl lr-add lr1 >> > > > > > + >> > > > > > +# lr1 DGP ts1 >> > > > > > +check ovn-nbctl ls-add ts1 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 >> > > 172.16.10.1/24 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-1 >> > > > > > + >> > > > > > +# lr1 DGP ts2 >> > > > > > +check ovn-nbctl ls-add ts2 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 >> > > 172.16.20.1/24 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-1 >> > > > > > + >> > > > > > +# lr1 DGP public >> > > > > > +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 >> > > 173.16.0.1/16 >> > > > > > +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 >> > > > > > +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 >> > > > > > + >> > > > > > +check ovn-nbctl ls-add s1 >> > > > > > +# s1 - lr1 >> > > > > > +check ovn-nbctl lsp-add s1 s1_lr1 >> > > > > > +check ovn-nbctl lsp-set-type s1_lr1 router >> > > > > > +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 >> > > 172.16.0.1" >> > > > > > +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 >> > > > > > + >> > > > > > +# s1 - backend vm1 >> > > > > > +check ovn-nbctl lsp-add s1 vm1 >> > > > > > +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 >> > > 172.16.0.101" >> > > > > > + >> > > > > > +# s1 - backend vm2 >> > > > > > +check ovn-nbctl lsp-add s1 vm2 >> > > > > > +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 >> > > 172.16.0.102" >> > > > > > + >> > > > > > +# s1 - backend vm3 >> > > > > > +check ovn-nbctl lsp-add s1 vm3 >> > > > > > +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 >> > > 172.16.0.103" >> > > > > > + >> > > > > > +# Add the lr1 DGP ts1 to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1_ts1 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts1 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Add the lr1 DGP ts2 to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1_ts2 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1_ts2 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Add the lr1 DGP public to the public switch >> > > > > > +check ovn-nbctl lsp-add public public_lr1 >> > > > > > +check ovn-nbctl lsp-set-type public_lr1 router >> > > > > > +check ovn-nbctl lsp-set-addresses public_lr1 router >> > > > > > +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public >> > > > > nat-addresses=router >> > > > > > + >> > > > > > +# Create the Load Balancer lb1 >> > > > > > +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" >> > > > > "172.16.0.103,172.16.0.102,172.16.0.101" >> > > > > > + >> > > > > > +# Associate load balancer to s1 >> > > > > > +check ovn-nbctl ls-lb-add s1 lb1 >> > > > > > +check ovn-nbctl --wait=sb sync >> > > > > > + >> > > > > > +ovn-sbctl dump-flows s1 > s1flows >> > > > > > +AT_CAPTURE_FILE([s1flows]) >> > > > > > + >> > > > > > +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | >> > > grep >> > > > > "30.0.0.1"], [0], [dnl >> > > > > > + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] >> > > == 1 >> > > > > && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) >> > > > > > +]) >> > > > > > +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep >> > > > > "30.0.0.1"], [0], [dnl >> > > > > > + table=??(ls_in_lb ), priority=110 , match=(ct.new && >> > > > > ip4.dst == 30.0.0.1), >> > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > +]) >> > > > > > + >> > > > > > +# Associate load balancer to lr1 with DGP >> > > > > > +check ovn-nbctl lr-lb-add lr1 lb1 >> > > > > > +check ovn-nbctl --wait=sb sync >> > > > > > + >> > > > > > +ovn-sbctl dump-flows lr1 > lr1flows >> > > > > > +AT_CAPTURE_FILE([lr1flows]) >> > > > > > + >> > > > > > +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep >> > > > > "30.0.0.1"], [0], [dnl >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && >> > > > > is_chassis_resident("cr-lr1-ts1")), >> > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && >> > > > > is_chassis_resident("cr-lr1-ts2")), >> > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > + table=??(lr_in_dnat ), priority=110 , match=(ct.new && >> > > > > !ct.rel && ip4 && ip4.dst == 30.0.0.1 && >> > > > > is_chassis_resident("cr-lr1_public")), >> > > > > action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) >> > > > > > +]) >> > > > > > + >> > > > > > +AT_CLEANUP >> > > > > > +]) >> > > > > > -- >> > > > > > 2.34.1 >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > _'Esta mensagem é direcionada apenas para os endereços constantes no >> > > > > > cabeçalho inicial. Se você não está listado nos endereços constantes >> > > no >> > > > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo >> > > dessa >> > > > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas >> > > > > estão >> > > > > > imediatamente anuladas e proibidas'._ >> > > > > > >> > > > > > >> > > > > > * **'Apesar do Magazine Luiza tomar >> > > > > > todas as precauções razoáveis para assegurar que nenhum vírus esteja >> > > > > > presente nesse e-mail, a empresa não poderá aceitar a >> > > responsabilidade >> > > > > por >> > > > > > quaisquer perdas ou danos causados por esse e-mail ou por seus >> > > anexos'.* >> > > > > > >> > > > > > >> > > > > > >> > > > > > _______________________________________________ >> > > > > > dev mailing list >> > > > > > dev@openvswitch.org >> > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev >> > > > > > >> > > > > >> > > > >> > > > -- >> > > > >> > > > >> > > > >> > > > >> > > > _‘Esta mensagem é direcionada apenas para os endereços constantes no >> > > > cabeçalho inicial. Se você não está listado nos endereços constantes no >> > > > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa >> > > > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas >> > > estão >> > > > imediatamente anuladas e proibidas’._ >> > > > >> > > > >> > > > * **‘Apesar do Magazine Luiza tomar >> > > > todas as precauções razoáveis para assegurar que nenhum vírus esteja >> > > > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade >> > > por >> > > > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* >> > > > >> > > > >> > > > >> > > > _______________________________________________ >> > > > dev mailing list >> > > > dev@openvswitch.org >> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev >> > > >> > >> > -- >> > >> > >> > >> > >> > _‘Esta mensagem é direcionada apenas para os endereços constantes no >> > cabeçalho inicial. Se você não está listado nos endereços constantes no >> > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa >> > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão >> > imediatamente anuladas e proibidas’._ >> > >> > >> > * **‘Apesar do Magazine Luiza tomar >> > todas as precauções razoáveis para assegurar que nenhum vírus esteja >> > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por >> > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.* >> > >> > >> > >> > _______________________________________________ >> > dev mailing list >> > dev@openvswitch.org >> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev >> > > > ‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’. > > ‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.
diff --git a/northd/en-lr-stateful.c b/northd/en-lr-stateful.c index baf1bd2f8..f09691af6 100644 --- a/northd/en-lr-stateful.c +++ b/northd/en-lr-stateful.c @@ -516,18 +516,6 @@ lr_stateful_record_create(struct lr_stateful_table *table, table->array[od->index] = lr_stateful_rec; - /* Load balancers are not supported (yet) if a logical router has multiple - * distributed gateway port. Log a warning. */ - if (lr_stateful_rec->has_lb_vip && lr_has_multiple_gw_ports(od)) { - static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1); - VLOG_WARN_RL(&rl, "Load-balancers are configured on logical " - "router %s, which has %"PRIuSIZE" distributed " - "gateway ports. Load-balancer is not supported " - "yet when there is more than one distributed " - "gateway port on the router.", - od->nbr->name, od->n_l3dgw_ports); - } - return lr_stateful_rec; } diff --git a/northd/northd.c b/northd/northd.c index a267cd5f8..bbe97acf8 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -11807,31 +11807,30 @@ static void build_distr_lrouter_nat_flows_for_lb(struct lrouter_nat_lb_flows_ctx *ctx, enum lrouter_nat_lb_flow_type type, struct ovn_datapath *od, - struct lflow_ref *lflow_ref) + struct lflow_ref *lflow_ref, + struct ovn_port *dgp, + bool stateless_nat) { - struct ovn_port *dgp = od->l3dgw_ports[0]; - - const char *undnat_action; - - switch (type) { - case LROUTER_NAT_LB_FLOW_FORCE_SNAT: - undnat_action = "flags.force_snat_for_lb = 1; next;"; - break; - case LROUTER_NAT_LB_FLOW_SKIP_SNAT: - undnat_action = "flags.skip_snat_for_lb = 1; next;"; - break; - case LROUTER_NAT_LB_FLOW_NORMAL: - case LROUTER_NAT_LB_FLOW_MAX: - undnat_action = lrouter_use_common_zone(od) - ? "ct_dnat_in_czone;" - : "ct_dnat;"; - break; - } + struct ds dnat_action = DS_EMPTY_INITIALIZER; /* Store the match lengths, so we can reuse the ds buffer. */ size_t new_match_len = ctx->new_match->length; size_t undnat_match_len = ctx->undnat_match->length; + /* dnat_action: Add the LB backend IPs as a destination action of the + * lr_in_dnat NAT rule with cumulative effect because any + * backend dst IP used in the action list will redirect the + * packet to the ct_lb pipeline. + */ + if (stateless_nat) { + for (size_t i = 0; i < ctx->lb_vip->n_backends; i++) { + struct ovn_lb_backend *backend = &ctx->lb_vip->backends[i]; + bool ipv6 = !IN6_IS_ADDR_V4MAPPED(&backend->ip); + ds_put_format(&dnat_action, "%s.dst=%s;", ipv6 ? "ip6" : "ip4", + backend->ip_str); + } + } + ds_put_format(&dnat_action, "%s", ctx->new_action[type]); const char *meter = NULL; @@ -11841,20 +11840,46 @@ build_distr_lrouter_nat_flows_for_lb(struct lrouter_nat_lb_flows_ctx *ctx, if (ctx->lb_vip->n_backends || !ctx->lb_vip->empty_backend_rej) { ds_put_format(ctx->new_match, " && is_chassis_resident(%s)", - od->l3dgw_ports[0]->cr_port->json_key); + dgp->cr_port->json_key); } ovn_lflow_add_with_hint__(ctx->lflows, od, S_ROUTER_IN_DNAT, ctx->prio, - ds_cstr(ctx->new_match), ctx->new_action[type], + ds_cstr(ctx->new_match), ds_cstr(&dnat_action), NULL, meter, &ctx->lb->nlb->header_, lflow_ref); ds_truncate(ctx->new_match, new_match_len); + ds_destroy(&dnat_action); if (!ctx->lb_vip->n_backends) { return; } + struct ds undnat_action = DS_EMPTY_INITIALIZER; + struct ds snat_action = DS_EMPTY_INITIALIZER; + + switch (type) { + case LROUTER_NAT_LB_FLOW_FORCE_SNAT: + ds_put_format(&undnat_action, "flags.force_snat_for_lb = 1; next;"); + break; + case LROUTER_NAT_LB_FLOW_SKIP_SNAT: + ds_put_format(&undnat_action, "flags.skip_snat_for_lb = 1; next;"); + break; + case LROUTER_NAT_LB_FLOW_NORMAL: + case LROUTER_NAT_LB_FLOW_MAX: + ds_put_format(&undnat_action, "%s", + lrouter_use_common_zone(od) ? "ct_dnat_in_czone;" + : "ct_dnat;"); + break; + } + + /* undnat_action: Remove the ct action from the lr_out_undenat NAT rule. + */ + if (stateless_nat) { + ds_clear(&undnat_action); + ds_put_format(&undnat_action, "next;"); + } + /* We need to centralize the LB traffic to properly perform * the undnat stage. */ @@ -11873,11 +11898,41 @@ build_distr_lrouter_nat_flows_for_lb(struct lrouter_nat_lb_flows_ctx *ctx, ds_put_format(ctx->undnat_match, ") && (inport == %s || outport == %s)" " && is_chassis_resident(%s)", dgp->json_key, dgp->json_key, dgp->cr_port->json_key); + /* Use the LB protocol as matching criteria for out undnat and snat when + * creating LBs with stateless NAT. */ + if (stateless_nat) { + ds_put_format(ctx->undnat_match, " && %s", ctx->lb->proto); + } ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_UNDNAT, 120, - ds_cstr(ctx->undnat_match), undnat_action, - &ctx->lb->nlb->header_, + ds_cstr(ctx->undnat_match), + ds_cstr(&undnat_action), &ctx->lb->nlb->header_, lflow_ref); + + /* snat_action: Add a new lr_out_snat rule with the LB VIP as source IP + * action to perform the NAT stateless pipeline completely. + */ + if (stateless_nat) { + if (ctx->lb_vip->port_str) { + ds_put_format(&snat_action, "%s.src=%s; %s.src=%s; next;", + ctx->lb_vip->address_family == AF_INET6 ? + "ip6" : "ip4", + ctx->lb_vip->vip_str, ctx->lb->proto, + ctx->lb_vip->port_str); + } else { + ds_put_format(&snat_action, "%s.src=%s; next;", + ctx->lb_vip->address_family == AF_INET6 ? + "ip6" : "ip4", + ctx->lb_vip->vip_str); + } + ovn_lflow_add_with_hint(ctx->lflows, od, S_ROUTER_OUT_SNAT, 160, + ds_cstr(ctx->undnat_match), + ds_cstr(&snat_action), &ctx->lb->nlb->header_, + lflow_ref); + } + ds_truncate(ctx->undnat_match, undnat_match_len); + ds_destroy(&undnat_action); + ds_destroy(&snat_action); } static void @@ -12022,6 +12077,8 @@ build_lrouter_nat_flows_for_lb( * lflow generation for them. */ size_t index; + bool use_stateless_nat = smap_get_bool(&lb->nlb->options, + "use_stateless_nat", false); BITMAP_FOR_EACH_1 (index, bitmap_len, lb_dps->nb_lr_map) { struct ovn_datapath *od = lr_datapaths->array[index]; enum lrouter_nat_lb_flow_type type; @@ -12043,8 +12100,17 @@ build_lrouter_nat_flows_for_lb( if (!od->n_l3dgw_ports) { bitmap_set1(gw_dp_bitmap[type], index); } else { - build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, - lb_dps->lflow_ref); + /* Create stateless LB NAT rules when using multiple DGPs and + * use_stateless_nat is true. + */ + bool stateless_nat = (od->n_l3dgw_ports > 1) + ? use_stateless_nat : false; + for (size_t i = 0; i < od->n_l3dgw_ports; i++) { + struct ovn_port *dgp = od->l3dgw_ports[i]; + build_distr_lrouter_nat_flows_for_lb(&ctx, type, od, + lb_dps->lflow_ref, dgp, + stateless_nat); + } } if (lb->affinity_timeout) { diff --git a/ovn-nb.xml b/ovn-nb.xml index 2836f58f5..ad03c6214 100644 --- a/ovn-nb.xml +++ b/ovn-nb.xml @@ -2302,6 +2302,16 @@ or local anymore by the ovn-controller. This option is set to <code>false</code> by default. </column> + + <column name="options" key="use_stateless_nat" + type='{"type": "boolean"}'> + If the load balancer is configured with <code>use_stateless_nat</code> + option to <code>true</code>, the logical router that references this + load balancer will use Stateless NAT rules when the logical router + has multiple distributed gateway ports(DGP). Otherwise, the outbound + traffic may be dropped in scenarios where we have different chassis + for each DGP. This option is set to <code>false</code> by default. + </column> </group> </table> diff --git a/tests/multinode-macros.at b/tests/multinode-macros.at index 757917626..2f69433fc 100644 --- a/tests/multinode-macros.at +++ b/tests/multinode-macros.at @@ -40,6 +40,27 @@ m4_define([M_START_TCPDUMP], ] ) +# M_EXEC([fake_node], [command]) +# +# Execute 'command' in 'fakenode' +m4_define([M_EXEC], + [podman exec $1 $2]) + +# M_CHECK_EXEC([fake_node], [command], other_params...) +# +# Wrapper for AT_CHECK that executes 'command' inside 'fake_node''s'. +# 'other_params' as passed as they are to AT_CHECK. +m4_define([M_CHECK_EXEC], + [ AT_CHECK([M_EXEC([$1], [$2])], m4_shift(m4_shift(m4_shift($@)))) ] +) + +# M_FORMAT_CT([ip-addr]) +# +# Strip content from the piped input which would differ from test to test +# and limit the output to the rows containing 'ip-addr'. +# +m4_define([M_FORMAT_CT], + [[grep -F "dst=$1," | sed -e 's/id=[0-9]*/id=<cleared>/g' -e 's/state=[0-9_A-Z]*/state=<cleared>/g' | sort | uniq | sed -e 's/zone=[[0-9]]*/zone=<cleared>/' -e 's/mark=[[0-9]]*/mark=<cleared>/' ]]) OVS_START_SHELL_HELPERS @@ -76,6 +97,25 @@ multinode_nbctl () { m_as ovn-central ovn-nbctl "$@" } +check_fake_multinode_setup_by_nodes() { + check m_as ovn-central ovn-nbctl --wait=sb sync + for c in $1 + do + AT_CHECK([m_as $c ovn-appctl -t ovn-controller version], [0], [ignore]) + done +} + +cleanup_multinode_resources_by_nodes() { + m_as ovn-central rm -f /etc/ovn/ovnnb_db.db + m_as ovn-central /usr/share/ovn/scripts/ovn-ctl restart_northd + check m_as ovn-central ovn-nbctl --wait=sb sync + for c in $1 + do + m_as $c ovs-vsctl del-br br-int + m_as $c ip --all netns delete + done +} + # m_count_rows TABLE [CONDITION...] # # Prints the number of rows in TABLE (that satisfy CONDITION). diff --git a/tests/multinode.at b/tests/multinode.at index a0eb8fc67..b1beb4d97 100644 --- a/tests/multinode.at +++ b/tests/multinode.at @@ -1591,3 +1591,559 @@ AT_CHECK([cat ch1_eth2.tcpdump], [0], [dnl ]) AT_CLEANUP + +AT_SETUP([ovn multinode load-balancer with multiple DGPs and multiple chassis]) + +# Check that ovn-fake-multinode setup is up and running - requires additional nodes +check_fake_multinode_setup_by_nodes 'ovn-chassis-1 ovn-chassis-2 ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' + +# Delete the multinode NB and OVS resources before starting the test. +cleanup_multinode_resources_by_nodes 'ovn-chassis-1 ovn-chassis-2 ovn-chassis-3 ovn-chassis-4 ovn-gw-1 ovn-gw-2' + +# Network topology +# +# publicp1 (ovn-chassis-3) (20.0.0.3/24) +# | +# overlay +# | +# DGP public1 (ovn-gw-1) (20.0.0.1/24) +# | +# | +# | +# lr0 ------- sw0 --- sw0p1 (ovn-chassis-1) 10.0.0.3/24 +# | | +# | + --- sw0p2 (ovn-chassis-2) 10.0.0.4/24 +# | +# DGP public2 (ovn-gw-2) (30.0.0.1/24) +# | +# overlay +# | +# publicp2 (ovn-chassis-4) (30.0.0.3/24) + +# Delete already used ovs-ports +m_as ovn-chassis-1 ovs-vsctl del-port br-int sw0p1-p +m_as ovn-chassis-2 ovs-vsctl del-port br-int sw0p2-p +m_as ovn-chassis-1 ip link del sw0p1-p +m_as ovn-chassis-2 ip link del sw0p2-p +m_as ovn-chassis-3 ovs-vsctl del-port br-int publicp1-p +m_as ovn-chassis-4 ovs-vsctl del-port br-int publicp2-p +m_as ovn-chassis-3 ip link del publicp1-p +m_as ovn-chassis-4 ip link del publicp2-p + +# Create East-West switch for LB backends +check multinode_nbctl ls-add sw0 +check multinode_nbctl lsp-add sw0 sw0-port1 +check multinode_nbctl lsp-set-addresses sw0-port1 "50:54:00:00:00:03 10.0.0.3 1000::3" +check multinode_nbctl lsp-add sw0 sw0-port2 +check multinode_nbctl lsp-set-addresses sw0-port2 "50:54:00:00:00:04 10.0.0.4 1000::4" + +m_as ovn-chassis-1 /data/create_fake_vm.sh sw0-port1 sw0p1 50:54:00:00:00:03 1400 10.0.0.3 24 10.0.0.1 1000::3/64 1000::a +m_as ovn-chassis-2 /data/create_fake_vm.sh sw0-port2 sw0p2 50:54:00:00:00:04 1400 10.0.0.4 24 10.0.0.1 1000::4/64 1000::a + +m_wait_for_ports_up + +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 10.0.0.4 | FORMAT_PING], \ +[0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 10.0.0.3 | FORMAT_PING], \ +[0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + +# Create a logical router and attach to sw0 +check multinode_nbctl lr-add lr0 +check multinode_nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24 1000::a/64 +check multinode_nbctl lsp-add sw0 sw0-lr0 +check multinode_nbctl lsp-set-type sw0-lr0 router +check multinode_nbctl lsp-set-addresses sw0-lr0 router +check multinode_nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0 + +# create external connection for N/S traffic using multiple DGPs +check multinode_nbctl ls-add public + +# DGP public1 +check multinode_nbctl lsp-add public ln-public-1 +check multinode_nbctl lsp-set-type ln-public-1 localnet +check multinode_nbctl lsp-set-addresses ln-public-1 unknown +check multinode_nbctl lsp-set-options ln-public-1 network_name=public1 + +# DGP public2 +# create exteranl connection for N/S traffic +check multinode_nbctl lsp-add public ln-public-2 +check multinode_nbctl lsp-set-type ln-public-2 localnet +check multinode_nbctl lsp-set-addresses ln-public-2 unknown +check multinode_nbctl lsp-set-options ln-public-2 network_name=public2 + +# Attach DGP public1 to GW-1 and chassis-3 (overlay connectivity) +m_as ovn-gw-1 ovs-vsctl set open . external-ids:ovn-bridge-mappings=public1:br-ex +m_as ovn-chassis-3 ovs-vsctl set open . external-ids:ovn-bridge-mappings=public1:br-ex + +# Attach DGP public2 to GW-2 and chassis-4 (overlay connectivity) +m_as ovn-gw-2 ovs-vsctl set open . external-ids:ovn-bridge-mappings=public2:br-ex +m_as ovn-chassis-4 ovs-vsctl set open . external-ids:ovn-bridge-mappings=public2:br-ex + +# Create the external LR0 port to the DGP public1 +check multinode_nbctl lsp-add public public-port1 +check multinode_nbctl lsp-set-addresses public-port1 "40:54:00:00:00:03 20.0.0.3 2000::3" + +check multinode_nbctl lrp-add lr0 lr0-public-p1 00:00:00:00:ff:02 20.0.0.1/24 2000::a/64 +check multinode_nbctl lsp-add public public-lr0-p1 +check multinode_nbctl lsp-set-type public-lr0-p1 router +check multinode_nbctl lsp-set-addresses public-lr0-p1 router +check multinode_nbctl lsp-set-options public-lr0-p1 router-port=lr0-public-p1 +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p1 ovn-gw-1 10 + +# Create a VM on ovn-chassis-3 in the same public1 overlay +m_as ovn-chassis-3 /data/create_fake_vm.sh public-port1 publicp1 40:54:00:00:00:03 1400 20.0.0.3 24 20.0.0.1 2000::4/64 2000::a + +m_wait_for_ports_up public-port1 + +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [ping -q -c 3 -i 0.3 -w 2 20.0.0.1 | FORMAT_PING], \ +[0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + +# Create the external LR0 port to the DGP public2 +check multinode_nbctl lsp-add public public-port2 +check multinode_nbctl lsp-set-addresses public-port2 "60:54:00:00:00:03 30.0.0.3 3000::3" + +check multinode_nbctl lrp-add lr0 lr0-public-p2 00:00:00:00:ff:03 30.0.0.1/24 3000::a/64 +check multinode_nbctl lsp-add public public-lr0-p2 +check multinode_nbctl lsp-set-type public-lr0-p2 router +check multinode_nbctl lsp-set-addresses public-lr0-p2 router +check multinode_nbctl lsp-set-options public-lr0-p2 router-port=lr0-public-p2 +check multinode_nbctl lrp-set-gateway-chassis lr0-public-p2 ovn-gw-2 10 + +# Create a VM on ovn-chassis-4 in the same public2 overlay +m_as ovn-chassis-4 /data/create_fake_vm.sh public-port2 publicp2 60:54:00:00:00:03 1400 30.0.0.3 24 30.0.0.1 3000::4/64 3000::a + +m_wait_for_ports_up public-port2 + +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [ping -q -c 3 -i 0.3 -w 2 30.0.0.1 | FORMAT_PING], \ +[0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + +# Add a default route for multiple DGPs - using ECMP +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 20.0.0.3 +####check multinode_nbctl --ecmp lr-route-add lr0 0.0.0.0/0 30.0.0.3 + +# Add SNAT rules using gateway-port +check multinode_nbctl --gateway-port lr0-public-p1 lr-nat-add lr0 snat 20.0.0.1 10.0.0.0/24 +check multinode_nbctl --gateway-port lr0-public-p2 lr-nat-add lr0 snat 30.0.0.1 10.0.0.0/24 + +M_NS_CHECK_EXEC([ovn-chassis-1], [sw0p1], [ping -q -c 3 -i 0.3 -w 2 20.0.0.3 | FORMAT_PING], \ +[0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + +M_NS_CHECK_EXEC([ovn-chassis-2], [sw0p2], [ping -q -c 3 -i 0.3 -w 2 30.0.0.3 | FORMAT_PING], \ +[0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + +# create LB +check multinode_nbctl lb-add lb0 "172.16.0.100:80" "10.0.0.3:80,10.0.0.4:80" +check multinode_nbctl lr-lb-add lr0 lb0 +check multinode_nbctl ls-lb-add sw0 lb0 + +# Set use_stateless_nat to true +check multinode_nbctl set load_balancer lb0 options:use_stateless_nat=true + +# Start backend http services +M_NS_DAEMONIZE([ovn-chassis-1], [sw0p1], [$PYTHON -m http.server --bind 10.0.0.3 80 >/dev/null 2>&1], [http1.pid]) +M_NS_DAEMONIZE([ovn-chassis-2], [sw0p2], [$PYTHON -m http.server --bind 10.0.0.4 80 >/dev/null 2>&1], [http2.pid]) + +# wait for http server be ready +sleep 2 + +# Flush conntrack entries for easier output parsing of next test. +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack + +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v 172.16.0.100:80 --retry 3 --max-time 1 --local-port 59002 2> curl.out']) +M_NS_CHECK_EXEC([ovn-chassis-3], [publicp1], [sh -c 'cat curl.out | grep -i -e connect | grep -v 'Server:''], \ +[0], [dnl +* Connected to 172.16.0.100 (172.16.0.100) port 80 +* Closing connection +]) + +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v 172.16.0.100:80 --retry 3 --max-time 1 --local-port 59003 2> curl.out']) +M_NS_CHECK_EXEC([ovn-chassis-4], [publicp2], [sh -c 'cat curl.out | grep -i -e connect | grep -v 'Server:''], \ +[0], [dnl +* Connected to 172.16.0.100 (172.16.0.100) port 80 +* Closing connection +]) + +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack + +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v 172.16.0.100:80 --retry 3 --max-time 1 --local-port 59001']) +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | M_FORMAT_CT(20.0.0.3) | \ +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59001,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59001),zone=<cleared>,protoinfo=(state=<cleared>) +]) + +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v 172.16.0.100:80 --retry 3 --max-time 1 --local-port 59000']) +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | M_FORMAT_CT(30.0.0.3) | \ +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59000,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59000),zone=<cleared>,protoinfo=(state=<cleared>) +]) + +# create a big file on web servers for download +M_NS_EXEC([ovn-chassis-1], [sw0p1], [dd bs=512 count=200000 if=/dev/urandom of=download_file]) +M_NS_EXEC([ovn-chassis-2], [sw0p2], [dd bs=512 count=200000 if=/dev/urandom of=download_file]) + +# Flush conntrack entries for easier output parsing of next test. +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack + +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59004 2>curl.out']) + +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') + +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat curl.out | \ +grep -i -e connect | grep -v 'Server:'], [0], [dnl +* Connected to 172.16.0.100 (172.16.0.100) port 80 +* Closing connection +]) + +# Check if we have only one backend for the same connection - orig + dest ports +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59004,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) +]) + +# Check if gw-2 is empty to ensure that the traffic only come from/to the originator chassis via DGP public1 +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl +0 +]) + +# Check the backend IP from ct entries on gw-1 (DGP public1) +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" -c) + +if [[ $backend_check -gt 0 ]]; then +# Backend resides on ovn-chassis-1 +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ +grep tcp], [0], [dnl +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59004,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) +]) + +# Ensure that the traffic only come from ovn-chassis-1 +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +else +# Backend resides on ovn-chassis-2 +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ +grep tcp], [0], [dnl +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59004,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59004),zone=<cleared>,protoinfo=(state=<cleared>) +]) + +# Ensure that the traffic only come from ovn-chassis-2 +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +fi + +# Flush conntrack entries for easier output parsing of next test. +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack + +# Check the flows again for a new source port +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59005 2>curl.out']) + +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') + +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 ip netns exec publicp1 cat curl.out | \ +grep -i -e connect | grep -v 'Server:'], [0], [dnl +* Connected to 172.16.0.100 (172.16.0.100) port 80 +* Closing connection +]) + +# Check if we have only one backend for the same connection - orig + dest ports +OVS_WAIT_FOR_OUTPUT([echo -e $gw1_ct | M_FORMAT_CT(20.0.0.3) | \ +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59005,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) +]) + +# Check if gw-2 is empty to ensure that the traffic only come from/to the originator chassis via DGP public1 +AT_CHECK([echo -e $gw2_ct | grep "20.0.0.3" -c], [1], [dnl +0 +]) + +# Check the backend IP from ct entries on gw-1 (DGP public1) +backend_check=$(echo -e $gw1_ct | grep "10.0.0.3" | grep "dport=80" -c) + +if [[ $backend_check -gt 0 ]]; then +# Backend resides on ovn-chassis-1 +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(20.0.0.3) | \ +grep tcp], [0], [dnl +tcp,orig=(src=20.0.0.3,dst=10.0.0.3,sport=59005,dport=80),reply=(src=10.0.0.3,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) +]) + +# Ensure that the traffic only come from ovn-chassis-1 +AT_CHECK([echo -e $chassis2_ct | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +AT_CHECK([echo -e $chassis2_flow | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +else +# Backend resides on ovn-chassis-2 +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(20.0.0.3) | \ +grep tcp], [0], [dnl +tcp,orig=(src=20.0.0.3,dst=10.0.0.4,sport=59005,dport=80),reply=(src=10.0.0.4,dst=20.0.0.3,sport=80,dport=59005),zone=<cleared>,protoinfo=(state=<cleared>) +]) + +# Ensure that the traffic only come from ovn-chassis-2 +AT_CHECK([echo -e $chassis1_ct | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +AT_CHECK([echo -e $chassis1_flow | grep "20.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +fi + +# Flush conntrack entries for easier output parsing of next test. +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack + +# Start a new test using the second DGP as origin (public2) +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59006 2>curl.out']) + +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') + +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat curl.out | \ +grep -i -e connect | grep -v 'Server:'], [0], [dnl +* Connected to 172.16.0.100 (172.16.0.100) port 80 +* Closing connection +]) + +# Check if we have only one backend for the same connection - orig + dest ports +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59006,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) +]) + +# Check if gw-1 is empty to ensure that the traffic only come from/to the originator chassis via DGP public2 +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl +0 +]) + +# Check the backend IP from ct entries on gw-2 (DGP public2) +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" -c) + +if [[ $backend_check -gt 0 ]]; then +# Backend resides on ovn-chassis-1 +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ +grep tcp], [0], [dnl +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59006,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) +]) + +# Ensure that the traffic only come from ovn-chassis-1 +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +else +# Backend resides on ovn-chassis-2 +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ +grep tcp], [0], [dnl +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59006,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59006),zone=<cleared>,protoinfo=(state=<cleared>) +]) + +# Ensure that the traffic only come from ovn-chassis-2 +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +fi + +# Flush conntrack entries for easier output parsing of next test. +m_as ovn-chassis-1 ovs-appctl dpctl/flush-conntrack +m_as ovn-chassis-2 ovs-appctl dpctl/flush-conntrack +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack + +# Check the flows again for a new source port using the second DGP as origin (public2) +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 --local-port 59007 2>curl.out']) + +gw1_ct=$(m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +gw2_ct=$(m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +chassis1_ct=$(m_as ovn-chassis-1 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +chassis2_ct=$(m_as ovn-chassis-2 ovs-appctl dpctl/dump-conntrack | sed ':a;N;$!ba;s/\n/\\n/g') +chassis1_flow=$(m_as ovn-chassis-1 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') +chassis2_flow=$(m_as ovn-chassis-2 ovs-dpctl dump-flows | sed ':a;N;$!ba;s/\n/\\n/g') + +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 ip netns exec publicp2 cat curl.out | \ +grep -i -e connect | grep -v 'Server:'], [0], [dnl +* Connected to 172.16.0.100 (172.16.0.100) port 80 +* Closing connection +]) + +# Check if we have only one backend for the same connection - orig + dest ports +OVS_WAIT_FOR_OUTPUT([echo -e $gw2_ct | M_FORMAT_CT(30.0.0.3) | \ +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59007,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) +]) + +# Check if gw-1 is empty to ensure that the traffic only come from/to the originator chassis via DGP public2 +AT_CHECK([echo -e $gw1_ct | grep "30.0.0.3" -c], [1], [dnl +0 +]) + +# Check the backend IP from ct entries on gw-1 (DGP public1) +backend_check=$(echo -e $gw2_ct | grep "10.0.0.3" | grep "dport=80" -c) + +if [[ $backend_check -gt 0 ]]; then +# Backend resides on ovn-chassis-1 +AT_CHECK([echo -e $chassis1_ct | M_FORMAT_CT(30.0.0.3) | \ +grep tcp], [0], [dnl +tcp,orig=(src=30.0.0.3,dst=10.0.0.3,sport=59007,dport=80),reply=(src=10.0.0.3,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) +]) + +# Ensure that the traffic only come from ovn-chassis-1 +AT_CHECK([echo -e $chassis2_ct | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +AT_CHECK([echo -e $chassis2_flow | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +else +# Backend resides on ovn-chassis-2 +AT_CHECK([echo -e $chassis2_ct | M_FORMAT_CT(30.0.0.3) | \ +grep tcp], [0], [dnl +tcp,orig=(src=30.0.0.3,dst=10.0.0.4,sport=59007,dport=80),reply=(src=10.0.0.4,dst=30.0.0.3,sport=80,dport=59007),zone=<cleared>,protoinfo=(state=<cleared>) +]) + +# Ensure that the traffic only come from ovn-chassis-2 +AT_CHECK([echo -e $chassis1_ct | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +AT_CHECK([echo -e $chassis1_flow | grep "30.0.0.3" | grep "dport=80" -c], [1], [dnl +0 +]) +fi + +# Check multiple requests coming from DGP's public1 and public2 + +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep -v 'Server:'], [0], [dnl +* Connected to 172.16.0.100 (172.16.0.100) port 80 +200 OK +* Closing connection +]) + +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep -v 'Server:'], [0], [dnl +* Connected to 172.16.0.100 (172.16.0.100) port 80 +200 OK +* Closing connection +]) + +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-4 cat curl.out | \ +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep -v 'Server:'], [0], [dnl +* Connected to 172.16.0.100 (172.16.0.100) port 80 +200 OK +* Closing connection +]) + +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O 172.16.0.100:80/download_file --retry 3 --max-time 1 2>curl.out']) +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep -v 'Server:'], [0], [dnl +* Connected to 172.16.0.100 (172.16.0.100) port 80 +200 OK +* Closing connection +]) + +# Remove the LB and change the VIP port - different from the backend ports +check multinode_nbctl lb-del lb0 + +# create LB again +check multinode_nbctl lb-add lb0 "172.16.0.100:9000" "10.0.0.3:80,10.0.0.4:80" +check multinode_nbctl lr-lb-add lr0 lb0 +check multinode_nbctl ls-lb-add sw0 lb0 + +# Set use_stateless_nat to true +check multinode_nbctl set load_balancer lb0 options:use_stateless_nat=true + +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack + +# Check end-to-end request using a new port for VIP +M_NS_EXEC([ovn-chassis-3], [publicp1], [sh -c 'curl -v -O 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port 59008 2>curl.out']) +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-1 ovs-appctl dpctl/dump-conntrack | M_FORMAT_CT(20.0.0.3) | \ +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) +tcp,orig=(src=20.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=20.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) +]) + +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep -v 'Server:'], [0], [dnl +* Connected to 172.16.0.100 (172.16.0.100) port 9000 +200 OK +* Closing connection +]) + +m_as ovn-gw-1 ovs-appctl dpctl/flush-conntrack +m_as ovn-gw-2 ovs-appctl dpctl/flush-conntrack + +# Check end-to-end request using a new port for VIP +M_NS_EXEC([ovn-chassis-4], [publicp2], [sh -c 'curl -v -O 172.16.0.100:9000/download_file --retry 3 --max-time 1 --local-port 59008 2>curl.out']) +OVS_WAIT_FOR_OUTPUT([m_as ovn-gw-2 ovs-appctl dpctl/dump-conntrack | M_FORMAT_CT(30.0.0.3) | \ +grep tcp | sed -E -e 's/10.0.0.3|10.0.0.4/<cleared>/g' | sort], [0], [dnl +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=80),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,protoinfo=(state=<cleared>) +tcp,orig=(src=30.0.0.3,dst=<cleared>,sport=59008,dport=9000),reply=(src=<cleared>,dst=30.0.0.3,sport=80,dport=59008),zone=<cleared>,mark=<cleared>,protoinfo=(state=<cleared>) +]) + +OVS_WAIT_FOR_OUTPUT([m_as ovn-chassis-3 cat curl.out | \ +sed 's/\(.*\)200 OK/200 OK\n/' | grep -i -e connect -e "200 OK" | grep -v 'Server:'], [0], [dnl +* Connected to 172.16.0.100 (172.16.0.100) port 9000 +200 OK +* Closing connection +]) + +AT_CLEANUP diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at index dcc3dbbc3..9e7a2f225 100644 --- a/tests/ovn-northd.at +++ b/tests/ovn-northd.at @@ -13864,3 +13864,323 @@ check_no_redirect AT_CLEANUP ]) + +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + NAT Stateless)]) +ovn_start + +check ovn-nbctl ls-add public +check ovn-nbctl lr-add lr1 + +# lr1 DGP ts1 +check ovn-nbctl ls-add ts1 +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 172.16.10.1/24 +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 + +# lr1 DGP ts2 +check ovn-nbctl ls-add ts2 +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 172.16.20.1/24 +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 + +# lr1 DGP public +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 173.16.0.1/16 +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 + +check ovn-nbctl ls-add s1 +# s1 - lr1 +check ovn-nbctl lsp-add s1 s1_lr1 +check ovn-nbctl lsp-set-type s1_lr1 router +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 172.16.0.1" +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 + +# s1 - backend vm1 +check ovn-nbctl lsp-add s1 vm1 +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 172.16.0.101" + +# s1 - backend vm2 +check ovn-nbctl lsp-add s1 vm2 +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 172.16.0.102" + +# s1 - backend vm3 +check ovn-nbctl lsp-add s1 vm3 +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 172.16.0.103" + +# Add the lr1 DGP ts1 to the public switch +check ovn-nbctl lsp-add public public_lr1_ts1 +check ovn-nbctl lsp-set-type public_lr1_ts1 router +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 nat-addresses=router + +# Add the lr1 DGP ts2 to the public switch +check ovn-nbctl lsp-add public public_lr1_ts2 +check ovn-nbctl lsp-set-type public_lr1_ts2 router +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 nat-addresses=router + +# Add the lr1 DGP public to the public switch +check ovn-nbctl lsp-add public public_lr1 +check ovn-nbctl lsp-set-type public_lr1 router +check ovn-nbctl lsp-set-addresses public_lr1 router +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public nat-addresses=router + +# Create the Load Balancer lb1 +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" "172.16.0.103,172.16.0.102,172.16.0.101" + +# Set use_stateless_nat to true +check ovn-nbctl --wait=sb set load_balancer lb1 options:use_stateless_nat=true + +# Associate load balancer to s1 +check ovn-nbctl ls-lb-add s1 lb1 +check ovn-nbctl --wait=sb sync + +ovn-sbctl dump-flows s1 > s1flows +AT_CAPTURE_FILE([s1flows]) + +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep "30.0.0.1"], [0], [dnl + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) +]) +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep "30.0.0.1"], [0], [dnl + table=??(ls_in_lb ), priority=110 , match=(ct.new && ip4.dst == 30.0.0.1), action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) +]) + +# Associate load balancer to lr1 with DGP +check ovn-nbctl lr-lb-add lr1 lb1 +check ovn-nbctl --wait=sb sync + +ovn-sbctl dump-flows lr1 > lr1flows +AT_CAPTURE_FILE([lr1flows]) + +# Check stateless NAT rules for load balancer with multiple DGP +# 1. Check if the backend IPs are in the ipX.dst action +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep "30.0.0.1"], [0], [dnl + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 30.0.0.1 && is_chassis_resident("cr-lr1-ts1")), action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 30.0.0.1 && is_chassis_resident("cr-lr1-ts2")), action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 30.0.0.1 && is_chassis_resident("cr-lr1_public")), action=(ip4.dst=172.16.0.103;ip4.dst=172.16.0.102;ip4.dst=172.16.0.101;ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) +]) + +# 2. Check if the DGP ports are in the match with action next +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], [dnl + table=??(lr_out_undnat ), priority=0 , match=(1), action=(next;) + table=??(lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) + table=??(lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) + table=??(lr_out_undnat ), priority=120 , match=(ip4 && ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), action=(next;) +]) + +# 3. Check if the VIP IP is in the ipX.src action +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl + table=??(lr_out_snat ), priority=0 , match=(1), action=(next;) + table=??(lr_out_snat ), priority=120 , match=(nd_ns), action=(next;) + table=??(lr_out_snat ), priority=160 , match=(ip4 && ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == 172.16.0.101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(ip4.src=30.0.0.1; next;) + table=??(lr_out_snat ), priority=160 , match=(ip4 && ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == 172.16.0.101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(ip4.src=30.0.0.1; next;) + table=??(lr_out_snat ), priority=160 , match=(ip4 && ((ip4.src == 172.16.0.103) || (ip4.src == 172.16.0.102) || (ip4.src == 172.16.0.101)) && (inport == "lr1_public" || outport == "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), action=(ip4.src=30.0.0.1; next;) +]) + +AT_CLEANUP +]) + +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ +AT_SETUP([Load balancer with Distributed Gateway Ports (LB + DGP + NAT Stateless) - IPv6]) +ovn_start + +check ovn-nbctl ls-add public +check ovn-nbctl lr-add lr1 + +# lr1 DGP ts1 +check ovn-nbctl ls-add ts1 +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 2001:db8:aaaa:1::1/64 +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-2 + +# lr1 DGP ts2 +check ovn-nbctl ls-add ts2 +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 2001:db8:aaaa:2::1/64 +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-3 + +# lr1 DGP public +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 2001:db8:bbbb::1/64 +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 2001:db8:aaaa:3::1/64 +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 + +check ovn-nbctl ls-add s1 +# s1 - lr1 +check ovn-nbctl lsp-add s1 s1_lr1 +check ovn-nbctl lsp-set-type s1_lr1 router +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 2001:db8:aaaa:3::1" +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 + +# s1 - backend vm1 +check ovn-nbctl lsp-add s1 vm1 +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 2001:db8:aaaa:3::101" + +# s1 - backend vm2 +check ovn-nbctl lsp-add s1 vm2 +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 2001:db8:aaaa:3::102" + +# s1 - backend vm3 +check ovn-nbctl lsp-add s1 vm3 +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 2001:db8:aaaa:3::103" + +# Add the lr1 DGP ts1 to the public switch +check ovn-nbctl lsp-add public public_lr1_ts1 +check ovn-nbctl lsp-set-type public_lr1_ts1 router +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 nat-addresses=router + +# Add the lr1 DGP ts2 to the public switch +check ovn-nbctl lsp-add public public_lr1_ts2 +check ovn-nbctl lsp-set-type public_lr1_ts2 router +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 nat-addresses=router + +# Add the lr1 DGP public to the public switch +check ovn-nbctl lsp-add public public_lr1 +check ovn-nbctl lsp-set-type public_lr1 router +check ovn-nbctl lsp-set-addresses public_lr1 router +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public nat-addresses=router + +# Create the Load Balancer lb1 +check ovn-nbctl --wait=sb lb-add lb1 "2001:db8:cccc::1" "2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101" + +# Set use_stateless_nat to true +check ovn-nbctl --wait=sb set load_balancer lb1 options:use_stateless_nat=true + +# Associate load balancer to s1 +check ovn-nbctl ls-lb-add s1 lb1 +check ovn-nbctl --wait=sb sync + +ovn-sbctl dump-flows s1 > s1flows +AT_CAPTURE_FILE([s1flows]) + +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep "2001:db8:cccc::1"], [0], [dnl + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 && ip6.dst == 2001:db8:cccc::1), action=(xxreg1 = 2001:db8:cccc::1; ct_lb_mark;) +]) +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep "2001:db8:cccc::1"], [0], [dnl + table=??(ls_in_lb ), priority=110 , match=(ct.new && ip6.dst == 2001:db8:cccc::1), action=(ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) +]) + +# Associate load balancer to lr1 with DGP +check ovn-nbctl lr-lb-add lr1 lb1 +check ovn-nbctl --wait=sb sync + +ovn-sbctl dump-flows lr1 > lr1flows +AT_CAPTURE_FILE([lr1flows]) + +# Check stateless NAT rules for load balancer with multiple DGP +# 1. Check if the backend IPs are in the ipX.dst action +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep "2001:db8:cccc::1"], [0], [dnl + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && is_chassis_resident("cr-lr1-ts1")), action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && is_chassis_resident("cr-lr1-ts2")), action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip6 && ip6.dst == 2001:db8:cccc::1 && is_chassis_resident("cr-lr1_public")), action=(ip6.dst=2001:db8:aaaa:3::103;ip6.dst=2001:db8:aaaa:3::102;ip6.dst=2001:db8:aaaa:3::101;ct_lb_mark(backends=2001:db8:aaaa:3::103,2001:db8:aaaa:3::102,2001:db8:aaaa:3::101);) +]) + +# 2. Check if the DGP ports are in the match with action next +AT_CHECK([grep "lr_out_undnat" lr1flows | ovn_strip_lflows], [0], [dnl + table=??(lr_out_undnat ), priority=0 , match=(1), action=(next;) + table=??(lr_out_undnat ), priority=120 , match=(ip6 && ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(next;) + table=??(lr_out_undnat ), priority=120 , match=(ip6 && ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(next;) + table=??(lr_out_undnat ), priority=120 , match=(ip6 && ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || outport == "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), action=(next;) +]) + +# 3. Check if the VIP IP is in the ipX.src action +AT_CHECK([grep "lr_out_snat" lr1flows | ovn_strip_lflows], [0], [dnl + table=??(lr_out_snat ), priority=0 , match=(1), action=(next;) + table=??(lr_out_snat ), priority=120 , match=(nd_ns), action=(next;) + table=??(lr_out_snat ), priority=160 , match=(ip6 && ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts1" || outport == "lr1-ts1") && is_chassis_resident("cr-lr1-ts1") && tcp), action=(ip6.src=2001:db8:cccc::1; next;) + table=??(lr_out_snat ), priority=160 , match=(ip6 && ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1-ts2" || outport == "lr1-ts2") && is_chassis_resident("cr-lr1-ts2") && tcp), action=(ip6.src=2001:db8:cccc::1; next;) + table=??(lr_out_snat ), priority=160 , match=(ip6 && ((ip6.src == 2001:db8:aaaa:3::103) || (ip6.src == 2001:db8:aaaa:3::102) || (ip6.src == 2001:db8:aaaa:3::101)) && (inport == "lr1_public" || outport == "lr1_public") && is_chassis_resident("cr-lr1_public") && tcp), action=(ip6.src=2001:db8:cccc::1; next;) +]) + +AT_CLEANUP +]) + +OVN_FOR_EACH_NORTHD_NO_HV_PARALLELIZATION([ +AT_SETUP([Load balancer with Distributed Gateway Ports (DGP)]) +ovn_start + +check ovn-nbctl ls-add public +check ovn-nbctl lr-add lr1 + +# lr1 DGP ts1 +check ovn-nbctl ls-add ts1 +check ovn-nbctl lrp-add lr1 lr1-ts1 00:00:01:02:03:04 172.16.10.1/24 +check ovn-nbctl lrp-set-gateway-chassis lr1-ts1 chassis-1 + +# lr1 DGP ts2 +check ovn-nbctl ls-add ts2 +check ovn-nbctl lrp-add lr1 lr1-ts2 00:00:01:02:03:05 172.16.20.1/24 +check ovn-nbctl lrp-set-gateway-chassis lr1-ts2 chassis-1 + +# lr1 DGP public +check ovn-nbctl lrp-add lr1 lr1_public 00:de:ad:ff:00:01 173.16.0.1/16 +check ovn-nbctl lrp-add lr1 lr1_s1 00:de:ad:fe:00:02 172.16.0.1/24 +check ovn-nbctl lrp-set-gateway-chassis lr1_public chassis-1 + +check ovn-nbctl ls-add s1 +# s1 - lr1 +check ovn-nbctl lsp-add s1 s1_lr1 +check ovn-nbctl lsp-set-type s1_lr1 router +check ovn-nbctl lsp-set-addresses s1_lr1 "00:de:ad:fe:00:02 172.16.0.1" +check ovn-nbctl lsp-set-options s1_lr1 router-port=lr1_s1 + +# s1 - backend vm1 +check ovn-nbctl lsp-add s1 vm1 +check ovn-nbctl lsp-set-addresses vm1 "00:de:ad:01:00:01 172.16.0.101" + +# s1 - backend vm2 +check ovn-nbctl lsp-add s1 vm2 +check ovn-nbctl lsp-set-addresses vm2 "00:de:ad:01:00:02 172.16.0.102" + +# s1 - backend vm3 +check ovn-nbctl lsp-add s1 vm3 +check ovn-nbctl lsp-set-addresses vm3 "00:de:ad:01:00:03 172.16.0.103" + +# Add the lr1 DGP ts1 to the public switch +check ovn-nbctl lsp-add public public_lr1_ts1 +check ovn-nbctl lsp-set-type public_lr1_ts1 router +check ovn-nbctl lsp-set-addresses public_lr1_ts1 router +check ovn-nbctl lsp-set-options public_lr1_ts1 router-port=lr1-ts1 nat-addresses=router + +# Add the lr1 DGP ts2 to the public switch +check ovn-nbctl lsp-add public public_lr1_ts2 +check ovn-nbctl lsp-set-type public_lr1_ts2 router +check ovn-nbctl lsp-set-addresses public_lr1_ts2 router +check ovn-nbctl lsp-set-options public_lr1_ts2 router-port=lr1-ts2 nat-addresses=router + +# Add the lr1 DGP public to the public switch +check ovn-nbctl lsp-add public public_lr1 +check ovn-nbctl lsp-set-type public_lr1 router +check ovn-nbctl lsp-set-addresses public_lr1 router +check ovn-nbctl lsp-set-options public_lr1 router-port=lr1_public nat-addresses=router + +# Create the Load Balancer lb1 +check ovn-nbctl --wait=sb lb-add lb1 "30.0.0.1" "172.16.0.103,172.16.0.102,172.16.0.101" + +# Associate load balancer to s1 +check ovn-nbctl ls-lb-add s1 lb1 +check ovn-nbctl --wait=sb sync + +ovn-sbctl dump-flows s1 > s1flows +AT_CAPTURE_FILE([s1flows]) + +AT_CHECK([grep "ls_in_pre_stateful" s1flows | ovn_strip_lflows | grep "30.0.0.1"], [0], [dnl + table=??(ls_in_pre_stateful ), priority=120 , match=(reg0[[2]] == 1 && ip4.dst == 30.0.0.1), action=(reg1 = 30.0.0.1; ct_lb_mark;) +]) +AT_CHECK([grep "ls_in_lb" s1flows | ovn_strip_lflows | grep "30.0.0.1"], [0], [dnl + table=??(ls_in_lb ), priority=110 , match=(ct.new && ip4.dst == 30.0.0.1), action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) +]) + +# Associate load balancer to lr1 with DGP +check ovn-nbctl lr-lb-add lr1 lb1 +check ovn-nbctl --wait=sb sync + +ovn-sbctl dump-flows lr1 > lr1flows +AT_CAPTURE_FILE([lr1flows]) + +AT_CHECK([grep "lr_in_dnat" lr1flows | ovn_strip_lflows | grep "30.0.0.1"], [0], [dnl + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 30.0.0.1 && is_chassis_resident("cr-lr1-ts1")), action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 30.0.0.1 && is_chassis_resident("cr-lr1-ts2")), action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) + table=??(lr_in_dnat ), priority=110 , match=(ct.new && !ct.rel && ip4 && ip4.dst == 30.0.0.1 && is_chassis_resident("cr-lr1_public")), action=(ct_lb_mark(backends=172.16.0.103,172.16.0.102,172.16.0.101);) +]) + +AT_CLEANUP +])
This commit fixes the build_distr_lrouter_nat_flows_for_lb function to include a DNAT flow entry for each DGP in use. Since we have added support to create multiple gateway ports per logical router, it's necessary to include in the LR NAT rules pipeline a specific entry for each attached DGP. Otherwise, the inbound traffic will only be redirected when the incoming LRP matches the chassis_resident field. Additionally, this patch includes the ability to use load-balancer with DGPs attached to multiple chassis. We can have each of the DGPs associated with a different chassis, and in this case the DNAT rules added by default will not be enough to guarantee outgoing traffic. To solve the multiple chassis for DGPs problem, this patch include a new config options to be configured in the load-balancer. If the use_stateless_nat is set to true, the logical router that references this load-balancer will use Stateless NAT rules when the logical router has multiple DGPs. After applying this patch and setting the use_stateless_nat option, the inbound and/or outbound traffic can pass through any chassis where the DGP resides without having problems with CT state. Reported-at: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2054322 Fixes: 15348b7b806f ("ovn-northd: Multiple distributed gateway port support.") Signed-off-by: Roberto Bartzen Acosta <roberto.acosta@luizalabs.com> --- northd/en-lr-stateful.c | 12 - northd/northd.c | 116 ++++++-- ovn-nb.xml | 10 + tests/multinode-macros.at | 40 +++ tests/multinode.at | 556 ++++++++++++++++++++++++++++++++++++++ tests/ovn-northd.at | 320 ++++++++++++++++++++++ 6 files changed, 1017 insertions(+), 37 deletions(-)