From patchwork Thu Oct 26 18:18:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Numan Siddique X-Patchwork-Id: 1855919 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SGYwH3rq5z23jV for ; Fri, 27 Oct 2023 05:19:11 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id A3FC142F3E; Thu, 26 Oct 2023 18:19:06 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org A3FC142F3E X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5MWQXPSKK_jJ; Thu, 26 Oct 2023 18:19:04 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 06A1842F1A; Thu, 26 Oct 2023 18:19:03 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 06A1842F1A Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id BFA40C0032; Thu, 26 Oct 2023 18:19:02 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5E3E1C0032 for ; Thu, 26 Oct 2023 18:19:01 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id A7FE5424D5 for ; Thu, 26 Oct 2023 18:18:33 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org A7FE5424D5 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LVSeT-SQfzbC for ; Thu, 26 Oct 2023 18:18:31 +0000 (UTC) Received: from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net [IPv6:2001:4b98:dc4:8::225]) by smtp2.osuosl.org (Postfix) with ESMTPS id B4C72424C3 for ; Thu, 26 Oct 2023 18:18:30 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org B4C72424C3 Received: by mail.gandi.net (Postfix) with ESMTPSA id 7B3CE1C0003; Thu, 26 Oct 2023 18:18:28 +0000 (UTC) From: numans@ovn.org To: dev@openvswitch.org Date: Thu, 26 Oct 2023 14:18:16 -0400 Message-ID: <20231026181816.3367311-1-numans@ovn.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231026181224.3366779-1-numans@ovn.org> References: <20231026181224.3366779-1-numans@ovn.org> MIME-Version: 1.0 X-GND-Sasl: numans@ovn.org Subject: [ovs-dev] [PATCH ovn v2 14/18] northd: Add lr_lb_nat_data handler for lflow engine node. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Numan Siddique Signed-off-by: Numan Siddique --- northd/en-lflow.c | 30 +++ northd/en-lflow.h | 1 + northd/en-lr-lb-nat-data.c | 41 ++++- northd/en-lr-lb-nat-data.h | 26 +++ northd/inc-proc-northd.c | 5 +- northd/northd.c | 365 +++++++++++++++++++++---------------- northd/northd.h | 10 + tests/ovn-northd.at | 48 +++-- 8 files changed, 335 insertions(+), 191 deletions(-) diff --git a/northd/en-lflow.c b/northd/en-lflow.c index 038aa90295..11f92222e9 100644 --- a/northd/en-lflow.c +++ b/northd/en-lflow.c @@ -165,6 +165,36 @@ lflow_port_group_handler(struct engine_node *node, void *data OVS_UNUSED) return true; } +bool +lflow_lr_lb_nat_data_handler(struct engine_node *node, void *data) +{ + struct ed_type_lr_lb_nat_data *lr_lb_nat_data = + engine_get_input_data("lr_lb_nat_data", node); + + if (!lr_lb_nat_data->tracked + || lr_lb_nat_data->tracked_data.vip_nats_changed + || !hmapx_is_empty(&lr_lb_nat_data->tracked_data.deleted)) { + return false; + } + + const struct engine_context *eng_ctx = engine_get_context(); + struct lflow_data *lflow_data = data; + + struct lflow_input lflow_input; + lflow_get_input_data(node, &lflow_input); + + if (!lflow_handle_lr_lb_nat_data_changes(eng_ctx->ovnsb_idl_txn, + &lr_lb_nat_data->tracked_data, + &lflow_input, + lflow_data->lflow_table)) { + return false; + } + + engine_set_node_state(node, EN_UPDATED); + + return true; +} + void *en_lflow_init(struct engine_node *node OVS_UNUSED, struct engine_arg *arg OVS_UNUSED) { diff --git a/northd/en-lflow.h b/northd/en-lflow.h index f7325c56b1..adbd8767c9 100644 --- a/northd/en-lflow.h +++ b/northd/en-lflow.h @@ -20,5 +20,6 @@ void *en_lflow_init(struct engine_node *node, struct engine_arg *arg); void en_lflow_cleanup(void *data); bool lflow_northd_handler(struct engine_node *, void *data); bool lflow_port_group_handler(struct engine_node *, void *data); +bool lflow_lr_lb_nat_data_handler(struct engine_node *, void *data); #endif /* EN_LFLOW_H */ diff --git a/northd/en-lr-lb-nat-data.c b/northd/en-lr-lb-nat-data.c index d816d2321d..f33ed5040d 100644 --- a/northd/en-lr-lb-nat-data.c +++ b/northd/en-lr-lb-nat-data.c @@ -39,6 +39,7 @@ #include "lib/ovn-sb-idl.h" #include "lib/ovn-util.h" #include "lib/stopwatch-names.h" +#include "lflow-mgr.h" #include "northd.h" VLOG_DEFINE_THIS_MODULE(en_lr_lb_nat_data); @@ -81,7 +82,7 @@ static void remove_lrouter_lb_reachable_ips(struct lr_lb_nat_data_record *, enum lb_neighbor_responder_mode, const struct sset *lb_ips_v4, const struct sset *lb_ips_v6); -static void lr_lb_nat_data_build_vip_nats(struct lr_lb_nat_data_record *); +static bool lr_lb_nat_data_build_vip_nats(struct lr_lb_nat_data_record *); /* 'lr_lb_nat_data' engine node manages the NB logical router LB data. */ @@ -119,6 +120,7 @@ en_lr_lb_nat_data_clear_tracked_data(void *data_) } hmapx_clear(&data->tracked_data.crupdated); + data->tracked_data.vip_nats_changed = false; data->tracked = false; } @@ -186,7 +188,6 @@ lr_lb_nat_data_lb_data_handler(struct engine_node *node, void *data_) const struct lr_nat_record *lrnat_rec = lr_nat_table_find_by_index( input_data.lr_nats, od->index); ovs_assert(lrnat_rec); - lr_lbnat_rec = lr_lb_nat_data_record_create(&data->lr_lbnats, lrnat_rec, input_data.lb_datapaths_map, @@ -194,6 +195,10 @@ lr_lb_nat_data_lb_data_handler(struct engine_node *node, void *data_) /* Add the lr_lbnat_rec rec to the tracking data. */ hmapx_add(&data->tracked_data.crupdated, lr_lbnat_rec); + + if (!sset_is_empty(&lr_lbnat_rec->vip_nats)) { + data->tracked_data.vip_nats_changed = true; + } continue; } @@ -302,7 +307,9 @@ lr_lb_nat_data_lb_data_handler(struct engine_node *node, void *data_) * vip nats and re-evaluate 'has_lb_vip'. */ HMAPX_FOR_EACH (hmapx_node, &data->tracked_data.crupdated) { lr_lbnat_rec = hmapx_node->data; - lr_lb_nat_data_build_vip_nats(lr_lbnat_rec); + if (lr_lb_nat_data_build_vip_nats(lr_lbnat_rec)) { + data->tracked_data.vip_nats_changed = true; + } lr_lbnat_rec->has_lb_vip = od_has_lb_vip(lr_lbnat_rec->od); } @@ -341,8 +348,13 @@ lr_lb_nat_data_lr_nat_handler(struct engine_node *node, void *data_) lrnat_rec, input_data.lb_datapaths_map, input_data.lbgrp_datapaths_map); + if (!sset_is_empty(&lr_lbnat_rec->vip_nats)) { + data->tracked_data.vip_nats_changed = true; + } } else { - lr_lb_nat_data_build_vip_nats(lr_lbnat_rec); + if (lr_lb_nat_data_build_vip_nats(lr_lbnat_rec)) { + data->tracked_data.vip_nats_changed = true; + } } /* Add the lr_lbnat_rec rec to the tracking data. */ @@ -442,6 +454,7 @@ lr_lb_nat_data_record_create(struct lr_lb_nat_data_table *table, lr_lbnat_rec->od = lrnat_rec->od; lr_lb_nat_data_record_init(lr_lbnat_rec, lb_datapaths_map, lbgrp_datapaths_map); + lr_lbnat_rec->lflow_ref = lflow_ref_alloc(lr_lbnat_rec->od->nbr->name); hmap_insert(&table->entries, &lr_lbnat_rec->key_node, uuid_hash(&lr_lbnat_rec->od->nbr->header_.uuid)); @@ -456,6 +469,7 @@ lr_lb_nat_data_record_destroy(struct lr_lb_nat_data_record *lr_lbnat_rec) ovn_lb_ip_set_destroy(lr_lbnat_rec->lb_ips); lr_lbnat_rec->lb_ips = NULL; sset_destroy(&lr_lbnat_rec->vip_nats); + lflow_ref_destroy(lr_lbnat_rec->lflow_ref); free(lr_lbnat_rec); } @@ -522,7 +536,7 @@ lr_lb_nat_data_record_init(struct lr_lb_nat_data_record *lr_lbnat_rec, sset_init(&lr_lbnat_rec->vip_nats); - if (!nbr->n_nat) { + if (nbr->n_nat) { lr_lb_nat_data_build_vip_nats(lr_lbnat_rec); } @@ -636,10 +650,13 @@ remove_lrouter_lb_reachable_ips(struct lr_lb_nat_data_record *lr_lbnat_rec, } } -static void +/* Builds the vip nats and returns true if the lr_lbnat_rec->vip_nats + * changed, false otherwise. */ +static bool lr_lb_nat_data_build_vip_nats(struct lr_lb_nat_data_record *lr_lbnat_rec) { - sset_clear(&lr_lbnat_rec->vip_nats); + struct sset old_vip_nats = SSET_INITIALIZER(&old_vip_nats); + sset_swap(&lr_lbnat_rec->vip_nats, &old_vip_nats); const char *external_ip; SSET_FOR_EACH (external_ip, &lr_lbnat_rec->lrnat_rec->external_ips) { bool is_vip_nat = false; @@ -655,4 +672,14 @@ lr_lb_nat_data_build_vip_nats(struct lr_lb_nat_data_record *lr_lbnat_rec) sset_add(&lr_lbnat_rec->vip_nats, external_ip); } } + + bool vip_nats_changed = + sset_count(&lr_lbnat_rec->vip_nats) != sset_count(&old_vip_nats); + if (!vip_nats_changed) { + vip_nats_changed = !sset_equals(&lr_lbnat_rec->vip_nats, + &old_vip_nats); + } + sset_destroy(&old_vip_nats); + + return vip_nats_changed; } diff --git a/northd/en-lr-lb-nat-data.h b/northd/en-lr-lb-nat-data.h index ac21d28a57..f1ed0cd2fa 100644 --- a/northd/en-lr-lb-nat-data.h +++ b/northd/en-lr-lb-nat-data.h @@ -32,6 +32,7 @@ struct ovn_datapath; struct lr_nat_record; +struct lflow_ref; struct lr_lb_nat_data_record { struct hmap_node key_node; /* Index on 'nbr->header_.uuid'. */ @@ -46,6 +47,27 @@ struct lr_lb_nat_data_record { /* sset of vips which are also part of lr nats. */ struct sset vip_nats; + + /* 'lflow_ref' is used to reference logical flows generated for + * this lr_lb_nat_data record. + * + * This data is initialized and destroyed by the en_lr_lb_nat_data node, + * but populated and used only by the en_lflow node. Ideally this data + * should be maintained as part of en_lflow's data. However, it would + * be less efficient and more complex: + * + * 1. It would require an extra search (using the index) to find the + * lflows. + * + * 2. Building the index needs to be thread-safe, using either a global + * lock which is obviously less efficient, or hash-based lock array which + * is more complex. + * + * Adding the lflow_ref here is more straightforward. The drawback is that + * we need to keep in mind that this data belongs to en_lflow node, so + * never access it from any other nodes. + */ + struct lflow_ref *lflow_ref; }; struct lr_lb_nat_data_table { @@ -68,6 +90,10 @@ struct lr_lb_nat_data_tracked_data { /* Deleted logical router with LB data. */ struct hmapx deleted; /* Stores 'struct lr_lb_nat_data_record'. */ + + /* Indicates if any router's NATs changed which were also LB vips + * or vice versa. */ + bool vip_nats_changed; }; struct ed_type_lr_lb_nat_data { diff --git a/northd/inc-proc-northd.c b/northd/inc-proc-northd.c index 3cedd502cb..955b5e2ed1 100644 --- a/northd/inc-proc-northd.c +++ b/northd/inc-proc-northd.c @@ -236,10 +236,11 @@ void inc_proc_northd_init(struct ovsdb_idl_loop *nb, engine_add_input(&en_lflow, &en_sb_multicast_group, NULL); engine_add_input(&en_lflow, &en_sb_igmp_group, NULL); engine_add_input(&en_lflow, &en_sb_logical_dp_group, NULL); + engine_add_input(&en_lflow, &en_ls_lbacls, NULL); engine_add_input(&en_lflow, &en_northd, lflow_northd_handler); engine_add_input(&en_lflow, &en_port_group, lflow_port_group_handler); - engine_add_input(&en_lflow, &en_lr_lb_nat_data, NULL); - engine_add_input(&en_lflow, &en_ls_lbacls, NULL); + engine_add_input(&en_lflow, &en_lr_lb_nat_data, + lflow_lr_lb_nat_data_handler); engine_add_input(&en_sync_to_sb_addr_set, &en_nb_address_set, sync_to_sb_addr_set_nb_address_set_handler); diff --git a/northd/northd.c b/northd/northd.c index ec51dca211..fbe4f3515f 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -1227,7 +1227,7 @@ ovn_port_create(struct hmap *ports, const char *key, op->lflow_ref = lflow_ref_alloc(key); op->lbnat_lflow_ref = lflow_ref_alloc(key); - + op->routable_lflow_ref = lflow_ref_alloc(key); return op; } @@ -1268,6 +1268,7 @@ ovn_port_destroy_orphan(struct ovn_port *port) free(port->key); lflow_ref_destroy(port->lflow_ref); lflow_ref_destroy(port->lbnat_lflow_ref); + lflow_ref_destroy(port->routable_lflow_ref); free(port); } @@ -12819,63 +12820,6 @@ build_ip_routing_flows_for_lrp( } } -/* Logical router ingress table IP_ROUTING : IP Routing. - * - * For the LSP 'op' of type router, if there are logical router ports other - * than the LSP's peer connected to the logical switch, then for routable - * addresses (such as NAT IPs, LB VIPs, etc.) on each of the connected router - * ports, add routes to the LSP's peer router. - */ -static void -build_ip_routing_flows_for_router_type_lsp( - struct ovn_port *op, const struct lr_lb_nat_data_table *lr_lbnats, - const struct hmap *lr_ports, struct lflow_table *lflows, - struct lflow_ref *lflow_ref) -{ - ovs_assert(op->nbsp); - if (!lsp_is_router(op->nbsp)) { - return; - } - - struct ovn_port *peer = ovn_port_get_peer(lr_ports, op); - if (!peer || !peer->nbrp || !peer->lrp_networks.n_ipv4_addrs - || !op->od->n_router_ports) { - return; - } - - for (int i = 0; i < op->od->n_router_ports; i++) { - struct ovn_port *router_port = ovn_port_get_peer( - lr_ports, op->od->router_ports[i]); - if (!router_port || !router_port->nbrp || router_port == peer) { - continue; - } - - const struct lr_lb_nat_data_record *lr_lbnat_rec = - lr_lb_nat_data_table_find_by_index(lr_lbnats, - router_port->od->index); - - if (router_port->nbrp->ha_chassis_group || - router_port->nbrp->n_gateway_chassis) { - struct ovn_port_routable_addresses ra = - get_op_routable_addresses(router_port, lr_lbnat_rec); - for (size_t j = 0; j < ra.n_addrs; j++) { - struct lport_addresses *laddrs = &ra.laddrs[j]; - for (size_t k = 0; k < laddrs->n_ipv4_addrs; k++) { - add_route(lflows, peer->od, peer, - peer->lrp_networks.ipv4_addrs[0].addr_s, - laddrs->ipv4_addrs[k].network_s, - laddrs->ipv4_addrs[k].plen, NULL, false, 0, - &peer->nbrp->header_, false, - ROUTE_PRIO_OFFSET_CONNECTED, - lflow_ref); - } - } - destroy_routable_addresses(&ra); - } - } - -} - static void build_static_route_flows_for_lrouter( struct ovn_datapath *od, const struct chassis_features *features, @@ -13117,42 +13061,6 @@ build_arp_resolve_flows_for_lrouter( lflow_ref); } -static void -routable_addresses_to_lflows(struct lflow_table *lflows, - struct ovn_port *router_port, - struct ovn_port *peer, - const struct lr_lb_nat_data_record *lr_lbnat_rec, - struct ds *match, struct ds *actions, - struct lflow_ref *lflow_ref) -{ - struct ovn_port_routable_addresses ra = - get_op_routable_addresses(router_port, lr_lbnat_rec); - if (!ra.n_addrs) { - return; - } - - for (size_t i = 0; i < ra.n_addrs; i++) { - ds_clear(match); - ds_put_format(match, "outport == %s && "REG_NEXT_HOP_IPV4" == {", - peer->json_key); - bool first = true; - for (size_t j = 0; j < ra.laddrs[i].n_ipv4_addrs; j++) { - if (!first) { - ds_put_cstr(match, ", "); - } - ds_put_cstr(match, ra.laddrs[i].ipv4_addrs[j].addr_s); - first = false; - } - ds_put_cstr(match, "}"); - - ds_clear(actions); - ds_put_format(actions, "eth.dst = %s; next;", ra.laddrs[i].ea_s); - ovn_lflow_add(lflows, peer->od, S_ROUTER_IN_ARP_RESOLVE, 100, - ds_cstr(match), ds_cstr(actions), lflow_ref); - } - destroy_routable_addresses(&ra); -} - /* Local router ingress table ARP_RESOLVE: ARP Resolution. * * Any unicast packet that reaches this table is an IP packet whose @@ -13395,52 +13303,6 @@ build_arp_resolve_flows_for_lsp( } } -static void -build_arp_resolve_flows_for_lsp_routable_addresses( - struct ovn_port *op, struct lflow_table *lflows, - const struct hmap *lr_ports, - const struct lr_lb_nat_data_table *lr_lbnats, - struct ds *match, struct ds *actions, - struct lflow_ref *lflow_ref) -{ - if (!lsp_is_router(op->nbsp)) { - return; - } - - struct ovn_port *peer = ovn_port_get_peer(lr_ports, op); - if (!peer || !peer->nbrp) { - return; - } - - if (peer->od->nbr && - smap_get_bool(&peer->od->nbr->options, - "dynamic_neigh_routers", false)) { - return; - } - - for (size_t i = 0; i < op->od->n_router_ports; i++) { - struct ovn_port *router_port = - ovn_port_get_peer(lr_ports, op->od->router_ports[i]); - if (!router_port || !router_port->nbrp) { - continue; - } - - /* Skip the router port under consideration. */ - if (router_port == peer) { - continue; - } - - if (smap_get(&peer->od->nbr->options, "chassis") || peer->cr_port) { - const struct lr_lb_nat_data_record *lr_lbnat_rec; - lr_lbnat_rec = lr_lb_nat_data_table_find_by_index(lr_lbnats, - router_port->od->index); - routable_addresses_to_lflows(lflows, router_port, peer, - lr_lbnat_rec, match, actions, - lflow_ref); - } - } -} - static void build_icmperr_pkt_big_flows(struct ovn_port *op, int mtu, struct lflow_table *lflows, @@ -15657,8 +15519,6 @@ static void build_lsp_lflows_for_lbnats(struct ovn_port *lsp, struct ovn_port *lrp_peer, const struct lr_lb_nat_data_record *lr_lbnat_rec, - const struct lr_lb_nat_data_table *lr_lbnats, - const struct hmap *lr_ports, struct lflow_table *lflows, struct ds *match, struct ds *actions, @@ -15668,19 +15528,101 @@ build_lsp_lflows_for_lbnats(struct ovn_port *lsp, build_lswitch_rport_arp_req_flows_for_lbnats( lrp_peer, lr_lbnat_rec, lsp->od, lsp, lflows, &lsp->nbsp->header_, lflow_ref); - build_ip_routing_flows_for_router_type_lsp(lsp, lr_lbnats, - lr_ports, lflows, - lflow_ref); - build_arp_resolve_flows_for_lsp_routable_addresses( - lsp, lflows, lr_ports, lr_lbnats, match, actions, lflow_ref); build_lswitch_ip_unicast_lookup_for_nats(lsp, lr_lbnat_rec, lflows, match, actions, lflow_ref); } +/* Logical router ingress table IP_ROUTING : IP Routing. + * + * For the LRP 'lrp's peer's logical switch, if there are logical router + * ports ('peer_lrp's) other than the 'lrp', then for routable addresses + * (such as NAT IPs, LB VIPs, etc.) of the 'lrp' add routes to the + * peer_lrp's datapath. + */ +static void +build_routable_flows_for_router_port( + struct ovn_port *lrp, const struct lr_lb_nat_data_record *lr_lbnat_rec, + struct lflow_table *lflows, + struct ds *match, + struct ds *actions, + struct lflow_ref *lflow_ref) +{ + ovs_assert(lrp->nbrp && lrp->od == lr_lbnat_rec->od); + + struct ovn_port *lsp_peer = lrp->peer; + if (!lsp_peer || !lsp_peer->nbsp) { + return; + } + + struct ovn_datapath *peer_ls = lsp_peer->od; + ovs_assert(peer_ls->nbs); + + struct ovn_port_routable_addresses ra = + get_op_routable_addresses(lrp, lr_lbnat_rec); + + struct ovn_port *router_port; + + for (size_t i = 0; i < peer_ls->n_router_ports; i++) { + router_port = peer_ls->router_ports[i]->peer; + + if (router_port == lrp) { + continue; + } + + if (lrp->nbrp->ha_chassis_group || + lrp->nbrp->n_gateway_chassis) { + for (size_t j = 0; j < ra.n_addrs; j++) { + struct lport_addresses *laddrs = &ra.laddrs[j]; + for (size_t k = 0; k < laddrs->n_ipv4_addrs; k++) { + add_route(lflows, router_port->od, router_port, + router_port->lrp_networks.ipv4_addrs[0].addr_s, + laddrs->ipv4_addrs[k].network_s, + laddrs->ipv4_addrs[k].plen, NULL, false, 0, + &router_port->nbrp->header_, false, + ROUTE_PRIO_OFFSET_CONNECTED, + lflow_ref); + } + } + } + + bool dynamic_neigh_router = + smap_get_bool(&router_port->od->nbr->options, + "dynamic_neigh_routers", false); + + if (!dynamic_neigh_router && + (router_port->od->is_gw_router || router_port->cr_port)) { + + for (size_t k = 0; k < ra.n_addrs; k++) { + ds_clear(match); + ds_put_format(match, "outport == %s && " + REG_NEXT_HOP_IPV4" == {", + router_port->json_key); + bool first = true; + for (size_t j = 0; j < ra.laddrs[k].n_ipv4_addrs; j++) { + if (!first) { + ds_put_cstr(match, ", "); + } + ds_put_cstr(match, ra.laddrs[k].ipv4_addrs[j].addr_s); + first = false; + } + ds_put_cstr(match, "}"); + + ds_clear(actions); + ds_put_format(actions, "eth.dst = %s; next;", + ra.laddrs[k].ea_s); + ovn_lflow_add(lflows, router_port->od, S_ROUTER_IN_ARP_RESOLVE, + 100, ds_cstr(match), ds_cstr(actions), + lflow_ref); + } + } + } + + destroy_routable_addresses(&ra); +} + static void build_lbnat_lflows_iterate_by_lsp(struct ovn_port *op, const struct lr_lb_nat_data_table *lr_lbnats, - const struct hmap *lr_ports, struct ds *match, struct ds *actions, struct lflow_table *lflows, @@ -15698,8 +15640,8 @@ build_lbnat_lflows_iterate_by_lsp(struct ovn_port *op, ovs_assert(lr_lbnat_rec); build_lsp_lflows_for_lbnats(op, op->peer, lr_lbnat_rec, - lr_lbnats, lr_ports, lflows, - match, actions, lflow_ref); + lflows,match, actions, + lflow_ref); } static void @@ -15749,7 +15691,8 @@ build_lbnat_lflows_iterate_by_lrp(struct ovn_port *op, struct ds *match, struct ds *actions, struct lflow_table *lflows, - struct lflow_ref *lflow_ref) + struct lflow_ref *lflow_ref, + struct lflow_ref *routable_lflow_ref) { ovs_assert(op->nbrp); @@ -15760,6 +15703,9 @@ build_lbnat_lflows_iterate_by_lrp(struct ovn_port *op, build_lrp_lflows_for_lbnats(op, lr_lbnat_rec, meter_groups, match, actions, lflows, lflow_ref); + + build_routable_flows_for_router_port(op, lr_lbnat_rec, lflows, match, + actions, routable_lflow_ref); } static void @@ -15774,15 +15720,15 @@ build_lr_lbnat_data_flows(const struct lr_lb_nat_data_record *lr_lbnat_rec, { build_lrouter_nat_defrag_and_lb(lr_lbnat_rec, lflows, ls_ports, lr_ports, match, actions, meter_groups, features, - NULL); + lr_lbnat_rec->lflow_ref); build_lr_gateway_redirect_flows_for_nats(lr_lbnat_rec->od, lr_lbnat_rec->lrnat_rec, lflows, match, actions, - NULL); + lr_lbnat_rec->lflow_ref); build_lrouter_arp_nd_for_datapath(lr_lbnat_rec->od, lr_lbnat_rec->lrnat_rec, lflows, meter_groups, - NULL); + lr_lbnat_rec->lflow_ref); } static void @@ -16014,7 +15960,6 @@ build_lflows_thread(void *arg) &lsi->actions, lsi->lflows); build_lbnat_lflows_iterate_by_lsp(op, lsi->lr_lbnats, - lsi->lr_ports, &lsi->match, &lsi->actions, lsi->lflows, @@ -16036,7 +15981,8 @@ build_lflows_thread(void *arg) &lsi->match, &lsi->actions, lsi->lflows, - op->lbnat_lflow_ref); + op->lbnat_lflow_ref, + op->routable_lflow_ref); } } for (bnum = control->id; @@ -16267,9 +16213,8 @@ build_lswitch_and_lrouter_flows(const struct ovn_datapaths *ls_datapaths, &lsi.match, &lsi.actions, lsi.lflows); - build_lbnat_lflows_iterate_by_lsp(op, lsi.lr_lbnats, lsi.lr_ports, - &lsi.match, &lsi.actions, - lsi.lflows, + build_lbnat_lflows_iterate_by_lsp(op, lsi.lr_lbnats, &lsi.match, + &lsi.actions, lsi.lflows, op->lbnat_lflow_ref); } HMAP_FOR_EACH (op, key_node, lr_ports) { @@ -16279,7 +16224,8 @@ build_lswitch_and_lrouter_flows(const struct ovn_datapaths *ls_datapaths, &lsi.match, &lsi.actions, lsi.lflows, - op->lbnat_lflow_ref); + op->lbnat_lflow_ref, + op->routable_lflow_ref); } stopwatch_stop(LFLOWS_PORTS_STOPWATCH_NAME, time_msec()); stopwatch_start(LFLOWS_LBS_STOPWATCH_NAME, time_msec()); @@ -16472,12 +16418,24 @@ void build_lflows(struct ovsdb_idl_txn *ovnsb_txn, void reset_lflow_refs_for_northd_resources(struct lflow_input *lflow_input) { + struct lr_lb_nat_data_record *lr_lbnat_rec; struct ovn_lb_datapaths *lb_dps; struct ovn_port *op; + LR_LB_NAT_DATA_TABLE_FOR_EACH (lr_lbnat_rec, lflow_input->lr_lbnats) { + lflow_ref_reset(lr_lbnat_rec->lflow_ref); + } + HMAP_FOR_EACH (op, key_node, lflow_input->ls_ports) { lflow_ref_reset(op->lflow_ref); lflow_ref_reset(op->lbnat_lflow_ref); + lflow_ref_reset(op->routable_lflow_ref); + } + + HMAP_FOR_EACH (op, key_node, lflow_input->lr_ports) { + lflow_ref_reset(op->lflow_ref); + lflow_ref_reset(op->lbnat_lflow_ref); + lflow_ref_reset(op->routable_lflow_ref); } HMAP_FOR_EACH (lb_dps, hmap_node, lflow_input->lb_datapaths_map) { @@ -16536,8 +16494,6 @@ lflow_handle_northd_port_changes(struct ovsdb_idl_txn *ovnsb_txn, ovs_assert(lr_lbnat_rec); lflow_ref_clear_lflows(op->lbnat_lflow_ref); build_lsp_lflows_for_lbnats(op, op->peer, lr_lbnat_rec, - lflow_input->lr_lbnats, - lflow_input->lr_ports, lflows, &match, &actions, op->lbnat_lflow_ref); lflow_ref_sync_lflows_to_sb(op->lbnat_lflow_ref, lflows, ovnsb_txn, @@ -16696,6 +16652,91 @@ lflow_handle_northd_lb_changes(struct ovsdb_idl_txn *ovnsb_txn, return true; } +bool +lflow_handle_lr_lb_nat_data_changes(struct ovsdb_idl_txn *ovnsb_txn, + struct lr_lb_nat_data_tracked_data *trk_data, + struct lflow_input *lflow_input, + struct lflow_table *lflows) +{ + struct lr_lb_nat_data_record *lr_lbnat_rec; + struct hmapx_node *hmapx_node; + + HMAPX_FOR_EACH (hmapx_node, &trk_data->crupdated) { + lr_lbnat_rec = hmapx_node->data; + + lflow_ref_clear_lflows(lr_lbnat_rec->lflow_ref); + + /* Generate new lflows. */ + struct ds match = DS_EMPTY_INITIALIZER; + struct ds actions = DS_EMPTY_INITIALIZER; + + build_lr_lbnat_data_flows(lr_lbnat_rec, lflows, lflow_input->ls_ports, + lflow_input->lr_ports, &match, &actions, + lflow_input->meter_groups, + lflow_input->features); + + /* Sync the new flows to SB. */ + lflow_ref_sync_lflows_to_sb(lr_lbnat_rec->lflow_ref, lflows, ovnsb_txn, + lflow_input->ls_datapaths, + lflow_input->lr_datapaths, + lflow_input->ovn_internal_version_changed, + lflow_input->sbrec_logical_flow_table, + lflow_input->sbrec_logical_dp_group_table); + + struct ovn_port *op; + HMAP_FOR_EACH (op, dp_node, &lr_lbnat_rec->od->ports) { + lflow_ref_clear_lflows(op->lbnat_lflow_ref); + lflow_ref_clear_lflows_for_all_dps(op->routable_lflow_ref, + ods_size(lflow_input->ls_datapaths), + ods_size(lflow_input->lr_datapaths)); + + build_lbnat_lflows_iterate_by_lrp(op, lflow_input->lr_lbnats, + lflow_input->meter_groups, + &match, &actions, + lflows, + op->lbnat_lflow_ref, + op->routable_lflow_ref); + + lflow_ref_sync_lflows_to_sb(op->lbnat_lflow_ref, lflows, ovnsb_txn, + lflow_input->ls_datapaths, + lflow_input->lr_datapaths, + lflow_input->ovn_internal_version_changed, + lflow_input->sbrec_logical_flow_table, + lflow_input->sbrec_logical_dp_group_table); + + lflow_ref_sync_lflows_to_sb(op->routable_lflow_ref, lflows, + ovnsb_txn, lflow_input->ls_datapaths, + lflow_input->lr_datapaths, + lflow_input->ovn_internal_version_changed, + lflow_input->sbrec_logical_flow_table, + lflow_input->sbrec_logical_dp_group_table); + + if (op->peer && op->peer->nbsp) { + lflow_ref_clear_lflows(op->peer->lbnat_lflow_ref); + + build_lbnat_lflows_iterate_by_lsp(op->peer, + lflow_input->lr_lbnats, + &match, &actions, + lflows, + op->peer->lbnat_lflow_ref); + + lflow_ref_sync_lflows_to_sb(op->peer->lbnat_lflow_ref, lflows, + ovnsb_txn, + lflow_input->ls_datapaths, + lflow_input->lr_datapaths, + lflow_input->ovn_internal_version_changed, + lflow_input->sbrec_logical_flow_table, + lflow_input->sbrec_logical_dp_group_table); + } + } + + ds_destroy(&match); + ds_destroy(&actions); + } + + return true; +} + static bool mirror_needs_update(const struct nbrec_mirror *nb_mirror, const struct sbrec_mirror *sb_mirror) diff --git a/northd/northd.h b/northd/northd.h index 849d1eeedc..606580df0b 100644 --- a/northd/northd.h +++ b/northd/northd.h @@ -677,9 +677,13 @@ struct ovn_port { * 'lbnat_lflow_ref' is used for logical switch ports of type * 'patch/router' to referenece logical flows generated fo this ovn_port * from the 'lr_lb_nat_data_table' record of the peer port's datapath. + * + * 'routable_lflow_ref' is used to reference logical flows generated for + * the routable ips of a logical router port. */ struct lflow_ref *lflow_ref; struct lflow_ref *lbnat_lflow_ref; + struct lflow_ref *routable_lflow_ref; }; void ovnnb_db_run(struct northd_input *input_data, @@ -704,6 +708,8 @@ void northd_indices_create(struct northd_data *data, struct ovsdb_idl *ovnsb_idl); struct lflow_table; +struct lr_lb_nat_data_tracked_data; + void build_lflows(struct ovsdb_idl_txn *ovnsb_txn, struct lflow_input *input_data, struct lflow_table *); @@ -717,6 +723,10 @@ bool lflow_handle_northd_lb_changes(struct ovsdb_idl_txn *ovnsb_txn, struct tracked_lbs *, struct lflow_input *, struct lflow_table *lflows); +bool lflow_handle_lr_lb_nat_data_changes(struct ovsdb_idl_txn *, + struct lr_lb_nat_data_tracked_data *, + struct lflow_input *, + struct lflow_table *lflows); bool northd_handle_sb_port_binding_changes( const struct sbrec_port_binding_table *, struct hmap *ls_ports, struct hmap *lr_ports); diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at index b276c77221..3c330a051f 100644 --- a/tests/ovn-northd.at +++ b/tests/ovn-northd.at @@ -12,6 +12,7 @@ m4_define([_DUMP_DB_TABLES], [ ovn-sbctl list meter >> $1 ovn-sbctl list meter_band >> $1 ovn-sbctl list port_group >> $1 + ovn-sbctl dump-flows > lflows_$1 ]) # CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -10586,7 +10587,7 @@ check ovn-nbctl --wait=sb lr-lb-add lr0 lb1 check_engine_stats lb_data norecompute compute check_engine_stats northd norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_lb recompute compute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -10596,7 +10597,7 @@ check ovn-nbctl --wait=sb set load_balancer lb1 vips:'"10.0.0.10:80"'='"10.0.0.1 check_engine_stats lb_data norecompute compute check_engine_stats northd norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_lb recompute compute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -10606,7 +10607,7 @@ check ovn-nbctl --wait=sb clear load_Balancer lb1 vips check_engine_stats lb_data norecompute compute check_engine_stats northd norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_lb recompute compute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -10616,7 +10617,7 @@ check ovn-nbctl --wait=sb lb-add lb1 10.0.0.10:80 10.0.0.3:80 check_engine_stats lb_data norecompute compute check_engine_stats northd norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_lb recompute compute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -10626,7 +10627,7 @@ check ovn-nbctl --wait=sb lb-add lb1 10.0.0.20:80 10.0.0.30:8080 check_engine_stats lb_data norecompute compute check_engine_stats northd norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_lb recompute compute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -10634,6 +10635,7 @@ check as northd ovn-appctl -t NORTHD_TYPE inc-engine/clear-stats check ovn-nbctl --wait=sb lr-lb-del lr0 lb1 check_engine_stats lb_data norecompute compute check_engine_stats northd recompute nocompute +check_engine_stats lr_lb_nat_data recompute nocompute check_engine_stats lflow recompute nocompute check_engine_stats sync_to_sb_lb recompute compute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -10748,7 +10750,7 @@ check ovn-nbctl add logical_router lr0 load_balancer_group $lbg1_uuid check_engine_stats lb_data norecompute compute check_engine_stats northd norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_lb recompute compute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -10758,7 +10760,7 @@ check ovn-nbctl --wait=sb set load_balancer lb1 vips:'"10.0.0.10:80"'='"10.0.0.1 check_engine_stats lb_data norecompute compute check_engine_stats northd norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_lb recompute compute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -10768,7 +10770,7 @@ check ovn-nbctl --wait=sb clear load_Balancer lb1 vips check_engine_stats lb_data norecompute compute check_engine_stats northd norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_lb recompute compute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -10778,7 +10780,7 @@ check ovn-nbctl --wait=sb lb-add lb1 10.0.0.10:80 10.0.0.3:80 check_engine_stats lb_data norecompute compute check_engine_stats northd norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_lb recompute compute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -10788,7 +10790,7 @@ check ovn-nbctl --wait=sb lb-add lb1 10.0.0.20:80 10.0.0.30:8080 check_engine_stats lb_data norecompute compute check_engine_stats northd norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_lb recompute compute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -10869,7 +10871,7 @@ check ovn-nbctl --wait=sb set logical_router lr1 load_balancer_group=$lbg1_uuid check_engine_stats lb_data norecompute compute check_engine_stats northd norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_lb recompute compute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -10897,7 +10899,7 @@ check ovn-nbctl --wait=sb lr-lb-add lr1 lb2 check_engine_stats lb_data norecompute compute check_engine_stats northd norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_lb recompute compute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -11141,7 +11143,7 @@ check as northd ovn-appctl -t NORTHD_TYPE inc-engine/clear-stats check ovn-nbctl --wait=sb lr-nat-add lr0 dnat_and_snat 172.168.0.110 10.0.0.4 check_engine_stats northd norecompute compute check_engine_stats lr_nat norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_pb recompute nocompute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -11150,7 +11152,7 @@ check as northd ovn-appctl -t NORTHD_TYPE inc-engine/clear-stats check ovn-nbctl --wait=sb set NAT . options:foo=bar check_engine_stats northd norecompute compute check_engine_stats lr_nat norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_pb recompute nocompute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -11160,7 +11162,7 @@ check ovn-nbctl --wait=sb set NAT . external_ip=172.168.0.120 check_engine_stats northd norecompute compute check_engine_stats lr_nat norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_pb recompute nocompute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -11170,7 +11172,7 @@ check ovn-nbctl --wait=sb set NAT . logical_ip=10.0.0.10 check_engine_stats northd norecompute compute check_engine_stats lr_nat norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_pb recompute nocompute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -11180,7 +11182,7 @@ check ovn-nbctl --wait=sb set NAT . type=snat check_engine_stats northd norecompute compute check_engine_stats lr_nat norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_pb recompute nocompute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -11190,7 +11192,7 @@ check ovn-nbctl --wait=sb lr-nat-add lr0 dnat_and_snat 172.168.0.110 10.0.0.4 sw check_engine_stats northd norecompute compute check_engine_stats lr_nat norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_pb recompute nocompute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -11201,7 +11203,7 @@ check ovn-nbctl --wait=sb set NAT $nat2_uuid external_mac='"30:54:00:00:00:04"' check_engine_stats northd norecompute compute check_engine_stats lr_nat norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_pb recompute nocompute CHECK_NO_CHANGE_AFTER_RECOMPUTE @@ -11222,6 +11224,8 @@ check_engine_stats sync_to_sb_pb recompute nocompute check_engine_stats lflow recompute nocompute CHECK_NO_CHANGE_AFTER_RECOMPUTE +# lflow engine should recompute since the nat ip 172.168.0.150 +# is a lb vip. check as northd ovn-appctl -t NORTHD_TYPE inc-engine/clear-stats check ovn-nbctl --wait=sb lr-nat-add lr0 dnat_and_snat 172.168.0.150 10.0.0.41 check_engine_stats northd norecompute compute @@ -11231,6 +11235,8 @@ check_engine_stats sync_to_sb_pb recompute nocompute check_engine_stats lflow recompute nocompute CHECK_NO_CHANGE_AFTER_RECOMPUTE +# lflow engine should recompute since the deleted nat ip 172.168.0.150 +# is a lb vip. check as northd ovn-appctl -t NORTHD_TYPE inc-engine/clear-stats check ovn-nbctl --wait=sb lr-nat-del lr0 dnat_and_snat 172.168.0.150 check_engine_stats northd norecompute compute @@ -11240,6 +11246,8 @@ check_engine_stats sync_to_sb_pb recompute nocompute check_engine_stats lflow recompute nocompute CHECK_NO_CHANGE_AFTER_RECOMPUTE +# lflow engine should recompute since the deleted nat ip 172.168.0.140 +# is a lb vip. check as northd ovn-appctl -t NORTHD_TYPE inc-engine/clear-stats check ovn-nbctl --wait=sb lr-nat-del lr0 dnat_and_snat 172.168.0.140 check_engine_stats northd norecompute compute @@ -11255,7 +11263,7 @@ check ovn-nbctl --wait=sb clear logical_router lr0 nat check_engine_stats northd norecompute compute check_engine_stats lr_nat norecompute compute check_engine_stats lr_lb_nat_data norecompute compute -check_engine_stats lflow recompute nocompute +check_engine_stats lflow norecompute compute check_engine_stats sync_to_sb_pb recompute nocompute CHECK_NO_CHANGE_AFTER_RECOMPUTE