From patchwork Fri Sep 9 21:32:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1676254 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MPTk36V9kz1ypF for ; Sat, 10 Sep 2022 07:32:59 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id F255D41938; Fri, 9 Sep 2022 21:32:53 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org F255D41938 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FnKMHF3aIwFQ; Fri, 9 Sep 2022 21:32:50 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id C6DBA4193B; Fri, 9 Sep 2022 21:32:47 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org C6DBA4193B Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CB509C007C; Fri, 9 Sep 2022 21:32:46 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id C00E0C0082 for ; Fri, 9 Sep 2022 21:32:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 4829F41934 for ; Fri, 9 Sep 2022 21:32:42 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 4829F41934 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zmDy94JNaA79 for ; Fri, 9 Sep 2022 21:32:41 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 2BFF441937 Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [IPv6:2001:4b98:dc4:8::227]) by smtp4.osuosl.org (Postfix) with ESMTPS id 2BFF441937 for ; Fri, 9 Sep 2022 21:32:37 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by mail.gandi.net (Postfix) with ESMTPSA id 17B182000B; Fri, 9 Sep 2022 21:32:30 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Fri, 9 Sep 2022 23:32:20 +0200 Message-Id: <20220909213223.824013-2-i.maximets@ovn.org> X-Mailer: git-send-email 2.34.3 In-Reply-To: <20220909213223.824013-1-i.maximets@ovn.org> References: <20220909213223.824013-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH ovn v3 1/4] northd: Optimize load balancer lookups for groups. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" ovn_northd_lb_lookup() takes significant percent of northd runtime in scenarios with large number of load balancers. In many cases, like in ovn-kubernetes, a lot of load balancers are actually groupped and applied to most of the switches and routers. So, instead of looking up all the same load balancers from the group for each datapath, we can look them up once and store as a group. Later we can lookup the entire group at once. Acked-by: Dumitru Ceara Signed-off-by: Ilya Maximets --- lib/lb.c | 49 +++++++++++++++++++++++++++++++- lib/lb.h | 21 ++++++++++++-- northd/en-northd.c | 2 ++ northd/northd.c | 70 ++++++++++++++++++++++++++++------------------ northd/northd.h | 3 ++ 5 files changed, 115 insertions(+), 30 deletions(-) diff --git a/lib/lb.c b/lib/lb.c index 7b0ed1abe..fca5f27da 100644 --- a/lib/lb.c +++ b/lib/lb.c @@ -226,7 +226,7 @@ ovn_northd_lb_create(const struct nbrec_load_balancer *nbrec_lb) } struct ovn_northd_lb * -ovn_northd_lb_find(struct hmap *lbs, const struct uuid *uuid) +ovn_northd_lb_find(const struct hmap *lbs, const struct uuid *uuid) { struct ovn_northd_lb *lb; size_t hash = uuid_hash(uuid); @@ -275,6 +275,53 @@ ovn_northd_lb_destroy(struct ovn_northd_lb *lb) free(lb); } +/* Constructs a new 'struct ovn_lb_group' object from the Nb LB Group record + * and a hash map of all existing 'struct ovn_northd_lb' objects. */ +struct ovn_lb_group * +ovn_lb_group_create(const struct nbrec_load_balancer_group *nbrec_lb_group, + const struct hmap *lbs) +{ + struct ovn_lb_group *lb_group; + + lb_group = xzalloc(sizeof *lb_group); + lb_group->uuid = nbrec_lb_group->header_.uuid; + lb_group->n_lbs = nbrec_lb_group->n_load_balancer; + lb_group->lbs = xmalloc(lb_group->n_lbs * sizeof *lb_group->lbs); + + for (size_t i = 0; i < nbrec_lb_group->n_load_balancer; i++) { + const struct uuid *lb_uuid = + &nbrec_lb_group->load_balancer[i]->header_.uuid; + lb_group->lbs[i] = ovn_northd_lb_find(lbs, lb_uuid); + } + + return lb_group; +} + +void +ovn_lb_group_destroy(struct ovn_lb_group *lb_group) +{ + if (!lb_group) { + return; + } + + free(lb_group->lbs); + free(lb_group); +} + +struct ovn_lb_group * +ovn_lb_group_find(const struct hmap *lb_groups, const struct uuid *uuid) +{ + struct ovn_lb_group *lb_group; + size_t hash = uuid_hash(uuid); + + HMAP_FOR_EACH_WITH_HASH (lb_group, hmap_node, hash, lb_groups) { + if (uuid_equals(&lb_group->uuid, uuid)) { + return lb_group; + } + } + return NULL; +} + struct ovn_controller_lb * ovn_controller_lb_create(const struct sbrec_load_balancer *sbrec_lb) { diff --git a/lib/lb.h b/lib/lb.h index 832ed31fb..1c70bfb6e 100644 --- a/lib/lb.h +++ b/lib/lb.h @@ -20,10 +20,12 @@ #include #include #include "openvswitch/hmap.h" -#include "sset.h" #include "ovn-util.h" +#include "sset.h" +#include "uuid.h" struct nbrec_load_balancer; +struct nbrec_load_balancer_group; struct sbrec_load_balancer; struct sbrec_datapath_binding; struct ovn_port; @@ -86,13 +88,28 @@ struct ovn_northd_lb_backend { }; struct ovn_northd_lb *ovn_northd_lb_create(const struct nbrec_load_balancer *); -struct ovn_northd_lb * ovn_northd_lb_find(struct hmap *, const struct uuid *); +struct ovn_northd_lb *ovn_northd_lb_find(const struct hmap *, + const struct uuid *); void ovn_northd_lb_destroy(struct ovn_northd_lb *); void ovn_northd_lb_add_lr(struct ovn_northd_lb *lb, struct ovn_datapath *od); void ovn_northd_lb_add_ls(struct ovn_northd_lb *lb, struct ovn_datapath *od); +struct ovn_lb_group { + struct hmap_node hmap_node; + struct uuid uuid; + size_t n_lbs; + struct ovn_northd_lb **lbs; +}; + +struct ovn_lb_group *ovn_lb_group_create( + const struct nbrec_load_balancer_group *, + const struct hmap *lbs); +void ovn_lb_group_destroy(struct ovn_lb_group *lb_group); +struct ovn_lb_group *ovn_lb_group_find(const struct hmap *lb_groups, + const struct uuid *); + struct ovn_controller_lb { const struct sbrec_load_balancer *slb; /* May be NULL. */ diff --git a/northd/en-northd.c b/northd/en-northd.c index 4907a1ff2..7fe83db64 100644 --- a/northd/en-northd.c +++ b/northd/en-northd.c @@ -68,6 +68,8 @@ void en_northd_run(struct engine_node *node, void *data) EN_OVSDB_GET(engine_get_input("NB_logical_router", node)); input_data.nbrec_load_balancer_table = EN_OVSDB_GET(engine_get_input("NB_load_balancer", node)); + input_data.nbrec_load_balancer_group_table = + EN_OVSDB_GET(engine_get_input("NB_load_balancer_group", node)); input_data.nbrec_port_group_table = EN_OVSDB_GET(engine_get_input("NB_port_group", node)); input_data.nbrec_address_set_table = diff --git a/northd/northd.c b/northd/northd.c index deda4a9d3..e00625e9c 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -3843,11 +3843,14 @@ build_lrouter_lb_ips(struct ovn_datapath *od, const struct ovn_northd_lb *lb) static void build_lbs(struct northd_input *input_data, struct hmap *datapaths, - struct hmap *lbs) + struct hmap *lbs, struct hmap *lb_groups) { + const struct nbrec_load_balancer_group *nbrec_lb_group; + struct ovn_lb_group *lb_group; struct ovn_northd_lb *lb; hmap_init(lbs); + hmap_init(lb_groups); const struct nbrec_load_balancer *nbrec_lb; NBREC_LOAD_BALANCER_TABLE_FOR_EACH (nbrec_lb, @@ -3857,6 +3860,13 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, uuid_hash(&nbrec_lb->header_.uuid)); } + NBREC_LOAD_BALANCER_GROUP_TABLE_FOR_EACH (nbrec_lb_group, + input_data->nbrec_load_balancer_group_table) { + lb_group = ovn_lb_group_create(nbrec_lb_group, lbs); + hmap_insert(lb_groups, &lb_group->hmap_node, + uuid_hash(&lb_group->uuid)); + } + struct ovn_datapath *od; HMAP_FOR_EACH (od, key_node, datapaths) { if (!od->nbs) { @@ -3871,13 +3881,11 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, } for (size_t i = 0; i < od->nbs->n_load_balancer_group; i++) { - const struct nbrec_load_balancer_group *lbg = - od->nbs->load_balancer_group[i]; - for (size_t j = 0; j < lbg->n_load_balancer; j++) { - const struct uuid *lb_uuid = - &lbg->load_balancer[j]->header_.uuid; - lb = ovn_northd_lb_find(lbs, lb_uuid); - ovn_northd_lb_add_ls(lb, od); + nbrec_lb_group = od->nbs->load_balancer_group[i]; + lb_group = ovn_lb_group_find(lb_groups, + &nbrec_lb_group->header_.uuid); + for (size_t j = 0; j < lb_group->n_lbs; j++) { + ovn_northd_lb_add_ls(lb_group->lbs[j], od); } } } @@ -3896,14 +3904,12 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, } for (size_t i = 0; i < od->nbr->n_load_balancer_group; i++) { - const struct nbrec_load_balancer_group *lbg = - od->nbr->load_balancer_group[i]; - for (size_t j = 0; j < lbg->n_load_balancer; j++) { - const struct uuid *lb_uuid = - &lbg->load_balancer[j]->header_.uuid; - lb = ovn_northd_lb_find(lbs, lb_uuid); - ovn_northd_lb_add_lr(lb, od); - build_lrouter_lb_ips(od, lb); + nbrec_lb_group = od->nbr->load_balancer_group[i]; + lb_group = ovn_lb_group_find(lb_groups, + &nbrec_lb_group->header_.uuid); + for (size_t j = 0; j < lb_group->n_lbs; j++) { + ovn_northd_lb_add_lr(lb_group->lbs[j], od); + build_lrouter_lb_ips(od, lb_group->lbs[j]); } } } @@ -4021,7 +4027,8 @@ build_lrouter_lbs_check(const struct hmap *datapaths) } static void -build_lrouter_lbs_reachable_ips(struct hmap *datapaths, struct hmap *lbs) +build_lrouter_lbs_reachable_ips(struct hmap *datapaths, struct hmap *lbs, + struct hmap *lb_groups) { struct ovn_datapath *od; @@ -4038,13 +4045,14 @@ build_lrouter_lbs_reachable_ips(struct hmap *datapaths, struct hmap *lbs) } for (size_t i = 0; i < od->nbr->n_load_balancer_group; i++) { - const struct nbrec_load_balancer_group *lbg = + const struct nbrec_load_balancer_group *nbrec_lb_group = od->nbr->load_balancer_group[i]; - for (size_t j = 0; j < lbg->n_load_balancer; j++) { - struct ovn_northd_lb *lb = - ovn_northd_lb_find(lbs, - &lbg->load_balancer[j]->header_.uuid); - build_lrouter_lb_reachable_ips(od, lb); + struct ovn_lb_group *lb_group; + + lb_group = ovn_lb_group_find(lb_groups, + &nbrec_lb_group->header_.uuid); + for (size_t j = 0; j < lb_group->n_lbs; j++) { + build_lrouter_lb_reachable_ips(od, lb_group->lbs[j]); } } } @@ -4105,11 +4113,12 @@ build_lswitch_lbs_from_lrouter(struct hmap *datapaths, struct hmap *lbs) */ static void build_lb_port_related_data(struct hmap *datapaths, struct hmap *ports, - struct hmap *lbs, struct northd_input *input_data, + struct hmap *lbs, struct hmap *lb_groups, + struct northd_input *input_data, struct ovsdb_idl_txn *ovnsb_txn) { build_lrouter_lbs_check(datapaths); - build_lrouter_lbs_reachable_ips(datapaths, lbs); + build_lrouter_lbs_reachable_ips(datapaths, lbs, lb_groups); build_lb_svcs(input_data, ovnsb_txn, ports, lbs); build_lswitch_lbs_from_lrouter(datapaths, lbs); } @@ -15373,6 +15382,7 @@ northd_init(struct northd_data *data) hmap_init(&data->port_groups); shash_init(&data->meter_groups); hmap_init(&data->lbs); + hmap_init(&data->lb_groups); hmap_init(&data->bfd_connections); ovs_list_init(&data->lr_list); data->features = (struct chassis_features) { @@ -15391,6 +15401,12 @@ northd_destroy(struct northd_data *data) } hmap_destroy(&data->lbs); + struct ovn_lb_group *lb_group; + HMAP_FOR_EACH_POP (lb_group, hmap_node, &data->lb_groups) { + ovn_lb_group_destroy(lb_group); + } + hmap_destroy(&data->lb_groups); + struct ovn_port_group *pg; HMAP_FOR_EACH_SAFE (pg, key_node, &data->port_groups) { ovn_port_group_destroy(&data->port_groups, pg); @@ -15500,12 +15516,12 @@ ovnnb_db_run(struct northd_input *input_data, build_chassis_features(input_data, &data->features); build_datapaths(input_data, ovnsb_txn, &data->datapaths, &data->lr_list); - build_lbs(input_data, &data->datapaths, &data->lbs); + build_lbs(input_data, &data->datapaths, &data->lbs, &data->lb_groups); build_ports(input_data, ovnsb_txn, sbrec_chassis_by_name, sbrec_chassis_by_hostname, &data->datapaths, &data->ports); build_lb_port_related_data(&data->datapaths, &data->ports, &data->lbs, - input_data, ovnsb_txn); + &data->lb_groups, input_data, ovnsb_txn); build_ipam(&data->datapaths, &data->ports); build_port_group_lswitches(input_data, &data->port_groups, &data->ports); build_lrouter_groups(&data->ports, &data->lr_list); diff --git a/northd/northd.h b/northd/northd.h index d9856af97..8d299864f 100644 --- a/northd/northd.h +++ b/northd/northd.h @@ -28,6 +28,8 @@ struct northd_input { const struct nbrec_logical_switch_table *nbrec_logical_switch; const struct nbrec_logical_router_table *nbrec_logical_router; const struct nbrec_load_balancer_table *nbrec_load_balancer_table; + const struct nbrec_load_balancer_group_table + *nbrec_load_balancer_group_table; const struct nbrec_port_group_table *nbrec_port_group_table; const struct nbrec_address_set_table *nbrec_address_set_table; const struct nbrec_meter_table *nbrec_meter_table; @@ -74,6 +76,7 @@ struct northd_data { struct hmap port_groups; struct shash meter_groups; struct hmap lbs; + struct hmap lb_groups; struct hmap bfd_connections; struct ovs_list lr_list; bool ovn_internal_version_changed; From patchwork Fri Sep 9 21:32:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1676252 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MPTjz131cz1ypB for ; Sat, 10 Sep 2022 07:32:53 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 3E88641BDC; Fri, 9 Sep 2022 21:32:48 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 3E88641BDC X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EJNjLy1GWmRV; Fri, 9 Sep 2022 21:32:46 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id EF2024198C; Fri, 9 Sep 2022 21:32:44 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org EF2024198C Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id F007BC007F; Fri, 9 Sep 2022 21:32:42 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id D6830C002D for ; Fri, 9 Sep 2022 21:32:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id A158B83F1A for ; Fri, 9 Sep 2022 21:32:40 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org A158B83F1A X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ug-r6_labd8M for ; Fri, 9 Sep 2022 21:32:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 1469483F0D Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by smtp1.osuosl.org (Postfix) with ESMTPS id 1469483F0D for ; Fri, 9 Sep 2022 21:32:36 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by mail.gandi.net (Postfix) with ESMTPSA id 6CBCF2000E; Fri, 9 Sep 2022 21:32:34 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Fri, 9 Sep 2022 23:32:21 +0200 Message-Id: <20220909213223.824013-3-i.maximets@ovn.org> X-Mailer: git-send-email 2.34.3 In-Reply-To: <20220909213223.824013-1-i.maximets@ovn.org> References: <20220909213223.824013-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH ovn v3 2/4] northd: Add datapaths to load balancers in bulk. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Filling arrays of switches and routers for load balancers one-by-one is not very efficient. Copying them in bulk allows to save a noticeable amount of time in setups with large load balancer groups. Acked-by: Dumitru Ceara Signed-off-by: Ilya Maximets --- lib/lb.c | 27 +++++++++++++++++++-------- lib/lb.h | 30 +++++++++++++++++++++++++----- northd/northd.c | 29 +++++++++++++++++++++-------- 3 files changed, 65 insertions(+), 21 deletions(-) diff --git a/lib/lb.c b/lib/lb.c index fca5f27da..0b229f828 100644 --- a/lib/lb.c +++ b/lib/lb.c @@ -239,23 +239,27 @@ ovn_northd_lb_find(const struct hmap *lbs, const struct uuid *uuid) } void -ovn_northd_lb_add_lr(struct ovn_northd_lb *lb, struct ovn_datapath *od) +ovn_northd_lb_add_lr(struct ovn_northd_lb *lb, size_t n, + struct ovn_datapath **ods) { - if (lb->n_allocated_nb_lr == lb->n_nb_lr) { + while (lb->n_allocated_nb_lr <= lb->n_nb_lr + n) { lb->nb_lr = x2nrealloc(lb->nb_lr, &lb->n_allocated_nb_lr, sizeof *lb->nb_lr); } - lb->nb_lr[lb->n_nb_lr++] = od; + memcpy(&lb->nb_lr[lb->n_nb_lr], ods, n * sizeof *ods); + lb->n_nb_lr += n; } void -ovn_northd_lb_add_ls(struct ovn_northd_lb *lb, struct ovn_datapath *od) +ovn_northd_lb_add_ls(struct ovn_northd_lb *lb, size_t n, + struct ovn_datapath **ods) { - if (lb->n_allocated_nb_ls == lb->n_nb_ls) { + while (lb->n_allocated_nb_ls <= lb->n_nb_ls + n) { lb->nb_ls = x2nrealloc(lb->nb_ls, &lb->n_allocated_nb_ls, sizeof *lb->nb_ls); } - lb->nb_ls[lb->n_nb_ls++] = od; + memcpy(&lb->nb_ls[lb->n_nb_ls], ods, n * sizeof *ods); + lb->n_nb_ls += n; } void @@ -276,10 +280,13 @@ ovn_northd_lb_destroy(struct ovn_northd_lb *lb) } /* Constructs a new 'struct ovn_lb_group' object from the Nb LB Group record - * and a hash map of all existing 'struct ovn_northd_lb' objects. */ + * and a hash map of all existing 'struct ovn_northd_lb' objects. Space will + * be allocated for 'max_datapaths' logical switches and the same amount of + * logical routers to which this LB Group is applied. Can be filled later + * with ovn_lb_group_add_ls() and ovn_lb_group_add_lr() respectively. */ struct ovn_lb_group * ovn_lb_group_create(const struct nbrec_load_balancer_group *nbrec_lb_group, - const struct hmap *lbs) + const struct hmap *lbs, size_t max_datapaths) { struct ovn_lb_group *lb_group; @@ -287,6 +294,8 @@ ovn_lb_group_create(const struct nbrec_load_balancer_group *nbrec_lb_group, lb_group->uuid = nbrec_lb_group->header_.uuid; lb_group->n_lbs = nbrec_lb_group->n_load_balancer; lb_group->lbs = xmalloc(lb_group->n_lbs * sizeof *lb_group->lbs); + lb_group->ls = xmalloc(max_datapaths * sizeof *lb_group->ls); + lb_group->lr = xmalloc(max_datapaths * sizeof *lb_group->lr); for (size_t i = 0; i < nbrec_lb_group->n_load_balancer; i++) { const struct uuid *lb_uuid = @@ -305,6 +314,8 @@ ovn_lb_group_destroy(struct ovn_lb_group *lb_group) } free(lb_group->lbs); + free(lb_group->ls); + free(lb_group->lr); free(lb_group); } diff --git a/lib/lb.h b/lib/lb.h index 1c70bfb6e..fbd4f19e8 100644 --- a/lib/lb.h +++ b/lib/lb.h @@ -28,6 +28,7 @@ struct nbrec_load_balancer; struct nbrec_load_balancer_group; struct sbrec_load_balancer; struct sbrec_datapath_binding; +struct ovn_datapath; struct ovn_port; struct uuid; @@ -91,25 +92,44 @@ struct ovn_northd_lb *ovn_northd_lb_create(const struct nbrec_load_balancer *); struct ovn_northd_lb *ovn_northd_lb_find(const struct hmap *, const struct uuid *); void ovn_northd_lb_destroy(struct ovn_northd_lb *); -void -ovn_northd_lb_add_lr(struct ovn_northd_lb *lb, struct ovn_datapath *od); -void -ovn_northd_lb_add_ls(struct ovn_northd_lb *lb, struct ovn_datapath *od); +void ovn_northd_lb_add_lr(struct ovn_northd_lb *lb, size_t n, + struct ovn_datapath **ods); +void ovn_northd_lb_add_ls(struct ovn_northd_lb *lb, size_t n, + struct ovn_datapath **ods); struct ovn_lb_group { struct hmap_node hmap_node; struct uuid uuid; size_t n_lbs; struct ovn_northd_lb **lbs; + + /* Datapaths to which this LB group is applied. */ + size_t n_ls; + struct ovn_datapath **ls; + size_t n_lr; + struct ovn_datapath **lr; }; struct ovn_lb_group *ovn_lb_group_create( const struct nbrec_load_balancer_group *, - const struct hmap *lbs); + const struct hmap *lbs, + size_t max_datapaths); void ovn_lb_group_destroy(struct ovn_lb_group *lb_group); struct ovn_lb_group *ovn_lb_group_find(const struct hmap *lb_groups, const struct uuid *); +static inline void +ovn_lb_group_add_ls(struct ovn_lb_group *lb_group, struct ovn_datapath *ls) +{ + lb_group->ls[lb_group->n_ls++] = ls; +} + +static inline void +ovn_lb_group_add_lr(struct ovn_lb_group *lb_group, struct ovn_datapath *lr) +{ + lb_group->lr[lb_group->n_lr++] = lr; +} + struct ovn_controller_lb { const struct sbrec_load_balancer *slb; /* May be NULL. */ diff --git a/northd/northd.c b/northd/northd.c index e00625e9c..b08ad932b 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -3862,7 +3862,8 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, NBREC_LOAD_BALANCER_GROUP_TABLE_FOR_EACH (nbrec_lb_group, input_data->nbrec_load_balancer_group_table) { - lb_group = ovn_lb_group_create(nbrec_lb_group, lbs); + lb_group = ovn_lb_group_create(nbrec_lb_group, lbs, + hmap_count(datapaths)); hmap_insert(lb_groups, &lb_group->hmap_node, uuid_hash(&lb_group->uuid)); } @@ -3877,16 +3878,21 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, const struct uuid *lb_uuid = &od->nbs->load_balancer[i]->header_.uuid; lb = ovn_northd_lb_find(lbs, lb_uuid); - ovn_northd_lb_add_ls(lb, od); + ovn_northd_lb_add_ls(lb, 1, &od); } for (size_t i = 0; i < od->nbs->n_load_balancer_group; i++) { nbrec_lb_group = od->nbs->load_balancer_group[i]; lb_group = ovn_lb_group_find(lb_groups, &nbrec_lb_group->header_.uuid); - for (size_t j = 0; j < lb_group->n_lbs; j++) { - ovn_northd_lb_add_ls(lb_group->lbs[j], od); - } + ovn_lb_group_add_ls(lb_group, od); + } + } + + HMAP_FOR_EACH (lb_group, hmap_node, lb_groups) { + for (size_t j = 0; j < lb_group->n_lbs; j++) { + ovn_northd_lb_add_ls(lb_group->lbs[j], lb_group->n_ls, + lb_group->ls); } } @@ -3899,7 +3905,7 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, const struct uuid *lb_uuid = &od->nbr->load_balancer[i]->header_.uuid; lb = ovn_northd_lb_find(lbs, lb_uuid); - ovn_northd_lb_add_lr(lb, od); + ovn_northd_lb_add_lr(lb, 1, &od); build_lrouter_lb_ips(od, lb); } @@ -3907,12 +3913,19 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, nbrec_lb_group = od->nbr->load_balancer_group[i]; lb_group = ovn_lb_group_find(lb_groups, &nbrec_lb_group->header_.uuid); + ovn_lb_group_add_lr(lb_group, od); for (size_t j = 0; j < lb_group->n_lbs; j++) { - ovn_northd_lb_add_lr(lb_group->lbs[j], od); build_lrouter_lb_ips(od, lb_group->lbs[j]); } } } + + HMAP_FOR_EACH (lb_group, hmap_node, lb_groups) { + for (size_t j = 0; j < lb_group->n_lbs; j++) { + ovn_northd_lb_add_lr(lb_group->lbs[j], lb_group->n_lr, + lb_group->lr); + } + } } static void @@ -4097,7 +4110,7 @@ build_lswitch_lbs_from_lrouter(struct hmap *datapaths, struct hmap *lbs) } } if (!installed) { - ovn_northd_lb_add_ls(lb, od); + ovn_northd_lb_add_ls(lb, 1, &od); } if (lb->nlb) { od->has_lb_vip |= lb_has_vip(lb->nlb); From patchwork Fri Sep 9 21:32:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1676255 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MPTkD4D1Tz1ynm for ; Sat, 10 Sep 2022 07:33:06 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id B62858407A; Fri, 9 Sep 2022 21:32:48 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org B62858407A X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eMVD9-RguK8o; Fri, 9 Sep 2022 21:32:47 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 2D47283F95; Fri, 9 Sep 2022 21:32:46 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 2D47283F95 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 01D78C0082; Fri, 9 Sep 2022 21:32:44 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 32AAAC007C for ; Fri, 9 Sep 2022 21:32:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id C8A104193B for ; Fri, 9 Sep 2022 21:32:41 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org C8A104193B X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3W6gzvDvWPjv for ; Fri, 9 Sep 2022 21:32:40 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 41A8541934 Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by smtp4.osuosl.org (Postfix) with ESMTPS id 41A8541934 for ; Fri, 9 Sep 2022 21:32:40 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by mail.gandi.net (Postfix) with ESMTPSA id 8453C20005; Fri, 9 Sep 2022 21:32:37 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Fri, 9 Sep 2022 23:32:22 +0200 Message-Id: <20220909213223.824013-4-i.maximets@ovn.org> X-Mailer: git-send-email 2.34.3 In-Reply-To: <20220909213223.824013-1-i.maximets@ovn.org> References: <20220909213223.824013-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH ovn v3 3/4] northd: Retrieve load balancer options only once. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Options like 'add_route' and 'neighbor_responder' are looked up in the smap for each load balancer for each datapath and it takes significant amount of time. Checking them only once to speed up processing. 'skip_snat' is not that critical, but it's better to get all of them in a single place. Also using enum for the neighbor responder mode to avoid costly string comparison on a hot path. Acked-by: Dumitru Ceara Signed-off-by: Ilya Maximets --- lib/lb.c | 7 +++++++ lib/lb.h | 10 ++++++++++ northd/northd.c | 38 ++++++++++++++++---------------------- 3 files changed, 33 insertions(+), 22 deletions(-) diff --git a/lib/lb.c b/lib/lb.c index 0b229f828..fa1a66d82 100644 --- a/lib/lb.c +++ b/lib/lb.c @@ -173,6 +173,13 @@ ovn_northd_lb_create(const struct nbrec_load_balancer *nbrec_lb) lb->n_vips = smap_count(&nbrec_lb->vips); lb->vips = xcalloc(lb->n_vips, sizeof *lb->vips); lb->vips_nb = xcalloc(lb->n_vips, sizeof *lb->vips_nb); + lb->controller_event = smap_get_bool(&nbrec_lb->options, "event", false); + lb->routable = smap_get_bool(&nbrec_lb->options, "add_route", false); + lb->skip_snat = smap_get_bool(&nbrec_lb->options, "skip_snat", false); + const char *mode = + smap_get_def(&nbrec_lb->options, "neighbor_responder", "reachable"); + lb->neigh_mode = strcmp(mode, "all") ? LB_NEIGH_RESPOND_REACHABLE + : LB_NEIGH_RESPOND_ALL; sset_init(&lb->ips_v4); sset_init(&lb->ips_v6); struct smap_node *node; diff --git a/lib/lb.h b/lib/lb.h index fbd4f19e8..e0b153fb3 100644 --- a/lib/lb.h +++ b/lib/lb.h @@ -32,6 +32,11 @@ struct ovn_datapath; struct ovn_port; struct uuid; +enum lb_neighbor_responder_mode { + LB_NEIGH_RESPOND_REACHABLE, + LB_NEIGH_RESPOND_ALL, +}; + struct ovn_northd_lb { struct hmap_node hmap_node; @@ -43,6 +48,11 @@ struct ovn_northd_lb { struct ovn_northd_lb_vip *vips_nb; size_t n_vips; + enum lb_neighbor_responder_mode neigh_mode; + bool controller_event; + bool routable; + bool skip_snat; + struct sset ips_v4; struct sset ips_v6; diff --git a/northd/northd.c b/northd/northd.c index b08ad932b..a4aed7f96 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -3824,18 +3824,17 @@ build_lb_vip_actions(struct ovn_lb_vip *lb_vip, static void build_lrouter_lb_ips(struct ovn_datapath *od, const struct ovn_northd_lb *lb) { - bool is_routable = smap_get_bool(&lb->nlb->options, "add_route", false); const char *ip_address; SSET_FOR_EACH (ip_address, &lb->ips_v4) { sset_add(&od->lb_ips_v4, ip_address); - if (is_routable) { + if (lb->routable) { sset_add(&od->lb_ips_v4_routable, ip_address); } } SSET_FOR_EACH (ip_address, &lb->ips_v6) { sset_add(&od->lb_ips_v6, ip_address); - if (is_routable) { + if (lb->routable) { sset_add(&od->lb_ips_v6_routable, ip_address); } } @@ -3972,13 +3971,10 @@ static void build_lrouter_lb_reachable_ips(struct ovn_datapath *od, const struct ovn_northd_lb *lb) { - const char *neighbor_responder_mode = - smap_get_def(&lb->nlb->options, "neighbor_responder", "reachable"); - /* If configured to reply to neighbor requests for all VIPs force them * all to be considered "reachable". */ - if (!strcmp(neighbor_responder_mode, "all")) { + if (lb->neigh_mode == LB_NEIGH_RESPOND_ALL) { for (size_t i = 0; i < lb->n_vips; i++) { if (IN6_IS_ADDR_V4MAPPED(&lb->vips[i].vip)) { sset_add(&od->lb_ips_v4_reachable, lb->vips[i].vip_str); @@ -5758,10 +5754,10 @@ ls_has_dns_records(const struct nbrec_logical_switch *nbs) static bool build_empty_lb_event_flow(struct ovn_lb_vip *lb_vip, - const struct nbrec_load_balancer *lb, + const struct ovn_northd_lb *lb, struct ds *match, struct ds *action) { - bool controller_event = smap_get_bool(&lb->options, "event", false) || + bool controller_event = lb->controller_event || controller_event_en; /* deprecated */ if (!controller_event || lb_vip->n_backends || lb_vip->empty_backend_rej) { @@ -5774,12 +5770,11 @@ build_empty_lb_event_flow(struct ovn_lb_vip *lb_vip, bool ipv4 = IN6_IS_ADDR_V4MAPPED(&lb_vip->vip); ds_put_format(match, "ip%s.dst == %s && %s", - ipv4 ? "4": "6", lb_vip->vip_str, lb->protocol); + ipv4 ? "4": "6", lb_vip->vip_str, lb->proto); char *vip = lb_vip->vip_str; if (lb_vip->vip_port) { - ds_put_format(match, " && %s.dst == %u", lb->protocol, - lb_vip->vip_port); + ds_put_format(match, " && %s.dst == %u", lb->proto, lb_vip->vip_port); vip = xasprintf("%s%s%s:%u", ipv4 ? "" : "[", lb_vip->vip_str, ipv4 ? "" : "]", lb_vip->vip_port); } @@ -5790,8 +5785,8 @@ build_empty_lb_event_flow(struct ovn_lb_vip *lb_vip, "protocol = \"%s\", " "load_balancer = \"" UUID_FMT "\");", event_to_string(OVN_EVENT_EMPTY_LB_BACKENDS), - vip, lb->protocol, - UUID_ARGS(&lb->header_.uuid)); + vip, lb->proto, + UUID_ARGS(&lb->nlb->header_.uuid)); if (lb_vip->vip_port) { free(vip); } @@ -9950,8 +9945,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip, lb_vip->vip_str); } - bool lb_skip_snat = smap_get_bool(&lb->nlb->options, "skip_snat", false); - if (lb_skip_snat) { + if (lb->skip_snat) { skip_snat_new_action = xasprintf("flags.skip_snat_for_lb = 1; %s", ds_cstr(action)); skip_snat_est_action = xasprintf("flags.skip_snat_for_lb = 1; " @@ -10035,7 +10029,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip, for (size_t i = 0; i < lb->n_nb_lr; i++) { struct ovn_datapath *od = lb->nb_lr[i]; if (!od->n_l3dgw_ports) { - if (lb_skip_snat) { + if (lb->skip_snat) { gw_router_skip_snat[n_gw_router_skip_snat++] = od; } else if (!lport_addresses_is_empty(&od->lb_force_snat_addrs) || od->lb_force_snat_router_ip) { @@ -10106,7 +10100,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip, od->l3dgw_ports[0]->cr_port->json_key); } - if (lb_skip_snat) { + if (lb->skip_snat) { ovn_lflow_add_with_hint__(lflows, od, S_ROUTER_IN_DNAT, prio, new_match_p, skip_snat_new_action, NULL, meter, &lb->nlb->header_); @@ -10146,7 +10140,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip, ds_cstr(&undnat_match), od->l3dgw_ports[0]->json_key, od->l3dgw_ports[0]->cr_port->json_key); - if (lb_skip_snat) { + if (lb->skip_snat) { ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120, undnat_match_p, skip_snat_est_action, &lb->nlb->header_); @@ -10193,7 +10187,7 @@ build_lswitch_flows_for_lb(struct ovn_northd_lb *lb, struct hmap *lflows, struct ovn_lb_vip *lb_vip = &lb->vips[i]; /* pre-stateful lb */ - if (!build_empty_lb_event_flow(lb_vip, lb->nlb, match, action)) { + if (!build_empty_lb_event_flow(lb_vip, lb, match, action)) { continue; } for (size_t j = 0; j < lb->n_nb_ls; j++) { @@ -10307,7 +10301,7 @@ build_lrouter_flows_for_lb(struct ovn_northd_lb *lb, struct hmap *lflows, lflows, match, action, meter_groups, features->ct_no_masked_label); - if (!build_empty_lb_event_flow(lb_vip, lb->nlb, match, action)) { + if (!build_empty_lb_event_flow(lb_vip, lb, match, action)) { continue; } for (size_t j = 0; j < lb->n_nb_lr; j++) { @@ -10322,7 +10316,7 @@ build_lrouter_flows_for_lb(struct ovn_northd_lb *lb, struct hmap *lflows, } } - if (smap_get_bool(&lb->nlb->options, "skip_snat", false)) { + if (lb->skip_snat) { for (size_t i = 0; i < lb->n_nb_lr; i++) { ovn_lflow_add(lflows, lb->nb_lr[i], S_ROUTER_OUT_SNAT, 120, "flags.skip_snat_for_lb == 1 && ip", "next;"); From patchwork Fri Sep 9 21:32:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1676253 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MPTk206ybz1ypD for ; Sat, 10 Sep 2022 07:32:57 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 4835261286; Fri, 9 Sep 2022 21:32:55 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 4835261286 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id emRBQ4K3TEiO; Fri, 9 Sep 2022 21:32:51 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id F42056125F; Fri, 9 Sep 2022 21:32:49 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org F42056125F Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CABC2C0084; Fri, 9 Sep 2022 21:32:48 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 47368C0032 for ; Fri, 9 Sep 2022 21:32:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 18EDA8415F for ; Fri, 9 Sep 2022 21:32:47 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 18EDA8415F X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id t5T3NF9BpVda for ; Fri, 9 Sep 2022 21:32:45 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 3D3D483F0D Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by smtp1.osuosl.org (Postfix) with ESMTPS id 3D3D483F0D for ; Fri, 9 Sep 2022 21:32:43 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by mail.gandi.net (Postfix) with ESMTPSA id 72A0F20007; Fri, 9 Sep 2022 21:32:41 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Fri, 9 Sep 2022 23:32:23 +0200 Message-Id: <20220909213223.824013-5-i.maximets@ovn.org> X-Mailer: git-send-email 2.34.3 In-Reply-To: <20220909213223.824013-1-i.maximets@ovn.org> References: <20220909213223.824013-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Ilya Maximets , Dumitru Ceara Subject: [ovs-dev] [PATCH ovn v3 4/4] northd: Re-use IP sets created for load balancer groups. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" If the same load balancer group is attached to multiple routers, IP sets will be constructed for each of them by sequentially calling sset_add() for each IP for each load balancer for each router. Instead of doing that, we can create IP sets for load balancer groups and clone them. That will avoid extra ssed_find__() call making the construction of IP sets 2x faster. Only first load balancer group is cloned, the rest as well as standalone load balancers are still added one by one, because there is no more efficient way to merge sets. The order of processing changed to make sure that we're actually optimizing copy of a large group. The code can be optimized further by introduction of a reference counter and not copying at all if the router has no standalone load balancers and only one group. Also, construction of "reachable" sets can be optimized for the "neighbor_responder = all" case. But, for now, only moved to a new structure for better readability. Acked-by: Dumitru Ceara Signed-off-by: Ilya Maximets --- lib/lb.c | 47 ++++++++++++++++++++ lib/lb.h | 16 +++++++ northd/northd.c | 114 ++++++++++++++++++++++++++---------------------- northd/northd.h | 13 ++---- 4 files changed, 129 insertions(+), 61 deletions(-) diff --git a/lib/lb.c b/lib/lb.c index fa1a66d82..477cf8f5e 100644 --- a/lib/lb.c +++ b/lib/lb.c @@ -26,6 +26,51 @@ VLOG_DEFINE_THIS_MODULE(lb); +struct ovn_lb_ip_set * +ovn_lb_ip_set_create(void) +{ + struct ovn_lb_ip_set *lb_ip_set = xzalloc(sizeof *lb_ip_set); + + sset_init(&lb_ip_set->ips_v4); + sset_init(&lb_ip_set->ips_v4_routable); + sset_init(&lb_ip_set->ips_v4_reachable); + sset_init(&lb_ip_set->ips_v6); + sset_init(&lb_ip_set->ips_v6_routable); + sset_init(&lb_ip_set->ips_v6_reachable); + + return lb_ip_set; +} + +void +ovn_lb_ip_set_destroy(struct ovn_lb_ip_set *lb_ip_set) +{ + if (!lb_ip_set) { + return; + } + sset_destroy(&lb_ip_set->ips_v4); + sset_destroy(&lb_ip_set->ips_v4_routable); + sset_destroy(&lb_ip_set->ips_v4_reachable); + sset_destroy(&lb_ip_set->ips_v6); + sset_destroy(&lb_ip_set->ips_v6_routable); + sset_destroy(&lb_ip_set->ips_v6_reachable); + free(lb_ip_set); +} + +struct ovn_lb_ip_set * +ovn_lb_ip_set_clone(struct ovn_lb_ip_set *lb_ip_set) +{ + struct ovn_lb_ip_set *clone = ovn_lb_ip_set_create(); + + sset_clone(&clone->ips_v4, &lb_ip_set->ips_v4); + sset_clone(&clone->ips_v4_routable, &lb_ip_set->ips_v4_routable); + sset_clone(&clone->ips_v4_reachable, &lb_ip_set->ips_v4_reachable); + sset_clone(&clone->ips_v6, &lb_ip_set->ips_v6); + sset_clone(&clone->ips_v6_routable, &lb_ip_set->ips_v6_routable); + sset_clone(&clone->ips_v6_reachable, &lb_ip_set->ips_v6_reachable); + + return clone; +} + static bool ovn_lb_vip_init(struct ovn_lb_vip *lb_vip, const char *lb_key, const char *lb_value) @@ -303,6 +348,7 @@ ovn_lb_group_create(const struct nbrec_load_balancer_group *nbrec_lb_group, lb_group->lbs = xmalloc(lb_group->n_lbs * sizeof *lb_group->lbs); lb_group->ls = xmalloc(max_datapaths * sizeof *lb_group->ls); lb_group->lr = xmalloc(max_datapaths * sizeof *lb_group->lr); + lb_group->lb_ips = ovn_lb_ip_set_create(); for (size_t i = 0; i < nbrec_lb_group->n_load_balancer; i++) { const struct uuid *lb_uuid = @@ -320,6 +366,7 @@ ovn_lb_group_destroy(struct ovn_lb_group *lb_group) return; } + ovn_lb_ip_set_destroy(lb_group->lb_ips); free(lb_group->lbs); free(lb_group->ls); free(lb_group->lr); diff --git a/lib/lb.h b/lib/lb.h index e0b153fb3..9b902f005 100644 --- a/lib/lb.h +++ b/lib/lb.h @@ -37,6 +37,21 @@ enum lb_neighbor_responder_mode { LB_NEIGH_RESPOND_ALL, }; +/* The "routable" ssets are subsets of the load balancer IPs for which IP + * routes and ARP resolution flows are automatically added. */ +struct ovn_lb_ip_set { + struct sset ips_v4; + struct sset ips_v4_routable; + struct sset ips_v4_reachable; + struct sset ips_v6; + struct sset ips_v6_routable; + struct sset ips_v6_reachable; +}; + +struct ovn_lb_ip_set *ovn_lb_ip_set_create(void); +void ovn_lb_ip_set_destroy(struct ovn_lb_ip_set *); +struct ovn_lb_ip_set *ovn_lb_ip_set_clone(struct ovn_lb_ip_set *); + struct ovn_northd_lb { struct hmap_node hmap_node; @@ -112,6 +127,7 @@ struct ovn_lb_group { struct uuid uuid; size_t n_lbs; struct ovn_northd_lb **lbs; + struct ovn_lb_ip_set *lb_ips; /* Datapaths to which this LB group is applied. */ size_t n_ls; diff --git a/northd/northd.c b/northd/northd.c index a4aed7f96..f2c0f5aed 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -821,13 +821,6 @@ lr_lb_address_set_ref(const struct ovn_datapath *od, int addr_family) static void init_lb_for_datapath(struct ovn_datapath *od) { - sset_init(&od->lb_ips_v4); - sset_init(&od->lb_ips_v4_routable); - sset_init(&od->lb_ips_v4_reachable); - sset_init(&od->lb_ips_v6); - sset_init(&od->lb_ips_v6_routable); - sset_init(&od->lb_ips_v6_reachable); - if (od->nbs) { od->has_lb_vip = ls_has_lb_vip(od); } else { @@ -838,16 +831,12 @@ init_lb_for_datapath(struct ovn_datapath *od) static void destroy_lb_for_datapath(struct ovn_datapath *od) { + ovn_lb_ip_set_destroy(od->lb_ips); + od->lb_ips = NULL; + if (!od->nbs && !od->nbr) { return; } - - sset_destroy(&od->lb_ips_v4); - sset_destroy(&od->lb_ips_v4_routable); - sset_destroy(&od->lb_ips_v4_reachable); - sset_destroy(&od->lb_ips_v6); - sset_destroy(&od->lb_ips_v6_routable); - sset_destroy(&od->lb_ips_v6_reachable); } /* A group of logical router datapaths which are connected - either @@ -2818,20 +2807,20 @@ get_nat_addresses(const struct ovn_port *op, size_t *n, bool routable_only, if (include_lb_ips) { const char *ip_address; if (routable_only) { - SSET_FOR_EACH (ip_address, &op->od->lb_ips_v4_routable) { + SSET_FOR_EACH (ip_address, &op->od->lb_ips->ips_v4_routable) { ds_put_format(&c_addresses, " %s", ip_address); central_ip_address = true; } - SSET_FOR_EACH (ip_address, &op->od->lb_ips_v6_routable) { + SSET_FOR_EACH (ip_address, &op->od->lb_ips->ips_v6_routable) { ds_put_format(&c_addresses, " %s", ip_address); central_ip_address = true; } } else { - SSET_FOR_EACH (ip_address, &op->od->lb_ips_v4) { + SSET_FOR_EACH (ip_address, &op->od->lb_ips->ips_v4) { ds_put_format(&c_addresses, " %s", ip_address); central_ip_address = true; } - SSET_FOR_EACH (ip_address, &op->od->lb_ips_v6) { + SSET_FOR_EACH (ip_address, &op->od->lb_ips->ips_v6) { ds_put_format(&c_addresses, " %s", ip_address); central_ip_address = true; } @@ -3822,20 +3811,21 @@ build_lb_vip_actions(struct ovn_lb_vip *lb_vip, } static void -build_lrouter_lb_ips(struct ovn_datapath *od, const struct ovn_northd_lb *lb) +build_lrouter_lb_ips(struct ovn_lb_ip_set *lb_ips, + const struct ovn_northd_lb *lb) { const char *ip_address; SSET_FOR_EACH (ip_address, &lb->ips_v4) { - sset_add(&od->lb_ips_v4, ip_address); + sset_add(&lb_ips->ips_v4, ip_address); if (lb->routable) { - sset_add(&od->lb_ips_v4_routable, ip_address); + sset_add(&lb_ips->ips_v4_routable, ip_address); } } SSET_FOR_EACH (ip_address, &lb->ips_v6) { - sset_add(&od->lb_ips_v6, ip_address); + sset_add(&lb_ips->ips_v6, ip_address); if (lb->routable) { - sset_add(&od->lb_ips_v6_routable, ip_address); + sset_add(&lb_ips->ips_v6_routable, ip_address); } } } @@ -3863,6 +3853,11 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, input_data->nbrec_load_balancer_group_table) { lb_group = ovn_lb_group_create(nbrec_lb_group, lbs, hmap_count(datapaths)); + + for (size_t i = 0; i < lb_group->n_lbs; i++) { + build_lrouter_lb_ips(lb_group->lb_ips, lb_group->lbs[i]); + } + hmap_insert(lb_groups, &lb_group->hmap_node, uuid_hash(&lb_group->uuid)); } @@ -3900,23 +3895,34 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, continue; } - for (size_t i = 0; i < od->nbr->n_load_balancer; i++) { - const struct uuid *lb_uuid = - &od->nbr->load_balancer[i]->header_.uuid; - lb = ovn_northd_lb_find(lbs, lb_uuid); - ovn_northd_lb_add_lr(lb, 1, &od); - build_lrouter_lb_ips(od, lb); - } - + /* Checking load balancer groups first to more efficiently copy + * IP sets for large groups. */ for (size_t i = 0; i < od->nbr->n_load_balancer_group; i++) { nbrec_lb_group = od->nbr->load_balancer_group[i]; lb_group = ovn_lb_group_find(lb_groups, &nbrec_lb_group->header_.uuid); ovn_lb_group_add_lr(lb_group, od); - for (size_t j = 0; j < lb_group->n_lbs; j++) { - build_lrouter_lb_ips(od, lb_group->lbs[j]); + + if (!od->lb_ips) { + od->lb_ips = ovn_lb_ip_set_clone(lb_group->lb_ips); + } else { + for (size_t j = 0; j < lb_group->n_lbs; j++) { + build_lrouter_lb_ips(od->lb_ips, lb_group->lbs[j]); + } } } + + if (!od->lb_ips) { + od->lb_ips = ovn_lb_ip_set_create(); + } + + for (size_t i = 0; i < od->nbr->n_load_balancer; i++) { + const struct uuid *lb_uuid = + &od->nbr->load_balancer[i]->header_.uuid; + lb = ovn_northd_lb_find(lbs, lb_uuid); + ovn_northd_lb_add_lr(lb, 1, &od); + build_lrouter_lb_ips(od->lb_ips, lb); + } } HMAP_FOR_EACH (lb_group, hmap_node, lb_groups) { @@ -3977,9 +3983,9 @@ build_lrouter_lb_reachable_ips(struct ovn_datapath *od, if (lb->neigh_mode == LB_NEIGH_RESPOND_ALL) { for (size_t i = 0; i < lb->n_vips; i++) { if (IN6_IS_ADDR_V4MAPPED(&lb->vips[i].vip)) { - sset_add(&od->lb_ips_v4_reachable, lb->vips[i].vip_str); + sset_add(&od->lb_ips->ips_v4_reachable, lb->vips[i].vip_str); } else { - sset_add(&od->lb_ips_v6_reachable, lb->vips[i].vip_str); + sset_add(&od->lb_ips->ips_v6_reachable, lb->vips[i].vip_str); } } return; @@ -3995,7 +4001,8 @@ build_lrouter_lb_reachable_ips(struct ovn_datapath *od, LIST_FOR_EACH (op, dp_node, &od->port_list) { if (lrouter_port_ipv4_reachable(op, vip_ip4)) { - sset_add(&od->lb_ips_v4_reachable, lb->vips[i].vip_str); + sset_add(&od->lb_ips->ips_v4_reachable, + lb->vips[i].vip_str); break; } } @@ -4004,7 +4011,8 @@ build_lrouter_lb_reachable_ips(struct ovn_datapath *od, LIST_FOR_EACH (op, dp_node, &od->port_list) { if (lrouter_port_ipv6_reachable(op, &lb->vips[i].vip)) { - sset_add(&od->lb_ips_v6_reachable, lb->vips[i].vip_str); + sset_add(&od->lb_ips->ips_v6_reachable, + lb->vips[i].vip_str); break; } } @@ -7467,7 +7475,7 @@ build_lswitch_rport_arp_req_flows(struct ovn_port *op, */ const char *ip_addr; - SSET_FOR_EACH (ip_addr, &op->od->lb_ips_v4) { + SSET_FOR_EACH (ip_addr, &op->od->lb_ips->ips_v4) { ovs_be32 ipv4_addr; /* Check if the ovn port has a network configured on which we could @@ -7480,7 +7488,7 @@ build_lswitch_rport_arp_req_flows(struct ovn_port *op, stage_hint); } } - SSET_FOR_EACH (ip_addr, &op->od->lb_ips_v6) { + SSET_FOR_EACH (ip_addr, &op->od->lb_ips->ips_v6) { struct in6_addr ipv6_addr; /* Check if the ovn port has a network configured on which we could @@ -7510,13 +7518,13 @@ build_lswitch_rport_arp_req_flows(struct ovn_port *op, * expect ARP requests/NS for the DNAT external_ip. */ if (nat_entry_is_v6(nat_entry)) { - if (!sset_contains(&op->od->lb_ips_v6, nat->external_ip)) { + if (!sset_contains(&op->od->lb_ips->ips_v6, nat->external_ip)) { build_lswitch_rport_arp_req_flow( nat->external_ip, AF_INET6, sw_op, sw_od, 80, lflows, stage_hint); } } else { - if (!sset_contains(&op->od->lb_ips_v4, nat->external_ip)) { + if (!sset_contains(&op->od->lb_ips->ips_v4, nat->external_ip)) { build_lswitch_rport_arp_req_flow( nat->external_ip, AF_INET, sw_op, sw_od, 80, lflows, stage_hint); @@ -10709,7 +10717,8 @@ build_lrouter_drop_own_dest(struct ovn_port *op, enum ovn_stage stage, const char *ip = op->lrp_networks.ipv4_addrs[i].addr_s; bool router_ip_in_snat_ips = !!shash_find(&op->od->snat_ips, ip); - bool router_ip_in_lb_ips = !!sset_find(&op->od->lb_ips_v4, ip); + bool router_ip_in_lb_ips = + !!sset_find(&op->od->lb_ips->ips_v4, ip); bool drop_router_ip = (drop_snat_ip == (router_ip_in_snat_ips || router_ip_in_lb_ips)); @@ -10737,7 +10746,8 @@ build_lrouter_drop_own_dest(struct ovn_port *op, enum ovn_stage stage, const char *ip = op->lrp_networks.ipv6_addrs[i].addr_s; bool router_ip_in_snat_ips = !!shash_find(&op->od->snat_ips, ip); - bool router_ip_in_lb_ips = !!sset_find(&op->od->lb_ips_v6, ip); + bool router_ip_in_lb_ips = + !!sset_find(&op->od->lb_ips->ips_v6, ip); bool drop_router_ip = (drop_snat_ip == (router_ip_in_snat_ips || router_ip_in_lb_ips)); @@ -12804,7 +12814,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, &op->nbrp->header_, lflows); } - if (sset_count(&op->od->lb_ips_v4_reachable)) { + if (sset_count(&op->od->lb_ips->ips_v4_reachable)) { ds_clear(match); if (is_l3dgw_port(op)) { ds_put_format(match, "is_chassis_resident(%s)", @@ -12819,7 +12829,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, free(lb_ips_v4_as); } - if (sset_count(&op->od->lb_ips_v6_reachable)) { + if (sset_count(&op->od->lb_ips->ips_v6_reachable)) { ds_clear(match); if (is_l3dgw_port(op)) { @@ -14632,23 +14642,25 @@ sync_address_sets(struct northd_input *input_data, continue; } - if (sset_count(&od->lb_ips_v4_reachable)) { + if (sset_count(&od->lb_ips->ips_v4_reachable)) { char *ipv4_addrs_name = lr_lb_address_set_name(od, AF_INET); - const char **ipv4_addrs = sset_array(&od->lb_ips_v4_reachable); + const char **ipv4_addrs = + sset_array(&od->lb_ips->ips_v4_reachable); sync_address_set(ovnsb_txn, ipv4_addrs_name, ipv4_addrs, - sset_count(&od->lb_ips_v4_reachable), + sset_count(&od->lb_ips->ips_v4_reachable), &sb_address_sets); free(ipv4_addrs_name); free(ipv4_addrs); } - if (sset_count(&od->lb_ips_v6_reachable)) { + if (sset_count(&od->lb_ips->ips_v6_reachable)) { char *ipv6_addrs_name = lr_lb_address_set_name(od, AF_INET6); - const char **ipv6_addrs = sset_array(&od->lb_ips_v6_reachable); + const char **ipv6_addrs = + sset_array(&od->lb_ips->ips_v6_reachable); sync_address_set(ovnsb_txn, ipv6_addrs_name, ipv6_addrs, - sset_count(&od->lb_ips_v6_reachable), + sset_count(&od->lb_ips->ips_v6_reachable), &sb_address_sets); free(ipv6_addrs_name); free(ipv6_addrs); diff --git a/northd/northd.h b/northd/northd.h index 8d299864f..0723d900a 100644 --- a/northd/northd.h +++ b/northd/northd.h @@ -236,16 +236,9 @@ struct ovn_datapath { struct lport_addresses dnat_force_snat_addrs; struct lport_addresses lb_force_snat_addrs; bool lb_force_snat_router_ip; - /* The "routable" ssets are subsets of the load balancer - * IPs for which IP routes and ARP resolution flows are automatically - * added - */ - struct sset lb_ips_v4; - struct sset lb_ips_v4_routable; - struct sset lb_ips_v4_reachable; - struct sset lb_ips_v6; - struct sset lb_ips_v6_routable; - struct sset lb_ips_v6_reachable; + + /* Load Balancer vIPs relevant for this datapath. */ + struct ovn_lb_ip_set *lb_ips; struct ovn_port **localnet_ports; size_t n_localnet_ports;