From patchwork Wed Aug 24 17:26:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1669907 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MCY104LkXz1ygF for ; Thu, 25 Aug 2022 03:26:28 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 85BD84179A; Wed, 24 Aug 2022 17:26:25 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 85BD84179A X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4b_x7kMsulo1; Wed, 24 Aug 2022 17:26:23 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 5DECE41760; Wed, 24 Aug 2022 17:26:21 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 5DECE41760 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 322DBC0035; Wed, 24 Aug 2022 17:26:21 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id DF2CFC0032 for ; Wed, 24 Aug 2022 17:26:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id AC40782AC3 for ; Wed, 24 Aug 2022 17:26:19 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org AC40782AC3 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vGoKdAyRSILU for ; Wed, 24 Aug 2022 17:26:18 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 7D7A082AAA Received: from relay11.mail.gandi.net (relay11.mail.gandi.net [217.70.178.231]) by smtp1.osuosl.org (Postfix) with ESMTPS id 7D7A082AAA for ; Wed, 24 Aug 2022 17:26:18 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by mail.gandi.net (Postfix) with ESMTPSA id A7D63100008; Wed, 24 Aug 2022 17:26:15 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Wed, 24 Aug 2022 19:26:05 +0200 Message-Id: <20220824172608.3092562-2-i.maximets@ovn.org> X-Mailer: git-send-email 2.34.3 In-Reply-To: <20220824172608.3092562-1-i.maximets@ovn.org> References: <20220824172608.3092562-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Dumitru Ceara , Ilya Maximets Subject: [ovs-dev] [PATCH ovn 1/4] northd: Optimize load balancer lookups for groups. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" ovn_northd_lb_lookup() takes significant percent of northd runtime in scenarios with large number of load balancers. In many cases, like in ovn-kubernetes, a lot of load balancers are actually groupped and applied to most of the switches and routers. So, instead of looking up all the same load balancers from the group for each datapath, we can look them up once and store as a group. Later we can lookup the entire group at once. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- northd/en-northd.c | 2 + northd/northd.c | 103 +++++++++++++++++++++++++++++++++------------ northd/northd.h | 3 ++ 3 files changed, 81 insertions(+), 27 deletions(-) diff --git a/northd/en-northd.c b/northd/en-northd.c index 4907a1ff2..7fe83db64 100644 --- a/northd/en-northd.c +++ b/northd/en-northd.c @@ -68,6 +68,8 @@ void en_northd_run(struct engine_node *node, void *data) EN_OVSDB_GET(engine_get_input("NB_logical_router", node)); input_data.nbrec_load_balancer_table = EN_OVSDB_GET(engine_get_input("NB_load_balancer", node)); + input_data.nbrec_load_balancer_group_table = + EN_OVSDB_GET(engine_get_input("NB_load_balancer_group", node)); input_data.nbrec_port_group_table = EN_OVSDB_GET(engine_get_input("NB_port_group", node)); input_data.nbrec_address_set_table = diff --git a/northd/northd.c b/northd/northd.c index 33943bfaf..b66843581 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -3841,13 +3841,37 @@ build_lrouter_lb_ips(struct ovn_datapath *od, const struct ovn_northd_lb *lb) } } +struct ovn_lb_group { + struct hmap_node hmap_node; + struct uuid uuid; + size_t n; + struct ovn_northd_lb *lbs[]; +}; + +static struct ovn_lb_group * +ovn_lb_group_find(struct hmap *lb_groups, const struct uuid *uuid) +{ + struct ovn_lb_group *lb_group; + size_t hash = uuid_hash(uuid); + + HMAP_FOR_EACH_WITH_HASH (lb_group, hmap_node, hash, lb_groups) { + if (uuid_equals(&lb_group->uuid, uuid)) { + return lb_group; + } + } + return NULL; +} + static void build_lbs(struct northd_input *input_data, struct hmap *datapaths, - struct hmap *lbs) + struct hmap *lbs, struct hmap *lb_groups) { + const struct nbrec_load_balancer_group *nbrec_lb_group; + struct ovn_lb_group *lb_group; struct ovn_northd_lb *lb; hmap_init(lbs); + hmap_init(lb_groups); const struct nbrec_load_balancer *nbrec_lb; NBREC_LOAD_BALANCER_TABLE_FOR_EACH (nbrec_lb, @@ -3857,6 +3881,25 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, uuid_hash(&nbrec_lb->header_.uuid)); } + NBREC_LOAD_BALANCER_GROUP_TABLE_FOR_EACH (nbrec_lb_group, + input_data->nbrec_load_balancer_group_table) { + size_t size = sizeof *lb_group + + nbrec_lb_group->n_load_balancer * sizeof(struct ovn_northd_lb *); + + lb_group = xzalloc(size); + lb_group->uuid = nbrec_lb_group->header_.uuid; + lb_group->n = nbrec_lb_group->n_load_balancer; + + for (size_t i = 0; i < nbrec_lb_group->n_load_balancer; i++) { + const struct uuid *lb_uuid = + &nbrec_lb_group->load_balancer[i]->header_.uuid; + lb_group->lbs[i] = ovn_northd_lb_find(lbs, lb_uuid); + } + + hmap_insert(lb_groups, &lb_group->hmap_node, + uuid_hash(&lb_group->uuid)); + } + struct ovn_datapath *od; HMAP_FOR_EACH (od, key_node, datapaths) { if (!od->nbs) { @@ -3871,13 +3914,11 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, } for (size_t i = 0; i < od->nbs->n_load_balancer_group; i++) { - const struct nbrec_load_balancer_group *lbg = - od->nbs->load_balancer_group[i]; - for (size_t j = 0; j < lbg->n_load_balancer; j++) { - const struct uuid *lb_uuid = - &lbg->load_balancer[j]->header_.uuid; - lb = ovn_northd_lb_find(lbs, lb_uuid); - ovn_northd_lb_add_ls(lb, od); + nbrec_lb_group = od->nbs->load_balancer_group[i]; + lb_group = ovn_lb_group_find(lb_groups, + &nbrec_lb_group->header_.uuid); + for (size_t j = 0; j < lb_group->n; j++) { + ovn_northd_lb_add_ls(lb_group->lbs[j], od); } } } @@ -3896,14 +3937,12 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, } for (size_t i = 0; i < od->nbr->n_load_balancer_group; i++) { - const struct nbrec_load_balancer_group *lbg = - od->nbr->load_balancer_group[i]; - for (size_t j = 0; j < lbg->n_load_balancer; j++) { - const struct uuid *lb_uuid = - &lbg->load_balancer[j]->header_.uuid; - lb = ovn_northd_lb_find(lbs, lb_uuid); - ovn_northd_lb_add_lr(lb, od); - build_lrouter_lb_ips(od, lb); + nbrec_lb_group = od->nbr->load_balancer_group[i]; + lb_group = ovn_lb_group_find(lb_groups, + &nbrec_lb_group->header_.uuid); + for (size_t j = 0; j < lb_group->n; j++) { + ovn_northd_lb_add_lr(lb_group->lbs[j], od); + build_lrouter_lb_ips(od, lb_group->lbs[j]); } } } @@ -4021,7 +4060,8 @@ build_lrouter_lbs_check(const struct hmap *datapaths) } static void -build_lrouter_lbs_reachable_ips(struct hmap *datapaths, struct hmap *lbs) +build_lrouter_lbs_reachable_ips(struct hmap *datapaths, struct hmap *lbs, + struct hmap *lb_groups) { struct ovn_datapath *od; @@ -4038,13 +4078,14 @@ build_lrouter_lbs_reachable_ips(struct hmap *datapaths, struct hmap *lbs) } for (size_t i = 0; i < od->nbr->n_load_balancer_group; i++) { - const struct nbrec_load_balancer_group *lbg = + const struct nbrec_load_balancer_group *nbrec_lb_group = od->nbr->load_balancer_group[i]; - for (size_t j = 0; j < lbg->n_load_balancer; j++) { - struct ovn_northd_lb *lb = - ovn_northd_lb_find(lbs, - &lbg->load_balancer[j]->header_.uuid); - build_lrouter_lb_reachable_ips(od, lb); + struct ovn_lb_group *lb_group; + + lb_group = ovn_lb_group_find(lb_groups, + &nbrec_lb_group->header_.uuid); + for (size_t j = 0; j < lb_group->n; j++) { + build_lrouter_lb_reachable_ips(od, lb_group->lbs[j]); } } } @@ -4105,11 +4146,12 @@ build_lswitch_lbs_from_lrouter(struct hmap *datapaths, struct hmap *lbs) */ static void build_lb_port_related_data(struct hmap *datapaths, struct hmap *ports, - struct hmap *lbs, struct northd_input *input_data, + struct hmap *lbs, struct hmap *lb_groups, + struct northd_input *input_data, struct ovsdb_idl_txn *ovnsb_txn) { build_lrouter_lbs_check(datapaths); - build_lrouter_lbs_reachable_ips(datapaths, lbs); + build_lrouter_lbs_reachable_ips(datapaths, lbs, lb_groups); build_lb_svcs(input_data, ovnsb_txn, ports, lbs); build_lswitch_lbs_from_lrouter(datapaths, lbs); } @@ -15340,6 +15382,7 @@ northd_init(struct northd_data *data) hmap_init(&data->port_groups); shash_init(&data->meter_groups); hmap_init(&data->lbs); + hmap_init(&data->lb_groups); hmap_init(&data->bfd_connections); ovs_list_init(&data->lr_list); data->features = (struct chassis_features) { @@ -15358,6 +15401,12 @@ northd_destroy(struct northd_data *data) } hmap_destroy(&data->lbs); + struct ovn_lb_group *lb_group; + HMAP_FOR_EACH_POP (lb_group, hmap_node, &data->lb_groups) { + free(lb_group); + } + hmap_destroy(&data->lb_groups); + struct ovn_port_group *pg; HMAP_FOR_EACH_SAFE (pg, key_node, &data->port_groups) { ovn_port_group_destroy(&data->port_groups, pg); @@ -15467,12 +15516,12 @@ ovnnb_db_run(struct northd_input *input_data, build_chassis_features(input_data, &data->features); build_datapaths(input_data, ovnsb_txn, &data->datapaths, &data->lr_list); - build_lbs(input_data, &data->datapaths, &data->lbs); + build_lbs(input_data, &data->datapaths, &data->lbs, &data->lb_groups); build_ports(input_data, ovnsb_txn, sbrec_chassis_by_name, sbrec_chassis_by_hostname, &data->datapaths, &data->ports); build_lb_port_related_data(&data->datapaths, &data->ports, &data->lbs, - input_data, ovnsb_txn); + &data->lb_groups, input_data, ovnsb_txn); build_ipam(&data->datapaths, &data->ports); build_port_group_lswitches(input_data, &data->port_groups, &data->ports); build_lrouter_groups(&data->ports, &data->lr_list); diff --git a/northd/northd.h b/northd/northd.h index d9856af97..8d299864f 100644 --- a/northd/northd.h +++ b/northd/northd.h @@ -28,6 +28,8 @@ struct northd_input { const struct nbrec_logical_switch_table *nbrec_logical_switch; const struct nbrec_logical_router_table *nbrec_logical_router; const struct nbrec_load_balancer_table *nbrec_load_balancer_table; + const struct nbrec_load_balancer_group_table + *nbrec_load_balancer_group_table; const struct nbrec_port_group_table *nbrec_port_group_table; const struct nbrec_address_set_table *nbrec_address_set_table; const struct nbrec_meter_table *nbrec_meter_table; @@ -74,6 +76,7 @@ struct northd_data { struct hmap port_groups; struct shash meter_groups; struct hmap lbs; + struct hmap lb_groups; struct hmap bfd_connections; struct ovs_list lr_list; bool ovn_internal_version_changed; From patchwork Wed Aug 24 17:26:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1669908 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MCY135VZrz1ygF for ; Thu, 25 Aug 2022 03:26:31 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 04AC960FEF; Wed, 24 Aug 2022 17:26:27 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 04AC960FEF X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WeQZLYPwjDox; Wed, 24 Aug 2022 17:26:26 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 0D8A260FDA; Wed, 24 Aug 2022 17:26:25 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 0D8A260FDA Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CA207C0035; Wed, 24 Aug 2022 17:26:24 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3AC23C002D for ; Wed, 24 Aug 2022 17:26:23 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id B7EF041759 for ; Wed, 24 Aug 2022 17:26:22 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org B7EF041759 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lkglek2Q0gbZ for ; Wed, 24 Aug 2022 17:26:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 4F3E8415DB Received: from relay11.mail.gandi.net (relay11.mail.gandi.net [217.70.178.231]) by smtp4.osuosl.org (Postfix) with ESMTPS id 4F3E8415DB for ; Wed, 24 Aug 2022 17:26:21 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by mail.gandi.net (Postfix) with ESMTPSA id C236E10000B; Wed, 24 Aug 2022 17:26:18 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Wed, 24 Aug 2022 19:26:06 +0200 Message-Id: <20220824172608.3092562-3-i.maximets@ovn.org> X-Mailer: git-send-email 2.34.3 In-Reply-To: <20220824172608.3092562-1-i.maximets@ovn.org> References: <20220824172608.3092562-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Dumitru Ceara , Ilya Maximets Subject: [ovs-dev] [PATCH ovn 2/4] northd: Add datapaths to load balancers in bulk. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Filling arrays of switches and routers for load balancers one-by-one is not very efficient. Copying them in bulk allows to save a noticeable amount of time in setups with large load balancer groups. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- lib/lb.c | 16 ++++++++++------ lib/lb.h | 8 ++++---- northd/northd.c | 41 +++++++++++++++++++++++++++++------------ 3 files changed, 43 insertions(+), 22 deletions(-) diff --git a/lib/lb.c b/lib/lb.c index 7b0ed1abe..fe6070a40 100644 --- a/lib/lb.c +++ b/lib/lb.c @@ -239,23 +239,27 @@ ovn_northd_lb_find(struct hmap *lbs, const struct uuid *uuid) } void -ovn_northd_lb_add_lr(struct ovn_northd_lb *lb, struct ovn_datapath *od) +ovn_northd_lb_add_lr(struct ovn_northd_lb *lb, size_t n, + struct ovn_datapath **ods) { - if (lb->n_allocated_nb_lr == lb->n_nb_lr) { + while (lb->n_allocated_nb_lr <= lb->n_nb_lr + n) { lb->nb_lr = x2nrealloc(lb->nb_lr, &lb->n_allocated_nb_lr, sizeof *lb->nb_lr); } - lb->nb_lr[lb->n_nb_lr++] = od; + memcpy(&lb->nb_lr[lb->n_nb_lr], ods, n * sizeof *ods); + lb->n_nb_lr += n; } void -ovn_northd_lb_add_ls(struct ovn_northd_lb *lb, struct ovn_datapath *od) +ovn_northd_lb_add_ls(struct ovn_northd_lb *lb, size_t n, + struct ovn_datapath **ods) { - if (lb->n_allocated_nb_ls == lb->n_nb_ls) { + while (lb->n_allocated_nb_ls <= lb->n_nb_ls + n) { lb->nb_ls = x2nrealloc(lb->nb_ls, &lb->n_allocated_nb_ls, sizeof *lb->nb_ls); } - lb->nb_ls[lb->n_nb_ls++] = od; + memcpy(&lb->nb_ls[lb->n_nb_ls], ods, n * sizeof *ods); + lb->n_nb_ls += n; } void diff --git a/lib/lb.h b/lib/lb.h index 832ed31fb..d7bc28e18 100644 --- a/lib/lb.h +++ b/lib/lb.h @@ -88,10 +88,10 @@ struct ovn_northd_lb_backend { struct ovn_northd_lb *ovn_northd_lb_create(const struct nbrec_load_balancer *); struct ovn_northd_lb * ovn_northd_lb_find(struct hmap *, const struct uuid *); void ovn_northd_lb_destroy(struct ovn_northd_lb *); -void -ovn_northd_lb_add_lr(struct ovn_northd_lb *lb, struct ovn_datapath *od); -void -ovn_northd_lb_add_ls(struct ovn_northd_lb *lb, struct ovn_datapath *od); +void ovn_northd_lb_add_lr(struct ovn_northd_lb *lb, size_t n, + struct ovn_datapath **ods); +void ovn_northd_lb_add_ls(struct ovn_northd_lb *lb, size_t n, + struct ovn_datapath **ods); struct ovn_controller_lb { const struct sbrec_load_balancer *slb; /* May be NULL. */ diff --git a/northd/northd.c b/northd/northd.c index b66843581..c202d27d5 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -3845,7 +3845,9 @@ struct ovn_lb_group { struct hmap_node hmap_node; struct uuid uuid; size_t n; - struct ovn_northd_lb *lbs[]; + struct ovn_northd_lb **lbs; + size_t n_od; + struct ovn_datapath **ods; }; static struct ovn_lb_group * @@ -3883,12 +3885,11 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, NBREC_LOAD_BALANCER_GROUP_TABLE_FOR_EACH (nbrec_lb_group, input_data->nbrec_load_balancer_group_table) { - size_t size = sizeof *lb_group + - nbrec_lb_group->n_load_balancer * sizeof(struct ovn_northd_lb *); - - lb_group = xzalloc(size); + lb_group = xzalloc(sizeof *lb_group); lb_group->uuid = nbrec_lb_group->header_.uuid; lb_group->n = nbrec_lb_group->n_load_balancer; + lb_group->lbs = xzalloc(lb_group->n * sizeof *lb_group->lbs); + lb_group->ods = xzalloc(hmap_count(datapaths) * sizeof *lb_group->ods); for (size_t i = 0; i < nbrec_lb_group->n_load_balancer; i++) { const struct uuid *lb_uuid = @@ -3910,19 +3911,25 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, const struct uuid *lb_uuid = &od->nbs->load_balancer[i]->header_.uuid; lb = ovn_northd_lb_find(lbs, lb_uuid); - ovn_northd_lb_add_ls(lb, od); + ovn_northd_lb_add_ls(lb, 1, &od); } for (size_t i = 0; i < od->nbs->n_load_balancer_group; i++) { nbrec_lb_group = od->nbs->load_balancer_group[i]; lb_group = ovn_lb_group_find(lb_groups, &nbrec_lb_group->header_.uuid); - for (size_t j = 0; j < lb_group->n; j++) { - ovn_northd_lb_add_ls(lb_group->lbs[j], od); - } + lb_group->ods[lb_group->n_od++] = od; } } + HMAP_FOR_EACH (lb_group, hmap_node, lb_groups) { + for (size_t j = 0; j < lb_group->n; j++) { + ovn_northd_lb_add_ls(lb_group->lbs[j], lb_group->n_od, + lb_group->ods); + } + lb_group->n_od = 0; + } + HMAP_FOR_EACH (od, key_node, datapaths) { if (!od->nbr) { continue; @@ -3932,7 +3939,7 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, const struct uuid *lb_uuid = &od->nbr->load_balancer[i]->header_.uuid; lb = ovn_northd_lb_find(lbs, lb_uuid); - ovn_northd_lb_add_lr(lb, od); + ovn_northd_lb_add_lr(lb, 1, &od); build_lrouter_lb_ips(od, lb); } @@ -3940,12 +3947,20 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, nbrec_lb_group = od->nbr->load_balancer_group[i]; lb_group = ovn_lb_group_find(lb_groups, &nbrec_lb_group->header_.uuid); + lb_group->ods[lb_group->n_od++] = od; for (size_t j = 0; j < lb_group->n; j++) { - ovn_northd_lb_add_lr(lb_group->lbs[j], od); build_lrouter_lb_ips(od, lb_group->lbs[j]); } } } + + HMAP_FOR_EACH (lb_group, hmap_node, lb_groups) { + for (size_t j = 0; j < lb_group->n; j++) { + ovn_northd_lb_add_lr(lb_group->lbs[j], lb_group->n_od, + lb_group->ods); + } + lb_group->n_od = 0; + } } static void @@ -4130,7 +4145,7 @@ build_lswitch_lbs_from_lrouter(struct hmap *datapaths, struct hmap *lbs) } } if (!installed) { - ovn_northd_lb_add_ls(lb, od); + ovn_northd_lb_add_ls(lb, 1, &od); } if (lb->nlb) { od->has_lb_vip |= lb_has_vip(lb->nlb); @@ -15403,6 +15418,8 @@ northd_destroy(struct northd_data *data) struct ovn_lb_group *lb_group; HMAP_FOR_EACH_POP (lb_group, hmap_node, &data->lb_groups) { + free(lb_group->lbs); + free(lb_group->ods); free(lb_group); } hmap_destroy(&data->lb_groups); From patchwork Wed Aug 24 17:26:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1669909 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MCY194z9lz1ygF for ; Thu, 25 Aug 2022 03:26:37 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id C564F40C88; Wed, 24 Aug 2022 17:26:35 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org C564F40C88 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wk9XeVI4Ccei; Wed, 24 Aug 2022 17:26:34 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id D489340BE4; Wed, 24 Aug 2022 17:26:33 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org D489340BE4 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 9BC56C0032; Wed, 24 Aug 2022 17:26:33 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2D294C002D for ; Wed, 24 Aug 2022 17:26:32 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 0ED774177D for ; Wed, 24 Aug 2022 17:26:26 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 0ED774177D X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sVSy8uIAqXqI for ; Wed, 24 Aug 2022 17:26:25 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 8C6E84179D Received: from relay11.mail.gandi.net (relay11.mail.gandi.net [217.70.178.231]) by smtp4.osuosl.org (Postfix) with ESMTPS id 8C6E84179D for ; Wed, 24 Aug 2022 17:26:24 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by mail.gandi.net (Postfix) with ESMTPSA id 94650100009; Wed, 24 Aug 2022 17:26:21 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Wed, 24 Aug 2022 19:26:07 +0200 Message-Id: <20220824172608.3092562-4-i.maximets@ovn.org> X-Mailer: git-send-email 2.34.3 In-Reply-To: <20220824172608.3092562-1-i.maximets@ovn.org> References: <20220824172608.3092562-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Dumitru Ceara , Ilya Maximets Subject: [ovs-dev] [PATCH ovn 3/4] northd: Retrieve load balancer options only once. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Options like 'add_route' and 'neighbor_responder' are looked up in the smap for each load balancer for each datapath and it takes significant amount of time. Checking them only once to speed up processing. 'skip_snat' is not that critical, but it's better to get all of them in a single place. Also using enum for the heighbor responder mode to avoid costly string comparison on a hot path. Signed-off-by: Ilya Maximets --- lib/lb.c | 7 +++++++ lib/lb.h | 9 +++++++++ northd/northd.c | 19 +++++++------------ 3 files changed, 23 insertions(+), 12 deletions(-) diff --git a/lib/lb.c b/lib/lb.c index fe6070a40..41564fe9b 100644 --- a/lib/lb.c +++ b/lib/lb.c @@ -173,6 +173,13 @@ ovn_northd_lb_create(const struct nbrec_load_balancer *nbrec_lb) lb->n_vips = smap_count(&nbrec_lb->vips); lb->vips = xcalloc(lb->n_vips, sizeof *lb->vips); lb->vips_nb = xcalloc(lb->n_vips, sizeof *lb->vips_nb); + lb->routable = smap_get_bool(&nbrec_lb->options, "add_route", false); + lb->skip_snat = smap_get_bool(&nbrec_lb->options, "skip_snat", false); + const char *mode = + smap_get_def(&nbrec_lb->options, "neighbor_responder", "reachable"); + lb->neighbor_responder_mode = strcmp(mode, "all") + ? LB_NEIGH_RESPOND_REACHABLE + : LB_NEIGH_RESPOND_ALL; sset_init(&lb->ips_v4); sset_init(&lb->ips_v6); struct smap_node *node; diff --git a/lib/lb.h b/lib/lb.h index d7bc28e18..add934fef 100644 --- a/lib/lb.h +++ b/lib/lb.h @@ -29,6 +29,11 @@ struct sbrec_datapath_binding; struct ovn_port; struct uuid; +enum lb_neighbor_responder_mode { + LB_NEIGH_RESPOND_REACHABLE, + LB_NEIGH_RESPOND_ALL, +}; + struct ovn_northd_lb { struct hmap_node hmap_node; @@ -40,6 +45,10 @@ struct ovn_northd_lb { struct ovn_northd_lb_vip *vips_nb; size_t n_vips; + bool routable; + bool skip_snat; + enum lb_neighbor_responder_mode neighbor_responder_mode; + struct sset ips_v4; struct sset ips_v6; diff --git a/northd/northd.c b/northd/northd.c index c202d27d5..a942346bd 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -3824,18 +3824,17 @@ build_lb_vip_actions(struct ovn_lb_vip *lb_vip, static void build_lrouter_lb_ips(struct ovn_datapath *od, const struct ovn_northd_lb *lb) { - bool is_routable = smap_get_bool(&lb->nlb->options, "add_route", false); const char *ip_address; SSET_FOR_EACH (ip_address, &lb->ips_v4) { sset_add(&od->lb_ips_v4, ip_address); - if (is_routable) { + if (lb->routable) { sset_add(&od->lb_ips_v4_routable, ip_address); } } SSET_FOR_EACH (ip_address, &lb->ips_v6) { sset_add(&od->lb_ips_v6, ip_address); - if (is_routable) { + if (lb->routable) { sset_add(&od->lb_ips_v6_routable, ip_address); } } @@ -4007,13 +4006,10 @@ static void build_lrouter_lb_reachable_ips(struct ovn_datapath *od, const struct ovn_northd_lb *lb) { - const char *neighbor_responder_mode = - smap_get_def(&lb->nlb->options, "neighbor_responder", "reachable"); - /* If configured to reply to neighbor requests for all VIPs force them * all to be considered "reachable". */ - if (!strcmp(neighbor_responder_mode, "all")) { + if (lb->neighbor_responder_mode == LB_NEIGH_RESPOND_ALL) { for (size_t i = 0; i < lb->n_vips; i++) { if (IN6_IS_ADDR_V4MAPPED(&lb->vips[i].vip)) { sset_add(&od->lb_ips_v4_reachable, lb->vips[i].vip_str); @@ -9958,8 +9954,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip, lb_vip->vip_str); } - bool lb_skip_snat = smap_get_bool(&lb->nlb->options, "skip_snat", false); - if (lb_skip_snat) { + if (lb->skip_snat) { skip_snat_new_action = xasprintf("flags.skip_snat_for_lb = 1; %s", ds_cstr(action)); skip_snat_est_action = xasprintf("flags.skip_snat_for_lb = 1; " @@ -10043,7 +10038,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip, for (size_t i = 0; i < lb->n_nb_lr; i++) { struct ovn_datapath *od = lb->nb_lr[i]; if (!od->n_l3dgw_ports) { - if (lb_skip_snat) { + if (lb->skip_snat) { gw_router_skip_snat[n_gw_router_skip_snat++] = od; } else if (!lport_addresses_is_empty(&od->lb_force_snat_addrs) || od->lb_force_snat_router_ip) { @@ -10114,7 +10109,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip, od->l3dgw_ports[0]->cr_port->json_key); } - if (lb_skip_snat) { + if (lb->skip_snat) { ovn_lflow_add_with_hint__(lflows, od, S_ROUTER_IN_DNAT, prio, new_match_p, skip_snat_new_action, NULL, meter, &lb->nlb->header_); @@ -10154,7 +10149,7 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip, ds_cstr(&undnat_match), od->l3dgw_ports[0]->json_key, od->l3dgw_ports[0]->cr_port->json_key); - if (lb_skip_snat) { + if (lb->skip_snat) { ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120, undnat_match_p, skip_snat_est_action, &lb->nlb->header_); From patchwork Wed Aug 24 17:26:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1669910 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MCY1P2DBxz1yhJ for ; Thu, 25 Aug 2022 03:26:49 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 1AA9A82CDE; Wed, 24 Aug 2022 17:26:47 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 1AA9A82CDE X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id whPkK0D0auKo; Wed, 24 Aug 2022 17:26:45 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 9F22282BFD; Wed, 24 Aug 2022 17:26:44 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 9F22282BFD Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4A39FC007D; Wed, 24 Aug 2022 17:26:44 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 065DCC002D for ; Wed, 24 Aug 2022 17:26:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 9132D417BD for ; Wed, 24 Aug 2022 17:26:30 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 9132D417BD X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z8EJT7RoTPzX for ; Wed, 24 Aug 2022 17:26:29 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 8993841770 Received: from relay11.mail.gandi.net (relay11.mail.gandi.net [IPv6:2001:4b98:dc4:8::231]) by smtp4.osuosl.org (Postfix) with ESMTPS id 8993841770 for ; Wed, 24 Aug 2022 17:26:28 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by mail.gandi.net (Postfix) with ESMTPSA id 72B91100008; Wed, 24 Aug 2022 17:26:25 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Wed, 24 Aug 2022 19:26:08 +0200 Message-Id: <20220824172608.3092562-5-i.maximets@ovn.org> X-Mailer: git-send-email 2.34.3 In-Reply-To: <20220824172608.3092562-1-i.maximets@ovn.org> References: <20220824172608.3092562-1-i.maximets@ovn.org> MIME-Version: 1.0 Cc: Dumitru Ceara , Ilya Maximets Subject: [ovs-dev] [PATCH ovn 4/4] northd: Re-use IP sets created for load balancer groups. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" If the same load balancer group is attached to multiple routers, IP sets will be constructed for each of them by sequentially calling sset_add() for each IP for each load balancer for each router. Instead of doing that, we can create IP sets for load balancer groups and clone them. That will avoid extra sset_find__() call making the construction of IP sets about 2x faster. Only first load balancer group is cloned, the rest as well as standalone load balancers are still added one by one, because there is no more efficient way to merge sets. The order of processing changed to make sure that we're actually optimizing copy of a large group. The code can be optimized further by introduction of a reference counter and not copying at all if the router has no standalone load balancers and only one group, but that is not the common case, unfortunately. Also, construction of "reachable" sets can be optimized for the "neighbor_responder = all" case. But, for now, only moved to a new structure for better readability. Signed-off-by: Ilya Maximets Acked-by: Dumitru Ceara --- northd/northd.c | 156 +++++++++++++++++++++++++++++++++--------------- northd/northd.h | 24 ++++---- 2 files changed, 121 insertions(+), 59 deletions(-) diff --git a/northd/northd.c b/northd/northd.c index a942346bd..4f8ffc124 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -818,16 +818,13 @@ lr_lb_address_set_ref(const struct ovn_datapath *od, int addr_family) return lr_lb_address_set_name_(od, "$", addr_family); } +static struct ovn_lb_ip_set *ovn_lb_ip_set_create(void); +static void ovn_lb_ip_set_destroy(struct ovn_lb_ip_set *); +static struct ovn_lb_ip_set *ovn_lb_ip_set_clone(struct ovn_lb_ip_set *); + static void init_lb_for_datapath(struct ovn_datapath *od) { - sset_init(&od->lb_ips_v4); - sset_init(&od->lb_ips_v4_routable); - sset_init(&od->lb_ips_v4_reachable); - sset_init(&od->lb_ips_v6); - sset_init(&od->lb_ips_v6_routable); - sset_init(&od->lb_ips_v6_reachable); - if (od->nbs) { od->has_lb_vip = ls_has_lb_vip(od); } else { @@ -838,16 +835,12 @@ init_lb_for_datapath(struct ovn_datapath *od) static void destroy_lb_for_datapath(struct ovn_datapath *od) { + ovn_lb_ip_set_destroy(od->lb_ips); + od->lb_ips = NULL; + if (!od->nbs && !od->nbr) { return; } - - sset_destroy(&od->lb_ips_v4); - sset_destroy(&od->lb_ips_v4_routable); - sset_destroy(&od->lb_ips_v4_reachable); - sset_destroy(&od->lb_ips_v6); - sset_destroy(&od->lb_ips_v6_routable); - sset_destroy(&od->lb_ips_v6_reachable); } /* A group of logical router datapaths which are connected - either @@ -2818,20 +2811,20 @@ get_nat_addresses(const struct ovn_port *op, size_t *n, bool routable_only, if (include_lb_ips) { const char *ip_address; if (routable_only) { - SSET_FOR_EACH (ip_address, &op->od->lb_ips_v4_routable) { + SSET_FOR_EACH (ip_address, &op->od->lb_ips->ips_v4_routable) { ds_put_format(&c_addresses, " %s", ip_address); central_ip_address = true; } - SSET_FOR_EACH (ip_address, &op->od->lb_ips_v6_routable) { + SSET_FOR_EACH (ip_address, &op->od->lb_ips->ips_v6_routable) { ds_put_format(&c_addresses, " %s", ip_address); central_ip_address = true; } } else { - SSET_FOR_EACH (ip_address, &op->od->lb_ips_v4) { + SSET_FOR_EACH (ip_address, &op->od->lb_ips->ips_v4) { ds_put_format(&c_addresses, " %s", ip_address); central_ip_address = true; } - SSET_FOR_EACH (ip_address, &op->od->lb_ips_v6) { + SSET_FOR_EACH (ip_address, &op->od->lb_ips->ips_v6) { ds_put_format(&c_addresses, " %s", ip_address); central_ip_address = true; } @@ -3822,29 +3815,76 @@ build_lb_vip_actions(struct ovn_lb_vip *lb_vip, } static void -build_lrouter_lb_ips(struct ovn_datapath *od, const struct ovn_northd_lb *lb) +build_lrouter_lb_ips(struct ovn_lb_ip_set *lb_ips, + const struct ovn_northd_lb *lb) { const char *ip_address; SSET_FOR_EACH (ip_address, &lb->ips_v4) { - sset_add(&od->lb_ips_v4, ip_address); + sset_add(&lb_ips->ips_v4, ip_address); if (lb->routable) { - sset_add(&od->lb_ips_v4_routable, ip_address); + sset_add(&lb_ips->ips_v4_routable, ip_address); } } SSET_FOR_EACH (ip_address, &lb->ips_v6) { - sset_add(&od->lb_ips_v6, ip_address); + sset_add(&lb_ips->ips_v6, ip_address); if (lb->routable) { - sset_add(&od->lb_ips_v6_routable, ip_address); + sset_add(&lb_ips->ips_v6_routable, ip_address); } } } +static struct ovn_lb_ip_set * +ovn_lb_ip_set_create(void) +{ + struct ovn_lb_ip_set *lb_ip_set = xzalloc(sizeof *lb_ip_set); + + sset_init(&lb_ip_set->ips_v4); + sset_init(&lb_ip_set->ips_v4_routable); + sset_init(&lb_ip_set->ips_v4_reachable); + sset_init(&lb_ip_set->ips_v6); + sset_init(&lb_ip_set->ips_v6_routable); + sset_init(&lb_ip_set->ips_v6_reachable); + + return lb_ip_set; +} + +static void +ovn_lb_ip_set_destroy(struct ovn_lb_ip_set *lb_ip_set) +{ + if (!lb_ip_set) { + return; + } + sset_destroy(&lb_ip_set->ips_v4); + sset_destroy(&lb_ip_set->ips_v4_routable); + sset_destroy(&lb_ip_set->ips_v4_reachable); + sset_destroy(&lb_ip_set->ips_v6); + sset_destroy(&lb_ip_set->ips_v6_routable); + sset_destroy(&lb_ip_set->ips_v6_reachable); + free(lb_ip_set); +} + +static struct ovn_lb_ip_set * +ovn_lb_ip_set_clone(struct ovn_lb_ip_set *lb_ip_set) +{ + struct ovn_lb_ip_set *clone = ovn_lb_ip_set_create(); + + sset_clone(&clone->ips_v4, &lb_ip_set->ips_v4); + sset_clone(&clone->ips_v4_routable, &lb_ip_set->ips_v4_routable); + sset_clone(&clone->ips_v4_reachable, &lb_ip_set->ips_v4_reachable); + sset_clone(&clone->ips_v6, &lb_ip_set->ips_v6); + sset_clone(&clone->ips_v6_routable, &lb_ip_set->ips_v6_routable); + sset_clone(&clone->ips_v6_reachable, &lb_ip_set->ips_v6_reachable); + + return clone; +} + struct ovn_lb_group { struct hmap_node hmap_node; struct uuid uuid; size_t n; struct ovn_northd_lb **lbs; + struct ovn_lb_ip_set *lb_ips; size_t n_od; struct ovn_datapath **ods; }; @@ -3889,11 +3929,13 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, lb_group->n = nbrec_lb_group->n_load_balancer; lb_group->lbs = xzalloc(lb_group->n * sizeof *lb_group->lbs); lb_group->ods = xzalloc(hmap_count(datapaths) * sizeof *lb_group->ods); + lb_group->lb_ips = ovn_lb_ip_set_create(); for (size_t i = 0; i < nbrec_lb_group->n_load_balancer; i++) { const struct uuid *lb_uuid = &nbrec_lb_group->load_balancer[i]->header_.uuid; lb_group->lbs[i] = ovn_northd_lb_find(lbs, lb_uuid); + build_lrouter_lb_ips(lb_group->lb_ips, lb_group->lbs[i]); } hmap_insert(lb_groups, &lb_group->hmap_node, @@ -3934,23 +3976,34 @@ build_lbs(struct northd_input *input_data, struct hmap *datapaths, continue; } - for (size_t i = 0; i < od->nbr->n_load_balancer; i++) { - const struct uuid *lb_uuid = - &od->nbr->load_balancer[i]->header_.uuid; - lb = ovn_northd_lb_find(lbs, lb_uuid); - ovn_northd_lb_add_lr(lb, 1, &od); - build_lrouter_lb_ips(od, lb); - } - + /* Checking load balancer groups first to more efficiently copy + * IP sets for large groups. */ for (size_t i = 0; i < od->nbr->n_load_balancer_group; i++) { nbrec_lb_group = od->nbr->load_balancer_group[i]; lb_group = ovn_lb_group_find(lb_groups, &nbrec_lb_group->header_.uuid); lb_group->ods[lb_group->n_od++] = od; - for (size_t j = 0; j < lb_group->n; j++) { - build_lrouter_lb_ips(od, lb_group->lbs[j]); + + if (!od->lb_ips) { + od->lb_ips = ovn_lb_ip_set_clone(lb_group->lb_ips); + } else { + for (size_t j = 0; j < lb_group->n; j++) { + build_lrouter_lb_ips(od->lb_ips, lb_group->lbs[j]); + } } } + + if (!od->lb_ips) { + od->lb_ips = ovn_lb_ip_set_create(); + } + + for (size_t i = 0; i < od->nbr->n_load_balancer; i++) { + const struct uuid *lb_uuid = + &od->nbr->load_balancer[i]->header_.uuid; + lb = ovn_northd_lb_find(lbs, lb_uuid); + ovn_northd_lb_add_lr(lb, 1, &od); + build_lrouter_lb_ips(od->lb_ips, lb); + } } HMAP_FOR_EACH (lb_group, hmap_node, lb_groups) { @@ -4012,9 +4065,9 @@ build_lrouter_lb_reachable_ips(struct ovn_datapath *od, if (lb->neighbor_responder_mode == LB_NEIGH_RESPOND_ALL) { for (size_t i = 0; i < lb->n_vips; i++) { if (IN6_IS_ADDR_V4MAPPED(&lb->vips[i].vip)) { - sset_add(&od->lb_ips_v4_reachable, lb->vips[i].vip_str); + sset_add(&od->lb_ips->ips_v4_reachable, lb->vips[i].vip_str); } else { - sset_add(&od->lb_ips_v6_reachable, lb->vips[i].vip_str); + sset_add(&od->lb_ips->ips_v6_reachable, lb->vips[i].vip_str); } } return; @@ -4030,7 +4083,8 @@ build_lrouter_lb_reachable_ips(struct ovn_datapath *od, LIST_FOR_EACH (op, dp_node, &od->port_list) { if (lrouter_port_ipv4_reachable(op, vip_ip4)) { - sset_add(&od->lb_ips_v4_reachable, lb->vips[i].vip_str); + sset_add(&od->lb_ips->ips_v4_reachable, + lb->vips[i].vip_str); break; } } @@ -4039,7 +4093,8 @@ build_lrouter_lb_reachable_ips(struct ovn_datapath *od, LIST_FOR_EACH (op, dp_node, &od->port_list) { if (lrouter_port_ipv6_reachable(op, &lb->vips[i].vip)) { - sset_add(&od->lb_ips_v6_reachable, lb->vips[i].vip_str); + sset_add(&od->lb_ips->ips_v6_reachable, + lb->vips[i].vip_str); break; } } @@ -7476,7 +7531,7 @@ build_lswitch_rport_arp_req_flows(struct ovn_port *op, */ const char *ip_addr; - SSET_FOR_EACH (ip_addr, &op->od->lb_ips_v4) { + SSET_FOR_EACH (ip_addr, &op->od->lb_ips->ips_v4) { ovs_be32 ipv4_addr; /* Check if the ovn port has a network configured on which we could @@ -7489,7 +7544,7 @@ build_lswitch_rport_arp_req_flows(struct ovn_port *op, stage_hint); } } - SSET_FOR_EACH (ip_addr, &op->od->lb_ips_v6) { + SSET_FOR_EACH (ip_addr, &op->od->lb_ips->ips_v6) { struct in6_addr ipv6_addr; /* Check if the ovn port has a network configured on which we could @@ -7519,13 +7574,13 @@ build_lswitch_rport_arp_req_flows(struct ovn_port *op, * expect ARP requests/NS for the DNAT external_ip. */ if (nat_entry_is_v6(nat_entry)) { - if (!sset_contains(&op->od->lb_ips_v6, nat->external_ip)) { + if (!sset_contains(&op->od->lb_ips->ips_v6, nat->external_ip)) { build_lswitch_rport_arp_req_flow( nat->external_ip, AF_INET6, sw_op, sw_od, 80, lflows, stage_hint); } } else { - if (!sset_contains(&op->od->lb_ips_v4, nat->external_ip)) { + if (!sset_contains(&op->od->lb_ips->ips_v4, nat->external_ip)) { build_lswitch_rport_arp_req_flow( nat->external_ip, AF_INET, sw_op, sw_od, 80, lflows, stage_hint); @@ -12807,7 +12862,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, &op->nbrp->header_, lflows); } - if (sset_count(&op->od->lb_ips_v4_reachable)) { + if (sset_count(&op->od->lb_ips->ips_v4_reachable)) { ds_clear(match); if (is_l3dgw_port(op)) { ds_put_format(match, "is_chassis_resident(%s)", @@ -12822,7 +12877,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op, free(lb_ips_v4_as); } - if (sset_count(&op->od->lb_ips_v6_reachable)) { + if (sset_count(&op->od->lb_ips->ips_v6_reachable)) { ds_clear(match); if (is_l3dgw_port(op)) { @@ -14635,23 +14690,25 @@ sync_address_sets(struct northd_input *input_data, continue; } - if (sset_count(&od->lb_ips_v4_reachable)) { + if (sset_count(&od->lb_ips->ips_v4_reachable)) { char *ipv4_addrs_name = lr_lb_address_set_name(od, AF_INET); - const char **ipv4_addrs = sset_array(&od->lb_ips_v4_reachable); + const char **ipv4_addrs = + sset_array(&od->lb_ips->ips_v4_reachable); sync_address_set(ovnsb_txn, ipv4_addrs_name, ipv4_addrs, - sset_count(&od->lb_ips_v4_reachable), + sset_count(&od->lb_ips->ips_v4_reachable), &sb_address_sets); free(ipv4_addrs_name); free(ipv4_addrs); } - if (sset_count(&od->lb_ips_v6_reachable)) { + if (sset_count(&od->lb_ips->ips_v6_reachable)) { char *ipv6_addrs_name = lr_lb_address_set_name(od, AF_INET6); - const char **ipv6_addrs = sset_array(&od->lb_ips_v6_reachable); + const char **ipv6_addrs = + sset_array(&od->lb_ips->ips_v6_reachable); sync_address_set(ovnsb_txn, ipv6_addrs_name, ipv6_addrs, - sset_count(&od->lb_ips_v6_reachable), + sset_count(&od->lb_ips->ips_v6_reachable), &sb_address_sets); free(ipv6_addrs_name); free(ipv6_addrs); @@ -15413,6 +15470,7 @@ northd_destroy(struct northd_data *data) struct ovn_lb_group *lb_group; HMAP_FOR_EACH_POP (lb_group, hmap_node, &data->lb_groups) { + ovn_lb_ip_set_destroy(lb_group->lb_ips); free(lb_group->lbs); free(lb_group->ods); free(lb_group); diff --git a/northd/northd.h b/northd/northd.h index 8d299864f..16ca0e051 100644 --- a/northd/northd.h +++ b/northd/northd.h @@ -174,6 +174,17 @@ struct mcast_port_info { * (e.g., IGMP join/leave). */ }; +/* The "routable" ssets are subsets of the load balancer IPs for which IP + * routes and ARP resolution flows are automatically added. */ +struct ovn_lb_ip_set { + struct sset ips_v4; + struct sset ips_v4_routable; + struct sset ips_v4_reachable; + struct sset ips_v6; + struct sset ips_v6_routable; + struct sset ips_v6_reachable; +}; + /* The 'key' comes from nbs->header_.uuid or nbr->header_.uuid or * sb->external_ids:logical-switch. */ struct ovn_datapath { @@ -236,16 +247,9 @@ struct ovn_datapath { struct lport_addresses dnat_force_snat_addrs; struct lport_addresses lb_force_snat_addrs; bool lb_force_snat_router_ip; - /* The "routable" ssets are subsets of the load balancer - * IPs for which IP routes and ARP resolution flows are automatically - * added - */ - struct sset lb_ips_v4; - struct sset lb_ips_v4_routable; - struct sset lb_ips_v4_reachable; - struct sset lb_ips_v6; - struct sset lb_ips_v6_routable; - struct sset lb_ips_v6_reachable; + + /* Load Balancer vIPs relevant for this datapath. */ + struct ovn_lb_ip_set *lb_ips; struct ovn_port **localnet_ports; size_t n_localnet_ports;