From patchwork Mon Aug 23 20:02:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1519969 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Gtjp76CzHz9sWX for ; Tue, 24 Aug 2021 06:02:39 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 0A6EA60782; Mon, 23 Aug 2021 20:02:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id m6F3tTxZ1xs2; Mon, 23 Aug 2021 20:02:34 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 51E7360780; Mon, 23 Aug 2021 20:02:33 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2C61CC001A; Mon, 23 Aug 2021 20:02:33 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id CD1D9C000E for ; Mon, 23 Aug 2021 20:02:31 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id A963960782 for ; Mon, 23 Aug 2021 20:02:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z5ni1NY41o2g for ; Mon, 23 Aug 2021 20:02:31 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay9-d.mail.gandi.net (relay9-d.mail.gandi.net [217.70.183.199]) by smtp3.osuosl.org (Postfix) with ESMTPS id AD03960780 for ; Mon, 23 Aug 2021 20:02:30 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by relay9-d.mail.gandi.net (Postfix) with ESMTPSA id 7EF74FF803; Mon, 23 Aug 2021 20:02:26 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Mon, 23 Aug 2021 22:02:23 +0200 Message-Id: <20210823200223.680933-1-i.maximets@ovn.org> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Cc: Anton Ivanov , Ilya Maximets Subject: [ovs-dev] [PATCH ovn] ovn-northd: Fix extremely inefficient usage of lflow hash map. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" 'lflow_map' is never expanded because northd always uses fast insertion. This leads to the case where we have a hash map with only 128 initial buckets and every ovn_lflow_find() ends up iterating over n_lflows / 128 entries. It's thousands of logical flows or even more. For example, it takes forever for ovn-northd to start up with the Northbound Db from the 120 node density-heavy test from ovn-heater, because every lookup is slower than previous one. I aborted the process after 15 minutes of waiting, because there was no sign that it will converge. With this change applied the loop completes in only 11 seconds. Hash map will be pre-allocated to the maximum seen number of logical flows on a second iteration, but this doesn't help for the first iteration when northd first time connects to a big Northbound database, which is a common case during failover or cluster upgrade. And there is an even trickier case where big NbDB transaction that explodes the number of logical flows received on not the first run. We can't expand the hash map in case of parallel build, so this should be fixed separately. CC: Anton Ivanov Fixes: 74daa0607c7f ("ovn-northd: Introduce parallel lflow build") Signed-off-by: Ilya Maximets --- northd/ovn-northd.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c index 3d8e21a4f..40cf957c0 100644 --- a/northd/ovn-northd.c +++ b/northd/ovn-northd.c @@ -4387,7 +4387,11 @@ do_ovn_lflow_add(struct hmap *lflow_map, struct ovn_datapath *od, nullable_xstrdup(ctrl_meter), ovn_lflow_hint(stage_hint), where); hmapx_add(&lflow->od_group, od); - hmap_insert_fast(lflow_map, &lflow->hmap_node, hash); + if (!use_parallel_build) { + hmap_insert(lflow_map, &lflow->hmap_node, hash); + } else { + hmap_insert_fast(lflow_map, &lflow->hmap_node, hash); + } } /* Adds a row with the specified contents to the Logical_Flow table. */