From patchwork Thu Sep 8 15:46:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 1675715 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MNk4v5wqDz1yj1 for ; Fri, 9 Sep 2022 01:46:39 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id DE22F6F8D9; Thu, 8 Sep 2022 15:46:33 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org DE22F6F8D9 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id L59W-LXjVSxc; Thu, 8 Sep 2022 15:46:32 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id D053D61376; Thu, 8 Sep 2022 15:46:31 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org D053D61376 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 7FF81C0033; Thu, 8 Sep 2022 15:46:31 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4FA7AC002D for ; Thu, 8 Sep 2022 15:46:30 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 16042841B4 for ; Thu, 8 Sep 2022 15:46:30 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 16042841B4 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LvKSiJ5Mtfzq for ; Thu, 8 Sep 2022 15:46:29 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 88CCB8404E Received: from relay7-d.mail.gandi.net (relay7-d.mail.gandi.net [217.70.183.200]) by smtp1.osuosl.org (Postfix) with ESMTPS id 88CCB8404E for ; Thu, 8 Sep 2022 15:46:28 +0000 (UTC) Received: (Authenticated sender: i.maximets@ovn.org) by mail.gandi.net (Postfix) with ESMTPSA id 6D04320004; Thu, 8 Sep 2022 15:46:24 +0000 (UTC) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Thu, 8 Sep 2022 17:46:20 +0200 Message-Id: <20220908154620.681323-1-i.maximets@ovn.org> X-Mailer: git-send-email 2.34.3 MIME-Version: 1.0 Cc: Dumitru Ceara , Ilya Maximets Subject: [ovs-dev] [PATCH ovn] northd: Accumulate more database updates before processing. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" northd doesn't process changes incrementally, so it makes sense to accumulate more database updates and process them in bulk, so we can cover everything in a single recompute. ovsdb-server has a mechanism to start accumulating changes if the client doesn't receive them fast enough, but it relies on the receive buffer size, which is a few hundreds of KB on a typical system. Unfortunately, that is enough to queue up several hundreds of small updates, and it takes northd a lot of time to process them if poll intervals are large, receiving at most 50 messages on each iteration (half of which are updates for a _Server database). Calling ovsdb_idl_run() as long as something changes. This allows to quickly process large bursts of database updates. For example, it takes only 30-40 seconds for 'ovn-nbctl --wait=hv sync' to finish on a 500-node cluster after the startup phase of the density-heavy ovn-heater test, instead of 6-8 minutes without this change. 500 ms seems like a reasonable hard limit to avoid spinning for too long if the database is changed constantly at a fast pace. Very long polling is also logged at INFO level to notify users. Not using WARN or higher because it may happen under normal conditions, e.g. on the initial connection to a large database or another type of a single large update. Other notable polling attempts are logged at debug level. Signed-off-by: Ilya Maximets --- northd/ovn-northd.c | 38 ++++++++++++++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 4 deletions(-) diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c index bd35802ed..96f17f15f 100644 --- a/northd/ovn-northd.c +++ b/northd/ovn-northd.c @@ -685,6 +685,36 @@ get_probe_interval(const char *db, const struct nbrec_nb_global *nb) return interval; } +static struct ovsdb_idl_txn * +run_idl_loop(struct ovsdb_idl_loop *idl_loop, const char *name) +{ + unsigned long long duration, start = time_msec(); + unsigned int seqno = UINT_MAX; + struct ovsdb_idl_txn *txn; + int n = 0; + + /* Accumulate database changes as long as there are some, + * but no longer than half a second. */ + while (seqno != ovsdb_idl_get_seqno(idl_loop->idl) + && time_msec() - start < 500) { + seqno = ovsdb_idl_get_seqno(idl_loop->idl); + ovsdb_idl_run(idl_loop->idl); + n++; + } + + txn = ovsdb_idl_loop_run(idl_loop); + + duration = time_msec() - start; + /* ovsdb_idl_run() is called at least 2 times. Once directly and + * once in the ovsdb_idl_loop_run(). n > 2 means that we received + * data on at least 2 subsequent calls. */ + if (n > 2 || duration > 100) { + VLOG(duration > 500 ? VLL_INFO : VLL_DBG, + "%s IDL run: %d iterations in %lld ms", name, n + 1, duration); + } + return txn; +} + int main(int argc, char *argv[]) { @@ -821,8 +851,8 @@ main(int argc, char *argv[]) ovsdb_idl_set_lock(ovnsb_idl_loop.idl, "ovn_northd"); } - struct ovsdb_idl_txn *ovnnb_txn = - ovsdb_idl_loop_run(&ovnnb_idl_loop); + struct ovsdb_idl_txn *ovnnb_txn = run_idl_loop(&ovnnb_idl_loop, + "OVN_Northbound"); unsigned int new_ovnnb_cond_seqno = ovsdb_idl_get_condition_seqno(ovnnb_idl_loop.idl); if (new_ovnnb_cond_seqno != ovnnb_cond_seqno) { @@ -833,8 +863,8 @@ main(int argc, char *argv[]) ovnnb_cond_seqno = new_ovnnb_cond_seqno; } - struct ovsdb_idl_txn *ovnsb_txn = - ovsdb_idl_loop_run(&ovnsb_idl_loop); + struct ovsdb_idl_txn *ovnsb_txn = run_idl_loop(&ovnsb_idl_loop, + "OVN_Southbound"); unsigned int new_ovnsb_cond_seqno = ovsdb_idl_get_condition_seqno(ovnsb_idl_loop.idl); if (new_ovnsb_cond_seqno != ovnsb_cond_seqno) {