From patchwork Fri Mar 15 10:04:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: LIU Yulong X-Patchwork-Id: 1912431 X-Patchwork-Delegate: i.maximets@samsung.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Tx0Ht0DZnz1yWn for ; Fri, 15 Mar 2024 21:05:44 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 46D44417F6; Fri, 15 Mar 2024 10:05:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XPviWKuDQfEY; Fri, 15 Mar 2024 10:05:40 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 2647140124 Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 2647140124; Fri, 15 Mar 2024 10:05:40 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id DF9C0C0077; Fri, 15 Mar 2024 10:05:39 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 15742C0037 for ; Fri, 15 Mar 2024 10:05:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id EB32681F1E for ; Fri, 15 Mar 2024 10:05:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0J6xlRwGdYH4 for ; Fri, 15 Mar 2024 10:05:37 +0000 (UTC) X-Greylist: delayed 114919 seconds by postgrey-1.37 at util1.osuosl.org; Fri, 15 Mar 2024 10:05:36 UTC DMARC-Filter: OpenDMARC Filter v1.4.2 smtp1.osuosl.org A77D781E87 Authentication-Results: smtp1.osuosl.org; dmarc=none (p=none dis=none) header.from=liuyulong.me DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org A77D781E87 Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=43.154.54.12; helo=bg4.exmail.qq.com; envelope-from=i@liuyulong.me; receiver= Received: from bg4.exmail.qq.com (bg4.exmail.qq.com [43.154.54.12]) by smtp1.osuosl.org (Postfix) with ESMTPS id A77D781E87 for ; Fri, 15 Mar 2024 10:05:34 +0000 (UTC) X-QQ-mid: bizesmtp90t1710497128taha3045 X-QQ-Originating-IP: 41WYbGQansr040udD5iVTtbLxOwlgUGv8D9O3D/HvyY= Received: from localhost.localdomain ( [111.202.93.194]) by bizesmtp.qq.com (ESMTP) with id ; Fri, 15 Mar 2024 18:05:27 +0800 (CST) X-QQ-SSF: 01100000000000D0Z000000A0000000 X-QQ-FEAT: QityeSR92A10/vn3SWff0lkPbDGC0i1owLqALAwi866b55ad36ObF86FgBpHX 2CVxxDNMJdCrquzdyES7QSVm37zlbyVdGQJ8VBJ3RRPYuqTm7KfPPgtzIztZe0Gx4oqBvjz 2cZlqNGAynTBzDFAx6qAu1p5of0pilOrL9mZ7ZKbT8qPQlckHwEn+hVQT9n0xhKZ7hkuu4k SDkb3I4ff68/EjsmtRrbJ3VIwNa+hiUKjPjHsZkxM+N6CpuQF/VxhrqszrlFKtbSEON3rzp EyANYy2xXO1Ks1gyo4bX1uZI8tSZxD/khOnp36lUQ+KNwsCtlSdihJ5nIHEf4GNyIshofgq 9bTshvP9kotAV/2ijRxiWWj5oYvsczVCDQmg6QlCjPh8MHHpqo= X-QQ-GoodBg: 0 X-BIZMAIL-ID: 15544580163938425825 From: LIU Yulong To: ovs-dev@openvswitch.org Date: Fri, 15 Mar 2024 18:04:03 +0800 Message-Id: <20240315100402.1979482-1-i@liuyulong.me> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:liuyulong.me:qybglogicsvrgz:qybglogicsvrgz5a-3 Subject: [ovs-dev] [PATCH v3] ofproto-dpif-upcall: Try lock for udpif_key map during sweep. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" A potential race condition happened with the following 3 threads: * PMD thread replaced the old_ukey and transitioned the state to UKEY_DELETED. * RCU thread is freeing the old_ukey mutex. * While the revalidator thread is trying to lock the old_ukey mutex. We added some timestamp to udpif_key state transition and the udpif_key mutex free action, as well as the sweep try lock for the same udpif_key. When the ovs-vswitchd goes core, the udpif_key try_lock mutex had always a bit later than the udpif_key mutex free action. For instance [3]: ukey_destroy_time = 13217289156490 sweep_now = 13217289156568 The second time is 78us behind the first time. Then vswitchd process aborts at the revalidator thread try_lock of ukey->mutex because of the NULL pointer. This patch adds the try_lock for the ukeys' basket udpif_key map to avoid the PMD and revalidator access to the same map for replacing the udpif_key and transitioning the udpif_key state. More details can be found at: [1] https://mail.openvswitch.org/pipermail/ovs-discuss/2024-March/052973.html [2] https://mail.openvswitch.org/pipermail/ovs-discuss/2024-February/052949.html [3] https://mail.openvswitch.org/pipermail/ovs-discuss/2024-March/052993.html Signed-off-by: LIU Yulong --- v2: - Updated commit message to make 0-day Robot happy. v3: - Updated commit message to make checkpatch.py happy. - Add some investigation details. --- --- ofproto/ofproto-dpif-upcall.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c index 9a5c5c29c..ef13f820a 100644 --- a/ofproto/ofproto-dpif-upcall.c +++ b/ofproto/ofproto-dpif-upcall.c @@ -2974,6 +2974,10 @@ revalidator_sweep__(struct revalidator *revalidator, bool purge) struct umap *umap = &udpif->ukeys[i]; size_t n_ops = 0; + if (ovs_mutex_trylock(&umap->mutex)) { + continue; + } + CMAP_FOR_EACH(ukey, cmap_node, &umap->cmap) { enum ukey_state ukey_state; @@ -3013,9 +3017,7 @@ revalidator_sweep__(struct revalidator *revalidator, bool purge) if (ukey_state == UKEY_EVICTED) { /* The common flow deletion case involves deletion of the flow * during the dump phase and ukey deletion here. */ - ovs_mutex_lock(&umap->mutex); ukey_delete(umap, ukey); - ovs_mutex_unlock(&umap->mutex); } if (n_ops == REVALIDATE_MAX_BATCH) { @@ -3025,6 +3027,7 @@ revalidator_sweep__(struct revalidator *revalidator, bool purge) n_ops = 0; } } + ovs_mutex_unlock(&umap->mutex); if (n_ops) { push_ukey_ops(udpif, umap, ops, n_ops);