From patchwork Mon Sep 9 15:07:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aleksandr Mikhalitsyn X-Patchwork-Id: 1982623 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4X2VZV4CkSz1y1q for ; Tue, 10 Sep 2024 01:08:01 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1snfzC-0002e7-Hr; Mon, 09 Sep 2024 15:07:26 +0000 Received: from smtp-relay-internal-1.internal ([10.131.114.114] helo=smtp-relay-internal-1.canonical.com) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1snfzB-0002dj-7S for kernel-team@lists.ubuntu.com; Mon, 09 Sep 2024 15:07:25 +0000 Received: from mail-ej1-f69.google.com (mail-ej1-f69.google.com [209.85.218.69]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 9BE843F664 for ; Mon, 9 Sep 2024 15:07:24 +0000 (UTC) Received: by mail-ej1-f69.google.com with SMTP id a640c23a62f3a-a8d274ff219so43899266b.2 for ; Mon, 09 Sep 2024 08:07:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725894444; x=1726499244; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3hlXWm420sYSRxmsrBMZyTKTUV2hr19TiCYsRSELGoI=; b=DUVi8mq9Fm+T8qv2YnseTYNoNivWJF++ZoUpALZWodkR7OAT/upEI+fFVyX0xBRFrn WB1YkPkfa2gN0gfYIfUWQgTK9BEXgB/FfPOvIbq2k2K81pUbR90go4JXHkzDyKj1OlBX 0fM1nDquVfJlFQ+sZ+aaSkkSiWvyfVIpVwQBJDWnUGSEUH2HQS3AJe54lKhtIWwdZrTX 4KUBRbnJPQOrOsF8i6SzzyANy0ffcsOXXMXuDbIjkUDlhn2ICYzFI1QLpXfLhgH5nrav OSX7RISsO/wDO2V5upN7hWhmoVYPLLfCfvjzkJGFq2/gfCnW2+aKsCekzStb5tySQttG jZTQ== X-Gm-Message-State: AOJu0YyYu4X4lH2hcLx+qEgU9GtqEcVyjuyVCacfPoyM5HcxAAYPPeYu VrPw9B6laaZx0i1LxsdmrSNhQ63WNPQlzHG8vI/d8s5JZAyl3zi7AqsB5p2K1jC39XxIfJpFEgx gEnH8nbSbR7KjVxugMzEmZK4vZJDS+uxom2D8pCaUVyfq0/hA9yEqlPqMuYWhj4nVtu2QwMuz0Q Mq3Xe3mYRU2g== X-Received: by 2002:a05:6402:51d4:b0:5c3:c530:e99a with SMTP id 4fb4d7f45d1cf-5c3dc795d36mr11824232a12.15.1725894443673; Mon, 09 Sep 2024 08:07:23 -0700 (PDT) X-Google-Smtp-Source: AGHT+IErJi17Y4XDhGw3W9H+DS6uX/ElCuyHfwtWzn5dKjtqFlx2X0JQscrp1pEYU8uLKNv9U8kFMw== X-Received: by 2002:a05:6402:51d4:b0:5c3:c530:e99a with SMTP id 4fb4d7f45d1cf-5c3dc795d36mr11824182a12.15.1725894443112; Mon, 09 Sep 2024 08:07:23 -0700 (PDT) Received: from amikhalitsyn.. ([188.192.113.77]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5c3ebd41bd4sm3097069a12.11.2024.09.09.08.07.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 08:07:22 -0700 (PDT) From: Alexander Mikhalitsyn To: kernel-team@lists.ubuntu.com Subject: [SRU][N][PATCH v2 1/2] UBUNTU: SAUCE: fan: release rcu_read_lock on skb discard path Date: Mon, 9 Sep 2024 17:07:08 +0200 Message-Id: <20240909150709.118761-2-aleksandr.mikhalitsyn@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240909150709.118761-1-aleksandr.mikhalitsyn@canonical.com> References: <20240909150709.118761-1-aleksandr.mikhalitsyn@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: tim.gardner@canonical.com Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" BugLink: https://bugs.launchpad.net/bugs/2064176 User can trigger (see steps in [1] and LP bug) the following RCU warning (which makes the whole system unresponsive and effectively forces system administrator to reboot). Aug 30 21:51:57 v1 kernel: ------------[ cut here ]------------ Aug 30 21:51:57 v1 kernel: Voluntary context switch within RCU read-side critical section! Aug 30 21:51:57 v1 kernel: WARNING: CPU: 1 PID: 2669 at kernel/rcu/tree_plugin.h:320 rcu_note_context_switch+0x2ce/0x2f0 Aug 30 21:51:57 v1 kernel: Modules linked in: veth vxlan ip6_udp_tunnel udp_tunnel dummy nft_masq nft_chain_nat bridge stp llc zfs(PO) spl(O) nvme_fabrics nvme_core nvme_auth ebtable_filter ebtables ip6table_raw ip6table_mangle ip6table_nat ip6table_filter ip6_tables iptable_raw iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter nf_tables libcrc32c vhost_vsock vhost vhost_iotlb binfmt_misc kvm_amd ccp kvm irqbypass crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 sha1_ssse3 nls_iso8859_1 joydev aesni_intel crypto_simd cryptd virtio_gpu 9pnet_virtio virtio_dma_buf xhci_pci psmouse ahci 9pnet virtiofs libahci vmw_vsock_virtio_transport xhci_pci_renesas vmw_vsock_virtio_transport_common vsock virtio_input input_leds serio_raw efi_pstore nfnetlink dmi_sysfs virtio_rng ip_tables x_tables autofs4 Aug 30 21:51:57 v1 kernel: CPU: 1 PID: 2669 Comm: systemd-resolve Tainted: P O 6.8.0-41-generic #41-Ubuntu Aug 30 21:51:57 v1 kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)/LXD, BIOS unknown 2/2/2022 Aug 30 21:51:57 v1 kernel: RIP: 0010:rcu_note_context_switch+0x2ce/0x2f0 Aug 30 21:51:57 v1 kernel: Code: fe ff ff ba 02 00 00 00 be 01 00 00 00 e8 fa d0 fe ff e9 6b fe ff ff 48 c7 c7 60 7d a6 a8 c6 05 ab 99 61 02 01 e8 d2 0d f2 ff <0f> 0b e9 96 fd ff ff 0f 0b e9 36 ff ff ff 0f 0b e9 18 ff ff ff 66 Aug 30 21:51:57 v1 kernel: RSP: 0018:ffffb611812bbd80 EFLAGS: 00010046 Aug 30 21:51:57 v1 kernel: RAX: 0000000000000000 RBX: ffff9613faeb5a00 RCX: 0000000000000000 Aug 30 21:51:57 v1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 Aug 30 21:51:57 v1 kernel: RBP: ffffb611812bbda0 R08: 0000000000000000 R09: 0000000000000000 Aug 30 21:51:57 v1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 Aug 30 21:51:57 v1 kernel: R13: ffff9613b89dd200 R14: 0000000000000000 R15: 0000000000000000 Aug 30 21:51:57 v1 kernel: FS: 00007ec3a402c5c0(0000) GS:ffff9613fae80000(0000) knlGS:0000000000000000 Aug 30 21:51:57 v1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 30 21:51:57 v1 kernel: CR2: 000062592dc892b8 CR3: 000000013890a000 CR4: 00000000007506f0 Aug 30 21:51:57 v1 kernel: PKRU: 55555554 Aug 30 21:51:57 v1 kernel: Call Trace: Aug 30 21:51:57 v1 kernel: Aug 30 21:51:57 v1 kernel: ? show_regs+0x6d/0x80 Aug 30 21:51:57 v1 kernel: ? __warn+0x89/0x160 Aug 30 21:51:57 v1 kernel: ? rcu_note_context_switch+0x2ce/0x2f0 Aug 30 21:51:57 v1 kernel: ? report_bug+0x17e/0x1b0 Aug 30 21:51:57 v1 kernel: ? handle_bug+0x51/0xa0 Aug 30 21:51:57 v1 kernel: ? exc_invalid_op+0x18/0x80 Aug 30 21:51:57 v1 kernel: ? asm_exc_invalid_op+0x1b/0x20 Aug 30 21:51:57 v1 kernel: ? rcu_note_context_switch+0x2ce/0x2f0 Aug 30 21:51:57 v1 kernel: __schedule+0x81/0x6b0 Aug 30 21:51:57 v1 kernel: schedule+0x33/0x110 Aug 30 21:51:57 v1 kernel: syscall_exit_to_user_mode+0x22d/0x260 Aug 30 21:51:57 v1 kernel: do_syscall_64+0x8c/0x180 Aug 30 21:51:57 v1 kernel: ? srso_alias_return_thunk+0x5/0xfbef5 Aug 30 21:51:57 v1 kernel: ? syscall_exit_to_user_mode+0x89/0x260 Aug 30 21:51:57 v1 kernel: ? srso_alias_return_thunk+0x5/0xfbef5 Aug 30 21:51:57 v1 kernel: ? do_syscall_64+0x8c/0x180 Aug 30 21:51:57 v1 kernel: ? srso_alias_return_thunk+0x5/0xfbef5 Aug 30 21:51:57 v1 kernel: ? irqentry_exit_to_user_mode+0x7e/0x260 Aug 30 21:51:57 v1 kernel: ? srso_alias_return_thunk+0x5/0xfbef5 Aug 30 21:51:57 v1 kernel: ? irqentry_exit+0x43/0x50 Aug 30 21:51:57 v1 kernel: ? srso_alias_return_thunk+0x5/0xfbef5 Aug 30 21:51:57 v1 kernel: ? exc_page_fault+0x94/0x1b0 Aug 30 21:51:57 v1 kernel: entry_SYSCALL_64_after_hwframe+0x78/0x80 Aug 30 21:51:57 v1 kernel: RIP: 0033:0x7ec3a3f14887 Aug 30 21:51:57 v1 kernel: Code: 10 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 Aug 30 21:51:57 v1 kernel: RSP: 002b:00007ffcbb32de08 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 Aug 30 21:51:57 v1 kernel: RAX: 000000000000002d RBX: 000062592dc882b0 RCX: 00007ec3a3f14887 Aug 30 21:51:57 v1 kernel: RDX: 000000000000002d RSI: 000062592dc88360 RDI: 0000000000000011 Aug 30 21:51:57 v1 kernel: RBP: 000062592dc7e690 R08: 00007ffcbb32dde4 R09: 0000000000000000 Aug 30 21:51:57 v1 kernel: R10: 00000000000005aa R11: 0000000000000246 R12: 0000000000000011 Aug 30 21:51:57 v1 kernel: R13: 0000000000000002 R14: 000000000000002d R15: 000062592dc88360 Aug 30 21:51:57 v1 kernel: Aug 30 21:51:57 v1 kernel: ---[ end trace 0000000000000000 ]--- This warning is a result of an RCU misuse (an RCU read lock is taken and not released). Let's fix it by releasing the RCU read lock before "goto tx_free" on the skb discard codepath. Link: https://github.com/canonical/lxd/issues/14025 [1] Reported-by: Max Asnaashari Signed-off-by: Alexander Mikhalitsyn --- drivers/net/vxlan/vxlan_core.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/net/vxlan/vxlan_core.c b/drivers/net/vxlan/vxlan_core.c index c8b1ff86535a..d19c1900e52b 100644 --- a/drivers/net/vxlan/vxlan_core.c +++ b/drivers/net/vxlan/vxlan_core.c @@ -2621,6 +2621,7 @@ void vxlan_xmit_one(struct sk_buff *skb, struct net_device *dev, if (fan_has_map(&vxlan->fan) && rt->rt_flags & RTCF_LOCAL) { netdev_dbg(dev, "discard fan to localhost %pI4\n", &rdst->remote_ip.sin.sin_addr.s_addr); + rcu_read_unlock(); ip_rt_put(rt); goto tx_free; } From patchwork Mon Sep 9 15:07:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aleksandr Mikhalitsyn X-Patchwork-Id: 1982622 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4X2VZV40WNz1y1S for ; Tue, 10 Sep 2024 01:08:01 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1snfzE-0002ed-MM; Mon, 09 Sep 2024 15:07:28 +0000 Received: from smtp-relay-internal-1.internal ([10.131.114.114] helo=smtp-relay-internal-1.canonical.com) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1snfzC-0002eC-L9 for kernel-team@lists.ubuntu.com; Mon, 09 Sep 2024 15:07:26 +0000 Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 7AD633F2F1 for ; Mon, 9 Sep 2024 15:07:26 +0000 (UTC) Received: by mail-ed1-f70.google.com with SMTP id 4fb4d7f45d1cf-5c24c93b3bcso1697500a12.0 for ; Mon, 09 Sep 2024 08:07:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725894446; x=1726499246; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/uZXFcE+kLztBRV386MdVfUjO+K9AatnvnhUq76s2tM=; b=ohEt4h9Do6Lx/Xm484S5ubRukhpOX+pH0KsKGNJzVXFdsOeHESmxTzXthFLuuIYgdh zPQMw5B7qFH/abf8NPRTGIFtxJPHVyMhhhwrvg3ndbt1d0udEFkp7CnVzoGqpFg8Exm9 Tgs2WH+zNOOPvKeCvVIb9Hagq/x3m5Dnczs/zWHTRzLaqNK+a1rNb1LinkeJk+wSAT32 Hphe2FQecEkoLgVJ4jywr+yBNg5vxxfs7CVhfQ0qtEbUkAQlOacBm5ZswdXhL6iEFKDO wSzrZRM1+vWzRSURoXUioSo+DMrEKWV71vm7fYpnlDNj6HNbXysNet5rovU0PnqrZVEM xCtQ== X-Gm-Message-State: AOJu0YxVlb3JsBkpo5IWn0IDIxIGyHjPoUXODsyJ633u3WT56aSxVVOi pAMKv52Ec1pD+6wWopvgB4ohK+vGHDrgAiMouFpE2/Fa3H57pHMl4r8D9AHylFdK8a4Do6JtQ6V f23Jg+3WFkdl3C+r2p12jHZJhzkmDRm0K1NMfqKn3ZNSw8xGKpWkXbEToyqqWJ2BU1xFoFjEZMs pSItxqSxwhSQ== X-Received: by 2002:a05:6402:40d4:b0:5c3:d0e1:9f81 with SMTP id 4fb4d7f45d1cf-5c3dc77ab97mr11870689a12.7.1725894445636; Mon, 09 Sep 2024 08:07:25 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGbpAdMBC34DGttZ/3hw6IUjm1O4eyFoJGi/aUCTsa9yEWEyqg6KwM6KlAQmti1Dh3aw03G/Q== X-Received: by 2002:a05:6402:40d4:b0:5c3:d0e1:9f81 with SMTP id 4fb4d7f45d1cf-5c3dc77ab97mr11870642a12.7.1725894445129; Mon, 09 Sep 2024 08:07:25 -0700 (PDT) Received: from amikhalitsyn.. ([188.192.113.77]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5c3ebd41bd4sm3097069a12.11.2024.09.09.08.07.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Sep 2024 08:07:24 -0700 (PDT) From: Alexander Mikhalitsyn To: kernel-team@lists.ubuntu.com Subject: [SRU][N][PATCH v2 2/2] UBUNTU: SAUCE: fan: fix racy device stat update Date: Mon, 9 Sep 2024 17:07:09 +0200 Message-Id: <20240909150709.118761-3-aleksandr.mikhalitsyn@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240909150709.118761-1-aleksandr.mikhalitsyn@canonical.com> References: <20240909150709.118761-1-aleksandr.mikhalitsyn@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: tim.gardner@canonical.com Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" BugLink: https://bugs.launchpad.net/bugs/2064176 Let's use a proper API to increase netdevice's tx_dropped statistics. Since upstream commit 6dee402daba4eb ("vxlan: Fix racy device stats updates.") we have to use dev_core_stats_tx_dropped_inc() function for that. This one was found while I was debugging an original issues from LP bug #2064176 and was reviewing a whole Ubuntu FAN implementation. Signed-off-by: Alexander Mikhalitsyn --- drivers/net/vxlan/vxlan_core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/vxlan/vxlan_core.c b/drivers/net/vxlan/vxlan_core.c index d19c1900e52b..9c9ad01e6027 100644 --- a/drivers/net/vxlan/vxlan_core.c +++ b/drivers/net/vxlan/vxlan_core.c @@ -2916,7 +2916,7 @@ static netdev_tx_t vxlan_xmit(struct sk_buff *skb, struct net_device *dev) netdev_dbg(vxlan->dev, "vxlan_xmit p %x d %pM\n", eth->h_proto, eth->h_dest); if (vxlan_fan_build_rdst(vxlan, skb, &fan_rdst)) { - dev->stats.tx_dropped++; + dev_core_stats_tx_dropped_inc(dev); kfree_skb(skb); return NETDEV_TX_OK; }