From patchwork Thu Mar 23 16:35:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thadeu Lima de Souza Cascardo X-Patchwork-Id: 1760426 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=rRUSqT5A; dkim-atps=neutral Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Pj9wP4pl3z1yXt for ; Fri, 24 Mar 2023 03:36:53 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1pfNvk-0002BR-K9; Thu, 23 Mar 2023 16:36:48 +0000 Received: from smtp-relay-canonical-1.internal ([10.131.114.174] helo=smtp-relay-canonical-1.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1pfNvj-0002Ar-8L for kernel-team@lists.ubuntu.com; Thu, 23 Mar 2023 16:36:47 +0000 Received: from localhost.localdomain (1.general.cascardo.us.vpn [10.172.70.58]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-canonical-1.canonical.com (Postfix) with ESMTPSA id 902AB41C8C for ; Thu, 23 Mar 2023 16:36:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1679589406; bh=8QH9QMYrt35wB33R1FU8H4oaAcH/9Sa5l9zqJ1xwUt8=; h=From:To:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rRUSqT5AtwHuN0Z3Bm3lO4umPXGwn0vDHTGzzwaynMRMcqG8vZofdraTyMui/taJB fUMSyq7YKXXlzrE2DtBRba29z/5LReFq99gKTl+h0REUzrmcJ/KvN6cYaI04+Xd4cF I1GzHB5GP+wt3J2MOrw1691d3hHaRY0g6OIXq2zLWhFnx5DE44TAT5JwRc1II9SsW4 vWtaXhy0qkbzjD1IRyKBJIYxgEr4Vb+qSOracrZp8AoA7dSH48RL0TZx/J7O8Ikgvl 37Ab5KcOGENLYurDtEZOCtEcZBBnmF/sUEgEiALbChJRtgpfKm/3TaYoMmunS8daRB DZnVD/yClTliw== From: Thadeu Lima de Souza Cascardo To: kernel-team@lists.ubuntu.com Subject: [UBUNTU Kinetic 1/2] io_uring: fix tw losing poll events Date: Thu, 23 Mar 2023 13:35:29 -0300 Message-Id: <20230323163530.1122686-2-cascardo@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230323163530.1122686-1-cascardo@canonical.com> References: <20230323163530.1122686-1-cascardo@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Pavel Begunkov We may never try to process a poll wake and its mask if there was multiple wake ups racing for queueing up a tw. Force io_poll_check_events() to update the mask by vfs_poll(). Cc: stable@vger.kernel.org Fixes: aa43477b04025 ("io_uring: poll rework") Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/00344d60f8b18907171178d7cf598de71d127b0b.1668710222.git.asml.silence@gmail.com Signed-off-by: Jens Axboe (backported from commit 539bcb57da2f58886d7d5c17134236b0ec9cd15d) CVE-2023-0468 Signed-off-by: Thadeu Lima de Souza Cascardo --- io_uring/io_uring.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9541353346f8..3a63cddcb2bf 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -6674,6 +6674,14 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) if (v & IO_POLL_CANCEL_FLAG) return -ECANCELED; + /* + * cqe.res contains only events of the first wake up + * and all others are be lost. Redo vfs_poll() to get + * up to date state. + */ + if ((v & IO_POLL_REF_MASK) != 1) + req->cqe.res = 0; + if (!req->cqe.res) { struct poll_table_struct pt = { ._key = req->apoll_events }; req->cqe.res = vfs_poll(req->file, &pt) & req->apoll_events; From patchwork Thu Mar 23 16:35:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thadeu Lima de Souza Cascardo X-Patchwork-Id: 1760427 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=aramHvKa; dkim-atps=neutral Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Pj9wS6q8dz1yXt for ; Fri, 24 Mar 2023 03:36:56 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1pfNvo-0002Es-0D; Thu, 23 Mar 2023 16:36:52 +0000 Received: from smtp-relay-canonical-1.internal ([10.131.114.174] helo=smtp-relay-canonical-1.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1pfNvk-0002Be-R4 for kernel-team@lists.ubuntu.com; Thu, 23 Mar 2023 16:36:48 +0000 Received: from localhost.localdomain (1.general.cascardo.us.vpn [10.172.70.58]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-canonical-1.canonical.com (Postfix) with ESMTPSA id A726641C8C for ; Thu, 23 Mar 2023 16:36:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1679589408; bh=2t3MZlovWU7R1MsQuR3PDcQQvV3cYJkKk7SLuRLB2EM=; h=From:To:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=aramHvKaBEfxg7nkkWKxwPAS2vks/gY9++OUaZIfZ+KicwIhNmgQOnuJ/KOWlfS0g ezruSnawAmgL7vr/X4eZ0uTEnffLAoP4DAZ2B2/Lf+ZGl9FACn9IjLVeYbAtOUNGaZ CgoFaYviSUMc3SYV3OklI785D8zeDx5GK5V/Ni+MJHHLg6iQLHhUAu5YqHUti2T993 VPw9iJZ3jxGU/QdTCor48sXf1wYSZcY72MvAN4TqWQrk8bWbPbM9O2LHIUmY23geGV Cgn/Y4u2T2vGf//Vuld7mpqCzGuMz+AwHcQeUUKoz2olpa90eewTqAZPPCWyADquSW g2E1J2reA4A1g== From: Thadeu Lima de Souza Cascardo To: kernel-team@lists.ubuntu.com Subject: [UBUNTU Kinetic 2/2] io_uring: make poll refs more robust Date: Thu, 23 Mar 2023 13:35:30 -0300 Message-Id: <20230323163530.1122686-3-cascardo@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230323163530.1122686-1-cascardo@canonical.com> References: <20230323163530.1122686-1-cascardo@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Pavel Begunkov poll_refs carry two functions, the first is ownership over the request. The second is notifying the io_poll_check_events() that there was an event but wake up couldn't grab the ownership, so io_poll_check_events() should retry. We want to make poll_refs more robust against overflows. Instead of always incrementing it, which covers two purposes with one atomic, check if poll_refs is elevated enough and if so set a retry flag without attempts to grab ownership. The gap between the bias check and following atomics may seem racy, but we don't need it to be strict. Moreover there might only be maximum 4 parallel updates: by the first and the second poll entries, __io_arm_poll_handler() and cancellation. From those four, only poll wake ups may be executed multiple times, but they're protected by a spin. Cc: stable@vger.kernel.org Reported-by: Lin Ma Fixes: aa43477b04025 ("io_uring: poll rework") Signed-off-by: Pavel Begunkov Link: https://lore.kernel.org/r/c762bc31f8683b3270f3587691348a7119ef9c9d.1668963050.git.asml.silence@gmail.com Signed-off-by: Jens Axboe (backported from commit a26a35e9019fd70bf3cf647dcfdae87abc7bacea) CVE-2023-0468 Signed-off-by: Thadeu Lima de Souza Cascardo --- io_uring/io_uring.c | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3a63cddcb2bf..b782d73e3be0 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -6550,7 +6550,29 @@ struct io_poll_table { }; #define IO_POLL_CANCEL_FLAG BIT(31) -#define IO_POLL_REF_MASK GENMASK(30, 0) +#define IO_POLL_RETRY_FLAG BIT(30) +#define IO_POLL_REF_MASK GENMASK(29, 0) + +/* + * We usually have 1-2 refs taken, 128 is more than enough and we want to + * maximise the margin between this amount and the moment when it overflows. + */ +#define IO_POLL_REF_BIAS 128 + +static bool io_poll_get_ownership_slowpath(struct io_kiocb *req) +{ + int v; + + /* + * poll_refs are already elevated and we don't have much hope for + * grabbing the ownership. Instead of incrementing set a retry flag + * to notify the loop that there might have been some change. + */ + v = atomic_fetch_or(IO_POLL_RETRY_FLAG, &req->poll_refs); + if (v & IO_POLL_REF_MASK) + return false; + return !(atomic_fetch_inc(&req->poll_refs) & IO_POLL_REF_MASK); +} /* * If refs part of ->poll_refs (see IO_POLL_REF_MASK) is 0, it's free. We can @@ -6560,6 +6582,8 @@ struct io_poll_table { */ static inline bool io_poll_get_ownership(struct io_kiocb *req) { + if (unlikely(atomic_read(&req->poll_refs) >= IO_POLL_REF_BIAS)) + return io_poll_get_ownership_slowpath(req); return !(atomic_fetch_inc(&req->poll_refs) & IO_POLL_REF_MASK); } @@ -6681,6 +6705,16 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) */ if ((v & IO_POLL_REF_MASK) != 1) req->cqe.res = 0; + if (v & IO_POLL_RETRY_FLAG) { + req->cqe.res = 0; + /* + * We won't find new events that came in between + * vfs_poll and the ref put unless we clear the flag + * in advance. + */ + atomic_andnot(IO_POLL_RETRY_FLAG, &req->poll_refs); + v &= ~IO_POLL_RETRY_FLAG; + } if (!req->cqe.res) { struct poll_table_struct pt = { ._key = req->apoll_events };