From patchwork Wed Jul 20 03:12:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Ruffell X-Patchwork-Id: 1658324 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=OmylfbOg; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lngjz6lD6z9s2R for ; Wed, 20 Jul 2022 13:12:39 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1oE08W-0008DH-Ee; Wed, 20 Jul 2022 03:12:32 +0000 Received: from smtp-relay-internal-1.internal ([10.131.114.114] helo=smtp-relay-internal-1.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1oE08Q-000867-7g for kernel-team@lists.ubuntu.com; Wed, 20 Jul 2022 03:12:26 +0000 Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id E49C43F0C8 for ; Wed, 20 Jul 2022 03:12:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1658286745; bh=gDHDt/srQTe5fpzQYuYYMiFsH7GWUkisZgdnjg8rkCs=; h=From:To:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OmylfbOgi9a4SvU8X5ae/THTMRu3X+avnTijCGbwA+goGV9lGZoHQiDAxI612cpKu 5uKxYzQKbk7KUPW1NLWUw1w6WChS7vYpucZzVCmFbFZ+/Cg27wdh2B1PIV7ukKXUsI Evnx3S22dg7d5X0HbQIQb3g8SdLr9A9RSaqV46NO3XB4evAtCJ/Y7Y3vz111mezz27 KEzsYIvmLgKr4zEtL448XOh74cZqUIYZeHcp4gJKSYsuKAE/B8FSeXpKqvw1wtpbC0 MKo0HE3RhItqVfUSUqUKvBiUWQe1w/GKw9VQ3Lu5eqE5CSBkbrtcnALM8uh3g6/mjV SW7CvjLz8cz7w== Received: by mail-pf1-f199.google.com with SMTP id 64-20020a621643000000b0052b51cf6b2aso2352375pfw.0 for ; Tue, 19 Jul 2022 20:12:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gDHDt/srQTe5fpzQYuYYMiFsH7GWUkisZgdnjg8rkCs=; b=S1ankKnnLMngF06BsvCbzsCnA/SyzDPw6LOmMb/MOuDk9QTUQos85qjkwM1Zak5XL2 J4yDu4eJwA1SNNV4uUuWQ6uJ/gVrvwBzPFhZKsx9qQrgk39w3abSSwzI/CsrAhZ1cj+H 5qdybNMyL69/9a+w+WnBIKxh1kATv15KO19On7OCqLab6oFPnIQcXyOLrHvUm0TQVnT4 +FfUC2Yle4TInlkGVdTKJUC4COyQ802l8lEVmXkiNwNlcPf1KnJgM8YEVq+t+fOQoAI/ fIvc8s0b12cnmki092W61yPrXhwge1fjYLaFLUN8hahLMiXlz57fKYeJBlhnT3pQmUoQ nA2w== X-Gm-Message-State: AJIora8K4yEazeIIgOx5xazIzVRxlSDIURwvaAxAX2V4ljQOIsI5rGFZ W2pd+JHJa8/tXlHMPeJv2RYxpQNBFTwK9sMjZFH4NtPnuVzCbiIXIwa4ICxJ5t+ft/zO9YZ5U+z 3LHf687qk9LBA7aPo+RHWT+O5BsEzPPjNqAdh9dyBSw== X-Received: by 2002:a17:90b:3907:b0:1f0:386e:c0ee with SMTP id ob7-20020a17090b390700b001f0386ec0eemr2853811pjb.141.1658286744515; Tue, 19 Jul 2022 20:12:24 -0700 (PDT) X-Google-Smtp-Source: AGRyM1sPfcF81NbMr6dOMlwF4Ic5v4v4NeHD3nz22KnLIcUG3RZ623Pe7NmY69gPdoGcO5se0l2F+g== X-Received: by 2002:a17:90b:3907:b0:1f0:386e:c0ee with SMTP id ob7-20020a17090b390700b001f0386ec0eemr2853787pjb.141.1658286744258; Tue, 19 Jul 2022 20:12:24 -0700 (PDT) Received: from desktop.. (125-239-70-54-fibre.sparkbb.co.nz. [125.239.70.54]) by smtp.gmail.com with ESMTPSA id i2-20020a17090ac40200b001efbc3ad105sm331812pjt.54.2022.07.19.20.12.22 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Jul 2022 20:12:23 -0700 (PDT) From: Matthew Ruffell To: kernel-team@lists.ubuntu.com Subject: [SRU][F][PATCH V2 4/6] nbd: make sure request completion won't concurrent Date: Wed, 20 Jul 2022 15:12:08 +1200 Message-Id: <20220720031210.17801-5-matthew.ruffell@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220720031210.17801-1-matthew.ruffell@canonical.com> References: <20220720031210.17801-1-matthew.ruffell@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Yu Kuai BugLink: https://bugs.launchpad.net/bugs/1896350 commit cddce0116058 ("nbd: Aovid double completion of a request") try to fix that nbd_clear_que() and recv_work() can complete a request concurrently. However, the problem still exists: t1 t2 t3 nbd_disconnect_and_put flush_workqueue recv_work blk_mq_complete_request blk_mq_complete_request_remote -> this is true WRITE_ONCE(rq->state, MQ_RQ_COMPLETE) blk_mq_raise_softirq blk_done_softirq blk_complete_reqs nbd_complete_rq blk_mq_end_request blk_mq_free_request WRITE_ONCE(rq->state, MQ_RQ_IDLE) nbd_clear_que blk_mq_tagset_busy_iter nbd_clear_req __blk_mq_free_request blk_mq_put_tag blk_mq_complete_request -> complete again There are three places where request can be completed in nbd: recv_work(), nbd_clear_que() and nbd_xmit_timeout(). Since they all hold cmd->lock before completing the request, it's easy to avoid the problem by setting and checking a cmd flag. Signed-off-by: Yu Kuai Reviewed-by: Ming Lei Reviewed-by: Josef Bacik Link: https://lore.kernel.org/r/20210916093350.1410403-3-yukuai3@huawei.com Signed-off-by: Jens Axboe (cherry picked from 07175cb1baf4c51051b1fbd391097e349f9a02a9) Signed-off-by: Matthew Ruffell --- drivers/block/nbd.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index b960e29b0b57..01d030e9f301 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -394,7 +394,11 @@ static enum blk_eh_timer_return nbd_xmit_timeout(struct request *req, if (!mutex_trylock(&cmd->lock)) return BLK_EH_RESET_TIMER; - __clear_bit(NBD_CMD_INFLIGHT, &cmd->flags); + if (!__test_and_clear_bit(NBD_CMD_INFLIGHT, &cmd->flags)) { + mutex_unlock(&cmd->lock); + return BLK_EH_DONE; + } + if (!refcount_inc_not_zero(&nbd->config_refs)) { cmd->status = BLK_STS_TIMEOUT; mutex_unlock(&cmd->lock); @@ -818,7 +822,10 @@ static bool nbd_clear_req(struct request *req, void *data, bool reserved) return true; mutex_lock(&cmd->lock); - __clear_bit(NBD_CMD_INFLIGHT, &cmd->flags); + if (!__test_and_clear_bit(NBD_CMD_INFLIGHT, &cmd->flags)) { + mutex_unlock(&cmd->lock); + return true; + } cmd->status = BLK_STS_IOERR; mutex_unlock(&cmd->lock);