From patchwork Tue Jul 16 22:15:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guilherme G. Piccoli" X-Patchwork-Id: 1132972 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=canonical.com Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45pF8s5T2Kz9sND; Wed, 17 Jul 2019 08:15:57 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1hnVje-0004ah-NR; Tue, 16 Jul 2019 22:15:46 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:128) (Exim 4.86_2) (envelope-from ) id 1hnVjc-0004Zq-NB for kernel-team@lists.ubuntu.com; Tue, 16 Jul 2019 22:15:44 +0000 Received: from mail-qt1-f200.google.com ([209.85.160.200]) by youngberry.canonical.com with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.76) (envelope-from ) id 1hnVjc-0007HY-9B for kernel-team@lists.ubuntu.com; Tue, 16 Jul 2019 22:15:44 +0000 Received: by mail-qt1-f200.google.com with SMTP id q26so19493240qtr.3 for ; Tue, 16 Jul 2019 15:15:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=gVwWeWxsgq+4UK5jQoG4X6abdXnaKBQ3CzZLgzVR0/M=; b=O0tmQTFc5ckt85h66eS9+Brgc9GBIeAM1xX7B6IkfAq0P7k9Ib7i6CfEBTSSsDCfdV uUG6f6ddTiM1xYIQ9tfIsm81J3Acll7OWNGEsK1fVDmxsZqnsr0JAV+V9EJzw5x9MYo4 uZvOFOwiRyFUUvSvcVNBYFsyw3v6YvtYtEld76BJk3mt/IHHx7g3NSHzkZmjv3BH9cQI 24HaruAb0Oahv5QOr8R0hWI7HpJ10YeH80l7CiQGg4GmGs/ecVKDBt5RMZvlediLWjqE WG3KGc2YnjBxtBPTjYpUeg7c+e0ggSR2szKVI1xbDOBiJUt/9UEe7RBswR0dUR44hT1L MB7w== X-Gm-Message-State: APjAAAW4uULCcD/MJwuYvCQpJ9i/eZceoB+Uz5HHdEemdGjdBLLfqq7a 6ON+UlS7UVN3stckjPtXvFC9dq3ohri0OAclrBi6MKiVpmOtbjsXy+A2Z4RbT1fmfLAbEN1ke7l Op6YRdeOzKIYQI0m+I8eOQHUCE+GaEgHLsCxzDshvYg== X-Received: by 2002:ac8:1418:: with SMTP id k24mr23491796qtj.54.1563315342969; Tue, 16 Jul 2019 15:15:42 -0700 (PDT) X-Google-Smtp-Source: APXvYqyjVhW0eL9U/pNWJwJ9VkBOnolip8SWmre6XkluD60P04Vk+PLYJodVk2eWHVoc7ed+++5Igg== X-Received: by 2002:ac8:1418:: with SMTP id k24mr23491779qtj.54.1563315342746; Tue, 16 Jul 2019 15:15:42 -0700 (PDT) Received: from localhost (201-42-108-61.dsl.telesp.net.br. [201.42.108.61]) by smtp.gmail.com with ESMTPSA id w9sm9339521qki.81.2019.07.16.15.15.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Jul 2019 15:15:41 -0700 (PDT) From: "Guilherme G. Piccoli" To: kernel-team@lists.ubuntu.com Subject: [SRU D][PATCH 0/2] Two crashes on raid0 error path (during a member device removal) Date: Tue, 16 Jul 2019 19:15:37 -0300 Message-Id: <20190716221539.31891-1-gpiccoli@canonical.com> X-Mailer: git-send-email 2.22.0 MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: gpiccoli@canonical.com Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" BugLink: https://bugs.launchpad.net/bugs/1836806 [Impact] * During raid0 error path testing, by removing one member of the array, we've noticed after kernel 4.18 we can trigger a crash depending if there's I/O in-flight during the array removal. When debugging the issue, a second problem was found, that could cause a different crash. * For the first and more relevant problem, commit cd4a4ae4683d ("block: don't use blocking queue entered for recursive bio submits") introduced the flag BIO_QUEUE_ENTERED in order BIOs that were split do bypass the blocking queue entering routine and use the live non-blocking version. What happens with md/raid0 though is that their BIOs have their underlying device changed to the physical disk (array member). If we remove this physical disk (or if it fails), we could have one BIO that had the flag changed to BIO_QUEUE_ENTERED and had the device changed to the removed array member (before its removal); this bio then skips a lot of checks in generic_make_request_checks(), triggering the following crash: BUG: unable to handle kernel NULL pointer dereference at 0000000000000155 PGD 0 P4D 0 Oops: 0000 [#1] SMP PTI RIP: 0010:blk_throtl_bio+0x45/0x970 [...] Call Trace: generic_make_request_checks+0x1bf/0x690 generic_make_request+0x64/0x3f0 raid0_make_request+0x184/0x620 [raid0] ? raid0_make_request+0x184/0x620 [raid0] md_handle_request+0x126/0x1a0 md_make_request+0x7b/0x180 generic_make_request+0x19e/0x3f0 submit_bio+0x73/0x140 [...] * When debugging the above issue, by rebuilding the kernel with CONFIG_BLK_CGROUP=n we've noticed a different crash. Commit 37f9579f4c31 ("blk-mq: Avoid that submitting a bio concurrently with device removal triggers a crash") introduced a NULL pointer dereference in generic_make_request(), that manifests as: BUG: unable to handle kernel NULL pointer dereference at 0000000000000078 PGD 0 P4D 0 Oops: 0000 [#1] SMP PTI RIP: 0010:generic_make_request+0x32b/0x400 Call Trace: submit_bio+0x73/0x140 ext4_io_submit+0x4d/0x60 ext4_writepages+0x626/0xe90 do_writepages+0x4b/0xe0 [...] * For both the issues, we have simple patches that are present in linux-stable but not in Linus tree. ## For issue 1 (md removal crash): 869eec894663 ("md/raid0: Do not bypass blocking queue entered for raid0 bios") https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=869eec894663 ## For issue 2 (generic_make_request() NULL dereference): c9d8d3e9d7a0 ("block: Fix a NULL pointer dereference in generic_make_request()") https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=c9d8d3e9d7a0 The reasoning for both patches not being present in Linus tree is explained in the commit messages, but in summary Ming Lei submitted a major clean-up series at the same time I've submitted both patches, it wouldn't make sense to accept my patches to soon after remove the code paths with his clean-up. But Ming's series rely on legacy I/O path removal, and so it's very hard to backport. Hence maintainers suggested me to submit my small fixes to stable tree only. [Test case] For both cases, the test is the same, the only change being a kernel config option. To reproduce issue 1 (md removal crash), a regular Ubuntu kernel config is enough. For the issue 2, a kernel rebuild with CONFIG_BLK_CGROUP=n is necessary. Steps to reproduce: a) Create a raid0 md array with 2 NVMe devices as members, and mount it with an ext4 filesystem. b) Run the following oneliner (supposing the raid0 is mounted in /mnt): (dd of=/mnt/tmp if=/dev/zero bs=1M count=999 &); sleep 0.3;\ echo 1 > /sys/block/nvme1n1/device/device/remove (whereas nvme1n1 is the 2nd array member) [Regression potential] The fixes are self-contained and small, both validated by a great number of subsystem maintainers (including block, raid and stable). Commit c9d8d3e9d7a0 was also validated by the author of the offender patch it fixes, and has no functional change. Commit 869eec894663 has only raid0 driver as scope, and fall-backs raid0 to a previous behavior before the introduction of BIO_QUEUE_ENTERED flag (which indeed increases the amount of checks performed in BIOs), so the regression potential is low and restricted to raid0. Guilherme G. Piccoli (2): block: Fix a NULL pointer dereference in generic_make_request() md/raid0: Do not bypass blocking queue entered for raid0 bios block/blk-core.c | 5 ++--- drivers/md/raid0.c | 2 ++ 2 files changed, 4 insertions(+), 3 deletions(-) Acked-by: Stefan Bader Acked-by: Connor Kuehl