From patchwork Wed Apr 1 21:40:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guilherme G. Piccoli" X-Patchwork-Id: 1265357 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=canonical.com Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48t04C08YDz9sRN; Thu, 2 Apr 2020 08:40:41 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1jJl6D-0000Vh-LE; Wed, 01 Apr 2020 21:40:37 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1jJl6C-0000VP-0q for kernel-team@lists.ubuntu.com; Wed, 01 Apr 2020 21:40:36 +0000 Received: from mail-qt1-f200.google.com ([209.85.160.200]) by youngberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1jJl6B-0006AP-Lp for kernel-team@lists.ubuntu.com; Wed, 01 Apr 2020 21:40:35 +0000 Received: by mail-qt1-f200.google.com with SMTP id i36so1246892qtd.9 for ; Wed, 01 Apr 2020 14:40:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9/P3tcVCWFkPYk8eFqg0i1qvR1Zeb0wYnI835K969yc=; b=GrBvGsBdVJePVymrUrCl1x7BILDqhhO7JH5V4AI7fp5S0jVfv+b27LcDiQX5cxsAOD ROwrJNfJyABdOVdft6o90OW2fpPuR83dNNtmj644MMbpwSGeORYR3V6mdByqbLEpnHyt y/sWNkbPHMJK8o0+eR0dhvVj349sv93upm5ii1UWLiyPmOthZgLfbUQ6YYrQ9+sraVQG b6OZrtYLQK31RzyltEZozQ16Jf2H7vMxpsLqrB6Hu7TKLlho1K1Mt8h1waXbpF/lydrB WMJy9dGnp+T24228VaJkSeS7pnIUbeN7QG0Ly/ZPF3X0cEklpd8jlbUX3J78f1mzMngy XHeA== X-Gm-Message-State: AGi0PuY3bzjaS+a5Kax2U12oKb6pobXOLCpb10KD8LtMZwemE2SbXQhc pF9ydLYM0JH1K1Se6HOvk/i7465FC2AwD4/ZsbgLPsqAATOEgYpdhHh1DEdLwYIPWMk4Y5Mutwv PbnL3PbPmAOkaQHj1gQg4VvNNL+INiZpunm/heN6wGA== X-Received: by 2002:a0c:e848:: with SMTP id l8mr187742qvo.82.1585777234168; Wed, 01 Apr 2020 14:40:34 -0700 (PDT) X-Google-Smtp-Source: APiQypJrBqltGRnUCq8Am1fp7sQpD74GDM0vr9NuAs3u8LQjktFhScYj906IKMTzR6NHdoj3P8E7Jg== X-Received: by 2002:a0c:e848:: with SMTP id l8mr187722qvo.82.1585777233879; Wed, 01 Apr 2020 14:40:33 -0700 (PDT) Received: from localhost ([179.98.72.80]) by smtp.gmail.com with ESMTPSA id z43sm2356454qtb.92.2020.04.01.14.40.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 01 Apr 2020 14:40:33 -0700 (PDT) From: "Guilherme G. Piccoli" To: kernel-team@lists.ubuntu.com Subject: [X/B] [PATCH 1/1] net: ena: Add PCI shutdown handler to allow safe kexec Date: Wed, 1 Apr 2020 18:40:26 -0300 Message-Id: <20200401214027.32062-2-gpiccoli@canonical.com> X-Mailer: git-send-email 2.25.2 In-Reply-To: <20200401214027.32062-1-gpiccoli@canonical.com> References: <20200401214027.32062-1-gpiccoli@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: gpiccoli@canonical.com Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" BugLink: https://bugs.launchpad.net/bugs/1869948 Currently ENA only provides the PCI remove() handler, used during rmmod for example. This is not called on shutdown/kexec path; we are potentially creating a failure scenario on kexec: (a) Kexec is triggered, no shutdown() / remove() handler is called for ENA; instead pci_device_shutdown() clears the master bit of the PCI device, stopping all DMA transactions; (b) Kexec reboot happens and the device gets enabled again, likely having its FW with that DMA transaction buffered; then it may trigger the (now invalid) memory operation in the new kernel, corrupting kernel memory area. This patch aims to prevent this, by implementing a shutdown() handler quite similar to the remove() one - the difference being the handling of the netdev, which is unregistered on remove(), but following the convention observed in other drivers, it's only detached on shutdown(). This prevents an odd issue in AWS Nitro instances, in which after the 2nd kexec the next one will fail with an initrd corruption, caused by a wild DMA write to invalid kernel memory. The lspci output for the adapter present in my instance is: 00:05.0 Ethernet controller [0200]: Amazon.com, Inc. Elastic Network Adapter (ENA) [1d0f:ec20] Suggested-by: Gavin Shan Signed-off-by: Guilherme G. Piccoli Acked-by: Sameeh Jubran Signed-off-by: David S. Miller (backported from commit 428c491332bca498c8eb2127669af51506c346c7) [gpiccoli: context adjustments.] Signed-off-by: Guilherme G. Piccoli --- drivers/net/ethernet/amazon/ena/ena_netdev.c | 51 ++++++++++++++++---- 1 file changed, 41 insertions(+), 10 deletions(-) diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c index 1db0d8fe7d38..ef92ac190083 100644 --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c @@ -3675,13 +3675,15 @@ static int ena_sriov_configure(struct pci_dev *dev, int numvfs) /*****************************************************************************/ /*****************************************************************************/ -/* ena_remove - Device Removal Routine +/* __ena_shutoff - Helper used in both PCI remove/shutdown routines * @pdev: PCI device information struct + * @shutdown: Is it a shutdown operation? If false, means it is a removal * - * ena_remove is called by the PCI subsystem to alert the driver - * that it should release a PCI device. + * __ena_shutoff is a helper routine that does the real work on shutdown and + * removal paths; the difference between those paths is with regards to whether + * dettach or unregister the netdevice. */ -static void ena_remove(struct pci_dev *pdev) +static void __ena_shutoff(struct pci_dev *pdev, bool shutdown) { struct ena_adapter *adapter = pci_get_drvdata(pdev); struct ena_com_dev *ena_dev; @@ -3700,13 +3702,17 @@ static void ena_remove(struct pci_dev *pdev) cancel_work_sync(&adapter->reset_task); - rtnl_lock(); + rtnl_lock(); /* lock released inside the below if-else block */ ena_destroy_device(adapter, true); - rtnl_unlock(); - - unregister_netdev(netdev); - - free_netdev(netdev); + if (shutdown) { + netif_device_detach(netdev); + dev_close(netdev); + rtnl_unlock(); + } else { + rtnl_unlock(); + unregister_netdev(netdev); + free_netdev(netdev); + } ena_com_rss_destroy(ena_dev); @@ -3721,6 +3727,30 @@ static void ena_remove(struct pci_dev *pdev) vfree(ena_dev); } +/* ena_remove - Device Removal Routine + * @pdev: PCI device information struct + * + * ena_remove is called by the PCI subsystem to alert the driver + * that it should release a PCI device. + */ + +static void ena_remove(struct pci_dev *pdev) +{ + __ena_shutoff(pdev, false); +} + +/* ena_shutdown - Device Shutdown Routine + * @pdev: PCI device information struct + * + * ena_shutdown is called by the PCI subsystem to alert the driver that + * a shutdown/reboot (or kexec) is happening and device must be disabled. + */ + +static void ena_shutdown(struct pci_dev *pdev) +{ + __ena_shutoff(pdev, true); +} + #ifdef CONFIG_PM /* ena_suspend - PM suspend callback * @pdev: PCI device information struct @@ -3770,6 +3800,7 @@ static struct pci_driver ena_pci_driver = { .id_table = ena_pci_tbl, .probe = ena_probe, .remove = ena_remove, + .shutdown = ena_shutdown, #ifdef CONFIG_PM .suspend = ena_suspend, .resume = ena_resume,