From patchwork Thu May 26 10:31:11 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Talat Batheesh X-Patchwork-Id: 626657 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) by ozlabs.org (Postfix) with ESMTP id 3rFq4X0ncvz9t3x; Thu, 26 May 2016 23:00:12 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=Mellanox.com header.i=@Mellanox.com header.b=CGQ5oVVG; dkim-atps=neutral Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.76) (envelope-from ) id 1b5utT-0001J2-En; Thu, 26 May 2016 13:00:07 +0000 Received: from mail-am1on0085.outbound.protection.outlook.com ([157.56.112.85] helo=emea01-am1-obe.outbound.protection.outlook.com) by huckleberry.canonical.com with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.76) (envelope-from ) id 1b5sZg-0003Jy-On for kernel-team@lists.ubuntu.com; Thu, 26 May 2016 10:31:32 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=+Yw3eGyyMtCm2vzfcb2lt6ok//+rdNrWDz6fKJZTu5s=; b=CGQ5oVVGS53tIo0S5VlNMWhBGDuxfLIk2YsZcATt26o9mM3JFFP3r13mIQusJzdnKSgoY9Tq5MGVhua1mazK8XHtfr6A14rfDLhmJGIQoWHZKrw1UQ7H6o9FHj5CqLTgaMLD5yl99+JUs14fTK1kEo4obReNdfn3BtkdFZjJqTQ= Authentication-Results: lists.ubuntu.com; dkim=none (message not signed) header.d=none;lists.ubuntu.com; dmarc=none action=none header.from=mellanox.com; Received: from dev-h-vrt-005.mth.labs.mlnx (193.47.165.251) by HE1PR05MB1306.eurprd05.prod.outlook.com (10.162.250.151) with Microsoft SMTP Server (TLS) id 15.1.501.7; Thu, 26 May 2016 10:31:30 +0000 From: Talat Batheesh To: Subject: [PATCH] [xenial] [PATCH] (upstream) net/mlx5: Add pci shutdown callback Date: Thu, 26 May 2016 13:31:11 +0300 Message-ID: <1464258671-18244-1-git-send-email-talatb@mellanox.com> X-Mailer: git-send-email 2.5.0 MIME-Version: 1.0 X-Originating-IP: [193.47.165.251] X-ClientProxiedBy: AMSPR04CA0046.eurprd04.prod.outlook.com (10.242.87.164) To HE1PR05MB1306.eurprd05.prod.outlook.com (10.162.250.151) X-MS-Office365-Filtering-Correlation-Id: 623341d5-d64b-41ca-6cc0-08d38550e747 X-Microsoft-Exchange-Diagnostics: 1; HE1PR05MB1306; 2:hzHG0R99TwYpZ7bp3Ivsg7DD3EcGjiDC94aOMzMnVsKD+nvN4WH2nLmlttRLfvQvTrl1iHKR8hGHh1xee8bjgsMD7DPoTt+4IIxXY/0Re+Sa5YcQXJvuj899hWSS93NB2m/LhlSlzCSjsI/o5AeXu4ye/8idI8jpITttLr+ljv+8xM01RLepVxyZLWbudEFi; 3:CVgthUzA+FA7KGoJur77dcInsvN1oQ88y0cpP0FRMZNPx1DaZ0+o3oEgbE7qhZoA9s03Z5qvgopM03uUQPU1bRB6yuvb+b/AsOxNWy5z1U0tSIRHJMqjllwKta87eiQz X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:HE1PR05MB1306; X-Microsoft-Exchange-Diagnostics: 1; HE1PR05MB1306; 25:hbgheYWlBaw8gd6ON3Cy2d0X612P6g/oNYpiQ2i9RYaurLn4R8aYHRGXSZULUJGGa5k07855zeHaG5iv94VecId9YqPduK3+xuVHiAmk4q0aFxQoRz9oPnvb/Cv2dXJS50Lh1zODTTJtPGwLWIIqdvSdcyBfHjly+xSLBesd8VmvQ8wFqwkReAOxMIjpb1liAdL1f6Z5z26xkzJ61x0lxVPvVKCxwpMCcv26hXXaEEczVeKuZ/KZwrXp+J9TygEB84Iye6YBGaH/fXu8+YlKe/VcMJKOXH7zCXjDMSTWpMfvea2c94tMYDv3OJ9yu06jbScuj9yIa/ZAAgQXOYOh3kUOpHeztp0aRvF8SGL+CLLRDm/hJ3cD26nDFXdVaM5R7GHbYiUoKTU0ssuD6/U3NK6E3KB9/bLRguxLtXskppYl2zS9Xvwk6ixx883lMzmPaZ682RZ9A8VM2V48tomGtx1kjRhlVhPUobCQP+meSZ/7pK5Z1Y1VnwJbtTksmg+AbkVUUaxUGnUwPbPnq/LUPeA0PXOjIW9ZQuovVaxTR0sBTp+JWs+YsQ7n95sxcn511rrTGM1IrtSX590b/sYbQrwYa9jqQVf38PHTgPjki77PEGE5dojOCMQ1/E5RDFUg/5QKXCPOiN4WzAYLKCLjnzLVbBkWOwsjZNwFfWkhXCnXC/9LKyR1afOwG3efA3cofl/4t94p4SOuKQ2Viy8ooj5MhQx0ftjYF7iDcxu4yQb7oYzzjbsDIq8V+eHufupn X-Microsoft-Exchange-Diagnostics: 1; HE1PR05MB1306; 20:+5kcz0Clj1vyxBZnwW1oTQh4SoxMM576fuRREnZaH6ii5gwfjtWE3NGGM1g7AbMh2dhQXRfis8pkhHDKWFiZWrs9e+8aZNRNwOpOxS5zgpmyYKzG2Mj/xQQ04rD8QyTDQaah5//ndNjZDKw+rnrTO/GDwSup4Go3F4eDcEBd0bx3B25Xy9WypAgolzrxoAwOZ9YTQFDESDWksOKYtU+IBWWIISjqwQ+831yeyaLD7gNOCMrGB4v7ixiYN8fTdDtJHVeQgMqitV3lkPnGQcvptHgt5m308q7/wTioUpIYobDo5oafeT7LtAKv1M+cY28qXe1hscjRehfAERMr4ZWkRs/YMAjSX+AOTSoelMpRs+9EzmNdaoU5omZO5SsWUXEAE3Z931GONCpvJABshSiJWUGl7O5RUHr2+haRXRVgXUGbwNSuCJiShSfeIMRACiYVQsCFUXAOu5XCm0YWQ2sxivv6vuPQZqHeOp5L6hCFocOIhJAbxWxMaDVMLiVrLI/w X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(211254476889703)(24339691489775); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6055026); SRVR:HE1PR05MB1306; BCL:0; PCL:0; RULEID:; SRVR:HE1PR05MB1306; X-Microsoft-Exchange-Diagnostics: 1; HE1PR05MB1306; 4:6oYN4I/JYozkmjLA1l5etb8Djmu+DpkmI+04keqUIIo/ITlEa6kjmOOyaiBDfuQFdfszIu2oG8mHHvTwz5IamSx9oA7HlOpfJBkTQ3/Er1fdk6qzWG+wkLiRUqhAHTjlBsyfqHEVNJAbp0vemR55Y8sA5ydy9BebsmX8xJqcEqwgIqclM22YFuy8/TWsPTJOT3KuswoAbh7dFP6WkQiya7t7D4CTyDcjTbyvulRWO7bfiCGsS96juBlOaqdZ1/MIejzgnIT5t4o4h/W5rv1v6ucSpeTPHCYaoSt6fRB5IGuORBCC21oMn/4PuNKHeT9sj+hNzEP5PB+kaZXf4qKB8C5mTkxrmx6os6qlm3WAQE6WEBdZiRDfVCjnDBQ9d9I5bPYBtSmeSbNY/csHULJLNOEwnUHZy2xYlzoPf/w4D2un5yz8u0FAxyu9nWzKLYxC4PCmg+fvg9EfFaMQIXLdNQ== X-Forefront-PRVS: 0954EE4910 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(50226002)(4326007)(66066001)(5008740100001)(2906002)(2351001)(229853001)(110136002)(5004730100002)(450100001)(189998001)(586003)(107886002)(6116002)(3846002)(81166006)(36756003)(5003940100001)(92566002)(8676002)(47776003)(19580395003)(77096005)(19580405001)(15975445007)(42186005)(33646002)(50986999)(86362001)(48376002)(50466002)(4001430100002); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR05MB1306; H:dev-h-vrt-005.mth.labs.mlnx; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; HE1PR05MB1306; 23:4xJ2y7TF7EZqOsbYmHGDLfX9MhHZVD3UoJegOGQ03Jy6C1xmw5AmZamDpGM+EjtVx19EmZmhVqzdOJxaCrWrSG/dkvJcrQoSFTEczT5gWj5jOps73bXnc2ZJmQ93XkbkdNz8zNRlcO4urVsYy99UGH+8Aev/DDT/BH8DOQ+6TreInJdhGsTEl4tQnndblaE6/4u1TQuL1Fz5AJjGF+xUQuo3bKlCsMhwnDo3TvAKzrFWf5mcJEbukzNq4Zlz0PbPgORXDh9StvTayIj/ocMrWhJyDHyGuT4UNHzp0qrZzC4OVMEuw9BS4Hs1afcw7Jjfy9achQdtlVvWPYumdTB+Aho4xgf5jqb0gl27PsT1Co/40sazv9+BhLDjyBGy+ZSdMYqKCGJYA3kMSa7oZKzUpO6JeKZNbRJBmiR2u3I8eeEQUGuiQLxXi5shP2iWKOr9ddXdTyjwp6Codvgj6BXUokzoEy+ZEGX+WTTH+nTTgkeWzZljji/ya0XcTMk6uZ4/NHyMwNxKc7Nz+1A+1Cvg8m9VYQDg5V1epf7l1LxCr3JxGqOHT3U0wcQ9kcWqcwpvwyRH/K3HmGp81cU7wCJCV7iXWZm7CCDm7fBd/9dGk5xeje3X6hNpi4t+4mRPzVOa262chCKmow6YCjXzjT8F2qNppLJ80VsJ0Lx9q0fy52IydKnkgHjEtA4+zYsc6A6Aw0ggIPQGZXOOj0Wla06lgy4O1LgwzXCbqiPjrpIrBd3APq6pGPmR4SGblcV+aIU1Wffqb+nofZ9cuvisW7R1tT4HmeoG4YUONWdlSdl/pYr96y/A/iASxOh89mnuOv5bhteGhI3oGoQdd7K5jjMwJ5dEhQUTjH36baN4o0aJLMjRKMXACoAeqIivajw/ShxEwNj9UVITnZbcFG3hIsnqMK4x2TaDqyx6ws0soWZ4uovJUYih0E4tugAZU8xwIJHr X-Microsoft-Exchange-Diagnostics: 1; HE1PR05MB1306; 5:gxwPHEHI9H8jyYyl6OzXsHEjXr3l7arxnNQesIAuyQk0XHYkwxMJCWUmLlKi9yuuvQ84zonLMX89hQleSGrHx+XDaVcM/d/wPFWIZFc2LCP/2Sp/t0Auay6FEHpzq4x9sVebOKuRcvaMvHMTfDpgKA==; 24:y4dKzi0+fItWnvJKw4sEucOcSdEKUky5zRxTNXGUQQLQqi8sOdX85YzK/v99Uul8cg2zaB+D7t5jsEJHQnczn7lSvWMGAR4KUaHqMjQnb5E=; 7:H0c3Re1tBMq7CSyqLqNoKOyIwwJ6tDcBOHlecEzMoBOWkvq5icufvPbvrqkWXDOWA9JtoObiR7ahnYBcsoEdb3qyvTgKclbrhSbvDzboaYxH89dWAIC33Zt0M1b7c9eZ50MGdtXHguk9WP5Vy4DCq80ckc/8/bggLT8WOdAusDY= X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 May 2016 10:31:30.8382 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR05MB1306 X-Mailman-Approved-At: Thu, 26 May 2016 13:00:02 +0000 Cc: shanim@mellanox.com X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.14 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: kernel-team-bounces@lists.ubuntu.com From: Majd Dibbiny BugLink:https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1585978 This patch introduces kexec support for mlx5. When switching kernels, kexec() calls shutdown, which unloads the driver and cleans its resources. In addition, remove unregister netdev from shutdown flow. This will allow a clean shutdown, even if some netdev clients did not release their reference from this netdev. Releasing The HW resources only is enough as the kernel is shutting down Signed-off-by: Majd Dibbiny Signed-off-by: Tariq Toukan Signed-off-by: Haggai Abramovsky Signed-off-by: Saeed Mahameed Signed-off-by: David S. Miller (cherry picked from commit 5fc7197d3a256d9c5de3134870304b24892a4908) Signed-off-by: Talat Batheesh --- drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 15 +++++++++++++-- drivers/net/ethernet/mellanox/mlx5/core/main.c | 23 +++++++++++++++++++---- include/linux/mlx5/driver.h | 7 ++++--- 3 files changed, 36 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c index 9923450..42e0542 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c @@ -2323,7 +2323,16 @@ static void mlx5e_destroy_netdev(struct mlx5_core_dev *mdev, void *vpriv) schedule_work(&priv->set_rx_mode_work); mlx5e_disable_async_events(priv); flush_scheduled_work(); - unregister_netdev(netdev); + if (test_bit(MLX5_INTERFACE_STATE_SHUTDOWN, &mdev->intf_state)) { + netif_device_detach(netdev); + mutex_lock(&priv->state_lock); + if (test_bit(MLX5E_STATE_OPENED, &priv->state)) + mlx5e_close_locked(netdev); + mutex_unlock(&priv->state_lock); + } else { + unregister_netdev(netdev); + } + mlx5e_destroy_flow_tables(priv); mlx5e_destroy_tirs(priv); mlx5e_destroy_rqt(priv, MLX5E_SINGLE_RQ_RQT); @@ -2334,7 +2343,9 @@ static void mlx5e_destroy_netdev(struct mlx5_core_dev *mdev, void *vpriv) mlx5_dealloc_transport_domain(priv->mdev, priv->tdn); mlx5_core_dealloc_pd(priv->mdev, priv->pdn); mlx5_unmap_free_uar(priv->mdev, &priv->cq_uar); - free_netdev(netdev); + + if (!test_bit(MLX5_INTERFACE_STATE_SHUTDOWN, &mdev->intf_state)) + free_netdev(netdev); } static void *mlx5e_get_netdev(void *vpriv) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index cde356f..2229b14 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -917,7 +917,7 @@ static int mlx5_load_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv) int err; mutex_lock(&dev->intf_state_mutex); - if (dev->interface_state == MLX5_INTERFACE_STATE_UP) { + if (test_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state)) { dev_warn(&dev->pdev->dev, "%s: interface is up, NOP\n", __func__); goto out; @@ -1080,7 +1080,8 @@ static int mlx5_load_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv) if (err) pr_info("failed request module on %s\n", MLX5_IB_MOD); - dev->interface_state = MLX5_INTERFACE_STATE_UP; + clear_bit(MLX5_INTERFACE_STATE_DOWN, &dev->intf_state); + set_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state); out: mutex_unlock(&dev->intf_state_mutex); @@ -1156,7 +1157,7 @@ static int mlx5_unload_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv) } mutex_lock(&dev->intf_state_mutex); - if (dev->interface_state == MLX5_INTERFACE_STATE_DOWN) { + if (test_bit(MLX5_INTERFACE_STATE_DOWN, &dev->intf_state)) { dev_warn(&dev->pdev->dev, "%s: interface is down, NOP\n", __func__); goto out; @@ -1190,7 +1191,8 @@ static int mlx5_unload_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv) mlx5_cmd_cleanup(dev); out: - dev->interface_state = MLX5_INTERFACE_STATE_DOWN; + clear_bit(MLX5_INTERFACE_STATE_UP, &dev->intf_state); + set_bit(MLX5_INTERFACE_STATE_DOWN, &dev->intf_state); mutex_unlock(&dev->intf_state_mutex); return err; } @@ -1401,6 +1403,18 @@ static const struct pci_error_handlers mlx5_err_handler = { .resume = mlx5_pci_resume }; +static void shutdown(struct pci_dev *pdev) +{ + struct mlx5_core_dev *dev = pci_get_drvdata(pdev); + struct mlx5_priv *priv = &dev->priv; + + dev_info(&pdev->dev, "Shutdown was called\n"); + /* Notify mlx5 clients that the kernel is being shut down */ + set_bit(MLX5_INTERFACE_STATE_SHUTDOWN, &dev->intf_state); + mlx5_unload_one(dev, priv); + mlx5_pci_disable_device(dev); +} + static const struct pci_device_id mlx5_core_pci_table[] = { { PCI_VDEVICE(MELLANOX, 0x1011) }, /* Connect-IB */ { PCI_VDEVICE(MELLANOX, 0x1012), MLX5_PCI_DEV_IS_VF}, /* Connect-IB VF */ @@ -1418,6 +1432,7 @@ static struct pci_driver mlx5_core_driver = { .id_table = mlx5_core_pci_table, .probe = init_one, .remove = remove_one, + .shutdown = shutdown, .err_handler = &mlx5_err_handler, .sriov_configure = mlx5_core_sriov_configure, }; diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 81066dc..2d2767d 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -510,8 +510,9 @@ enum mlx5_device_state { }; enum mlx5_interface_state { - MLX5_INTERFACE_STATE_DOWN, - MLX5_INTERFACE_STATE_UP, + MLX5_INTERFACE_STATE_DOWN = BIT(0), + MLX5_INTERFACE_STATE_UP = BIT(1), + MLX5_INTERFACE_STATE_SHUTDOWN = BIT(2), }; enum mlx5_pci_status { @@ -535,7 +536,7 @@ struct mlx5_core_dev { enum mlx5_device_state state; /* sync interface state */ struct mutex intf_state_mutex; - enum mlx5_interface_state interface_state; + unsigned long intf_state; void (*event) (struct mlx5_core_dev *dev, enum mlx5_dev_event event, unsigned long param);