From patchwork Mon Sep 23 22:35:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: William Tu X-Patchwork-Id: 1988704 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XCHrt4v3Rz1xsy for ; Tue, 24 Sep 2024 08:35:58 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1ssrem-0006va-Ab; Mon, 23 Sep 2024 22:35:48 +0000 Received: from mail-dm6nam11on2089.outbound.protection.outlook.com ([40.107.223.89] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1ssrek-0006ut-74 for kernel-team@lists.ubuntu.com; Mon, 23 Sep 2024 22:35:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Zy58kCwruXPZP7hZ7TJTpvO/Rx+tnW0BfRcdDeDS456jMAhHb+zKf3bClHDwSDxMaguw01juQyx9S9+1cobaoxFU+3QarE5zeDRudestoqBsuNvsD4rPbef/wNHYFAGWj5c8OY0saEt/MGcQSX0DorISwiN5Ip4VuH5pkBmYOHfGVxn91Ghba2lbbmEMJkfZggRvzf0GzIQz6MzPrxeIIK3E2zb8roLZBuWfFh/lWJGPPKM6sjyqxK/QlwmwKZh8daOtVhm+c4JAEAZDYpzwiNpWMpySK5vOZX+p70kXGNGKnUMYA8101obIFOLl/6AA0pjeLkCIE0h6s9FtZ7dGoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=klsfW9HkwzPlamWt5pGlj/VhPAI5WbaOyt5J3natptI=; b=WCzAIfJ/hY5Q8OzhEw1Rp22sih3/cUl3k8JqGy6CFNUraSyhtUvAxVn6wUkDtufo6zD+BMH/srDj86Decl2uAI5SsCSAwP64aCK17E8OohsbvA36EaxlPosLGk5fLTAheQW9OzmuLmOW9vfp/Cml5F5PoG8l39Su4o65JfkY3v+qC1RWmX5sFBv+ySv8Isa6MJCy3e+dSyP9tKvkkWb5O12I1ev3UE3a3+XRnekV/cYYz86AAu3QHzMhLHou6dJ07rdoFWyeY4pUZDHV+hObnYvQvPIAG0OsNZPGRSPfZ9febOg9h4GDHwGBrzGfcHiNgxoljkbRPJ22pRpk3O8Pqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=lists.ubuntu.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) Received: from BN9PR03CA0799.namprd03.prod.outlook.com (2603:10b6:408:13f::24) by SN7PR12MB7978.namprd12.prod.outlook.com (2603:10b6:806:34b::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.26; Mon, 23 Sep 2024 22:35:42 +0000 Received: from BL6PEPF0001AB78.namprd02.prod.outlook.com (2603:10b6:408:13f:cafe::21) by BN9PR03CA0799.outlook.office365.com (2603:10b6:408:13f::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7982.25 via Frontend Transport; Mon, 23 Sep 2024 22:35:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BL6PEPF0001AB78.mail.protection.outlook.com (10.167.242.171) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Mon, 23 Sep 2024 22:35:41 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 23 Sep 2024 15:35:27 -0700 Received: from drhqmail203.nvidia.com (10.126.190.182) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Mon, 23 Sep 2024 15:35:26 -0700 Received: from vdi.nvidia.com (10.127.8.12) by mail.nvidia.com (10.126.190.182) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Mon, 23 Sep 2024 15:35:26 -0700 From: William Tu To: Subject: [SRU][J:linux-bluefield][PATCH v6 0/6] Add VFIO P2P support Date: Mon, 23 Sep 2024 22:35:16 +0000 Message-ID: <20240923223522.1203691-1-witu@nvidia.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB78:EE_|SN7PR12MB7978:EE_ X-MS-Office365-Filtering-Correlation-Id: ab95a1b1-2685-4473-e11b-08dcdc200e53 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|36860700013|82310400026|376014|1800799024; X-Microsoft-Antispam-Message-Info: k8So/Sv23Ffa9tRdqHEHGX7pbLdWJN5h9RYs3ZB7YKUrizfsFw+RnawGdnVYW6zcnnsJ+en8kzTs/4rp7YEwUJDmMLNvAfuFGcGYd55Nu95hg0n3ZDOuYEYyIT/A7pNN+xTi3NNFb5CW7KjBd84IA8bWSg2TeINpbPUkDAuvEuu3Zj0wY1YOAhvV5ZWi3FP7vMxLFOHv5l9xJsKX++v6de+1fdndHx+cV3nThCJniAjhdAcO8cWqleGg9FhCGRSPAEci5Iv4B9fgAMEUi21oe3CyL9ycoaMizray/kCbHDgo1pEWomAlVF5S9pZRyNT/+XkmbaxmfGkRJg/RsVYuyybJHaXNXcCFBkBIZhdrpqADJnEsezVocewALI4bst///wVLigDCIgF3fZ/nb032/N86nyvlwskWDrVIEGeH0UdqxY7RkdlhFBZ5bWeCSEgPAtgP5DT4057fAggR9ibdGGHUizTy97eXmsaCT+02slLhD9OYOi1zgy38KOjZBYY3OfJA7Dqu79E03ZSC+kcE0HaZrjGXYI6pQor2ig9tiswBLksi9GcP6ZEO79lg6q5C22+FarrY45LIReUewXtZktQc7PJdtnJ38/7v1zdMVUjNF+4CZJ6FiQSWvFhrXHbm3wx98+eWbDB/xwCrI/ECm/f0EBf4qXx4zAErfh23oQJdeKRvYNTE84YVqGC9rsnlSoga0Ntav0aPm57R2K/UnVnO5cacPzh4uJJZq19rDZNdk8A911ToRVwQ6II+3hQKua0KHN178TphEY7otGAwnfdtQJCuxNY/k1hHJlFuiBd1UUZvu4jr7dH5jVpo9zr9sJSt0m5C5EiCJVlZDlxEU5pydIo3ZKGljDoA7scTwzpxroBDgyzLoWrfe4YzHUEDMdqzwjjuKGRxZGCgcBKSaSJWoNddabU70h75uwFH/rRJvdVbQGLraVJmf67gTodZ5zpTiIqrzZrk37EaZIWH9hxvHT1WCTpl+wHd0ICRwZKkYxe9NSG712jhJd3Ar6WmBDKIeVQRHsGHb1jxTkpLaCR9EqmwB3Vihb2Cg2X6tMsl9JBB6IdAEykv2jjHiMRhASQPW9eozozGu5aS5Q52511weQd7bwW3KXAlcotvKGoOVeyaXic5vVEWYUCNbj/yFvRWQak3YkG+5bniDHBe9nYhgvpMD16lLxjYevIiMUI5Dd2JLHzGQwWMuU446FZKKmSKoUm2vV2bPrj8/oaKTyKBvJoLXsj0jr6Kz1xk+i+i+URtyZQnxbtjY7Trcn2YlpBGFyXo7KSdURVaLMSkB6fRzRSLAFqA4O1UvhjBc9puLBiw3UOQhtEQ4UTGDp8xLPsOc/5ixHc8SzOP1UlmNsfRc3SVOwCXRuyiBDVXSdNg14+r7CFDBsyPC8Ex5c7ThWT4H66r6exPvIS8IUONcxSzwxzIaWcRsCmrLMS6mgy5eNKP5c6LobL11Fq6hRVX X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230040)(36860700013)(82310400026)(376014)(1800799024); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Sep 2024 22:35:41.5335 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ab95a1b1-2685-4473-e11b-08dcdc200e53 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB78.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7978 Received-SPF: softfail client-ip=40.107.223.89; envelope-from=witu@nvidia.com; helo=NAM11-DM6-obe.outbound.protection.outlook.com X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: vlad@nvidia.com, bodong@nvidia.com, sergeygo@nvidia.com, jgg@nvidia.com, zwaksman@nvidia.com Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" BugLink: https://bugs.launchpad.net/bugs/2077887 The series adds support for VFIO P2P feature for NVMe target offload. We took the patch (not upstreamed) below with some fix to make it work. https://patchwork.kernel.org/project/linux-rdma/cover/0-v2-472615b3877e+28f7-vfio_dma_buf_jgg@nvidia.com/ * Kernel Config Need to enable CONFIG_PCI_P2PDMA (enable p2p dma transaction) CONFIG_DMABUF_MOVE_NOTIFY (no need to pin down the memory) The feature doesn't work with IOMMU, so either we disable it for the system, "iommu.passthrough=1 iommu=off" in kernel boot parameter, or conditionally disable it for VFIO. CONFIG_VFIO_NOIOMMU (optional) - You should enable CONFIG_VFIO_NOIOMMU in the kernel config if your vfio module does not have vfio enable_unsafe_noiommu_mode in the parameter list. * /etc/modprobe.d/vfio.conf $ cat /etc/modprobe.d/vfio.conf # The kernel hungs when 0000:06:00.0 (Non-Volatile memory controller: KIOXIA # Corporation Device 0001) is bound to the vfio-pci module, and the following # error is printed: # # vfio-pci 0000:06:00.0: Unable to change power state from D3hot to D0, device inaccessible # # The configuration below is a workarround for the issue. options vfio-pci disable_idle_d3=1 # Allow using VFIO without IOMMU. This is only required when SMMU is disabled. options vfio enable_unsafe_noiommu_mode=1 * Setup and Test 1. For ease of testing, we take a BF3 and boot the OS into mmc instead of NVMe drive. So leaving NVMe drive free to use. 2. use a test program to verify its correctness. The program create ibv rdma program, open nvme drive, create dma vfio buf from nvme, and let the rdma program read (DMA P2P) the version number of the nvme drive. https://gitlab.com/Mellanox/spdk_team/vfio-dmabuf-test * UAPI changes see "vfio/pci: Allow MMIO regions to be exported through dma-buf" for detail v6: - remove comments, fix the vfio_pci_set_power_state code v5: - add back the missing power management code, suggested by Bartlomiej v4: - fix Makefile v1->v2: - introduce new ioctl uAPI for vfio dma-buf v2->v3: - squash the ioctl uapi change into "vfio/pci: Allow MMIO regions to be exported through dma-buf" Jason Gunthorpe (4): UBUNTU: SAUCE: dma-buf: Add dma_buf_try_get() UBUNTU: SAUCE: vfio: Add vfio_device_get() UBUNTU: SAUCE: vfio_pci: Do not open code pci_try_reset_function() UBUNTU: SAUCE: vfio/pci: Allow MMIO regions to be exported through dma-buf Sergey Gorenko (1): UBUNTU: SAUCE: vfio/pci: Fix p2p address William Tu (1): UBUNTU: [Config] bluefield: add config for VFIO P2P debian.bluefield/config/annotations | 2 + drivers/vfio/pci/Makefile | 1 + drivers/vfio/pci/dma_buf.c | 265 ++++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_config.c | 24 +-- drivers/vfio/pci/vfio_pci_core.c | 72 +++++--- drivers/vfio/pci/vfio_pci_priv.h | 28 +++ include/linux/dma-buf.h | 13 ++ include/linux/vfio.h | 5 + include/linux/vfio_pci_core.h | 1 + include/uapi/linux/vfio.h | 17 ++ 10 files changed, 392 insertions(+), 36 deletions(-) create mode 100644 drivers/vfio/pci/dma_buf.c create mode 100644 drivers/vfio/pci/vfio_pci_priv.h