From patchwork Thu Jun 21 19:06:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 932890 Return-Path: X-Original-To: incoming-dt@patchwork.ozlabs.org Delivered-To: patchwork-incoming-dt@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=devicetree-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=arm.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 41BWXg1NFjz9rxs for ; Fri, 22 Jun 2018 05:12:03 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932965AbeFUTLz (ORCPT ); Thu, 21 Jun 2018 15:11:55 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:53772 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753972AbeFUTLy (ORCPT ); Thu, 21 Jun 2018 15:11:54 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 87B791435; Thu, 21 Jun 2018 12:11:53 -0700 (PDT) Received: from ostrya.cambridge.arm.com (ostrya.cambridge.arm.com [10.1.210.39]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 086FD3F246; Thu, 21 Jun 2018 12:11:49 -0700 (PDT) From: Jean-Philippe Brucker To: iommu@lists.linux-foundation.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, virtio-dev@lists.oasis-open.org, devicetree@vger.kernel.org, kvmarm@lists.cs.columbia.edu Cc: joro@8bytes.org, alex.williamson@redhat.com, robh+dt@kernel.org, mark.rutland@arm.com, mst@redhat.com, jasowang@redhat.com, marc.zyngier@arm.com, robin.murphy@arm.com, will.deacon@arm.com, lorenzo.pieralisi@arm.com, eric.auger@redhat.com, peterx@redhat.com, bharat.bhushan@nxp.com, tnowicki@caviumnetworks.com, jayachandran.nair@cavium.com, kevin.tian@intel.com Subject: [PATCH v2 0/5] Add virtio-iommu driver Date: Thu, 21 Jun 2018 20:06:50 +0100 Message-Id: <20180621190655.56391-1-jean-philippe.brucker@arm.com> X-Mailer: git-send-email 2.17.0 MIME-Version: 1.0 Sender: devicetree-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org Implement the base virtio-iommu driver, following version 0.7 of the specification [1]. Changes since last version [2]: * Address comments, thanks again for the review. * As suggested, add a DT binding description in patch 1. * Depend on VIRTIO_MMIO=y to fix a build failure¹ * Switch to v0.7 of the spec, which changes resv_mem parameters and adds an MMIO flag. These are trivial but not backward compatible. Once device or driver is upstream, updates to the spec will rely on feature bits to stay compatible with this code. * Implement the new tlb_sync interface, by splitting add_req() and sync_req(). I noticed a small improvement on netperf stream because the synchronous iommu_unmap() also benefits from this. Other experiments, such as using kmem_cache for requests instead of kmalloc, didn't show any improvement. Driver is available on branch virtio-iommu/v0.7 [3], and the x86+ACPI prototype is on branch virtio-iommu/devel. That x86 code hasn't changed, it still requires the DMA ifdeffery and I lack the expertise to tidy it up. The kvmtool example device has been cleaned up and is available on branch virtio-iommu/v0.7 [4]. Feedback welcome! Thanks, Jean [1] virtio-iommu specification v0.7 https://www.spinics.net/lists/linux-virtualization/msg34127.html [2] virtio-iommu driver v1 https://www.spinics.net/lists/kvm/msg164322.html [3] git://linux-arm.org/linux-jpb.git virtio-iommu/v0.7 http://www.linux-arm.org/git?p=linux-jpb.git;a=shortlog;h=refs/heads/virtio-iommu/v0.7 [4] git://linux-arm.org/kvmtool-jpb.git virtio-iommu/v0.7 --- ¹ A word on the module story. Because of complex dependencies IOMMU drivers cannot yet be .ko modules. Enabling it is outside the scope of this series but I have a small prototype on branch virtio-iommu/ module-devel. It seems desirable since some distros currently ship the transport code as module and are unlikely to change this on our account. Note that this series works fine with arm64 defconfig, which selects VIRTIO_MMIO=y. I could use some help to clean this up. Currently my solution is to split virtio-iommu into a module and a 3-lines built-in stub, which isn't graceful but could have been worse. Keeping the whole virtio-iommu driver as builtin would require accessing any virtio utility through get_symbol. So far we only have seven of those and could keep a list of pointer ops, but I find this solution terrible. If virtio or virtio_mmio is a module, virtio-iommu also needs to be one. The minimal set of changes to make a module out of an OF-based IOMMU driver seems to be: * Export IOMMU symbols used by drivers * Don't give up deferring probe in of_iommu * Move IOMMU OF tables to .rodata * Create a static virtio-iommu stub that declares the virtio-iommu OF table entry. The build system doesn't pick up IOMMU_OF_DECLARE when done from an object destined to be built as module :( * Create a device_link between endpoint and IOMMU, to ensure that removing the IOMMU driver also unbinds the endpoint driver. Putting this in IOMMU core seems like a better approach since even when not built as module, unbinding an IOMMU device via sysfs will cause crashes. With this, as long as virtio-mmio isn't loaded, the OF code defers probe of endpoints that depend on virtio-iommu. Once virtio-mmio is loaded, the probe is still deferred until virtio-iommu registers itself to the virtio bus. Once virtio-iommu is loaded, probe of endpoints managed by the IOMMU follows. I'll investigate ACPI IORT when I find some time, though I don't expect much complication and suggest we tackle one problem at a time. Since it is a luxury that requires changes to the IOMMU core, module support is left as a future improvement. --- Jean-Philippe Brucker (5): dt-bindings: virtio: Specify #iommu-cells value for a virtio-iommu iommu: Add virtio-iommu driver iommu/virtio: Add probe request iommu/virtio: Add event queue vfio: Allow type-1 IOMMU instantiation for ARM .../devicetree/bindings/virtio/mmio.txt | 8 + MAINTAINERS | 6 + drivers/iommu/Kconfig | 11 + drivers/iommu/Makefile | 1 + drivers/iommu/virtio-iommu.c | 1164 +++++++++++++++++ drivers/vfio/Kconfig | 2 +- include/uapi/linux/virtio_ids.h | 1 + include/uapi/linux/virtio_iommu.h | 172 +++ 8 files changed, 1364 insertions(+), 1 deletion(-) create mode 100644 drivers/iommu/virtio-iommu.c create mode 100644 include/uapi/linux/virtio_iommu.h