From patchwork Mon Oct 10 20:28:35 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 680560 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3stBg04WnQz9sBR for ; Tue, 11 Oct 2016 07:34:04 +1100 (AEDT) Received: from localhost ([::1]:52461 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bthGq-0004xW-Aq for incoming@patchwork.ozlabs.org; Mon, 10 Oct 2016 16:34:00 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54181) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bthAt-0008UO-OL for qemu-devel@nongnu.org; Mon, 10 Oct 2016 16:27:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bthAq-0001bm-EA for qemu-devel@nongnu.org; Mon, 10 Oct 2016 16:27:51 -0400 Received: from hqemgate14.nvidia.com ([216.228.121.143]:2716) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bthAq-0001bR-2V for qemu-devel@nongnu.org; Mon, 10 Oct 2016 16:27:48 -0400 Received: from hqnvupgp07.nvidia.com (Not Verified[216.228.121.13]) by hqemgate14.nvidia.com id ; Mon, 10 Oct 2016 13:27:36 -0700 Received: from HQMAIL106.nvidia.com ([172.20.13.39]) by hqnvupgp07.nvidia.com (PGP Universal service); Mon, 10 Oct 2016 13:21:06 -0700 X-PGP-Universal: processed; by hqnvupgp07.nvidia.com on Mon, 10 Oct 2016 13:21:06 -0700 Received: from HQMAIL104.nvidia.com (172.18.146.11) by HQMAIL106.nvidia.com (172.18.146.12) with Microsoft SMTP Server (TLS) id 15.0.1210.3; Mon, 10 Oct 2016 20:27:46 +0000 Received: from kwankhede-dev.nvidia.com (172.20.13.39) by HQMAIL104.nvidia.com (172.18.146.11) with Microsoft SMTP Server (TLS) id 15.0.1210.3 via Frontend Transport; Mon, 10 Oct 2016 20:27:43 +0000 From: Kirti Wankhede To: , , , Date: Tue, 11 Oct 2016 01:58:35 +0530 Message-ID: <1476131317-358-5-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1476131317-358-1-git-send-email-kwankhede@nvidia.com> References: <1476131317-358-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.143 Subject: [Qemu-devel] [PATCH v8 4/6] docs: Add Documentation for Mediated devices X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: jike.song@intel.com, kvm@vger.kernel.org, kevin.tian@intel.com, qemu-devel@nongnu.org, Kirti Wankhede , bjsdjshi@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Add file Documentation/vfio-mediated-device.txt that include details of mediated device framework. Signed-off-by: Kirti Wankhede Signed-off-by: Neo Jia Change-Id: I137dd646442936090d92008b115908b7b2c7bc5d --- Documentation/vfio-mdev/vfio-mediated-device.txt | 219 +++++++++++++++++++++++ 1 file changed, 219 insertions(+) create mode 100644 Documentation/vfio-mdev/vfio-mediated-device.txt diff --git a/Documentation/vfio-mdev/vfio-mediated-device.txt b/Documentation/vfio-mdev/vfio-mediated-device.txt new file mode 100644 index 000000000000..c1eacb83807b --- /dev/null +++ b/Documentation/vfio-mdev/vfio-mediated-device.txt @@ -0,0 +1,219 @@ +/* + * VFIO Mediated devices + * + * Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved. + * Author: Neo Jia + * Kirti Wankhede + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +VFIO Mediated devices [1] +------------------------- + +There are more and more use cases/demands to virtualize the DMA devices which +don't have SR_IOV capability built-in. To do this, drivers of different +devices had to develop their own management interface and set of APIs and then +integrate it to user space software. We've identified common requirements and +unified management interface for such devices to make user space software +integration easier and simplify the device driver implementation to support such +I/O virtualization solution. + +The VFIO driver framework provides unified APIs for direct device access from +user space. It is an IOMMU/device agnostic framework for exposing direct device +access to user space, in a secure, IOMMU protected environment. This framework +is used for multiple devices like GPUs, network adapters and compute +accelerators. With direct device access, virtual machines or user space +applications have direct access of physical device. This framework is reused +for mediated devices. + +Mediated core driver provides a common interface for mediated device management +that can be used by drivers of different devices. This module provides a generic +interface to create/destroy a mediated device, add/remove it to mediated bus +driver and add/remove device to an IOMMU group. It also provides an interface to +register bus driver, for example, Mediated VFIO mdev driver is designed for +mediated devices and supports VFIO APIs. Mediated bus driver add/delete mediated +device to VFIO Group. + +Below is the high Level block diagram, with NVIDIA, Intel and IBM devices +as example, since these are the devices which are going to actively use +this module as of now. + + +---------------+ + | | + | +-----------+ | mdev_register_driver() +--------------+ + | | | +<------------------------+ | + | | mdev | | | | + | | bus | +------------------------>+ vfio_mdev.ko |<-> VFIO user + | | driver | | probe()/remove() | | APIs + | | | | +--------------+ + | +-----------+ | + | | + | MDEV CORE | + | MODULE | + | mdev.ko | + | +-----------+ | mdev_register_device() +--------------+ + | | | +<------------------------+ | + | | | | | nvidia.ko |<-> physical + | | | +------------------------>+ | device + | | | | callbacks +--------------+ + | | Physical | | + | | device | | mdev_register_device() +--------------+ + | | interface | |<------------------------+ | + | | | | | i915.ko |<-> physical + | | | +------------------------>+ | device + | | | | callbacks +--------------+ + | | | | + | | | | mdev_register_device() +--------------+ + | | | +<------------------------+ | + | | | | | ccw_device.ko|<-> physical + | | | +------------------------>+ | device + | | | | callbacks +--------------+ + | +-----------+ | + +---------------+ + + +Registration Interfaces: +------------------------ + +Mediated core driver provides two types of registration interfaces: + +1. Registration interface for mediated bus driver: +-------------------------------------------------- + /* + * struct mdev_driver [2] - Mediated device's driver + * @name: driver name + * @probe: called when new device created + * @remove: called when device removed + * @driver: device driver structure + */ + struct mdev_driver { + const char *name; + int (*probe) (struct device *dev); + void (*remove) (struct device *dev); + struct device_driver driver; + }; + +Mediated bus driver for mdev should use this interface to register and +unregister with core driver respectively: + +extern int mdev_register_driver(struct mdev_driver *drv, struct module *owner); +extern void mdev_unregister_driver(struct mdev_driver *drv); + +Mediated bus driver is responsible to add/delete mediated devices to/from VFIO +group when devices are bound and unbound to the driver. + +2. Physical device driver interface: +------------------------------------ +This interface [3] provides a set of APIs to manage physical device related work +in its driver. APIs are: + +* dev_attr_groups: attributes of parent device. +* mdev_attr_groups: attributes of mediated device. +* supported_type_groups: attributes to define supported types. It is mandatory + to provide supported types. +* create: to allocate basic resources in driver for a mediated device. +* remove: to free resources in driver when mediated device is destroyed. +* open: open callback of mediated device +* release: close callback of mediated device +* read : read emulation callback. +* write: write emulation callback. +* mmap: mmap emulation callback. +* ioctl: ioctl callback. + +Drivers should use these interfaces to register and unregister device to mdev +core driver respectively: + +extern int mdev_register_device(struct device *dev, + const struct parent_ops *ops); +extern void mdev_unregister_device(struct device *dev); + +Mediated device management interface via sysfs +---------------------------------------------- +Management interface via sysfs allows user space software, like libvirt, to +query and configure mediated device in a HW agnostic fashion. This management +interface provide flexibility to underlying physical device's driver to support +mediated device hotplug, multiple mediated devices per virtual machine, multiple +mediated devices from different physical devices, etc. + +Under per-physical device sysfs: +-------------------------------- + +* mdev_supported_types: + List of current supported mediated device types and its details are added +in this directory in following format: + +|- +|--- Vendor-specific-attributes [optional] +|--- mdev_supported_types +| |--- +| | |--- create +| | |--- name +| | |--- available_instances +| | |--- description /class +| | |--- [devices] +| |--- +| | |--- create +| | |--- name +| | |--- available_instances +| | |--- description /class +| | |--- [devices] +| |--- +| |--- create +| |--- name +| |--- available_instances +| |--- description /class +| |--- [devices] + +[TBD : description or class is yet to be decided. This will change.] + +Under per mdev device: +---------------------- + +|- +|--- $MDEV_UUID + |--- remove + |--- {link to its type} + |--- vendor-specific-attributes [optional] + +* remove: (write only) + Write '1' to 'remove' file would destroy mdev device. Vendor driver can + fail remove() callback if that device is active and vendor driver + doesn't support hot-unplug. + Example: + # echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove + + +Mediated device Hotplug: +------------------------ + +Mediated devices can be created and assigned during runtime. Procedure to +hot-plug mediated device is same as hot-plug PCI device. + +Translation APIs for Mediated device +------------------------------------ + +Below APIs are provided for user pfn to host pfn translation in VFIO driver: + +extern long vfio_pin_pages(struct device *dev, unsigned long *user_pfn, + long npage, int prot, unsigned long *phys_pfn); + +extern long vfio_unpin_pages(struct device *dev, unsigned long *pfn, + long npage); + +These functions call back into the backend IOMMU module using two callbacks of +struct vfio_iommu_driver_ops, pin_pages and unpin_pages [4]. Currently these are +supported in TYPE1 IOMMU module. To enable the same for other IOMMU backend +modules, such as PPC64 sPAPR module, they need to provide these two callback +functions. + +References +---------- + +[1] See Documentation/vfio.txt for more information on VFIO. +[2] struct mdev_driver in include/linux/mdev.h +[3] struct parent_ops in include/linux/mdev.h +[4] struct vfio_iommu_driver_ops in include/linux/vfio.h +