[for-9.2,v6,12/12] docs/specs: add riscv-iommu

Add a simple guideline to use the existing RISC-V IOMMU support we just
added.

This doc will be updated once we add the riscv-iommu-sys device.

Signed-off-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>
---
 docs/specs/index.rst       |  1 +
 docs/specs/riscv-iommu.rst | 80 ++++++++++++++++++++++++++++++++++++++
 docs/system/riscv/virt.rst | 13 +++++++
 3 files changed, 94 insertions(+)
 create mode 100644 docs/specs/riscv-iommu.rst

On Fri, Aug 2, 2024 at 1:46 AM Daniel Henrique Barboza
<dbarboza@ventanamicro.com> wrote:
>
> Add a simple guideline to use the existing RISC-V IOMMU support we just
> added.
>
> This doc will be updated once we add the riscv-iommu-sys device.
>
> Signed-off-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>

Reviewed-by: Alistair Francis <alistair.francis@wdc.com>

Alistair

> ---
>  docs/specs/index.rst       |  1 +
>  docs/specs/riscv-iommu.rst | 80 ++++++++++++++++++++++++++++++++++++++
>  docs/system/riscv/virt.rst | 13 +++++++
>  3 files changed, 94 insertions(+)
>  create mode 100644 docs/specs/riscv-iommu.rst
>
> diff --git a/docs/specs/index.rst b/docs/specs/index.rst
> index 1484e3e760..c68cd9ae6c 100644
> --- a/docs/specs/index.rst
> +++ b/docs/specs/index.rst
> @@ -33,3 +33,4 @@ guest hardware that is specific to QEMU.
>     virt-ctlr
>     vmcoreinfo
>     vmgenid
> +   riscv-iommu
> diff --git a/docs/specs/riscv-iommu.rst b/docs/specs/riscv-iommu.rst
> new file mode 100644
> index 0000000000..e3b8f7208e
> --- /dev/null
> +++ b/docs/specs/riscv-iommu.rst
> @@ -0,0 +1,80 @@
> +.. _riscv-iommu:
> +
> +RISC-V IOMMU support for RISC-V machines
> +========================================
> +
> +QEMU implements a RISC-V IOMMU emulation based on the RISC-V IOMMU spec
> +version 1.0 `iommu1.0`_.
> +
> +The emulation includes a PCI reference device, riscv-iommu-pci, that QEMU
> +RISC-V boards can use.  The 'virt' RISC-V machine is compatible with this
> +device.
> +
> +riscv-iommu-pci reference device
> +--------------------------------
> +
> +This device implements the RISC-V IOMMU emulation as recommended by the section
> +"Integrating an IOMMU as a PCIe device" of `iommu1.0`_: a PCI device with base
> +class 08h, sub-class 06h and programming interface 00h.
> +
> +As a reference device it doesn't implement anything outside of the specification,
> +so it uses a generic default PCI ID given by QEMU: 1b36:0014.
> +
> +To include the device in the 'virt' machine:
> +
> +.. code-block:: bash
> +
> +  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci,[optional_pci_opts] (...)
> +
> +This will add a RISC-V IOMMU PCI device in the board following any additional
> +PCI parameters (like PCI bus address).  The behavior of the RISC-V IOMMU is
> +defined by the spec but its operation is OS dependent.  As of this writing the
> +existing Linux kernel support `linux-v8`_, not yet merged, will configure the IOMMU
> +to create IOMMU groups with any eligible cards available in the system,
> +regardless of factors such as the order in which the devices are added in the
> +command line.
> +
> +This means that these command lines are equivalent as far as the current
> +IOMMU kernel driver behaves:
> +
> +.. code-block:: bash
> +
> +  $ qemu-system-riscv64 \
> +        -M virt,aia=aplic-imsic,aia-guests=5 \
> +        -device riscv-iommu-pci,addr=1.0,vendor-id=0x1efd,device-id=0xedf1 \
> +        -device e1000e,netdev=net1 -netdev user,id=net1,net=192.168.0.0/24 \
> +        -device e1000e,netdev=net2 -netdev user,id=net2,net=192.168.200.0/24 \
> +        (...)
> +
> +  $ qemu-system-riscv64 \
> +        -M virt,aia=aplic-imsic,aia-guests=5 \
> +        -device e1000e,netdev=net1 -netdev user,id=net1,net=192.168.0.0/24 \
> +        -device e1000e,netdev=net2 -netdev user,id=net2,net=192.168.200.0/24 \
> +        -device riscv-iommu-pci,addr=1.0,vendor-id=0x1efd,device-id=0xedf1 \
> +        (...)
> +
> +Both will create iommu groups for the two e1000e cards.
> +
> +Another thing to notice on `linux-v8`_ is that the kernel driver considers an IOMMU
> +identified as a Rivos device, i.e. it uses Rivos vendor ID.  To use the riscv-iommu-pci
> +device with the existing kernel support we need to emulate a Rivos PCI IOMMU by
> +setting 'vendor-id' and 'device-id':
> +
> +.. code-block:: bash
> +
> +  $ qemu-system-riscv64 -M virt        \
> +     -device riscv-iommu-pci,vendor-id=0x1efd,device-id=0xedf1 (...)
> +
> +Several options are available to control the capabilities of the device, namely:
> +
> +- "bus": the bus that the IOMMU device uses
> +- "ioatc-limit": size of the Address Translation Cache (default to 2Mb)
> +- "intremap": enable/disable MSI support
> +- "ats": enable ATS support
> +- "off" (Out-of-reset translation mode: 'on' for DMA disabled, 'off' for 'BARE' (passthrough))
> +- "s-stage": enable s-stage support
> +- "g-stage": enable g-stage support
> +
> +.. _iommu1.0: https://github.com/riscv-non-isa/riscv-iommu/releases/download/v1.0/riscv-iommu.pdf
> +
> +.. _linux-v8: https://lore.kernel.org/linux-riscv/cover.1718388908.git.tjeznach@rivosinc.com/
> diff --git a/docs/system/riscv/virt.rst b/docs/system/riscv/virt.rst
> index c3f94f63da..dd884ea919 100644
> --- a/docs/system/riscv/virt.rst
> +++ b/docs/system/riscv/virt.rst
> @@ -84,6 +84,19 @@ none``, as in
>
>  Firmware images used for pflash must be exactly 32 MiB in size.
>
> +riscv-iommu support
> +-------------------
> +
> +The board has support for the riscv-iommu-pci device by using the following
> +command line:
> +
> +.. code-block:: bash
> +
> +  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci (...)
> +
> +Refer to :ref:`riscv-iommu` for more information on how the RISC-V IOMMU support
> +works.
> +
>  Machine-specific options
>  ------------------------
>
> --
> 2.45.2
>
>

On 8/1/24 12:43 PM, Daniel Henrique Barboza wrote:
> Add a simple guideline to use the existing RISC-V IOMMU support we just
> added.
> 
> This doc will be updated once we add the riscv-iommu-sys device.
> 
> Signed-off-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>
> ---
>   docs/specs/index.rst       |  1 +
>   docs/specs/riscv-iommu.rst | 80 ++++++++++++++++++++++++++++++++++++++
>   docs/system/riscv/virt.rst | 13 +++++++
>   3 files changed, 94 insertions(+)
>   create mode 100644 docs/specs/riscv-iommu.rst
> 
> diff --git a/docs/specs/index.rst b/docs/specs/index.rst
> index 1484e3e760..c68cd9ae6c 100644
> --- a/docs/specs/index.rst
> +++ b/docs/specs/index.rst
> @@ -33,3 +33,4 @@ guest hardware that is specific to QEMU.
>      virt-ctlr
>      vmcoreinfo
>      vmgenid
> +   riscv-iommu
> diff --git a/docs/specs/riscv-iommu.rst b/docs/specs/riscv-iommu.rst
> new file mode 100644
> index 0000000000..e3b8f7208e
> --- /dev/null
> +++ b/docs/specs/riscv-iommu.rst
> @@ -0,0 +1,80 @@
> +.. _riscv-iommu:
> +
> +RISC-V IOMMU support for RISC-V machines
> +========================================
> +
> +QEMU implements a RISC-V IOMMU emulation based on the RISC-V IOMMU spec
> +version 1.0 `iommu1.0`_.
> +
> +The emulation includes a PCI reference device, riscv-iommu-pci, that QEMU
> +RISC-V boards can use.  The 'virt' RISC-V machine is compatible with this
> +device.
> +
> +riscv-iommu-pci reference device
> +--------------------------------
> +
> +This device implements the RISC-V IOMMU emulation as recommended by the section
> +"Integrating an IOMMU as a PCIe device" of `iommu1.0`_: a PCI device with base
> +class 08h, sub-class 06h and programming interface 00h.
> +
> +As a reference device it doesn't implement anything outside of the specification,
> +so it uses a generic default PCI ID given by QEMU: 1b36:0014.
> +
> +To include the device in the 'virt' machine:
> +
> +.. code-block:: bash
> +
> +  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci,[optional_pci_opts] (...)
> +
> +This will add a RISC-V IOMMU PCI device in the board following any additional
> +PCI parameters (like PCI bus address).  The behavior of the RISC-V IOMMU is
> +defined by the spec but its operation is OS dependent.  As of this writing the
> +existing Linux kernel support `linux-v8`_, not yet merged, will configure the IOMMU
> +to create IOMMU groups with any eligible cards available in the system,
> +regardless of factors such as the order in which the devices are added in the
> +command line.

I'm receiving emails from users about the kernel being cited here not being able to
do a VFIO passthrough.

They are correct. The explanation is that the current kernel patches does not have all
the support needed for VFIO passthrough and other features like irqbypass. For my
testing I've been using a kernel that has this additional support that is made available
in the Ventana Micro github mirror:

https://github.com/ventanamicro/linux/tree/dev-upstream


Until the kernel support is feature complete, feel free to use this kernel that Ventana
is maintaining updated with the latest IOMMU kernel version + other goodies.


I'll update v7 with this additional info in the docs. Thanks,


Daniel


> +
> +This means that these command lines are equivalent as far as the current
> +IOMMU kernel driver behaves:
> +
> +.. code-block:: bash
> +
> +  $ qemu-system-riscv64 \
> +        -M virt,aia=aplic-imsic,aia-guests=5 \
> +        -device riscv-iommu-pci,addr=1.0,vendor-id=0x1efd,device-id=0xedf1 \
> +        -device e1000e,netdev=net1 -netdev user,id=net1,net=192.168.0.0/24 \
> +        -device e1000e,netdev=net2 -netdev user,id=net2,net=192.168.200.0/24 \
> +        (...)
> +
> +  $ qemu-system-riscv64 \
> +        -M virt,aia=aplic-imsic,aia-guests=5 \
> +        -device e1000e,netdev=net1 -netdev user,id=net1,net=192.168.0.0/24 \
> +        -device e1000e,netdev=net2 -netdev user,id=net2,net=192.168.200.0/24 \
> +        -device riscv-iommu-pci,addr=1.0,vendor-id=0x1efd,device-id=0xedf1 \
> +        (...)
> +
> +Both will create iommu groups for the two e1000e cards.
> +
> +Another thing to notice on `linux-v8`_ is that the kernel driver considers an IOMMU
> +identified as a Rivos device, i.e. it uses Rivos vendor ID.  To use the riscv-iommu-pci
> +device with the existing kernel support we need to emulate a Rivos PCI IOMMU by
> +setting 'vendor-id' and 'device-id':
> +
> +.. code-block:: bash
> +
> +  $ qemu-system-riscv64 -M virt	\
> +     -device riscv-iommu-pci,vendor-id=0x1efd,device-id=0xedf1 (...)
> +
> +Several options are available to control the capabilities of the device, namely:
> +
> +- "bus": the bus that the IOMMU device uses
> +- "ioatc-limit": size of the Address Translation Cache (default to 2Mb)
> +- "intremap": enable/disable MSI support
> +- "ats": enable ATS support
> +- "off" (Out-of-reset translation mode: 'on' for DMA disabled, 'off' for 'BARE' (passthrough))
> +- "s-stage": enable s-stage support
> +- "g-stage": enable g-stage support
> +
> +.. _iommu1.0: https://github.com/riscv-non-isa/riscv-iommu/releases/download/v1.0/riscv-iommu.pdf
> +
> +.. _linux-v8: https://lore.kernel.org/linux-riscv/cover.1718388908.git.tjeznach@rivosinc.com/
> diff --git a/docs/system/riscv/virt.rst b/docs/system/riscv/virt.rst
> index c3f94f63da..dd884ea919 100644
> --- a/docs/system/riscv/virt.rst
> +++ b/docs/system/riscv/virt.rst
> @@ -84,6 +84,19 @@ none``, as in
>   
>   Firmware images used for pflash must be exactly 32 MiB in size.
>   
> +riscv-iommu support
> +-------------------
> +
> +The board has support for the riscv-iommu-pci device by using the following
> +command line:
> +
> +.. code-block:: bash
> +
> +  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci (...)
> +
> +Refer to :ref:`riscv-iommu` for more information on how the RISC-V IOMMU support
> +works.
> +
>   Machine-specific options
>   ------------------------
>