[v5,13/13] docs/specs: add riscv-iommu

Add a simple guideline to use the existing RISC-V IOMMU support we just
added.

This doc will be updated once we add the riscv-iommu-sys device.

Signed-off-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>
---
 docs/specs/index.rst       |  1 +
 docs/specs/riscv-iommu.rst | 55 ++++++++++++++++++++++++++++++++++++++
 docs/system/riscv/virt.rst | 13 +++++++++
 3 files changed, 69 insertions(+)
 create mode 100644 docs/specs/riscv-iommu.rst

On Tue, Jul 9, 2024 at 3:37 AM Daniel Henrique Barboza
<dbarboza@ventanamicro.com> wrote:
>
> Add a simple guideline to use the existing RISC-V IOMMU support we just
> added.
>
> This doc will be updated once we add the riscv-iommu-sys device.
>
> Signed-off-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>
> ---
>  docs/specs/index.rst       |  1 +
>  docs/specs/riscv-iommu.rst | 55 ++++++++++++++++++++++++++++++++++++++
>  docs/system/riscv/virt.rst | 13 +++++++++
>  3 files changed, 69 insertions(+)
>  create mode 100644 docs/specs/riscv-iommu.rst
>
> diff --git a/docs/specs/index.rst b/docs/specs/index.rst
> index 1484e3e760..c68cd9ae6c 100644
> --- a/docs/specs/index.rst
> +++ b/docs/specs/index.rst
> @@ -33,3 +33,4 @@ guest hardware that is specific to QEMU.
>     virt-ctlr
>     vmcoreinfo
>     vmgenid
> +   riscv-iommu
> diff --git a/docs/specs/riscv-iommu.rst b/docs/specs/riscv-iommu.rst
> new file mode 100644
> index 0000000000..fa38ff7667
> --- /dev/null
> +++ b/docs/specs/riscv-iommu.rst
> @@ -0,0 +1,55 @@
> +.. _riscv-iommu:
> +
> +RISC-V IOMMU support for RISC-V machines
> +========================================
> +
> +QEMU implements a RISC-V IOMMU emulation based on the RISC-V IOMMU spec
> +version 1.0 [1].
> +
> +The emulation includes a PCI reference device, riscv-iommu-pci, that QEMU
> +RISC-V boards can use.  The 'virt' RISC-V machine is compatible with this
> +device.
> +
> +A platform device that implements the RISC-V IOMMU will be added in the
> +future.
> +
> +
> +riscv-iommu-pci reference device
> +--------------------------------
> +
> +This device implements the RISC-V IOMMU emulation as recommended by the section
> +"Integrating an IOMMU as a PCIe device" of [1]: a PCI device with base class 08h,
> +sub-class 06h and programming interface 00h.
> +
> +As a reference device it doesn't implement anything outside of the specification,
> +so it uses a generic default PCI ID given by QEMU: 1b36:0014.
> +
> +To include the device in the 'virt' machine:
> +
> +.. code-block:: bash
> +
> +  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci (...)

We should add a sentence saying what this does. As in what should a
user expect after they have done this

Alistair

> +
> +As of this writing the existing Linux kernel support [2], not yet merged, is being
> +created as a Rivos device, i.e. it uses Rivos vendor ID.  To use the riscv-iommu-pci
> +device with the existing kernel support we need to emulate a Rivos PCI IOMMU by
> +setting 'vendor-id' and 'device-id':
> +
> +.. code-block:: bash
> +
> +  $ qemu-system-riscv64 -M virt        \
> +     -device riscv-iommu-pci,vendor-id=0x1efd,device-id=0xedf1 (...)
> +
> +Several options are available to control the capabilities of the device, namely:
> +
> +- "bus"
> +- "ioatc-limit"
> +- "intremap"
> +- "ats"
> +- "off" (Out-of-reset translation mode: 'on' for DMA disabled, 'off' for 'BARE' (passthrough))
> +- "s-stage"
> +- "g-stage"
> +
> +
> +[1] https://github.com/riscv-non-isa/riscv-iommu/releases/download/v1.0/riscv-iommu.pdf
> +[2] https://lore.kernel.org/linux-riscv/cover.1718388908.git.tjeznach@rivosinc.com/
> diff --git a/docs/system/riscv/virt.rst b/docs/system/riscv/virt.rst
> index 9a06f95a34..8e9a2e4dda 100644
> --- a/docs/system/riscv/virt.rst
> +++ b/docs/system/riscv/virt.rst
> @@ -84,6 +84,19 @@ none``, as in
>
>  Firmware images used for pflash must be exactly 32 MiB in size.
>
> +riscv-iommu support
> +-------------------
> +
> +The board has support for the riscv-iommu-pci device by using the following
> +command line:
> +
> +.. code-block:: bash
> +
> +  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci (...)
> +
> +Refer to :ref:`riscv-iommu` for more information on how the RISC-V IOMMU support
> +works.
> +
>  Machine-specific options
>  ------------------------
>
> --
> 2.45.2
>
>

On 7/19/24 6:34 AM, Alistair Francis wrote:
> On Tue, Jul 9, 2024 at 3:37 AM Daniel Henrique Barboza
> <dbarboza@ventanamicro.com> wrote:
>>
>> Add a simple guideline to use the existing RISC-V IOMMU support we just
>> added.
>>
>> This doc will be updated once we add the riscv-iommu-sys device.
>>
>> Signed-off-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>
>> ---
>>   docs/specs/index.rst       |  1 +
>>   docs/specs/riscv-iommu.rst | 55 ++++++++++++++++++++++++++++++++++++++
>>   docs/system/riscv/virt.rst | 13 +++++++++
>>   3 files changed, 69 insertions(+)
>>   create mode 100644 docs/specs/riscv-iommu.rst
>>
>> diff --git a/docs/specs/index.rst b/docs/specs/index.rst
>> index 1484e3e760..c68cd9ae6c 100644
>> --- a/docs/specs/index.rst
>> +++ b/docs/specs/index.rst
>> @@ -33,3 +33,4 @@ guest hardware that is specific to QEMU.
>>      virt-ctlr
>>      vmcoreinfo
>>      vmgenid
>> +   riscv-iommu
>> diff --git a/docs/specs/riscv-iommu.rst b/docs/specs/riscv-iommu.rst
>> new file mode 100644
>> index 0000000000..fa38ff7667
>> --- /dev/null
>> +++ b/docs/specs/riscv-iommu.rst
>> @@ -0,0 +1,55 @@
>> +.. _riscv-iommu:
>> +
>> +RISC-V IOMMU support for RISC-V machines
>> +========================================
>> +
>> +QEMU implements a RISC-V IOMMU emulation based on the RISC-V IOMMU spec
>> +version 1.0 [1].
>> +
>> +The emulation includes a PCI reference device, riscv-iommu-pci, that QEMU
>> +RISC-V boards can use.  The 'virt' RISC-V machine is compatible with this
>> +device.
>> +
>> +A platform device that implements the RISC-V IOMMU will be added in the
>> +future.
>> +
>> +
>> +riscv-iommu-pci reference device
>> +--------------------------------
>> +
>> +This device implements the RISC-V IOMMU emulation as recommended by the section
>> +"Integrating an IOMMU as a PCIe device" of [1]: a PCI device with base class 08h,
>> +sub-class 06h and programming interface 00h.
>> +
>> +As a reference device it doesn't implement anything outside of the specification,
>> +so it uses a generic default PCI ID given by QEMU: 1b36:0014.
>> +
>> +To include the device in the 'virt' machine:
>> +
>> +.. code-block:: bash
>> +
>> +  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci (...)
> 
> We should add a sentence saying what this does. As in what should a
> user expect after they have done this

I suppose we can add a boilerplate sentence about what an IOMMU does, but aside
from that I'm not sure what to say about user expectation. If you're adding an
IOMMU (or any device in QEMU) it's expected that the user has some idea of what
the device is supposed to do in the board and so on hehe


Thanks,

Daniel

> 
> Alistair
> 
>> +
>> +As of this writing the existing Linux kernel support [2], not yet merged, is being
>> +created as a Rivos device, i.e. it uses Rivos vendor ID.  To use the riscv-iommu-pci
>> +device with the existing kernel support we need to emulate a Rivos PCI IOMMU by
>> +setting 'vendor-id' and 'device-id':
>> +
>> +.. code-block:: bash
>> +
>> +  $ qemu-system-riscv64 -M virt        \
>> +     -device riscv-iommu-pci,vendor-id=0x1efd,device-id=0xedf1 (...)
>> +
>> +Several options are available to control the capabilities of the device, namely:
>> +
>> +- "bus"
>> +- "ioatc-limit"
>> +- "intremap"
>> +- "ats"
>> +- "off" (Out-of-reset translation mode: 'on' for DMA disabled, 'off' for 'BARE' (passthrough))
>> +- "s-stage"
>> +- "g-stage"
>> +
>> +
>> +[1] https://github.com/riscv-non-isa/riscv-iommu/releases/download/v1.0/riscv-iommu.pdf
>> +[2] https://lore.kernel.org/linux-riscv/cover.1718388908.git.tjeznach@rivosinc.com/
>> diff --git a/docs/system/riscv/virt.rst b/docs/system/riscv/virt.rst
>> index 9a06f95a34..8e9a2e4dda 100644
>> --- a/docs/system/riscv/virt.rst
>> +++ b/docs/system/riscv/virt.rst
>> @@ -84,6 +84,19 @@ none``, as in
>>
>>   Firmware images used for pflash must be exactly 32 MiB in size.
>>
>> +riscv-iommu support
>> +-------------------
>> +
>> +The board has support for the riscv-iommu-pci device by using the following
>> +command line:
>> +
>> +.. code-block:: bash
>> +
>> +  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci (...)
>> +
>> +Refer to :ref:`riscv-iommu` for more information on how the RISC-V IOMMU support
>> +works.
>> +
>>   Machine-specific options
>>   ------------------------
>>
>> --
>> 2.45.2
>>
>>

On Wed, Jul 24, 2024 at 10:56 PM Daniel Henrique Barboza
<dbarboza@ventanamicro.com> wrote:
>
>
>
> On 7/19/24 6:34 AM, Alistair Francis wrote:
> > On Tue, Jul 9, 2024 at 3:37 AM Daniel Henrique Barboza
> > <dbarboza@ventanamicro.com> wrote:
> >>
> >> Add a simple guideline to use the existing RISC-V IOMMU support we just
> >> added.
> >>
> >> This doc will be updated once we add the riscv-iommu-sys device.
> >>
> >> Signed-off-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>
> >> ---
> >>   docs/specs/index.rst       |  1 +
> >>   docs/specs/riscv-iommu.rst | 55 ++++++++++++++++++++++++++++++++++++++
> >>   docs/system/riscv/virt.rst | 13 +++++++++
> >>   3 files changed, 69 insertions(+)
> >>   create mode 100644 docs/specs/riscv-iommu.rst
> >>
> >> diff --git a/docs/specs/index.rst b/docs/specs/index.rst
> >> index 1484e3e760..c68cd9ae6c 100644
> >> --- a/docs/specs/index.rst
> >> +++ b/docs/specs/index.rst
> >> @@ -33,3 +33,4 @@ guest hardware that is specific to QEMU.
> >>      virt-ctlr
> >>      vmcoreinfo
> >>      vmgenid
> >> +   riscv-iommu
> >> diff --git a/docs/specs/riscv-iommu.rst b/docs/specs/riscv-iommu.rst
> >> new file mode 100644
> >> index 0000000000..fa38ff7667
> >> --- /dev/null
> >> +++ b/docs/specs/riscv-iommu.rst
> >> @@ -0,0 +1,55 @@
> >> +.. _riscv-iommu:
> >> +
> >> +RISC-V IOMMU support for RISC-V machines
> >> +========================================
> >> +
> >> +QEMU implements a RISC-V IOMMU emulation based on the RISC-V IOMMU spec
> >> +version 1.0 [1].
> >> +
> >> +The emulation includes a PCI reference device, riscv-iommu-pci, that QEMU
> >> +RISC-V boards can use.  The 'virt' RISC-V machine is compatible with this
> >> +device.
> >> +
> >> +A platform device that implements the RISC-V IOMMU will be added in the
> >> +future.
> >> +
> >> +
> >> +riscv-iommu-pci reference device
> >> +--------------------------------
> >> +
> >> +This device implements the RISC-V IOMMU emulation as recommended by the section
> >> +"Integrating an IOMMU as a PCIe device" of [1]: a PCI device with base class 08h,
> >> +sub-class 06h and programming interface 00h.
> >> +
> >> +As a reference device it doesn't implement anything outside of the specification,
> >> +so it uses a generic default PCI ID given by QEMU: 1b36:0014.
> >> +
> >> +To include the device in the 'virt' machine:
> >> +
> >> +.. code-block:: bash
> >> +
> >> +  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci (...)
> >
> > We should add a sentence saying what this does. As in what should a
> > user expect after they have done this
>
> I suppose we can add a boilerplate sentence about what an IOMMU does, but aside
> from that I'm not sure what to say about user expectation. If you're adding an
> IOMMU (or any device in QEMU) it's expected that the user has some idea of what
> the device is supposed to do in the board and so on hehe

That's fair and I don't think we need to describe what an IOMMU does,
I more meant the fact it's attached via PCI.

We specifically say a "platform device will be added later" so I think
we should just be clear on what a PCI IOMMU is

Alistair

On 7/19/24 6:34 AM, Alistair Francis wrote:
> On Tue, Jul 9, 2024 at 3:37 AM Daniel Henrique Barboza
> <dbarboza@ventanamicro.com> wrote:
>>
>> Add a simple guideline to use the existing RISC-V IOMMU support we just
>> added.
>>
>> This doc will be updated once we add the riscv-iommu-sys device.
>>
>> Signed-off-by: Daniel Henrique Barboza <dbarboza@ventanamicro.com>
>> ---
>>   docs/specs/index.rst       |  1 +
>>   docs/specs/riscv-iommu.rst | 55 ++++++++++++++++++++++++++++++++++++++
>>   docs/system/riscv/virt.rst | 13 +++++++++
>>   3 files changed, 69 insertions(+)
>>   create mode 100644 docs/specs/riscv-iommu.rst
>>
>> diff --git a/docs/specs/index.rst b/docs/specs/index.rst
>> index 1484e3e760..c68cd9ae6c 100644
>> --- a/docs/specs/index.rst
>> +++ b/docs/specs/index.rst
>> @@ -33,3 +33,4 @@ guest hardware that is specific to QEMU.
>>      virt-ctlr
>>      vmcoreinfo
>>      vmgenid
>> +   riscv-iommu
>> diff --git a/docs/specs/riscv-iommu.rst b/docs/specs/riscv-iommu.rst
>> new file mode 100644
>> index 0000000000..fa38ff7667
>> --- /dev/null
>> +++ b/docs/specs/riscv-iommu.rst
>> @@ -0,0 +1,55 @@
>> +.. _riscv-iommu:
>> +
>> +RISC-V IOMMU support for RISC-V machines
>> +========================================
>> +
>> +QEMU implements a RISC-V IOMMU emulation based on the RISC-V IOMMU spec
>> +version 1.0 [1].
>> +
>> +The emulation includes a PCI reference device, riscv-iommu-pci, that QEMU
>> +RISC-V boards can use.  The 'virt' RISC-V machine is compatible with this
>> +device.
>> +
>> +A platform device that implements the RISC-V IOMMU will be added in the
>> +future.
>> +
>> +
>> +riscv-iommu-pci reference device
>> +--------------------------------
>> +
>> +This device implements the RISC-V IOMMU emulation as recommended by the section
>> +"Integrating an IOMMU as a PCIe device" of [1]: a PCI device with base class 08h,
>> +sub-class 06h and programming interface 00h.
>> +
>> +As a reference device it doesn't implement anything outside of the specification,
>> +so it uses a generic default PCI ID given by QEMU: 1b36:0014.
>> +
>> +To include the device in the 'virt' machine:
>> +
>> +.. code-block:: bash
>> +
>> +  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci (...)
> 
> We should add a sentence saying what this does. As in what should a
> user expect after they have done this

Here's what was added at this point:

  .. code-block:: bash
  
-  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci (...)
+  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci,[optional_pci_opts] (...)
+
+This will add a RISC-V IOMMU PCI device in the board following any additional
+PCI parameters (like PCI bus address).  The behavior of the RISC-V IOMMU is
+defined by the spec but its operation is OS dependent.  As of this writing the
+existing Linux kernel support `linux-v8`_, not yet merged, will configure the IOMMU
+to create IOMMU groups with any eligible cards available in the system,
+regardless of factors such as the order in which the devices are added in the
+command line.
+
+This means that these command lines are equivalent as far as the current
+IOMMU kernel driver behaves:
+
+.. code-block:: bash
+
+  $ qemu-system-riscv64 \
+        -M virt,aia=aplic-imsic,aia-guests=5 \
+        -device riscv-iommu-pci,addr=1.0,vendor-id=0x1efd,device-id=0xedf1 \
+        -device e1000e,netdev=net1 -netdev user,id=net1,net=192.168.0.0/24 \
+        -device e1000e,netdev=net2 -netdev user,id=net2,net=192.168.200.0/24 \
+        (...)
+
+  $ qemu-system-riscv64 \
+        -M virt,aia=aplic-imsic,aia-guests=5 \
+        -device e1000e,netdev=net1 -netdev user,id=net1,net=192.168.0.0/24 \
+        -device e1000e,netdev=net2 -netdev user,id=net2,net=192.168.200.0/24 \
+        -device riscv-iommu-pci,addr=1.0,vendor-id=0x1efd,device-id=0xedf1 \
+        (...)
+
+Both will create iommu groups for the two e1000e cards.


What I tried to say here is that the operation of the IOMMU device is OS dependent,
and the current Linux driver doesn't care to things such as the order in which the
devices were added. Is this what you had in mind?


I also removed the platform device reference at the start of the doc. I think it was
doing more harm than good. We'll need to change this doc when that device is added,
might as well leave it as is for now.


Thanks,

Daniel


> 
> Alistair
> 
>> +
>> +As of this writing the existing Linux kernel support [2], not yet merged, is being
>> +created as a Rivos device, i.e. it uses Rivos vendor ID.  To use the riscv-iommu-pci
>> +device with the existing kernel support we need to emulate a Rivos PCI IOMMU by
>> +setting 'vendor-id' and 'device-id':
>> +
>> +.. code-block:: bash
>> +
>> +  $ qemu-system-riscv64 -M virt        \
>> +     -device riscv-iommu-pci,vendor-id=0x1efd,device-id=0xedf1 (...)
>> +
>> +Several options are available to control the capabilities of the device, namely:
>> +
>> +- "bus"
>> +- "ioatc-limit"
>> +- "intremap"
>> +- "ats"
>> +- "off" (Out-of-reset translation mode: 'on' for DMA disabled, 'off' for 'BARE' (passthrough))
>> +- "s-stage"
>> +- "g-stage"
>> +
>> +
>> +[1] https://github.com/riscv-non-isa/riscv-iommu/releases/download/v1.0/riscv-iommu.pdf
>> +[2] https://lore.kernel.org/linux-riscv/cover.1718388908.git.tjeznach@rivosinc.com/
>> diff --git a/docs/system/riscv/virt.rst b/docs/system/riscv/virt.rst
>> index 9a06f95a34..8e9a2e4dda 100644
>> --- a/docs/system/riscv/virt.rst
>> +++ b/docs/system/riscv/virt.rst
>> @@ -84,6 +84,19 @@ none``, as in
>>
>>   Firmware images used for pflash must be exactly 32 MiB in size.
>>
>> +riscv-iommu support
>> +-------------------
>> +
>> +The board has support for the riscv-iommu-pci device by using the following
>> +command line:
>> +
>> +.. code-block:: bash
>> +
>> +  $ qemu-system-riscv64 -M virt -device riscv-iommu-pci (...)
>> +
>> +Refer to :ref:`riscv-iommu` for more information on how the RISC-V IOMMU support
>> +works.
>> +
>>   Machine-specific options
>>   ------------------------
>>
>> --
>> 2.45.2
>>
>>