[v3,00/13] riscv: QEMU RISC-V IOMMU Support

Hi,

In this new version a lot of changes were made throughout all the code,
most notably on patch 3. Link for the previous version is [1].

* How it was tested *

This series was tested using an emulated QEMU RISC-V host booting a QEMU
KVM guest, passing through an emulated e1000 network card from the host
to the guest. I can provide more details (e.g. QEMU command lines) if
required, just let me know. For now this cover-letter is too much of an
essay as is.

The Linux kernel used for tests can be found here:

https://github.com/tjeznach/linux/tree/riscv_iommu_v6-rc3

This is a newer version of the following work from Tomasz: 

https://lore.kernel.org/linux-riscv/cover.1715708679.git.tjeznach@rivosinc.com/
("[PATCH v5 0/7] Linux RISC-V IOMMU Support")

The v5 wasn't enough for the testing being done. v6-rc3 did the trick.

Note that to test this work using riscv-iommu-pci we'll need to provide
the Rivos PCI ID in the command line. More details down below.

* Highlights of this version *

- patches removed from v2: platform driver (riscv-iommu-sys, former
patch 05) and the EDU changes (patches 14 and 15). The platform driver
will be sent later with a working example on the 'virt' machine,
either on a newer version of this series or via a follow-up series. We
already have a PoC on [2] created by Sunil. More tests are needed, so
it'll be left behind for now. The EDU changes will be sent in separate
after I finish the doc changes that Frank cited in v2.

- patch 3 contains the bulk of changes made from v2. Please give special
attention to the following functions since this is entirely new code I
ended up adding:
 
 - riscv_iommu_report_fault()
 - riscv_iommu_validate_device_ctx() 
 - riscv_iommu_update_ipsr() 
 
  Aside from these helpers most of the changes made in this patch 3 were
punctual.

- Red HAT PCI ID related changes. A new patch (4) that introduces a
generic RISC-V IOMMU PCI ID was added. This PCI ID was gracefully given
to us by Red Hat and Gerd Hoffman from their ID space. The
riscv-iommu-pci device now defaults to this PCI ID instead of Rivos PCI
ID. The device was changed slightly to allow vendor-id and device-id to
be set in the command-line, so it's now possible to use this reference
device as another RISC-V IOMMU PCI device to ease the burden of
testing/development.

  To instantiate the riscv-iommu-pci device using the previous Rivos PCI
ID, use the following cmd line:

  -device riscv-iommu-pci,vendor-id=0x1efd,device-id=0xedf1 

  I'm using these options to test the series with the existing Linux RISC-V
IOMMU support that uses just a Rivos ID to identify the device.


Series based on alistair/riscv-to-apply.next. It's also applicable on
current QEMU master. It can also be fetched from:

https://gitlab.com/danielhb/qemu/-/tree/riscv_iommu_v3
 

Patches missing reviews/acks: 3, 5, 9, 10, 11.

Changes from v2 [1]:
- patch 05 (hw/riscv: add riscv-iommu-sys platform device): dropped
  - will be reintroduced in a later review or as a follow-up series

- patches 14 and 15: dropped
  - will be sent in separate 

- patches 2, 3, 4 and 5:
  - removed all 'Ziommu' references

- patch 2:
  - added extra bits that patch 3 ended up using

- patch 3:
  - fixed blank line at EOF in hw/riscv/trace.h
  - added a riscv_iommu_report_fault() helper to report faults. The helper checks if
    a given fault is eligible to be reported if DTF is 1
  - Use riscv_iommu_report_fault() in riscv_iommu_ctx() and riscv_iommu_translate()
    to avoid code repetition
  - added a riscv_iommu_validate_device_ctx() helper to validate the device context
    as specified in "Device configuration checks" section. This helper is being used
    in riscv_iommu_ctx_fetch()
  - added a new riscv_iommu_update_ipsr() helper to handle IPSR updates
    in riscv_iommu_mmio_write()
  - riscv_iommmu_msi_write() now reports a fault in all error paths
  - check for fctl.WSI before issuing a MSI interrupt in riscv_iommu_notify()
  - change riscv-iommu region name to 'riscv-iommu'
  - change address_space_init() name for PCI devices to 'name' instead of using TYPE_RISCV_IOMMU_PCI
  - changed riscv_iommu_mmio_ops min_access_size to 4
  - do not check for min and max sizes on riscv_iommu_mmio_write()
  - changed riscv_iommu_trap_ops  min_access_size to 4
  - removed IOMMU qemu_thread thread:
    - riscv_iommu_mmio_write() will now execute a riscv_iommu_process_fn by holding
      'core_lock'
  - init FSCR as zero explicitly
  - check for bus->iommu_opaque == NULL before calling pci_setup_iommu()

- patch 4 (new):
  - add Red-Hat PCI RISC-V IOMMU ID

- patch 5 (former 4):
  - create vendor-id and device-id properties
  - set Red-hat PCI RISC-V IOMMU ID as default ID

- patch 8:
  - use IOMMU_NONE instead of '0' in relevant 'iot->perm = 0' instances

- patch 9:
  - add s-stage and g-stage steps in riscv_iommu_validate_device_ctx()
  - removed 'gpa' boolean from riscv_iommu_spa_fetch()
  - 'en_s' is no longer used for early MSI address match

- patch 10:
  - add ATS steps in riscv_iommu_validate_device_ctx()
  - check for 's->enable_ats' before adding RISCV_IOMMU_DC_TC_EN_ATS in device context
  - check for 's->enable_ats' before processing ATS commands in riscv_iommu_process_cq_tail()
  - remove ambiguous trace_riscv_iommu_ats() from riscv_iommu_translate()

- patch 11:
  - removed unused bits
  - added RISCV_IOMMU_TR_REQ_CTL_NW and RISCV_IOMMU_TR_RESPONSE_S
    bits
  - set IOMMUTLBEntry 'perm' using RISCV_IOMMU_TR_REQ_CTL_NW in riscv_iommu_process_dbg()
  - clear RISCV_IOMMU_TR_RESPONSE_S in riscv_iommu_process_dbg(). Added a comment talking about the (lack of) superpage support
 
[1] https://lore.kernel.org/qemu-riscv/20240307160319.675044-1-dbarboza@ventanamicro.com/
[2] https://github.com/vlsunil/qemu/commits/acpi_rimt_poc_v1/

Andrew Jones (1):
  hw/riscv/riscv-iommu: Add another irq for mrif notifications

Daniel Henrique Barboza (3):
  pci-ids.rst: add Red Hat pci-id for RISC-V IOMMU device
  test/qtest: add riscv-iommu-pci tests
  qtest/riscv-iommu-test: add init queues test

Tomasz Jeznach (9):
  exec/memtxattr: add process identifier to the transaction attributes
  hw/riscv: add riscv-iommu-bits.h
  hw/riscv: add RISC-V IOMMU base emulation
  hw/riscv: add riscv-iommu-pci reference device
  hw/riscv/virt.c: support for RISC-V IOMMU PCIDevice hotplug
  hw/riscv/riscv-iommu: add Address Translation Cache (IOATC)
  hw/riscv/riscv-iommu: add s-stage and g-stage support
  hw/riscv/riscv-iommu: add ATS support
  hw/riscv/riscv-iommu: add DBG support

 docs/specs/pci-ids.rst           |    2 +
 hw/riscv/Kconfig                 |    4 +
 hw/riscv/meson.build             |    1 +
 hw/riscv/riscv-iommu-bits.h      |  416 ++++++
 hw/riscv/riscv-iommu-pci.c       |  177 +++
 hw/riscv/riscv-iommu.c           | 2283 ++++++++++++++++++++++++++++++
 hw/riscv/riscv-iommu.h           |  146 ++
 hw/riscv/trace-events            |   15 +
 hw/riscv/trace.h                 |    1 +
 hw/riscv/virt.c                  |   33 +-
 include/exec/memattrs.h          |    5 +
 include/hw/pci/pci.h             |    1 +
 include/hw/riscv/iommu.h         |   36 +
 meson.build                      |    1 +
 tests/qtest/libqos/meson.build   |    4 +
 tests/qtest/libqos/riscv-iommu.c |   76 +
 tests/qtest/libqos/riscv-iommu.h |  100 ++
 tests/qtest/meson.build          |    1 +
 tests/qtest/riscv-iommu-test.c   |  234 +++
 19 files changed, 3535 insertions(+), 1 deletion(-)
 create mode 100644 hw/riscv/riscv-iommu-bits.h
 create mode 100644 hw/riscv/riscv-iommu-pci.c
 create mode 100644 hw/riscv/riscv-iommu.c
 create mode 100644 hw/riscv/riscv-iommu.h
 create mode 100644 hw/riscv/trace-events
 create mode 100644 hw/riscv/trace.h
 create mode 100644 include/hw/riscv/iommu.h
 create mode 100644 tests/qtest/libqos/riscv-iommu.c
 create mode 100644 tests/qtest/libqos/riscv-iommu.h
 create mode 100644 tests/qtest/riscv-iommu-test.c

On Fri, May 24, 2024 at 3:43 AM Daniel Henrique Barboza
<dbarboza@ventanamicro.com> wrote:
>
> Hi,
>
> In this new version a lot of changes were made throughout all the code,
> most notably on patch 3. Link for the previous version is [1].
>
> * How it was tested *
>
> This series was tested using an emulated QEMU RISC-V host booting a QEMU
> KVM guest, passing through an emulated e1000 network card from the host
> to the guest. I can provide more details (e.g. QEMU command lines) if
> required, just let me know. For now this cover-letter is too much of an
> essay as is.

It would probably be helpful to document these somewhere, so others
can use them as a starting point for running this

Alistair

>
> The Linux kernel used for tests can be found here:
>
> https://github.com/tjeznach/linux/tree/riscv_iommu_v6-rc3
>
> This is a newer version of the following work from Tomasz:
>
> https://lore.kernel.org/linux-riscv/cover.1715708679.git.tjeznach@rivosinc.com/
> ("[PATCH v5 0/7] Linux RISC-V IOMMU Support")
>
> The v5 wasn't enough for the testing being done. v6-rc3 did the trick.
>
> Note that to test this work using riscv-iommu-pci we'll need to provide
> the Rivos PCI ID in the command line. More details down below.
>
> * Highlights of this version *
>
> - patches removed from v2: platform driver (riscv-iommu-sys, former
> patch 05) and the EDU changes (patches 14 and 15). The platform driver
> will be sent later with a working example on the 'virt' machine,
> either on a newer version of this series or via a follow-up series. We
> already have a PoC on [2] created by Sunil. More tests are needed, so
> it'll be left behind for now. The EDU changes will be sent in separate
> after I finish the doc changes that Frank cited in v2.
>
> - patch 3 contains the bulk of changes made from v2. Please give special
> attention to the following functions since this is entirely new code I
> ended up adding:
>
>  - riscv_iommu_report_fault()
>  - riscv_iommu_validate_device_ctx()
>  - riscv_iommu_update_ipsr()
>
>   Aside from these helpers most of the changes made in this patch 3 were
> punctual.
>
> - Red HAT PCI ID related changes. A new patch (4) that introduces a
> generic RISC-V IOMMU PCI ID was added. This PCI ID was gracefully given
> to us by Red Hat and Gerd Hoffman from their ID space. The
> riscv-iommu-pci device now defaults to this PCI ID instead of Rivos PCI
> ID. The device was changed slightly to allow vendor-id and device-id to
> be set in the command-line, so it's now possible to use this reference
> device as another RISC-V IOMMU PCI device to ease the burden of
> testing/development.
>
>   To instantiate the riscv-iommu-pci device using the previous Rivos PCI
> ID, use the following cmd line:
>
>   -device riscv-iommu-pci,vendor-id=0x1efd,device-id=0xedf1
>
>   I'm using these options to test the series with the existing Linux RISC-V
> IOMMU support that uses just a Rivos ID to identify the device.
>
>
> Series based on alistair/riscv-to-apply.next. It's also applicable on
> current QEMU master. It can also be fetched from:
>
> https://gitlab.com/danielhb/qemu/-/tree/riscv_iommu_v3
>
>
> Patches missing reviews/acks: 3, 5, 9, 10, 11.
>
> Changes from v2 [1]:
> - patch 05 (hw/riscv: add riscv-iommu-sys platform device): dropped
>   - will be reintroduced in a later review or as a follow-up series
>
> - patches 14 and 15: dropped
>   - will be sent in separate
>
> - patches 2, 3, 4 and 5:
>   - removed all 'Ziommu' references
>
> - patch 2:
>   - added extra bits that patch 3 ended up using
>
> - patch 3:
>   - fixed blank line at EOF in hw/riscv/trace.h
>   - added a riscv_iommu_report_fault() helper to report faults. The helper checks if
>     a given fault is eligible to be reported if DTF is 1
>   - Use riscv_iommu_report_fault() in riscv_iommu_ctx() and riscv_iommu_translate()
>     to avoid code repetition
>   - added a riscv_iommu_validate_device_ctx() helper to validate the device context
>     as specified in "Device configuration checks" section. This helper is being used
>     in riscv_iommu_ctx_fetch()
>   - added a new riscv_iommu_update_ipsr() helper to handle IPSR updates
>     in riscv_iommu_mmio_write()
>   - riscv_iommmu_msi_write() now reports a fault in all error paths
>   - check for fctl.WSI before issuing a MSI interrupt in riscv_iommu_notify()
>   - change riscv-iommu region name to 'riscv-iommu'
>   - change address_space_init() name for PCI devices to 'name' instead of using TYPE_RISCV_IOMMU_PCI
>   - changed riscv_iommu_mmio_ops min_access_size to 4
>   - do not check for min and max sizes on riscv_iommu_mmio_write()
>   - changed riscv_iommu_trap_ops  min_access_size to 4
>   - removed IOMMU qemu_thread thread:
>     - riscv_iommu_mmio_write() will now execute a riscv_iommu_process_fn by holding
>       'core_lock'
>   - init FSCR as zero explicitly
>   - check for bus->iommu_opaque == NULL before calling pci_setup_iommu()
>
> - patch 4 (new):
>   - add Red-Hat PCI RISC-V IOMMU ID
>
> - patch 5 (former 4):
>   - create vendor-id and device-id properties
>   - set Red-hat PCI RISC-V IOMMU ID as default ID
>
> - patch 8:
>   - use IOMMU_NONE instead of '0' in relevant 'iot->perm = 0' instances
>
> - patch 9:
>   - add s-stage and g-stage steps in riscv_iommu_validate_device_ctx()
>   - removed 'gpa' boolean from riscv_iommu_spa_fetch()
>   - 'en_s' is no longer used for early MSI address match
>
> - patch 10:
>   - add ATS steps in riscv_iommu_validate_device_ctx()
>   - check for 's->enable_ats' before adding RISCV_IOMMU_DC_TC_EN_ATS in device context
>   - check for 's->enable_ats' before processing ATS commands in riscv_iommu_process_cq_tail()
>   - remove ambiguous trace_riscv_iommu_ats() from riscv_iommu_translate()
>
> - patch 11:
>   - removed unused bits
>   - added RISCV_IOMMU_TR_REQ_CTL_NW and RISCV_IOMMU_TR_RESPONSE_S
>     bits
>   - set IOMMUTLBEntry 'perm' using RISCV_IOMMU_TR_REQ_CTL_NW in riscv_iommu_process_dbg()
>   - clear RISCV_IOMMU_TR_RESPONSE_S in riscv_iommu_process_dbg(). Added a comment talking about the (lack of) superpage support
>
> [1] https://lore.kernel.org/qemu-riscv/20240307160319.675044-1-dbarboza@ventanamicro.com/
> [2] https://github.com/vlsunil/qemu/commits/acpi_rimt_poc_v1/
>
> Andrew Jones (1):
>   hw/riscv/riscv-iommu: Add another irq for mrif notifications
>
> Daniel Henrique Barboza (3):
>   pci-ids.rst: add Red Hat pci-id for RISC-V IOMMU device
>   test/qtest: add riscv-iommu-pci tests
>   qtest/riscv-iommu-test: add init queues test
>
> Tomasz Jeznach (9):
>   exec/memtxattr: add process identifier to the transaction attributes
>   hw/riscv: add riscv-iommu-bits.h
>   hw/riscv: add RISC-V IOMMU base emulation
>   hw/riscv: add riscv-iommu-pci reference device
>   hw/riscv/virt.c: support for RISC-V IOMMU PCIDevice hotplug
>   hw/riscv/riscv-iommu: add Address Translation Cache (IOATC)
>   hw/riscv/riscv-iommu: add s-stage and g-stage support
>   hw/riscv/riscv-iommu: add ATS support
>   hw/riscv/riscv-iommu: add DBG support
>
>  docs/specs/pci-ids.rst           |    2 +
>  hw/riscv/Kconfig                 |    4 +
>  hw/riscv/meson.build             |    1 +
>  hw/riscv/riscv-iommu-bits.h      |  416 ++++++
>  hw/riscv/riscv-iommu-pci.c       |  177 +++
>  hw/riscv/riscv-iommu.c           | 2283 ++++++++++++++++++++++++++++++
>  hw/riscv/riscv-iommu.h           |  146 ++
>  hw/riscv/trace-events            |   15 +
>  hw/riscv/trace.h                 |    1 +
>  hw/riscv/virt.c                  |   33 +-
>  include/exec/memattrs.h          |    5 +
>  include/hw/pci/pci.h             |    1 +
>  include/hw/riscv/iommu.h         |   36 +
>  meson.build                      |    1 +
>  tests/qtest/libqos/meson.build   |    4 +
>  tests/qtest/libqos/riscv-iommu.c |   76 +
>  tests/qtest/libqos/riscv-iommu.h |  100 ++
>  tests/qtest/meson.build          |    1 +
>  tests/qtest/riscv-iommu-test.c   |  234 +++
>  19 files changed, 3535 insertions(+), 1 deletion(-)
>  create mode 100644 hw/riscv/riscv-iommu-bits.h
>  create mode 100644 hw/riscv/riscv-iommu-pci.c
>  create mode 100644 hw/riscv/riscv-iommu.c
>  create mode 100644 hw/riscv/riscv-iommu.h
>  create mode 100644 hw/riscv/trace-events
>  create mode 100644 hw/riscv/trace.h
>  create mode 100644 include/hw/riscv/iommu.h
>  create mode 100644 tests/qtest/libqos/riscv-iommu.c
>  create mode 100644 tests/qtest/libqos/riscv-iommu.h
>  create mode 100644 tests/qtest/riscv-iommu-test.c
>
> --
> 2.44.0
>
>

On June 10, 2024 2:34:58 AM GMT+02:00, Alistair Francis <alistair23@gmail.com> wrote:
>On Fri, May 24, 2024 at 3:43 AM Daniel Henrique Barboza
><dbarboza@ventanamicro.com> wrote:
>>
>> Hi,
>>
>> In this new version a lot of changes were made throughout all the code,
>> most notably on patch 3. Link for the previous version is [1].
>>
>> * How it was tested *
>>
>> This series was tested using an emulated QEMU RISC-V host booting a QEMU
>> KVM guest, passing through an emulated e1000 network card from the host
>> to the guest. I can provide more details (e.g. QEMU command lines) if
>> required, just let me know. For now this cover-letter is too much of an
>> essay as is.
>
>It would probably be helpful to document these somewhere, so others
>can use them as a starting point for running this
>

I've written up a testing procedure which I shared internally with Daniel. I'll sanitize it and post it somewhere public.

Thanks,
drew

>Alistair
>
>>
>> The Linux kernel used for tests can be found here:
>>
>> https://github.com/tjeznach/linux/tree/riscv_iommu_v6-rc3
>>
>> This is a newer version of the following work from Tomasz:
>>
>> https://lore.kernel.org/linux-riscv/cover.1715708679.git.tjeznach@rivosinc.com/
>> ("[PATCH v5 0/7] Linux RISC-V IOMMU Support")
>>
>> The v5 wasn't enough for the testing being done. v6-rc3 did the trick.
>>
>> Note that to test this work using riscv-iommu-pci we'll need to provide
>> the Rivos PCI ID in the command line. More details down below.
>>
>> * Highlights of this version *
>>
>> - patches removed from v2: platform driver (riscv-iommu-sys, former
>> patch 05) and the EDU changes (patches 14 and 15). The platform driver
>> will be sent later with a working example on the 'virt' machine,
>> either on a newer version of this series or via a follow-up series. We
>> already have a PoC on [2] created by Sunil. More tests are needed, so
>> it'll be left behind for now. The EDU changes will be sent in separate
>> after I finish the doc changes that Frank cited in v2.
>>
>> - patch 3 contains the bulk of changes made from v2. Please give special
>> attention to the following functions since this is entirely new code I
>> ended up adding:
>>
>>  - riscv_iommu_report_fault()
>>  - riscv_iommu_validate_device_ctx()
>>  - riscv_iommu_update_ipsr()
>>
>>   Aside from these helpers most of the changes made in this patch 3 were
>> punctual.
>>
>> - Red HAT PCI ID related changes. A new patch (4) that introduces a
>> generic RISC-V IOMMU PCI ID was added. This PCI ID was gracefully given
>> to us by Red Hat and Gerd Hoffman from their ID space. The
>> riscv-iommu-pci device now defaults to this PCI ID instead of Rivos PCI
>> ID. The device was changed slightly to allow vendor-id and device-id to
>> be set in the command-line, so it's now possible to use this reference
>> device as another RISC-V IOMMU PCI device to ease the burden of
>> testing/development.
>>
>>   To instantiate the riscv-iommu-pci device using the previous Rivos PCI
>> ID, use the following cmd line:
>>
>>   -device riscv-iommu-pci,vendor-id=0x1efd,device-id=0xedf1
>>
>>   I'm using these options to test the series with the existing Linux RISC-V
>> IOMMU support that uses just a Rivos ID to identify the device.
>>
>>
>> Series based on alistair/riscv-to-apply.next. It's also applicable on
>> current QEMU master. It can also be fetched from:
>>
>> https://gitlab.com/danielhb/qemu/-/tree/riscv_iommu_v3
>>
>>
>> Patches missing reviews/acks: 3, 5, 9, 10, 11.
>>
>> Changes from v2 [1]:
>> - patch 05 (hw/riscv: add riscv-iommu-sys platform device): dropped
>>   - will be reintroduced in a later review or as a follow-up series
>>
>> - patches 14 and 15: dropped
>>   - will be sent in separate
>>
>> - patches 2, 3, 4 and 5:
>>   - removed all 'Ziommu' references
>>
>> - patch 2:
>>   - added extra bits that patch 3 ended up using
>>
>> - patch 3:
>>   - fixed blank line at EOF in hw/riscv/trace.h
>>   - added a riscv_iommu_report_fault() helper to report faults. The helper checks if
>>     a given fault is eligible to be reported if DTF is 1
>>   - Use riscv_iommu_report_fault() in riscv_iommu_ctx() and riscv_iommu_translate()
>>     to avoid code repetition
>>   - added a riscv_iommu_validate_device_ctx() helper to validate the device context
>>     as specified in "Device configuration checks" section. This helper is being used
>>     in riscv_iommu_ctx_fetch()
>>   - added a new riscv_iommu_update_ipsr() helper to handle IPSR updates
>>     in riscv_iommu_mmio_write()
>>   - riscv_iommmu_msi_write() now reports a fault in all error paths
>>   - check for fctl.WSI before issuing a MSI interrupt in riscv_iommu_notify()
>>   - change riscv-iommu region name to 'riscv-iommu'
>>   - change address_space_init() name for PCI devices to 'name' instead of using TYPE_RISCV_IOMMU_PCI
>>   - changed riscv_iommu_mmio_ops min_access_size to 4
>>   - do not check for min and max sizes on riscv_iommu_mmio_write()
>>   - changed riscv_iommu_trap_ops  min_access_size to 4
>>   - removed IOMMU qemu_thread thread:
>>     - riscv_iommu_mmio_write() will now execute a riscv_iommu_process_fn by holding
>>       'core_lock'
>>   - init FSCR as zero explicitly
>>   - check for bus->iommu_opaque == NULL before calling pci_setup_iommu()
>>
>> - patch 4 (new):
>>   - add Red-Hat PCI RISC-V IOMMU ID
>>
>> - patch 5 (former 4):
>>   - create vendor-id and device-id properties
>>   - set Red-hat PCI RISC-V IOMMU ID as default ID
>>
>> - patch 8:
>>   - use IOMMU_NONE instead of '0' in relevant 'iot->perm = 0' instances
>>
>> - patch 9:
>>   - add s-stage and g-stage steps in riscv_iommu_validate_device_ctx()
>>   - removed 'gpa' boolean from riscv_iommu_spa_fetch()
>>   - 'en_s' is no longer used for early MSI address match
>>
>> - patch 10:
>>   - add ATS steps in riscv_iommu_validate_device_ctx()
>>   - check for 's->enable_ats' before adding RISCV_IOMMU_DC_TC_EN_ATS in device context
>>   - check for 's->enable_ats' before processing ATS commands in riscv_iommu_process_cq_tail()
>>   - remove ambiguous trace_riscv_iommu_ats() from riscv_iommu_translate()
>>
>> - patch 11:
>>   - removed unused bits
>>   - added RISCV_IOMMU_TR_REQ_CTL_NW and RISCV_IOMMU_TR_RESPONSE_S
>>     bits
>>   - set IOMMUTLBEntry 'perm' using RISCV_IOMMU_TR_REQ_CTL_NW in riscv_iommu_process_dbg()
>>   - clear RISCV_IOMMU_TR_RESPONSE_S in riscv_iommu_process_dbg(). Added a comment talking about the (lack of) superpage support
>>
>> [1] https://lore.kernel.org/qemu-riscv/20240307160319.675044-1-dbarboza@ventanamicro.com/
>> [2] https://github.com/vlsunil/qemu/commits/acpi_rimt_poc_v1/
>>
>> Andrew Jones (1):
>>   hw/riscv/riscv-iommu: Add another irq for mrif notifications
>>
>> Daniel Henrique Barboza (3):
>>   pci-ids.rst: add Red Hat pci-id for RISC-V IOMMU device
>>   test/qtest: add riscv-iommu-pci tests
>>   qtest/riscv-iommu-test: add init queues test
>>
>> Tomasz Jeznach (9):
>>   exec/memtxattr: add process identifier to the transaction attributes
>>   hw/riscv: add riscv-iommu-bits.h
>>   hw/riscv: add RISC-V IOMMU base emulation
>>   hw/riscv: add riscv-iommu-pci reference device
>>   hw/riscv/virt.c: support for RISC-V IOMMU PCIDevice hotplug
>>   hw/riscv/riscv-iommu: add Address Translation Cache (IOATC)
>>   hw/riscv/riscv-iommu: add s-stage and g-stage support
>>   hw/riscv/riscv-iommu: add ATS support
>>   hw/riscv/riscv-iommu: add DBG support
>>
>>  docs/specs/pci-ids.rst           |    2 +
>>  hw/riscv/Kconfig                 |    4 +
>>  hw/riscv/meson.build             |    1 +
>>  hw/riscv/riscv-iommu-bits.h      |  416 ++++++
>>  hw/riscv/riscv-iommu-pci.c       |  177 +++
>>  hw/riscv/riscv-iommu.c           | 2283 ++++++++++++++++++++++++++++++
>>  hw/riscv/riscv-iommu.h           |  146 ++
>>  hw/riscv/trace-events            |   15 +
>>  hw/riscv/trace.h                 |    1 +
>>  hw/riscv/virt.c                  |   33 +-
>>  include/exec/memattrs.h          |    5 +
>>  include/hw/pci/pci.h             |    1 +
>>  include/hw/riscv/iommu.h         |   36 +
>>  meson.build                      |    1 +
>>  tests/qtest/libqos/meson.build   |    4 +
>>  tests/qtest/libqos/riscv-iommu.c |   76 +
>>  tests/qtest/libqos/riscv-iommu.h |  100 ++
>>  tests/qtest/meson.build          |    1 +
>>  tests/qtest/riscv-iommu-test.c   |  234 +++
>>  19 files changed, 3535 insertions(+), 1 deletion(-)
>>  create mode 100644 hw/riscv/riscv-iommu-bits.h
>>  create mode 100644 hw/riscv/riscv-iommu-pci.c
>>  create mode 100644 hw/riscv/riscv-iommu.c
>>  create mode 100644 hw/riscv/riscv-iommu.h
>>  create mode 100644 hw/riscv/trace-events
>>  create mode 100644 hw/riscv/trace.h
>>  create mode 100644 include/hw/riscv/iommu.h
>>  create mode 100644 tests/qtest/libqos/riscv-iommu.c
>>  create mode 100644 tests/qtest/libqos/riscv-iommu.h
>>  create mode 100644 tests/qtest/riscv-iommu-test.c
>>
>> --
>> 2.44.0
>>
>>

On 6/10/24 3:32 PM, Andrew Jones wrote:
> On June 10, 2024 2:34:58 AM GMT+02:00, Alistair Francis <alistair23@gmail.com> wrote:
>> On Fri, May 24, 2024 at 3:43 AM Daniel Henrique Barboza
>> <dbarboza@ventanamicro.com> wrote:
>>>
>>> Hi,
>>>
>>> In this new version a lot of changes were made throughout all the code,
>>> most notably on patch 3. Link for the previous version is [1].
>>>
>>> * How it was tested *
>>>
>>> This series was tested using an emulated QEMU RISC-V host booting a QEMU
>>> KVM guest, passing through an emulated e1000 network card from the host
>>> to the guest. I can provide more details (e.g. QEMU command lines) if
>>> required, just let me know. For now this cover-letter is too much of an
>>> essay as is.
>>
>> It would probably be helpful to document these somewhere, so others
>> can use them as a starting point for running this
>>
> 
> I've written up a testing procedure which I shared internally with Daniel. I'll sanitize it and post it somewhere public.
>

I can also add a QEMU docs under docs/system/riscv, both as a
subsection of virt.rst and perhaps a new doc that describes the
devices itself (riscv-iommu-pci and later on riscv-iommu-sys).


Thanks,


Daniel
  
> Thanks,
> drew
> 
>> Alistair
>>
>>>
>>> The Linux kernel used for tests can be found here:
>>>
>>> https://github.com/tjeznach/linux/tree/riscv_iommu_v6-rc3
>>>
>>> This is a newer version of the following work from Tomasz:
>>>
>>> https://lore.kernel.org/linux-riscv/cover.1715708679.git.tjeznach@rivosinc.com/
>>> ("[PATCH v5 0/7] Linux RISC-V IOMMU Support")
>>>
>>> The v5 wasn't enough for the testing being done. v6-rc3 did the trick.
>>>
>>> Note that to test this work using riscv-iommu-pci we'll need to provide
>>> the Rivos PCI ID in the command line. More details down below.
>>>
>>> * Highlights of this version *
>>>
>>> - patches removed from v2: platform driver (riscv-iommu-sys, former
>>> patch 05) and the EDU changes (patches 14 and 15). The platform driver
>>> will be sent later with a working example on the 'virt' machine,
>>> either on a newer version of this series or via a follow-up series. We
>>> already have a PoC on [2] created by Sunil. More tests are needed, so
>>> it'll be left behind for now. The EDU changes will be sent in separate
>>> after I finish the doc changes that Frank cited in v2.
>>>
>>> - patch 3 contains the bulk of changes made from v2. Please give special
>>> attention to the following functions since this is entirely new code I
>>> ended up adding:
>>>
>>>   - riscv_iommu_report_fault()
>>>   - riscv_iommu_validate_device_ctx()
>>>   - riscv_iommu_update_ipsr()
>>>
>>>    Aside from these helpers most of the changes made in this patch 3 were
>>> punctual.
>>>
>>> - Red HAT PCI ID related changes. A new patch (4) that introduces a
>>> generic RISC-V IOMMU PCI ID was added. This PCI ID was gracefully given
>>> to us by Red Hat and Gerd Hoffman from their ID space. The
>>> riscv-iommu-pci device now defaults to this PCI ID instead of Rivos PCI
>>> ID. The device was changed slightly to allow vendor-id and device-id to
>>> be set in the command-line, so it's now possible to use this reference
>>> device as another RISC-V IOMMU PCI device to ease the burden of
>>> testing/development.
>>>
>>>    To instantiate the riscv-iommu-pci device using the previous Rivos PCI
>>> ID, use the following cmd line:
>>>
>>>    -device riscv-iommu-pci,vendor-id=0x1efd,device-id=0xedf1
>>>
>>>    I'm using these options to test the series with the existing Linux RISC-V
>>> IOMMU support that uses just a Rivos ID to identify the device.
>>>
>>>
>>> Series based on alistair/riscv-to-apply.next. It's also applicable on
>>> current QEMU master. It can also be fetched from:
>>>
>>> https://gitlab.com/danielhb/qemu/-/tree/riscv_iommu_v3
>>>
>>>
>>> Patches missing reviews/acks: 3, 5, 9, 10, 11.
>>>
>>> Changes from v2 [1]:
>>> - patch 05 (hw/riscv: add riscv-iommu-sys platform device): dropped
>>>    - will be reintroduced in a later review or as a follow-up series
>>>
>>> - patches 14 and 15: dropped
>>>    - will be sent in separate
>>>
>>> - patches 2, 3, 4 and 5:
>>>    - removed all 'Ziommu' references
>>>
>>> - patch 2:
>>>    - added extra bits that patch 3 ended up using
>>>
>>> - patch 3:
>>>    - fixed blank line at EOF in hw/riscv/trace.h
>>>    - added a riscv_iommu_report_fault() helper to report faults. The helper checks if
>>>      a given fault is eligible to be reported if DTF is 1
>>>    - Use riscv_iommu_report_fault() in riscv_iommu_ctx() and riscv_iommu_translate()
>>>      to avoid code repetition
>>>    - added a riscv_iommu_validate_device_ctx() helper to validate the device context
>>>      as specified in "Device configuration checks" section. This helper is being used
>>>      in riscv_iommu_ctx_fetch()
>>>    - added a new riscv_iommu_update_ipsr() helper to handle IPSR updates
>>>      in riscv_iommu_mmio_write()
>>>    - riscv_iommmu_msi_write() now reports a fault in all error paths
>>>    - check for fctl.WSI before issuing a MSI interrupt in riscv_iommu_notify()
>>>    - change riscv-iommu region name to 'riscv-iommu'
>>>    - change address_space_init() name for PCI devices to 'name' instead of using TYPE_RISCV_IOMMU_PCI
>>>    - changed riscv_iommu_mmio_ops min_access_size to 4
>>>    - do not check for min and max sizes on riscv_iommu_mmio_write()
>>>    - changed riscv_iommu_trap_ops  min_access_size to 4
>>>    - removed IOMMU qemu_thread thread:
>>>      - riscv_iommu_mmio_write() will now execute a riscv_iommu_process_fn by holding
>>>        'core_lock'
>>>    - init FSCR as zero explicitly
>>>    - check for bus->iommu_opaque == NULL before calling pci_setup_iommu()
>>>
>>> - patch 4 (new):
>>>    - add Red-Hat PCI RISC-V IOMMU ID
>>>
>>> - patch 5 (former 4):
>>>    - create vendor-id and device-id properties
>>>    - set Red-hat PCI RISC-V IOMMU ID as default ID
>>>
>>> - patch 8:
>>>    - use IOMMU_NONE instead of '0' in relevant 'iot->perm = 0' instances
>>>
>>> - patch 9:
>>>    - add s-stage and g-stage steps in riscv_iommu_validate_device_ctx()
>>>    - removed 'gpa' boolean from riscv_iommu_spa_fetch()
>>>    - 'en_s' is no longer used for early MSI address match
>>>
>>> - patch 10:
>>>    - add ATS steps in riscv_iommu_validate_device_ctx()
>>>    - check for 's->enable_ats' before adding RISCV_IOMMU_DC_TC_EN_ATS in device context
>>>    - check for 's->enable_ats' before processing ATS commands in riscv_iommu_process_cq_tail()
>>>    - remove ambiguous trace_riscv_iommu_ats() from riscv_iommu_translate()
>>>
>>> - patch 11:
>>>    - removed unused bits
>>>    - added RISCV_IOMMU_TR_REQ_CTL_NW and RISCV_IOMMU_TR_RESPONSE_S
>>>      bits
>>>    - set IOMMUTLBEntry 'perm' using RISCV_IOMMU_TR_REQ_CTL_NW in riscv_iommu_process_dbg()
>>>    - clear RISCV_IOMMU_TR_RESPONSE_S in riscv_iommu_process_dbg(). Added a comment talking about the (lack of) superpage support
>>>
>>> [1] https://lore.kernel.org/qemu-riscv/20240307160319.675044-1-dbarboza@ventanamicro.com/
>>> [2] https://github.com/vlsunil/qemu/commits/acpi_rimt_poc_v1/
>>>
>>> Andrew Jones (1):
>>>    hw/riscv/riscv-iommu: Add another irq for mrif notifications
>>>
>>> Daniel Henrique Barboza (3):
>>>    pci-ids.rst: add Red Hat pci-id for RISC-V IOMMU device
>>>    test/qtest: add riscv-iommu-pci tests
>>>    qtest/riscv-iommu-test: add init queues test
>>>
>>> Tomasz Jeznach (9):
>>>    exec/memtxattr: add process identifier to the transaction attributes
>>>    hw/riscv: add riscv-iommu-bits.h
>>>    hw/riscv: add RISC-V IOMMU base emulation
>>>    hw/riscv: add riscv-iommu-pci reference device
>>>    hw/riscv/virt.c: support for RISC-V IOMMU PCIDevice hotplug
>>>    hw/riscv/riscv-iommu: add Address Translation Cache (IOATC)
>>>    hw/riscv/riscv-iommu: add s-stage and g-stage support
>>>    hw/riscv/riscv-iommu: add ATS support
>>>    hw/riscv/riscv-iommu: add DBG support
>>>
>>>   docs/specs/pci-ids.rst           |    2 +
>>>   hw/riscv/Kconfig                 |    4 +
>>>   hw/riscv/meson.build             |    1 +
>>>   hw/riscv/riscv-iommu-bits.h      |  416 ++++++
>>>   hw/riscv/riscv-iommu-pci.c       |  177 +++
>>>   hw/riscv/riscv-iommu.c           | 2283 ++++++++++++++++++++++++++++++
>>>   hw/riscv/riscv-iommu.h           |  146 ++
>>>   hw/riscv/trace-events            |   15 +
>>>   hw/riscv/trace.h                 |    1 +
>>>   hw/riscv/virt.c                  |   33 +-
>>>   include/exec/memattrs.h          |    5 +
>>>   include/hw/pci/pci.h             |    1 +
>>>   include/hw/riscv/iommu.h         |   36 +
>>>   meson.build                      |    1 +
>>>   tests/qtest/libqos/meson.build   |    4 +
>>>   tests/qtest/libqos/riscv-iommu.c |   76 +
>>>   tests/qtest/libqos/riscv-iommu.h |  100 ++
>>>   tests/qtest/meson.build          |    1 +
>>>   tests/qtest/riscv-iommu-test.c   |  234 +++
>>>   19 files changed, 3535 insertions(+), 1 deletion(-)
>>>   create mode 100644 hw/riscv/riscv-iommu-bits.h
>>>   create mode 100644 hw/riscv/riscv-iommu-pci.c
>>>   create mode 100644 hw/riscv/riscv-iommu.c
>>>   create mode 100644 hw/riscv/riscv-iommu.h
>>>   create mode 100644 hw/riscv/trace-events
>>>   create mode 100644 hw/riscv/trace.h
>>>   create mode 100644 include/hw/riscv/iommu.h
>>>   create mode 100644 tests/qtest/libqos/riscv-iommu.c
>>>   create mode 100644 tests/qtest/libqos/riscv-iommu.h
>>>   create mode 100644 tests/qtest/riscv-iommu-test.c
>>>
>>> --
>>> 2.44.0
>>>
>>>
>

On Tue, Jun 11, 2024 at 5:16 AM Daniel Henrique Barboza
<dbarboza@ventanamicro.com> wrote:
>
>
>
> On 6/10/24 3:32 PM, Andrew Jones wrote:
> > On June 10, 2024 2:34:58 AM GMT+02:00, Alistair Francis <alistair23@gmail.com> wrote:
> >> On Fri, May 24, 2024 at 3:43 AM Daniel Henrique Barboza
> >> <dbarboza@ventanamicro.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> In this new version a lot of changes were made throughout all the code,
> >>> most notably on patch 3. Link for the previous version is [1].
> >>>
> >>> * How it was tested *
> >>>
> >>> This series was tested using an emulated QEMU RISC-V host booting a QEMU
> >>> KVM guest, passing through an emulated e1000 network card from the host
> >>> to the guest. I can provide more details (e.g. QEMU command lines) if
> >>> required, just let me know. For now this cover-letter is too much of an
> >>> essay as is.
> >>
> >> It would probably be helpful to document these somewhere, so others
> >> can use them as a starting point for running this
> >>
> >
> > I've written up a testing procedure which I shared internally with Daniel. I'll sanitize it and post it somewhere public.
> >
>
> I can also add a QEMU docs under docs/system/riscv, both as a
> subsection of virt.rst and perhaps a new doc that describes the
> devices itself (riscv-iommu-pci and later on riscv-iommu-sys).

I think that would be great. Even if it isn't a simple "copy this
command and it works" it at least gives users a place to start to
figure out how to use this

Alistair

Hi Daniel,

I want to know if we can use the IOMMU and IOPMP at the same time.

The relationship between them is more similar to MMU and sPMP or to MMU 
and PMP?

Thanks,
Zhiwei

On 2024/5/24 1:39, Daniel Henrique Barboza wrote:
> Hi,
>
> In this new version a lot of changes were made throughout all the code,
> most notably on patch 3. Link for the previous version is [1].
>
> * How it was tested *
>
> This series was tested using an emulated QEMU RISC-V host booting a QEMU
> KVM guest, passing through an emulated e1000 network card from the host
> to the guest. I can provide more details (e.g. QEMU command lines) if
> required, just let me know. For now this cover-letter is too much of an
> essay as is.
>
> The Linux kernel used for tests can be found here:
>
> https://github.com/tjeznach/linux/tree/riscv_iommu_v6-rc3
>
> This is a newer version of the following work from Tomasz:
>
> https://lore.kernel.org/linux-riscv/cover.1715708679.git.tjeznach@rivosinc.com/
> ("[PATCH v5 0/7] Linux RISC-V IOMMU Support")
>
> The v5 wasn't enough for the testing being done. v6-rc3 did the trick.
>
> Note that to test this work using riscv-iommu-pci we'll need to provide
> the Rivos PCI ID in the command line. More details down below.
>
> * Highlights of this version *
>
> - patches removed from v2: platform driver (riscv-iommu-sys, former
> patch 05) and the EDU changes (patches 14 and 15). The platform driver
> will be sent later with a working example on the 'virt' machine,
> either on a newer version of this series or via a follow-up series. We
> already have a PoC on [2] created by Sunil. More tests are needed, so
> it'll be left behind for now. The EDU changes will be sent in separate
> after I finish the doc changes that Frank cited in v2.
>
> - patch 3 contains the bulk of changes made from v2. Please give special
> attention to the following functions since this is entirely new code I
> ended up adding:
>   
>   - riscv_iommu_report_fault()
>   - riscv_iommu_validate_device_ctx()
>   - riscv_iommu_update_ipsr()
>   
>    Aside from these helpers most of the changes made in this patch 3 were
> punctual.
>
> - Red HAT PCI ID related changes. A new patch (4) that introduces a
> generic RISC-V IOMMU PCI ID was added. This PCI ID was gracefully given
> to us by Red Hat and Gerd Hoffman from their ID space. The
> riscv-iommu-pci device now defaults to this PCI ID instead of Rivos PCI
> ID. The device was changed slightly to allow vendor-id and device-id to
> be set in the command-line, so it's now possible to use this reference
> device as another RISC-V IOMMU PCI device to ease the burden of
> testing/development.
>
>    To instantiate the riscv-iommu-pci device using the previous Rivos PCI
> ID, use the following cmd line:
>
>    -device riscv-iommu-pci,vendor-id=0x1efd,device-id=0xedf1
>
>    I'm using these options to test the series with the existing Linux RISC-V
> IOMMU support that uses just a Rivos ID to identify the device.
>
>
> Series based on alistair/riscv-to-apply.next. It's also applicable on
> current QEMU master. It can also be fetched from:
>
> https://gitlab.com/danielhb/qemu/-/tree/riscv_iommu_v3
>   
>
> Patches missing reviews/acks: 3, 5, 9, 10, 11.
>
> Changes from v2 [1]:
> - patch 05 (hw/riscv: add riscv-iommu-sys platform device): dropped
>    - will be reintroduced in a later review or as a follow-up series
>
> - patches 14 and 15: dropped
>    - will be sent in separate
>
> - patches 2, 3, 4 and 5:
>    - removed all 'Ziommu' references
>
> - patch 2:
>    - added extra bits that patch 3 ended up using
>
> - patch 3:
>    - fixed blank line at EOF in hw/riscv/trace.h
>    - added a riscv_iommu_report_fault() helper to report faults. The helper checks if
>      a given fault is eligible to be reported if DTF is 1
>    - Use riscv_iommu_report_fault() in riscv_iommu_ctx() and riscv_iommu_translate()
>      to avoid code repetition
>    - added a riscv_iommu_validate_device_ctx() helper to validate the device context
>      as specified in "Device configuration checks" section. This helper is being used
>      in riscv_iommu_ctx_fetch()
>    - added a new riscv_iommu_update_ipsr() helper to handle IPSR updates
>      in riscv_iommu_mmio_write()
>    - riscv_iommmu_msi_write() now reports a fault in all error paths
>    - check for fctl.WSI before issuing a MSI interrupt in riscv_iommu_notify()
>    - change riscv-iommu region name to 'riscv-iommu'
>    - change address_space_init() name for PCI devices to 'name' instead of using TYPE_RISCV_IOMMU_PCI
>    - changed riscv_iommu_mmio_ops min_access_size to 4
>    - do not check for min and max sizes on riscv_iommu_mmio_write()
>    - changed riscv_iommu_trap_ops  min_access_size to 4
>    - removed IOMMU qemu_thread thread:
>      - riscv_iommu_mmio_write() will now execute a riscv_iommu_process_fn by holding
>        'core_lock'
>    - init FSCR as zero explicitly
>    - check for bus->iommu_opaque == NULL before calling pci_setup_iommu()
>
> - patch 4 (new):
>    - add Red-Hat PCI RISC-V IOMMU ID
>
> - patch 5 (former 4):
>    - create vendor-id and device-id properties
>    - set Red-hat PCI RISC-V IOMMU ID as default ID
>
> - patch 8:
>    - use IOMMU_NONE instead of '0' in relevant 'iot->perm = 0' instances
>
> - patch 9:
>    - add s-stage and g-stage steps in riscv_iommu_validate_device_ctx()
>    - removed 'gpa' boolean from riscv_iommu_spa_fetch()
>    - 'en_s' is no longer used for early MSI address match
>
> - patch 10:
>    - add ATS steps in riscv_iommu_validate_device_ctx()
>    - check for 's->enable_ats' before adding RISCV_IOMMU_DC_TC_EN_ATS in device context
>    - check for 's->enable_ats' before processing ATS commands in riscv_iommu_process_cq_tail()
>    - remove ambiguous trace_riscv_iommu_ats() from riscv_iommu_translate()
>
> - patch 11:
>    - removed unused bits
>    - added RISCV_IOMMU_TR_REQ_CTL_NW and RISCV_IOMMU_TR_RESPONSE_S
>      bits
>    - set IOMMUTLBEntry 'perm' using RISCV_IOMMU_TR_REQ_CTL_NW in riscv_iommu_process_dbg()
>    - clear RISCV_IOMMU_TR_RESPONSE_S in riscv_iommu_process_dbg(). Added a comment talking about the (lack of) superpage support
>   
> [1] https://lore.kernel.org/qemu-riscv/20240307160319.675044-1-dbarboza@ventanamicro.com/
> [2] https://github.com/vlsunil/qemu/commits/acpi_rimt_poc_v1/
>
> Andrew Jones (1):
>    hw/riscv/riscv-iommu: Add another irq for mrif notifications
>
> Daniel Henrique Barboza (3):
>    pci-ids.rst: add Red Hat pci-id for RISC-V IOMMU device
>    test/qtest: add riscv-iommu-pci tests
>    qtest/riscv-iommu-test: add init queues test
>
> Tomasz Jeznach (9):
>    exec/memtxattr: add process identifier to the transaction attributes
>    hw/riscv: add riscv-iommu-bits.h
>    hw/riscv: add RISC-V IOMMU base emulation
>    hw/riscv: add riscv-iommu-pci reference device
>    hw/riscv/virt.c: support for RISC-V IOMMU PCIDevice hotplug
>    hw/riscv/riscv-iommu: add Address Translation Cache (IOATC)
>    hw/riscv/riscv-iommu: add s-stage and g-stage support
>    hw/riscv/riscv-iommu: add ATS support
>    hw/riscv/riscv-iommu: add DBG support
>
>   docs/specs/pci-ids.rst           |    2 +
>   hw/riscv/Kconfig                 |    4 +
>   hw/riscv/meson.build             |    1 +
>   hw/riscv/riscv-iommu-bits.h      |  416 ++++++
>   hw/riscv/riscv-iommu-pci.c       |  177 +++
>   hw/riscv/riscv-iommu.c           | 2283 ++++++++++++++++++++++++++++++
>   hw/riscv/riscv-iommu.h           |  146 ++
>   hw/riscv/trace-events            |   15 +
>   hw/riscv/trace.h                 |    1 +
>   hw/riscv/virt.c                  |   33 +-
>   include/exec/memattrs.h          |    5 +
>   include/hw/pci/pci.h             |    1 +
>   include/hw/riscv/iommu.h         |   36 +
>   meson.build                      |    1 +
>   tests/qtest/libqos/meson.build   |    4 +
>   tests/qtest/libqos/riscv-iommu.c |   76 +
>   tests/qtest/libqos/riscv-iommu.h |  100 ++
>   tests/qtest/meson.build          |    1 +
>   tests/qtest/riscv-iommu-test.c   |  234 +++
>   19 files changed, 3535 insertions(+), 1 deletion(-)
>   create mode 100644 hw/riscv/riscv-iommu-bits.h
>   create mode 100644 hw/riscv/riscv-iommu-pci.c
>   create mode 100644 hw/riscv/riscv-iommu.c
>   create mode 100644 hw/riscv/riscv-iommu.h
>   create mode 100644 hw/riscv/trace-events
>   create mode 100644 hw/riscv/trace.h
>   create mode 100644 include/hw/riscv/iommu.h
>   create mode 100644 tests/qtest/libqos/riscv-iommu.c
>   create mode 100644 tests/qtest/libqos/riscv-iommu.h
>   create mode 100644 tests/qtest/riscv-iommu-test.c
>

Hi Zhiwei,

On 6/10/24 10:51 PM, LIU Zhiwei wrote:
> Hi Daniel,
> 
> I want to know if we can use the IOMMU and IOPMP at the same time.

AFAIK we can. They're not mutually exclusive since they offer protection
and isolation at different layers/stages.


> 
> The relationship between them is more similar to MMU and sPMP or to MMU and PMP?

I'd say MMU and PMP since the IOMMU can isolate devices regardless of
s-mode context or not.


Thanks,

Daniel

> 
> Thanks,
> Zhiwei
> 
> On 2024/5/24 1:39, Daniel Henrique Barboza wrote:
>> Hi,
>>
>> In this new version a lot of changes were made throughout all the code,
>> most notably on patch 3. Link for the previous version is [1].
>>
>> * How it was tested *
>>
>> This series was tested using an emulated QEMU RISC-V host booting a QEMU
>> KVM guest, passing through an emulated e1000 network card from the host
>> to the guest. I can provide more details (e.g. QEMU command lines) if
>> required, just let me know. For now this cover-letter is too much of an
>> essay as is.
>>
>> The Linux kernel used for tests can be found here:
>>
>> https://github.com/tjeznach/linux/tree/riscv_iommu_v6-rc3
>>
>> This is a newer version of the following work from Tomasz:
>>
>> https://lore.kernel.org/linux-riscv/cover.1715708679.git.tjeznach@rivosinc.com/
>> ("[PATCH v5 0/7] Linux RISC-V IOMMU Support")
>>
>> The v5 wasn't enough for the testing being done. v6-rc3 did the trick.
>>
>> Note that to test this work using riscv-iommu-pci we'll need to provide
>> the Rivos PCI ID in the command line. More details down below.
>>
>> * Highlights of this version *
>>
>> - patches removed from v2: platform driver (riscv-iommu-sys, former
>> patch 05) and the EDU changes (patches 14 and 15). The platform driver
>> will be sent later with a working example on the 'virt' machine,
>> either on a newer version of this series or via a follow-up series. We
>> already have a PoC on [2] created by Sunil. More tests are needed, so
>> it'll be left behind for now. The EDU changes will be sent in separate
>> after I finish the doc changes that Frank cited in v2.
>>
>> - patch 3 contains the bulk of changes made from v2. Please give special
>> attention to the following functions since this is entirely new code I
>> ended up adding:
>>   - riscv_iommu_report_fault()
>>   - riscv_iommu_validate_device_ctx()
>>   - riscv_iommu_update_ipsr()
>>    Aside from these helpers most of the changes made in this patch 3 were
>> punctual.
>>
>> - Red HAT PCI ID related changes. A new patch (4) that introduces a
>> generic RISC-V IOMMU PCI ID was added. This PCI ID was gracefully given
>> to us by Red Hat and Gerd Hoffman from their ID space. The
>> riscv-iommu-pci device now defaults to this PCI ID instead of Rivos PCI
>> ID. The device was changed slightly to allow vendor-id and device-id to
>> be set in the command-line, so it's now possible to use this reference
>> device as another RISC-V IOMMU PCI device to ease the burden of
>> testing/development.
>>
>>    To instantiate the riscv-iommu-pci device using the previous Rivos PCI
>> ID, use the following cmd line:
>>
>>    -device riscv-iommu-pci,vendor-id=0x1efd,device-id=0xedf1
>>
>>    I'm using these options to test the series with the existing Linux RISC-V
>> IOMMU support that uses just a Rivos ID to identify the device.
>>
>>
>> Series based on alistair/riscv-to-apply.next. It's also applicable on
>> current QEMU master. It can also be fetched from:
>>
>> https://gitlab.com/danielhb/qemu/-/tree/riscv_iommu_v3
>>
>> Patches missing reviews/acks: 3, 5, 9, 10, 11.
>>
>> Changes from v2 [1]:
>> - patch 05 (hw/riscv: add riscv-iommu-sys platform device): dropped
>>    - will be reintroduced in a later review or as a follow-up series
>>
>> - patches 14 and 15: dropped
>>    - will be sent in separate
>>
>> - patches 2, 3, 4 and 5:
>>    - removed all 'Ziommu' references
>>
>> - patch 2:
>>    - added extra bits that patch 3 ended up using
>>
>> - patch 3:
>>    - fixed blank line at EOF in hw/riscv/trace.h
>>    - added a riscv_iommu_report_fault() helper to report faults. The helper checks if
>>      a given fault is eligible to be reported if DTF is 1
>>    - Use riscv_iommu_report_fault() in riscv_iommu_ctx() and riscv_iommu_translate()
>>      to avoid code repetition
>>    - added a riscv_iommu_validate_device_ctx() helper to validate the device context
>>      as specified in "Device configuration checks" section. This helper is being used
>>      in riscv_iommu_ctx_fetch()
>>    - added a new riscv_iommu_update_ipsr() helper to handle IPSR updates
>>      in riscv_iommu_mmio_write()
>>    - riscv_iommmu_msi_write() now reports a fault in all error paths
>>    - check for fctl.WSI before issuing a MSI interrupt in riscv_iommu_notify()
>>    - change riscv-iommu region name to 'riscv-iommu'
>>    - change address_space_init() name for PCI devices to 'name' instead of using TYPE_RISCV_IOMMU_PCI
>>    - changed riscv_iommu_mmio_ops min_access_size to 4
>>    - do not check for min and max sizes on riscv_iommu_mmio_write()
>>    - changed riscv_iommu_trap_ops  min_access_size to 4
>>    - removed IOMMU qemu_thread thread:
>>      - riscv_iommu_mmio_write() will now execute a riscv_iommu_process_fn by holding
>>        'core_lock'
>>    - init FSCR as zero explicitly
>>    - check for bus->iommu_opaque == NULL before calling pci_setup_iommu()
>>
>> - patch 4 (new):
>>    - add Red-Hat PCI RISC-V IOMMU ID
>>
>> - patch 5 (former 4):
>>    - create vendor-id and device-id properties
>>    - set Red-hat PCI RISC-V IOMMU ID as default ID
>>
>> - patch 8:
>>    - use IOMMU_NONE instead of '0' in relevant 'iot->perm = 0' instances
>>
>> - patch 9:
>>    - add s-stage and g-stage steps in riscv_iommu_validate_device_ctx()
>>    - removed 'gpa' boolean from riscv_iommu_spa_fetch()
>>    - 'en_s' is no longer used for early MSI address match
>>
>> - patch 10:
>>    - add ATS steps in riscv_iommu_validate_device_ctx()
>>    - check for 's->enable_ats' before adding RISCV_IOMMU_DC_TC_EN_ATS in device context
>>    - check for 's->enable_ats' before processing ATS commands in riscv_iommu_process_cq_tail()
>>    - remove ambiguous trace_riscv_iommu_ats() from riscv_iommu_translate()
>>
>> - patch 11:
>>    - removed unused bits
>>    - added RISCV_IOMMU_TR_REQ_CTL_NW and RISCV_IOMMU_TR_RESPONSE_S
>>      bits
>>    - set IOMMUTLBEntry 'perm' using RISCV_IOMMU_TR_REQ_CTL_NW in riscv_iommu_process_dbg()
>>    - clear RISCV_IOMMU_TR_RESPONSE_S in riscv_iommu_process_dbg(). Added a comment talking about the (lack of) superpage support
>> [1] https://lore.kernel.org/qemu-riscv/20240307160319.675044-1-dbarboza@ventanamicro.com/
>> [2] https://github.com/vlsunil/qemu/commits/acpi_rimt_poc_v1/
>>
>> Andrew Jones (1):
>>    hw/riscv/riscv-iommu: Add another irq for mrif notifications
>>
>> Daniel Henrique Barboza (3):
>>    pci-ids.rst: add Red Hat pci-id for RISC-V IOMMU device
>>    test/qtest: add riscv-iommu-pci tests
>>    qtest/riscv-iommu-test: add init queues test
>>
>> Tomasz Jeznach (9):
>>    exec/memtxattr: add process identifier to the transaction attributes
>>    hw/riscv: add riscv-iommu-bits.h
>>    hw/riscv: add RISC-V IOMMU base emulation
>>    hw/riscv: add riscv-iommu-pci reference device
>>    hw/riscv/virt.c: support for RISC-V IOMMU PCIDevice hotplug
>>    hw/riscv/riscv-iommu: add Address Translation Cache (IOATC)
>>    hw/riscv/riscv-iommu: add s-stage and g-stage support
>>    hw/riscv/riscv-iommu: add ATS support
>>    hw/riscv/riscv-iommu: add DBG support
>>
>>   docs/specs/pci-ids.rst           |    2 +
>>   hw/riscv/Kconfig                 |    4 +
>>   hw/riscv/meson.build             |    1 +
>>   hw/riscv/riscv-iommu-bits.h      |  416 ++++++
>>   hw/riscv/riscv-iommu-pci.c       |  177 +++
>>   hw/riscv/riscv-iommu.c           | 2283 ++++++++++++++++++++++++++++++
>>   hw/riscv/riscv-iommu.h           |  146 ++
>>   hw/riscv/trace-events            |   15 +
>>   hw/riscv/trace.h                 |    1 +
>>   hw/riscv/virt.c                  |   33 +-
>>   include/exec/memattrs.h          |    5 +
>>   include/hw/pci/pci.h             |    1 +
>>   include/hw/riscv/iommu.h         |   36 +
>>   meson.build                      |    1 +
>>   tests/qtest/libqos/meson.build   |    4 +
>>   tests/qtest/libqos/riscv-iommu.c |   76 +
>>   tests/qtest/libqos/riscv-iommu.h |  100 ++
>>   tests/qtest/meson.build          |    1 +
>>   tests/qtest/riscv-iommu-test.c   |  234 +++
>>   19 files changed, 3535 insertions(+), 1 deletion(-)
>>   create mode 100644 hw/riscv/riscv-iommu-bits.h
>>   create mode 100644 hw/riscv/riscv-iommu-pci.c
>>   create mode 100644 hw/riscv/riscv-iommu.c
>>   create mode 100644 hw/riscv/riscv-iommu.h
>>   create mode 100644 hw/riscv/trace-events
>>   create mode 100644 hw/riscv/trace.h
>>   create mode 100644 include/hw/riscv/iommu.h
>>   create mode 100644 tests/qtest/libqos/riscv-iommu.c
>>   create mode 100644 tests/qtest/libqos/riscv-iommu.h
>>   create mode 100644 tests/qtest/riscv-iommu-test.c
>>

On 2024/6/11 18:13, Daniel Henrique Barboza wrote:
> Hi Zhiwei,
>
> On 6/10/24 10:51 PM, LIU Zhiwei wrote:
>> Hi Daniel,
>>
>> I want to know if we can use the IOMMU and IOPMP at the same time.
>
> AFAIK we can. They're not mutually exclusive since they offer protection
> and isolation at different layers/stages.

OK. Thanks. I will dive into more details.

I see the IOMMU and IOPMP implementations on mail list both set IOMMU 
for PCI root bus.
Is it right?

Thanks,
Zhiwei

>
>
>>
>> The relationship between them is more similar to MMU and sPMP or to 
>> MMU and PMP?
>
> I'd say MMU and PMP since the IOMMU can isolate devices regardless of
> s-mode context or not.
>
>
> Thanks,
>
> Daniel
>
>>
>> Thanks,
>> Zhiwei
>>
>> On 2024/5/24 1:39, Daniel Henrique Barboza wrote:
>>> Hi,
>>>
>>> In this new version a lot of changes were made throughout all the code,
>>> most notably on patch 3. Link for the previous version is [1].
>>>
>>> * How it was tested *
>>>
>>> This series was tested using an emulated QEMU RISC-V host booting a 
>>> QEMU
>>> KVM guest, passing through an emulated e1000 network card from the host
>>> to the guest. I can provide more details (e.g. QEMU command lines) if
>>> required, just let me know. For now this cover-letter is too much of an
>>> essay as is.
>>>
>>> The Linux kernel used for tests can be found here:
>>>
>>> https://github.com/tjeznach/linux/tree/riscv_iommu_v6-rc3
>>>
>>> This is a newer version of the following work from Tomasz:
>>>
>>> https://lore.kernel.org/linux-riscv/cover.1715708679.git.tjeznach@rivosinc.com/ 
>>>
>>> ("[PATCH v5 0/7] Linux RISC-V IOMMU Support")
>>>
>>> The v5 wasn't enough for the testing being done. v6-rc3 did the trick.
>>>
>>> Note that to test this work using riscv-iommu-pci we'll need to provide
>>> the Rivos PCI ID in the command line. More details down below.
>>>
>>> * Highlights of this version *
>>>
>>> - patches removed from v2: platform driver (riscv-iommu-sys, former
>>> patch 05) and the EDU changes (patches 14 and 15). The platform driver
>>> will be sent later with a working example on the 'virt' machine,
>>> either on a newer version of this series or via a follow-up series. We
>>> already have a PoC on [2] created by Sunil. More tests are needed, so
>>> it'll be left behind for now. The EDU changes will be sent in separate
>>> after I finish the doc changes that Frank cited in v2.
>>>
>>> - patch 3 contains the bulk of changes made from v2. Please give 
>>> special
>>> attention to the following functions since this is entirely new code I
>>> ended up adding:
>>>   - riscv_iommu_report_fault()
>>>   - riscv_iommu_validate_device_ctx()
>>>   - riscv_iommu_update_ipsr()
>>>    Aside from these helpers most of the changes made in this patch 3 
>>> were
>>> punctual.
>>>
>>> - Red HAT PCI ID related changes. A new patch (4) that introduces a
>>> generic RISC-V IOMMU PCI ID was added. This PCI ID was gracefully given
>>> to us by Red Hat and Gerd Hoffman from their ID space. The
>>> riscv-iommu-pci device now defaults to this PCI ID instead of Rivos PCI
>>> ID. The device was changed slightly to allow vendor-id and device-id to
>>> be set in the command-line, so it's now possible to use this reference
>>> device as another RISC-V IOMMU PCI device to ease the burden of
>>> testing/development.
>>>
>>>    To instantiate the riscv-iommu-pci device using the previous 
>>> Rivos PCI
>>> ID, use the following cmd line:
>>>
>>>    -device riscv-iommu-pci,vendor-id=0x1efd,device-id=0xedf1
>>>
>>>    I'm using these options to test the series with the existing 
>>> Linux RISC-V
>>> IOMMU support that uses just a Rivos ID to identify the device.
>>>
>>>
>>> Series based on alistair/riscv-to-apply.next. It's also applicable on
>>> current QEMU master. It can also be fetched from:
>>>
>>> https://gitlab.com/danielhb/qemu/-/tree/riscv_iommu_v3
>>>
>>> Patches missing reviews/acks: 3, 5, 9, 10, 11.
>>>
>>> Changes from v2 [1]:
>>> - patch 05 (hw/riscv: add riscv-iommu-sys platform device): dropped
>>>    - will be reintroduced in a later review or as a follow-up series
>>>
>>> - patches 14 and 15: dropped
>>>    - will be sent in separate
>>>
>>> - patches 2, 3, 4 and 5:
>>>    - removed all 'Ziommu' references
>>>
>>> - patch 2:
>>>    - added extra bits that patch 3 ended up using
>>>
>>> - patch 3:
>>>    - fixed blank line at EOF in hw/riscv/trace.h
>>>    - added a riscv_iommu_report_fault() helper to report faults. The 
>>> helper checks if
>>>      a given fault is eligible to be reported if DTF is 1
>>>    - Use riscv_iommu_report_fault() in riscv_iommu_ctx() and 
>>> riscv_iommu_translate()
>>>      to avoid code repetition
>>>    - added a riscv_iommu_validate_device_ctx() helper to validate 
>>> the device context
>>>      as specified in "Device configuration checks" section. This 
>>> helper is being used
>>>      in riscv_iommu_ctx_fetch()
>>>    - added a new riscv_iommu_update_ipsr() helper to handle IPSR 
>>> updates
>>>      in riscv_iommu_mmio_write()
>>>    - riscv_iommmu_msi_write() now reports a fault in all error paths
>>>    - check for fctl.WSI before issuing a MSI interrupt in 
>>> riscv_iommu_notify()
>>>    - change riscv-iommu region name to 'riscv-iommu'
>>>    - change address_space_init() name for PCI devices to 'name' 
>>> instead of using TYPE_RISCV_IOMMU_PCI
>>>    - changed riscv_iommu_mmio_ops min_access_size to 4
>>>    - do not check for min and max sizes on riscv_iommu_mmio_write()
>>>    - changed riscv_iommu_trap_ops  min_access_size to 4
>>>    - removed IOMMU qemu_thread thread:
>>>      - riscv_iommu_mmio_write() will now execute a 
>>> riscv_iommu_process_fn by holding
>>>        'core_lock'
>>>    - init FSCR as zero explicitly
>>>    - check for bus->iommu_opaque == NULL before calling 
>>> pci_setup_iommu()
>>>
>>> - patch 4 (new):
>>>    - add Red-Hat PCI RISC-V IOMMU ID
>>>
>>> - patch 5 (former 4):
>>>    - create vendor-id and device-id properties
>>>    - set Red-hat PCI RISC-V IOMMU ID as default ID
>>>
>>> - patch 8:
>>>    - use IOMMU_NONE instead of '0' in relevant 'iot->perm = 0' 
>>> instances
>>>
>>> - patch 9:
>>>    - add s-stage and g-stage steps in riscv_iommu_validate_device_ctx()
>>>    - removed 'gpa' boolean from riscv_iommu_spa_fetch()
>>>    - 'en_s' is no longer used for early MSI address match
>>>
>>> - patch 10:
>>>    - add ATS steps in riscv_iommu_validate_device_ctx()
>>>    - check for 's->enable_ats' before adding 
>>> RISCV_IOMMU_DC_TC_EN_ATS in device context
>>>    - check for 's->enable_ats' before processing ATS commands in 
>>> riscv_iommu_process_cq_tail()
>>>    - remove ambiguous trace_riscv_iommu_ats() from 
>>> riscv_iommu_translate()
>>>
>>> - patch 11:
>>>    - removed unused bits
>>>    - added RISCV_IOMMU_TR_REQ_CTL_NW and RISCV_IOMMU_TR_RESPONSE_S
>>>      bits
>>>    - set IOMMUTLBEntry 'perm' using RISCV_IOMMU_TR_REQ_CTL_NW in 
>>> riscv_iommu_process_dbg()
>>>    - clear RISCV_IOMMU_TR_RESPONSE_S in riscv_iommu_process_dbg(). 
>>> Added a comment talking about the (lack of) superpage support
>>> [1] 
>>> https://lore.kernel.org/qemu-riscv/20240307160319.675044-1-dbarboza@ventanamicro.com/
>>> [2] https://github.com/vlsunil/qemu/commits/acpi_rimt_poc_v1/
>>>
>>> Andrew Jones (1):
>>>    hw/riscv/riscv-iommu: Add another irq for mrif notifications
>>>
>>> Daniel Henrique Barboza (3):
>>>    pci-ids.rst: add Red Hat pci-id for RISC-V IOMMU device
>>>    test/qtest: add riscv-iommu-pci tests
>>>    qtest/riscv-iommu-test: add init queues test
>>>
>>> Tomasz Jeznach (9):
>>>    exec/memtxattr: add process identifier to the transaction attributes
>>>    hw/riscv: add riscv-iommu-bits.h
>>>    hw/riscv: add RISC-V IOMMU base emulation
>>>    hw/riscv: add riscv-iommu-pci reference device
>>>    hw/riscv/virt.c: support for RISC-V IOMMU PCIDevice hotplug
>>>    hw/riscv/riscv-iommu: add Address Translation Cache (IOATC)
>>>    hw/riscv/riscv-iommu: add s-stage and g-stage support
>>>    hw/riscv/riscv-iommu: add ATS support
>>>    hw/riscv/riscv-iommu: add DBG support
>>>
>>>   docs/specs/pci-ids.rst           |    2 +
>>>   hw/riscv/Kconfig                 |    4 +
>>>   hw/riscv/meson.build             |    1 +
>>>   hw/riscv/riscv-iommu-bits.h      |  416 ++++++
>>>   hw/riscv/riscv-iommu-pci.c       |  177 +++
>>>   hw/riscv/riscv-iommu.c           | 2283 
>>> ++++++++++++++++++++++++++++++
>>>   hw/riscv/riscv-iommu.h           |  146 ++
>>>   hw/riscv/trace-events            |   15 +
>>>   hw/riscv/trace.h                 |    1 +
>>>   hw/riscv/virt.c                  |   33 +-
>>>   include/exec/memattrs.h          |    5 +
>>>   include/hw/pci/pci.h             |    1 +
>>>   include/hw/riscv/iommu.h         |   36 +
>>>   meson.build                      |    1 +
>>>   tests/qtest/libqos/meson.build   |    4 +
>>>   tests/qtest/libqos/riscv-iommu.c |   76 +
>>>   tests/qtest/libqos/riscv-iommu.h |  100 ++
>>>   tests/qtest/meson.build          |    1 +
>>>   tests/qtest/riscv-iommu-test.c   |  234 +++
>>>   19 files changed, 3535 insertions(+), 1 deletion(-)
>>>   create mode 100644 hw/riscv/riscv-iommu-bits.h
>>>   create mode 100644 hw/riscv/riscv-iommu-pci.c
>>>   create mode 100644 hw/riscv/riscv-iommu.c
>>>   create mode 100644 hw/riscv/riscv-iommu.h
>>>   create mode 100644 hw/riscv/trace-events
>>>   create mode 100644 hw/riscv/trace.h
>>>   create mode 100644 include/hw/riscv/iommu.h
>>>   create mode 100644 tests/qtest/libqos/riscv-iommu.c
>>>   create mode 100644 tests/qtest/libqos/riscv-iommu.h
>>>   create mode 100644 tests/qtest/riscv-iommu-test.c
>>>

On 6/12/24 4:50 AM, LIU Zhiwei wrote:
> 
> On 2024/6/11 18:13, Daniel Henrique Barboza wrote:
>> Hi Zhiwei,
>>
>> On 6/10/24 10:51 PM, LIU Zhiwei wrote:
>>> Hi Daniel,
>>>
>>> I want to know if we can use the IOMMU and IOPMP at the same time.
>>
>> AFAIK we can. They're not mutually exclusive since they offer protection
>> and isolation at different layers/stages.
> 
> OK. Thanks. I will dive into more details.
> 
> I see the IOMMU and IOPMP implementations on mail list both set IOMMU for PCI root bus.
> Is it right?

For now the riscv-iommu-pci device must be placed at a root bus for the
sake of simplicity. We'll want to lift this restriction in the future as
the support matures.


Thanks,

Daniel

> 
> Thanks,
> Zhiwei
> 
>>
>>
>>>
>>> The relationship between them is more similar to MMU and sPMP or to MMU and PMP?
>>
>> I'd say MMU and PMP since the IOMMU can isolate devices regardless of
>> s-mode context or not.
>>
>>
>> Thanks,
>>
>> Daniel
>>
>>>
>>> Thanks,
>>> Zhiwei
>>>
>>> On 2024/5/24 1:39, Daniel Henrique Barboza wrote:
>>>> Hi,
>>>>
>>>> In this new version a lot of changes were made throughout all the code,
>>>> most notably on patch 3. Link for the previous version is [1].
>>>>
>>>> * How it was tested *
>>>>
>>>> This series was tested using an emulated QEMU RISC-V host booting a QEMU
>>>> KVM guest, passing through an emulated e1000 network card from the host
>>>> to the guest. I can provide more details (e.g. QEMU command lines) if
>>>> required, just let me know. For now this cover-letter is too much of an
>>>> essay as is.
>>>>
>>>> The Linux kernel used for tests can be found here:
>>>>
>>>> https://github.com/tjeznach/linux/tree/riscv_iommu_v6-rc3
>>>>
>>>> This is a newer version of the following work from Tomasz:
>>>>
>>>> https://lore.kernel.org/linux-riscv/cover.1715708679.git.tjeznach@rivosinc.com/
>>>> ("[PATCH v5 0/7] Linux RISC-V IOMMU Support")
>>>>
>>>> The v5 wasn't enough for the testing being done. v6-rc3 did the trick.
>>>>
>>>> Note that to test this work using riscv-iommu-pci we'll need to provide
>>>> the Rivos PCI ID in the command line. More details down below.
>>>>
>>>> * Highlights of this version *
>>>>
>>>> - patches removed from v2: platform driver (riscv-iommu-sys, former
>>>> patch 05) and the EDU changes (patches 14 and 15). The platform driver
>>>> will be sent later with a working example on the 'virt' machine,
>>>> either on a newer version of this series or via a follow-up series. We
>>>> already have a PoC on [2] created by Sunil. More tests are needed, so
>>>> it'll be left behind for now. The EDU changes will be sent in separate
>>>> after I finish the doc changes that Frank cited in v2.
>>>>
>>>> - patch 3 contains the bulk of changes made from v2. Please give special
>>>> attention to the following functions since this is entirely new code I
>>>> ended up adding:
>>>>   - riscv_iommu_report_fault()
>>>>   - riscv_iommu_validate_device_ctx()
>>>>   - riscv_iommu_update_ipsr()
>>>>    Aside from these helpers most of the changes made in this patch 3 were
>>>> punctual.
>>>>
>>>> - Red HAT PCI ID related changes. A new patch (4) that introduces a
>>>> generic RISC-V IOMMU PCI ID was added. This PCI ID was gracefully given
>>>> to us by Red Hat and Gerd Hoffman from their ID space. The
>>>> riscv-iommu-pci device now defaults to this PCI ID instead of Rivos PCI
>>>> ID. The device was changed slightly to allow vendor-id and device-id to
>>>> be set in the command-line, so it's now possible to use this reference
>>>> device as another RISC-V IOMMU PCI device to ease the burden of
>>>> testing/development.
>>>>
>>>>    To instantiate the riscv-iommu-pci device using the previous Rivos PCI
>>>> ID, use the following cmd line:
>>>>
>>>>    -device riscv-iommu-pci,vendor-id=0x1efd,device-id=0xedf1
>>>>
>>>>    I'm using these options to test the series with the existing Linux RISC-V
>>>> IOMMU support that uses just a Rivos ID to identify the device.
>>>>
>>>>
>>>> Series based on alistair/riscv-to-apply.next. It's also applicable on
>>>> current QEMU master. It can also be fetched from:
>>>>
>>>> https://gitlab.com/danielhb/qemu/-/tree/riscv_iommu_v3
>>>>
>>>> Patches missing reviews/acks: 3, 5, 9, 10, 11.
>>>>
>>>> Changes from v2 [1]:
>>>> - patch 05 (hw/riscv: add riscv-iommu-sys platform device): dropped
>>>>    - will be reintroduced in a later review or as a follow-up series
>>>>
>>>> - patches 14 and 15: dropped
>>>>    - will be sent in separate
>>>>
>>>> - patches 2, 3, 4 and 5:
>>>>    - removed all 'Ziommu' references
>>>>
>>>> - patch 2:
>>>>    - added extra bits that patch 3 ended up using
>>>>
>>>> - patch 3:
>>>>    - fixed blank line at EOF in hw/riscv/trace.h
>>>>    - added a riscv_iommu_report_fault() helper to report faults. The helper checks if
>>>>      a given fault is eligible to be reported if DTF is 1
>>>>    - Use riscv_iommu_report_fault() in riscv_iommu_ctx() and riscv_iommu_translate()
>>>>      to avoid code repetition
>>>>    - added a riscv_iommu_validate_device_ctx() helper to validate the device context
>>>>      as specified in "Device configuration checks" section. This helper is being used
>>>>      in riscv_iommu_ctx_fetch()
>>>>    - added a new riscv_iommu_update_ipsr() helper to handle IPSR updates
>>>>      in riscv_iommu_mmio_write()
>>>>    - riscv_iommmu_msi_write() now reports a fault in all error paths
>>>>    - check for fctl.WSI before issuing a MSI interrupt in riscv_iommu_notify()
>>>>    - change riscv-iommu region name to 'riscv-iommu'
>>>>    - change address_space_init() name for PCI devices to 'name' instead of using TYPE_RISCV_IOMMU_PCI
>>>>    - changed riscv_iommu_mmio_ops min_access_size to 4
>>>>    - do not check for min and max sizes on riscv_iommu_mmio_write()
>>>>    - changed riscv_iommu_trap_ops  min_access_size to 4
>>>>    - removed IOMMU qemu_thread thread:
>>>>      - riscv_iommu_mmio_write() will now execute a riscv_iommu_process_fn by holding
>>>>        'core_lock'
>>>>    - init FSCR as zero explicitly
>>>>    - check for bus->iommu_opaque == NULL before calling pci_setup_iommu()
>>>>
>>>> - patch 4 (new):
>>>>    - add Red-Hat PCI RISC-V IOMMU ID
>>>>
>>>> - patch 5 (former 4):
>>>>    - create vendor-id and device-id properties
>>>>    - set Red-hat PCI RISC-V IOMMU ID as default ID
>>>>
>>>> - patch 8:
>>>>    - use IOMMU_NONE instead of '0' in relevant 'iot->perm = 0' instances
>>>>
>>>> - patch 9:
>>>>    - add s-stage and g-stage steps in riscv_iommu_validate_device_ctx()
>>>>    - removed 'gpa' boolean from riscv_iommu_spa_fetch()
>>>>    - 'en_s' is no longer used for early MSI address match
>>>>
>>>> - patch 10:
>>>>    - add ATS steps in riscv_iommu_validate_device_ctx()
>>>>    - check for 's->enable_ats' before adding RISCV_IOMMU_DC_TC_EN_ATS in device context
>>>>    - check for 's->enable_ats' before processing ATS commands in riscv_iommu_process_cq_tail()
>>>>    - remove ambiguous trace_riscv_iommu_ats() from riscv_iommu_translate()
>>>>
>>>> - patch 11:
>>>>    - removed unused bits
>>>>    - added RISCV_IOMMU_TR_REQ_CTL_NW and RISCV_IOMMU_TR_RESPONSE_S
>>>>      bits
>>>>    - set IOMMUTLBEntry 'perm' using RISCV_IOMMU_TR_REQ_CTL_NW in riscv_iommu_process_dbg()
>>>>    - clear RISCV_IOMMU_TR_RESPONSE_S in riscv_iommu_process_dbg(). Added a comment talking about the (lack of) superpage support
>>>> [1] https://lore.kernel.org/qemu-riscv/20240307160319.675044-1-dbarboza@ventanamicro.com/
>>>> [2] https://github.com/vlsunil/qemu/commits/acpi_rimt_poc_v1/
>>>>
>>>> Andrew Jones (1):
>>>>    hw/riscv/riscv-iommu: Add another irq for mrif notifications
>>>>
>>>> Daniel Henrique Barboza (3):
>>>>    pci-ids.rst: add Red Hat pci-id for RISC-V IOMMU device
>>>>    test/qtest: add riscv-iommu-pci tests
>>>>    qtest/riscv-iommu-test: add init queues test
>>>>
>>>> Tomasz Jeznach (9):
>>>>    exec/memtxattr: add process identifier to the transaction attributes
>>>>    hw/riscv: add riscv-iommu-bits.h
>>>>    hw/riscv: add RISC-V IOMMU base emulation
>>>>    hw/riscv: add riscv-iommu-pci reference device
>>>>    hw/riscv/virt.c: support for RISC-V IOMMU PCIDevice hotplug
>>>>    hw/riscv/riscv-iommu: add Address Translation Cache (IOATC)
>>>>    hw/riscv/riscv-iommu: add s-stage and g-stage support
>>>>    hw/riscv/riscv-iommu: add ATS support
>>>>    hw/riscv/riscv-iommu: add DBG support
>>>>
>>>>   docs/specs/pci-ids.rst           |    2 +
>>>>   hw/riscv/Kconfig                 |    4 +
>>>>   hw/riscv/meson.build             |    1 +
>>>>   hw/riscv/riscv-iommu-bits.h      |  416 ++++++
>>>>   hw/riscv/riscv-iommu-pci.c       |  177 +++
>>>>   hw/riscv/riscv-iommu.c           | 2283 ++++++++++++++++++++++++++++++
>>>>   hw/riscv/riscv-iommu.h           |  146 ++
>>>>   hw/riscv/trace-events            |   15 +
>>>>   hw/riscv/trace.h                 |    1 +
>>>>   hw/riscv/virt.c                  |   33 +-
>>>>   include/exec/memattrs.h          |    5 +
>>>>   include/hw/pci/pci.h             |    1 +
>>>>   include/hw/riscv/iommu.h         |   36 +
>>>>   meson.build                      |    1 +
>>>>   tests/qtest/libqos/meson.build   |    4 +
>>>>   tests/qtest/libqos/riscv-iommu.c |   76 +
>>>>   tests/qtest/libqos/riscv-iommu.h |  100 ++
>>>>   tests/qtest/meson.build          |    1 +
>>>>   tests/qtest/riscv-iommu-test.c   |  234 +++
>>>>   19 files changed, 3535 insertions(+), 1 deletion(-)
>>>>   create mode 100644 hw/riscv/riscv-iommu-bits.h
>>>>   create mode 100644 hw/riscv/riscv-iommu-pci.c
>>>>   create mode 100644 hw/riscv/riscv-iommu.c
>>>>   create mode 100644 hw/riscv/riscv-iommu.h
>>>>   create mode 100644 hw/riscv/trace-events
>>>>   create mode 100644 hw/riscv/trace.h
>>>>   create mode 100644 include/hw/riscv/iommu.h
>>>>   create mode 100644 tests/qtest/libqos/riscv-iommu.c
>>>>   create mode 100644 tests/qtest/libqos/riscv-iommu.h
>>>>   create mode 100644 tests/qtest/riscv-iommu-test.c
>>>>

On 2024/6/12 20:10, Daniel Henrique Barboza wrote:
>
>
> On 6/12/24 4:50 AM, LIU Zhiwei wrote:
>>
>> On 2024/6/11 18:13, Daniel Henrique Barboza wrote:
>>> Hi Zhiwei,
>>>
>>> On 6/10/24 10:51 PM, LIU Zhiwei wrote:
>>>> Hi Daniel,
>>>>
>>>> I want to know if we can use the IOMMU and IOPMP at the same time.
>>>
>>> AFAIK we can. They're not mutually exclusive since they offer 
>>> protection
>>> and isolation at different layers/stages.
>>
>> OK. Thanks. I will dive into more details.
>>
>> I see the IOMMU and IOPMP implementations on mail list both set IOMMU 
>> for PCI root bus.
>> Is it right?
>
> For now the riscv-iommu-pci device must be placed at a root bus for the
> sake of simplicity. 
Agree.
> We'll want to lift this restriction in the future as
> the support matures.

I think it's OK if it only belongs to virt machine.

If we both support IOPMP and IOMMU for PCI, I think we should call 
pci_setup_iommu only once and use the same address space finding 
function for them.

At least, currently I don't find a reason that they can't share the same 
address space finding function.

Thanks,
Zhiwei

>
>
> Thanks,
>
> Daniel
>
>>
>> Thanks,
>> Zhiwei
>>
>>>
>>>
>>>>
>>>> The relationship between them is more similar to MMU and sPMP or to 
>>>> MMU and PMP?
>>>
>>> I'd say MMU and PMP since the IOMMU can isolate devices regardless of
>>> s-mode context or not.
>>>
>>>
>>> Thanks,
>>>
>>> Daniel
>>>
>>>>
>>>> Thanks,
>>>> Zhiwei
>>>>
>>>> On 2024/5/24 1:39, Daniel Henrique Barboza wrote:
>>>>> Hi,
>>>>>
>>>>> In this new version a lot of changes were made throughout all the 
>>>>> code,
>>>>> most notably on patch 3. Link for the previous version is [1].
>>>>>
>>>>> * How it was tested *
>>>>>
>>>>> This series was tested using an emulated QEMU RISC-V host booting 
>>>>> a QEMU
>>>>> KVM guest, passing through an emulated e1000 network card from the 
>>>>> host
>>>>> to the guest. I can provide more details (e.g. QEMU command lines) if
>>>>> required, just let me know. For now this cover-letter is too much 
>>>>> of an
>>>>> essay as is.
>>>>>
>>>>> The Linux kernel used for tests can be found here:
>>>>>
>>>>> https://github.com/tjeznach/linux/tree/riscv_iommu_v6-rc3
>>>>>
>>>>> This is a newer version of the following work from Tomasz:
>>>>>
>>>>> https://lore.kernel.org/linux-riscv/cover.1715708679.git.tjeznach@rivosinc.com/ 
>>>>>
>>>>> ("[PATCH v5 0/7] Linux RISC-V IOMMU Support")
>>>>>
>>>>> The v5 wasn't enough for the testing being done. v6-rc3 did the 
>>>>> trick.
>>>>>
>>>>> Note that to test this work using riscv-iommu-pci we'll need to 
>>>>> provide
>>>>> the Rivos PCI ID in the command line. More details down below.
>>>>>
>>>>> * Highlights of this version *
>>>>>
>>>>> - patches removed from v2: platform driver (riscv-iommu-sys, former
>>>>> patch 05) and the EDU changes (patches 14 and 15). The platform 
>>>>> driver
>>>>> will be sent later with a working example on the 'virt' machine,
>>>>> either on a newer version of this series or via a follow-up 
>>>>> series. We
>>>>> already have a PoC on [2] created by Sunil. More tests are needed, so
>>>>> it'll be left behind for now. The EDU changes will be sent in 
>>>>> separate
>>>>> after I finish the doc changes that Frank cited in v2.
>>>>>
>>>>> - patch 3 contains the bulk of changes made from v2. Please give 
>>>>> special
>>>>> attention to the following functions since this is entirely new 
>>>>> code I
>>>>> ended up adding:
>>>>>   - riscv_iommu_report_fault()
>>>>>   - riscv_iommu_validate_device_ctx()
>>>>>   - riscv_iommu_update_ipsr()
>>>>>    Aside from these helpers most of the changes made in this patch 
>>>>> 3 were
>>>>> punctual.
>>>>>
>>>>> - Red HAT PCI ID related changes. A new patch (4) that introduces a
>>>>> generic RISC-V IOMMU PCI ID was added. This PCI ID was gracefully 
>>>>> given
>>>>> to us by Red Hat and Gerd Hoffman from their ID space. The
>>>>> riscv-iommu-pci device now defaults to this PCI ID instead of 
>>>>> Rivos PCI
>>>>> ID. The device was changed slightly to allow vendor-id and 
>>>>> device-id to
>>>>> be set in the command-line, so it's now possible to use this 
>>>>> reference
>>>>> device as another RISC-V IOMMU PCI device to ease the burden of
>>>>> testing/development.
>>>>>
>>>>>    To instantiate the riscv-iommu-pci device using the previous 
>>>>> Rivos PCI
>>>>> ID, use the following cmd line:
>>>>>
>>>>>    -device riscv-iommu-pci,vendor-id=0x1efd,device-id=0xedf1
>>>>>
>>>>>    I'm using these options to test the series with the existing 
>>>>> Linux RISC-V
>>>>> IOMMU support that uses just a Rivos ID to identify the device.
>>>>>
>>>>>
>>>>> Series based on alistair/riscv-to-apply.next. It's also applicable on
>>>>> current QEMU master. It can also be fetched from:
>>>>>
>>>>> https://gitlab.com/danielhb/qemu/-/tree/riscv_iommu_v3
>>>>>
>>>>> Patches missing reviews/acks: 3, 5, 9, 10, 11.
>>>>>
>>>>> Changes from v2 [1]:
>>>>> - patch 05 (hw/riscv: add riscv-iommu-sys platform device): dropped
>>>>>    - will be reintroduced in a later review or as a follow-up series
>>>>>
>>>>> - patches 14 and 15: dropped
>>>>>    - will be sent in separate
>>>>>
>>>>> - patches 2, 3, 4 and 5:
>>>>>    - removed all 'Ziommu' references
>>>>>
>>>>> - patch 2:
>>>>>    - added extra bits that patch 3 ended up using
>>>>>
>>>>> - patch 3:
>>>>>    - fixed blank line at EOF in hw/riscv/trace.h
>>>>>    - added a riscv_iommu_report_fault() helper to report faults. 
>>>>> The helper checks if
>>>>>      a given fault is eligible to be reported if DTF is 1
>>>>>    - Use riscv_iommu_report_fault() in riscv_iommu_ctx() and 
>>>>> riscv_iommu_translate()
>>>>>      to avoid code repetition
>>>>>    - added a riscv_iommu_validate_device_ctx() helper to validate 
>>>>> the device context
>>>>>      as specified in "Device configuration checks" section. This 
>>>>> helper is being used
>>>>>      in riscv_iommu_ctx_fetch()
>>>>>    - added a new riscv_iommu_update_ipsr() helper to handle IPSR 
>>>>> updates
>>>>>      in riscv_iommu_mmio_write()
>>>>>    - riscv_iommmu_msi_write() now reports a fault in all error paths
>>>>>    - check for fctl.WSI before issuing a MSI interrupt in 
>>>>> riscv_iommu_notify()
>>>>>    - change riscv-iommu region name to 'riscv-iommu'
>>>>>    - change address_space_init() name for PCI devices to 'name' 
>>>>> instead of using TYPE_RISCV_IOMMU_PCI
>>>>>    - changed riscv_iommu_mmio_ops min_access_size to 4
>>>>>    - do not check for min and max sizes on riscv_iommu_mmio_write()
>>>>>    - changed riscv_iommu_trap_ops  min_access_size to 4
>>>>>    - removed IOMMU qemu_thread thread:
>>>>>      - riscv_iommu_mmio_write() will now execute a 
>>>>> riscv_iommu_process_fn by holding
>>>>>        'core_lock'
>>>>>    - init FSCR as zero explicitly
>>>>>    - check for bus->iommu_opaque == NULL before calling 
>>>>> pci_setup_iommu()
>>>>>
>>>>> - patch 4 (new):
>>>>>    - add Red-Hat PCI RISC-V IOMMU ID
>>>>>
>>>>> - patch 5 (former 4):
>>>>>    - create vendor-id and device-id properties
>>>>>    - set Red-hat PCI RISC-V IOMMU ID as default ID
>>>>>
>>>>> - patch 8:
>>>>>    - use IOMMU_NONE instead of '0' in relevant 'iot->perm = 0' 
>>>>> instances
>>>>>
>>>>> - patch 9:
>>>>>    - add s-stage and g-stage steps in 
>>>>> riscv_iommu_validate_device_ctx()
>>>>>    - removed 'gpa' boolean from riscv_iommu_spa_fetch()
>>>>>    - 'en_s' is no longer used for early MSI address match
>>>>>
>>>>> - patch 10:
>>>>>    - add ATS steps in riscv_iommu_validate_device_ctx()
>>>>>    - check for 's->enable_ats' before adding 
>>>>> RISCV_IOMMU_DC_TC_EN_ATS in device context
>>>>>    - check for 's->enable_ats' before processing ATS commands in 
>>>>> riscv_iommu_process_cq_tail()
>>>>>    - remove ambiguous trace_riscv_iommu_ats() from 
>>>>> riscv_iommu_translate()
>>>>>
>>>>> - patch 11:
>>>>>    - removed unused bits
>>>>>    - added RISCV_IOMMU_TR_REQ_CTL_NW and RISCV_IOMMU_TR_RESPONSE_S
>>>>>      bits
>>>>>    - set IOMMUTLBEntry 'perm' using RISCV_IOMMU_TR_REQ_CTL_NW in 
>>>>> riscv_iommu_process_dbg()
>>>>>    - clear RISCV_IOMMU_TR_RESPONSE_S in riscv_iommu_process_dbg(). 
>>>>> Added a comment talking about the (lack of) superpage support
>>>>> [1] 
>>>>> https://lore.kernel.org/qemu-riscv/20240307160319.675044-1-dbarboza@ventanamicro.com/
>>>>> [2] https://github.com/vlsunil/qemu/commits/acpi_rimt_poc_v1/
>>>>>
>>>>> Andrew Jones (1):
>>>>>    hw/riscv/riscv-iommu: Add another irq for mrif notifications
>>>>>
>>>>> Daniel Henrique Barboza (3):
>>>>>    pci-ids.rst: add Red Hat pci-id for RISC-V IOMMU device
>>>>>    test/qtest: add riscv-iommu-pci tests
>>>>>    qtest/riscv-iommu-test: add init queues test
>>>>>
>>>>> Tomasz Jeznach (9):
>>>>>    exec/memtxattr: add process identifier to the transaction 
>>>>> attributes
>>>>>    hw/riscv: add riscv-iommu-bits.h
>>>>>    hw/riscv: add RISC-V IOMMU base emulation
>>>>>    hw/riscv: add riscv-iommu-pci reference device
>>>>>    hw/riscv/virt.c: support for RISC-V IOMMU PCIDevice hotplug
>>>>>    hw/riscv/riscv-iommu: add Address Translation Cache (IOATC)
>>>>>    hw/riscv/riscv-iommu: add s-stage and g-stage support
>>>>>    hw/riscv/riscv-iommu: add ATS support
>>>>>    hw/riscv/riscv-iommu: add DBG support
>>>>>
>>>>>   docs/specs/pci-ids.rst           |    2 +
>>>>>   hw/riscv/Kconfig                 |    4 +
>>>>>   hw/riscv/meson.build             |    1 +
>>>>>   hw/riscv/riscv-iommu-bits.h      |  416 ++++++
>>>>>   hw/riscv/riscv-iommu-pci.c       |  177 +++
>>>>>   hw/riscv/riscv-iommu.c           | 2283 
>>>>> ++++++++++++++++++++++++++++++
>>>>>   hw/riscv/riscv-iommu.h           |  146 ++
>>>>>   hw/riscv/trace-events            |   15 +
>>>>>   hw/riscv/trace.h                 |    1 +
>>>>>   hw/riscv/virt.c                  |   33 +-
>>>>>   include/exec/memattrs.h          |    5 +
>>>>>   include/hw/pci/pci.h             |    1 +
>>>>>   include/hw/riscv/iommu.h         |   36 +
>>>>>   meson.build                      |    1 +
>>>>>   tests/qtest/libqos/meson.build   |    4 +
>>>>>   tests/qtest/libqos/riscv-iommu.c |   76 +
>>>>>   tests/qtest/libqos/riscv-iommu.h |  100 ++
>>>>>   tests/qtest/meson.build          |    1 +
>>>>>   tests/qtest/riscv-iommu-test.c   |  234 +++
>>>>>   19 files changed, 3535 insertions(+), 1 deletion(-)
>>>>>   create mode 100644 hw/riscv/riscv-iommu-bits.h
>>>>>   create mode 100644 hw/riscv/riscv-iommu-pci.c
>>>>>   create mode 100644 hw/riscv/riscv-iommu.c
>>>>>   create mode 100644 hw/riscv/riscv-iommu.h
>>>>>   create mode 100644 hw/riscv/trace-events
>>>>>   create mode 100644 hw/riscv/trace.h
>>>>>   create mode 100644 include/hw/riscv/iommu.h
>>>>>   create mode 100644 tests/qtest/libqos/riscv-iommu.c
>>>>>   create mode 100644 tests/qtest/libqos/riscv-iommu.h
>>>>>   create mode 100644 tests/qtest/riscv-iommu-test.c
>>>>>