| Message ID | 20220412021009.582424-1-atishp@rivosinc.com |
|---|---|
| Headers | show |
| Series | Introduce a new Qemu machine for RISC-V | expand |
On Mon, Apr 11, 2022 at 07:10:06PM -0700, Atish Patra wrote: > > The RISC-V virt machine has helped RISC-V software eco system to evolve at a > rapid pace even in absense of the real hardware. It is definitely commendable. > However, the number of devices & commandline options keeps growing as a result > of that as well. That adds flexibility but will also become bit difficult > to manage in the future as more extension support will be added. As it is the > most commonly used qemu machine, it needs to support all kinds of device and > interrupts as well. Moreover, virt machine has limitations on the maximum > number of harts it can support because of all the MMIO devices it has to support. > > The RISC-V IMSIC specification allows to develop machines completely relying > on MSI and don't care about the wired interrupts at all. It just requires > all the devices to be present behind a PCI bus or present themselves as platform > MSI device. The former is a more common scenario in x86 world where most > of the devices are behind PCI bus. As there is very limited MMIO device > support, it can also scale to very large number of harts. > > That's why, this patch series introduces a minimalistic yet very extensible > forward looking machine called as "RISC-V Mini Computer" or "minic". The > idea is to build PC or server like systems with this machine. The machine can > work with or without virtio framework. The current implementation only > supports RV64. I am not sure if building a RV32 machine would be of interest > for such machines. The only mmio device it requires is clint to emulate > the mtimecmp. I would ask what you see as the long term future usage for 'virt' vs 'minic' machine types ? Would you expect all existing users of 'virt' to ultimately switch to 'minic', or are there distinct non-overlapping use cases for 'virt' vs 'minic' such that both end up widely used ? Is 'minic' intended to be able to mimic real physical hardware at all, or is it still intended as a purely virtual machine, like a 'virt-ng' ? Essentially 'virt' was positioned as the standard machine to use if you want to run a virtual machine, without any particular match to physical hardware. It feels like 'minic' is creating a second machine type to fill the same purpose, so how do users decide which to use ? > "Naming is hard". I am not too attached with the name "minic". > I just chose least bad one out of the few on my mind :). I am definitely > open to any other name as well. > > The other alternative to provide MSI only option to aia in the > existing virt machine to build MSI only machines. This is certainly doable > and here is the patch that supports that kind of setup. > > https://github.com/atishp04/qemu/tree/virt_imsic_only > > However, it even complicates the virt machine even further with additional > command line option, branches in the code. I believe virt machine will become > very complex if we continue this path. I am interested to learn what everyone > else think. > > It is needless to say that the current version of minic machine > is inspired from virt machine and tries to reuse as much as code possible. > The first patch in this series adds MSI support for serial-pci device so > console can work on such a machine. The 2nd patch moves some common functions > between minic and the virt machine to a helper file. The PATCH3 actually > implements the new minic machine. > > I have not added the fw-cfg/flash support. We probably should add those > but I just wanted to start small and get the feedback first. > This is a work in progress and have few more TODO items before becoming the > new world order :) > > 1. OpenSBI doesn't have PCI support. Thus, no console support for OpenSBI > for now. > 2. The ns16550 driver in OpenSBI also need to support MSI/MSI-X. > 3. Add MSI-X support for serial-pci device. > > This series can boot Linux distros with the minic machine with or without virtio > devices with out-of-tree Linux kernel patches[1]. Here is an example commandline > > Without virtio devices (nvme, serial-pci & e1000e): > ===================================================== > /scratch/workspace/qemu/build/qemu-system-riscv64 -cpu rv64 -M minic -m 1G -smp 4 -nographic -nodefaults \ > -display none -bios /scratch/workspace/opensbi/build/platform/generic/firmware/fw_dynamic.elf \ > -kernel /scratch/workspace/linux/arch/riscv/boot/Image \ > -chardev stdio,mux=on,signal=off,id=charconsole0 \ > -mon chardev=charconsole0,mode=readline \ > -device pci-serial,msi=true,chardev=charconsole0 \ > -drive id=disk3,file=/scratch/workspace/rootfs_images//fedora/Fedora-Developer-Rawhide-20211110.n.0-sda.raw,format=raw,if=none,id=drive-system-disk,cache=none,format=raw \ > -device nvme,serial=deadbeef,drive=disk3 \ > -netdev user,id=usernet,hostfwd=tcp::10000-:22 -device e1000e,netdev=usernet,bus=pcie.0 \ > -append 'root=/dev/nvme0n1p2 rw loglevel=8 memblock=debug console=ttyS0 earlycon' -d in_asm -D log.txt -s > > With virtio devices (virtio-scsi-pci, serial-pci & virtio-net-pci) > ================================================================== > /scratch/workspace/qemu/build/qemu-system-riscv64 -cpu rv64 -M minic -m 1G -smp 4 -nographic -nodefaults \ > -display none -bios /scratch/workspace/opensbi/build/platform/generic/firmware/fw_dynamic.elf \ > -kernel /scratch/workspace/linux/arch/riscv/boot/Image \ > -chardev stdio,mux=on,signal=off,id=charconsole0 \ > -mon chardev=charconsole0,mode=readline \ > -device pci-serial,msi=true,chardev=charconsole0 \ > -drive file=/scratch/workspace/rootfs_images//fedora/Fedora-Developer-Rawhide-20211110.n.0-sda.raw,format=raw,if=none,id=drive-system-disk,cache=none \ > -device virtio-scsi-pci,id=scsi0 -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=1 \ > -netdev user,id=n1,hostfwd=tcp::10000-:22 -device virtio-net-pci,netdev=n1 \ > -append 'root=/dev/sda2 rw loglevel=8 memblock=debug console=ttyS0 earlycon' > > The objective of this series is to engage the community to solve this problem. > Please suggest if you have another alternatve solution. > > [1] https://github.com/atishp04/linux/tree/msi_only_console > > Atish Patra (3): > serial: Enable MSI capablity and option > hw/riscv: virt: Move common functions to a separate helper file > hw/riscv: Create a new qemu machine for RISC-V > > configs/devices/riscv64-softmmu/default.mak | 1 + > hw/char/serial-pci.c | 36 +- > hw/riscv/Kconfig | 11 + > hw/riscv/machine_helper.c | 417 +++++++++++++++++++ > hw/riscv/meson.build | 2 + > hw/riscv/minic.c | 438 ++++++++++++++++++++ > hw/riscv/virt.c | 403 ++---------------- > include/hw/riscv/machine_helper.h | 87 ++++ > include/hw/riscv/minic.h | 65 +++ > include/hw/riscv/virt.h | 13 - > 10 files changed, 1090 insertions(+), 383 deletions(-) > create mode 100644 hw/riscv/machine_helper.c > create mode 100644 hw/riscv/minic.c > create mode 100644 include/hw/riscv/machine_helper.h > create mode 100644 include/hw/riscv/minic.h > > -- > 2.25.1 > > With regards, Daniel
On Tue, Apr 19, 2022 at 9:51 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Mon, Apr 11, 2022 at 07:10:06PM -0700, Atish Patra wrote: > > > > The RISC-V virt machine has helped RISC-V software eco system to evolve at a > > rapid pace even in absense of the real hardware. It is definitely commendable. > > However, the number of devices & commandline options keeps growing as a result > > of that as well. That adds flexibility but will also become bit difficult > > to manage in the future as more extension support will be added. As it is the > > most commonly used qemu machine, it needs to support all kinds of device and > > interrupts as well. Moreover, virt machine has limitations on the maximum > > number of harts it can support because of all the MMIO devices it has to support. > > > > The RISC-V IMSIC specification allows to develop machines completely relying > > on MSI and don't care about the wired interrupts at all. It just requires > > all the devices to be present behind a PCI bus or present themselves as platform > > MSI device. The former is a more common scenario in x86 world where most > > of the devices are behind PCI bus. As there is very limited MMIO device > > support, it can also scale to very large number of harts. > > > > That's why, this patch series introduces a minimalistic yet very extensible > > forward looking machine called as "RISC-V Mini Computer" or "minic". The > > idea is to build PC or server like systems with this machine. The machine can > > work with or without virtio framework. The current implementation only > > supports RV64. I am not sure if building a RV32 machine would be of interest > > for such machines. The only mmio device it requires is clint to emulate > > the mtimecmp. > > I would ask what you see as the long term future usage for 'virt' vs > 'minic' machine types ? Would you expect all existing users of 'virt' > to ultimately switch to 'minic', or are there distinct non-overlapping > use cases for 'virt' vs 'minic' such that both end up widely used ? > Nope. I don't expect existing 'virt' users to switch to 'minic' as they aim to cater to different users. Here are the major differences 1. virt machine supports MMIO devices & wired interrupts. Minic doesn't 2. virt machine doesn't support the MSI only option yet (can be added though[1]). Minic does. 3. Number of cpu supported by virt machine are limited because of the MMIO devices. Minic can scale to very large numbers of cpu. 4. 'Minic' only supports PCI based MSI capable devices. Thus, MSI is a mandatory requirement for 'minic' while it is optional for 'virt'. 'Minic' aims towards the users who want to create virtual machines that are MSI based and don't care about a million options that virt machines provide. Virt machine is more complex so that it can be flexible in terms of what it supports. Minic is a minimalistic machine which doesn't need to be expanded a lot in the future given that most of the devices can be behind PCI. [1] https://github.com/atishp04/qemu/tree/virt_imsic_only > Is 'minic' intended to be able to mimic real physical hardware at all, > or is it still intended as a purely virtual machine, like a 'virt-ng' ? > Any future hardware that relies only on PCI-MSI based devices, they can be created on top of minic. At that point, minic will provide a useful abstract for all those machines as well. minic doesn't need a virtio framework. Thus, it can closely emulate such hardware as well. > Essentially 'virt' was positioned as the standard machine to use if > you want to run a virtual machine, without any particular match to > physical hardware. It feels like 'minic' is creating a second machine > type to fill the same purpose, so how do users decide which to use ? > I envision 'minic' to be a standard machine for a specific set of user requirements (x86 style PCI based machines). Virt machine will continue to be a standard machine for more generic use cases with MMIO devices. > > "Naming is hard". I am not too attached with the name "minic". > > I just chose least bad one out of the few on my mind :). I am definitely > > open to any other name as well. > > > > The other alternative to provide MSI only option to aia in the > > existing virt machine to build MSI only machines. This is certainly doable > > and here is the patch that supports that kind of setup. > > > > https://github.com/atishp04/qemu/tree/virt_imsic_only > > > > However, it even complicates the virt machine even further with additional > > command line option, branches in the code. I believe virt machine will become > > very complex if we continue this path. I am interested to learn what everyone > > else think. > > > > It is needless to say that the current version of minic machine > > is inspired from virt machine and tries to reuse as much as code possible. > > The first patch in this series adds MSI support for serial-pci device so > > console can work on such a machine. The 2nd patch moves some common functions > > between minic and the virt machine to a helper file. The PATCH3 actually > > implements the new minic machine. > > > > I have not added the fw-cfg/flash support. We probably should add those > > but I just wanted to start small and get the feedback first. > > This is a work in progress and have few more TODO items before becoming the > > new world order :) > > > > 1. OpenSBI doesn't have PCI support. Thus, no console support for OpenSBI > > for now. > > 2. The ns16550 driver in OpenSBI also need to support MSI/MSI-X. > > 3. Add MSI-X support for serial-pci device. > > > > This series can boot Linux distros with the minic machine with or without virtio > > devices with out-of-tree Linux kernel patches[1]. Here is an example commandline > > > > Without virtio devices (nvme, serial-pci & e1000e): > > ===================================================== > > /scratch/workspace/qemu/build/qemu-system-riscv64 -cpu rv64 -M minic -m 1G -smp 4 -nographic -nodefaults \ > > -display none -bios /scratch/workspace/opensbi/build/platform/generic/firmware/fw_dynamic.elf \ > > -kernel /scratch/workspace/linux/arch/riscv/boot/Image \ > > -chardev stdio,mux=on,signal=off,id=charconsole0 \ > > -mon chardev=charconsole0,mode=readline \ > > -device pci-serial,msi=true,chardev=charconsole0 \ > > -drive id=disk3,file=/scratch/workspace/rootfs_images//fedora/Fedora-Developer-Rawhide-20211110.n.0-sda.raw,format=raw,if=none,id=drive-system-disk,cache=none,format=raw \ > > -device nvme,serial=deadbeef,drive=disk3 \ > > -netdev user,id=usernet,hostfwd=tcp::10000-:22 -device e1000e,netdev=usernet,bus=pcie.0 \ > > -append 'root=/dev/nvme0n1p2 rw loglevel=8 memblock=debug console=ttyS0 earlycon' -d in_asm -D log.txt -s > > > > With virtio devices (virtio-scsi-pci, serial-pci & virtio-net-pci) > > ================================================================== > > /scratch/workspace/qemu/build/qemu-system-riscv64 -cpu rv64 -M minic -m 1G -smp 4 -nographic -nodefaults \ > > -display none -bios /scratch/workspace/opensbi/build/platform/generic/firmware/fw_dynamic.elf \ > > -kernel /scratch/workspace/linux/arch/riscv/boot/Image \ > > -chardev stdio,mux=on,signal=off,id=charconsole0 \ > > -mon chardev=charconsole0,mode=readline \ > > -device pci-serial,msi=true,chardev=charconsole0 \ > > -drive file=/scratch/workspace/rootfs_images//fedora/Fedora-Developer-Rawhide-20211110.n.0-sda.raw,format=raw,if=none,id=drive-system-disk,cache=none \ > > -device virtio-scsi-pci,id=scsi0 -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=1 \ > > -netdev user,id=n1,hostfwd=tcp::10000-:22 -device virtio-net-pci,netdev=n1 \ > > -append 'root=/dev/sda2 rw loglevel=8 memblock=debug console=ttyS0 earlycon' > > > > The objective of this series is to engage the community to solve this problem. > > Please suggest if you have another alternatve solution. > > > > [1] https://github.com/atishp04/linux/tree/msi_only_console > > > > Atish Patra (3): > > serial: Enable MSI capablity and option > > hw/riscv: virt: Move common functions to a separate helper file > > hw/riscv: Create a new qemu machine for RISC-V > > > > configs/devices/riscv64-softmmu/default.mak | 1 + > > hw/char/serial-pci.c | 36 +- > > hw/riscv/Kconfig | 11 + > > hw/riscv/machine_helper.c | 417 +++++++++++++++++++ > > hw/riscv/meson.build | 2 + > > hw/riscv/minic.c | 438 ++++++++++++++++++++ > > hw/riscv/virt.c | 403 ++---------------- > > include/hw/riscv/machine_helper.h | 87 ++++ > > include/hw/riscv/minic.h | 65 +++ > > include/hw/riscv/virt.h | 13 - > > 10 files changed, 1090 insertions(+), 383 deletions(-) > > create mode 100644 hw/riscv/machine_helper.c > > create mode 100644 hw/riscv/minic.c > > create mode 100644 include/hw/riscv/machine_helper.h > > create mode 100644 include/hw/riscv/minic.h > > > > -- > > 2.25.1 > > > > > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > >
On Tue, Apr 19, 2022 at 5:26 PM Atish Patra <atishp@atishpatra.org> wrote: > > On Tue, Apr 19, 2022 at 9:51 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > On Mon, Apr 11, 2022 at 07:10:06PM -0700, Atish Patra wrote: > > > > > > The RISC-V virt machine has helped RISC-V software eco system to evolve at a > > > rapid pace even in absense of the real hardware. It is definitely commendable. > > > However, the number of devices & commandline options keeps growing as a result > > > of that as well. That adds flexibility but will also become bit difficult > > > to manage in the future as more extension support will be added. As it is the > > > most commonly used qemu machine, it needs to support all kinds of device and > > > interrupts as well. Moreover, virt machine has limitations on the maximum > > > number of harts it can support because of all the MMIO devices it has to support. > > > > > > The RISC-V IMSIC specification allows to develop machines completely relying > > > on MSI and don't care about the wired interrupts at all. It just requires > > > all the devices to be present behind a PCI bus or present themselves as platform > > > MSI device. The former is a more common scenario in x86 world where most > > > of the devices are behind PCI bus. As there is very limited MMIO device > > > support, it can also scale to very large number of harts. > > > > > > That's why, this patch series introduces a minimalistic yet very extensible > > > forward looking machine called as "RISC-V Mini Computer" or "minic". The > > > idea is to build PC or server like systems with this machine. The machine can > > > work with or without virtio framework. The current implementation only > > > supports RV64. I am not sure if building a RV32 machine would be of interest > > > for such machines. The only mmio device it requires is clint to emulate > > > the mtimecmp. > > Any other thoughts ? > > I would ask what you see as the long term future usage for 'virt' vs > > 'minic' machine types ? Would you expect all existing users of 'virt' > > to ultimately switch to 'minic', or are there distinct non-overlapping > > use cases for 'virt' vs 'minic' such that both end up widely used ? > > > > Nope. I don't expect existing 'virt' users to switch to 'minic' as > they aim to cater to different users. > > Here are the major differences > 1. virt machine supports MMIO devices & wired interrupts. Minic doesn't > 2. virt machine doesn't support the MSI only option yet (can be added > though[1]). Minic does. > 3. Number of cpu supported by virt machine are limited because of the > MMIO devices. Minic can scale to very > large numbers of cpu. > 4. 'Minic' only supports PCI based MSI capable devices. Thus, MSI is a > mandatory requirement for 'minic' while > it is optional for 'virt'. > > 'Minic' aims towards the users who want to create virtual machines > that are MSI based and don't care about > a million options that virt machines provide. Virt machine is more > complex so that it can be flexible in terms of > what it supports. Minic is a minimalistic machine which doesn't need > to be expanded a lot in the future given that > most of the devices can be behind PCI. > > [1] https://github.com/atishp04/qemu/tree/virt_imsic_only > > > Is 'minic' intended to be able to mimic real physical hardware at all, > > or is it still intended as a purely virtual machine, like a 'virt-ng' ? > > > > Any future hardware that relies only on PCI-MSI based devices, they > can be created on top of minic. > At that point, minic will provide a useful abstract for all those > machines as well. minic doesn't need a virtio framework. > Thus, it can closely emulate such hardware as well. > > > Essentially 'virt' was positioned as the standard machine to use if > > you want to run a virtual machine, without any particular match to > > physical hardware. It feels like 'minic' is creating a second machine > > type to fill the same purpose, so how do users decide which to use ? > > > > I envision 'minic' to be a standard machine for a specific set of user > requirements (x86 style PCI based > machines). Virt machine will continue to be a standard machine for > more generic use cases with MMIO devices. > > > > "Naming is hard". I am not too attached with the name "minic". > > > I just chose least bad one out of the few on my mind :). I am definitely > > > open to any other name as well. > > > > > > The other alternative to provide MSI only option to aia in the > > > existing virt machine to build MSI only machines. This is certainly doable > > > and here is the patch that supports that kind of setup. > > > > > > https://github.com/atishp04/qemu/tree/virt_imsic_only > > > > > > However, it even complicates the virt machine even further with additional > > > command line option, branches in the code. I believe virt machine will become > > > very complex if we continue this path. I am interested to learn what everyone > > > else think. > > > > > > It is needless to say that the current version of minic machine > > > is inspired from virt machine and tries to reuse as much as code possible. > > > The first patch in this series adds MSI support for serial-pci device so > > > console can work on such a machine. The 2nd patch moves some common functions > > > between minic and the virt machine to a helper file. The PATCH3 actually > > > implements the new minic machine. > > > > > > I have not added the fw-cfg/flash support. We probably should add those > > > but I just wanted to start small and get the feedback first. > > > This is a work in progress and have few more TODO items before becoming the > > > new world order :) > > > > > > 1. OpenSBI doesn't have PCI support. Thus, no console support for OpenSBI > > > for now. > > > 2. The ns16550 driver in OpenSBI also need to support MSI/MSI-X. > > > 3. Add MSI-X support for serial-pci device. > > > > > > This series can boot Linux distros with the minic machine with or without virtio > > > devices with out-of-tree Linux kernel patches[1]. Here is an example commandline > > > > > > Without virtio devices (nvme, serial-pci & e1000e): > > > ===================================================== > > > /scratch/workspace/qemu/build/qemu-system-riscv64 -cpu rv64 -M minic -m 1G -smp 4 -nographic -nodefaults \ > > > -display none -bios /scratch/workspace/opensbi/build/platform/generic/firmware/fw_dynamic.elf \ > > > -kernel /scratch/workspace/linux/arch/riscv/boot/Image \ > > > -chardev stdio,mux=on,signal=off,id=charconsole0 \ > > > -mon chardev=charconsole0,mode=readline \ > > > -device pci-serial,msi=true,chardev=charconsole0 \ > > > -drive id=disk3,file=/scratch/workspace/rootfs_images//fedora/Fedora-Developer-Rawhide-20211110.n.0-sda.raw,format=raw,if=none,id=drive-system-disk,cache=none,format=raw \ > > > -device nvme,serial=deadbeef,drive=disk3 \ > > > -netdev user,id=usernet,hostfwd=tcp::10000-:22 -device e1000e,netdev=usernet,bus=pcie.0 \ > > > -append 'root=/dev/nvme0n1p2 rw loglevel=8 memblock=debug console=ttyS0 earlycon' -d in_asm -D log.txt -s > > > > > > With virtio devices (virtio-scsi-pci, serial-pci & virtio-net-pci) > > > ================================================================== > > > /scratch/workspace/qemu/build/qemu-system-riscv64 -cpu rv64 -M minic -m 1G -smp 4 -nographic -nodefaults \ > > > -display none -bios /scratch/workspace/opensbi/build/platform/generic/firmware/fw_dynamic.elf \ > > > -kernel /scratch/workspace/linux/arch/riscv/boot/Image \ > > > -chardev stdio,mux=on,signal=off,id=charconsole0 \ > > > -mon chardev=charconsole0,mode=readline \ > > > -device pci-serial,msi=true,chardev=charconsole0 \ > > > -drive file=/scratch/workspace/rootfs_images//fedora/Fedora-Developer-Rawhide-20211110.n.0-sda.raw,format=raw,if=none,id=drive-system-disk,cache=none \ > > > -device virtio-scsi-pci,id=scsi0 -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=1 \ > > > -netdev user,id=n1,hostfwd=tcp::10000-:22 -device virtio-net-pci,netdev=n1 \ > > > -append 'root=/dev/sda2 rw loglevel=8 memblock=debug console=ttyS0 earlycon' > > > > > > The objective of this series is to engage the community to solve this problem. > > > Please suggest if you have another alternatve solution. > > > > > > [1] https://github.com/atishp04/linux/tree/msi_only_console > > > > > > Atish Patra (3): > > > serial: Enable MSI capablity and option > > > hw/riscv: virt: Move common functions to a separate helper file > > > hw/riscv: Create a new qemu machine for RISC-V > > > > > > configs/devices/riscv64-softmmu/default.mak | 1 + > > > hw/char/serial-pci.c | 36 +- > > > hw/riscv/Kconfig | 11 + > > > hw/riscv/machine_helper.c | 417 +++++++++++++++++++ > > > hw/riscv/meson.build | 2 + > > > hw/riscv/minic.c | 438 ++++++++++++++++++++ > > > hw/riscv/virt.c | 403 ++---------------- > > > include/hw/riscv/machine_helper.h | 87 ++++ > > > include/hw/riscv/minic.h | 65 +++ > > > include/hw/riscv/virt.h | 13 - > > > 10 files changed, 1090 insertions(+), 383 deletions(-) > > > create mode 100644 hw/riscv/machine_helper.c > > > create mode 100644 hw/riscv/minic.c > > > create mode 100644 include/hw/riscv/machine_helper.h > > > create mode 100644 include/hw/riscv/minic.h > > > > > > -- > > > 2.25.1 > > > > > > > > > > With regards, > > Daniel > > -- > > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > > |: https://libvirt.org -o- https://fstop138.berrange.com :| > > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > > > > > > > -- > Regards, > Atish
On Tue, May 3, 2022 at 5:57 PM Atish Patra <atishp@atishpatra.org> wrote: > > On Tue, Apr 19, 2022 at 5:26 PM Atish Patra <atishp@atishpatra.org> wrote: > > > > On Tue, Apr 19, 2022 at 9:51 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > On Mon, Apr 11, 2022 at 07:10:06PM -0700, Atish Patra wrote: > > > > > > > > The RISC-V virt machine has helped RISC-V software eco system to evolve at a > > > > rapid pace even in absense of the real hardware. It is definitely commendable. > > > > However, the number of devices & commandline options keeps growing as a result > > > > of that as well. That adds flexibility but will also become bit difficult > > > > to manage in the future as more extension support will be added. As it is the > > > > most commonly used qemu machine, it needs to support all kinds of device and > > > > interrupts as well. Moreover, virt machine has limitations on the maximum > > > > number of harts it can support because of all the MMIO devices it has to support. > > > > > > > > The RISC-V IMSIC specification allows to develop machines completely relying > > > > on MSI and don't care about the wired interrupts at all. It just requires > > > > all the devices to be present behind a PCI bus or present themselves as platform > > > > MSI device. The former is a more common scenario in x86 world where most > > > > of the devices are behind PCI bus. As there is very limited MMIO device > > > > support, it can also scale to very large number of harts. > > > > > > > > That's why, this patch series introduces a minimalistic yet very extensible > > > > forward looking machine called as "RISC-V Mini Computer" or "minic". The > > > > idea is to build PC or server like systems with this machine. The machine can > > > > work with or without virtio framework. The current implementation only > > > > supports RV64. I am not sure if building a RV32 machine would be of interest > > > > for such machines. The only mmio device it requires is clint to emulate > > > > the mtimecmp. > > > > > Any other thoughts ? I don't *love* this idea. I think the virt machine is useful, but I'm not convinced we need a second one. This feels a little bit more like a "none" machine, as it contains just the bare minimum to work. > > > > I would ask what you see as the long term future usage for 'virt' vs > > > 'minic' machine types ? Would you expect all existing users of 'virt' > > > to ultimately switch to 'minic', or are there distinct non-overlapping > > > use cases for 'virt' vs 'minic' such that both end up widely used ? > > > > > > > Nope. I don't expect existing 'virt' users to switch to 'minic' as > > they aim to cater to different users. > > > > Here are the major differences > > 1. virt machine supports MMIO devices & wired interrupts. Minic doesn't This seems like the main difference > > 2. virt machine doesn't support the MSI only option yet (can be added > > though[1]). Minic does. This could be fixed > > 3. Number of cpu supported by virt machine are limited because of the > > MMIO devices. Minic can scale to very > > large numbers of cpu. Similar to 1 > > 4. 'Minic' only supports PCI based MSI capable devices. Thus, MSI is a > > mandatory requirement for 'minic' while > > it is optional for 'virt'. I'm not fully convinced we need this, but it also doesn't seem to cost us a lot in terms of maintenance. It would be beneficial if we could share a bit more of the code. Can we share the socket creation code as well? I don't like the name minic though. What about something like `virt-hpc`, `virt-pcie-minimal` or something like that? That way we indicate it's still a virt board Alistair > > > > 'Minic' aims towards the users who want to create virtual machines > > that are MSI based and don't care about > > a million options that virt machines provide. Virt machine is more > > complex so that it can be flexible in terms of > > what it supports. Minic is a minimalistic machine which doesn't need > > to be expanded a lot in the future given that > > most of the devices can be behind PCI. > > > > [1] https://github.com/atishp04/qemu/tree/virt_imsic_only > > > > > Is 'minic' intended to be able to mimic real physical hardware at all, > > > or is it still intended as a purely virtual machine, like a 'virt-ng' ? > > > > > > > Any future hardware that relies only on PCI-MSI based devices, they > > can be created on top of minic. > > At that point, minic will provide a useful abstract for all those > > machines as well. minic doesn't need a virtio framework. > > Thus, it can closely emulate such hardware as well. > > > > > Essentially 'virt' was positioned as the standard machine to use if > > > you want to run a virtual machine, without any particular match to > > > physical hardware. It feels like 'minic' is creating a second machine > > > type to fill the same purpose, so how do users decide which to use ? > > > > > > > I envision 'minic' to be a standard machine for a specific set of user > > requirements (x86 style PCI based > > machines). Virt machine will continue to be a standard machine for > > more generic use cases with MMIO devices. > > > > > > "Naming is hard". I am not too attached with the name "minic". > > > > I just chose least bad one out of the few on my mind :). I am definitely > > > > open to any other name as well. > > > > > > > > The other alternative to provide MSI only option to aia in the > > > > existing virt machine to build MSI only machines. This is certainly doable > > > > and here is the patch that supports that kind of setup. > > > > > > > > https://github.com/atishp04/qemu/tree/virt_imsic_only > > > > > > > > However, it even complicates the virt machine even further with additional > > > > command line option, branches in the code. I believe virt machine will become > > > > very complex if we continue this path. I am interested to learn what everyone > > > > else think. > > > > > > > > It is needless to say that the current version of minic machine > > > > is inspired from virt machine and tries to reuse as much as code possible. > > > > The first patch in this series adds MSI support for serial-pci device so > > > > console can work on such a machine. The 2nd patch moves some common functions > > > > between minic and the virt machine to a helper file. The PATCH3 actually > > > > implements the new minic machine. > > > > > > > > I have not added the fw-cfg/flash support. We probably should add those > > > > but I just wanted to start small and get the feedback first. > > > > This is a work in progress and have few more TODO items before becoming the > > > > new world order :) > > > > > > > > 1. OpenSBI doesn't have PCI support. Thus, no console support for OpenSBI > > > > for now. > > > > 2. The ns16550 driver in OpenSBI also need to support MSI/MSI-X. > > > > 3. Add MSI-X support for serial-pci device. > > > > > > > > This series can boot Linux distros with the minic machine with or without virtio > > > > devices with out-of-tree Linux kernel patches[1]. Here is an example commandline > > > > > > > > Without virtio devices (nvme, serial-pci & e1000e): > > > > ===================================================== > > > > /scratch/workspace/qemu/build/qemu-system-riscv64 -cpu rv64 -M minic -m 1G -smp 4 -nographic -nodefaults \ > > > > -display none -bios /scratch/workspace/opensbi/build/platform/generic/firmware/fw_dynamic.elf \ > > > > -kernel /scratch/workspace/linux/arch/riscv/boot/Image \ > > > > -chardev stdio,mux=on,signal=off,id=charconsole0 \ > > > > -mon chardev=charconsole0,mode=readline \ > > > > -device pci-serial,msi=true,chardev=charconsole0 \ > > > > -drive id=disk3,file=/scratch/workspace/rootfs_images//fedora/Fedora-Developer-Rawhide-20211110.n.0-sda.raw,format=raw,if=none,id=drive-system-disk,cache=none,format=raw \ > > > > -device nvme,serial=deadbeef,drive=disk3 \ > > > > -netdev user,id=usernet,hostfwd=tcp::10000-:22 -device e1000e,netdev=usernet,bus=pcie.0 \ > > > > -append 'root=/dev/nvme0n1p2 rw loglevel=8 memblock=debug console=ttyS0 earlycon' -d in_asm -D log.txt -s > > > > > > > > With virtio devices (virtio-scsi-pci, serial-pci & virtio-net-pci) > > > > ================================================================== > > > > /scratch/workspace/qemu/build/qemu-system-riscv64 -cpu rv64 -M minic -m 1G -smp 4 -nographic -nodefaults \ > > > > -display none -bios /scratch/workspace/opensbi/build/platform/generic/firmware/fw_dynamic.elf \ > > > > -kernel /scratch/workspace/linux/arch/riscv/boot/Image \ > > > > -chardev stdio,mux=on,signal=off,id=charconsole0 \ > > > > -mon chardev=charconsole0,mode=readline \ > > > > -device pci-serial,msi=true,chardev=charconsole0 \ > > > > -drive file=/scratch/workspace/rootfs_images//fedora/Fedora-Developer-Rawhide-20211110.n.0-sda.raw,format=raw,if=none,id=drive-system-disk,cache=none \ > > > > -device virtio-scsi-pci,id=scsi0 -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=1 \ > > > > -netdev user,id=n1,hostfwd=tcp::10000-:22 -device virtio-net-pci,netdev=n1 \ > > > > -append 'root=/dev/sda2 rw loglevel=8 memblock=debug console=ttyS0 earlycon' > > > > > > > > The objective of this series is to engage the community to solve this problem. > > > > Please suggest if you have another alternatve solution. > > > > > > > > [1] https://github.com/atishp04/linux/tree/msi_only_console > > > > > > > > Atish Patra (3): > > > > serial: Enable MSI capablity and option > > > > hw/riscv: virt: Move common functions to a separate helper file > > > > hw/riscv: Create a new qemu machine for RISC-V > > > > > > > > configs/devices/riscv64-softmmu/default.mak | 1 + > > > > hw/char/serial-pci.c | 36 +- > > > > hw/riscv/Kconfig | 11 + > > > > hw/riscv/machine_helper.c | 417 +++++++++++++++++++ > > > > hw/riscv/meson.build | 2 + > > > > hw/riscv/minic.c | 438 ++++++++++++++++++++ > > > > hw/riscv/virt.c | 403 ++---------------- > > > > include/hw/riscv/machine_helper.h | 87 ++++ > > > > include/hw/riscv/minic.h | 65 +++ > > > > include/hw/riscv/virt.h | 13 - > > > > 10 files changed, 1090 insertions(+), 383 deletions(-) > > > > create mode 100644 hw/riscv/machine_helper.c > > > > create mode 100644 hw/riscv/minic.c > > > > create mode 100644 include/hw/riscv/machine_helper.h > > > > create mode 100644 include/hw/riscv/minic.h > > > > > > > > -- > > > > 2.25.1 > > > > > > > > > > > > > > With regards, > > > Daniel > > > -- > > > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > > > |: https://libvirt.org -o- https://fstop138.berrange.com :| > > > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > > > > > > > > > > > > -- > > Regards, > > Atish > > > > -- > Regards, > Atish >
On Thu, May 05, 2022 at 07:36:51PM +1000, Alistair Francis wrote: > On Tue, May 3, 2022 at 5:57 PM Atish Patra <atishp@atishpatra.org> wrote: > > > > On Tue, Apr 19, 2022 at 5:26 PM Atish Patra <atishp@atishpatra.org> wrote: > > > > > > On Tue, Apr 19, 2022 at 9:51 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > On Mon, Apr 11, 2022 at 07:10:06PM -0700, Atish Patra wrote: > > > > > > > > > > The RISC-V virt machine has helped RISC-V software eco system to evolve at a > > > > > rapid pace even in absense of the real hardware. It is definitely commendable. > > > > > However, the number of devices & commandline options keeps growing as a result > > > > > of that as well. That adds flexibility but will also become bit difficult > > > > > to manage in the future as more extension support will be added. As it is the > > > > > most commonly used qemu machine, it needs to support all kinds of device and > > > > > interrupts as well. Moreover, virt machine has limitations on the maximum > > > > > number of harts it can support because of all the MMIO devices it has to support. > > > > > > > > > > The RISC-V IMSIC specification allows to develop machines completely relying > > > > > on MSI and don't care about the wired interrupts at all. It just requires > > > > > all the devices to be present behind a PCI bus or present themselves as platform > > > > > MSI device. The former is a more common scenario in x86 world where most > > > > > of the devices are behind PCI bus. As there is very limited MMIO device > > > > > support, it can also scale to very large number of harts. > > > > > > > > > > That's why, this patch series introduces a minimalistic yet very extensible > > > > > forward looking machine called as "RISC-V Mini Computer" or "minic". The > > > > > idea is to build PC or server like systems with this machine. The machine can > > > > > work with or without virtio framework. The current implementation only > > > > > supports RV64. I am not sure if building a RV32 machine would be of interest > > > > > for such machines. The only mmio device it requires is clint to emulate > > > > > the mtimecmp. > > > > > > > > Any other thoughts ? > > I don't *love* this idea. I think the virt machine is useful, but I'm > not convinced we need a second one. > > This feels a little bit more like a "none" machine, as it contains > just the bare minimum to work. > > > > > > > I would ask what you see as the long term future usage for 'virt' vs > > > > 'minic' machine types ? Would you expect all existing users of 'virt' > > > > to ultimately switch to 'minic', or are there distinct non-overlapping > > > > use cases for 'virt' vs 'minic' such that both end up widely used ? > > > > > > > > > > Nope. I don't expect existing 'virt' users to switch to 'minic' as > > > they aim to cater to different users. > > > > > > Here are the major differences > > > 1. virt machine supports MMIO devices & wired interrupts. Minic doesn't > > This seems like the main difference > > > > 2. virt machine doesn't support the MSI only option yet (can be added > > > though[1]). Minic does. > > This could be fixed > > > > 3. Number of cpu supported by virt machine are limited because of the > > > MMIO devices. Minic can scale to very > > > large numbers of cpu. > > Similar to 1 > > > > 4. 'Minic' only supports PCI based MSI capable devices. Thus, MSI is a > > > mandatory requirement for 'minic' while > > > it is optional for 'virt'. > > I'm not fully convinced we need this, but it also doesn't seem to cost > us a lot in terms of maintenance. It would be beneficial if we could > share a bit more of the code. Can we share the socket creation code as > well? > > I don't like the name minic though. What about something like > `virt-hpc`, `virt-pcie-minimal` or something like that? That way we > indicate it's still a virt board We're not versioning the 'virt' machine type right so. IOW, we've not made any promises about its long term featureset. If the virt machine type isn't the perfect match right now, should we change it, in a potentially incompatible way, to give us the right solution long term, rather than introducing a brand new machine type with significant overlap. With regards, Daniel
On Thu, May 5, 2022 at 8:04 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Thu, May 05, 2022 at 07:36:51PM +1000, Alistair Francis wrote: > > On Tue, May 3, 2022 at 5:57 PM Atish Patra <atishp@atishpatra.org> wrote: > > > > > > On Tue, Apr 19, 2022 at 5:26 PM Atish Patra <atishp@atishpatra.org> wrote: > > > > > > > > On Tue, Apr 19, 2022 at 9:51 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > > > On Mon, Apr 11, 2022 at 07:10:06PM -0700, Atish Patra wrote: > > > > > > > > > > > > The RISC-V virt machine has helped RISC-V software eco system to evolve at a > > > > > > rapid pace even in absense of the real hardware. It is definitely commendable. > > > > > > However, the number of devices & commandline options keeps growing as a result > > > > > > of that as well. That adds flexibility but will also become bit difficult > > > > > > to manage in the future as more extension support will be added. As it is the > > > > > > most commonly used qemu machine, it needs to support all kinds of device and > > > > > > interrupts as well. Moreover, virt machine has limitations on the maximum > > > > > > number of harts it can support because of all the MMIO devices it has to support. > > > > > > > > > > > > The RISC-V IMSIC specification allows to develop machines completely relying > > > > > > on MSI and don't care about the wired interrupts at all. It just requires > > > > > > all the devices to be present behind a PCI bus or present themselves as platform > > > > > > MSI device. The former is a more common scenario in x86 world where most > > > > > > of the devices are behind PCI bus. As there is very limited MMIO device > > > > > > support, it can also scale to very large number of harts. > > > > > > > > > > > > That's why, this patch series introduces a minimalistic yet very extensible > > > > > > forward looking machine called as "RISC-V Mini Computer" or "minic". The > > > > > > idea is to build PC or server like systems with this machine. The machine can > > > > > > work with or without virtio framework. The current implementation only > > > > > > supports RV64. I am not sure if building a RV32 machine would be of interest > > > > > > for such machines. The only mmio device it requires is clint to emulate > > > > > > the mtimecmp. > > > > > > > > > > > Any other thoughts ? > > > > I don't *love* this idea. I think the virt machine is useful, but I'm > > not convinced we need a second one. > > > > This feels a little bit more like a "none" machine, as it contains > > just the bare minimum to work. > > > > > > > > > > I would ask what you see as the long term future usage for 'virt' vs > > > > > 'minic' machine types ? Would you expect all existing users of 'virt' > > > > > to ultimately switch to 'minic', or are there distinct non-overlapping > > > > > use cases for 'virt' vs 'minic' such that both end up widely used ? > > > > > > > > > > > > > Nope. I don't expect existing 'virt' users to switch to 'minic' as > > > > they aim to cater to different users. > > > > > > > > Here are the major differences > > > > 1. virt machine supports MMIO devices & wired interrupts. Minic doesn't > > > > This seems like the main difference > > > > > > 2. virt machine doesn't support the MSI only option yet (can be added > > > > though[1]). Minic does. > > > > This could be fixed > > > > > > 3. Number of cpu supported by virt machine are limited because of the > > > > MMIO devices. Minic can scale to very > > > > large numbers of cpu. > > > > Similar to 1 > > > > > > 4. 'Minic' only supports PCI based MSI capable devices. Thus, MSI is a > > > > mandatory requirement for 'minic' while > > > > it is optional for 'virt'. > > > > I'm not fully convinced we need this, but it also doesn't seem to cost > > us a lot in terms of maintenance. It would be beneficial if we could > > share a bit more of the code. Can we share the socket creation code as > > well? > > > > I don't like the name minic though. What about something like > > `virt-hpc`, `virt-pcie-minimal` or something like that? That way we > > indicate it's still a virt board > > We're not versioning the 'virt' machine type right so. IOW, we've not > made any promises about its long term featureset. > > If the virt machine type isn't the perfect match right now, should > we change it, in a potentially incompatible way, to give us the right > solution long term, rather than introducing a brand new machine type > with significant overlap. Even if we didn't worry about backwards compatibility the current virt machine would still be what most users want. It's just a small number of users who don't want MMIO devices and instead want to use PCIe for everything. Realistically it's only HPC users who would want this type of machine, at least that's my understanding. Alistair > > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
On Thu, May 5, 2022 at 2:37 AM Alistair Francis <alistair23@gmail.com> wrote: > > On Tue, May 3, 2022 at 5:57 PM Atish Patra <atishp@atishpatra.org> wrote: > > > > On Tue, Apr 19, 2022 at 5:26 PM Atish Patra <atishp@atishpatra.org> wrote: > > > > > > On Tue, Apr 19, 2022 at 9:51 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > On Mon, Apr 11, 2022 at 07:10:06PM -0700, Atish Patra wrote: > > > > > > > > > > The RISC-V virt machine has helped RISC-V software eco system to evolve at a > > > > > rapid pace even in absense of the real hardware. It is definitely commendable. > > > > > However, the number of devices & commandline options keeps growing as a result > > > > > of that as well. That adds flexibility but will also become bit difficult > > > > > to manage in the future as more extension support will be added. As it is the > > > > > most commonly used qemu machine, it needs to support all kinds of device and > > > > > interrupts as well. Moreover, virt machine has limitations on the maximum > > > > > number of harts it can support because of all the MMIO devices it has to support. > > > > > > > > > > The RISC-V IMSIC specification allows to develop machines completely relying > > > > > on MSI and don't care about the wired interrupts at all. It just requires > > > > > all the devices to be present behind a PCI bus or present themselves as platform > > > > > MSI device. The former is a more common scenario in x86 world where most > > > > > of the devices are behind PCI bus. As there is very limited MMIO device > > > > > support, it can also scale to very large number of harts. > > > > > > > > > > That's why, this patch series introduces a minimalistic yet very extensible > > > > > forward looking machine called as "RISC-V Mini Computer" or "minic". The > > > > > idea is to build PC or server like systems with this machine. The machine can > > > > > work with or without virtio framework. The current implementation only > > > > > supports RV64. I am not sure if building a RV32 machine would be of interest > > > > > for such machines. The only mmio device it requires is clint to emulate > > > > > the mtimecmp. > > > > > > > > Any other thoughts ? > > I don't *love* this idea. I think the virt machine is useful, but I'm > not convinced we need a second one. > > This feels a little bit more like a "none" machine, as it contains > just the bare minimum to work. > Ha ha :). That was the whole point. To create a minimal machine that is easy to maintain and emulate platforms in the real world. > > > > > > I would ask what you see as the long term future usage for 'virt' vs > > > > 'minic' machine types ? Would you expect all existing users of 'virt' > > > > to ultimately switch to 'minic', or are there distinct non-overlapping > > > > use cases for 'virt' vs 'minic' such that both end up widely used ? > > > > > > > > > > Nope. I don't expect existing 'virt' users to switch to 'minic' as > > > they aim to cater to different users. > > > > > > Here are the major differences > > > 1. virt machine supports MMIO devices & wired interrupts. Minic doesn't > > This seems like the main difference > > > > 2. virt machine doesn't support the MSI only option yet (can be added > > > though[1]). Minic does. > > This could be fixed > > > > 3. Number of cpu supported by virt machine are limited because of the > > > MMIO devices. Minic can scale to very > > > large numbers of cpu. > > Similar to 1 > Yes. I already had the patch to fix that in the virt machine in the cover letter. I did not send it to the mailing list to avoid confusion. > > > 4. 'Minic' only supports PCI based MSI capable devices. Thus, MSI is a > > > mandatory requirement for 'minic' while > > > it is optional for 'virt'. > > I'm not fully convinced we need this, but it also doesn't seem to cost > us a lot in terms of maintenance. It would be beneficial if we could > share a bit more of the code. Can we share the socket creation code as > well? > Yeah. We can move the socket creation to the common code as well. There are few others small ones (virt_set/get_aia_guests) can be moved to common code. In the future, real world hpc machines may just build their machine on top of this machine instead of developing from scratch if we allow some configurability (e.g memory map, machine name, max cpus etc.) > I don't like the name minic though. What about something like > `virt-hpc`, `virt-pcie-minimal` or something like that? That way we > indicate it's still a virt board > Fair enough. I can rename it to virt-hpc or something similar. > Alistair > > > > > > > 'Minic' aims towards the users who want to create virtual machines > > > that are MSI based and don't care about > > > a million options that virt machines provide. Virt machine is more > > > complex so that it can be flexible in terms of > > > what it supports. Minic is a minimalistic machine which doesn't need > > > to be expanded a lot in the future given that > > > most of the devices can be behind PCI. > > > > > > [1] https://github.com/atishp04/qemu/tree/virt_imsic_only > > > > > > > Is 'minic' intended to be able to mimic real physical hardware at all, > > > > or is it still intended as a purely virtual machine, like a 'virt-ng' ? > > > > > > > > > > Any future hardware that relies only on PCI-MSI based devices, they > > > can be created on top of minic. > > > At that point, minic will provide a useful abstract for all those > > > machines as well. minic doesn't need a virtio framework. > > > Thus, it can closely emulate such hardware as well. > > > > > > > Essentially 'virt' was positioned as the standard machine to use if > > > > you want to run a virtual machine, without any particular match to > > > > physical hardware. It feels like 'minic' is creating a second machine > > > > type to fill the same purpose, so how do users decide which to use ? > > > > > > > > > > I envision 'minic' to be a standard machine for a specific set of user > > > requirements (x86 style PCI based > > > machines). Virt machine will continue to be a standard machine for > > > more generic use cases with MMIO devices. > > > > > > > > "Naming is hard". I am not too attached with the name "minic". > > > > > I just chose least bad one out of the few on my mind :). I am definitely > > > > > open to any other name as well. > > > > > > > > > > The other alternative to provide MSI only option to aia in the > > > > > existing virt machine to build MSI only machines. This is certainly doable > > > > > and here is the patch that supports that kind of setup. > > > > > > > > > > https://github.com/atishp04/qemu/tree/virt_imsic_only > > > > > > > > > > However, it even complicates the virt machine even further with additional > > > > > command line option, branches in the code. I believe virt machine will become > > > > > very complex if we continue this path. I am interested to learn what everyone > > > > > else think. > > > > > > > > > > It is needless to say that the current version of minic machine > > > > > is inspired from virt machine and tries to reuse as much as code possible. > > > > > The first patch in this series adds MSI support for serial-pci device so > > > > > console can work on such a machine. The 2nd patch moves some common functions > > > > > between minic and the virt machine to a helper file. The PATCH3 actually > > > > > implements the new minic machine. > > > > > > > > > > I have not added the fw-cfg/flash support. We probably should add those > > > > > but I just wanted to start small and get the feedback first. > > > > > This is a work in progress and have few more TODO items before becoming the > > > > > new world order :) > > > > > > > > > > 1. OpenSBI doesn't have PCI support. Thus, no console support for OpenSBI > > > > > for now. > > > > > 2. The ns16550 driver in OpenSBI also need to support MSI/MSI-X. > > > > > 3. Add MSI-X support for serial-pci device. > > > > > > > > > > This series can boot Linux distros with the minic machine with or without virtio > > > > > devices with out-of-tree Linux kernel patches[1]. Here is an example commandline > > > > > > > > > > Without virtio devices (nvme, serial-pci & e1000e): > > > > > ===================================================== > > > > > /scratch/workspace/qemu/build/qemu-system-riscv64 -cpu rv64 -M minic -m 1G -smp 4 -nographic -nodefaults \ > > > > > -display none -bios /scratch/workspace/opensbi/build/platform/generic/firmware/fw_dynamic.elf \ > > > > > -kernel /scratch/workspace/linux/arch/riscv/boot/Image \ > > > > > -chardev stdio,mux=on,signal=off,id=charconsole0 \ > > > > > -mon chardev=charconsole0,mode=readline \ > > > > > -device pci-serial,msi=true,chardev=charconsole0 \ > > > > > -drive id=disk3,file=/scratch/workspace/rootfs_images//fedora/Fedora-Developer-Rawhide-20211110.n.0-sda.raw,format=raw,if=none,id=drive-system-disk,cache=none,format=raw \ > > > > > -device nvme,serial=deadbeef,drive=disk3 \ > > > > > -netdev user,id=usernet,hostfwd=tcp::10000-:22 -device e1000e,netdev=usernet,bus=pcie.0 \ > > > > > -append 'root=/dev/nvme0n1p2 rw loglevel=8 memblock=debug console=ttyS0 earlycon' -d in_asm -D log.txt -s > > > > > > > > > > With virtio devices (virtio-scsi-pci, serial-pci & virtio-net-pci) > > > > > ================================================================== > > > > > /scratch/workspace/qemu/build/qemu-system-riscv64 -cpu rv64 -M minic -m 1G -smp 4 -nographic -nodefaults \ > > > > > -display none -bios /scratch/workspace/opensbi/build/platform/generic/firmware/fw_dynamic.elf \ > > > > > -kernel /scratch/workspace/linux/arch/riscv/boot/Image \ > > > > > -chardev stdio,mux=on,signal=off,id=charconsole0 \ > > > > > -mon chardev=charconsole0,mode=readline \ > > > > > -device pci-serial,msi=true,chardev=charconsole0 \ > > > > > -drive file=/scratch/workspace/rootfs_images//fedora/Fedora-Developer-Rawhide-20211110.n.0-sda.raw,format=raw,if=none,id=drive-system-disk,cache=none \ > > > > > -device virtio-scsi-pci,id=scsi0 -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=1 \ > > > > > -netdev user,id=n1,hostfwd=tcp::10000-:22 -device virtio-net-pci,netdev=n1 \ > > > > > -append 'root=/dev/sda2 rw loglevel=8 memblock=debug console=ttyS0 earlycon' > > > > > > > > > > The objective of this series is to engage the community to solve this problem. > > > > > Please suggest if you have another alternatve solution. > > > > > > > > > > [1] https://github.com/atishp04/linux/tree/msi_only_console > > > > > > > > > > Atish Patra (3): > > > > > serial: Enable MSI capablity and option > > > > > hw/riscv: virt: Move common functions to a separate helper file > > > > > hw/riscv: Create a new qemu machine for RISC-V > > > > > > > > > > configs/devices/riscv64-softmmu/default.mak | 1 + > > > > > hw/char/serial-pci.c | 36 +- > > > > > hw/riscv/Kconfig | 11 + > > > > > hw/riscv/machine_helper.c | 417 +++++++++++++++++++ > > > > > hw/riscv/meson.build | 2 + > > > > > hw/riscv/minic.c | 438 ++++++++++++++++++++ > > > > > hw/riscv/virt.c | 403 ++---------------- > > > > > include/hw/riscv/machine_helper.h | 87 ++++ > > > > > include/hw/riscv/minic.h | 65 +++ > > > > > include/hw/riscv/virt.h | 13 - > > > > > 10 files changed, 1090 insertions(+), 383 deletions(-) > > > > > create mode 100644 hw/riscv/machine_helper.c > > > > > create mode 100644 hw/riscv/minic.c > > > > > create mode 100644 include/hw/riscv/machine_helper.h > > > > > create mode 100644 include/hw/riscv/minic.h > > > > > > > > > > -- > > > > > 2.25.1 > > > > > > > > > > > > > > > > > > With regards, > > > > Daniel > > > > -- > > > > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > > > > |: https://libvirt.org -o- https://fstop138.berrange.com :| > > > > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > > > > > > > > > > > > > > > > > -- > > > Regards, > > > Atish > > > > > > > > -- > > Regards, > > Atish > >
On Thu, May 5, 2022 at 1:35 PM Alistair Francis <alistair23@gmail.com> wrote: > > On Thu, May 5, 2022 at 8:04 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > On Thu, May 05, 2022 at 07:36:51PM +1000, Alistair Francis wrote: > > > On Tue, May 3, 2022 at 5:57 PM Atish Patra <atishp@atishpatra.org> wrote: > > > > > > > > On Tue, Apr 19, 2022 at 5:26 PM Atish Patra <atishp@atishpatra.org> wrote: > > > > > > > > > > On Tue, Apr 19, 2022 at 9:51 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > > > > > On Mon, Apr 11, 2022 at 07:10:06PM -0700, Atish Patra wrote: > > > > > > > > > > > > > > The RISC-V virt machine has helped RISC-V software eco system to evolve at a > > > > > > > rapid pace even in absense of the real hardware. It is definitely commendable. > > > > > > > However, the number of devices & commandline options keeps growing as a result > > > > > > > of that as well. That adds flexibility but will also become bit difficult > > > > > > > to manage in the future as more extension support will be added. As it is the > > > > > > > most commonly used qemu machine, it needs to support all kinds of device and > > > > > > > interrupts as well. Moreover, virt machine has limitations on the maximum > > > > > > > number of harts it can support because of all the MMIO devices it has to support. > > > > > > > > > > > > > > The RISC-V IMSIC specification allows to develop machines completely relying > > > > > > > on MSI and don't care about the wired interrupts at all. It just requires > > > > > > > all the devices to be present behind a PCI bus or present themselves as platform > > > > > > > MSI device. The former is a more common scenario in x86 world where most > > > > > > > of the devices are behind PCI bus. As there is very limited MMIO device > > > > > > > support, it can also scale to very large number of harts. > > > > > > > > > > > > > > That's why, this patch series introduces a minimalistic yet very extensible > > > > > > > forward looking machine called as "RISC-V Mini Computer" or "minic". The > > > > > > > idea is to build PC or server like systems with this machine. The machine can > > > > > > > work with or without virtio framework. The current implementation only > > > > > > > supports RV64. I am not sure if building a RV32 machine would be of interest > > > > > > > for such machines. The only mmio device it requires is clint to emulate > > > > > > > the mtimecmp. > > > > > > > > > > > > > > Any other thoughts ? > > > > > > I don't *love* this idea. I think the virt machine is useful, but I'm > > > not convinced we need a second one. > > > > > > This feels a little bit more like a "none" machine, as it contains > > > just the bare minimum to work. > > > > > > > > > > > > > I would ask what you see as the long term future usage for 'virt' vs > > > > > > 'minic' machine types ? Would you expect all existing users of 'virt' > > > > > > to ultimately switch to 'minic', or are there distinct non-overlapping > > > > > > use cases for 'virt' vs 'minic' such that both end up widely used ? > > > > > > > > > > > > > > > > Nope. I don't expect existing 'virt' users to switch to 'minic' as > > > > > they aim to cater to different users. > > > > > > > > > > Here are the major differences > > > > > 1. virt machine supports MMIO devices & wired interrupts. Minic doesn't > > > > > > This seems like the main difference > > > > > > > > 2. virt machine doesn't support the MSI only option yet (can be added > > > > > though[1]). Minic does. > > > > > > This could be fixed > > > > > > > > 3. Number of cpu supported by virt machine are limited because of the > > > > > MMIO devices. Minic can scale to very > > > > > large numbers of cpu. > > > > > > Similar to 1 > > > > > > > > 4. 'Minic' only supports PCI based MSI capable devices. Thus, MSI is a > > > > > mandatory requirement for 'minic' while > > > > > it is optional for 'virt'. > > > > > > I'm not fully convinced we need this, but it also doesn't seem to cost > > > us a lot in terms of maintenance. It would be beneficial if we could > > > share a bit more of the code. Can we share the socket creation code as > > > well? > > > > > > I don't like the name minic though. What about something like > > > `virt-hpc`, `virt-pcie-minimal` or something like that? That way we > > > indicate it's still a virt board > > > > We're not versioning the 'virt' machine type right so. IOW, we've not > > made any promises about its long term featureset. > > > > If the virt machine type isn't the perfect match right now, should > > we change it, in a potentially incompatible way, to give us the right > > solution long term, rather than introducing a brand new machine type > > with significant overlap. > > Even if we didn't worry about backwards compatibility the current virt > machine would still be what most users want. It's just a small number > of users who don't want MMIO devices and instead want to use PCIe for > everything. Realistically it's only HPC users who would want this type > of machine, at least that's my understanding. > Yes. You are correct. Virt machine will continue to be useful for platforms with MMIO or wired irq devices. The new machine is more suitable for platforms where everything is behind PCI. HPC platforms certainly is one market segment. However, future RISC-V PCs (similar to x86 PCs) may also fall in this category. > Alistair > > > > > > > With regards, > > Daniel > > -- > > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > > |: https://libvirt.org -o- https://fstop138.berrange.com :| > > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > >
On Thu, May 5, 2022 at 4:24 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Thu, May 05, 2022 at 07:36:51PM +1000, Alistair Francis wrote: > > On Tue, May 3, 2022 at 5:57 PM Atish Patra <atishp@atishpatra.org> wrote: > > > > > > On Tue, Apr 19, 2022 at 5:26 PM Atish Patra <atishp@atishpatra.org> wrote: > > > > > > > > On Tue, Apr 19, 2022 at 9:51 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > > > On Mon, Apr 11, 2022 at 07:10:06PM -0700, Atish Patra wrote: > > > > > > > > > > > > The RISC-V virt machine has helped RISC-V software eco system to evolve at a > > > > > > rapid pace even in absense of the real hardware. It is definitely commendable. > > > > > > However, the number of devices & commandline options keeps growing as a result > > > > > > of that as well. That adds flexibility but will also become bit difficult > > > > > > to manage in the future as more extension support will be added. As it is the > > > > > > most commonly used qemu machine, it needs to support all kinds of device and > > > > > > interrupts as well. Moreover, virt machine has limitations on the maximum > > > > > > number of harts it can support because of all the MMIO devices it has to support. > > > > > > > > > > > > The RISC-V IMSIC specification allows to develop machines completely relying > > > > > > on MSI and don't care about the wired interrupts at all. It just requires > > > > > > all the devices to be present behind a PCI bus or present themselves as platform > > > > > > MSI device. The former is a more common scenario in x86 world where most > > > > > > of the devices are behind PCI bus. As there is very limited MMIO device > > > > > > support, it can also scale to very large number of harts. > > > > > > > > > > > > That's why, this patch series introduces a minimalistic yet very extensible > > > > > > forward looking machine called as "RISC-V Mini Computer" or "minic". The > > > > > > idea is to build PC or server like systems with this machine. The machine can > > > > > > work with or without virtio framework. The current implementation only > > > > > > supports RV64. I am not sure if building a RV32 machine would be of interest > > > > > > for such machines. The only mmio device it requires is clint to emulate > > > > > > the mtimecmp. > > > > > > > > > > > Any other thoughts ? > > > > I don't *love* this idea. I think the virt machine is useful, but I'm > > not convinced we need a second one. > > > > This feels a little bit more like a "none" machine, as it contains > > just the bare minimum to work. > > > > > > > > > > I would ask what you see as the long term future usage for 'virt' vs > > > > > 'minic' machine types ? Would you expect all existing users of 'virt' > > > > > to ultimately switch to 'minic', or are there distinct non-overlapping > > > > > use cases for 'virt' vs 'minic' such that both end up widely used ? > > > > > > > > > > > > > Nope. I don't expect existing 'virt' users to switch to 'minic' as > > > > they aim to cater to different users. > > > > > > > > Here are the major differences > > > > 1. virt machine supports MMIO devices & wired interrupts. Minic doesn't > > > > This seems like the main difference > > > > > > 2. virt machine doesn't support the MSI only option yet (can be added > > > > though[1]). Minic does. > > > > This could be fixed > > > > > > 3. Number of cpu supported by virt machine are limited because of the > > > > MMIO devices. Minic can scale to very > > > > large numbers of cpu. > > > > Similar to 1 > > > > > > 4. 'Minic' only supports PCI based MSI capable devices. Thus, MSI is a > > > > mandatory requirement for 'minic' while > > > > it is optional for 'virt'. > > > > I'm not fully convinced we need this, but it also doesn't seem to cost > > us a lot in terms of maintenance. It would be beneficial if we could > > share a bit more of the code. Can we share the socket creation code as > > well? > > > > I don't like the name minic though. What about something like > > `virt-hpc`, `virt-pcie-minimal` or something like that? That way we > > indicate it's still a virt board > > We're not versioning the 'virt' machine type right so. IOW, we've not > made any promises about its long term featureset. > > If the virt machine type isn't the perfect match right now, should > we change it, in a potentially incompatible way, to give us the right > solution long term, rather than introducing a brand new machine type > with significant overlap. Versioning of "virt" machine has been a long pending item for enterprise class Guest/VM migration. Since "virt" machine is QEMU RISC-V specific, we can do the following: 1) Detailed documentation of "virt" machine layout along with versioning in the QEMU documentation 2) Re-structure "virt" machine implementation to support future changes "virt" machine. Regards, Anup > > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > >
On Fri, May 06, 2022 at 06:34:47AM +1000, Alistair Francis wrote: > On Thu, May 5, 2022 at 8:04 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > On Thu, May 05, 2022 at 07:36:51PM +1000, Alistair Francis wrote: > > > On Tue, May 3, 2022 at 5:57 PM Atish Patra <atishp@atishpatra.org> wrote: > > > > > > > > On Tue, Apr 19, 2022 at 5:26 PM Atish Patra <atishp@atishpatra.org> wrote: > > > > > > > > > > On Tue, Apr 19, 2022 at 9:51 AM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > > > > > On Mon, Apr 11, 2022 at 07:10:06PM -0700, Atish Patra wrote: > > > > > > > > > > > > > > The RISC-V virt machine has helped RISC-V software eco system to evolve at a > > > > > > > rapid pace even in absense of the real hardware. It is definitely commendable. > > > > > > > However, the number of devices & commandline options keeps growing as a result > > > > > > > of that as well. That adds flexibility but will also become bit difficult > > > > > > > to manage in the future as more extension support will be added. As it is the > > > > > > > most commonly used qemu machine, it needs to support all kinds of device and > > > > > > > interrupts as well. Moreover, virt machine has limitations on the maximum > > > > > > > number of harts it can support because of all the MMIO devices it has to support. > > > > > > > > > > > > > > The RISC-V IMSIC specification allows to develop machines completely relying > > > > > > > on MSI and don't care about the wired interrupts at all. It just requires > > > > > > > all the devices to be present behind a PCI bus or present themselves as platform > > > > > > > MSI device. The former is a more common scenario in x86 world where most > > > > > > > of the devices are behind PCI bus. As there is very limited MMIO device > > > > > > > support, it can also scale to very large number of harts. > > > > > > > > > > > > > > That's why, this patch series introduces a minimalistic yet very extensible > > > > > > > forward looking machine called as "RISC-V Mini Computer" or "minic". The > > > > > > > idea is to build PC or server like systems with this machine. The machine can > > > > > > > work with or without virtio framework. The current implementation only > > > > > > > supports RV64. I am not sure if building a RV32 machine would be of interest > > > > > > > for such machines. The only mmio device it requires is clint to emulate > > > > > > > the mtimecmp. > > > > > > > > > > > > > > Any other thoughts ? > > > > > > I don't *love* this idea. I think the virt machine is useful, but I'm > > > not convinced we need a second one. > > > > > > This feels a little bit more like a "none" machine, as it contains > > > just the bare minimum to work. > > > > > > > > > > > > > I would ask what you see as the long term future usage for 'virt' vs > > > > > > 'minic' machine types ? Would you expect all existing users of 'virt' > > > > > > to ultimately switch to 'minic', or are there distinct non-overlapping > > > > > > use cases for 'virt' vs 'minic' such that both end up widely used ? > > > > > > > > > > > > > > > > Nope. I don't expect existing 'virt' users to switch to 'minic' as > > > > > they aim to cater to different users. > > > > > > > > > > Here are the major differences > > > > > 1. virt machine supports MMIO devices & wired interrupts. Minic doesn't > > > > > > This seems like the main difference > > > > > > > > 2. virt machine doesn't support the MSI only option yet (can be added > > > > > though[1]). Minic does. > > > > > > This could be fixed > > > > > > > > 3. Number of cpu supported by virt machine are limited because of the > > > > > MMIO devices. Minic can scale to very > > > > > large numbers of cpu. > > > > > > Similar to 1 > > > > > > > > 4. 'Minic' only supports PCI based MSI capable devices. Thus, MSI is a > > > > > mandatory requirement for 'minic' while > > > > > it is optional for 'virt'. > > > > > > I'm not fully convinced we need this, but it also doesn't seem to cost > > > us a lot in terms of maintenance. It would be beneficial if we could > > > share a bit more of the code. Can we share the socket creation code as > > > well? > > > > > > I don't like the name minic though. What about something like > > > `virt-hpc`, `virt-pcie-minimal` or something like that? That way we > > > indicate it's still a virt board > > > > We're not versioning the 'virt' machine type right so. IOW, we've not > > made any promises about its long term featureset. > > > > If the virt machine type isn't the perfect match right now, should > > we change it, in a potentially incompatible way, to give us the right > > solution long term, rather than introducing a brand new machine type > > with significant overlap. > > Even if we didn't worry about backwards compatibility the current virt > machine would still be what most users want. It's just a small number > of users who don't want MMIO devices and instead want to use PCIe for > everything. Realistically it's only HPC users who would want this type > of machine, at least that's my understanding. I'm not so sure about that. Every other architecture has ended up standardizing on PCI for general purpose virtual machines. IIRC, aarch64 started off with MMIO, but switched to PCI as it matured. In terms of having VM mgmt tools "just work" for risc-v, I think it will be very compelling for the general 'virt' machine to be PCI based, otherwise all the assumptions about PCI in mgmt apps are going to break requiring never ending riscv fixes. With regards, Daniel
On Fri, 6 May 2022 at 09:18, Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Fri, May 06, 2022 at 06:34:47AM +1000, Alistair Francis wrote: > > Even if we didn't worry about backwards compatibility the current virt > > machine would still be what most users want. It's just a small number > > of users who don't want MMIO devices and instead want to use PCIe for > > everything. Realistically it's only HPC users who would want this type > > of machine, at least that's my understanding. > > I'm not so sure about that. Every other architecture has ended up > standardizing on PCI for general purpose virtual machines. IIRC, > aarch64 started off with MMIO, but switched to PCI as it matured. > > In terms of having VM mgmt tools "just work" for risc-v, I think > it will be very compelling for the general 'virt' machine to be > PCI based, otherwise all the assumptions about PCI in mgmt apps > are going to break requiring never ending riscv fixes. Mmm, my experience with aarch64 virt is that PCI is much nicer as a general preference. aarch64 virt has some MMIO devices for historical reasons and some because you can't reasonably do the necessary things with PCI, but I'm actively trying to push people who submit new MMIO device features for virt to try to use a PCI-based solution instead if they possibly can. thanks -- PMM
On Fri, May 6, 2022 at 4:00 AM Peter Maydell <peter.maydell@linaro.org> wrote: > > On Fri, 6 May 2022 at 09:18, Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > On Fri, May 06, 2022 at 06:34:47AM +1000, Alistair Francis wrote: > > > Even if we didn't worry about backwards compatibility the current virt > > > machine would still be what most users want. It's just a small number > > > of users who don't want MMIO devices and instead want to use PCIe for > > > everything. Realistically it's only HPC users who would want this type > > > of machine, at least that's my understanding. > > > > I'm not so sure about that. Every other architecture has ended up > > standardizing on PCI for general purpose virtual machines. IIRC, > > aarch64 started off with MMIO, but switched to PCI as it matured. > > > > In terms of having VM mgmt tools "just work" for risc-v, I think > > it will be very compelling for the general 'virt' machine to be > > PCI based, otherwise all the assumptions about PCI in mgmt apps > > are going to break requiring never ending riscv fixes. > > Mmm, my experience with aarch64 virt is that PCI is much nicer > as a general preference. aarch64 virt has some MMIO devices > for historical reasons and some because you can't reasonably > do the necessary things with PCI, but I'm actively trying to > push people who submit new MMIO device features for virt to > try to use a PCI-based solution instead if they possibly can. > Yeah. That was one of the primary goals of this series. If we have an alternate PCI only machine, folks will be more motivated to add only PCI based solutions in the future. In that case, there will be minimal or no change required to the machine code itself. Just for clarification: We can achieve the same with the current virt machine. But it is already bloated with a bunch of MMIO devices and will probably continue to do so because of its flexibility. In addition to that, any real PCI based platform emulation can not reuse the virt machine code which will result in more vendor specific implementations in the future.. > thanks > -- PMM
On Sat, May 7, 2022 at 6:30 AM Atish Kumar Patra <atishp@rivosinc.com> wrote: > > On Fri, May 6, 2022 at 4:00 AM Peter Maydell <peter.maydell@linaro.org> wrote: > > > > On Fri, 6 May 2022 at 09:18, Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > On Fri, May 06, 2022 at 06:34:47AM +1000, Alistair Francis wrote: > > > > Even if we didn't worry about backwards compatibility the current virt > > > > machine would still be what most users want. It's just a small number > > > > of users who don't want MMIO devices and instead want to use PCIe for > > > > everything. Realistically it's only HPC users who would want this type > > > > of machine, at least that's my understanding. > > > > > > I'm not so sure about that. Every other architecture has ended up > > > standardizing on PCI for general purpose virtual machines. IIRC, > > > aarch64 started off with MMIO, but switched to PCI as it matured. > > > > > > In terms of having VM mgmt tools "just work" for risc-v, I think > > > it will be very compelling for the general 'virt' machine to be > > > PCI based, otherwise all the assumptions about PCI in mgmt apps > > > are going to break requiring never ending riscv fixes. > > > > Mmm, my experience with aarch64 virt is that PCI is much nicer > > as a general preference. aarch64 virt has some MMIO devices > > for historical reasons and some because you can't reasonably > > do the necessary things with PCI, but I'm actively trying to > > push people who submit new MMIO device features for virt to > > try to use a PCI-based solution instead if they possibly can. Interesting... Ok, maybe calling this "virt-pcie" might be a good start, with the expectation to eventually replace the current virt with the new virt-pcie at some point. The other option would be to try and gradually change from the current virt machine to this new virt machine Alistair > > > > Yeah. That was one of the primary goals of this series. If we have an > alternate PCI only machine, > folks will be more motivated to add only PCI based solutions in the > future. In that case, there will be minimal > or no change required to the machine code itself. > > Just for clarification: We can achieve the same with the current virt > machine. But it is already bloated with a bunch of MMIO devices > and will probably continue to do so because of its flexibility. In > addition to that, any real PCI based platform emulation can not reuse > the virt machine code which will result in more vendor specific > implementations in the future.. > > > thanks > > -- PMM
On Tue, May 17, 2022 at 03:03:38PM +1000, Alistair Francis wrote: > On Sat, May 7, 2022 at 6:30 AM Atish Kumar Patra <atishp@rivosinc.com> wrote: > > > > On Fri, May 6, 2022 at 4:00 AM Peter Maydell <peter.maydell@linaro.org> wrote: > > > > > > On Fri, 6 May 2022 at 09:18, Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > On Fri, May 06, 2022 at 06:34:47AM +1000, Alistair Francis wrote: > > > > > Even if we didn't worry about backwards compatibility the current virt > > > > > machine would still be what most users want. It's just a small number > > > > > of users who don't want MMIO devices and instead want to use PCIe for > > > > > everything. Realistically it's only HPC users who would want this type > > > > > of machine, at least that's my understanding. > > > > > > > > I'm not so sure about that. Every other architecture has ended up > > > > standardizing on PCI for general purpose virtual machines. IIRC, > > > > aarch64 started off with MMIO, but switched to PCI as it matured. > > > > > > > > In terms of having VM mgmt tools "just work" for risc-v, I think > > > > it will be very compelling for the general 'virt' machine to be > > > > PCI based, otherwise all the assumptions about PCI in mgmt apps > > > > are going to break requiring never ending riscv fixes. > > > > > > Mmm, my experience with aarch64 virt is that PCI is much nicer > > > as a general preference. aarch64 virt has some MMIO devices > > > for historical reasons and some because you can't reasonably > > > do the necessary things with PCI, but I'm actively trying to > > > push people who submit new MMIO device features for virt to > > > try to use a PCI-based solution instead if they possibly can. > > Interesting... > > Ok, maybe calling this "virt-pcie" might be a good start, with the > expectation to eventually replace the current virt with the new > virt-pcie at some point. Delaying the inevitable by leaving PCIE support in a separate machine type initially is going to be more painful long term. > The other option would be to try and gradually change from the current > virt machine to this new virt machine Yes, I really think the 'virt' machine type needs to aim for PCIE support sooner rather than later, if RISC-V wants to get on part with other architectures. The best time to have added PCIE support to 'virt' was when it was first created, the next best time is now. With regards, Daniel
On Tue, May 17, 2022 at 6:52 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Tue, May 17, 2022 at 03:03:38PM +1000, Alistair Francis wrote: > > On Sat, May 7, 2022 at 6:30 AM Atish Kumar Patra <atishp@rivosinc.com> wrote: > > > > > > On Fri, May 6, 2022 at 4:00 AM Peter Maydell <peter.maydell@linaro.org> wrote: > > > > > > > > On Fri, 6 May 2022 at 09:18, Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > > > On Fri, May 06, 2022 at 06:34:47AM +1000, Alistair Francis wrote: > > > > > > Even if we didn't worry about backwards compatibility the current virt > > > > > > machine would still be what most users want. It's just a small number > > > > > > of users who don't want MMIO devices and instead want to use PCIe for > > > > > > everything. Realistically it's only HPC users who would want this type > > > > > > of machine, at least that's my understanding. > > > > > > > > > > I'm not so sure about that. Every other architecture has ended up > > > > > standardizing on PCI for general purpose virtual machines. IIRC, > > > > > aarch64 started off with MMIO, but switched to PCI as it matured. > > > > > > > > > > In terms of having VM mgmt tools "just work" for risc-v, I think > > > > > it will be very compelling for the general 'virt' machine to be > > > > > PCI based, otherwise all the assumptions about PCI in mgmt apps > > > > > are going to break requiring never ending riscv fixes. > > > > > > > > Mmm, my experience with aarch64 virt is that PCI is much nicer > > > > as a general preference. aarch64 virt has some MMIO devices > > > > for historical reasons and some because you can't reasonably > > > > do the necessary things with PCI, but I'm actively trying to > > > > push people who submit new MMIO device features for virt to > > > > try to use a PCI-based solution instead if they possibly can. > > > > Interesting... > > > > Ok, maybe calling this "virt-pcie" might be a good start, with the > > expectation to eventually replace the current virt with the new > > virt-pcie at some point. > > Delaying the inevitable by leaving PCIE support in a separate > machine type initially is going to be more painful long term. > > > The other option would be to try and gradually change from the current > > virt machine to this new virt machine > > Yes, I really think the 'virt' machine type needs to aim for PCIE > support sooner rather than later, if RISC-V wants to get on part > with other architectures. The best time to have added PCIE support > to 'virt' was when it was first created, the next best time is now. So maybe instead we lock in the current virt machine as the 7.1 virt machine for QEMU 7.1, then work on migrating to a PCIe only machine with versions (similar to the other archs) Alistair > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
On Tue, May 17, 2022 at 1:54 PM Alistair Francis <alistair23@gmail.com> wrote: > > On Tue, May 17, 2022 at 6:52 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > On Tue, May 17, 2022 at 03:03:38PM +1000, Alistair Francis wrote: > > > On Sat, May 7, 2022 at 6:30 AM Atish Kumar Patra <atishp@rivosinc.com> wrote: > > > > > > > > On Fri, May 6, 2022 at 4:00 AM Peter Maydell <peter.maydell@linaro.org> wrote: > > > > > > > > > > On Fri, 6 May 2022 at 09:18, Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > > > > > On Fri, May 06, 2022 at 06:34:47AM +1000, Alistair Francis wrote: > > > > > > > Even if we didn't worry about backwards compatibility the current virt > > > > > > > machine would still be what most users want. It's just a small number > > > > > > > of users who don't want MMIO devices and instead want to use PCIe for > > > > > > > everything. Realistically it's only HPC users who would want this type > > > > > > > of machine, at least that's my understanding. > > > > > > > > > > > > I'm not so sure about that. Every other architecture has ended up > > > > > > standardizing on PCI for general purpose virtual machines. IIRC, > > > > > > aarch64 started off with MMIO, but switched to PCI as it matured. > > > > > > > > > > > > In terms of having VM mgmt tools "just work" for risc-v, I think > > > > > > it will be very compelling for the general 'virt' machine to be > > > > > > PCI based, otherwise all the assumptions about PCI in mgmt apps > > > > > > are going to break requiring never ending riscv fixes. > > > > > > > > > > Mmm, my experience with aarch64 virt is that PCI is much nicer > > > > > as a general preference. aarch64 virt has some MMIO devices > > > > > for historical reasons and some because you can't reasonably > > > > > do the necessary things with PCI, but I'm actively trying to > > > > > push people who submit new MMIO device features for virt to > > > > > try to use a PCI-based solution instead if they possibly can. > > > > > > Interesting... > > > > > > Ok, maybe calling this "virt-pcie" might be a good start, with the > > > expectation to eventually replace the current virt with the new > > > virt-pcie at some point. > > > > Delaying the inevitable by leaving PCIE support in a separate > > machine type initially is going to be more painful long term. > > > > > The other option would be to try and gradually change from the current > > > virt machine to this new virt machine > > > > Yes, I really think the 'virt' machine type needs to aim for PCIE > > support sooner rather than later, if RISC-V wants to get on part > > with other architectures. The best time to have added PCIE support > > to 'virt' was when it was first created, the next best time is now. > > So maybe instead we lock in the current virt machine as the 7.1 virt > machine for QEMU 7.1, then work on migrating to a PCIe only machine > with versions (similar to the other archs) > I am not quite sure what exactly you mean here. Do you mean to modify the current virt machine to be PCIE only after QEMU 7.1 or the new PCIE only machine (with the versioning) which will be the default machine in the future If you intend to say the former, few issues with that approach. 1. virt machine is not well documented and already bloated. There is no specification for virt machine as such. Putting restrictions after a certain release will lead to confusion. 2. Do we support existing MMIO devices after that specific version or not ? 3. The user has to be aware of which version of virt machine it is running in order to test specific features (i.e. flash, reset, wired interrupts) 4. Based on the version of the virt machine, the command line options will change. This may also be confusing. On the other hand, the second approach will be much cleaner without any baggage. The RISC-V eco-system is still maturing and this is the right time to start something fresh. Let's call the new machine virt-pcie for now. Here are a few things that can be implemented for this machine. 1. It must support versioning and any changes to the machine code must modify the version of the machine. 2. It must only support MSI based PCIe devices. No support for wired interrupts. The only allowed MMIO devices are -- mtimer/mtimecmp (there is no other option provided in ISA) -- reset/rtc device (If there is a way we can emulate these two over PCIe, that would be great) -- flash Beyond this, adding any other MMIO device must be strongly discouraged. 3. The virt-pcie machine must be well documented similar to a hardware specification (including address range, restrictions and implemented/allowed devices) 4. No dependence on virtio framework but must work with virtio-pcie backend. 5. Migration must be supported. 6. No command line option is required. 7. Any other ? Once we have these policies in place, this can be the preferred virt machine and the current virt machine can be phased out or continue to co-exist as a more flexible experimental platform. Thoughts ? > Alistair > > > > > With regards, > > Daniel > > -- > > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > > |: https://libvirt.org -o- https://fstop138.berrange.com :| > > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > >
On Tue, May 17, 2022 at 11:38:39PM -0700, Atish Patra wrote: > On Tue, May 17, 2022 at 1:54 PM Alistair Francis <alistair23@gmail.com> wrote: > > > > On Tue, May 17, 2022 at 6:52 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > On Tue, May 17, 2022 at 03:03:38PM +1000, Alistair Francis wrote: > > > > On Sat, May 7, 2022 at 6:30 AM Atish Kumar Patra <atishp@rivosinc.com> wrote: > > > > > > > > > > On Fri, May 6, 2022 at 4:00 AM Peter Maydell <peter.maydell@linaro.org> wrote: > > > > > > > > > > > > On Fri, 6 May 2022 at 09:18, Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > > > > > > > On Fri, May 06, 2022 at 06:34:47AM +1000, Alistair Francis wrote: > > > > > > > > Even if we didn't worry about backwards compatibility the current virt > > > > > > > > machine would still be what most users want. It's just a small number > > > > > > > > of users who don't want MMIO devices and instead want to use PCIe for > > > > > > > > everything. Realistically it's only HPC users who would want this type > > > > > > > > of machine, at least that's my understanding. > > > > > > > > > > > > > > I'm not so sure about that. Every other architecture has ended up > > > > > > > standardizing on PCI for general purpose virtual machines. IIRC, > > > > > > > aarch64 started off with MMIO, but switched to PCI as it matured. > > > > > > > > > > > > > > In terms of having VM mgmt tools "just work" for risc-v, I think > > > > > > > it will be very compelling for the general 'virt' machine to be > > > > > > > PCI based, otherwise all the assumptions about PCI in mgmt apps > > > > > > > are going to break requiring never ending riscv fixes. > > > > > > > > > > > > Mmm, my experience with aarch64 virt is that PCI is much nicer > > > > > > as a general preference. aarch64 virt has some MMIO devices > > > > > > for historical reasons and some because you can't reasonably > > > > > > do the necessary things with PCI, but I'm actively trying to > > > > > > push people who submit new MMIO device features for virt to > > > > > > try to use a PCI-based solution instead if they possibly can. > > > > > > > > Interesting... > > > > > > > > Ok, maybe calling this "virt-pcie" might be a good start, with the > > > > expectation to eventually replace the current virt with the new > > > > virt-pcie at some point. > > > > > > Delaying the inevitable by leaving PCIE support in a separate > > > machine type initially is going to be more painful long term. > > > > > > > The other option would be to try and gradually change from the current > > > > virt machine to this new virt machine > > > > > > Yes, I really think the 'virt' machine type needs to aim for PCIE > > > support sooner rather than later, if RISC-V wants to get on part > > > with other architectures. The best time to have added PCIE support > > > to 'virt' was when it was first created, the next best time is now. > > > > So maybe instead we lock in the current virt machine as the 7.1 virt > > machine for QEMU 7.1, then work on migrating to a PCIe only machine > > with versions (similar to the other archs) > > > > I am not quite sure what exactly you mean here. Do you mean to modify > the current virt > machine to be PCIE only after QEMU 7.1 or the new PCIE only machine > (with the versioning) > which will be the default machine in the future > > If you intend to say the former, few issues with that approach. > > 1. virt machine is not well documented and already bloated. There is > no specification for virt machine as such. > Putting restrictions after a certain release will lead to confusion. > 2. Do we support existing MMIO devices after that specific version or not ? > 3. The user has to be aware of which version of virt machine it is > running in order to test specific features (i.e. flash, reset, wired > interrupts) > 4. Based on the version of the virt machine, the command line options > will change. This may also be confusing. > > On the other hand, the second approach will be much cleaner without > any baggage. The RISC-V eco-system is still maturing and this is the > right time > to start something fresh. Let's call the new machine virt-pcie for > now. Here are a few things that can be implemented for this machine. The fact that RISC-V ecosystem is so young & has relatively few users, and even fewer expecting long term stability, is precisely why we should just modify the existing 'virt' machine now rather than introducing a new 'virt-pcie'. We can afford to have the limited incompatibility in the short term given the small userbase. We went through this same exercise with aarch64 virt machine and despite the short term disruption, it was a good success IMHO to get it switched from MMIO to PCI, instead of having two machines in parallel long term. With regards, Daniel
On Wed, 18 May 2022 at 09:25, Daniel P. Berrangé <berrange@redhat.com> wrote: > The fact that RISC-V ecosystem is so young & has relatively few > users, and even fewer expecting long term stability, is precisely > why we should just modify the existing 'virt' machine now rather > than introducing a new 'virt-pcie'. We can afford to have the > limited incompatibility in the short term given the small userbase. > We went through this same exercise with aarch64 virt machine and > despite the short term disruption, it was a good success IMHO to > get it switched from MMIO to PCI, instead of having two machines > in parallel long term. The aarch64 virt board does still carry around the mmio devices, though...it's just that we have pci as well now. Personally I don't think that switching to a new machine type is likely to help escape from the "bloat" problem, which arises from two conflicting desires: (1) people want this kind of board to be nice and small and simple, with a minimal set of devices (2) everybody has their own "but this specific one device is really important and it should be in the minimal set" (watchdog? acpi? ability to power the machine on and off? second UART? i2c? etc etc etc) So either your 'minimal' board is only serving a small subset of the users who want a minimal board; or else it's not as minimal as any of them would like; or else it acquires a growing set of -machine options to turn various devices on and off... -- PMM
On Wed, May 18, 2022 at 3:46 AM Peter Maydell <peter.maydell@linaro.org> wrote: > > On Wed, 18 May 2022 at 09:25, Daniel P. Berrangé <berrange@redhat.com> wrote: > > The fact that RISC-V ecosystem is so young & has relatively few > > users, and even fewer expecting long term stability, is precisely > > why we should just modify the existing 'virt' machine now rather > > than introducing a new 'virt-pcie'. We can afford to have the > > limited incompatibility in the short term given the small userbase. > > We went through this same exercise with aarch64 virt machine and > > despite the short term disruption, it was a good success IMHO to > > get it switched from MMIO to PCI, instead of having two machines > > in parallel long term. > > The aarch64 virt board does still carry around the mmio devices, > though...it's just that we have pci as well now. > > Personally I don't think that switching to a new machine type > is likely to help escape from the "bloat" problem, which arises > from two conflicting desires: > > (1) people want this kind of board to be nice and small and > simple, with a minimal set of devices > (2) everybody has their own "but this specific one device is > really important and it should be in the minimal set" > (watchdog? acpi? ability to power the machine on and off? > second UART? i2c? etc etc etc) > Both acpi/device tree support should be there anyways. MMIO based reset will probably needed as well (I listed that earlier with mandatory MMIO devices) AFAIK everything else can be PCIe based which the new board will mandate. It must strictly enforce the rules about what can be added to it. The bar to allow new MMIO devices must be very high and must have a wide range of usage. This will make life easier for the entire ecosystem as well. AFAIK, libvirt uses PCIe devices only to build VMs. I understand that is probably a big ask but if odd mmio devices sneak into this platform, then that defeats the purpose. On other hand, having a flag day for virt machines creates a lot of incompatibility for the users until everyone transitions. The transition also has to happen based on Qemu version as virt machine doesn't have any versioning right now. Do we make users' life difficult by having a flag date based on the Qemu version or take additional responsibility of maintaining another board ? I hope the new board will continue to be small so the maintenance burden is not too much. Personally, I feel the latter approach will have minimum inconvenience for everybody but I am okay with whatever is decided by the community. > So either your 'minimal' board is only serving a small subset > of the users who want a minimal board; or else it's not as > minimal as any of them would like; or else it acquires a > growing set of -machine options to turn various devices on > and off... > > -- PMM
On Wed, May 18, 2022 at 4:38 PM Atish Patra <atishp@atishpatra.org> wrote: > > On Tue, May 17, 2022 at 1:54 PM Alistair Francis <alistair23@gmail.com> wrote: > > > > On Tue, May 17, 2022 at 6:52 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > On Tue, May 17, 2022 at 03:03:38PM +1000, Alistair Francis wrote: > > > > On Sat, May 7, 2022 at 6:30 AM Atish Kumar Patra <atishp@rivosinc.com> wrote: > > > > > > > > > > On Fri, May 6, 2022 at 4:00 AM Peter Maydell <peter.maydell@linaro.org> wrote: > > > > > > > > > > > > On Fri, 6 May 2022 at 09:18, Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > > > > > > > On Fri, May 06, 2022 at 06:34:47AM +1000, Alistair Francis wrote: > > > > > > > > Even if we didn't worry about backwards compatibility the current virt > > > > > > > > machine would still be what most users want. It's just a small number > > > > > > > > of users who don't want MMIO devices and instead want to use PCIe for > > > > > > > > everything. Realistically it's only HPC users who would want this type > > > > > > > > of machine, at least that's my understanding. > > > > > > > > > > > > > > I'm not so sure about that. Every other architecture has ended up > > > > > > > standardizing on PCI for general purpose virtual machines. IIRC, > > > > > > > aarch64 started off with MMIO, but switched to PCI as it matured. > > > > > > > > > > > > > > In terms of having VM mgmt tools "just work" for risc-v, I think > > > > > > > it will be very compelling for the general 'virt' machine to be > > > > > > > PCI based, otherwise all the assumptions about PCI in mgmt apps > > > > > > > are going to break requiring never ending riscv fixes. > > > > > > > > > > > > Mmm, my experience with aarch64 virt is that PCI is much nicer > > > > > > as a general preference. aarch64 virt has some MMIO devices > > > > > > for historical reasons and some because you can't reasonably > > > > > > do the necessary things with PCI, but I'm actively trying to > > > > > > push people who submit new MMIO device features for virt to > > > > > > try to use a PCI-based solution instead if they possibly can. > > > > > > > > Interesting... > > > > > > > > Ok, maybe calling this "virt-pcie" might be a good start, with the > > > > expectation to eventually replace the current virt with the new > > > > virt-pcie at some point. > > > > > > Delaying the inevitable by leaving PCIE support in a separate > > > machine type initially is going to be more painful long term. > > > > > > > The other option would be to try and gradually change from the current > > > > virt machine to this new virt machine > > > > > > Yes, I really think the 'virt' machine type needs to aim for PCIE > > > support sooner rather than later, if RISC-V wants to get on part > > > with other architectures. The best time to have added PCIE support > > > to 'virt' was when it was first created, the next best time is now. > > > > So maybe instead we lock in the current virt machine as the 7.1 virt > > machine for QEMU 7.1, then work on migrating to a PCIe only machine > > with versions (similar to the other archs) > > > > I am not quite sure what exactly you mean here. Do you mean to modify > the current virt > machine to be PCIE only after QEMU 7.1 or the new PCIE only machine > (with the versioning) > which will be the default machine in the future I mean that we call the current virt machine the virt machine for QEMU 7.1. Then for future releases we can make breaking changes, where we have the old 7.1 virt machine for backwards compatibility. > > If you intend to say the former, few issues with that approach. > > 1. virt machine is not well documented and already bloated. There is > no specification for virt machine as such. > Putting restrictions after a certain release will lead to confusion. > 2. Do we support existing MMIO devices after that specific version or not ? Yeah, so I guess this doesn't achieve the same outcome you want. I would say we would still include some MMIO devices, like UART for example. But we could simplify things a bit. So for example maybe we could use AIA by default and then remove the PLIC code. That would help cleanup the board file. Then we could add a `msi-only` option that would be similar to https://github.com/atishp04/qemu/commit/d7fc1c6aa7855b414b3484672a076140166a2dcd. But without the PLIC it should hopefully be cleaner We would need AIA ratified before we could remove the PLIC though. > 3. The user has to be aware of which version of virt machine it is > running in order to test specific features (i.e. flash, reset, wired > interrupts) That's true, but I think we are going to have this issue someday anyway. We don't want to use the SiFive CLINT and PLIC forever, eventually we will want to use the newer hardware. > 4. Based on the version of the virt machine, the command line options > will change. This may also be confusing. > > On the other hand, the second approach will be much cleaner without > any baggage. The RISC-V eco-system is still maturing and this is the > right time > to start something fresh. Let's call the new machine virt-pcie for > now. Here are a few things that can be implemented for this machine. > > 1. It must support versioning and any changes to the machine code must > modify the version of the machine. > 2. It must only support MSI based PCIe devices. No support for wired interrupts. > The only allowed MMIO devices are > -- mtimer/mtimecmp (there is no other option provided in ISA) > -- reset/rtc device (If there is a way we can emulate these > two over PCIe, that would be great) > -- flash > Beyond this, adding any other MMIO device must be strongly discouraged. > 3. The virt-pcie machine must be well documented similar to a hardware > specification (including address range, restrictions and > implemented/allowed devices) > 4. No dependence on virtio framework but must work with virtio-pcie backend. > 5. Migration must be supported. I'm on board with these! They would all be great to have. I'm open to a virt-pcie, but as others have pointed out it might be easier to just make the switch now to the general board. > 6. No command line option is required. I don't see this being achievable unfortunately > 7. Any other ? > > Once we have these policies in place, this can be the preferred virt > machine and the current virt machine can be phased out or continue to > co-exist > as a more flexible experimental platform. > > Thoughts ? > > > Alistair > > > > > > > > With regards, > > > Daniel > > > -- > > > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > > > |: https://libvirt.org -o- https://fstop138.berrange.com :| > > > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > > > > > > > -- > Regards, > Atish
On Sun, May 22, 2022 at 10:59 PM Alistair Francis <alistair23@gmail.com> wrote: > > On Wed, May 18, 2022 at 4:38 PM Atish Patra <atishp@atishpatra.org> wrote: > > > > On Tue, May 17, 2022 at 1:54 PM Alistair Francis <alistair23@gmail.com> wrote: > > > > > > On Tue, May 17, 2022 at 6:52 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > On Tue, May 17, 2022 at 03:03:38PM +1000, Alistair Francis wrote: > > > > > On Sat, May 7, 2022 at 6:30 AM Atish Kumar Patra <atishp@rivosinc.com> wrote: > > > > > > > > > > > > On Fri, May 6, 2022 at 4:00 AM Peter Maydell <peter.maydell@linaro.org> wrote: > > > > > > > > > > > > > > On Fri, 6 May 2022 at 09:18, Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > > > > > > > > > > > On Fri, May 06, 2022 at 06:34:47AM +1000, Alistair Francis wrote: > > > > > > > > > Even if we didn't worry about backwards compatibility the current virt > > > > > > > > > machine would still be what most users want. It's just a small number > > > > > > > > > of users who don't want MMIO devices and instead want to use PCIe for > > > > > > > > > everything. Realistically it's only HPC users who would want this type > > > > > > > > > of machine, at least that's my understanding. > > > > > > > > > > > > > > > > I'm not so sure about that. Every other architecture has ended up > > > > > > > > standardizing on PCI for general purpose virtual machines. IIRC, > > > > > > > > aarch64 started off with MMIO, but switched to PCI as it matured. > > > > > > > > > > > > > > > > In terms of having VM mgmt tools "just work" for risc-v, I think > > > > > > > > it will be very compelling for the general 'virt' machine to be > > > > > > > > PCI based, otherwise all the assumptions about PCI in mgmt apps > > > > > > > > are going to break requiring never ending riscv fixes. > > > > > > > > > > > > > > Mmm, my experience with aarch64 virt is that PCI is much nicer > > > > > > > as a general preference. aarch64 virt has some MMIO devices > > > > > > > for historical reasons and some because you can't reasonably > > > > > > > do the necessary things with PCI, but I'm actively trying to > > > > > > > push people who submit new MMIO device features for virt to > > > > > > > try to use a PCI-based solution instead if they possibly can. > > > > > > > > > > Interesting... > > > > > > > > > > Ok, maybe calling this "virt-pcie" might be a good start, with the > > > > > expectation to eventually replace the current virt with the new > > > > > virt-pcie at some point. > > > > > > > > Delaying the inevitable by leaving PCIE support in a separate > > > > machine type initially is going to be more painful long term. > > > > > > > > > The other option would be to try and gradually change from the current > > > > > virt machine to this new virt machine > > > > > > > > Yes, I really think the 'virt' machine type needs to aim for PCIE > > > > support sooner rather than later, if RISC-V wants to get on part > > > > with other architectures. The best time to have added PCIE support > > > > to 'virt' was when it was first created, the next best time is now. > > > > > > So maybe instead we lock in the current virt machine as the 7.1 virt > > > machine for QEMU 7.1, then work on migrating to a PCIe only machine > > > with versions (similar to the other archs) > > > > > > > I am not quite sure what exactly you mean here. Do you mean to modify > > the current virt > > machine to be PCIE only after QEMU 7.1 or the new PCIE only machine > > (with the versioning) > > which will be the default machine in the future > > I mean that we call the current virt machine the virt machine for QEMU > 7.1. Then for future releases we can make breaking changes, where we > have the old 7.1 virt machine for backwards compatibility. > > > > > If you intend to say the former, few issues with that approach. > > > > 1. virt machine is not well documented and already bloated. There is > > no specification for virt machine as such. > > Putting restrictions after a certain release will lead to confusion. > > 2. Do we support existing MMIO devices after that specific version or not ? > > Yeah, so I guess this doesn't achieve the same outcome you want. I > would say we would still include some MMIO devices, like UART for > example. > Why ? We can just rely on the pcie based uart (virtio-serial-pci or serial-pci) should be enough. The only MMIO devices that should be allowed are the ones that can't be behind pcie. > But we could simplify things a bit. So for example maybe we could use > AIA by default and then remove the PLIC code. That would help cleanup > the board file. Then we could add a `msi-only` option that would be > similar to https://github.com/atishp04/qemu/commit/d7fc1c6aa7855b414b3484672a076140166a2dcd. > But without the PLIC it should hopefully be cleaner > > We would need AIA ratified before we could remove the PLIC though. > And AIA patches available in the upstream Linux kernel. Even after that, can we just remove the PLIC ? That would mean everybody has to use the latest kernel with AIA support after that. > > 3. The user has to be aware of which version of virt machine it is > > running in order to test specific features (i.e. flash, reset, wired > > interrupts) > > That's true, but I think we are going to have this issue someday > anyway. We don't want to use the SiFive CLINT and PLIC forever, > eventually we will want to use the newer hardware. > Agreed. But It should be disabed by default at first. In the future it can be removed. Otherwise, we end up breaking a bunch of user configurations. > > 4. Based on the version of the virt machine, the command line options > > will change. This may also be confusing. > > > > On the other hand, the second approach will be much cleaner without > > any baggage. The RISC-V eco-system is still maturing and this is the > > right time > > to start something fresh. Let's call the new machine virt-pcie for > > now. Here are a few things that can be implemented for this machine. > > > > 1. It must support versioning and any changes to the machine code must > > modify the version of the machine. > > 2. It must only support MSI based PCIe devices. No support for wired interrupts. > > The only allowed MMIO devices are > > -- mtimer/mtimecmp (there is no other option provided in ISA) > > -- reset/rtc device (If there is a way we can emulate these > > two over PCIe, that would be great) > > -- flash > > Beyond this, adding any other MMIO device must be strongly discouraged. > > 3. The virt-pcie machine must be well documented similar to a hardware > > specification (including address range, restrictions and > > implemented/allowed devices) > > 4. No dependence on virtio framework but must work with virtio-pcie backend. > > 5. Migration must be supported. > > I'm on board with these! They would all be great to have. > > I'm open to a virt-pcie, but as others have pointed out it might be > easier to just make the switch now to the general board. > > > 6. No command line option is required. > > I don't see this being achievable unfortunately > I meant no command line configurability options for the machine itself. We probably should allow using different versions of the machine via command line. > > 7. Any other ? > > > > Once we have these policies in place, this can be the preferred virt > > machine and the current virt machine can be phased out or continue to > > co-exist > > as a more flexible experimental platform. > > > > Thoughts ? > > > > > Alistair > > > > > > > > > > > With regards, > > > > Daniel > > > > -- > > > > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > > > > |: https://libvirt.org -o- https://fstop138.berrange.com :| > > > > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > > > > > > > > > > > > -- > > Regards, > > Atish
On Mon, May 23, 2022 at 08:16:40PM -0700, Atish Patra wrote: > On Sun, May 22, 2022 at 10:59 PM Alistair Francis <alistair23@gmail.com> wrote: > > On Wed, May 18, 2022 at 4:38 PM Atish Patra <atishp@atishpatra.org> wrote: > > > 1. virt machine is not well documented and already bloated. There is > > > no specification for virt machine as such. > > > Putting restrictions after a certain release will lead to confusion. > > > 2. Do we support existing MMIO devices after that specific version or not ? > > > > Yeah, so I guess this doesn't achieve the same outcome you want. I > > would say we would still include some MMIO devices, like UART for > > example. > > Why ? We can just rely on the pcie based uart (virtio-serial-pci or > serial-pci) should be enough. > The only MMIO devices that should be allowed are the ones that can't > be behind pcie. IIRC virtio-serial is initialized too late to catch messages produced very early by the firmware (and possibly the kernel), which means it's okay for regular usage but not when trying to debug an entire class of boot issues. Either way, it looks like you wouldn't be able to completely get rid of MMIO even if you introduced a new virt-pcie machine type. That's the same for the aarch64 virt machine. I agree with Dan that we should follow the example set by that architecture - it has worked out pretty well. If there is a desire to reduce the complexity of the "standard" machine type, can we just take the current virt machine type and rename it to something else? And have your simpler code take over the virt name? Sure, it will cause some pain in the short term, but the RISC-V ecosystem is still young enough for it to not be a deal breaker.
On Wed, May 25, 2022 at 1:56 AM Andrea Bolognani <abologna@redhat.com> wrote: > > On Mon, May 23, 2022 at 08:16:40PM -0700, Atish Patra wrote: > > On Sun, May 22, 2022 at 10:59 PM Alistair Francis <alistair23@gmail.com> wrote: > > > On Wed, May 18, 2022 at 4:38 PM Atish Patra <atishp@atishpatra.org> wrote: > > > > 1. virt machine is not well documented and already bloated. There is > > > > no specification for virt machine as such. > > > > Putting restrictions after a certain release will lead to confusion. > > > > 2. Do we support existing MMIO devices after that specific version or not ? > > > > > > Yeah, so I guess this doesn't achieve the same outcome you want. I > > > would say we would still include some MMIO devices, like UART for > > > example. > > > > Why ? We can just rely on the pcie based uart (virtio-serial-pci or > > serial-pci) should be enough. > > The only MMIO devices that should be allowed are the ones that can't > > be behind pcie. > > IIRC virtio-serial is initialized too late to catch messages produced > very early by the firmware (and possibly the kernel), which means > it's okay for regular usage but not when trying to debug an entire > class of boot issues. Agreed. OpenSBI doesn't even support PCIe so we need an MMIO UART for OpenSBI to be able to print messages Alistair > > Either way, it looks like you wouldn't be able to completely get rid > of MMIO even if you introduced a new virt-pcie machine type. That's > the same for the aarch64 virt machine. I agree with Dan that we > should follow the example set by that architecture - it has worked > out pretty well. > > If there is a desire to reduce the complexity of the "standard" > machine type, can we just take the current virt machine type and > rename it to something else? And have your simpler code take over the > virt name? Sure, it will cause some pain in the short term, but the > RISC-V ecosystem is still young enough for it to not be a deal > breaker. > > -- > Andrea Bolognani / Red Hat / Virtualization >