| Message ID | 1572902923-8096-1-git-send-email-linuxram@us.ibm.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Enable IOMMU support for pseries Secure VMs | expand |
On 05/11/2019 08:28, Ram Pai wrote: > This patch series enables IOMMU support for pseries Secure VMs. > > > Tested using QEMU command line option: > > "-device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4, > iommu_platform=on,disable-modern=off,disable-legacy=on" > and > > "-device virtio-blk-pci,scsi=off,bus=pci.0, > addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0, > iommu_platform=on,disable-modern=off,disable-legacy=on" Worth mentioning that SLOF won't boot with such devices as SLOF does not know about iommu_platform=on. > > Ram Pai (2): > powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor. > powerpc/pseries/iommu: Use dma_iommu_ops for Secure VMs aswell. > > arch/powerpc/platforms/pseries/iommu.c | 30 ++++++++++++++++++------------ > 1 file changed, 18 insertions(+), 12 deletions(-) >
On Wed, Nov 06, 2019 at 12:59:50PM +1100, Alexey Kardashevskiy wrote: > > > On 05/11/2019 08:28, Ram Pai wrote: > > This patch series enables IOMMU support for pseries Secure VMs. > > > > > > Tested using QEMU command line option: > > > > "-device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4, > > iommu_platform=on,disable-modern=off,disable-legacy=on" > > and > > > > "-device virtio-blk-pci,scsi=off,bus=pci.0, > > addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0, > > iommu_platform=on,disable-modern=off,disable-legacy=on" > > > Worth mentioning that SLOF won't boot with such devices as SLOF does not know about iommu_platform=on. I did see SLOF complaining, but that did not stop it from booting the guest. My boot device was a virtio device, with iommu platform flag enabled. So I am not sure, under what conditions SLOF will fail to boot. Any idea? RP
On Wed, Nov 06, 2019 at 12:59:50PM +1100, Alexey Kardashevskiy wrote: > > > On 05/11/2019 08:28, Ram Pai wrote: > > This patch series enables IOMMU support for pseries Secure VMs. > > > > > > Tested using QEMU command line option: > > > > "-device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4, > > iommu_platform=on,disable-modern=off,disable-legacy=on" > > and > > > > "-device virtio-blk-pci,scsi=off,bus=pci.0, > > addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0, > > iommu_platform=on,disable-modern=off,disable-legacy=on" > > > Worth mentioning that SLOF won't boot with such devices as SLOF does not know about iommu_platform=on. Shouldn't be hard to support: set up the iommu to allow everything and ack the feature. Right? > > > > Ram Pai (2): > > powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor. > > powerpc/pseries/iommu: Use dma_iommu_ops for Secure VMs aswell. > > > > arch/powerpc/platforms/pseries/iommu.c | 30 ++++++++++++++++++------------ > > 1 file changed, 18 insertions(+), 12 deletions(-) > > > > -- > Alexey
Quoting Michael S. Tsirkin (2019-11-06 12:06:37) > On Wed, Nov 06, 2019 at 12:59:50PM +1100, Alexey Kardashevskiy wrote: > > > > > > On 05/11/2019 08:28, Ram Pai wrote: > > > This patch series enables IOMMU support for pseries Secure VMs. > > > > > > > > > Tested using QEMU command line option: > > > > > > "-device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4, > > > iommu_platform=on,disable-modern=off,disable-legacy=on" > > > and > > > > > > "-device virtio-blk-pci,scsi=off,bus=pci.0, > > > addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0, > > > iommu_platform=on,disable-modern=off,disable-legacy=on" > > > > > > Worth mentioning that SLOF won't boot with such devices as SLOF does not know about iommu_platform=on. > > Shouldn't be hard to support: set up the iommu to allow everything > and ack the feature. Right? It's not a static/linear mapping in this case so we need calls to map DMA buffers as-needed. I've gotten it to boot with virtio-blk, but the patches have some hacks and need cleanup, hoping to post them soon. I'm a bit perplexed how we would manage to boot a guest without those changes though, this is what I get with qemu 4.1.0: qemu-system-ppc64 -M pseries,ic-mode=xics -m 512M -bios /home/mdroth/w/build/qemu-4.1.0-build/pc-bios/slof.bin -device virtio-blk-pci,drive=drive0,id=blk0,disable-modern=off,disable-legacy=on,iommu_platform=on -drive file=/home/mdroth/vm/bionic-server-cloudimg-ppc64el.img,if=none,id=drive0 -trace enable=spapr_iommu\*,file=trace.out -monitor unix:/tmp/mon.sock,server,nowait -vga none -nographic qemu-system-ppc64: warning: TCG doesn't support requested feature, cap-cfpc=workaround qemu-system-ppc64: warning: TCG doesn't support requested feature, cap-sbbc=workaround qemu-system-ppc64: warning: TCG doesn't support requested feature, cap-ibs=workaround SLOF ********************************************************************** QEMU Starting Build Date = Jul 3 2019 12:26:14 FW Version = git-ba1ab360eebe6338 Press "s" to enter Open Firmware. Populating /vdevice methods Populating /vdevice/vty@71000000 Populating /vdevice/nvram@71000001 Populating /vdevice/l-lan@71000002 Populating /vdevice/v-scsi@71000003 SCSI: Looking for devices 8200000000000000 CD-ROM : "QEMU QEMU CD-ROM 2.5+" Populating /pci@800000020000000 00 0000 (D) : 1af4 1042 virtio [ block ] No NVRAM common partition, re-initializing... Scanning USB Using default console: /vdevice/vty@71000000 Welcome to Open Firmware Copyright (c) 2004, 2017 IBM Corporation All rights reserved. This program and the accompanying materials are made available under the terms of the BSD License available at http://www.opensource.org/licenses/bsd-license.php Trying to load: from: /pci@800000020000000/scsi@0 ... virtioblk_init: failed virtioblk_transfer: Access beyond end of device! And then it hangs. This is with TCG so maybe it behaves differently with KVM, but that's the result I would expect with the current SLOF code. > > > > > > > Ram Pai (2): > > > powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor. > > > powerpc/pseries/iommu: Use dma_iommu_ops for Secure VMs aswell. > > > > > > arch/powerpc/platforms/pseries/iommu.c | 30 ++++++++++++++++++------------ > > > 1 file changed, 18 insertions(+), 12 deletions(-) > > > > > > > -- > > Alexey >
On Wed, Nov 06, 2019 at 12:52:02PM -0600, Michael Roth wrote: > Quoting Michael S. Tsirkin (2019-11-06 12:06:37) > > On Wed, Nov 06, 2019 at 12:59:50PM +1100, Alexey Kardashevskiy wrote: > > > > > > > > > On 05/11/2019 08:28, Ram Pai wrote: > > > > This patch series enables IOMMU support for pseries Secure VMs. > > > > > > > > > > > > Tested using QEMU command line option: > > > > > > > > "-device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4, > > > > iommu_platform=on,disable-modern=off,disable-legacy=on" > > > > and > > > > > > > > "-device virtio-blk-pci,scsi=off,bus=pci.0, > > > > addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0, > > > > iommu_platform=on,disable-modern=off,disable-legacy=on" > > > > > > > > > Worth mentioning that SLOF won't boot with such devices as SLOF does not know about iommu_platform=on. > > > > Shouldn't be hard to support: set up the iommu to allow everything > > and ack the feature. Right? > > It's not a static/linear mapping in this case so we need calls to map DMA > buffers as-needed. I've gotten it to boot with virtio-blk, but the patches > have some hacks and need cleanup, hoping to post them soon. > > I'm a bit perplexed how we would manage to boot a guest without those > changes though, this is what I get with qemu 4.1.0: > > qemu-system-ppc64 -M pseries,ic-mode=xics -m 512M -bios /home/mdroth/w/build/qemu-4.1.0-build/pc-bios/slof.bin -device virtio-blk-pci,drive=drive0,id=blk0,disable-modern=off,disable-legacy=on,iommu_platform=on -drive file=/home/mdroth/vm/bionic-server-cloudimg-ppc64el.img,if=none,id=drive0 -trace enable=spapr_iommu\*,file=trace.out -monitor unix:/tmp/mon.sock,server,nowait -vga none -nographic > qemu-system-ppc64: warning: TCG doesn't support requested feature, cap-cfpc=workaround > qemu-system-ppc64: warning: TCG doesn't support requested feature, cap-sbbc=workaround > qemu-system-ppc64: warning: TCG doesn't support requested feature, cap-ibs=workaround > > > SLOF ********************************************************************** > QEMU Starting > Build Date = Jul 3 2019 12:26:14 > FW Version = git-ba1ab360eebe6338 > Press "s" to enter Open Firmware. > > Populating /vdevice methods > Populating /vdevice/vty@71000000 > Populating /vdevice/nvram@71000001 > Populating /vdevice/l-lan@71000002 > Populating /vdevice/v-scsi@71000003 > SCSI: Looking for devices > 8200000000000000 CD-ROM : "QEMU QEMU CD-ROM 2.5+" > Populating /pci@800000020000000 > 00 0000 (D) : 1af4 1042 virtio [ block ] > No NVRAM common partition, re-initializing... > Scanning USB > Using default console: /vdevice/vty@71000000 > > Welcome to Open Firmware > > Copyright (c) 2004, 2017 IBM Corporation All rights reserved. > This program and the accompanying materials are made available > under the terms of the BSD License available at > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.opensource.org_licenses_bsd-2Dlicense.php&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=m-UrKChQVkZtnPpjbF6YY99NbT8FBByQ-E-ygV8luxw&m=-xG4gaWE7BANKGOFx0wmF5ZgZVd8A1r-tsN44n4JUW4&s=QcpPgRqeZAk1pICsA-kk2gNKKiMJLASiPVi-hPinur0&e= > > > Trying to load: from: /pci@800000020000000/scsi@0 ... virtioblk_init: failed > virtioblk_transfer: Access beyond end of device! > > And then it hangs. This is with TCG so maybe it behaves differently with > KVM, but that's the result I would expect with the current SLOF code. you are right. In my case, I was providing the kernel image on the qemu command line. So there was no need for SLOF to read the virtio disk. Hence it continued without hanging. SLOF needs to be enhanced to support IOMMU. RP
On 07/11/2019 05:06, Michael S. Tsirkin wrote: > On Wed, Nov 06, 2019 at 12:59:50PM +1100, Alexey Kardashevskiy wrote: >> >> >> On 05/11/2019 08:28, Ram Pai wrote: >>> This patch series enables IOMMU support for pseries Secure VMs. >>> >>> >>> Tested using QEMU command line option: >>> >>> "-device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4, >>> iommu_platform=on,disable-modern=off,disable-legacy=on" >>> and >>> >>> "-device virtio-blk-pci,scsi=off,bus=pci.0, >>> addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0, >>> iommu_platform=on,disable-modern=off,disable-legacy=on" >> >> >> Worth mentioning that SLOF won't boot with such devices as SLOF does not know about iommu_platform=on. > > Shouldn't be hard to support: set up the iommu to allow everything > and ack the feature. Right? With the defaults we have (32bit window limited by 1GB and SLOF residing close to the end of the second GB), it is not straight forward; also SLOF's XHCI and E1000 already use IOMMU so we should just do this for virtio as well, I am hacking it now. Thanks, > >>> >>> Ram Pai (2): >>> powerpc/pseries/iommu: Share the per-cpu TCE page with the hypervisor. >>> powerpc/pseries/iommu: Use dma_iommu_ops for Secure VMs aswell. >>> >>> arch/powerpc/platforms/pseries/iommu.c | 30 ++++++++++++++++++------------ >>> 1 file changed, 18 insertions(+), 12 deletions(-) >>> >> >> -- >> Alexey >