Message ID | 20190507062559.20295-2-aik@ozlabs.ru (mailing list archive) |
---|---|
State | Accepted |
Commit | efd176a04bef41aab5b3087e977fea2b69915174 |
Headers | show |
Series | pseries: Enable SWIOTLB | expand |
Alexey Kardashevskiy <aik@ozlabs.ru> writes: > The commit 8617a5c5bc00 ("powerpc/dma: handle iommu bypass in > dma_iommu_ops") merged direct DMA ops into the IOMMU DMA ops allowing > SWIOTLB as well but only for mapping; the unmapping and bouncing parts > were left unmodified. > > This adds missing direct unmapping calls to .unmap_page() and .unmap_sg(). > > This adds missing sync callbacks and directs them to the direct DMA hooks. > > Fixes: 8617a5c5bc00 (powerpc/dma: handle iommu bypass in dma_iommu_ops) > Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> Nice! Thanks for working on this. I have the patch at the end of this email to get virtio-scsi-pci and virtio-blk-pci working in a secure guest. I applied your patch and reverted my patch and unfortunately the guest hangs right after mounting the disk: [ 0.185659] virtio-pci 0000:00:04.0: enabling device (0100 -> 0102) [ 0.187082] virtio-pci 0000:00:04.0: ibm,query-pe-dma-windows(2026) 2000 8000000 20000000 returned 0 [ 0.187497] virtio-pci 0000:00:04.0: ibm,create-pe-dma-window(2027) 2000 8000000 20000000 10 20 returned 0 (liobn = 0x80000001 startin g addr = 8000000 0) [ 0.226654] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled [ 0.227094] Non-volatile memory driver v1.3 [ 0.228950] brd: module loaded [ 0.230666] loop: module loaded [ 0.230773] ipr: IBM Power RAID SCSI Device Driver version: 2.6.4 (March 14, 2017) [ 0.233323] scsi host0: Virtio SCSI HBA [ 0.235439] scsi 0:0:0:0: Direct-Access QEMU QEMU HARDDISK 2.5+ PQ: 0 ANSI: 5 [ 0.369009] random: fast init done [ 0.370819] sd 0:0:0:0: Attached scsi generic sg0 type 0 [ 0.371320] sd 0:0:0:0: Power-on or device reset occurred <snip> [ 0.380378] sd 0:0:0:0: [sda] 31457280 512-byte logical blocks: (16.1 GB/15.0 GiB) [ 0.381102] sd 0:0:0:0: [sda] Write Protect is off [ 0.381195] sd 0:0:0:0: [sda] Mode Sense: 63 00 00 08 [ 0.382436] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 0.383630] sd 0:0:0:0: [sda] Optimal transfer size 0 bytes < PAGE_SIZE (65536 bytes) [ 0.391562] sda: sda1 sda2 [ 0.398101] sd 0:0:0:0: [sda] Attached SCSI disk [ 0.398205] md: Waiting for all devices to be available before autodetect [ 0.398318] md: If you don't use raid, use raid=noautodetect [ 0.398515] md: Autodetecting RAID arrays. [ 0.398585] md: autorun ... [ 0.398631] md: ... autorun DONE. [ 0.403552] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null) [ 0.403700] VFS: Mounted root (ext4 filesystem) readonly on device 8:2. [ 0.405258] devtmpfs: mounted [ 0.406427] Freeing unused kernel memory: 4224K [ 0.406519] This architecture does not have kernel memory protection. [ 0.406633] Run /sbin/init as init process Sorry, I don't have any information on where the guest is stuck. I tried <sysrq>+l, <sysrq>+t and <sysrq>+w but nothing out of the ordinary showed up. Will try something else later. -- Thiago Jung Bauermann IBM Linux Technology Center From 70d2fba809119ae2d35c9ca4269405bb5c28413a Mon Sep 17 00:00:00 2001 From: Thiago Jung Bauermann <bauerman@linux.ibm.com> Date: Thu, 24 Jan 2019 22:40:16 -0200 Subject: [PATCH 1/1] powerpc/pseries/iommu: Don't use dma_iommu_ops on secure guests Secure guest memory is inacessible to devices so regular DMA isn't possible. In that case set devices' dma_map_ops to NULL so that the generic DMA code path will use SWIOTLB and DMA to bounce buffers. Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com> --- arch/powerpc/platforms/pseries/iommu.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c index 36eb1ddbac69..1636306007eb 100644 --- a/arch/powerpc/platforms/pseries/iommu.c +++ b/arch/powerpc/platforms/pseries/iommu.c @@ -50,6 +50,7 @@ #include <asm/udbg.h> #include <asm/mmzone.h> #include <asm/plpar_wrappers.h> +#include <asm/svm.h> #include "pseries.h" @@ -1335,7 +1336,10 @@ void iommu_init_early_pSeries(void) of_reconfig_notifier_register(&iommu_reconfig_nb); register_memory_notifier(&iommu_mem_nb); - set_pci_dma_ops(&dma_iommu_ops); + if (is_secure_guest()) + set_pci_dma_ops(NULL); + else + set_pci_dma_ops(&dma_iommu_ops); } static int __init disable_multitce(char *str)
On 11/05/2019 08:36, Thiago Jung Bauermann wrote: > > Alexey Kardashevskiy <aik@ozlabs.ru> writes: > >> The commit 8617a5c5bc00 ("powerpc/dma: handle iommu bypass in >> dma_iommu_ops") merged direct DMA ops into the IOMMU DMA ops allowing >> SWIOTLB as well but only for mapping; the unmapping and bouncing parts >> were left unmodified. >> >> This adds missing direct unmapping calls to .unmap_page() and .unmap_sg(). >> >> This adds missing sync callbacks and directs them to the direct DMA hooks. >> >> Fixes: 8617a5c5bc00 (powerpc/dma: handle iommu bypass in dma_iommu_ops) >> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> > > Nice! Thanks for working on this. I have the patch at the end of this > email to get virtio-scsi-pci and virtio-blk-pci working in a secure > guest. I saw the set_pci_dma_ops(NULL) patch but could not figure out how pass NULL there sets the DMA ops to direct. > > I applied your patch and reverted my patch and unfortunately the guest > hangs right after mounting the disk: Have you applied it on upstream kernel? I cannot see how it affects current guests as it is... > > [ 0.185659] virtio-pci 0000:00:04.0: enabling device (0100 -> 0102) > [ 0.187082] virtio-pci 0000:00:04.0: ibm,query-pe-dma-windows(2026) 2000 8000000 20000000 returned 0 > [ 0.187497] virtio-pci 0000:00:04.0: ibm,create-pe-dma-window(2027) 2000 8000000 20000000 10 20 returned 0 (liobn = 0x80000001 startin > g addr = 8000000 0) > [ 0.226654] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled > [ 0.227094] Non-volatile memory driver v1.3 > [ 0.228950] brd: module loaded > [ 0.230666] loop: module loaded > [ 0.230773] ipr: IBM Power RAID SCSI Device Driver version: 2.6.4 (March 14, 2017) > [ 0.233323] scsi host0: Virtio SCSI HBA > [ 0.235439] scsi 0:0:0:0: Direct-Access QEMU QEMU HARDDISK 2.5+ PQ: 0 ANSI: 5 > [ 0.369009] random: fast init done > [ 0.370819] sd 0:0:0:0: Attached scsi generic sg0 type 0 > [ 0.371320] sd 0:0:0:0: Power-on or device reset occurred > > <snip> > > [ 0.380378] sd 0:0:0:0: [sda] 31457280 512-byte logical blocks: (16.1 GB/15.0 GiB) > [ 0.381102] sd 0:0:0:0: [sda] Write Protect is off > [ 0.381195] sd 0:0:0:0: [sda] Mode Sense: 63 00 00 08 > [ 0.382436] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA > [ 0.383630] sd 0:0:0:0: [sda] Optimal transfer size 0 bytes < PAGE_SIZE (65536 bytes) > [ 0.391562] sda: sda1 sda2 > [ 0.398101] sd 0:0:0:0: [sda] Attached SCSI disk > [ 0.398205] md: Waiting for all devices to be available before autodetect > [ 0.398318] md: If you don't use raid, use raid=noautodetect > [ 0.398515] md: Autodetecting RAID arrays. > [ 0.398585] md: autorun ... > [ 0.398631] md: ... autorun DONE. > [ 0.403552] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null) > [ 0.403700] VFS: Mounted root (ext4 filesystem) readonly on device 8:2. > [ 0.405258] devtmpfs: mounted > [ 0.406427] Freeing unused kernel memory: 4224K > [ 0.406519] This architecture does not have kernel memory protection. > [ 0.406633] Run /sbin/init as init process > > Sorry, I don't have any information on where the guest is stuck. I tried > <sysrq>+l, <sysrq>+t and <sysrq>+w but nothing out of the ordinary > showed up. Will try something else later. > > -- > Thiago Jung Bauermann > IBM Linux Technology Center > > > > From 70d2fba809119ae2d35c9ca4269405bb5c28413a Mon Sep 17 00:00:00 2001 > From: Thiago Jung Bauermann <bauerman@linux.ibm.com> > Date: Thu, 24 Jan 2019 22:40:16 -0200 > Subject: [PATCH 1/1] powerpc/pseries/iommu: Don't use dma_iommu_ops on secure > guests > > Secure guest memory is inacessible to devices so regular DMA isn't > possible. > > In that case set devices' dma_map_ops to NULL so that the generic > DMA code path will use SWIOTLB and DMA to bounce buffers. > > Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com> > --- > arch/powerpc/platforms/pseries/iommu.c | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c > index 36eb1ddbac69..1636306007eb 100644 > --- a/arch/powerpc/platforms/pseries/iommu.c > +++ b/arch/powerpc/platforms/pseries/iommu.c > @@ -50,6 +50,7 @@ > #include <asm/udbg.h> > #include <asm/mmzone.h> > #include <asm/plpar_wrappers.h> > +#include <asm/svm.h> > > #include "pseries.h" > > @@ -1335,7 +1336,10 @@ void iommu_init_early_pSeries(void) > of_reconfig_notifier_register(&iommu_reconfig_nb); > register_memory_notifier(&iommu_mem_nb); > > - set_pci_dma_ops(&dma_iommu_ops); > + if (is_secure_guest()) > + set_pci_dma_ops(NULL); > + else > + set_pci_dma_ops(&dma_iommu_ops); > } > > static int __init disable_multitce(char *str) >
Alexey Kardashevskiy <aik@ozlabs.ru> writes: > On 11/05/2019 08:36, Thiago Jung Bauermann wrote: >> >> Alexey Kardashevskiy <aik@ozlabs.ru> writes: >> >>> The commit 8617a5c5bc00 ("powerpc/dma: handle iommu bypass in >>> dma_iommu_ops") merged direct DMA ops into the IOMMU DMA ops allowing >>> SWIOTLB as well but only for mapping; the unmapping and bouncing parts >>> were left unmodified. >>> >>> This adds missing direct unmapping calls to .unmap_page() and .unmap_sg(). >>> >>> This adds missing sync callbacks and directs them to the direct DMA hooks. >>> >>> Fixes: 8617a5c5bc00 (powerpc/dma: handle iommu bypass in dma_iommu_ops) >>> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> >> >> Nice! Thanks for working on this. I have the patch at the end of this >> email to get virtio-scsi-pci and virtio-blk-pci working in a secure >> guest. > > I saw the set_pci_dma_ops(NULL) patch but could not figure out how pass > NULL there sets the DMA ops to direct. That causes pcibios_setup_device() to call set_dma_ops(&dev->dev, NULL), which in turn causes dma_is_direct(get_dma_ops(dev)) to return true. >> I applied your patch and reverted my patch and unfortunately the guest >> hangs right after mounting the disk: > > Have you applied it on upstream kernel? I cannot see how it affects > current guests as it is... I applied it on a branch containing both Claudio Carvalho's "kvmppc: Paravirtualize KVM to support ultravisor" series as well as my "Secure Virtual Machine Enablement" patch series. https://lore.kernel.org/linuxppc-dev/20190518142524.28528-1-cclaudio@linux.ibm.com/ https://lore.kernel.org/linuxppc-dev/20190521044912.1375-1-bauerman@linux.ibm.com/ -- Thiago Jung Bauermann IBM Linux Technology Center
On Tue, 2019-05-07 at 06:25:58 UTC, Alexey Kardashevskiy wrote: > The commit 8617a5c5bc00 ("powerpc/dma: handle iommu bypass in > dma_iommu_ops") merged direct DMA ops into the IOMMU DMA ops allowing > SWIOTLB as well but only for mapping; the unmapping and bouncing parts > were left unmodified. > > This adds missing direct unmapping calls to .unmap_page() and .unmap_sg(). > > This adds missing sync callbacks and directs them to the direct DMA hooks. > > Fixes: 8617a5c5bc00 (powerpc/dma: handle iommu bypass in dma_iommu_ops) > Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> > Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com> Series applied to powerpc next, thanks. https://git.kernel.org/powerpc/c/efd176a04bef41aab5b3087e977fea2b69915174 cheers
diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c index 09231ef06d01..92b318df1aa1 100644 --- a/arch/powerpc/kernel/dma-iommu.c +++ b/arch/powerpc/kernel/dma-iommu.c @@ -82,6 +82,8 @@ static void dma_iommu_unmap_page(struct device *dev, dma_addr_t dma_handle, if (!dma_iommu_map_bypass(dev, attrs)) iommu_unmap_page(get_iommu_table_base(dev), dma_handle, size, direction, attrs); + else + dma_direct_unmap_page(dev, dma_handle, size, direction, attrs); } @@ -102,6 +104,8 @@ static void dma_iommu_unmap_sg(struct device *dev, struct scatterlist *sglist, if (!dma_iommu_map_bypass(dev, attrs)) ppc_iommu_unmap_sg(get_iommu_table_base(dev), sglist, nelems, direction, attrs); + else + dma_direct_unmap_sg(dev, sglist, nelems, direction, attrs); } static bool dma_iommu_bypass_supported(struct device *dev, u64 mask) @@ -163,6 +167,34 @@ u64 dma_iommu_get_required_mask(struct device *dev) return mask; } +static void dma_iommu_sync_for_cpu(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir) +{ + if (dma_iommu_alloc_bypass(dev)) + dma_direct_sync_single_for_cpu(dev, addr, size, dir); +} + +static void dma_iommu_sync_for_device(struct device *dev, dma_addr_t addr, + size_t sz, enum dma_data_direction dir) +{ + if (dma_iommu_alloc_bypass(dev)) + dma_direct_sync_single_for_device(dev, addr, sz, dir); +} + +extern void dma_iommu_sync_sg_for_cpu(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir) +{ + if (dma_iommu_alloc_bypass(dev)) + dma_direct_sync_sg_for_cpu(dev, sgl, nents, dir); +} + +extern void dma_iommu_sync_sg_for_device(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir) +{ + if (dma_iommu_alloc_bypass(dev)) + dma_direct_sync_sg_for_device(dev, sgl, nents, dir); +} + const struct dma_map_ops dma_iommu_ops = { .alloc = dma_iommu_alloc_coherent, .free = dma_iommu_free_coherent, @@ -172,4 +204,8 @@ const struct dma_map_ops dma_iommu_ops = { .map_page = dma_iommu_map_page, .unmap_page = dma_iommu_unmap_page, .get_required_mask = dma_iommu_get_required_mask, + .sync_single_for_cpu = dma_iommu_sync_for_cpu, + .sync_single_for_device = dma_iommu_sync_for_device, + .sync_sg_for_cpu = dma_iommu_sync_sg_for_cpu, + .sync_sg_for_device = dma_iommu_sync_sg_for_device, };
The commit 8617a5c5bc00 ("powerpc/dma: handle iommu bypass in dma_iommu_ops") merged direct DMA ops into the IOMMU DMA ops allowing SWIOTLB as well but only for mapping; the unmapping and bouncing parts were left unmodified. This adds missing direct unmapping calls to .unmap_page() and .unmap_sg(). This adds missing sync callbacks and directs them to the direct DMA hooks. Fixes: 8617a5c5bc00 (powerpc/dma: handle iommu bypass in dma_iommu_ops) Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru> --- arch/powerpc/kernel/dma-iommu.c | 36 +++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)