| Message ID | 1570843519-8696-3-git-send-email-linuxram@us.ibm.com (mailing list archive) |
|---|---|
| State | Rejected, archived |
| Headers | show |
| Series | virtio: Support encrypted memory on powerpc secure guests | expand |
| Context | Check | Description |
|---|---|---|
| snowpatch_ozlabs/apply_patch | success | Successfully applied on branch next (600802af9049be799465b24d14162918545634bf) |
| snowpatch_ozlabs/build-ppc64le | success | Build succeeded |
| snowpatch_ozlabs/build-ppc64be | success | Build succeeded |
| snowpatch_ozlabs/build-ppc64e | success | Build succeeded |
| snowpatch_ozlabs/build-pmac32 | success | Build succeeded |
| snowpatch_ozlabs/checkpatch | success | total: 0 errors, 0 warnings, 0 checks, 70 lines checked |
On Fri, Oct 11, 2019 at 06:25:19PM -0700, Ram Pai wrote: > From: Thiago Jung Bauermann <bauerman@linux.ibm.com> > > Normally, virtio enables DMA API with VIRTIO_F_IOMMU_PLATFORM, which must > be set by both device and guest driver. However, as a hack, when DMA API > returns physical addresses, guest driver can use the DMA API; even though > device does not set VIRTIO_F_IOMMU_PLATFORM and just uses physical > addresses. > > Doing this works-around POWER secure guests for which only the bounce > buffer is accessible to the device, but which don't set > VIRTIO_F_IOMMU_PLATFORM due to a set of hypervisor and architectural bugs. > To guard against platform changes, breaking any of these assumptions down > the road, we check at probe time and fail if that's not the case. > > cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> > cc: David Gibson <david@gibson.dropbear.id.au> > cc: Michael Ellerman <mpe@ellerman.id.au> > cc: Paul Mackerras <paulus@ozlabs.org> > cc: Michael Roth <mdroth@linux.vnet.ibm.com> > cc: Alexey Kardashevskiy <aik@linux.ibm.com> > cc: Jason Wang <jasowang@redhat.com> > cc: Christoph Hellwig <hch@lst.de> > Suggested-by: Michael S. Tsirkin <mst@redhat.com> > Signed-off-by: Ram Pai <linuxram@us.ibm.com> > Signed-off-by: Thiago Jung Bauermann <bauerman@linux.ibm.com> Reviewed-by: David Gibson <david@gibson.dropbear.id.au> I don't know that this is the most elegant solution possible. But it's simple, gets the job done and pretty unlikely to cause mysterious breakage down the road.
On Fri, Oct 11, 2019 at 06:25:19PM -0700, Ram Pai wrote: > From: Thiago Jung Bauermann <bauerman@linux.ibm.com> > > Normally, virtio enables DMA API with VIRTIO_F_IOMMU_PLATFORM, which must > be set by both device and guest driver. However, as a hack, when DMA API > returns physical addresses, guest driver can use the DMA API; even though > device does not set VIRTIO_F_IOMMU_PLATFORM and just uses physical > addresses. Sorry, but this is a complete bullshit hack. Driver must always use the DMA API if they do DMA, and if virtio devices use physical addresses that needs to be returned through the platform firmware interfaces for the dma setup. If you don't do that yet (which based on previous informations you don't), you need to fix it, and we can then quirk old implementations that already are out in the field. In other words: we finally need to fix that virtio mess and not pile hacks on top of hacks.
On Tue, Oct 15, 2019 at 09:35:01AM +0200, Christoph Hellwig wrote: > On Fri, Oct 11, 2019 at 06:25:19PM -0700, Ram Pai wrote: > > From: Thiago Jung Bauermann <bauerman@linux.ibm.com> > > > > Normally, virtio enables DMA API with VIRTIO_F_IOMMU_PLATFORM, which must > > be set by both device and guest driver. However, as a hack, when DMA API > > returns physical addresses, guest driver can use the DMA API; even though > > device does not set VIRTIO_F_IOMMU_PLATFORM and just uses physical > > addresses. > > Sorry, but this is a complete bullshit hack. Driver must always use > the DMA API if they do DMA, and if virtio devices use physical addresses > that needs to be returned through the platform firmware interfaces for > the dma setup. If you don't do that yet (which based on previous > informations you don't), you need to fix it, and we can then quirk > old implementations that already are out in the field. > > In other words: we finally need to fix that virtio mess and not pile > hacks on top of hacks. So force all virtio devices to use DMA API, except when VIRTIO_F_IOMMU_PLATFORM is not enabled? Any help detailing the idea, will enable us fix this issue once for all. Will something like below work? It removes the prior hacks, and always uses DMA API; except when VIRTIO_F_IOMMU_PLATFORM is not enabled. diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index c8be1c4..b593d3d 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -240,22 +240,10 @@ static inline bool virtqueue_use_indirect(struct virtqueue *_vq, static bool vring_use_dma_api(struct virtio_device *vdev) { - if (!virtio_has_iommu_quirk(vdev)) - return true; - - /* Otherwise, we are left to guess. */ - /* - * In theory, it's possible to have a buggy QEMU-supposed - * emulated Q35 IOMMU and Xen enabled at the same time. On - * such a configuration, virtio has never worked and will - * not work without an even larger kludge. Instead, enable - * the DMA API if we're a Xen guest, which at least allows - * all of the sensible Xen configurations to work correctly. - */ - if (xen_domain()) - return true; + if (virtio_has_iommu_quirk(vdev)) + return false; - return false; + return true; } size_t virtio_max_dma_size(struct virtio_device *vdev)
On 2019/10/15 下午3:35, Christoph Hellwig wrote: > On Fri, Oct 11, 2019 at 06:25:19PM -0700, Ram Pai wrote: >> From: Thiago Jung Bauermann <bauerman@linux.ibm.com> >> >> Normally, virtio enables DMA API with VIRTIO_F_IOMMU_PLATFORM, which must >> be set by both device and guest driver. However, as a hack, when DMA API >> returns physical addresses, guest driver can use the DMA API; even though >> device does not set VIRTIO_F_IOMMU_PLATFORM and just uses physical >> addresses. > Sorry, but this is a complete bullshit hack. Driver must always use > the DMA API if they do DMA, and if virtio devices use physical addresses > that needs to be returned through the platform firmware interfaces for > the dma setup. If you don't do that yet (which based on previous > informations you don't), you need to fix it, and we can then quirk > old implementations that already are out in the field. > > In other words: we finally need to fix that virtio mess and not pile > hacks on top of hacks. I agree, the only reason for IOMMU_PLATFORM is to make sure guest works for some old and buggy qemu when vIOMMU is enabled which seems useless (note we don't even support vIOMMU migration in that case). Thanks
On Tue, Oct 15, 2019 at 09:35:01AM +0200, Christoph Hellwig wrote: > On Fri, Oct 11, 2019 at 06:25:19PM -0700, Ram Pai wrote: > > From: Thiago Jung Bauermann <bauerman@linux.ibm.com> > > > > Normally, virtio enables DMA API with VIRTIO_F_IOMMU_PLATFORM, which must > > be set by both device and guest driver. However, as a hack, when DMA API > > returns physical addresses, guest driver can use the DMA API; even though > > device does not set VIRTIO_F_IOMMU_PLATFORM and just uses physical > > addresses. > > Sorry, but this is a complete bullshit hack. Driver must always use > the DMA API if they do DMA, and if virtio devices use physical addresses > that needs to be returned through the platform firmware interfaces for > the dma setup. If you don't do that yet (which based on previous > informations you don't), you need to fix it, and we can then quirk > old implementations that already are out in the field. > > In other words: we finally need to fix that virtio mess and not pile > hacks on top of hacks. Christoph, if I understand correctly, your objection isn't so much to the proposed change here of itself, except insofar as it entrenches virtio's existing code allowing it to either use the DMA api or bypass it and use physical addresses directly. Is that right, or have I missed something? Where do you envisage the decision to bypass the IOMMU being made? The virtio spec more or less states that virtio devices use hypervisor magic to access physical addresses directly, rather than using normal DMA channels. The F_IOMMU_PLATFORM flag then overrides that, since it obviously won't work for hardware devices. The platform code isn't really in a position to know that virtio devices are (usually) magic. So were you envisaging the virtio driver explicitly telling the platform to use bypassing DMA operations?
diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c index a977e32..77a3baf 100644 --- a/drivers/virtio/virtio.c +++ b/drivers/virtio/virtio.c @@ -4,6 +4,7 @@ #include <linux/virtio_config.h> #include <linux/module.h> #include <linux/idr.h> +#include <linux/dma-mapping.h> #include <uapi/linux/virtio_ids.h> /* Unique numbering for virtio devices. */ @@ -245,6 +246,23 @@ static int virtio_dev_probe(struct device *_d) if (err) goto err; + /* + * If memory is encrypted, but VIRTIO_F_IOMMU_PLATFORM is not set, then + * the device is broken: DMA API is required for these platforms, but + * the only way using the DMA API is going to work at all is if the + * device is ready for it. So we need a flag on the virtio device, + * exposed by the hypervisor (or hardware for hw virtio devices) that + * says: hey, I'm real, don't take a shortcut. + * + * There's one exception where guest can make things work, and that is + * when DMA API is guaranteed to always return physical addresses. + */ + if (mem_encrypt_active() && !virtio_can_use_dma_api(dev)) { + dev_err(_d, "virtio: device unable to access encrypted memory\n"); + err = -EINVAL; + goto err; + } + err = drv->probe(dev); if (err) goto err; diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index c8be1c4..9c56b61 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -255,6 +255,14 @@ static bool vring_use_dma_api(struct virtio_device *vdev) if (xen_domain()) return true; + /* + * Also, if guest memory is encrypted the host can't access it + * directly. We need to either use an IOMMU or do bounce buffering. + * Both are done via the DMA API. + */ + if (mem_encrypt_active() && virtio_can_use_dma_api(vdev)) + return true; + return false; } diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h index bb4cc49..57bc25c 100644 --- a/include/linux/virtio_config.h +++ b/include/linux/virtio_config.h @@ -4,6 +4,7 @@ #include <linux/err.h> #include <linux/bug.h> +#include <linux/dma-mapping.h> #include <linux/virtio.h> #include <linux/virtio_byteorder.h> #include <uapi/linux/virtio_config.h> @@ -174,6 +175,19 @@ static inline bool virtio_has_iommu_quirk(const struct virtio_device *vdev) return !virtio_has_feature(vdev, VIRTIO_F_IOMMU_PLATFORM); } +/** + * virtio_can_use_dma_api - determine whether the DMA API can be used + * @vdev: the device + * + * The DMA API can be used either when the device doesn't have the IOMMU quirk, + * or when the DMA API is guaranteed to always return physical addresses. + */ +static inline bool virtio_can_use_dma_api(const struct virtio_device *vdev) +{ + return !virtio_has_iommu_quirk(vdev) || + dma_addr_is_phys_addr(vdev->dev.parent); +} + static inline struct virtqueue *virtio_find_single_vq(struct virtio_device *vdev, vq_callback_t *c, const char *n)