Message ID | 20170829214929.31136.21144.stgit@gimli.home |
---|---|
Headers | show |
Series | vfio/pci: Add NVIDIA GPUDirect P2P clique support | expand |
There seem to be some bugs in these patches, causing my VM failed to boot. Test case: 0. Merge these 3 patches in to release 2.10.1 1. qemu-system-x86_64_2.10.1 ... \ -device vfio-pci,host=04:00.0 \ -device vfio-pci,host=05:00.0 \ -device vfio-pci,host=08:00.0 \ -device vfio-pci,host=09:00.0 \ -device vfio-pci,host=85:00.0 \ -device vfio-pci,host=86:00.0 \ -device vfio-pci,host=89:00.0 \ -device vfio-pci,host=8a:00.0 ... The guest was able to boot up. 2. qemu-system-x86_64_2.10.1 ... \ -device vfio-pci,host=04:00.0,x-nv-gpudirect-clique=0 \ -device vfio-pci,host=05:00.0,x-nv-gpudirect-clique=0 \ -device vfio-pci,host=08:00.0,x-nv-gpudirect-clique=0 \ -device vfio-pci,host=09:00.0,x-nv-gpudirect-clique=0 \ -device vfio-pci,host=85:00.0,x-nv-gpudirect-clique=8 \ -device vfio-pci,host=86:00.0,x-nv-gpudirect-clique=8 \ -device vfio-pci,host=89:00.0,x-nv-gpudirect-clique=8 \ -device vfio-pci,host=8a:00.0,x-nv-gpudirect-clique=8 \ Hang. VNC couldn't connect. My personal patch used to work, although it was done by straightforward hacking and not that friendly to read. --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@static int vfio_initfn(PCIDevice *pdev) vfio_add_emulated_long(vdev, 0xc8, 0x50080009, ~0); if (count < 4) { vfio_add_emulated_long(vdev, 0xcc, 0x00005032, ~0); } else { vfio_add_emulated_long(vdev, 0xcc, 0x00085032, ~0); } vfio_add_emulated_word(vdev, 0x78, 0xc810, ~0); 2017-08-30 6:05 GMT+08:00 Alex Williamson <alex.williamson@redhat.com>: > NVIDIA has a specification for exposing a virtual vendor capability > which provides a hint to guest drivers as to which sets of GPUs can > support direct peer-to-peer DMA. Devices with the same clique ID are > expected to support this. The user can specify a clique ID for an > NVIDIA graphics device using the new vfio-pci x-nv-gpudirect-clique= > option, where valid clique IDs are a 4-bit integer. It's entirely the > user's responsibility to specify sets of devices for which P2P works > correctly and provides some benefit. This is only useful for DMA > between NVIDIA GPUs, therefore it's only useful to specify cliques > comprised of more than one GPU. Furthermore, this does not enable DMA > between VMs, there is no change to VM DMA mapping, this only exposes > hints about existing DMA paths to the guest driver. Thanks, > > Alex > > --- > > Alex Williamson (3): > vfio/pci: Do not unwind on error > vfio/pci: Add virtual capabilities quirk infrastructure > vfio/pci: Add NVIDIA GPUDirect Cliques support > > > hw/vfio/pci-quirks.c | 114 ++++++++++++++++++++++++++++++ > ++++++++++++++++++++ > hw/vfio/pci.c | 17 +++++++ > hw/vfio/pci.h | 4 ++ > 3 files changed, 133 insertions(+), 2 deletions(-) >
It's a mistake, please ignore. This patch is able to work. 2017-10-26 18:45 GMT+08:00 Bob Chen <a175818323@gmail.com>: > There seem to be some bugs in these patches, causing my VM failed to boot. > > Test case: > > 0. Merge these 3 patches in to release 2.10.1 > > 1. qemu-system-x86_64_2.10.1 ... \ > -device vfio-pci,host=04:00.0 \ > -device vfio-pci,host=05:00.0 \ > -device vfio-pci,host=08:00.0 \ > -device vfio-pci,host=09:00.0 \ > -device vfio-pci,host=85:00.0 \ > -device vfio-pci,host=86:00.0 \ > -device vfio-pci,host=89:00.0 \ > -device vfio-pci,host=8a:00.0 ... > > The guest was able to boot up. > > 2. qemu-system-x86_64_2.10.1 ... \ > -device vfio-pci,host=04:00.0,x-nv-gpudirect-clique=0 \ > -device vfio-pci,host=05:00.0,x-nv-gpudirect-clique=0 \ > -device vfio-pci,host=08:00.0,x-nv-gpudirect-clique=0 \ > -device vfio-pci,host=09:00.0,x-nv-gpudirect-clique=0 \ > -device vfio-pci,host=85:00.0,x-nv-gpudirect-clique=8 \ > -device vfio-pci,host=86:00.0,x-nv-gpudirect-clique=8 \ > -device vfio-pci,host=89:00.0,x-nv-gpudirect-clique=8 \ > -device vfio-pci,host=8a:00.0,x-nv-gpudirect-clique=8 \ > > Hang. VNC couldn't connect. > > > My personal patch used to work, although it was done by straightforward > hacking and not that friendly to read. > > --- a/hw/vfio/pci.c > +++ b/hw/vfio/pci.c > > @@static int vfio_initfn(PCIDevice *pdev) > > vfio_add_emulated_long(vdev, 0xc8, 0x50080009, ~0); > if (count < 4) > { > vfio_add_emulated_long(vdev, 0xcc, 0x00005032, ~0); > } > else > { > vfio_add_emulated_long(vdev, 0xcc, 0x00085032, ~0); > } > vfio_add_emulated_word(vdev, 0x78, 0xc810, ~0); > > 2017-08-30 6:05 GMT+08:00 Alex Williamson <alex.williamson@redhat.com>: > >> NVIDIA has a specification for exposing a virtual vendor capability >> which provides a hint to guest drivers as to which sets of GPUs can >> support direct peer-to-peer DMA. Devices with the same clique ID are >> expected to support this. The user can specify a clique ID for an >> NVIDIA graphics device using the new vfio-pci x-nv-gpudirect-clique= >> option, where valid clique IDs are a 4-bit integer. It's entirely the >> user's responsibility to specify sets of devices for which P2P works >> correctly and provides some benefit. This is only useful for DMA >> between NVIDIA GPUs, therefore it's only useful to specify cliques >> comprised of more than one GPU. Furthermore, this does not enable DMA >> between VMs, there is no change to VM DMA mapping, this only exposes >> hints about existing DMA paths to the guest driver. Thanks, >> >> Alex >> >> --- >> >> Alex Williamson (3): >> vfio/pci: Do not unwind on error >> vfio/pci: Add virtual capabilities quirk infrastructure >> vfio/pci: Add NVIDIA GPUDirect Cliques support >> >> >> hw/vfio/pci-quirks.c | 114 ++++++++++++++++++++++++++++++ >> ++++++++++++++++++++ >> hw/vfio/pci.c | 17 +++++++ >> hw/vfio/pci.h | 4 ++ >> 3 files changed, 133 insertions(+), 2 deletions(-) >> > >