vfio-pci: PCI_DEV_FLAGS_ASSIGNED flag not set when PCI device is assigned

A number of PCI device drivers supporting SR-IOV use pci_vfs_assigned() to
check if there are any VF devices assigned by a VMM prior to disabling
SR-IOV (i.e. bnx2x, be2net, fm10k, i40e, igb, ixgbe, qlcnic).  This check
works fine with the legacy device assignment (pci-stub enables the device)
which calls pci_set_dev_assigned().  The newer VFIO-based assignment
(vfio-pci enables device) doesn't call pci_set_dev_assigned() potentially
leading to issues in those drivers when disabling SR-IOV with VFs assigned.

Add calls to pci_[set|clear]_dev_assigned() to set the flag as expected.

Signed-off-by: Bruce Allan <bruce.w.allan@intel.com>
---
 drivers/vfio/pci/vfio_pci.c |    4 ++++
 1 file changed, 4 insertions(+)

On Tue, 2015-05-12 at 18:35 -0700, Bruce Allan wrote:
> A number of PCI device drivers supporting SR-IOV use pci_vfs_assigned() to
> check if there are any VF devices assigned by a VMM prior to disabling
> SR-IOV (i.e. bnx2x, be2net, fm10k, i40e, igb, ixgbe, qlcnic).  This check
> works fine with the legacy device assignment (pci-stub enables the device)
> which calls pci_set_dev_assigned().  The newer VFIO-based assignment
> (vfio-pci enables device) doesn't call pci_set_dev_assigned() potentially
> leading to issues in those drivers when disabling SR-IOV with VFs assigned.
> 
> Add calls to pci_[set|clear]_dev_assigned() to set the flag as expected.
> 
> Signed-off-by: Bruce Allan <bruce.w.allan@intel.com>
> ---
>  drivers/vfio/pci/vfio_pci.c |    4 ++++
>  1 file changed, 4 insertions(+)


Why should a device exposed to the user be handled differently than a
device used by a host driver?  The reason the assigned flag exists is
because legacy KVM device assignment doesn't actually claim the device
using the driver model, it relies on pci-stub to act as a placeholder to
prevent other drivers from binding to the device, but pci-stub has no
visibility to the device usage and will immediately release it
regardless of it being in use.  vfio-pci participates in the driver
model, signals the user for the device to be released and blocks until
it is released.  Beyond that, the assigned flag is a racy hack.  There's
no locking whatsoever to imply that the flag as any meaning beyond the
instant that it's tested.  I'd like to see the assigned flag deprecated
along with legacy KVM device assignment, not perpetuated in newer
drivers.  The patch is also wrong because we can tell when the device is
actually opened by the user, not just bound to the vfio-pci driver.
Thanks,

Alex

> diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
> index 69fab0f..3f368ce 100644
> --- a/drivers/vfio/pci/vfio_pci.c
> +++ b/drivers/vfio/pci/vfio_pci.c
> @@ -956,6 +956,8 @@ static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>  		return ret;
>  	}
>  
> +	pci_set_dev_assigned(pdev);
> +
>  	if (vfio_pci_is_vga(pdev)) {
>  		vga_client_register(pdev, vdev, NULL, vfio_pci_set_vga_decode);
>  		vga_set_legacy_decoding(pdev,
> @@ -990,6 +992,8 @@ static void vfio_pci_remove(struct pci_dev *pdev)
>  	iommu_group_put(pdev->dev.iommu_group);
>  	kfree(vdev);
>  
> +	pci_clear_dev_assigned(pdev);
> +
>  	if (vfio_pci_is_vga(pdev)) {
>  		vga_client_register(pdev, NULL, NULL, NULL);
>  		vga_set_legacy_decoding(pdev,
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Tuesday, May 12, 2015 7:42 PM
> To: Allan, Bruce W
> Cc: kvm@vger.kernel.org; netdev@vger.kernel.org; intel-wired-
> lan@lists.osuosl.org
> Subject: Re: [PATCH] vfio-pci: PCI_DEV_FLAGS_ASSIGNED flag not set when
> PCI device is assigned
> 
> On Tue, 2015-05-12 at 18:35 -0700, Bruce Allan wrote:
> > A number of PCI device drivers supporting SR-IOV use pci_vfs_assigned()
> to
> > check if there are any VF devices assigned by a VMM prior to disabling
> > SR-IOV (i.e. bnx2x, be2net, fm10k, i40e, igb, ixgbe, qlcnic).  This check
> > works fine with the legacy device assignment (pci-stub enables the
> device)
> > which calls pci_set_dev_assigned().  The newer VFIO-based assignment
> > (vfio-pci enables device) doesn't call pci_set_dev_assigned() potentially
> > leading to issues in those drivers when disabling SR-IOV with VFs assigned.
> >
> > Add calls to pci_[set|clear]_dev_assigned() to set the flag as expected.
> >
> > Signed-off-by: Bruce Allan <bruce.w.allan@intel.com>
> > ---
> >  drivers/vfio/pci/vfio_pci.c |    4 ++++
> >  1 file changed, 4 insertions(+)
> 
> 
> Why should a device exposed to the user be handled differently than a
> device used by a host driver?  The reason the assigned flag exists is
> because legacy KVM device assignment doesn't actually claim the device
> using the driver model, it relies on pci-stub to act as a placeholder to
> prevent other drivers from binding to the device, but pci-stub has no
> visibility to the device usage and will immediately release it
> regardless of it being in use.  vfio-pci participates in the driver
> model, signals the user for the device to be released and blocks until
> it is released.  Beyond that, the assigned flag is a racy hack.  There's
> no locking whatsoever to imply that the flag as any meaning beyond the
> instant that it's tested.  I'd like to see the assigned flag deprecated
> along with legacy KVM device assignment, not perpetuated in newer
> drivers.  The patch is also wrong because we can tell when the device is
> actually opened by the user, not just bound to the vfio-pci driver.
> Thanks,
> 
> Alex

So for clarification, in the drivers mentioned above would you recommend
rather than checking pci_vfs_assigned() before disabling SR-IOV the drivers
should instead check if any of the VF devices are bound/opened by another
driver?

Thanks,
Bruce.

On Wed, 2015-05-13 at 17:50 +0000, Allan, Bruce W wrote:
> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > Sent: Tuesday, May 12, 2015 7:42 PM
> > To: Allan, Bruce W
> > Cc: kvm@vger.kernel.org; netdev@vger.kernel.org; intel-wired-
> > lan@lists.osuosl.org
> > Subject: Re: [PATCH] vfio-pci: PCI_DEV_FLAGS_ASSIGNED flag not set when
> > PCI device is assigned
> > 
> > On Tue, 2015-05-12 at 18:35 -0700, Bruce Allan wrote:
> > > A number of PCI device drivers supporting SR-IOV use pci_vfs_assigned()
> > to
> > > check if there are any VF devices assigned by a VMM prior to disabling
> > > SR-IOV (i.e. bnx2x, be2net, fm10k, i40e, igb, ixgbe, qlcnic).  This check
> > > works fine with the legacy device assignment (pci-stub enables the
> > device)
> > > which calls pci_set_dev_assigned().  The newer VFIO-based assignment
> > > (vfio-pci enables device) doesn't call pci_set_dev_assigned() potentially
> > > leading to issues in those drivers when disabling SR-IOV with VFs assigned.
> > >
> > > Add calls to pci_[set|clear]_dev_assigned() to set the flag as expected.
> > >
> > > Signed-off-by: Bruce Allan <bruce.w.allan@intel.com>
> > > ---
> > >  drivers/vfio/pci/vfio_pci.c |    4 ++++
> > >  1 file changed, 4 insertions(+)
> > 
> > 
> > Why should a device exposed to the user be handled differently than a
> > device used by a host driver?  The reason the assigned flag exists is
> > because legacy KVM device assignment doesn't actually claim the device
> > using the driver model, it relies on pci-stub to act as a placeholder to
> > prevent other drivers from binding to the device, but pci-stub has no
> > visibility to the device usage and will immediately release it
> > regardless of it being in use.  vfio-pci participates in the driver
> > model, signals the user for the device to be released and blocks until
> > it is released.  Beyond that, the assigned flag is a racy hack.  There's
> > no locking whatsoever to imply that the flag as any meaning beyond the
> > instant that it's tested.  I'd like to see the assigned flag deprecated
> > along with legacy KVM device assignment, not perpetuated in newer
> > drivers.  The patch is also wrong because we can tell when the device is
> > actually opened by the user, not just bound to the vfio-pci driver.
> > Thanks,
> > 
> > Alex
> 
> So for clarification, in the drivers mentioned above would you recommend
> rather than checking pci_vfs_assigned() before disabling SR-IOV the drivers
> should instead check if any of the VF devices are bound/opened by another
> driver?

That would also be racy, it could be unbound at your test point and
bound immediately after.  Driver binding also doesn't necessarily imply
that the device is in use; vfio-pci will immediately release an unused
device.  Again, why are we trying to handle this as a special case?
vfio-pci is just another driver.  If it can release the device, it will.
It will even try to signal the user to release it.  Beyond that, we're
bound by the driver model.  If we want drivers to be able to return
-EBUSY if the device is in use, fix the driver model, don't hack around
it with racy tests like the assigned flag.  Thanks,

Alex