From patchwork Thu May 9 16:30:38 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 242789 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id E93022C00F3 for ; Fri, 10 May 2013 02:31:11 +1000 (EST) Received: from localhost ([::1]:55257 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UaTkL-0005Ew-VV for incoming@patchwork.ozlabs.org; Thu, 09 May 2013 12:31:09 -0400 Received: from eggs.gnu.org ([208.118.235.92]:38674) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UaTk2-0005En-Tp for qemu-devel@nongnu.org; Thu, 09 May 2013 12:30:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UaTk1-0002Xw-CX for qemu-devel@nongnu.org; Thu, 09 May 2013 12:30:50 -0400 Received: from mx1.redhat.com ([209.132.183.28]:17395) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UaTk1-0002XD-3P for qemu-devel@nongnu.org; Thu, 09 May 2013 12:30:49 -0400 Received: from int-mx01.intmail.prod.int.phx2.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id r49GUh1c009200 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Thu, 9 May 2013 12:30:44 -0400 Received: from bling.home (ovpn-113-58.phx2.redhat.com [10.3.113.58]) by int-mx01.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id r49GUcJ0030024; Thu, 9 May 2013 12:30:43 -0400 To: alex.williamson@redhat.com From: Alex Williamson Date: Thu, 09 May 2013 10:30:38 -0600 Message-ID: <20130509162935.22443.21491.stgit@bling.home> User-Agent: StGit/0.16 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.67 on 10.5.11.11 X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id r49GUh1c009200 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 209.132.183.28 Cc: qemu-devel@nongnu.org Subject: [Qemu-devel] [PATCH] vfio-pci: Add support for MSI affinity X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org When MSI is accelerated through KVM the vectors are only programmed when the guest first enables MSI support.  Subsequent writes to the vector address or data fields are ignored.  Unfortunately that means we're ignore updates done to adjust SMP affinity of the vectors. MSI SMP affinity already works in non-KVM mode because the address and data fields are read from their backing store on each interrupt. This patch stores the MSIMessage programmed into KVM so that we can determine when changes are made and update the routes. Signed-off-by: Alex Williamson --- For qemu-1.6 hw/misc/vfio.c | 47 ++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 40 insertions(+), 7 deletions(-) diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c index 693a9ff..67a1580 100644 --- a/hw/misc/vfio.c +++ b/hw/misc/vfio.c @@ -102,6 +102,7 @@ typedef struct VFIOINTx { typedef struct VFIOMSIVector { EventNotifier interrupt; /* eventfd triggered on interrupt */ struct VFIODevice *vdev; /* back pointer to device */ + MSIMessage msg; /* cache the MSI message so we know when it changes */ int virq; /* KVM irqchip route for QEMU bypass */ bool use; } VFIOMSIVector; @@ -776,7 +777,6 @@ retry: vdev->msi_vectors = g_malloc0(vdev->nr_vectors * sizeof(VFIOMSIVector)); for (i = 0; i < vdev->nr_vectors; i++) { - MSIMessage msg; VFIOMSIVector *vector = &vdev->msi_vectors[i]; vector->vdev = vdev; @@ -786,13 +786,13 @@ retry: error_report("vfio: Error: event_notifier_init failed"); } - msg = msi_get_message(&vdev->pdev, i); + vector->msg = msi_get_message(&vdev->pdev, i); /* * Attempt to enable route through KVM irqchip, * default to userspace handling if unavailable. */ - vector->virq = kvm_irqchip_add_msi_route(kvm_state, msg); + vector->virq = kvm_irqchip_add_msi_route(kvm_state, vector->msg); if (vector->virq < 0 || kvm_irqchip_add_irqfd_notifier(kvm_state, &vector->interrupt, vector->virq) < 0) { @@ -898,6 +898,33 @@ static void vfio_disable_msi(VFIODevice *vdev) vdev->host.bus, vdev->host.slot, vdev->host.function); } +static void vfio_update_msi(VFIODevice *vdev) +{ + int i; + + for (i = 0; i < vdev->nr_vectors; i++) { + VFIOMSIVector *vector = &vdev->msi_vectors[i]; + MSIMessage msg; + + if (!vector->use || vector->virq < 0) { + continue; + } + + msg = msi_get_message(&vdev->pdev, i); + + if (msg.address != vector->msg.address || + msg.data != vector->msg.data) { + + DPRINTF("%s(%04x:%02x:%02x.%x) MSI vector %d changed\n", + __func__, vdev->host.domain, vdev->host.bus, + vdev->host.slot, vdev->host.function, i); + + kvm_irqchip_update_msi_route(kvm_state, vector->virq, msg); + vector->msg = msg; + } + } +} + /* * IO Port/MMIO - Beware of the endians, VFIO is always little endian */ @@ -1850,10 +1877,16 @@ static void vfio_pci_write_config(PCIDevice *pdev, uint32_t addr, is_enabled = msi_enabled(pdev); - if (!was_enabled && is_enabled) { - vfio_enable_msi(vdev); - } else if (was_enabled && !is_enabled) { - vfio_disable_msi(vdev); + if (!was_enabled) { + if (is_enabled) { + vfio_enable_msi(vdev); + } + } else { + if (!is_enabled) { + vfio_disable_msi(vdev); + } else { + vfio_update_msi(vdev); + } } } else if (pdev->cap_present & QEMU_PCI_CAP_MSIX && ranges_overlap(addr, len, pdev->msix_cap, MSIX_CAP_LENGTH)) {