From patchwork Wed Nov 16 03:16:12 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 695412 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3tJTvV5vMdz9t0p for ; Wed, 16 Nov 2016 14:17:10 +1100 (AEDT) Received: from localhost ([::1]:50349 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c6qif-0004Lm-35 for incoming@patchwork.ozlabs.org; Tue, 15 Nov 2016 22:17:05 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:40165) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c6qhw-000435-Nc for qemu-devel@nongnu.org; Tue, 15 Nov 2016 22:16:21 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1c6qhs-0007LO-3i for qemu-devel@nongnu.org; Tue, 15 Nov 2016 22:16:20 -0500 Received: from mx1.redhat.com ([209.132.183.28]:33978) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1c6qhr-0007LI-Rs for qemu-devel@nongnu.org; Tue, 15 Nov 2016 22:16:16 -0500 Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 08DF62BA0EF; Wed, 16 Nov 2016 03:16:14 +0000 (UTC) Received: from t450s.home (ovpn03.gateway.prod.ext.phx2.redhat.com [10.5.9.3]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id uAG3GDtj025899; Tue, 15 Nov 2016 22:16:13 -0500 Date: Tue, 15 Nov 2016 20:16:12 -0700 From: Alex Williamson To: Kirti Wankhede Message-ID: <20161115201612.103893d7@t450s.home> In-Reply-To: References: <1479223805-22895-1-git-send-email-kwankhede@nvidia.com> <1479223805-22895-12-git-send-email-kwankhede@nvidia.com> <20161115151950.1e8ab7d6@t450s.home> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Wed, 16 Nov 2016 03:16:14 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: Re: [Qemu-devel] [PATCH v13 11/22] vfio iommu: Add blocking notifier to notify DMA_UNMAP X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kevin.tian@intel.com, cjia@nvidia.com, kvm@vger.kernel.org, qemu-devel@nongnu.org, linux-kernel@vger.kernel.org, jike.song@intel.com, kraxel@redhat.com, pbonzini@redhat.com, bjsdjshi@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" On Wed, 16 Nov 2016 08:16:15 +0530 Kirti Wankhede wrote: > On 11/16/2016 3:49 AM, Alex Williamson wrote: > > On Tue, 15 Nov 2016 20:59:54 +0530 > > Kirti Wankhede wrote: > > > ... > > >> @@ -854,7 +857,28 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, > >> */ > >> if (dma->task->mm != current->mm) > >> break; > >> + > >> unmapped += dma->size; > >> + > >> + if (iommu->external_domain && !RB_EMPTY_ROOT(&dma->pfn_list)) { > >> + struct vfio_iommu_type1_dma_unmap nb_unmap; > >> + > >> + nb_unmap.iova = dma->iova; > >> + nb_unmap.size = dma->size; > >> + > >> + /* > >> + * Notifier callback would call vfio_unpin_pages() which > >> + * would acquire iommu->lock. Release lock here and > >> + * reacquire it again. > >> + */ > >> + mutex_unlock(&iommu->lock); > >> + blocking_notifier_call_chain(&iommu->notifier, > >> + VFIO_IOMMU_NOTIFY_DMA_UNMAP, > >> + &nb_unmap); > >> + mutex_lock(&iommu->lock); > >> + if (WARN_ON(!RB_EMPTY_ROOT(&dma->pfn_list))) > >> + break; > >> + } > > > > > > Why exactly do we need to notify per vfio_dma rather than per unmap > > request? If we do the latter we can send the notify first, limiting us > > to races where a page is pinned between the notify and the locking, > > whereas here, even our dma pointer is suspect once we re-acquire the > > lock, we don't technically know if another unmap could have removed > > that already. Perhaps something like this (untested): > > > > There are checks to validate unmap request, like v2 check and who is > calling unmap and is it allowed for that task to unmap. Before these > checks its not sure that unmap region range which asked for would be > unmapped all. Notify call should be at the place where its sure that the > range provided to notify call is definitely going to be removed. My > change do that. Ok, but that does solve the problem. What about this (untested): diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index ee9a680..50cafdf 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -782,9 +782,9 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, struct vfio_iommu_type1_dma_unmap *unmap) { uint64_t mask; - struct vfio_dma *dma; + struct vfio_dma *dma, *dma_last = NULL; size_t unmapped = 0; - int ret = 0; + int ret = 0, retries; mask = ((uint64_t)1 << __ffs(vfio_pgsize_bitmap(iommu))) - 1; @@ -794,7 +794,7 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, return -EINVAL; WARN_ON(mask & PAGE_MASK); - +again: mutex_lock(&iommu->lock); /* @@ -851,11 +851,16 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, if (dma->task->mm != current->mm) break; - unmapped += dma->size; - - if (iommu->external_domain && !RB_EMPTY_ROOT(&dma->pfn_list)) { + if (!RB_EMPTY_ROOT(&dma->pfn_list)) { struct vfio_iommu_type1_dma_unmap nb_unmap; + if (dma_last == dma) { + BUG_ON(++retries > 10); + } else { + dma_last = dma; + retries = 0; + } + nb_unmap.iova = dma->iova; nb_unmap.size = dma->size; @@ -868,11 +873,11 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu, blocking_notifier_call_chain(&iommu->notifier, VFIO_IOMMU_NOTIFY_DMA_UNMAP, &nb_unmap); - mutex_lock(&iommu->lock); - if (WARN_ON(!RB_EMPTY_ROOT(&dma->pfn_list))) - break; + goto again: } + unmapped += dma->size; vfio_remove_dma(iommu, dma); + } unlock: