From patchwork Fri Jun 23 08:43:13 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jason Wang X-Patchwork-Id: 779884 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3wvBnT4KC4z9s7h for ; Fri, 23 Jun 2017 18:43:57 +1000 (AEST) Received: from localhost ([::1]:34133 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dOKC1-0004er-Rx for incoming@patchwork.ozlabs.org; Fri, 23 Jun 2017 04:43:53 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36644) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dOKBe-0004dY-3C for qemu-devel@nongnu.org; Fri, 23 Jun 2017 04:43:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dOKBZ-0005jm-5e for qemu-devel@nongnu.org; Fri, 23 Jun 2017 04:43:30 -0400 Received: from mx1.redhat.com ([209.132.183.28]:34058) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dOKBY-0005iy-TC for qemu-devel@nongnu.org; Fri, 23 Jun 2017 04:43:25 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0091543A25; Fri, 23 Jun 2017 08:43:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 0091543A25 Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=jasowang@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 0091543A25 Received: from [10.72.12.55] (ovpn-12-55.pek2.redhat.com [10.72.12.55]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 707146A451; Fri, 23 Jun 2017 08:43:15 +0000 (UTC) To: "Michael S. Tsirkin" , jean-philippe menil References: <20170605050423-mutt-send-email-mst@kernel.org> <20170606024211-mutt-send-email-mst@kernel.org> <20170622215125-mutt-send-email-mst@kernel.org> From: Jason Wang Message-ID: <4f6d4342-8476-882e-e7a8-b3f0ebd20d95@redhat.com> Date: Fri, 23 Jun 2017 16:43:13 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: <20170622215125-mutt-send-email-mst@kernel.org> Content-Language: en-US X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Fri, 23 Jun 2017 08:43:22 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: Re: [Qemu-devel] BUG: KASAN: use-after-free in free_old_xmit_skbs X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: netdev@vger.kernel.org, John Fastabend , qemu-devel@nongnu.org, virtualization@lists.linux-foundation.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" On 2017年06月23日 02:53, Michael S. Tsirkin wrote: > On Thu, Jun 22, 2017 at 08:15:58AM +0200, jean-philippe menil wrote: >> 2017-06-06 1:52 GMT+02:00 Michael S. Tsirkin : >> >> On Mon, Jun 05, 2017 at 05:08:25AM +0300, Michael S. Tsirkin wrote: >> > On Mon, Jun 05, 2017 at 12:48:53AM +0200, Jean-Philippe Menil wrote: >> > > Hi, >> > > >> > > while playing with xdp and ebpf, i'm hitting the following: >> > > >> > > [ 309.993136] >> > > ================================================================== >> > > [ 309.994735] BUG: KASAN: use-after-free in >> > > free_old_xmit_skbs.isra.29+0x2b7/0x2e0 [virtio_net] >> > > [ 309.998396] Read of size 8 at addr ffff88006aa64220 by task sshd/323 >> > > [ 310.000650] >> > > [ 310.002305] CPU: 1 PID: 323 Comm: sshd Not tainted 4.12.0-rc3+ #2 >> > > [ 310.004018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), >> BIOS >> > > 1.10.2-20170228_101828-anatol 04/01/2014 > ... > >> > >> > Since commit 680557cf79f82623e2c4fd42733077d60a843513 >> > virtio_net: rework mergeable buffer handling >> > >> > we no longer must do the resets, we now have enough space >> > to store a bit saying whether a buffer is xdp one or not. >> > >> > And that's probably a cleaner way to fix these issues than >> > try to find and fix the race condition. >> > >> > John? >> > >> > -- >> > MST >> >> >> I think I see the source of the race. virtio net calls >> netif_device_detach and assumes no packets will be sent after >> this point. However, all it does is stop all queues so >> no new packets will be transmitted. >> >> Try locking with HARD_TX_LOCK? >> >> >> -- >> MST >> >> >> Hi Michael, >> >> from what i see, the race appear when we hit virtnet_reset in virtnet_xdp_set. >> virtnet_reset >> _remove_vq_common >> virtnet_del_vqs >> virtnet_free_queues >> kfree(vi->sq) >> when the xdp program (with two instances of the program to trigger it faster) >> is added or removed. >> >> It's easily repeatable, with 2 cpus and 4 queues on the qemu command line, >> running the xdp_ttl tool from Jesper. >> >> For now, i'm able to continue my qualification, testing if xdp_qp is not null, >> but do not seem to be a sustainable trick. >> if (xdp_qp && vi->xdp_queues_pairs != xdp_qp) >> >> Maybe it will be more clear to you with theses informations. >> >> Best regards. >> >> Jean-Philippe > > I'm pretty clear about the issue here, I was trying to figure out a fix. > Jason, any thoughts? > > Hi Jean: Does the following fix this issue? (I can't reproduce it locally through xdp_ttl) Thanks diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 1f8c15c..3e65c3f 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -1801,7 +1801,9 @@ static void virtnet_freeze_down(struct virtio_device *vdev) /* Make sure no work handler is accessing the device */ flush_work(&vi->config_work); + netif_tx_lock_bh(vi->dev); netif_device_detach(vi->dev); + netif_tx_unlock_bh(vi->dev); cancel_delayed_work_sync(&vi->refill);