From patchwork Thu Feb 14 18:21:57 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luigi Rizzo X-Patchwork-Id: 220477 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 452052C0087 for ; Fri, 15 Feb 2013 05:22:05 +1100 (EST) Received: from localhost ([::1]:49275 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1U63Rb-0005sg-5K for incoming@patchwork.ozlabs.org; Thu, 14 Feb 2013 13:22:03 -0500 Received: from eggs.gnu.org ([208.118.235.92]:47127) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1U63RS-0005sI-LM for qemu-devel@nongnu.org; Thu, 14 Feb 2013 13:21:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1U63RN-0007kg-NT for qemu-devel@nongnu.org; Thu, 14 Feb 2013 13:21:54 -0500 Received: from onelab2.iet.unipi.it ([131.114.59.238]:49204) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1U63RN-0007jQ-Gi for qemu-devel@nongnu.org; Thu, 14 Feb 2013 13:21:49 -0500 Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id 5D50A7300A; Thu, 14 Feb 2013 19:21:57 +0100 (CET) Date: Thu, 14 Feb 2013 19:21:57 +0100 From: Luigi Rizzo To: qemu-devel@nongnu.org Message-ID: <20130214182157.GA44257@onelab2.iet.unipi.it> MIME-Version: 1.0 Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) X-detected-operating-system: by eggs.gnu.org: Mac OS X 10.x X-Received-From: 131.114.59.238 Subject: [Qemu-devel] RFC: handling "backend too fast" in virtio-net X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org virtio-style network devices (where the producer and consumer chase each other through a shared memory block) can enter into a bad operating regime when the consumer is too fast. I am hitting this case right now when virtio is used on top of the netmap/VALE backend that I posted a few weeks ago: what happens is that the backend is so fast that the io thread keeps re-enabling notifications every few packets. This results, on my test machine, in a throughput of 250-300Kpps (and extremely unstable, oscillating between 200 and 600Kpps). I'd like to get some feedback on the following trivial patch to have the thread keep spinning for a bounded amount of time when the producer is slower than the consumer. This gives a relatively stable throughput between 700 and 800 Kpps (we have something similar in our paravirtualized e1000 driver, which is slightly faster at 900-1100 Kpps). EXISTING LOGIC: reschedule the bh when at least a burst of packets has been received. Otherwise enable notifications and do another race-prevention check. NEW LOGIC: when the bh is scheduled the first time, establish a budget (currently 20) of reschedule attempts. Every time virtio_net_flush_tx() returns 0 packets [this could be changed to 'less than a full burst'] the counter is increased. The bh is rescheduled until the counter reaches the budget, at which point we re-enable notifications as above. I am not 100% happy with this patch because there is a "magic" constant (the maximum number of retries) which should be really adaptive, or perhaps we should even bound the amount of time the bh runs, rather than the number of retries. In practice, having the thread spin (or sleep) for 10-20us even without new packets is probably preferable to re-enable notifications and request a kick. opinions ? cheers luigi diff --git a/hw/virtio-net.c b/hw/virtio-net.c index 573c669..5389088 100644 --- a/hw/virtio-net.c +++ b/hw/virtio-net.c @@ -49,6 +51,7 @@ typedef struct VirtIONet NICState *nic; uint32_t tx_timeout; int32_t tx_burst; + int32_t tx_retries; /* keep spinning a bit on tx */ uint32_t has_vnet_hdr; size_t host_hdr_len; size_t guest_hdr_len; @@ -1062,7 +1065,9 @@ static void virtio_net_tx_bh(void *opaque) /* If we flush a full burst of packets, assume there are * more coming and immediately reschedule */ - if (ret >= n->tx_burst) { + if (ret == 0) + n->tx_retries++; + if (n->tx_retries < 20) { qemu_bh_schedule(q->tx_bh); q->tx_waiting = 1; return; @@ -1076,6 +1082,8 @@ static void virtio_net_tx_bh(void *opaque) virtio_queue_set_notification(q->tx_vq, 0); qemu_bh_schedule(q->tx_bh); q->tx_waiting = 1; + } else { + n->tx_retries = 0; } }