From patchwork Tue Sep 5 22:35:50 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petar Penkov X-Patchwork-Id: 810300 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.b="J8Y0k5dq"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 3xn1ll0CKdz9sR9 for ; Wed, 6 Sep 2017 08:36:19 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753992AbdIEWgQ (ORCPT ); Tue, 5 Sep 2017 18:36:16 -0400 Received: from mail-pf0-f169.google.com ([209.85.192.169]:36479 "EHLO mail-pf0-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752834AbdIEWgM (ORCPT ); Tue, 5 Sep 2017 18:36:12 -0400 Received: by mail-pf0-f169.google.com with SMTP id e199so9921270pfh.3 for ; Tue, 05 Sep 2017 15:36:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=jUwVaVG3BlQ0DQXmCHrO6iqiVEnIg/gkGPxid4qQYro=; b=J8Y0k5dqPERDO1mKNRq2JSTmMPmea2bjBdX8fSJ/cbbs3R0Zf72ceYCNuAN5WT0D7W tKErBwiP6oI9Ttrxa6WYecx9elaC1O8R82qPpt0kLj0SOb/O2Hco2okwb8eO598IgHkj uFSb6rd2DFodXqM8erTsiBPWEHr+vDx4a3DGnb48rMKGwEYLV91T94ogyWY5Cp15BufN gaxle43UjD6IO9ruGn2Dqeqizxv+tj/rHuUOmv6P2E4FmH2IwxjHh6nnYVmNipPI9YKT 9lTmGg86BKb4HPpsCw2EqIJ8342HH9FRsvzO3YYIReQWJKrO2RwpvUneonH/pTnP7VR7 eOsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=jUwVaVG3BlQ0DQXmCHrO6iqiVEnIg/gkGPxid4qQYro=; b=is9iVCO95HKwer9AKTZNB2Qnx0kXCOu2SZwQNU/oT6DsfePzeBDwWtnZUrblSL0GHS hl7Ew+o+Ub9MlWuTU10BiiiByRYRBHAherviBBsDaDyQtTUL7jdxQrDheAlIMYhy9cPV ZTGOYx77VU3Fz72mQmIkp9gdGv4c/BL8lBU7QNe6pG3PFzptaYcSsO0ViKPsr23DrxYE ALFCMFrfjOBXIfb24hAXUYorLkjp+CZZ10TYdJ9DIx4jr3KoZvtZ6/2updrP/09SvJgI YbQ4cT3OkjtpP97RzKrLM1RZM3VSgHe9llaPMrDlbJuWcyVAWBnROrNckAh4KCKUzapk vBGg== X-Gm-Message-State: AHPjjUhDUtla7uE0RP03GHd4AJ5QVRZH+C6rEtWZLIz1avAwtCXwMvb0 IayAGhItRQ1Qry5HNb8HmH0T X-Google-Smtp-Source: ADKCNb7EahMEvdGlXAWuAyvJTL23Tdv3F9vA/3JP5L/zeTU+FGsBRu9hBzRUkr4UPfRcz6jYS6d9zg== X-Received: by 10.98.10.12 with SMTP id s12mr5284530pfi.127.1504650970777; Tue, 05 Sep 2017 15:36:10 -0700 (PDT) Received: from localhost ([2620:15c:2cb:1:183d:cea1:ba48:3c2f]) by smtp.gmail.com with ESMTPSA id l85sm1058pfb.176.2017.09.05.15.36.10 (version=TLS1_2 cipher=AES128-SHA bits=128/128); Tue, 05 Sep 2017 15:36:10 -0700 (PDT) From: Petar Penkov To: netdev@vger.kernel.org Cc: Petar Penkov , Eric Dumazet , Mahesh Bandewar , Willem de Bruijn , davem@davemloft.net, ppenkov@stanford.edu Subject: [PATCH net-next RFC 1/2] tun: enable NAPI for TUN/TAP driver Date: Tue, 5 Sep 2017 15:35:50 -0700 Message-Id: <20170905223551.27925-2-ppenkov@google.com> X-Mailer: git-send-email 2.14.1.581.gf28d330327-goog In-Reply-To: <20170905223551.27925-1-ppenkov@google.com> References: <20170905223551.27925-1-ppenkov@google.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Changes TUN driver to use napi_gro_receive() upon receiving packets rather than netif_rx_ni(). Adds flag CONFIG_TUN_NAPI that enables these changes and operation is not affected if the flag is disabled. SKBs are constructed upon packet arrival and are queued to be processed later. The new path was evaluated with a benchmark with the following setup: Open two tap devices and a receiver thread that reads in a loop for each device. Start one sender thread and pin all threads to different CPUs. Send 1M minimum UDP packets to each device and measure sending time for each of the sending methods: napi_gro_receive(): 4.90s netif_rx_ni(): 4.90s netif_receive_skb(): 7.20s Signed-off-by: Petar Penkov Cc: Eric Dumazet Cc: Mahesh Bandewar Cc: Willem de Bruijn Cc: davem@davemloft.net Cc: ppenkov@stanford.edu --- drivers/net/Kconfig | 8 ++++ drivers/net/tun.c | 120 +++++++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 118 insertions(+), 10 deletions(-) diff --git a/drivers/net/Kconfig b/drivers/net/Kconfig index 83a1616903f8..34850d71ddd1 100644 --- a/drivers/net/Kconfig +++ b/drivers/net/Kconfig @@ -307,6 +307,14 @@ config TAP This option is selected by any driver implementing tap user space interface for a virtual interface to re-use core tap functionality. +config TUN_NAPI + bool "NAPI support on tx path for TUN/TAP driver" + default n + depends on TUN + ---help--- + This option allows the TUN/TAP driver to use NAPI to pass packets to + the kernel when receiving packets from user space via write()/send(). + config TUN_VNET_CROSS_LE bool "Support for cross-endian vnet headers on little-endian kernels" default n diff --git a/drivers/net/tun.c b/drivers/net/tun.c index 06e8f0bb2dab..d5c824e3ec42 100644 --- a/drivers/net/tun.c +++ b/drivers/net/tun.c @@ -172,6 +172,7 @@ struct tun_file { u16 queue_index; unsigned int ifindex; }; + struct napi_struct napi; struct list_head next; struct tun_struct *detached; struct skb_array tx_array; @@ -229,6 +230,67 @@ struct tun_struct { struct bpf_prog __rcu *xdp_prog; }; +static int tun_napi_receive(struct napi_struct *napi, int budget) +{ + struct tun_file *tfile = container_of(napi, struct tun_file, napi); + struct sk_buff_head *queue = &tfile->sk.sk_write_queue; + struct sk_buff_head process_queue; + struct sk_buff *skb; + int received = 0; + + __skb_queue_head_init(&process_queue); + + spin_lock(&queue->lock); + skb_queue_splice_tail_init(queue, &process_queue); + spin_unlock(&queue->lock); + + while (received < budget && (skb = __skb_dequeue(&process_queue))) { + napi_gro_receive(napi, skb); + ++received; + } + + if (!skb_queue_empty(&process_queue)) { + spin_lock(&queue->lock); + skb_queue_splice(&process_queue, queue); + spin_unlock(&queue->lock); + } + + return received; +} + +static int tun_napi_poll(struct napi_struct *napi, int budget) +{ + unsigned int received; + + received = tun_napi_receive(napi, budget); + + if (received < budget) + napi_complete_done(napi, received); + + return received; +} + +static void tun_napi_init(struct tun_struct *tun, struct tun_file *tfile) +{ + if (IS_ENABLED(CONFIG_TUN_NAPI)) { + netif_napi_add(tun->dev, &tfile->napi, tun_napi_poll, + NAPI_POLL_WEIGHT); + napi_enable(&tfile->napi); + } +} + +static void tun_napi_disable(struct tun_file *tfile) +{ + if (IS_ENABLED(CONFIG_TUN_NAPI)) + napi_disable(&tfile->napi); +} + +static void tun_napi_del(struct tun_file *tfile) +{ + if (IS_ENABLED(CONFIG_TUN_NAPI)) + netif_napi_del(&tfile->napi); +} + #ifdef CONFIG_TUN_VNET_CROSS_LE static inline bool tun_legacy_is_little_endian(struct tun_struct *tun) { @@ -541,6 +603,11 @@ static void __tun_detach(struct tun_file *tfile, bool clean) tun = rtnl_dereference(tfile->tun); + if (tun && clean) { + tun_napi_disable(tfile); + tun_napi_del(tfile); + } + if (tun && !tfile->detached) { u16 index = tfile->queue_index; BUG_ON(index >= tun->numqueues); @@ -598,6 +665,7 @@ static void tun_detach_all(struct net_device *dev) for (i = 0; i < n; i++) { tfile = rtnl_dereference(tun->tfiles[i]); BUG_ON(!tfile); + tun_napi_disable(tfile); tfile->socket.sk->sk_shutdown = RCV_SHUTDOWN; tfile->socket.sk->sk_data_ready(tfile->socket.sk); RCU_INIT_POINTER(tfile->tun, NULL); @@ -613,6 +681,7 @@ static void tun_detach_all(struct net_device *dev) synchronize_net(); for (i = 0; i < n; i++) { tfile = rtnl_dereference(tun->tfiles[i]); + tun_napi_del(tfile); /* Drop read queue */ tun_queue_purge(tfile); sock_put(&tfile->sk); @@ -677,10 +746,12 @@ static int tun_attach(struct tun_struct *tun, struct file *file, bool skip_filte rcu_assign_pointer(tun->tfiles[tun->numqueues], tfile); tun->numqueues++; - if (tfile->detached) + if (tfile->detached) { tun_enable_queue(tfile); - else + } else { sock_hold(&tfile->sk); + tun_napi_init(tun, tfile); + } tun_set_real_num_queues(tun); @@ -956,13 +1027,28 @@ static void tun_poll_controller(struct net_device *dev) * Tun only receives frames when: * 1) the char device endpoint gets data from user space * 2) the tun socket gets a sendmsg call from user space - * Since both of those are synchronous operations, we are guaranteed - * never to have pending data when we poll for it - * so there is nothing to do here but return. + * If NAPI is not enabled, since both of those are synchronous + * operations, we are guaranteed never to have pending data when we poll + * for it so there is nothing to do here but return. * We need this though so netpoll recognizes us as an interface that * supports polling, which enables bridge devices in virt setups to * still use netconsole + * If NAPI is enabled, however, we need to schedule polling for all + * queues. */ + + if (IS_ENABLED(CONFIG_TUN_NAPI)) { + struct tun_struct *tun = netdev_priv(dev); + struct tun_file *tfile; + int i; + + rcu_read_lock(); + for (i = 0; i < tun->numqueues; i++) { + tfile = rcu_dereference(tun->tfiles[i]); + napi_schedule(&tfile->napi); + } + rcu_read_unlock(); + } return; } #endif @@ -1535,11 +1621,25 @@ static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile, } rxhash = __skb_get_hash_symmetric(skb); -#ifndef CONFIG_4KSTACKS - tun_rx_batched(tun, tfile, skb, more); -#else - netif_rx_ni(skb); -#endif + + if (IS_ENABLED(CONFIG_TUN_NAPI)) { + struct sk_buff_head *queue = &tfile->sk.sk_write_queue; + int queue_len; + + spin_lock_bh(&queue->lock); + __skb_queue_tail(queue, skb); + queue_len = skb_queue_len(queue); + spin_unlock(&queue->lock); + + if (!more || queue_len > NAPI_POLL_WEIGHT) + napi_schedule(&tfile->napi); + + local_bh_enable(); + } else if (!IS_ENABLED(CONFIG_4KSTACKS)) { + tun_rx_batched(tun, tfile, skb, more); + } else { + netif_rx_ni(skb); + } stats = get_cpu_ptr(tun->pcpu_stats); u64_stats_update_begin(&stats->syncp);