From patchwork Tue Apr 24 14:39:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Toshiaki Makita X-Patchwork-Id: 903539 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="F3use0DH"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40VmGW3jgsz9s02 for ; Wed, 25 Apr 2018 00:40:51 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753426AbeDXOkL (ORCPT ); Tue, 24 Apr 2018 10:40:11 -0400 Received: from mail-pf0-f194.google.com ([209.85.192.194]:35354 "EHLO mail-pf0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754462AbeDXOjz (ORCPT ); Tue, 24 Apr 2018 10:39:55 -0400 Received: by mail-pf0-f194.google.com with SMTP id j5so12435973pfh.2 for ; Tue, 24 Apr 2018 07:39:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=XIlrRLnvrbt+NyF/5Fo+OAhKn1d2GVwO37Lb8gF5cM8=; b=F3use0DHwK1X7OvmsiDxZymU1qRpoI2nYR8p0pCA5V1SZlumIeqiOKRdiyHfZrCHJe 3i9k2iavuOHJEcnq0Zs/wSSR8iVThxjSfwXxg57i6N36sre5WMVkBZymEp6wYuQqxu0A vd3lIwuta97ZqxuaWRAUw61qYMyIwpOgXEBVL9rtF/AVhZ/dTe998KSYsqg+S1HUiefw LTduQZMKePDVxdJeGS3Je2eVyvrKehq/XNLjkmFU34ftPf73+7KPxRHEQDVFy5RfVd1m x7Fa8a7pPoW6xt5s9nmBPBCp193eUEUE8JYT7oHL0abh2m68c13oC6F8gfcaDsJumscz Zf6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=XIlrRLnvrbt+NyF/5Fo+OAhKn1d2GVwO37Lb8gF5cM8=; b=JpMmwtVRCSSS9Y8x/45KtFcHcvSbsCvc7j1klnxUx1rCZCKXcvtZnJA/KBJwhFltMb PYa6clVDStc+XCJMn2tMrWZhVDxHsNimMFEzw8+jTXEhhObShJgMC8Z8fvGVFaken+4Y +trJM7wb+Zqub/izs/hpRNgw2ju/yyTjNnKT49f0ZHZTQnwmq+LzatCWs1WMZstjJ0/B 2qlvxh3enPQ+c/CGXoqjCZMi91sOxErlq/wArtUlJlsIasw43UnKC8D7xRk1FCJ9I4qb M/GyFpiRJN9Tt+/IpYqUjs6cBkUMIZkwWdI6ZmeoymzpkFFJ+kiGA/eueOKMfLelPVFy JEEg== X-Gm-Message-State: ALQs6tBo7NHxmMWvAcWwJ37D71Z5zvitqqOem3J3zXncD1iQynY6YCqu KFlGbiuRzcC3yIp44PXiikyT2LYy5tI= X-Google-Smtp-Source: AIpwx4/xMlct4lN84h6d0x46POLNFoWltL4d8K1jhIVBROalEj3RXOubIbG9fZlmK3ysnpgJI+ioyg== X-Received: by 2002:a17:902:9a8b:: with SMTP id w11-v6mr25278766plp.75.1524580795168; Tue, 24 Apr 2018 07:39:55 -0700 (PDT) Received: from localhost.localdomain (i121-115-166-6.s42.a013.ap.plala.or.jp. [121.115.166.6]) by smtp.gmail.com with ESMTPSA id o64sm28179970pfb.62.2018.04.24.07.39.53 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 24 Apr 2018 07:39:54 -0700 (PDT) From: Toshiaki Makita To: netdev@vger.kernel.org Cc: Toshiaki Makita Subject: [PATCH RFC 9/9] veth: Avoid per-packet spinlock of XDP napi ring on enqueueing Date: Tue, 24 Apr 2018 23:39:23 +0900 Message-Id: <20180424143923.26519-10-toshiaki.makita1@gmail.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180424143923.26519-1-toshiaki.makita1@gmail.com> References: <20180424143923.26519-1-toshiaki.makita1@gmail.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Toshiaki Makita Use percpu temporary storage to avoid per-packet spinlock. This is different from dequeue in that multiple veth devices can be redirect target in one napi loop so allocate percpu storage in veth private structure. Signed-off-by: Toshiaki Makita --- drivers/net/veth.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 65 insertions(+), 1 deletion(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 1592119e3873..5978d76f2c00 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -38,12 +38,18 @@ struct pcpu_vstats { struct u64_stats_sync syncp; }; +struct xdp_queue { + void *q[VETH_XDP_QUEUE_SIZE]; + unsigned int len; +}; + struct veth_priv { struct napi_struct xdp_napi; struct net_device *dev; struct bpf_prog __rcu *xdp_prog; struct net_device __rcu *peer; atomic64_t dropped; + struct xdp_queue __percpu *xdp_produce_q; struct xdp_mem_info xdp_mem; unsigned requested_headroom; bool rx_notify_masked; @@ -147,8 +153,48 @@ static void veth_ptr_free(void *ptr) } } +static void veth_xdp_cleanup_queues(struct veth_priv *priv) +{ + int cpu; + + for_each_possible_cpu(cpu) { + struct xdp_queue *q = per_cpu_ptr(priv->xdp_produce_q, cpu); + int i; + + for (i = 0; i < q->len; i++) + veth_ptr_free(q->q[i]); + + q->len = 0; + } +} + +static bool veth_xdp_flush_queue(struct veth_priv *priv) +{ + struct xdp_queue *q = this_cpu_ptr(priv->xdp_produce_q); + int i; + + if (unlikely(!q->len)) + return false; + + spin_lock(&priv->xdp_ring.producer_lock); + for (i = 0; i < q->len; i++) { + void *ptr = q->q[i]; + + if (unlikely(__ptr_ring_produce(&priv->xdp_ring, ptr))) + veth_ptr_free(ptr); + } + spin_unlock(&priv->xdp_ring.producer_lock); + + q->len = 0; + + return true; +} + static void __veth_xdp_flush(struct veth_priv *priv) { + if (unlikely(!veth_xdp_flush_queue(priv))) + return; + /* Write ptr_ring before reading rx_notify_masked */ smp_mb(); if (!priv->rx_notify_masked) { @@ -159,9 +205,13 @@ static void __veth_xdp_flush(struct veth_priv *priv) static int veth_xdp_enqueue(struct veth_priv *priv, void *ptr) { - if (unlikely(ptr_ring_produce(&priv->xdp_ring, ptr))) + struct xdp_queue *q = this_cpu_ptr(priv->xdp_produce_q); + + if (unlikely(q->len >= VETH_XDP_QUEUE_SIZE)) return -ENOSPC; + q->q[q->len++] = ptr; + return 0; } @@ -644,6 +694,7 @@ static void veth_napi_del(struct net_device *dev) napi_disable(&priv->xdp_napi); netif_napi_del(&priv->xdp_napi); + veth_xdp_cleanup_queues(priv); ptr_ring_cleanup(&priv->xdp_ring, veth_ptr_free); } @@ -711,15 +762,28 @@ static int is_valid_veth_mtu(int mtu) static int veth_dev_init(struct net_device *dev) { + struct veth_priv *priv = netdev_priv(dev); + dev->vstats = netdev_alloc_pcpu_stats(struct pcpu_vstats); if (!dev->vstats) return -ENOMEM; + + priv->xdp_produce_q = __alloc_percpu(sizeof(*priv->xdp_produce_q), + sizeof (void *)); + if (!priv->xdp_produce_q) { + free_percpu(dev->vstats); + return -ENOMEM; + } + return 0; } static void veth_dev_free(struct net_device *dev) { + struct veth_priv *priv = netdev_priv(dev); + free_percpu(dev->vstats); + free_percpu(priv->xdp_produce_q); } #ifdef CONFIG_NET_POLL_CONTROLLER