From patchwork Tue Apr 24 14:39:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Toshiaki Makita X-Patchwork-Id: 903538 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="iXETD/TP"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40VmGP4Nzjz9s02 for ; Wed, 25 Apr 2018 00:40:45 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754505AbeDXOkO (ORCPT ); Tue, 24 Apr 2018 10:40:14 -0400 Received: from mail-pg0-f66.google.com ([74.125.83.66]:36236 "EHLO mail-pg0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754401AbeDXOjy (ORCPT ); Tue, 24 Apr 2018 10:39:54 -0400 Received: by mail-pg0-f66.google.com with SMTP id i6so11038718pgv.3 for ; Tue, 24 Apr 2018 07:39:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=phSjkHfCr1vZGLL1OQQbAbkAUiQQEVL6XnBxa987jiE=; b=iXETD/TPbgAWoelSnDwMn1R40Fj6YvCgfG4MxEhdm1/pqHJagUSbgkcoTMb63BnDim KBta50ypkTYTBMlASkO9H8ElAWpzq+LEisOA6/HVNEtdN93TQqb/Vemu+G9xuvPBjdjX RbAH9JgQFKR7GX7vnw10YKouafrxuNW/sB9gNfsCW/PJUGUAlM27CZkVhIkTLim1AmsC IDxnslm5ZME5yWVpAXanAml8XjN3Z7SWHT+t+2i9NywrL5MU0ufvuld4msUHQYPPG4d3 GEkrfkcVbGP9lvdBOTlrD3oudVRoI781VCRfyUQO6DUpWMgf97fFGC/FxDurNd4gfNCJ xfMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=phSjkHfCr1vZGLL1OQQbAbkAUiQQEVL6XnBxa987jiE=; b=e8gKv8EcHF0rnbGmJAJvKkOFwqigrbGILbGEGYQ0Df+VSiyNxNGuGWnDjY1ktHYOJo rnbzyM74VRptjm5OKFWkc1R8KZgsunbEuv1Jp1TSJvnzF9Qs3tbIPXzyV6rjx4PEByNz XUPH4IAovoE2NRtDL5jw3IHuSK093tHXUNHrHjpfGc3o90ovP/tb2VacUNgeYJzdp/cb 9n2kmBb+Asx6A6iPE6Ma4OX6K7MBQXGIzBMic+YSfsa98SZfdJVTznMNgzPwKVCtmBVN FLWD2QjyNyzIAN+iOhgCV5toQNJNHVWRSSo6KEOpqW+WMN5PN2zEE4lPrsRXOOU619q1 h7sQ== X-Gm-Message-State: ALQs6tB0YC3rDrkmL+1+63XVFcsdSi/8yjQ6HJPj9BTr9zMvpgmNzEKZ 1eI0kX6JnWKKKakDAWDrzbOu6P6r4NU= X-Google-Smtp-Source: AIpwx49SBYM1zAtpvHx3lGJi/X88n0PCD6jr2XmSXtm1MEy6ooEARpVBbo5OxLvXCbKLCH1pzwrWtw== X-Received: by 2002:a17:902:274a:: with SMTP id j10-v6mr25801585plg.393.1524580793673; Tue, 24 Apr 2018 07:39:53 -0700 (PDT) Received: from localhost.localdomain (i121-115-166-6.s42.a013.ap.plala.or.jp. [121.115.166.6]) by smtp.gmail.com with ESMTPSA id o64sm28179970pfb.62.2018.04.24.07.39.52 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 24 Apr 2018 07:39:53 -0700 (PDT) From: Toshiaki Makita To: netdev@vger.kernel.org Cc: Toshiaki Makita Subject: [PATCH RFC 8/9] veth: Avoid per-packet spinlock of XDP napi ring on dequeueing Date: Tue, 24 Apr 2018 23:39:22 +0900 Message-Id: <20180424143923.26519-9-toshiaki.makita1@gmail.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180424143923.26519-1-toshiaki.makita1@gmail.com> References: <20180424143923.26519-1-toshiaki.makita1@gmail.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Toshiaki Makita Use percpu temporary storage to avoid per-packet spinlock. Signed-off-by: Toshiaki Makita --- drivers/net/veth.c | 46 +++++++++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 19 deletions(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 98fc91a64e29..1592119e3873 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -30,6 +30,7 @@ #define VETH_XDP_FLAG 0x1UL #define VETH_RING_SIZE 256 #define VETH_XDP_HEADROOM (XDP_PACKET_HEADROOM + NET_IP_ALIGN) +#define VETH_XDP_QUEUE_SIZE NAPI_POLL_WEIGHT struct pcpu_vstats { u64 packets; @@ -50,6 +51,8 @@ struct veth_priv { struct xdp_rxq_info xdp_rxq; }; +static DEFINE_PER_CPU(void *[VETH_XDP_QUEUE_SIZE], xdp_consume_q); + static bool veth_is_xdp_frame(void *ptr) { return (unsigned long)ptr & VETH_XDP_FLAG; @@ -563,27 +566,32 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_priv *priv, static int veth_xdp_rcv(struct veth_priv *priv, int budget, bool *xdp_xmit, bool *xdp_redir) { - int i, done = 0; - - for (i = 0; i < budget; i++) { - void *ptr = ptr_ring_consume(&priv->xdp_ring); - struct sk_buff *skb; - - if (!ptr) - break; + void **q = this_cpu_ptr(xdp_consume_q); + int num, lim, done = 0; + + do { + int i; + + lim = min(budget - done, VETH_XDP_QUEUE_SIZE); + num = ptr_ring_consume_batched(&priv->xdp_ring, q, lim); + for (i = 0; i < num; i++) { + struct sk_buff *skb; + void *ptr = q[i]; + + if (veth_is_xdp_frame(ptr)) { + skb = veth_xdp_rcv_one(priv, + veth_ptr_to_xdp(ptr), + xdp_xmit, xdp_redir); + } else { + skb = veth_xdp_rcv_skb(priv, ptr, xdp_xmit, + xdp_redir); + } - if (veth_is_xdp_frame(ptr)) { - skb = veth_xdp_rcv_one(priv, veth_ptr_to_xdp(ptr), - xdp_xmit, xdp_redir); - } else { - skb = veth_xdp_rcv_skb(priv, ptr, xdp_xmit, xdp_redir); + if (skb) + napi_gro_receive(&priv->xdp_napi, skb); } - - if (skb) - napi_gro_receive(&priv->xdp_napi, skb); - - done++; - } + done += num; + } while (unlikely(num == lim && done < budget)); return done; }