From patchwork Sun Jun 10 16:02:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Toshiaki Makita X-Patchwork-Id: 927380 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="jiCTttC7"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 413gsF5Yszz9rvt for ; Mon, 11 Jun 2018 02:02:41 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932296AbeFJQCh (ORCPT ); Sun, 10 Jun 2018 12:02:37 -0400 Received: from mail-pf0-f193.google.com ([209.85.192.193]:34271 "EHLO mail-pf0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932163AbeFJQCe (ORCPT ); Sun, 10 Jun 2018 12:02:34 -0400 Received: by mail-pf0-f193.google.com with SMTP id a63-v6so8962683pfl.1 for ; Sun, 10 Jun 2018 09:02:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uLnUamuckBFjDEk74ZdINumU6xe4kM+jHRMGBq4wCoE=; b=jiCTttC7yVnhkeZ01Wh1k2lsFt1NMcqed+ryjV+WK1ENLY3sepg5DOsNhFkMu0zlrz vIqnsfsZtyE96+1vwAHpSH66/F67N3shHxjlUP3HxytqsA6Zy0UZOPSjL8IfrR6ecK7A UfnH+ZcT3NIqZqc90PV77pOmzGf+E7wo5ZkWt97egmss8k4I24wABJrAQImJq8eSEdYQ 7XUmtAe3bDBTlOo2YpY0aPOusQZ+GKESSrPrWOWwp7JHXHpkDNcUk1edPNSrfeOva4uH Gy46IbbLrWoBZPfKDOGDO3UKzi3wMqB58N9KBjRj1TIL349w25JT40jY4V9ButKFqdNB Cq1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uLnUamuckBFjDEk74ZdINumU6xe4kM+jHRMGBq4wCoE=; b=flE3xE3H3rIiiy6L9Qwt3LHQE3RP7mYvOfngOTouZSCClybOkLF6I4kfHFZ9bzG4Hy 4deSFR6BXfLJj125qvUp2o0XzEsM38HnCb3H83C2TG1aj/4+e9XTrbx+jn1G5h7/PUCa o8ssn7/tKPZKSTUl0yNhUe6e8wsjmam1B3fRf0T4hsPl8MLII4Be2reo8e3V4s6dvZ6M Wn3zdqE/95rfEyNZoHG3le3V4Et7su3JrHcJe8ApPnX4EjXJh6L7iFHl94F+c492f53s 9pE7rVHWBVIsSxeVjhchJsjwZRVz932JED3dBL/X2OTkh0rVub5Ef4oiEfaQWacDms5n bMmA== X-Gm-Message-State: APt69E0IACP0quNRY4DyKpmXtgaG07w1X8+m3nNZ6OdyGOOw+1x5nFaE HstYyG/J5Dqe0SgVNV1ROGnqzg== X-Google-Smtp-Source: ADUXVKL7KDU/6RJqt84l59rWtIT+OgK+eiquaK6L+4Gz7lnR09qfnsLdc4wKTkl4zJddO4WZ8w/wgg== X-Received: by 2002:a63:af50:: with SMTP id s16-v6mr12111502pgo.263.1528646553319; Sun, 10 Jun 2018 09:02:33 -0700 (PDT) Received: from localhost.localdomain (i153-145-22-9.s42.a013.ap.plala.or.jp. [153.145.22.9]) by smtp.gmail.com with ESMTPSA id o87-v6sm56068211pfa.106.2018.06.10.09.02.31 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sun, 10 Jun 2018 09:02:32 -0700 (PDT) From: Toshiaki Makita To: netdev@vger.kernel.org Cc: Toshiaki Makita , Jesper Dangaard Brouer , Alexei Starovoitov , Daniel Borkmann Subject: [PATCH RFC v2 4/9] veth: Add another napi ring for ndo_xdp_xmit and handle xdp_frames Date: Mon, 11 Jun 2018 01:02:12 +0900 Message-Id: <20180610160217.3146-5-toshiaki.makita1@gmail.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180610160217.3146-1-toshiaki.makita1@gmail.com> References: <20180610160217.3146-1-toshiaki.makita1@gmail.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Toshiaki Makita This is preparation for XDP TX and ndo_xdp_xmit. Add another napi ring and handle redirected xdp_frames through it. v2: - Use another ring instead of using flag to differentiate skb and xdp_frame. This approach makes bulk skb transmit possible in veth_xmit later. - Clear xdp_frame feilds in skb->head. - Implement adjust_tail. Signed-off-by: Toshiaki Makita --- drivers/net/veth.c | 125 ++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 114 insertions(+), 11 deletions(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 88d349da72cc..cb3fa558fbe0 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -46,6 +46,7 @@ struct veth_priv { unsigned requested_headroom; bool rx_notify_masked; struct ptr_ring xdp_ring; + struct ptr_ring xdp_tx_ring; struct xdp_rxq_info xdp_rxq; }; @@ -114,6 +115,11 @@ static const struct ethtool_ops veth_ethtool_ops = { /* general routines */ +static void veth_xdp_free(void *frame) +{ + xdp_return_frame(frame); +} + static void __veth_xdp_flush(struct veth_priv *priv) { /* Write ptr_ring before reading rx_notify_masked */ @@ -248,6 +254,61 @@ static struct sk_buff *veth_build_skb(void *head, int headroom, int len, return skb; } +static struct sk_buff *veth_xdp_rcv_one(struct veth_priv *priv, + struct xdp_frame *frame) +{ + int len = frame->len, delta = 0; + struct bpf_prog *xdp_prog; + unsigned int headroom; + struct sk_buff *skb; + + rcu_read_lock(); + xdp_prog = rcu_dereference(priv->xdp_prog); + if (xdp_prog) { + struct xdp_buff xdp; + u32 act; + + xdp.data_hard_start = frame->data - frame->headroom; + xdp.data = frame->data; + xdp.data_end = frame->data + frame->len; + xdp.data_meta = frame->data - frame->metasize; + xdp.rxq = &priv->xdp_rxq; + + act = bpf_prog_run_xdp(xdp_prog, &xdp); + + switch (act) { + case XDP_PASS: + delta = frame->data - xdp.data; + len = xdp.data_end - xdp.data; + break; + default: + bpf_warn_invalid_xdp_action(act); + case XDP_ABORTED: + trace_xdp_exception(priv->dev, xdp_prog, act); + case XDP_DROP: + goto err_xdp; + } + } + rcu_read_unlock(); + + headroom = frame->data - delta - (void *)frame; + skb = veth_build_skb(frame, headroom, len, 0); + if (!skb) { + xdp_return_frame(frame); + goto err; + } + + memset(frame, 0, sizeof(*frame)); + skb->protocol = eth_type_trans(skb, priv->dev); +err: + return skb; +err_xdp: + rcu_read_unlock(); + xdp_return_frame(frame); + + return NULL; +} + static struct sk_buff *veth_xdp_rcv_skb(struct veth_priv *priv, struct sk_buff *skb) { @@ -352,21 +413,53 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_priv *priv, static int veth_xdp_rcv(struct veth_priv *priv, int budget) { - int i, done = 0; + int done = 0; + bool more; - for (i = 0; i < budget; i++) { - struct sk_buff *skb = __ptr_ring_consume(&priv->xdp_ring); + do { + int curr_budget, i; + bool curr_more; - if (!skb) - break; + more = false; - skb = veth_xdp_rcv_skb(priv, skb); + curr_more = true; + curr_budget = min(budget - done, budget >> 1); + for (i = 0; i < curr_budget; i++) { + struct xdp_frame *frame; + struct sk_buff *skb; - if (skb) - napi_gro_receive(&priv->xdp_napi, skb); + frame = __ptr_ring_consume(&priv->xdp_tx_ring); + if (!frame) { + curr_more = false; + break; + } - done++; - } + skb = veth_xdp_rcv_one(priv, frame); + if (skb) + napi_gro_receive(&priv->xdp_napi, skb); + + done++; + } + more |= curr_more; + + curr_more = true; + curr_budget = min(budget - done, budget >> 1); + for (i = 0; i < curr_budget; i++) { + struct sk_buff *skb = __ptr_ring_consume(&priv->xdp_ring); + + if (!skb) { + curr_more = false; + break; + } + + skb = veth_xdp_rcv_skb(priv, skb); + if (skb) + napi_gro_receive(&priv->xdp_napi, skb); + + done++; + } + more |= curr_more; + } while (more && done < budget); return done; } @@ -382,7 +475,8 @@ static int veth_poll(struct napi_struct *napi, int budget) if (done < budget && napi_complete_done(napi, done)) { /* Write rx_notify_masked before reading ptr_ring */ smp_store_mb(priv->rx_notify_masked, false); - if (unlikely(!__ptr_ring_empty(&priv->xdp_ring))) { + if (unlikely(!__ptr_ring_empty(&priv->xdp_tx_ring) || + !__ptr_ring_empty(&priv->xdp_ring))) { priv->rx_notify_masked = true; napi_schedule(&priv->xdp_napi); } @@ -400,10 +494,18 @@ static int veth_napi_add(struct net_device *dev) if (err) return err; + err = ptr_ring_init(&priv->xdp_tx_ring, VETH_RING_SIZE, GFP_KERNEL); + if (err) + goto err_xdp_tx_ring; + netif_napi_add(dev, &priv->xdp_napi, veth_poll, NAPI_POLL_WEIGHT); napi_enable(&priv->xdp_napi); return 0; +err_xdp_tx_ring: + ptr_ring_cleanup(&priv->xdp_ring, __skb_array_destroy_skb); + + return err; } static void veth_napi_del(struct net_device *dev) @@ -413,6 +515,7 @@ static void veth_napi_del(struct net_device *dev) napi_disable(&priv->xdp_napi); netif_napi_del(&priv->xdp_napi); ptr_ring_cleanup(&priv->xdp_ring, __skb_array_destroy_skb); + ptr_ring_cleanup(&priv->xdp_tx_ring, veth_xdp_free); } static int veth_enable_xdp(struct net_device *dev)