From patchwork Wed Dec 26 20:27:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: William Tu X-Patchwork-Id: 1018731 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="X0MwM1VF"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43Q4L42WYfz9sCr for ; Thu, 27 Dec 2018 07:28:28 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727685AbeLZU2X (ORCPT ); Wed, 26 Dec 2018 15:28:23 -0500 Received: from mail-pf1-f195.google.com ([209.85.210.195]:45714 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727513AbeLZU2W (ORCPT ); Wed, 26 Dec 2018 15:28:22 -0500 Received: by mail-pf1-f195.google.com with SMTP id g62so8215847pfd.12 for ; Wed, 26 Dec 2018 12:28:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=C8sDSQVHPnIyd6UC0lHfEe89QhjRPtO9aa9eLwG/GjE=; b=X0MwM1VF8LO7eYRL6nxQR6X4m7ukgeNyudk0EhKkBeVgZ5mYFndlceR3OtCPIzGAxb /STyKY0m78mHiofGRpXNIYkSn7aRkdkJx2C8PWlWPmY4ZLRstXQzP/eTNCuGZipfZCzC CbgUrvyfjmIU395KZS/5VrHQYZIwDOcd/JZIi0M0qZOO8pcqqzZnSazkl6psRFnUXKnM qU0Pl31uB6Z/UKF7giND12RAZnyT0ySWPE6nQgaFQHo8PlpxCO6WAiPvKwuMsCyyBPRY IF8xX8kbUaZMAT1oHa65u8cNWzPrvflgM73bfp1D54iTYCkzDF/ZvoXyHlXXH40iwwMY uEhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=C8sDSQVHPnIyd6UC0lHfEe89QhjRPtO9aa9eLwG/GjE=; b=XA44c7MLhNyBHdnO8KS7ucEFo/rMUlfKUSax5cnivHEax7TxfKCgy32jXLqI3Gq8jb EWcUvYgBjwLaQjWihiPoRW/uQKaR9aiab3tLD3CUp3w8mBa7lYNZlv2xd5+HoEs2jjvq Wjj8TC5lQMGvXbJWkAGtlurgc4mNgQZr4WeR8pdBD3GdvLpYxrAOgYgFQHLjZSPTzmj1 92krKzBeWWAZ57vObHD3/uGu3QpLUKxSZPBPdnrQxyyH7ZDRyA5JonGqpz25wrZ9B/kw T+hnCsp33tM7XQG8hvzTcPpNlXDO1a9CIPdPNT2GYTiclA72YAoNiILw8rK4af1AMQoi 1CWg== X-Gm-Message-State: AJcUukdMnT5hBjXr9ZCCTSJUgMbflBtEYwMnTZnviUIaaBbEFU4QwHVw B/J7kQkvmzZ2e41SbOKlZ7IA+/rf X-Google-Smtp-Source: ALg8bN63eeHwdIKNIhtol25XyvtUVWljm2XKl5a3O5ZnRFRHE1/l7r4mUuQsKUXjrye2W/IIMMEj7Q== X-Received: by 2002:a63:d301:: with SMTP id b1mr20348429pgg.61.1545856100351; Wed, 26 Dec 2018 12:28:20 -0800 (PST) Received: from sc9-mailhost2.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id t90sm72712460pfj.23.2018.12.26.12.28.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 26 Dec 2018 12:28:19 -0800 (PST) From: William Tu To: bjorn.topel@gmail.com, magnus.karlsson@gmail.com, ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org, makita.toshiaki@lab.ntt.co.jp, yihung.wei@gmail.com, magnus.karlsson@intel.com Subject: [PATCH bpf-next RFCv3 1/6] xsk: add xsk_umem_consume_tx_virtual. Date: Wed, 26 Dec 2018 12:27:48 -0800 Message-Id: <1545856073-8680-2-git-send-email-u9012063@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1545856073-8680-1-git-send-email-u9012063@gmail.com> References: <1545856073-8680-1-git-send-email-u9012063@gmail.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Currently the xsk_umem_consume_tx expects only the physical NICs so the api returns a dma address. This patch introduce the new function to return the virtual address, when XSK is used by a virtual device. Signed-off-by: William Tu --- include/net/xdp_sock.h | 7 +++++++ net/xdp/xdp_umem.c | 1 + net/xdp/xsk.c | 38 +++++++++++++++++++++++++++++++------- 3 files changed, 39 insertions(+), 7 deletions(-) diff --git a/include/net/xdp_sock.h b/include/net/xdp_sock.h index 13acb9803a6d..7fefe74f7fb5 100644 --- a/include/net/xdp_sock.h +++ b/include/net/xdp_sock.h @@ -81,6 +81,7 @@ u64 *xsk_umem_peek_addr(struct xdp_umem *umem, u64 *addr); void xsk_umem_discard_addr(struct xdp_umem *umem); void xsk_umem_complete_tx(struct xdp_umem *umem, u32 nb_entries); bool xsk_umem_consume_tx(struct xdp_umem *umem, dma_addr_t *dma, u32 *len); +bool xsk_umem_consume_tx_virtual(struct xdp_umem *umem, void **addr, u32 *len); void xsk_umem_consume_tx_done(struct xdp_umem *umem); struct xdp_umem_fq_reuse *xsk_reuseq_prepare(u32 nentries); struct xdp_umem_fq_reuse *xsk_reuseq_swap(struct xdp_umem *umem, @@ -165,6 +166,12 @@ static inline bool xsk_umem_consume_tx(struct xdp_umem *umem, dma_addr_t *dma, return false; } +static inline bool xsk_umem_consume_tx_virtual(struct xdp_umem *umem, + void **vaddr, u32 *len) +{ + return false; +} + static inline void xsk_umem_consume_tx_done(struct xdp_umem *umem) { } diff --git a/net/xdp/xdp_umem.c b/net/xdp/xdp_umem.c index a264cf2accd0..424ae2538f9f 100644 --- a/net/xdp/xdp_umem.c +++ b/net/xdp/xdp_umem.c @@ -60,6 +60,7 @@ struct xdp_umem *xdp_get_umem_from_qid(struct net_device *dev, return NULL; } +EXPORT_SYMBOL(xdp_get_umem_from_qid); static void xdp_clear_umem_at_qid(struct net_device *dev, u16 queue_id) { diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index 07156f43d295..0e252047f55f 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -170,22 +170,19 @@ void xsk_umem_consume_tx_done(struct xdp_umem *umem) } EXPORT_SYMBOL(xsk_umem_consume_tx_done); -bool xsk_umem_consume_tx(struct xdp_umem *umem, dma_addr_t *dma, u32 *len) +static __always_inline bool __xsk_umem_consume_tx(struct xdp_umem *umem, + struct xdp_desc *desc) { - struct xdp_desc desc; struct xdp_sock *xs; rcu_read_lock(); list_for_each_entry_rcu(xs, &umem->xsk_list, list) { - if (!xskq_peek_desc(xs->tx, &desc)) + if (!xskq_peek_desc(xs->tx, desc)) continue; - if (xskq_produce_addr_lazy(umem->cq, desc.addr)) + if (xskq_produce_addr_lazy(umem->cq, desc->addr)) goto out; - *dma = xdp_umem_get_dma(umem, desc.addr); - *len = desc.len; - xskq_discard_desc(xs->tx); rcu_read_unlock(); return true; @@ -195,8 +192,35 @@ bool xsk_umem_consume_tx(struct xdp_umem *umem, dma_addr_t *dma, u32 *len) rcu_read_unlock(); return false; } + +bool xsk_umem_consume_tx(struct xdp_umem *umem, dma_addr_t *dma, u32 *len) +{ + struct xdp_desc desc; + + if (!__xsk_umem_consume_tx(umem, &desc)) + return false; + + *dma = xdp_umem_get_dma(umem, desc.addr); + *len = desc.len; + + return true; +} EXPORT_SYMBOL(xsk_umem_consume_tx); +bool xsk_umem_consume_tx_virtual(struct xdp_umem *umem, void **vaddr, u32 *len) +{ + struct xdp_desc desc; + + if (!__xsk_umem_consume_tx(umem, &desc)) + return false; + + *vaddr = xdp_umem_get_data(umem, desc.addr); + *len = desc.len; + + return true; +} +EXPORT_SYMBOL(xsk_umem_consume_tx_virtual); + static int xsk_zc_xmit(struct sock *sk) { struct xdp_sock *xs = xdp_sk(sk); From patchwork Wed Dec 26 20:27:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: William Tu X-Patchwork-Id: 1018730 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="tI6oJEXe"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43Q4L30crnz9s3l for ; Thu, 27 Dec 2018 07:28:27 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727697AbeLZU2Y (ORCPT ); Wed, 26 Dec 2018 15:28:24 -0500 Received: from mail-pg1-f196.google.com ([209.85.215.196]:41187 "EHLO mail-pg1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727674AbeLZU2W (ORCPT ); Wed, 26 Dec 2018 15:28:22 -0500 Received: by mail-pg1-f196.google.com with SMTP id m1so7916879pgq.8 for ; Wed, 26 Dec 2018 12:28:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=Rvp4tCA03NhWaP8VAiDM4rXCpO7gQ18Uxv5ij1P/5yE=; b=tI6oJEXeK7J2MBdNXhTWqLbMh6mNbwiutn2EvY4QOhO8ORYk+11sty6PFgZ59X9blB FqcPUpxWvDby1Y2corAnCS1ygNNLpBs9lzeAWN/u4Aw2rVTIWg98qV/ZnDgPGmg0p+0g youIYVN6NO8iagbylXddInqRdU+di45+4TCNqXBupEgdrfmQjrwxKXnaNNIBDwpShmFh dNOP9tgj9c0PzE53KyfttcoH1dKhPmrvTBNrdahnJ3lvgg+Flg8eqed8WCxVm5iuFMwq s3AiMCsMnRhOY3c+Z2dI9QG4HHb4L/2t3IkWt5abSD847IdEbLgpexE1AntJB63u91yv tj6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=Rvp4tCA03NhWaP8VAiDM4rXCpO7gQ18Uxv5ij1P/5yE=; b=VidbEjRYqyZG3xyfOVGzR7C9PtbUq7Fs7jlzwydhoJfOy+fIRX/C5eNcEbfZ57yBxq mw18N4H8dGMEdkzGQfTN/YbLRaKLVhPYvuW7EaR0EQcBpRH4OYG9hAzYZ0GWy+nkA5Bw t8MYL6RD80nEplv4EDuLeu+N3d3OmKoHxouNiQtr6J7PRag9J0sRj+eZXcfgSuGAq0nn CkYT/5TX3btOvYBuKmQBIDwNP79ZhJKcinf4Qf6sS+Kj1MuSDlln4NHlCuw036buwDHv YhyfDwDGTk/LhvGIUZ/lC0s2rm7kPGhGxFDfCXcVwfwODgkxOl7qVVAb0jauaG/IhXYt 8e7Q== X-Gm-Message-State: AA+aEWaQfgb5OcX51qIwUlcLfZshecz+k0lbAAYR9PwMWkhFFKdgd5N0 eiYqMNu7hIpkQ27GDrVQSlg= X-Google-Smtp-Source: AFSGD/VRk9dp4IWh6MFVxuYoBrzeULoKTFQjvYocCZ02vIPjGQaSe9dqih2u8fYNIOa7fK3DmtVN3A== X-Received: by 2002:a62:7c47:: with SMTP id x68mr21726538pfc.209.1545856101269; Wed, 26 Dec 2018 12:28:21 -0800 (PST) Received: from sc9-mailhost2.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id t90sm72712460pfj.23.2018.12.26.12.28.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 26 Dec 2018 12:28:20 -0800 (PST) From: William Tu To: bjorn.topel@gmail.com, magnus.karlsson@gmail.com, ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org, makita.toshiaki@lab.ntt.co.jp, yihung.wei@gmail.com, magnus.karlsson@intel.com Subject: [PATCH bpf-next RFCv3 2/6] veth: support AF_XDP TX copy-mode. Date: Wed, 26 Dec 2018 12:27:49 -0800 Message-Id: <1545856073-8680-3-git-send-email-u9012063@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1545856073-8680-1-git-send-email-u9012063@gmail.com> References: <1545856073-8680-1-git-send-email-u9012063@gmail.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org The patch adds support for AF_XDP async xmit. Users can use AF_XDP on both sides of the veth and get better performance, with the cost of ksoftirqd doing the xmit. The veth_xsk_async_xmit simply kicks the napi function, veth_poll, to receive the packets that are on the umem transmit ring at the _peer_ side. Tested using two namespaces, one runs xdpsock and the other runs xdp_rxq_info. A simple script comparing the performance with/without AF_XDP shows improvement from 724Kpps to 1.1Mpps. ip netns add at_ns0 ip link add p0 type veth peer name p1 ip link set p0 netns at_ns0 ip link set dev p1 up ip netns exec at_ns0 ip link set dev p0 up # receiver ip netns exec at_ns0 xdp_rxq_info --dev p0 --action XDP_DROP # sender xdpsock -i p1 -t -N -z or xdpsock -i p1 -t -S Signed-off-by: William Tu --- drivers/net/veth.c | 200 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 199 insertions(+), 1 deletion(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index f412ea1cef18..10cf9ded59f1 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -25,6 +25,10 @@ #include #include #include +#include +#include +#include +#include #define DRV_NAME "veth" #define DRV_VERSION "1.0" @@ -53,6 +57,8 @@ struct veth_rq { bool rx_notify_masked; struct ptr_ring xdp_ring; struct xdp_rxq_info xdp_rxq; + struct xdp_umem *xsk_umem; + u16 qid; }; struct veth_priv { @@ -737,11 +743,95 @@ static int veth_xdp_rcv(struct veth_rq *rq, int budget, unsigned int *xdp_xmit) return done; } +static int veth_xsk_poll(struct napi_struct *napi, int budget) +{ + struct veth_priv *priv, *peer_priv; + struct net_device *dev, *peer_dev; + struct veth_rq *peer_rq; + struct veth_rq *rq = + container_of(napi, struct veth_rq, xdp_napi); + int done = 0; + + dev = rq->dev; + priv = netdev_priv(dev); + peer_dev = priv->peer; + peer_priv = netdev_priv(peer_dev); + peer_rq = &peer_priv->rq[rq->qid]; + + while (peer_rq->xsk_umem && budget--) { + unsigned int inner_xdp_xmit = 0; + unsigned int metasize = 0; + struct xdp_frame *xdpf; + bool dropped = false; + struct sk_buff *skb; + struct page *page; + void *vaddr; + void *addr; + u32 len; + + if (!xsk_umem_consume_tx_virtual(peer_rq->xsk_umem, &vaddr, &len)) + break; + + page = dev_alloc_page(); + if (!page) { + xsk_umem_complete_tx(peer_rq->xsk_umem, 1); + xsk_umem_consume_tx_done(peer_rq->xsk_umem); + return -ENOMEM; + } + + addr = page_to_virt(page); + xdpf = addr; + memset(xdpf, 0, sizeof(*xdpf)); + + addr += sizeof(*xdpf); + memcpy(addr, vaddr, len); + + xdpf->data = addr + metasize; + xdpf->len = len; + xdpf->headroom = 0; + xdpf->metasize = metasize; + xdpf->mem.type = MEM_TYPE_PAGE_SHARED; + + /* put into rq */ + skb = veth_xdp_rcv_one(rq, xdpf, &inner_xdp_xmit); + if (!skb) { + /* Peer side has XDP program attached */ + if (inner_xdp_xmit & VETH_XDP_TX) { + /* Not supported */ + pr_warn("veth: peer XDP_TX not supported\n"); + xdp_return_frame(xdpf); + dropped = true; + goto skip_tx; + } else if (inner_xdp_xmit & VETH_XDP_REDIR) { + xdp_do_flush_map(); + } else { + dropped = true; + } + } else { + napi_gro_receive(&rq->xdp_napi, skb); + } +skip_tx: + xsk_umem_complete_tx(peer_rq->xsk_umem, 1); + xsk_umem_consume_tx_done(peer_rq->xsk_umem); + + /* update rq stats */ + u64_stats_update_begin(&rq->stats.syncp); + rq->stats.xdp_packets++; + rq->stats.xdp_bytes += len; + if (dropped) + rq->stats.xdp_drops++; + u64_stats_update_end(&rq->stats.syncp); + done++; + } + return done; +} + static int veth_poll(struct napi_struct *napi, int budget) { struct veth_rq *rq = container_of(napi, struct veth_rq, xdp_napi); unsigned int xdp_xmit = 0; + int tx_done; int done; xdp_set_return_frame_no_direct(); @@ -756,13 +846,17 @@ static int veth_poll(struct napi_struct *napi, int budget) } } + tx_done = veth_xsk_poll(napi, budget); + if (tx_done > 0) + done += tx_done; + if (xdp_xmit & VETH_XDP_TX) veth_xdp_flush(rq->dev); if (xdp_xmit & VETH_XDP_REDIR) xdp_do_flush_map(); xdp_clear_return_frame_no_direct(); - return done; + return done > budget ? budget : done; } static int veth_napi_add(struct net_device *dev) @@ -776,6 +870,7 @@ static int veth_napi_add(struct net_device *dev) err = ptr_ring_init(&rq->xdp_ring, VETH_RING_SIZE, GFP_KERNEL); if (err) goto err_xdp_ring; + rq->qid = i; } for (i = 0; i < dev->real_num_rx_queues; i++) { @@ -812,6 +907,7 @@ static void veth_napi_del(struct net_device *dev) netif_napi_del(&rq->xdp_napi); rq->rx_notify_masked = false; ptr_ring_cleanup(&rq->xdp_ring, veth_ptr_free); + rq->qid = -1; } } @@ -836,6 +932,7 @@ static int veth_enable_xdp(struct net_device *dev) /* Save original mem info as it can be overwritten */ rq->xdp_mem = rq->xdp_rxq.mem; + rq->qid = i; } err = veth_napi_add(dev); @@ -1115,6 +1212,84 @@ static u32 veth_xdp_query(struct net_device *dev) return 0; } +int veth_xsk_umem_query(struct net_device *dev, struct xdp_umem **umem, + u16 qid) +{ + struct xdp_umem *queried_umem; + + queried_umem = xdp_get_umem_from_qid(dev, qid); + + if (!queried_umem) + return -EINVAL; + + *umem = queried_umem; + return 0; +} + +static int veth_xsk_umem_enable(struct net_device *dev, + struct xdp_umem *umem, + u16 qid) +{ + struct veth_priv *priv = netdev_priv(dev); + struct xdp_umem_fq_reuse *reuseq; + int err = 0; + + if (qid >= dev->real_num_rx_queues) + return -EINVAL; + + reuseq = xsk_reuseq_prepare(priv->rq[0].xdp_ring.size); + if (!reuseq) + return -ENOMEM; + + xsk_reuseq_free(xsk_reuseq_swap(umem, reuseq)); + + priv->rq[qid].xsk_umem = umem; + + return err; +} + +static int veth_xsk_umem_disable(struct net_device *dev, + u16 qid) +{ + struct veth_priv *priv = netdev_priv(dev); + struct xdp_umem *umem; + + umem = xdp_get_umem_from_qid(dev, qid); + if (!umem) + return -EINVAL; + + priv->rq[qid].xsk_umem = NULL; + return 0; +} + +int veth_xsk_umem_setup(struct net_device *dev, struct xdp_umem *umem, + u16 qid) +{ + return umem ? veth_xsk_umem_enable(dev, umem, qid) : + veth_xsk_umem_disable(dev, qid); +} + +int veth_xsk_async_xmit(struct net_device *dev, u32 qid) +{ + struct veth_priv *priv, *peer_priv; + struct net_device *peer_dev; + struct veth_rq *peer_rq; + + priv = netdev_priv(dev); + peer_dev = priv->peer; + peer_priv = netdev_priv(peer_dev); + peer_rq = &peer_priv->rq[qid]; + + if (qid >= dev->real_num_rx_queues) + return -ENXIO; + + /* Schedule the peer side NAPI to receive */ + if (!napi_if_scheduled_mark_missed(&peer_rq->xdp_napi)) + napi_schedule(&peer_rq->xdp_napi); + + return 0; +} + static int veth_xdp(struct net_device *dev, struct netdev_bpf *xdp) { switch (xdp->command) { @@ -1123,6 +1298,28 @@ static int veth_xdp(struct net_device *dev, struct netdev_bpf *xdp) case XDP_QUERY_PROG: xdp->prog_id = veth_xdp_query(dev); return 0; + case XDP_QUERY_XSK_UMEM: + return veth_xsk_umem_query(dev, &xdp->xsk.umem, + xdp->xsk.queue_id); + case XDP_SETUP_XSK_UMEM: { + struct veth_priv *priv; + int err; + + /* Enable NAPI on both sides, by enabling + * their XDP. + */ + err = veth_enable_xdp(dev); + if (err) + return err; + + priv = netdev_priv(dev); + err = veth_enable_xdp(priv->peer); + if (err) + return err; + + return veth_xsk_umem_setup(dev, xdp->xsk.umem, + xdp->xsk.queue_id); + } default: return -EINVAL; } @@ -1145,6 +1342,7 @@ static const struct net_device_ops veth_netdev_ops = { .ndo_set_rx_headroom = veth_set_rx_headroom, .ndo_bpf = veth_xdp, .ndo_xdp_xmit = veth_xdp_xmit, + .ndo_xsk_async_xmit = veth_xsk_async_xmit, }; #define VETH_FEATURES (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_HW_CSUM | \ From patchwork Wed Dec 26 20:27:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: William Tu X-Patchwork-Id: 1018735 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="EYogZ3VY"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43Q4LD3F7Nz9sCr for ; Thu, 27 Dec 2018 07:28:36 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727859AbeLZU2f (ORCPT ); Wed, 26 Dec 2018 15:28:35 -0500 Received: from mail-pg1-f196.google.com ([209.85.215.196]:44809 "EHLO mail-pg1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727676AbeLZU2X (ORCPT ); Wed, 26 Dec 2018 15:28:23 -0500 Received: by mail-pg1-f196.google.com with SMTP id t13so7907296pgr.11 for ; Wed, 26 Dec 2018 12:28:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=pFiWK5NTUhS+xzWXgght5NgoJ++lA1LDOOXLe9Bk4kc=; b=EYogZ3VYI8TVRf5KIoo0hkrwLUiDVxj9n0CYUEU4ie2IdyZXAvi3zWFEpVQme0leBc 0zm/bTqPor203+JBLT3Gfhs/rBku24d0SE8r6NeqVh6aEBVcsm/2viX+xRJLKDjmaf6+ 0cu099XmzAw4a64b5vuCb+IYC3wYVB/YCHKmFtTlOqC2qAvh83zjsHXPWxnLWGvp7HlC pY3e2dae0FeiqU2f1GRMkzngDbZoCPuJ/9bvy97ovlFXWY3STbH5xyNPoZG73FCbQuz+ QNyK/ojxklQkICZlJDrdJ2/9HkWSF125BGQpwV3aCu5qV0AOOD7i30PGjGyWZl2U/aRf 43pQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=pFiWK5NTUhS+xzWXgght5NgoJ++lA1LDOOXLe9Bk4kc=; b=B/H5sKvZs7uruv0cnmIosQV+0mcO7P1jLEogKStDdch3RCg3qbOr/UtRJQ/yv51E0y ckEO2Z1bMSEhNMSc0rNCbwntPHlvLOMutLTKpBLpSXNmtIv4v6AmMGVQ8Z4wdo84jG46 z4VQYR4Va9VTiQ0o+oyMMjK175Nzz9UmST6gQV9JhTM33NKNSoNig2RUzzjRDAXTwxvM mfv68Nw+eE0dF2tKG9OK0PAQsU+E+quuWgRnvuIrIWbPRb6sVpb4vcyxsDFldQrE9pha TJuGB1R8f627MiQ8O1ykGLDEEkmIl6LN0gk9EudiTTk4pa7pkSNx5xDgQm8MoDRowVZ2 cbow== X-Gm-Message-State: AJcUukdUFqDRXDaGc2i8GvTJMReEcOkexe6gY3bZ6X6/0MRlM3j6Uarh 8fSrcglF53hnYb56I0z/XOA= X-Google-Smtp-Source: ALg8bN501HUYJsbjAlbJzPOUu7ESc2yABhmX6CEzParyUGJETpdp2q+zuJ4pqbmbN40lzlWvw7ppmQ== X-Received: by 2002:a63:78cd:: with SMTP id t196mr20270427pgc.62.1545856102292; Wed, 26 Dec 2018 12:28:22 -0800 (PST) Received: from sc9-mailhost2.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id t90sm72712460pfj.23.2018.12.26.12.28.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 26 Dec 2018 12:28:21 -0800 (PST) From: William Tu To: bjorn.topel@gmail.com, magnus.karlsson@gmail.com, ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org, makita.toshiaki@lab.ntt.co.jp, yihung.wei@gmail.com, magnus.karlsson@intel.com Subject: [PATCH bpf-next RFCv3 3/6] xsk: add new MEM type for virtual device. Date: Wed, 26 Dec 2018 12:27:50 -0800 Message-Id: <1545856073-8680-4-git-send-email-u9012063@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1545856073-8680-1-git-send-email-u9012063@gmail.com> References: <1545856073-8680-1-git-send-email-u9012063@gmail.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Add MEM_TYPE_ZERO_COPY_VDEV for supporting the veth AF_XDP. For zero-copy veth, the memory comes from userspace and does not need to be freed at kernel. Thus, when xdp returns the frame, detecting this type and doing nothing. Signed-off-by: William Tu --- include/net/xdp.h | 1 + net/core/xdp.c | 1 + net/xdp/xsk.c | 3 ++- 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/include/net/xdp.h b/include/net/xdp.h index 0f25b3675c5c..010cb9efca90 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -38,6 +38,7 @@ enum xdp_mem_type { MEM_TYPE_PAGE_ORDER0, /* Orig XDP full page model */ MEM_TYPE_PAGE_POOL, MEM_TYPE_ZERO_COPY, + MEM_TYPE_ZERO_COPY_VDEV, MEM_TYPE_MAX, }; diff --git a/net/core/xdp.c b/net/core/xdp.c index 4b2b194f4f1f..d47f003714c4 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -355,6 +355,7 @@ static void __xdp_return(void *data, struct xdp_mem_info *mem, bool napi_direct, xa = rhashtable_lookup(mem_id_ht, &mem->id, mem_id_rht_params); xa->zc_alloc->free(xa->zc_alloc, handle); rcu_read_unlock(); + case MEM_TYPE_ZERO_COPY_VDEV: default: /* Not possible, checked in xdp_rxq_info_reg_mem_model() */ break; diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index 0e252047f55f..3d151d64fc79 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -109,7 +109,8 @@ int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp) len = xdp->data_end - xdp->data; - return (xdp->rxq->mem.type == MEM_TYPE_ZERO_COPY) ? + return (xdp->rxq->mem.type == MEM_TYPE_ZERO_COPY || + xdp->rxq->mem.type == MEM_TYPE_ZERO_COPY_VDEV) ? __xsk_rcv_zc(xs, xdp, len) : __xsk_rcv(xs, xdp, len); } From patchwork Wed Dec 26 20:27:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: William Tu X-Patchwork-Id: 1018733 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ikXwQqk7"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43Q4L73kXTz9sCr for ; Thu, 27 Dec 2018 07:28:31 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727819AbeLZU23 (ORCPT ); Wed, 26 Dec 2018 15:28:29 -0500 Received: from mail-pf1-f196.google.com ([209.85.210.196]:42757 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727689AbeLZU2Y (ORCPT ); Wed, 26 Dec 2018 15:28:24 -0500 Received: by mail-pf1-f196.google.com with SMTP id 64so8216756pfr.9 for ; Wed, 26 Dec 2018 12:28:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=5IK0vZiHxM5SDcSN+srCKjjCaT5uK7+dKJmAZ7vek3Y=; b=ikXwQqk7rG1LxucPFQgQxA3oigaOjl/VJXXKQXfETMtwXZPryr5GjkwXoHMazNCjiK Cs/k8DtszJ3XEhhWph7BxYsEdg8a/XiJcajxjEDu/6uGxujrhQsO/37zbkIr2OL1qSKV qMR4sSg/T3qmTAc+ww1Lxy1ITkXDC8xmoJGmYXcYYA/ZObfLRQJ1hf6nuPen49DIsPBN paQUDM83rzWr8YirHNR2Hub8XRj94rvP8CC/wEuABeTzbbWGEJMXUbTWa0xLV6NV+Cao 9y4dYtoOlS4NMRx8XV/r/A2TX5tQI0rxtt3c/MpeBoYuaca1J2rPHWs3aB0sHxkDt/kW KfxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=5IK0vZiHxM5SDcSN+srCKjjCaT5uK7+dKJmAZ7vek3Y=; b=RdiBM+vY7j7o3ZD37UJBl4FXepiu5TjWrCRW33AiraFRWallJLhoJquAaDO5+tZWe7 5nWGIApbIOkXrrd1KOBcsuMpyUwAi734Mq7mGs4K688MPl9xm9O0DzK501qSVzXJQTiJ S/dt3BU+fwzIjaZjxu1EwzBeX8LWQCtdP8ueT+rUiYNN49fC3yyk71GuEKZpaPfD0+n+ SaOsa3ObqmX1F/Xoy178r34mWfUWcZHLERd/HuSeXmNo7PsEUL9MnKo5HZubBIGWGGdS wVMKuH/AKiXb7lHUemYlUAq/S+oSUZaA0S9fYr0GjgV2/B55bXyNEVriVG9AYQVnh9ZX kUAg== X-Gm-Message-State: AA+aEWZ48BNf8GbfRjGXS7VCdZxKE3w9Z/nlPvxFE52NGdRJzORqoOFW 5A2387cdgJU6DNnevJVzrTFQFwaA X-Google-Smtp-Source: AFSGD/U87PsfUz3TeZfjsXGjLyLQ8IVqSdBocwJSh57G0jRNwI2z/CQRzH6pGMP2WbuDskumpcVCHQ== X-Received: by 2002:a62:a99:: with SMTP id 25mr21164077pfk.121.1545856103317; Wed, 26 Dec 2018 12:28:23 -0800 (PST) Received: from sc9-mailhost2.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id t90sm72712460pfj.23.2018.12.26.12.28.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 26 Dec 2018 12:28:22 -0800 (PST) From: William Tu To: bjorn.topel@gmail.com, magnus.karlsson@gmail.com, ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org, makita.toshiaki@lab.ntt.co.jp, yihung.wei@gmail.com, magnus.karlsson@intel.com Subject: [PATCH bpf-next RFCv3 4/6] veth: add zero-copy AF_XDP TX support. Date: Wed, 26 Dec 2018 12:27:51 -0800 Message-Id: <1545856073-8680-5-git-send-email-u9012063@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1545856073-8680-1-git-send-email-u9012063@gmail.com> References: <1545856073-8680-1-git-send-email-u9012063@gmail.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Remove the extra copy when doing AF_XDP TX. The xdp frame comes directly from the umem element and passes to the receiving logic. Also, only depending on async_xmit to kick napi poll isn't fast enough. So re-schedule the napi at the end of poll so the ksoftirqd can keep processing the packets. The performance increases from 1.1Mpps to 1.4Mpps, when running zero copy xdpsock as sender and XDP_DROP at the receiver side. Signed-off-by: William Tu --- drivers/net/veth.c | 41 ++++++++++++++++++----------------------- 1 file changed, 18 insertions(+), 23 deletions(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 10cf9ded59f1..551444195398 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -761,45 +761,34 @@ static int veth_xsk_poll(struct napi_struct *napi, int budget) while (peer_rq->xsk_umem && budget--) { unsigned int inner_xdp_xmit = 0; unsigned int metasize = 0; - struct xdp_frame *xdpf; + struct xdp_frame xdpf; bool dropped = false; struct sk_buff *skb; struct page *page; void *vaddr; - void *addr; u32 len; if (!xsk_umem_consume_tx_virtual(peer_rq->xsk_umem, &vaddr, &len)) break; - page = dev_alloc_page(); - if (!page) { - xsk_umem_complete_tx(peer_rq->xsk_umem, 1); - xsk_umem_consume_tx_done(peer_rq->xsk_umem); - return -ENOMEM; - } - - addr = page_to_virt(page); - xdpf = addr; - memset(xdpf, 0, sizeof(*xdpf)); - - addr += sizeof(*xdpf); - memcpy(addr, vaddr, len); + xdpf.data = vaddr + metasize; + xdpf.len = len; + xdpf.headroom = 0; + xdpf.metasize = metasize; + xdpf.mem.type = MEM_TYPE_ZERO_COPY_VDEV; - xdpf->data = addr + metasize; - xdpf->len = len; - xdpf->headroom = 0; - xdpf->metasize = metasize; - xdpf->mem.type = MEM_TYPE_PAGE_SHARED; + page = virt_to_head_page(vaddr); + if (page->mem_cgroup) + page->mem_cgroup = NULL; /* put into rq */ - skb = veth_xdp_rcv_one(rq, xdpf, &inner_xdp_xmit); + skb = veth_xdp_rcv_one(rq, &xdpf, &inner_xdp_xmit); if (!skb) { /* Peer side has XDP program attached */ if (inner_xdp_xmit & VETH_XDP_TX) { /* Not supported */ pr_warn("veth: peer XDP_TX not supported\n"); - xdp_return_frame(xdpf); + xdp_return_frame(&xdpf); dropped = true; goto skip_tx; } else if (inner_xdp_xmit & VETH_XDP_REDIR) { @@ -808,7 +797,8 @@ static int veth_xsk_poll(struct napi_struct *napi, int budget) dropped = true; } } else { - napi_gro_receive(&rq->xdp_napi, skb); + napi_gro_receive(&rq->xdp_napi, skb_copy(skb, GFP_KERNEL)); + kfree(skb); } skip_tx: xsk_umem_complete_tx(peer_rq->xsk_umem, 1); @@ -856,6 +846,11 @@ static int veth_poll(struct napi_struct *napi, int budget) xdp_do_flush_map(); xdp_clear_return_frame_no_direct(); + /* schedule again so the CPU can keep receiving + * at higher rate + */ + napi_schedule(&rq->xdp_napi); + return done > budget ? budget : done; } From patchwork Wed Dec 26 20:27:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: William Tu X-Patchwork-Id: 1018732 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="WZAiDi6I"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43Q4L61wsJz9s3l for ; Thu, 27 Dec 2018 07:28:30 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727797AbeLZU22 (ORCPT ); Wed, 26 Dec 2018 15:28:28 -0500 Received: from mail-pl1-f195.google.com ([209.85.214.195]:36749 "EHLO mail-pl1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727674AbeLZU2Z (ORCPT ); Wed, 26 Dec 2018 15:28:25 -0500 Received: by mail-pl1-f195.google.com with SMTP id g9so7897136plo.3 for ; Wed, 26 Dec 2018 12:28:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=KBTn+8dsvS6Ket37B9sbkPQt2QppL5cxtBOM4iUJIRU=; b=WZAiDi6IcJMxOrW48pWl2syG3vXGzO7d53sg3QIrvDZgtuZmtETTlwpXEogLtZPEZD CDTu+WGotxu3mGnAHXyO5gSk1GDAEtq4TMz78rljvxezkhtAEIKKHdI/UQRbO4Yz/uxf 6OdH8Fi5GbD9ecZNZP9zg6DVHIzcQg8ulCPsi9CNx1qLdKxx7NNnRM06p7l0kE5k19uR GvUqHy5cm5IfC71knjwJeD0WCTI1opg0Y0rcNFMz3lWPE4RgzDvWC/NW++npk/B1at8F 77QnC8o8bAftGO9hUiPEIir7hUvyQHv7u4taTYEZR3AeXj+EZW6YYvKnTZMX6vWOz93f pomw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=KBTn+8dsvS6Ket37B9sbkPQt2QppL5cxtBOM4iUJIRU=; b=T4R5sojCuZJFASr92aP5WDSqm+90dKAOEoCREe0EYHNy8wh2PAAhHIVCxg0F5Ut1we 2BOxL4lSJzkaJYK40F1nnkW9ZgCFITdLipVBk7fL7mAr86o8AQs7u1JA54qkULISiuYl y/sY2CKJb8wmZlV1jtlnzkbWrWAgjOIDYfys2RJTyJGYl1S1tBSzQb2m11JNafH5M5GJ 5W0bihcVMK9uY+0kPMKqVTtE8pDegsvoTzgrsE+9GPv49sUe4YIn/45paiP3oAxJWBJi BesXhV6AZSRCJafixQrXmeCkMImnDzfE8exOGKLFNa/GTd9z/jWsfFcDlTf//BnL40z2 M6Nw== X-Gm-Message-State: AJcUukcSXNto98/7D5kB3/yvohA4s3rPJTPhEGgJ6dnx4eXGFPXkwqOj /LPg5nukaCooqexlFBuB4Lk= X-Google-Smtp-Source: ALg8bN7SFFcKRxxfu3+GcZhWmX8OhbsICV4h+Y87xhwUxNTxg07qSHG9izBZ0vqN6BKg+a7HQh9avg== X-Received: by 2002:a17:902:8641:: with SMTP id y1mr21019652plt.159.1545856104327; Wed, 26 Dec 2018 12:28:24 -0800 (PST) Received: from sc9-mailhost2.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id t90sm72712460pfj.23.2018.12.26.12.28.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 26 Dec 2018 12:28:23 -0800 (PST) From: William Tu To: bjorn.topel@gmail.com, magnus.karlsson@gmail.com, ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org, makita.toshiaki@lab.ntt.co.jp, yihung.wei@gmail.com, magnus.karlsson@intel.com Subject: [PATCH bpf-next RFCv3 5/6] veth: add AF_XDP RX support. Date: Wed, 26 Dec 2018 12:27:52 -0800 Message-Id: <1545856073-8680-6-git-send-email-u9012063@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1545856073-8680-1-git-send-email-u9012063@gmail.com> References: <1545856073-8680-1-git-send-email-u9012063@gmail.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org If the receiving veth side has umem rx enabled, the patch directly copy the packet from the peer side's send buffer to the umem receive buffer. This requires running AF_XDP as both side of the veth peer. For example: Receiver: # ip netns exec at_ns0 xdpsock -r -N -z -i p0 Sender: # xdpsock -i p1 -t -N -z The performance increases from 1.4Mpps to 2.3Mpps. Signed-off-by: William Tu --- drivers/net/veth.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 551444195398..8aac67554880 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -766,11 +766,28 @@ static int veth_xsk_poll(struct napi_struct *napi, int budget) struct sk_buff *skb; struct page *page; void *vaddr; + u64 handle; u32 len; if (!xsk_umem_consume_tx_virtual(peer_rq->xsk_umem, &vaddr, &len)) break; + if (rq->xsk_umem && xsk_umem_peek_addr(rq->xsk_umem, &handle)) { + char *daddr; + u64 hr = 0; + + /* the peer side also has umem enabled, + * copy directly to it. + */ + handle &= rq->xsk_umem->chunk_mask; + hr = rq->xsk_umem->headroom + XDP_PACKET_HEADROOM; + daddr = xdp_umem_get_data(rq->xsk_umem, handle); + daddr += hr; + memcpy((void *)daddr, vaddr, len); + xsk_umem_discard_addr(rq->xsk_umem); + vaddr = daddr; + } + xdpf.data = vaddr + metasize; xdpf.len = len; xdpf.headroom = 0; From patchwork Wed Dec 26 20:27:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: William Tu X-Patchwork-Id: 1018734 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="MfxGNACQ"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43Q4LB1KK7z9sCr for ; Thu, 27 Dec 2018 07:28:34 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727837AbeLZU2d (ORCPT ); Wed, 26 Dec 2018 15:28:33 -0500 Received: from mail-pf1-f196.google.com ([209.85.210.196]:40665 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727788AbeLZU20 (ORCPT ); Wed, 26 Dec 2018 15:28:26 -0500 Received: by mail-pf1-f196.google.com with SMTP id i12so8218833pfo.7 for ; Wed, 26 Dec 2018 12:28:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=gSM1tnW/B/PC4eiQ0xa6IkqOUcAdNgVZzbxSnDBz0O8=; b=MfxGNACQh+/YiRk0eJDl2KQQe9j4p/4KL8w67VY5nbyC3FuPzvwVwMn7MDqpxuMqL/ JJT4ZVuH/KdiStQVxJcWISrsJFGuktMOxbrW4AQKV/CuMibzGAgjLwTJkwXUm13jojgU 0jg4LMT2Lm8HgA4BpUhacJdodvIHbMDTlrn6K5xmz3nPdPbTXdGTCVRltwDOfJ2ZbIHH 4HAdW4DFFaIKzLcFFKoyvQrFVjZgYU0XRmFGkW++PTmn3Oew8RHgnJn71LR+mze7nFpo ehgACIJCTdbhxp/9e3hYOLmSmN56gwaKrhdsn8oWhaXPo7i/SHWdRRSdkRj3N8LOtLg0 0T7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=gSM1tnW/B/PC4eiQ0xa6IkqOUcAdNgVZzbxSnDBz0O8=; b=jC6yRtQCsMZjJQGssoCloEOeWo3Iy2yqOYY5E63t4qz2oFZn0IzhgI1cDk7czFiSRd cBjRspSslGTx9104eWg0Xdg4AO1JCSZmJsT+pvPhC9gXUnSXeURNPz6bIp+ts5lPBqCx yS1NTfnc6T+Rtz4NY3IfN+lFR9w6wK4DCwHKBWEO58fqNWAFquaMxDjkp2wFqgrnUS16 cHpbRkfVJGI1IT01dTXhPIpTn1Cx3dSjsuXNZNST/YhLLG5pne4+9w+5HOJG7d9uLwzr RvwSbfvCRGwSmAO911VYCKehDs2ukNPwh7YJ90vVb8cNsyugBlIFQOrGzC9GXz6OVcej 8Ymg== X-Gm-Message-State: AJcUuke+dnmDxeW1BaP2SbaqRQaN2Fnt7ioDkZad36VAXzrO2Tfp/LoW kb4X3n8o9iwOz3uyNtS/GuM= X-Google-Smtp-Source: ALg8bN4hTGNyMb+dZgLfNb2mvHVxb5Yj7RicXVIHt2w2jcAj+/6AwrGHR1EsKgzYoaZ6ihcq0xReHw== X-Received: by 2002:a63:2bc4:: with SMTP id r187mr19961815pgr.306.1545856105332; Wed, 26 Dec 2018 12:28:25 -0800 (PST) Received: from sc9-mailhost2.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id t90sm72712460pfj.23.2018.12.26.12.28.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 26 Dec 2018 12:28:24 -0800 (PST) From: William Tu To: bjorn.topel@gmail.com, magnus.karlsson@gmail.com, ast@kernel.org, daniel@iogearbox.net, netdev@vger.kernel.org, makita.toshiaki@lab.ntt.co.jp, yihung.wei@gmail.com, magnus.karlsson@intel.com Subject: [PATCH bpf-next RFCv3 6/6] samples: bpf: add veth AF_XDP example. Date: Wed, 26 Dec 2018 12:27:53 -0800 Message-Id: <1545856073-8680-7-git-send-email-u9012063@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1545856073-8680-1-git-send-email-u9012063@gmail.com> References: <1545856073-8680-1-git-send-email-u9012063@gmail.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Add example use cases for AF_XDP socket on two namespaces. The script runs sender at the root namespace, and receiver at the at_ns0 namespace with different XDP actions. Signed-off-by: William Tu --- samples/bpf/test_veth_afxdp.sh | 82 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100755 samples/bpf/test_veth_afxdp.sh diff --git a/samples/bpf/test_veth_afxdp.sh b/samples/bpf/test_veth_afxdp.sh new file mode 100755 index 000000000000..d51c6031db47 --- /dev/null +++ b/samples/bpf/test_veth_afxdp.sh @@ -0,0 +1,82 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +# The script runs the sender at the root namespace, and +# the receiver at the namespace at_ns0, with different mode +# 1. XDP_DROP +# 2. XDP_TX +# 3. XDP_PASS +# 4. XDP_REDIRECT +# 5. Generic XDP + +XDPSOCK=./xdpsock +XDP_RXQ_INFO=./xdp_rxq_info + +ip netns add at_ns0 +ip link add p0 type veth peer name p1 +ip link set p0 netns at_ns0 +ip link set dev p1 up +ip netns exec at_ns0 ip link set dev p0 up + +test_xdp_drop() +{ + echo "[Peer] XDP_DROP" + ip netns exec at_ns0 $XDP_RXQ_INFO --dev p0 --action XDP_DROP & +} + +test_xdp_pass() +{ + echo "[Peer] XDP_PASS" + ip netns exec at_ns0 $XDP_RXQ_INFO --dev p0 --action XDP_PASS & +} + +test_xdp_tx() +{ + echo "[Peer] XDP_TX" + ip netns exec at_ns0 $XDP_RXQ_INFO --dev p0 --action XDP_TX & +} + +test_generic_xdp() +{ + echo "[Peer] Generic XDP" + ip netns exec at_ns0 $XDPSOCK -i p0 -r -S & +} + +test_xdp_redirect() +{ + echo "[Peer] XDP_REDIRECT" + ip netns exec at_ns0 $XDPSOCK -i p0 -r -N & +} + +test_xdp_zcrx() +{ + echo "[Peer] AF_XDP RX" + ip netns exec at_ns0 $XDPSOCK -i p0 -r -N -z & +} + +cleanup() { + killall xdpsock + sleep 1 + killall xdp_rxq_info + ip netns del at_ns0 + ip link del p1 +} + +trap cleanup 0 3 6 + +if [ "$1" == "drop" ]; then + test_xdp_drop +elif [ "$1" == "pass" ]; then + test_xdp_pass +elif [ "$1" == "tx" ]; then + test_xdp_tx +elif [ "$1" == "redirect" ]; then + test_xdp_redirect +elif [ "$1" == "zcrx" ]; then + test_xdp_zcrx +else + test_xdp_drop +fi + +# send at root namespace +$XDPSOCK -i p1 -t -N -z