From patchwork Thu Aug 15 03:46:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Samudrala, Sridhar" X-Patchwork-Id: 1147392 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=osuosl.org (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=intel.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 468C6z5M5Hz9sNx for ; Thu, 15 Aug 2019 13:46:35 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 0B3C786964; Thu, 15 Aug 2019 03:46:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kKMDwjjVmpdz; Thu, 15 Aug 2019 03:46:31 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id DC5C286940; Thu, 15 Aug 2019 03:46:29 +0000 (UTC) X-Original-To: intel-wired-lan@lists.osuosl.org Delivered-To: intel-wired-lan@lists.osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 2E0051BF3BD for ; Thu, 15 Aug 2019 03:46:27 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 2A27C203F5 for ; Thu, 15 Aug 2019 03:46:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nWsavshwBQbQ for ; Thu, 15 Aug 2019 03:46:24 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by silver.osuosl.org (Postfix) with ESMTPS id 9083920459 for ; Thu, 15 Aug 2019 03:46:24 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 14 Aug 2019 20:46:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,387,1559545200"; d="scan'208";a="352124051" Received: from arch-p28.jf.intel.com ([10.166.187.31]) by orsmga005.jf.intel.com with ESMTP; 14 Aug 2019 20:46:23 -0700 From: Sridhar Samudrala To: magnus.karlsson@intel.com, bjorn.topel@intel.com, netdev@vger.kernel.org, bpf@vger.kernel.org, sridhar.samudrala@intel.com, intel-wired-lan@lists.osuosl.org, maciej.fijalkowski@intel.com, tom.herbert@intel.com Date: Wed, 14 Aug 2019 20:46:19 -0700 Message-Id: <1565840783-8269-2-git-send-email-sridhar.samudrala@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1565840783-8269-1-git-send-email-sridhar.samudrala@intel.com> References: <1565840783-8269-1-git-send-email-sridhar.samudrala@intel.com> Subject: [Intel-wired-lan] [PATCH bpf-next 1/5] xsk: Convert bool 'zc' field in struct xdp_umem to a u32 bitmap X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" The bool 'zc' field in struct xdp_uem is replaced with a u32 flags field and a bit within flags is used to indicate zerocopy. This flags field will be used in later patches for other bit fields. Also, removed the bool 'zc' field from struct xdp_sock as it can be accessed via flags in xs->umem. Signed-off-by: Sridhar Samudrala --- include/net/xdp_sock.h | 12 ++++++++++-- net/xdp/xdp_umem.c | 6 +++--- net/xdp/xsk.c | 12 +++++++++--- net/xdp/xsk_diag.c | 3 ++- 4 files changed, 24 insertions(+), 9 deletions(-) diff --git a/include/net/xdp_sock.h b/include/net/xdp_sock.h index 69796d264f06..b6716dbdce1a 100644 --- a/include/net/xdp_sock.h +++ b/include/net/xdp_sock.h @@ -27,6 +27,9 @@ struct xdp_umem_fq_reuse { u64 handles[]; }; +/* Bits for the umem flags field. */ +#define XDP_UMEM_F_ZEROCOPY (1 << 0) + struct xdp_umem { struct xsk_queue *fq; struct xsk_queue *cq; @@ -45,7 +48,7 @@ struct xdp_umem { struct net_device *dev; struct xdp_umem_fq_reuse *fq_reuse; u16 queue_id; - bool zc; + u32 flags; spinlock_t xsk_list_lock; struct list_head xsk_list; }; @@ -58,7 +61,6 @@ struct xdp_sock { struct xdp_umem *umem; struct list_head flush_node; u16 queue_id; - bool zc; enum { XSK_READY = 0, XSK_BOUND, @@ -95,6 +97,7 @@ struct xdp_umem_fq_reuse *xsk_reuseq_swap(struct xdp_umem *umem, struct xdp_umem_fq_reuse *newq); void xsk_reuseq_free(struct xdp_umem_fq_reuse *rq); struct xdp_umem *xdp_get_umem_from_qid(struct net_device *dev, u16 queue_id); +bool xsk_umem_zerocopy(struct xdp_umem *umem); static inline char *xdp_umem_get_data(struct xdp_umem *umem, u64 addr) { @@ -213,6 +216,11 @@ static inline struct xdp_umem *xdp_get_umem_from_qid(struct net_device *dev, return NULL; } +static inline bool xsk_umem_zerocopy(struct xdp_umem *umem) +{ + return false; +} + static inline char *xdp_umem_get_data(struct xdp_umem *umem, u64 addr) { return NULL; diff --git a/net/xdp/xdp_umem.c b/net/xdp/xdp_umem.c index a0607969f8c0..411b3e3498c4 100644 --- a/net/xdp/xdp_umem.c +++ b/net/xdp/xdp_umem.c @@ -126,7 +126,7 @@ int xdp_umem_assign_dev(struct xdp_umem *umem, struct net_device *dev, if (err) goto err_unreg_umem; - umem->zc = true; + umem->flags |= XDP_UMEM_F_ZEROCOPY; return 0; err_unreg_umem: @@ -147,7 +147,7 @@ void xdp_umem_clear_dev(struct xdp_umem *umem) if (!umem->dev) return; - if (umem->zc) { + if (xsk_umem_zerocopy(umem)) { bpf.command = XDP_SETUP_XSK_UMEM; bpf.xsk.umem = NULL; bpf.xsk.queue_id = umem->queue_id; @@ -162,7 +162,7 @@ void xdp_umem_clear_dev(struct xdp_umem *umem) dev_put(umem->dev); umem->dev = NULL; - umem->zc = false; + umem->flags &= ~XDP_UMEM_F_ZEROCOPY; } static void xdp_umem_unmap_pages(struct xdp_umem *umem) diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index 59b57d708697..ca95676ef75d 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -295,6 +295,12 @@ static int xsk_generic_xmit(struct sock *sk, struct msghdr *m, return err; } +bool xsk_umem_zerocopy(struct xdp_umem *umem) +{ + return (umem && (umem->flags & XDP_UMEM_F_ZEROCOPY)); +} +EXPORT_SYMBOL(xsk_umem_zerocopy); + static int xsk_sendmsg(struct socket *sock, struct msghdr *m, size_t total_len) { bool need_wait = !(m->msg_flags & MSG_DONTWAIT); @@ -310,7 +316,8 @@ static int xsk_sendmsg(struct socket *sock, struct msghdr *m, size_t total_len) if (need_wait) return -EOPNOTSUPP; - return (xs->zc) ? xsk_zc_xmit(sk) : xsk_generic_xmit(sk, m, total_len); + return xsk_umem_zerocopy(xs->umem) ? xsk_zc_xmit(sk) : + xsk_generic_xmit(sk, m, total_len); } static unsigned int xsk_poll(struct file *file, struct socket *sock, @@ -503,7 +510,6 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len) } xs->dev = dev; - xs->zc = xs->umem->zc; xs->queue_id = qid; xskq_set_umem(xs->rx, xs->umem->size, xs->umem->chunk_mask); xskq_set_umem(xs->tx, xs->umem->size, xs->umem->chunk_mask); @@ -683,7 +689,7 @@ static int xsk_getsockopt(struct socket *sock, int level, int optname, return -EINVAL; mutex_lock(&xs->mutex); - if (xs->zc) + if (xsk_umem_zerocopy(xs->umem)) opts.flags |= XDP_OPTIONS_ZEROCOPY; mutex_unlock(&xs->mutex); diff --git a/net/xdp/xsk_diag.c b/net/xdp/xsk_diag.c index d5e06c8e0cbf..8a19b7e87cfb 100644 --- a/net/xdp/xsk_diag.c +++ b/net/xdp/xsk_diag.c @@ -10,6 +10,7 @@ #include #include #include +#include #include "xsk_queue.h" #include "xsk.h" @@ -61,7 +62,7 @@ static int xsk_diag_put_umem(const struct xdp_sock *xs, struct sk_buff *nlskb) du.ifindex = umem->dev ? umem->dev->ifindex : 0; du.queue_id = umem->queue_id; du.flags = 0; - if (umem->zc) + if (xsk_umem_zerocopy(umem)) du.flags |= XDP_DU_F_ZEROCOPY; du.refs = refcount_read(&umem->users); From patchwork Thu Aug 15 03:46:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Samudrala, Sridhar" X-Patchwork-Id: 1147393 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=osuosl.org (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=intel.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 468C702Zbcz9sN1 for ; Thu, 15 Aug 2019 13:46:35 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 72ACB204CB; Thu, 15 Aug 2019 03:46:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id te4D8Twlx1aB; Thu, 15 Aug 2019 03:46:30 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 340D720460; Thu, 15 Aug 2019 03:46:30 +0000 (UTC) X-Original-To: intel-wired-lan@lists.osuosl.org Delivered-To: intel-wired-lan@lists.osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id B37231BF3BD for ; Thu, 15 Aug 2019 03:46:27 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id B0EF6203F5 for ; Thu, 15 Aug 2019 03:46:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HXkTvBbsjYQl for ; Thu, 15 Aug 2019 03:46:24 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by silver.osuosl.org (Postfix) with ESMTPS id AFDEC20460 for ; Thu, 15 Aug 2019 03:46:24 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 14 Aug 2019 20:46:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,387,1559545200"; d="scan'208";a="352124054" Received: from arch-p28.jf.intel.com ([10.166.187.31]) by orsmga005.jf.intel.com with ESMTP; 14 Aug 2019 20:46:23 -0700 From: Sridhar Samudrala To: magnus.karlsson@intel.com, bjorn.topel@intel.com, netdev@vger.kernel.org, bpf@vger.kernel.org, sridhar.samudrala@intel.com, intel-wired-lan@lists.osuosl.org, maciej.fijalkowski@intel.com, tom.herbert@intel.com Date: Wed, 14 Aug 2019 20:46:20 -0700 Message-Id: <1565840783-8269-3-git-send-email-sridhar.samudrala@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1565840783-8269-1-git-send-email-sridhar.samudrala@intel.com> References: <1565840783-8269-1-git-send-email-sridhar.samudrala@intel.com> Subject: [Intel-wired-lan] [PATCH bpf-next 2/5] xsk: Introduce XDP_SKIP_BPF bind option X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" This option enables an AF_XDP socket to specify XDP_SKIP_BPF flag with the bind() call to skip calling the BPF program in the receive path and pass the XDP buffer directly to the socket. When a single AF_XDP socket is associated with a queue and a HW filter is used to redirect the packets and the app is interested in receiving all the packets on that queue, we don't need an additional BPF program to do further filtering or lookup/redirect to a socket. Signed-off-by: Sridhar Samudrala --- include/net/xdp_sock.h | 9 +++++++++ include/uapi/linux/if_xdp.h | 1 + include/uapi/linux/xdp_diag.h | 1 + net/xdp/xdp_umem.c | 5 ++++- net/xdp/xsk.c | 31 +++++++++++++++++++++++++++++-- net/xdp/xsk_diag.c | 2 ++ 6 files changed, 46 insertions(+), 3 deletions(-) diff --git a/include/net/xdp_sock.h b/include/net/xdp_sock.h index b6716dbdce1a..ad132a69db7c 100644 --- a/include/net/xdp_sock.h +++ b/include/net/xdp_sock.h @@ -29,6 +29,7 @@ struct xdp_umem_fq_reuse { /* Bits for the umem flags field. */ #define XDP_UMEM_F_ZEROCOPY (1 << 0) +#define XDP_UMEM_F_SKIP_BPF (1 << 1) struct xdp_umem { struct xsk_queue *fq; @@ -98,6 +99,9 @@ struct xdp_umem_fq_reuse *xsk_reuseq_swap(struct xdp_umem *umem, void xsk_reuseq_free(struct xdp_umem_fq_reuse *rq); struct xdp_umem *xdp_get_umem_from_qid(struct net_device *dev, u16 queue_id); bool xsk_umem_zerocopy(struct xdp_umem *umem); +bool xsk_umem_skip_bpf(struct xdp_umem *umem); +void xsk_umem_flush(struct xdp_umem *umem); +int xsk_umem_rcv(struct xdp_umem *umem, struct xdp_buff *xdp); static inline char *xdp_umem_get_data(struct xdp_umem *umem, u64 addr) { @@ -221,6 +225,11 @@ static inline bool xsk_umem_zerocopy(struct xdp_umem *umem) return false; } +static inline bool xsk_umem_skip_bpf(struct xdp_umem *umem) +{ + return false; +} + static inline char *xdp_umem_get_data(struct xdp_umem *umem, u64 addr) { return NULL; diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h index faaa5ca2a117..881447ebf3c9 100644 --- a/include/uapi/linux/if_xdp.h +++ b/include/uapi/linux/if_xdp.h @@ -16,6 +16,7 @@ #define XDP_SHARED_UMEM (1 << 0) #define XDP_COPY (1 << 1) /* Force copy-mode */ #define XDP_ZEROCOPY (1 << 2) /* Force zero-copy mode */ +#define XDP_SKIP_BPF (1 << 3) /* Skip running BPF program */ struct sockaddr_xdp { __u16 sxdp_family; diff --git a/include/uapi/linux/xdp_diag.h b/include/uapi/linux/xdp_diag.h index 78b2591a7782..6caf3d9c9abe 100644 --- a/include/uapi/linux/xdp_diag.h +++ b/include/uapi/linux/xdp_diag.h @@ -56,6 +56,7 @@ struct xdp_diag_ring { }; #define XDP_DU_F_ZEROCOPY (1 << 0) +#define XDP_DU_F_SKIP_BPF (2 << 0) struct xdp_diag_umem { __u64 size; diff --git a/net/xdp/xdp_umem.c b/net/xdp/xdp_umem.c index 411b3e3498c4..cbc02509dc90 100644 --- a/net/xdp/xdp_umem.c +++ b/net/xdp/xdp_umem.c @@ -106,6 +106,9 @@ int xdp_umem_assign_dev(struct xdp_umem *umem, struct net_device *dev, umem->dev = dev; umem->queue_id = queue_id; + if (flags & XDP_SKIP_BPF) + umem->flags |= XDP_UMEM_F_SKIP_BPF; + dev_hold(dev); if (force_copy) @@ -162,7 +165,7 @@ void xdp_umem_clear_dev(struct xdp_umem *umem) dev_put(umem->dev); umem->dev = NULL; - umem->flags &= ~XDP_UMEM_F_ZEROCOPY; + umem->flags &= ~(XDP_UMEM_F_ZEROCOPY | XDP_UMEM_F_SKIP_BPF); } static void xdp_umem_unmap_pages(struct xdp_umem *umem) diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index ca95676ef75d..bcb6a77fae22 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -166,6 +166,27 @@ int xsk_generic_rcv(struct xdp_sock *xs, struct xdp_buff *xdp) return err; } +void xsk_umem_flush(struct xdp_umem *umem) +{ + struct xdp_sock *xs; + + if (!list_empty(&umem->xsk_list)) { + xs = list_first_entry(&umem->xsk_list, struct xdp_sock, list); + xsk_flush(xs); + } +} +EXPORT_SYMBOL(xsk_umem_flush); + +int xsk_umem_rcv(struct xdp_umem *umem, struct xdp_buff *xdp) +{ + struct xdp_sock *xs; + + xs = list_first_entry(&umem->xsk_list, struct xdp_sock, list); + xdp->handle += xdp->data - xdp->data_hard_start; + return xsk_rcv(xs, xdp); +} +EXPORT_SYMBOL(xsk_umem_rcv); + void xsk_umem_complete_tx(struct xdp_umem *umem, u32 nb_entries) { xskq_produce_flush_addr_n(umem->cq, nb_entries); @@ -301,6 +322,12 @@ bool xsk_umem_zerocopy(struct xdp_umem *umem) } EXPORT_SYMBOL(xsk_umem_zerocopy); +bool xsk_umem_skip_bpf(struct xdp_umem *umem) +{ + return (umem && (umem->flags & XDP_UMEM_F_SKIP_BPF)); +} +EXPORT_SYMBOL(xsk_umem_skip_bpf); + static int xsk_sendmsg(struct socket *sock, struct msghdr *m, size_t total_len) { bool need_wait = !(m->msg_flags & MSG_DONTWAIT); @@ -434,7 +461,7 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len) return -EINVAL; flags = sxdp->sxdp_flags; - if (flags & ~(XDP_SHARED_UMEM | XDP_COPY | XDP_ZEROCOPY)) + if (flags & ~(XDP_SHARED_UMEM | XDP_COPY | XDP_ZEROCOPY | XDP_SKIP_BPF)) return -EINVAL; rtnl_lock(); @@ -461,7 +488,7 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len) struct xdp_sock *umem_xs; struct socket *sock; - if ((flags & XDP_COPY) || (flags & XDP_ZEROCOPY)) { + if (flags & (XDP_COPY | XDP_ZEROCOPY | XDP_SKIP_BPF)) { /* Cannot specify flags for shared sockets. */ err = -EINVAL; goto out_unlock; diff --git a/net/xdp/xsk_diag.c b/net/xdp/xsk_diag.c index 8a19b7e87cfb..f6f4b7912a22 100644 --- a/net/xdp/xsk_diag.c +++ b/net/xdp/xsk_diag.c @@ -64,6 +64,8 @@ static int xsk_diag_put_umem(const struct xdp_sock *xs, struct sk_buff *nlskb) du.flags = 0; if (xsk_umem_zerocopy(umem)) du.flags |= XDP_DU_F_ZEROCOPY; + if (xsk_umem_skip_bpf(umem)) + du.flags |= XDP_DU_F_SKIP_BPF; du.refs = refcount_read(&umem->users); err = nla_put(nlskb, XDP_DIAG_UMEM, sizeof(du), &du); From patchwork Thu Aug 15 03:46:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Samudrala, Sridhar" X-Patchwork-Id: 1147390 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=osuosl.org (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=intel.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 468C6y3zRKz9sN6 for ; Thu, 15 Aug 2019 13:46:33 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id C98A08695E; Thu, 15 Aug 2019 03:46:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RX7bE9Zhozgs; Thu, 15 Aug 2019 03:46:30 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 89C8486936; Thu, 15 Aug 2019 03:46:28 +0000 (UTC) X-Original-To: intel-wired-lan@lists.osuosl.org Delivered-To: intel-wired-lan@lists.osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 4654C1BF9B9 for ; Thu, 15 Aug 2019 03:46:26 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 44064203F5 for ; Thu, 15 Aug 2019 03:46:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OvkTgVJI8CcW for ; Thu, 15 Aug 2019 03:46:24 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by silver.osuosl.org (Postfix) with ESMTPS id C4A0D2046B for ; Thu, 15 Aug 2019 03:46:24 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 14 Aug 2019 20:46:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,387,1559545200"; d="scan'208";a="352124058" Received: from arch-p28.jf.intel.com ([10.166.187.31]) by orsmga005.jf.intel.com with ESMTP; 14 Aug 2019 20:46:23 -0700 From: Sridhar Samudrala To: magnus.karlsson@intel.com, bjorn.topel@intel.com, netdev@vger.kernel.org, bpf@vger.kernel.org, sridhar.samudrala@intel.com, intel-wired-lan@lists.osuosl.org, maciej.fijalkowski@intel.com, tom.herbert@intel.com Date: Wed, 14 Aug 2019 20:46:21 -0700 Message-Id: <1565840783-8269-4-git-send-email-sridhar.samudrala@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1565840783-8269-1-git-send-email-sridhar.samudrala@intel.com> References: <1565840783-8269-1-git-send-email-sridhar.samudrala@intel.com> Subject: [Intel-wired-lan] [PATCH bpf-next 3/5] i40e: Enable XDP_SKIP_BPF option for AF_XDP sockets X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" This patch skips calling BPF program in the receive path if the queue is associated with UMEM that is not shared and bound to an AF_XDP socket that has enabled skip bpf during bind() call. Here are some performance numbers collected on - 2 socket 28 core Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz - Intel 40Gb Ethernet NIC (i40e) All tests use 2 cores and the results are in Mpps. turbo on (default) --------------------------------------------- no-skip-bpf skip-bpf --------------------------------------------- rxdrop zerocopy 21.9 38.5 l2fwd zerocopy 17.0 20.5 rxdrop copy 11.1 13.3 l2fwd copy 1.9 2.0 no turbo : echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo --------------------------------------------- no-skip-bpf skip-bpf --------------------------------------------- rxdrop zerocopy 15.4 29.0 l2fwd zerocopy 11.8 18.2 rxdrop copy 8.2 10.5 l2fwd copy 1.7 1.7 --------------------------------------------- Signed-off-by: Sridhar Samudrala --- drivers/net/ethernet/intel/i40e/i40e_txrx.c | 22 +++++++++++++++++++-- drivers/net/ethernet/intel/i40e/i40e_xsk.c | 6 ++++++ 2 files changed, 26 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c index e3f29dc8b290..5e63e3644e87 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c @@ -2199,6 +2199,7 @@ static struct sk_buff *i40e_run_xdp(struct i40e_ring *rx_ring, int err, result = I40E_XDP_PASS; struct i40e_ring *xdp_ring; struct bpf_prog *xdp_prog; + struct xdp_umem *umem; u32 act; rcu_read_lock(); @@ -2209,6 +2210,13 @@ static struct sk_buff *i40e_run_xdp(struct i40e_ring *rx_ring, prefetchw(xdp->data_hard_start); /* xdp_frame write */ + umem = xdp_get_umem_from_qid(rx_ring->netdev, rx_ring->queue_index); + if (xsk_umem_skip_bpf(umem)) { + err = xsk_umem_rcv(umem, xdp); + result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED; + goto xdp_out; + } + act = bpf_prog_run_xdp(xdp_prog, xdp); switch (act) { case XDP_PASS: @@ -2303,8 +2311,18 @@ void i40e_update_rx_stats(struct i40e_ring *rx_ring, **/ void i40e_finalize_xdp_rx(struct i40e_ring *rx_ring, unsigned int xdp_res) { - if (xdp_res & I40E_XDP_REDIR) - xdp_do_flush_map(); + if (xdp_res & I40E_XDP_REDIR) { + struct xdp_umem *umem; + + umem = rx_ring->xsk_umem; + if (!umem) + umem = xdp_get_umem_from_qid(rx_ring->netdev, + rx_ring->queue_index); + if (xsk_umem_skip_bpf(umem)) + xsk_umem_flush(umem); + else + xdp_do_flush_map(); + } if (xdp_res & I40E_XDP_TX) { struct i40e_ring *xdp_ring = diff --git a/drivers/net/ethernet/intel/i40e/i40e_xsk.c b/drivers/net/ethernet/intel/i40e/i40e_xsk.c index 32bad014d76c..cc538479c95d 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_xsk.c +++ b/drivers/net/ethernet/intel/i40e/i40e_xsk.c @@ -195,6 +195,12 @@ static int i40e_run_xdp_zc(struct i40e_ring *rx_ring, struct xdp_buff *xdp) struct bpf_prog *xdp_prog; u32 act; + if (xsk_umem_skip_bpf(rx_ring->xsk_umem)) { + err = xsk_umem_rcv(rx_ring->xsk_umem, xdp); + result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED; + return result; + } + rcu_read_lock(); /* NB! xdp_prog will always be !NULL, due to the fact that * this path is enabled by setting an XDP program. From patchwork Thu Aug 15 03:46:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Samudrala, Sridhar" X-Patchwork-Id: 1147391 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=osuosl.org (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=intel.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 468C6y6tjhz9sNm for ; Thu, 15 Aug 2019 13:46:34 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 4751186938; Thu, 15 Aug 2019 03:46:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OLEQl-OXUN9R; Thu, 15 Aug 2019 03:46:31 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 9242586955; Thu, 15 Aug 2019 03:46:30 +0000 (UTC) X-Original-To: intel-wired-lan@lists.osuosl.org Delivered-To: intel-wired-lan@lists.osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 108261BF3BD for ; Thu, 15 Aug 2019 03:46:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id BD65287729 for ; Thu, 15 Aug 2019 03:46:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IJ2w9DFFHxLw for ; Thu, 15 Aug 2019 03:46:25 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by whitealder.osuosl.org (Postfix) with ESMTPS id D25A387726 for ; Thu, 15 Aug 2019 03:46:24 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 14 Aug 2019 20:46:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,387,1559545200"; d="scan'208";a="352124060" Received: from arch-p28.jf.intel.com ([10.166.187.31]) by orsmga005.jf.intel.com with ESMTP; 14 Aug 2019 20:46:23 -0700 From: Sridhar Samudrala To: magnus.karlsson@intel.com, bjorn.topel@intel.com, netdev@vger.kernel.org, bpf@vger.kernel.org, sridhar.samudrala@intel.com, intel-wired-lan@lists.osuosl.org, maciej.fijalkowski@intel.com, tom.herbert@intel.com Date: Wed, 14 Aug 2019 20:46:22 -0700 Message-Id: <1565840783-8269-5-git-send-email-sridhar.samudrala@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1565840783-8269-1-git-send-email-sridhar.samudrala@intel.com> References: <1565840783-8269-1-git-send-email-sridhar.samudrala@intel.com> Subject: [Intel-wired-lan] [PATCH bpf-next 4/5] ixgbe: Enable XDP_SKIP_BPF option for AF_XDP sockets X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" This patch skips calling BPF program in the receive path if the queue is associated with UMEM that is not shared and bound to an AF_XDP socket that has enabled skip bpf during bind() call. Signed-off-by: Sridhar Samudrala --- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 20 +++++++++++++++++-- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 16 +++++++++++++-- 2 files changed, 32 insertions(+), 4 deletions(-) diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c index dc7b128c780e..594792860cdd 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c @@ -2197,6 +2197,7 @@ static struct sk_buff *ixgbe_run_xdp(struct ixgbe_adapter *adapter, int err, result = IXGBE_XDP_PASS; struct bpf_prog *xdp_prog; struct xdp_frame *xdpf; + struct xdp_umem *umem; u32 act; rcu_read_lock(); @@ -2207,6 +2208,13 @@ static struct sk_buff *ixgbe_run_xdp(struct ixgbe_adapter *adapter, prefetchw(xdp->data_hard_start); /* xdp_frame write */ + umem = xdp_get_umem_from_qid(rx_ring->netdev, rx_ring->queue_index); + if (xsk_umem_skip_bpf(umem)) { + err = xsk_umem_rcv(umem, xdp); + result = !err ? IXGBE_XDP_REDIR : IXGBE_XDP_CONSUMED; + goto xdp_out; + } + act = bpf_prog_run_xdp(xdp_prog, xdp); switch (act) { case XDP_PASS: @@ -2400,8 +2408,16 @@ static int ixgbe_clean_rx_irq(struct ixgbe_q_vector *q_vector, total_rx_packets++; } - if (xdp_xmit & IXGBE_XDP_REDIR) - xdp_do_flush_map(); + if (xdp_xmit & IXGBE_XDP_REDIR) { + struct xdp_umem *umem; + + umem = xdp_get_umem_from_qid(rx_ring->netdev, + rx_ring->queue_index); + if (xsk_umem_skip_bpf(umem)) + xsk_umem_flush(umem); + else + xdp_do_flush_map(); + } if (xdp_xmit & IXGBE_XDP_TX) { struct ixgbe_ring *ring = adapter->xdp_ring[smp_processor_id()]; diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c index 6b609553329f..9ea8a769d7a8 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c @@ -148,6 +148,12 @@ static int ixgbe_run_xdp_zc(struct ixgbe_adapter *adapter, struct xdp_frame *xdpf; u32 act; + if (xsk_umem_skip_bpf(rx_ring->xsk_umem)) { + err = xsk_umem_rcv(rx_ring->xsk_umem, xdp); + result = !err ? IXGBE_XDP_REDIR : IXGBE_XDP_CONSUMED; + return result; + } + rcu_read_lock(); xdp_prog = READ_ONCE(rx_ring->xdp_prog); act = bpf_prog_run_xdp(xdp_prog, xdp); @@ -527,8 +533,14 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector, ixgbe_rx_skb(q_vector, skb); } - if (xdp_xmit & IXGBE_XDP_REDIR) - xdp_do_flush_map(); + if (xdp_xmit & IXGBE_XDP_REDIR) { + struct xdp_umem *umem = rx_ring->xsk_umem; + + if (xsk_umem_skip_bpf(umem)) + xsk_umem_flush(umem); + else + xdp_do_flush_map(); + } if (xdp_xmit & IXGBE_XDP_TX) { struct ixgbe_ring *ring = adapter->xdp_ring[smp_processor_id()]; From patchwork Thu Aug 15 03:46:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Samudrala, Sridhar" X-Patchwork-Id: 1147388 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=osuosl.org (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=intel.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 468C6y4lYVz9sNf for ; Thu, 15 Aug 2019 13:46:32 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 692B186946; Thu, 15 Aug 2019 03:46:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9Xdv8CqLQtet; Thu, 15 Aug 2019 03:46:29 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 3BA328692E; Thu, 15 Aug 2019 03:46:28 +0000 (UTC) X-Original-To: intel-wired-lan@lists.osuosl.org Delivered-To: intel-wired-lan@lists.osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 2E9731BF3BD for ; Thu, 15 Aug 2019 03:46:26 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 29619203F5 for ; Thu, 15 Aug 2019 03:46:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id B8itxSy0cK7M for ; Thu, 15 Aug 2019 03:46:25 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by silver.osuosl.org (Postfix) with ESMTPS id 14AAE20483 for ; Thu, 15 Aug 2019 03:46:24 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 14 Aug 2019 20:46:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,387,1559545200"; d="scan'208";a="352124063" Received: from arch-p28.jf.intel.com ([10.166.187.31]) by orsmga005.jf.intel.com with ESMTP; 14 Aug 2019 20:46:23 -0700 From: Sridhar Samudrala To: magnus.karlsson@intel.com, bjorn.topel@intel.com, netdev@vger.kernel.org, bpf@vger.kernel.org, sridhar.samudrala@intel.com, intel-wired-lan@lists.osuosl.org, maciej.fijalkowski@intel.com, tom.herbert@intel.com Date: Wed, 14 Aug 2019 20:46:23 -0700 Message-Id: <1565840783-8269-6-git-send-email-sridhar.samudrala@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1565840783-8269-1-git-send-email-sridhar.samudrala@intel.com> References: <1565840783-8269-1-git-send-email-sridhar.samudrala@intel.com> Subject: [Intel-wired-lan] [PATCH bpf-next 5/5] xdpsock_user: Add skip_bpf option X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" Signed-off-by: Sridhar Samudrala --- samples/bpf/xdpsock_user.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/samples/bpf/xdpsock_user.c b/samples/bpf/xdpsock_user.c index 93eaaf7239b2..509fc6a18af9 100644 --- a/samples/bpf/xdpsock_user.c +++ b/samples/bpf/xdpsock_user.c @@ -123,6 +123,9 @@ static void print_benchmark(bool running) if (opt_poll) printf("poll() "); + if (opt_xdp_bind_flags & XDP_SKIP_BPF) + printf("skip-bpf "); + if (running) { printf("running..."); fflush(stdout); @@ -352,6 +355,7 @@ static struct option long_options[] = { {"zero-copy", no_argument, 0, 'z'}, {"copy", no_argument, 0, 'c'}, {"frame-size", required_argument, 0, 'f'}, + {"skip-bpf", no_argument, 0, 's'}, {0, 0, 0, 0} }; @@ -372,6 +376,7 @@ static void usage(const char *prog) " -z, --zero-copy Force zero-copy mode.\n" " -c, --copy Force copy mode.\n" " -f, --frame-size=n Set the frame size (must be a power of two, default is %d).\n" + " -s, --skip-bpf Skip running bpf program.\n" "\n"; fprintf(stderr, str, prog, XSK_UMEM__DEFAULT_FRAME_SIZE); exit(EXIT_FAILURE); @@ -430,6 +435,9 @@ static void parse_command_line(int argc, char **argv) case 'f': opt_xsk_frame_size = atoi(optarg); break; + case 's': + opt_xdp_bind_flags |= XDP_SKIP_BPF; + break; default: usage(basename(argv[0])); }