From patchwork Fri Jul 12 08:55:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kurt Kanzenbach X-Patchwork-Id: 1959721 X-Patchwork-Delegate: anthony.l.nguyen@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=osuosl.org header.i=@osuosl.org header.a=rsa-sha256 header.s=default header.b=3Ca+L76u; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=osuosl.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WL56j4sNZz1xqj for ; Fri, 12 Jul 2024 18:56:13 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 2C8BF4144F; Fri, 12 Jul 2024 08:56:09 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id J5DSLQKmWy5c; Fri, 12 Jul 2024 08:56:07 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.166.34; helo=ash.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 96582412DB DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osuosl.org; s=default; t=1720774567; bh=+Q70TNir7IBy52CbPauTJ3qNqdXl8vBAfbwUA/7ETBE=; h=From:Date:References:In-Reply-To:To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: Cc:From; b=3Ca+L76uFyWGd8HpUxc1JOHuJFsA9Z9Qdyp97Vrry+yYWVLSvyPZESm4+fEk40xf7 0Y3XcQfEovDpJjMZ73tneVttHNC98JAAc/AIfiXchR4sCJyzkTibUl2QJ+cFIejYY2 VOZOoxFo6XLn9A0AD2KRrCQxnEdNwO1X91GeTxSa6Bew0hNzB7d9BeFp6L2lyRb2Fi e1/x5WDtuyGMYzFaEZtT62ol4VXvRE6M98YWdMkwIq2j5tdjIL+LOxk+qlkgt2hmM4 BPWWmgH7B5Bh9CFOymgQONeRPMW5Rxr2QX0/YJFW1pvU0DD/6iuCXIBaI2Yq3TjSyl o98GDMKljpqWw== Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp2.osuosl.org (Postfix) with ESMTP id 96582412DB; Fri, 12 Jul 2024 08:56:07 +0000 (UTC) X-Original-To: intel-wired-lan@lists.osuosl.org Delivered-To: intel-wired-lan@lists.osuosl.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 4BDBF1BF584 for ; Fri, 12 Jul 2024 08:56:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 441EE60AE3 for ; Fri, 12 Jul 2024 08:55:59 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 2t0TAfp9M6-S for ; Fri, 12 Jul 2024 08:55:58 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=193.142.43.55; helo=galois.linutronix.de; envelope-from=kurt@linutronix.de; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org 05148608C3 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 05148608C3 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by smtp3.osuosl.org (Postfix) with ESMTPS id 05148608C3 for ; Fri, 12 Jul 2024 08:55:57 +0000 (UTC) From: Kurt Kanzenbach Date: Fri, 12 Jul 2024 10:55:29 +0200 MIME-Version: 1.0 Message-Id: <20240711-b4-igb_zero_copy-v5-1-f3f455113b11@linutronix.de> References: <20240711-b4-igb_zero_copy-v5-0-f3f455113b11@linutronix.de> In-Reply-To: <20240711-b4-igb_zero_copy-v5-0-f3f455113b11@linutronix.de> To: Tony Nguyen , Przemek Kitszel X-Developer-Signature: v=1; a=openpgp-sha256; l=6788; i=kurt@linutronix.de; h=from:subject:message-id; bh=karr5T8HuyUwA088auP3gGhvt/no1iXDgJPYDayi7vg=; b=owEBbQKS/ZANAwAKAcGT0fKqRnOCAcsmYgBmkO+WOBYC3tcGL39gclFbXDK6Zi4EbWSdTfPU/ xRsxl8QEb6JAjMEAAEKAB0WIQS8ub+yyMN909/bWZLBk9HyqkZzggUCZpDvlgAKCRDBk9HyqkZz gnDXEACNzkOn+7o+SynoIJo0o4KOg8X3oFaBWoR4wnZEHC3Cn2wfz2h3ZqY9lGBXaDI6Gu2F1Xq M2ATwtv9yXGq/sCu6rEUwceLIENnYwfQLrpBkXMx5+QtkMkoF5iF7/50hP3x/3CmB6lKAhlYh7o 2oYXd7lQu6LLYQmtq+Y8CLGxGYa0RG00UrmWlt/twLAp1EMN0tPUfItCRlXSTi7ZX8+5Iqnks4+ J1h+xBA0ezCVWRjNeDh4DvEFDibhcFcJWS+xGBBhxUUbCv7f9n+TmE3mlOOwY3NbYrMz3Mcr9DZ Ci2mgQQbQ351IccXPMpPOm+Bgrw5AznueCbR7sAIIyaOlXhaafvI7CrKmqa64ZYB+5W9B0i/NhC AdPvPjPXnEdPhioMphW3fBcfT330CYgGe9drw3GNsKIT/ZrN86sunYAI24TQtiTXUGh9nWgZrmU e/SIly8i+THhmScgb5mkmwjULqPdB7oLvVkcF6Gf0mUqLZfraaTuyofHWbtRBSGa8LOkontffmt IR0UsytjJiBeYe/XatpM1frb4WblIcHdhrPFj3i09JnCXAMmJeF9Wkw+lc1M09cxnYmUdsW0Kv1 XHU2aitnrYmbJB4L3eEm6DFcr4d+1ArzdKPyJ0KvIP3jzdOFpsAVaNlxP/MrUDi7G6CoQ25gVSx mkD2m6N16oyWBFw== X-Developer-Key: i=kurt@linutronix.de; a=openpgp; fpr=BCB9BFB2C8C37DD3DFDB5992C193D1F2AA467382 X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1720774553; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+Q70TNir7IBy52CbPauTJ3qNqdXl8vBAfbwUA/7ETBE=; b=OUKpw4dY7+c8OeaHbG3nWN6OYBKwGO8cTUZI71w300cPkMdCG78VVT8RAkg/SpG2m2Gnsy HOBel1qqCaYgrj68jaWZv5YO/6BSXxnde4tM9abeyGinRKlg8MjBLV6kc7p72ujih6v5k6 N6zhw0i9xJM8RWv3TNc38QhuAz3tl1NoW4PKeZOr33nPzlcRkd7e+X8o0OkI0iNCK0AIS0 of8feF74iT41cTeKIhdTHLLq45JwODQ/O20g/m23rWE7HZHogzez0QiPOC133gQ6MoK+J8 vbLvgjG1SeIOxNOvfXN2K0Mak33WPJuYEHTyOTz69lno29l/6F3xZjdE+9yYpg== X-Mailman-Original-DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1720774553; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+Q70TNir7IBy52CbPauTJ3qNqdXl8vBAfbwUA/7ETBE=; b=0qJ0+cHkqB7yPwcVrmGgPzKyU94wvR9IeZ+KtC7MZxppq20puzFwWEr9SH5CQhZwFRbmlC YEqrKb/75GGKh8DA== X-Mailman-Original-Authentication-Results: smtp3.osuosl.org; dmarc=pass (p=none dis=none) header.from=linutronix.de X-Mailman-Original-Authentication-Results: smtp3.osuosl.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.a=rsa-sha256 header.s=2020 header.b=OUKpw4dY; dkim=pass header.d=linutronix.de header.i=@linutronix.de header.a=ed25519-sha256 header.s=2020e header.b=0qJ0+cHk Subject: [Intel-wired-lan] [PATCH iwl-next v5 1/4] igb: prepare for AF_XDP zero-copy support X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jesper Dangaard Brouer , Daniel Borkmann , Sriram Yagnaraman , Richard Cochran , Kurt Kanzenbach , John Fastabend , Alexei Starovoitov , Benjamin Steinke , Eric Dumazet , Sriram Yagnaraman , intel-wired-lan@lists.osuosl.org, netdev@vger.kernel.org, Jakub Kicinski , bpf@vger.kernel.org, Paolo Abeni , "David S. Miller" , Sebastian Andrzej Siewior Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" From: Sriram Yagnaraman Always call igb_xdp_ring_update_tail under __netif_tx_lock, add a comment to indicate that. This is needed to share the same TX ring between XDP, XSK and slow paths. Remove static qualifiers on the following functions to be able to call from XSK specific file that is added in the later patches - igb_xdp_tx_queue_mapping - igb_xdp_ring_update_tail - igb_clean_tx_ring - igb_clean_rx_ring - igb_run_xdp - igb_process_skb_fields Introduce igb_xdp_is_enabled() to check if an XDP program is assigned to the device. Signed-off-by: Sriram Yagnaraman Signed-off-by: Kurt Kanzenbach Tested-by: Chandan Kumar Rout (A Contingent Worker at Intel) --- drivers/net/ethernet/intel/igb/igb.h | 15 +++++++++++++++ drivers/net/ethernet/intel/igb/igb_main.c | 29 ++++++++++++++--------------- 2 files changed, 29 insertions(+), 15 deletions(-) diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h index 3c2dc7bdebb5..0de71ec324ed 100644 --- a/drivers/net/ethernet/intel/igb/igb.h +++ b/drivers/net/ethernet/intel/igb/igb.h @@ -718,6 +718,8 @@ extern char igb_driver_name[]; int igb_xmit_xdp_ring(struct igb_adapter *adapter, struct igb_ring *ring, struct xdp_frame *xdpf); +struct igb_ring *igb_xdp_tx_queue_mapping(struct igb_adapter *adapter); +void igb_xdp_ring_update_tail(struct igb_ring *ring); int igb_open(struct net_device *netdev); int igb_close(struct net_device *netdev); int igb_up(struct igb_adapter *); @@ -731,12 +733,20 @@ int igb_setup_tx_resources(struct igb_ring *); int igb_setup_rx_resources(struct igb_ring *); void igb_free_tx_resources(struct igb_ring *); void igb_free_rx_resources(struct igb_ring *); +void igb_clean_tx_ring(struct igb_ring *tx_ring); +void igb_clean_rx_ring(struct igb_ring *rx_ring); void igb_configure_tx_ring(struct igb_adapter *, struct igb_ring *); void igb_configure_rx_ring(struct igb_adapter *, struct igb_ring *); void igb_setup_tctl(struct igb_adapter *); void igb_setup_rctl(struct igb_adapter *); void igb_setup_srrctl(struct igb_adapter *, struct igb_ring *); netdev_tx_t igb_xmit_frame_ring(struct sk_buff *, struct igb_ring *); +struct sk_buff *igb_run_xdp(struct igb_adapter *adapter, + struct igb_ring *rx_ring, + struct xdp_buff *xdp); +void igb_process_skb_fields(struct igb_ring *rx_ring, + union e1000_adv_rx_desc *rx_desc, + struct sk_buff *skb); void igb_alloc_rx_buffers(struct igb_ring *, u16); void igb_update_stats(struct igb_adapter *); bool igb_has_link(struct igb_adapter *adapter); @@ -797,6 +807,11 @@ static inline struct netdev_queue *txring_txq(const struct igb_ring *tx_ring) return netdev_get_tx_queue(tx_ring->netdev, tx_ring->queue_index); } +static inline bool igb_xdp_is_enabled(struct igb_adapter *adapter) +{ + return !!adapter->xdp_prog; +} + int igb_add_filter(struct igb_adapter *adapter, struct igb_nfc_filter *input); int igb_erase_filter(struct igb_adapter *adapter, diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index 3af03a211c3c..7f84e4117fa7 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -115,8 +115,6 @@ static void igb_configure_tx(struct igb_adapter *); static void igb_configure_rx(struct igb_adapter *); static void igb_clean_all_tx_rings(struct igb_adapter *); static void igb_clean_all_rx_rings(struct igb_adapter *); -static void igb_clean_tx_ring(struct igb_ring *); -static void igb_clean_rx_ring(struct igb_ring *); static void igb_set_rx_mode(struct net_device *); static void igb_update_phy_info(struct timer_list *); static void igb_watchdog(struct timer_list *); @@ -2915,7 +2913,8 @@ static int igb_xdp(struct net_device *dev, struct netdev_bpf *xdp) } } -static void igb_xdp_ring_update_tail(struct igb_ring *ring) +/* This function assumes __netif_tx_lock is held by the caller. */ +void igb_xdp_ring_update_tail(struct igb_ring *ring) { /* Force memory writes to complete before letting h/w know there * are new descriptors to fetch. @@ -2924,7 +2923,7 @@ static void igb_xdp_ring_update_tail(struct igb_ring *ring) writel(ring->next_to_use, ring->tail); } -static struct igb_ring *igb_xdp_tx_queue_mapping(struct igb_adapter *adapter) +struct igb_ring *igb_xdp_tx_queue_mapping(struct igb_adapter *adapter) { unsigned int r_idx = smp_processor_id(); @@ -3001,11 +3000,11 @@ static int igb_xdp_xmit(struct net_device *dev, int n, nxmit++; } - __netif_tx_unlock(nq); - if (unlikely(flags & XDP_XMIT_FLUSH)) igb_xdp_ring_update_tail(tx_ring); + __netif_tx_unlock(nq); + return nxmit; } @@ -4880,7 +4879,7 @@ static void igb_free_all_tx_resources(struct igb_adapter *adapter) * igb_clean_tx_ring - Free Tx Buffers * @tx_ring: ring to be cleaned **/ -static void igb_clean_tx_ring(struct igb_ring *tx_ring) +void igb_clean_tx_ring(struct igb_ring *tx_ring) { u16 i = tx_ring->next_to_clean; struct igb_tx_buffer *tx_buffer = &tx_ring->tx_buffer_info[i]; @@ -4999,7 +4998,7 @@ static void igb_free_all_rx_resources(struct igb_adapter *adapter) * igb_clean_rx_ring - Free Rx Buffers per Queue * @rx_ring: ring to free buffers from **/ -static void igb_clean_rx_ring(struct igb_ring *rx_ring) +void igb_clean_rx_ring(struct igb_ring *rx_ring) { u16 i = rx_ring->next_to_clean; @@ -6614,7 +6613,7 @@ static int igb_change_mtu(struct net_device *netdev, int new_mtu) struct igb_adapter *adapter = netdev_priv(netdev); int max_frame = new_mtu + IGB_ETH_PKT_HDR_PAD; - if (adapter->xdp_prog) { + if (igb_xdp_is_enabled(adapter)) { int i; for (i = 0; i < adapter->num_rx_queues; i++) { @@ -8570,9 +8569,9 @@ static struct sk_buff *igb_build_skb(struct igb_ring *rx_ring, return skb; } -static struct sk_buff *igb_run_xdp(struct igb_adapter *adapter, - struct igb_ring *rx_ring, - struct xdp_buff *xdp) +struct sk_buff *igb_run_xdp(struct igb_adapter *adapter, + struct igb_ring *rx_ring, + struct xdp_buff *xdp) { int err, result = IGB_XDP_PASS; struct bpf_prog *xdp_prog; @@ -8768,9 +8767,9 @@ static bool igb_cleanup_headers(struct igb_ring *rx_ring, * order to populate the hash, checksum, VLAN, timestamp, protocol, and * other fields within the skb. **/ -static void igb_process_skb_fields(struct igb_ring *rx_ring, - union e1000_adv_rx_desc *rx_desc, - struct sk_buff *skb) +void igb_process_skb_fields(struct igb_ring *rx_ring, + union e1000_adv_rx_desc *rx_desc, + struct sk_buff *skb) { struct net_device *dev = rx_ring->netdev; From patchwork Fri Jul 12 08:55:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kurt Kanzenbach X-Patchwork-Id: 1959717 X-Patchwork-Delegate: anthony.l.nguyen@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=osuosl.org header.i=@osuosl.org header.a=rsa-sha256 header.s=default header.b=lSNG+8bY; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=osuosl.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WL56W2s9yz1xqj for ; Fri, 12 Jul 2024 18:56:02 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 6C5A5412DB; Fri, 12 Jul 2024 08:56:01 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id BtpZrfCjsO6f; Fri, 12 Jul 2024 08:55:59 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.166.34; helo=ash.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 3EA9A412C0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osuosl.org; s=default; t=1720774559; bh=ih6UbIZGDMnr/iZr8mWx107M7mRUF4icdpIKWbHtISs=; h=From:Date:References:In-Reply-To:To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: Cc:From; b=lSNG+8bYdElZrv3haeAKRgWKdc1gtSOvOYA2k2h4DwWXDTP6jfKQrc5vPuzjiIlR1 YdzUJJ7Q4jtKI5flbjtm4hP3HIOGd1YVi5ziY8bYrMBEKQZbJb8NmxMdwQ2NkjPRuC bGGQoJ/Nl5KMHflapFTcAX9FTqLBXG5kJcLzR+XglMn4Z0uByMgiHUOlR0pHC/+rci T4UrisGV9Pu6c+/D8q7ZNVJrizHLChb5wFFUT7XXt9OCquj0oOnLbLO7OjxmR1oyP9 4VnUOGtdywhYpc0d50ZhZ04AgefSqzTYvGkCFlpnLzx2VjghT8c8EbCQBwLSJp2Sci +whhDjKhxh3Zg== Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp2.osuosl.org (Postfix) with ESMTP id 3EA9A412C0; Fri, 12 Jul 2024 08:55:59 +0000 (UTC) X-Original-To: intel-wired-lan@lists.osuosl.org Delivered-To: intel-wired-lan@lists.osuosl.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id D8CC01BF584 for ; Fri, 12 Jul 2024 08:55:57 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id D0AF44150E for ; Fri, 12 Jul 2024 08:55:57 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 7_7bpmt4JOp4 for ; Fri, 12 Jul 2024 08:55:56 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a0a:51c0:0:12e:550::1; helo=galois.linutronix.de; envelope-from=kurt@linutronix.de; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp4.osuosl.org 0DECD41486 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 0DECD41486 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by smtp4.osuosl.org (Postfix) with ESMTPS id 0DECD41486 for ; Fri, 12 Jul 2024 08:55:55 +0000 (UTC) From: Kurt Kanzenbach Date: Fri, 12 Jul 2024 10:55:30 +0200 MIME-Version: 1.0 Message-Id: <20240711-b4-igb_zero_copy-v5-2-f3f455113b11@linutronix.de> References: <20240711-b4-igb_zero_copy-v5-0-f3f455113b11@linutronix.de> In-Reply-To: <20240711-b4-igb_zero_copy-v5-0-f3f455113b11@linutronix.de> To: Tony Nguyen , Przemek Kitszel X-Developer-Signature: v=1; a=openpgp-sha256; l=10554; i=kurt@linutronix.de; h=from:subject:message-id; bh=VG6jHt8qOitfnJ9fdfNZPgCPEQaFR5JTFwYUKjoZL6A=; b=owEBbQKS/ZANAwAKAcGT0fKqRnOCAcsmYgBmkO+W+bzewYAxri7m9jb+JG9oJmYX9yao4QFqB S8J0d1bNb+JAjMEAAEKAB0WIQS8ub+yyMN909/bWZLBk9HyqkZzggUCZpDvlgAKCRDBk9HyqkZz gpFDD/wIXn3CgJiCPKthCqcI/CZIjcl1X4ztwAXmbVJbuu7WMFv5YJDGTWp7ntx9RGTusVfbPcM DU+YoAagYS8m9qhYh1Gt222Dn/dCwq3qXsUMpSBUTA/dh3D6edA19AOD5Oc2UHVplsJlIvKGkDd 0HLD9onJvIYlnxNTGwJgf7amBh3kGD/44WBK9BAnAmzNy22dJuA988yRI4mQP03x0P5Q1dubQAN yiahuSkcF4bM2muP23VvJQTvFfCfI1xRP69lCE5F3o6UYvhjztrTVHhtVyFeFi2te6CbCNksWu9 svbxUVJ7cxCMfv32wEulGjO1ohWEIL7TIwdxiPVcL+G/zAyKwWMbQ1uZNmQ5H1vcrMDBvENxiLT cO3+5LUyxWPog07HQsfwDNRH9YbFHaRUF+ketC4yGKPBQvzpE34rVreZ3L23pMTTopzVsMVyFUP l2hL4hEpvcdzT+7vafzy784B2fG4eEqrVup/IM5c9Xd8gpr7hM6yE3/AcwS4fzdADd0lUV1/HZu c14wBBVRAH2sb18/uQvlTY3EqVuuRHoxQ8D9HNPPVetWch+yHndQtAXeQvnnIKygdMjDhJiBMlo pjRQIjMTCCgUB6izCq9x0FPT4CjDZabNvIIFAt8CiS3kBFZ5gkFd5fakSug4UkwtAZrbaYSKpJ0 xALLP8dBXbphiEw== X-Developer-Key: i=kurt@linutronix.de; a=openpgp; fpr=BCB9BFB2C8C37DD3DFDB5992C193D1F2AA467382 X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1720774554; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ih6UbIZGDMnr/iZr8mWx107M7mRUF4icdpIKWbHtISs=; b=dLJJvv7ghyD0PkwAPammaxDRiVzvbz8ko4ZKJQ2eMFuUxXIFye4crxHquvUN8P0uOCMK70 UqtDqpXK3aysE4D10lfk+Oh6TpWETdZrcjjbJ6WzMheXds6N6VSvPkx3ZRKzlDhUcUAg+i 9FZC3WqXesxsFkuVG7eiulahvL07SquWyLOduzc/1MqQZ68SKsku53b9D29ixPefzT0+H7 ZcMF0Byj0N7AjPB4TLnw6XvIradfW83LHiKXGr/vP9dv/ySvwZ7fH+SLevgMNZWlotcuCG wcleYKobT9gAcyR+z1mhTps3PPCpBormxRvfGqu7AoKDOfvEDwrnHWuJueJ4rw== X-Mailman-Original-DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1720774554; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ih6UbIZGDMnr/iZr8mWx107M7mRUF4icdpIKWbHtISs=; b=iScPox6mlg4H1uix9DxNNEz9FGyUdGCmKmD/gG9JMBYHh4qJOHBUXTnwTtrTEbAocfSKQS q/S5kH/zz4dtNQAQ== X-Mailman-Original-Authentication-Results: smtp4.osuosl.org; dmarc=pass (p=none dis=none) header.from=linutronix.de X-Mailman-Original-Authentication-Results: smtp4.osuosl.org; dkim=pass (2048-bit key, unprotected) header.d=linutronix.de header.i=@linutronix.de header.a=rsa-sha256 header.s=2020 header.b=dLJJvv7g; dkim=pass header.d=linutronix.de header.i=@linutronix.de header.a=ed25519-sha256 header.s=2020e header.b=iScPox6m Subject: [Intel-wired-lan] [PATCH iwl-next v5 2/4] igb: Introduce XSK data structures and helpers X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jesper Dangaard Brouer , Daniel Borkmann , Sriram Yagnaraman , Richard Cochran , Kurt Kanzenbach , John Fastabend , Alexei Starovoitov , Benjamin Steinke , Eric Dumazet , Sriram Yagnaraman , intel-wired-lan@lists.osuosl.org, netdev@vger.kernel.org, Jakub Kicinski , bpf@vger.kernel.org, Paolo Abeni , "David S. Miller" , Sebastian Andrzej Siewior Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" From: Sriram Yagnaraman Add the following ring flags - IGB_RING_FLAG_TX_DISABLED (when xsk pool is being setup) - IGB_RING_FLAG_AF_XDP_ZC (xsk pool is active) Add a xdp_buff array for use with XSK receive batch API, and a pointer to xsk_pool in igb_adapter. Add enable/disable functions for TX and RX rings Add enable/disable functions for XSK pool Add xsk wakeup function None of the above functionality will be active until NETDEV_XDP_ACT_XSK_ZEROCOPY is advertised in netdev->xdp_features. Signed-off-by: Sriram Yagnaraman Signed-off-by: Kurt Kanzenbach Tested-by: Chandan Kumar Rout (A Contingent Worker at Intel) --- drivers/net/ethernet/intel/igb/Makefile | 2 +- drivers/net/ethernet/intel/igb/igb.h | 14 +- drivers/net/ethernet/intel/igb/igb_main.c | 9 ++ drivers/net/ethernet/intel/igb/igb_xsk.c | 210 ++++++++++++++++++++++++++++++ 4 files changed, 233 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/intel/igb/Makefile b/drivers/net/ethernet/intel/igb/Makefile index 463c0d26b9d4..6c1b702fd992 100644 --- a/drivers/net/ethernet/intel/igb/Makefile +++ b/drivers/net/ethernet/intel/igb/Makefile @@ -8,4 +8,4 @@ obj-$(CONFIG_IGB) += igb.o igb-y := igb_main.o igb_ethtool.o e1000_82575.o \ e1000_mac.o e1000_nvm.o e1000_phy.o e1000_mbx.o \ - e1000_i210.o igb_ptp.o igb_hwmon.o + e1000_i210.o igb_ptp.o igb_hwmon.o igb_xsk.o diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h index 0de71ec324ed..053130c01480 100644 --- a/drivers/net/ethernet/intel/igb/igb.h +++ b/drivers/net/ethernet/intel/igb/igb.h @@ -20,6 +20,7 @@ #include #include +#include struct igb_adapter; @@ -320,6 +321,7 @@ struct igb_ring { union { /* array of buffer info structs */ struct igb_tx_buffer *tx_buffer_info; struct igb_rx_buffer *rx_buffer_info; + struct xdp_buff **rx_buffer_info_zc; }; void *desc; /* descriptor ring memory */ unsigned long flags; /* ring specific flags */ @@ -357,6 +359,7 @@ struct igb_ring { }; }; struct xdp_rxq_info xdp_rxq; + struct xsk_buff_pool *xsk_pool; } ____cacheline_internodealigned_in_smp; struct igb_q_vector { @@ -384,7 +387,9 @@ enum e1000_ring_flags_t { IGB_RING_FLAG_RX_SCTP_CSUM, IGB_RING_FLAG_RX_LB_VLAN_BSWAP, IGB_RING_FLAG_TX_CTX_IDX, - IGB_RING_FLAG_TX_DETECT_HANG + IGB_RING_FLAG_TX_DETECT_HANG, + IGB_RING_FLAG_TX_DISABLED, + IGB_RING_FLAG_AF_XDP_ZC }; #define ring_uses_large_buffer(ring) \ @@ -822,4 +827,11 @@ int igb_add_mac_steering_filter(struct igb_adapter *adapter, int igb_del_mac_steering_filter(struct igb_adapter *adapter, const u8 *addr, u8 queue, u8 flags); +struct xsk_buff_pool *igb_xsk_pool(struct igb_adapter *adapter, + struct igb_ring *ring); +int igb_xsk_pool_setup(struct igb_adapter *adapter, + struct xsk_buff_pool *pool, + u16 qid); +int igb_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags); + #endif /* _IGB_H_ */ diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index 7f84e4117fa7..28abe9664c44 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -2905,9 +2905,14 @@ static int igb_xdp_setup(struct net_device *dev, struct netdev_bpf *bpf) static int igb_xdp(struct net_device *dev, struct netdev_bpf *xdp) { + struct igb_adapter *adapter = netdev_priv(dev); + switch (xdp->command) { case XDP_SETUP_PROG: return igb_xdp_setup(dev, xdp); + case XDP_SETUP_XSK_POOL: + return igb_xsk_pool_setup(adapter, xdp->xsk.pool, + xdp->xsk.queue_id); default: return -EINVAL; } @@ -3034,6 +3039,7 @@ static const struct net_device_ops igb_netdev_ops = { .ndo_setup_tc = igb_setup_tc, .ndo_bpf = igb_xdp, .ndo_xdp_xmit = igb_xdp_xmit, + .ndo_xsk_wakeup = igb_xsk_wakeup, }; /** @@ -4356,6 +4362,8 @@ void igb_configure_tx_ring(struct igb_adapter *adapter, u64 tdba = ring->dma; int reg_idx = ring->reg_idx; + ring->xsk_pool = igb_xsk_pool(adapter, ring); + wr32(E1000_TDLEN(reg_idx), ring->count * sizeof(union e1000_adv_tx_desc)); wr32(E1000_TDBAL(reg_idx), @@ -4751,6 +4759,7 @@ void igb_configure_rx_ring(struct igb_adapter *adapter, xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq); WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq, MEM_TYPE_PAGE_SHARED, NULL)); + ring->xsk_pool = igb_xsk_pool(adapter, ring); /* disable the queue */ wr32(E1000_RXDCTL(reg_idx), 0); diff --git a/drivers/net/ethernet/intel/igb/igb_xsk.c b/drivers/net/ethernet/intel/igb/igb_xsk.c new file mode 100644 index 000000000000..925bf97f7caa --- /dev/null +++ b/drivers/net/ethernet/intel/igb/igb_xsk.c @@ -0,0 +1,210 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2018 Intel Corporation. */ + +#include +#include +#include + +#include "e1000_hw.h" +#include "igb.h" + +static int igb_realloc_rx_buffer_info(struct igb_ring *ring, bool pool_present) +{ + int size = pool_present ? + sizeof(*ring->rx_buffer_info_zc) * ring->count : + sizeof(*ring->rx_buffer_info) * ring->count; + void *buff_info = vmalloc(size); + + if (!buff_info) + return -ENOMEM; + + if (pool_present) { + vfree(ring->rx_buffer_info); + ring->rx_buffer_info = NULL; + ring->rx_buffer_info_zc = buff_info; + } else { + vfree(ring->rx_buffer_info_zc); + ring->rx_buffer_info_zc = NULL; + ring->rx_buffer_info = buff_info; + } + + return 0; +} + +static void igb_txrx_ring_disable(struct igb_adapter *adapter, u16 qid) +{ + struct igb_ring *tx_ring = adapter->tx_ring[qid]; + struct igb_ring *rx_ring = adapter->rx_ring[qid]; + struct e1000_hw *hw = &adapter->hw; + + set_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags); + + wr32(E1000_TXDCTL(tx_ring->reg_idx), 0); + wr32(E1000_RXDCTL(rx_ring->reg_idx), 0); + + /* Rx/Tx share the same napi context. */ + napi_disable(&rx_ring->q_vector->napi); + + igb_clean_tx_ring(tx_ring); + igb_clean_rx_ring(rx_ring); + + memset(&rx_ring->rx_stats, 0, sizeof(rx_ring->rx_stats)); + memset(&tx_ring->tx_stats, 0, sizeof(tx_ring->tx_stats)); +} + +static void igb_txrx_ring_enable(struct igb_adapter *adapter, u16 qid) +{ + struct igb_ring *tx_ring = adapter->tx_ring[qid]; + struct igb_ring *rx_ring = adapter->rx_ring[qid]; + + igb_configure_tx_ring(adapter, tx_ring); + igb_configure_rx_ring(adapter, rx_ring); + + clear_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags); + + /* call igb_desc_unused which always leaves + * at least 1 descriptor unused to make sure + * next_to_use != next_to_clean + */ + igb_alloc_rx_buffers(rx_ring, igb_desc_unused(rx_ring)); + + /* Rx/Tx share the same napi context. */ + napi_enable(&rx_ring->q_vector->napi); +} + +struct xsk_buff_pool *igb_xsk_pool(struct igb_adapter *adapter, + struct igb_ring *ring) +{ + int qid = ring->queue_index; + + if (!igb_xdp_is_enabled(adapter) || + !test_bit(IGB_RING_FLAG_AF_XDP_ZC, &ring->flags)) + return NULL; + + return xsk_get_pool_from_qid(adapter->netdev, qid); +} + +static int igb_xsk_pool_enable(struct igb_adapter *adapter, + struct xsk_buff_pool *pool, + u16 qid) +{ + struct net_device *netdev = adapter->netdev; + struct igb_ring *tx_ring, *rx_ring; + bool if_running; + int err; + + if (qid >= adapter->num_rx_queues) + return -EINVAL; + + if (qid >= netdev->real_num_rx_queues || + qid >= netdev->real_num_tx_queues) + return -EINVAL; + + err = xsk_pool_dma_map(pool, &adapter->pdev->dev, IGB_RX_DMA_ATTR); + if (err) + return err; + + tx_ring = adapter->tx_ring[qid]; + rx_ring = adapter->rx_ring[qid]; + if_running = netif_running(adapter->netdev) && igb_xdp_is_enabled(adapter); + if (if_running) + igb_txrx_ring_disable(adapter, qid); + + set_bit(IGB_RING_FLAG_AF_XDP_ZC, &tx_ring->flags); + set_bit(IGB_RING_FLAG_AF_XDP_ZC, &rx_ring->flags); + + if (if_running) { + err = igb_realloc_rx_buffer_info(rx_ring, true); + if (!err) { + igb_txrx_ring_enable(adapter, qid); + /* Kick start the NAPI context so that receiving will start */ + err = igb_xsk_wakeup(adapter->netdev, qid, XDP_WAKEUP_RX); + } + + if (err) { + clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &tx_ring->flags); + clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &rx_ring->flags); + xsk_pool_dma_unmap(pool, IGB_RX_DMA_ATTR); + return err; + } + } + + return 0; +} + +static int igb_xsk_pool_disable(struct igb_adapter *adapter, u16 qid) +{ + struct igb_ring *tx_ring, *rx_ring; + struct xsk_buff_pool *pool; + bool if_running; + int err; + + pool = xsk_get_pool_from_qid(adapter->netdev, qid); + if (!pool) + return -EINVAL; + + tx_ring = adapter->tx_ring[qid]; + rx_ring = adapter->rx_ring[qid]; + if_running = netif_running(adapter->netdev) && igb_xdp_is_enabled(adapter); + if (if_running) + igb_txrx_ring_disable(adapter, qid); + + xsk_pool_dma_unmap(pool, IGB_RX_DMA_ATTR); + clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &tx_ring->flags); + clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &rx_ring->flags); + + if (if_running) { + err = igb_realloc_rx_buffer_info(rx_ring, false); + if (err) + return err; + + igb_txrx_ring_enable(adapter, qid); + } + + return 0; +} + +int igb_xsk_pool_setup(struct igb_adapter *adapter, + struct xsk_buff_pool *pool, + u16 qid) +{ + return pool ? igb_xsk_pool_enable(adapter, pool, qid) : + igb_xsk_pool_disable(adapter, qid); +} + +int igb_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags) +{ + struct igb_adapter *adapter = netdev_priv(dev); + struct e1000_hw *hw = &adapter->hw; + struct igb_ring *ring; + u32 eics = 0; + + if (test_bit(__IGB_DOWN, &adapter->state)) + return -ENETDOWN; + + if (!igb_xdp_is_enabled(adapter)) + return -EINVAL; + + if (qid >= adapter->num_tx_queues) + return -EINVAL; + + ring = adapter->tx_ring[qid]; + + if (test_bit(IGB_RING_FLAG_TX_DISABLED, &ring->flags)) + return -ENETDOWN; + + if (!ring->xsk_pool) + return -EINVAL; + + if (!napi_if_scheduled_mark_missed(&ring->q_vector->napi)) { + /* Cause software interrupt to ensure Rx ring is cleaned */ + if (adapter->flags & IGB_FLAG_HAS_MSIX) { + eics |= ring->q_vector->eims_value; + wr32(E1000_EICS, eics); + } else { + wr32(E1000_ICS, E1000_ICS_RXDMT0); + } + } + + return 0; +} From patchwork Fri Jul 12 08:55:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kurt Kanzenbach X-Patchwork-Id: 1959720 X-Patchwork-Delegate: anthony.l.nguyen@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=osuosl.org header.i=@osuosl.org header.a=rsa-sha256 header.s=default header.b=fSjmwR7t; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=osuosl.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WL56g3RvQz1xqj for ; Fri, 12 Jul 2024 18:56:11 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 1460F41433; Fri, 12 Jul 2024 08:56:08 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 3wGqIacXuKdL; Fri, 12 Jul 2024 08:56:04 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.166.34; helo=ash.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org BB2BA412D3 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osuosl.org; s=default; t=1720774564; bh=ssrE1dENjYOWX+omaQIOEmmXuhB42nS6FGCvY6DG/YA=; h=From:Date:References:In-Reply-To:To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: Cc:From; b=fSjmwR7tddKotvZjKxQNesUJF/vuiYqMrWVSnEWSnKIebqUI5TRsEfmNGDZ6yibMZ daaoDGmMNjp8sSAmtLRMc2TKllqIj8hvcW5clVvDi6dQcG+B+SnnvH5pJASz7ntgLZ zwvu5t/a/HeNi2q412BO5FQLC4/rBxEDyTKvVpOzxGtefsyZQ8YZkhZW0dnB5nk5IF W2mX7gGHwdwxxchl2GRIAC8R2zaNxODrLE5RFmzdm77lyBNVT0V3y/vfAptLqhhdws +blu7bi2H14N5mXfiQy4k7Teaewy8M2H7AqtL7p/BEnhunBXgmUGLVYSHLzZLmi2IF KYroa5fk6sNxA== Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp2.osuosl.org (Postfix) with ESMTP id BB2BA412D3; Fri, 12 Jul 2024 08:56:04 +0000 (UTC) X-Original-To: intel-wired-lan@lists.osuosl.org Delivered-To: intel-wired-lan@lists.osuosl.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 40A401BF584 for ; Fri, 12 Jul 2024 08:55:58 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 2C3B984601 for ; Fri, 12 Jul 2024 08:55:58 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id RokYa8tUxBtQ for ; Fri, 12 Jul 2024 08:55:57 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a0a:51c0:0:12e:550::1; helo=galois.linutronix.de; envelope-from=kurt@linutronix.de; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp1.osuosl.org 980B984600 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 980B984600 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by smtp1.osuosl.org (Postfix) with ESMTPS id 980B984600 for ; Fri, 12 Jul 2024 08:55:56 +0000 (UTC) From: Kurt Kanzenbach Date: Fri, 12 Jul 2024 10:55:31 +0200 MIME-Version: 1.0 Message-Id: <20240711-b4-igb_zero_copy-v5-3-f3f455113b11@linutronix.de> References: <20240711-b4-igb_zero_copy-v5-0-f3f455113b11@linutronix.de> In-Reply-To: <20240711-b4-igb_zero_copy-v5-0-f3f455113b11@linutronix.de> To: Tony Nguyen , Przemek Kitszel X-Developer-Signature: v=1; a=openpgp-sha256; l=17084; i=kurt@linutronix.de; h=from:subject:message-id; bh=0JIgbz4/m8tYGtwWiNrFP+wsDia4eif+jduJm6RTo9U=; b=owEBbQKS/ZANAwAKAcGT0fKqRnOCAcsmYgBmkO+WBlTW9k3Hmo1/4DmvdYH2NWKc+BYMnNUqW U/2onvrWK2JAjMEAAEKAB0WIQS8ub+yyMN909/bWZLBk9HyqkZzggUCZpDvlgAKCRDBk9HyqkZz gubGD/4vabZ9jJ9V01emJ8nSYFvHbvxdT7wFUUR6vd2fC0BudOq8KFt39VXpFFlWuMy33VnBzqe 5d2iYHoaPZXiCYSC7j1bfGhXhDGCZ5vUmM/RmzysJvQ4qs8dCiy9SPj34MapaNWUNr2QsPQjMb5 J13ITjFraP3n4zRV+RXgRF/5inyBifpFx4ITu++ZiEKPzphPom+fSJiqL6LDg/AwvUCmJZ1/CpM MmdJFMgLLvIsJEn8oUtm1e8Ge387MPzgIy7LA6Qv1Sm8X2v2DFRe2RlQEJaqniVsC1/GOnAHVCM PO+h8fES8rYfwtdqiDIEKJ6G2DlMzfUZo/1fYHahmoK1ztTawTS2MFJEbLZWV2FX3p9cpl15Zyu bQYjL5bb8jWem5RXv2E3ADzxP20fPbE3d8zXkRowYE10+2SEbHifOTvL92jRtnsqMtgHDFdoR2k wVFsy3WqZRTUPKM09yFyt0Qoe/l5Fa73GBNlGHzgES2rD9WjGA8l+fD3zyZooaWQa+/UWdDNTky +E+wEWJ2l07Ryly4d6VBgAv3bY0bT7MNWGxOziWhyhzDhD1+zIJCut8L6T2l5jHcuULhgZDwROw A8Il3m0oJ1N81iu6hNSSFy4eec6tEQuFxiWJoR/z5k3LhT+4bDLBE8FCZfZds0ILDaHKrZRx1NM ngQ/U4HIR18/4ug== X-Developer-Key: i=kurt@linutronix.de; a=openpgp; fpr=BCB9BFB2C8C37DD3DFDB5992C193D1F2AA467382 X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1720774554; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ssrE1dENjYOWX+omaQIOEmmXuhB42nS6FGCvY6DG/YA=; b=kfsuBc/HI0BxZ7eW3Paw0zo9bA2yzaGAls2TBsk5NWEXdmVHe1T39D0ZJsono0bNn9+bkK k0ZUNb9IM+UbDx0IDuvbog+9IrGKUWgzOcM7crhhcBIkbwVruJE73JuhGAN0bNNr3AG+6R w11Z7K4t9yNMjJwhAkrXyFxeMd4r9dL8m1JFIiwsKlgKpr/O9idyaV4S1Bb7/9Ns5VuWFI sVTVXtRPX5v78QGSxbzVOhZiPZxkqrwUjT4NSAdaZQKT+I069RxqDMVeNdumLWHUh7xuSZ tyHlmcCh2pzJYiPyiZ5BHjpn/Zevrhyo8cECuNvmFKrfeGvc1E0GBSgDWhWcdA== X-Mailman-Original-DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1720774554; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ssrE1dENjYOWX+omaQIOEmmXuhB42nS6FGCvY6DG/YA=; b=ZV6NWZFC2JB884RYw5Ai5L7jS97qaHoyzJaF6giDmynSD3tlhKFuSgccvR8dVMeXg9JLJ7 eqbs5AidcOdRAfAw== X-Mailman-Original-Authentication-Results: smtp1.osuosl.org; dmarc=pass (p=none dis=none) header.from=linutronix.de X-Mailman-Original-Authentication-Results: smtp1.osuosl.org; dkim=pass (2048-bit key, unprotected) header.d=linutronix.de header.i=@linutronix.de header.a=rsa-sha256 header.s=2020 header.b=kfsuBc/H; dkim=pass header.d=linutronix.de header.i=@linutronix.de header.a=ed25519-sha256 header.s=2020e header.b=ZV6NWZFC Subject: [Intel-wired-lan] [PATCH iwl-next v5 3/4] igb: add AF_XDP zero-copy Rx support X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jesper Dangaard Brouer , Daniel Borkmann , Sriram Yagnaraman , Richard Cochran , Kurt Kanzenbach , John Fastabend , Alexei Starovoitov , Benjamin Steinke , Eric Dumazet , Sriram Yagnaraman , intel-wired-lan@lists.osuosl.org, netdev@vger.kernel.org, Jakub Kicinski , bpf@vger.kernel.org, Paolo Abeni , "David S. Miller" , Sebastian Andrzej Siewior Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" From: Sriram Yagnaraman Add support for AF_XDP zero-copy receive path. When AF_XDP zero-copy is enabled, the rx buffers are allocated from the xsk buff pool using igb_alloc_rx_buffers_zc. Use xsk_pool_get_rx_frame_size to set SRRCTL rx buf size when zero-copy is enabled. Signed-off-by: Sriram Yagnaraman [Kurt: Port to v6.10 and provide napi_id for xdp_rxq_info_reg()] Signed-off-by: Kurt Kanzenbach Tested-by: Chandan Kumar Rout (A Contingent Worker at Intel) --- drivers/net/ethernet/intel/igb/igb.h | 4 + drivers/net/ethernet/intel/igb/igb_main.c | 95 ++++++++--- drivers/net/ethernet/intel/igb/igb_xsk.c | 261 +++++++++++++++++++++++++++++- 3 files changed, 337 insertions(+), 23 deletions(-) diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h index 053130c01480..4983a6ec718e 100644 --- a/drivers/net/ethernet/intel/igb/igb.h +++ b/drivers/net/ethernet/intel/igb/igb.h @@ -87,6 +87,7 @@ struct igb_adapter; #define IGB_XDP_CONSUMED BIT(0) #define IGB_XDP_TX BIT(1) #define IGB_XDP_REDIR BIT(2) +#define IGB_XDP_EXIT BIT(3) struct vf_data_storage { unsigned char vf_mac_addresses[ETH_ALEN]; @@ -832,6 +833,9 @@ struct xsk_buff_pool *igb_xsk_pool(struct igb_adapter *adapter, int igb_xsk_pool_setup(struct igb_adapter *adapter, struct xsk_buff_pool *pool, u16 qid); +bool igb_alloc_rx_buffers_zc(struct igb_ring *rx_ring, u16 count); +void igb_clean_rx_ring_zc(struct igb_ring *rx_ring); +int igb_clean_rx_irq_zc(struct igb_q_vector *q_vector, const int budget); int igb_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags); #endif /* _IGB_H_ */ diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index 28abe9664c44..8d65c9eca102 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -473,12 +473,17 @@ static void igb_dump(struct igb_adapter *adapter) for (i = 0; i < rx_ring->count; i++) { const char *next_desc; - struct igb_rx_buffer *buffer_info; - buffer_info = &rx_ring->rx_buffer_info[i]; + dma_addr_t dma = (dma_addr_t)0; + struct igb_rx_buffer *buffer_info = NULL; rx_desc = IGB_RX_DESC(rx_ring, i); u0 = (struct my_u0 *)rx_desc; staterr = le32_to_cpu(rx_desc->wb.upper.status_error); + if (!rx_ring->xsk_pool) { + buffer_info = &rx_ring->rx_buffer_info[i]; + dma = buffer_info->dma; + } + if (i == rx_ring->next_to_use) next_desc = " NTU"; else if (i == rx_ring->next_to_clean) @@ -498,11 +503,11 @@ static void igb_dump(struct igb_adapter *adapter) "R ", i, le64_to_cpu(u0->a), le64_to_cpu(u0->b), - (u64)buffer_info->dma, + (u64)dma, next_desc); if (netif_msg_pktdata(adapter) && - buffer_info->dma && buffer_info->page) { + buffer_info && dma && buffer_info->page) { print_hex_dump(KERN_INFO, "", DUMP_PREFIX_ADDRESS, 16, 1, @@ -1984,7 +1989,10 @@ static void igb_configure(struct igb_adapter *adapter) */ for (i = 0; i < adapter->num_rx_queues; i++) { struct igb_ring *ring = adapter->rx_ring[i]; - igb_alloc_rx_buffers(ring, igb_desc_unused(ring)); + if (ring->xsk_pool) + igb_alloc_rx_buffers_zc(ring, igb_desc_unused(ring)); + else + igb_alloc_rx_buffers(ring, igb_desc_unused(ring)); } } @@ -3336,7 +3344,8 @@ static int igb_probe(struct pci_dev *pdev, const struct pci_device_id *ent) netdev->priv_flags |= IFF_SUPP_NOFCS; netdev->priv_flags |= IFF_UNICAST_FLT; - netdev->xdp_features = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT; + netdev->xdp_features = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT | + NETDEV_XDP_ACT_XSK_ZEROCOPY; /* MTU range: 68 - 9216 */ netdev->min_mtu = ETH_MIN_MTU; @@ -4424,7 +4433,8 @@ int igb_setup_rx_resources(struct igb_ring *rx_ring) if (xdp_rxq_info_is_reg(&rx_ring->xdp_rxq)) xdp_rxq_info_unreg(&rx_ring->xdp_rxq); res = xdp_rxq_info_reg(&rx_ring->xdp_rxq, rx_ring->netdev, - rx_ring->queue_index, 0); + rx_ring->queue_index, + rx_ring->q_vector->napi.napi_id); if (res < 0) { dev_err(dev, "Failed to register xdp_rxq index %u\n", rx_ring->queue_index); @@ -4720,12 +4730,17 @@ void igb_setup_srrctl(struct igb_adapter *adapter, struct igb_ring *ring) struct e1000_hw *hw = &adapter->hw; int reg_idx = ring->reg_idx; u32 srrctl = 0; + u32 buf_size; - srrctl = IGB_RX_HDR_LEN << E1000_SRRCTL_BSIZEHDRSIZE_SHIFT; - if (ring_uses_large_buffer(ring)) - srrctl |= IGB_RXBUFFER_3072 >> E1000_SRRCTL_BSIZEPKT_SHIFT; + if (ring->xsk_pool) + buf_size = xsk_pool_get_rx_frame_size(ring->xsk_pool); + else if (ring_uses_large_buffer(ring)) + buf_size = IGB_RXBUFFER_3072; else - srrctl |= IGB_RXBUFFER_2048 >> E1000_SRRCTL_BSIZEPKT_SHIFT; + buf_size = IGB_RXBUFFER_2048; + + srrctl = IGB_RX_HDR_LEN << E1000_SRRCTL_BSIZEHDRSIZE_SHIFT; + srrctl |= buf_size >> E1000_SRRCTL_BSIZEPKT_SHIFT; srrctl |= E1000_SRRCTL_DESCTYPE_ADV_ONEBUF; if (hw->mac.type >= e1000_82580) srrctl |= E1000_SRRCTL_TIMESTAMP; @@ -4757,9 +4772,17 @@ void igb_configure_rx_ring(struct igb_adapter *adapter, u32 rxdctl = 0; xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq); - WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq, - MEM_TYPE_PAGE_SHARED, NULL)); ring->xsk_pool = igb_xsk_pool(adapter, ring); + if (ring->xsk_pool) { + WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq, + MEM_TYPE_XSK_BUFF_POOL, + NULL)); + xsk_pool_set_rxq_info(ring->xsk_pool, &ring->xdp_rxq); + } else { + WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq, + MEM_TYPE_PAGE_SHARED, + NULL)); + } /* disable the queue */ wr32(E1000_RXDCTL(reg_idx), 0); @@ -4786,9 +4809,12 @@ void igb_configure_rx_ring(struct igb_adapter *adapter, rxdctl |= IGB_RX_HTHRESH << 8; rxdctl |= IGB_RX_WTHRESH << 16; - /* initialize rx_buffer_info */ - memset(ring->rx_buffer_info, 0, - sizeof(struct igb_rx_buffer) * ring->count); + if (ring->xsk_pool) + memset(ring->rx_buffer_info_zc, 0, + sizeof(*ring->rx_buffer_info_zc) * ring->count); + else + memset(ring->rx_buffer_info, 0, + sizeof(*ring->rx_buffer_info) * ring->count); /* initialize Rx descriptor 0 */ rx_desc = IGB_RX_DESC(ring, 0); @@ -4975,8 +5001,13 @@ void igb_free_rx_resources(struct igb_ring *rx_ring) rx_ring->xdp_prog = NULL; xdp_rxq_info_unreg(&rx_ring->xdp_rxq); - vfree(rx_ring->rx_buffer_info); - rx_ring->rx_buffer_info = NULL; + if (rx_ring->xsk_pool) { + vfree(rx_ring->rx_buffer_info_zc); + rx_ring->rx_buffer_info_zc = NULL; + } else { + vfree(rx_ring->rx_buffer_info); + rx_ring->rx_buffer_info = NULL; + } /* if not set, then don't free */ if (!rx_ring->desc) @@ -5014,6 +5045,11 @@ void igb_clean_rx_ring(struct igb_ring *rx_ring) dev_kfree_skb(rx_ring->skb); rx_ring->skb = NULL; + if (rx_ring->xsk_pool) { + igb_clean_rx_ring_zc(rx_ring); + goto skip_for_xsk; + } + /* Free all the Rx ring sk_buffs */ while (i != rx_ring->next_to_alloc) { struct igb_rx_buffer *buffer_info = &rx_ring->rx_buffer_info[i]; @@ -5041,6 +5077,7 @@ void igb_clean_rx_ring(struct igb_ring *rx_ring) i = 0; } +skip_for_xsk: rx_ring->next_to_alloc = 0; rx_ring->next_to_clean = 0; rx_ring->next_to_use = 0; @@ -8196,7 +8233,9 @@ static int igb_poll(struct napi_struct *napi, int budget) clean_complete = igb_clean_tx_irq(q_vector, budget); if (q_vector->rx.ring) { - int cleaned = igb_clean_rx_irq(q_vector, budget); + int cleaned = q_vector->rx.ring->xsk_pool ? + igb_clean_rx_irq_zc(q_vector, budget) : + igb_clean_rx_irq(q_vector, budget); work_done += cleaned; if (cleaned >= budget) @@ -8604,8 +8643,15 @@ struct sk_buff *igb_run_xdp(struct igb_adapter *adapter, break; case XDP_REDIRECT: err = xdp_do_redirect(adapter->netdev, xdp, xdp_prog); - if (err) + if (err) { + if (rx_ring->xsk_pool && + xsk_uses_need_wakeup(rx_ring->xsk_pool) && + err == -ENOBUFS) + result = IGB_XDP_EXIT; + else + result = IGB_XDP_CONSUMED; goto out_failure; + } result = IGB_XDP_REDIR; break; default: @@ -8862,12 +8908,14 @@ static void igb_put_rx_buffer(struct igb_ring *rx_ring, static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget) { + unsigned int total_bytes = 0, total_packets = 0; struct igb_adapter *adapter = q_vector->adapter; struct igb_ring *rx_ring = q_vector->rx.ring; - struct sk_buff *skb = rx_ring->skb; - unsigned int total_bytes = 0, total_packets = 0; u16 cleaned_count = igb_desc_unused(rx_ring); + struct sk_buff *skb = rx_ring->skb; + int cpu = smp_processor_id(); unsigned int xdp_xmit = 0; + struct netdev_queue *nq; struct xdp_buff xdp; u32 frame_sz = 0; int rx_buf_pgcnt; @@ -8995,7 +9043,10 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget) if (xdp_xmit & IGB_XDP_TX) { struct igb_ring *tx_ring = igb_xdp_tx_queue_mapping(adapter); + nq = txring_txq(tx_ring); + __netif_tx_lock(nq, cpu); igb_xdp_ring_update_tail(tx_ring); + __netif_tx_unlock(nq); } u64_stats_update_begin(&rx_ring->rx_syncp); diff --git a/drivers/net/ethernet/intel/igb/igb_xsk.c b/drivers/net/ethernet/intel/igb/igb_xsk.c index 925bf97f7caa..66cdc30e9b6e 100644 --- a/drivers/net/ethernet/intel/igb/igb_xsk.c +++ b/drivers/net/ethernet/intel/igb/igb_xsk.c @@ -66,7 +66,10 @@ static void igb_txrx_ring_enable(struct igb_adapter *adapter, u16 qid) * at least 1 descriptor unused to make sure * next_to_use != next_to_clean */ - igb_alloc_rx_buffers(rx_ring, igb_desc_unused(rx_ring)); + if (rx_ring->xsk_pool) + igb_alloc_rx_buffers_zc(rx_ring, igb_desc_unused(rx_ring)); + else + igb_alloc_rx_buffers(rx_ring, igb_desc_unused(rx_ring)); /* Rx/Tx share the same napi context. */ napi_enable(&rx_ring->q_vector->napi); @@ -172,6 +175,262 @@ int igb_xsk_pool_setup(struct igb_adapter *adapter, igb_xsk_pool_disable(adapter, qid); } +static u16 igb_fill_rx_descs(struct xsk_buff_pool *pool, struct xdp_buff **xdp, + union e1000_adv_rx_desc *rx_desc, u16 count) +{ + dma_addr_t dma; + u16 buffs; + int i; + + /* nothing to do */ + if (!count) + return 0; + + buffs = xsk_buff_alloc_batch(pool, xdp, count); + for (i = 0; i < buffs; i++) { + dma = xsk_buff_xdp_get_dma(*xdp); + rx_desc->read.pkt_addr = cpu_to_le64(dma); + rx_desc->wb.upper.length = 0; + + rx_desc++; + xdp++; + } + + return buffs; +} + +bool igb_alloc_rx_buffers_zc(struct igb_ring *rx_ring, u16 count) +{ + u32 nb_buffs_extra = 0, nb_buffs = 0; + union e1000_adv_rx_desc *rx_desc; + u16 ntu = rx_ring->next_to_use; + u16 total_count = count; + struct xdp_buff **xdp; + + rx_desc = IGB_RX_DESC(rx_ring, ntu); + xdp = &rx_ring->rx_buffer_info_zc[ntu]; + + if (ntu + count >= rx_ring->count) { + nb_buffs_extra = igb_fill_rx_descs(rx_ring->xsk_pool, xdp, + rx_desc, + rx_ring->count - ntu); + if (nb_buffs_extra != rx_ring->count - ntu) { + ntu += nb_buffs_extra; + goto exit; + } + rx_desc = IGB_RX_DESC(rx_ring, 0); + xdp = rx_ring->rx_buffer_info_zc; + ntu = 0; + count -= nb_buffs_extra; + } + + nb_buffs = igb_fill_rx_descs(rx_ring->xsk_pool, xdp, rx_desc, count); + ntu += nb_buffs; + if (ntu == rx_ring->count) + ntu = 0; + + /* clear the length for the next_to_use descriptor */ + rx_desc = IGB_RX_DESC(rx_ring, ntu); + rx_desc->wb.upper.length = 0; + +exit: + if (rx_ring->next_to_use != ntu) { + rx_ring->next_to_use = ntu; + + /* Force memory writes to complete before letting h/w + * know there are new descriptors to fetch. (Only + * applicable for weak-ordered memory model archs, + * such as IA-64). + */ + wmb(); + writel(ntu, rx_ring->tail); + } + + return total_count == (nb_buffs + nb_buffs_extra); +} + +void igb_clean_rx_ring_zc(struct igb_ring *rx_ring) +{ + u16 ntc = rx_ring->next_to_clean; + u16 ntu = rx_ring->next_to_use; + + while (ntc != ntu) { + struct xdp_buff *xdp = rx_ring->rx_buffer_info_zc[ntc]; + + xsk_buff_free(xdp); + ntc++; + if (ntc >= rx_ring->count) + ntc = 0; + } +} + +static struct sk_buff *igb_construct_skb_zc(struct igb_ring *rx_ring, + struct xdp_buff *xdp, + ktime_t timestamp) +{ + unsigned int totalsize = xdp->data_end - xdp->data_meta; + unsigned int metasize = xdp->data - xdp->data_meta; + struct sk_buff *skb; + + net_prefetch(xdp->data_meta); + + /* allocate a skb to store the frags */ + skb = napi_alloc_skb(&rx_ring->q_vector->napi, totalsize); + if (unlikely(!skb)) + return NULL; + + if (timestamp) + skb_hwtstamps(skb)->hwtstamp = timestamp; + + memcpy(__skb_put(skb, totalsize), xdp->data_meta, + ALIGN(totalsize, sizeof(long))); + + if (metasize) { + skb_metadata_set(skb, metasize); + __skb_pull(skb, metasize); + } + + return skb; +} + +static void igb_update_ntc(struct igb_ring *rx_ring) +{ + u32 ntc = rx_ring->next_to_clean + 1; + + /* fetch, update, and store next to clean */ + ntc = (ntc < rx_ring->count) ? ntc : 0; + rx_ring->next_to_clean = ntc; + + prefetch(IGB_RX_DESC(rx_ring, ntc)); +} + +int igb_clean_rx_irq_zc(struct igb_q_vector *q_vector, const int budget) +{ + struct igb_adapter *adapter = q_vector->adapter; + unsigned int total_bytes = 0, total_packets = 0; + struct igb_ring *rx_ring = q_vector->rx.ring; + int cpu = smp_processor_id(); + unsigned int xdp_xmit = 0; + struct netdev_queue *nq; + bool failure = false; + u16 entries_to_alloc; + struct sk_buff *skb; + + while (likely(total_packets < budget)) { + union e1000_adv_rx_desc *rx_desc; + struct xdp_buff *xdp; + ktime_t timestamp = 0; + unsigned int size; + + rx_desc = IGB_RX_DESC(rx_ring, rx_ring->next_to_clean); + size = le16_to_cpu(rx_desc->wb.upper.length); + if (!size) + break; + + /* This memory barrier is needed to keep us from reading + * any other fields out of the rx_desc until we know the + * descriptor has been written back + */ + dma_rmb(); + + xdp = rx_ring->rx_buffer_info_zc[rx_ring->next_to_clean]; + xsk_buff_set_size(xdp, size); + xsk_buff_dma_sync_for_cpu(xdp); + + /* pull rx packet timestamp if available and valid */ + if (igb_test_staterr(rx_desc, E1000_RXDADV_STAT_TSIP)) { + int ts_hdr_len; + + ts_hdr_len = igb_ptp_rx_pktstamp(rx_ring->q_vector, + xdp->data, + ×tamp); + + xdp->data += ts_hdr_len; + xdp->data_meta += ts_hdr_len; + size -= ts_hdr_len; + } + + skb = igb_run_xdp(adapter, rx_ring, xdp); + + if (IS_ERR(skb)) { + unsigned int xdp_res = -PTR_ERR(skb); + + if (likely(xdp_res & (IGB_XDP_TX | IGB_XDP_REDIR))) { + xdp_xmit |= xdp_res; + } else if (xdp_res == IGB_XDP_EXIT) { + failure = true; + break; + } else if (xdp_res == IGB_XDP_CONSUMED) { + xsk_buff_free(xdp); + } + + total_packets++; + total_bytes += size; + + igb_update_ntc(rx_ring); + continue; + } + + skb = igb_construct_skb_zc(rx_ring, xdp, timestamp); + + /* exit if we failed to retrieve a buffer */ + if (!skb) { + rx_ring->rx_stats.alloc_failed++; + break; + } + + xsk_buff_free(xdp); + igb_update_ntc(rx_ring); + + if (eth_skb_pad(skb)) + continue; + + /* probably a little skewed due to removing CRC */ + total_bytes += skb->len; + + /* populate checksum, timestamp, VLAN, and protocol */ + igb_process_skb_fields(rx_ring, rx_desc, skb); + + napi_gro_receive(&q_vector->napi, skb); + + /* update budget accounting */ + total_packets++; + } + + if (xdp_xmit & IGB_XDP_REDIR) + xdp_do_flush(); + + if (xdp_xmit & IGB_XDP_TX) { + struct igb_ring *tx_ring = igb_xdp_tx_queue_mapping(adapter); + + nq = txring_txq(tx_ring); + __netif_tx_lock(nq, cpu); + igb_xdp_ring_update_tail(tx_ring); + __netif_tx_unlock(nq); + } + + u64_stats_update_begin(&rx_ring->rx_syncp); + rx_ring->rx_stats.packets += total_packets; + rx_ring->rx_stats.bytes += total_bytes; + u64_stats_update_end(&rx_ring->rx_syncp); + q_vector->rx.total_packets += total_packets; + q_vector->rx.total_bytes += total_bytes; + + entries_to_alloc = igb_desc_unused(rx_ring); + if (entries_to_alloc >= IGB_RX_BUFFER_WRITE) + failure |= !igb_alloc_rx_buffers_zc(rx_ring, entries_to_alloc); + + if (xsk_uses_need_wakeup(rx_ring->xsk_pool)) { + if (failure || rx_ring->next_to_clean == rx_ring->next_to_use) + xsk_set_rx_need_wakeup(rx_ring->xsk_pool); + else + xsk_clear_rx_need_wakeup(rx_ring->xsk_pool); + + return (int)total_packets; + } + return failure ? budget : (int)total_packets; +} + int igb_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags) { struct igb_adapter *adapter = netdev_priv(dev); From patchwork Fri Jul 12 08:55:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kurt Kanzenbach X-Patchwork-Id: 1959719 X-Patchwork-Delegate: anthony.l.nguyen@intel.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=osuosl.org header.i=@osuosl.org header.a=rsa-sha256 header.s=default header.b=MlH2HwGx; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=osuosl.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WL56d3QKzz1xqj for ; Fri, 12 Jul 2024 18:56:09 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id BCE7D41441; Fri, 12 Jul 2024 08:56:07 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id ljzp48xTXV2H; Fri, 12 Jul 2024 08:56:06 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.166.34; helo=ash.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 5A33441436 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osuosl.org; s=default; t=1720774566; bh=QoZTiw4RzynW2aXOJczHFuJNP1z24G6tT8kwFl99k9U=; h=From:Date:References:In-Reply-To:To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: Cc:From; b=MlH2HwGxzb8PZUUnJEIvnpD0Cd4PxZ5DRuAwUZ6KLmhKMGuD50+I2nqSE3ZBzTH3j bzdPFj7/5eMKn1yqlKwt+OJv+P6dH/2k7Vq1QzVPKjWwRsE9YV58L4hcXIOoIqM39O jdP7U52iCgi6uZx70ngHoYwb5oI43fg4BoSJVsq/bZX9KDP7EiGsJuxZQBrIQ+iEzv /4qyT4ydPvqow8FsnqNNSqcLd43hhSMROHoZ8lZ8MpzpMjD9mtQ067TixNy7gsfxgD AjnFcSsyW3QbLqflIYckrHJFqGkDX54DgmVZV6yL3flaJEgEhFYadQijXoQkX80EA3 x2Wuq7F1wZkWw== Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp2.osuosl.org (Postfix) with ESMTP id 5A33441436; Fri, 12 Jul 2024 08:56:06 +0000 (UTC) X-Original-To: intel-wired-lan@lists.osuosl.org Delivered-To: intel-wired-lan@lists.osuosl.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 30F1C1BF584 for ; Fri, 12 Jul 2024 08:55:59 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 1EF1560B0E for ; Fri, 12 Jul 2024 08:55:59 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id fX4HZJiO9bmE for ; Fri, 12 Jul 2024 08:55:58 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a0a:51c0:0:12e:550::1; helo=galois.linutronix.de; envelope-from=kurt@linutronix.de; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org BA45160AE3 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org BA45160AE3 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by smtp3.osuosl.org (Postfix) with ESMTPS id BA45160AE3 for ; Fri, 12 Jul 2024 08:55:57 +0000 (UTC) From: Kurt Kanzenbach Date: Fri, 12 Jul 2024 10:55:32 +0200 MIME-Version: 1.0 Message-Id: <20240711-b4-igb_zero_copy-v5-4-f3f455113b11@linutronix.de> References: <20240711-b4-igb_zero_copy-v5-0-f3f455113b11@linutronix.de> In-Reply-To: <20240711-b4-igb_zero_copy-v5-0-f3f455113b11@linutronix.de> To: Tony Nguyen , Przemek Kitszel X-Developer-Signature: v=1; a=openpgp-sha256; l=8780; i=kurt@linutronix.de; h=from:subject:message-id; bh=ceURveCToMKlf3R2KjKtvvECaRqa6XN5wCqFUVUmODc=; b=owEBbQKS/ZANAwAKAcGT0fKqRnOCAcsmYgBmkO+XmWfgAHNA0vXxIyoOT+NnAsUarZYwm8Pq4 pmRyZh/6sqJAjMEAAEKAB0WIQS8ub+yyMN909/bWZLBk9HyqkZzggUCZpDvlwAKCRDBk9HyqkZz gobsD/49YiLFMiX4NJVSW0Rije8NP/1BXwm3i8o0cziR6yx5RZeAqMNSz202bbHPGKqtS/aJmVh N9GtVmANvM4dthYCQ+MMVUNjAwstI1a3Lup6sOQ5qzlKMDKzlPVEM1LhXEMiv8SutNr26UdkgJS kI1v1x/nuDQh8aAj2nbx6YH9Z+S232M4MJsj9vTczMwFbUpRU0hnV2rPHiNdejr8K2DMc8LA1qC 3ycflQ/n+7wFyfPlocuhsv9CimFmBFxvAKgDl7fY2he4lAqMUGMupM1FnErxcID39TqoYm1dTbT scjltSwapD7JX8+7NQp3ueg4splVVCyvsjvMqWzbJAk45dm9Z1U+HP9DuksUnrEk7zvOAHOAdre T1bNYbWARP/ENYls3pd96//z4MmpIIoNHWboTwuNvHFvDB6fAAfQ/fWectwc6iSwgVeG96aAURO UNwMoEPTcmHPh9SuyyuTs9/E/K4Z0M9A0gCnupVkJ4tQRR36zIThEzVGjUnbm4NLVpDBl+fc7TI lpHoXA2BYZp7UX0IiY7Xmm5AturjD9bt4kx4jnqxndBSjpfCjV6tz+TbDRI67pNG5R3MGb0zOHs ZHLsJuqT6x6u0U2m30mynF0aqEgsv5vUe91eN8hLn2lAxli3sExmNVkMtdpZfylmHV4RDeXCnlN b+yngWlVdB/6u3A== X-Developer-Key: i=kurt@linutronix.de; a=openpgp; fpr=BCB9BFB2C8C37DD3DFDB5992C193D1F2AA467382 X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1720774555; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QoZTiw4RzynW2aXOJczHFuJNP1z24G6tT8kwFl99k9U=; b=xJ1yaWZDejz5cl7akHxay/euK9vN7nxlCaKP7Nl9Ghys+HRAIC2pW4rpMIapp8d8XFNVKW k7ty4nzyZxocjEdBs1I26qqvFMxzV0r4Qv/msMRs6fSt3xvp234tokvyAMORYux2fm9Wd5 O/+cAkt2Dh4US4emoSUEI4ObPIQ7mtz9V+9uqRebfnM67slsFuKRfNWbeEqNH91f1cquIS FG/Pngk7PD/E75hcDDikyLaOKuBjoFYaaSgCi9g1PFJUSewTYXGctEWSIc0z6nDD152C4F dEo8LaUYVqP+M8+JlLrxmx1MH4wqt31+JpOZoU6Fnf+FVDq3C9d3SpLtSG0ELQ== X-Mailman-Original-DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1720774555; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QoZTiw4RzynW2aXOJczHFuJNP1z24G6tT8kwFl99k9U=; b=yAlQ9ecET4qayZQ0ukthAoNgCrzcD6eFo6utmJEkA0pwwt8pyW+a7/Pi7+6dqK+u27Bip7 2HiBNvSaksgD7HDA== X-Mailman-Original-Authentication-Results: smtp3.osuosl.org; dmarc=pass (p=none dis=none) header.from=linutronix.de X-Mailman-Original-Authentication-Results: smtp3.osuosl.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.a=rsa-sha256 header.s=2020 header.b=xJ1yaWZD; dkim=pass header.d=linutronix.de header.i=@linutronix.de header.a=ed25519-sha256 header.s=2020e header.b=yAlQ9ecE Subject: [Intel-wired-lan] [PATCH iwl-next v5 4/4] igb: add AF_XDP zero-copy Tx support X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jesper Dangaard Brouer , Daniel Borkmann , Sriram Yagnaraman , Richard Cochran , Kurt Kanzenbach , John Fastabend , Alexei Starovoitov , Benjamin Steinke , Eric Dumazet , Sriram Yagnaraman , intel-wired-lan@lists.osuosl.org, netdev@vger.kernel.org, Jakub Kicinski , bpf@vger.kernel.org, Paolo Abeni , "David S. Miller" , Sebastian Andrzej Siewior Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" From: Sriram Yagnaraman Add support for AF_XDP zero-copy transmit path. A new TX buffer type IGB_TYPE_XSK is introduced to indicate that the Tx frame was allocated from the xsk buff pool, so igb_clean_tx_ring and igb_clean_tx_irq can clean the buffers correctly based on type. igb_xmit_zc performs the actual packet transmit when AF_XDP zero-copy is enabled. We share the TX ring between slow path, XDP and AF_XDP zero-copy, so we use the netdev queue lock to ensure mutual exclusion. Signed-off-by: Sriram Yagnaraman [Kurt: Set olinfo_status in igb_xmit_zc() so that frames are transmitted] Signed-off-by: Kurt Kanzenbach Tested-by: Chandan Kumar Rout (A Contingent Worker at Intel) --- drivers/net/ethernet/intel/igb/igb.h | 2 ++ drivers/net/ethernet/intel/igb/igb_main.c | 56 ++++++++++++++++++++++++++----- drivers/net/ethernet/intel/igb/igb_xsk.c | 53 +++++++++++++++++++++++++++++ 3 files changed, 102 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h index 4983a6ec718e..9ee18ac1ba47 100644 --- a/drivers/net/ethernet/intel/igb/igb.h +++ b/drivers/net/ethernet/intel/igb/igb.h @@ -257,6 +257,7 @@ enum igb_tx_flags { enum igb_tx_buf_type { IGB_TYPE_SKB = 0, IGB_TYPE_XDP, + IGB_TYPE_XSK }; /* wrapper around a pointer to a socket buffer, @@ -836,6 +837,7 @@ int igb_xsk_pool_setup(struct igb_adapter *adapter, bool igb_alloc_rx_buffers_zc(struct igb_ring *rx_ring, u16 count); void igb_clean_rx_ring_zc(struct igb_ring *rx_ring); int igb_clean_rx_irq_zc(struct igb_q_vector *q_vector, const int budget); +bool igb_xmit_zc(struct igb_ring *tx_ring); int igb_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags); #endif /* _IGB_H_ */ diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index 8d65c9eca102..4069732e2f4c 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -2997,6 +2997,9 @@ static int igb_xdp_xmit(struct net_device *dev, int n, if (unlikely(!tx_ring)) return -ENXIO; + if (unlikely(test_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags))) + return -ENXIO; + nq = txring_txq(tx_ring); __netif_tx_lock(nq, cpu); @@ -4918,15 +4921,20 @@ void igb_clean_tx_ring(struct igb_ring *tx_ring) { u16 i = tx_ring->next_to_clean; struct igb_tx_buffer *tx_buffer = &tx_ring->tx_buffer_info[i]; + u32 xsk_frames = 0; while (i != tx_ring->next_to_use) { union e1000_adv_tx_desc *eop_desc, *tx_desc; /* Free all the Tx ring sk_buffs or xdp frames */ - if (tx_buffer->type == IGB_TYPE_SKB) + if (tx_buffer->type == IGB_TYPE_SKB) { dev_kfree_skb_any(tx_buffer->skb); - else + } else if (tx_buffer->type == IGB_TYPE_XDP) { xdp_return_frame(tx_buffer->xdpf); + } else if (tx_buffer->type == IGB_TYPE_XSK) { + xsk_frames++; + goto skip_for_xsk; + } /* unmap skb header data */ dma_unmap_single(tx_ring->dev, @@ -4957,6 +4965,7 @@ void igb_clean_tx_ring(struct igb_ring *tx_ring) DMA_TO_DEVICE); } +skip_for_xsk: tx_buffer->next_to_watch = NULL; /* move us one more past the eop_desc for start of next pkt */ @@ -4971,6 +4980,9 @@ void igb_clean_tx_ring(struct igb_ring *tx_ring) /* reset BQL for queue */ netdev_tx_reset_queue(txring_txq(tx_ring)); + if (tx_ring->xsk_pool && xsk_frames) + xsk_tx_completed(tx_ring->xsk_pool, xsk_frames); + /* reset next_to_use and next_to_clean */ tx_ring->next_to_use = 0; tx_ring->next_to_clean = 0; @@ -6504,6 +6516,9 @@ netdev_tx_t igb_xmit_frame_ring(struct sk_buff *skb, return NETDEV_TX_BUSY; } + if (unlikely(test_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags))) + return NETDEV_TX_BUSY; + /* record the location of the first descriptor for this packet */ first = &tx_ring->tx_buffer_info[tx_ring->next_to_use]; first->type = IGB_TYPE_SKB; @@ -8264,13 +8279,17 @@ static int igb_poll(struct napi_struct *napi, int budget) **/ static bool igb_clean_tx_irq(struct igb_q_vector *q_vector, int napi_budget) { - struct igb_adapter *adapter = q_vector->adapter; - struct igb_ring *tx_ring = q_vector->tx.ring; - struct igb_tx_buffer *tx_buffer; - union e1000_adv_tx_desc *tx_desc; unsigned int total_bytes = 0, total_packets = 0; + struct igb_adapter *adapter = q_vector->adapter; unsigned int budget = q_vector->tx.work_limit; + struct igb_ring *tx_ring = q_vector->tx.ring; unsigned int i = tx_ring->next_to_clean; + union e1000_adv_tx_desc *tx_desc; + struct igb_tx_buffer *tx_buffer; + int cpu = smp_processor_id(); + bool xsk_xmit_done = true; + struct netdev_queue *nq; + u32 xsk_frames = 0; if (test_bit(__IGB_DOWN, &adapter->state)) return true; @@ -8301,10 +8320,14 @@ static bool igb_clean_tx_irq(struct igb_q_vector *q_vector, int napi_budget) total_packets += tx_buffer->gso_segs; /* free the skb */ - if (tx_buffer->type == IGB_TYPE_SKB) + if (tx_buffer->type == IGB_TYPE_SKB) { napi_consume_skb(tx_buffer->skb, napi_budget); - else + } else if (tx_buffer->type == IGB_TYPE_XDP) { xdp_return_frame(tx_buffer->xdpf); + } else if (tx_buffer->type == IGB_TYPE_XSK) { + xsk_frames++; + goto skip_for_xsk; + } /* unmap skb header data */ dma_unmap_single(tx_ring->dev, @@ -8336,6 +8359,7 @@ static bool igb_clean_tx_irq(struct igb_q_vector *q_vector, int napi_budget) } } +skip_for_xsk: /* move us one more past the eop_desc for start of next pkt */ tx_buffer++; tx_desc++; @@ -8364,6 +8388,20 @@ static bool igb_clean_tx_irq(struct igb_q_vector *q_vector, int napi_budget) q_vector->tx.total_bytes += total_bytes; q_vector->tx.total_packets += total_packets; + if (tx_ring->xsk_pool) { + if (xsk_frames) + xsk_tx_completed(tx_ring->xsk_pool, xsk_frames); + if (xsk_uses_need_wakeup(tx_ring->xsk_pool)) + xsk_set_tx_need_wakeup(tx_ring->xsk_pool); + + nq = txring_txq(tx_ring); + __netif_tx_lock(nq, cpu); + /* Avoid transmit queue timeout since we share it with the slow path */ + txq_trans_cond_update(nq); + xsk_xmit_done = igb_xmit_zc(tx_ring); + __netif_tx_unlock(nq); + } + if (test_bit(IGB_RING_FLAG_TX_DETECT_HANG, &tx_ring->flags)) { struct e1000_hw *hw = &adapter->hw; @@ -8426,7 +8464,7 @@ static bool igb_clean_tx_irq(struct igb_q_vector *q_vector, int napi_budget) } } - return !!budget; + return !!budget && xsk_xmit_done; } /** diff --git a/drivers/net/ethernet/intel/igb/igb_xsk.c b/drivers/net/ethernet/intel/igb/igb_xsk.c index 66cdc30e9b6e..4e530e1eb3c0 100644 --- a/drivers/net/ethernet/intel/igb/igb_xsk.c +++ b/drivers/net/ethernet/intel/igb/igb_xsk.c @@ -431,6 +431,59 @@ int igb_clean_rx_irq_zc(struct igb_q_vector *q_vector, const int budget) return failure ? budget : (int)total_packets; } +bool igb_xmit_zc(struct igb_ring *tx_ring) +{ + unsigned int budget = igb_desc_unused(tx_ring); + struct xsk_buff_pool *pool = tx_ring->xsk_pool; + u32 cmd_type, olinfo_status, nb_pkts, i = 0; + struct xdp_desc *descs = pool->tx_descs; + union e1000_adv_tx_desc *tx_desc = NULL; + struct igb_tx_buffer *tx_buffer_info; + unsigned int total_bytes = 0; + dma_addr_t dma; + + nb_pkts = xsk_tx_peek_release_desc_batch(pool, budget); + if (!nb_pkts) + return true; + + while (nb_pkts-- > 0) { + dma = xsk_buff_raw_get_dma(pool, descs[i].addr); + xsk_buff_raw_dma_sync_for_device(pool, dma, descs[i].len); + + tx_buffer_info = &tx_ring->tx_buffer_info[tx_ring->next_to_use]; + tx_buffer_info->bytecount = descs[i].len; + tx_buffer_info->type = IGB_TYPE_XSK; + tx_buffer_info->xdpf = NULL; + tx_buffer_info->gso_segs = 1; + tx_buffer_info->time_stamp = jiffies; + + tx_desc = IGB_TX_DESC(tx_ring, tx_ring->next_to_use); + tx_desc->read.buffer_addr = cpu_to_le64(dma); + + /* put descriptor type bits */ + cmd_type = E1000_ADVTXD_DTYP_DATA | E1000_ADVTXD_DCMD_DEXT | + E1000_ADVTXD_DCMD_IFCS; + olinfo_status = descs[i].len << E1000_ADVTXD_PAYLEN_SHIFT; + + cmd_type |= descs[i].len | IGB_TXD_DCMD; + tx_desc->read.cmd_type_len = cpu_to_le32(cmd_type); + tx_desc->read.olinfo_status = cpu_to_le32(olinfo_status); + + total_bytes += descs[i].len; + + i++; + tx_ring->next_to_use++; + tx_buffer_info->next_to_watch = tx_desc; + if (tx_ring->next_to_use == tx_ring->count) + tx_ring->next_to_use = 0; + } + + netdev_tx_sent_queue(txring_txq(tx_ring), total_bytes); + igb_xdp_ring_update_tail(tx_ring); + + return nb_pkts < budget; +} + int igb_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags) { struct igb_adapter *adapter = netdev_priv(dev);