From patchwork Thu Feb 27 17:30:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: William Tu X-Patchwork-Id: 1245969 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=iPhLKd+w; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48T07R0fC1z9sP7 for ; Fri, 28 Feb 2020 04:30:42 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 67EE787ED9; Thu, 27 Feb 2020 17:30:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RUcSP6NLi8db; Thu, 27 Feb 2020 17:30:39 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by hemlock.osuosl.org (Postfix) with ESMTP id 097D487E99; Thu, 27 Feb 2020 17:30:39 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id DECCEC1D80; Thu, 27 Feb 2020 17:30:38 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4272CC0177 for ; Thu, 27 Feb 2020 17:30:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 2432787E99 for ; Thu, 27 Feb 2020 17:30:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HAg6MyTw+2Rx for ; Thu, 27 Feb 2020 17:30:36 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) by hemlock.osuosl.org (Postfix) with ESMTPS id 5B29D87E7A for ; Thu, 27 Feb 2020 17:30:36 +0000 (UTC) Received: by mail-pl1-f195.google.com with SMTP id y1so64055plp.7 for ; Thu, 27 Feb 2020 09:30:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=LRQnXckoib1bBdFkSVHx2z6+r2v5tA67M6jmA2g+3Bg=; b=iPhLKd+wG6WZ4glAxIKE8vzNnxAjxJYfkAkd75fRHRFi6XNPrYXMAjEsWrhc+M1cXF fM/ZAcVcYqn1fFo04CyHQmIpl8TuNlDVDmD4z/nNYtjUynbvaPgAZMfSZMSNIa1Gs6T+ sKAGpytCUaRJmQkLIPLDofy7l9oJYEnOtRSZmJyEplORzsUZ1tiq63o8X0RxCsiaAstz sApZYNtkj83HSOjS13QHMzyrVhc9zOccMOU3Dt6biBF9tsr8tiod+MidkRzFwTMuUhwv +HRTPHLVC3GNH+GgBZLB4yt0uWHdzRzei1xV5E4Yn4Pd7cs3ofGpLVj/0/Qi5W0BVX1G LZgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=LRQnXckoib1bBdFkSVHx2z6+r2v5tA67M6jmA2g+3Bg=; b=hL3HyIQhPo0++E16QKJTnHo0U2x28nowUxLxyfiJ95Fk9VwkhUBtFi8LTMowCkBy1y ia1ZgDUQfYRYI5vBXbGEPNhJx0v3tbBOTP7VARnI0u2VktuqCMn+ZSVeeqvd3GAjmcLz O8cf+4aeRcIQy2ipyKtjN6Bzci0XOx5UiFKGP/s7fmUjeQ+VFcp4SpyGit9Bq52IVya+ 5iE7XlzJaoLZMRFTrKYTDr/HmhFeDvuJBkmA7Z0iZmpFRSyBuQ9x8prdhTYaTZ1L1vng Kwp9MofYuELzrm+JwbwNZH1Z1f15YD9JxQkilUp8u/vx0mWovS0LS/SLAktxufR53tEP vgiw== X-Gm-Message-State: APjAAAU1akmg5klBfV9gF+2hoCrgZva3+vTdbgAh7E/GyxmW2q+tZ0b0 LKiSZE46YjN5sJcvE5qUWo6ZRJty X-Google-Smtp-Source: APXvYqzauH3+lydYjI7UhTTIGbE9bwdGacEXR0XanHx4xMI2OVgPwxR1d7Yqv6+O4AN7KPW9YAhLzQ== X-Received: by 2002:a17:902:be0e:: with SMTP id r14mr748152pls.33.1582824635127; Thu, 27 Feb 2020 09:30:35 -0800 (PST) Received: from sc9-mailhost2.vmware.com ([96.76.194.129]) by smtp.gmail.com with ESMTPSA id z27sm8192823pfj.107.2020.02.27.09.30.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 27 Feb 2020 09:30:33 -0800 (PST) From: William Tu To: dev@openvswitch.org Date: Thu, 27 Feb 2020 09:30:04 -0800 Message-Id: <1582824604-113843-1-git-send-email-u9012063@gmail.com> X-Mailer: git-send-email 2.7.4 Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCHv2] netdev-afxdp: Add interrupt mode netdev class. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The patch adds a new netdev class 'afxdp-nonpmd' to enable afxdp interrupt mode. This is similar to 'type=afxdp', except that the is_pmd field is set to false. As a result, the packet processing is handled by main thread, not pmd thread. This avoids burning the CPU to always 100% when there is no traffic. Tested-at: https://travis-ci.org/williamtu/ovs-travis/builds/655885506 Signed-off-by: William Tu --- NEWS | 3 +++ lib/netdev-afxdp.c | 20 +++++++++++++++++++- lib/netdev-linux.c | 20 ++++++++++++++++++++ lib/netdev-provider.h | 1 + lib/netdev.c | 1 + tests/system-afxdp.at | 23 +++++++++++++++++++++++ 6 files changed, 67 insertions(+), 1 deletion(-) diff --git a/NEWS b/NEWS index f62ef1f47ea8..594c55dc11d6 100644 --- a/NEWS +++ b/NEWS @@ -4,6 +4,9 @@ Post-v2.13.0 - OpenFlow: * The OpenFlow ofp_desc/serial_num may now be configured by setting the value of other-config:dp-sn in the Bridge table. + - AF_XDP: + * New netdev class 'afxdp-nonpmd' for netdev-afxdp to save CPU cycles + by enabling interrupt mode. v2.13.0 - 14 Feb 2020 diff --git a/lib/netdev-afxdp.c b/lib/netdev-afxdp.c index 482400d8d135..cd2c7c381139 100644 --- a/lib/netdev-afxdp.c +++ b/lib/netdev-afxdp.c @@ -169,6 +169,12 @@ struct netdev_afxdp_tx_lock { ); }; +static int nonpmd_cnt; /* Number of afxdp netdevs in non-pmd mode. */ +static bool +netdev_is_afxdp_nonpmd(struct netdev *netdev) { + return netdev_get_class(netdev) == &netdev_afxdp_nonpmd_class; +} + #ifdef HAVE_XDP_NEED_WAKEUP static inline void xsk_rx_wakeup_if_needed(struct xsk_umem_info *umem, @@ -1115,7 +1121,10 @@ netdev_afxdp_batch_send(struct netdev *netdev, int qid, struct netdev_linux *dev; int ret; - if (concurrent_txq) { + /* Lock is required when mixing afxdp pmd and nonpmd mode. + * ex: one device is created 'afxdp', the other is 'afxdp-nonpmd'. + */ + if (concurrent_txq || (nonpmd_cnt != 0)) { dev = netdev_linux_cast(netdev); qid = qid % netdev_n_txq(netdev); @@ -1159,6 +1168,7 @@ libbpf_print(enum libbpf_print_level level, int netdev_afxdp_init(void) { libbpf_set_print(libbpf_print); + nonpmd_cnt = 0; return 0; } @@ -1188,6 +1198,10 @@ netdev_afxdp_construct(struct netdev *netdev) dev->tx_locks = NULL; netdev_request_reconfigure(netdev); + + if (netdev_is_afxdp_nonpmd(netdev)) { + nonpmd_cnt++; + } return 0; } @@ -1208,6 +1222,10 @@ netdev_afxdp_destruct(struct netdev *netdev) xsk_destroy_all(netdev); ovs_mutex_destroy(&dev->mutex); + + if (netdev_is_afxdp_nonpmd(netdev)) { + nonpmd_cnt--; + } } int diff --git a/lib/netdev-linux.c b/lib/netdev-linux.c index c6f3d27409b6..75b3b55d3cad 100644 --- a/lib/netdev-linux.c +++ b/lib/netdev-linux.c @@ -3605,6 +3605,26 @@ const struct netdev_class netdev_afxdp_class = { .rxq_destruct = netdev_afxdp_rxq_destruct, .rxq_recv = netdev_afxdp_rxq_recv, }; + +const struct netdev_class netdev_afxdp_nonpmd_class = { + NETDEV_LINUX_CLASS_COMMON, + .type = "afxdp-nonpmd", + .is_pmd = false, + .init = netdev_afxdp_init, + .construct = netdev_afxdp_construct, + .destruct = netdev_afxdp_destruct, + .get_stats = netdev_afxdp_get_stats, + .get_custom_stats = netdev_afxdp_get_custom_stats, + .get_status = netdev_linux_get_status, + .set_config = netdev_afxdp_set_config, + .get_config = netdev_afxdp_get_config, + .reconfigure = netdev_afxdp_reconfigure, + .get_numa_id = netdev_linux_get_numa_id, + .send = netdev_afxdp_batch_send, + .rxq_construct = netdev_afxdp_rxq_construct, + .rxq_destruct = netdev_afxdp_rxq_destruct, + .rxq_recv = netdev_afxdp_rxq_recv, +}; #endif diff --git a/lib/netdev-provider.h b/lib/netdev-provider.h index 22f4cde3337a..06c1d98a0b07 100644 --- a/lib/netdev-provider.h +++ b/lib/netdev-provider.h @@ -848,6 +848,7 @@ extern const struct netdev_class netdev_tap_class; #ifdef HAVE_AF_XDP extern const struct netdev_class netdev_afxdp_class; +extern const struct netdev_class netdev_afxdp_nonpmd_class; #endif #ifdef __cplusplus } diff --git a/lib/netdev.c b/lib/netdev.c index f95b19af4da0..6d9723ebc3c1 100644 --- a/lib/netdev.c +++ b/lib/netdev.c @@ -154,6 +154,7 @@ netdev_initialize(void) netdev_register_flow_api_provider(&netdev_offload_tc); #ifdef HAVE_AF_XDP netdev_register_provider(&netdev_afxdp_class); + netdev_register_provider(&netdev_afxdp_nonpmd_class); #endif #endif #if defined(__FreeBSD__) || defined(__NetBSD__) diff --git a/tests/system-afxdp.at b/tests/system-afxdp.at index e4451624f882..0d09906fb6c8 100644 --- a/tests/system-afxdp.at +++ b/tests/system-afxdp.at @@ -22,3 +22,26 @@ AT_CHECK([grep "ovs-p0: could not set configuration" ovs-vswitchd.log | wc -l], OVS_TRAFFIC_VSWITCHD_STOP(["/ovs-p0: Too big 'n_rxq'/d /ovs-p0: could not set configuration/d"]) AT_CLEANUP + + +AT_SETUP([AF_XDP - ping between pmd and non-pmd ports]) +AT_KEYWORDS([afxdp nonpmd]) +OVS_TRAFFIC_VSWITCHD_START() + +AT_CHECK([ovs-ofctl add-flow br0 "actions=normal"]) + +ADD_NAMESPACES(at_ns0, at_ns1) +ADD_VETH(p0, at_ns0, br0, "10.1.1.1/24") +ADD_VETH(p1, at_ns1, br0, "10.1.1.2/24") + +AT_CHECK([ovs-vsctl del-port ovs-p0]) +AT_CHECK([ovs-vsctl add-port br0 ovs-p0 -- \ + set interface ovs-p0 type=afxdp-nonpmd options:n_rxq=1], + [0], [], [stderr]) + +NS_CHECK_EXEC([at_ns0], [ping -q -c 3 -i 0.3 -w 2 10.1.1.2 | FORMAT_PING], [0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + +OVS_TRAFFIC_VSWITCHD_STOP +AT_CLEANUP