From patchwork Fri Feb 21 20:24:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: William Tu X-Patchwork-Id: 1242329 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=fsTUHdRf; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48PNHl4WM0z9sNg for ; Sat, 22 Feb 2020 07:25:23 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 0EC0C86637; Fri, 21 Feb 2020 20:25:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id L7v0R9_VZZpC; Fri, 21 Feb 2020 20:25:20 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by fraxinus.osuosl.org (Postfix) with ESMTP id 1E423864B3; Fri, 21 Feb 2020 20:25:20 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id F35A2C07FE; Fri, 21 Feb 2020 20:25:19 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 7AE9EC013E for ; Fri, 21 Feb 2020 20:25:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 64BDC87639 for ; Fri, 21 Feb 2020 20:25:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vKJhDK-Xzsi0 for ; Fri, 21 Feb 2020 20:25:17 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by whitealder.osuosl.org (Postfix) with ESMTPS id 95CB38761D for ; Fri, 21 Feb 2020 20:25:17 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id y30so1522790pga.13 for ; Fri, 21 Feb 2020 12:25:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=+iyuanaaKf/C0aGVTsW4R9+UMvTbKMgGyilj78Cwu/8=; b=fsTUHdRfStkNaAyGx4NO0w/4PYhdCo4VsodxoZHvwhJgReP2xlzvCJ3m3N8ygaJney gyiZrcrpe/F0YLp1/WyavOX2zkVdjFgJwjkdp1Nc4qABPRlqtJqXbPqNxf9B/zr2FZrl LIz3nu0yRg40aDDAtO+IV6uGzzt9luxiMoAcpt/dQCmtMx0UKVUtlSxyt+vVDYJxGhs/ XW4saH/M7CczWZ7117EU+s3IDlGYUcdaMkKUaddC9wfr1uue16Ab1Zpv9cytqWuBo0Ou LRWpIFPJOhfGmRvuPDp5RRQ+vdgDAzEg0YG0LoXreLPTgdcUThHlvpgVBFMo8gYwgUtI UNOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=+iyuanaaKf/C0aGVTsW4R9+UMvTbKMgGyilj78Cwu/8=; b=YgoAitQ8/ibRBSWIzAMKtkoixQyknX4YKcTIdNkZtqFR57GskkE7zmOk0XzEPyhsU1 FjfRAgggONKlborztDpBTCGxRaJkZfmywYw+du2xsGByeN66IluejMc0a5EbKMWdujP7 5NgM4t/EjRO+4C+mTAU6V7d3lnKvmMDqFWCSHMJx9zkzoU49CnyI8+VhYkFBahn6oG5y tzbq/zHdmyfq2r2jP81++dLYTaDh970uYiYdyG5aF3Grt570478KyfCSmVsx1IipXuQ2 ATmdJ8dUHLoregwzm4oJSILydS5BKCpj000h/QjAC8oSYtlBOCuyQYWhH/cJCNC3cbaU 4EDA== X-Gm-Message-State: APjAAAWhNTqdLyMYaHkD/qj2mi1nYf7wwZTIPPnOsTvsodaogJp7Wnpj M0/qt/0Uzx+goUULw/cjJyDxeFt6 X-Google-Smtp-Source: APXvYqwogDde7RyttMB6voJjf2dL1plNh2ALDM5aY9IkT0gXDtILEWvD7iA1rGWe8xn0F5sZDEVD2Q== X-Received: by 2002:a63:b08:: with SMTP id 8mr13121154pgl.118.1582316716545; Fri, 21 Feb 2020 12:25:16 -0800 (PST) Received: from sc9-mailhost3.vmware.com ([66.170.99.95]) by smtp.gmail.com with ESMTPSA id c68sm3795744pfc.156.2020.02.21.12.25.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 21 Feb 2020 12:25:15 -0800 (PST) From: William Tu To: dev@openvswitch.org Date: Fri, 21 Feb 2020 12:24:50 -0800 Message-Id: <1582316690-8678-1-git-send-email-u9012063@gmail.com> X-Mailer: git-send-email 2.7.4 Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH] netdev-afxdp: Add interrupt mode using poll syscall. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The patch adds a new option 'use-intr' to enable afxdp interrupt mode. At receive path, add a poll() syscall so that when there is no packet arrived, the pmd thread will be blocked and this saves some CPU time for other processes. This avoids burning the CPU to always 100% when there is no traffic. Disabled by default. Tested-at: https://travis-ci.org/williamtu/ovs-travis/builds/653613549 Signed-off-by: William Tu --- NEWS | 3 +++ lib/netdev-afxdp.c | 45 ++++++++++++++++++++++++++++++++++++++++++++- lib/netdev-linux-private.h | 2 ++ vswitchd/vswitch.xml | 12 ++++++++++++ 4 files changed, 61 insertions(+), 1 deletion(-) diff --git a/NEWS b/NEWS index f62ef1f47ea8..d91002a76fff 100644 --- a/NEWS +++ b/NEWS @@ -4,6 +4,9 @@ Post-v2.13.0 - OpenFlow: * The OpenFlow ofp_desc/serial_num may now be configured by setting the value of other-config:dp-sn in the Bridge table. + - AF_XDP: + * New option 'use-intr' for netdev-afxdp to save CPU cycles by enabling + interrupt mode. Disabled by default. v2.13.0 - 14 Feb 2020 diff --git a/lib/netdev-afxdp.c b/lib/netdev-afxdp.c index 482400d8d135..7b60acde1b02 100644 --- a/lib/netdev-afxdp.c +++ b/lib/netdev-afxdp.c @@ -604,6 +604,7 @@ netdev_afxdp_set_config(struct netdev *netdev, const struct smap *args, enum afxdp_mode xdp_mode; bool need_wakeup; int new_n_rxq; + bool use_intr; ovs_mutex_lock(&dev->mutex); new_n_rxq = MAX(smap_get_int(args, "n_rxq", NR_QUEUE), 1); @@ -637,12 +638,16 @@ netdev_afxdp_set_config(struct netdev *netdev, const struct smap *args, } #endif + use_intr = smap_get_bool(args, "use-intr", false); + if (dev->requested_n_rxq != new_n_rxq || dev->requested_xdp_mode != xdp_mode - || dev->requested_need_wakeup != need_wakeup) { + || dev->requested_need_wakeup != need_wakeup + || dev->requested_use_intr != use_intr) { dev->requested_n_rxq = new_n_rxq; dev->requested_xdp_mode = xdp_mode; dev->requested_need_wakeup = need_wakeup; + dev->requested_use_intr = use_intr; netdev_request_reconfigure(netdev); } ovs_mutex_unlock(&dev->mutex); @@ -661,6 +666,8 @@ netdev_afxdp_get_config(const struct netdev *netdev, struct smap *args) xdp_modes[dev->xdp_mode_in_use].name); smap_add_format(args, "use-need-wakeup", "%s", dev->use_need_wakeup ? "true" : "false"); + smap_add_format(args, "use-intr", "%s", + dev->use_intr ? "true" : "false"); ovs_mutex_unlock(&dev->mutex); return 0; } @@ -696,6 +703,7 @@ netdev_afxdp_reconfigure(struct netdev *netdev) if (netdev->n_rxq == dev->requested_n_rxq && dev->xdp_mode == dev->requested_xdp_mode && dev->use_need_wakeup == dev->requested_need_wakeup + && dev->use_intr == dev->requested_use_intr && dev->xsks) { goto out; } @@ -713,6 +721,7 @@ netdev_afxdp_reconfigure(struct netdev *netdev) VLOG_ERR("setrlimit(RLIMIT_MEMLOCK) failed: %s", ovs_strerror(errno)); } dev->use_need_wakeup = dev->requested_need_wakeup; + dev->use_intr = dev->requested_use_intr; err = xsk_configure_all(netdev); if (err) { @@ -815,6 +824,32 @@ prepare_fill_queue(struct xsk_socket_info *xsk_info) xsk_info->available_rx += BATCH_SIZE; } +static int +enable_intr_mode(int fd, struct netdev *netdev) +{ + struct pollfd fds[1]; + int ret; + + memset(fds, 0, sizeof fds); + fds[0].fd = fd; + fds[0].events = POLLIN; + + ret = poll(fds, 1, 1); + if (OVS_UNLIKELY(ret == 0)) { + /* Timeout. */ + ovsrcu_quiesce_start(); + return EAGAIN; + } + if (OVS_UNLIKELY(ret < 0)) { + VLOG_WARN_RL(&rl, "%s: error polling rx fd: %s.", + netdev_get_name(netdev), + ovs_strerror(errno)); + return EAGAIN; + } + + return 0; +} + int netdev_afxdp_rxq_recv(struct netdev_rxq *rxq_, struct dp_packet_batch *batch, int *qfill) @@ -827,6 +862,7 @@ netdev_afxdp_rxq_recv(struct netdev_rxq *rxq_, struct dp_packet_batch *batch, uint32_t idx_rx = 0; int qid = rxq_->queue_id; unsigned int rcvd, i; + int ret; xsk_info = dev->xsks[qid]; if (!xsk_info || !xsk_info->xsk) { @@ -838,6 +874,13 @@ netdev_afxdp_rxq_recv(struct netdev_rxq *rxq_, struct dp_packet_batch *batch, umem = xsk_info->umem; rx->fd = xsk_socket__fd(xsk_info->xsk); + if (OVS_UNLIKELY(dev->use_intr)) { + ret = enable_intr_mode(rx->fd, netdev); + if (ret > 0) { + return ret; + } + } + rcvd = xsk_ring_cons__peek(&xsk_info->rx, BATCH_SIZE, &idx_rx); if (!rcvd) { xsk_rx_wakeup_if_needed(umem, netdev, rx->fd); diff --git a/lib/netdev-linux-private.h b/lib/netdev-linux-private.h index c7c515f70700..a89eb36b5e3c 100644 --- a/lib/netdev-linux-private.h +++ b/lib/netdev-linux-private.h @@ -116,6 +116,8 @@ struct netdev_linux { bool use_need_wakeup; bool requested_need_wakeup; + bool use_intr; + bool requested_use_intr; struct netdev_afxdp_tx_lock *tx_locks; /* Array of locks for TX queues. */ #endif diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index 4a74ed3ef24e..046018179578 100644 --- a/vswitchd/vswitch.xml +++ b/vswitchd/vswitch.xml @@ -3186,6 +3186,18 @@ ovs-vsctl add-port br0 p0 -- set Interface p0 type=patch options:peer=p1 \

+ +

+ Specifies whether to use interrupt mode feature in afxdp netdev. + If enabled, the pmd thread calls poll() syscall on the RX path. + If there is no packet at rx queue, the pmd is blocked at poll() + until it is timeout or new packets arrive. This saves CPU cycles + when packet arrival rate is low. + Defaults to false. +

+ +