From patchwork Thu Aug 31 13:11:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Jarry X-Patchwork-Id: 1828343 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=O+oRlU/i; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Rc1lQ6jBHz1yfm for ; Thu, 31 Aug 2023 23:11:46 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id A19BF612F4; Thu, 31 Aug 2023 13:11:44 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org A19BF612F4 Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=O+oRlU/i X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mZrCWuD14WJU; Thu, 31 Aug 2023 13:11:43 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 8EE1760BA7; Thu, 31 Aug 2023 13:11:42 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 8EE1760BA7 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4FB5CC0039; Thu, 31 Aug 2023 13:11:42 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id B0062C0032 for ; Thu, 31 Aug 2023 13:11:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 84613833FB for ; Thu, 31 Aug 2023 13:11:40 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 84613833FB Authentication-Results: smtp1.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=O+oRlU/i X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id n88dYt0P9S_4 for ; Thu, 31 Aug 2023 13:11:39 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 169D88177A for ; Thu, 31 Aug 2023 13:11:38 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 169D88177A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693487497; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=jkNsIUStnjmbUrm6+yX3fK2ev2bpNBGAE8X43RjwqFc=; b=O+oRlU/i8yc3ynAlqSEbzV8jqwGWGYJG23ueWZUBTn9Rc73SG3yC8pG1Q9gqw2mBxC7FXI T/9YOeGHq3+wGfeh+e41gGVLyqyT70/iJLfdTrQ64Qq/wbHhnuDPyHqLafJqxYgic8Dh6E 4C90mePy/iQebxBtFKqQb56tke8/pHY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-41-41HYZ3ATO5KxctBJ57xcVQ-1; Thu, 31 Aug 2023 09:11:36 -0400 X-MC-Unique: 41HYZ3ATO5KxctBJ57xcVQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8A6BA856DED for ; Thu, 31 Aug 2023 13:11:34 +0000 (UTC) Received: from ringo.redhat.com (unknown [10.39.208.40]) by smtp.corp.redhat.com (Postfix) with ESMTP id 82AD9C15BAE; Thu, 31 Aug 2023 13:11:33 +0000 (UTC) From: Robin Jarry To: dev@openvswitch.org Date: Thu, 31 Aug 2023 15:11:22 +0200 Message-ID: <20230831131122.284913-1-rjarry@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: David Marchand Subject: [ovs-dev] [PATCH] dpdk: expose cpu usage stats on telemetry socket X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Since DPDK 23.03, it is possible to register a callback to report lcore TSC cycles usage. Reuse the busy/idle cycles gathering in dpif-netdev and expose them to the DPDK telemetry socket. Upon dpdk_attach_thread, record the mapping between the DPDK lcore_id and the dpif-netdev core_id. Reuse that mapping in the lcore usage callback to invoke dpif_netdev_get_pmd_cycles. Here is an example output: ~# ovs-appctl dpif-netdev/pmd-stats-show | grep -e ^pmd -e cycles: pmd thread numa_id 0 core_id 8: idle cycles: 2720796781680 (100.00%) processing cycles: 3566020 (0.00%) pmd thread numa_id 0 core_id 9: idle cycles: 2718974371440 (100.00%) processing cycles: 3136840 (0.00%) pmd thread numa_id 0 core_id 72: pmd thread numa_id 0 core_id 73: ~# echo /eal/lcore/usage | dpdk-telemetry.py | jq { "/eal/lcore/usage": { "lcore_ids": [ 3, 5, 11, 15 ], "total_cycles": [ 2725722342740, 2725722347480, 2723899464040, 2725722354980 ], "busy_cycles": [ 3566020, 3566020, 3136840, 3566020 ] } } Link: https://git.dpdk.org/dpdk/commit/?id=9ab1804922ba583b0b16 Cc: David Marchand Cc: Kevin Traynor Signed-off-by: Robin Jarry --- lib/dpdk-stub.c | 5 +++ lib/dpdk.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++- lib/dpdk.h | 5 +++ lib/dpif-netdev.c | 38 +++++++++++++++++++ 4 files changed, 142 insertions(+), 1 deletion(-) diff --git a/lib/dpdk-stub.c b/lib/dpdk-stub.c index 58ebf6cb62cd..02fb561bea7b 100644 --- a/lib/dpdk-stub.c +++ b/lib/dpdk-stub.c @@ -49,6 +49,11 @@ dpdk_detach_thread(void) { } +void +dpdk_register_core_usage_callback(dpdk_core_usage_cb *cb OVS_UNUSED) +{ +} + bool dpdk_available(void) { diff --git a/lib/dpdk.c b/lib/dpdk.c index d76d53f8f16c..31871300f719 100644 --- a/lib/dpdk.c +++ b/lib/dpdk.c @@ -23,6 +23,7 @@ #include #include +#include #include #include #include @@ -310,6 +311,10 @@ malloc_dump_stats_wrapper(FILE *stream) rte_malloc_dump_stats(stream, NULL); } +#ifdef ALLOW_EXPERIMENTAL_API +static int dpdk_get_lcore_cycles(unsigned int, struct rte_lcore_usage *); +#endif + static bool dpdk_init__(const struct smap *ovs_other_config) { @@ -440,6 +445,10 @@ dpdk_init__(const struct smap *ovs_other_config) /* We are called from the main thread here */ RTE_PER_LCORE(_lcore_id) = NON_PMD_CORE_ID; +#ifdef ALLOW_EXPERIMENTAL_API + rte_lcore_register_usage_cb(dpdk_get_lcore_cycles); +#endif + /* Finally, register the dpdk classes */ netdev_dpdk_register(ovs_other_config); netdev_register_flow_api_provider(&netdev_offload_dpdk); @@ -490,9 +499,52 @@ dpdk_available(void) return initialized; } +struct lcore_id_map { + unsigned int lcore_id; + unsigned int pmd_core_id; +}; + +/* Protects against changes to 'lcore_id_maps'. */ +struct ovs_mutex lcore_id_maps_mutex = OVS_MUTEX_INITIALIZER; + +/* Contains all 'struct lcore_id_map's. */ +static struct shash lcore_id_maps OVS_GUARDED_BY(lcore_id_maps_mutex) + = SHASH_INITIALIZER(&lcore_id_maps); + +static void +lcore_id_to_str(char *buf, size_t len, unsigned int lcore_id) +{ + int n; + + n = snprintf(buf, len, "%u", lcore_id); + if (n < 0) { + VLOG_WARN("Failed to format lcore_id: %s", ovs_strerror(errno)); + n = 0; + } + buf[n] = '\0'; +} + +static void +lcore_id_map_update(unsigned int lcore_id, unsigned int cpu, bool add) +{ + char buf[128]; + + lcore_id_to_str(buf, sizeof buf, lcore_id); + + ovs_mutex_lock(&lcore_id_maps_mutex); + if (add) { + shash_replace(&lcore_id_maps, buf, (void *) (uintptr_t) cpu); + } else { + shash_find_and_delete(&lcore_id_maps, buf); + } + ovs_mutex_unlock(&lcore_id_maps_mutex); +} + bool dpdk_attach_thread(unsigned cpu) { + unsigned int lcore_id; + /* NON_PMD_CORE_ID is reserved for use by non pmd threads. */ ovs_assert(cpu != NON_PMD_CORE_ID); @@ -506,7 +558,9 @@ dpdk_attach_thread(unsigned cpu) return false; } - VLOG_INFO("PMD thread uses DPDK lcore %u.", rte_lcore_id()); + lcore_id = rte_lcore_id(); + lcore_id_map_update(lcore_id, cpu, true); + VLOG_INFO("PMD thread uses DPDK lcore %u.", lcore_id); return true; } @@ -516,10 +570,49 @@ dpdk_detach_thread(void) unsigned int lcore_id; lcore_id = rte_lcore_id(); + lcore_id_map_update(lcore_id, 0, false); + rte_thread_unregister(); VLOG_INFO("PMD thread released DPDK lcore %u.", lcore_id); } +static dpdk_core_usage_cb_t *core_usage_cb; + +void +dpdk_register_core_usage_callback(dpdk_core_usage_cb_t *cb) +{ + core_usage_cb = cb; +} + +#ifdef ALLOW_EXPERIMENTAL_API +static int +dpdk_get_lcore_cycles(unsigned int lcore_id, struct rte_lcore_usage *usage) +{ + struct shash_node *node; + unsigned int core_id; + char buf[128]; + + if (!core_usage_cb) { + return -1; + } + + lcore_id_to_str(buf, sizeof buf, lcore_id); + + ovs_mutex_lock(&lcore_id_maps_mutex); + node = shash_find(&lcore_id_maps, buf); + ovs_mutex_unlock(&lcore_id_maps_mutex); + + if (!node) { + return -1; + } + + core_id = (unsigned int) (uintptr_t) node->data; + core_usage_cb(core_id, &usage->busy_cycles, &usage->total_cycles); + + return 0; +} +#endif + void print_dpdk_version(void) { diff --git a/lib/dpdk.h b/lib/dpdk.h index 1b790e682e4d..95a51698c74d 100644 --- a/lib/dpdk.h +++ b/lib/dpdk.h @@ -18,6 +18,7 @@ #define DPDK_H #include +#include #ifdef DPDK_NETDEV @@ -42,4 +43,8 @@ bool dpdk_available(void); void print_dpdk_version(void); void dpdk_status(const struct ovsrec_open_vswitch *); +typedef void (dpdk_core_usage_cb_t)(unsigned cpu, uint64_t *busy_cycles, + uint64_t *total_cycles); +void dpdk_register_core_usage_callback(dpdk_core_usage_cb_t *cb); + #endif /* dpdk.h */ diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 70b953ae6dd3..ebf43a0f62e4 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -1427,6 +1427,41 @@ dpif_netdev_pmd_rebalance(struct unixctl_conn *conn, int argc, ds_destroy(&reply); } +static void +dpif_netdev_get_pmd_cycles(unsigned int core_id, + uint64_t *busy_cycles, uint64_t *total_cycles) +{ + struct dp_netdev_pmd_thread **pmd_list = NULL; + uint64_t stats[PMD_N_STATS]; + struct dp_netdev *dp; + size_t num_pmds; + + ovs_mutex_lock(&dp_netdev_mutex); + + if (shash_count(&dp_netdevs) != 1) { + goto out; + } + + dp = shash_first(&dp_netdevs)->data; + sorted_poll_thread_list(dp, &pmd_list, &num_pmds); + + for (size_t i = 0; i < num_pmds; i++) { + struct dp_netdev_pmd_thread *pmd = pmd_list[i]; + + if (pmd->core_id == core_id) { + continue; + } + pmd_perf_read_counters(&pmd->perf_stats, stats); + *busy_cycles = stats[PMD_CYCLES_ITER_BUSY]; + *total_cycles = *busy_cycles + stats[PMD_CYCLES_ITER_IDLE]; + break; + } + +out: + free(pmd_list); + ovs_mutex_unlock(&dp_netdev_mutex); +} + static void dpif_netdev_pmd_info(struct unixctl_conn *conn, int argc, const char *argv[], void *aux) @@ -1661,6 +1696,9 @@ dpif_netdev_init(void) unixctl_command_register("dpif-netdev/miniflow-parser-get", "", 0, 0, dpif_miniflow_extract_impl_get, NULL); + + dpdk_register_core_usage_callback(dpif_netdev_get_pmd_cycles); + return 0; }