From patchwork Thu Mar 2 07:16:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengen Du X-Patchwork-Id: 1750398 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=XNftVUpe; dkim-atps=neutral Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4PS2TY0c7tz2460 for ; Thu, 2 Mar 2023 18:16:32 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1pXdAu-00069r-A8; Thu, 02 Mar 2023 07:16:24 +0000 Received: from smtp-relay-internal-0.internal ([10.131.114.225] helo=smtp-relay-internal-0.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1pXdAr-00069d-L3 for kernel-team@lists.ubuntu.com; Thu, 02 Mar 2023 07:16:21 +0000 Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id 4BD4C3F144 for ; Thu, 2 Mar 2023 07:16:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1677741381; bh=/rbPZpd2/NPwCwgVfPCtV4KPxddQs5FMpg2TKSmhgvI=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=XNftVUperjUeYgoaZEbHeUTiqkO0Ku9tbbniV6+rwuRV9JKwwl10x319ObDffHwGW RT1hA00AY3F8/zwd1xOvn6/UzewL/DJQBfw4aqmY+DVSe5osZbxh394SIAZifbCbwn ed4OOxHcGB10a6KTuYHEYd1Fsyc2KCpED4/K2NtJUoybrjfD2zb5Wbi7mJb1YlexJC jWe9fxVNHJgVclNB/hqbg7y3hqRTqNP8JDBDlEAigkgZ3lUP4axMKoNNEFxgzhPqdi /PhKKikw/Jo6oPuMpxBjRzBm1uheE1GidO1BBIelIJbm40PQdlq+Hbr0W8mi4MYbPX 2m6RVAoomr6+w== Received: by mail-pl1-f198.google.com with SMTP id l10-20020a17090270ca00b0019caa6e6bd1so8195862plt.2 for ; Wed, 01 Mar 2023 23:16:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677741380; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/rbPZpd2/NPwCwgVfPCtV4KPxddQs5FMpg2TKSmhgvI=; b=focE26I/h81IZDA/eAwkKm1/umpj6b2t611UvxdSjVjfG+kWaqMNf2AQevL7UJVoQ2 mYAT87/USFuloT5ppZUTm47MFipNHZRnUx4Oh5p2T7EZwBxOdOm9zcdQwQc9+JsAitQI e8OEmgltaO3bK4s+e67cWzpo3U10MAqYrM2dRsvgU7o6+2rzSRDpWVmo69IY7GFVMIHA rdzp3Pv4BjoUXJKdgkuJKBmhigMxrdchqm+5HDJSel93jPneZBEQaj06O1BFor5V7V0K dhw1bw8N0+P1HywMYhwHPRqCrp0xKL7wx7MCjMyMuMTETvmX/NnKxrTiBzhHsu2PiNIa vj/A== X-Gm-Message-State: AO0yUKVZGEqz9mE4MqEqWQdsCJRqWTEvzKksv03BS0qrhiiNKsX9PaZ/ kb4kGC5RCEbMk/E6l8nnRSNWIhYaGMxmKoUEEJSj72hsnVAcHc8hj2mMkVEPVDNJjPTJ2FDe4HH 1YVpNIXditYGlW63w0fQuNtdGwYghVsg/ewmV92zn1B/FxK4= X-Received: by 2002:a17:903:41c1:b0:19c:d550:8cd4 with SMTP id u1-20020a17090341c100b0019cd5508cd4mr11568884ple.7.1677741379628; Wed, 01 Mar 2023 23:16:19 -0800 (PST) X-Google-Smtp-Source: AK7set9WEfjk+hvMnQHnAEtJ7KJL9SgxmAeg6YvD22ei1O0gOJyb0ORTL6wzVEk9p2YBbJzzEdy7BA== X-Received: by 2002:a17:903:41c1:b0:19c:d550:8cd4 with SMTP id u1-20020a17090341c100b0019cd5508cd4mr11568862ple.7.1677741379207; Wed, 01 Mar 2023 23:16:19 -0800 (PST) Received: from chengendu.. (111-248-105-208.dynamic-ip.hinet.net. [111.248.105.208]) by smtp.gmail.com with ESMTPSA id a11-20020a170902900b00b0019a9637b2d3sm9576523plp.279.2023.03.01.23.16.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Mar 2023 23:16:18 -0800 (PST) From: Chengen Du To: kernel-team@lists.ubuntu.com Subject: [SRU][Jammy][PATCH 0/1] Do not sort the task scan result from /proc when synthesizing perf events Date: Thu, 2 Mar 2023 15:16:15 +0800 Message-Id: <20230302071616.40489-1-chengen.du@canonical.com> X-Mailer: git-send-email 2.37.2 MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" BugLink: https://bugs.launchpad.net/bugs/2008971 SRU Justification: [Impact] The perf tool use scandir() to iterate threads and sort in alphabetical order when synthesizing PERF_RECORD_ events. If the process ID is 9999 and it has one thread (tid = 10000), the thread will be processed before the process. It results in PERF_RECORD_FORK events that come before PERF_RECORD_MMAP2 events. The callstack will have missing symbols for threads where `PERF_RECORD_FORK` events are processed before the `PERF_RECORD_MMAP2` event for the corresponding process. [Fix] Do not use alphasort when calling scandir() 363afa3aef24f5e08df6a539f5dc3aae4cddcc1a (perf synthetic-events: Don't sort the task scan result from /proc) [Test Plan] function update_last_pid() local file = io.open("/proc/sys/kernel/ns_last_pid", "w") file:write(9997) end update_last_pid() os.execute("~/reproducer") #include #include #include constexpr int kThreadNum = 10; void thread_job() { sleep(30); } int main(void) { std::thread threads[kThreadNum]; std::cout << "Parent process with pid " << getpid() << std::endl; for (int i = 0; i < kThreadNum; ++i) { threads[i] = std::thread(thread_job); } for (int i = 0; i < kThreadNum; ++i) { threads[i].join(); } std::cout << "All threads have finished" << std::endl; return 0; } The flow is to set /proc/sys/kernel/ns_last_pid first, which represents the last pid allocated in the current pid namespace. The script (test.lua) sets ns_last_pid to 9997 and executes the reproducer (reproducer.cpp). After the reproducer creates ten threads, we execute the perf command as follows: perf record -F 49 -e cpu-clock -a -g sleep 20. Here is the result of command: perf report -f --tasks --mmaps -D | egrep -i 'perf_record_fork|perf_record_mmap' | grep 9999 Before applying the patch, the output of the perf command was as follows: 0 0 0x34910 [0x40]: PERF_RECORD_FORK(9999:10000):(9999:9999) 0 0 0x34990 [0x40]: PERF_RECORD_FORK(9999:10001):(9999:9999) 0 0 0x34a10 [0x40]: PERF_RECORD_FORK(9999:10002):(9999:9999) 0 0 0x34a90 [0x40]: PERF_RECORD_FORK(9999:10003):(9999:9999) 0 0 0x34b10 [0x40]: PERF_RECORD_FORK(9999:10004):(9999:9999) 0 0 0x34b90 [0x40]: PERF_RECORD_FORK(9999:10005):(9999:9999) 0 0 0x34c10 [0x40]: PERF_RECORD_FORK(9999:10006):(9999:9999) 0 0 0x34c90 [0x40]: PERF_RECORD_FORK(9999:10007):(9999:9999) 0 0 0x34d10 [0x40]: PERF_RECORD_FORK(9999:10008):(9999:9999) 0 0 0x34d90 [0x40]: PERF_RECORD_FORK(9999:10009):(9999:9999) 0 0 0x34e10 [0x40]: PERF_RECORD_FORK(9999:9999):(9998:9998) 0 0 0x34e90 [0x80]: PERF_RECORD_MMAP2 9999/9999: [0x555de0159000(0x1000) @ 0x1000 fd:00 2097758 0]: r-xp /root/reproducer 0 0 0x34f10 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd17f8000(0x7c000) @ 0xe000 fd:00 1055098 0]: r-xp /usr/lib/x86_64-linux-gnu/libm.so.6 0 0 0x34fa0 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd18f9000(0x195000) @ 0x28000 fd:00 1054988 0]: r-xp /usr/lib/x86_64-linux-gnu/libc.so.6 0 0 0x35030 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd1afc000(0x17000) @ 0x3000 fd:00 1054745 0]: r-xp /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 0 0 0x350c0 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd1bb3000(0x110000) @ 0x9a000 fd:00 1068050 0]: r-xp /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30 0 0 0x35158 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f9cd1d4e000(0x2a000) @ 0x2000 fd:00 1054949 0]: r-xp /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 0 0 0x351f0 [0x70]: PERF_RECORD_MMAP2 9999/9999: [0x7ffc983bb000(0x2000) @ 0 00:00 0 0]: r-xp [vdso] 0 0 0x35260 [0x78]: PERF_RECORD_MMAP2 9999/9999: [0xffffffffff600000(0x1000) @ 0 00:00 0 0]: --xp [vsyscall] After applying the patch, the output of the perf command is as follows: 0 0 0x30c28 [0x40]: PERF_RECORD_FORK(9999:9999):(9998:9998) 0 0 0x30ca8 [0x80]: PERF_RECORD_MMAP2 9999/9999: [0x5642c7635000(0x1000) @ 0x1000 fd:00 2097758 0]: r-xp /root/reproducer 0 0 0x30d28 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada03d000(0x7c000) @ 0xe000 fd:00 1055098 0]: r-xp /usr/lib/x86_64-linux-gnu/libm.so.6 0 0 0x30db8 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada13e000(0x195000) @ 0x28000 fd:00 1054988 0]: r-xp /usr/lib/x86_64-linux-gnu/libc.so.6 0 0 0x30e48 [0x90]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada341000(0x17000) @ 0x3000 fd:00 1054745 0]: r-xp /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 0 0 0x30ed8 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada3f8000(0x110000) @ 0x9a000 fd:00 1068050 0]: r-xp /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30 0 0 0x30f70 [0x98]: PERF_RECORD_MMAP2 9999/9999: [0x7f8ada593000(0x2a000) @ 0x2000 fd:00 1054949 0]: r-xp /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 0 0 0x31008 [0x70]: PERF_RECORD_MMAP2 9999/9999: [0x7ffcb2116000(0x2000) @ 0 00:00 0 0]: r-xp [vdso] 0 0 0x31078 [0x78]: PERF_RECORD_MMAP2 9999/9999: [0xffffffffff600000(0x1000) @ 0 00:00 0 0]: --xp [vsyscall] 0 0 0x310f0 [0x40]: PERF_RECORD_FORK(9999:10000):(9999:9999) 0 0 0x31170 [0x40]: PERF_RECORD_FORK(9999:10001):(9999:9999) 0 0 0x311f0 [0x40]: PERF_RECORD_FORK(9999:10002):(9999:9999) 0 0 0x31270 [0x40]: PERF_RECORD_FORK(9999:10003):(9999:9999) 0 0 0x312f0 [0x40]: PERF_RECORD_FORK(9999:10004):(9999:9999) 0 0 0x31370 [0x40]: PERF_RECORD_FORK(9999:10005):(9999:9999) 0 0 0x313f0 [0x40]: PERF_RECORD_FORK(9999:10006):(9999:9999) 0 0 0x31470 [0x40]: PERF_RECORD_FORK(9999:10007):(9999:9999) 0 0 0x314f0 [0x40]: PERF_RECORD_FORK(9999:10008):(9999:9999) 0 0 0x31570 [0x40]: PERF_RECORD_FORK(9999:10009):(9999:9999) [Where problems could occur] The fix will apply upstream commits, so the regression can be considered as low. Namhyung Kim (1): perf synthetic-events: Don't sort the task scan result from /proc tools/perf/util/synthetic-events.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) Acked-by: Tim Gardner Acked-by: Luke Nowakowski-Krijger