From patchwork Tue Feb 20 17:14:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: K Prateek Nayak X-Patchwork-Id: 1901595 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=amd.com header.i=@amd.com header.a=rsa-sha256 header.s=selector1 header.b=UgYZZLGp; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=ozlabs.org (client-ip=2404:9400:2221:ea00::3; helo=gandalf.ozlabs.org; envelope-from=srs0=ucn6=j5=vger.kernel.org=sparclinux+bounces-359-patchwork-incoming=ozlabs.org@ozlabs.org; receiver=patchwork.ozlabs.org) Received: from gandalf.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4TfR194R2tz20RF for ; Wed, 21 Feb 2024 04:17:33 +1100 (AEDT) Received: from gandalf.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by gandalf.ozlabs.org (Postfix) with ESMTP id 4TfR193tR0z4wxZ for ; Wed, 21 Feb 2024 04:17:33 +1100 (AEDT) Received: by gandalf.ozlabs.org (Postfix) id 4TfR192JSjz4wyg; Wed, 21 Feb 2024 04:17:33 +1100 (AEDT) Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: gandalf.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: gandalf.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=amd.com header.i=@amd.com header.a=rsa-sha256 header.s=selector1 header.b=UgYZZLGp; dkim-atps=neutral Authentication-Results: gandalf.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=139.178.88.99; helo=sv.mirrors.kernel.org; envelope-from=sparclinux+bounces-359-patchwork-incoming=ozlabs.org@vger.kernel.org; receiver=ozlabs.org) Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org [139.178.88.99]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by gandalf.ozlabs.org (Postfix) with ESMTPS id 4TfR186mlKz4wxZ for ; Wed, 21 Feb 2024 04:17:32 +1100 (AEDT) Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id A97A62857E1 for ; Tue, 20 Feb 2024 17:17:31 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0B8AB76917; Tue, 20 Feb 2024 17:17:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="UgYZZLGp" X-Original-To: sparclinux@vger.kernel.org Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2085.outbound.protection.outlook.com [40.107.223.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7BF64626C6; Tue, 20 Feb 2024 17:17:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.85 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708449447; cv=fail; b=WzugrEL9z8mKGs2k/bRHniyD7nk9oGEeMdqAmDBVf5xHkwqjKGJrPEaUlVbrhevPQMMyHxdNwvoxJw0ylzlQY+JWQyPI82ulzVyDOhOR4AmrIWAt7SPHVJdKJEMp0N/AUagffgHAp7vxVYF4+SqOtSk1JD0qD90HjGDSkw6KD5c= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708449447; c=relaxed/simple; bh=q73GbydjQ9EyweCHfZP9AYAB0qit57DKnvbQuJYJ/tU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=MLE7/13RN6k3F/bF/lAzPGXLe2r6HrvcVg3bTFPMMtm1ygAfdGOgaRw2YH4K07RgDeDuCGMRe8BZ2JYGqdYVcjt3hhLSKnppjwYgrvzEgHyKs8SgTKQhJ7emJJhqzbZVFq1GlbY2CJpT0co5VChC5DqP9jSWAnuO2xXe9P+cyoc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=UgYZZLGp; arc=fail smtp.client-ip=40.107.223.85 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=I4M+sz5envWScVYjmU7ne/7hYsu99wMXjycwJlJBSYkggP2SZdRpHHaOg+cpAeycAu4b40wlBd1Zwgii7Il+XW1aAFh3oHR+TVfMfgqkSsy1tB8Uhez3zi5Qv3eAo+g8z/1YACsSiltEaTOmdi5flKfyH/eHEC+WjQXS5VMAQ8mPcy7LL49y39Z/DV2xAVNesK/El7tqOJemQGng6V3t3Q7AddupHS5BCrBCucVdwtltNzkD77CG22vYSgmbxQPzoUbdhMpsqSSsb7Vy5kTbx0xEt2z3i5fHHb0ZASNcws+aMe473yTySGhWNIF+uy3rl+BJkeCmTG1n8BpGBKIqsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=6zn7Cdv/hE7tDImP8Q7FT7F4wn3Jxa6COi5JJIJEBPQ=; b=QYKUsHjPapafVmoq4lbhS2FZbNHC7SgpTgS21Zn36cGF4XWZwn0pgeaTWwuZK4nLoJSqcetDwbv1TKRb3o6F+foGOsc8J+O9sx4XstkuIKTHW2RYayKmzaqQ5RQL801RLykbWZZYdAU2LBopqX1Wd8fFE9WX+KKZqm+M7K7432AT9mUP0c1nanhwIaqT8kUSkZLT+LCxIU0kmsZdqML8OhiX98RFLbFThMrpS7YOAFWRDDOIxu/SbSA/vlMiBudtfciXz35Gv5hyEdVooN5T/N0xEScLJYm9wcKKzMXxnTeSGc4Bta2dbiDyY98IXzi3U3sQW1SAc9qaRUT92+e1qQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6zn7Cdv/hE7tDImP8Q7FT7F4wn3Jxa6COi5JJIJEBPQ=; b=UgYZZLGpvKCll6WzVEGghqVfehpGDE3Dln3V4qRPkS+tvMXIEDqCI1BWxy+QlEInStqCw+tunb+ZTzzwHtZXfDDxEjXIwqb1SMky1rSOoDIXvoZ21UdvDtjVdxW5ZNqX0yjwA3hC9Sro5SWyaxIPVfKBKsFtW1cmKp+Yw58XNJk= Received: from BN8PR04CA0042.namprd04.prod.outlook.com (2603:10b6:408:d4::16) by CY5PR12MB6430.namprd12.prod.outlook.com (2603:10b6:930:3a::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7316.19; Tue, 20 Feb 2024 17:17:22 +0000 Received: from BN2PEPF000044A5.namprd04.prod.outlook.com (2603:10b6:408:d4:cafe::30) by BN8PR04CA0042.outlook.office365.com (2603:10b6:408:d4::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7292.40 via Frontend Transport; Tue, 20 Feb 2024 17:17:21 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN2PEPF000044A5.mail.protection.outlook.com (10.167.243.104) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7292.25 via Frontend Transport; Tue, 20 Feb 2024 17:17:21 +0000 Received: from BLR5CG134614W.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Tue, 20 Feb 2024 11:17:04 -0600 From: K Prateek Nayak To: CC: "Gautham R. Shenoy" , Richard Henderson , Ivan Kokshaysky , Matt Turner , Russell King , "Guo Ren" , Michal Simek , Dinh Nguyen , Jonas Bonn , Stefan Kristiansson , Stafford Horne , "James E.J. Bottomley" , Helge Deller , Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Yoshinori Sato , Rich Felker , John Paul Adrian Glaubitz , "David S. Miller" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , "Rafael J. Wysocki" , Daniel Lezcano , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , "Mel Gorman" , Daniel Bristot de Oliveira , Valentin Schneider , Al Viro , Linus Walleij , Ard Biesheuvel , Andrew Donnellan , Nicholas Miehlbradt , Andrew Morton , "Arnd Bergmann" , Josh Poimboeuf , "Kirill A. Shutemov" , Rick Edgecombe , Tony Battersby , "Brian Gerst" , Tim Chen , "David Vernet" , , , , , , , , , , , K Prateek Nayak Subject: [RFC PATCH 03/14] sched/core: Use TIF_NOTIFY_IPI to notify an idle CPU in TIF_POLLING mode of pending IPI Date: Tue, 20 Feb 2024 22:44:46 +0530 Message-ID: <20240220171457.703-4-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240220171457.703-1-kprateek.nayak@amd.com> References: <20240220171457.703-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: sparclinux@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF000044A5:EE_|CY5PR12MB6430:EE_ X-MS-Office365-Filtering-Correlation-Id: 0186f5ac-8e33-4c3d-0671-08dc3237ccae X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ncHCayupuzmtB3AtAKoRY3cty31XhyLDo/3xPrfDWidenBe585b245/PFNNNuV3DkaV5Pe/xQ93dwP8naAMX5O9pBHeSR6bRKqU75A/alAYiSNaPdotwr9+USt6ppg7m0irux8100pf9HnhN0b6BsC49Dly8LMjfx5IT2Y/oay6j/6ZT0lClXjl5tzFl35naHty/EyLoXCAqQ2L2Oaqu8GBa7agAvtv7hDjpKUBYjjZvjJ4IcDbdnzRexC82F1hVgpQJnpWB4NepV9Ne12FbeibY0IlPu+OgxnDUd1lGVlANks28buXQCS7F3nM9RxVD15MAQPfery8l2Bl0jH8kkHISMcby6wUCmlyGgdn2d3aUkqk5Qmy4nPEtTYGKSxXLGjmJR9A0P29C82TDMZz4JtkHQUtLK9G1k9V4FjwPhFkigABnKpedMvd7/lqLeL5UnIfdV2Qd3Ptar35AH/V1AXjr2Dl+L4+CgfbkKZRI741Q8RL4e533Z8L9ApJVHnM6C0KDwJoX8eYLW1eGDdDQo2LuDs4/CqO3ky2nuiVRzvutOXEoZ9MJ+oCNHy/PBurZOf2UzvdJxQRwm+kS+NT56k/qQcRqh93Ir10tanXU4B4fy3a2exN4TKYI/JJs789B192StSWndQ9FtLsnGvA2Gw== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230031)(36860700004)(46966006)(40470700004);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2024 17:17:21.7377 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0186f5ac-8e33-4c3d-0671-08dc3237ccae X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF000044A5.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6430 From: "Gautham R. Shenoy" Problem statement ================= When measuring IPI throughput using a modified version of Anton Blanchard's ipistorm benchmark [1], configured to measure time taken to perform a fixed number of smp_call_function_single() (with wait set to 1), an increase in benchmark time was observed between v5.7 and the upstream kernel (v6.7-rc6). Bisection pointed to commit b2a02fc43a1f ("smp: Optimize send_call_function_single_ipi()") as the reason behind this increase in runtime. Reverting the optimization introduced by the above commit fixed the regression in ipistorm, however benchmarks like tbench and netperf regressed with the revert, supporting the validity of the optimization. Following are the benchmark results on top of tip:sched/core with the optimization reverted on a dual socket 3rd Generation aMD EPYC system (2 x 64C/128T) running with boost enabled and C2 disabled: (tip:sched/core at tag "sched-core-2024-01-08" for all the testing done below) ================================================================== Test : ipistorm (modified) Units : Normalized runtime Interpretation: Lower is better Statistic : AMean cmdline : insmod ipistorm.ko numipi=100000 single=1 offset=8 cpulist=8 wait=1 ================================================================== kernel: time [pct imp] tip:sched/core 1.00 [0.00] tip:sched/core + revert 0.81 [19.36] ================================================================== Test : tbench Units : Normalized throughput Interpretation: Higher is better Statistic : AMean ================================================================== Clients: tip[pct imp](CV) revert[pct imp](CV) 1 1.00 [ 0.00]( 0.24) 0.91 [ -8.96]( 0.30) 2 1.00 [ 0.00]( 0.25) 0.92 [ -8.20]( 0.97) 4 1.00 [ 0.00]( 0.23) 0.91 [ -9.20]( 1.75) 8 1.00 [ 0.00]( 0.69) 0.91 [ -9.48]( 1.56) 16 1.00 [ 0.00]( 0.66) 0.92 [ -8.49]( 2.43) 32 1.00 [ 0.00]( 0.96) 0.89 [-11.13]( 0.96) 64 1.00 [ 0.00]( 1.06) 0.90 [ -9.72]( 2.49) 128 1.00 [ 0.00]( 0.70) 0.92 [ -8.36]( 1.26) 256 1.00 [ 0.00]( 0.72) 0.97 [ -3.30]( 1.10) 512 1.00 [ 0.00]( 0.42) 0.98 [ -1.73]( 0.37) 1024 1.00 [ 0.00]( 0.28) 0.99 [ -1.39]( 0.43) ================================================================== Test : netperf Units : Normalized Througput Interpretation: Higher is better Statistic : AMean ================================================================== Clients: tip[pct imp](CV) revert[pct imp](CV) 1-clients 1.00 [ 0.00]( 0.50) 0.89 [-10.51]( 0.20) 2-clients 1.00 [ 0.00]( 1.16) 0.89 [-11.10]( 0.59) 4-clients 1.00 [ 0.00]( 1.03) 0.89 [-10.68]( 0.38) 8-clients 1.00 [ 0.00]( 0.99) 0.89 [-10.54]( 0.50) 16-clients 1.00 [ 0.00]( 0.87) 0.89 [-10.92]( 0.95) 32-clients 1.00 [ 0.00]( 1.24) 0.89 [-10.85]( 0.63) 64-clients 1.00 [ 0.00]( 1.58) 0.90 [-10.11]( 1.18) 128-clients 1.00 [ 0.00]( 0.87) 0.89 [-10.94]( 1.11) 256-clients 1.00 [ 0.00]( 4.77) 1.00 [ -0.16]( 3.45) 512-clients 1.00 [ 0.00](56.16) 1.02 [ 2.10](56.05) Since a simple revert is not a viable solution, the changes in the code path of call_function_single_prep_ipi(), with and without the optimization were audited to better understand the effect of the commit. Effects of call_function_single_prep_ipi() ========================================== To pull a TIF_POLLING thread out of idle to process an IPI, the sender sets the TIF_NEED_RESCHED bit in the idle task's thread info in call_function_single_prep_ipi() and avoids sending an actual IPI to the target. As a result, the scheduler expects a task to be enqueued when exiting the idle path. This is not the case with non-polling idle states where the idle CPU exits the non-polling idle state to process the interrupt, and since need_resched() returns false, soon goes back to idle again. When TIF_NEED_RESCHED flag is set, do_idle() will call schedule_idle(), a large part of which runs with local IRQ disabled. In case of ipistorm, when measuring IPI throughput, this large IRQ disabled section delays processing of IPIs. Further auditing revealed that in absence of any runnable tasks, pick_next_task_fair(), which is called from the pick_next_task() fast path, will always call newidle_balance() in this scenario, further increasing the time spent in the IRQ disabled section. Following is the crude visualization of the problem with relevant functions expanded: --- CPU0 CPU1 ==== ==== do_idle() { __current_set_polling(); ... monitor(addr); if (!need_resched()) { mwait() { /* Waiting */ smp_call_function_single(CPU1, func, wait = 1) { ... ... ... set_nr_if_polling(CPU1) { ... /* Realizes CPU1 is polling */ ... try_cmpxchg(addr, ... &val, ... val | _TIF_NEED_RESCHED); ... } /* Does not send an IPI */ ... ... } /* mwait exit due to write at addr */ csd_lock_wait() { } /* Waiting */ preempt_set_need_resched(); ... __current_clr_polling(); ... flush_smp_call_function_queue() { ... func(); } /* End of wait */ } } schedule_idle() { ... local_irq_disable(); smp_call_function_single(CPU1, func, wait = 1) { ... ... ... arch_send_call_function_single_ipi(CPU1); ... \ ... \ newidle_balance() { \ ... /* Delay */ ... \ } \ ... \--------------> local_irq_enable(); /* Processes the IPI */ -- Skipping newidle_balance() ========================== In an earlier attempt to solve the challenge of the long IRQ disabled section, newidle_balance() was skipped when a CPU waking up from idle was found to have no runnable tasks, and was transitioning back to idle [2]. Tim [3] and David [4] had pointed out that newidle_balance() may be viable for CPUs that are idling with tick enabled, where the newidle_balance() has the opportunity to pull tasks onto the idle CPU. Vincent [5] pointed out a case where the idle load kick will fail to run on an idle CPU since the IPI handler launching the ILB will check for need_resched(). In such cases, the idle CPU relies on newidle_balance() to pull tasks towards itself. Using an alternate flag instead of NEED_RESCHED to indicate a pending IPI was suggested as the correct approach to solve this problem on the same thread. Proposed solution: TIF_NOTIFY_IPI ================================= Instead of reusing TIF_NEED_RESCHED bit to pull an TIF_POLLING CPU out of idle, TIF_NOTIFY_IPI is a newly introduced flag that call_function_single_prep_ipi() sets on a target TIF_POLLING CPU to indicate a pending IPI, which the idle CPU promises to process soon. On architectures that do not support the TIF_NOTIFY_IPI flag, call_function_single_prep_ipi() will fallback to setting TIF_NEED_RESCHED bit to pull the TIF_POLLING CPU out of idle. Since the pending IPI handlers are processed before the call to schedule_idle() in do_idle(), schedule_idle() will only be called if the IPI handler have woken / migrated a new task on the idle CPU and has set TIF_NEED_RESCHED bit to indicate the same. This avoids running into the long IRQ disabled section in schedule_idle() unnecessarily, and any need_resched() check within a call function will accurately notify if a task is waiting for CPU time on the CPU handling the IPI. Following is the crude visualization of how the situation changes with the newly introduced TIF_NOTIFY_IPI flag: -- CPU0 CPU1 ==== ==== do_idle() { __current_set_polling(); ... monitor(addr); if (!need_resched_or_ipi()) { mwait() { /* Waiting */ smp_call_function_single(CPU1, func, wait = 1) { ... ... ... set_nr_if_polling(CPU1) { ... /* Realizes CPU1 is polling */ ... try_cmpxchg(addr, ... &val, ... val | _TIF_NOTIFY_IPI); ... } /* Does not send an IPI */ ... ... } /* mwait exit due to write at addr */ csd_lock_wait() { } /* Waiting */ preempt_fold_need_resched(); /* fold if NEED_RESCHED */ ... __current_clr_polling(); ... flush_smp_call_function_queue() { ... func(); /* Will set NEED_RESCHED if sched_ttwu_pending() */ } /* End of wait */ } } if (need_resched()) { schedule_idle(); smp_call_function_single(CPU1, func, wait = 1) { } ... ... /* IRQs remain enabled */ arch_send_call_function_single_ipi(CPU1); -----------> /* Processes the IPI */ -- Results ======= With the TIF_NOTIFY_IPI, the time taken to complete a fixed set of IPIs using ipistorm improves drastically. Following are the numbers from the same dual socket 3rd Generation EPYC system (2 x 64C/128T) (boost on, C2 disabled) running ipistorm between CPU8 and CPU16: cmdline: insmod ipistorm.ko numipi=100000 single=1 offset=8 cpulist=8 wait=1 ================================================================== Test : ipistorm (modified) Units : Normalized runtime Interpretation: Lower is better Statistic : AMean ================================================================== kernel: time [pct imp] tip:sched/core 1.00 [0.00] tip:sched/core + revert 0.81 [19.36] tip:sched/core + TIF_NOTIFY_IPI 0.20 [80.99] Same experiment was repeated on an dual socket ARM server (2 x 64C) which too saw a significant improvement in the ipistorm performance: ================================================================== Test : ipistorm (modified) Units : Normalized runtime Interpretation: Lower is better Statistic : AMean ================================================================== kernel: time [pct imp] tip:sched/core 1.00 [0.00] tip:sched/core + TIF_NOTIFY_IPI 0.41 [59.29] netperf and tbench results with the patch match the results on tip on the dual socket 3rd Generation AMD system (2 x 64C/128T). Additionally, hackbench, stream, and schbench too were tested, with results from the patched kernel matching that of the tip. [ prateek: Split the changes into a separate patch, added the TIF_NEED_RESCHED optimization in notify_ipi_if_polling(). TIF_WAKE_FLAG macro, commit log ] Link: https://github.com/antonblanchard/ipistorm [1] Link: https://lore.kernel.org/lkml/20240119084548.2788-1-kprateek.nayak@amd.com/ [2] Link: https://lore.kernel.org/lkml/b4f5ac150685456cf45a342e3bb1f28cdd557a53.camel@linux.intel.com/ [3] Link: https://lore.kernel.org/lkml/20240123211756.GA221793@maniforge/ [4] Link: https://lore.kernel.org/lkml/CAKfTPtC446Lo9CATPp7PExdkLhHQFoBuY-JMGC7agOHY4hs-Pw@mail.gmail.com/ [5] Cc: Richard Henderson Cc: Ivan Kokshaysky Cc: Matt Turner Cc: Russell King Cc: Guo Ren Cc: Michal Simek Cc: Dinh Nguyen Cc: Jonas Bonn Cc: Stefan Kristiansson Cc: Stafford Horne Cc: "James E.J. Bottomley" Cc: Helge Deller Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: "Aneesh Kumar K.V" Cc: "Naveen N. Rao" Cc: Yoshinori Sato Cc: Rich Felker Cc: John Paul Adrian Glaubitz Cc: "David S. Miller" Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: "Rafael J. Wysocki" Cc: Daniel Lezcano Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Steven Rostedt Cc: Ben Segall Cc: Mel Gorman Cc: Daniel Bristot de Oliveira Cc: Valentin Schneider Cc: Al Viro Cc: Linus Walleij Cc: Ard Biesheuvel Cc: Andrew Donnellan Cc: Nicholas Miehlbradt Cc: Andrew Morton Cc: Arnd Bergmann Cc: Josh Poimboeuf Cc: "Kirill A. Shutemov" Cc: Rick Edgecombe Cc: Tony Battersby Cc: Brian Gerst Cc: Tim Chen Cc: David Vernet Cc: x86@kernel.org Cc: linux-kernel@vger.kernel.org Cc: linux-alpha@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org Cc: linux-csky@vger.kernel.org Cc: linux-openrisc@vger.kernel.org Cc: linux-parisc@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org Cc: linux-sh@vger.kernel.org Cc: sparclinux@vger.kernel.org Cc: linux-pm@vger.kernel.org Signed-off-by: Gautham R. Shenoy Co-developed-by: K Prateek Nayak Signed-off-by: K Prateek Nayak --- include/linux/sched/idle.h | 8 ++++---- kernel/sched/core.c | 41 ++++++++++++++++++++++++++++++-------- kernel/sched/idle.c | 16 +++++++++++---- 3 files changed, 49 insertions(+), 16 deletions(-) diff --git a/include/linux/sched/idle.h b/include/linux/sched/idle.h index d739ab810e00..c22312087c30 100644 --- a/include/linux/sched/idle.h +++ b/include/linux/sched/idle.h @@ -58,8 +58,8 @@ static __always_inline bool __must_check current_set_polling_and_test(void) __current_set_polling(); /* - * Polling state must be visible before we test NEED_RESCHED, - * paired by resched_curr() + * Polling state must be visible before we test NEED_RESCHED or + * NOTIFY_IPI paired by resched_curr() or notify_ipi_if_polling() */ smp_mb__after_atomic(); @@ -71,8 +71,8 @@ static __always_inline bool __must_check current_clr_polling_and_test(void) __current_clr_polling(); /* - * Polling state must be visible before we test NEED_RESCHED, - * paired by resched_curr() + * Polling state must be visible before we test NEED_RESCHED or + * NOTIFY_IPI paired by resched_curr() or notify_ipi_if_polling() */ smp_mb__after_atomic(); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index db4be4921e7f..6fb6e5b75724 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -909,12 +909,30 @@ static inline bool set_nr_and_not_polling(struct task_struct *p) } /* - * Atomically set TIF_NEED_RESCHED if TIF_POLLING_NRFLAG is set. + * Certain architectures that support TIF_POLLING_NRFLAG may not support + * TIF_NOTIFY_IPI to notify an idle CPU in TIF_POLLING mode of a pending + * IPI. On such architectures, set TIF_NEED_RESCHED instead to wake the + * idle CPU and process the pending IPI. + */ +#ifdef _TIF_NOTIFY_IPI +#define _TIF_WAKE_FLAG _TIF_NOTIFY_IPI +#else +#define _TIF_WAKE_FLAG _TIF_NEED_RESCHED +#endif + +/* + * Atomically set TIF_WAKE_FLAG when TIF_POLLING_NRFLAG is set. + * + * On architectures that define TIF_NOTIFY_IPI, the same is set in the + * idle task's thread_info to pull the CPU out of idle and process + * the pending interrupt. On architectures that don't support + * TIF_NOTIFY_IPI, TIF_NEED_RESCHED is set instead to notify the + * pending IPI. * - * If this returns true, then the idle task promises to call - * sched_ttwu_pending() and reschedule soon. + * If this returns true, then the idle task promises to process the + * call function soon. */ -static bool set_nr_if_polling(struct task_struct *p) +static bool notify_ipi_if_polling(struct task_struct *p) { struct thread_info *ti = task_thread_info(p); typeof(ti->flags) val = READ_ONCE(ti->flags); @@ -922,9 +940,16 @@ static bool set_nr_if_polling(struct task_struct *p) do { if (!(val & _TIF_POLLING_NRFLAG)) return false; - if (val & _TIF_NEED_RESCHED) + /* + * If TIF_NEED_RESCHED flag is set in addition to + * TIF_POLLING_NRFLAG, the CPU will soon fall out of + * idle. Since flush_smp_call_function_queue() is called + * soon after the idle exit, setting TIF_WAKE_FLAG is + * not necessary. + */ + if (val & (_TIF_NEED_RESCHED | _TIF_WAKE_FLAG)) return true; - } while (!try_cmpxchg(&ti->flags, &val, val | _TIF_NEED_RESCHED)); + } while (!try_cmpxchg(&ti->flags, &val, val | _TIF_WAKE_FLAG)); return true; } @@ -937,7 +962,7 @@ static inline bool set_nr_and_not_polling(struct task_struct *p) } #ifdef CONFIG_SMP -static inline bool set_nr_if_polling(struct task_struct *p) +static inline bool notify_ipi_if_polling(struct task_struct *p) { return false; } @@ -3918,7 +3943,7 @@ void sched_ttwu_pending(void *arg) */ bool call_function_single_prep_ipi(int cpu) { - if (set_nr_if_polling(cpu_rq(cpu)->idle)) { + if (notify_ipi_if_polling(cpu_rq(cpu)->idle)) { trace_sched_wake_idle_without_ipi(cpu); return false; } diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c index fcc734f45a2a..b91dc1f62a56 100644 --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -315,13 +315,13 @@ static void do_idle(void) } /* - * Since we fell out of the loop above, we know TIF_NEED_RESCHED must - * be set, propagate it into PREEMPT_NEED_RESCHED. + * Since we fell out of the loop above, TIF_NEED_RESCHED may be set. + * Propagate it into PREEMPT_NEED_RESCHED. * * This is required because for polling idle loops we will not have had * an IPI to fold the state for us. */ - preempt_set_need_resched(); + preempt_fold_need_resched(); tick_nohz_idle_exit(); __current_clr_polling(); @@ -338,7 +338,15 @@ static void do_idle(void) */ current_clr_notify_ipi(); flush_smp_call_function_queue(); - schedule_idle(); + + /* + * When NEED_RESCHED is set, the idle thread promises to call + * schedule_idle(). schedule_idle() can be skipped when an idle CPU + * was woken up to process an IPI that does not queue a task on the + * idle CPU, facilitating faster idle re-entry. + */ + if (need_resched()) + schedule_idle(); if (unlikely(klp_patch_pending(current))) klp_update_patch_state(current);