From patchwork Wed Jul 31 07:56:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ritesh Harjani (IBM)" X-Patchwork-Id: 1966917 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=HslJSdTH; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WYkwD0Nczz1ybb for ; Wed, 31 Jul 2024 17:57:32 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=HslJSdTH; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WYkwC1QkRz3cmk for ; Wed, 31 Jul 2024 17:57:31 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=HslJSdTH; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::433; helo=mail-pf1-x433.google.com; envelope-from=ritesh.list@gmail.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WYkvN3bG5z3dBs for ; Wed, 31 Jul 2024 17:56:48 +1000 (AEST) Received: by mail-pf1-x433.google.com with SMTP id d2e1a72fcca58-70d2ae44790so3814009b3a.2 for ; Wed, 31 Jul 2024 00:56:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722412603; x=1723017403; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HbfN1xqNnjayEqbAepIgV5lEDapyyBy+LBsW9BWPpLw=; b=HslJSdTHKhYrs3BRuIo/cgmhJGE2RFohRVqr4vdrEuM5GavG+PgEdjUWZLnJMTNX2T JP63ENbjMpjXzzUbvviXLm27XKEaAyyOdURL9V6G3SujwwMIZtZAdeMHwYWUGQ3c3yZV C9acBHgH30reJPkvXgy4WwlQ/D1TiaXZA7Qq5Wp2lgfrG3mnBNPV9St50KqVvEltzaZG DLbQxfedHa22E1i/Y+4eR7U+RxiDhDoeMr/JROqBX+4DieNrVTlGok1tLeM0R9g4FFSL Z9cTDikYWh4hDjd8BU2rMbRm72S+6ylxaBN6vM41DczKWjUjm+KYwvVGdG8CIBf1RA0H hGpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722412603; x=1723017403; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HbfN1xqNnjayEqbAepIgV5lEDapyyBy+LBsW9BWPpLw=; b=iQIWmKy9BHC69BBkpi7vcI8w9iWL3ctho6J0Llqd1FD/2hoKxJg2DC4mxBCdOAUX6Y o2nfIRUccuLJMx9epvuhVH1kJPL4LkcDMVS/suYriXehloADK0kII9TGsOd7tk+Kp/2M z79rdmBdq+CG/TnR3UJ3seQY4Oci4NZs7DQ9sBEU1wrkujGin7ojdITdeUX8ByLMW51z 9uzcpHY6FLNyjDXxoSqP4z/nErY3RleF//FysEu8+dAcKzn4Ngl4ku82T38EDF+FcDu5 ZqzSkagEHj1Ex12YCqf9r1HDeA/AEfmowf1QWu3/0JF4S3xgf5gcYLYSHlKMmzCky+UL s2Ag== X-Gm-Message-State: AOJu0Yxf9oInFwbHxVqMK8qnRBCJMGVDSrqnxCM4Z3JKBfbAqyTv42Ih po9TwyoHsvffxSD1UY60Zai2/MmlKyNypoaNm4zi7tYWaTf/TSS+f74rJQ== X-Google-Smtp-Source: AGHT+IHLMpw4Jt1eipQva3mgTiVMENiX1gmC3joMUjc75SQ0Lae2I151M27hDR+kDNlQYxT/2VxHAw== X-Received: by 2002:a05:6a00:1891:b0:705:bd9d:a7c0 with SMTP id d2e1a72fcca58-70ecea14176mr12965846b3a.8.1722412602412; Wed, 31 Jul 2024 00:56:42 -0700 (PDT) Received: from dw-tp.ibmuc.com ([171.76.83.154]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70ead6e16fbsm9788372b3a.19.2024.07.31.00.56.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:56:41 -0700 (PDT) From: "Ritesh Harjani (IBM)" To: linuxppc-dev Subject: [RFC v1 01/10] book3s64/hash: Remove kfence support temporarily Date: Wed, 31 Jul 2024 13:26:05 +0530 Message-ID: X-Mailer: git-send-email 2.45.2 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Donet Tom , Madhavan Srinivasan , "Ritesh Harjani \(IBM\)" , Christophe Leroy , Pavithra Prakash , "Aneesh Kumar K . V" , Nicholas Piggin , Hari Bathini Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Kfence on book3s Hash on pseries is anyways broken. It fails to boot due to RMA size limitation. That is because, kfence with Hash uses debug_pagealloc infrastructure. debug_pagealloc allocates linear map for entire dram size instead of just kfence relevant objects. This means for 16TB of DRAM it will require (16TB >> PAGE_SHIFT) which is 256MB which is half of RMA region on P8. crash kernel reserves 256MB and we also need 2048 * 16KB * 3 for emergency stack and some more for paca allocations. That means there is not enough memory for reserving the full linear map in the RMA region, if the DRAM size is too big (>=16TB) (The issue is seen above 8TB with crash kernel 256 MB reservation). Now Kfence does not require linear memory map for entire DRAM. It only needs for kfence objects. So this patch temporarily removes the kfence functionality since debug_pagealloc code needs some refactoring. We will bring in kfence on Hash support in later patches. Signed-off-by: Ritesh Harjani (IBM) --- arch/powerpc/include/asm/kfence.h | 5 +++++ arch/powerpc/mm/book3s64/hash_utils.c | 16 +++++++++++----- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/arch/powerpc/include/asm/kfence.h b/arch/powerpc/include/asm/kfence.h index fab124ada1c7..f3a9476a71b3 100644 --- a/arch/powerpc/include/asm/kfence.h +++ b/arch/powerpc/include/asm/kfence.h @@ -10,6 +10,7 @@ #include #include +#include #ifdef CONFIG_PPC64_ELF_ABI_V1 #define ARCH_FUNC_PREFIX "." @@ -25,6 +26,10 @@ static inline void disable_kfence(void) static inline bool arch_kfence_init_pool(void) { +#ifdef CONFIG_PPC64 + if (!radix_enabled()) + return false; +#endif return !kfence_disabled; } #endif diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index 01c3b4b65241..1a1b50735fa0 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -431,7 +431,7 @@ int htab_bolt_mapping(unsigned long vstart, unsigned long vend, break; cond_resched(); - if (debug_pagealloc_enabled_or_kfence() && + if (debug_pagealloc_enabled() && (paddr >> PAGE_SHIFT) < linear_map_hash_count) linear_map_hash_slots[paddr >> PAGE_SHIFT] = ret | 0x80; } @@ -814,7 +814,7 @@ static void __init htab_init_page_sizes(void) bool aligned = true; init_hpte_page_sizes(); - if (!debug_pagealloc_enabled_or_kfence()) { + if (!debug_pagealloc_enabled()) { /* * Pick a size for the linear mapping. Currently, we only * support 16M, 1M and 4K which is the default @@ -1134,7 +1134,7 @@ static void __init htab_initialize(void) prot = pgprot_val(PAGE_KERNEL); - if (debug_pagealloc_enabled_or_kfence()) { + if (debug_pagealloc_enabled()) { linear_map_hash_count = memblock_end_of_DRAM() >> PAGE_SHIFT; linear_map_hash_slots = memblock_alloc_try_nid( linear_map_hash_count, 1, MEMBLOCK_LOW_LIMIT, @@ -2117,7 +2117,7 @@ void hpt_do_stress(unsigned long ea, unsigned long hpte_group) } } -#if defined(CONFIG_DEBUG_PAGEALLOC) || defined(CONFIG_KFENCE) +#if defined(CONFIG_DEBUG_PAGEALLOC) static DEFINE_RAW_SPINLOCK(linear_map_hash_lock); static void kernel_map_linear_page(unsigned long vaddr, unsigned long lmi) @@ -2191,7 +2191,13 @@ int hash__kernel_map_pages(struct page *page, int numpages, int enable) local_irq_restore(flags); return 0; } -#endif /* CONFIG_DEBUG_PAGEALLOC || CONFIG_KFENCE */ +#else /* CONFIG_DEBUG_PAGEALLOC */ +int hash__kernel_map_pages(struct page *page, int numpages, + int enable) +{ + return 0; +} +#endif /* CONFIG_DEBUG_PAGEALLOC */ void hash__setup_initial_memory_limit(phys_addr_t first_memblock_base, phys_addr_t first_memblock_size) From patchwork Wed Jul 31 07:56:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ritesh Harjani (IBM)" X-Patchwork-Id: 1966918 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=BtcJnGcn; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WYkwz5Zslz1ybb for ; Wed, 31 Jul 2024 17:58:11 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=BtcJnGcn; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WYkwz1ssVz3d4F for ; Wed, 31 Jul 2024 17:58:11 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=BtcJnGcn; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::436; helo=mail-pf1-x436.google.com; envelope-from=ritesh.list@gmail.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WYkvV1P8Tz3dBq for ; Wed, 31 Jul 2024 17:56:53 +1000 (AEST) Received: by mail-pf1-x436.google.com with SMTP id d2e1a72fcca58-70d18d4b94cso3980788b3a.2 for ; Wed, 31 Jul 2024 00:56:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722412611; x=1723017411; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=65ywP7/O6fQAO9Uxn3NOVkm8J4vsqs8B0sDGst0fbN0=; b=BtcJnGcnpQPb57got4A+Ny4BPxlw4wN6SL6V+r9xifZ8Z1l9cmV/wz9N+ZVQdli8BL LSGc6ntHRvVfwrRRhmuy2iTdG3k8ucTrgqBO3RZm5TRx1f+kCj00YTdYWeWCI3B3ibun yLIwg9Uc9Q6fVdxG7Wc6coJqEa9IfrA0KasPTch6Vz8wb4eKXUzaHM4WLelmdWAMtvME 7h4MUv1G/gXIlVE2eZ8rkQpgOi6tgWHmROsRkT9xeoLVLRzWlp3RocAR++GfICXSYe9E acgiXHkEE9z98rKjZ9FAsgqX6kSVYkinGo7j7G7DZKHkuMDuiKmahytMtSR53Fovg9Vk U0vA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722412611; x=1723017411; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=65ywP7/O6fQAO9Uxn3NOVkm8J4vsqs8B0sDGst0fbN0=; b=nYklKOLkn/2LjbLqfc7LVtutUEzgsjxKIw9WdheiKKCSYRh0MBOpsVTW0fmGsLcNU1 a9ubT5FYqecsGuVkY0t5WpbJVpUq20Yb2BMjCtN+zDtmXP1RC3AmFJzmux7osy75y/kU GxoQ5pLtPkdio4PmfuJ7FjkBte8ECtQhV8w3CnGoLZj2E5d57B+1IIjrd3Hp5Rzw17Hj QcditxXrwawpQmL6cy8XJ9Ig8fwnXYjhaYmgBxWXGx+Z4XZ+El1vjvaX9cbTjZA9OPja aMHPJ9DSF7LOcNDtHjnkkwm5iqci8iiwMm1jGSkPXUqHBi0tD/POak9JHpHX3t+TBlCc 0TZg== X-Gm-Message-State: AOJu0YwW5/l87SUG2UWRi6Jpug9XnzR+8g5Vqq2pYRcLtETaOj/Vakv/ 6nZBSQgqtPuRNMJ8rlnJ56ecHWQm0jZPn6qLekYD7+d15qRgI59ZYg/1zA== X-Google-Smtp-Source: AGHT+IFWuWgg8/OtIwgco2BJ+kY4lwdnIJJmdcufARbpnK82Rfv+NqBTW6bkkEKQP2wPKcclw0fhHQ== X-Received: by 2002:a05:6a21:3992:b0:1c4:a55b:8146 with SMTP id adf61e73a8af0-1c4a55ba9e3mr11835793637.26.1722412610683; Wed, 31 Jul 2024 00:56:50 -0700 (PDT) Received: from dw-tp.ibmuc.com ([171.76.83.154]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70ead6e16fbsm9788372b3a.19.2024.07.31.00.56.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:56:50 -0700 (PDT) From: "Ritesh Harjani (IBM)" To: linuxppc-dev Subject: [RFC v1 02/10] book3s64/hash: Refactor kernel linear map related calls Date: Wed, 31 Jul 2024 13:26:06 +0530 Message-ID: <3e764710d8097965b00fc2ed7b54f4f0e1487781.1722408881.git.ritesh.list@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Donet Tom , Madhavan Srinivasan , "Ritesh Harjani \(IBM\)" , Christophe Leroy , Pavithra Prakash , "Aneesh Kumar K . V" , Nicholas Piggin , Hari Bathini Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This just brings all linear map related handling at one place instead of having those functions scattered in hash_utils file. Makes it easy for review. No functionality changes in this patch. Signed-off-by: Ritesh Harjani (IBM) --- arch/powerpc/mm/book3s64/hash_utils.c | 164 +++++++++++++------------- 1 file changed, 82 insertions(+), 82 deletions(-) diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index 1a1b50735fa0..b6ae955971bf 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -273,6 +273,88 @@ void hash__tlbiel_all(unsigned int action) WARN(1, "%s called on pre-POWER7 CPU\n", __func__); } +#if defined(CONFIG_DEBUG_PAGEALLOC) +static DEFINE_RAW_SPINLOCK(linear_map_hash_lock); + +static void kernel_map_linear_page(unsigned long vaddr, unsigned long lmi) +{ + unsigned long hash; + unsigned long vsid = get_kernel_vsid(vaddr, mmu_kernel_ssize); + unsigned long vpn = hpt_vpn(vaddr, vsid, mmu_kernel_ssize); + unsigned long mode = htab_convert_pte_flags(pgprot_val(PAGE_KERNEL), HPTE_USE_KERNEL_KEY); + long ret; + + hash = hpt_hash(vpn, PAGE_SHIFT, mmu_kernel_ssize); + + /* Don't create HPTE entries for bad address */ + if (!vsid) + return; + + if (linear_map_hash_slots[lmi] & 0x80) + return; + + ret = hpte_insert_repeating(hash, vpn, __pa(vaddr), mode, + HPTE_V_BOLTED, + mmu_linear_psize, mmu_kernel_ssize); + + BUG_ON (ret < 0); + raw_spin_lock(&linear_map_hash_lock); + BUG_ON(linear_map_hash_slots[lmi] & 0x80); + linear_map_hash_slots[lmi] = ret | 0x80; + raw_spin_unlock(&linear_map_hash_lock); +} + +static void kernel_unmap_linear_page(unsigned long vaddr, unsigned long lmi) +{ + unsigned long hash, hidx, slot; + unsigned long vsid = get_kernel_vsid(vaddr, mmu_kernel_ssize); + unsigned long vpn = hpt_vpn(vaddr, vsid, mmu_kernel_ssize); + + hash = hpt_hash(vpn, PAGE_SHIFT, mmu_kernel_ssize); + raw_spin_lock(&linear_map_hash_lock); + if (!(linear_map_hash_slots[lmi] & 0x80)) { + raw_spin_unlock(&linear_map_hash_lock); + return; + } + hidx = linear_map_hash_slots[lmi] & 0x7f; + linear_map_hash_slots[lmi] = 0; + raw_spin_unlock(&linear_map_hash_lock); + if (hidx & _PTEIDX_SECONDARY) + hash = ~hash; + slot = (hash & htab_hash_mask) * HPTES_PER_GROUP; + slot += hidx & _PTEIDX_GROUP_IX; + mmu_hash_ops.hpte_invalidate(slot, vpn, mmu_linear_psize, + mmu_linear_psize, + mmu_kernel_ssize, 0); +} + +int hash__kernel_map_pages(struct page *page, int numpages, int enable) +{ + unsigned long flags, vaddr, lmi; + int i; + + local_irq_save(flags); + for (i = 0; i < numpages; i++, page++) { + vaddr = (unsigned long)page_address(page); + lmi = __pa(vaddr) >> PAGE_SHIFT; + if (lmi >= linear_map_hash_count) + continue; + if (enable) + kernel_map_linear_page(vaddr, lmi); + else + kernel_unmap_linear_page(vaddr, lmi); + } + local_irq_restore(flags); + return 0; +} +#else /* CONFIG_DEBUG_PAGEALLOC */ +int hash__kernel_map_pages(struct page *page, int numpages, + int enable) +{ + return 0; +} +#endif /* CONFIG_DEBUG_PAGEALLOC */ + /* * 'R' and 'C' update notes: * - Under pHyp or KVM, the updatepp path will not set C, thus it *will* @@ -2117,88 +2199,6 @@ void hpt_do_stress(unsigned long ea, unsigned long hpte_group) } } -#if defined(CONFIG_DEBUG_PAGEALLOC) -static DEFINE_RAW_SPINLOCK(linear_map_hash_lock); - -static void kernel_map_linear_page(unsigned long vaddr, unsigned long lmi) -{ - unsigned long hash; - unsigned long vsid = get_kernel_vsid(vaddr, mmu_kernel_ssize); - unsigned long vpn = hpt_vpn(vaddr, vsid, mmu_kernel_ssize); - unsigned long mode = htab_convert_pte_flags(pgprot_val(PAGE_KERNEL), HPTE_USE_KERNEL_KEY); - long ret; - - hash = hpt_hash(vpn, PAGE_SHIFT, mmu_kernel_ssize); - - /* Don't create HPTE entries for bad address */ - if (!vsid) - return; - - if (linear_map_hash_slots[lmi] & 0x80) - return; - - ret = hpte_insert_repeating(hash, vpn, __pa(vaddr), mode, - HPTE_V_BOLTED, - mmu_linear_psize, mmu_kernel_ssize); - - BUG_ON (ret < 0); - raw_spin_lock(&linear_map_hash_lock); - BUG_ON(linear_map_hash_slots[lmi] & 0x80); - linear_map_hash_slots[lmi] = ret | 0x80; - raw_spin_unlock(&linear_map_hash_lock); -} - -static void kernel_unmap_linear_page(unsigned long vaddr, unsigned long lmi) -{ - unsigned long hash, hidx, slot; - unsigned long vsid = get_kernel_vsid(vaddr, mmu_kernel_ssize); - unsigned long vpn = hpt_vpn(vaddr, vsid, mmu_kernel_ssize); - - hash = hpt_hash(vpn, PAGE_SHIFT, mmu_kernel_ssize); - raw_spin_lock(&linear_map_hash_lock); - if (!(linear_map_hash_slots[lmi] & 0x80)) { - raw_spin_unlock(&linear_map_hash_lock); - return; - } - hidx = linear_map_hash_slots[lmi] & 0x7f; - linear_map_hash_slots[lmi] = 0; - raw_spin_unlock(&linear_map_hash_lock); - if (hidx & _PTEIDX_SECONDARY) - hash = ~hash; - slot = (hash & htab_hash_mask) * HPTES_PER_GROUP; - slot += hidx & _PTEIDX_GROUP_IX; - mmu_hash_ops.hpte_invalidate(slot, vpn, mmu_linear_psize, - mmu_linear_psize, - mmu_kernel_ssize, 0); -} - -int hash__kernel_map_pages(struct page *page, int numpages, int enable) -{ - unsigned long flags, vaddr, lmi; - int i; - - local_irq_save(flags); - for (i = 0; i < numpages; i++, page++) { - vaddr = (unsigned long)page_address(page); - lmi = __pa(vaddr) >> PAGE_SHIFT; - if (lmi >= linear_map_hash_count) - continue; - if (enable) - kernel_map_linear_page(vaddr, lmi); - else - kernel_unmap_linear_page(vaddr, lmi); - } - local_irq_restore(flags); - return 0; -} -#else /* CONFIG_DEBUG_PAGEALLOC */ -int hash__kernel_map_pages(struct page *page, int numpages, - int enable) -{ - return 0; -} -#endif /* CONFIG_DEBUG_PAGEALLOC */ - void hash__setup_initial_memory_limit(phys_addr_t first_memblock_base, phys_addr_t first_memblock_size) { From patchwork Wed Jul 31 07:56:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ritesh Harjani (IBM)" X-Patchwork-Id: 1966919 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=Jjps5A7Q; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WYkxt6LGTz1yfG for ; Wed, 31 Jul 2024 17:58:58 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=Jjps5A7Q; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WYkxt4mgQz3d32 for ; Wed, 31 Jul 2024 17:58:58 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=Jjps5A7Q; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::533; helo=mail-pg1-x533.google.com; envelope-from=ritesh.list@gmail.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WYkvm2ByDz3d87 for ; Wed, 31 Jul 2024 17:57:08 +1000 (AEST) Received: by mail-pg1-x533.google.com with SMTP id 41be03b00d2f7-7a23fbb372dso3530416a12.0 for ; Wed, 31 Jul 2024 00:57:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722412624; x=1723017424; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vcPszrP5nDAVyyhdxZhwDa2dtlM+ONcznOdsoeX7+5Q=; b=Jjps5A7QftC73zMQvLmQCcRUhA/xGoDrRxH8aOUuxQJFDZw3WbN4QjXFhveFHoyvTg O+sVbmybsemMCtrXmjJdMkwQYNSiB+kNuj2Bh+wj0gRhXOUh0p9J8dfQZurnPaaykhhF yaJapEmzCwaR/aaeLmQVjRQHslPNcuyyrjIgRMe7XrfZhSBa5VuA5NmOL2fJ78BshcWA irpuU4PUKFzr00OWhi0yVvoKTIXYDHm4z1gMfPlE+5UFo0IhP4iVITqYw3TgJauhWX5L KO5eOUTeU23lA3KLjhUprjs4k3JDe6izzLvEpXfQc6KUz/p7oq5dDsUbj2QYSgkY/qAR Re4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722412624; x=1723017424; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vcPszrP5nDAVyyhdxZhwDa2dtlM+ONcznOdsoeX7+5Q=; b=TOggjPtWNXGJ3mgyh3G+dFVIEW+fvZ3qPc2qD6YUo16Ybz6UAnhOMwUeQPv7YhwHKt 5Eyk1xHj1tUEEbROmGpb/EKb08zCseDu1pQZju+2lVOBvTsTQfk0RjhIFNfH35eXSnwx 6hfWFB8xpCRJA07ie7qm6vdfa+1FNezsGni8KqzeaAHRRDMSc0omd6RvxmbCalQK0Uq0 XFWNm9X6gETXcJOF1jU2ifSDe8ASA8DfvfFLjPgEWk0BEFbvRCwkqMoHBKY6Qo+RBQl0 Zh1hTnLysoVZeMLGf30/K2W3oO6m0opXTrpYNXDHu4/cjuF5+suFR9C5xeyIjuPs4BJ6 P9AA== X-Gm-Message-State: AOJu0YwOjxxdsZNfIghlw7Z+/vmSwloXZsuNzspdCROlj4vrRnlmMWZI jP6W/vdwaI2t5btl4ywXvitySrFPqtTVDArjuK4oOL2cn9SOjJQQm8jZLQ== X-Google-Smtp-Source: AGHT+IHXQVX/nkFW9qGUznlg3Vm08dkDE6A6Sz1MOQ6mxgY8IoDHpkrOVkSdXVk9SkR3accdWASViw== X-Received: by 2002:a05:6a20:9c88:b0:1c4:dae8:c72f with SMTP id adf61e73a8af0-1c4dae8c982mr6926811637.19.1722412624282; Wed, 31 Jul 2024 00:57:04 -0700 (PDT) Received: from dw-tp.ibmuc.com ([171.76.83.154]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70ead6e16fbsm9788372b3a.19.2024.07.31.00.56.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:57:03 -0700 (PDT) From: "Ritesh Harjani (IBM)" To: linuxppc-dev Subject: [RFC v1 03/10] book3s64/hash: Add hash_debug_pagealloc_add_slot() function Date: Wed, 31 Jul 2024 13:26:07 +0530 Message-ID: <09f67a7d467ebb1e90fa48a3c6944c1ab5f89f64.1722408881.git.ritesh.list@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Donet Tom , Madhavan Srinivasan , "Ritesh Harjani \(IBM\)" , Christophe Leroy , Pavithra Prakash , "Aneesh Kumar K . V" , Nicholas Piggin , Hari Bathini Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This adds hash_debug_pagealloc_add_slot() function instead of open coding that in htab_bolt_mapping(). This is required since we will be separating kfence functionality to not depend upon debug_pagealloc. No functionality change in this patch. Signed-off-by: Ritesh Harjani (IBM) --- arch/powerpc/mm/book3s64/hash_utils.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index b6ae955971bf..47b40b9b49d6 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -328,6 +328,14 @@ static void kernel_unmap_linear_page(unsigned long vaddr, unsigned long lmi) mmu_kernel_ssize, 0); } +static inline void hash_debug_pagealloc_add_slot(phys_addr_t paddr, int slot) +{ + if (!debug_pagealloc_enabled()) + return; + if ((paddr >> PAGE_SHIFT) < linear_map_hash_count) + linear_map_hash_slots[paddr >> PAGE_SHIFT] = slot | 0x80; +} + int hash__kernel_map_pages(struct page *page, int numpages, int enable) { unsigned long flags, vaddr, lmi; @@ -353,6 +361,7 @@ int hash__kernel_map_pages(struct page *page, int numpages, { return 0; } +static inline void hash_debug_pagealloc_add_slot(phys_addr_t paddr, int slot) {} #endif /* CONFIG_DEBUG_PAGEALLOC */ /* @@ -513,9 +522,7 @@ int htab_bolt_mapping(unsigned long vstart, unsigned long vend, break; cond_resched(); - if (debug_pagealloc_enabled() && - (paddr >> PAGE_SHIFT) < linear_map_hash_count) - linear_map_hash_slots[paddr >> PAGE_SHIFT] = ret | 0x80; + hash_debug_pagealloc_add_slot(paddr, ret); } return ret < 0 ? ret : 0; } From patchwork Wed Jul 31 07:56:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ritesh Harjani (IBM)" X-Patchwork-Id: 1966920 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=lDU8Zqqp; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WYkyg0fL2z1ybb for ; Wed, 31 Jul 2024 17:59:38 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=lDU8Zqqp; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WYkyf3rF7z3d32 for ; Wed, 31 Jul 2024 17:59:38 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=lDU8Zqqp; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::42f; helo=mail-pf1-x42f.google.com; envelope-from=ritesh.list@gmail.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x42f.google.com (mail-pf1-x42f.google.com [IPv6:2607:f8b0:4864:20::42f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WYkvx1l5qz3dBg for ; Wed, 31 Jul 2024 17:57:17 +1000 (AEST) Received: by mail-pf1-x42f.google.com with SMTP id d2e1a72fcca58-70d18112b60so571822b3a.1 for ; Wed, 31 Jul 2024 00:57:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722412634; x=1723017434; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JAZ02YqVGjEzeBr1jmCD9nUbyKKhVBudwW64T2PfzZQ=; b=lDU8Zqqpeg3E9BGsadyWMD4Br5REUGezERt7huT+5BrSipMILT1TjqkvHVj6jVJIuV l1g42dXPA0TOiweljEwGRdSkgHcY/sGdo06ti4/iYejf+D47bXgX9bvraHn5xOimD3vA ANqbysvaTzP5SaP9NvgewM2CwiUZ48sDxiBJetOQ1XfmA/tKE7Y7y4mblKXXJnRHo7Nb uqIm+IzG3CqZUcHTy5AVtiEd8hNdwmYVFP44zkyQ1TkFfAH38NzJ8U/K4feb0gdKD+3w Z+JMFnULWmaQZs8YvnGyZnL8D3s9IhAkXVh5diGbE3bCDeolOi9QPWIU4J21XDxx2SFX IUvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722412634; x=1723017434; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JAZ02YqVGjEzeBr1jmCD9nUbyKKhVBudwW64T2PfzZQ=; b=MOHBWNmR8HFwqXXlN4SXAy2HjSE3o4eOCvOM/8UrAejESvu0fvjKia5f9/jDUbil7x b7xmaPTVMVBF58cvpgu0kcMthekDd0jaEDcEuVLyAztedhxKW/I2hwVNyrhiumlSrdYp hiQqm/tpU6V5QO2JfvBv5Mzs46osV/qZqIpixXN45nzzjTsephwrHp07E/gAMr/sZrg5 ublShwb9iYt56s+p9a0LJG9ikWHtF5b3V7zh0Nj7PNGwmXLvSaAZ9+qbTvy584LtVDo5 NH08wfJzuo1afYY/sVk6mpscPMhRZRsrg6dHwyUdaaPDDh+31xUNzThiGT+3AN25Y5kZ cOug== X-Gm-Message-State: AOJu0YzOQS0kwvnv4tnSwRV6jjusLWwdyuIy71NYjEVYig+dcnwyaHy1 8j5ieEYRDk8kvmKFeIDm5RxmxcUT/tO6kG8obX/SXwdTY5cVoLaDrOMhRg== X-Google-Smtp-Source: AGHT+IEuYQnNGc7Vk7wl6xFoDWB9T4HMFW/wcNm/VzpYsK/KTWIZ5+fIK8RwM9xDJXbF/g9KIYbM+g== X-Received: by 2002:a05:6a00:178f:b0:710:4e4c:a4ba with SMTP id d2e1a72fcca58-7104e4ca553mr1655257b3a.13.1722412634054; Wed, 31 Jul 2024 00:57:14 -0700 (PDT) Received: from dw-tp.ibmuc.com ([171.76.83.154]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70ead6e16fbsm9788372b3a.19.2024.07.31.00.57.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:57:13 -0700 (PDT) From: "Ritesh Harjani (IBM)" To: linuxppc-dev Subject: [RFC v1 04/10] book3s64/hash: Add hash_debug_pagealloc_alloc_slots() function Date: Wed, 31 Jul 2024 13:26:08 +0530 Message-ID: <147c55d66eeeb8de4b7332b4717c85cb24c17418.1722408881.git.ritesh.list@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Donet Tom , Madhavan Srinivasan , "Ritesh Harjani \(IBM\)" , Christophe Leroy , Pavithra Prakash , "Aneesh Kumar K . V" , Nicholas Piggin , Hari Bathini Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This adds hash_debug_pagealloc_alloc_slots() function instead of open coding that in htab_initialize(). This is required since we will be separating the kfence functionality to not depend upon debug_pagealloc. Now that everything required for debug_pagealloc is under a #ifdef config. Bring in linear_map_hash_slots and linear_map_hash_count variables under the same config too. Signed-off-by: Ritesh Harjani (IBM) --- arch/powerpc/mm/book3s64/hash_utils.c | 29 ++++++++++++++++----------- 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index 47b40b9b49d6..6af47b996e79 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -123,8 +123,6 @@ EXPORT_SYMBOL_GPL(mmu_slb_size); #ifdef CONFIG_PPC_64K_PAGES int mmu_ci_restrictions; #endif -static u8 *linear_map_hash_slots; -static unsigned long linear_map_hash_count; struct mmu_hash_ops mmu_hash_ops; EXPORT_SYMBOL(mmu_hash_ops); @@ -274,6 +272,8 @@ void hash__tlbiel_all(unsigned int action) } #if defined(CONFIG_DEBUG_PAGEALLOC) +static u8 *linear_map_hash_slots; +static unsigned long linear_map_hash_count; static DEFINE_RAW_SPINLOCK(linear_map_hash_lock); static void kernel_map_linear_page(unsigned long vaddr, unsigned long lmi) @@ -328,6 +328,19 @@ static void kernel_unmap_linear_page(unsigned long vaddr, unsigned long lmi) mmu_kernel_ssize, 0); } +static inline void hash_debug_pagealloc_alloc_slots(void) +{ + if (!debug_pagealloc_enabled()) + return; + linear_map_hash_count = memblock_end_of_DRAM() >> PAGE_SHIFT; + linear_map_hash_slots = memblock_alloc_try_nid( + linear_map_hash_count, 1, MEMBLOCK_LOW_LIMIT, + ppc64_rma_size, NUMA_NO_NODE); + if (!linear_map_hash_slots) + panic("%s: Failed to allocate %lu bytes max_addr=%pa\n", + __func__, linear_map_hash_count, &ppc64_rma_size); +} + static inline void hash_debug_pagealloc_add_slot(phys_addr_t paddr, int slot) { if (!debug_pagealloc_enabled()) @@ -361,6 +374,7 @@ int hash__kernel_map_pages(struct page *page, int numpages, { return 0; } +static inline void hash_debug_pagealloc_alloc_slots(void) {} static inline void hash_debug_pagealloc_add_slot(phys_addr_t paddr, int slot) {} #endif /* CONFIG_DEBUG_PAGEALLOC */ @@ -1223,16 +1237,7 @@ static void __init htab_initialize(void) prot = pgprot_val(PAGE_KERNEL); - if (debug_pagealloc_enabled()) { - linear_map_hash_count = memblock_end_of_DRAM() >> PAGE_SHIFT; - linear_map_hash_slots = memblock_alloc_try_nid( - linear_map_hash_count, 1, MEMBLOCK_LOW_LIMIT, - ppc64_rma_size, NUMA_NO_NODE); - if (!linear_map_hash_slots) - panic("%s: Failed to allocate %lu bytes max_addr=%pa\n", - __func__, linear_map_hash_count, &ppc64_rma_size); - } - + hash_debug_pagealloc_alloc_slots(); /* create bolted the linear mapping in the hash table */ for_each_mem_range(i, &base, &end) { size = end - base; From patchwork Wed Jul 31 07:56:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ritesh Harjani (IBM)" X-Patchwork-Id: 1966921 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=N+IhfqaU; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WYkzR2QXLz1yYq for ; Wed, 31 Jul 2024 18:00:19 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=N+IhfqaU; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WYkzR1PbTz3dC5 for ; Wed, 31 Jul 2024 18:00:19 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=N+IhfqaU; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::431; helo=mail-pf1-x431.google.com; envelope-from=ritesh.list@gmail.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WYkw73jNhz3d87 for ; Wed, 31 Jul 2024 17:57:27 +1000 (AEST) Received: by mail-pf1-x431.google.com with SMTP id d2e1a72fcca58-70d25b5b6b0so3967262b3a.2 for ; Wed, 31 Jul 2024 00:57:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722412644; x=1723017444; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iCk5qyX5EiGs1jnjvmC4tFgRzaEbSCoUcjAyI+/hRpE=; b=N+IhfqaUBObbqZXnERxcE0v2EUcj5ozaFT4e7LR44NvyIEZdjZIC8nwHs6R/fsY8Ks cQjq9bqcgMj6QCswPxUrnIujcA5ZT3X2uKmW5++9qEGLcEs5sOoJyGYRlKP7WIUh6KF2 1DH9sbxt2rbw+rZZ6/BPeUEwTGOf+5jFcBWU4IYsXFwUb5kNr/Fa5niddRjPoXrfQ8Vj hEimTo7+ezx2/0h2VCz/WJfrY4cmq232Us/EMcyRogjgdH7zCuOjNTbqKNF7Wo5kb40b L0s8zOhyBvLmbK/iEvIy01lDurgCDlN6dd+g6thXYufH69V3VUwwdHZoVYHd4KKwHolU DRxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722412644; x=1723017444; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iCk5qyX5EiGs1jnjvmC4tFgRzaEbSCoUcjAyI+/hRpE=; b=kFP0/i9eaUkSqPulz4qnHhA0116X4d7jM1Bx7nbH1UATh+iQYSqN3U7MKF2M+RKxZ4 SXMynXPx7Ai1/4pxDWCmbUQBLyxH01dxUgTemSnrIkKBaYhhBQeEWiQQhZ7kKlJZLl/e QOs9ICu5mPeOsa4vaWg3/2tjyXe6mUJkTRHgY/VHiGo7bVQW+4QjwlpW7lb5F09vk8tf 3dnKEf33zR3ZoYodsI8THHH9RhUAG7Mq1Y4l5sB8yZX50kRPyt4pq4mkAdC3AKXEz4av RYe6W4iwvKYKuf1USfAwmLEp3SCz0U2iEQUVV8l+UM08nVWYC/K3UQTH6amRqyJ2kcLE NJAw== X-Gm-Message-State: AOJu0Yy/mLHae6TdOWFeqL3XJvYR+GVorSWg4yFITxghOdTS+QCdIX7g p/vGS/KUEaktrn7zMZi3SL8l9mry7DmEYlDBz2ae5wVi1Vb7S1PaDyCSiQ== X-Google-Smtp-Source: AGHT+IG8QLOsWSlBzAJF3KQen4GILrtrhJ4wWDH2SHSC9DjRMhP8zoXIX7GlbIOjsEMOoX44iOr7Ng== X-Received: by 2002:a05:6a00:4f94:b0:70a:f65e:b13d with SMTP id d2e1a72fcca58-70ecedbc284mr12237333b3a.27.1722412643644; Wed, 31 Jul 2024 00:57:23 -0700 (PDT) Received: from dw-tp.ibmuc.com ([171.76.83.154]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70ead6e16fbsm9788372b3a.19.2024.07.31.00.57.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:57:23 -0700 (PDT) From: "Ritesh Harjani (IBM)" To: linuxppc-dev Subject: [RFC v1 05/10] book3s64/hash: Refactor hash__kernel_map_pages() function Date: Wed, 31 Jul 2024 13:26:09 +0530 Message-ID: <2ef6b884c2ec5ce974c09e5290a66d3ae00c6be4.1722408881.git.ritesh.list@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Donet Tom , Madhavan Srinivasan , "Ritesh Harjani \(IBM\)" , Christophe Leroy , Pavithra Prakash , "Aneesh Kumar K . V" , Nicholas Piggin , Hari Bathini Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This refactors hash__kernel_map_pages() function to call hash_debug_pagealloc_map_pages(). This will come useful when we will add kfence support. No functionality changes in this patch. Signed-off-by: Ritesh Harjani (IBM) --- arch/powerpc/mm/book3s64/hash_utils.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index 6af47b996e79..b96bbb0025fb 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -349,7 +349,8 @@ static inline void hash_debug_pagealloc_add_slot(phys_addr_t paddr, int slot) linear_map_hash_slots[paddr >> PAGE_SHIFT] = slot | 0x80; } -int hash__kernel_map_pages(struct page *page, int numpages, int enable) +static int hash_debug_pagealloc_map_pages(struct page *page, int numpages, + int enable) { unsigned long flags, vaddr, lmi; int i; @@ -368,6 +369,12 @@ int hash__kernel_map_pages(struct page *page, int numpages, int enable) local_irq_restore(flags); return 0; } + +int hash__kernel_map_pages(struct page *page, int numpages, int enable) +{ + return hash_debug_pagealloc_map_pages(page, numpages, enable); +} + #else /* CONFIG_DEBUG_PAGEALLOC */ int hash__kernel_map_pages(struct page *page, int numpages, int enable) From patchwork Wed Jul 31 07:56:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ritesh Harjani (IBM)" X-Patchwork-Id: 1966922 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=L+rgMB4w; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WYl0D2s0zz1yYq for ; Wed, 31 Jul 2024 18:01:00 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=L+rgMB4w; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WYl0C5pwxz3d8x for ; Wed, 31 Jul 2024 18:00:59 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=L+rgMB4w; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::d32; helo=mail-io1-xd32.google.com; envelope-from=ritesh.list@gmail.com; receiver=lists.ozlabs.org) Received: from mail-io1-xd32.google.com (mail-io1-xd32.google.com [IPv6:2607:f8b0:4864:20::d32]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WYkwM1kT1z3dBZ for ; Wed, 31 Jul 2024 17:57:39 +1000 (AEST) Received: by mail-io1-xd32.google.com with SMTP id ca18e2360f4ac-81fb80d3887so77456039f.2 for ; Wed, 31 Jul 2024 00:57:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722412655; x=1723017455; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oOl+VUtIQmTRaKfzajaG2TO9gNGalFY7Fmi1+sFRLoU=; b=L+rgMB4wvl3/bejj4FQakJ7NezXLJLZA5vkBUL+6xgiE/xD4eVRZ3p+4wiz2rqBnjm Nqwx/1QZ7AUrGQTLXqSo1zg8Jj1NrKdHZsxQsQJTLr9srV7i6bGx/wU3sTj+WLi8AK+f 0mFh0Um8LcWIrJQjdCKw/9s/Jz7Wvx8vjTWfLDeykJmzYAcrJQomTBNd5WkCTPSFHYlF wStdAlWskfl3cDTMRLUlQmYu/l/ZqCfpoTacosasRrxQ5mg/zfilHo8dM+aQlwDJaST5 fdTWGlbJk71krabWg9+aA6tfdqs+i2HBEsETlKKhijXrh7t/sCYJdYMfGtCMtTYkp0zq UYng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722412655; x=1723017455; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oOl+VUtIQmTRaKfzajaG2TO9gNGalFY7Fmi1+sFRLoU=; b=pxP1Et+OxaCBY09iDKOOJi7tFWsqw1cewHiViInP0dAI0opPIuGVRLOh0FY1ZIJDME D/qRDgTSqeTF7ci8fXMGzlwl07f5kidvTh6KtNROW+CF9iEM5VHm/hh+33VLlK6Jz/qA Ubc0cPntTeXFgodLk0X3MoqRsXMEcKOrIQ9tw/Dz7+XE2QS59QSUUB1McqFAPOMX/8mq GMeue/AbMp1/usqXqNUBuecgKKRK/Io2lUBUtqhMFPt1ApUe7gMk2KRBf6C9/DpUNud6 Qdqf3uoJ1zK/nDlkE8fUDWBYet9T3cXA/YEIODJpxODc65V4L2+P+ZHrYhXyFChLuZ9I TlBg== X-Gm-Message-State: AOJu0Ywbdo/mjU46s4oJxcZ+wRCwNH1QupF3ibgycW0titJEI571YuH1 Ivyek2D80Ik8ZN/8uM+OJq1D6mAwE+3N3Pn0vy1cZZLeSLgPniXXuuJKog== X-Google-Smtp-Source: AGHT+IHj48im2wqaa993Km8z6CrobozyyRPr6j+K3RhCHQnjEj4mh/WFGCzHySu88zW6R7HdGn9lQg== X-Received: by 2002:a05:6e02:1908:b0:395:ee4b:333 with SMTP id e9e14a558f8ab-39aec41ab77mr157029505ab.28.1722412655080; Wed, 31 Jul 2024 00:57:35 -0700 (PDT) Received: from dw-tp.ibmuc.com ([171.76.83.154]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70ead6e16fbsm9788372b3a.19.2024.07.31.00.57.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:57:34 -0700 (PDT) From: "Ritesh Harjani (IBM)" To: linuxppc-dev Subject: [RFC v1 06/10] book3s64/hash: Make kernel_map_linear_page() generic Date: Wed, 31 Jul 2024 13:26:10 +0530 Message-ID: <25bfe0937e58855e354b307466eff809bbd40609.1722408881.git.ritesh.list@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Donet Tom , Madhavan Srinivasan , "Ritesh Harjani \(IBM\)" , Christophe Leroy , Pavithra Prakash , "Aneesh Kumar K . V" , Nicholas Piggin , Hari Bathini Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Currently kernel_map_linear_page() function assumes to be working on linear_map_hash_slots array. But since in later patches we need a separate linear map array for kfence, hence make kernel_map_linear_page() take a linear map array and lock in it's function argument. This is needed to separate out kfence from debug_pagealloc infrastructure. Signed-off-by: Ritesh Harjani (IBM) --- arch/powerpc/mm/book3s64/hash_utils.c | 47 ++++++++++++++------------- 1 file changed, 25 insertions(+), 22 deletions(-) diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index b96bbb0025fb..3f3eaf0a254b 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -272,11 +272,8 @@ void hash__tlbiel_all(unsigned int action) } #if defined(CONFIG_DEBUG_PAGEALLOC) -static u8 *linear_map_hash_slots; -static unsigned long linear_map_hash_count; -static DEFINE_RAW_SPINLOCK(linear_map_hash_lock); - -static void kernel_map_linear_page(unsigned long vaddr, unsigned long lmi) +static void kernel_map_linear_page(unsigned long vaddr, unsigned long idx, + u8 *slots, raw_spinlock_t *lock) { unsigned long hash; unsigned long vsid = get_kernel_vsid(vaddr, mmu_kernel_ssize); @@ -290,7 +287,7 @@ static void kernel_map_linear_page(unsigned long vaddr, unsigned long lmi) if (!vsid) return; - if (linear_map_hash_slots[lmi] & 0x80) + if (slots[idx] & 0x80) return; ret = hpte_insert_repeating(hash, vpn, __pa(vaddr), mode, @@ -298,36 +295,40 @@ static void kernel_map_linear_page(unsigned long vaddr, unsigned long lmi) mmu_linear_psize, mmu_kernel_ssize); BUG_ON (ret < 0); - raw_spin_lock(&linear_map_hash_lock); - BUG_ON(linear_map_hash_slots[lmi] & 0x80); - linear_map_hash_slots[lmi] = ret | 0x80; - raw_spin_unlock(&linear_map_hash_lock); + raw_spin_lock(lock); + BUG_ON(slots[idx] & 0x80); + slots[idx] = ret | 0x80; + raw_spin_unlock(lock); } -static void kernel_unmap_linear_page(unsigned long vaddr, unsigned long lmi) +static void kernel_unmap_linear_page(unsigned long vaddr, unsigned long idx, + u8 *slots, raw_spinlock_t *lock) { - unsigned long hash, hidx, slot; + unsigned long hash, hslot, slot; unsigned long vsid = get_kernel_vsid(vaddr, mmu_kernel_ssize); unsigned long vpn = hpt_vpn(vaddr, vsid, mmu_kernel_ssize); hash = hpt_hash(vpn, PAGE_SHIFT, mmu_kernel_ssize); - raw_spin_lock(&linear_map_hash_lock); - if (!(linear_map_hash_slots[lmi] & 0x80)) { - raw_spin_unlock(&linear_map_hash_lock); + raw_spin_lock(lock); + if (!(slots[idx] & 0x80)) { + raw_spin_unlock(lock); return; } - hidx = linear_map_hash_slots[lmi] & 0x7f; - linear_map_hash_slots[lmi] = 0; - raw_spin_unlock(&linear_map_hash_lock); - if (hidx & _PTEIDX_SECONDARY) + hslot = slots[idx] & 0x7f; + slots[idx] = 0; + raw_spin_unlock(lock); + if (hslot & _PTEIDX_SECONDARY) hash = ~hash; slot = (hash & htab_hash_mask) * HPTES_PER_GROUP; - slot += hidx & _PTEIDX_GROUP_IX; + slot += hslot & _PTEIDX_GROUP_IX; mmu_hash_ops.hpte_invalidate(slot, vpn, mmu_linear_psize, mmu_linear_psize, mmu_kernel_ssize, 0); } +static u8 *linear_map_hash_slots; +static unsigned long linear_map_hash_count; +static DEFINE_RAW_SPINLOCK(linear_map_hash_lock); static inline void hash_debug_pagealloc_alloc_slots(void) { if (!debug_pagealloc_enabled()) @@ -362,9 +363,11 @@ static int hash_debug_pagealloc_map_pages(struct page *page, int numpages, if (lmi >= linear_map_hash_count) continue; if (enable) - kernel_map_linear_page(vaddr, lmi); + kernel_map_linear_page(vaddr, lmi, + linear_map_hash_slots, &linear_map_hash_lock); else - kernel_unmap_linear_page(vaddr, lmi); + kernel_unmap_linear_page(vaddr, lmi, + linear_map_hash_slots, &linear_map_hash_lock); } local_irq_restore(flags); return 0; From patchwork Wed Jul 31 07:56:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ritesh Harjani (IBM)" X-Patchwork-Id: 1966923 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=WJscwdTW; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WYl106btJz1yfG for ; Wed, 31 Jul 2024 18:01:40 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=WJscwdTW; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WYl104x9Kz3dRP for ; Wed, 31 Jul 2024 18:01:40 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=WJscwdTW; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::433; helo=mail-pf1-x433.google.com; envelope-from=ritesh.list@gmail.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WYkwW37F1z3d2S for ; Wed, 31 Jul 2024 17:57:47 +1000 (AEST) Received: by mail-pf1-x433.google.com with SMTP id d2e1a72fcca58-70d2b921cdfso4772672b3a.0 for ; Wed, 31 Jul 2024 00:57:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722412665; x=1723017465; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UbALxTqSqlEVKnoz62IgytvWvcETkWPldZLYnVFh9W4=; b=WJscwdTWHcTNW9NVoq386sdgb2rW5o4UgXusccq6CdmOO1Pn53DuWhGZDMkoButWy2 cgTbj6+lh7KYuK4J5onQe7hGuYlIvByHVoQJbv7Td4QMxJVCkr0O0jUlYLZyJiHPLshc c2y2sP7h0WC4q+AsRyy1Z3JC/p9t4N4Xd30PC/RE57FvrXCTepyVWf+2I9d2ih8kdTpZ X7k0U3zct0BOrk/u+3V4xkfXmhpXOzh6vsyo6aPt0tVUM4Q/HOCbfNYFvpOaKhJeMHTB 8Kz8AlbRuX3s9tPXwTfhRkrANnWQVPagisd1i8GDQLmUIZBJpkzNb3wjUSNN4NH5Au3w Y3Pg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722412665; x=1723017465; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UbALxTqSqlEVKnoz62IgytvWvcETkWPldZLYnVFh9W4=; b=lpu9AzYKeDUe6oBrOZuF4jHVOtd+iO/dGVKizOSbFiv75RylAL7479kDHCg549X9Ai adtccykt0SHHAoN/KkK1c0vz7mWJ2H+iLcYKa2T+OZotrJvybKAYcNnya34B6PaRkbaA BLy/LZUGLQOxgNj4rWPkCzIByIucs8ef4hFZmez/Ot2FrjbIPFC6AfPmgGWWs9NfElQj NHhBM/Eej9kYEnIUKCnzzfukNMza2b1WKXFQ/LYzHhNqXcrKAuzwul7ocnkb4IO3eVgY McZaEl9gsW6tUo1tVxPUy8yiC0G179ZGpFyKD/wIKDLV5aKsgNmLIDms+soKHavHmCJl ALEA== X-Gm-Message-State: AOJu0YxLoHOl9QvBr+E7VVWhNqZHMSTDh3dhE+h45SjLXWlKVrSFscIl iX7LtkI2P81LwReKwkFIwXG/YvIJGELDFNyio6FjCygi1IxwlzihpsIJHA== X-Google-Smtp-Source: AGHT+IEzxyRVOeCaCY954crrhmdRysIFEVHaRyW4RQuGke8M+TLewXN4ywzl7YIlQvNN50l5nIQxvA== X-Received: by 2002:a05:6a00:138e:b0:705:6a0a:de14 with SMTP id d2e1a72fcca58-70ece9ebba7mr16338761b3a.1.1722412664651; Wed, 31 Jul 2024 00:57:44 -0700 (PDT) Received: from dw-tp.ibmuc.com ([171.76.83.154]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70ead6e16fbsm9788372b3a.19.2024.07.31.00.57.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:57:44 -0700 (PDT) From: "Ritesh Harjani (IBM)" To: linuxppc-dev Subject: [RFC v1 07/10] book3s64/hash: Disable debug_pagealloc if it requires more memory Date: Wed, 31 Jul 2024 13:26:11 +0530 Message-ID: <771cddcc8298575d27cde85e0dc6b85755f9dd9e.1722408881.git.ritesh.list@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Donet Tom , Madhavan Srinivasan , "Ritesh Harjani \(IBM\)" , Christophe Leroy , Pavithra Prakash , "Aneesh Kumar K . V" , Nicholas Piggin , Hari Bathini Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Make size of the linear map to be allocated in RMA region to be of ppc64_rma_size / 4. If debug_pagealloc requires more memory than that then do not allocate any memory and disable debug_pagealloc. Signed-off-by: Ritesh Harjani (IBM) --- arch/powerpc/mm/book3s64/hash_utils.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index 3f3eaf0a254b..906cd167180a 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -331,9 +331,19 @@ static unsigned long linear_map_hash_count; static DEFINE_RAW_SPINLOCK(linear_map_hash_lock); static inline void hash_debug_pagealloc_alloc_slots(void) { + unsigned long max_hash_count = (ppc64_rma_size / 4) >> PAGE_SHIFT; + if (!debug_pagealloc_enabled()) return; linear_map_hash_count = memblock_end_of_DRAM() >> PAGE_SHIFT; + if (unlikely(linear_map_hash_count > max_hash_count)) { + pr_info("linear map size (%llu) greater than 4 times RMA region (%llu). Disabling debug_pagealloc\n", + ((u64)linear_map_hash_count << PAGE_SHIFT), + ppc64_rma_size); + linear_map_hash_count = 0; + return; + } + linear_map_hash_slots = memblock_alloc_try_nid( linear_map_hash_count, 1, MEMBLOCK_LOW_LIMIT, ppc64_rma_size, NUMA_NO_NODE); @@ -344,7 +354,7 @@ static inline void hash_debug_pagealloc_alloc_slots(void) static inline void hash_debug_pagealloc_add_slot(phys_addr_t paddr, int slot) { - if (!debug_pagealloc_enabled()) + if (!debug_pagealloc_enabled() || !linear_map_hash_count) return; if ((paddr >> PAGE_SHIFT) < linear_map_hash_count) linear_map_hash_slots[paddr >> PAGE_SHIFT] = slot | 0x80; @@ -356,6 +366,9 @@ static int hash_debug_pagealloc_map_pages(struct page *page, int numpages, unsigned long flags, vaddr, lmi; int i; + if (!debug_pagealloc_enabled() || !linear_map_hash_count) + return 0; + local_irq_save(flags); for (i = 0; i < numpages; i++, page++) { vaddr = (unsigned long)page_address(page); From patchwork Wed Jul 31 07:56:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ritesh Harjani (IBM)" X-Patchwork-Id: 1966924 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=U1ngZo3A; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WYl1y1WXJz1yYq for ; Wed, 31 Jul 2024 18:02:30 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=U1ngZo3A; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WYl1y0RhDz3d8R for ; Wed, 31 Jul 2024 18:02:30 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=U1ngZo3A; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::42c; helo=mail-pf1-x42c.google.com; envelope-from=ritesh.list@gmail.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WYkwk4D0Gz3dC5 for ; Wed, 31 Jul 2024 17:57:58 +1000 (AEST) Received: by mail-pf1-x42c.google.com with SMTP id d2e1a72fcca58-70d23caf8ddso4781108b3a.0 for ; Wed, 31 Jul 2024 00:57:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722412675; x=1723017475; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SlswoT2b+orJh1kLQ/S6utrJaivvXqzqOuHKXYuEfSY=; b=U1ngZo3Am7T+s60woAo7H4GbC/awXFpvLTS6k4RcODzZ6ONnghKPc0AGb86bdAfLZL tYfIs3cPW6wUtfy8kwepvqQ01p1RvcYnvimq85KLRk9hJDxqfEMj7o26HK/1afuj5lG4 DfogM7DIEpa3HByMqGFw/3AZwbmuj1PCWYNky34V2RA41MNuKcfglc3LuFJqfyuoR5I/ pejREdFdYYQWG53SGDlXnQDsRcCzV5933xv2U3UuXkicyf5j4HrmqGagZWdaDsKqQRjP hVjC5LmS2jBndWRfgRGOM+sVCFARs3mkPvcPUEBkDfbj9ivXmCjv+D+tO1x0alD4yHxQ kNnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722412675; x=1723017475; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SlswoT2b+orJh1kLQ/S6utrJaivvXqzqOuHKXYuEfSY=; b=jqow4f/mLdPK040E/uP2jWG3T7T88jkfhp3oIrY8/agEpQhkZjrT0nNUhRuiTikIjH oHsZJe4TVsxUktDbbItZWDNFYAEHOi+Frs2gC9RsRq0YUwHAypnEQ3FnaJ/XE52EOK4I jMbU7AfyZGqgeSw+nbQ2giy/9sdQP42XxGf0y+8U9D0jC61jQ5sla08zP99xQHPTqVgm KRXgVHGIE2QLD8+KYV5WUrklnxEOkfI7oslL2qkMWV3wsYYNASuVFvcEQXYah5LClwFr MW1CQ5VWDz3fGChpBBALKuloEssJhYJykUEwjwXIrZxD2Z4VCxboNaLUH8ObbksHEz24 QnvQ== X-Gm-Message-State: AOJu0Yw9weWCZrleURTKDz0ak2zdP0Da+Fu1dRDJ9Yiubf54rnexQ9Hr 5qAPA1BLCKzXWDgTIDm3S/03gUgIGvVuy0GExOm9dUR/1eGlJt4wG0nv9Q== X-Google-Smtp-Source: AGHT+IHm+gtmm0YCt3trxm82F89ieYgjTcfOm8zqblI880B+4sGI4rBJ/zXAFGEY2gBXIBmare9gNQ== X-Received: by 2002:a05:6a00:4f83:b0:706:74be:686e with SMTP id d2e1a72fcca58-70ecedb2fa1mr16092812b3a.26.1722412675176; Wed, 31 Jul 2024 00:57:55 -0700 (PDT) Received: from dw-tp.ibmuc.com ([171.76.83.154]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70ead6e16fbsm9788372b3a.19.2024.07.31.00.57.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:57:54 -0700 (PDT) From: "Ritesh Harjani (IBM)" To: linuxppc-dev Subject: [RFC v1 08/10] book3s64/hash: Add kfence functionality Date: Wed, 31 Jul 2024 13:26:12 +0530 Message-ID: <7450b71d41093795674fe0314444492b2ff70e58.1722408881.git.ritesh.list@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Donet Tom , Madhavan Srinivasan , "Ritesh Harjani \(IBM\)" , Christophe Leroy , Pavithra Prakash , "Aneesh Kumar K . V" , Nicholas Piggin , Hari Bathini Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Now that linear map functionality of debug_pagealloc is made generic, enable kfence to use this generic infrastructure. 1. Define kfence related linear map variables. - u8 *linear_map_kf_hash_slots; - unsigned long linear_map_kf_hash_count; - DEFINE_RAW_SPINLOCK(linear_map_kf_hash_lock); 2. The linear map size allocated in RMA region is quite small (KFENCE_POOL_SIZE >> PAGE_SHIFT) which is 512 bytes by default. 3. kfence pool memory is reserved using memblock_phys_alloc() which has can come from anywhere. (default 255 objects => ((1+255) * 2) << PAGE_SHIFT = 32MB) 4. The hash slot information for kfence memory gets added in linear map in hash_linear_map_add_slot() (which also adds for debug_pagealloc). Reported-by: Pavithra Prakash Signed-off-by: Ritesh Harjani (IBM) --- arch/powerpc/include/asm/kfence.h | 5 - arch/powerpc/mm/book3s64/hash_utils.c | 162 +++++++++++++++++++++++--- 2 files changed, 149 insertions(+), 18 deletions(-) diff --git a/arch/powerpc/include/asm/kfence.h b/arch/powerpc/include/asm/kfence.h index f3a9476a71b3..fab124ada1c7 100644 --- a/arch/powerpc/include/asm/kfence.h +++ b/arch/powerpc/include/asm/kfence.h @@ -10,7 +10,6 @@ #include #include -#include #ifdef CONFIG_PPC64_ELF_ABI_V1 #define ARCH_FUNC_PREFIX "." @@ -26,10 +25,6 @@ static inline void disable_kfence(void) static inline bool arch_kfence_init_pool(void) { -#ifdef CONFIG_PPC64 - if (!radix_enabled()) - return false; -#endif return !kfence_disabled; } #endif diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index 906cd167180a..c66b9921fc7d 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -40,6 +40,7 @@ #include #include #include +#include #include #include @@ -66,6 +67,7 @@ #include #include #include +#include #include @@ -271,7 +273,7 @@ void hash__tlbiel_all(unsigned int action) WARN(1, "%s called on pre-POWER7 CPU\n", __func__); } -#if defined(CONFIG_DEBUG_PAGEALLOC) +#if defined(CONFIG_DEBUG_PAGEALLOC) || defined(CONFIG_KFENCE) static void kernel_map_linear_page(unsigned long vaddr, unsigned long idx, u8 *slots, raw_spinlock_t *lock) { @@ -325,11 +327,13 @@ static void kernel_unmap_linear_page(unsigned long vaddr, unsigned long idx, mmu_linear_psize, mmu_kernel_ssize, 0); } +#endif +#if defined(CONFIG_DEBUG_PAGEALLOC) static u8 *linear_map_hash_slots; static unsigned long linear_map_hash_count; static DEFINE_RAW_SPINLOCK(linear_map_hash_lock); -static inline void hash_debug_pagealloc_alloc_slots(void) +static void hash_debug_pagealloc_alloc_slots(void) { unsigned long max_hash_count = (ppc64_rma_size / 4) >> PAGE_SHIFT; @@ -352,7 +356,8 @@ static inline void hash_debug_pagealloc_alloc_slots(void) __func__, linear_map_hash_count, &ppc64_rma_size); } -static inline void hash_debug_pagealloc_add_slot(phys_addr_t paddr, int slot) +static inline void hash_debug_pagealloc_add_slot(phys_addr_t paddr, + int slot) { if (!debug_pagealloc_enabled() || !linear_map_hash_count) return; @@ -386,20 +391,148 @@ static int hash_debug_pagealloc_map_pages(struct page *page, int numpages, return 0; } -int hash__kernel_map_pages(struct page *page, int numpages, int enable) +#else /* CONFIG_DEBUG_PAGEALLOC */ +static inline void hash_debug_pagealloc_alloc_slots(void) {} +static inline void hash_debug_pagealloc_add_slot(phys_addr_t paddr, int slot) {} +static int __maybe_unused +hash_debug_pagealloc_map_pages(struct page *page, int numpages, int enable) { - return hash_debug_pagealloc_map_pages(page, numpages, enable); + return 0; } +#endif /* CONFIG_DEBUG_PAGEALLOC */ -#else /* CONFIG_DEBUG_PAGEALLOC */ -int hash__kernel_map_pages(struct page *page, int numpages, - int enable) +#if defined(CONFIG_KFENCE) +static u8 *linear_map_kf_hash_slots; +static unsigned long linear_map_kf_hash_count; +static DEFINE_RAW_SPINLOCK(linear_map_kf_hash_lock); + +static phys_addr_t kfence_pool; + +static inline void hash_kfence_alloc_pool(void) +{ + + // allocate linear map for kfence within RMA region + linear_map_kf_hash_count = KFENCE_POOL_SIZE >> PAGE_SHIFT; + linear_map_kf_hash_slots = memblock_alloc_try_nid( + linear_map_kf_hash_count, 1, + MEMBLOCK_LOW_LIMIT, ppc64_rma_size, + NUMA_NO_NODE); + if (!linear_map_kf_hash_slots) { + pr_err("%s: memblock for linear map (%lu) failed\n", __func__, + linear_map_kf_hash_count); + goto err; + } + + // allocate kfence pool early + kfence_pool = memblock_phys_alloc_range(KFENCE_POOL_SIZE, PAGE_SIZE, + MEMBLOCK_LOW_LIMIT, MEMBLOCK_ALLOC_ANYWHERE); + if (!kfence_pool) { + pr_err("%s: memblock for kfence pool (%lu) failed\n", __func__, + KFENCE_POOL_SIZE); + memblock_free(linear_map_kf_hash_slots, + linear_map_kf_hash_count); + linear_map_kf_hash_count = 0; + goto err; + } + memblock_mark_nomap(kfence_pool, KFENCE_POOL_SIZE); + + return; +err: + pr_info("Disabling kfence\n"); + disable_kfence(); +} + +static inline void hash_kfence_map_pool(void) +{ + unsigned long kfence_pool_start, kfence_pool_end; + unsigned long prot = pgprot_val(PAGE_KERNEL); + + if (!kfence_pool) + return; + + kfence_pool_start = (unsigned long) __va(kfence_pool); + kfence_pool_end = kfence_pool_start + KFENCE_POOL_SIZE; + __kfence_pool = (char *) kfence_pool_start; + BUG_ON(htab_bolt_mapping(kfence_pool_start, kfence_pool_end, + kfence_pool, prot, mmu_linear_psize, + mmu_kernel_ssize)); + memblock_clear_nomap(kfence_pool, KFENCE_POOL_SIZE); +} + +static inline void hash_kfence_add_slot(phys_addr_t paddr, int slot) { + unsigned long vaddr = (unsigned long) __va(paddr); + unsigned long lmi = (vaddr - (unsigned long)__kfence_pool) + >> PAGE_SHIFT; + + if (!kfence_pool) + return; + BUG_ON(!is_kfence_address((void *)vaddr)); + BUG_ON(lmi >= linear_map_kf_hash_count); + linear_map_kf_hash_slots[lmi] = slot | 0x80; +} + +static int hash_kfence_map_pages(struct page *page, int numpages, int enable) +{ + unsigned long flags, vaddr, lmi; + int i; + + WARN_ON_ONCE(!linear_map_kf_hash_count); + local_irq_save(flags); + for (i = 0; i < numpages; i++, page++) { + vaddr = (unsigned long)page_address(page); + lmi = (vaddr - (unsigned long)__kfence_pool) >> PAGE_SHIFT; + + /* Ideally this should never happen */ + if (lmi >= linear_map_kf_hash_count) { + WARN_ON_ONCE(1); + continue; + } + + if (enable) + kernel_map_linear_page(vaddr, lmi, + linear_map_kf_hash_slots, + &linear_map_kf_hash_lock); + else + kernel_unmap_linear_page(vaddr, lmi, + linear_map_kf_hash_slots, + &linear_map_kf_hash_lock); + } + local_irq_restore(flags); return 0; } -static inline void hash_debug_pagealloc_alloc_slots(void) {} -static inline void hash_debug_pagealloc_add_slot(phys_addr_t paddr, int slot) {} -#endif /* CONFIG_DEBUG_PAGEALLOC */ +#else +static inline void hash_kfence_alloc_pool(void) {} +static inline void hash_kfence_map_pool(void) {} +static inline void hash_kfence_add_slot(phys_addr_t paddr, int slot) {} +static int __maybe_unused +hash_kfence_map_pages(struct page *page, int numpages, int enable) +{ + return 0; +} +#endif + +#if defined(CONFIG_DEBUG_PAGEALLOC) || defined(CONFIG_KFENCE) +int hash__kernel_map_pages(struct page *page, int numpages, int enable) +{ + void *vaddr = page_address(page); + + if (is_kfence_address(vaddr)) + return hash_kfence_map_pages(page, numpages, enable); + else + return hash_debug_pagealloc_map_pages(page, numpages, enable); +} + +static void hash_linear_map_add_slot(phys_addr_t paddr, int slot) +{ + if (is_kfence_address(__va(paddr))) + hash_kfence_add_slot(paddr, slot); + else + hash_debug_pagealloc_add_slot(paddr, slot); +} +#else +static void hash_linear_map_add_slot(phys_addr_t paddr, int slot) {} +#endif /* * 'R' and 'C' update notes: @@ -559,7 +692,8 @@ int htab_bolt_mapping(unsigned long vstart, unsigned long vend, break; cond_resched(); - hash_debug_pagealloc_add_slot(paddr, ret); + // add slot info in debug_pagealloc / kfence linear map + hash_linear_map_add_slot(paddr, ret); } return ret < 0 ? ret : 0; } @@ -940,7 +1074,7 @@ static void __init htab_init_page_sizes(void) bool aligned = true; init_hpte_page_sizes(); - if (!debug_pagealloc_enabled()) { + if (!debug_pagealloc_enabled_or_kfence()) { /* * Pick a size for the linear mapping. Currently, we only * support 16M, 1M and 4K which is the default @@ -1261,6 +1395,7 @@ static void __init htab_initialize(void) prot = pgprot_val(PAGE_KERNEL); hash_debug_pagealloc_alloc_slots(); + hash_kfence_alloc_pool(); /* create bolted the linear mapping in the hash table */ for_each_mem_range(i, &base, &end) { size = end - base; @@ -1277,6 +1412,7 @@ static void __init htab_initialize(void) BUG_ON(htab_bolt_mapping(base, base + size, __pa(base), prot, mmu_linear_psize, mmu_kernel_ssize)); } + hash_kfence_map_pool(); memblock_set_current_limit(MEMBLOCK_ALLOC_ANYWHERE); /* From patchwork Wed Jul 31 07:56:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ritesh Harjani (IBM)" X-Patchwork-Id: 1966925 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=JLg++IN8; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WYl2l0zHSz1yYq for ; Wed, 31 Jul 2024 18:03:11 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=JLg++IN8; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WYl2k71Kvz3dKV for ; Wed, 31 Jul 2024 18:03:10 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=JLg++IN8; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::52d; helo=mail-pg1-x52d.google.com; envelope-from=ritesh.list@gmail.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WYkww33RQz3d9G for ; Wed, 31 Jul 2024 17:58:08 +1000 (AEST) Received: by mail-pg1-x52d.google.com with SMTP id 41be03b00d2f7-7a103ac7be3so3296878a12.3 for ; Wed, 31 Jul 2024 00:58:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722412685; x=1723017485; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ohBW4WWXljfsS+w6lK1TGfOL2L5ph++1vfcBaUGToUM=; b=JLg++IN8GFQD6Dj4cGLuPU7F5Ul4b3zLU/EbbK3g3ezaWMf2FT9PQbbn/g01CkrmR6 CaWvlHSCVhxtUsrbgs7hhG2utTfG+sHeAtP0wIKH2226h4jaEONxi03XJh7cwR5C+4u/ pwy/uZQBNyIWYcM5SAdYnwQycY1RJRSIw88HQDYTGXTfnN0M2JmZj+BL00F9O9EIaus4 GbNpuvrBfaPMw1/ilnm1lhsvAbAq1LWyFcDajlwY8P5PQEHUKxdgaN7NHt8bm9EXV31b bUAyiKmLOCcxEfeXRY5noSNd5H9QmCYK1a2EqBNtUtntHp69S4OXkH7rKtzUd8/MS0iM 4sPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722412685; x=1723017485; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ohBW4WWXljfsS+w6lK1TGfOL2L5ph++1vfcBaUGToUM=; b=DspiaV+JSoPW0tJO6fEkF6QshIPHzBLSAj/l73LoiQ48og+nqthK1moYe2XrAduhKz /Az4idnNz52uAR+fCe/GFC9mrv62lY/k1p4W/0+YcSP23c1VRlgPjhCL908tBoLjEzTb MowaJf+jWNMgF2lStoKny2vFNuZcwow25x6IfcEBDEbjYrvALzTi/5rHkoy19G5XdVsO jbIXq53EAwVEfsCcTFC5K3GloJYX8NwPG8WunIV5t9Xnpz+ns1dwTK7ZFv0LJu3yddTS l1FWbuFzXkWhjB8cqHiKzhFKWZjB3sYEoPuqrJB1ejXQ7JYtem03k6WOWqWy3D8JvcP8 5dzA== X-Gm-Message-State: AOJu0Yz7H/NekPbnajSm/QxYOcpap0nLzjo0t+E0zh2o+w2N9NYumTT/ Vm8Ee/a+XFeThljBVPuFwjsMbCpQVQEkkcX/k20CO2T3OfcCibMGkS0Y/A== X-Google-Smtp-Source: AGHT+IFB3hPBsysOmUsQsmP0ufXzP62nw33VuDXJmxWMYGx+DaG2r9KImMZcQNojLZt4Ylq3/lNHSw== X-Received: by 2002:a05:6a21:6da0:b0:1c2:8ece:97ae with SMTP id adf61e73a8af0-1c4a13afd52mr11270732637.34.1722412685200; Wed, 31 Jul 2024 00:58:05 -0700 (PDT) Received: from dw-tp.ibmuc.com ([171.76.83.154]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70ead6e16fbsm9788372b3a.19.2024.07.31.00.57.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:58:04 -0700 (PDT) From: "Ritesh Harjani (IBM)" To: linuxppc-dev Subject: [RFC v1 09/10] book3s64/radix: Refactoring common kfence related functions Date: Wed, 31 Jul 2024 13:26:13 +0530 Message-ID: <292790d676ed9e31507a0ba4b11800cbc6670e44.1722408881.git.ritesh.list@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Donet Tom , Madhavan Srinivasan , "Ritesh Harjani \(IBM\)" , Christophe Leroy , Pavithra Prakash , "Aneesh Kumar K . V" , Nicholas Piggin , Hari Bathini Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Both radix and hash on book3s requires to detect if kfence early init is enabled or not. Hash needs to disable kfence if early init is not enabled because with kfence the linear map is mapped using PAGE_SIZE rather than 16M mapping. We don't support multiple page sizes for slb entry used for kernel linear map in book3s64. This patch refactors out the common functions required to detect kfence early init is enabled or not. Signed-off-by: Ritesh Harjani (IBM) --- arch/powerpc/include/asm/kfence.h | 2 ++ arch/powerpc/mm/book3s64/radix_pgtable.c | 12 ------------ arch/powerpc/mm/init-common.c | 12 ++++++++++++ 3 files changed, 14 insertions(+), 12 deletions(-) diff --git a/arch/powerpc/include/asm/kfence.h b/arch/powerpc/include/asm/kfence.h index fab124ada1c7..5975688d8de1 100644 --- a/arch/powerpc/include/asm/kfence.h +++ b/arch/powerpc/include/asm/kfence.h @@ -15,6 +15,8 @@ #define ARCH_FUNC_PREFIX "." #endif +extern bool kfence_early_init; + #ifdef CONFIG_KFENCE extern bool kfence_disabled; diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index b0d927009af8..311e2112d782 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -363,18 +363,6 @@ static int __meminit create_physical_mapping(unsigned long start, } #ifdef CONFIG_KFENCE -static bool __ro_after_init kfence_early_init = !!CONFIG_KFENCE_SAMPLE_INTERVAL; - -static int __init parse_kfence_early_init(char *arg) -{ - int val; - - if (get_option(&arg, &val)) - kfence_early_init = !!val; - return 0; -} -early_param("kfence.sample_interval", parse_kfence_early_init); - static inline phys_addr_t alloc_kfence_pool(void) { phys_addr_t kfence_pool; diff --git a/arch/powerpc/mm/init-common.c b/arch/powerpc/mm/init-common.c index 21131b96d209..259821a4db62 100644 --- a/arch/powerpc/mm/init-common.c +++ b/arch/powerpc/mm/init-common.c @@ -33,6 +33,18 @@ bool disable_kuep = !IS_ENABLED(CONFIG_PPC_KUEP); bool disable_kuap = !IS_ENABLED(CONFIG_PPC_KUAP); #ifdef CONFIG_KFENCE bool __ro_after_init kfence_disabled; +bool __ro_after_init kfence_early_init = !!CONFIG_KFENCE_SAMPLE_INTERVAL; +static int __init parse_kfence_early_init(char *arg) +{ + int val; + + if (get_option(&arg, &val)) + kfence_early_init = !!val; + return 0; +} +early_param("kfence.sample_interval", parse_kfence_early_init); +#else +bool __ro_after_init kfence_early_init; #endif static int __init parse_nosmep(char *p) From patchwork Wed Jul 31 07:56:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ritesh Harjani (IBM)" X-Patchwork-Id: 1966926 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=KS+PdhBi; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WYl3W2cDgz1yYq for ; Wed, 31 Jul 2024 18:03:51 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=KS+PdhBi; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WYl3W1ZQSz3dRt for ; Wed, 31 Jul 2024 18:03:51 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=KS+PdhBi; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2001:4860:4864:20::34; helo=mail-oa1-x34.google.com; envelope-from=ritesh.list@gmail.com; receiver=lists.ozlabs.org) Received: from mail-oa1-x34.google.com (mail-oa1-x34.google.com [IPv6:2001:4860:4864:20::34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WYkxC3qtZz3cnc for ; Wed, 31 Jul 2024 17:58:23 +1000 (AEST) Received: by mail-oa1-x34.google.com with SMTP id 586e51a60fabf-260f94067bcso3715516fac.3 for ; Wed, 31 Jul 2024 00:58:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722412695; x=1723017495; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4sjNoON398ZsIx0bmEQJty2zbqjlCUYWeF9pzoiIutU=; b=KS+PdhBi2Kra1Hv0eSi8mn9096NY34PPznqE7ZjmBqazBPEz7BdWbPHjKM+UYPWiBo 0LoTARQRF4/+ErvgMyyNIwLgItkQNy22fRIlpcTCBctdf1Nv7YQdT28Baxb4VTC/e5uk fCrTRWz3T36U12/F+Kf+rsxfA0QDyeNROZuKfRUzXlRncP0bzGsC0h3/Yle24eWP5605 Kmn4agRbIzVsO4DmKkg68Sc3GO9CUJrU8NsfoWoJYOiR6HaJfu3fLHtPAkbh6mDCnzBX 9ubeTJhPp/zVxnjKPKtOj3+zw36b6SzesoMl5ADbW6x9N3VAqU5sRW6hdZfIKFHS1LuJ jtzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722412695; x=1723017495; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4sjNoON398ZsIx0bmEQJty2zbqjlCUYWeF9pzoiIutU=; b=WlISPER8/IW7IIBtbAviPjslnKeJBlGXCkAd5HWevd7DJgTZQOGGRxqIkyH0l9NAc7 5WuBKL8PpIYM45GtHT8RrlkSRzN96fP6pbE80SkKWkSmJyOCd77Lb9of1o6DD73AGCy6 Hrr5BG++DgXyQJuO49UYl5DWRitavu5Wc/1+ZBY+QjQ2yByen5Q5y1XkPb4rpFnkZ2JQ 0Jm1dTTmXWR2NBEbNYY27J7/4lsgzXhQFmC6X4CyavoV174v/WYu7sCmDOJ3gg1jhLZC DvGB4zZMephLQCyFmVyue8fXykaKCjjUGzVtJrg9JgUhc2A67xAYcyqXJZ0dHIqABDSa I6GQ== X-Gm-Message-State: AOJu0YzAA+fW9HDEQTlwIwGuUuEjf9i2T18ckFiYdWsLuNiTFQXR7+y+ E+dEXF2COSdKJI2JXaFUdB673M3iU7diMQf4mPy5R/2J/JLNCn+ToCJD7A== X-Google-Smtp-Source: AGHT+IGW4hGBbauSzQ7W8AKrqOHSQ/x7Lo+FNQaq84PeyRJgK4W33/QqxduNJ9zsws/o/hDmiPXKkw== X-Received: by 2002:a05:6870:1605:b0:261:10b2:7bc1 with SMTP id 586e51a60fabf-267d4f21b01mr17781102fac.31.1722412694597; Wed, 31 Jul 2024 00:58:14 -0700 (PDT) Received: from dw-tp.ibmuc.com ([171.76.83.154]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70ead6e16fbsm9788372b3a.19.2024.07.31.00.58.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:58:14 -0700 (PDT) From: "Ritesh Harjani (IBM)" To: linuxppc-dev Subject: [RFC v1 10/10] book3s64/hash: Disable kfence if not early init Date: Wed, 31 Jul 2024 13:26:14 +0530 Message-ID: X-Mailer: git-send-email 2.45.2 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Donet Tom , Madhavan Srinivasan , "Ritesh Harjani \(IBM\)" , Christophe Leroy , Pavithra Prakash , "Aneesh Kumar K . V" , Nicholas Piggin , Hari Bathini Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Enable kfence on book3s64 hash only when early init is enabled. This is because, kfence could cause the kernel linear map to be mapped at PAGE_SIZE level instead of 16M (which I guess we don't want). Also currently there is no way to - 1. Make multiple page size entries for the SLB used for kernel linear map. 2. No easy way of getting the hash slot details after the page table mapping for kernel linear setup. So even if kfence allocate the pool in late init, we won't be able to get the hash slot details in kfence linear map. Thus this patch disables kfence on hash if kfence early init is not enabled. Signed-off-by: Ritesh Harjani (IBM) --- arch/powerpc/mm/book3s64/hash_utils.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) -- 2.45.2 diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index c66b9921fc7d..759dbcbf1483 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -410,6 +410,8 @@ static phys_addr_t kfence_pool; static inline void hash_kfence_alloc_pool(void) { + if (!kfence_early_init) + goto err; // allocate linear map for kfence within RMA region linear_map_kf_hash_count = KFENCE_POOL_SIZE >> PAGE_SHIFT; @@ -1074,7 +1076,8 @@ static void __init htab_init_page_sizes(void) bool aligned = true; init_hpte_page_sizes(); - if (!debug_pagealloc_enabled_or_kfence()) { + if (!debug_pagealloc_enabled() && + !(IS_ENABLED(CONFIG_KFENCE) && kfence_early_init)) { /* * Pick a size for the linear mapping. Currently, we only * support 16M, 1M and 4K which is the default