From patchwork Wed Apr 24 11:09:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hari Bathini X-Patchwork-Id: 1927112 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=WplcODHK; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VPbrM0rzcz1yZr for ; Wed, 24 Apr 2024 21:10:43 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=WplcODHK; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4VPbrL6tdRz3cgk for ; Wed, 24 Apr 2024 21:10:42 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=WplcODHK; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=hbathini@linux.ibm.com; receiver=lists.ozlabs.org) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4VPbqD3rmRz3cJl for ; Wed, 24 Apr 2024 21:09:43 +1000 (AEST) Received: from pps.filterd (m0353726.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 43OAw7oF024942; Wed, 24 Apr 2024 11:09:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : content-transfer-encoding : mime-version; s=pp1; bh=pZwrcCGPr2IFK7BeGC4ojUZLtSftfzcRsbiZ791Q1vk=; b=WplcODHK4Zf4iYr2LwSrzaKWTrvZ15wr8tYl8pR65FL9JGa51Urk3WtMKbBctNp6TYIM 0Lb970dOeU5zBZ+hM8k6dyJQPqmM+c6S6ULk3cneHAGuhQLqzlb73sLzBtAnhsxddCmB DKf7tLqljVhjHN8Em/KKuvTA+UF/QFMAEw1Wgmq5Mu8U4hye+7kBF9vLIVXYmZwxJTYG xk5U1tu7arekE6wHNiAFomtVcfqumD7ePxsIstciZ7RlJgeeHC/WD4eu8e4b795gDuoM VAS3sl4bthAHJYPKXdW2/c58lS3Uwy7+e2V6aSshZQogVkoYUtnxOcjDao3YDyrmus1u gw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3xq0sar0qx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 24 Apr 2024 11:09:36 +0000 Received: from m0353726.ppops.net (m0353726.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 43OB9aNR012359; Wed, 24 Apr 2024 11:09:36 GMT Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3xq0sar0qt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 24 Apr 2024 11:09:36 +0000 Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 43O9jsba005736; Wed, 24 Apr 2024 11:09:35 GMT Received: from smtprelay06.fra02v.mail.ibm.com ([9.218.2.230]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 3xmx3cj0hp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 24 Apr 2024 11:09:35 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay06.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 43OB9Tov34996772 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 24 Apr 2024 11:09:32 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D2A0420040; Wed, 24 Apr 2024 11:09:29 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C3D4820043; Wed, 24 Apr 2024 11:09:27 +0000 (GMT) Received: from li-bd3f974c-2712-11b2-a85c-df1cec4d728e.in.ibm.com (unknown [9.203.115.195]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Wed, 24 Apr 2024 11:09:27 +0000 (GMT) From: Hari Bathini To: linuxppc-dev , Michael Ellerman Subject: [PATCH 1/2] radix/kfence: map __kfence_pool at page granularity Date: Wed, 24 Apr 2024 16:39:25 +0530 Message-ID: <20240424110926.184077-1-hbathini@linux.ibm.com> X-Mailer: git-send-email 2.44.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: P_sYw4f4LeE6qeLOqrjv5qGsNWvOfXGU X-Proofpoint-GUID: xBDnxHZi2FSHS4f3HMj3915KJXVg0Mwd X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1011,Hydra:6.0.650,FMLib:17.11.176.26 definitions=2024-04-24_08,2024-04-23_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 malwarescore=0 mlxlogscore=999 priorityscore=1501 phishscore=0 clxscore=1015 bulkscore=0 lowpriorityscore=0 spamscore=0 mlxscore=0 suspectscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2404010000 definitions=main-2404240045 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Marco Elver , "Aneesh Kumar K.V" , Nicholas Piggin , Alexander Potapenko , "Naveen N. Rao" , Dmitry Vyukov Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" When KFENCE is enabled, total system memory is mapped at page level granularity. But in radix MMU mode, ~3GB additional memory is needed to map 100GB of system memory at page level granularity when compared to using 2MB direct mapping. This is not desired considering KFENCE is designed to be enabled in production kernels [1]. Also, mapping memory allocated for KFENCE pool at page granularity seems sufficient enough to enable KFENCE support. So, allocate __kfence_pool during bootup and map it at page granularity instead of mapping all system memory at page granularity. Without patch: # cat /proc/meminfo MemTotal: 101201920 kB With patch: # cat /proc/meminfo MemTotal: 104483904 kB All kfence_test.c testcases passed with this patch. [1] https://lore.kernel.org/all/20201103175841.3495947-2-elver@google.com/ Signed-off-by: Hari Bathini --- arch/powerpc/include/asm/kfence.h | 5 ++++ arch/powerpc/mm/book3s64/radix_pgtable.c | 34 ++++++++++++++++++------ arch/powerpc/mm/init_64.c | 14 ++++++++++ 3 files changed, 45 insertions(+), 8 deletions(-) diff --git a/arch/powerpc/include/asm/kfence.h b/arch/powerpc/include/asm/kfence.h index 424ceef82ae6..18ec2b06ba1e 100644 --- a/arch/powerpc/include/asm/kfence.h +++ b/arch/powerpc/include/asm/kfence.h @@ -8,6 +8,7 @@ #ifndef __ASM_POWERPC_KFENCE_H #define __ASM_POWERPC_KFENCE_H +#include #include #include @@ -15,6 +16,10 @@ #define ARCH_FUNC_PREFIX "." #endif +#ifdef CONFIG_KFENCE +extern bool kfence_early_init; +#endif + static inline bool arch_kfence_init_pool(void) { return true; diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 15e88f1439ec..fccbf92f279b 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -31,6 +31,7 @@ #include #include #include +#include #include @@ -291,9 +292,8 @@ static unsigned long next_boundary(unsigned long addr, unsigned long end) return end; } -static int __meminit create_physical_mapping(unsigned long start, - unsigned long end, - int nid, pgprot_t _prot) +static int __meminit create_physical_mapping(unsigned long start, unsigned long end, int nid, + pgprot_t _prot, unsigned long mapping_sz_limit) { unsigned long vaddr, addr, mapping_size = 0; bool prev_exec, exec = false; @@ -301,7 +301,10 @@ static int __meminit create_physical_mapping(unsigned long start, int psize; unsigned long max_mapping_size = memory_block_size; - if (debug_pagealloc_enabled_or_kfence()) + if (mapping_sz_limit < max_mapping_size) + max_mapping_size = mapping_sz_limit; + + if (debug_pagealloc_enabled()) max_mapping_size = PAGE_SIZE; start = ALIGN(start, PAGE_SIZE); @@ -358,6 +361,7 @@ static int __meminit create_physical_mapping(unsigned long start, static void __init radix_init_pgtable(void) { + phys_addr_t kfence_pool __maybe_unused; unsigned long rts_field; phys_addr_t start, end; u64 i; @@ -365,6 +369,13 @@ static void __init radix_init_pgtable(void) /* We don't support slb for radix */ slb_set_size(0); +#ifdef CONFIG_KFENCE + if (kfence_early_init) { + kfence_pool = memblock_phys_alloc(KFENCE_POOL_SIZE, PAGE_SIZE); + memblock_mark_nomap(kfence_pool, KFENCE_POOL_SIZE); + } +#endif + /* * Create the linear mapping */ @@ -380,10 +391,18 @@ static void __init radix_init_pgtable(void) continue; } - WARN_ON(create_physical_mapping(start, end, - -1, PAGE_KERNEL)); + WARN_ON(create_physical_mapping(start, end, -1, PAGE_KERNEL, ~0UL)); } +#ifdef CONFIG_KFENCE + if (kfence_early_init) { + create_physical_mapping(kfence_pool, kfence_pool + KFENCE_POOL_SIZE, -1, + PAGE_KERNEL, PAGE_SIZE); + memblock_clear_nomap(kfence_pool, KFENCE_POOL_SIZE); + __kfence_pool = __va(kfence_pool); + } +#endif + if (!cpu_has_feature(CPU_FTR_HVMODE) && cpu_has_feature(CPU_FTR_P9_RADIX_PREFETCH_BUG)) { /* @@ -874,8 +893,7 @@ int __meminit radix__create_section_mapping(unsigned long start, return -1; } - return create_physical_mapping(__pa(start), __pa(end), - nid, prot); + return create_physical_mapping(__pa(start), __pa(end), nid, prot, ~0UL); } int __meminit radix__remove_section_mapping(unsigned long start, unsigned long end) diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c index d96bbc001e73..8155bfd6c16b 100644 --- a/arch/powerpc/mm/init_64.c +++ b/arch/powerpc/mm/init_64.c @@ -64,6 +64,20 @@ #include +#ifdef CONFIG_KFENCE +bool __ro_after_init kfence_early_init = !!CONFIG_KFENCE_SAMPLE_INTERVAL; + +static int __init parse_kfence_early_init(char *arg) +{ + int val; + + if (get_option(&arg, &val)) + kfence_early_init = !!val; + return 0; +} +early_param("kfence.sample_interval", parse_kfence_early_init); +#endif + #ifdef CONFIG_SPARSEMEM_VMEMMAP /* * Given an address within the vmemmap, determine the page that From patchwork Wed Apr 24 11:09:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hari Bathini X-Patchwork-Id: 1927111 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=VBp/BXDV; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VPbqf57S9z1yZr for ; Wed, 24 Apr 2024 21:10:05 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=VBp/BXDV; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4VPbqY5cC5z2yD6 for ; Wed, 24 Apr 2024 21:10:01 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=VBp/BXDV; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=hbathini@linux.ibm.com; receiver=lists.ozlabs.org) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4VPbqD37NGz3cBK for ; Wed, 24 Apr 2024 21:09:43 +1000 (AEST) Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 43OB4gCR002566; Wed, 24 Apr 2024 11:09:40 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=HLU7ONeu9Nvh3ksMCOGNKf5vsMDI83v7vnizH8GI6FE=; b=VBp/BXDVo2v/El3aXQgjRosQuZo6jlNO7eXnZ+uMxcOAC4kXwXKuFbazbW6g3X9lG3Tc exLtLgU9Ylb7es9tyu8euXpRVZ4GjlyKHG3hR/VmKKTkIfJBPd/l0voGDgfmQTq3+L+2 Mh7f2uc9L85+kL8RXQGNfB4t4GvaIwB44pMvuq3pOc9c56YcwSQVCf+P8q1/Y7ZeNJNI MxnL1ZH+M3pCpeDX0ICmB3xO8gTk4KAztVmaC3xWRVqwZcMdMsCRIsPCFTQZjSZjBEsa 9IAGkOzKYiGB6Xnsn9ytWLDdF9W8TTRk2yjLpbBPgLqnlgor1gNKHZCbqyZJq8YnrCcJ 0Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3xq0w0r087-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 24 Apr 2024 11:09:39 +0000 Received: from m0356517.ppops.net (m0356517.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 43OB9d4I009343; Wed, 24 Apr 2024 11:09:39 GMT Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3xq0w0r082-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 24 Apr 2024 11:09:39 +0000 Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 43O81EeT023051; Wed, 24 Apr 2024 11:09:37 GMT Received: from smtprelay03.fra02v.mail.ibm.com ([9.218.2.224]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3xms1p38et-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 24 Apr 2024 11:09:37 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay03.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 43OB9WJ953412294 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 24 Apr 2024 11:09:34 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 40D8520040; Wed, 24 Apr 2024 11:09:32 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 36B3420043; Wed, 24 Apr 2024 11:09:30 +0000 (GMT) Received: from li-bd3f974c-2712-11b2-a85c-df1cec4d728e.in.ibm.com (unknown [9.203.115.195]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Wed, 24 Apr 2024 11:09:30 +0000 (GMT) From: Hari Bathini To: linuxppc-dev , Michael Ellerman Subject: [PATCH 2/2] radix/kfence: support late __kfence_pool allocation Date: Wed, 24 Apr 2024 16:39:26 +0530 Message-ID: <20240424110926.184077-2-hbathini@linux.ibm.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240424110926.184077-1-hbathini@linux.ibm.com> References: <20240424110926.184077-1-hbathini@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: bJt_UXpwhSWuME7RNrIQvHiyTHhhsv8k X-Proofpoint-ORIG-GUID: m0z3pf3zByM-oeghEgxjC-eftHIYcbAs X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1011,Hydra:6.0.650,FMLib:17.11.176.26 definitions=2024-04-24_08,2024-04-23_02,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 adultscore=0 impostorscore=0 mlxlogscore=999 lowpriorityscore=0 spamscore=0 phishscore=0 mlxscore=0 malwarescore=0 priorityscore=1501 bulkscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2404010000 definitions=main-2404240045 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Marco Elver , "Aneesh Kumar K.V" , Nicholas Piggin , Alexander Potapenko , "Naveen N. Rao" , Dmitry Vyukov Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" With commit b33f778bba5ef ("kfence: alloc kfence_pool after system startup"), KFENCE pool can be allocated after system startup via the page allocator. This can lead to problems as all memory is not mapped at page granularity anymore with CONFIG_KFENCE. Address this by direct mapping all memory at PMD level and split the mapping for PMD pages that overlap with __kfence_pool to page level granularity if and when __kfence_pool is allocated after system startup. Signed-off-by: Hari Bathini --- arch/powerpc/include/asm/book3s/64/radix.h | 2 + arch/powerpc/include/asm/kfence.h | 14 +++++- arch/powerpc/mm/book3s64/radix_pgtable.c | 50 +++++++++++++++++++++- 3 files changed, 64 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index 8f55ff74bb68..0423ddbcf73c 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -340,6 +340,8 @@ extern void radix__vmemmap_remove_mapping(unsigned long start, extern int radix__map_kernel_page(unsigned long ea, unsigned long pa, pgprot_t flags, unsigned int psz); +extern bool radix_kfence_init_pool(void); + static inline unsigned long radix__get_tree_size(void) { unsigned long rts_field; diff --git a/arch/powerpc/include/asm/kfence.h b/arch/powerpc/include/asm/kfence.h index 18ec2b06ba1e..c5d2fb2f9ecb 100644 --- a/arch/powerpc/include/asm/kfence.h +++ b/arch/powerpc/include/asm/kfence.h @@ -18,12 +18,24 @@ #ifdef CONFIG_KFENCE extern bool kfence_early_init; -#endif + +static inline bool kfence_alloc_pool_late(void) +{ + return !kfence_early_init; +} static inline bool arch_kfence_init_pool(void) { +#ifdef CONFIG_PPC_BOOK3S_64 + if (radix_enabled()) + return radix_kfence_init_pool(); +#endif + return true; } +#else +static inline bool kfence_alloc_pool_late(void) { return false; } +#endif #ifdef CONFIG_PPC64 static inline bool kfence_protect_page(unsigned long addr, bool protect) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index fccbf92f279b..f4374e3e31e1 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -253,6 +253,53 @@ void radix__mark_initmem_nx(void) } #endif /* CONFIG_STRICT_KERNEL_RWX */ +#ifdef CONFIG_KFENCE +static inline int radix_split_pmd_page(pmd_t *pmd, unsigned long addr) +{ + pte_t *pte = pte_alloc_one_kernel(&init_mm); + unsigned long pfn = PFN_DOWN(__pa(addr)); + int i; + + if (!pte) + return -ENOMEM; + + for (i = 0; i < PTRS_PER_PTE; i++) { + __set_pte_at(&init_mm, addr, pte + i, pfn_pte(pfn + i, PAGE_KERNEL), 0); + asm volatile("ptesync": : :"memory"); + } + pmd_populate_kernel(&init_mm, pmd, pte); + + flush_tlb_kernel_range(addr, addr + PMD_SIZE); + return 0; +} + +bool radix_kfence_init_pool(void) +{ + unsigned int page_psize, pmd_psize; + unsigned long addr; + pmd_t *pmd; + + if (!kfence_alloc_pool_late()) + return true; + + page_psize = shift_to_mmu_psize(PAGE_SHIFT); + pmd_psize = shift_to_mmu_psize(PMD_SHIFT); + for (addr = (unsigned long)__kfence_pool; is_kfence_address((void *)addr); + addr += PAGE_SIZE) { + pmd = pmd_off_k(addr); + + if (pmd_leaf(*pmd)) { + if (radix_split_pmd_page(pmd, addr & PMD_MASK)) + return false; + update_page_count(pmd_psize, -1); + update_page_count(page_psize, PTRS_PER_PTE); + } + } + + return true; +} +#endif + static inline void __meminit print_mapping(unsigned long start, unsigned long end, unsigned long size, bool exec) { @@ -391,7 +438,8 @@ static void __init radix_init_pgtable(void) continue; } - WARN_ON(create_physical_mapping(start, end, -1, PAGE_KERNEL, ~0UL)); + WARN_ON(create_physical_mapping(start, end, -1, PAGE_KERNEL, + kfence_alloc_pool_late() ? PMD_SIZE : ~0UL)); } #ifdef CONFIG_KFENCE