From patchwork Wed Jul 31 07:56:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Ritesh Harjani (IBM)" X-Patchwork-Id: 1966924 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=U1ngZo3A; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WYl1y1WXJz1yYq for ; Wed, 31 Jul 2024 18:02:30 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=U1ngZo3A; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WYl1y0RhDz3d8R for ; Wed, 31 Jul 2024 18:02:30 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=U1ngZo3A; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::42c; helo=mail-pf1-x42c.google.com; envelope-from=ritesh.list@gmail.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WYkwk4D0Gz3dC5 for ; Wed, 31 Jul 2024 17:57:58 +1000 (AEST) Received: by mail-pf1-x42c.google.com with SMTP id d2e1a72fcca58-70d23caf8ddso4781108b3a.0 for ; Wed, 31 Jul 2024 00:57:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722412675; x=1723017475; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SlswoT2b+orJh1kLQ/S6utrJaivvXqzqOuHKXYuEfSY=; b=U1ngZo3Am7T+s60woAo7H4GbC/awXFpvLTS6k4RcODzZ6ONnghKPc0AGb86bdAfLZL tYfIs3cPW6wUtfy8kwepvqQ01p1RvcYnvimq85KLRk9hJDxqfEMj7o26HK/1afuj5lG4 DfogM7DIEpa3HByMqGFw/3AZwbmuj1PCWYNky34V2RA41MNuKcfglc3LuFJqfyuoR5I/ pejREdFdYYQWG53SGDlXnQDsRcCzV5933xv2U3UuXkicyf5j4HrmqGagZWdaDsKqQRjP hVjC5LmS2jBndWRfgRGOM+sVCFARs3mkPvcPUEBkDfbj9ivXmCjv+D+tO1x0alD4yHxQ kNnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722412675; x=1723017475; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SlswoT2b+orJh1kLQ/S6utrJaivvXqzqOuHKXYuEfSY=; b=jqow4f/mLdPK040E/uP2jWG3T7T88jkfhp3oIrY8/agEpQhkZjrT0nNUhRuiTikIjH oHsZJe4TVsxUktDbbItZWDNFYAEHOi+Frs2gC9RsRq0YUwHAypnEQ3FnaJ/XE52EOK4I jMbU7AfyZGqgeSw+nbQ2giy/9sdQP42XxGf0y+8U9D0jC61jQ5sla08zP99xQHPTqVgm KRXgVHGIE2QLD8+KYV5WUrklnxEOkfI7oslL2qkMWV3wsYYNASuVFvcEQXYah5LClwFr MW1CQ5VWDz3fGChpBBALKuloEssJhYJykUEwjwXIrZxD2Z4VCxboNaLUH8ObbksHEz24 QnvQ== X-Gm-Message-State: AOJu0Yw9weWCZrleURTKDz0ak2zdP0Da+Fu1dRDJ9Yiubf54rnexQ9Hr 5qAPA1BLCKzXWDgTIDm3S/03gUgIGvVuy0GExOm9dUR/1eGlJt4wG0nv9Q== X-Google-Smtp-Source: AGHT+IHm+gtmm0YCt3trxm82F89ieYgjTcfOm8zqblI880B+4sGI4rBJ/zXAFGEY2gBXIBmare9gNQ== X-Received: by 2002:a05:6a00:4f83:b0:706:74be:686e with SMTP id d2e1a72fcca58-70ecedb2fa1mr16092812b3a.26.1722412675176; Wed, 31 Jul 2024 00:57:55 -0700 (PDT) Received: from dw-tp.ibmuc.com ([171.76.83.154]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-70ead6e16fbsm9788372b3a.19.2024.07.31.00.57.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jul 2024 00:57:54 -0700 (PDT) From: "Ritesh Harjani (IBM)" To: linuxppc-dev Subject: [RFC v1 08/10] book3s64/hash: Add kfence functionality Date: Wed, 31 Jul 2024 13:26:12 +0530 Message-ID: <7450b71d41093795674fe0314444492b2ff70e58.1722408881.git.ritesh.list@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Donet Tom , Madhavan Srinivasan , "Ritesh Harjani \(IBM\)" , Christophe Leroy , Pavithra Prakash , "Aneesh Kumar K . V" , Nicholas Piggin , Hari Bathini Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Now that linear map functionality of debug_pagealloc is made generic, enable kfence to use this generic infrastructure. 1. Define kfence related linear map variables. - u8 *linear_map_kf_hash_slots; - unsigned long linear_map_kf_hash_count; - DEFINE_RAW_SPINLOCK(linear_map_kf_hash_lock); 2. The linear map size allocated in RMA region is quite small (KFENCE_POOL_SIZE >> PAGE_SHIFT) which is 512 bytes by default. 3. kfence pool memory is reserved using memblock_phys_alloc() which has can come from anywhere. (default 255 objects => ((1+255) * 2) << PAGE_SHIFT = 32MB) 4. The hash slot information for kfence memory gets added in linear map in hash_linear_map_add_slot() (which also adds for debug_pagealloc). Reported-by: Pavithra Prakash Signed-off-by: Ritesh Harjani (IBM) --- arch/powerpc/include/asm/kfence.h | 5 - arch/powerpc/mm/book3s64/hash_utils.c | 162 +++++++++++++++++++++++--- 2 files changed, 149 insertions(+), 18 deletions(-) diff --git a/arch/powerpc/include/asm/kfence.h b/arch/powerpc/include/asm/kfence.h index f3a9476a71b3..fab124ada1c7 100644 --- a/arch/powerpc/include/asm/kfence.h +++ b/arch/powerpc/include/asm/kfence.h @@ -10,7 +10,6 @@ #include #include -#include #ifdef CONFIG_PPC64_ELF_ABI_V1 #define ARCH_FUNC_PREFIX "." @@ -26,10 +25,6 @@ static inline void disable_kfence(void) static inline bool arch_kfence_init_pool(void) { -#ifdef CONFIG_PPC64 - if (!radix_enabled()) - return false; -#endif return !kfence_disabled; } #endif diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index 906cd167180a..c66b9921fc7d 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -40,6 +40,7 @@ #include #include #include +#include #include #include @@ -66,6 +67,7 @@ #include #include #include +#include #include @@ -271,7 +273,7 @@ void hash__tlbiel_all(unsigned int action) WARN(1, "%s called on pre-POWER7 CPU\n", __func__); } -#if defined(CONFIG_DEBUG_PAGEALLOC) +#if defined(CONFIG_DEBUG_PAGEALLOC) || defined(CONFIG_KFENCE) static void kernel_map_linear_page(unsigned long vaddr, unsigned long idx, u8 *slots, raw_spinlock_t *lock) { @@ -325,11 +327,13 @@ static void kernel_unmap_linear_page(unsigned long vaddr, unsigned long idx, mmu_linear_psize, mmu_kernel_ssize, 0); } +#endif +#if defined(CONFIG_DEBUG_PAGEALLOC) static u8 *linear_map_hash_slots; static unsigned long linear_map_hash_count; static DEFINE_RAW_SPINLOCK(linear_map_hash_lock); -static inline void hash_debug_pagealloc_alloc_slots(void) +static void hash_debug_pagealloc_alloc_slots(void) { unsigned long max_hash_count = (ppc64_rma_size / 4) >> PAGE_SHIFT; @@ -352,7 +356,8 @@ static inline void hash_debug_pagealloc_alloc_slots(void) __func__, linear_map_hash_count, &ppc64_rma_size); } -static inline void hash_debug_pagealloc_add_slot(phys_addr_t paddr, int slot) +static inline void hash_debug_pagealloc_add_slot(phys_addr_t paddr, + int slot) { if (!debug_pagealloc_enabled() || !linear_map_hash_count) return; @@ -386,20 +391,148 @@ static int hash_debug_pagealloc_map_pages(struct page *page, int numpages, return 0; } -int hash__kernel_map_pages(struct page *page, int numpages, int enable) +#else /* CONFIG_DEBUG_PAGEALLOC */ +static inline void hash_debug_pagealloc_alloc_slots(void) {} +static inline void hash_debug_pagealloc_add_slot(phys_addr_t paddr, int slot) {} +static int __maybe_unused +hash_debug_pagealloc_map_pages(struct page *page, int numpages, int enable) { - return hash_debug_pagealloc_map_pages(page, numpages, enable); + return 0; } +#endif /* CONFIG_DEBUG_PAGEALLOC */ -#else /* CONFIG_DEBUG_PAGEALLOC */ -int hash__kernel_map_pages(struct page *page, int numpages, - int enable) +#if defined(CONFIG_KFENCE) +static u8 *linear_map_kf_hash_slots; +static unsigned long linear_map_kf_hash_count; +static DEFINE_RAW_SPINLOCK(linear_map_kf_hash_lock); + +static phys_addr_t kfence_pool; + +static inline void hash_kfence_alloc_pool(void) +{ + + // allocate linear map for kfence within RMA region + linear_map_kf_hash_count = KFENCE_POOL_SIZE >> PAGE_SHIFT; + linear_map_kf_hash_slots = memblock_alloc_try_nid( + linear_map_kf_hash_count, 1, + MEMBLOCK_LOW_LIMIT, ppc64_rma_size, + NUMA_NO_NODE); + if (!linear_map_kf_hash_slots) { + pr_err("%s: memblock for linear map (%lu) failed\n", __func__, + linear_map_kf_hash_count); + goto err; + } + + // allocate kfence pool early + kfence_pool = memblock_phys_alloc_range(KFENCE_POOL_SIZE, PAGE_SIZE, + MEMBLOCK_LOW_LIMIT, MEMBLOCK_ALLOC_ANYWHERE); + if (!kfence_pool) { + pr_err("%s: memblock for kfence pool (%lu) failed\n", __func__, + KFENCE_POOL_SIZE); + memblock_free(linear_map_kf_hash_slots, + linear_map_kf_hash_count); + linear_map_kf_hash_count = 0; + goto err; + } + memblock_mark_nomap(kfence_pool, KFENCE_POOL_SIZE); + + return; +err: + pr_info("Disabling kfence\n"); + disable_kfence(); +} + +static inline void hash_kfence_map_pool(void) +{ + unsigned long kfence_pool_start, kfence_pool_end; + unsigned long prot = pgprot_val(PAGE_KERNEL); + + if (!kfence_pool) + return; + + kfence_pool_start = (unsigned long) __va(kfence_pool); + kfence_pool_end = kfence_pool_start + KFENCE_POOL_SIZE; + __kfence_pool = (char *) kfence_pool_start; + BUG_ON(htab_bolt_mapping(kfence_pool_start, kfence_pool_end, + kfence_pool, prot, mmu_linear_psize, + mmu_kernel_ssize)); + memblock_clear_nomap(kfence_pool, KFENCE_POOL_SIZE); +} + +static inline void hash_kfence_add_slot(phys_addr_t paddr, int slot) { + unsigned long vaddr = (unsigned long) __va(paddr); + unsigned long lmi = (vaddr - (unsigned long)__kfence_pool) + >> PAGE_SHIFT; + + if (!kfence_pool) + return; + BUG_ON(!is_kfence_address((void *)vaddr)); + BUG_ON(lmi >= linear_map_kf_hash_count); + linear_map_kf_hash_slots[lmi] = slot | 0x80; +} + +static int hash_kfence_map_pages(struct page *page, int numpages, int enable) +{ + unsigned long flags, vaddr, lmi; + int i; + + WARN_ON_ONCE(!linear_map_kf_hash_count); + local_irq_save(flags); + for (i = 0; i < numpages; i++, page++) { + vaddr = (unsigned long)page_address(page); + lmi = (vaddr - (unsigned long)__kfence_pool) >> PAGE_SHIFT; + + /* Ideally this should never happen */ + if (lmi >= linear_map_kf_hash_count) { + WARN_ON_ONCE(1); + continue; + } + + if (enable) + kernel_map_linear_page(vaddr, lmi, + linear_map_kf_hash_slots, + &linear_map_kf_hash_lock); + else + kernel_unmap_linear_page(vaddr, lmi, + linear_map_kf_hash_slots, + &linear_map_kf_hash_lock); + } + local_irq_restore(flags); return 0; } -static inline void hash_debug_pagealloc_alloc_slots(void) {} -static inline void hash_debug_pagealloc_add_slot(phys_addr_t paddr, int slot) {} -#endif /* CONFIG_DEBUG_PAGEALLOC */ +#else +static inline void hash_kfence_alloc_pool(void) {} +static inline void hash_kfence_map_pool(void) {} +static inline void hash_kfence_add_slot(phys_addr_t paddr, int slot) {} +static int __maybe_unused +hash_kfence_map_pages(struct page *page, int numpages, int enable) +{ + return 0; +} +#endif + +#if defined(CONFIG_DEBUG_PAGEALLOC) || defined(CONFIG_KFENCE) +int hash__kernel_map_pages(struct page *page, int numpages, int enable) +{ + void *vaddr = page_address(page); + + if (is_kfence_address(vaddr)) + return hash_kfence_map_pages(page, numpages, enable); + else + return hash_debug_pagealloc_map_pages(page, numpages, enable); +} + +static void hash_linear_map_add_slot(phys_addr_t paddr, int slot) +{ + if (is_kfence_address(__va(paddr))) + hash_kfence_add_slot(paddr, slot); + else + hash_debug_pagealloc_add_slot(paddr, slot); +} +#else +static void hash_linear_map_add_slot(phys_addr_t paddr, int slot) {} +#endif /* * 'R' and 'C' update notes: @@ -559,7 +692,8 @@ int htab_bolt_mapping(unsigned long vstart, unsigned long vend, break; cond_resched(); - hash_debug_pagealloc_add_slot(paddr, ret); + // add slot info in debug_pagealloc / kfence linear map + hash_linear_map_add_slot(paddr, ret); } return ret < 0 ? ret : 0; } @@ -940,7 +1074,7 @@ static void __init htab_init_page_sizes(void) bool aligned = true; init_hpte_page_sizes(); - if (!debug_pagealloc_enabled()) { + if (!debug_pagealloc_enabled_or_kfence()) { /* * Pick a size for the linear mapping. Currently, we only * support 16M, 1M and 4K which is the default @@ -1261,6 +1395,7 @@ static void __init htab_initialize(void) prot = pgprot_val(PAGE_KERNEL); hash_debug_pagealloc_alloc_slots(); + hash_kfence_alloc_pool(); /* create bolted the linear mapping in the hash table */ for_each_mem_range(i, &base, &end) { size = end - base; @@ -1277,6 +1412,7 @@ static void __init htab_initialize(void) BUG_ON(htab_bolt_mapping(base, base + size, __pa(base), prot, mmu_linear_psize, mmu_kernel_ssize)); } + hash_kfence_map_pool(); memblock_set_current_limit(MEMBLOCK_ALLOC_ANYWHERE); /*