From patchwork Wed May 8 19:19:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 1933174 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=GyqFZeJT; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=SIfVnHkC; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VZQDg6jWZz1xnT for ; Thu, 9 May 2024 05:28:51 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=6O587SGlsYbHpYBL7jL2MtJgyFRrQIaqIeTyMVgZDaY=; b=GyqFZeJTLarFm6 lemw3K28QdQpuZpNmX/S/YYrcEtMgwerqFjdwV7A4qCPBi6bsPwHjR3fNl3DHYE9YgHXRSN8Xqy7i PLlzesstmxPMHGmeA0qGptSLj8Zc7YzKMLUH2zJFyPB3Ai+HrGpljdMVbKeKQRZ4CAW2l0AHFOLRP JrMDkYuZp+JYTy6msxOzQfQYjKEXSLiaBqt+y6rXDP079LeuMriew5mRuEgWbDgBjLhcjq13AHmXR HalppTebsvAYgD08de84PDPSFq/J4I26J7M1KNOhdSSGkXi++tnenS0Yk+0gMnEVgjkWINM5w0FSa auFBKWkppc9OiT8SPcMQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4myA-0000000GkCt-2bZa; Wed, 08 May 2024 19:28:50 +0000 Received: from mail-wm1-x336.google.com ([2a00:1450:4864:20::336]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4my7-0000000GkBM-2ohB for kvm-riscv@lists.infradead.org; Wed, 08 May 2024 19:28:49 +0000 Received: by mail-wm1-x336.google.com with SMTP id 5b1f17b1804b1-41b794510cdso601655e9.2 for ; Wed, 08 May 2024 12:28:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196526; x=1715801326; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SrlnOIvJhP/2Gfg4T9gM6ETbApI/dif7rW/ZR+x6Q9w=; b=SIfVnHkCie+hKEQP1F11dwW6kROYkdvWaUxuxpBoppwOY/guxPVLWdVta2q9ujrRM9 ZJKoH1kPSMkqErmAv4gefMavdTGZDksQfDKr5/CBfuAb6tmkz9Sc2BvDAz4fl6QMVSFP 2v4lhwYRo90MZIKQ97SToHSnOhE2kg4fYyyXNUHj8EFAylmH37WjzL2m1AE7urJ6IgwX mWv2+qdvpXIlZpGEcIYUXqlkbxzsFG3j5WPGLTIKAYzRTekRaGZNhpWerBILSJ8K0PBA 1/lBsEPoTzXQMzjf0wpDIRZmhB8c6wiL6s5nABSgECoJhoAqfnEHkDT1+OmSq8isBeUc mFlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196526; x=1715801326; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SrlnOIvJhP/2Gfg4T9gM6ETbApI/dif7rW/ZR+x6Q9w=; b=Cz/YIobYMnEtfLxFn4lz7pdHiXKHwi0sfgJxfkeFh2pUli1VFxAU0oQgms9WzxC3Ig Tg+LZ1SbezSA6PUDa4qYDvjumrWhGJMaRfy+aJeOyehE0RIh4rk7htNLbDNWtIJogggA EsxCgSwCoZLUoU2VrpNSWQl+UI0Mka6KQ6t7aGRsF1oF9MwYudavvfLARtkzGU3h7pAD GCxXXWMUtkRAs9qP+zCKCFWVCOrqSy3X9VjmMyaf1C7AV41102sRWOqInjh4AiYiG/GA AbDgfOenUf2eMtzWl4ryt/EQWnZJi38Fp5IKMsLrbFwRWR/k+LMBhGTZK8CDKKnKd2dL QTPw== X-Forwarded-Encrypted: i=1; AJvYcCVkakYCP9xVhcgBxr/R9Y/xxn0qG6ae5ImMUHuZ2e1fMi77AiqUNRF6Ikrcja8JUd4ZlxnjE/HfgBGTUdJXeUZERLcrtE7b9zru+WxgiQ== X-Gm-Message-State: AOJu0YxmRetBwrRnS9UaDDLGqr3LIc5pd4NZAyodWz3swJwcSyfhQooF LCJwqsHxugj59hxNsUtopmw5Z00naNVjbW9qjf0I044nKnOjl39cz7KhafpV41s= X-Google-Smtp-Source: AGHT+IEyIjL1IsEMqXGLkJGiy2tkR0X5tVHC2TKSU1+3Y1r/Xzm4QA5qwQ487USNifryKvOLnO9vbQ== X-Received: by 2002:a5d:6350:0:b0:34c:65ba:5d43 with SMTP id ffacd0b85a97d-34fca621699mr2523239f8f.46.1715196526270; Wed, 08 May 2024 12:28:46 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id cx8-20020a056000092800b0034e01a80176sm16002694wrb.114.2024.05.08.12.28.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:28:45 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 09/12] mm, riscv, arm64: Use common ptep_clear_flush_young() function Date: Wed, 8 May 2024 21:19:28 +0200 Message-Id: <20240508191931.46060-10-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240508_122847_742194_3A21F19B X-CRM114-Status: GOOD ( 16.74 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Make riscv use the contpte aware ptep_clear_flush_young() function from arm64. Note that riscv used to not flush the tlb after clearing the accessed bit, which it does now: this will be improved when we implement svinval support. Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:336 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Make riscv use the contpte aware ptep_clear_flush_young() function from arm64. Note that riscv used to not flush the tlb after clearing the accessed bit, which it does now: this will be improved when we implement svinval support. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 22 ++++++++---------- arch/arm64/mm/contpte.c | 21 ----------------- arch/riscv/include/asm/pgtable.h | 12 +++++++--- include/linux/contpte.h | 2 ++ mm/contpte.c | 40 ++++++++++++++++++++++++++++++++ 5 files changed, 61 insertions(+), 36 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 9a8702d1ad00..92c12fb85cb4 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1389,8 +1389,6 @@ extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full); -extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep); extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr); extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, @@ -1479,16 +1477,8 @@ extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH -static inline int ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep) -{ - pte_t orig_pte = __ptep_get(ptep); - - if (likely(!pte_valid_cont(orig_pte))) - return __ptep_clear_flush_young(vma, addr, ptep); - - return contpte_ptep_clear_flush_young(vma, addr, ptep); -} +extern int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #define wrprotect_ptes wrprotect_ptes static __always_inline void wrprotect_ptes(struct mm_struct *mm, @@ -1616,6 +1606,14 @@ static inline void arch_contpte_flush_tlb_range(struct vm_area_struct *vma, __flush_tlb_range(vma, start, end, stride, true, 3); } +static inline void arch_contpte_flush_tlb_range_nosync(struct vm_area_struct *vma, + unsigned long start, + unsigned long end, + unsigned long stride) +{ + __flush_tlb_range_nosync(vma, start, end, stride, true, 3); +} + static inline int arch_contpte_get_first_ncontig(size_t *pgsize) { if (pgsize) diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 9bf471633ca4..16940511943c 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -45,27 +45,6 @@ pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, } EXPORT_SYMBOL_GPL(contpte_get_and_clear_full_ptes); -int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep) -{ - int young; - - young = contpte_ptep_test_and_clear_young(vma, addr, ptep); - - if (young) { - /* - * See comment in __ptep_clear_flush_young(); same rationale for - * eliding the trailing DSB applies here. - */ - addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); - __flush_tlb_range_nosync(vma, addr, addr + CONT_PTE_SIZE, - PAGE_SIZE, true, 3); - } - - return young; -} -EXPORT_SYMBOL_GPL(contpte_ptep_clear_flush_young); - void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr) { diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index d39cb24c6c4a..42c7884b8d2e 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -615,6 +615,8 @@ static inline void arch_contpte_flush_tlb_range(struct vm_area_struct *vma, flush_tlb_mm_range(vma->vm_mm, start, end, stride); } +#define arch_contpte_flush_tlb_range_nosync arch_contpte_flush_tlb_range + static inline int arch_contpte_get_first_ncontig(size_t *pgsize) { if (pgsize) @@ -758,9 +760,8 @@ static inline void __ptep_set_wrprotect(struct mm_struct *mm, atomic_long_and(~(unsigned long)_PAGE_WRITE, (atomic_long_t *)ptep); } -#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH -static inline int ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep) +static inline int __ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep) { /* * This comment is borrowed from x86, but applies equally to RISC-V: @@ -799,6 +800,9 @@ extern pte_t ptep_get_and_clear(struct mm_struct *mm, #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH +extern int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #else /* CONFIG_THP_CONTPTE */ @@ -810,6 +814,8 @@ extern int ptep_test_and_clear_young(struct vm_area_struct *vma, #define ptep_get_and_clear __ptep_get_and_clear #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG #define ptep_test_and_clear_young __ptep_test_and_clear_young +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH +#define ptep_clear_flush_young __ptep_clear_flush_young #endif /* CONFIG_THP_CONTPTE */ diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 38092adbe0d4..76a49ac8b6f5 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -21,5 +21,7 @@ void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 220e9d81f401..600277b1196c 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -48,6 +48,7 @@ * - pte_clear() * - ptep_get_and_clear() * - ptep_test_and_clear_young() + * - ptep_clear_flush_young() */ pte_t huge_ptep_get(pte_t *ptep) @@ -729,4 +730,43 @@ __always_inline int ptep_test_and_clear_young(struct vm_area_struct *vma, return contpte_ptep_test_and_clear_young(vma, addr, ptep); } + +int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + int young; + + young = contpte_ptep_test_and_clear_young(vma, addr, ptep); + + if (young) { + /* + * See comment in __ptep_clear_flush_young(); same rationale for + * eliding the trailing DSB applies here. + */ + size_t pgsize; + int ncontig; + + ncontig = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, + 0, &pgsize); + + addr = ALIGN_DOWN(addr, ncontig * pgsize); + arch_contpte_flush_tlb_range_nosync(vma, addr, + addr + ncontig * pgsize, + pgsize); + } + + return young; +} +EXPORT_SYMBOL_GPL(contpte_ptep_clear_flush_young); + +__always_inline int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + pte_t orig_pte = __ptep_get(ptep); + + if (likely(!pte_valid_cont(orig_pte))) + return __ptep_clear_flush_young(vma, addr, ptep); + + return contpte_ptep_clear_flush_young(vma, addr, ptep); +} #endif /* CONFIG_THP_CONTPTE */