From patchwork Thu Aug 22 07:13:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1975284 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=iX7A4UIL; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-305-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WqDw727whz1ydn for ; Thu, 22 Aug 2024 17:14:15 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WqDvp3v7fz2yWy; Thu, 22 Aug 2024 17:13:58 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::22d" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=iX7A4UIL; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::22d; helo=mail-oi1-x22d.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-oi1-x22d.google.com (mail-oi1-x22d.google.com [IPv6:2607:f8b0:4864:20::22d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WqDvn5Fwgz2xDM for ; Thu, 22 Aug 2024 17:13:57 +1000 (AEST) Received: by mail-oi1-x22d.google.com with SMTP id 5614622812f47-3db16129143so245228b6e.0 for ; Thu, 22 Aug 2024 00:13:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310834; x=1724915634; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CO0xfYDzsVihapVFXw+qMWwHYfpsvq03XEdZOsUU02k=; b=iX7A4UILZ23lk9Ov1RyeDs5e36QYJzp0K068J5s77LBmJYboRj4SZxgzVUFhm+Dofu psgKkY534ZfkyYg7ZsqxtZaA+/jSntlsA/VjDXz4nTZvmVY3My7HNLysUIEd9DIXDy6a wgyXxma6rWeKCS3xajW675e8ke/4y5JjTd1o3d71dkB5TNEPJJsVucppk9vO3bcd+PO1 pwyVbdwUp4bv3nxcrKe1m/q39QI3BAsvOkO7U8EBQR9yvV9m9VWbKgu/6B8whZCXuryQ djKRECoFDQbBc3rPDaWemDu4wZ/wLt+l3BM7syvodf2aGwsRxdts6VH3p5d+ms+ketsV Qyag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310834; x=1724915634; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CO0xfYDzsVihapVFXw+qMWwHYfpsvq03XEdZOsUU02k=; b=VF1Co3tE+WvsTfJUjR65yigLlMPW46p4SHxDxrCTCT8lklvP32j0r5KCVTuEMNrH0c kTRlCwAjSjO8W2rrxqT6b6K/QESZ2wo81ZYFBgYpOxOLWkFnjQh2zFcjIMVZQ6oQCJe9 2lpz+vNsCIL/oG4xcsoIAEFjkvPgSQkriqYNwqoekxJTpbMDMfexk2s+PMZRC7h16JDi qLBZbYzJQcRmKE6QAoIu3l6v4UlYIhhPjIX6YrJPOyb1YGMTwQh4nDFQSoKsr0nxqFBo GchQ6cJ9LPUJZmrraBKcXVF4OlfXYdqArLQcu+x1CvG5ppsEoJorRehJ3sgC8W4dHjlx nNDg== X-Forwarded-Encrypted: i=1; AJvYcCVbviX+gAX7jXXd82RVyjIGTYaFYU897VGCI4BKhxg+p6Y9XdpZMErnfJCPCWuGtkxElBbExEp58FIloJI=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yy4PAhneEBwmkIevbqDh7mevbuH/IMSrJmH4NOQxefFWmvdEH5n 7SyqtIcrMwH5iWBYCdrLLS5IniD3wzDCqD0/i8xaNe6R41nEmX+aO6HJmyisePg= X-Google-Smtp-Source: AGHT+IHZvtclVw9AATG67WMtfqsJZD4bpWoHNpEdN87kwQejpgQ5CLrsmOGdaNcwygIvYs6P1O1W6A== X-Received: by 2002:a05:6808:148a:b0:3db:15ed:2a24 with SMTP id 5614622812f47-3de1950bb78mr6135912b6e.24.1724310834483; Thu, 22 Aug 2024 00:13:54 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.13.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:13:54 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 01/14] mm: pgtable: introduce pte_offset_map_{ro|rw}_nolock() Date: Thu, 22 Aug 2024 15:13:16 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 Currently, the usage of pte_offset_map_nolock() can be divided into the following two cases: 1) After acquiring PTL, only read-only operations are performed on the PTE page. In this case, the RCU lock in pte_offset_map_nolock() will ensure that the PTE page will not be freed, and there is no need to worry about whether the pmd entry is modified. 2) After acquiring PTL, the pte or pmd entries may be modified. At this time, we need to ensure that the pmd entry has not been modified concurrently. To more clearing distinguish between these two cases, this commit introduces two new helper functions to replace pte_offset_map_nolock(). For 1), just rename it to pte_offset_map_ro_nolock(). For 2), in addition to changing the name to pte_offset_map_rw_nolock(), it also outputs the pmdval when successful. This can help the caller recheck *pmd once the PTL is taken. In some cases, that is, either the mmap_lock for write, or pte_same() check on contents, is also enough to ensure that the pmd entry is stable. But in order to prevent the interface from being abused, we choose to pass in a dummy local variable instead of NULL. Subsequent commits will convert pte_offset_map_nolock() into the above two functions one by one, and finally completely delete it. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 7 ++++ include/linux/mm.h | 5 +++ mm/pgtable-generic.c | 43 ++++++++++++++++++++++ 3 files changed, 55 insertions(+) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index e4f6972eb6c04..08d0e706a32db 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -19,6 +19,13 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_ro_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_rw_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken) and the value of its pmd entry, or returns NULL + if no PTE table; - pte_offset_map() maps PTE, returns pointer to PTE, or returns NULL if no PTE table; - pte_unmap() diff --git a/include/linux/mm.h b/include/linux/mm.h index da29b066495d6..a00cb35ce065f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2954,6 +2954,11 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp); #define pte_unmap_unlock(pte, ptl) do { \ spin_unlock(ptl); \ diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index a78a4adf711ac..9a1666574c959 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -317,6 +317,33 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, return pte; } +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + return pte; +} + +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + BUG_ON(!pmdvalp); + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + *pmdvalp = pmdval; + return pte; +} + /* * pte_offset_map_lock(mm, pmd, addr, ptlp), and its internal implementation * __pte_offset_map_lock() below, is usually called with the pmd pointer for @@ -356,6 +383,22 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, * recheck *pmd once the lock is taken; in practice, no callsite needs that - * either the mmap_lock for write, or pte_same() check on contents, is enough. * + * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like + * pte_offset_map(); but when successful, it also outputs a pointer to the + * spinlock in ptlp - as pte_offset_map_lock() does, but in this case without + * locking it. This helps the caller to avoid a later pte_lockptr(mm, *pmd), + * which might by that time act on a changed *pmd: pte_offset_map_ro_nolock() + * provides the correct spinlock pointer for the page table that it returns. + * For readonly case, the caller does not need to recheck *pmd after the lock is + * taken, because the RCU lock will ensure that the PTE page will not be freed. + * + * pte_offset_map_rw_nolock(mm, pmd, addr, pmdvalp, ptlp), above, is like + * pte_offset_map_ro_nolock(); but when successful, it also outputs the + * pdmval. For cases where pte or pmd entries may be modified, that is, maywrite + * case, this can help the caller recheck *pmd once the lock is taken. In some + * cases, that is, either the mmap_lock for write, or pte_same() check on + * contents, is also enough to ensure that the pmd entry is stable. + * * Note that free_pgtables(), used after unmapping detached vmas, or when * exiting the whole mm, does not take page table lock before freeing a page * table, and may not use RCU at all: "outsiders" like khugepaged should avoid From patchwork Thu Aug 22 07:13:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1975285 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lMiRpoOs; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-306-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WqDwH0DyKz1ydn for ; Thu, 22 Aug 2024 17:14:23 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WqDvt1bf2z2yY9; Thu, 22 Aug 2024 17:14:02 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::532" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lMiRpoOs; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::532; helo=mail-pg1-x532.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x532.google.com (mail-pg1-x532.google.com [IPv6:2607:f8b0:4864:20::532]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WqDvs6x49z2xDM for ; Thu, 22 Aug 2024 17:14:01 +1000 (AEST) Received: by mail-pg1-x532.google.com with SMTP id 41be03b00d2f7-7cd70404bd5so164264a12.1 for ; Thu, 22 Aug 2024 00:14:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310840; x=1724915640; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=d/m1IXPfk7xvDU+Qn41r4NiqBFnWc/gEWMU/opSilGI=; b=lMiRpoOsp0jkh3gVw4x2u9+2sWo0jA2EhiUrXMsbTfXCBBJi2jRJkjMpMbUuBiAPcf ZLpXTuZYORHpZqNh+q4jCT8yCr21Wb9hQWBecakjeR/9HtWecp2G0ifm2GPS8Vz0JFhp DukgkOv5L/t34kLofm1AB+WioF6wL3hqvS3rSK6wMedJmiUUoXh61rNwhhETc3unmidW 8YDi5Ju9MyR0iD5fFDg4qWyfBy4M6R/E62JrQdJUgwWrBkSmhYyT2Usk3xsyvShLI80n VsBW+m6oVGwR7EbLY7Ldd6hezejlwgA1WV+mmeWyR3b1Q3cjUHL4jCIx2+Ak4JMOi9M3 Yzxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310840; x=1724915640; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d/m1IXPfk7xvDU+Qn41r4NiqBFnWc/gEWMU/opSilGI=; b=ZkoRwfcFEbvIk0Xr/z2PodL2xRwiT2iDxoHgzsZvnBCSDAUdb5phI1B0Wz+N3R0rkk 0S78B//GhR2LHu4P/R7AQ8BvXnk5GqdzyMEMZlDpwOqzt7UwbhZFSft045iLFL4YR8QN 93CB9ldKcgbnn6/WItt09anHoXnKTAXkb2t/AtLB1faJiJ04ANvwjWFwqJq8ghnPcdlo r9Ftkyr1/3ow310bWl2jh+p6q7NRXkNDqalOaPC9zd1IlVNfNPHqT6qoKeZZmJadGB9G XxTSLKsp/VV51IyQfxA4K63OTs/T1pOeKTtO3lTwiukEh7FtspTjjlfWeBlzayWYWxxm ER6w== X-Forwarded-Encrypted: i=1; AJvYcCXtii4v1rICrah3TUHMqP8dkW/rOrRfTSfJoCZ9Ejl+BS2Sz5Klu0HV7Yno/SZSJT6i3C5ycPeaIzUyjUQ=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yx2BtDS5IngKgwuNmnQUm4lOhkW94hhLiTjVonil866hN8SkgdY ptfAn+O1K7Dc5PM10nEXz9rNpwfU5sLD26jqmIwSlX5LKUEmB1XPf5cLKMoAyTM= X-Google-Smtp-Source: AGHT+IEhsa9RMy8FC6eeC7Dc/qzW9nWoiqPGY6DtNfgP/dDTOo2T0oKgHsMPC17UGp2n78kSJG5uNg== X-Received: by 2002:a05:6a20:6a2a:b0:1c4:9c08:5df3 with SMTP id adf61e73a8af0-1cae51cf01cmr3205085637.7.1724310840022; Thu, 22 Aug 2024 00:14:00 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.13.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:13:59 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 02/14] arm: adjust_pte() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:17 +0800 Message-Id: <7915acf5887e7bf0c5cc71ff30ad2fe8447d005d.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In do_adjust_pte(), we may modify the pte entry. At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the PTL held. The corresponding pmd entry may have been modified concurrently. Therefore, in order to ensure the stability if pmd entry, use pte_offset_map_rw_nolock() to replace pte_offset_map_nolock(), and do pmd_same() check after holding the PTL. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/arm/mm/fault-armv.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 831793cd6ff94..de6c7d8a2ddfc 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -94,6 +94,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, pud_t *pud; pmd_t *pmd; pte_t *pte; + pmd_t pmdval; int ret; pgd = pgd_offset(vma->vm_mm, address); @@ -112,16 +113,22 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, if (pmd_none_or_clear_bad(pmd)) return 0; +again: /* * This is called while another page table is mapped, so we * must use the nested version. This also means we need to * open-code the spin-locking. */ - pte = pte_offset_map_nolock(vma->vm_mm, pmd, address, &ptl); + pte = pte_offset_map_rw_nolock(vma->vm_mm, pmd, address, &pmdval, &ptl); if (!pte) return 0; do_pte_lock(ptl); + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + do_pte_unlock(ptl); + pte_unmap(pte); + goto again; + } ret = do_adjust_pte(vma, address, pfn, pte); From patchwork Thu Aug 22 07:13:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1975287 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=TWZPtvue; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-308-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WqDwb0Tx4z1ydn for ; Thu, 22 Aug 2024 17:14:39 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WqDw071kcz2yZ3; Thu, 22 Aug 2024 17:14:08 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::22c" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=TWZPtvue; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::22c; helo=mail-oi1-x22c.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-oi1-x22c.google.com (mail-oi1-x22c.google.com [IPv6:2607:f8b0:4864:20::22c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WqDw051jPz2yZ5 for ; Thu, 22 Aug 2024 17:14:08 +1000 (AEST) Received: by mail-oi1-x22c.google.com with SMTP id 5614622812f47-3db14339fb0so294269b6e.2 for ; Thu, 22 Aug 2024 00:14:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310846; x=1724915646; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O1mpKMHwwZ1CFkcRRT4IxjhKZ5nh5A3TknIoqewquLI=; b=TWZPtvueDriJIMI6gtUVEu7qiK1ZbgeQjRaNdAxxh4I4++Dl23p10U4iHNrvnJsARx ejfBcA7cdqPBMQx/Hdxtg3yZiiv1Nc8js3X1bVEjyGklwzJsHRWXajf0PGra+OQIH+eI kxSgBXdP9SeK0c34pRi6D7gPg8NA33rMno+sb8//FU+2LChvYuSSUAnWUzgVAp8ebAEv FCutsfAm1ETJ6/K3VvK4XFr6wayrtQTZIzYQsfAIbZvkk6AVHqoN/ioKkFb0KDwRN7EY eq6yFfOadPO9yuFbvPvFqHwrAnugjp6ctk7GVGXZGJentXFp1wlaoWpKaegloZMvmmKu 3pkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310846; x=1724915646; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O1mpKMHwwZ1CFkcRRT4IxjhKZ5nh5A3TknIoqewquLI=; b=g9ZZWGxtvfyf2FQYZw+b3tBLYyVx5gwOMIlduC+wWSNWlQ0MfNRoPIo7ZCvpOtsM1z euVXtx88GRUiDIZsqcCIlj59Ja1rJpyvyJjiltz3f6g3FTrjHpHaLxZxdfesbnAcq8PP iB20Fil2WTyySQcDE0WoLeytvPS0CDwG3+DakC3enaopQA9AH+iECI5R9AVnJEEbaiER X2iYC0AIS040jmPR+eF/KiP210+6sVytsh1ZhXlc0xpoZq/d8Std7YZMyC2CR0d9ItqW Jtafeh8F9LsItg0/y7iFsk0LQ5+/a/SB+dJUSSDvoHXNG0eTEJISDLspCjw+WXpc6Fu8 WVtw== X-Forwarded-Encrypted: i=1; AJvYcCVF28MoDqmE9sAcJLaWpxe9IybqjKGRWvmCTSACBeKNSimLAA5VHGQ6Qbsnr7UidvtL7MdTC70L1j8pqW4=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxA6tzgdMc67PNeROpRw5R8bVjh0W0/SG2skaz+YJxkzXbE0ocz HMMBD050tU9QdGWSU726zlpgDKDv6dc0n1dA1zuQGvEHDPGvVnkrKP1ruyjH4g8= X-Google-Smtp-Source: AGHT+IFDX2PqWoIoI6wCA8pFxmw/BmBf/PDUvtd9hijDBvTgLhn0KCKCIBnod1yVuIa/+/M4c6qbig== X-Received: by 2002:a05:6808:1a08:b0:3da:a16e:1764 with SMTP id 5614622812f47-3de194e0e97mr6243309b6e.4.1724310845744; Thu, 22 Aug 2024 00:14:05 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:05 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 03/14] powerpc: assert_pte_locked() use pte_offset_map_ro_nolock() Date: Thu, 22 Aug 2024 15:13:18 +0800 Message-Id: <89ae94f722996b4ee7ef20d44cd7f5e2f0495c47.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In assert_pte_locked(), we just get the ptl and assert if it was already held, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/powerpc/mm/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 7316396e452d8..61df5aed79894 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -398,7 +398,7 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr) */ if (pmd_none(*pmd)) return; - pte = pte_offset_map_nolock(mm, pmd, addr, &ptl); + pte = pte_offset_map_ro_nolock(mm, pmd, addr, &ptl); BUG_ON(!pte); assert_spin_locked(ptl); pte_unmap(pte); From patchwork Thu Aug 22 07:13:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1975289 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Jh46hr6X; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-310-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WqDwt17wMz1ydn for ; Thu, 22 Aug 2024 17:14:54 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WqDw7004pz2yK9; Thu, 22 Aug 2024 17:14:14 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::431" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Jh46hr6X; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::431; helo=mail-pf1-x431.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WqDw64mJqz2ygy for ; Thu, 22 Aug 2024 17:14:14 +1000 (AEST) Received: by mail-pf1-x431.google.com with SMTP id d2e1a72fcca58-7142a93ea9cso394260b3a.3 for ; Thu, 22 Aug 2024 00:14:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310851; x=1724915651; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jnbwyZ8nm7OA7xAj7wrWrabgK1t5zgYyH3DelQTt/t4=; b=Jh46hr6Xw1i3jrNTYvvpAYRvQTeq4fDqTOX6C8hjg9bWXrwSEDmEXEPN+Oz4W/iWJv JpC7AQGB4rlLQxKO1HfnvJ9QtYFPH4yvYez1q5scNK0dMlwnvARovb+fpGuh8ErXYz2W 3gOGBF2exieTsSVkgH3bRlljsnBTeNW2wxd7dF8GvVMtU7JzqCm9HdEYq7tD5kzftKBF jxUCAyeiF/qKuHePIxKO3Wg03wCOmvMSj6YoJBl3XK7uaW1G9GUpaY/We5quXAPJrems g+g9APXnQxtGD3ebcZIKve4HM9E6A9MGzjd4psWSMRKDsyDJ3OtuMnnuCs6UOJROmWB1 l3Zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310851; x=1724915651; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jnbwyZ8nm7OA7xAj7wrWrabgK1t5zgYyH3DelQTt/t4=; b=mpANmvgDVOs6WaaOGshTjxJPEupbqcjMGN441F7vHHSuPXa8OF7lPkZL6CNITjzInW DpWnPTVwN5HZdMyHX3dIwBVhLr1ZEw5Zr7ARKHUkBOu0ZqSZM8s3jka9wDqoYtFrRtYD Pl5yiaU2QC1m/Xe4PDVbe1aSxbKVYTjmy2dK+xWKhTizyIFPh4ksKLYnXeYX3FAmSv67 3dCmiFI+DU4PleuSLJf7OIGnIdP3WXnnTMCJZchgU5XReDvwI02z7tpUegqQGZEL+h1l YKU7iUSBRt+ndZOdgsE/bbHqE1HHHZpQMOaCKhHr3nYO+1U/c11fnNHB2n8vL5gzjGkj HveA== X-Forwarded-Encrypted: i=1; AJvYcCWRFI2t0syca0ATkJqqMVlSfmQ9IgIThtv/AtCIL2iDBYfwWn8tBCxCh7Caiz+qzcwIKYX00waj2YmerxM=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yxw4w9pd60GBBAckGmmLpw+uHH/br+mLDAWMW1dKm98yr1VrLe4 H5fq7u4bdJooCpvX/IDWVm3gHhTm7uy6QXRm2qFmP4rsNKQ3mgsVPhKuxRthqSY= X-Google-Smtp-Source: AGHT+IGXmNRuv+mQEijx6lEwkmYjHk6XirTogWRG+N5Gk1h3xzTz9eMzqUkZv9fzF0cNQyeHFRUhTA== X-Received: by 2002:a05:6a00:c93:b0:70d:3777:da8b with SMTP id d2e1a72fcca58-714235a5404mr6267050b3a.25.1724310851578; Thu, 22 Aug 2024 00:14:11 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:10 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 04/14] mm: filemap: filemap_fault_recheck_pte_none() use pte_offset_map_ro_nolock() Date: Thu, 22 Aug 2024 15:13:19 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In filemap_fault_recheck_pte_none(), we just do pte_none() check, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 0f13126b43b08..c98da9af6b9bd 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3234,8 +3234,8 @@ static vm_fault_t filemap_fault_recheck_pte_none(struct vm_fault *vmf) if (!(vmf->flags & FAULT_FLAG_ORIG_PTE_VALID)) return 0; - ptep = pte_offset_map_nolock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + ptep = pte_offset_map_ro_nolock(vma->vm_mm, vmf->pmd, vmf->address, + &vmf->ptl); if (unlikely(!ptep)) return VM_FAULT_NOPAGE; From patchwork Thu Aug 22 07:13:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1975290 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hkhjhs1w; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-311-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WqDx13L77z1ydn for ; Thu, 22 Aug 2024 17:15:01 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WqDwC2LKrz2yPj; Thu, 22 Aug 2024 17:14:19 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::22d" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hkhjhs1w; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::22d; helo=mail-oi1-x22d.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-oi1-x22d.google.com (mail-oi1-x22d.google.com [IPv6:2607:f8b0:4864:20::22d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WqDwB6l4bz2yLT for ; Thu, 22 Aug 2024 17:14:18 +1000 (AEST) Received: by mail-oi1-x22d.google.com with SMTP id 5614622812f47-3db1d0fca58so279838b6e.3 for ; Thu, 22 Aug 2024 00:14:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310857; x=1724915657; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=toohyZrQNwk2kNUrhQFlaPWHYnggLi3YB3M65Yq8QzA=; b=hkhjhs1wVytFCBaYErD756Jzgt+B6ppJIz5j15FWmjx8WPntZDghQ0Szbe6pUC+GUy Og0V+ww+fGjGzDgnmtdhZWieE4I7YJT7jKjCrwteRuWCWPQOO3XLK+ABx9B6S6YTAaPT DT1qp18hGKnOuj8EY6C4TQ0JkAab8jpsYGDgRAAZx5LxI3noRB4V4cO29frUO3r0TwD9 t+EHRbYG/WpkNAEQbIF/Rv1Rv9F0uKASrh7t2jThtuA08zT6hRkPB9jGfP+AsbSPW9UQ wuu1ODG8Yb8FJtJxEBgmoNR1yE5aAMeAjBGY1Qq9xwordWS2k7o0e/nISosp+RV1BHvA lmCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310857; x=1724915657; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=toohyZrQNwk2kNUrhQFlaPWHYnggLi3YB3M65Yq8QzA=; b=tIY1VtN+24tkSlJNz7L064CUrZkjUomFVe+FiCSkqFHAVXDC36pzqiN9OJLz0QtpZY k+cXdlP1zBbE4MjoveTn/RKY3CIy88X/XhmjRDbVZk2KY+NmL5eApP94lm2v4dpfmerg cvAkN3bANQJgyCetLCz9uDPhLodH2gjBJ+O5snZt6UFxIFfmtnxE20/pTg4VzxfxMybx p7kGD7+JYctnr5FyXv7yjRYaOMVj7/vnwADfScTAojRz5gOcOL2N0nVoxD8oS5Ep8Fot PgWnfWX8jZbtIUXE7sijKyKcZpvIBKohfTF8ia20sgMVS4lhB5JXOtrM3kdgwb/O8vFV U0BA== X-Forwarded-Encrypted: i=1; AJvYcCVRv/QwyMNyAydXdoWN1QrPTgPXtnIfZ/Tr8WtyKo8vwfbMiHq3TRxnO1CSEAMZSnwe5B5J/NB5tLyH3G8=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yx8HF0GjQfduEyG9odszlml/dy7mqQzgYZzyfX0pR/vl2B8Fr6C YuLXzd8aAK+uTVLWuOxM3XVv/zsKypByUU0wOLQ7JObFAnp1h1KGaW+MVnoAX/E= X-Google-Smtp-Source: AGHT+IGgjighdWtbrKc2Se12Iutexgq8AwJLXy8Rdygo/b6qt6ZS1aEL6K5S14xFU0XlAJpbImifsA== X-Received: by 2002:a05:6808:179b:b0:3d9:38e2:5392 with SMTP id 5614622812f47-3de195c380emr6158328b6e.36.1724310857179; Thu, 22 Aug 2024 00:14:17 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:16 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 05/14] mm: khugepaged: __collapse_huge_page_swapin() use pte_offset_map_ro_nolock() Date: Thu, 22 Aug 2024 15:13:20 +0800 Message-Id: <2b7dceaa12da273c6decf92e3bf2ebc4f4e85ec5.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In __collapse_huge_page_swapin(), we just use the ptl for pte_same() check in do_swap_page(). In other places, we directly use pte_offset_map_lock(), so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/khugepaged.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 4a83c40d90538..53bfa7f4b7f82 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1011,7 +1011,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, }; if (!pte++) { - pte = pte_offset_map_nolock(mm, pmd, address, &ptl); + /* + * Here the ptl is only used to check pte_same() in + * do_swap_page(), so readonly version is enough. + */ + pte = pte_offset_map_ro_nolock(mm, pmd, address, &ptl); if (!pte) { mmap_read_unlock(mm); result = SCAN_PMD_NULL; From patchwork Thu Aug 22 07:13:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1975292 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OciNZDeO; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-313-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WqDxK3Z3cz1ybW for ; Thu, 22 Aug 2024 17:15:17 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WqDwL55ctz2yjV; Thu, 22 Aug 2024 17:14:26 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::433" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OciNZDeO; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::433; helo=mail-pf1-x433.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WqDwL1XCYz2yLT for ; Thu, 22 Aug 2024 17:14:26 +1000 (AEST) Received: by mail-pf1-x433.google.com with SMTP id d2e1a72fcca58-71430e7eaf8so391984b3a.1 for ; Thu, 22 Aug 2024 00:14:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310863; x=1724915663; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YTJMrM+GLhYLeZwUuiK6dImcBAbTGjqYt196yQPATUA=; b=OciNZDeORUbI63yazH0oDkuhSMpppsE20FtTPXLv4w/liCgRyP1LfAGtiZRfJM77rA S4pZ6n0WBycwWw1EJKiPfgK4DTxbwNk7EEZ2mYgqMqFVV0+JBW//52Q2K/MKhYcJNjyw rVeStUvE6knAeP6ZVTwDknL8iIa/YTpbb0KZ/GTM00QYf9bPxZJ6fWSSlDofU2iogPOr Be1gWKWnjEJvysQL1qgD2L3PDbl/kpdLR27BDRrX0Ldm3JG5SWHTMATSTx6Bl/dENZXb PZIGgZW+8URMzNEtlUxOWxVryrSlXJUK05nB6h2PonwW5ke0EMXuU7R1VLKWHch0jmq+ JclQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310863; x=1724915663; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YTJMrM+GLhYLeZwUuiK6dImcBAbTGjqYt196yQPATUA=; b=ppRfz6rMVwdgeqOgo0yOP+2Xi2tU2rcxZNMvI4ZHhB1H00SvXvQBOKOWO3956YuFkh n4M2Kw6sAP7wdPFDHXbfWRW2hEBqmsPkzOFspAMmGisnMGbhHBR4QfLvDoipRQc8gS1p oKhgLhaatpMX82NPiGxn+Bl73NqKcBEMno8r/+q457ypmgrDdf8phcIZUrjRckz+XThN eJMLM6ISLlTz48NB1QmesDwqj/HTl5+9wmcW6A8z5lf/weXq97e4YE6HkDyQblZIZD9q ujshILdaYVZASGSEsKVcsfCLwDdl7uU2CYc1zVobQ0pM5hy7PS0XUtUJnmm7a37qVEzg PoPw== X-Forwarded-Encrypted: i=1; AJvYcCVBdIPR0nJcQ2Xj8A0SUOZ4ZUZ0JW4VtUn4psjOTieUGZCwxtZNgRiYwmbmctG9xi4h7wRT+5XE83tWNWQ=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwVVuBXfiNYwc0OpGsWARiTlw1XoQC2vk789qLyI1JYmuklvzrL EgLiFVahAUAjuWUVK4XYWHfbdSHtysGLdJI0DdkKcTwqIxmcV+pYQqG7RY79zkM= X-Google-Smtp-Source: AGHT+IEJaUpX6CQIEutPHebqIOZXfyMaKxdLW1UElMR3gb10gY3PgzFG7UEaK+UwdtskMN8KUpfIGw== X-Received: by 2002:a05:6a00:cd4:b0:714:1bce:913a with SMTP id d2e1a72fcca58-7142356e960mr5852557b3a.21.1724310862916; Thu, 22 Aug 2024 00:14:22 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:22 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 06/14] mm: handle_pte_fault() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:21 +0800 Message-Id: <5acabedfae7ded01b075960b4a91f2e15b4d76b5.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In handle_pte_fault(), we may modify the vmf->pte after acquiring the vmf->ptl, so convert it to using pte_offset_map_rw_nolock(). But since we will do the pte_same() check, so there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/memory.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 93c0c25433d02..7b6071a0e21e2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5499,14 +5499,22 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) vmf->pte = NULL; vmf->flags &= ~FAULT_FLAG_ORIG_PTE_VALID; } else { + pmd_t dummy_pmdval; + /* * A regular pmd is established and it can't morph into a huge * pmd by anon khugepaged, since that takes mmap_lock in write * mode; but shmem or file collapse to THP could still morph * it into a huge pmd: just retry later if so. + * + * Use the maywrite version to indicate that vmf->pte will be + * modified, but since we will use pte_same() to detect the + * change of the pte entry, there is no need to get pmdval, so + * just pass a dummy variable to it. */ - vmf->pte = pte_offset_map_nolock(vmf->vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + vmf->pte = pte_offset_map_rw_nolock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &dummy_pmdval, + &vmf->ptl); if (unlikely(!vmf->pte)) return 0; vmf->orig_pte = ptep_get_lockless(vmf->pte); From patchwork Thu Aug 22 07:13:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1975294 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=EXAv4B3b; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-315-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WqDxb5SNDz1ybW for ; Thu, 22 Aug 2024 17:15:31 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WqDwQ5bq0z2yLT; Thu, 22 Aug 2024 17:14:30 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::433" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=EXAv4B3b; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::433; helo=mail-pf1-x433.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WqDwQ2SmQz2yks for ; Thu, 22 Aug 2024 17:14:30 +1000 (AEST) Received: by mail-pf1-x433.google.com with SMTP id d2e1a72fcca58-70cec4aa1e4so342424b3a.1 for ; Thu, 22 Aug 2024 00:14:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310868; x=1724915668; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jy8AvBGPXewIN0f06tSdN0J5TZbHLAwXxEIbEfEZm70=; b=EXAv4B3b0nds3d7opW+skIbhZBQUFH7EsjqvDcmwPO+5CD+sjq/B0LhJNpeNFmomtt WVfGdXjuhwx/blq7cVTMbMvpNp3w14v/MW2NVxfnOpjkBKKqJzRNi4FwGijnNe4B6RWJ Lxq1lWa7qisLRgM4imtmfakaKE5voERau/IOplfCOfO8xGtu03b7I4oQ2XNXTAdNPiSM r/VyIBzIxLVcHbgTXo+vcY7k7fCm7RGCSIV3KZoTQLF7HC14f8d0c26DObB/8ioN51XB S4k3j78WQmxgXr2r72SCm4ZjRIhUlm81MAxN/l3AC8kr/slIShdipRfTkxUolXhbOYKR yfbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310868; x=1724915668; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jy8AvBGPXewIN0f06tSdN0J5TZbHLAwXxEIbEfEZm70=; b=KeTqnPrCRI6BESeqN/r+KMJ5k6eG3m3D/XvTYLtSLhCWrtbhDXkV1jdEztLJkZukWG uOGv5fC7w27Nl8b25Cmhj3oA3b8TAajWcTb71Nkgf0AW+/Hq0TNzEQ/rox4KQ5HiYbDd O1yHxBku7bgzQrZTGmBVMGSdq58tu9yfRf3seMUwgMZP/aqKlUsGNiuhjdx56ZUTX7ik c+pU9VEVONWvCjxN8QG/aKosUOg+gGPHxFX7uDaaJqDhR/zxIOlMV7YuW5zLS5tADIHy ycJXDvJ18vlo4ClOFhPAhqYgCNAQFzcvFURJeHKMJgE+IymgxiTLxARrvu/SFv7u2qe3 yZlQ== X-Forwarded-Encrypted: i=1; AJvYcCWyXuS2LReSoNret/N9GipJYpML477nHbX0Oa52311nM+/n6zx6FgIfqTcBKuNErY6VuPZoDrSHkUV9/t8=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyPkbrt+nxqyB3QY/8K1rNsPUVZQnxDfU2YfzSefaidgxwKDcXD N/7hciqzjRgkSs2+I9JaA9CzBqSn+MBwUB+pKEGcjsH/lftSlDXbtAkofOMfomA= X-Google-Smtp-Source: AGHT+IEV/3obR+4TCbDGR6rVI1h9FT7v1/u/+fxVcK1o1ztnph09CHD4OJqmjxMXnPNwi8/vaSiIPQ== X-Received: by 2002:a05:6a21:6b0a:b0:1c0:e728:a99e with SMTP id adf61e73a8af0-1cada078d94mr5247385637.26.1724310868515; Thu, 22 Aug 2024 00:14:28 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:28 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 07/14] mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:22 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In collapse_pte_mapped_thp(), we may modify the pte and pmd entry after acquring the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the PTL held. So we should get pgt_pmd and do pmd_same() check after the ptl held. For the case where the ptl is released first and then the pml is acquired, the PTE page may have been freed, so we must do pmd_same() check before reacquiring the ptl. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 53bfa7f4b7f82..15d3f7f3c65f2 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1604,7 +1604,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml = pmd_lock(mm, pmd); - start_pte = pte_offset_map_nolock(mm, pmd, haddr, &ptl); + start_pte = pte_offset_map_rw_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1612,6 +1612,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, else if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i = 0, addr = haddr, pte = start_pte; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) { @@ -1657,6 +1660,16 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 4: remove empty page table */ if (!pml) { pml = pmd_lock(mm, pmd); + /* + * We called pte_unmap() and release the ptl before acquiring + * the pml, which means we left the RCU critical section, so the + * PTE page may have been freed, so we must do pmd_same() check + * before reacquiring the ptl. + */ + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + spin_unlock(pml); + goto pmd_change; + } if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); } @@ -1688,6 +1701,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, pte_unmap_unlock(start_pte, ptl); if (pml && pml != ptl) spin_unlock(pml); +pmd_change: if (notified) mmu_notifier_invalidate_range_end(&range); drop_folio: From patchwork Thu Aug 22 07:13:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1975297 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OT4WiPvT; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-317-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WqDxv1d3Qz1ybW for ; Thu, 22 Aug 2024 17:15:47 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WqDwX2JRrz2ymr; Thu, 22 Aug 2024 17:14:36 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::42d" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OT4WiPvT; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::42d; helo=mail-pf1-x42d.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WqDwW5QB5z2yks for ; Thu, 22 Aug 2024 17:14:35 +1000 (AEST) Received: by mail-pf1-x42d.google.com with SMTP id d2e1a72fcca58-71439adca73so165294b3a.0 for ; Thu, 22 Aug 2024 00:14:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310874; x=1724915674; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lG9D96+xW+VlDYZHTIMi8bcfezh9NEOiBopQjA6w29o=; b=OT4WiPvT8MpC6CrUrCKiEselPk4ucfCvPcv2rsC17Y7Wno0DaYOpkWa4vNrk1U6EG7 kDVbGBXIHPigzdt1iU7jKStumbKfLmzTRPvfB/Q/3FzVK5FKkWPKMLjfsDX04dpupoUm 61MJlGkNVm2mSC5P04PjP9lK+PIdMV15RqDInC9qK69pBIeDiLH5a+63swdiuTfxamNv rg1MuuROR7ODsXPSeXXPLRGT7SFZzfDp9G+LAAL4ookHvVIkbXEXem0EKpbL8Hvs4KhL gpn7mWCTQLVupmQm3F52miIwW2MQ4egYn9ODZjkRbk6pxli3eHHbXg8YhSC5VESbSwFs nc+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310874; x=1724915674; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lG9D96+xW+VlDYZHTIMi8bcfezh9NEOiBopQjA6w29o=; b=NqhG+lhRHqSIbb1gKNzEqVO6XJR1LJNhFQuhsvQk/zN2P0CtVnjKsnte4dIWG6v+/i vV6w9ATJAvSjVWk5Lnio4UisKH+VfWxFT7eduxfYZRlC97cg+tT80fwMKOjJkdWB0DzA zufmpgs22NXh76ELnRYr7Y4IOQe6HIP+O3D0uRi9MaF81j35gMmiUczBZ+GUc20Xefut t/4FtCbkL4kpVWv1u8nONgBxwPLYewBaNmgunZgLF0EEydKXD/csKCNBZ2lXOmXp00jS yJHlOsvEr7l1sRrYHi8fwzwsFNZezuy8aOCrDm904MRAYhc7tTsIM+MLtXDCjUuSQRy7 ZB0Q== X-Forwarded-Encrypted: i=1; AJvYcCUf3462cEqK1F/SCotGZBwDsumSV800cNzZd/Xi7Ca9Oop/IWhyOd7C3aL+dVw/TgdSZWKntJlRr+5YbZo=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxQXq6RnKXjfl/rjjszNPxpQnlMpCP9iSV022tjGfhKBmLmMTBr pKerAJAdbkD11YCBZ0NQ1wbS4ZZahDiuLAUctF9i2L2n5C/8fikMPou/dzOoF7g= X-Google-Smtp-Source: AGHT+IE1Azw7tBspdYFUO9+EydXCiNKc3KqrNzafyoF+69WNl4yC4JRDYA6cJOAAfX+V+yHVonAuzQ== X-Received: by 2002:a05:6a20:9d91:b0:1c4:f209:f1ea with SMTP id adf61e73a8af0-1cad80f657emr5853305637.31.1724310873985; Thu, 22 Aug 2024 00:14:33 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:33 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 08/14] mm: copy_pte_range() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:23 +0800 Message-Id: <71100c3867c4cf6f5f429ce9f2db8432066d0e99.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In copy_pte_range(), we may modify the src_pte entry after holding the src_ptl, so convert it to using pte_offset_map_rw_nolock(). But since we already hold the write lock of mmap_lock, there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/memory.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 7b6071a0e21e2..30d98025b2a40 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1083,6 +1083,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, struct mm_struct *src_mm = src_vma->vm_mm; pte_t *orig_src_pte, *orig_dst_pte; pte_t *src_pte, *dst_pte; + pmd_t dummy_pmdval; pte_t ptent; spinlock_t *src_ptl, *dst_ptl; int progress, max_nr, ret = 0; @@ -1108,7 +1109,15 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, ret = -ENOMEM; goto out; } - src_pte = pte_offset_map_nolock(src_mm, src_pmd, addr, &src_ptl); + + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we already hold the write lock of mmap_lock, there is no + * need to get pmdval to do pmd_same() check, just pass a dummy variable + * to it. + */ + src_pte = pte_offset_map_rw_nolock(src_mm, src_pmd, addr, &dummy_pmdval, + &src_ptl); if (!src_pte) { pte_unmap_unlock(dst_pte, dst_ptl); /* ret == 0 */ From patchwork Thu Aug 22 07:13:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1975300 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=cs8MKNLk; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-320-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WqDyK3BQkz1ybW for ; Thu, 22 Aug 2024 17:16:09 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WqDwk4yXzz2xYl; Thu, 22 Aug 2024 17:14:46 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2001:4860:4864:20::36" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=cs8MKNLk; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2001:4860:4864:20::36; helo=mail-oa1-x36.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-oa1-x36.google.com (mail-oa1-x36.google.com [IPv6:2001:4860:4864:20::36]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WqDwj6pvtz2yp9 for ; Thu, 22 Aug 2024 17:14:45 +1000 (AEST) Received: by mail-oa1-x36.google.com with SMTP id 586e51a60fabf-2702ed1054fso295141fac.3 for ; Thu, 22 Aug 2024 00:14:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310879; x=1724915679; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kCwbwV/YmKb9rcRCZPyi2vEyGZwxl6QKnoBllHH347k=; b=cs8MKNLkGr7bmSr1XA06ECR8qdSnOuZnFbLP8OPOLQL3mY6rZ+ZWJaCIjINMuFTFFD K+Dc9m6ZP3k+99y2q4HOVnotrZXkBRY+Hj8MZg27SFKZjuDQ8SU1ym7ZLApfMEC8HZKY 6tIRnct2xCOpRh8cuXv5XdzWydPHhIVEuC6c6GSiA9JaWT0M1UZBE2IYhWI+UA/qg6su RnXmD8Jza31vBpQZ/4ebNll+uW5BPUdyVhHxzC9Z4QKsfKLIvRYQeF8VI/Zq2FKF8ae6 hj9oo6L6M568uyziduUmUg0RCrSg09IUAr/Chj3nznoPYypzRcEe30yeR4wDb32LsagJ 0cqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310879; x=1724915679; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kCwbwV/YmKb9rcRCZPyi2vEyGZwxl6QKnoBllHH347k=; b=kkNs1JqQH6WwRLwk2fIrITsc/keJ/zHSaY27ZwJ8PhonUM197nEWQWcd7r5Q2zNT+6 4sP1VquS2C9A74N8H6KQZf1ze+Vj/bU0h1lLGux87r7DlborFNhZ9sBHQcxL3+GIjuwO zk3+ARk4GQ/KwQHGTS7dV9A47untcNPK0DBJPd/8eu4HZMWfaMG1ZU1Oqh8hTrAwtb/6 tVvB+3gzqWSikGbjwRAPymJIeOucZdstN9bUVBhvGZzSNqc6l5jJhDUiV8LalOcrIo2j QiugJTGF2iIdGvLbwWWabVkghPWsGWR616QC6Qv0cLHq04dtnUQ7T8+3RJ/PhEZFDc6M /m6w== X-Forwarded-Encrypted: i=1; AJvYcCWDW4kJac0/cdJoYRfe2OVNf3VwTKld55HdrV3Ald/6M1Jx6oG1272W52otJKCYCTgLIBemrISYCpUbCzg=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxPe0PX5p5DimddKcu9HhVhFoId7vK1zXuYK8aHxoZHO4FaNbCN 8RqjyCDIgoLEBC0ycqVmJd52l5I6YCh62UbO/nUfFZZegIWVB0R3FeSKqEUvfhM= X-Google-Smtp-Source: AGHT+IFsTn+Q5A1WOhbyPTAjHclYiOvt3nhLZV/Hi5KOu9NOtbxQa/3JykhMvHE3PrFxCVBhO2k7lw== X-Received: by 2002:a05:6871:149:b0:268:afc3:648e with SMTP id 586e51a60fabf-273cffcb557mr1073020fac.42.1724310879532; Thu, 22 Aug 2024 00:14:39 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:39 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 09/14] mm: mremap: move_ptes() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:24 +0800 Message-Id: <12097a187fee5f21860d4f87e7aa103f822e298e.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In move_ptes(), we may modify the new_pte after acquiring the new_ptl, so convert it to using pte_offset_map_rw_nolock(). But since we already hold the exclusive mmap_lock, there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng --- mm/mremap.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/mremap.c b/mm/mremap.c index 24712f8dbb6b5..f96b025c09079 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -143,6 +143,7 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, spinlock_t *old_ptl, *new_ptl; bool force_flush = false; unsigned long len = old_end - old_addr; + pmd_t dummy_pmdval; int err = 0; /* @@ -175,7 +176,13 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, err = -EAGAIN; goto out; } - new_pte = pte_offset_map_nolock(mm, new_pmd, new_addr, &new_ptl); + /* + * Use the maywrite version to indicate that new_pte will be modified, + * but since we hold the exclusive mmap_lock, there is no need to + * recheck pmd_same() after acquiring the new_ptl. + */ + new_pte = pte_offset_map_rw_nolock(mm, new_pmd, new_addr, &dummy_pmdval, + &new_ptl); if (!new_pte) { pte_unmap_unlock(old_pte, old_ptl); err = -EAGAIN; From patchwork Thu Aug 22 07:13:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1975301 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=gyhe6DEZ; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-321-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WqDyT1p4Sz1ybW for ; Thu, 22 Aug 2024 17:16:17 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WqDwl2q68z2yp9; Thu, 22 Aug 2024 17:14:47 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::22c" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=gyhe6DEZ; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::22c; helo=mail-oi1-x22c.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-oi1-x22c.google.com (mail-oi1-x22c.google.com [IPv6:2607:f8b0:4864:20::22c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WqDwl04g4z2yps for ; Thu, 22 Aug 2024 17:14:47 +1000 (AEST) Received: by mail-oi1-x22c.google.com with SMTP id 5614622812f47-3db145c8010so274223b6e.3 for ; Thu, 22 Aug 2024 00:14:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310885; x=1724915685; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Crbz/R3rJcbpbcovfp9fh4ozsAEMZSaAxrx7zvZfRRw=; b=gyhe6DEZSH95h0MAnuhTnzacosoLdJ2D9yDsHcbuX6lCEFDxXAHEaZPKTOUUQ71+Sk Xrx7AtHiiBIQa/iFFzIQYPanq1USiA8wagx3wkXiblVvGrNiwIaVvnJr/5SdznAMdAsi pAjBhwQJ46oIL/6hse7tJaNBwLrBj3XUA3p0pWjyHGGzD4Wkon6QlA0NlSst8mb+n93n 3jSm6HiYkGo1QVY4u1sGFJROez+DXVjwOUxZPLgiDc4sXDcoqN5IM03re5p8kwMcXmwm 3qdmJtmaLX76zLyd+0i5/1E1OguN9mUGjOB5fH+MNq84hS4tZ4TkUoEGgzXGqzEtAQGn JLKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310885; x=1724915685; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Crbz/R3rJcbpbcovfp9fh4ozsAEMZSaAxrx7zvZfRRw=; b=pryqqKOQ1zdjsS0zxscbChpeU0q5TpjZkXcmEEpD9CwAR9buI1jUEwr8vxYdzEf3Xn LFPugTP/BpIBhEoF4Q+ckKDqNvgRUru5a59/pxgS3EmeGIsGUSA3Ghq9YHH6OjT2atdT IOLHaZIuh+/UTX4f/Jr3NJf2NU/rn4tOwts3+hPfsAPogbzekW+P9hz4XleDTUggkkMm JmHKjb+poq9RZhxvSDIHq9aK9l9aDTgBjSMclEvxck8pL5TV5JX9/raicikpmMmtvUQ5 vl8EtIWFkhcgbcPNGypBHXm7pfi1qt3BNfu8ahYeo6Gpu1kAva5HakLIXeeZdxXbh+tB dcIw== X-Forwarded-Encrypted: i=1; AJvYcCXOTsyvFtRKiaDBUSsLZLKGKWekq1195s7kZfvPS/5zDAUaKMNqEdG4nPKIgF+n2kgP7/eUhdJa3nGjTYM=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyETsjQuhmLfGRNdiFukySCwJcxOjlhTCE1tBHYiDg58GT8IwGz SVmUtydNZKuWgKrPvhYMAqESjLYbHZZ9Lswb3pDt4rQmZcUdGqfzGo0I50m2M3w= X-Google-Smtp-Source: AGHT+IF73IAzwqsx0nomJCVJryAV7cndKc0EgZ0+2TYoDy25bfQlq+9JSwXFUBQmZZcxXSkt111lxw== X-Received: by 2002:a05:6808:d46:b0:3d9:29c1:be5c with SMTP id 5614622812f47-3de194fbf2bmr5329807b6e.12.1724310885053; Thu, 22 Aug 2024 00:14:45 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:44 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 10/14] mm: page_vma_mapped_walk: map_pte() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:25 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In the caller of map_pte(), we may modify the pvmw->pte after acquiring the pvmw->ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the pvmw->ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pvmw->pmd. Signed-off-by: Qi Zheng --- mm/page_vma_mapped.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index ae5cc42aa2087..f1d73fd448708 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,9 +13,11 @@ static inline bool not_found(struct page_vma_mapped_walk *pvmw) return false; } -static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) +static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp, + spinlock_t **ptlp) { pte_t ptent; + pmd_t pmdval; if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ @@ -25,6 +27,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) return !!pvmw->pte; } +again: /* * It is important to return the ptl corresponding to pte, * in case *pvmw->pmd changes underneath us; so we need to @@ -32,10 +35,11 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) * proceeds to loop over next ptes, and finds a match later. * Though, in most cases, page lock already protects this. */ - pvmw->pte = pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, - pvmw->address, ptlp); + pvmw->pte = pte_offset_map_rw_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, &pmdval, ptlp); if (!pvmw->pte) return false; + *pmdvalp = pmdval; ptent = ptep_get(pvmw->pte); @@ -69,6 +73,12 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) } pvmw->ptl = *ptlp; spin_lock(pvmw->ptl); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pvmw->pmd)))) { + spin_unlock(pvmw->ptl); + goto again; + } + return true; } @@ -278,7 +288,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw, &ptl)) { + if (!map_pte(pvmw, &pmde, &ptl)) { if (!pvmw->pte) goto restart; goto next_pte; @@ -307,6 +317,12 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (!pvmw->ptl) { pvmw->ptl = ptl; spin_lock(pvmw->ptl); + if (unlikely(!pmd_same(pmde, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, pvmw->ptl); + pvmw->ptl = NULL; + pvmw->pte = NULL; + goto restart; + } } goto this_pte; } while (pvmw->address < end); From patchwork Thu Aug 22 07:13:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1975303 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Nrg+QyPZ; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-324-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WqDyv2WQRz1ybW for ; Thu, 22 Aug 2024 17:16:39 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WqDwt0sq1z2ytV; Thu, 22 Aug 2024 17:14:54 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::232" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Nrg+QyPZ; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::232; helo=mail-oi1-x232.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-oi1-x232.google.com (mail-oi1-x232.google.com [IPv6:2607:f8b0:4864:20::232]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WqDws58Ynz2ysd for ; Thu, 22 Aug 2024 17:14:53 +1000 (AEST) Received: by mail-oi1-x232.google.com with SMTP id 5614622812f47-3db16b2c1d2so306875b6e.2 for ; Thu, 22 Aug 2024 00:14:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310891; x=1724915691; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7wvuIpVK8M95P7HYfgMO/FZeMoPfoEIc6BtRbN48KOY=; b=Nrg+QyPZRo7HAbdfrHvMAOTK/ugOWyPnbUj8atL/DZAHAGUR9x9Wx5RtIE3ZARG0OS k/rjmQRdLcGyueWIVWdJCAixzVhw64LDN4q8uCueOqD2iQGBQHEM68L7mXOzUputg4rt 6bEhjeSpmLOaB1lNoKVqXrD1pBdj7XVtCet4LEQ69RpbK49ICXQc5e0CIibS0gXsL2oN d+po/AUPScIxh8WyeT45ns1Lni+6KLQkBcNNjC9RcQhQy6baBzPKs86dGTfwr2DBHnoF nfQxisF/pH32C9t76lswwicqaVKCexeW+ilMe6+Ytmcx8UtCfu3yMwxeSoV8T+A2mOAM xacg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310891; x=1724915691; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7wvuIpVK8M95P7HYfgMO/FZeMoPfoEIc6BtRbN48KOY=; b=gIki08nSLdsWZdsyMvK16mz3BoXO/Ql3Fq1SP5U7JVRKddMPi+4FAGVRxy6Lu1jKhy SE9qfK7bL0uifsK5EEq/oKEJchkdfSGKwi6N7ryMB12eISRgKbLpf9zeMHD0Skrc7c1q B8RLe5TP/aR6rc+EeREDEAAURFH8X7PrcP6tm8SqUMS+xXlKk01Cp7ByDDw8v9k1QErh yUxJhmZNEtOEUz2xhHNXDDOLVNV7d0NS9X2SvVEeD41ab5x2p9M25Qz93BbM53azJw0Y 1HZXT+66zK3itrf9vvCvNF3H3q8cUgrD1vyp2qdCc7BlWcbegpPIq2FLWyr5CDuHjAEN 0hmg== X-Forwarded-Encrypted: i=1; AJvYcCUAXEH1NV6jSWrYGXKJPnfVEc7kv4y4HksvEF4GPss5Ya7iuAEe9u93h6qqg98ybhLthvdyhXGvEfE9KVw=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yxgyxjmt+wlbYZGfCu4cjNJMls0bLMsJQ1HgeVay+ZBaC8H518A SssRQPNL76T2ffXOnDv68WL8t2rwNL7g2WC0jMoXnYjSGWDrcue9ual2I0CKM75rHLO+apREJxb it3Y= X-Google-Smtp-Source: AGHT+IH3dTXEnimLAtKAfnTveP36n8fQueYjLUMvugh9RpWMjyLI+ldrjyQ+VF2Ub3qToXnGQhbMbg== X-Received: by 2002:a05:6808:3847:b0:3d6:2b12:7dc0 with SMTP id 5614622812f47-3de22ff6ae0mr1325714b6e.20.1724310890896; Thu, 22 Aug 2024 00:14:50 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:50 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 11/14] mm: userfaultfd: move_pages_pte() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:26 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In move_pages_pte(), we may modify the dst_pte and src_pte after acquiring the ptl, so convert it to using pte_offset_map_rw_nolock(). But since we already do the pte_same() check, there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng --- mm/userfaultfd.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 6ef42d9cd482e..414ee744257b7 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1135,7 +1135,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, spinlock_t *src_ptl, *dst_ptl; pte_t *src_pte = NULL; pte_t *dst_pte = NULL; - + pmd_t dummy_pmdval; struct folio *src_folio = NULL; struct anon_vma *src_anon_vma = NULL; struct mmu_notifier_range range; @@ -1146,7 +1146,14 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, src_addr, src_addr + PAGE_SIZE); mmu_notifier_invalidate_range_start(&range); retry: - dst_pte = pte_offset_map_nolock(mm, dst_pmd, dst_addr, &dst_ptl); + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we will use pte_same() to detect the change of the pte + * entry, there is no need to get pmdval, so just pass a dummy variable + * to it. + */ + dst_pte = pte_offset_map_rw_nolock(mm, dst_pmd, dst_addr, &dummy_pmdval, + &dst_ptl); /* Retry if a huge pmd materialized from under us */ if (unlikely(!dst_pte)) { @@ -1154,7 +1161,9 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, goto out; } - src_pte = pte_offset_map_nolock(mm, src_pmd, src_addr, &src_ptl); + /* same as dst_pte */ + src_pte = pte_offset_map_rw_nolock(mm, src_pmd, src_addr, &dummy_pmdval, + &src_ptl); /* * We held the mmap_lock for reading so MADV_DONTNEED From patchwork Thu Aug 22 07:13:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1975304 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=LtgK/lsJ; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-325-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WqDz31htqz1ybW for ; Thu, 22 Aug 2024 17:16:47 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WqDwz1SBkz2ytm; Thu, 22 Aug 2024 17:14:59 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62c" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=LtgK/lsJ; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62c; helo=mail-pl1-x62c.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62c.google.com (mail-pl1-x62c.google.com [IPv6:2607:f8b0:4864:20::62c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WqDwy6Mh9z2ysd for ; Thu, 22 Aug 2024 17:14:58 +1000 (AEST) Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-202089e57d8so3197855ad.0 for ; Thu, 22 Aug 2024 00:14:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310897; x=1724915697; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TKuz+eT/BjM4oBQXjKjArJ5u0hGAcRe4llal02NX1d4=; b=LtgK/lsJk3/ahGjfxuGiDFPOCt1n9xIpLit235SLiM0S+NmTxV1dIALR2G9prhtOkp HWDBGLuijGo/NeHPgUbSi5k4xCInkfeNkJvlzGkjR3qopton0uZ4URvSRRKR9cNg98yJ Ia9SH9lA+PRGDyG0S1e/WtHh3ts4QV+mqIVIpnp6pN7T7SyzO/W0unJoxY2zDFQaAvNo TRY8Ipv+6cGDVOx3818mJ5Yv4gujIFQW1rZeP7NmA9QvMflqAhKf6w09Ewu+bcG3iW5C ma1uWqb4gw7ueH4r0Cy0S6mYC7U09vU61gH1QBpXOEaFzeDh6DIuZa6VcieuVWx+zCp1 1scQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310897; x=1724915697; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TKuz+eT/BjM4oBQXjKjArJ5u0hGAcRe4llal02NX1d4=; b=F3W3e23r1BVk3zGSQwdu0B1PIncL7i/JMEuQGIuxG+vkxH540FXbJgj+Fyb8penPOj TJJxQ+QoYLX/mMalmsP/EGP8eG8eelnPUMNp/4uaFlykZPtZJr72qth2ZejTTMBC9pXC nHjNj6a7Xl5DlBdn8bVpXOywirYOU0LizS42s3F5dIXxMvCM6EB0mRCJeSpp8SyCLsu6 +/3dVZ9KPQUK9/KF0V+FtjvZlnGLVR0J9IMFFgMiP56cBhNggIpNAaVEA06bOEgdKQX5 dnosIA7BrU+iIs3yLE8tVlXxr3bXupfhvkK/wlRW3/xqNZlQLT3O1q9E9RIX1bzSQBg5 Yuhw== X-Forwarded-Encrypted: i=1; AJvYcCXJjb0vXT33XOaQn4Ij0zylui4hMeNmeK24CnWyd55g9o670c39Phh+pGjfNFnYA9j4QoZtttM+nh7pmLA=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yz4eUSqEKRmO/dUk72DnpcyrU9mrXOBeTGjLXne3Lg/DwFyXEmu vl7BBDLcJgu6T0FqFB7ILXcB/bohOD7m1o2cdIP5TvXFDBXw74lh705V0JoC6/k= X-Google-Smtp-Source: AGHT+IF6H0tDfgQqKcPaVFXqnNnvtvkdaP6bepcTCK17Vph5rD33YztktUCyS9gQ0weOULsun7jJQw== X-Received: by 2002:a17:902:e88d:b0:1ff:43a8:22f2 with SMTP id d9443c01a7336-2037fe188b8mr31159405ad.24.1724310896888; Thu, 22 Aug 2024 00:14:56 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:56 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 12/14] mm: multi-gen LRU: walk_pte_range() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:27 +0800 Message-Id: <0e2980e9a31d3503a5dfb350c04f4ede4d05220e.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In walk_pte_range(), we may modify the pte entry after holding the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pmd entry. Signed-off-by: Qi Zheng --- mm/vmscan.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 5dc96a843466a..89ef1ac8eb1a6 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3396,8 +3396,10 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); int old_gen, new_gen = lru_gen_from_seq(max_seq); + pmd_t pmdval; - pte = pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + pte = pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, + &ptl); if (!pte) return false; if (!spin_trylock(ptl)) { @@ -3405,6 +3407,11 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, return false; } + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + return false; + } + arch_enter_lazy_mmu_mode(); restart: for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) { From patchwork Thu Aug 22 07:13:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1975306 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=fnQsOP6y; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-327-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WqDzK6776z1ybW for ; Thu, 22 Aug 2024 17:17:01 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WqDx46j29z2yvx; Thu, 22 Aug 2024 17:15:04 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::433" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=fnQsOP6y; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::433; helo=mail-pf1-x433.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WqDx44H5Wz2ysd for ; Thu, 22 Aug 2024 17:15:04 +1000 (AEST) Received: by mail-pf1-x433.google.com with SMTP id d2e1a72fcca58-7106cf5771bso456060b3a.2 for ; Thu, 22 Aug 2024 00:15:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310903; x=1724915703; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gFdM7jnff0z8zF8oYknH78hRu/cQudWIIbslP9zGsAs=; b=fnQsOP6yixVAPuVjU4jC/uwZSD1bnfJlQidlOA65WfL8mhCM023F0M2kIuqlRD8o1r SLyEFpo6gA2pWIRVdHrBv1zlNAjvH+sG71nghpAuRyAcbdAvGNsWGRM9fouu4RF3pzr3 ul595/EWTbpNM/0nnYolleTsVvacY4WtGxijsLsk3/yXL/lBTqCpp14gtAyure2okJlV 4j4zsZbLLTocXzLDxBoVjtGMon+/hWkixklLBe3b8hHO+XbpNa0n5fEQqVJ9zYDTJjfV dC6K/1eTX6CroUjSeBUj/cAj5+1g1ebVtiHjuU+yhGEEYEOgP29VfUQox7DE9SXshoOJ thiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310903; x=1724915703; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gFdM7jnff0z8zF8oYknH78hRu/cQudWIIbslP9zGsAs=; b=o9J4kQ/JjHPR+gPnCpwdJv8waHvzMvx7Wn7OaYhmgD+b8vk0F8L9q9z0c6hppCTuQt NiCdSluDrPg9vH6zWc6PXkOOzgeuj9h44vfilvKxHG8C5S1lccWnUEpcTg4E7qyoh3pB pk12PdSaxt6PYUpcZvnKYgMktgiJOC1ug03zIqo7dER7JJ3rx+p6tUHkCHR14DyZOIsh pmOyZ7wYFUxPd3hq4CU6tJbXu5eXJpN8Gkec67iBfFNvkQm/flyd4dR4Is27EMdQKlgl ovkfsu7h88uCHML26k07YmlrNyRMDlAaopKvLybV4KCUp8TdK7iLKIsr8N5o4GQ9+CkK 703A== X-Forwarded-Encrypted: i=1; AJvYcCUhfM8mHbaFym7kmvhk4jgrpFiYDyNTSig5rdBioGas3rremYq2Q/Xvuv7sS06VD7SNsRzPnsCkSGz84lc=@lists.ozlabs.org X-Gm-Message-State: AOJu0Ywuk71nPbUTmSSc1ALGXfGJhRtU3v6/JCbFYJvgbSi0l6My8k7O YUTPI+jPDDC+yjXjyhMu/bek50Wj+U3Ihqs1FQRL5NR5oQCzwi4KoWKSXZmlzzw= X-Google-Smtp-Source: AGHT+IElSFOoF4KMPSsF0Gy5tbhJtfEKgqkCJ+pjJm8bK93AAf5vkcbpbvWiZRIPotD+TEYG0fABhg== X-Received: by 2002:a05:6a00:b53:b0:70d:2fb5:f996 with SMTP id d2e1a72fcca58-714234922a8mr5431869b3a.11.1724310902611; Thu, 22 Aug 2024 00:15:02 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:15:02 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 13/14] mm: pgtable: remove pte_offset_map_nolock() Date: Thu, 22 Aug 2024 15:13:28 +0800 Message-Id: <061a61c5871948680488ad02fd1e03547948a500.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 Now no users are using the pte_offset_map_nolock(), remove it. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 3 --- include/linux/mm.h | 2 -- mm/pgtable-generic.c | 21 --------------------- 3 files changed, 26 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index 08d0e706a32db..581446d4a4eba 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -16,9 +16,6 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_lock() maps PTE and takes PTE table lock, returns pointer to PTE with pointer to its PTE table lock, or returns NULL if no PTE table; - - pte_offset_map_nolock() - maps PTE, returns pointer to PTE with pointer to its PTE table - lock (not taken), or returns NULL if no PTE table; - pte_offset_map_ro_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; diff --git a/include/linux/mm.h b/include/linux/mm.h index a00cb35ce065f..1d3e8882f6ba1 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2952,8 +2952,6 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, return pte; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 9a1666574c959..3b6dd8bb5b134 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -305,18 +305,6 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) return NULL; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp) -{ - pmd_t pmdval; - pte_t *pte; - - pte = __pte_offset_map(pmd, addr, &pmdval); - if (likely(pte)) - *ptlp = pte_lockptr(mm, &pmdval); - return pte; -} - pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp) { @@ -374,15 +362,6 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, * and disconnected table. Until pte_unmap(pte) unmaps and rcu_read_unlock()s * afterwards. * - * pte_offset_map_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); - * but when successful, it also outputs a pointer to the spinlock in ptlp - as - * pte_offset_map_lock() does, but in this case without locking it. This helps - * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that time - * act on a changed *pmd: pte_offset_map_nolock() provides the correct spinlock - * pointer for the page table that it returns. In principle, the caller should - * recheck *pmd once the lock is taken; in practice, no callsite needs that - - * either the mmap_lock for write, or pte_same() check on contents, is enough. - * * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like * pte_offset_map(); but when successful, it also outputs a pointer to the * spinlock in ptlp - as pte_offset_map_lock() does, but in this case without From patchwork Thu Aug 22 07:13:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1975308 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=ZvtQBeIA; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-329-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WqDzZ0nb4z1ybW for ; Thu, 22 Aug 2024 17:17:14 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WqDxC13k0z2ywh; Thu, 22 Aug 2024 17:15:11 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::432" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=ZvtQBeIA; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::432; helo=mail-pf1-x432.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WqDxB5ftWz2ywR for ; Thu, 22 Aug 2024 17:15:10 +1000 (AEST) Received: by mail-pf1-x432.google.com with SMTP id d2e1a72fcca58-7142448aaf9so338316b3a.1 for ; Thu, 22 Aug 2024 00:15:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310908; x=1724915708; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3XLeH1tU+payUby5GohREupClIvRGZdbeLsp/8eA5FY=; b=ZvtQBeIAYrrR2IbbELeSuHKFoym6qJ2jyqz2VVi5UmGhTCHUKIkaqhSuZNEANQt7FA y8sSSZdx8hTJKRNX2yNJ1MBIEgl6hRMlg9rWjqHC6hcfD68+huo11vv6Y+1hA2NhzhIM R0c24o8cP+MlxikeQYwZ5+xBc5eHORMYXqsMDg/y0hukdFx3TrIYwDlMrmZ50AJAEfBn /l/3v5L+H8YfjFEQpRYpo+LBxpI4edYCAQr0NkppDUTb7SL7uSZQ+URY93VLENr0p6+o cwmxJmgq/i6cFruqUi3Nccibaya0OHp85rbYrMhTQLYkG9nFuiyiSeTYAWSuk7nmxYq+ SPrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310908; x=1724915708; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3XLeH1tU+payUby5GohREupClIvRGZdbeLsp/8eA5FY=; b=pDxUJErllgs3hMAoSci5ZjNdgwyJe8uEvTELvrPkdVFDQUCANGEQT80aQs7z00A424 FNqcenZV8/lLy2WKGTFbRZzb1mnKhlGW6sFS0yOf1gVj7mpJMOZwsQR1kwO3ovDNd7Ji dupL4kUZNA45cWUYjUCqpfsj5EPn2IoCebmcTZWi0M8+XFBCGDz8KcqNYUBAl9o+b/Qb PV4HvkVE8ZT5VRvon+DdRe7NBG7sMhPD8ZBc6BQE6+i4IBISo5HEO08iMX3hsahG2mbz J5jIiD7DOnPo47s41Idg0ZW8hAdBRi5lydgOyaAUI1AVW4lfzpB9lpEnGzyoHecB7nD5 pmmw== X-Forwarded-Encrypted: i=1; AJvYcCXTxFpvlP0Z6J+xj5jnkKXJ1Up2IY6aw6MMhaYCYU4IYEo7ZIq9QygektJlr1oe0MoYdHHrTsLJzmus8uE=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yw1x45HFBguqlPqIP3CHUmhT2k2GbX9G3C9GN8lwKs4fbKtwjPr 8J1OBALU89MYMPSjIJ3fvXMIjQmC4BgHo4adCbcuxKOeLG1fv90dU5yxxiNGD9I= X-Google-Smtp-Source: AGHT+IGjO58kdb6JjGpOfA0UnXFn2ByeOkp/uIZzxQt7YtlV+9PVIdBNBOEj4fOrvZDCpXHiWUn49Q== X-Received: by 2002:aa7:8244:0:b0:710:d294:a2b with SMTP id d2e1a72fcca58-7143176773amr3032765b3a.8.1724310908168; Thu, 22 Aug 2024 00:15:08 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.15.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:15:07 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 14/14] mm: khugepaged: retract_page_tables() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:29 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In retract_page_tables(), we may modify the pmd entry after acquiring the pml and ptl, so we should also check whether the pmd entry is stable. Using pte_offset_map_rw_nolock() + pmd_same() to do it. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 15d3f7f3c65f2..799412041d262 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1723,6 +1723,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) spinlock_t *pml; spinlock_t *ptl; bool skipped_uffd = false; + pte_t *pte; /* * Check vma->anon_vma to exclude MAP_PRIVATE mappings that @@ -1758,11 +1759,25 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) addr, addr + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); + pte = pte_offset_map_rw_nolock(mm, pmd, addr, &pgt_pmd, &ptl); + if (!pte) { + mmu_notifier_invalidate_range_end(&range); + continue; + } + pml = pmd_lock(mm, pmd); - ptl = pte_lockptr(mm, pmd); if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + if (ptl != pml) + spin_unlock(pml); + mmu_notifier_invalidate_range_end(&range); + continue; + } + pte_unmap(pte); + /* * Huge page lock is still held, so normally the page table * must remain empty; and we have already skipped anon_vma