From patchwork Wed Aug 7 19:48:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1970208 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=YRhbpR9/; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=YRhbpR9/; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WfLP16KHlz1yfC for ; Thu, 8 Aug 2024 05:49:57 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=YRhbpR9/; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=YRhbpR9/; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WfLP15G39z3dDp for ; Thu, 8 Aug 2024 05:49:57 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=YRhbpR9/; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=YRhbpR9/; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=redhat.com (client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=peterx@redhat.com; receiver=lists.ozlabs.org) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WfLMD550Rz3d96 for ; Thu, 8 Aug 2024 05:48:22 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723060099; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9Kq72BfB1Q9PD26Udwsuz2Gt7rCvKvwunl3vkpU7op8=; b=YRhbpR9/bEBQ8FUkOxax6RVR0ouj58Zp+y5gK9iyFahINpBqNP8GnN5ZpwUHd4OH997LhK /b6YVpDcM9UIEtnJB0OX1GAk/Z9bxN+vLq+sohBtKQA8Ez7UPrgh3UHuZDuDGkSOw3Ay1X UkdnqvIku3b/RUybWhHrb51jgftYABk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723060099; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9Kq72BfB1Q9PD26Udwsuz2Gt7rCvKvwunl3vkpU7op8=; b=YRhbpR9/bEBQ8FUkOxax6RVR0ouj58Zp+y5gK9iyFahINpBqNP8GnN5ZpwUHd4OH997LhK /b6YVpDcM9UIEtnJB0OX1GAk/Z9bxN+vLq+sohBtKQA8Ez7UPrgh3UHuZDuDGkSOw3Ay1X UkdnqvIku3b/RUybWhHrb51jgftYABk= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-447-nczqYeA1PGCtz_hvbNZ4Lw-1; Wed, 07 Aug 2024 15:48:18 -0400 X-MC-Unique: nczqYeA1PGCtz_hvbNZ4Lw-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-6b7a0e7b823so476426d6.0 for ; Wed, 07 Aug 2024 12:48:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723060097; x=1723664897; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9Kq72BfB1Q9PD26Udwsuz2Gt7rCvKvwunl3vkpU7op8=; b=Qa/ZTkEeaSWU0dT410edeOrWMnZB90pupl/3H/sI+THnLD1AfUV9iRk18SeHSi660w Hy7kHIffiWhNy5hffQXub0NLxANzTGdhg80PJUKJwAO02PLtkT2Oop4A7im9+X/gSCHq eP5Vu5g7XaBU4cyIPuN7g9RA3A+7yiFdS+XGn2lzGufOFBfzZoSSAGiEmWTjvR0bwpb6 81nChzP1/MqJw9B7TiVcM0PU3Ar5cHqXqX6MeEVOOZRMLvJECdT0zT7vDsz5xe/SxBZI +5f6db6ptAkaINaWSXaaN8T7bRnfqo+tbPhTLPKdHQ6jywNhc2FInf/SZpuckcSXUTqv yL0w== X-Forwarded-Encrypted: i=1; AJvYcCV03ILoiKUklJKs1X4cxFwsER71XarvIz4JZHsvKYC/MEwO4bfZncqX99bfnRLjzQwzUQ3hkG+MTufQAU8=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yym5BKXX9nn95ukTYsIXG10OVgfuwFGPpLKrrczPIbLSXxJFKYd PaWuoIBrGHCrSCLKHWaFkUH2mFjXkNgIqeTe5ChDKpIOqlFaDXSNpHKIeC9ICKrpSjd9bFwaKZ9 KVUxckMhRS+rLu/xE7D0DZ28Q2kKbLcF3jN6RJoI/PTLdJuwBD5ysVQvPH1pGE7M= X-Received: by 2002:ad4:5f89:0:b0:6b2:b5b5:124e with SMTP id 6a1803df08f44-6bb98201fe5mr130641396d6.0.1723060097407; Wed, 07 Aug 2024 12:48:17 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE2kwiMDYWwjAOmnvJrXFpiL3DqhuBYXy1c4XYYU13it34Yz8NGLRxRmyIIbMBrd7nnJuhzgA== X-Received: by 2002:ad4:5f89:0:b0:6b2:b5b5:124e with SMTP id 6a1803df08f44-6bb98201fe5mr130641186d6.0.1723060096990; Wed, 07 Aug 2024 12:48:16 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6bb9c78ae4asm59853256d6.33.2024.08.07.12.48.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Aug 2024 12:48:16 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v4 1/7] mm/dax: Dump start address in fault handler Date: Wed, 7 Aug 2024 15:48:05 -0400 Message-ID: <20240807194812.819412-2-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240807194812.819412-1-peterx@redhat.com> References: <20240807194812.819412-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: James Houghton , David Hildenbrand , Dave Hansen , peterx@redhat.com, Christophe Leroy , Dan Williams , Dave Jiang , "Aneesh Kumar K . V" , x86@kernel.org, Hugh Dickins , Matthew Wilcox , Ingo Molnar , Huang Ying , Rik van Riel , Nicholas Piggin , Borislav Petkov , "Kirill A . Shutemov" , Thomas Gleixner , Vlastimil Babka , Oscar Salvador , linuxppc-dev@lists.ozlabs.org, Andrew Morton , Rick P Edgecombe , Mel Gorman Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Currently the dax fault handler dumps the vma range when dynamic debugging enabled. That's mostly not useful. Dump the (aligned) address instead with the order info. Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- drivers/dax/device.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index 2051e4f73c8a..9c1a729cd77e 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -235,9 +235,9 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, unsigned int order) int id; struct dev_dax *dev_dax = filp->private_data; - dev_dbg(&dev_dax->dev, "%s: %s (%#lx - %#lx) order:%d\n", current->comm, - (vmf->flags & FAULT_FLAG_WRITE) ? "write" : "read", - vmf->vma->vm_start, vmf->vma->vm_end, order); + dev_dbg(&dev_dax->dev, "%s: op=%s addr=%#lx order=%d\n", current->comm, + (vmf->flags & FAULT_FLAG_WRITE) ? "write" : "read", + vmf->address & ~((1UL << (order + PAGE_SHIFT)) - 1), order); id = dax_read_lock(); if (order == 0) From patchwork Wed Aug 7 19:48:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1970207 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=XGAWFOvG; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=PODobSHL; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WfLNG0YCkz1yfC for ; Thu, 8 Aug 2024 05:49:18 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=XGAWFOvG; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=PODobSHL; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WfLNF66KJz3dJs for ; Thu, 8 Aug 2024 05:49:17 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=XGAWFOvG; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=PODobSHL; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=redhat.com (client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=peterx@redhat.com; receiver=lists.ozlabs.org) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WfLMD4XFGz3d8K for ; Thu, 8 Aug 2024 05:48:24 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723060101; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6aC2RF+//p0GJfjgH/iZrlW9ATLJl4mVfe8Rdc4jHgk=; b=XGAWFOvG4zuw1lxUHs7LeSvHNCITA/C2mNJZXN9EGvTUgcye0ReTxjX7whZ71KtY/Usvhz NnogIvD+pOob5/EbH9RUIJFIl38Lsd3Ml01rQmDuaJhpiDvQbZsQizJ75e1oj1Hw4ZZR93 hXkSMymI5PhkkM15GGtLleUIE7AYeyk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723060102; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6aC2RF+//p0GJfjgH/iZrlW9ATLJl4mVfe8Rdc4jHgk=; b=PODobSHL9U/z+Itf4YhoZ/dBLtHitahnMQqNSlrQuYMJxxQV3svaChA4tUsbR5shpZUGVl V61dGf9mgUtQqPvg6kd/LCQFGjlhuORrmTmG2mvaL4D4SHvWKPUHuKL5S7uSqt1Uqri63b dhHB9kUdGkK16gkybykW/tZtX60DTmk= Received: from mail-ua1-f71.google.com (mail-ua1-f71.google.com [209.85.222.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-335-GvuIPZ8xNsSd5UoA1UA4vQ-1; Wed, 07 Aug 2024 15:48:20 -0400 X-MC-Unique: GvuIPZ8xNsSd5UoA1UA4vQ-1 Received: by mail-ua1-f71.google.com with SMTP id a1e0cc1a2514c-8257c3b4904so17715241.1 for ; Wed, 07 Aug 2024 12:48:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723060099; x=1723664899; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6aC2RF+//p0GJfjgH/iZrlW9ATLJl4mVfe8Rdc4jHgk=; b=na51piB9kETl2Rh+9qiu9wUVORoHiyt7wA0fdmqWXBR8rKZEFxzccJt5PpoMjhys+D dM4CFvfWkRvVWfyTXPnj4BXUIqI6hiMvRL42cbeHMjwk+cvhsf0ORH3yO+83eYgylmlC MU9UM9owzNjYFAtXcb0xm0xU+aQjOczYKQdKqsz+hN4qhEkoLfBUKKdxDPiOu1ZMRVXD /KPvZ4CYiCPYlAA59TJmh/z15AQjK42gI+pzaX9sdCXay8zpBGye7v6irbw7wHyu3BRh Hjzlb/XPuJU3JccDLe/nfuWQ73oxvIgEreYxhG/KUK5d9m1PFM9u+rcFnvZ1KWpZFXfw O+sQ== X-Forwarded-Encrypted: i=1; AJvYcCVk2CL+FRuPCngkxtgDvsjiJOl5uQWia8Hic8/LrE8g/ja9h3XQoT3ryfQXS5Ys+mkvV9Xm4r7ggUvlxr8o8F4U9bUrKJhOwXszIjfH4Q== X-Gm-Message-State: AOJu0YwbbPHsflkz5lNlOFDxYIH96bSX7GX+kzLUvDqHYs3D+OuyJl97 PqdABk21t1MAT8cZvdx8enHAII6ujQoDZWuafVE3Ffn4r9D8ULH6aLmyb4p8GXUHh6l3VnAhWOu DU2E35WU0h74oZt4YD2kLFtsEEYuqPci9xDj0X52i/Noo6LUqpRmDz4Fl4phIrYU= X-Received: by 2002:a67:f445:0:b0:493:c75f:4c71 with SMTP id ada2fe7eead31-4945bf7f9bemr10182699137.4.1723060099540; Wed, 07 Aug 2024 12:48:19 -0700 (PDT) X-Google-Smtp-Source: AGHT+IExnSrOtwcHXkdqrS3UwyseZPMUb/72Vd4NIKyn9I98RgoSLE2TM2KVGSHUSlSxMHs5MS4l7Q== X-Received: by 2002:a67:f445:0:b0:493:c75f:4c71 with SMTP id ada2fe7eead31-4945bf7f9bemr10182670137.4.1723060099151; Wed, 07 Aug 2024 12:48:19 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6bb9c78ae4asm59853256d6.33.2024.08.07.12.48.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Aug 2024 12:48:18 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v4 2/7] mm/mprotect: Push mmu notifier to PUDs Date: Wed, 7 Aug 2024 15:48:06 -0400 Message-ID: <20240807194812.819412-3-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240807194812.819412-1-peterx@redhat.com> References: <20240807194812.819412-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: James Houghton , kvm@vger.kernel.org, David Hildenbrand , Dave Hansen , peterx@redhat.com, Christophe Leroy , Dan Williams , David Rientjes , Dave Jiang , "Aneesh Kumar K . V" , x86@kernel.org, Hugh Dickins , Matthew Wilcox , Ingo Molnar , Huang Ying , Rik van Riel , Sean Christopherson , Nicholas Piggin , Borislav Petkov , "Kirill A . Shutemov" , Thomas Gleixner , Vlastimil Babka , Oscar Salvador , linuxppc-dev@lists.ozlabs.org, Paolo Bonzini , Andrew Morton , Rick P Edgecombe , Mel Gorman Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" mprotect() does mmu notifiers in PMD levels. It's there since 2014 of commit a5338093bfb4 ("mm: move mmu notifier call from change_protection to change_pmd_range"). At that time, the issue was that NUMA balancing can be applied on a huge range of VM memory, even if nothing was populated. The notification can be avoided in this case if no valid pmd detected, which includes either THP or a PTE pgtable page. Now to pave way for PUD handling, this isn't enough. We need to generate mmu notifications even on PUD entries properly. mprotect() is currently broken on PUD (e.g., one can easily trigger kernel error with dax 1G mappings already), this is the start to fix it. To fix that, this patch proposes to push such notifications to the PUD layers. There is risk on regressing the problem Rik wanted to resolve before, but I think it shouldn't really happen, and I still chose this solution because of a few reasons: 1) Consider a large VM that should definitely contain more than GBs of memory, it's highly likely that PUDs are also none. In this case there will have no regression. 2) KVM has evolved a lot over the years to get rid of rmap walks, which might be the major cause of the previous soft-lockup. At least TDP MMU already got rid of rmap as long as not nested (which should be the major use case, IIUC), then the TDP MMU pgtable walker will simply see empty VM pgtable (e.g. EPT on x86), the invalidation of a full empty region in most cases could be pretty fast now, comparing to 2014. 3) KVM has explicit code paths now to even give way for mmu notifiers just like this one, e.g. in commit d02c357e5bfa ("KVM: x86/mmu: Retry fault before acquiring mmu_lock if mapping is changing"). It'll also avoid contentions that may also contribute to a soft-lockup. 4) Stick with PMD layer simply don't work when PUD is there... We need one way or another to fix PUD mappings on mprotect(). Pushing it to PUD should be the safest approach as of now, e.g. there's yet no sign of huge P4D coming on any known archs. Cc: kvm@vger.kernel.org Cc: Sean Christopherson Cc: Paolo Bonzini Cc: David Rientjes Cc: Rik van Riel Signed-off-by: Peter Xu --- mm/mprotect.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index 37cf8d249405..d423080e6509 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -363,9 +363,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, unsigned long next; long pages = 0; unsigned long nr_huge_updates = 0; - struct mmu_notifier_range range; - - range.start = 0; pmd = pmd_offset(pud, addr); do { @@ -383,14 +380,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, if (pmd_none(*pmd)) goto next; - /* invoke the mmu notifier if the pmd is populated */ - if (!range.start) { - mmu_notifier_range_init(&range, - MMU_NOTIFY_PROTECTION_VMA, 0, - vma->vm_mm, addr, end); - mmu_notifier_invalidate_range_start(&range); - } - _pmd = pmdp_get_lockless(pmd); if (is_swap_pmd(_pmd) || pmd_trans_huge(_pmd) || pmd_devmap(_pmd)) { if ((next - addr != HPAGE_PMD_SIZE) || @@ -431,9 +420,6 @@ static inline long change_pmd_range(struct mmu_gather *tlb, cond_resched(); } while (pmd++, addr = next, addr != end); - if (range.start) - mmu_notifier_invalidate_range_end(&range); - if (nr_huge_updates) count_vm_numa_events(NUMA_HUGE_PTE_UPDATES, nr_huge_updates); return pages; @@ -443,22 +429,36 @@ static inline long change_pud_range(struct mmu_gather *tlb, struct vm_area_struct *vma, p4d_t *p4d, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { + struct mmu_notifier_range range; pud_t *pud; unsigned long next; long pages = 0, ret; + range.start = 0; + pud = pud_offset(p4d, addr); do { next = pud_addr_end(addr, end); ret = change_prepare(vma, pud, pmd, addr, cp_flags); - if (ret) - return ret; + if (ret) { + pages = ret; + break; + } if (pud_none_or_clear_bad(pud)) continue; + if (!range.start) { + mmu_notifier_range_init(&range, + MMU_NOTIFY_PROTECTION_VMA, 0, + vma->vm_mm, addr, end); + mmu_notifier_invalidate_range_start(&range); + } pages += change_pmd_range(tlb, vma, pud, addr, next, newprot, cp_flags); } while (pud++, addr = next, addr != end); + if (range.start) + mmu_notifier_invalidate_range_end(&range); + return pages; } From patchwork Wed Aug 7 19:48:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1970209 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Eq8hOgk3; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ZvlrfpGu; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WfLPv6517z1ydt for ; Thu, 8 Aug 2024 05:50:43 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Eq8hOgk3; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ZvlrfpGu; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WfLPv53Sxz3d8K for ; Thu, 8 Aug 2024 05:50:43 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Eq8hOgk3; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ZvlrfpGu; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=redhat.com (client-ip=170.10.133.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=peterx@redhat.com; receiver=lists.ozlabs.org) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WfLMG673zz3dFL for ; Thu, 8 Aug 2024 05:48:26 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723060103; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DPSgdrK+/9txvlBVeWURx0wHDXuvLt3nB8aMlUhEkV8=; b=Eq8hOgk3eeaZwWxSy7wISbkn0rD4pV0TMF5t2z5CgEyjm2siE+ntbXQ7Lq6vdX70spKXaK fCVOrS54DGe6L9CTz1GzQ/+hUy0kHtAVjfaG4gdTVI9KsUy/pCiKtmbjgCPjnRsPfqJ5W6 DjW5g+nkKgqvD6EgsrlGw5sf+2rJkPc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723060104; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DPSgdrK+/9txvlBVeWURx0wHDXuvLt3nB8aMlUhEkV8=; b=ZvlrfpGunAYik7lEe5hIbJxFhcW+bWZfQdbDPutfp6Vj01XLL+0Ja8GP2aplk8F8i82Uep wP/eDoKYEqsurGaK3rNPkJwQxWGuOLi5OUNeGc6lovXdncsOzLyOi4bY/k6Q3RQH7gJ2D0 6Lgw6MBtKmE795T8pxFdLwJKi/F1IbE= Received: from mail-oi1-f200.google.com (mail-oi1-f200.google.com [209.85.167.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-518-rPDZ2hX3Mb6e7lwu638kiA-1; Wed, 07 Aug 2024 15:48:22 -0400 X-MC-Unique: rPDZ2hX3Mb6e7lwu638kiA-1 Received: by mail-oi1-f200.google.com with SMTP id 5614622812f47-3db16c5eabeso58298b6e.1 for ; Wed, 07 Aug 2024 12:48:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723060101; x=1723664901; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DPSgdrK+/9txvlBVeWURx0wHDXuvLt3nB8aMlUhEkV8=; b=RYD1VHNVib3laOv6keaxVYNf3WxxpTEN8sYPoicXi053WZbsWUplGYduyMSIUeB+LD M7EtQsCc4Z/DKjpcN0uNWUX5ZS9YXIZNKM47Ic47hpZyPv46ElQgdHU4247IOyPJnTBw KRDwR73yUj3+OYWPVgTEat893QTBODV4OgujmajYGE+SQRZ5LuewbDjpGljCWBfY76He P9Q1f3k8wM2jhHHMKD/hHAlIei5BjxKJr6MrI88cfH7tkeLHEMauBbdb41qv5MABBFsD O2rjsO9kkJGVAG4s7EFDaM5Gfi6zQTamVE3SkmnZ3ZA6gF0cDAHfySMi0P8fZWJnzJB4 W8Hw== X-Forwarded-Encrypted: i=1; AJvYcCXtrpMyQo2BhY/LLe1CSds1tWVzhpwT3VKitNzHCciDRwHW1VAd1zFKHjXYSwFY+uIcKpHURSsYLhHfUs+UysqIGOi2UVFE8Ezie+mAMw== X-Gm-Message-State: AOJu0YxIY0dMIto4XccAvunS2+F1KQSoDv0KKGfausy25dvJeIQQ3OxN 3nwPzjIrkXklCtdQPGNLJCjxBu39Qe3xP4FV+8UQ8ATMdnGRqhGvlR0fJRysTtkEFYaCz/0PS/h I524palRHUh4CBGO+dKkmhxno9hw2t/hBaoCbJoVZO5sA3EKyNVsdgxdfHSnHcEc= X-Received: by 2002:a05:6358:a096:b0:1aa:c73d:5a95 with SMTP id e5c5f4694b2df-1af3b89cddemr1270642555d.0.1723060101665; Wed, 07 Aug 2024 12:48:21 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH//9GjXWmqzA+Y0OZdeiBZvOf2mh0xJt9rLdJnCDcBWQbkH3n97H6Shuw3uIhtGYCg2pjapA== X-Received: by 2002:a05:6358:a096:b0:1aa:c73d:5a95 with SMTP id e5c5f4694b2df-1af3b89cddemr1270641255d.0.1723060101168; Wed, 07 Aug 2024 12:48:21 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6bb9c78ae4asm59853256d6.33.2024.08.07.12.48.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Aug 2024 12:48:20 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v4 3/7] mm/powerpc: Add missing pud helpers Date: Wed, 7 Aug 2024 15:48:07 -0400 Message-ID: <20240807194812.819412-4-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240807194812.819412-1-peterx@redhat.com> References: <20240807194812.819412-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: James Houghton , David Hildenbrand , Dave Hansen , peterx@redhat.com, Christophe Leroy , Dan Williams , Dave Jiang , "Aneesh Kumar K . V" , x86@kernel.org, Hugh Dickins , Matthew Wilcox , Ingo Molnar , Huang Ying , Rik van Riel , Nicholas Piggin , Borislav Petkov , "Kirill A . Shutemov" , Thomas Gleixner , Vlastimil Babka , Oscar Salvador , linuxppc-dev@lists.ozlabs.org, Andrew Morton , Rick P Edgecombe , Mel Gorman Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" These new helpers will be needed for pud entry updates soon. Introduce them by referencing the pmd ones. Namely: - pudp_invalidate() - pud_modify() Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: linuxppc-dev@lists.ozlabs.org Cc: Aneesh Kumar K.V Signed-off-by: Peter Xu --- arch/powerpc/include/asm/book3s/64/pgtable.h | 3 +++ arch/powerpc/mm/book3s64/pgtable.c | 20 ++++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index 519b1743a0f4..5da92ba68a45 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -1124,6 +1124,7 @@ extern pmd_t pfn_pmd(unsigned long pfn, pgprot_t pgprot); extern pud_t pfn_pud(unsigned long pfn, pgprot_t pgprot); extern pmd_t mk_pmd(struct page *page, pgprot_t pgprot); extern pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot); +extern pud_t pud_modify(pud_t pud, pgprot_t newprot); extern void set_pmd_at(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd); extern void set_pud_at(struct mm_struct *mm, unsigned long addr, @@ -1384,6 +1385,8 @@ static inline pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, #define __HAVE_ARCH_PMDP_INVALIDATE extern pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +extern pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); #define pmd_move_must_withdraw pmd_move_must_withdraw struct spinlock; diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index f4d8d3c40e5c..5a4a75369043 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -176,6 +176,17 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, return __pmd(old_pmd); } +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + unsigned long old_pud; + + VM_WARN_ON_ONCE(!pud_present(*pudp)); + old_pud = pud_hugepage_update(vma->vm_mm, address, pudp, _PAGE_PRESENT, _PAGE_INVALID); + flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE); + return __pud(old_pud); +} + pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp, int full) { @@ -259,6 +270,15 @@ pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) pmdv &= _HPAGE_CHG_MASK; return pmd_set_protbits(__pmd(pmdv), newprot); } + +pud_t pud_modify(pud_t pud, pgprot_t newprot) +{ + unsigned long pudv; + + pudv = pud_val(pud); + pudv &= _HPAGE_CHG_MASK; + return pud_set_protbits(__pud(pudv), newprot); +} #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /* For use by kexec, called with MMU off */ From patchwork Wed Aug 7 19:48:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1970211 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EP5EY6UF; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=IZR87+RI; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WfLRQ5wrwz1ydt for ; Thu, 8 Aug 2024 05:52:02 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EP5EY6UF; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=IZR87+RI; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WfLRQ4vpcz3cXd for ; Thu, 8 Aug 2024 05:52:02 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EP5EY6UF; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=IZR87+RI; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=redhat.com (client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=peterx@redhat.com; receiver=lists.ozlabs.org) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WfLMM1VDrz3dLd for ; Thu, 8 Aug 2024 05:48:30 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723060107; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0ts+Sj64QZG1FGRDLVoa9nQimNVqUC6U/k8fGPxCUtI=; b=EP5EY6UFfks1Ce2q5WPA9TLSenl+gvnrQNUPK4EvGC8WZQcaXja70fS3j3d1D2ebQaNtTL PZ7T4jk6C4z/Uss1Z0WsFvQwcyXjy/184NeD/+1zUMIBMGIX49RbiQsvr5ckzrV3ymFLuv sR2ZAHqFjKL8eDD0EIlgkeF2/9pmZY0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723060108; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0ts+Sj64QZG1FGRDLVoa9nQimNVqUC6U/k8fGPxCUtI=; b=IZR87+RIamqQDto+wtO24buQyMy2d6lHUt6V53JgCVGnU6U6EDuO8mAM8DnQET88ndgBYj rdOcE+Bb2cXmwyVl5aHZpyAbw17ApoqgPBb38EPjbwLwshjgfRbfCJShCIHCztg73bpbv/ nG0t7vH/DAeQC0WgtqFJJHBwq+q2fMc= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-317-hTJtGU_WPuiB3h_xMwZQLw-1; Wed, 07 Aug 2024 15:48:24 -0400 X-MC-Unique: hTJtGU_WPuiB3h_xMwZQLw-1 Received: by mail-qv1-f69.google.com with SMTP id 6a1803df08f44-6ba92dd246fso407706d6.0 for ; Wed, 07 Aug 2024 12:48:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723060104; x=1723664904; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0ts+Sj64QZG1FGRDLVoa9nQimNVqUC6U/k8fGPxCUtI=; b=LAao0NhxNf2AF0btrMbh+kFbAE3pWhGwFiteThMRD/oV2XJJb1OtRmJKqt1j7BAgNQ /zoeOKb37jgbN1nvyR21OP3uS2KLsAzFBFzY4gx+Nri6Co0g20L8EYTOa2iyf0QoXZ2y Y/UsN+wXpsS7FIt13vGd+1K53G0dBo6Sos2gQ39efPBV3dFf82y+MktOWjo2xl+Qedza epKJK3jGPyQPlZj+aRITa2jb3MMkm9PiFporBS52w8ObSs214JtEUCIgx/9nWabBDjzd rRhRAHy8x6/2tCX56s+WnUuEHzvzC3sevqXlChVq7ElSUrRMUOjKIYh0WndhG9RVQQw/ Ix0w== X-Forwarded-Encrypted: i=1; AJvYcCXSw1z0rg1s1006WCYr1UR/abeMUO2/iJRJQIKqkSqXwACWRou5wongXCJ6TseLCZy0Z5mQOequEsgrIZs=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyxEPIuWD537WcK76pVqJPpijU5a5HvenUD70ABE1Z2sk0yseDf 6BTTslc7MMV4akouPc1PrlfagzEK3bqejbk8+OWE+H6CWI3zWACVFbJWmJAngnqc0oX/202+q2L I/ND6hbHo62DwmkYGUCEXAAT5n7ZOsg/sMpMRsPjOGcFn+2YEU+kRJ6LqYXfy4/c= X-Received: by 2002:ad4:5dca:0:b0:6b5:e3bc:af9a with SMTP id 6a1803df08f44-6bb9832d2dbmr147037496d6.2.1723060103630; Wed, 07 Aug 2024 12:48:23 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEOqKDpzCy2XKs/l7qwihRy4ZY0EbFICMSu3L2U6zTcNQWVR9cvlOSLvEsobqbXC66x3CvadA== X-Received: by 2002:ad4:5dca:0:b0:6b5:e3bc:af9a with SMTP id 6a1803df08f44-6bb9832d2dbmr147037356d6.2.1723060103178; Wed, 07 Aug 2024 12:48:23 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6bb9c78ae4asm59853256d6.33.2024.08.07.12.48.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Aug 2024 12:48:22 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v4 4/7] mm/x86: Make pud_leaf() only care about PSE bit Date: Wed, 7 Aug 2024 15:48:08 -0400 Message-ID: <20240807194812.819412-5-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240807194812.819412-1-peterx@redhat.com> References: <20240807194812.819412-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: James Houghton , David Hildenbrand , Dave Hansen , peterx@redhat.com, Christophe Leroy , Dan Williams , Dave Jiang , "Aneesh Kumar K . V" , x86@kernel.org, Hugh Dickins , Matthew Wilcox , Ingo Molnar , Huang Ying , Rik van Riel , Nicholas Piggin , Borislav Petkov , "Kirill A . Shutemov" , Thomas Gleixner , Vlastimil Babka , Oscar Salvador , linuxppc-dev@lists.ozlabs.org, Andrew Morton , Rick P Edgecombe , Mel Gorman Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" An entry should be reported as PUD leaf even if it's PROT_NONE, in which case PRESENT bit isn't there. I hit bad pud without this when testing dax 1G on zapping a PROT_NONE PUD. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Acked-by: Dave Hansen Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index e39311a89bf4..a2a3bd4c1bda 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1078,8 +1078,7 @@ static inline pmd_t *pud_pgtable(pud_t pud) #define pud_leaf pud_leaf static inline bool pud_leaf(pud_t pud) { - return (pud_val(pud) & (_PAGE_PSE | _PAGE_PRESENT)) == - (_PAGE_PSE | _PAGE_PRESENT); + return pud_val(pud) & _PAGE_PSE; } static inline int pud_bad(pud_t pud) From patchwork Wed Aug 7 19:48:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1970210 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EoTQY8MX; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EoTQY8MX; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WfLQg4P3jz1ydt for ; Thu, 8 Aug 2024 05:51:23 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EoTQY8MX; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EoTQY8MX; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WfLQg3dDkz3dLj for ; Thu, 8 Aug 2024 05:51:23 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EoTQY8MX; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EoTQY8MX; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=redhat.com (client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=peterx@redhat.com; receiver=lists.ozlabs.org) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WfLMM35bWz3dL2 for ; Thu, 8 Aug 2024 05:48:31 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723060108; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FAGFj0FHVAq5pVRnc576bB34mI1HflD3zt6t4lJRJgM=; b=EoTQY8MX/sjXJgghtqS7Zec906j68az8Rwxns7odhETIK2WTcCiWZY6nkyBTACEI0gx8c9 vUlha9QSFM+nTnenWoT+wshS2GTdi93sOAPUapByQWj+aoWlVmbNchGh3LEhBOa4H6y+UW oaRVjYLgxD8LJoZFuQa0tnvzdM1MGbA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723060108; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FAGFj0FHVAq5pVRnc576bB34mI1HflD3zt6t4lJRJgM=; b=EoTQY8MX/sjXJgghtqS7Zec906j68az8Rwxns7odhETIK2WTcCiWZY6nkyBTACEI0gx8c9 vUlha9QSFM+nTnenWoT+wshS2GTdi93sOAPUapByQWj+aoWlVmbNchGh3LEhBOa4H6y+UW oaRVjYLgxD8LJoZFuQa0tnvzdM1MGbA= Received: from mail-oo1-f69.google.com (mail-oo1-f69.google.com [209.85.161.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-465-yTy3uU3PM6-_PBJnK6JPHw-1; Wed, 07 Aug 2024 15:48:26 -0400 X-MC-Unique: yTy3uU3PM6-_PBJnK6JPHw-1 Received: by mail-oo1-f69.google.com with SMTP id 006d021491bc7-5d5cc01aee6so12866eaf.0 for ; Wed, 07 Aug 2024 12:48:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723060106; x=1723664906; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FAGFj0FHVAq5pVRnc576bB34mI1HflD3zt6t4lJRJgM=; b=kwqmW/qVfjXs+ZDoXDd01GrbMBKTu3MWyl0O2ZjPEYxRL3cd+uSMWKmnlkCO5AT+Dn msZJnexzXrjeZCKdGqLWA3Tf7ZumE4XugFz4+wnek8960eg0J94B4eXX7+5O++OenRHx GepMcmIDsN+rcrgBvsd7bUzGLdqm8fRskuDCfswCBLQ4Kv+/+7AIpmy7lZg+kuuVGRI2 3NrmcMMMTCMKPI/lKbC8z+3T4c+J5uZPstWskRMu5A0xpONflYRpiyNXUz4wk5NrVx9z axzzT/8gLTC9itPNNtun8NprgZ7cmggJf7u/kODyj2p4C8wp58x4pcRwOC10RnHHedhf qVQQ== X-Forwarded-Encrypted: i=1; AJvYcCXoARvwZ8UjuibatPNZ7np6U231lAoGxUu271lfM4vXFItsgYVMZb2+1YDUeorOarIfRkeks+C/2119x9E=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxlIHa/aGLdEbJ60XVgbsehkgddqXTF8x31LgApRJkOjoKcVUzY T0XXgZAKY6fnonlTstjRoGVuihfKm5ELhma2z3KE8latDIjROCwovq4n2VCr5VO9/3Dx5QLouPv cHeQWkIROHDcAYAGCsr7CnmgT+6wIxBLOxR/+HsA2K0AfNG2JUievK9449mxTK10= X-Received: by 2002:a4a:b90c:0:b0:5cd:920:de44 with SMTP id 006d021491bc7-5d6636caa6bmr12611330eaf.2.1723060105978; Wed, 07 Aug 2024 12:48:25 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE55jmIe/WZbCw3l3NTgyyno7WLqIqKlQdXAtW30hkxt69+d9woKPh92QMpZkY/maA+61+r6A== X-Received: by 2002:a4a:b90c:0:b0:5cd:920:de44 with SMTP id 006d021491bc7-5d6636caa6bmr12611307eaf.2.1723060105619; Wed, 07 Aug 2024 12:48:25 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6bb9c78ae4asm59853256d6.33.2024.08.07.12.48.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Aug 2024 12:48:24 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v4 5/7] mm/x86: arch_check_zapped_pud() Date: Wed, 7 Aug 2024 15:48:09 -0400 Message-ID: <20240807194812.819412-6-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240807194812.819412-1-peterx@redhat.com> References: <20240807194812.819412-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: James Houghton , David Hildenbrand , Dave Hansen , peterx@redhat.com, Christophe Leroy , Dan Williams , Dave Jiang , "Aneesh Kumar K . V" , x86@kernel.org, Hugh Dickins , Matthew Wilcox , Ingo Molnar , Huang Ying , Rik van Riel , Nicholas Piggin , Borislav Petkov , "Kirill A . Shutemov" , Thomas Gleixner , Vlastimil Babka , Oscar Salvador , linuxppc-dev@lists.ozlabs.org, Andrew Morton , Rick P Edgecombe , Mel Gorman Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Introduce arch_check_zapped_pud() to sanity check shadow stack on PUD zaps. It has the same logic of the PMD helper. One thing to mention is, it might be a good idea to use page_table_check in the future for trapping wrong setups of shadow stack pgtable entries [1]. That is left for the future as a separate effort. [1] https://lore.kernel.org/all/59d518698f664e07c036a5098833d7b56b953305.camel@intel.com Cc: "Edgecombe, Rick P" Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 10 ++++++++++ arch/x86/mm/pgtable.c | 7 +++++++ include/linux/pgtable.h | 7 +++++++ mm/huge_memory.c | 4 +++- 4 files changed, 27 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index a2a3bd4c1bda..fdb8ac9e7030 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -174,6 +174,13 @@ static inline int pud_young(pud_t pud) return pud_flags(pud) & _PAGE_ACCESSED; } +static inline bool pud_shstk(pud_t pud) +{ + return cpu_feature_enabled(X86_FEATURE_SHSTK) && + (pud_flags(pud) & (_PAGE_RW | _PAGE_DIRTY | _PAGE_PSE)) == + (_PAGE_DIRTY | _PAGE_PSE); +} + static inline int pte_write(pte_t pte) { /* @@ -1667,6 +1674,9 @@ void arch_check_zapped_pte(struct vm_area_struct *vma, pte_t pte); #define arch_check_zapped_pmd arch_check_zapped_pmd void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd); +#define arch_check_zapped_pud arch_check_zapped_pud +void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud); + #ifdef CONFIG_XEN_PV #define arch_has_hw_nonleaf_pmd_young arch_has_hw_nonleaf_pmd_young static inline bool arch_has_hw_nonleaf_pmd_young(void) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index f5931499c2d6..d4b3ccf90236 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -926,3 +926,10 @@ void arch_check_zapped_pmd(struct vm_area_struct *vma, pmd_t pmd) VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && pmd_shstk(pmd)); } + +void arch_check_zapped_pud(struct vm_area_struct *vma, pud_t pud) +{ + /* See note in arch_check_zapped_pte() */ + VM_WARN_ON_ONCE(!(vma->vm_flags & VM_SHADOW_STACK) && + pud_shstk(pud)); +} diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 2a6a3cccfc36..2289e9f7aa1b 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -447,6 +447,13 @@ static inline void arch_check_zapped_pmd(struct vm_area_struct *vma, } #endif +#ifndef arch_check_zapped_pud +static inline void arch_check_zapped_pud(struct vm_area_struct *vma, + pud_t pud) +{ +} +#endif + #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 0024266dea0a..81c5da0708ed 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2293,12 +2293,14 @@ int zap_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, pud_t *pud, unsigned long addr) { spinlock_t *ptl; + pud_t orig_pud; ptl = __pud_trans_huge_lock(pud, vma); if (!ptl) return 0; - pudp_huge_get_and_clear_full(vma, addr, pud, tlb->fullmm); + orig_pud = pudp_huge_get_and_clear_full(vma, addr, pud, tlb->fullmm); + arch_check_zapped_pud(vma, orig_pud); tlb_remove_pud_tlb_entry(tlb, pud, addr); if (vma_is_special_huge(vma)) { spin_unlock(ptl); From patchwork Wed Aug 7 19:48:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1970212 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Hf2mV4kZ; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Hf2mV4kZ; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WfLSN0gSlz1ydt for ; Thu, 8 Aug 2024 05:52:52 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Hf2mV4kZ; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Hf2mV4kZ; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WfLSK0Fn6z3dK5 for ; Thu, 8 Aug 2024 05:52:49 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Hf2mV4kZ; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Hf2mV4kZ; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=redhat.com (client-ip=170.10.133.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=peterx@redhat.com; receiver=lists.ozlabs.org) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WfLMR1MZrz3dJk for ; Thu, 8 Aug 2024 05:48:34 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723060112; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DDb5b0nOIqaqJ1HBArfkZKBp6ttQDN26GIh6owjfgNA=; b=Hf2mV4kZxRlLiQ9bZGWLp3IthTKPBoeyliaw1o4T7vFcVUzeLcxvZS08mku5EeOoWSLaJV 7V8kqcyu+EjD2i4D5m9IU2ybJ0TXa0/UYwoDE0KikKBVSFVhVhw1FQXtTE9LYvSu06U8cC jy9G2T+XrkHGIVg8l951eSSAalkISno= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723060112; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DDb5b0nOIqaqJ1HBArfkZKBp6ttQDN26GIh6owjfgNA=; b=Hf2mV4kZxRlLiQ9bZGWLp3IthTKPBoeyliaw1o4T7vFcVUzeLcxvZS08mku5EeOoWSLaJV 7V8kqcyu+EjD2i4D5m9IU2ybJ0TXa0/UYwoDE0KikKBVSFVhVhw1FQXtTE9LYvSu06U8cC jy9G2T+XrkHGIVg8l951eSSAalkISno= Received: from mail-oi1-f198.google.com (mail-oi1-f198.google.com [209.85.167.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-206-Cg_y7RqLO-K0TLWbm7QqiQ-1; Wed, 07 Aug 2024 15:48:29 -0400 X-MC-Unique: Cg_y7RqLO-K0TLWbm7QqiQ-1 Received: by mail-oi1-f198.google.com with SMTP id 5614622812f47-3dc2b094f1aso66534b6e.0 for ; Wed, 07 Aug 2024 12:48:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723060108; x=1723664908; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DDb5b0nOIqaqJ1HBArfkZKBp6ttQDN26GIh6owjfgNA=; b=L3pkoUOlrCcYhXAUMn2BSKy7aylo4lv2+v3iC3d3QqkxyqzEjE4ffGgBj7zeUhamrk pPxCJlIS/vpAvr7PHj4iLgRjJIV0GNJHH4myIICPUoDFqlHmPhs4/hYeBEK7oIUrD8xh LXgKIwVXeqpPHKlJWM1Yh8p+oZz3zrNRVXyINrRI8D915jiFCHI7Wgaz4zSvIWYu5w6/ fyPf9nvICkYsP/hEev0L70rcrXKLDYy6x+ufIS9g6mnHhTNTG/cNB/M8Z0XBPcClh14A YVR6ssOO2pyDYm2AeMpB6K4WLMPWthxnbV+nPAsNAm/MCgcjP3g6ajGnwFgtcjSpXsZy 9b4Q== X-Forwarded-Encrypted: i=1; AJvYcCWYXdwnxhSCXyzTq47p0Rfw1fYolVq6BEij/Xd1TqXIKjOIMvGvxkA1sZX8r+MDryFVRCK9XhzngAWtIZKrgz26zy3Y/vzA1kyZv500WA== X-Gm-Message-State: AOJu0YxiTDQx1icCPVeHLCNkDAxzQJRgQBFNHLFfvg3W1+I2Rf7Mp3vC Wp7mxQAMc2OMVW44m3bc/6kRbSjTUg/+9vL/3VRdX66dYWtX1qbujjdzk5uU+IhQfOQUMof772C 5RYhWv+/vZ37K2UQe2yhIan36bOKpEC/QbI2Sz0h5aPY0hiT1n4ym62g570AB3BU= X-Received: by 2002:a05:6358:e49f:b0:1ac:f6e3:dbcd with SMTP id e5c5f4694b2df-1af3babf83fmr1244372055d.3.1723060108244; Wed, 07 Aug 2024 12:48:28 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGF6PD4SDnTvqRZFf7QOZAHAr54Efqd3zzz8coMqAlruM+AKE9ZSLZCO533yzyaTsEnxMNyzw== X-Received: by 2002:a05:6358:e49f:b0:1ac:f6e3:dbcd with SMTP id e5c5f4694b2df-1af3babf83fmr1244369055d.3.1723060107772; Wed, 07 Aug 2024 12:48:27 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6bb9c78ae4asm59853256d6.33.2024.08.07.12.48.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Aug 2024 12:48:27 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v4 6/7] mm/x86: Add missing pud helpers Date: Wed, 7 Aug 2024 15:48:10 -0400 Message-ID: <20240807194812.819412-7-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240807194812.819412-1-peterx@redhat.com> References: <20240807194812.819412-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: James Houghton , David Hildenbrand , Dave Hansen , peterx@redhat.com, Christophe Leroy , Dan Williams , Dave Jiang , "Aneesh Kumar K . V" , x86@kernel.org, Hugh Dickins , Matthew Wilcox , Ingo Molnar , Huang Ying , Rik van Riel , Nicholas Piggin , Borislav Petkov , "Kirill A . Shutemov" , Thomas Gleixner , Vlastimil Babka , Oscar Salvador , linuxppc-dev@lists.ozlabs.org, Andrew Morton , Rick P Edgecombe , Mel Gorman Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" These new helpers will be needed for pud entry updates soon. Introduce these helpers by referencing the pmd ones. Namely: - pudp_invalidate() - pud_modify() Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 55 +++++++++++++++++++++++++++++----- arch/x86/mm/pgtable.c | 12 ++++++++ 2 files changed, 59 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index fdb8ac9e7030..a7c1e9cfea41 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -787,6 +787,12 @@ static inline pmd_t pmd_mkinvalid(pmd_t pmd) __pgprot(pmd_flags(pmd) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); } +static inline pud_t pud_mkinvalid(pud_t pud) +{ + return pfn_pud(pud_pfn(pud), + __pgprot(pud_flags(pud) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); +} + static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask); static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) @@ -834,14 +840,8 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) pmd_result = __pmd(val); /* - * To avoid creating Write=0,Dirty=1 PMDs, pte_modify() needs to avoid: - * 1. Marking Write=0 PMDs Dirty=1 - * 2. Marking Dirty=1 PMDs Write=0 - * - * The first case cannot happen because the _PAGE_CHG_MASK will filter - * out any Dirty bit passed in newprot. Handle the second case by - * going through the mksaveddirty exercise. Only do this if the old - * value was Write=1 to avoid doing this on Shadow Stack PTEs. + * Avoid creating shadow stack PMD by accident. See comment in + * pte_modify(). */ if (oldval & _PAGE_RW) pmd_result = pmd_mksaveddirty(pmd_result); @@ -851,6 +851,29 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) return pmd_result; } +static inline pud_t pud_modify(pud_t pud, pgprot_t newprot) +{ + pudval_t val = pud_val(pud), oldval = val; + pud_t pud_result; + + val &= _HPAGE_CHG_MASK; + val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val = flip_protnone_guard(oldval, val, PHYSICAL_PUD_PAGE_MASK); + + pud_result = __pud(val); + + /* + * Avoid creating shadow stack PUD by accident. See comment in + * pte_modify(). + */ + if (oldval & _PAGE_RW) + pud_result = pud_mksaveddirty(pud_result); + else + pud_result = pud_clear_saveddirty(pud_result); + + return pud_result; +} + /* * mprotect needs to preserve PAT and encryption bits when updating * vm_page_prot @@ -1389,10 +1412,26 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, } #endif +static inline pud_t pudp_establish(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp, pud_t pud) +{ + page_table_check_pud_set(vma->vm_mm, pudp, pud); + if (IS_ENABLED(CONFIG_SMP)) { + return xchg(pudp, pud); + } else { + pud_t old = *pudp; + WRITE_ONCE(*pudp, pud); + return old; + } +} + #define __HAVE_ARCH_PMDP_INVALIDATE_AD extern pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); + /* * Page table pages are page-aligned. The lower half of the top * level is used for userspace and the top half for the kernel. diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index d4b3ccf90236..9fc2dabf8427 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -641,6 +641,18 @@ pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, } #endif +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ + defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + VM_WARN_ON_ONCE(!pud_present(*pudp)); + pud_t old = pudp_establish(vma, address, pudp, pud_mkinvalid(*pudp)); + flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE); + return old; +} +#endif + /** * reserve_top_address - reserves a hole in the top of kernel address space * @reserve - size of hole to reserve From patchwork Wed Aug 7 19:48:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1970213 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ItOjf3t6; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=MBADa7Hs; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WfLT42r5tz1ydt for ; Thu, 8 Aug 2024 05:53:28 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ItOjf3t6; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=MBADa7Hs; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WfLT41pldz2y8W for ; Thu, 8 Aug 2024 05:53:28 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ItOjf3t6; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=MBADa7Hs; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=redhat.com (client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=peterx@redhat.com; receiver=lists.ozlabs.org) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WfLMS0dZjz3dHV for ; Thu, 8 Aug 2024 05:48:35 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723060112; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A/6PEtzQTa+drbYKkEnodTZuUMYm4tSA4vStMaJG0Ro=; b=ItOjf3t6DCj11voM/a7dWGDp0KOMK9De9u6szfhL9mdDakEyB8/wGEUO1gmPFx4YTMWZ2o Alib2V7drqOYM3p+TOSL41kv2EJA3jrgTlsE5Cx4yD26WxCnO6W0HzDmUCQ7QfFoTsmpw7 Fxh3+c+zUx0lRvB3BWHx46aT7weqoWw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723060113; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A/6PEtzQTa+drbYKkEnodTZuUMYm4tSA4vStMaJG0Ro=; b=MBADa7HsXwZ4dezjd7aQnCYzHlTnZxbcLiJFPNGb5rkQHEOPHyey3pi2KIzFNbCqkRnDT8 PPi9D5qws50yr5qK0F/0m/wOVdna3V1wr0n2HccpfCmf3Vl04P51SCSA46c/Jd0RN8UNC1 +QTx7w+8FFJA/oA0uKWMzvGabm/qAJc= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-577-NVDkJUPoPMy8X0uC4Yv8gA-1; Wed, 07 Aug 2024 15:48:31 -0400 X-MC-Unique: NVDkJUPoPMy8X0uC4Yv8gA-1 Received: by mail-qv1-f69.google.com with SMTP id 6a1803df08f44-6b7ad98c1f8so389786d6.1 for ; Wed, 07 Aug 2024 12:48:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723060111; x=1723664911; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A/6PEtzQTa+drbYKkEnodTZuUMYm4tSA4vStMaJG0Ro=; b=dDg6C3Ph7G9Iw1BoOwPdysDq48i4ltDMWaiDs+0YY6M1+W3MutlEuNxQO2r8YWUuT7 Nrf3jvKih8rVuPeE5LOVtRSXMTAz5Z6iAiHYX65lXsEm/FLadt2pYmu7PoNGiVYhhHdV hZvtN250sg2z8dJLpJxeCivVZMGAMQBOvDuKIWcN89HQ3lJQHlggrVx32LnhZeRNqnMx 5VACBsy24fz04G5KXrLDZGxV6G8AvW+B/nAklBs7vvsrT9+ZP0pp3mQWgtHf+nf4srxN APpoz/p8rEeTt7uKsP4sRSkNxyJiXyZsOB/AR0dtP9PnHrSW4b8AYE5Q0+OOya1Urdk3 WBxA== X-Forwarded-Encrypted: i=1; AJvYcCUdbssA74cxw1LzRliqISyjPJc6Dt+m0PxbQ93yXzoJSvSXuIHhgOKy1thrKvmTBcwroAw40uJl/U7z0iA=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yzz7FwljABn1CYvfEbSvImEteCluUnNfgYxt6RWC66ozQGv/7E6 1t2sU5i8CKTDtjAc4CPYOggk6D/1VQ+SB9kUOtBCukoO1PoZOuahnVkT44hwJoCcievUdexR2X4 y4g6UNl9FqC4MbG56QTdFGGbMqixk0A0PufoslfdS7dAPjjbonTCbf4/kqFO7B6g= X-Received: by 2002:a05:6214:e64:b0:6b0:8202:5c4e with SMTP id 6a1803df08f44-6bb983f0fa6mr145729316d6.5.1723060110597; Wed, 07 Aug 2024 12:48:30 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFjs/meRFYsBYhQNSwMvbRLZa9/TlQmQfQAUysM61b8jRz23PUki/2rojCFyrJzg3cPY3hNlw== X-Received: by 2002:a05:6214:e64:b0:6b0:8202:5c4e with SMTP id 6a1803df08f44-6bb983f0fa6mr145728896d6.5.1723060110042; Wed, 07 Aug 2024 12:48:30 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6bb9c78ae4asm59853256d6.33.2024.08.07.12.48.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Aug 2024 12:48:29 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v4 7/7] mm/mprotect: fix dax pud handlings Date: Wed, 7 Aug 2024 15:48:11 -0400 Message-ID: <20240807194812.819412-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240807194812.819412-1-peterx@redhat.com> References: <20240807194812.819412-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: James Houghton , David Hildenbrand , Dave Hansen , peterx@redhat.com, Christophe Leroy , Dan Williams , Dave Jiang , "Aneesh Kumar K . V" , x86@kernel.org, Hugh Dickins , Matthew Wilcox , Ingo Molnar , Huang Ying , Rik van Riel , Nicholas Piggin , Borislav Petkov , "Kirill A . Shutemov" , Thomas Gleixner , Vlastimil Babka , Oscar Salvador , linuxppc-dev@lists.ozlabs.org, Andrew Morton , Rick P Edgecombe , Mel Gorman Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This is only relevant to the two archs that support PUD dax, aka, x86_64 and ppc64. PUD THPs do not yet exist elsewhere, and hugetlb PUDs do not count in this case. DAX have had PUD mappings for years, but change protection path never worked. When the path is triggered in any form (a simple test program would be: call mprotect() on a 1G dev_dax mapping), the kernel will report "bad pud". This patch should fix that. The new change_huge_pud() tries to keep everything simple. For example, it doesn't optimize write bit as that will need even more PUD helpers. It's not too bad anyway to have one more write fault in the worst case once for 1G range; may be a bigger thing for each PAGE_SIZE, though. Neither does it support userfault-wp bits, as there isn't such PUD mappings that is supported; file mappings always need a split there. The same to TLB shootdown: the pmd path (which was for x86 only) has the trick of using _ad() version of pmdp_invalidate*() which can avoid one redundant TLB, but let's also leave that for later. Again, the larger the mapping, the smaller of such effect. There's some difference on handling "retry" for change_huge_pud() (where it can return 0): it isn't like change_huge_pmd(), as the pmd version is safe with all conditions handled in change_pte_range() later, thanks to Hugh's new pte_offset_map_lock(). In short, change_pte_range() is simply smarter. For that, change_pud_range() will need proper retry if it races with something else when a huge PUD changed from under us. The last thing to mention is currently the PUD path ignores the huge pte numa counter (NUMA_HUGE_PTE_UPDATES), not only because DAX is not applicable to NUMA, but also that it's ambiguous on its own to decide how to account pud in this case. In one earlier version of this patchset I proposed to remove the counter as it doesn't even look right to do the accounting as of now [1], but then a further discussion suggests we can leave that for later, as that doesn't block this series if we choose to ignore that counter. That's what this patch does, by ignoring it. When at it, touch up the comment in pgtable_split_needed() to make it generic to either pmd or pud file THPs. [1] https://lore.kernel.org/all/20240715192142.3241557-3-peterx@redhat.com/ [2] https://lore.kernel.org/r/added2d0-b8be-4108-82ca-1367a388d0b1@redhat.com Cc: Dan Williams Cc: Matthew Wilcox Cc: Dave Jiang Cc: Hugh Dickins Cc: Kirill A. Shutemov Cc: Vlastimil Babka Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Michael Ellerman Cc: Aneesh Kumar K.V Cc: Oscar Salvador Cc: x86@kernel.org Cc: linuxppc-dev@lists.ozlabs.org Fixes: a00cc7d9dd93 ("mm, x86: add support for PUD-sized transparent hugepages") Fixes: 27af67f35631 ("powerpc/book3s64/mm: enable transparent pud hugepage") Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 24 +++++++++++++++++++ mm/huge_memory.c | 52 +++++++++++++++++++++++++++++++++++++++++ mm/mprotect.c | 39 ++++++++++++++++++++++++------- 3 files changed, 107 insertions(+), 8 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index ce44caa40eed..6370026689e0 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -342,6 +342,17 @@ void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, unsigned long address); +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags); +#else +static inline int +change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) { return 0; } +#endif + #define split_huge_pud(__vma, __pud, __address) \ do { \ pud_t *____pud = (__pud); \ @@ -585,6 +596,19 @@ static inline int next_order(unsigned long *orders, int prev) { return 0; } + +static inline void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, + unsigned long address) +{ +} + +static inline int change_huge_pud(struct mmu_gather *tlb, + struct vm_area_struct *vma, pud_t *pudp, + unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) +{ + return 0; +} #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ static inline int split_folio_to_list_to_order(struct folio *folio, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 81c5da0708ed..0aafd26d7a53 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2114,6 +2114,53 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, return ret; } +/* + * Returns: + * + * - 0: if pud leaf changed from under us + * - 1: if pud can be skipped + * - HPAGE_PUD_NR: if pud was successfully processed + */ +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) +{ + struct mm_struct *mm = vma->vm_mm; + pud_t oldpud, entry; + spinlock_t *ptl; + + tlb_change_page_size(tlb, HPAGE_PUD_SIZE); + + /* NUMA balancing doesn't apply to dax */ + if (cp_flags & MM_CP_PROT_NUMA) + return 1; + + /* + * Huge entries on userfault-wp only works with anonymous, while we + * don't have anonymous PUDs yet. + */ + if (WARN_ON_ONCE(cp_flags & MM_CP_UFFD_WP_ALL)) + return 1; + + ptl = __pud_trans_huge_lock(pudp, vma); + if (!ptl) + return 0; + + /* + * Can't clear PUD or it can race with concurrent zapping. See + * change_huge_pmd(). + */ + oldpud = pudp_invalidate(vma, addr, pudp); + entry = pud_modify(oldpud, newprot); + set_pud_at(mm, addr, pudp, entry); + tlb_flush_pud_range(tlb, addr, HPAGE_PUD_SIZE); + + spin_unlock(ptl); + return HPAGE_PUD_NR; +} +#endif + #ifdef CONFIG_USERFAULTFD /* * The PT lock for src_pmd and dst_vma/src_vma (for reading) are locked by @@ -2344,6 +2391,11 @@ void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, spin_unlock(ptl); mmu_notifier_invalidate_range_end(&range); } +#else +void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, + unsigned long address) +{ +} #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma, diff --git a/mm/mprotect.c b/mm/mprotect.c index d423080e6509..446f8e5f10d9 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -302,8 +302,9 @@ pgtable_split_needed(struct vm_area_struct *vma, unsigned long cp_flags) { /* * pte markers only resides in pte level, if we need pte markers, - * we need to split. We cannot wr-protect shmem thp because file - * thp is handled differently when split by erasing the pmd so far. + * we need to split. For example, we cannot wr-protect a file thp + * (e.g. 2M shmem) because file thp is handled differently when + * split by erasing the pmd so far. */ return (cp_flags & MM_CP_UFFD_WP) && !vma_is_anonymous(vma); } @@ -430,31 +431,53 @@ static inline long change_pud_range(struct mmu_gather *tlb, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { struct mmu_notifier_range range; - pud_t *pud; + pud_t *pudp, pud; unsigned long next; long pages = 0, ret; range.start = 0; - pud = pud_offset(p4d, addr); + pudp = pud_offset(p4d, addr); do { +again: next = pud_addr_end(addr, end); - ret = change_prepare(vma, pud, pmd, addr, cp_flags); + ret = change_prepare(vma, pudp, pmd, addr, cp_flags); if (ret) { pages = ret; break; } - if (pud_none_or_clear_bad(pud)) + + pud = READ_ONCE(*pudp); + if (pud_none(pud)) continue; + if (!range.start) { mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0, vma->vm_mm, addr, end); mmu_notifier_invalidate_range_start(&range); } - pages += change_pmd_range(tlb, vma, pud, addr, next, newprot, + + if (pud_leaf(pud)) { + if ((next - addr != PUD_SIZE) || + pgtable_split_needed(vma, cp_flags)) { + __split_huge_pud(vma, pudp, addr); + goto again; + } else { + ret = change_huge_pud(tlb, vma, pudp, + addr, newprot, cp_flags); + if (ret == 0) + goto again; + /* huge pud was handled */ + if (ret == HPAGE_PUD_NR) + pages += HPAGE_PUD_NR; + continue; + } + } + + pages += change_pmd_range(tlb, vma, pudp, addr, next, newprot, cp_flags); - } while (pud++, addr = next, addr != end); + } while (pudp++, addr = next, addr != end); if (range.start) mmu_notifier_invalidate_range_end(&range);