| Message ID | cover.1724173828.git.christophe.leroy@csgroup.eu (mailing list archive) |
|---|---|
| Headers | show |
| Series | Reduce alignment constraint on STRICT_KERNEL_RWX and speed-up TLB misses on 8xx and 603 | expand |
On Tue, 20 Aug 2024 19:23:44 +0200, Christophe Leroy wrote: > This series does mainly two things: > - Remove the 8M alignment constraint on STRICT_KERNEL_RWX on 8xx to > avoid wasting memory. > - Speed-up TLB misses by duplicating kernel PGD entries into each > task's PGDIRs to avoid the address comparison in TLB miss handler. > > On 8xx, the address comparison takes a significant part of CPU cycles > as it requires saving/restoring CR, and because a taken branch > requires 2 cycles. > On 603 it is less significant because CR is saved automatically and > has to be restored anyway but it is still worth it. > > [...] Applied to powerpc/next. [01/14] powerpc/8xx: Fix initial memory mapping https://git.kernel.org/powerpc/c/f9f2bff64c2f0dbee57be3d8c2741357ad3d05e6 [02/14] powerpc/8xx: Fix kernel vs user address comparison https://git.kernel.org/powerpc/c/65a82e117ffeeab0baf6f871a1cab11a28ace183 [03/14] powerpc/8xx: Copy kernel PGD entries into all PGDIRs https://git.kernel.org/powerpc/c/985db026c34dfc45213649023d5505822a5dcd78 [04/14] Revert "powerpc/8xx: Always pin kernel text TLB" https://git.kernel.org/powerpc/c/1a736d98c84acd38e40fff69528ce7aaa55dd22d [05/14] powerpc/8xx: Allow setting DATA alignment even with STRICT_KERNEL_RWX https://git.kernel.org/powerpc/c/bcf77a70c4ffc9b01044229de87f5b6f9c1f7913 [06/14] powerpc/8xx: Reduce default size of module/execmem area https://git.kernel.org/powerpc/c/c5eec4df25c34f4bee8c757ed157f5d96eaba554 [07/14] powerpc/8xx: Preallocate execmem page tables https://git.kernel.org/powerpc/c/16a71c045186a11c1c743934e330de78162b86dd [08/14] powerpc/8xx: Inconditionally use task PGDIR in ITLB misses https://git.kernel.org/powerpc/c/33c527522f394f63cc589a6f7af990b2232444c8 [09/14] powerpc/8xx: Inconditionally use task PGDIR in DTLB misses https://git.kernel.org/powerpc/c/ac9f97ff8b324905d457f2694490c63b9deccbc6 [10/14] powerpc/32s: Reduce default size of module/execmem area https://git.kernel.org/powerpc/c/2f2b9a3adc66e978a1248ffb38df8477e8e97c57 [11/14] powerpc/603: Copy kernel PGD entries into all PGDIRs and preallocate execmem page tables https://git.kernel.org/powerpc/c/82ef440f9a38a1fd7f4854397633a35af33840a5 [12/14] powerpc/603: Switch r0 and r3 in TLB miss handlers https://git.kernel.org/powerpc/c/31c0e137ec609f36877ea39cd343ef2476d080aa [13/14] powerpc/603: Inconditionally use task PGDIR in ITLB misses https://git.kernel.org/powerpc/c/3f57d90c231d3329aaed7079dd05b5a2f7692a58 [14/14] powerpc/603: Inconditionally use task PGDIR in DTLB misses https://git.kernel.org/powerpc/c/062e825a336017c0334c7497690826c95aa1a84f cheers
This series does mainly two things: - Remove the 8M alignment constraint on STRICT_KERNEL_RWX on 8xx to avoid wasting memory. - Speed-up TLB misses by duplicating kernel PGD entries into each task's PGDIRs to avoid the address comparison in TLB miss handler. On 8xx, the address comparison takes a significant part of CPU cycles as it requires saving/restoring CR, and because a taken branch requires 2 cycles. On 603 it is less significant because CR is saved automatically and has to be restored anyway but it is still worth it. For ITLB misses: - Kernel PGD entries are setup for once during init, before creation of new PGDIRs. - Module PGD entries are setup also at init by preallocating page tables for a very few number of pages For DTLB misses: - Some handling is added in 8xx DATA TLB error interrupt and in 603 DATA read and store TLB miss interrupts to copy missing PGD entries into child. The cost of that additional handling on error paths is worth the gain on hot TLB miss pathes. Because 8xx and 603 don't use leaf kernel pages at PGD level, there is no need to care about PGD entries cleanup, page tables are never freed. Christophe Leroy (14): powerpc/8xx: Fix initial memory mapping powerpc/8xx: Fix kernel vs user address comparison powerpc/8xx: Copy kernel PGD entries into all PGDIRs Revert "powerpc/8xx: Always pin kernel text TLB" powerpc/8xx: Allow setting DATA alignment even with STRICT_KERNEL_RWX powerpc/8xx: Reduce default size of module/execmem area powerpc/8xx: Preallocate execmem page tables powerpc/8xx: Inconditionally use task PGDIR in ITLB misses powerpc/8xx: Inconditionally use task PGDIR in DTLB misses powerpc/32s: Reduce default size of module/execmem area powerpc/603: Copy kernel PGD entries into all PGDIRs and preallocate execmem page tables powerpc/603: Switch r0 and r3 in TLB miss handlers powerpc/603: Inconditionally use task PGDIR in ITLB misses powerpc/603: Inconditionally use task PGDIR in DTLB misses arch/powerpc/Kconfig | 31 +++- arch/powerpc/include/asm/book3s/32/pgtable.h | 3 +- arch/powerpc/include/asm/nohash/32/mmu-8xx.h | 3 +- arch/powerpc/include/asm/nohash/pgalloc.h | 8 +- arch/powerpc/kernel/head_8xx.S | 78 +++++----- arch/powerpc/kernel/head_book3s_32.S | 144 +++++++++---------- arch/powerpc/mm/book3s32/mmu.c | 2 + arch/powerpc/mm/mem.c | 14 ++ arch/powerpc/mm/nohash/8xx.c | 9 +- arch/powerpc/platforms/8xx/Kconfig | 7 + 10 files changed, 173 insertions(+), 126 deletions(-)