| Message ID | 20240129124649.189745-1-david@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | mm/memory: optimize fork() with PTE-mapped THP | expand |
On 29/01/2024 12:46, David Hildenbrand wrote: > Now that the rmap overhaul[1] is upstream that provides a clean interface > for rmap batching, let's implement PTE batching during fork when processing > PTE-mapped THPs. > > This series is partially based on Ryan's previous work[2] to implement > cont-pte support on arm64, but its a complete rewrite based on [1] to > optimize all architectures independent of any such PTE bits, and to > use the new rmap batching functions that simplify the code and prepare > for further rmap accounting changes. > > We collect consecutive PTEs that map consecutive pages of the same large > folio, making sure that the other PTE bits are compatible, and (a) adjust > the refcount only once per batch, (b) call rmap handling functions only > once per batch and (c) perform batch PTE setting/updates. > > While this series should be beneficial for adding cont-pte support on > ARM64[2], it's one of the requirements for maintaining a total mapcount[3] > for large folios with minimal added overhead and further changes[4] that > build up on top of the total mapcount. > > Independent of all that, this series results in a speedup during fork with > PTE-mapped THP, which is the default with THPs that are smaller than a PMD > (for example, 16KiB to 1024KiB mTHPs for anonymous memory[5]). > > On an Intel Xeon Silver 4210R CPU, fork'ing with 1GiB of PTE-mapped folios > of the same size (stddev < 1%) results in the following runtimes > for fork() (shorter is better): > > Folio Size | v6.8-rc1 | New | Change > ------------------------------------------ > 4KiB | 0.014328 | 0.014035 | - 2% > 16KiB | 0.014263 | 0.01196 | -16% > 32KiB | 0.014334 | 0.01094 | -24% > 64KiB | 0.014046 | 0.010444 | -26% > 128KiB | 0.014011 | 0.010063 | -28% > 256KiB | 0.013993 | 0.009938 | -29% > 512KiB | 0.013983 | 0.00985 | -30% > 1024KiB | 0.013986 | 0.00982 | -30% > 2048KiB | 0.014305 | 0.010076 | -30% Just a heads up that I'm seeing some strange results on Apple M2. Fork for order-0 is seemingly costing ~17% more. I'm using GCC 13.2 and was pretty sure I didn't see this problem with version 1; although that was on a different baseline and I've thrown the numbers away so will rerun and try to debug this. | kernel | mean_rel | std_rel | |:------------|-----------:|----------:| | mm-unstable | 0.0% | 1.1% | | patch 1 | -2.3% | 1.3% | | patch 10 | -2.9% | 2.7% | | patch 11 | 13.5% | 0.5% | | patch 12 | 15.2% | 1.2% | | patch 13 | 18.2% | 0.7% | | patch 14 | 20.5% | 1.0% | | patch 15 | 17.1% | 1.6% | | patch 15 | 16.7% | 0.8% | fork for order-9 is looking good (-20%), and for the zap series, munmap is looking good, but dontneed is looking poor for both order-0 and 9. But one thing at a time... let's concentrate on fork order-0 first. Note that I'm still using the "old" benchmark code. Could you resend me the link to the new code? Although I don't think there should be any effect for order-0 anyway, if I understood your changes correctly? > > Note that these numbers are even better than the ones from v1 (verified > over multiple reboots), even though there were only minimal code changes. > Well, I removed a pte_mkclean() call for anon folios, maybe that also > plays a role. > > But my experience is that fork() is extremely sensitive to code size, > inlining, ... so I suspect we'll see on other architectures rather a change > of -20% instead of -30%, and it will be easy to "lose" some of that speedup > in the future by subtle code changes. > > Next up is PTE batching when unmapping. Only tested on x86-64. > Compile-tested on most other architectures. > > v2 -> v3: > * Rebased on mm-unstable > * Picked up RB's > * Updated documentation of wrprotect_ptes(). > > v1 -> v2: > * "arm64/mm: Make set_ptes() robust when OAs cross 48-bit boundary" > -> Added patch from Ryan > * "arm/pgtable: define PFN_PTE_SHIFT" > -> Removed the arm64 bits > * "mm/pgtable: make pte_next_pfn() independent of set_ptes()" > * "arm/mm: use pte_next_pfn() in set_ptes()" > * "powerpc/mm: use pte_next_pfn() in set_ptes()" > -> Added to use pte_next_pfn() in some arch set_ptes() implementations > I tried to make use of pte_next_pfn() also in the others, but it's > not trivial because the other archs implement set_ptes() in their > asm/pgtable.h. Future work. > * "mm/memory: factor out copying the actual PTE in copy_present_pte()" > -> Move common folio_get() out of if/else > * "mm/memory: optimize fork() with PTE-mapped THP" > -> Add doc for wrprotect_ptes > -> Extend description to mention handling of pinned folios > -> Move common folio_ref_add() out of if/else > * "mm/memory: ignore dirty/accessed/soft-dirty bits in folio_pte_batch()" > -> Be more conservative with dirt/soft-dirty, let the caller specify > using flags > > [1] https://lkml.kernel.org/r/20231220224504.646757-1-david@redhat.com > [2] https://lkml.kernel.org/r/20231218105100.172635-1-ryan.roberts@arm.com > [3] https://lkml.kernel.org/r/20230809083256.699513-1-david@redhat.com > [4] https://lkml.kernel.org/r/20231124132626.235350-1-david@redhat.com > [5] https://lkml.kernel.org/r/20231207161211.2374093-1-ryan.roberts@arm.com > > Cc: Andrew Morton <akpm@linux-foundation.org> > Cc: Matthew Wilcox (Oracle) <willy@infradead.org> > Cc: Ryan Roberts <ryan.roberts@arm.com> > Cc: Russell King <linux@armlinux.org.uk> > Cc: Catalin Marinas <catalin.marinas@arm.com> > Cc: Will Deacon <will@kernel.org> > Cc: Dinh Nguyen <dinguyen@kernel.org> > Cc: Michael Ellerman <mpe@ellerman.id.au> > Cc: Nicholas Piggin <npiggin@gmail.com> > Cc: Christophe Leroy <christophe.leroy@csgroup.eu> > Cc: "Aneesh Kumar K.V" <aneesh.kumar@kernel.org> > Cc: "Naveen N. Rao" <naveen.n.rao@linux.ibm.com> > Cc: Paul Walmsley <paul.walmsley@sifive.com> > Cc: Palmer Dabbelt <palmer@dabbelt.com> > Cc: Albert Ou <aou@eecs.berkeley.edu> > Cc: Alexander Gordeev <agordeev@linux.ibm.com> > Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> > Cc: Heiko Carstens <hca@linux.ibm.com> > Cc: Vasily Gorbik <gor@linux.ibm.com> > Cc: Christian Borntraeger <borntraeger@linux.ibm.com> > Cc: Sven Schnelle <svens@linux.ibm.com> > Cc: "David S. Miller" <davem@davemloft.net> > Cc: linux-arm-kernel@lists.infradead.org > Cc: linuxppc-dev@lists.ozlabs.org > Cc: linux-riscv@lists.infradead.org > Cc: linux-s390@vger.kernel.org > Cc: sparclinux@vger.kernel.org > > --- > > Andrew asked for a resend based on latest mm-unstable. I am sending this > out earlier than I would usually have sent out the next version, so we can > pull it into mm-unstable again now that v1 was dropped. > > David Hildenbrand (14): > arm/pgtable: define PFN_PTE_SHIFT > nios2/pgtable: define PFN_PTE_SHIFT > powerpc/pgtable: define PFN_PTE_SHIFT > riscv/pgtable: define PFN_PTE_SHIFT > s390/pgtable: define PFN_PTE_SHIFT > sparc/pgtable: define PFN_PTE_SHIFT > mm/pgtable: make pte_next_pfn() independent of set_ptes() > arm/mm: use pte_next_pfn() in set_ptes() > powerpc/mm: use pte_next_pfn() in set_ptes() > mm/memory: factor out copying the actual PTE in copy_present_pte() > mm/memory: pass PTE to copy_present_pte() > mm/memory: optimize fork() with PTE-mapped THP > mm/memory: ignore dirty/accessed/soft-dirty bits in folio_pte_batch() > mm/memory: ignore writable bit in folio_pte_batch() > > Ryan Roberts (1): > arm64/mm: Make set_ptes() robust when OAs cross 48-bit boundary > > arch/arm/include/asm/pgtable.h | 2 + > arch/arm/mm/mmu.c | 2 +- > arch/arm64/include/asm/pgtable.h | 28 ++-- > arch/nios2/include/asm/pgtable.h | 2 + > arch/powerpc/include/asm/pgtable.h | 2 + > arch/powerpc/mm/pgtable.c | 5 +- > arch/riscv/include/asm/pgtable.h | 2 + > arch/s390/include/asm/pgtable.h | 2 + > arch/sparc/include/asm/pgtable_64.h | 2 + > include/linux/pgtable.h | 33 ++++- > mm/memory.c | 212 ++++++++++++++++++++++------ > 11 files changed, 229 insertions(+), 63 deletions(-) > > > base-commit: d162e170f1181b4305494843e1976584ddf2b72e
On 31.01.24 11:43, Ryan Roberts wrote: > On 29/01/2024 12:46, David Hildenbrand wrote: >> Now that the rmap overhaul[1] is upstream that provides a clean interface >> for rmap batching, let's implement PTE batching during fork when processing >> PTE-mapped THPs. >> >> This series is partially based on Ryan's previous work[2] to implement >> cont-pte support on arm64, but its a complete rewrite based on [1] to >> optimize all architectures independent of any such PTE bits, and to >> use the new rmap batching functions that simplify the code and prepare >> for further rmap accounting changes. >> >> We collect consecutive PTEs that map consecutive pages of the same large >> folio, making sure that the other PTE bits are compatible, and (a) adjust >> the refcount only once per batch, (b) call rmap handling functions only >> once per batch and (c) perform batch PTE setting/updates. >> >> While this series should be beneficial for adding cont-pte support on >> ARM64[2], it's one of the requirements for maintaining a total mapcount[3] >> for large folios with minimal added overhead and further changes[4] that >> build up on top of the total mapcount. >> >> Independent of all that, this series results in a speedup during fork with >> PTE-mapped THP, which is the default with THPs that are smaller than a PMD >> (for example, 16KiB to 1024KiB mTHPs for anonymous memory[5]). >> >> On an Intel Xeon Silver 4210R CPU, fork'ing with 1GiB of PTE-mapped folios >> of the same size (stddev < 1%) results in the following runtimes >> for fork() (shorter is better): >> >> Folio Size | v6.8-rc1 | New | Change >> ------------------------------------------ >> 4KiB | 0.014328 | 0.014035 | - 2% >> 16KiB | 0.014263 | 0.01196 | -16% >> 32KiB | 0.014334 | 0.01094 | -24% >> 64KiB | 0.014046 | 0.010444 | -26% >> 128KiB | 0.014011 | 0.010063 | -28% >> 256KiB | 0.013993 | 0.009938 | -29% >> 512KiB | 0.013983 | 0.00985 | -30% >> 1024KiB | 0.013986 | 0.00982 | -30% >> 2048KiB | 0.014305 | 0.010076 | -30% > > Just a heads up that I'm seeing some strange results on Apple M2. Fork for > order-0 is seemingly costing ~17% more. I'm using GCC 13.2 and was pretty sure I > didn't see this problem with version 1; although that was on a different > baseline and I've thrown the numbers away so will rerun and try to debug this. > So far, on my x86 tests (Intel, AMD EPYC), I was not able to observe this. fork() for order-0 was consistently effectively unchanged. Do you observe that on other ARM systems as well? > | kernel | mean_rel | std_rel | > |:------------|-----------:|----------:| > | mm-unstable | 0.0% | 1.1% | > | patch 1 | -2.3% | 1.3% | > | patch 10 | -2.9% | 2.7% | > | patch 11 | 13.5% | 0.5% | > | patch 12 | 15.2% | 1.2% | > | patch 13 | 18.2% | 0.7% | > | patch 14 | 20.5% | 1.0% | > | patch 15 | 17.1% | 1.6% | > | patch 15 | 16.7% | 0.8% | > > fork for order-9 is looking good (-20%), and for the zap series, munmap is > looking good, but dontneed is looking poor for both order-0 and 9. But one thing > at a time... let's concentrate on fork order-0 first. munmap and dontneed end up calling the exact same call paths. So a big performance difference is rather surprising and might indicate something else. (I think I told you that I was running in some kind of VMA merging problem where one would suddenly get with my benchmark 1 VMA per page. The new benchmark below works around that, but I am not sure if that was fixed in the meantime) VMA merging can of course explain a big difference in fork and munmap vs. dontneed times, especially when comparing different code base where that VMA merging behavior was different. > > Note that I'm still using the "old" benchmark code. Could you resend me the link > to the new code? Although I don't think there should be any effect for order-0 > anyway, if I understood your changes correctly? This is the combined one (small and large PTEs): https://gitlab.com/davidhildenbrand/scratchspace/-/raw/main/pte-mapped-folio-benchmarks.c?inline=false
On 31/01/2024 11:06, David Hildenbrand wrote: > On 31.01.24 11:43, Ryan Roberts wrote: >> On 29/01/2024 12:46, David Hildenbrand wrote: >>> Now that the rmap overhaul[1] is upstream that provides a clean interface >>> for rmap batching, let's implement PTE batching during fork when processing >>> PTE-mapped THPs. >>> >>> This series is partially based on Ryan's previous work[2] to implement >>> cont-pte support on arm64, but its a complete rewrite based on [1] to >>> optimize all architectures independent of any such PTE bits, and to >>> use the new rmap batching functions that simplify the code and prepare >>> for further rmap accounting changes. >>> >>> We collect consecutive PTEs that map consecutive pages of the same large >>> folio, making sure that the other PTE bits are compatible, and (a) adjust >>> the refcount only once per batch, (b) call rmap handling functions only >>> once per batch and (c) perform batch PTE setting/updates. >>> >>> While this series should be beneficial for adding cont-pte support on >>> ARM64[2], it's one of the requirements for maintaining a total mapcount[3] >>> for large folios with minimal added overhead and further changes[4] that >>> build up on top of the total mapcount. >>> >>> Independent of all that, this series results in a speedup during fork with >>> PTE-mapped THP, which is the default with THPs that are smaller than a PMD >>> (for example, 16KiB to 1024KiB mTHPs for anonymous memory[5]). >>> >>> On an Intel Xeon Silver 4210R CPU, fork'ing with 1GiB of PTE-mapped folios >>> of the same size (stddev < 1%) results in the following runtimes >>> for fork() (shorter is better): >>> >>> Folio Size | v6.8-rc1 | New | Change >>> ------------------------------------------ >>> 4KiB | 0.014328 | 0.014035 | - 2% >>> 16KiB | 0.014263 | 0.01196 | -16% >>> 32KiB | 0.014334 | 0.01094 | -24% >>> 64KiB | 0.014046 | 0.010444 | -26% >>> 128KiB | 0.014011 | 0.010063 | -28% >>> 256KiB | 0.013993 | 0.009938 | -29% >>> 512KiB | 0.013983 | 0.00985 | -30% >>> 1024KiB | 0.013986 | 0.00982 | -30% >>> 2048KiB | 0.014305 | 0.010076 | -30% >> >> Just a heads up that I'm seeing some strange results on Apple M2. Fork for >> order-0 is seemingly costing ~17% more. I'm using GCC 13.2 and was pretty sure I >> didn't see this problem with version 1; although that was on a different >> baseline and I've thrown the numbers away so will rerun and try to debug this. >> > > So far, on my x86 tests (Intel, AMD EPYC), I was not able to observe this. > fork() for order-0 was consistently effectively unchanged. Do you observe that > on other ARM systems as well? Nope; running the exact same kernel binary and user space on Altra, I see sensible numbers; fork order-0: -1.3% fork order-9: -7.6% dontneed order-0: -0.5% dontneed order-9: 0.1% munmap order-0: 0.0% munmap order-9: -67.9% So I guess some pipelining issue that causes the M2 to stall more? > > >> | kernel | mean_rel | std_rel | >> |:------------|-----------:|----------:| >> | mm-unstable | 0.0% | 1.1% | >> | patch 1 | -2.3% | 1.3% | >> | patch 10 | -2.9% | 2.7% | >> | patch 11 | 13.5% | 0.5% | >> | patch 12 | 15.2% | 1.2% | >> | patch 13 | 18.2% | 0.7% | >> | patch 14 | 20.5% | 1.0% | >> | patch 15 | 17.1% | 1.6% | >> | patch 15 | 16.7% | 0.8% | >> >> fork for order-9 is looking good (-20%), and for the zap series, munmap is >> looking good, but dontneed is looking poor for both order-0 and 9. But one thing >> at a time... let's concentrate on fork order-0 first. > > munmap and dontneed end up calling the exact same call paths. So a big > performance difference is rather surprising and might indicate something else. > > (I think I told you that I was running in some kind of VMA merging problem where > one would suddenly get with my benchmark 1 VMA per page. The new benchmark below > works around that, but I am not sure if that was fixed in the meantime) > > VMA merging can of course explain a big difference in fork and munmap vs. > dontneed times, especially when comparing different code base where that VMA > merging behavior was different. > >> >> Note that I'm still using the "old" benchmark code. Could you resend me the link >> to the new code? Although I don't think there should be any effect for order-0 >> anyway, if I understood your changes correctly? > > This is the combined one (small and large PTEs): > > https://gitlab.com/davidhildenbrand/scratchspace/-/raw/main/pte-mapped-folio-benchmarks.c?inline=false I'll have a go with this. >
On 31.01.24 12:16, Ryan Roberts wrote: > On 31/01/2024 11:06, David Hildenbrand wrote: >> On 31.01.24 11:43, Ryan Roberts wrote: >>> On 29/01/2024 12:46, David Hildenbrand wrote: >>>> Now that the rmap overhaul[1] is upstream that provides a clean interface >>>> for rmap batching, let's implement PTE batching during fork when processing >>>> PTE-mapped THPs. >>>> >>>> This series is partially based on Ryan's previous work[2] to implement >>>> cont-pte support on arm64, but its a complete rewrite based on [1] to >>>> optimize all architectures independent of any such PTE bits, and to >>>> use the new rmap batching functions that simplify the code and prepare >>>> for further rmap accounting changes. >>>> >>>> We collect consecutive PTEs that map consecutive pages of the same large >>>> folio, making sure that the other PTE bits are compatible, and (a) adjust >>>> the refcount only once per batch, (b) call rmap handling functions only >>>> once per batch and (c) perform batch PTE setting/updates. >>>> >>>> While this series should be beneficial for adding cont-pte support on >>>> ARM64[2], it's one of the requirements for maintaining a total mapcount[3] >>>> for large folios with minimal added overhead and further changes[4] that >>>> build up on top of the total mapcount. >>>> >>>> Independent of all that, this series results in a speedup during fork with >>>> PTE-mapped THP, which is the default with THPs that are smaller than a PMD >>>> (for example, 16KiB to 1024KiB mTHPs for anonymous memory[5]). >>>> >>>> On an Intel Xeon Silver 4210R CPU, fork'ing with 1GiB of PTE-mapped folios >>>> of the same size (stddev < 1%) results in the following runtimes >>>> for fork() (shorter is better): >>>> >>>> Folio Size | v6.8-rc1 | New | Change >>>> ------------------------------------------ >>>> 4KiB | 0.014328 | 0.014035 | - 2% >>>> 16KiB | 0.014263 | 0.01196 | -16% >>>> 32KiB | 0.014334 | 0.01094 | -24% >>>> 64KiB | 0.014046 | 0.010444 | -26% >>>> 128KiB | 0.014011 | 0.010063 | -28% >>>> 256KiB | 0.013993 | 0.009938 | -29% >>>> 512KiB | 0.013983 | 0.00985 | -30% >>>> 1024KiB | 0.013986 | 0.00982 | -30% >>>> 2048KiB | 0.014305 | 0.010076 | -30% >>> >>> Just a heads up that I'm seeing some strange results on Apple M2. Fork for >>> order-0 is seemingly costing ~17% more. I'm using GCC 13.2 and was pretty sure I >>> didn't see this problem with version 1; although that was on a different >>> baseline and I've thrown the numbers away so will rerun and try to debug this. >>> >> >> So far, on my x86 tests (Intel, AMD EPYC), I was not able to observe this. >> fork() for order-0 was consistently effectively unchanged. Do you observe that >> on other ARM systems as well? > > Nope; running the exact same kernel binary and user space on Altra, I see > sensible numbers; > > fork order-0: -1.3% > fork order-9: -7.6% > dontneed order-0: -0.5% > dontneed order-9: 0.1% > munmap order-0: 0.0% > munmap order-9: -67.9% > > So I guess some pipelining issue that causes the M2 to stall more? With one effective added folio_test_large(), it could only be a code layout problem? Or the compiler does something stupid, but you say that you run the exact same kernel binary, so that doesn't make sense. I'm also surprised about the dontneed vs. munmap numbers. Doesn't make any sense (again, there was this VMA merging problem but it would still allow for batching within a single VMA that spans exactly one large folio). What are you using as baseline? Really just mm-unstable vs. mm-unstable+patches? Let's see if the new test changes the numbers you measure.
On 31/01/2024 11:28, David Hildenbrand wrote: > On 31.01.24 12:16, Ryan Roberts wrote: >> On 31/01/2024 11:06, David Hildenbrand wrote: >>> On 31.01.24 11:43, Ryan Roberts wrote: >>>> On 29/01/2024 12:46, David Hildenbrand wrote: >>>>> Now that the rmap overhaul[1] is upstream that provides a clean interface >>>>> for rmap batching, let's implement PTE batching during fork when processing >>>>> PTE-mapped THPs. >>>>> >>>>> This series is partially based on Ryan's previous work[2] to implement >>>>> cont-pte support on arm64, but its a complete rewrite based on [1] to >>>>> optimize all architectures independent of any such PTE bits, and to >>>>> use the new rmap batching functions that simplify the code and prepare >>>>> for further rmap accounting changes. >>>>> >>>>> We collect consecutive PTEs that map consecutive pages of the same large >>>>> folio, making sure that the other PTE bits are compatible, and (a) adjust >>>>> the refcount only once per batch, (b) call rmap handling functions only >>>>> once per batch and (c) perform batch PTE setting/updates. >>>>> >>>>> While this series should be beneficial for adding cont-pte support on >>>>> ARM64[2], it's one of the requirements for maintaining a total mapcount[3] >>>>> for large folios with minimal added overhead and further changes[4] that >>>>> build up on top of the total mapcount. >>>>> >>>>> Independent of all that, this series results in a speedup during fork with >>>>> PTE-mapped THP, which is the default with THPs that are smaller than a PMD >>>>> (for example, 16KiB to 1024KiB mTHPs for anonymous memory[5]). >>>>> >>>>> On an Intel Xeon Silver 4210R CPU, fork'ing with 1GiB of PTE-mapped folios >>>>> of the same size (stddev < 1%) results in the following runtimes >>>>> for fork() (shorter is better): >>>>> >>>>> Folio Size | v6.8-rc1 | New | Change >>>>> ------------------------------------------ >>>>> 4KiB | 0.014328 | 0.014035 | - 2% >>>>> 16KiB | 0.014263 | 0.01196 | -16% >>>>> 32KiB | 0.014334 | 0.01094 | -24% >>>>> 64KiB | 0.014046 | 0.010444 | -26% >>>>> 128KiB | 0.014011 | 0.010063 | -28% >>>>> 256KiB | 0.013993 | 0.009938 | -29% >>>>> 512KiB | 0.013983 | 0.00985 | -30% >>>>> 1024KiB | 0.013986 | 0.00982 | -30% >>>>> 2048KiB | 0.014305 | 0.010076 | -30% >>>> >>>> Just a heads up that I'm seeing some strange results on Apple M2. Fork for >>>> order-0 is seemingly costing ~17% more. I'm using GCC 13.2 and was pretty >>>> sure I >>>> didn't see this problem with version 1; although that was on a different >>>> baseline and I've thrown the numbers away so will rerun and try to debug this. Numbers for v1 of the series, both on top of 6.8-rc1 and rebased to the same mm-unstable base as v3 of the series (first 2 rows are from what I just posted for context): | kernel | mean_rel | std_rel | |:-------------------|-----------:|----------:| | mm-unstabe (base) | 0.0% | 1.1% | | mm-unstable + v3 | 16.7% | 0.8% | | mm-unstable + v1 | -2.5% | 1.7% | | v6.8-rc1 + v1 | -6.6% | 1.1% | So all looks good with v1. And seems to suggest mm-unstable has regressed by ~4% vs v6.8-rc1. Is this really a useful benchmark? Does the raw performance of fork() syscall really matter? Evidence suggests its moving all over the place - breath on the code and it changes - not a great place to be when using the test for gating purposes! Still with the old tests - I'll move to the new ones now. >>>> >>> >>> So far, on my x86 tests (Intel, AMD EPYC), I was not able to observe this. >>> fork() for order-0 was consistently effectively unchanged. Do you observe that >>> on other ARM systems as well? >> >> Nope; running the exact same kernel binary and user space on Altra, I see >> sensible numbers; >> >> fork order-0: -1.3% >> fork order-9: -7.6% >> dontneed order-0: -0.5% >> dontneed order-9: 0.1% >> munmap order-0: 0.0% >> munmap order-9: -67.9% >> >> So I guess some pipelining issue that causes the M2 to stall more? > > With one effective added folio_test_large(), it could only be a code layout > problem? Or the compiler does something stupid, but you say that you run the > exact same kernel binary, so that doesn't make sense. Yup, same binary. We know this code is very sensitive - 1 cycle makes a big difference. So could easily be code layout, branch prediction, etc... > > I'm also surprised about the dontneed vs. munmap numbers. You mean the ones for Altra that I posted? (I didn't post any for M2). The altra numbers look ok to me; dontneed has no change, and munmap has no change for order-0 and is massively improved for order-9. Doesn't make any sense > (again, there was this VMA merging problem but it would still allow for batching > within a single VMA that spans exactly one large folio). > > What are you using as baseline? Really just mm-unstable vs. mm-unstable+patches? yes. except for "v6.8-rc1 + v1" above. > > Let's see if the new test changes the numbers you measure. >
On 31/01/2024 11:49, Ryan Roberts wrote: > On 31/01/2024 11:28, David Hildenbrand wrote: >> On 31.01.24 12:16, Ryan Roberts wrote: >>> On 31/01/2024 11:06, David Hildenbrand wrote: >>>> On 31.01.24 11:43, Ryan Roberts wrote: >>>>> On 29/01/2024 12:46, David Hildenbrand wrote: >>>>>> Now that the rmap overhaul[1] is upstream that provides a clean interface >>>>>> for rmap batching, let's implement PTE batching during fork when processing >>>>>> PTE-mapped THPs. >>>>>> >>>>>> This series is partially based on Ryan's previous work[2] to implement >>>>>> cont-pte support on arm64, but its a complete rewrite based on [1] to >>>>>> optimize all architectures independent of any such PTE bits, and to >>>>>> use the new rmap batching functions that simplify the code and prepare >>>>>> for further rmap accounting changes. >>>>>> >>>>>> We collect consecutive PTEs that map consecutive pages of the same large >>>>>> folio, making sure that the other PTE bits are compatible, and (a) adjust >>>>>> the refcount only once per batch, (b) call rmap handling functions only >>>>>> once per batch and (c) perform batch PTE setting/updates. >>>>>> >>>>>> While this series should be beneficial for adding cont-pte support on >>>>>> ARM64[2], it's one of the requirements for maintaining a total mapcount[3] >>>>>> for large folios with minimal added overhead and further changes[4] that >>>>>> build up on top of the total mapcount. >>>>>> >>>>>> Independent of all that, this series results in a speedup during fork with >>>>>> PTE-mapped THP, which is the default with THPs that are smaller than a PMD >>>>>> (for example, 16KiB to 1024KiB mTHPs for anonymous memory[5]). >>>>>> >>>>>> On an Intel Xeon Silver 4210R CPU, fork'ing with 1GiB of PTE-mapped folios >>>>>> of the same size (stddev < 1%) results in the following runtimes >>>>>> for fork() (shorter is better): >>>>>> >>>>>> Folio Size | v6.8-rc1 | New | Change >>>>>> ------------------------------------------ >>>>>> 4KiB | 0.014328 | 0.014035 | - 2% >>>>>> 16KiB | 0.014263 | 0.01196 | -16% >>>>>> 32KiB | 0.014334 | 0.01094 | -24% >>>>>> 64KiB | 0.014046 | 0.010444 | -26% >>>>>> 128KiB | 0.014011 | 0.010063 | -28% >>>>>> 256KiB | 0.013993 | 0.009938 | -29% >>>>>> 512KiB | 0.013983 | 0.00985 | -30% >>>>>> 1024KiB | 0.013986 | 0.00982 | -30% >>>>>> 2048KiB | 0.014305 | 0.010076 | -30% >>>>> >>>>> Just a heads up that I'm seeing some strange results on Apple M2. Fork for >>>>> order-0 is seemingly costing ~17% more. I'm using GCC 13.2 and was pretty >>>>> sure I >>>>> didn't see this problem with version 1; although that was on a different >>>>> baseline and I've thrown the numbers away so will rerun and try to debug this. > > Numbers for v1 of the series, both on top of 6.8-rc1 and rebased to the same > mm-unstable base as v3 of the series (first 2 rows are from what I just posted > for context): > > | kernel | mean_rel | std_rel | > |:-------------------|-----------:|----------:| > | mm-unstabe (base) | 0.0% | 1.1% | > | mm-unstable + v3 | 16.7% | 0.8% | > | mm-unstable + v1 | -2.5% | 1.7% | > | v6.8-rc1 + v1 | -6.6% | 1.1% | > > So all looks good with v1. And seems to suggest mm-unstable has regressed by ~4% > vs v6.8-rc1. Is this really a useful benchmark? Does the raw performance of > fork() syscall really matter? Evidence suggests its moving all over the place - > breath on the code and it changes - not a great place to be when using the test > for gating purposes! > > Still with the old tests - I'll move to the new ones now. > > >>>>> >>>> >>>> So far, on my x86 tests (Intel, AMD EPYC), I was not able to observe this. >>>> fork() for order-0 was consistently effectively unchanged. Do you observe that >>>> on other ARM systems as well? >>> >>> Nope; running the exact same kernel binary and user space on Altra, I see >>> sensible numbers; >>> >>> fork order-0: -1.3% >>> fork order-9: -7.6% >>> dontneed order-0: -0.5% >>> dontneed order-9: 0.1% >>> munmap order-0: 0.0% >>> munmap order-9: -67.9% >>> >>> So I guess some pipelining issue that causes the M2 to stall more? >> >> With one effective added folio_test_large(), it could only be a code layout >> problem? Or the compiler does something stupid, but you say that you run the >> exact same kernel binary, so that doesn't make sense. > > Yup, same binary. We know this code is very sensitive - 1 cycle makes a big > difference. So could easily be code layout, branch prediction, etc... > >> >> I'm also surprised about the dontneed vs. munmap numbers. > > You mean the ones for Altra that I posted? (I didn't post any for M2). The altra > numbers look ok to me; dontneed has no change, and munmap has no change for > order-0 and is massively improved for order-9. > > Doesn't make any sense >> (again, there was this VMA merging problem but it would still allow for batching >> within a single VMA that spans exactly one large folio). >> >> What are you using as baseline? Really just mm-unstable vs. mm-unstable+patches? > > yes. except for "v6.8-rc1 + v1" above. > >> >> Let's see if the new test changes the numbers you measure. Nope: looks the same. I've taken my test harness out of the picture and done everything manually from the ground up, with the old tests and the new. Headline is that I see similar numbers from both. Some details: - I'm running for 10 seconds then averaging the output - test is bimodal; first run (of 10 seconds) after boot is a bit faster on average (up to 10%) than the rest; I could guess this is due to the memory being allocated more contiguously the first few times through, so struct pages have better locality, but that's a guess. - test is 5-10% slower when output is printed to terminal vs when redirected to file. I've always effectively been redirecting. Not sure if this overhead could start to dominate the regression and that's why you don't see it? I'm inclined to run this test for the last N kernel releases and if the number moves around significantly, conclude that these tests don't really matter. Otherwise its an exercise in randomly refactoring code until it works well, but that's just overfitting to the compiler and hw. What do you think? Thanks, Ryan
On 31.01.24 13:37, Ryan Roberts wrote: > On 31/01/2024 11:49, Ryan Roberts wrote: >> On 31/01/2024 11:28, David Hildenbrand wrote: >>> On 31.01.24 12:16, Ryan Roberts wrote: >>>> On 31/01/2024 11:06, David Hildenbrand wrote: >>>>> On 31.01.24 11:43, Ryan Roberts wrote: >>>>>> On 29/01/2024 12:46, David Hildenbrand wrote: >>>>>>> Now that the rmap overhaul[1] is upstream that provides a clean interface >>>>>>> for rmap batching, let's implement PTE batching during fork when processing >>>>>>> PTE-mapped THPs. >>>>>>> >>>>>>> This series is partially based on Ryan's previous work[2] to implement >>>>>>> cont-pte support on arm64, but its a complete rewrite based on [1] to >>>>>>> optimize all architectures independent of any such PTE bits, and to >>>>>>> use the new rmap batching functions that simplify the code and prepare >>>>>>> for further rmap accounting changes. >>>>>>> >>>>>>> We collect consecutive PTEs that map consecutive pages of the same large >>>>>>> folio, making sure that the other PTE bits are compatible, and (a) adjust >>>>>>> the refcount only once per batch, (b) call rmap handling functions only >>>>>>> once per batch and (c) perform batch PTE setting/updates. >>>>>>> >>>>>>> While this series should be beneficial for adding cont-pte support on >>>>>>> ARM64[2], it's one of the requirements for maintaining a total mapcount[3] >>>>>>> for large folios with minimal added overhead and further changes[4] that >>>>>>> build up on top of the total mapcount. >>>>>>> >>>>>>> Independent of all that, this series results in a speedup during fork with >>>>>>> PTE-mapped THP, which is the default with THPs that are smaller than a PMD >>>>>>> (for example, 16KiB to 1024KiB mTHPs for anonymous memory[5]). >>>>>>> >>>>>>> On an Intel Xeon Silver 4210R CPU, fork'ing with 1GiB of PTE-mapped folios >>>>>>> of the same size (stddev < 1%) results in the following runtimes >>>>>>> for fork() (shorter is better): >>>>>>> >>>>>>> Folio Size | v6.8-rc1 | New | Change >>>>>>> ------------------------------------------ >>>>>>> 4KiB | 0.014328 | 0.014035 | - 2% >>>>>>> 16KiB | 0.014263 | 0.01196 | -16% >>>>>>> 32KiB | 0.014334 | 0.01094 | -24% >>>>>>> 64KiB | 0.014046 | 0.010444 | -26% >>>>>>> 128KiB | 0.014011 | 0.010063 | -28% >>>>>>> 256KiB | 0.013993 | 0.009938 | -29% >>>>>>> 512KiB | 0.013983 | 0.00985 | -30% >>>>>>> 1024KiB | 0.013986 | 0.00982 | -30% >>>>>>> 2048KiB | 0.014305 | 0.010076 | -30% >>>>>> >>>>>> Just a heads up that I'm seeing some strange results on Apple M2. Fork for >>>>>> order-0 is seemingly costing ~17% more. I'm using GCC 13.2 and was pretty >>>>>> sure I >>>>>> didn't see this problem with version 1; although that was on a different >>>>>> baseline and I've thrown the numbers away so will rerun and try to debug this. >> >> Numbers for v1 of the series, both on top of 6.8-rc1 and rebased to the same >> mm-unstable base as v3 of the series (first 2 rows are from what I just posted >> for context): >> >> | kernel | mean_rel | std_rel | >> |:-------------------|-----------:|----------:| >> | mm-unstabe (base) | 0.0% | 1.1% | >> | mm-unstable + v3 | 16.7% | 0.8% | >> | mm-unstable + v1 | -2.5% | 1.7% | >> | v6.8-rc1 + v1 | -6.6% | 1.1% | >> >> So all looks good with v1. And seems to suggest mm-unstable has regressed by ~4% >> vs v6.8-rc1. Is this really a useful benchmark? Does the raw performance of >> fork() syscall really matter? Evidence suggests its moving all over the place - >> breath on the code and it changes - not a great place to be when using the test >> for gating purposes! >> >> Still with the old tests - I'll move to the new ones now. >> >> >>>>>> >>>>> >>>>> So far, on my x86 tests (Intel, AMD EPYC), I was not able to observe this. >>>>> fork() for order-0 was consistently effectively unchanged. Do you observe that >>>>> on other ARM systems as well? >>>> >>>> Nope; running the exact same kernel binary and user space on Altra, I see >>>> sensible numbers; >>>> >>>> fork order-0: -1.3% >>>> fork order-9: -7.6% >>>> dontneed order-0: -0.5% >>>> dontneed order-9: 0.1% >>>> munmap order-0: 0.0% >>>> munmap order-9: -67.9% >>>> >>>> So I guess some pipelining issue that causes the M2 to stall more? >>> >>> With one effective added folio_test_large(), it could only be a code layout >>> problem? Or the compiler does something stupid, but you say that you run the >>> exact same kernel binary, so that doesn't make sense. >> >> Yup, same binary. We know this code is very sensitive - 1 cycle makes a big >> difference. So could easily be code layout, branch prediction, etc... >> >>> >>> I'm also surprised about the dontneed vs. munmap numbers. >> >> You mean the ones for Altra that I posted? (I didn't post any for M2). The altra >> numbers look ok to me; dontneed has no change, and munmap has no change for >> order-0 and is massively improved for order-9. >> >> Doesn't make any sense >>> (again, there was this VMA merging problem but it would still allow for batching >>> within a single VMA that spans exactly one large folio). >>> >>> What are you using as baseline? Really just mm-unstable vs. mm-unstable+patches? >> >> yes. except for "v6.8-rc1 + v1" above. >> >>> >>> Let's see if the new test changes the numbers you measure. > > Nope: looks the same. I've taken my test harness out of the picture and done > everything manually from the ground up, with the old tests and the new. Headline > is that I see similar numbers from both. I took me a while to get really reproducible numbers on Intel. Most importantly: * Set a fixed CPU frequency, disabling any boost and avoiding any thermal throttling. * Pin the test to CPUs and set a nice level. Another thing is, to avoid systems where you can have NUMA effects within a single socket. Otherwise, memory access latency is just random and depends on what the buddy enjoys giving you. But you seem to get the same +17 even after reboots, so that indicates that the CPU is not happy about the code for some reason. And the weird thing is, that nothing significantly changed for order-0 folios between v1 and v3 that could explain any of this. I'm not worried about 5% or so, nobody cares. But it would be good to have at least an explanation why only that system shows +17%. > > Some details: > - I'm running for 10 seconds then averaging the output Same here. > - test is bimodal; first run (of 10 seconds) after boot is a bit faster on > average (up to 10%) than the rest; I could guess this is due to the memory > being allocated more contiguously the first few times through, so struct > pages have better locality, but that's a guess. I think it also has to do with the PCP lists, and the high-pcp auto tuning (I played with disabling that). Running on a freshly booted system gave me reproducible results. But yes: I was observing something similar on AMD EPYC, where you get consecutive pages from the buddy, but once you allocate from the PCP it might no longer be consecutive. > - test is 5-10% slower when output is printed to terminal vs when redirected to > file. I've always effectively been redirecting. Not sure if this overhead > could start to dominate the regression and that's why you don't see it? That's weird, because we don't print while measuring? Anyhow, 5/10% variance on some system is not the end of the world. > > I'm inclined to run this test for the last N kernel releases and if the number > moves around significantly, conclude that these tests don't really matter. > Otherwise its an exercise in randomly refactoring code until it works well, but > that's just overfitting to the compiler and hw. What do you think? Personally, I wouldn't lose sleep if you see weird, unexplainable behavior on some system (not even architecture!). Trying to optimize for that would indeed be random refactorings. But I would not be so fast to say that "these tests don't really matter" and then go wild and degrade them as much as you want. There are use cases that care about fork performance especially with order-0 pages -- such as Redis.
>> >> I'm also surprised about the dontneed vs. munmap numbers. > > You mean the ones for Altra that I posted? (I didn't post any for M2). The altra > numbers look ok to me; dontneed has no change, and munmap has no change for > order-0 and is massively improved for order-9. I would expect that dontneed would similarly benefit -- same code path. But I focused on munmap measurements for now, I'll try finding time to confirm that it's the same on Intel.
On 31/01/2024 12:56, David Hildenbrand wrote: > On 31.01.24 13:37, Ryan Roberts wrote: >> On 31/01/2024 11:49, Ryan Roberts wrote: >>> On 31/01/2024 11:28, David Hildenbrand wrote: >>>> On 31.01.24 12:16, Ryan Roberts wrote: >>>>> On 31/01/2024 11:06, David Hildenbrand wrote: >>>>>> On 31.01.24 11:43, Ryan Roberts wrote: >>>>>>> On 29/01/2024 12:46, David Hildenbrand wrote: >>>>>>>> Now that the rmap overhaul[1] is upstream that provides a clean interface >>>>>>>> for rmap batching, let's implement PTE batching during fork when processing >>>>>>>> PTE-mapped THPs. >>>>>>>> >>>>>>>> This series is partially based on Ryan's previous work[2] to implement >>>>>>>> cont-pte support on arm64, but its a complete rewrite based on [1] to >>>>>>>> optimize all architectures independent of any such PTE bits, and to >>>>>>>> use the new rmap batching functions that simplify the code and prepare >>>>>>>> for further rmap accounting changes. >>>>>>>> >>>>>>>> We collect consecutive PTEs that map consecutive pages of the same large >>>>>>>> folio, making sure that the other PTE bits are compatible, and (a) adjust >>>>>>>> the refcount only once per batch, (b) call rmap handling functions only >>>>>>>> once per batch and (c) perform batch PTE setting/updates. >>>>>>>> >>>>>>>> While this series should be beneficial for adding cont-pte support on >>>>>>>> ARM64[2], it's one of the requirements for maintaining a total mapcount[3] >>>>>>>> for large folios with minimal added overhead and further changes[4] that >>>>>>>> build up on top of the total mapcount. >>>>>>>> >>>>>>>> Independent of all that, this series results in a speedup during fork with >>>>>>>> PTE-mapped THP, which is the default with THPs that are smaller than a PMD >>>>>>>> (for example, 16KiB to 1024KiB mTHPs for anonymous memory[5]). >>>>>>>> >>>>>>>> On an Intel Xeon Silver 4210R CPU, fork'ing with 1GiB of PTE-mapped folios >>>>>>>> of the same size (stddev < 1%) results in the following runtimes >>>>>>>> for fork() (shorter is better): >>>>>>>> >>>>>>>> Folio Size | v6.8-rc1 | New | Change >>>>>>>> ------------------------------------------ >>>>>>>> 4KiB | 0.014328 | 0.014035 | - 2% >>>>>>>> 16KiB | 0.014263 | 0.01196 | -16% >>>>>>>> 32KiB | 0.014334 | 0.01094 | -24% >>>>>>>> 64KiB | 0.014046 | 0.010444 | -26% >>>>>>>> 128KiB | 0.014011 | 0.010063 | -28% >>>>>>>> 256KiB | 0.013993 | 0.009938 | -29% >>>>>>>> 512KiB | 0.013983 | 0.00985 | -30% >>>>>>>> 1024KiB | 0.013986 | 0.00982 | -30% >>>>>>>> 2048KiB | 0.014305 | 0.010076 | -30% >>>>>>> >>>>>>> Just a heads up that I'm seeing some strange results on Apple M2. Fork for >>>>>>> order-0 is seemingly costing ~17% more. I'm using GCC 13.2 and was pretty >>>>>>> sure I >>>>>>> didn't see this problem with version 1; although that was on a different >>>>>>> baseline and I've thrown the numbers away so will rerun and try to debug >>>>>>> this. >>> >>> Numbers for v1 of the series, both on top of 6.8-rc1 and rebased to the same >>> mm-unstable base as v3 of the series (first 2 rows are from what I just posted >>> for context): >>> >>> | kernel | mean_rel | std_rel | >>> |:-------------------|-----------:|----------:| >>> | mm-unstabe (base) | 0.0% | 1.1% | >>> | mm-unstable + v3 | 16.7% | 0.8% | >>> | mm-unstable + v1 | -2.5% | 1.7% | >>> | v6.8-rc1 + v1 | -6.6% | 1.1% | >>> >>> So all looks good with v1. And seems to suggest mm-unstable has regressed by ~4% >>> vs v6.8-rc1. Is this really a useful benchmark? Does the raw performance of >>> fork() syscall really matter? Evidence suggests its moving all over the place - >>> breath on the code and it changes - not a great place to be when using the test >>> for gating purposes! >>> >>> Still with the old tests - I'll move to the new ones now. >>> >>> >>>>>>> >>>>>> >>>>>> So far, on my x86 tests (Intel, AMD EPYC), I was not able to observe this. >>>>>> fork() for order-0 was consistently effectively unchanged. Do you observe >>>>>> that >>>>>> on other ARM systems as well? >>>>> >>>>> Nope; running the exact same kernel binary and user space on Altra, I see >>>>> sensible numbers; >>>>> >>>>> fork order-0: -1.3% >>>>> fork order-9: -7.6% >>>>> dontneed order-0: -0.5% >>>>> dontneed order-9: 0.1% >>>>> munmap order-0: 0.0% >>>>> munmap order-9: -67.9% >>>>> >>>>> So I guess some pipelining issue that causes the M2 to stall more? >>>> >>>> With one effective added folio_test_large(), it could only be a code layout >>>> problem? Or the compiler does something stupid, but you say that you run the >>>> exact same kernel binary, so that doesn't make sense. >>> >>> Yup, same binary. We know this code is very sensitive - 1 cycle makes a big >>> difference. So could easily be code layout, branch prediction, etc... >>> >>>> >>>> I'm also surprised about the dontneed vs. munmap numbers. >>> >>> You mean the ones for Altra that I posted? (I didn't post any for M2). The altra >>> numbers look ok to me; dontneed has no change, and munmap has no change for >>> order-0 and is massively improved for order-9. >>> >>> Doesn't make any sense >>>> (again, there was this VMA merging problem but it would still allow for >>>> batching >>>> within a single VMA that spans exactly one large folio). >>>> >>>> What are you using as baseline? Really just mm-unstable vs. >>>> mm-unstable+patches? >>> >>> yes. except for "v6.8-rc1 + v1" above. >>> >>>> >>>> Let's see if the new test changes the numbers you measure. >> >> Nope: looks the same. I've taken my test harness out of the picture and done >> everything manually from the ground up, with the old tests and the new. Headline >> is that I see similar numbers from both. > > I took me a while to get really reproducible numbers on Intel. Most importantly: > * Set a fixed CPU frequency, disabling any boost and avoiding any > thermal throttling. > * Pin the test to CPUs and set a nice level. I'm already pinning the test to cpu 0. But for M2, at least, I'm running in a VM on top of macos, and I don't have a mechanism to pin the QEMU threads to the physical CPUs. Anyway, I don't think these are problems because for a given kernel build I can accurately repro numbers. > > Another thing is, to avoid systems where you can have NUMA effects within a > single socket. Otherwise, memory access latency is just random and depends on > what the buddy enjoys giving you. Yep; same. M2 is 1 NUMA node. On Altra, I'm disabling the second NUMA node to remove those effects. > > But you seem to get the same +17 even after reboots, so that indicates that the > CPU is not happy about the code for some reason. And the weird thing is, that > nothing significantly changed for order-0 folios between v1 and v3 that could > explain any of this. > > I'm not worried about 5% or so, nobody cares. But it would be good to have at > least an explanation why only that system shows +17%. Yep understood. > >> >> Some details: >> - I'm running for 10 seconds then averaging the output > > Same here. > >> - test is bimodal; first run (of 10 seconds) after boot is a bit faster on >> average (up to 10%) than the rest; I could guess this is due to the memory >> being allocated more contiguously the first few times through, so struct >> pages have better locality, but that's a guess. > > I think it also has to do with the PCP lists, and the high-pcp auto tuning (I > played with disabling that). Running on a freshly booted system gave me > reproducible results. > > But yes: I was observing something similar on AMD EPYC, where you get > consecutive pages from the buddy, but once you allocate from the PCP it might no > longer be consecutive. > >> - test is 5-10% slower when output is printed to terminal vs when redirected to >> file. I've always effectively been redirecting. Not sure if this overhead >> could start to dominate the regression and that's why you don't see it? > > That's weird, because we don't print while measuring? Anyhow, 5/10% variance on > some system is not the end of the world. I imagine its cache effects? More work to do to print the output could be evicting some code that's in the benchmark path? > >> >> I'm inclined to run this test for the last N kernel releases and if the number >> moves around significantly, conclude that these tests don't really matter. >> Otherwise its an exercise in randomly refactoring code until it works well, but >> that's just overfitting to the compiler and hw. What do you think? > > Personally, I wouldn't lose sleep if you see weird, unexplainable behavior on > some system (not even architecture!). Trying to optimize for that would indeed > be random refactorings. > > But I would not be so fast to say that "these tests don't really matter" and > then go wild and degrade them as much as you want. There are use cases that care > about fork performance especially with order-0 pages -- such as Redis. Indeed. But also remember that my fork baseline time is ~2.5ms, and I think you said yours was 14ms :) I'll continue to mess around with it until the end of the day. But I'm not making any headway, then I'll change tack; I'll just measure the performance of my contpte changes using your fork/zap stuff as the baseline and post based on that.
>>> Nope: looks the same. I've taken my test harness out of the picture and done >>> everything manually from the ground up, with the old tests and the new. Headline >>> is that I see similar numbers from both. >> >> I took me a while to get really reproducible numbers on Intel. Most importantly: >> * Set a fixed CPU frequency, disabling any boost and avoiding any >> thermal throttling. >> * Pin the test to CPUs and set a nice level. > > I'm already pinning the test to cpu 0. But for M2, at least, I'm running in a VM > on top of macos, and I don't have a mechanism to pin the QEMU threads to the > physical CPUs. Anyway, I don't think these are problems because for a given > kernel build I can accurately repro numbers. Oh, you do have a layer of virtualization in there. I *suspect* that might amplify some odd things regarding code layout, caching effects, etc. I guess especially the fork() benchmark is too sensible (fast) for things like that, so I would just focus on bare metal results where you can control the environment completely. Note that regarding NUMA effects, I mean when some memory access within the same socket is faster/slower even with only a single node. On AMD EPYC that's possible, depending on which core you are running and on which memory controller the memory you want to access is located. If both are in different quadrants IIUC, the access latency will be different. >> But yes: I was observing something similar on AMD EPYC, where you get >> consecutive pages from the buddy, but once you allocate from the PCP it might no >> longer be consecutive. >> >>> - test is 5-10% slower when output is printed to terminal vs when redirected to >>> file. I've always effectively been redirecting. Not sure if this overhead >>> could start to dominate the regression and that's why you don't see it? >> >> That's weird, because we don't print while measuring? Anyhow, 5/10% variance on >> some system is not the end of the world. > > I imagine its cache effects? More work to do to print the output could be > evicting some code that's in the benchmark path? Maybe. Do you also see these oddities on the bare metal system? > >> >>> >>> I'm inclined to run this test for the last N kernel releases and if the number >>> moves around significantly, conclude that these tests don't really matter. >>> Otherwise its an exercise in randomly refactoring code until it works well, but >>> that's just overfitting to the compiler and hw. What do you think? >> >> Personally, I wouldn't lose sleep if you see weird, unexplainable behavior on >> some system (not even architecture!). Trying to optimize for that would indeed >> be random refactorings. >> >> But I would not be so fast to say that "these tests don't really matter" and >> then go wild and degrade them as much as you want. There are use cases that care >> about fork performance especially with order-0 pages -- such as Redis. > > Indeed. But also remember that my fork baseline time is ~2.5ms, and I think you > said yours was 14ms :) Yes, no idea why M2 is that fast (BTW, which page size? 4k or 16k? ) :) > > I'll continue to mess around with it until the end of the day. But I'm not > making any headway, then I'll change tack; I'll just measure the performance of > my contpte changes using your fork/zap stuff as the baseline and post based on that. You should likely not focus on M2 results. Just pick a representative bare metal machine where you get consistent, explainable results. Nothing in the code is fine-tuned for a particular architecture so far, only order-0 handling is kept separate. BTW: I see the exact same speedups for dontneed that I see for munmap. For example, for order-9, it goes from 0.023412s -> 0.009785, so -58%. So I'm curious why you see a speedup for munmap but not for dontneed.
On 31/01/2024 13:38, David Hildenbrand wrote: >>>> Nope: looks the same. I've taken my test harness out of the picture and done >>>> everything manually from the ground up, with the old tests and the new. >>>> Headline >>>> is that I see similar numbers from both. >>> >>> I took me a while to get really reproducible numbers on Intel. Most importantly: >>> * Set a fixed CPU frequency, disabling any boost and avoiding any >>> thermal throttling. >>> * Pin the test to CPUs and set a nice level. >> >> I'm already pinning the test to cpu 0. But for M2, at least, I'm running in a VM >> on top of macos, and I don't have a mechanism to pin the QEMU threads to the >> physical CPUs. Anyway, I don't think these are problems because for a given >> kernel build I can accurately repro numbers. > > Oh, you do have a layer of virtualization in there. I *suspect* that might > amplify some odd things regarding code layout, caching effects, etc. > > I guess especially the fork() benchmark is too sensible (fast) for things like > that, so I would just focus on bare metal results where you can control the > environment completely. Yeah, maybe. OK I'll park M2 for now. > > Note that regarding NUMA effects, I mean when some memory access within the same > socket is faster/slower even with only a single node. On AMD EPYC that's > possible, depending on which core you are running and on which memory controller > the memory you want to access is located. If both are in different quadrants > IIUC, the access latency will be different. I've configured the NUMA to only bring the RAM and CPUs for a single socket online, so I shouldn't be seeing any of these effects. Anyway, I've been using the Altra as a secondary because its so much slower than the M2. Let me move over to it and see if everything looks more straightforward there. > >>> But yes: I was observing something similar on AMD EPYC, where you get >>> consecutive pages from the buddy, but once you allocate from the PCP it might no >>> longer be consecutive. >>> >>>> - test is 5-10% slower when output is printed to terminal vs when >>>> redirected to >>>> file. I've always effectively been redirecting. Not sure if this overhead >>>> could start to dominate the regression and that's why you don't see it? >>> >>> That's weird, because we don't print while measuring? Anyhow, 5/10% variance on >>> some system is not the end of the world. >> >> I imagine its cache effects? More work to do to print the output could be >> evicting some code that's in the benchmark path? > > Maybe. Do you also see these oddities on the bare metal system? > >> >>> >>>> >>>> I'm inclined to run this test for the last N kernel releases and if the number >>>> moves around significantly, conclude that these tests don't really matter. >>>> Otherwise its an exercise in randomly refactoring code until it works well, but >>>> that's just overfitting to the compiler and hw. What do you think? >>> >>> Personally, I wouldn't lose sleep if you see weird, unexplainable behavior on >>> some system (not even architecture!). Trying to optimize for that would indeed >>> be random refactorings. >>> >>> But I would not be so fast to say that "these tests don't really matter" and >>> then go wild and degrade them as much as you want. There are use cases that care >>> about fork performance especially with order-0 pages -- such as Redis. >> >> Indeed. But also remember that my fork baseline time is ~2.5ms, and I think you >> said yours was 14ms :) > > Yes, no idea why M2 is that fast (BTW, which page size? 4k or 16k? ) :) The guest kernel is using 4K pages. I'm not quite sure what is happening at stage2; QEMU doesn't expose any options to explicitly request huge pages for macos AFAICT. > >> >> I'll continue to mess around with it until the end of the day. But I'm not >> making any headway, then I'll change tack; I'll just measure the performance of >> my contpte changes using your fork/zap stuff as the baseline and post based on >> that. > > You should likely not focus on M2 results. Just pick a representative bare metal > machine where you get consistent, explainable results. > > Nothing in the code is fine-tuned for a particular architecture so far, only > order-0 handling is kept separate. > > BTW: I see the exact same speedups for dontneed that I see for munmap. For > example, for order-9, it goes from 0.023412s -> 0.009785, so -58%. So I'm > curious why you see a speedup for munmap but not for dontneed. Ugh... ok, coming up.
>> Note that regarding NUMA effects, I mean when some memory access within the same >> socket is faster/slower even with only a single node. On AMD EPYC that's >> possible, depending on which core you are running and on which memory controller >> the memory you want to access is located. If both are in different quadrants >> IIUC, the access latency will be different. > > I've configured the NUMA to only bring the RAM and CPUs for a single socket > online, so I shouldn't be seeing any of these effects. Anyway, I've been using > the Altra as a secondary because its so much slower than the M2. Let me move > over to it and see if everything looks more straightforward there. Better use a system where people will actually run Linux production workloads on, even if it is slower :) [...] >>> >>> I'll continue to mess around with it until the end of the day. But I'm not >>> making any headway, then I'll change tack; I'll just measure the performance of >>> my contpte changes using your fork/zap stuff as the baseline and post based on >>> that. >> >> You should likely not focus on M2 results. Just pick a representative bare metal >> machine where you get consistent, explainable results. >> >> Nothing in the code is fine-tuned for a particular architecture so far, only >> order-0 handling is kept separate. >> >> BTW: I see the exact same speedups for dontneed that I see for munmap. For >> example, for order-9, it goes from 0.023412s -> 0.009785, so -58%. So I'm >> curious why you see a speedup for munmap but not for dontneed. > > Ugh... ok, coming up. Hopefully you were just staring at the wrong numbers (e.g., only with fork patches). Because both (munmap/pte-dontneed) are using the exact same code path.
On 31/01/2024 14:29, David Hildenbrand wrote: >>> Note that regarding NUMA effects, I mean when some memory access within the same >>> socket is faster/slower even with only a single node. On AMD EPYC that's >>> possible, depending on which core you are running and on which memory controller >>> the memory you want to access is located. If both are in different quadrants >>> IIUC, the access latency will be different. >> >> I've configured the NUMA to only bring the RAM and CPUs for a single socket >> online, so I shouldn't be seeing any of these effects. Anyway, I've been using >> the Altra as a secondary because its so much slower than the M2. Let me move >> over to it and see if everything looks more straightforward there. > > Better use a system where people will actually run Linux production workloads > on, even if it is slower :) > > [...] > >>>> >>>> I'll continue to mess around with it until the end of the day. But I'm not >>>> making any headway, then I'll change tack; I'll just measure the performance of >>>> my contpte changes using your fork/zap stuff as the baseline and post based on >>>> that. >>> >>> You should likely not focus on M2 results. Just pick a representative bare metal >>> machine where you get consistent, explainable results. >>> >>> Nothing in the code is fine-tuned for a particular architecture so far, only >>> order-0 handling is kept separate. >>> >>> BTW: I see the exact same speedups for dontneed that I see for munmap. For >>> example, for order-9, it goes from 0.023412s -> 0.009785, so -58%. So I'm >>> curious why you see a speedup for munmap but not for dontneed. >> >> Ugh... ok, coming up. > > Hopefully you were just staring at the wrong numbers (e.g., only with fork > patches). Because both (munmap/pte-dontneed) are using the exact same code path. > Ahh... I'm doing pte-dontneed, which is the only option in your original benchmark - it does MADV_DONTNEED one page at a time. It looks like your new benchmark has an additional "dontneed" option that does it in one shot. Which option are you running? Assuming the latter, I think that explains it.
On 31.01.24 16:02, Ryan Roberts wrote: > On 31/01/2024 14:29, David Hildenbrand wrote: >>>> Note that regarding NUMA effects, I mean when some memory access within the same >>>> socket is faster/slower even with only a single node. On AMD EPYC that's >>>> possible, depending on which core you are running and on which memory controller >>>> the memory you want to access is located. If both are in different quadrants >>>> IIUC, the access latency will be different. >>> >>> I've configured the NUMA to only bring the RAM and CPUs for a single socket >>> online, so I shouldn't be seeing any of these effects. Anyway, I've been using >>> the Altra as a secondary because its so much slower than the M2. Let me move >>> over to it and see if everything looks more straightforward there. >> >> Better use a system where people will actually run Linux production workloads >> on, even if it is slower :) >> >> [...] >> >>>>> >>>>> I'll continue to mess around with it until the end of the day. But I'm not >>>>> making any headway, then I'll change tack; I'll just measure the performance of >>>>> my contpte changes using your fork/zap stuff as the baseline and post based on >>>>> that. >>>> >>>> You should likely not focus on M2 results. Just pick a representative bare metal >>>> machine where you get consistent, explainable results. >>>> >>>> Nothing in the code is fine-tuned for a particular architecture so far, only >>>> order-0 handling is kept separate. >>>> >>>> BTW: I see the exact same speedups for dontneed that I see for munmap. For >>>> example, for order-9, it goes from 0.023412s -> 0.009785, so -58%. So I'm >>>> curious why you see a speedup for munmap but not for dontneed. >>> >>> Ugh... ok, coming up. >> >> Hopefully you were just staring at the wrong numbers (e.g., only with fork >> patches). Because both (munmap/pte-dontneed) are using the exact same code path. >> > > Ahh... I'm doing pte-dontneed, which is the only option in your original > benchmark - it does MADV_DONTNEED one page at a time. It looks like your new > benchmark has an additional "dontneed" option that does it in one shot. Which > option are you running? Assuming the latter, I think that explains it. I temporarily removed that option and then re-added it. Guess you got a wrong snapshot of the benchmark :D pte-dontneed not observing any change is great (no batching possible). dontneed should hopefully/likely see a speedup. Great!
On 31/01/2024 15:05, David Hildenbrand wrote: > On 31.01.24 16:02, Ryan Roberts wrote: >> On 31/01/2024 14:29, David Hildenbrand wrote: >>>>> Note that regarding NUMA effects, I mean when some memory access within the >>>>> same >>>>> socket is faster/slower even with only a single node. On AMD EPYC that's >>>>> possible, depending on which core you are running and on which memory >>>>> controller >>>>> the memory you want to access is located. If both are in different quadrants >>>>> IIUC, the access latency will be different. >>>> >>>> I've configured the NUMA to only bring the RAM and CPUs for a single socket >>>> online, so I shouldn't be seeing any of these effects. Anyway, I've been using >>>> the Altra as a secondary because its so much slower than the M2. Let me move >>>> over to it and see if everything looks more straightforward there. >>> >>> Better use a system where people will actually run Linux production workloads >>> on, even if it is slower :) >>> >>> [...] >>> >>>>>> >>>>>> I'll continue to mess around with it until the end of the day. But I'm not >>>>>> making any headway, then I'll change tack; I'll just measure the >>>>>> performance of >>>>>> my contpte changes using your fork/zap stuff as the baseline and post >>>>>> based on >>>>>> that. >>>>> >>>>> You should likely not focus on M2 results. Just pick a representative bare >>>>> metal >>>>> machine where you get consistent, explainable results. >>>>> >>>>> Nothing in the code is fine-tuned for a particular architecture so far, only >>>>> order-0 handling is kept separate. >>>>> >>>>> BTW: I see the exact same speedups for dontneed that I see for munmap. For >>>>> example, for order-9, it goes from 0.023412s -> 0.009785, so -58%. So I'm >>>>> curious why you see a speedup for munmap but not for dontneed. >>>> >>>> Ugh... ok, coming up. >>> >>> Hopefully you were just staring at the wrong numbers (e.g., only with fork >>> patches). Because both (munmap/pte-dontneed) are using the exact same code path. >>> >> >> Ahh... I'm doing pte-dontneed, which is the only option in your original >> benchmark - it does MADV_DONTNEED one page at a time. It looks like your new >> benchmark has an additional "dontneed" option that does it in one shot. Which >> option are you running? Assuming the latter, I think that explains it. > > I temporarily removed that option and then re-added it. Guess you got a wrong > snapshot of the benchmark :D > > pte-dontneed not observing any change is great (no batching possible). indeed. > > dontneed should hopefully/likely see a speedup. Yes, but that's almost exactly the same path as munmap, so I'm sure it really adds much for this particular series. Anyway, on Altra at least, I'm seeing no regressions, so: Tested-by: Ryan Roberts <ryan.roberts@arm.com> > > Great! >
>> dontneed should hopefully/likely see a speedup. > > Yes, but that's almost exactly the same path as munmap, so I'm sure it really > adds much for this particular series. Right, that's why I'm not including these measurements. dontneed vs. munmap is more about measuring the overhead of VMA modifications + page table reclaim. > Anyway, on Altra at least, I'm seeing no > regressions, so: > > Tested-by: Ryan Roberts <ryan.roberts@arm.com> > Thanks!
Hello: This series was applied to riscv/linux.git (fixes) by Andrew Morton <akpm@linux-foundation.org>: On Mon, 29 Jan 2024 13:46:34 +0100 you wrote: > Now that the rmap overhaul[1] is upstream that provides a clean interface > for rmap batching, let's implement PTE batching during fork when processing > PTE-mapped THPs. > > This series is partially based on Ryan's previous work[2] to implement > cont-pte support on arm64, but its a complete rewrite based on [1] to > optimize all architectures independent of any such PTE bits, and to > use the new rmap batching functions that simplify the code and prepare > for further rmap accounting changes. > > [...] Here is the summary with links: - [v3,01/15] arm64/mm: Make set_ptes() robust when OAs cross 48-bit boundary (no matching commit) - [v3,02/15] arm/pgtable: define PFN_PTE_SHIFT (no matching commit) - [v3,03/15] nios2/pgtable: define PFN_PTE_SHIFT (no matching commit) - [v3,04/15] powerpc/pgtable: define PFN_PTE_SHIFT (no matching commit) - [v3,05/15] riscv/pgtable: define PFN_PTE_SHIFT https://git.kernel.org/riscv/c/57c254b2fb31 - [v3,06/15] s390/pgtable: define PFN_PTE_SHIFT (no matching commit) - [v3,07/15] sparc/pgtable: define PFN_PTE_SHIFT (no matching commit) - [v3,08/15] mm/pgtable: make pte_next_pfn() independent of set_ptes() (no matching commit) - [v3,09/15] arm/mm: use pte_next_pfn() in set_ptes() (no matching commit) - [v3,10/15] powerpc/mm: use pte_next_pfn() in set_ptes() (no matching commit) - [v3,11/15] mm/memory: factor out copying the actual PTE in copy_present_pte() (no matching commit) - [v3,12/15] mm/memory: pass PTE to copy_present_pte() (no matching commit) - [v3,13/15] mm/memory: optimize fork() with PTE-mapped THP (no matching commit) - [v3,14/15] mm/memory: ignore dirty/accessed/soft-dirty bits in folio_pte_batch() (no matching commit) - [v3,15/15] mm/memory: ignore writable bit in folio_pte_batch() (no matching commit) You are awesome, thank you!
Now that the rmap overhaul[1] is upstream that provides a clean interface for rmap batching, let's implement PTE batching during fork when processing PTE-mapped THPs. This series is partially based on Ryan's previous work[2] to implement cont-pte support on arm64, but its a complete rewrite based on [1] to optimize all architectures independent of any such PTE bits, and to use the new rmap batching functions that simplify the code and prepare for further rmap accounting changes. We collect consecutive PTEs that map consecutive pages of the same large folio, making sure that the other PTE bits are compatible, and (a) adjust the refcount only once per batch, (b) call rmap handling functions only once per batch and (c) perform batch PTE setting/updates. While this series should be beneficial for adding cont-pte support on ARM64[2], it's one of the requirements for maintaining a total mapcount[3] for large folios with minimal added overhead and further changes[4] that build up on top of the total mapcount. Independent of all that, this series results in a speedup during fork with PTE-mapped THP, which is the default with THPs that are smaller than a PMD (for example, 16KiB to 1024KiB mTHPs for anonymous memory[5]). On an Intel Xeon Silver 4210R CPU, fork'ing with 1GiB of PTE-mapped folios of the same size (stddev < 1%) results in the following runtimes for fork() (shorter is better): Folio Size | v6.8-rc1 | New | Change ------------------------------------------ 4KiB | 0.014328 | 0.014035 | - 2% 16KiB | 0.014263 | 0.01196 | -16% 32KiB | 0.014334 | 0.01094 | -24% 64KiB | 0.014046 | 0.010444 | -26% 128KiB | 0.014011 | 0.010063 | -28% 256KiB | 0.013993 | 0.009938 | -29% 512KiB | 0.013983 | 0.00985 | -30% 1024KiB | 0.013986 | 0.00982 | -30% 2048KiB | 0.014305 | 0.010076 | -30% Note that these numbers are even better than the ones from v1 (verified over multiple reboots), even though there were only minimal code changes. Well, I removed a pte_mkclean() call for anon folios, maybe that also plays a role. But my experience is that fork() is extremely sensitive to code size, inlining, ... so I suspect we'll see on other architectures rather a change of -20% instead of -30%, and it will be easy to "lose" some of that speedup in the future by subtle code changes. Next up is PTE batching when unmapping. Only tested on x86-64. Compile-tested on most other architectures. v2 -> v3: * Rebased on mm-unstable * Picked up RB's * Updated documentation of wrprotect_ptes(). v1 -> v2: * "arm64/mm: Make set_ptes() robust when OAs cross 48-bit boundary" -> Added patch from Ryan * "arm/pgtable: define PFN_PTE_SHIFT" -> Removed the arm64 bits * "mm/pgtable: make pte_next_pfn() independent of set_ptes()" * "arm/mm: use pte_next_pfn() in set_ptes()" * "powerpc/mm: use pte_next_pfn() in set_ptes()" -> Added to use pte_next_pfn() in some arch set_ptes() implementations I tried to make use of pte_next_pfn() also in the others, but it's not trivial because the other archs implement set_ptes() in their asm/pgtable.h. Future work. * "mm/memory: factor out copying the actual PTE in copy_present_pte()" -> Move common folio_get() out of if/else * "mm/memory: optimize fork() with PTE-mapped THP" -> Add doc for wrprotect_ptes -> Extend description to mention handling of pinned folios -> Move common folio_ref_add() out of if/else * "mm/memory: ignore dirty/accessed/soft-dirty bits in folio_pte_batch()" -> Be more conservative with dirt/soft-dirty, let the caller specify using flags [1] https://lkml.kernel.org/r/20231220224504.646757-1-david@redhat.com [2] https://lkml.kernel.org/r/20231218105100.172635-1-ryan.roberts@arm.com [3] https://lkml.kernel.org/r/20230809083256.699513-1-david@redhat.com [4] https://lkml.kernel.org/r/20231124132626.235350-1-david@redhat.com [5] https://lkml.kernel.org/r/20231207161211.2374093-1-ryan.roberts@arm.com Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Russell King <linux@armlinux.org.uk> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Cc: Dinh Nguyen <dinguyen@kernel.org> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Nicholas Piggin <npiggin@gmail.com> Cc: Christophe Leroy <christophe.leroy@csgroup.eu> Cc: "Aneesh Kumar K.V" <aneesh.kumar@kernel.org> Cc: "Naveen N. Rao" <naveen.n.rao@linux.ibm.com> Cc: Paul Walmsley <paul.walmsley@sifive.com> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Alexander Gordeev <agordeev@linux.ibm.com> Cc: Gerald Schaefer <gerald.schaefer@linux.ibm.com> Cc: Heiko Carstens <hca@linux.ibm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Sven Schnelle <svens@linux.ibm.com> Cc: "David S. Miller" <davem@davemloft.net> Cc: linux-arm-kernel@lists.infradead.org Cc: linuxppc-dev@lists.ozlabs.org Cc: linux-riscv@lists.infradead.org Cc: linux-s390@vger.kernel.org Cc: sparclinux@vger.kernel.org --- Andrew asked for a resend based on latest mm-unstable. I am sending this out earlier than I would usually have sent out the next version, so we can pull it into mm-unstable again now that v1 was dropped. David Hildenbrand (14): arm/pgtable: define PFN_PTE_SHIFT nios2/pgtable: define PFN_PTE_SHIFT powerpc/pgtable: define PFN_PTE_SHIFT riscv/pgtable: define PFN_PTE_SHIFT s390/pgtable: define PFN_PTE_SHIFT sparc/pgtable: define PFN_PTE_SHIFT mm/pgtable: make pte_next_pfn() independent of set_ptes() arm/mm: use pte_next_pfn() in set_ptes() powerpc/mm: use pte_next_pfn() in set_ptes() mm/memory: factor out copying the actual PTE in copy_present_pte() mm/memory: pass PTE to copy_present_pte() mm/memory: optimize fork() with PTE-mapped THP mm/memory: ignore dirty/accessed/soft-dirty bits in folio_pte_batch() mm/memory: ignore writable bit in folio_pte_batch() Ryan Roberts (1): arm64/mm: Make set_ptes() robust when OAs cross 48-bit boundary arch/arm/include/asm/pgtable.h | 2 + arch/arm/mm/mmu.c | 2 +- arch/arm64/include/asm/pgtable.h | 28 ++-- arch/nios2/include/asm/pgtable.h | 2 + arch/powerpc/include/asm/pgtable.h | 2 + arch/powerpc/mm/pgtable.c | 5 +- arch/riscv/include/asm/pgtable.h | 2 + arch/s390/include/asm/pgtable.h | 2 + arch/sparc/include/asm/pgtable_64.h | 2 + include/linux/pgtable.h | 33 ++++- mm/memory.c | 212 ++++++++++++++++++++++------ 11 files changed, 229 insertions(+), 63 deletions(-) base-commit: d162e170f1181b4305494843e1976584ddf2b72e