| Message ID | 20211102122945.117744-5-agruenba@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | gfs2: Fix mmap + page fault deadlocks | expand |
On Tue, Nov 02, 2021 at 01:29:32PM +0100, Andreas Gruenbacher wrote: > Turn iov_iter_fault_in_readable into a function that returns the number > of bytes not faulted in, similar to copy_to_user, instead of returning a > non-zero value when any of the requested pages couldn't be faulted in. > This supports the existing users that require all pages to be faulted in > as well as new users that are happy if any pages can be faulted in. > > Rename iov_iter_fault_in_readable to fault_in_iov_iter_readable to make > sure this change doesn't silently break things. > > Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com> [...] > diff --git a/mm/filemap.c b/mm/filemap.c > index ff34f4087f87..4dd5edcd39fd 100644 > --- a/mm/filemap.c > +++ b/mm/filemap.c > @@ -3757,7 +3757,7 @@ ssize_t generic_perform_write(struct file *file, > * same page as we're writing to, without it being marked > * up-to-date. > */ > - if (unlikely(iov_iter_fault_in_readable(i, bytes))) { > + if (unlikely(fault_in_iov_iter_readable(i, bytes))) { > status = -EFAULT; > break; > } Now that fault_in_iov_iter_readable() returns the number of bytes, we could change the above test to: if (unlikely(fault_in_iov_iter_readable(i, bytes) == bytes)) { Assuming we have a pointer 'a', accessible, and 'a + PAGE_SIZE' unmapped: write(fd, a + PAGE_SIZE - 1, 2); can still copy one byte but it returns -EFAULT instead since the second page is not accessible. While writing some test-cases for MTE (sub-page faults, 16-byte granularity), we noticed that reading 2 bytes from 'a + 15' with 'a + 16' tagged for faulting: write(fd, a + 15, 2); succeeds as long as 'a + 16' is not at a page boundary. Checking against 'bytes' above makes this consistent. The downside is that it's an ABI change though not sure anyone is relying on it.
On Thu, Nov 4, 2021 at 7:22 PM Catalin Marinas <catalin.marinas@arm.com> wrote: > On Tue, Nov 02, 2021 at 01:29:32PM +0100, Andreas Gruenbacher wrote: > > Turn iov_iter_fault_in_readable into a function that returns the number > > of bytes not faulted in, similar to copy_to_user, instead of returning a > > non-zero value when any of the requested pages couldn't be faulted in. > > This supports the existing users that require all pages to be faulted in > > as well as new users that are happy if any pages can be faulted in. > > > > Rename iov_iter_fault_in_readable to fault_in_iov_iter_readable to make > > sure this change doesn't silently break things. > > > > Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com> > [...] > > diff --git a/mm/filemap.c b/mm/filemap.c > > index ff34f4087f87..4dd5edcd39fd 100644 > > --- a/mm/filemap.c > > +++ b/mm/filemap.c > > @@ -3757,7 +3757,7 @@ ssize_t generic_perform_write(struct file *file, > > * same page as we're writing to, without it being marked > > * up-to-date. > > */ > > - if (unlikely(iov_iter_fault_in_readable(i, bytes))) { > > + if (unlikely(fault_in_iov_iter_readable(i, bytes))) { > > status = -EFAULT; > > break; > > } > > Now that fault_in_iov_iter_readable() returns the number of bytes, we > could change the above test to: > > if (unlikely(fault_in_iov_iter_readable(i, bytes) == bytes)) { > > Assuming we have a pointer 'a', accessible, and 'a + PAGE_SIZE' unmapped: > > write(fd, a + PAGE_SIZE - 1, 2); > > can still copy one byte but it returns -EFAULT instead since the second > page is not accessible. > > While writing some test-cases for MTE (sub-page faults, 16-byte > granularity), we noticed that reading 2 bytes from 'a + 15' with > 'a + 16' tagged for faulting: > > write(fd, a + 15, 2); > > succeeds as long as 'a + 16' is not at a page boundary. Checking against > 'bytes' above makes this consistent. > > The downside is that it's an ABI change though not sure anyone is > relying on it. The same pattern exists in iomap_write_iter too, of course. In the very light testing I did for eliminating the pre-faulting, this kind of change was working fine. I have no performance numbers though. https://lore.kernel.org/linux-fsdevel/20211026094430.3669156-1-agruenba@redhat.com/ https://lore.kernel.org/linux-fsdevel/20211027212138.3722977-1-agruenba@redhat.com/ Thanks, Andreas
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c index 7ff577005d0f..f37211d3bb69 100644 --- a/fs/btrfs/file.c +++ b/fs/btrfs/file.c @@ -1710,7 +1710,7 @@ static noinline ssize_t btrfs_buffered_write(struct kiocb *iocb, * Fault pages before locking them in prepare_pages * to avoid recursive lock */ - if (unlikely(iov_iter_fault_in_readable(i, write_bytes))) { + if (unlikely(fault_in_iov_iter_readable(i, write_bytes))) { ret = -EFAULT; break; } diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c index 9c8ef33bd8d3..eb971e1e7227 100644 --- a/fs/f2fs/file.c +++ b/fs/f2fs/file.c @@ -4276,7 +4276,7 @@ static ssize_t f2fs_file_write_iter(struct kiocb *iocb, struct iov_iter *from) size_t target_size = 0; int err; - if (iov_iter_fault_in_readable(from, iov_iter_count(from))) + if (fault_in_iov_iter_readable(from, iov_iter_count(from))) set_inode_flag(inode, FI_NO_PREALLOC); if ((iocb->ki_flags & IOCB_NOWAIT)) { diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 11404f8c21c7..4b6d8e13322d 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1164,7 +1164,7 @@ static ssize_t fuse_fill_write_pages(struct fuse_io_args *ia, again: err = -EFAULT; - if (iov_iter_fault_in_readable(ii, bytes)) + if (fault_in_iov_iter_readable(ii, bytes)) break; err = -ENOMEM; diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 9cc5798423d1..1753c26c8e76 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -750,7 +750,7 @@ static loff_t iomap_write_iter(struct iomap_iter *iter, struct iov_iter *i) * same page as we're writing to, without it being marked * up-to-date. */ - if (unlikely(iov_iter_fault_in_readable(i, bytes))) { + if (unlikely(fault_in_iov_iter_readable(i, bytes))) { status = -EFAULT; break; } diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c index ab4f3362466d..a43adeacd930 100644 --- a/fs/ntfs/file.c +++ b/fs/ntfs/file.c @@ -1829,7 +1829,7 @@ static ssize_t ntfs_perform_write(struct file *file, struct iov_iter *i, * pages being swapped out between us bringing them into memory * and doing the actual copying. */ - if (unlikely(iov_iter_fault_in_readable(i, bytes))) { + if (unlikely(fault_in_iov_iter_readable(i, bytes))) { status = -EFAULT; break; } diff --git a/fs/ntfs3/file.c b/fs/ntfs3/file.c index 424450e77ad5..a52388387175 100644 --- a/fs/ntfs3/file.c +++ b/fs/ntfs3/file.c @@ -987,7 +987,7 @@ static ssize_t ntfs_compress_write(struct kiocb *iocb, struct iov_iter *from) frame_vbo = pos & ~(frame_size - 1); index = frame_vbo >> PAGE_SHIFT; - if (unlikely(iov_iter_fault_in_readable(from, bytes))) { + if (unlikely(fault_in_iov_iter_readable(from, bytes))) { err = -EFAULT; goto out; } diff --git a/include/linux/uio.h b/include/linux/uio.h index 207101a9c5c3..d18458af6681 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -133,7 +133,7 @@ size_t copy_page_from_iter_atomic(struct page *page, unsigned offset, size_t bytes, struct iov_iter *i); void iov_iter_advance(struct iov_iter *i, size_t bytes); void iov_iter_revert(struct iov_iter *i, size_t bytes); -int iov_iter_fault_in_readable(const struct iov_iter *i, size_t bytes); +size_t fault_in_iov_iter_readable(const struct iov_iter *i, size_t bytes); size_t iov_iter_single_seg_count(const struct iov_iter *i); size_t copy_page_to_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i); diff --git a/lib/iov_iter.c b/lib/iov_iter.c index c88908f0f138..ce3d4f610626 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -430,33 +430,42 @@ static size_t copy_page_to_iter_pipe(struct page *page, size_t offset, size_t by } /* + * fault_in_iov_iter_readable - fault in iov iterator for reading + * @i: iterator + * @size: maximum length + * * Fault in one or more iovecs of the given iov_iter, to a maximum length of - * bytes. For each iovec, fault in each page that constitutes the iovec. + * @size. For each iovec, fault in each page that constitutes the iovec. + * + * Returns the number of bytes not faulted in (like copy_to_user() and + * copy_from_user()). * - * Return 0 on success, or non-zero if the memory could not be accessed (i.e. - * because it is an invalid address). + * Always returns 0 for non-userspace iterators. */ -int iov_iter_fault_in_readable(const struct iov_iter *i, size_t bytes) +size_t fault_in_iov_iter_readable(const struct iov_iter *i, size_t size) { if (iter_is_iovec(i)) { + size_t count = min(size, iov_iter_count(i)); const struct iovec *p; size_t skip; - if (bytes > i->count) - bytes = i->count; - for (p = i->iov, skip = i->iov_offset; bytes; p++, skip = 0) { - size_t len = min(bytes, p->iov_len - skip); + size -= count; + for (p = i->iov, skip = i->iov_offset; count; p++, skip = 0) { + size_t len = min(count, p->iov_len - skip); + size_t ret; if (unlikely(!len)) continue; - if (fault_in_readable(p->iov_base + skip, len)) - return -EFAULT; - bytes -= len; + ret = fault_in_readable(p->iov_base + skip, len); + count -= len - ret; + if (ret) + break; } + return count + size; } return 0; } -EXPORT_SYMBOL(iov_iter_fault_in_readable); +EXPORT_SYMBOL(fault_in_iov_iter_readable); void iov_iter_init(struct iov_iter *i, unsigned int direction, const struct iovec *iov, unsigned long nr_segs, diff --git a/mm/filemap.c b/mm/filemap.c index ff34f4087f87..4dd5edcd39fd 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3757,7 +3757,7 @@ ssize_t generic_perform_write(struct file *file, * same page as we're writing to, without it being marked * up-to-date. */ - if (unlikely(iov_iter_fault_in_readable(i, bytes))) { + if (unlikely(fault_in_iov_iter_readable(i, bytes))) { status = -EFAULT; break; }
Turn iov_iter_fault_in_readable into a function that returns the number of bytes not faulted in, similar to copy_to_user, instead of returning a non-zero value when any of the requested pages couldn't be faulted in. This supports the existing users that require all pages to be faulted in as well as new users that are happy if any pages can be faulted in. Rename iov_iter_fault_in_readable to fault_in_iov_iter_readable to make sure this change doesn't silently break things. Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com> --- fs/btrfs/file.c | 2 +- fs/f2fs/file.c | 2 +- fs/fuse/file.c | 2 +- fs/iomap/buffered-io.c | 2 +- fs/ntfs/file.c | 2 +- fs/ntfs3/file.c | 2 +- include/linux/uio.h | 2 +- lib/iov_iter.c | 33 +++++++++++++++++++++------------ mm/filemap.c | 2 +- 9 files changed, 29 insertions(+), 20 deletions(-)