Message ID | 1346579549-25122-1-git-send-email-xiaoqiangnk@gmail.com |
---|---|
State | Superseded, archived |
Headers | show |
Hi Ted, I noticed that you send out the patch enabling resizing on filesystems with meta_bg and 64bits, so I send out the corresponding patch on e2fsprogs. Unfortunately, there is still a bug in e2fsprogs as my last email says. Thanks, Yongqiang. On Sun, Sep 2, 2012 at 5:52 PM, Yongqiang Yang <xiaoqiangnk@gmail.com> wrote: > This patch supports online-resizing for meata_bg and 64bits features. > > Signed-off-by: Yongqiang Yang <xiaoqiangnk@gmail.com> > --- > resize/main.c | 5 +++-- > resize/online.c | 6 +++--- > 2 files changed, 6 insertions(+), 5 deletions(-) > > diff --git a/resize/main.c b/resize/main.c > index ffefe01..b3a7840 100644 > --- a/resize/main.c > +++ b/resize/main.c > @@ -324,9 +324,10 @@ int main (int argc, char ** argv) > * This is something we should fix eventually, though. > */ > if ((fs->super->s_feature_incompat & EXT4_FEATURE_INCOMPAT_FLEX_BG) && > - !(fs->super->s_feature_compat & EXT2_FEATURE_COMPAT_RESIZE_INODE)) { > + !(fs->super->s_feature_compat & EXT2_FEATURE_COMPAT_RESIZE_INODE) && > + !(fs->super->s_feature_incompat & EXT2_FEATURE_INCOMPAT_META_BG)) { > com_err(program_name, 0, _("%s: The combination of flex_bg " > - "and\n\t!resize_inode features " > + "and\n\t!resize_inode and !meta_bg features " > "is not supported by resize2fs.\n"), > device_name); > exit(1); > diff --git a/resize/online.c b/resize/online.c > index 966ea1e..e7891bd 100644 > --- a/resize/online.c > +++ b/resize/online.c > @@ -56,9 +56,9 @@ errcode_t online_resize_fs(ext2_filsys fs, const char *mtpt, > EXT2_DESC_PER_BLOCK(fs->super)); > printf("old_desc_blocks = %lu, new_desc_blocks = %lu\n", > fs->desc_blocks, new_desc_blocks); > - if (!(fs->super->s_feature_compat & > - EXT2_FEATURE_COMPAT_RESIZE_INODE) && > - new_desc_blocks != fs->desc_blocks) { > + if (!(fs->super->s_feature_compat & EXT2_FEATURE_COMPAT_RESIZE_INODE) && > + new_desc_blocks != fs->desc_blocks && > + !(fs->super->s_feature_incompat & EXT2_FEATURE_INCOMPAT_META_BG)) { > com_err(program_name, 0, > _("Filesystem does not support online resizing")); > exit(1); > -- > 1.7.5.1 >
On Sun, Sep 02, 2012 at 06:31:19PM +0800, Yongqiang Yang wrote: > Hi Ted, > > I noticed that you send out the patch enabling resizing on > filesystems with meta_bg and 64bits, so I send out the corresponding > patch on e2fsprogs. Unfortunately, there is still a bug in e2fsprogs > as my last email says. Great, thanks, I'll take a look at it. I've made some changes of my own in resize2fs to enable using meta_bg && flex_bg && !resize_inode, but it isn't fully done yet, so I hadn't sent them out. (I must have missed your initial patch to resize2fs to enable things for testing purposes, but it's not that hard.) I'm guessing you must have only tested relatively small increases in the file system size when you were developing your kernel patch set. I was doing tests using a 1k block size with "mke2fs -t ext4 -b 1024 -O meta_bg,64bit,^resize_inode /dev/vdd 256000" and then growing the file system from 243M to 5G, which increased the number of meta_bg's from 2 to 40 meta_bg's. I also tried using the MKE2FS_FIRST_META_BG environment variable to emulate a file system which was originally grown using resize_inode scheme, and then later converted over to the meta_bg scheme. All of this flushed out a number of bugs in your original patch series, which I've fixed up. Anyway, the reason why I haven't released the changes I had made to resize2fs is I know there are still some fixes needed so that we can transition a file system from resize_inode to meta_bg once we run out of reserved gdt blocks, and I need to test what happens with file systems that are greater than 16TB --- so far, my testing has been with smaller file systems, and since I was still finding things to fix, I haven't progressed yet to testing really big file systems. We will probably also want to advertise a flag indicating that the kernel will support meta_bg resizing, and then allow mke2fs to use meta_bg by default (even for file systems < 16TB) since it has much less overhead and doesn't have the fixed limits that the original resize_inode has. So there is still some work that we need to do to improve the online resizing, but what we have so far is a very good start, and your patches definitely helped to push us forward. My apologies for not having the time to really seriously finish things up until now, but hopefully we'll be able to get something fully robust and tested soon (realistically speaking, looking at the likely release schedules for the kernel and e2fsprogs, I'm shooting for getting this all done before the end of the year). Regards, - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, Sep 02, 2012 at 06:31:19PM +0800, Yongqiang Yang wrote: > Hi Ted, > > I noticed that you send out the patch enabling resizing on > filesystems with meta_bg and 64bits, so I send out the corresponding > patch on e2fsprogs. Unfortunately, there is still a bug in e2fsprogs > as my last email says. The following two patches were sufficient for me to resize a test file system from 14 TB to 18 TB. We still need to handle automatic enabling of the meta_bg flag in resize2fs, but this should be sufficient for more adventurous folks to test the online resize patches. (Note that I also posted a v2 version of patch 5/5 which fixed a number of 64-bit overflow bugs that I found in my testing.) - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Am 03.09.2012 18:45, schrieb Theodore Ts'o: > On Sun, Sep 02, 2012 at 06:31:19PM +0800, Yongqiang Yang wrote: >> Hi Ted, >> >> I noticed that you send out the patch enabling resizing on >> filesystems with meta_bg and 64bits, so I send out the corresponding >> patch on e2fsprogs. Unfortunately, there is still a bug in e2fsprogs >> as my last email says. > The following two patches were sufficient for me to resize a test file > system from 14 TB to 18 TB. We still need to handle automatic > enabling of the meta_bg flag in resize2fs, but this should be > sufficient for more adventurous folks to test the online resize > patches. (Note that I also posted a v2 version of patch 5/5 which > fixed a number of 64-bit overflow bugs that I found in my testing.) > > - Ted Hi Ted, is offline resizeing possible with this patch without online resize kernel patches (i use 3.2.0) with this features? Filesystem features: has_journal ext_attr dir_index filetype needs_recovery meta_bg extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize I had no luck offline resizing my 24TB fs with e2fstools version 1.42 and 1.42.5. thx for your time. Regards, Kai Grosshaus -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Sep 03, 2012 at 11:02:17PM +0200, Kai Grosshaus wrote: > > is offline resizeing possible with this patch without online resize > kernel patches (i use 3.2.0) with this features? > > Filesystem features: has_journal ext_attr dir_index filetype > needs_recovery meta_bg extent 64bit flex_bg sparse_super large_file > huge_file uninit_bg dir_nlink extra_isize > > I had no luck offline resizing my 24TB fs with e2fstools version > 1.42 and 1.42.5. No, offline resizing for 64-bit file systems (and file systems with meta_bg) hasn't been implemented yet. It's on the todo list, but the main issue is that for many of the ext4 developers, our employers don't require the use of large RAID arrays, so the development has been somewhat delayed between my work responsibilities and organizing the kernel summit last week in San Diego. Yongqiang's patches were posted for review quite some time ago, but I knew they needed some review and reworking, and it just didn't rise to the top of the priority heap until now. (I'm technically on vacation this week in Santa Fe, and hacking online resizing for ext4 is fun, so I've been using the early morning and late nights to finally get around to making forward progress for online resize. :-) Technically, off-line shrink for meta_bg/64-bit file systems is easy, and off-line growth isn't fundamentally hard, but it simply requires teaching resize2fs about the new meta_bg and flex_bg formats. It's probably only a week or so, full-time, by someone who really understands the code in question. The only problem is the only person who understands that code deeply is me, and I've just been completely swamped these last couple of months. Hopefully, now that the kernel summit is behind me, I'll have a bit more time to work on off-line resize, as well as reviewing the many other patches for ext4 which have been submitted in the past few months (i.e., the extent status tree, which enables SEEK_HOLE and fixing the bigalloc ENOSPC problem with delalloc bufferred writes, and the inline data patches). I suspect that for people who really want to be able to grow their large 64-bit ext4 file systems, the best path forward will be for me to backport the on-line resizing (for enlarging 64-bit file systems) patches to 3.2.x, once I'm convinced we've shaken out all of the bugs and they are well-tested. Technically, they wouldn't be material for the 3.2.x stable branch since they're not a bugfix, but if some distro using 3.2 is willing to sponsor them, we could make the case that since one or more distro's are willing to use them in shipping product, they should be in 3.2.x. (Or I could just abuse the fact that no one is currently actively enforcing rules around the criteria for stable tree patches, but that would just be bad kernel developer citizenship. :-) Regards, - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 04.09.2012 01:00, Theodore Ts'o wrote: > On Mon, Sep 03, 2012 at 11:02:17PM +0200, Kai Grosshaus wrote: >> is offline resizeing possible with this patch without online resize >> kernel patches (i use 3.2.0) with this features? >> >> Filesystem features: has_journal ext_attr dir_index filetype >> needs_recovery meta_bg extent 64bit flex_bg sparse_super large_file >> huge_file uninit_bg dir_nlink extra_isize >> >> I had no luck offline resizing my 24TB fs with e2fstools version >> 1.42 and 1.42.5. > No, offline resizing for 64-bit file systems (and file systems with > meta_bg) hasn't been implemented yet. It's on the todo list, but the > main issue is that for many of the ext4 developers, our employers > don't require the use of large RAID arrays, so the development has > been somewhat delayed between my work responsibilities and organizing > the kernel summit last week in San Diego. Yongqiang's patches were > posted for review quite some time ago, but I knew they needed some > review and reworking, and it just didn't rise to the top of the > priority heap until now. > > (I'm technically on vacation this week in Santa Fe, and hacking online > resizing for ext4 is fun, so I've been using the early morning and > late nights to finally get around to making forward progress for > online resize. :-) interesting what you call fun ;) But thx, i appreciate your work! > > Technically, off-line shrink for meta_bg/64-bit file systems is easy, > and off-line growth isn't fundamentally hard, but it simply requires > teaching resize2fs about the new meta_bg and flex_bg formats. It's > probably only a week or so, full-time, by someone who really > understands the code in question. The only problem is the only person > who understands that code deeply is me, and I've just been completely > swamped these last couple of months. Hopefully, now that the kernel > summit is behind me, I'll have a bit more time to work on off-line > resize, as well as reviewing the many other patches for ext4 which > have been submitted in the past few months (i.e., the extent status > tree, which enables SEEK_HOLE and fixing the bigalloc ENOSPC problem > with delalloc bufferred writes, and the inline data patches). > > I suspect that for people who really want to be able to grow their > large 64-bit ext4 file systems, the best path forward will be for me > to backport the on-line resizing (for enlarging 64-bit file systems) > patches to 3.2.x, once I'm convinced we've shaken out all of the bugs > and they are well-tested. > > Technically, they wouldn't be material for the 3.2.x stable branch > since they're not a bugfix, but if some distro using 3.2 is willing to > sponsor them, we could make the case that since one or more distro's > are willing to use them in shipping product, they should be in 3.2.x. > (Or I could just abuse the fact that no one is currently actively > enforcing rules around the criteria for stable tree patches, but that > would just be bad kernel developer citizenship. :-) I didn't want to press on it, but thx for the detailed response. Ill stay tuned here to read about the progress. But could you do me the favor and answer me an additional question. If i create a fs with the kernel, e2fsprog versions and the features i mentioned, could i grow it in future when the work on e2fsprogs is done. I mean recreating the fs is no option for me ;) thx a lot and have a good vacation, Regards, Kai Grosshaus -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Sep 4, 2012 at 12:45 AM, Theodore Ts'o <tytso@mit.edu> wrote: > On Sun, Sep 02, 2012 at 06:31:19PM +0800, Yongqiang Yang wrote: >> Hi Ted, >> >> I noticed that you send out the patch enabling resizing on >> filesystems with meta_bg and 64bits, so I send out the corresponding >> patch on e2fsprogs. Unfortunately, there is still a bug in e2fsprogs >> as my last email says. > > The following two patches were sufficient for me to resize a test file > system from 14 TB to 18 TB. We still need to handle automatic > enabling of the meta_bg flag in resize2fs, but this should be > sufficient for more adventurous folks to test the online resize > patches. (Note that I also posted a v2 version of patch 5/5 which > fixed a number of 64-bit overflow bugs that I found in my testing.) Great! I have emailed Kevin Liao who tested my patches before to have a try. Thanks, Yongqiang. > > - Ted
diff --git a/resize/main.c b/resize/main.c index ffefe01..b3a7840 100644 --- a/resize/main.c +++ b/resize/main.c @@ -324,9 +324,10 @@ int main (int argc, char ** argv) * This is something we should fix eventually, though. */ if ((fs->super->s_feature_incompat & EXT4_FEATURE_INCOMPAT_FLEX_BG) && - !(fs->super->s_feature_compat & EXT2_FEATURE_COMPAT_RESIZE_INODE)) { + !(fs->super->s_feature_compat & EXT2_FEATURE_COMPAT_RESIZE_INODE) && + !(fs->super->s_feature_incompat & EXT2_FEATURE_INCOMPAT_META_BG)) { com_err(program_name, 0, _("%s: The combination of flex_bg " - "and\n\t!resize_inode features " + "and\n\t!resize_inode and !meta_bg features " "is not supported by resize2fs.\n"), device_name); exit(1); diff --git a/resize/online.c b/resize/online.c index 966ea1e..e7891bd 100644 --- a/resize/online.c +++ b/resize/online.c @@ -56,9 +56,9 @@ errcode_t online_resize_fs(ext2_filsys fs, const char *mtpt, EXT2_DESC_PER_BLOCK(fs->super)); printf("old_desc_blocks = %lu, new_desc_blocks = %lu\n", fs->desc_blocks, new_desc_blocks); - if (!(fs->super->s_feature_compat & - EXT2_FEATURE_COMPAT_RESIZE_INODE) && - new_desc_blocks != fs->desc_blocks) { + if (!(fs->super->s_feature_compat & EXT2_FEATURE_COMPAT_RESIZE_INODE) && + new_desc_blocks != fs->desc_blocks && + !(fs->super->s_feature_incompat & EXT2_FEATURE_INCOMPAT_META_BG)) { com_err(program_name, 0, _("Filesystem does not support online resizing")); exit(1);
This patch supports online-resizing for meata_bg and 64bits features. Signed-off-by: Yongqiang Yang <xiaoqiangnk@gmail.com> --- resize/main.c | 5 +++-- resize/online.c | 6 +++--- 2 files changed, 6 insertions(+), 5 deletions(-)