Message ID | ce6bff3e-49f4-2bed-67ce-a7be604ce1cf@redhat.com |
---|---|
State | Superseded, archived |
Headers | show |
On Jul 18, 2017, at 3:10 PM, Eric Sandeen <sandeen@redhat.com> wrote: > > Time & mount-count based checks have been off by default for quite some > time now, but the dire warning about disabling them remains in the > tune2fs manpage, which is confusing. We did "strongly consider > the consequences" and disabled it by default, no need to scare the > user about it now. Sigh, I still think this is going in the wrong direction. I'm happily running a weekly e2fsck on a snapshot of the filesystem, and then reset the time and mount-count fields in the superblock with tune2fs. That way I never see any warnings, or have slow boots because of a scan, but I'm also notified if there are ever problems on the filesystem (which happens occasionally, since I'm sometimes running experimental code). Since virtually everyone is using MD/LVM devices these days, I don't think that is hard to do. I offered up my "lvcheck" script a few times, but nobody at RH or on the DM team seemed interested at the time... I'd also be happy if there was some other similar mechanism included with the distro to do periodic background checks of the filesystem, rather than letting them find any problem at some random time. This is pretty standard for RAID systems, I think it makes sense for the filesystem too. Cheers, Andreas > Signed-off-by: Eric Sandeen <sandeen@redhat.com> > --- > > diff --git a/misc/tune2fs.8.in b/misc/tune2fs.8.in > index 5c885f9..a8cacc7 100644 > --- a/misc/tune2fs.8.in > +++ b/misc/tune2fs.8.in > @@ -134,17 +134,6 @@ Staggering the mount-counts at which filesystems are forcibly > checked will avoid all filesystems being checked at one time > when using journaled filesystems. > .sp > -You should strongly consider the consequences of disabling > -mount-count-dependent checking entirely. Bad disk drives, cables, > -memory, and kernel bugs could all corrupt a filesystem without > -marking the filesystem dirty or in error. If you are using > -journaling on your filesystem, your filesystem will > -.B never > -be marked dirty, so it will not normally be checked. A > -filesystem error detected by the kernel will still force > -an fsck on the next reboot, but it may already be too late > -to prevent data loss at that point. > -.sp > See also the > .B \-i > option for time-dependent checking. > Cheers, Andreas
On Tue, Jul 18, 2017 at 04:28:16PM -0600, Andreas Dilger wrote: > > Sigh, I still think this is going in the wrong direction. I'm happily > running a weekly e2fsck on a snapshot of the filesystem, and then reset > the time and mount-count fields in the superblock with tune2fs. That > way I never see any warnings, or have slow boots because of a scan, but > I'm also notified if there are ever problems on the filesystem (which > happens occasionally, since I'm sometimes running experimental code). > > Since virtually everyone is using MD/LVM devices these days, I don't > think that is hard to do. I offered up my "lvcheck" script a few times, > but nobody at RH or on the DM team seemed interested at the time... > I'd also be happy if there was some other similar mechanism included with > the distro to do periodic background checks of the filesystem, rather > than letting them find any problem at some random time. This is pretty > standard for RAID systems, I think it makes sense for the filesystem too. I've had e2croncheck in the contrib directory for a long time. I suspect it wouldn't be that hard to make a version of it which scans /proc/mounts, and for those devices that are in an LVM, or dm-thin, and if there is room for a snapshot, it would create a snapshot, run fsck on the snapshot, and if there are any errors, sends an e-mail report to root by default. (We would need to have some kind of configuration file in /etc to control where to send the reports, what the default snapshot size should be, etc., but if we have intelligent defaults than the config file could be optional.) We could try to make it a bit nicer, and then move it to the misc directory and start installing it by default with "make install". That might make it easier for more users to set it up. Maybe some distros will even decide to install a crontab entry by default. - Ted
On 07/18/2017 05:28 PM, Andreas Dilger wrote: > On Jul 18, 2017, at 3:10 PM, Eric Sandeen <sandeen@redhat.com> wrote: >> >> Time & mount-count based checks have been off by default for quite some >> time now, but the dire warning about disabling them remains in the >> tune2fs manpage, which is confusing. We did "strongly consider >> the consequences" and disabled it by default, no need to scare the >> user about it now. > > Sigh, I still think this is going in the wrong direction. Well, the upstream defaults have been not-check for /years/ now, this just makes the docs match reality. > I'm happily > running a weekly e2fsck on a snapshot of the filesystem, and then reset > the time and mount-count fields in the superblock with tune2fs. That > way I never see any warnings, or have slow boots because of a scan, but > I'm also notified if there are ever problems on the filesystem (which > happens occasionally, since I'm sometimes running experimental code). *nod* it's a nice mechanism. > Since virtually everyone is using MD/LVM devices these days, I don't > think that is hard to do. I offered up my "lvcheck" script a few times, > but nobody at RH or on the DM team seemed interested at the time... No, I think it's great. It needs to go into some package, somewhere, and not just float around on the internet ... e2sfprogs comes to mind. or util-linux, or lvm-tools, or whatever... ;) Send a proper patch to the appropriate package maintainer, and it'll get into fedora and every other distro. > I'd also be happy if there was some other similar mechanism included with > the distro to do periodic background checks of the filesystem, rather > than letting them find any problem at some random time. This is pretty > standard for RAID systems, I think it makes sense for the filesystem too. well, tbh "every 27th boot" was pretty random, too, in practice. ;) Ok, I see ted pointed out e2croncheck, too - and yes, actually installing it and dropping someting in cron.d would complete the circle, to get it out of the some-assembly-required mode. -Eric > Cheers, Andreas > >> Signed-off-by: Eric Sandeen <sandeen@redhat.com> >> --- >> >> diff --git a/misc/tune2fs.8.in b/misc/tune2fs.8.in >> index 5c885f9..a8cacc7 100644 >> --- a/misc/tune2fs.8.in >> +++ b/misc/tune2fs.8.in >> @@ -134,17 +134,6 @@ Staggering the mount-counts at which filesystems are forcibly >> checked will avoid all filesystems being checked at one time >> when using journaled filesystems. >> .sp >> -You should strongly consider the consequences of disabling >> -mount-count-dependent checking entirely. Bad disk drives, cables, >> -memory, and kernel bugs could all corrupt a filesystem without >> -marking the filesystem dirty or in error. If you are using >> -journaling on your filesystem, your filesystem will >> -.B never >> -be marked dirty, so it will not normally be checked. A >> -filesystem error detected by the kernel will still force >> -an fsck on the next reboot, but it may already be too late >> -to prevent data loss at that point. >> -.sp >> See also the >> .B \-i >> option for time-dependent checking. >> > > > Cheers, Andreas > > > >
On Tue, Jul 18, 2017 at 09:15:17PM -0400, Theodore Ts'o wrote: > On Tue, Jul 18, 2017 at 04:28:16PM -0600, Andreas Dilger wrote: > > > > Sigh, I still think this is going in the wrong direction. I'm happily > > running a weekly e2fsck on a snapshot of the filesystem, and then reset > > the time and mount-count fields in the superblock with tune2fs. That > > way I never see any warnings, or have slow boots because of a scan, but > > I'm also notified if there are ever problems on the filesystem (which > > happens occasionally, since I'm sometimes running experimental code). > > > > Since virtually everyone is using MD/LVM devices these days, I don't > > think that is hard to do. I offered up my "lvcheck" script a few times, > > but nobody at RH or on the DM team seemed interested at the time... > > I'd also be happy if there was some other similar mechanism included with > > the distro to do periodic background checks of the filesystem, rather > > than letting them find any problem at some random time. This is pretty > > standard for RAID systems, I think it makes sense for the filesystem too. > > I've had e2croncheck in the contrib directory for a long time. I > suspect it wouldn't be that hard to make a version of it which scans > /proc/mounts, and for those devices that are in an LVM, or dm-thin, > and if there is room for a snapshot, it would create a snapshot, run > fsck on the snapshot, and if there are any errors, sends an e-mail > report to root by default. (We would need to have some kind of > configuration file in /etc to control where to send the reports, what > the default snapshot size should be, etc., but if we have intelligent > defaults than the config file could be optional.) > > We could try to make it a bit nicer, and then move it to the misc > directory and start installing it by default with "make install". > That might make it easier for more users to set it up. Maybe some > distros will even decide to install a crontab entry by default. > > - Ted I am actually worried that with this approach we are, simply by adding complexity, making situation worse than just not running periodic e2fsck. What we should be aiming for I think is the online file system check and scrub. This would of course not replace the need rof e2fsck, but we would be able to catch errors early while fixing some of those that we can. But that's long term. Short term I think we're better off without this snapshotting/checking complexity. Those who are concerned can still enable the time/mount based checks right ? -Lukas
On Tue, Jul 18, 2017 at 04:10:49PM -0500, Eric Sandeen wrote: > Time & mount-count based checks have been off by default for quite some > time now, but the dire warning about disabling them remains in the > tune2fs manpage, which is confusing. We did "strongly consider > the consequences" and disabled it by default, no need to scare the > user about it now. > > Signed-off-by: Eric Sandeen <sandeen@redhat.com> > --- > > diff --git a/misc/tune2fs.8.in b/misc/tune2fs.8.in > index 5c885f9..a8cacc7 100644 > --- a/misc/tune2fs.8.in > +++ b/misc/tune2fs.8.in > @@ -134,17 +134,6 @@ Staggering the mount-counts at which filesystems are forcibly > checked will avoid all filesystems being checked at one time > when using journaled filesystems. > .sp > -You should strongly consider the consequences of disabling > -mount-count-dependent checking entirely. Bad disk drives, cables, > -memory, and kernel bugs could all corrupt a filesystem without > -marking the filesystem dirty or in error. If you are using > -journaling on your filesystem, your filesystem will > -.B never > -be marked dirty, so it will not normally be checked. A > -filesystem error detected by the kernel will still force > -an fsck on the next reboot, but it may already be too late > -to prevent data loss at that point. > -.sp > See also the > .B \-i > option for time-dependent checking. > There is one more paragraph about this in the section about -i option. Also I'd not remove it entirely, but adding a note of possible benefits of this setting as well as disadvantages. -Lukas
On Wed, Jul 19, 2017 at 09:21:57AM +0200, Lukas Czerner wrote: > I am actually worried that with this approach we are, simply by adding > complexity, making situation worse than just not running periodic > e2fsck. How would it make things worse? If you don't trust lvm or dm-thin to create a read-only snapshot, you've got **way** worse problems. I acutally think relying on e2fsck on a r/o snapshot to be much simpler than trying to add an on-line file systme check. That requires much more kernel code which almost by definition is higher risk (e.g., to bugs of the sort found by AFL) than already-existing userspace code. > What we should be aiming for I think is the online file system check and > scrub. This would of course not replace the need rof e2fsck, but we > would be able to catch errors early while fixing some of those that we > can. But that's long term. Short term I think we're better off without > this snapshotting/checking complexity. Those who are concerned can still > enable the time/mount based checks right ? time/mount-based checks only help if you reboot; the advantage of doing a check on read-only snapshot is you can schedule it once a week, or once a month, during idle times. Picking idle times might be tricky, but distro's when they decide on a default for running updatedb(8) for the locate command. And whether the crontab entry is installed by default, or has to be explicitly enabled by the user, or e2croncheck is put in a separate package for distributions to use are all distro decisions. I would probably go for the last, with a debian-style "recommends" or "suggests" dependency for easy discoverability but different distributions can do what they like --- including not packaging e2croncheck at all. But in terms of a short-term solution it's really not hard to add. And I don't believe I've heard any reports of instability for r/o snapshot functionality. That's been around for a long, long, time at least for LVM snapshots. dm-thin might be considered more flakey, but that reputation seems to apply for dm-thin as a whole, as opposed to just its snapshot functionality. If a user is willing to trust their data to dm-thin, are taking a bigger risk by using dm-thin snapshots? - Ted
On Tue, Jul 18, 2017 at 09:15:17PM -0400, Theodore Ts'o wrote: > On Tue, Jul 18, 2017 at 04:28:16PM -0600, Andreas Dilger wrote: > > > > Sigh, I still think this is going in the wrong direction. I'm happily > > running a weekly e2fsck on a snapshot of the filesystem, and then reset > > the time and mount-count fields in the superblock with tune2fs. That > > way I never see any warnings, or have slow boots because of a scan, but > > I'm also notified if there are ever problems on the filesystem (which > > happens occasionally, since I'm sometimes running experimental code). > > > > Since virtually everyone is using MD/LVM devices these days, I don't > > think that is hard to do. I offered up my "lvcheck" script a few times, > > but nobody at RH or on the DM team seemed interested at the time... > > I'd also be happy if there was some other similar mechanism included with > > the distro to do periodic background checks of the filesystem, rather > > than letting them find any problem at some random time. This is pretty > > standard for RAID systems, I think it makes sense for the filesystem too. > > I've had e2croncheck in the contrib directory for a long time. I > suspect it wouldn't be that hard to make a version of it which scans > /proc/mounts, and for those devices that are in an LVM, or dm-thin, > and if there is room for a snapshot, it would create a snapshot, run > fsck on the snapshot, and if there are any errors, sends an e-mail > report to root by default. (We would need to have some kind of > configuration file in /etc to control where to send the reports, what > the default snapshot size should be, etc., but if we have intelligent > defaults than the config file could be optional.) > > We could try to make it a bit nicer, and then move it to the misc > directory and start installing it by default with "make install". > That might make it easier for more users to set it up. Maybe some > distros will even decide to install a crontab entry by default. So... I've had a private debian package for years that does most of this. There are two scripts -- one that uses lvs and blkid to identify potential ext4 LVS and calls the second script, which sets up the snapshot, runs e2fsck on that, and (optionally) calls fstrim on the original fs if the snapshot fscks cleanly. There's also a udev rules script to discourage udev from "managing" /dev/disk/ symlinks to the fsck snapshot. Newer versions of the package integrate systemd support to (clumsily) isolate the e2fsck process, send email if things fail, and run automatically a la cron. There are some missing pieces, however -- I didn't modify d-i to reserve free space in the VG; there needs to be a monitoring daemon to kill fsck and the snapshot if the snapshot exhausts all of its space; a boot time script to kill the fsck snapshots if the system happened to go down while fsck was in progress. It also assumes that the fs is idle enough that 256M for the snapshot will be sufficient. I've never bothered to submit any of it because I haven't had the time to implement any of those missing bits. --D > > - Ted
On Wed, Jul 19, 2017 at 10:42:20AM -0400, Theodore Ts'o wrote: > On Wed, Jul 19, 2017 at 09:21:57AM +0200, Lukas Czerner wrote: > > I am actually worried that with this approach we are, simply by adding > > complexity, making situation worse than just not running periodic > > e2fsck. > > How would it make things worse? If you don't trust lvm or dm-thin to > create a read-only snapshot, you've got **way** worse problems. I > acutally think relying on e2fsck on a r/o snapshot to be much simpler > than trying to add an on-line file systme check. That requires much > more kernel code which almost by definition is higher risk (e.g., to > bugs of the sort found by AFL) than already-existing userspace code. Because by adding complexity we're introducing bugs, problems and unexpected scenarios to what's supposed to be just a caution check. I feel like the problems caused by this setup are more likely than file system problems that would be caught by this check. But maybe I did not explain myself very well. I think that the dm-thin solution to run e2fsck is great, for those that already run dm-thin and those that are aware of what it means it's a great solution. But I was under assumption that we're talking about general recommendation - that's where I see the problem. It's not that I do not trust dm-thin, or lvm. They have their problems and bugs like everyone else. Not only that, but it comes with some caveats, like unresolved ENOSPC handling, or performance problems with legacy snapshots. It only takes to run a cron job in just the right time for the user to be terribly surprised. > > > What we should be aiming for I think is the online file system check and > > scrub. This would of course not replace the need rof e2fsck, but we > > would be able to catch errors early while fixing some of those that we > > can. But that's long term. Short term I think we're better off without > > this snapshotting/checking complexity. Those who are concerned can still > > enable the time/mount based checks right ? > > time/mount-based checks only help if you reboot; the advantage of > doing a check on read-only snapshot is you can schedule it once a > week, or once a month, during idle times. Picking idle times might be > tricky, but distro's when they decide on a default for running > updatedb(8) for the locate command. And whether the crontab entry is > installed by default, or has to be explicitly enabled by the user, or > e2croncheck is put in a separate package for distributions to use are > all distro decisions. > > I would probably go for the last, with a debian-style "recommends" or > "suggests" dependency for easy discoverability but different > distributions can do what they like --- including not packaging > e2croncheck at all. But in terms of a short-term solution it's really > not hard to add. And I don't believe I've heard any reports of > instability for r/o snapshot functionality. That's been around for a > long, long, time at least for LVM snapshots. dm-thin might be > considered more flakey, but that reputation seems to apply for dm-thin > as a whole, as opposed to just its snapshot functionality. If a user > is willing to trust their data to dm-thin, are taking a bigger risk by > using dm-thin snapshots? Right, for those that already use dm-thin that's, I thing, a good solution and it's easy enough to do. Having a distribution package to install to enable this is also fine. Even though my worry about this potentially causing more problems than it sovles still applies. Again, having this be a general recommendation (as it was the case with time/mount based checks) that's what I have much bigger problem with. Thanks! -Lukas > > - Ted
diff --git a/misc/tune2fs.8.in b/misc/tune2fs.8.in index 5c885f9..a8cacc7 100644 --- a/misc/tune2fs.8.in +++ b/misc/tune2fs.8.in @@ -134,17 +134,6 @@ Staggering the mount-counts at which filesystems are forcibly checked will avoid all filesystems being checked at one time when using journaled filesystems. .sp -You should strongly consider the consequences of disabling -mount-count-dependent checking entirely. Bad disk drives, cables, -memory, and kernel bugs could all corrupt a filesystem without -marking the filesystem dirty or in error. If you are using -journaling on your filesystem, your filesystem will -.B never -be marked dirty, so it will not normally be checked. A -filesystem error detected by the kernel will still force -an fsck on the next reboot, but it may already be too late -to prevent data loss at that point. -.sp See also the .B \-i option for time-dependent checking.
Time & mount-count based checks have been off by default for quite some time now, but the dire warning about disabling them remains in the tune2fs manpage, which is confusing. We did "strongly consider the consequences" and disabled it by default, no need to scare the user about it now. Signed-off-by: Eric Sandeen <sandeen@redhat.com> ---