|
Looking at usage over the last 30 weeks all of a sudden at the end of February something happened and you can see it very obviously in the graph attached. We don't believe our workflows have changed.
'df -i' output shows it only 35% full inode-wise.
We've tried mounting the MDT as ldiskfs and having a look around. Nothing obvious (to my eyes) immediately showing as strange.
I did try removing an old directory in there just to see if I could free up a bit of space (knowing this would leave orphans on the OSTs we'd later have to clean up with lfsck). Watching with 'df -k' I can see the used space go down but the free space remains at 0.
dumpe2fs -h shows free blocks.
No changelog users:
Following that I tried remounting it and then using the MDS as a client to try and delete files. This immediately kernel panics.
df -i output:
df -k output:
dumpe2fs -h output:
|
|
Also worth noting, we tried an e2fsck on /dev/md0 (the block device under /mnt/mdt) - this threw a huge number of errors but most seemed to be down to not being able to change anything because the filesystem is full.
One other thing we noticed which just seems strange is that the total size reported is 3.5TB, despite md0 being just over 6TB. The block count in dumpe2fs suggests that it should be just over 6TB.
|
|
Things you could check:
- run "e2fsck -fp" on the MDS to check for errors
- mount as type ldiskfs and check the "lost+found" directory to see if there are files there
- run "du -sk O/*" to see if there are a lot of llog files or other objects that are not visible in the namespace. If there is a particular directory there (eg. "1" or "10" that might be consuming a lot of space for Changelogs, recovery logs, unlink logs, etc.
Do you have multiple MDTs?
|
|
Note my previous comment was written without having seen your comment first. It would be helpful to get some of the output from e2fsck.
The 3.5TB vs 6TB discrepancy is expected. The 2.4B inodes each consume 1KB for a total of 2.4TB, and are not listed in the total/used space for "df", otherwise users would be unhappy because their filesystem was 40% full after formatting.
There are still almost 6M blocks free, about 24GB, so it isn't clear to me what is "full"? One possibility is that it is unhappy when the free blocks count < reserved block count? This should only be possible from writes by root. You could try setting the reserved block count to zero via "tune2fs -m 0 /dev/md0" and see if that allows you to modify the filesystem again.
As for what is consuming the space, it is pretty clear that something was aggressively and steadily consuming blocks from the MDT. You could also check the oi16.N files are not outrageously large. For 800M used inodes they might be in the neighborhood of 50GB in size. Not that we've seen a problem there, but just trying to figure it out. You could also check "du -sk" of the other top-level directories besides "ROOT/" to see if they are consuming a lot of space.
|
|
Thanks for the quick reply. To go through the bits...
- lost+found is empty
- du -sk O/* shows O/1 19M and O/10 136K
- Only one MDT
- I'll try the tune2fs and report back
I have a 9.3GB screenlog from the e2fsck that I'm just gzip -9 ing. We stopped the fsck early because it seemed quite stuck. Let me know if there's somewhere I can upload a large file.
|
|
Following the tune2fs, mounted as ldiskfs it does now show free space:
When mounted as lustre is still shows 100%:
Unfortunately trying to delete anything from the filesystem seems to have immediately panic'd it again. No output in dmesg before the crash but two lines in syslog:
|
|
FYI, nothing outside of ROOT seems big:
|
|
Can you get the full stack trace from the LBUG/crash via a serial console or IPMI, or just take a photo with a connected monitor?
As for the e2fsck output, if it is many, many MB of the same message over and over again, I don't need to see all of that. The important thing would be the start of the run until it starts hitting errors, and then a sampling of the most common messages after that.
|
|
Also, collecting the full debug logs from Lustre during mount would be useful using the following process:
|
|
Based on the statfs results from mounting as "lustre" and "ldiskfs", it looks like the difference is 0.78%, which exactly matches OSD_STATFS_RESERVED_SHIFT=7, or 1/128th of the space. In order to get above that threshold, it looks like we would need to free up about 5GB of space, but there doesn't appear to be any easy place to get that space. Since you are running Lustre 2.10 you won't have Data-on-MDT, so there shouldn't be (m)any large files in the ROOT/ directory, but there could be large (empty?) directories, which is why the e2fsck output may be interesting.
Also, the debug log and/or stack trace from the MDS will shed light on what piece of code is hitting the problem.
|
|
Was watching the Dell iDRAC after settings the debug params you noted. Interestingly it let me rm a single file, so I ran a recursive rm on an old directory and it pretty much immediately hung up. The only errors output are:
The e2fsck output is a lot of:
and
Probably worth saying the RAID itself has reported no errors anywhere I can see.
|
it looks like we would need to free up about 5GB of space
As briefly mentioned before, we have a bunch of old stuff on this filesystem that we can definitely delete. In order to free up that extra 5GB or more I could remove directly from the ldiskfs mount and would have to rely on lfsck later to clean up the orphaned objects....?
Also since you mention we're on 2.10, we're not averse to upgrading if it brings with it more useful debug output or bug fixes etc.
|
|
OK, based on your e2fsck summary, it is pointing to a problem in the ldiskfs code where it allocates the directory inode, then hit an out-of-space error trying to allocate the block for the directory, but it doesn't clean up the directory inode on error? Then, when e2fsck is running on the filesystem, it is trying to allocate blocks for all of these zero-length directories and failing, and then hitting errors trying to insert "." and ".." entries into the non-existent directory blocks. That is just speculation based on the errors, and not something we can do anything about yet.
On the MDS, when you dumped the debug log, did you previously set debug=-1 to capture all the logs? You could check for files like /tmp/lustre-log.<timestamp> that may contain the log dumped by the LBUG (if MDS has not been rebooted yet). Also, is there a stack trace shown by dmesg? That would also be useful to attach here.
|
|
Stack trace from dmesg (strangely wasn't on iDRAC console so I'd missed it):
Attaching the lustre-log file.
|
|
I wouldn't suggest upgrading to 2.12 before this situation is resolved. I am not aware of any particular fix that would address this issue.
It's not clear whether the "delete 5GB of space" will help, or if there is some other data structure that is having problems. AFAICS, the 0.78% free threshold for reserved space is only affecting the df output, but there is already 23GB of free space on the MDT, and at least root should be able to delete files, if the MDS didn't keep crashing. Getting the console log (via dmesg if the MDS doesn't actually reboot) would be helpful. You might need to increase the debug level for the console messages if it isn't showing anything, like "echo 7 > /proc/sys/kernel/printk" to print more information to the console.
One other question - what version of e2fsck are you using? You should use the latest release (1.45.2-wc1) from https://downloads.whamcloud.com/public/e2fsprogs/latest/el7/RPMS/x86_64/ which may help with repairing or removing the broken directories.
|
|
We are using:
I can upgrade these two now. Do you recommend starting another e2fsck at this point or wait to see what the log/stack trace show first?
|
|
Is it possible that you have a tunable parameter in your config log to set debug=0? It seems that there are lots of debug logs until a certain point in the mount, then it stops entirely until the osp_sync_declare_add()) ASSERTION( ctxt ) failed message is printed:
so there is about a 20s gap in the logs. It seems that after mounting the MDS you may need to run "lctl set_param debug=-1" again and then re-capture the logs to get the background of what the client was doing before the LBUG, but I'm still looking through what is in the current log to see what I can find.
|
|
I've been running that debug=-1 line each time. Do you know where I'd look for the tunable you mention?
Edit: Nevermind - just re-read your comment and see I need to set it again after the mount.
|
|
i would recommend to update e2fsck, and then run read-only "e2fsck -fn" (under script so the output is logged) to see if it behaves better.
The stack trace at least shows us what the client is doing that is causing problems. Clearly unlinking, which isn't a surprise, but details are helpful.
Is it possible that you changed the default layout at the start of March to have a very wide striping, or use PFL with a large number of components, that would cause all the files to allocate large xattr blocks? Alternately, did you add a lot of ACLs, or applications using other xattrs (SELinux?) that might have done the same?
|
|
None of those were done as far as I know. It's just been sat there for months being a target for rsync to back up the same set of servers.
I'll try running the fsck now in a logged screen. Guessing this will take several hours so if there's anything else you'd like me to do first, let me know.
|
so there is about a 20s gap in the logs
Isn't this just the time I'm spending navigating to the directory where I'm going to do the rm? Think I mentioned before, I'm using the MDS as the client
|
|
Hmm, is your backup something like "BackupPC" which is creating large numbers of hard links to the same files? That would also cause Lustre to allocate an external xattr block for each file after a certain number of hard links to each file.
|
|
Yes - we use dirvish which does create hardlinks for incrementals. We have been using it for years though without change.
|
|
Given the nature of what's on here and how long the fsck is going to take to run, I'm tempted to run it with -y unless you recommend otherwise now that those 23GB are free.
|
|
Another engineer working on this issue has informed me that this problem may already be resolved via patch https://review.whamcloud.com/38224 "LU-12674 osp: handle -EINPROGRESS on llog objects", which was backported to b2_10.
|
|
Would we need to upgrade/downgrade to any particular version?
|
|
There is a build at https://build.whamcloud.com/job/lustre-reviews/73314/ for RHEL7.6 x86_64 that should be installed on the MDS and preferably OSS nodes also. It is based on the 2.10.8 release, or you can build your own if that is normally what you do.
|
|
Note that you should be able to install the Lustre server kernel on pretty much any RHEL7 distro, unless you have other kernel modules (e.g. IB) that also have dependencies. In that case, you might be better off to rebuild the Lustre modules for your specific kernel.
|
|
I'll install that on the OSSs now while I wait on the fsck.
fsck seems to be running faster than expected. 22% through on pass 1.
|
|
The previously-mentioned LU-12674 patch should fix the "ASSERTION ( ctxt )" crashes on the MDS, so it might be possible to delete some files without crashing.
However, there are still directories that have problems due to running out of space that will need e2fsck to fix. I suspect if you try to delete them via Lustre or ldiskfs they will cause the MDT to be remounted read-only (-30 error).
|
|
Ok. I'm gonna leave the e2fsck to complete (hopefully) and in the meantime I'll install the updated lustre packages. Once it's done, reboots all 'round and I'll let you know how it goes.
Thanks for all your help so far.
|
|
The other problem to understand/address is what is consuming the space on the MDT. You could check this if you know some files that were backed up recently, and run "debugfs -c -R 'stat /ROOT/path/to/backup/file' /dev/md0" to dump the inode information. The "catastrophic mode" message is not cause for alarm, just the "-c" flag skipping loading of unnecessary filesystem metadata to run faster. This should dump something like:
and of interest is the "File ACL:" line being non-zero (indicating an external xattr block) and the "trusted.link (968)" xattr line showing that it is large (at least larger than can fit into the remaining ~450 bytes of xattr space available in the inode). If that is the case, you might be able to reduce this xattr space usage by removing one of your older backup copies, or some old directory trees that have lots of files with these large xattrs.
|
|
You could also use "debugfs -c -R 'stat <inode_number> [<inode_number> ...]' /dev/md0" to dump the attributes of of a few problematic inodes previously reported by e2fsck, if they haven't already been fixed. That might give us an idea of what went wrong with them.
|
|
I'll do that as soon as the fsck completes and before I mount as lustre to try and delete stuff. It's certainly likely that we'll exceed the 450 bytes on almost all the files because we maintain (usually) 2 or 3 weeks of daily hard linked backups for each server so up to roughly 21 hard links per file.
Again though this hasn't changed recently, it's been this way for some time and I'm not sure what would have caused such an incredible linear increase in the numbers of files/size used.
|
|
When you have a chance (not urgent) it would be useful to check whether the trend for "df -i" on the MDT (if you track this history) is increasing similarly as shown in df.png. That would at least tell us if the extra usage is new files (it both used space and inodes are growing at about the same rate) or it is extra space usage per file or elsewhere (if space usage is increasing while inode usage is roughly constant).
|
|
Very annoyingly we noticed that we didn't have inode stats turned on in collectd. We've corrected this but only as of today unfortunately.
|
|
Unfortunately:
|
|
A couple of debugfs examples:
|
|
I am however able now to delete files without a panic.
I'm going to try and clear space and see if we can get a backup through overnight and then check the logs tomorrow. Looks like another fsck is going to be required...
|
|
The dir_info e2fsck error appears to be the same as LU-13197, which has a patch to fix it. There is a RHEL7 build of e2fsprogs that is known to fix this specific issue:
https://build.whamcloud.com/job/e2fsprogs-reviews/arch=x86_64,distro=el7/862/artifact/_topdir/RPMS/x86_64/
This e2fsck bug was hit at another site that has a very large number of directories (over 180M directories), which is unusual for most cases, but in the case of your symlink trees there are lots of directories with relatively few directories. The updated e2fsck was confirmed to fix the problem on their filesystem.
|
|
As for the debugfs stat output, it definitely shows that the "link" xattr is large in at least some cases, and would consume an extra block for each such inode. Also, based on the previous e2fsck issue, it seems that there are a very large number of directories compared to regular files, and each directory will also consume at least one block. Based on LU-13197, the filesystem must have at least 180M directories for only 850M inodes, so only about 5 files per directory (although this doesn't take into account the number of hard links).
|
|
Our directory trees are ridiculously deep and overused for structure of data so this doesn't surprise me. I'm still not sure what changed end of Feb though so we're gonna have to watch this carefully.
Checking this morning, backups are still running and seem to be somewhat stable so I'll let a good backup complete then try and take it offline to run a new e2fsck.
After we've had a successful e2fsck, I'd like to upgrade to 2.12.4 but would it be sensible to run an lfsck prior to doing that, or after to get all the updates/bug-fixes?
|
|
If you run e2fsck on the MDT to repair any problems in the local MDT filesystem, then running LFSCK is not strictly required, as it is mostly doing garbage collection and handling cases where there is some inconsistency between the MDT and OSTs. Generally, LFSCK has been getting better with newer releases of Lustre, so it is probably better to wait until after the upgrade if you want to run it, and unless there are visible problems with the filesystem you may want to wait until there is a good time to run it (e.g. planned system outage).
|
|
Will do. The priority of this ticket can be dropped if you like since the filesystem is now up and running. I'll continue to report back on the e2fsck progress.
Thank you for your help getting things working so quickly.
|
|
Any update cmcl
|
|
I ran some tests on a local filesystem with master, filling up the MDT with DOM files and directories, and while there were some -28 = -ENOSPC errors printed on the console, I didn't have any problems with deleting the files afterward.
|
|
After deleting a huge number of files and getting a couple of good backups through, I managed to run an fsck over the weekend (results attached). With that done, everything remounted and seemed to function fine.
I've since upgraded to 2.12.4 and we seem to be back in business.
Thanks again for all your help.
|
|
Excellent - thanks for the update!
|
Generated at Sat Feb 10 03:01:23 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
df.png
lustre-log.1586896595.11759.gz