|
Lai
Can you please advise?
Robin
Even if Lai can proceed without a crash dump on this occasion, I think that having this option is bound to be useful at some point.
Peter
|
|
Hi,
there was a -116 in the logs in todays hang/crash. syslog attached.
in case it's not clear - this ticket is our highest priority of all our tickets.
thanks.
Peter - we have some compute node crash dump's working now, and will start to work on servers.
cheers,
robin
|
|
another hang. I'll attach the syslog. this hang has no rc = -116.
first hung process at
Sep 24 14:57:22
for the 2 crashes today, I can't see any chmod/chgrp's on the system. there are a about 8 rsync's running on the login nodes from various users, so I suspect they are probably triggering it.
probably tangential, but there are new mdt_cross_open errors in the logs as of today too. can't see them in previous logs.
cheers,
robin
|
|
The log looks similar to that of LU-11301, so the previous fix doesn't really solve this issue, I'm checking related code.
|
|
Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/33248
Subject: LU-11418 osd-zfs: stop trans if transaction abort
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 518eff16a77d077d82bc92b85912327dbfe6a271
|
|
I just uploaded a patch which fixes a corner case if transaction aborts, and also adds a few debug messages. You can cherry-pick it to 2.10 and test again.
|
|
Hi Lai,
thanks very much!
I was just about to add that in, but then I noticed sanity-sec fails in gerrit. is that something we should be worried about?
cheers,
robin
|
|
Hi Lai,
has this patch stabilised enough to try it out now do you think?
cheers,
robin
|
|
Yes, you can try it now.
|
|
Hi Lai,
ok. that does seem to have helped a lot.
the MDTs were "repeatably" hanging today, and then I added the version3 patch and they seem ok so far.
success? 
longer version: there are a lot of rsync's running on login nodes and data-movers today, and warble2 (which has all 3 DNE dagg MDTs on it) hung again (as it did 2 days ago). warble2 needed to be reset to fix it but it just hung again a few minutes later. and then the same again.
the one odd thing about today's hangs was that for the first two hangs all 3 MDTs wouldn't umount. usually it's just one.
so then I added your version 3 patch from https://review.whamcloud.com/#/c/33248/ and warble2 hasn't hung again yet.
only warble2 is running this patched lustre at the moment.
dmesg below.
there are multiple -116 messages, but no stuck MDT threads.
there are definitely new corrupted directories from today's resets/hangs, but I can't tell if they are worse than usual or not.
cheers,
robin
|
|
so the corruption is perhaps different today...
I'm putting it in this ticket as it's perhaps related to these changes.
there are high level directories (not new leaf dirs) that now report "stale file handle".
2 users have reported this so far.
these are not new directories, but have been there for weeks/months . eg.
and there are 1000's of old files under that directory tree.
it seems unlikely that these directories should be "normal" corruption from the MDS resets or client evicts today.
I'm going to reboot back into the unpatched version and see if the directories come back to life.
cheers,
robin
|
|
under the unpatched lustre the directory is ok
also another directory that the other user reported as "stale"
please advise.
cheers,
robin
|
|
Can you apply the patch https://review.whamcloud.com/#/c/33248/, and collect debug log on MDS when mkdir fail with -ESTALE? And it's better to enable more debug with 'lctl set_param debug=+trace' on MDS.
|
|
I rebooted back into the patched lustre and now there are no "stale" messages from either of those 3 dirs. WTF.
I also created a test dir in all
level dirs in the basic hierarchy, and they worked ok.
dmesg for this boot is
warble2 was hung again before I rebooted it into this patched version, which is the same as the previous time we were in this patched kernel.
some more info on the "stale" state ->
the "stale" was repeatable and from multiple nodes.
all dirs below the "stale" dir were also "stale" when trying to create a new dir in them.
files and dirs in the "stale" hierarchy were readable, just basically immutable.
I'm very unsure whether it's safe to leave it in this lustre version...
it's not really practical to go over every dir and create a test file or dir in there and see if it succeeds or not.
any thoughts?
cheers,
robin
|
|
Hi Lai,
I've done some more sweeps through what dirs I can, and (I think) somewhere in
the below messages appear but I haven't narrowed down where.
I haven't found a definitively "stale" dir tree again yet.
anyway, I've left it in this patched lustre version for now.
I don't understand why "stale" directory trees would come and go though.
further advice would be appreciated.
thanks
cheers,
robin
|
|
The error messages report that distributed log fail to cancel, which is not fatal and won't cause anything wrong normally, and I'll review code to see whether mdt_cross_open() failure is related.
Fine, next time you met with any failure, if you can collect debug logs in time, it may help.
|
|
Hi Lai,
can you please explain more about why we would get "stale" directories on all clients on 1 boot and not on another. we have never seen this behaviour before.
I will try to remember to gather logs more diligently, but obviously I was expecting the same behaviour the next time, so thought I could gather logs for you if requested by rebooting back into the patched lustre again. at the time it was more important to stop users jobs dying. as I said, we haven't seen behaviours change randomly from boot to boot like this before, and are concerned that it will just randomly happen again.
I can't see any significant differences in the two dmesg connection logs posted above (same lustre, one "stale", one not), but hopefully you folks can.
thanks.
cheers,
robin
|
|
llog_cat_cancel_records() is called after distributed transaction commit to remove replay logs, if for unknown reason it's not found, it will return -ESTALE, but since transaction was committed already, it won't cause any trouble.
mdt_cross_open() is to open a remote file (whose inode and dirent are located on different MDTs), if the inode corresponding to the FID doesn't exist, it returns -ESTALE, this normally happens when a concurrent unlink removed this inode.
But for lack of debug logs, I can't really know what happened in your case.
|
|
Hi Lai,
thanks, but those are just the ongoing messages.
instead can you please look at the 2 dmesg's in the comments above that start with "LNet: Using FMR for registration".
these are the mount logs for the 2 attempts with the same lustre version, but one is "stale" and the other isn't.
as the "stale" issue started at boot time, this is the place to look. no?
thanks.
cheers,
robin
|
|
Hi Lai,
the stale directory problem has re-emerged.
I've uploaded debug logs to ftp in uploads/lu11418/
I'll attach warble2/john99/Lustre syslog for the day.
I started debug_daemon on the MDS that has all 3 dagg MDTs on it, with -1 debugging.
then on the client ran
so a couple of directory creates succeeded, but then "blah102" failed.
maybe 30% to 50% of the time they fail.
I also picked a random directory on the filesystem, made a dir for myself, and created 21 dirs and the results are below.
so it looks like all attempts to mkdir on mdt0 are failing?
this "stale" problem has only just started again this evening.
the filesystem has been fine for a few days now until this.
I ran chgrp/chmod sweeps and they completed ok.
I ran a namespace lfsck this afternoon and that corrected a bunch of things, and seemed to complete ok too.
sometime during the lfsck, lustre started getting quota enforcement badly wrong, so I turned off inode group quota enforcing on the dagg filesystem until we can look at it further, and now these stale directories seem to be appearing.
no idea if that is all coincidence or is interrelated, so I'm just writing it all here for you to decide about.
I suspect nobody is around to help on a weekend, so I'm going to reboot the MDS now.
I'll try rebooting back into this same osd-zfs patched lustre and see if they're still stale.
cheers,
robin
|
|
Hi Lai,
MDS crashes on reboot into any of our recent lustre versions. see LU-11516.
I've applied Andreas's patch to allow the filesystem to mount.
I've now removed the version 3 patch from https://review.whamcloud.com/#/c/33248/ from our build.
we only started to see "stale" directories after applying that patch.
please let us know what you glean from the debug logs about stale dirs above.
cheers,
robin
|
|
Hi Lai,
so firstly, I've just noticed that the chmod/chgrp sweep that we did on the filesystem starting on 2018-10-13-14:44:52 also came up with a bunch of "stale" directories. I started the lfsck around Oct 13 19:49:28 which is well after the chmod/chgrp sweeps had finished.
so that means that the "stale" issue is definitively (IMHO) related to the patch in this ticket, and not to the lfsck.
secondly, all 1739 of these " Stale file handle" were directories (no files) and all on mdt0.
thirdly, this sweep also hit 396 "Disk quota exceeded" errors on directories (no files) messages (which I think should never happen as root), and these were heavily on mdt1 and mdt2 and not on mdt0. ie.
more information is below...
so I actually did 2 chmod/chgrp sweeps. the first ran from 2018-10-10-14:11:22 to Oct 13 07:37 and it saw no "stale" or "quota" problems.
the second sweep was expected to be quick as almost all incorrect mode/ownership had already been fixed by the first. it ran from 2018-10-13-14:44:52 to Oct 13 18:18. now I have noticed that this second sweep did see "stale" and "quota" error directories.
everything from Oct 13 about warble2 in syslog in this time interval is already attached to this ticket in messages-john99.warble2.Lustre.20181014.txt.gz
please let me know if you'd like everything from 2018-10-10 to say, 2018-10-14.
(perhaps fourthly), the other thing I've noticed is that all the groups (== directories on Lustre) that saw "Stale file handle" on some tiny fraction of their mdt0 directories in this second sweep were the same as the groups that saw the "Disk quota exceeded" errors ->
the exception in the above is the group oz123, but that kinda makes sense 'cos the the number of "Disk quota exceeded" messages for each group's sweep is less than their "Stale file handle" errors, and oz123 had the least of those.
so perhaps the "stale" problem and the "quota" problem are related somehow.
cheers,
robin
|
|
-116 comes from llog_osd_declare_write_rec() --> osp_md_declare_write(), where 'opo_stale' is set:
I'm still trying to understand why it's set, and because the stale flag is set for dagg-MDT0002-osp-MDT0000, all the create starting from MDT0 failed.
|
|
Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/33401
Subject: LU-11418 llog: refresh remote llog upon -ESTALE
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a46b01230162e12b82e85c1c84d9f07c9350a71d
|
|
Hi Lai,
so is the idea to apply just https://review.whamcloud.com/33401 or is that in addition to https://review.whamcloud.com/33248 ?
cheers,
robin
|
|
I think both are needed.
|
|
Hi Lai,
do you think this also explains the quota errors that we were seeing, or could that be a separate (also stale?) problem?
I've built an image with these 2 patches in, and will reboot all lustre servers into it later today. I guess I just really have to do the MDS's though... ?
cheers,
robin
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33248/
Subject: LU-11418 osd-zfs: call stop_cb if transaction start fail
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d614ff561cb6bd9a9b35310c99dd991c55797d97
|
|
Hi Lai,
we've been running with these 2 patches for a few hours now. so far so good, but one client LBUG seen. presumably it's not a coincidence, but I guess it might be.
cheers,
robin
|
|
Hi Lai,
2 more MDT hung threads/crashes today. will get logs in a min.
at the moment this process is using 100% cpu on the DNE MDS.
root 50259 99.7 0.0 0 0 ? R 12:44 53:47 [orph_cleanup_da]
load is 28, and perf top says
cheers,
robin
|
|
Hi Lai,
conman for today on warble2 is attached.
summary:
I rebooted warble2 (and warble1) into the lustre with the 2 patches in this ticket at 2018-10-23 23:20
it lasted ok until 4:40am at which point the filesystem was hung. a find for directories that I was running hung at that time.
the hung MDT thread stack trace was something we haven't seen before.
the MDS was powered off (a MDT wouldn't umount like usual, had to power off) and rebooted at ~11:20 am.
almost immediately multiple MDT threads were hung
and the same again after another reboot at 12:45
I took a -1 debug log at ~13:30.
at 14:00 we gave up on this, and rebooted warble2 back into the centos-7.5-minimal-server-06/ image which is the same as 07/ except does not have these 2 patches in it.
we then got a "normal" MDT thread hang very quickly after boot
I suspended all jobs, then rebooted warble1 into this 06/ image without these 2 patches and then rebooted warble2 again at ~2018-10-24 14:39
better luck this time, and it's been running jobs and ok for half an hour or so now... presumably the reduced load from having jobs suspended when it mounted the fs's helped.
I'll upload the -1 debug daemon logs to the ftp now.
additional information is that when umount'ing the MDTs at 14:30 I noticed the same [orph_cleanup_da] process running as in the previous comment in this ticket which around ~13:30. so it kinda looked like the MDTs were umounting/reconnecting even though they up but hung.
cheers,
robin
|
|
darn, sorry. forgot to upload the debug log until now. kinda assumed it would be useless to you so it slipped my mind.
it's on the ftp at /uploads/lu11418/20181024/
cheers,
robin
|
|
https://review.whamcloud.com/33401 is not correct, which may cause such hung, I'll update it soon.
|
|
Hi Lai,
should we be giving patch set 2 a try?
BTW for some reason the machine has been fairly stable (without the patches in this ticket) for almost a couple of weeks now. perhaps the namespace lfsck that finished and corrected some things has helped.
cheers,
robin
|
|
yes, I added a test for set 2, which shouldn't cause system hung any more, you can give it a try now.
|
|
Hi Lai,
we've just started running version 3 of the patch now. will let you know how it goes.
thanks!
cheers,
robin
|
|
Hi Lai,
no obvious signs of problems yet. I'll start a chgrp/chmod correcting sweep soon I think.
FYI syslog messages so far are
as the last namespace lfsck completed ok (with the previous version of the patches in this ticket), should we now try a lfsck with -C to try and get rid of some of the above?
we currently mount with -o skip_lfsck. should we umount and mount without this first, in order that an oi scrub runs first? or just try an "all" types lfsck (dry-run to start with I guess)?
cheers,
robin
|
|
Do you know which operation generated these messages? And did it cause any error?
'lfsck' with '-C' is to create missing MDT objects, I'm afraid it won't fix the the orphan_destory() failure. I'd suggest you do a dry-run and post the result here. IMhO for the orphan issue, we need to wait for https://review.whamcloud.com/#/c/32959/ to become ready.
|
|
Hi Lai,
perhaps those error messages could print a bit more info eg. uid/gid/fid/path? playing "guess which file or dir" across a fs with 100M files and 100's of users is kinda hard...
or if you have a systemtap script or similar then I could try that...
hmm. so I have just noticed that this process has been running 100% of the time since boot.
from 'top':
ie. ~= 3688 minutes uptime and 3671 minutes of system time used by "orph_cleanup_da".
the process is only running on the MDS that has the 3 DNE MDTs on it. and not the other MDS that has non-DNE MDTs for the smaller filesystems.
I presume this process is a problem? I haven't noticed it using a whole core (or any time) previously, but TBH I haven't run 'top' than many times on the MDS.
otherwise, things seem to be going well... no hangs, no crashes, no reports from users, and the chgrp/chmod script ran and finished ok.
thanks for the lfsck advice. I ran this on the DNE MDS
and got this from layout (which is confusing 'cos it seems to never have started a phase2, nor completed)
and this from namespace
further advice would be much appreciated, but I will try not to do anything further today 'cos I don't want to break things over the weekend.
cheers,
robin
|
|
Hi Lai,
FYI the "orph_cleanup_da" thread seems to be permanently looping doing ~900k reads/second to ZFS.
the 900k rate is independent of (and at least 50x greater than) the actual iops we graph going to the dagg MDTs (typically 1k to 5k iops - probably 766 over the lifetime of the pool - the first number in arcstat). so I think the reads/s to ZFS's ARC are just that thread doing something crazy and nothing to do with any actual iops to the MDTs.
presumably it's in an infinite loop of some sort, and 900k is just as fast as it can go. hence the 100% system time on that core.
would it help to get a sysrq stack listing or a 'perf top' or something for you?
I don't see any large increase in IB traffic to the MDS that would explain 900k anything, but I could probably grab a lustre debug log if you like too.
cheers,
robin
|
|
ah, ok - the MDS graphs of ZFS stats seem to indicate that the LU-11516 https://review.whamcloud.com/33368 workaround is probably the thing causing the thread to use 100% system time and do all the ZFS reads (rather than LBUG). ARC reads are 900k for most of the last month, but with a break of a week until this latest reboot.
cheers,
robin
|
|
Yes, it looks to be so, I'll check code to see how to improve it.
As for the lfsck result, I need some time to verify, will update you later.
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33401/
Subject: LU-11418 llog: refresh remote llog upon -ESTALE
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 71f409c9b31b90fa432f1f46ad4e612fb65c7fcc
|
|
So do I understand correctly that there will likely be at least one more patch to come tracked by this ticket?
|
|
yes, Peter.
|
|
I just uploaded a patch to fix the issue of orphan cleanup dead loop:
master branch: https://review.whamcloud.com/#/c/33661/
2.10 branch: https://review.whamcloud.com/#/c/33662/
You can wait autotest finish and then update MDS, if everything is okay. Then you can try lfsck in dry-run mode again, and post the result here.
|
|
Hi Lai,
thanks. we've been running the 2nd patch in https://review.whamcloud.com/#/c/33662/ on the MDS's for almost 2 days now.
all good so far
I will update the OSS's to this version too and lfsck when there's a quiet time - unfortunately the filesystem is very busy at the moment.
cheers,
robin
|
|
Any update?
|
|
Hi Lai,
lustre's been stable for the last 2 week since that patch.
all OSS's were rebooted to match the MDS version about 10 days ago.
so looking really good.
I haven't run the lfsck yet, or explicitly tried to run stress tests to try to break anything.
just enjoying the stability for a while
I'll kick off a lfsck dry run today.
cheers,
robin
|
|
Robin
Is no news good news?
Peter
|
|
Hi Peter and Lai,
not sure if it's good or bad
the lfsck just takes a while 'cos I set it to a slow rate...
this is namespace which has completed now
layout is still running. I think it's stuck
the only thing changing in the output above is the time counters. the lfsck/lfsk_layout processes are still there but not showing up as being active in 'top'.
I've stopped it now ->
I should point out that we're still mounting these 3 MDTs with -o skip_lfsck.
should we mount them in the normal way first, before doing these lfsck's? I'm not sure of the difference.
cheers,
robin
|
|
Robin, I'm still working on a patch to fix bugs in striped directory lfsck, I'll update you when it's finished, and I think it may help because many inconsistencies in your system are in striped directory. And next time when you think system stalls, can you use sysrq to dump all processes backtraces on all servers, and it's also helps to dump debug logs.
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33661/
Subject: LU-11418 mdd: delete name if orphan doesn't exist
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: fffef5c29e3bdf0f96168abc3d0488bad06f33bb
|
|
The existing patch has landed for 2.13 and could now potentially be included in 2.10.x or 2.12.x maintenance releases. Lai, if you're still working on a further patch, what ticket is it being tracked under?
|
|
Peter, it's tracked under LU-11681, when it passed reviews, I'll backport them to 2.10.
|
|
Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34326
Subject: LU-11418 mdd: delete name if orphan doesn't exist
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: ec0944fc22a351b44332984050606f0efb1d3b63
|
|
Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34327
Subject: LU-11418 mdd: delete name if orphan doesn't exist
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 694a92ec774d5bd958a61f457fc64380feb95db2
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34326/
Subject: LU-11418 mdd: delete name if orphan doesn't exist
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 6a412a8671d3d76b5da55c08ada011e7aeea1e8c
|
Generated at Sat Feb 10 02:43:42 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
lustre-log.1537573059.458124.gz