|
Robin
Just to acknowledge that we have seen this ticket. I am discussing with engineering
Peter
|
|
Robin
Could you please attach mdt logs while this is happening?
Peter
|
|
Hi Peter,
logs attached
LBUG at Apr 7 16:38:10
and then various attempts to get them connecting after that.
happy to try 2.11 if you like 
cheers,
robin
|
|
Hello Robin,
have you tried to abort recovery on these 'WAITING' servers?
Mike
|
|
Hi Mike,
haven't heard of abort recovery before... will give it a try though.
and the same on MDT0 (may as well, even though that one seems fine) and it didn't do anything.
all the recovery_status's didn't change and nothing in syslog/dmesg.
try umount and remount with -o abort_recov ->
is so far hanging.
in another window I did
which is also hanging.
this is recovery_status now
cheers,
robin
|
|
nothing happened after waiting an hour or 2, so abort recovery didn't work.
^C'd the mounts and because lustre and mount were now conflicted about what was mounted and what wasn't, I was forced to reboot both MDS's. pacemaker/corosync is now in tiny broken shards, sigh.
however still the same from lustre after mounting everything by hand: dagg-MDT0 COMPLETED, and MDT1,2 in WAITING state.
I suspect we'll have to rollback a zfs transaction or two in order to get this to mount. does that sound right to y'all? anyone done it before and know where to start without losing too much data?
cheers,
robin
|
|
Can you attach zfs status from mdt 1 and 2? Hopefully the pools aren’t corrupt. Also the messages file?
|
|
Hi Nathaniel,
messages are already attached to this ticket. we have central syslog'd messages and minimal state on lustre servers.
currently everything is mounted on warble1 with nothing running on warble2, so I'll attach zpool status and zfs list for those. but as per usual with zfs, these look very neat an tidy and there's nothing interesting. it's not what you'd call verbose about what it's doing...
cheers,
robin
|
|
it looks like dagg-mdt1 and 2 cannot talk to each other and mdt0, this is evident in mdt2 saying that mdt0 and 1 are "not ready" and DNE recovery depends on all MDT targets being available otherwise you need to abort recovery.
That said it's strange that your mdt1 mount hangs, are there any additional stacktraces printed into messages/dmesg (since the snapshot you provided appears to have been before the hang).
Did you try to mount with -o abort recovery after your latest reboot? That seems like the thing to do.
|
|
DNE doesn't allow abort_recovery (though maybe it should?). We need MDT0000 to be able to connect to MDT0001 and MDT0002.
|
|
In my discussions with Di at the time of DNE landing he told me that the way to recover from a lost MDT server is by administrative action (which I think was abort recovery) followed by fsck or some such to ensure the namespace is coherent.
|
|
Could you please dump the stack strace of that hanging MDS? I suspect the update log (for DNE recovery) might be screwed up somehow, if that happens, you might also need delete those update log manually from ZFS to abort recovery, (hmm, these should be added to abort recovery process  ). We did this a lot for ldiskfs MDTs, but not sure how to do it for zfs. Let me do a few tests.
|
|
Hmm, I tried to mount zfs osd, delete those update log files, then mount MDT again, but failed
According to the debug log, it seems rm over zfs mount can not really delete those update log files.
I remembered Alex mentioned this once, but forgot the details. Any suggestions how to delete the llog file on zfs osd. Alex, Andreas, Oleg?
|
|
Di, maybe try running "lctl lfsck_start -A" on the MDS after removing the files, so the OI entries are cleaned up?
|
|
Andreas, I deleted the local llog file over zfs mount,, instead of lustre, after that the MDT mount will fail, can lfsck be used in this case? Besides this failure seems happened during object creation, instead of OI insertion, so it looks like to me this local llog file becomes an orphan after deletion. Is there a ZFS tool to fix this? Thanks.
|
|
Hi Di,
sysrq traces attached now.
I did show-backtrace-all-active-cpus(l) then show-task-states(t) then show-blocked-tasks(w)
cheers,
robin
|
|
Hi Oleg and Nathaniel,
all of central messages for warbles for yesterday is attached now - warbles-messages-20180408.txt
there are indeed stack traces for the hung mount -o abort_recov's in there from mount.lustre around Apr 7 23:02:41
sorry for not noticing these before.
cheers,
robin
|
|
there's also about 100kbytes/sec of IB network traffic and a load of 2 or 3 on warble1 (all the MDTs are on this machine still)
|
|
yesterday I noticed in recovery_status that when dagg-MDT1 and dagg-MDT2 first mount they say "non-ready MDTs: 0000 0002" and "non-ready MDTs: 0000 0001" respectively, but then quickly see each other and go to the above "non-ready MDTs: 0000".
so even though MDT0 is the one that's mounted and completed in recovery_status, I suspect there's actually something wrong with that one...?
I did try mounting MDT1,2 first and waiting a minute or so, and then mounting MDT0 afterwards, but it didn't change anything.
as a bit of background, dagg-MDT1 and 2 are usually on warble1 (as the zpool name in the df above indicates) and dagg-MDT0 is usually on warble2, but when warble1 LBUG'd on umount (as part of the initial failover everything, failback everything attempt) then actually all the MDTs were on warble1, so I'm not sure that means anything.
all the lnet connections look ok BTW - lctl ping, list_nids - I just thought I'd better check that 'cos I forgot to do it yesterday, and this seemed like it might be part of a "connection issue" related problem.
cheers,
robin
|
|
Did you try mounting the MDTs (in particular MDT0000 on the normal MDS? It may be that the failover config for the MDTs is not configured properly, so when they are started on the backup MDS it isn't possible to find MDT0000 and the startup/recovery of MDT0001 and MDT0002 is stuck.
|
|
Di, actually, it might be possible to try editing the OI directly... With ZFS the oi "files" just appear to be directories with filenames that are FIDs, so if you also know the FID of the files being deleted (lfs path2fid <file> should print the FID even if it is mounted locally), then you can also try deleting that file from the appropriate oi.N directory. I haven't tried this myself, but should work on your test system.
|
|
Hmm, according to the trace MDT1 and 2 are indeed still recovering.
According to these 2 traces, MDT1/2 indeed keep reading update log. It is either because of network issue or corrupted llog. I did see some timeout connecting request from the console, but seems temporary issue. Do you have recent console log, i.e. when the stack strace is created around Apr 8 13:24:53.
And could you please also mount your mdt0 as zfs to see how much update log here. After you mount your MDT0 as zfs, it should be here.
|
|
Andreas, ah, ok. thanks
|
|
Hi Andreas,
I checked that the zfs lustre params are fine. also these filesystems have been failed over and back dozens of times before so it seems unlikely. but anyway, sure, why not
this is with MDT0 mounted on warble1:
MDT1,2 saw no change after the above, so I umount/mount'd them again just to see and for a while they said:
and after ~60s they saw each other, but still not MDT0:
syslog for this stuff
it's now over 900s later and no change.
cheers,
robin
|
|
Hi Di,
attached is conman-warble1-traces.txt which is console log that I took the same time as the syslog warble1-traces.txt
I'll remount as zfs now.
cheers,
robin
|
|
and there has now been a change - all 3 of the MDT's are now in WAITING state.
syslog for this stuff is ->
cheers,
robin
|
|
please note we have had a ticket open for this
https://jira.hpdd.intel.com/browse/LU-10677
for quite a while now, so some corruption on the MDT is probably expected.
cheers,
robin
|
|
Robin,
According to Di's analysis, it is suspected that there is some trouble in DNE update logs. If that is true, then we can try to remove and and re-create the DNE update logs, that will allow us to mount the system up as Lustre. The risk is that if there are some uncompleted cross-MDTs modifications before former MDT umount, then those modifications can NOT be recovered any longer. Under such case, Lustre namespace will become inconsistent, we will have to run namespace LFSCK to repair that. But the namespace LFSCK cannot guarantee that the system globally recover to the status before the former umount. But consider another ticket LU-10677, since your system is already inconsistent, it is worth to try as following:
1) mount the MDT0 up as "zfs"
2) mv /warble2-dagg-MDT0-pool/MDT0/update_log /warble2-dagg-MDT0-pool/MDT0/update_log.bak
3) mv /warble2-dagg-MDT0-pool/MDT0/oi.9/0x200000009:0x0:0x0 /warble2-dagg-MDT0-pool/MDT0/oi.9/0x200000009:0x0:0x0.bak
Please do the same for MDT1 and MDT2, replace the 0x200000009:0x0:0x0 as 0x200000009:0x1:0x0/0x200000009:0x2:0x0 on the MDT1/MDT2. Then remount all the MDTs as "lustre". If the system can be mounted up, then we will leave related update log objects on the MDTs, but comparing with the unmountable system, that is not fatal. If it does now work, then please rename them back and collect -1 level Lustre kernel debug logs on the MDTs when mount as "lustre".
|
|
Hi nasf,
so I'm clear on 1,2,3 and doing more or less the same on mdt1,2
I get confused after that...
a) did you want me collect -1 logs as I try to mount lustre the first time?
b) (if all that worked ok) you want me to then umount and mount as zfs again, and mv the OI files back again, and then try to mount as lustre again, and collect -1 logs there too?
c) lfsck at which stage? and this is done with "lctl lfsck_start" ?
cheers,
robin
|
a) did you want me collect -1 logs as I try to mount lustre the first time?
If your system can mount up after removing the update logs, then needs NOT Lustre debug logs.
b) (if all that worked ok) you want me to then umount and mount as zfs again, and mv the OI files back again, and then try to mount as lustre again, and collect -1 logs there too?
No, if the MDT can be mounted as "lustre" after removing the update logs, then the system will re-create new update logs after the mount.
c) lfsck at which stage? and this is done with "lctl lfsck_start" ?
namespace LFSCK needs to be done after the MDTs mounted up as "lustre" on the MDT0 (if the system can be mounted after removing the update logs):
To be safe, you can use "--dryrun" option to check namespace consistency firstly before the repair mode LFSCK.
|
|
ok. thanks!
I'll try to do that in an hour or two.
cheers,
robin
|
|
Anyway, all depends on that we can mount the MDTs up as "lustre" after removing the update logs.
|
|
yay! moving update_log and oi.9/0x200000009:0x[012]:0x0 to bak has worked
thank you all very much!
I just mounted them all on warble1 for simplicity.
I'll try a dry-run of lfsck now.
cheers,
robin
|
|
lfsck went well (included below if you are interested). looks like only minor repairs. impressively quick too.
many thanks again to everybody involved, especially as this happened over a weekend.
we'll probably try more mount/umount and failover stuff during the week, just to give ourselves a bit of confidence that it's fully back to being ok again. we'll have to reconstruct the pacemaker/corosync config first though 'cos that fell in a heap after one of the reboots, so might take a few days...
please let us know if there's more that we should do to make sure it's happy. I presume the namespace lfsck was sufficient?
I guess it goes without saying that it would be nice to know why the LBUG and subsequent WAITING happened in the first place, and have future lustre be more robust against such a failure, and have an easier way to put it back together if something like this happens again too.
but at the moment I'm just happy it's back to running. thanks again
cheers,
robin
|
dangling_repaired: 3
Please add "-C" option to repair the dangling name entry via re-creating the lost MDT-object (empty) when you run repair mode LFSCK; otherwise by default, it will only warning you that there is dangling name entry. Please note that there are two possible cases that may cause the dangling name entry:
1) The uncompleted rm operation before crash, that only destroyed the MDT-object but left the name entry in namespace.
2) The uncompleted create operation, that only inserted the name entry but without creating the MDT-obejct.
From the LFSCK view, it cannot distinguish the two cases. For data safe, we assume that it is the 2nd case. You need to check the LFSCK logs (Lustre kernel debug logs on the MDT with "lfsck" enabled: lctl set_param debug+=lfsck) and find out related repaired items, then decide whether it is the 2nd case or not. If it is the 1st case, then you can remove the stale name entry manually.
Anyway, even if it was the 1st case, but LFSCK repaired it as the 2nd case, equal to that some removed name entry (and related MDT-object) is "rollback" because of the MDT corruption and LFSCK. But Lustre namespace is still consistent.
|
|
thanks for checking that output and for clarifying.
so we have 3 Lustre files/dirs somewhere that point to something that doesn't actually exist on an MDT?
'lfsck' seems to already be in the default 'debug' setting on our MDS's.
unfortunately it looks a bit like there's a lot of lines in the 'lctl dk' during a lfsck, perhaps 1 line for every inode? but there's only about 290k lines in the 'dk' buffer. I did 'dk' 3 times during a lfsck sweep and got ~850k unique lines.
even with a new fs with only ~7m files in the filesystem, I'm not sure how this is going to work... is the idea to gather 'lctl dk' in a loop every few seconds, and then grep for something 'dangling' related? that sounds wrong... I'm guessing I'm missing something...
cheers,
robin
|
|
if it's only 3 files then I'm not sure I care too much. 3 dirs might be different.
but yeah, (tomorrow) perhaps it would be better to find out what they are before running with -C.
cheers,
robin
|
|
Hi, do you still have logs with this mesage:
I think we still want to learn the root cause..
|
|
Robin,
To collect a large number of debug logs, you can use "lctl debug_daemon /path/to/file", which writes the debug pages to a file (optionally with a maximum size), rather than running "lctl dk" in a loop.
Typically, lfsck should write any repairs done to the console log (check "lctl get_param printk" for "lfsck" being enabled to print on the console.
|
so we have 3 Lustre files/dirs somewhere that point to something that doesn't actually exist on an MDT?
Yes.
even with a new fs with only ~7m files in the filesystem, I'm not sure how this is going to work... is the idea to gather 'lctl dk' in a loop every few seconds, and then grep for something 'dangling' related? that sounds wrong... I'm guessing I'm missing something...
If there are too much logs as to the buffer overflow, then you can only enable "lfsck" debug via "lctl set_param debug=lfsck" on the MDT, then the logs will only contain LFSCK related operations.
if it's only 3 files then I'm not sure I care too much. 3 dirs might be different.
I am not sure whether the dangling entry is for file or dir. But there may not be too much different, because even if it was for dir, the LFSCK only can create empty dir object to repair if "-C" specified.
|
|
Hi Alex,
the LBUG is in warbles.txt above, and also warbles-messages-20180408.txt
console logs attached now too - warble1.log-20180408.gz , but looks the same to me.
the stack trace plus a little bit of important looking context (from warbles.txt) is ->
cheers,
robin
|
|
hmm. this did not go too well.
setting this ->
lctl set_param printk='warning error emerg console lfsck'
and then starting an lfsck effectively killed warble1.
I think what's happened is that all the lfsck traffic (about 2.4M lines of it) goes at serial baud rate console=ttyS0,115200n8 which stops the machine from working. it basically halted. couldn't ping on some interfaces. all clients evicted from dagg-MDT. eventually I think warble1 was stonith'd (which shouldn't have happened but that's another story).
on remount after reboot the lfsck seems to have automatically continued, which is unexpected, but not tragic in this case as 'printk' was back to default.
the debug_daemon approach seems to work.
thanks for that. haven't used that before.
I guess these are the problematic files? there's a lot of output
so I'm guessing the grep line a bit ->
which looks like 1 file and 2 dirs.
is there a tool to see which/where these are and who owns them or the parent?
cheers,
robin
|
|
You can find out the parent name path via "lfs fid2path" on the client, for example:
|
|
These two are for the broken striped directory [0x28001c995:0x1b3c7:0x0].
|
|
Hi,
we've hit a new LBUG when running lfsck with -C.
warble1 then failed over to warble2, the lfsck automatically continued, and then warble2 got the same LBUG.
and now we are back to the situation of site down, and 2 MDT's in WAITING state
hmm, actually I think perhaps it eventually completed, because it's LBUG'd again now.
I had this command entered in terminal and it appeared to be hung and waiting for the MDT's to complete, but perhaps it wasn't fast enough to stop the LBUG?
maybe it only turned it off on MDT0 and not MDT1,2?
is there a way to turn off 'lfsck' before mounting the MDTs, or do you have another suggestion for how to recover.
thanks.
cheers,
robin
|
|
ah, -o skip_lfsck.
perhaps this should be put into your recommended HA config defaults?
otherwise it's pretty hard to stop...
cheers,
robin
|
ah, -o skip_lfsck.
perhaps this should be put into your recommended HA config defaults?
otherwise it's pretty hard to stop...
Yes, please add "-o skip_lfsck" mount option if you want to prevent lfsck auto resuming from last checkpoint. I am sorry for the inconvenience. Please repeat the steps as described in the comment https://jira.hpdd.intel.com/browse/LU-10887?focusedCommentId=225392&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-225392 to make your system mountable again. I will work on the patch for the LBUG() caused by LFSCK.
|
|
nope.
(edit) my bad. had a leading '/'. seems to mount now.
seeing if they complete now...
cheers,
robin
|
|
yup. that worked. cool. filesystems all mounted up again, and I'll try to add -o skip_lfsck to our HA config.
here is the namespace lfsck info if you are interested.
thanks for your help.
cheers,
robin
|
|
Fan Yong (fan.yong@intel.com) uploaded a new patch: https://review.whamcloud.com/31915
Subject: LU-10887 lfsck: offer shard's mode when re-create it
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 99cf33f36e7951b6e166b6bb06243e60dda65aac
|
|
so this is probably a stupid question, but how do I see if a mounted lustre zfs dataset has been mounted with the -o skip_lfsck option?
I'm trying to add this to our HA config (the obvious place being /usr/lib/ocf/resource.d/lustre/Lustre), but having done a failover that should have been safe either way, now I can't see if -o skip_lfsck was set or not.
'mount' just says 'lustre (ro)' for everything. /proc/mounts the same. mtab is a symlink to /proc/self/mounts these days.
cheers,
robin
|
|
Here is my "mount" output for "-o skip_lfsck" specified when mount the MDT:
On the other hand, the mount option "-o skip_lfsck" is only used to prevent auto resuming LFSCK when mount, after that, such option has no other usage. So as long as the LFSCK is not auto started when mount the target as "lustre", then that is OK.
|
|
another LBUG. this one on umount of a MDT.
|
|
fs's mounted up ok.
I'm starting to think there's something broken in the hardware of warble1 that could be compounding these problems.
filesystems seem much faster on warble2. I'll try to test this theory tomorrow.
cheers,
robin
|
|
Robin, I might suggest that the system be left as-is for the time being, until a suitable maintenance window is available? That will allow users to get some work done while we test and review Fan Yong's patch (which should address at least one of the problems you are seeing).
|
|
Fan Yong (fan.yong@intel.com) uploaded a new patch: https://review.whamcloud.com/31929
Subject: LU-10887 mdt: ldlm lock should not pin object
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bc77552f9fbc09b1fcc3f29151fbfc0b47fcfbb1
|
|
Hi Andreas,
the system seems stable with all the MDTs on warble2 and mounted by-hand with -o skip_lfsck. so yup, happy to leave it that way for a while - both for patches to mature, and for us to swap out and try to diagnose potentially bad bits of warble1 hardware.
obviously at some stage we'll have to test (a hopefully improved) warble1 with Lustre again though, and unfortunately there are shared components (raid controller) that we can't test separately and offline.
Hi Fan Yong, so centos7 doesn't show any extra options in the 'mount' output unfortunately. even when I mount by hand I get just 'lustre (ro)'. I guess you're using a different OS. I don't suppose there's another way to see if -o skip_lfsck was asked for? 'dk', /proc, /sys? but anyway, no hurry 'cos we don't have any HA now so can't re-test anyway.
cheers,
robin
|
|
What is the output on the MDT?
|
|
all the dagg's were definitely mounted with -o skip_lfsck.
odd.
cheers,
robin
|
|
I know the reason. You need the patch: https://review.whamcloud.com/29228
Such patch has been landed to Lustre-2.11, not Lustre-2.10.3 yet.
|
|
oh, that makes sense. nice find!
hopefully we'll move to 2.11 about the 2.11.1 timeframe.
cheers,
robin
|
|
Robin
We have no immediate plans to do a 2.11.1 release but we can queue up this fix for inclusion in 2.10.4
Peter
|
|
Hi Peter,
oh, I see I haven't been reading roadmaps enough. I thought the plan was to keep folks rolling forward with 2.x releases and we were ok with that. I didn't realise 2.10.x was a LTS.
is it appropriate to drop this from severity 1 now? the fs is up and we're reasonably confident it'll stay that way.
cheers,
robin
|
|
Robin
Yes - 2.10.x is an LTS branch. We'd prefer to keep the ticket severity at S1 so that we correctly categorize it when we run future reports but we understand that the "all hands on deck" period is past and we're focusing on RCA and preventive actions to avoid future scenarios.
Peter
|
Fan Yong (fan.yong@intel.com) uploaded a new patch: https://review.whamcloud.com/31929
Subject: LU-10887 mdt: ldlm lock should not pin object
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bc77552f9fbc09b1fcc3f29151fbfc0b47fcfbb1
The object reference leak issue will be fixed via https://review.whamcloud.com/#/c/31431/ (2.10.4)
|
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/31915/
Subject: LU-10887 lfsck: offer shard's mode when re-create it
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 7d48050b7a3ba0b9db2ff823bc6fbc3091506597
|
|
Robin,
https://review.whamcloud.com/31915/ has been landed to master, https://review.whamcloud.com/#/c/31431/ has been landed to 2.10.4.
Any else do you need for this ticket?
|
|
Marking this is fixed in 2.12 so that this fix is also queued up for 2.10.4. Our goal would be for Swinburne to be running a vanilla release without the need for additional patches.
|
|
Minh Diep (minh.diep@intel.com) uploaded a new patch: https://review.whamcloud.com/32076
Subject: LU-10887 lfsck: offer shard's mode when re-create it
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 6510f5123b273419ee33a0ec6aaf297124dae155
|
|
Hi,
we tested warble1 hardware all we could for about a week and found no hardware issues. we also replaced sas cards and cables just to be safe.
warble1 now is 3.10.0-693.21.1.el7.x86_64 and zfs 0.7.8 and has these patches applied
when the dagg MDT's were mounted on warble1 they COMPLETED ok and then about 5 seconds later it hit an LBUG in lfsck.
I've failed the MDT's back to warble2 and mounted them by hand with -o skip_lfsck
cheers,
robin
|
usr/src/lustre-2.10.3/lu10887-lfsck.patch
So you have applied the patch https://review.whamcloud.com/31915/ on the MDT, right? If yes, there should be other corners for lfsck_namespace_repair_dangling() that missed to set the “mode”. I will investigate more. Anyway, the existed patches are still valid.
|
|
I've created a new ticket - LU-10988 - to track this further investigation. Please can we move any further discussion there? It helps with tracking fixes landing to releases to keep 1:1 mapping between tickets and fixes....
|
|
yes, that patch is what I've called lu10887-lfsck.patch and is applied.
cheers,
robin
|
|
John L. Hammond (john.hammond@intel.com) merged in patch https://review.whamcloud.com/32076/
Subject: LU-10887 lfsck: offer shard's mode when re-create it
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 4118b5317b23d6d2d6c09ca1cdc797ec027622c8
|
Generated at Sat Feb 10 02:39:03 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
conman-warble1-traces.txt
warble1.log-20180408.gz