|
oh, and our lustre (client and server) were patched with https://jira.hpdd.intel.com/browse/LU-10212 https://review.whamcloud.com/#/c/29992/ 'cos we were hitting ESTALE on clients.
|
|
Robin, after the "rm -r" fails, if you do "ls -lR" of the directory, does it show any entries left behind, or is the directory empty? It is possible there is some problem iterating the directory entries during the rm that leaves something behind.
Lai, are you able to reproduce this?
|
|
Hi Andreas,
the directory 2556/ is empty, but can't be rmdir'd.
cheers,
robin
|
|
I'm trying to reproduce this, but haven't seen in smaller scale on master and 2.10 yet. In the mean time could you do the following to dump debug logs?
1. 'lctl set_param debug=-1' on 2 MDS's.
2. 'lctl clear' on 2 MDS's to clear previous debug logs.
3. 'rm /dagg/old_stuff/rjh/mdtest/mdt0-2/2556'
4. 'lctl dk /tmp/10677-`hostname`.log' to collect debug logs on 2 MDS, and attach them here.
Also could you run 'lfs getdirstripe /dagg/old_stuff/rjh/mdtest/mdt0-2/2556' and 'stat /dagg/old_stuff/rjh/mdtest/mdt0-2/2556', and post the output here?
|
|
Hi Lai,
yeah, sorry, I should have mentioned - scale is important.
the above reproducer makes 10k dirs, each with 100 subdirs.
I also ran a bunch with 1k dirs, each with 100 subdirs, and they all worked fine.
so something about 10k (and presumably above) triggers it for us.
cheers,
robin
|
|
Hi Lai,
I couldn't delete these dirs (there are 2 now), but I did mv them so I could do more mdtests, so in the below and the attached the paths differ slightly from the above, but it's the same dir.
|
|
and just in case it isn't obvious from the logs, the current arrangement of MDTs is
so dagg mdt0 on warble2, and mdt1,2 on warble1. each has their own zpool.
|
|
the 'stat' result shows 'Links' count is 3, while the correct link count of an empty dir should be 2, I'm afraid some error in 'rm' caused one link not released. As you said there is nothing in dmesg or syslog, did you check them on MDS's?
|
|
Hi Lai,
there is nothing in dmesg or syslog. either when this happens the first time, or on subsequent attempts to remove the directory.
to check this again, I re-ran my reproducer above. 1 attempt with 10k dirs and 1 attempt with 20k dirs didn't produce any corrupted dirs, but the next run got 2 corrupted (unremoveable) dirs, and again only on the -c3 -D directory ->
looking at the broken dirs ->
one difference between the 2 recent ok runs, and this incorrect run, is that I moved the 3 MDTs apart onto their (normal) 2 separate MDS's for this run. for the 2 previous runs all the MDTs had been on one MDS. however as the first post in this ticket said, I've also been able to hit the bug when all the MDTs were on all MDS. it does seem slightly easier to trigger when they're spread out though.
please note we have also moved to zfs 0.7.6. the first corruption in this ticket was with 0.7.5. so ZFS version doesn't seem to matter.
cheers,
robin
|
|
I see, it looks like there is race in refcounting somewhere, I'll check related code.
|
|
Hi,
I'd also greatly appreciate advice on how to delete the residual directories created from the above testing in order to tidy up the filesystem.
thanks.
I presume I have to mount the MDTs as zfs and rm something by hand?
cheers,
robin
|
|
yes, I think so.
|
|
can you be more specific please.
thanks.
cheers,
robin
|
|
The 10677-warble2.log shows that:
==> for "dot".
==> for "dotdot".
==> anther unknown entry.
If the log is for the trying remove the trouble directory, then means that the first stripe of the striped directory contains at least one unknown entry (neither "dot" nor "dotdot"). But according to your description, such entry is invisible to client via "ls -al". Since it is test environment, I would suggest that if you can reproduce the trouble, please mount related MDT as "zfs" and "ls -alR" on such trouble directory (its first stripe) via ZPL. If still show nothing, then I suspected that there are some dummy entries left in such stripe.
NOTE: Please NOT remove such trouble directory via ZPL directly.
|
|
Hi. thanks for looking into this.
actually it's not a test filesystem now, it's the main production filesystem.
it doesn't seem possible to mount a zfs snapshot whilst lustre is running (please let me know how to do this if it's possible), so I did a short cluster downtime.
we have (at least) 4 dirs so far ->
which can't be deleted
with striping
I picked the first one and mounted MDT1 as zfs
and there is indeed something in there. 2 levels of dirs in fact ->
cheers,
robin
|
/warble1-dagg-MDT1-pool/MDT1/oi.65/0x280019a41:0xa9dc:0x0/58:
total 75
drwxrwxr-x 5 rhumble hpcadmin 2 Feb 16 19:28 .
drwxrwxr-x 3 rhumble hpcadmin 2 Feb 16 20:07 ..
drwxrwxr-x 2 rhumble hpcadmin 2 Feb 16 19:28 [0x280019a41:0xa9f1:0x0]:0
This is not a common directory, instead, it is the shard of another striped directory - "58". Normally, if you did not set stripe for the directory "58", it should have 3 shards by default that was inherited from its parent. But as shown, there is only one shard. So I suspected that something wrong when removed the striped directory "58", as to the other two shards has been destroyed, but failed to removed the local shard. With the corrupted striped directory "58", we cannot remove the parent directory - I assume it is the "0291".
Please find a new client, run the following:
|
|
|
[root@john6 corrupted]# ls -ailR 0291/58
0291/58:
total 150
180145747155549269 drwxrwxr-x 2 rhumble hpcadmin 76800 Feb 16 19:28 .
180145747155549248 drwxrwxr-x 3 rhumble hpcadmin 76800 Feb 16 20:07 ..
So the corrupted directory "58" is still visible. Please try luck to do "rm -rf 0291/58" although I am afraid you cannot do that successfully. Please collect -1 level debug logs on the MDT when rm. Thanks!
|
|
wow! that worked. well done! 
and I could remove the parent now too
so now 3 problem dirs instead of 4.
I suspect I can probably just
in each of those and they'll be ok again too... sweet. I won't do it just yet though in case you want to experiment with other things.
cheers,
robin
|
|
ok, this is crazy - each of the 3 remaining dirs has a '58' subdir ->
and each of the '58' shards is on the same MDT as the parent dir ->
I thought 42 was the magic number, not 58
cheers,
robin
|
|
That is unbelievable..., in spit of "58" or "42", just the name, nothing special. May be related with your test parameters? What are the parameters you used to reproduce the trouble? Have you adjusted them during different instances? Change it, you may got different results.
|
|
the reproducer is the 1-liner in the first post. I've run it with 1k, 10k and 20k for the first level dirs, but always (as far as I can remember)
{00..99}
for the second level dirs. 10k and 20k don't always create problems, but eventually do. 1k seems too small to generate problems.
could there be something in Lustre directory randomisation for which 58 is special?
that's the only thing that really makes sense...
'58' is the 59th dir created in the parent, so that's 111011. is 4 special somehow because we have 3 MDT's to hash across? more buckets in a hash table? hash seed? dunno... I'm just guessing.
I'll run a few more tests the same and see if they also come up with '58', and then try a few variations.
cheers,
robin
|
|
Please adjust the 2nd level sub-dirs counts for try.
|
|
I ran some more. they take a while, so apologies for the delay.
after a couple of runs through via the usual command line I got an invisible directory:
so, darn, '58' isn't so special, that would have been awesome
I also tried new variants with more subdirs but didn't hit any problems. then I tried a variant with less subdirs
and got 4 busted dirs on the first run. as per usual these 4 dirs appear empty to ls -l
but actually have these dirs in them which I can see if I explicitly ls -l them
so oddly 5* subdirs still seems preferred, but there's one 9* now, so it's possibly just a preference for mid-located dirs for some reason.
cheers,
robin
|
|
rjh, thanks for the further verification. The results are in the expectation. The remaining shard of striped directory was not related with any special name, instead, I suspect that it may be related with some in-complete rmdir (striped directory) operation. Although we try to guarantee the atomicity for cross-MDTs modification, but it is difficult to say (or prove) that all the corner cases are properly handled, especially for error cases handing. Now, we need the logs, please help to collect the MDS side Lustre kernel debug logs (the details the better) when reproduce the trouble. Thanks!
|
|
Hi,
the MDS logs for rm'ing the '58' dir has been attached to this ticket for a while now.
are you asking for logs whilst I create 1M dirs? surely that will overflow all the log buffers on the MDS's and be useless?
cheers,
robin
|
|
What I need is the Lustre kernel debug logs when failed to rmdir the striped directory (such as the "58"). It is difficult to know in advance which striped directory will be the one.
|
|
ok. I understand now. thanks.
I'll try to find a shorter reproducer and run the 'lctl debug_daemon' with '-1' whilst I running the 'rm -rf' parts.
cheers,
robin
|
|
Hi Fan Yong,
just to let you know I've put this on the backburner and won't be doing any more on this bug until we have resolved most of LU-10887 and have repaired the current fs errors. basically I'm reluctant to make things worse by adding more errors to the fs at the moment.
it'll be interesting to see if a successful lfsck pass can repair the current 93410/5 59234/5 46886/5 24538/9 'hidden' directories.
cheers,
robin
|
|
Hi Robin,
As you can see, we already have three patches for LU-10887, one is for LFSCK trouble (https://review.whamcloud.com/31915), another is for object leak (https://review.whamcloud.com/31929), the 3rd one (https://review.whamcloud.com/29228) for showing mount options. Please keep update on such ticket when you need more helps.
|
|
just so you know, these directories that can't be deleted 'cos they have 'hidden' dirs in them are still there and behave the same after the namespace lfsck in LU-10988. so lfsck missed them somehow.
there are at least 9 directories like this at the moment.
cheers,
robin
|
just so you know, these directories that can't be deleted 'cos they have 'hidden' dirs in them are still there and behave the same after the namespace lfsck in LU-10988. so lfsck missed them somehow.
there are at least 9 directories like this at the moment.
I am afraid that these 9 directories are the ones skipped by the namespace LFSCK. I would suggest to enable LFSCK debug (lctl set_param debug+=lfsck) on the MDTs, then re-run namespace LFSCK. Please collect the debug logs on the MDT that will tell us which directories are skipped and why. Please use large debug buffer and mask unnecessary debug components to avoid debug buffer overflow.
|
|
scadmin when do you expect to be able to run this test?
|
|
Hi,
sorry. yes, I should have got back to this, but it's been a pretty low priority for us. and high risk as it turns out 'cos the MDS crashed whilst running the lfsck (see LU-11111)
we did get as far as this though ->
which perhaps is enough to help.
would you like me to upload the debug_file.txt.gz to somewhere, or grep for something? it's about 300M gzip'd.
cheers,
robin
|
|
Robin
There is a Whamcloud ftp site that you could upload large files to. I can give you details directly (i.e via email) if you wish to do this
Peter
|
|
I'm working on patch to improve lfsck on this, will provide a patch later.
|
Generated at Sat Feb 10 02:37:13 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
10677-rm58-warble1.log