|
Lai, could you please take a look?
Stephane, is there anything particular about the mdtest directory that might trigger this (number of files, striping, etc)?
|
|
Hey Andreas,
The mdtest directory contains only a few sub directories (shown below), but the whole tree has 298,653 files.
Actually, I now noticed that lmv_stripe_offset for 32142854 is 1, and the directory is already stripped over 2 MDTs. Unfortunately I don't have the output of getdirstripe before the lfs migrate sorry... The lmv_hash_type: -2147483646 looks suspicious too. Please note that this test filesystem was formatted using 2.11.x with an old version of e2fsprogs (1.42.13.wc6-7).
|
|
The lmv_hash_type printing is just a minor bug in lfs getdirstripe - it should be printing the hash type as an unsigned hex value 0x80000002. There is a special flag (#define LMV_HASH_FLAG_MIGRATION 0x80000000) set on migrating directories to indicate that clients may need to check multiple MDTs to find filenames that are in the middle of migration.
|
|
It looks that there is bug in readdir code between MDTs, normally this code is not called, so this bug was not seen before, but dir migration needs to readdir from remote MDT, which triggers this BUG, I'm reviewing related code and working on the fix.
|
|
can you use 'echo t > /proc/sysrq-triggers' to dump all processes backtrace on MDT1? this crash should have happened on MDT0, right?
|
|
Hi Lai,
Yes it happened on MDT0 (on server fir-md1-s1 which is running MDT0 and MDT2). The other server fir-md1-s2 is running MDT1 and MDT3 and I'm attaching the output of sysrq t there, however, please note that since the crash, I have restarted fir-md1-s1 and MDT0,2. If you want the dump just after the crash, I can retry to trigger it tomorrow.
|
|
Here is another occurence of this issue when doing a lfs migrate -m 1 of a copy of the directory:
MDT0 crashed
commands on clients:
vmcore (mdt0) uploaded to the ftp as LU-11753-vmcore-fir-md1-s1-2018-12-11-14-48-11
debuginfo rpms available there too
output of sysrq trigger "t" on mdt1 (fir-md1-s2) while fir-md1-s1 was NOT restarted yet have been attached as fir-md1-s2_sysrq_t_2.log
|
|
Thanks, I'll check that, can you also post the script of the whole process, which can help to add a test for this.
|
|
Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/33837
Subject: LU-11753 obdclass: index_page support variable length rec
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 7924513ad8ac82971b4d93914c1dc6d7b679610d
|
|
Hi Lai,
I just tried your patch on top of 2.12.0 RC2 and did a lfs migrate -m 1 on a copy of the original directory, and a crash occurred again (sorry!). Please see attached file vmcore-dmesg-fir-md1-s1-2018-12-12-12_18_37.txt
The whole process is that I recursively copied a directory where I previously generated a lot of files using mdtest. The directory is stripped on MDT0, then I just lfs migrate it to MDT1.
Perhaps something is wrong with the original directory 32142854? It's possible that is is only partially migrated. But in any case, it would be nice to avoid the crash. Thanks!
|
|
Is it possible that this is related to LU-11330, which causes errors like "rm: cannot remove '/mnt/lustre/d70d.replay-single/test1': Directory not empty"?
|
|
Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/33843
Subject: LU-11753 utils: print out DNE2 directory hash flags
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 3e8cc55044cf097450493f0cf9bfded24f356ef7
|
|
Stephane, can you apply the above patch on MDS and test again?
|
|
Lai, I already tried with https://review.whamcloud.com/33837 (patchset 1), see my comment above.
|
|
Stephane, can you tar the whole directory of 32142854-copy2 and upload it? I want to test with it myself.
|
|
Lai, sure, this is done!
|
|
What is strange are the errors reported during copy in Stephane's test:
Is that because the mdtest/32142854-copy directory already exists before the copy and those directories already exist, or are these errors generated when copying to a new directory?
|
|
Indeed that is strange. The destination directory doesn't exist before the copy. I noticed that these two directories have several Links, esp #test-dir.2/mdtest_tree.0 which has many of them..
|
|
Never mind about the link count, it's just that this directory has a lot of sub directories.
|
|
Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/33865
Subject: LU-11753 obdclass: lu_dirent record length missing '0'
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: aea1bb0102fb7eb23e5df592ab401f470faef27c
|
|
Stephane, I uploaded another patch, can you apply both and test again?
|
|
Lai,
Good news, MDS is not crashing anymore and lfs migrate worked!!
Version tested (servers and client): 2.12.0 RC2 + 3 patches:
Results (no crash!):
=> Success 
Interestingly, during lfs migrate, the dir is stripped on the 2 MDTs (source and destination) and seen as migrating, which seems perfectly normal:
Another minor thing to note, is that after the migration, I got the following error from another terminal with a CWD within the migrated directory:
This is probably a limitation of lfs migrate at the moment but I'm fine with that!
Thanks much Lai!
|
|
Hey, I just wanted to add that BOTH patches from Lai really seem to be needed to make the MDS crash go away and lfs migrate actually work. I already knew that the first one (https://review.whamcloud.com/33837) alone was not enough, and so FYI, I just tried with only this one https://review.whamcloud.com/33865 and got a crash in dt_index_page_build.
|
|
sthiell could you please re-try with the latest versions of the patches - Alex made a tweak because of some issues that came up during regression testing....
|
|
Peter, ok great, no problem! working on it now.
|
|
Patch tested on top of 2.12.0 RC2:
https://review.whamcloud.com/#/c/33837/4
https://review.whamcloud.com/#/c/33865/1
Result: success!
BTW I believe the "File exists" errors are due to the fact that the original directory 32142854 is partially migrated and MDS crashed several times, so maybe the two directories are present in both MDT? I don't want to resume its migration now because the directory is part of my reproducer for the current issue, but we can probably confirm that later.
Thank you all!
|
|
Fantastic news - thanks Stephane!
|
I got the following error from another terminal with a CWD within the migrated directory:
Yes, migrating files/directories between MDTs changes their FIDs, which is important to note if your HSM copytool depends on FIDs... There is LU-7607 "Preserve inode number after MDT migration" but it is not implemented yet.
|
|
Ok I see, thanks Andreas! (the correct LU# seems to be LU-7607)
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33843/
Subject: LU-11753 utils: print out DNE2 directory hash flags
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 795eea12271f5c1bab3414803db3538d7c266d66
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33837/
Subject: LU-11753 obdclass: index_page support variable length rec
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a13e4991a1350f54f97c6ba13686d33c7a3eeb57
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33865/
Subject: LU-11753 obdclass: lu_dirent record length missing '0'
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 77f01308c5095030ee84f83339c085bcbcf04155
|
|
Landed for 2.12
|
|
Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34376
Subject: LU-11753 utils: print out DNE2 directory hash flags
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 47096b801de9b70f01caa7aae4104f2b851bc474
|
Generated at Sat Feb 10 02:46:37 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
32142854-copy2.tar.gz
fir-md1-s2_sysrq_t.log