Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Lustre 2.15.2
-
ZFS MDT
-
3
-
9223372036854775807
Description
While doing several directory restripes (parent dir only with "lfs migrate -m -d") at the same time, the Lustre client lost connection with the typical messages on the MDS asking to resume the migration process.
Resuming doesn't work, starting the migration process on affected directories returns "File descriptor in bad state (77)".
The current stripe setting of the affected directories show hash types of "none" and "bad_type", e.g.
lmv_stripe_count: 5 lmv_stripe_offset: 2 lmv_hash_type: none,bad_type mdtidx FID[seq:oid:ver] 2 [0x280000bd1:0x6fb:0x0] 0 [0x200000400:0xd16:0x0] 2 [0x280000402:0xd16:0x0] 1 [0x240000401:0xd16:0x0] 3 [0x2c0000402:0xd16:0x0]
The directories can be traversed, changes on the directories can't be done (adding or removing files).
Several directory trees are affected through multiple levels. Trying to migrate the directories from the top level results in "Invalid argument (22)" instead of -EBADFD.
Debug messages on the MDS show that lmv_is_sane() is failing when trying to access the broken directories:
Thu Apr 6 09:36:53 2023] LustreError: 1952126:0:(lustre_lmv.h:468:lmv_is_sane2()) insane LMV: magic=0xcd20cd0 count=5 index=1 hash=none:0x20000000 version=3 migrate offset=1 migrate hash=fnv_1a_64:33554434. [Thu Apr 6 09:36:53 2023] LustreError: 1952126:0:(lustre_lmv.h:446:lmv_is_sane()) insane LMV: magic=0xcd20cd0 count=5 index=1 hash=none:0x20000000 version=3 migrate offset=1 migrate hash=fnv_1a_64:33554434.
Removing of the directories is also imposssible ("cannot remove <path>: invalid argument").
LFSCK only has "striped_shards_skipped", but doesn't repair anything.
The migration might have been started in parallel on several levels within a directory tree, maybe causing races or corruption. lfs migrate
If folders are affected, they are part of a whole affected path of directories from root+1 to a directory with no more subdirectories within, e.g.
/root/dir1/dir2/dir3 with directories dir1 dir2 dir3 being affected.
Is there any way to repair or remove the affected files? Data loss is not an issue for us, as this is a pre-production test setup, I want to report the issue anyway, because the problem immediately occured on starting the migration.
Attachments
Issue Links
Activity
2023-05-29: Both two patches being worked on are ready to land(on master-next branch).
2023-05-20: Two patches being worked on, the first patch is being reviewed, another one passed code-review and is depending on the first one.
Sorry for the late response (LUG was in the way).
I can confirm that removing the directories was possible in all cases with the patched server. LFSCK didn't seem to repair anything, although I'm not sure if its supposed to change anything for directories that have already been broken by my previous LFSCK runs.
I will have to setup the filesystem again, as we are going into production and can't do any more testing.
Patrick, I just pushed two patches to address the issues listed above, will you try the latter one https://review.whamcloud.com/c/fs/lustre-release/+/50797 to see if it can fix your issue?
"Lai Siyao <lai.siyao@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50797
Subject: LU-16717 mdt: resume dir migration with bad_type
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4054056c6268ee47850f54a57019b819f108738f
"Lai Siyao <lai.siyao@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50796
Subject: LU-16717 mdt: treat unknown hash type as sane type
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: be2d62766e7a0d75846c9482f8105fd5ae7bf461
Lai, I've always wondered about cases like this (eg. bad hash type). It should be possible for the MDS to "complete" the migration in any case. At worst it could pick a new hash type and move more entries around, but at least it would get the directory out of the "MIGRATION" state.
It's not supported to resume directory migration upon bad hash type. Under this condition, user can still access this directory and sub files, but can't unlink them because server code doesn't support name lookup with bad hash (it should try each stripe as what's done on client).
I will look into these two issues:
- upon bad hash type, directory migration can try to resume any way, and there is big chance of success.
- support unlink of such directory and files.
I've attached logs (debug_migrate_rmdir_leafdir) that show the debug output when trying to migrate and remove an affected directory. I ran the command on the MDS with MDT0002 because I think that's where the issue occurs.
I created the directory list that i used for directory migration after the first "rc = -22" errors apeared, so the issue might have been present before restriping of the directories even started. The fids affected at that time can't be resolved bs lfs fid2path right now due to -EBADFD.
I also did restriping of DoM prior on the same day, so that might actually be the root cause of the issue. attached the kernel messages for Lustre on the client doing the DoM migration in dmesg_dommigrate. I left the dump_lsm() messages in for completeness, although the messages seem to be redundant.
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50796/
Subject:
LU-16717mdt: treat unknown hash type as sane typeProject: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 05cdb71ba6813570123613993f3cfcf74fc83561