[LU-15913] rename stress test leads to REMOTE_PARENT_DIR corruption

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.16.0
• Severity: 3
• Assignee: Andreas Dilger
• Reporter: Artem Blagodarenko (Inactive)

Description

Streess test with active renaming of files and directories inside a striped directory finished with an error:

Performing actions for 'rename(2,676pp674,aaaaaaaaaa,bbbbbbbbbb,cccccccccc,dddddddddd,eeeeeeeeee,ffffffffff,gggggggggg,hhhhhhhhhh,iiiiiiiiii,jjjjjjjjjj,kkkkkkkkkk,llllllllll,mmmmmmmmmm,nnnnnnnnnn,oooooo -> 3,676pp674,aaaaaaaaaa,bbbbbbbbbb,cccccccccc,dddddddddd,eeeeeeeeee,ffffffffff,gggggggggg,hhhhhhhhhh,i) error Input/output error (5)'
numerrs=1
Fri Jun  3 01:06:30 CDT 2022 

leads to corrupted MDT partition. E2fsck reports next symptoms:

1) incorrect filetype

2) a link to directory

3) .. points to the REMOTE_PARENT_DIR but should point to the special directory with sequence name

./data.20220423/server/e2fsck.pre_read_only.kjcf04n03.6.1-010.43.20220504165818.out.kjcf04n03:Entry '14,37275pp37273,aaaaaaaaaa,bbbbbbbbbb,cccccccccc,dddddddddd,eeeeeeeeee,ffffffffff,gggggggggg,hhh' in /REMOTE_PARENT_DIR/0x24006497a:0x129bc:0x0 (4014289161) has an incorrect filetype (was 17, should be 2).

./data.20220423/server/e2fsck.pre_read_only.kjcf04n03.6.1-010.43.20220504165818.out.kjcf04n03:Entry '0x24006b05d:0xa4c8:0x0' in /REMOTE_PARENT_DIR (4030089985) is a link to directory /REMOTE_PARENT_DIR/0x24006497a:0x129bc:0x0/14,37275pp37273,aaaaaaaaaa,bbbbbbbbbb,cccccccccc,dddddddddd,eeeeeeeeee,ffffffffff,gggggggggg,hhh (4014289162 fid=[0x20006a4b1:0x42c6:0x0]).

./data.20220423/server/e2fsck.pre_read_only.kjcf04n03.6.1-010.43.20220504165818.out.kjcf04n03:'..' in /REMOTE_PARENT_DIR/0x24006497a:0x129bc:0x0/14,37275pp37273,aaaaaaaaaa,bbbbbbbbbb,cccccccccc,dddddddddd,eeeeeeeeee,ffffffffff,gggggggggg,hhh (4014289162) is /REMOTE_PARENT_DIR (4030089985), should be /REMOTE_PARENT_DIR/0x24006497a:0x129bc:0x0 (4014289161).  

 

Issue Links

• is related to: LU-12834 MDT hung during failover
• is related to: LU-15491 rename deadlock
• is related to: LU-17448 LBUG in racer with layout change
• is related to: LU-14570 e2fsck reports "Entry '..' an incorrect filetype (was 18, should be 1)"
• is related to: LU-12125 Allow parallel rename of regular files
• is related to: LU-15830 distribute mkdir should lookup target name
• is related to: LU-17426 parallel cross-directory rename of regular files on single MDT

Comments

[Andreas Dilger]

Patch was landed to master in 2.15.51.

[Gerrit Updater]

"Etienne AUJAMES <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57815
Subject: LU-15913 mdt: disable parallel rename for striped dirs
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: 67c9076e4aed3b20f1bae576de4c9f3f9619a130

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/55143/
Subject: LU-15913 tests: fix set_params_xxx
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 214432e2595715458102365870e4573eb672439c

[Gerrit Updater]

"Sergey Cheremencev <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/55143
Subject: LU-15913 tests: fix set_params_xxx
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bcf4095b63d7e46ab017afaaae0bfb7a5e51112c

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/47643/
Subject: LU-15913 tests: add rename stress test via racer
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9a1d68f9b8d9dc7edcbcbc9543450e046c8303b4

[Gerrit Updater]

"Sergey Cheremencev <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53981
Subject: LU-15913 tests: clean between racer 1 and 2
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 81cb6f72de12dc6a3aca486a8585405dca755fc3

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53768
Subject: LU-15913 tests: add rename stress test via racer (testing)
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b49b73d379b33705688a82c535b824c660c64641

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/47593/
Subject: LU-15913 mdt: disable parallel rename for striped dirs
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: f238540c879dc668e18cf99cba62f117ccae64d6

[Cory Spitz]

> Was testing not being done with 2.15.0-RC5?
Correct. The report was filed for the previous RC. I regret that that wasn't made clear. But, at least it has uncovered this broken rollback, which could still happen in theory for other reasons (say, an ENOMEM experienced along the way).

[Alex Zhuravlev]

one option is to make the failing update (e.g. insert) the very first update in the batch so there would be nothing to rollback.