Details
-
Bug
-
Resolution: Fixed
-
Major
-
None
-
None
-
None
-
3
-
9223372036854775807
Description
e2fsck spends 72 hours to check ~113T OST. Most time in phase 2.
Profiler shows that most expensive path is:
e2fsck_run -> e2fsck_pass2 -> check_dir_block -> e2fsck_process_bad_inode -> e2fsck_read_inode
After adding -d option we get millions of such messages:
e2fsck_pass1:1543: increase inode 12485987 badness 0 to 2
Which correspond to this piece of code:
else if (EXT4_XTIME_ANCIENT(ctx, sb, inode->i_ctime, ctx->time_fudge)) e2fsck_mark_inode_bad(ctx, ino, BADNESS_HIGH);
Code check if ctime is too old. But Lustre FS adds precreated object with zeroed time. So every such object assumed as bad by e2fsck. By some reasons (this is other issue topic) we get millions of precreated files and e2fsck spends a lot of time to process this file during phase 2.
After removing check above e2fsck completes its work after 20 minutes.
Two possible solutions are suggested: 1) remove this check, because having zeroed chime is possible situation for Lustre FS 2) remove inode badness patches. Does it give enough advantages for such overhead.
Attachments
Issue Links
Activity
| Status | Original: Resolved [ 5 ] | New: Closed [ 6 ] |
| Resolution | New: Fixed [ 1 ] | |
| Status | Original: Open [ 1 ] | New: Resolved [ 5 ] |
| Description |
Original:
e2fsck spends 72 hours to check ~113T OST. Most time in phase 2.
Profiler shows that most expensive path is: {code} e2fsck_run -> e2fsck_pass2 -> check_dir_block -> e2fsck_process_bad_inode -> e2fsck_read_inode {code} After adding -d option we get millions of such messages: {code} e2fsck_pass1:1543: increase inode 12485987 badness 0 to 2 {code} Which correspond to this piece of code: {code} else if (EXT4_XTIME_ANCIENT(ctx, sb, inode->i_ctime, ctx->time_fudge)) e2fsck_mark_inode_bad(ctx, ino, BADNESS_HIGH); {code} Code check if ctime is too old. But Lustre FS adds recreated object with zeroed time. So every such object assumed as bad by e2fsck. By some reasons (this is other issue topic) we get millions of recreated files and e2fsck spends a lot of time to proceed this file during phase 2. After removing check above e2fsck completes its work after 20 minutes. Two possible solutions are suggested: 1) remove this check, because having zeroed chime is possible situation for Lustre FS 2) remove inode badness patches. Does it give enough advantages for such overhead. |
New:
e2fsck spends 72 hours to check ~113T OST. Most time in phase 2.
Profiler shows that most expensive path is: {code} e2fsck_run -> e2fsck_pass2 -> check_dir_block -> e2fsck_process_bad_inode -> e2fsck_read_inode {code} After adding -d option we get millions of such messages: {code} e2fsck_pass1:1543: increase inode 12485987 badness 0 to 2 {code} Which correspond to this piece of code: {code} else if (EXT4_XTIME_ANCIENT(ctx, sb, inode->i_ctime, ctx->time_fudge)) e2fsck_mark_inode_bad(ctx, ino, BADNESS_HIGH); {code} Code check if ctime is too old. But Lustre FS adds precreated object with zeroed time. So every such object assumed as bad by e2fsck. By some reasons (this is other issue topic) we get millions of precreated files and e2fsck spends a lot of time to process this file during phase 2. After removing check above e2fsck completes its work after 20 minutes. Two possible solutions are suggested: 1) remove this check, because having zeroed chime is possible situation for Lustre FS 2) remove inode badness patches. Does it give enough advantages for such overhead. |