[LU-3719] divide error in ldiskfs_mb_normalize_request on MDT

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.1.5
• Fix Version/s: Lustre 2.5.0, Lustre 2.4.2, Lustre 2.11.0, Lustre 2.10.2
• Type: Bug
• Priority: Major
• Reporter: Kit Westneat (Inactive)
• Assignee: Zhenyu Xu
• Resolution: Fixed
• Labels: mn1
• Severity: 3

Description

IU ran into an issue on their MDT where it would constantly crash after recovery. We finally got a good core dump and were able to get this bt:
#6 [ffff882ff8ded040] divide_error at ffffffff8100bdfb
[exception RIP: ldiskfs_mb_normalize_request+244]
[exception RIP: ldiskfs_mb_normalize_request+244]

This looks almost identical to LU-2480, except that it is occurring on the MDT.

We were able to get the MDT mounted after running e2fsck and then tune2fs -E stripe_width=0,stride=0. Apparently mke2fs had set them based on values from LVM.

I checked the line at ldiskfs_mb_normalize_request+244 in mballoc.c:
wind = sbi->s_mb_prealloc_table[i - 1];
tstart = ac->ac_o_ex.fe_logical;
do_div(tstart, wind);

It looks like the s_mb_prealloc_table isn't getting fully populated. I inspected it with crash, and that looks to be the case:
crash> x/3xg 0xffff881822cc4d40
0xffff881822cc4d40: 0x0000000000000c00 0x0000000000001800
0xffff881822cc4d50: 0x0000000000000000

0xc00 (3072) was the reported stripe_width by dumpe2fs. It appears that ldiskfs_mb_init attempts to create three entries in the table, stripe * 1, *2, and * 4. However, ldiskfs_mb_prealloc_table_add can silently fail if the entry value is > (sbi->s_blocks_per_group - 1 - 1 - sbi->s_itb_per_group). This can cause a situation where the table size is still 3, but one or more entry is zero.

I'm not sure what the best fix is. It seems as if returning an error from ldiskfs_mb_prealloc_table_add and adjusting the table size would be ideal. Alternatively, ldiskfs_mb_normalize_request could check to make sure the table doesn't have a zero, something like:
+ for (i = 0; i < sbi>s_mb_prealloc_table_size; i++) {
++ for (i = 0; i < sbi->s_mb_prealloc_table_size && sbi->s_mb_prealloc_table[i]; i++) {

Thanks.

Issue Links

• is related to: LU-9814 prealloc table change don't work afte...
• is related to: LU-9617 RHEL-7 ext4-prealloc.patch missing er...

Comments

[Peter Jones]

Thanks for the report Kit.

[Peter Jones]

Bobijam

Could you please advise on this one?

Thanks

Peter

[Zhenyu Xu]

patch tracking at http://review.whamcloud.com/7297

[Zhenyu Xu]

master version http://review.whamcloud.com/7591

[Peter Jones]

Landed for 2.5.0

[Bob Glossman (Inactive)]

I notice only the 6.4 version of the ldiskfs patch has been changed. Do other versions also need adjustment with similar changes?

[James A Simmons]

Yes the SLES11 platforms need to be updated as well.

[Bob Glossman (Inactive)]

Reopened to address other versions of the ldiskfs patch. In particular need similar changes in the sles11 version.

[Bob Glossman (Inactive)]

sles11 sp2/sp3 version
http://review.whamcloud.com/7781

[Peter Jones]

Landed for 2.5

[Gerrit Updater]

Yang Sheng ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27748
Subject: LU-3719 ldiskfs: adjust s_mb_prealloc_table_size correctly
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: cd2ab45b96718107d20cb95fcf91ef1a8ac2c41d

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/27748/
Subject: LU-3719 ldiskfs: adjust s_mb_prealloc_table_size correctly
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b9ecf2671cba08530cc49d47713a360f2c6c25ca

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28228
Subject: LU-3719 ldiskfs: adjust s_mb_prealloc_table_size correctly
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: f6cc8c6f7825cefa7a33ed8438cc203cbe8d7183

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/28228/
Subject: LU-3719 ldiskfs: adjust s_mb_prealloc_table_size correctly
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: ec1dfdfae8b860af54ca0fdc4d035870e6338017