[LU-13187] sanity test_129: current dir size 4096, previous limit 20480

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.14.0, Lustre 2.12.6
• Affects Version/s: Lustre 2.12.5
• Severity: 3
• Assignee: Dongyang Li
• Reporter: Maloo

Description

This issue was created by maloo for Andreas Dilger <[e-mail]>

This issue relates to the following test suite run:
https://testing.whamcloud.com/test_sets/2c10b9da-44b8-11ea-bffa-52540065bddc

test_129 failed with the following error:

current dir size 4096,  previous limit 20480

It looks like this started on 2020-01-28 when a number of patches landed.

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity test_129 - current dir size 4096, previous limit 20480
sanity test_129 - dirsize 4096 < 32768 after 93 files

Issue Links

• is related to: LU-13916 sanity test_129: dirsize 4096 < 32768 after 93 files
• is related to: LU-11310 support for SLES 15

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/39882/
Subject: LU-13187 osd-ldiskfs: don't enforce max dir size limit on IAM objects
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: a73f4e566debadfc156b6d8c48237a2e34ac75ba

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39882
Subject: LU-13187 osd-ldiskfs: don't enforce max dir size limit on IAM objects
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: d8c40507e87798e37d05988d190ccef78b528c42

[Peter Jones]

Landed for 2.14

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/39823/
Subject: LU-13187 osd-ldiskfs: don't enforce max dir size limit on IAM objects
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 03e6db505be90d35ccacb3af7e15277784e5d448

[Gerrit Updater]

Li Dongyang ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39823
Subject: LU-13187 osd-ldiskfs: don't enforce max dir size limit on IAM objects
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 07c97b04411a396572d7124f3217a7d561a96d2b

[Andreas Dilger]

Neil, thanks for tracking this down.

Dongyang, I think it would be better avoid changing the API for ext4_append(), as that would need even more changes to the core ext4 code.

I think there are two options that are relatively simple:

• split ext4_append() into an outer function of the same name that checks the directory size limit, and a second internal function (e.g. ext4_append_nolimit() or similar) that can be called directly from the iam_* functions
• set a new EXT4_STATE_IAM flag on the IAM objects when they are opened, and check that inside ext4_append() when checking the size limit

Probably the second option is less intrusive, as it is likely that patch could avoid conflicts with ext4-pdirop.patch and ext4-misc.patch, and hopefully would not need to be different for every kernel.

[Dongyang Li]

Great, I just could not reproduce the problem.

if it's failing with ENOSPC from osd_ldiskfs_append(), I think we can add a new param to ldiskfs/ext4_append() to bypass the limit check for the oi related code path, like iam_new_node(), iam_lfix_create() and iam_lvar_create().

the normal dir is using a different code path, osd_ldiskfs_add_entry()->__ldiskfs/ext4_add_entry()

 

[Neil Brown]

I think this problem is caused by some metadata directory trying to grow.

I added some tracing and found that the call to osd_ldiskfs_append() in iam_new_node() was failing with ENOSPC.

Maybe the best fix would be to add a test to ldiskfs_append() to check if it is a special lustre metadata directory, and if so to bypass the dir limit.

Is there an easy way to detect lustre metadata directories?

[James A Simmons]

RHE8 and Ubuntu overlap for the ldiskfs patches so they will need to be updated at the same time. I can update Ubuntu but I don't have a RHEL8 system to fix it up on.

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39773
Subject: LU-13187 ldiskfs: Fix max_dir_size_kb for RHEL7
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 43fe1051ee6cab1c9f8b85863ec91aec2c06b251