[LU-10465] increase default stripe size to 4MB

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.16.0
• Affects Version/s: Lustre 2.11.0
• Labels: LTS12
• Severity: 3
• Assignee: Mikhail Pershin
• Reporter: Jian Yu

Description

Patch https://review.whamcloud.com/25336 for LU-9090 has increased the default OST BRW size to 4MB. Patch for this ticket will increase the default stripe size from 1MB to 4MB so that widely-striped files can generate full RPCs without pinning so much memory on the client.

Issue Links

• is related to: LU-17747 interop: sanity test_130g: filefrag printed 175 < 700 extents
• is related to: LU-10786 sanity-flr test_45: Create /mnt/lustre/d45.sanity-flr/f45.sanity-flr failed
• is related to: LU-17076 ptlrpc_nrs_req_stop_nolock() use after free with ORR NRS policy
• is related to: LU-10808 DoM: component end should align with dom_stripesize
• is related to: LU-11918 Allow setting default file layout on root directory at mkfs time
• is related to: LU-10463 Poor write performance periodically on repeated test runs
• is related to: LU-9090 increase default RPC size to 4MB

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/37318/
Subject: LU-10465 lov: increase default stripe size to 4MB
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: ea18d7da59d369f093e340e150544f51b2f229a1

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39957
Subject: LU-10465 lod: adjust component stripe size
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: bf76de5430fa104b2992f40768cdf8c97699f1de

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/37661/
Subject: LU-10465 lod: adjust component stripe size
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e67a7f7b960a042ae7369e81e4365c6f7e095d25

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37661
Subject: LU-10465 lod: adjust component stripe size
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6b589349595efb4eba73b40c7e7e8f07c41bd8b3

[Mikhail Pershin]

Probably I see where problem is, test does the following:

$LFS migrate -E 2M -c1 -E -1 -c2 $dom

without implicitly set stripe size, LOD uses default size 4MB and fails because it is over whole component 2MB. Considering the user may not know default MDT stripe size, I'd say it is not his fault to use 2MB component, and either lfs or LOD should take care and reduce stripe size to the component size maybe?

[Mikhail Pershin]

Test failed due to that error:

(lod_lov.c:1934:lod_verify_striping()) stripe size isn't aligned, stripe_sz: 4194304, [0, 2097152)

So this check in lod_verify_striping() is not quite correct, isn't it? If original file has 1MB for first component and 3MB, 4MB ... for the second, then it can be migrated to 2MB for the first and then 2MB, 4MB, ... for the second component.
 

[Andreas Dilger]

The second chunk (start of second component, after DoM component) should just be a bit shorter, starting at 1MB and ending at 4MB, with a "hole" at the start where the DoM opponent is. Then the rest of the chunks in the second component would be properly sized/aligned at 4MB.

It is done this way so that eg. the DoM data could be written into the second component without having to move all the later data.

[Mikhail Pershin]

Andreas, this happens because new component stripes are not aligned with an old ones. We have original file with 1MB DOM component and not defined second component, e.g. it become 4MB stripes by default. Test tries to migrate to PFL layout with 2MB as first component stripe and second component by default. With 1MB default it works, but with 4MB boundaries are not aligned. So test should be changed to migrate to the same first stripe size it seems.

Meanwhile I wonder, is that OK that components are not aligned by stripe size in a file? E.g. for original file we have [0, 1MB) for DOM and then 4MB stripes, so whole file has stripes as 1MB, 4MB, 4, ... and each 4MB stripe is not aligned at 4MB from the file beginning. I am not sure is that a problem or not, but wouldn't be better to adjust new component stripe sizes to be aligned with that component start?

[Andreas Dilger]

Mike, it looks like there are still failures in the DoM tests when the default stripe size is changed to 4MB. Could you please take a look.
https://testing.whamcloud.com/test_sets/c361e79a-3e3d-11ea-bc97-52540065bddc

== sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ===================== 23:32:52 (1579822372)
lfs migrate: cannot create composite file '/mnt/lustre/d272c.sanity/.:VOLATILE:0000:5EB80E61': Invalid argument
error: lfs migrate: /mnt/lustre/d272c.sanity/f272c.sanity: cannot create volatile file: Operation not permitted
 sanity test_272c: @@@@@@ FAIL: failed to migrate to the new composite layout 

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37318
Subject: Revert "LU-10465 lov: decrease default stripe size to 1MB"
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 973a7e7042eda757080884ead71468d575516402

[Andreas Dilger]

I think with LU-10808 fixed that we could try reintroducing this change (essentially revert https://review.whamcloud.com/31589 again).

Ihara, if you get a chance, could you please run a 4MB default stripe size for IOR FPP and SSF to see the performance impact.

Vitaly, it would also be good to know if this change improves or hurts performance on your systems before it becomes the default.