[LU-9840] LU-3529 causes 25% metadata performance regressions even without DNE

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.2
• Environment: master
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Shuichi Ihara (Inactive)

Description

Finally, we found a commit and root cause of 25% metadata performance regression.
(File creation into single shared directory) This regression introduced on middle of lustre-2.5 and lustre-2.6 and this regression are still exist.
After our investigation with "git bisect" the follwoing patch caseus perforamnce regression.

5f3e926ac9ff8ad134ad920d0e8545e16395ef3b is the first bad commit
commit 5f3e926ac9ff8ad134ad920d0e8545e16395ef3b
Author: wang di <[e-mail]>
Date:   Wed Jul 31 00:00:40 2013 -0700

    LU-3529 lod: create striped directory
    
    1. Add "lfs setdirstripe -i -c" to create striped
    directory.
    
    2. client send create request to the master MDT, which
    will allocate FIDs and create slaves. for all of slaves.
    
    3. Client needs to revalidate slaves during intent getattr
    and open request.
    
    4. lmv_stripe_md will include attributes(size, nlink etc)
    from all of stripe, which will be protected by UPDATE lock.
    client needs to merge these attributes when update inode.
    
    5. send create request to the MDT where the file is located,
    which can help creating master stripe of striped directory.
    
    Signed-off-by: wang di <[e-mail]>
    Change-Id: I7ac560e39dcb415e310dc5e6ade531d76227ffae
    Reviewed-on: http://review.whamcloud.com/7196
    Tested-by: Jenkins
    Tested-by: Maloo <[e-mail]>
    Reviewed-by: Andreas Dilger <[e-mail]>
    Reviewed-by: John L. Hammond <[e-mail]>

Here is test configuration
1 x MDS (2 x E5-2690 v3, 128GB memory)
32 x Client(2 x CPU E5-2650, 128GB memory)
4 x OSS and 40 OST
RHEL6.5

1. mpirun -np 128 -ppn 4 -hostfile ./hostfile.32 mdtest -n 5000 -v -d /scratch/mdtest.out -p 30 -i 3 -F

[e19b51372ad94818a7a79b1fbae5b55c665ba59f] LU-4196 build: Reenable OFED-3.5 support on SLES11
SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :      88017.734      82493.520      84509.613       2489.825
   File stat         :     151997.740     142649.444     148656.020       4256.270
   File read         :     162847.716     154605.697     158734.759       3364.809
   File removal      :      84993.971      78063.127      80600.677       3118.973
   Tree creation     :       3692.169       2931.030       3262.108        318.519
   Tree removal      :         51.245         47.758         49.678          1.446
V-1: Entering timestamp...

[5f3e926ac9ff8ad134ad920d0e8545e16395ef3b] LU-3529 lod: create striped directory
SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :      66695.074      66242.158      66524.596        201.138
   File stat         :     152026.405     143681.866     148948.529       3741.740
   File read         :     165470.364     163291.085     164307.211        895.740
   File removal      :      86953.641      82117.776      84285.373       2005.726
   Tree creation     :       4165.148       2841.669       3603.150        558.417
   Tree removal      :         59.119         52.690         55.581          2.664
V-1: Entering timestamp...

Even no DNE, we are losing 25% performance regression.

Comments

[Peter Jones]

This patch has landed to master but is more work still to come?

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/28962/
Subject: LU-9840 lod: add ldo_dir_stripe_loaded
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 744fe412fa101609a38f7ccc77efc4f1c540e008

[Di Wang (Inactive)]

Ihara, thanks for testing. hmm, I did not expect master could be better here, maybe there are some other optimization here. Anyway it is good news.

Could you please also try https://jira.hpdd.intel.com/secure/attachment/28000/28000_LU-9840.patch directly based on [5f3e926ac9ff8ad134ad920d0e8545e16395ef3b] LU-3529 lod: create striped directory. ? Thanks.

[Shuichi Ihara (Inactive)]

Thanks WangDi. I did quick test your patches against laster master.

test case: mdtest to a shared directory, 32 clients, 128 processes and 10,000 files per process which means 1.28 Million files for total.
mpirun -np 128 /work/tools/bin/mdtest -n 10000 -v -d /scratch0/mdtest.out -F -i 3 -p 30 -w 0

master: 1fc4ed3ac40ab0e11b1c59d7d147a100636cbda0 without any patches.

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :      87877.948      77161.656      81981.331       4440.689
   File stat         :     146500.548     139206.464     143471.400       3103.363
   File read         :     157850.695     155719.694     157052.803        948.731
   File removal      :     109180.271     103474.828     105885.648       2411.618
   Tree creation     :       4812.597       1625.810       3218.841       1301.001
   Tree removal      :         41.205         38.867         39.730          1.048
V-1: Entering timestamp...

SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :      94691.817      87748.601      90106.652       3242.642
   File stat         :     145824.020     139999.453     142233.530       2564.019
   File read         :     154804.912     152446.911     153784.582        988.457
   File removal      :     110741.797     105265.586     107834.815       2248.374
   Tree creation     :       4708.453       2226.195       3100.325       1138.556
   Tree removal      :         40.618         21.520         33.672          8.622
V-1: Entering timestamp...

Somehow, master branch without patches are a bit better than lustre-2.7 we saw before. gap between 2.5 and current master is small.
Also, patch 28962 helps performance improvment too. we are seeing average ~10% impromvents at file creation.

[Gerrit Updater]

wangdi ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28962
Subject: LU-9840 lod: add lod_dir_nonstripe
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d173f7738f5c8053f5e07a685c428f3e263224c3

[Di Wang (Inactive)]

Sure, Ihara

[Shuichi Ihara (Inactive)]

WangDi,
I wonder if your patch could adapt to latest master branch? it seems codes changes a lot..

[Di Wang (Inactive)]

Any news?

[Di Wang (Inactive)]

Ihara:

Could you please try this patch? Thanks

LU-9840.patch

[Shuichi Ihara (Inactive)]

Could you please try this patch ?

patch improved ~15%, but still another 5% performamnce regressions and patch helped for File creation, but go regressions on unlink.
Here is results.

[e19b51372ad94818a7a79b1fbae5b55c665ba59f] LU-4196 build: Reenable OFED-3.5 support on SLES11
SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :      85568.624      84412.561      84814.381        533.711
   File stat         :     156095.695     145840.993     152336.768       4612.121
   File read         :     161833.820     157226.792     160061.457       2025.264
   File removal      :      89463.742      83852.552      85970.898       2488.413
   Tree creation     :       4185.932       2597.092       3482.057        661.160
   Tree removal      :         58.671         47.865         52.494          4.545
V-1: Entering timestamp...

[5f3e926ac9ff8ad134ad920d0e8545e16395ef3b] LU-3529 lod: create striped directory
SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :      72231.370      71757.985      71941.350        207.453
   File stat         :     156218.994     146766.402     152100.221       3953.490
   File read         :     151082.612     148981.339     149687.568        986.470
   File removal      :      86520.243      85361.115      85847.648        491.161
   Tree creation     :       3731.587       1801.677       3001.790        855.194
   Tree removal      :         59.288         55.627         57.821          1.581
V-1: Entering timestamp...

[5f3e926ac9ff8ad134ad920d0e8545e16395ef3b] LU-3529 lod: create striped directory + patch
SUMMARY: (of 3 iterations)
   Operation                      Max            Min           Mean        Std Dev
   ---------                      ---            ---           ----        -------
   File creation     :      82139.647      80821.907      81312.553        588.222
   File stat         :     157364.652     151950.461     153937.210       2433.799
   File read         :     153314.454     151299.630     152332.901        823.361
   File removal      :      79767.310      77557.914      78420.017        965.027
   Tree creation     :       3905.311       3134.756       3433.015        337.792
   Tree removal      :         53.836         50.449         51.712          1.511
V-1: Entering timestamp...

[Shuichi Ihara (Inactive)]

WangDi, Sure, I know what you poined out. I will test them and feedback you sooner.