Details
-
Bug
-
Resolution: Fixed
-
Minor
-
None
-
None
-
master
-
3
-
9223372036854775807
Description
Finally, we found a commit and root cause of 25% metadata performance regression.
(File creation into single shared directory) This regression introduced on middle of lustre-2.5 and lustre-2.6 and this regression are still exist.
After our investigation with "git bisect" the follwoing patch caseus perforamnce regression.
5f3e926ac9ff8ad134ad920d0e8545e16395ef3b is the first bad commit
commit 5f3e926ac9ff8ad134ad920d0e8545e16395ef3b
Author: wang di <di.wang@intel.com>
Date: Wed Jul 31 00:00:40 2013 -0700
LU-3529 lod: create striped directory
1. Add "lfs setdirstripe -i -c" to create striped
directory.
2. client send create request to the master MDT, which
will allocate FIDs and create slaves. for all of slaves.
3. Client needs to revalidate slaves during intent getattr
and open request.
4. lmv_stripe_md will include attributes(size, nlink etc)
from all of stripe, which will be protected by UPDATE lock.
client needs to merge these attributes when update inode.
5. send create request to the MDT where the file is located,
which can help creating master stripe of striped directory.
Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I7ac560e39dcb415e310dc5e6ade531d76227ffae
Reviewed-on: http://review.whamcloud.com/7196
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Here is test configuration
1 x MDS (2 x E5-2690 v3, 128GB memory)
32 x Client(2 x CPU E5-2650, 128GB memory)
4 x OSS and 40 OST
RHEL6.5
- mpirun -np 128 -ppn 4 -hostfile ./hostfile.32 mdtest -n 5000 -v -d /scratch/mdtest.out -p 30 -i 3 -F
[e19b51372ad94818a7a79b1fbae5b55c665ba59f] LU-4196 build: Reenable OFED-3.5 support on SLES11 SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 88017.734 82493.520 84509.613 2489.825 File stat : 151997.740 142649.444 148656.020 4256.270 File read : 162847.716 154605.697 158734.759 3364.809 File removal : 84993.971 78063.127 80600.677 3118.973 Tree creation : 3692.169 2931.030 3262.108 318.519 Tree removal : 51.245 47.758 49.678 1.446 V-1: Entering timestamp...
[5f3e926ac9ff8ad134ad920d0e8545e16395ef3b] LU-3529 lod: create striped directory SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 66695.074 66242.158 66524.596 201.138 File stat : 152026.405 143681.866 148948.529 3741.740 File read : 165470.364 163291.085 164307.211 895.740 File removal : 86953.641 82117.776 84285.373 2005.726 Tree creation : 4165.148 2841.669 3603.150 558.417 Tree removal : 59.119 52.690 55.581 2.664 V-1: Entering timestamp...
Even no DNE, we are losing 25% performance regression.
Attachments
Activity
Minh Diep (minh.diep@intel.com) uploaded a new patch: https://review.whamcloud.com/29143
Subject: LU-9840 lod: add ldo_dir_stripe_loaded
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 91f613ee25e5a39e28b1c2ad45fff15cc65b79de
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/28962/
Subject: LU-9840 lod: add ldo_dir_stripe_loaded
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 744fe412fa101609a38f7ccc77efc4f1c540e008
Ihara, thanks for testing. hmm, I did not expect master could be better here, maybe there are some other optimization here. Anyway it is good news.
Could you please also try https://jira.hpdd.intel.com/secure/attachment/28000/28000_LU-9840.patch directly based on [5f3e926ac9ff8ad134ad920d0e8545e16395ef3b] LU-3529 lod: create striped directory. ? Thanks.
Thanks WangDi. I did quick test your patches against laster master.
test case: mdtest to a shared directory, 32 clients, 128 processes and 10,000 files per process which means 1.28 Million files for total.
mpirun -np 128 /work/tools/bin/mdtest -n 10000 -v -d /scratch0/mdtest.out -F -i 3 -p 30 -w 0
master: 1fc4ed3ac40ab0e11b1c59d7d147a100636cbda0 without any patches.
SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 87877.948 77161.656 81981.331 4440.689 File stat : 146500.548 139206.464 143471.400 3103.363 File read : 157850.695 155719.694 157052.803 948.731 File removal : 109180.271 103474.828 105885.648 2411.618 Tree creation : 4812.597 1625.810 3218.841 1301.001 Tree removal : 41.205 38.867 39.730 1.048 V-1: Entering timestamp...
SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 94691.817 87748.601 90106.652 3242.642 File stat : 145824.020 139999.453 142233.530 2564.019 File read : 154804.912 152446.911 153784.582 988.457 File removal : 110741.797 105265.586 107834.815 2248.374 Tree creation : 4708.453 2226.195 3100.325 1138.556 Tree removal : 40.618 21.520 33.672 8.622 V-1: Entering timestamp...
Somehow, master branch without patches are a bit better than lustre-2.7 we saw before. gap between 2.5 and current master is small.
Also, patch 28962 helps performance improvment too. we are seeing average ~10% impromvents at file creation.
John L. Hammond (john.hammond@intel.com) merged in patch https://review.whamcloud.com/29143/
Subject:
LU-9840lod: add ldo_dir_stripe_loadedProject: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 507852d2fb49da5da6ed792e25964dd14169b30d