https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10400 Lustre <p>We have been noticing decreased performance in any stat-intense operation on lustre 2.10.<span class="error">&#91;0,2&#93;</span> when compared to 2.7. The difference is more significant when testing on HDDs than when testing on SSDs, but is visible for us on both. Between runs I am dropping cache on the client, mds, and oss via "echo 3 &gt; /proc/sys/vm/drop_caches"</p> <p>For example, in a single directory containing 100000 files:</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <th class='confluenceTh'> Client Version</th> <th class='confluenceTh'>2.10.2</th> <th class='confluenceTh'>2.10.2</th> <th class='confluenceTh'>2.7</th> <th class='confluenceTh'>2.7</th> </tr> <tr> <th class='confluenceTh'>Server Version</th> <th class='confluenceTh'>2.10.2 </th> <th class='confluenceTh'>2.7</th> <th class='confluenceTh'>2.10.2</th> <th class='confluenceTh'>2.7</th> </tr> <tr> <td class='confluenceTd'> ls -l time (seconds)</td> <td class='confluenceTd'>54</td> <td class='confluenceTd'>5</td> <td class='confluenceTd'>53</td> <td class='confluenceTd'>6</td> </tr> <tr> <td class='confluenceTd'>du -s time (seconds)</td> <td class='confluenceTd'>150</td> <td class='confluenceTd'>22</td> <td class='confluenceTd'>150</td> <td class='confluenceTd'>29</td> </tr> </tbody></table> </div> <p>We are running 2.10 server on centos7 and 2.7 on rhel6.6.</p> LU-10400

Reduced stat performance with lustre 2.10

Bug Minor Open Unresolved Saurabh Tandan Tim McMullan Fri, 15 Dec 2017 18:51:46 +0000 Thu, 22 Mar 2018 19:36:01 +0000 0 12 <p>Saraubh</p> <p>Can you please see whether you can reproduce these results?</p> <p>Thanks</p> <p>Peter</p> <p>Hi Tim, are there any tunable or formatting options that are used, or default file striping that is used at your site? We’d like to reproduce this locally to debug the problem, but want to make sure that what we are testing matches what you have. </p> <p><tt>Hey Andreas, </tt><br/> The physical setup is 1mds/mgs, 1 oss with 4 osts (2 SSD. 2 HDD). The stripe size is 1MB, and we tested with stripe 2 so everything hits both osts of the same type (SSD and HDD are in separate pools).  The files we used were all 2MB.  All the testing above was done on the HDDs.<br/>  <br/> I'm setting the following on the MGS, but otherwise the setup is default for both 2.7 and 2.10<br/> <tt>lctl set_param </tt><tt>-</tt><tt>P llite.</tt><tt>*</tt><tt>.lazystatfs</tt><tt>=</tt><tt>1</tt><br/> <tt>lctl set_param </tt><tt>-</tt><tt>P osc.</tt><tt>*</tt><tt>.max_rpcs_in_flight</tt><tt>=</tt><tt>32</tt><br/> <tt>lctl set_param </tt><tt>-</tt><tt>P osc.</tt><tt>*</tt><tt>.max_dirty_mb</tt><tt>=</tt><tt>256</tt><br/>  <br/> <tt>I'm formatting with the following:</tt><br/> MDS - the mgs and mdt are on the same host sharing a drive, the underlying device is a RAID1 of 10k RPM disks.<br/> <tt>mkfs.lustre --fsname=${name} --mgs /dev/sdc1</tt><br/> <tt>mkfs.lustre --fsname=${name} --mdt --mgsnode=${mgs_ip}@o2ib --index=0 /dev/sdc2</tt><br/>  <br/> <tt>OSS -</tt><br/> <tt>ssd, no raid</tt><br/> <tt>mkfs.lustre –fsname=${name} -<del>ost </del>-mgsnode=${mgs_ip}@o2ib --index=0 /dev/sdb</tt><br/> <tt>mkfs.lustre –fsname=${name} -<del>ost </del>-mgsnode=${mgs_ip}@o2ib --index=1 /dev/sdc</tt><br/>  <br/>  <br/> <tt>10k RPM disks, no raid:</tt><br/> <tt>mkfs.lustre –fsname=${name} -<del>ost </del>-mgsnode=${mgs_ip}@o2ib --index=2 /dev/sdd</tt><br/> <tt>mkfs.lustre –fsname=${name} -<del>ost </del>-mgsnode=${mgs_ip}@o2ib --index=3 /dev/sde</tt><br/>  <br/> <tt>Thank you!</tt><br/> <tt>--Tim</tt><br/>  </p> <p>Hi Tim,<br/> Can you please clarify if you are using el7 or el7.4 with 2.10.x ? <br/> Also can you please clarify if you are using two separate systems with 2.7 and 2.10.x or upgrading from 2.7 to 2.10.x ? <br/> Statement - "We are running 2.10 server on centos7 and 2.7 on rhel6.6." is creating a bit of confusion. <br/> Can you please clarify a bit on your lustre version setup please.</p> <p>The lustre 2.10.x system is running: CentOS Linux release 7.4.1708 (Core)<br/> The lustre 2.7 system is running: RedHatEnterpriseServer 6.6</p> <p>Both filesystems are new builds in a lab with no preexisting data.</p> <p>Tried to verify the performance drop between Lustre version 2.7.19.6 and 2.10.0 using the same kernel but I was not able to identify any huge delta between their performance numbers for file creation of 100000 files and later stat using 'time ls -l'. The numbers below are average of 3 runs for each. We will still continue to investigate further into this issue and see if we may identify anything.<br/> File creation using Touch</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Build Version Real user sys b_ieel3_0 build 159 2.7.19.6 85.389 0.33 14.625 kernel-3.10.0-514.el7 b2_10 build 5 2.10.0 99.75 0.325 18.363 kernel-3.10.0-514.el7 </pre> </div></div> <p>time ls -l for touch</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Build Version Real user sys b_ieel3_0 build 159 2.7.19.6 4.444 0.835 2.098 kernel-3.10.0-514.el7 b2_10 build 5 2.10.0 3.848 0.824 2.338 kernel-3.10.0-514.el7 </pre> </div></div> <p>File creation using Mcreate:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Build Version Real usr sys b_ieel3_0 build 159 2.7.19.6 183.02 38.133 137.687 kernel-3.10.0-514.el7 b2_10 build 5 2.10.0 196.111 38.003 152.28 kernel-3.10.0-514.el7 </pre> </div></div> <p>time ls -l for Mcreate:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Build Version Real usr sys b_ieel3_0 build 159 2.7.19.6 3.266 0.76 1.464 kernel-3.10.0-514.el7 b2_10 build 5 2.10.0 3.27 0.738 1.782 kernel-3.10.0-514.el7 </pre> </div></div> <p>Thanks for checking it out!  After your test I decided to try running a test with the same lustre version on the el6 and 7 kernels.  I ran this with lustre 2.8 on rhel6 and rhel7 since it happens to be easy with the released packages.  The results are below, but times appear to be significantly different between the two.  </p> <p>time ls -l </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Kernel real user sys 2.6.32-573.12.1.el6_lustre.x86_64 2.848 0.824 1.808 3.10.0-693.11.6.el7_lustre.x86_64 4.322 0.832 2.188</pre> </div></div> <p>time du -s</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Kernel real user sys 2.6.32-573.12.1.el6_lustre.x86_64 20.450 0.188 5.280 3.10.0-693.11.6.el7_lustre.x86_64 34.830 0.192 5.448</pre> </div></div> <p>I'll keep looking and see what more I can come up with.  Thanks!</p> <p> </p> <p> </p> <p>Tim,</p> <p>That version of CentOS 7 includes the KPTI/Meltdown fix, and that version of CentOS 6 does not.  That's a huge difference, and should account for the differences you're seeing, unless you've specifically disabled KPTI.</p> <p>I'm sorry Patrick, my mistake.  I grabbed some output for the wrong host...  </p> <p>This is the run from 3.10.0-327.3.1.el7_lustre.x86_64 (packaged one for 2.8)</p> <p>time ls -l</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Kernel real user sys 2.6.32-573.12.1.el6_lustre.x86_64 2.848 0.824 1.808 3.10.0-327.3.1.el7_lustre.x86_64 3.391 0.820 1.876</pre> </div></div> <p>time du -s</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Kernel real user sys 2.6.32-573.12.1.el6_lustre.x86_64 20.450 0.188 5.280 3.10.0-327.3.1.el7_lustre.x86_64 32.417 0.252 5.272</pre> </div></div> <p> </p> Development Rank 1|hzzpkv: Rank (Obsolete) 9223372036854775807 Severity