https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5106 Lustre <p>This issue was created by maloo for Nathaniel Clark &lt;nathaniel.l.clark@intel.com&gt;</p> <p>This issue relates to the following test suite run:<br/> <a href="http://maloo.whamcloud.com/test_sets/a4d76d66-baba-11e3-a27d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/a4d76d66-baba-11e3-a27d-52540035b04c</a><br/> <a href="http://maloo.whamcloud.com/test_sets/e3f486ac-e317-11e3-93d9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/e3f486ac-e317-11e3-93d9-52540035b04c</a></p> <p>The sub-test test_123a failed with the following error:</p> <blockquote> <p>ls 10000 files is slower with statahead!</p></blockquote> <p>Info required for matching: sanity 123a</p> LU-5106

Test failure sanity test_123a: ls 10000 files is slower with statahead!

Bug Major Resolved Fixed Lai Siyao Maloo Tue, 27 May 2014 14:50:38 +0000 Sat, 13 Feb 2021 09:55:37 +0000 Wed, 19 Jul 2017 03:41:58 +0000 Lustre 2.11.0 0 12 <p>This is most likely related to the VM being slow or similar.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/61/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/61/</a><br/> Distro/Arch: RHEL6.5/x86_64 + SLES11SP3/x86_64 (Server + Client)</p> <p>The same failure occurred: <a href="https://maloo.whamcloud.com/test_sets/82588a38-ef83-11e3-b8c2-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/82588a38-ef83-11e3-b8c2-52540035b04c</a></p> <p>another seen with el6.7 client/server on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/69e7442c-6321-11e5-b25a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/69e7442c-6321-11e5-b25a-5254006e85c2</a></p> <p>More failures on master:<br/> 2015-11-14 06:00:54 - <a href="https://testing.hpdd.intel.com/test_sets/1d980b04-8add-11e5-86aa-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1d980b04-8add-11e5-86aa-5254006e85c2</a><br/> 2015-11-18 02:21:57 - <a href="https://testing.hpdd.intel.com/test_sets/588d486c-8de1-11e5-a4b1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/588d486c-8de1-11e5-a4b1-5254006e85c2</a><br/> 2015-12-02 06:56:45 - <a href="https://testing.hpdd.intel.com/test_sets/ae2b6ed8-9913-11e5-aeec-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ae2b6ed8-9913-11e5-aeec-5254006e85c2</a><br/> 2015-12-07 14:40:39 - <a href="https://testing.hpdd.intel.com/test_sets/17b4eaa2-9d40-11e5-ade6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/17b4eaa2-9d40-11e5-ade6-5254006e85c2</a><br/> 2015-12-09 05:53:36 - <a href="https://testing.hpdd.intel.com/test_sets/cd3cb85e-9e86-11e5-87a9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cd3cb85e-9e86-11e5-87a9-5254006e85c2</a><br/> 2015-12-09 18:40:23 - <a href="https://testing.hpdd.intel.com/test_sets/24479cda-9ef6-11e5-ba94-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/24479cda-9ef6-11e5-ba94-5254006e85c2</a><br/> 2016-01-04 03:38:41 - <a href="https://testing.hpdd.intel.com/test_sets/21928502-b2df-11e5-aa1f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/21928502-b2df-11e5-aa1f-5254006e85c2</a><br/> 2016-01-14 21:54:15 - <a href="https://testing.hpdd.intel.com/test_sets/2efcd76e-bb50-11e5-acbb-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2efcd76e-bb50-11e5-acbb-5254006e85c2</a><br/> 2016-01-20 09:06:34 - <a href="https://testing.hpdd.intel.com/test_sets/642dee9c-bf9b-11e5-a659-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/642dee9c-bf9b-11e5-a659-5254006e85c2</a><br/> 2016-01-24 07:57:29 - <a href="https://testing.hpdd.intel.com/test_sets/e1d05f34-c2b5-11e5-8d4d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e1d05f34-c2b5-11e5-8d4d-5254006e85c2</a><br/> 2016-01-26 09:25:22 - <a href="https://testing.hpdd.intel.com/test_sets/2a3e8606-c455-11e5-8866-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2a3e8606-c455-11e5-8866-5254006e85c2</a><br/> 2016-02-02 08:13:41 - <a href="https://testing.hpdd.intel.com/test_sets/264d4cb4-c9d0-11e5-b71a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/264d4cb4-c9d0-11e5-b71a-5254006e85c2</a><br/> 2016-02-15 14:02:16 - <a href="https://testing.hpdd.intel.com/test_sets/5798cb4c-d437-11e5-aabf-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5798cb4c-d437-11e5-aabf-5254006e85c2</a><br/> 2016-02-22 16:05:58 - <a href="https://testing.hpdd.intel.com/test_sets/b40373ea-d9c9-11e5-8b17-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b40373ea-d9c9-11e5-8b17-5254006e85c2</a></p> <p>Another failure on Master:</p> <p><a href="https://testing.hpdd.intel.com/test_sets/287c20f2-e48b-11e5-bbef-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/287c20f2-e48b-11e5-bbef-5254006e85c2</a></p> <p>I reproduced this in local system, and found the root cause is that on DNE system, it takes long time to prepare a page for readdir, because dirents are distributed on several stripes, but the directory page should be in hash order, so it will iterate each stripes for a single dirent until it fills a directory page, it looks like statahead is often slower than 'ls' in building directory page, so 'lookup' from 'ls' couldn't find cached statahead entries, thus statahead failed and quit later. In the end 'ls' without statahead is often faster.</p> <p>I'll see how this can be improved.</p> <p>Lai, the generation of DNE2 readdir pages is something that I discussed with Di in the past. Essentially, the llite-level readdir is a merge sort of the individual readdir pages from the various MDTs, which are already in hash order. Currently, it appears that this is implemented in a sub-optimal manner - essentially an O(n^2) sort currently.</p> <p>One option would be to implement a secondary readdir cache at the llite level that maintains the entries in sorted order, possibly in a linked list of dentries. This would allow fixing the readdir vs. unlink problem in <a href="https://jira.whamcloud.com/browse/LU-3308" title="large readdir chunk size slows unlink/&quot;rm -r&quot; performance" class="issue-link" data-issue-key="LU-3308">LU-3308</a>, and would potentially allow unlinks to drop dentries from the readdir list at lock cancellation, rather than having to drop all whenever the directory lock is lost.</p> <p>Lai Siyao (lai.siyao@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/27663" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27663</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5106" title="Test failure sanity test_123a: ls 10000 files is slower with statahead!" class="issue-link" data-issue-key="LU-5106"><del>LU-5106</del></a> readdir: improve striped readdir<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 596f4129e8d9763febe297f6c82cd95efb405fbd</p> <p>Lai Siyao (lai.siyao@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/27683" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27683</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5106" title="Test failure sanity test_123a: ls 10000 files is slower with statahead!" class="issue-link" data-issue-key="LU-5106"><del>LU-5106</del></a> statahead: support striped directory<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 852c199f31da0ae9a64ef0ee63398c973373cca4</p> <p>Andreas, this may not be so complicated, in <a href="https://review.whamcloud.com/27663" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27663</a> I introduced a struct lmv_dir_ctxt which saves directory page and current dirent for all stripes, so to get the next dirent for this directory, it only needs to compare current dirent of all stripes and find the one with the smallest hash. This is O<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/thumbs_down.png" height="16" width="16" align="absmiddle" alt="" border="0"/> IMO.</p> <p>BTW, to pass sanity.sh 123a, both patches are needed.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/27683/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27683/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5106" title="Test failure sanity test_123a: ls 10000 files is slower with statahead!" class="issue-link" data-issue-key="LU-5106"><del>LU-5106</del></a> statahead: support striped directory<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9e7952c045a3ce2041a2fa325cc4a147be6549bb</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/27663/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27663/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5106" title="Test failure sanity test_123a: ls 10000 files is slower with statahead!" class="issue-link" data-issue-key="LU-5106"><del>LU-5106</del></a> readdir: improve striped readdir<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 98fc9a77446a1539bca18215ad57f21712218ecc</p> <p>Landed for 2.11</p> <p>Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38826" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38826</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5106" title="Test failure sanity test_123a: ls 10000 files is slower with statahead!" class="issue-link" data-issue-key="LU-5106"><del>LU-5106</del></a> readdir: improve striped readdir<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 23935cdc099876e5945979b11d9ef80863a5cb2f</p> Related LU-8643 Development Rank 1|hzwn3z: Rank (Obsolete) 14086 Severity