https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10195 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/5de11e12-bf6f-11e7-a943-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5de11e12-bf6f-11e7-a943-52540065bddc</a>.</p> <p>The sub-test test_24v failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Timeout occurred after 71 mins, last suite running was sanity, restarting cluster to continue tests </pre> </div></div> <p>This might be a DCO problem, not a lustre bug. I can't find evidence of a lustre failure.</p> <p>Info required for matching: sanity 24v</p> LU-10195

sanity test_24v: timeout occurred after 71 mins

Bug Major Resolved Duplicate WC Triage Bob Glossman Thu, 2 Nov 2017 15:18:50 +0000 Fri, 3 Aug 2018 13:24:14 +0000 Fri, 3 Aug 2018 13:23:54 +0000 Lustre 2.11.0 Lustre 2.10.2 Lustre 2.12.0 Lustre 2.10.4 0 6 <p>sanity test 24v started failing on the master branch on 2017-10-09 and on the b2_10 branch on 2017-11-17.</p> <p>Although number of files created and create times differ, the last lines in the client test_log for this failure are a variation of the following: </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 24v: list large directory (test hash collision, b=17560) == 12:27:33 (1510921653) striped dir -i0 -c2 /mnt/lustre/d24v.sanity - create 9317 (time 1510921664.06 total 10.03 last 928.56) - create 10000 (time 1510921665.04 total 11.02 last 691.01) - create 19063 (time 1510921675.05 total 21.02 last 906.28) - create 20000 (time 1510921676.03 total 22.01 last 953.02) - create 28885 (time 1510921686.03 total 32.01 last 888.44) - create 30000 (time 1510921687.25 total 33.22 last 916.65) - create 38856 (time 1510921697.25 total 43.22 last 885.56) - create 40000 (time 1510921698.48 total 44.46 last 925.93) - create 48554 (time 1510921708.48 total 54.46 last 855.36) - create 50000 (time 1510921710.31 total 56.29 last 790.30) - create 58537 (time 1510921720.31 total 66.29 last 853.60) - create 60000 (time 1510921721.91 total 67.89 last 913.75) - create 68831 (time 1510921731.91 total 77.89 last 883.04) - create 70000 (time 1510921733.16 total 79.13 last 940.86) - create 78502 (time 1510921743.16 total 89.13 last 850.16) - create 80000 (time 1510921745.04 total 91.02 last 794.76) - create 88521 (time 1510921755.04 total 101.02 last 852.07) - create 90000 (time 1510921756.64 total 102.61 last 928.39) - create 99080 (time 1510921766.64 total 112.61 last 907.98) total: 100000 create in 113.62 seconds: 880.16 ops/second mdc.lustre-MDT0000-mdc-ffff880078d24400.stats=clear mdc.lustre-MDT0001-mdc-ffff880078d24400.stats=clear mdc.lustre-MDT0002-mdc-ffff880078d24400.stats=clear mdc.lustre-MDT0003-mdc-ffff880078d24400.stats=clear readpages: 6 rpc_max: 7 </pre> </div></div> <p>More failure instances on Lustre b2_10 branch in DNE test session:<br/> <a href="https://testing.whamcloud.com/test_sets/fc89f1a4-88b1-11e8-b376-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/fc89f1a4-88b1-11e8-b376-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/1c7e0108-8c90-11e8-a9f7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1c7e0108-8c90-11e8-a9f7-52540065bddc</a></p> <p>+1 on b2_10: <a href="https://testing.whamcloud.com/test_sets/d879223a-94fa-11e8-8ee3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d879223a-94fa-11e8-8ee3-52540065bddc</a></p> <p>I ran this locally and it looks like during cleanup each unlink is triggering statahead and reading the whole directory from <tt>is_first_dirent()</tt>. Here is the stack trace for rm:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>o:~# stack1 rm 17437 rm [&lt;ffffffffc0d16140&gt;] ptlrpc_set_wait+0x4c0/0x920 [ptlrpc] [&lt;ffffffffc0d1661d&gt;] ptlrpc_queue_wait+0x7d/0x220 [ptlrpc] [&lt;ffffffffc0f04496&gt;] mdc_getpage+0x1e6/0x6a0 [mdc] [&lt;ffffffffc0f04a7d&gt;] mdc_read_page_remote+0x12d/0x640 [mdc] [&lt;ffffffffb8b94c01&gt;] do_read_cache_page+0x81/0x190 [&lt;ffffffffb8b94d4c&gt;] read_cache_page+0x1c/0x30 [&lt;ffffffffc0efec0a&gt;] mdc_read_page+0x1aa/0x970 [mdc] [&lt;ffffffffc0c8f9f4&gt;] lmv_read_page+0x1c4/0x550 [lmv] [&lt;ffffffffc156d6de&gt;] ll_get_dir_page+0xce/0x380 [lustre] [&lt;ffffffffc15c0ca4&gt;] start_statahead_thread+0x94/0xea0 [lustre] [&lt;ffffffffc15c5a4b&gt;] ll_statahead+0x10b/0x120 [lustre] [&lt;ffffffffc15af0b4&gt;] ll_lookup_it+0xb54/0xef0 [lustre] [&lt;ffffffffc15b07eb&gt;] ll_lookup_nd+0xbb/0x190 [lustre] [&lt;ffffffffb8c25cf3&gt;] lookup_real+0x23/0x60 [&lt;ffffffffb8c26702&gt;] __lookup_hash+0x42/0x60 [&lt;ffffffffb8c2cfec&gt;] do_unlinkat+0x14c/0x2d0 [&lt;ffffffffb8c2e0cb&gt;] SyS_unlinkat+0x1b/0x40 [&lt;ffffffffb912082f&gt;] system_call_fastpath+0x1c/0x21 [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> <p>Attaching strace to rm shows that it's doing about 1 unlink per second:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>o:~# strace -ttt -p 17437 strace: Process 17437 attached 1533157403.718270 unlinkat(4, "f24v.sanity1160", 0) = 0 1533157404.349918 unlinkat(4, "f24v.sanity1161", 0) = 0 1533157404.962529 unlinkat(4, "f24v.sanity1162", 0) = 0 1533157405.580449 unlinkat(4, "f24v.sanity1163", 0) = 0 1533157406.196802 unlinkat(4, "f24v.sanity1164", 0) = 0 1533157406.812760 unlinkat(4, "f24v.sanity1165", 0) = 0 1533157407.681777 unlinkat(4, "f24v.sanity1166", 0) = 0 1533157408.311224 unlinkat(4, "f24v.sanity1167", 0^Cstrace: Process 17437 detached &lt;detached ...&gt; </pre> </div></div> <p> Every time I get the stack trace of rm it appears as above. On disabling statahead, unlinks go to about 100 per second.</p> <p>Also this test is basically disabled in a non-DNE setup since it uses <tt>$LFS getdirstripe -c $(dirname $fname)</tt> to compute the stripe count of the directory which returns 0 for an unstriped directory.</p> <p>Lai, is this <a href="https://jira.whamcloud.com/browse/LU-10165" title="sanity test_24v: test took quite long time to finish" class="issue-link" data-issue-key="LU-10165"><del>LU-10165</del></a> again?</p> <p>Yes, it looks to be <a href="https://jira.whamcloud.com/browse/LU-10165" title="sanity test_24v: test took quite long time to finish" class="issue-link" data-issue-key="LU-10165"><del>LU-10165</del></a>, which is not landed on b2_10 yet.</p> Duplicate LU-10165 Development Rank 1|hzzn1r: Rank (Obsolete) 9223372036854775807 Severity