https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5244 Lustre <p>This issue was created by maloo for wangdi &lt;di.wang@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/1c06a92c-fa14-11e3-883f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/1c06a92c-fa14-11e3-883f-52540035b04c</a>.</p> <p>The sub-test test_32b failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p> <p>04:48:35:LustreError: 985:0:(osp_sync.c:994:osp_sync_thread()) ASSERTION( count &lt; 10 ) failed: t32fs-OST0000-osc-MDT0000: 5 5 empty<br/> 04:48:35:LustreError: 985:0:(osp_sync.c:994:osp_sync_thread()) LBUG<br/> 04:48:35:Pid: 985, comm: osp-syn-0-0<br/> 04:48:35:<br/> 04:48:35:Call Trace:<br/> 04:48:35: <span class="error">&#91;&lt;ffffffffa0742895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> 04:48:35: <span class="error">&#91;&lt;ffffffffa0742e97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> 04:48:35: <span class="error">&#91;&lt;ffffffffa16ca112&gt;&#93;</span> osp_sync_thread+0x6c2/0x7d0 <span class="error">&#91;osp&#93;</span><br/> 04:48:35: <span class="error">&#91;&lt;ffffffff81061d00&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> 04:48:35: <span class="error">&#91;&lt;ffffffffa16c9a50&gt;&#93;</span> ? osp_sync_thread+0x0/0x7d0 <span class="error">&#91;osp&#93;</span><br/> 04:48:35: <span class="error">&#91;&lt;ffffffff8109ab56&gt;&#93;</span> kthread+0x96/0xa0<br/> 04:48:35: <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> child_rip+0xa/0x20<br/> 04:48:35: <span class="error">&#91;&lt;ffffffff8109aac0&gt;&#93;</span> ? kthread+0x0/0xa0<br/> 04:48:35: <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20<br/> 04:48:35:<br/> 04:48:35:Kernel panic - not syncing: LBUG<br/> 04:48:35:Pid: 985, comm: osp-syn-0-0 Tainted: G W --------------- 2.6.32-431.17.1.el6_lustre.g0eed638.x86_64 #1<br/> 04:48:35:Call Trace:<br/> 04:48:35: <span class="error">&#91;&lt;ffffffff8152795f&gt;&#93;</span> ? panic+0xa7/0x16f<br/> 04:48:35: <span class="error">&#91;&lt;ffffffffa0742eeb&gt;&#93;</span> ? lbug_with_loc+0x9b/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> 04:48:35: <span class="error">&#91;&lt;ffffffffa16ca112&gt;&#93;</span> ? osp_sync_thread+0x6c2/0x7d0 <span class="error">&#91;osp&#93;</span><br/> 04:48:35: <span class="error">&#91;&lt;ffffffff81061d00&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> 04:48:35: <span class="error">&#91;&lt;ffffffffa16c9a50&gt;&#93;</span> ? osp_sync_thread+0x0/0x7d0 <span class="error">&#91;osp&#93;</span><br/> 04:48:35: <span class="error">&#91;&lt;ffffffff8109ab56&gt;&#93;</span> ? kthread+0x96/0xa0<br/> 04:48:35: <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> ? child_rip+0xa/0x20<br/> 04:48:35: <span class="error">&#91;&lt;ffffffff8109aac0&gt;&#93;</span> ? kthread+0x0/0xa0<br/> 04:48:35: <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20<br/> 04:48:35:Initializing cgroup subsys cpuset</p></blockquote> <p>Info required for matching: conf-sanity 32b</p> LU-5244

conf-sanity test_32b: osp_sync_thread()) ASSERTION( count < 10 )

Bug Blocker Closed Duplicate Nathaniel Clark Maloo Mon, 23 Jun 2014 16:36:44 +0000 Mon, 30 Jun 2014 17:37:36 +0000 Mon, 30 Jun 2014 17:37:36 +0000 Lustre 2.6.0 Lustre 2.6.0 0 8 <p>Nathaniel,<br/> Could you have a look at this one?<br/> Thank you!</p> <p>This is a bad LASSERT(). I can't see any reason why "10" is a magic number before which the remote RPCs need to be completed? If we hit this on a test system, we will definitely hit this on some customer system when the MDS is busy, or the network is overloaded.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* wait till all the requests are completed */</span> count = 0; <span class="code-keyword">while</span> (d-&gt;opd_syn_rpc_in_progress &gt; 0) { osp_sync_process_committed(&amp;env, d); lwi = LWI_TIMEOUT(cfs_time_seconds(5), NULL, NULL); rc = l_wait_event(d-&gt;opd_syn_waitq, d-&gt;opd_syn_rpc_in_progress == 0, &amp;lwi); <span class="code-keyword">if</span> (rc == -ETIMEDOUT) count++; LASSERTF(count &lt; 10, <span class="code-quote">"%s: %d %d %sempty\n"</span>, d-&gt;opd_obd-&gt;obd_name, d-&gt;opd_syn_rpc_in_progress, d-&gt;opd_syn_rpc_in_flight, list_empty(&amp;d-&gt;opd_syn_committed_there) ? <span class="code-quote">"" : "</span>!"); } </pre> </div></div> <p>There needs to be proper error handling here, either just to continue looping, or to break out and return an error.</p> <p>This was landed as commit 08f093ce2c799faf7a580f53850ecb13d2b71603:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> LU-2701 osp: wake up sync thread osp_sync_process_committed() to wake up sync thread when it is requested to stop (e.g. umount) and there is no pending work left. the patch adds a sanity check to ensure this process is not taking too long. </pre> </div></div> <p>"sanity check" != LASSERT()...</p> <p>I've bumped this to be a blocker, since it is causing very regular test failures in review-dne-part-1.</p> <p>the idea was that at umount we invalidate the import and this should cause RPCs in-flight to abort quickly. I'm not very familiar with lnet internals and not sure the abort is very promptly in all the cases. I think it makes sense to see what's going on and why the RPCs weren't aborted in time.</p> <p><a href="http://review.whamcloud.com/10805" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10805</a></p> <p>The patch avoids the crash, but so far there isn't any explanation about why this started failing so seriously.</p> <p>Unfortunately, both <a href="https://jira.whamcloud.com/browse/LU-5244" title="conf-sanity test_32b: osp_sync_thread()) ASSERTION( count &lt; 10 ) " class="issue-link" data-issue-key="LU-5244"><del>LU-5244</del></a> and <a href="https://jira.whamcloud.com/browse/LU-5249" title="conf-sanity test_32a: NULL pointer in fld_local_lookup" class="issue-link" data-issue-key="LU-5249"><del>LU-5249</del></a> are causing so many test failures that it may not be possible for them to land independently, so re triggering them may not be enough. Instead, basing one patch on the other would allow the second to pass, then it could be landed, then the first one rebased and landed. </p> <p>Also, reverting the patch that is the root of these problems may fix both issues at once. </p> <p>Duplicate of <a href="https://jira.whamcloud.com/browse/LU-5188" title="nbp6-OST002f-osc-MDT0000: invalid setattr record, lsr_valid:0" class="issue-link" data-issue-key="LU-5188"><del>LU-5188</del></a></p> Related LU-5249 LU-5188 Development Rank 1|hzwppj: Rank (Obsolete) 14626 Severity