https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5489 Lustre <p>Attached file (service164.gz) has complete trace of all threads.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LNet: Service thread pid 8805 was inactive <span class="code-keyword">for</span> 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes:^M Pid: 8805, comm: ll_ost03_040^M ^M Call Trace:^M [&lt;ffffffffa04cf6fe&gt;] cfs_waitq_wait+0xe/0x10 [libcfs]^M [&lt;ffffffffa062c6b3&gt;] lu_object_find_at+0xb3/0x360 [obdclass]^M [&lt;ffffffff81063be0&gt;] ? default_wake_function+0x0/0x20^M [&lt;ffffffffa0e74cb9&gt;] ? ofd_key_init+0x59/0x1a0 [ofd]^M [&lt;ffffffffa062c976&gt;] lu_object_find+0x16/0x20 [obdclass]^M [&lt;ffffffffa0e886c5&gt;] ofd_object_find+0x35/0xf0 [ofd]^M [&lt;ffffffffa062d57e&gt;] ? lu_env_init+0x1e/0x30 [obdclass]^M [&lt;ffffffffa0e98649&gt;] ofd_lvbo_update+0x6d9/0xea8 [ofd]^M [&lt;ffffffffa0e7df77&gt;] ofd_setattr+0x7e7/0xb80 [ofd]^M [&lt;ffffffffa0e4ec1c&gt;] ost_setattr+0x31c/0x990 [ost]^M [&lt;ffffffffa0e52746&gt;] ost_handle+0x21e6/0x48e0 [ost]^M [&lt;ffffffffa04db124&gt;] ? libcfs_id2str+0x74/0xb0 [libcfs]^M [&lt;ffffffffa07c53b8&gt;] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]^M [&lt;ffffffffa04cf5de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs]^M [&lt;ffffffffa04e0d6f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs]^M [&lt;ffffffffa07bc719&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]^M [&lt;ffffffff81063be0&gt;] ? default_wake_function+0x0/0x20^M [&lt;ffffffffa07c674e&gt;] ptlrpc_main+0xace/0x1700 [ptlrpc]^M [&lt;ffffffffa07c5c80&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc]^M [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20^M [&lt;ffffffffa07c5c80&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc]^M [&lt;ffffffffa07c5c80&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc]^M [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20^M </pre> </div></div> source at <a href="https://github.com/jlan/lustre-nas">https://github.com/jlan/lustre-nas</a><br/> running version 2.4.3-5.1nas LU-5489

ll_ost thread stuck at lu_object_find_at

Bug Major Resolved Duplicate Hongchao Zhang Mahmoud Hanafi Thu, 14 Aug 2014 18:03:07 +0000 Thu, 2 Oct 2014 21:05:59 +0000 Wed, 3 Sep 2014 14:34:02 +0000 Lustre 2.4.3 0 4 <p>I guess this is similar in nature to lu4725 only this time in ofd code. This should be a somewhat rare race.</p> <p>ofd_setattr does ofd_object_find and pins and object. Then some other thread destroys and object and then ldlm_res_lvbo_update-&gt;ofd_lvbo_update() does ofd_object_find, finds the now destroyed object and starts to wait till the referenes go away, but they cannot because it's this same thread that's holding the reference.</p> <p>Technically the object should not ever be deleted because we are supposed to hold an ldlm lock on it, but I imagine if the lock was somehow lost (held by a client, so if a client was evicted for example) - this might happen.</p> <p>We need to add some sort of a non-racy check to make sure the object is till alive before going into lvbo update for a fix.</p> <p>Looking in the logs we can see there was a bunch of evictions indeed, sso this indeed is a plausible scenario.</p> <p>Hongchao</p> <p>Could you please look into the feasibility of reworking this code in the manner Oleg suggests?</p> <p>Thanks</p> <p>Peter</p> <p>After some additional digging I found that this is actually a dup of <a href="https://jira.whamcloud.com/browse/LU-4019" title="today&#39;s master stick on shutdown on test == sanity test 132: on lu_object_find_at" class="issue-link" data-issue-key="LU-4019"><del>LU-4019</del></a>, the patch from there should help you too, I verified that it applies to b2_4: <a href="http://review.whamcloud.com/7795" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7795</a></p> Development Rank 1|hzwtrj: Rank (Obsolete) 15312 Severity