https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5913 Lustre <p>?DUP of <a href="https://jira.whamcloud.com/browse/LU-10" title="Client stuck in ptlrpc_invalidate_import() after eviction" class="issue-link" data-issue-key="LU-10"><del>LU-10</del></a>?<br/> 1. Client get a odb_ping failed<br/> 2. client is evicted<br/> 3. client can't reconnect and stuck in ptlrpc_invalidate_import</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Nov 12 04:32:30 pfe22 kernel: [69643.816473] LustreError: 11-0: nbp9-OST0075-osc-ffff880a28404400: Communicating with 10.151.26.11@o2ib, operation obd_ping failed with -107. Nov 12 04:32:30 pfe22 kernel: [69643.854158] Lustre: nbp9-OST0075-osc-ffff880a28404400: Connection to nbp9-OST0075 (at 10.151.26.11@o2ib) was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete Nov 12 04:32:30 pfe22 kernel: [69643.922758] LustreError: 167-0: nbp9-OST0075-osc-ffff880a28404400: This client was evicted by nbp9-OST0075; in progress operations using <span class="code-keyword">this</span> service will fail. Nov 12 04:34:11 pfe22 kernel: [69743.757653] LustreError: 92481:0:(<span class="code-keyword">import</span>.c:324:ptlrpc_invalidate_import()) nbp9-OST0075_UUID: rc = -110 waiting <span class="code-keyword">for</span> callback (1 != 0) Nov 12 04:34:11 pfe22 kernel: [69743.793518] LustreError: 92481:0:(<span class="code-keyword">import</span>.c:350:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8802204cec00 x1484496199485132/t0(0) o4-&gt;nbp9-OST0075-osc-ffff880a28404400@10.151.26.11@o2ib:6/4 lens 488/448 e 0 to 0 dl 1415726912 ref 2 fl Rpc:RE/0/ffffffff rc -5/-1 Nov 12 04:34:11 pfe22 kernel: [69743.866993] LustreError: 92481:0:(<span class="code-keyword">import</span>.c:366:ptlrpc_invalidate_import()) nbp9-OST0075_UUID: RPCs in <span class="code-quote">"Unregistering"</span> phase found (0). Network is sluggish? Waiting them to error out. </pre> </div></div> <p>We find ldlm_bl_ threads stuck in D state.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Stack traceback <span class="code-keyword">for</span> pid 8080 0xffff8802569b2540 8080 2 0 1 D 0xffff8802569b2bb0 ldlm_bl_04 [&lt;ffffffff8146fb6b&gt;] thread_return+0x0/0x295 [&lt;ffffffff81470c58&gt;] __mutex_lock_slowpath+0xf8/0x150 [&lt;ffffffff814706ea&gt;] mutex_lock+0x1a/0x40 [&lt;ffffffffa08bd28e&gt;] cl_lock_mutex_get+0x6e/0xc0 [obdclass] [&lt;ffffffffa0b8988e&gt;] osc_dlm_blocking_ast0+0x5e/0x210 [osc] [&lt;ffffffffa0b89a8c&gt;] osc_ldlm_blocking_ast+0x4c/0x100 [osc] [&lt;ffffffffa09e30a0&gt;] ldlm_handle_bl_callback+0xc0/0x420 [ptlrpc] [&lt;ffffffffa09e3609&gt;] ldlm_bl_thread_main+0x209/0x430 [ptlrpc] [&lt;ffffffff8147ade4&gt;] kernel_thread_helper+0x4/0x10 [0]kdb&gt; btp 8090 Stack traceback <span class="code-keyword">for</span> pid 8090 0xffff88024c9a6640 8090 2 0 6 D 0xffff88024c9a6cb0 ldlm_bl_12 [&lt;ffffffff8146fb6b&gt;] thread_return+0x0/0x295 [&lt;ffffffff81470c58&gt;] __mutex_lock_slowpath+0xf8/0x150 [&lt;ffffffff814706ea&gt;] mutex_lock+0x1a/0x40 [&lt;ffffffffa08bd28e&gt;] cl_lock_mutex_get+0x6e/0xc0 [obdclass] [&lt;ffffffffa0b8988e&gt;] osc_dlm_blocking_ast0+0x5e/0x210 [osc] [&lt;ffffffffa0b89a8c&gt;] osc_ldlm_blocking_ast+0x4c/0x100 [osc] [&lt;ffffffffa09e30a0&gt;] ldlm_handle_bl_callback+0xc0/0x420 [ptlrpc] [&lt;ffffffffa09e3609&gt;] ldlm_bl_thread_main+0x209/0x430 [ptlrpc] [&lt;ffffffff8147ade4&gt;] kernel_thread_helper+0x4/0x10 [0]kdb&gt; go </pre> </div></div> client and server version lustre2.4.3-7nas LU-5913

client stuck in ptlrpc_invalidate_import

Bug Major Resolved Duplicate Zhenyu Xu Mahmoud Hanafi Wed, 12 Nov 2014 19:30:00 +0000 Fri, 16 Oct 2015 04:27:42 +0000 Fri, 16 Oct 2015 04:27:42 +0000 Lustre 2.4.3 0 6 <p>Hello Mahmoud,<br/> Have you had the chance to at least collect the full stacks trace and the Lustre debug log ?? Or may be a Client's full crash-dump has been taken ??</p> <p>Bobijam</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>I have uploaded lustre debug logs and stacktrace for all processes to the ftp site.</p> <p>ftp.whamcloud.com/uploads/<a href="https://jira.whamcloud.com/browse/LU-5913" title="client stuck in ptlrpc_invalidate_import" class="issue-link" data-issue-key="LU-5913"><del>LU-5913</del></a>/r447i2n10.stacktrace.gz<br/> ftp.whamcloud.com/uploads/<a href="https://jira.whamcloud.com/browse/LU-5913" title="client stuck in ptlrpc_invalidate_import" class="issue-link" data-issue-key="LU-5913"><del>LU-5913</del></a>/r447i2n10.lustredebug.gz</p> <p>I saw the statehead thread below was trying to enqueue a lock, then the enqueue process was stuck at writing back dirty pages. This is bad as it's doing this by holding parent lock, which may cause deadlock.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[1415909369.169600] ll_sa_96389 S ffff880734597460 0 96438 2 0x00000000 [1415909369.169600] ffff880734597390 0000000000000046 ffff880734596010 0000000000011800 [1415909369.169600] 0000000000011800 0000000000011800 0000000000011800 ffff880734597fd8 [1415909369.169600] ffff880734597fd8 0000000000011800 ffff880688818500 ffffffff81a11020 [1415909369.169600] Call Trace: [1415909369.169600] [&lt;ffffffffa09a4b7b&gt;] osc_extent_wait+0x5eb/0x680 [osc] [1415909369.169600] [&lt;ffffffffa09a5146&gt;] osc_cache_wait_range+0x536/0x820 [osc] [1415909369.169600] [&lt;ffffffffa09a5f55&gt;] osc_cache_writeback_range+0xb25/0x1140 [osc] [1415909369.169600] [&lt;ffffffffa099025d&gt;] osc_lock_flush+0x7d/0x260 [osc] [1415909369.169600] [&lt;ffffffffa0990539&gt;] osc_lock_cancel+0xf9/0x1e0 [osc] [1415909369.169600] [&lt;ffffffffa06c3225&gt;] cl_lock_cancel0+0x65/0x150 [obdclass] [1415909369.169600] [&lt;ffffffffa06c3f7b&gt;] cl_lock_cancel+0x14b/0x150 [obdclass] [1415909369.169600] [&lt;ffffffffa099107d&gt;] osc_lock_blocking+0x5d/0xf0 [osc] [1415909369.169600] [&lt;ffffffffa0991929&gt;] osc_dlm_blocking_ast0+0xf9/0x210 [osc] [1415909369.169600] [&lt;ffffffffa0991a8c&gt;] osc_ldlm_blocking_ast+0x4c/0x100 [osc] [1415909369.169600] [&lt;ffffffffa07d4e0f&gt;] ldlm_cancel_callback+0x5f/0x180 [ptlrpc] [1415909369.169600] [&lt;ffffffffa07e320f&gt;] ldlm_cli_cancel_local+0x7f/0x480 [ptlrpc] [1415909369.169600] [&lt;ffffffffa07e65a2&gt;] ldlm_cli_cancel_list_local+0xf2/0x290 [ptlrpc] [1415909369.169600] [&lt;ffffffffa07e839f&gt;] ldlm_prep_elc_req+0x3df/0x490 [ptlrpc] [1415909369.169600] [&lt;ffffffffa07e846f&gt;] ldlm_prep_enqueue_req+0x1f/0x30 [ptlrpc] [1415909369.169600] [&lt;ffffffffa0977280&gt;] osc_enqueue_base+0x150/0x660 [osc] [1415909369.169600] [&lt;ffffffffa09914a0&gt;] osc_lock_enqueue+0x310/0x4c0 [osc] [1415909369.169600] [&lt;ffffffffa06c4722&gt;] cl_enqueue_kick+0x62/0x150 [obdclass] [1415909369.169600] [&lt;ffffffffa06c7f31&gt;] cl_enqueue_try+0xc1/0x210 [obdclass] [1415909369.169600] [&lt;ffffffffa0a27e2d&gt;] lov_lock_enqueue_one+0x4d/0x170 [lov] [1415909369.169600] [&lt;ffffffffa0a2b07b&gt;] lov_lock_enqueue+0x16b/0x3e0 [lov] [1415909369.169600] [&lt;ffffffffa06c4722&gt;] cl_enqueue_kick+0x62/0x150 [obdclass] [1415909369.169600] [&lt;ffffffffa06c7f31&gt;] cl_enqueue_try+0xc1/0x210 [obdclass] [1415909369.169600] [&lt;ffffffffa06c8817&gt;] cl_enqueue_locked+0x77/0x1e0 [obdclass] [1415909369.169600] [&lt;ffffffffa06c9509&gt;] cl_lock_request+0x79/0x280 [obdclass] [1415909369.169600] [&lt;ffffffffa0afcd2d&gt;] cl_glimpse_lock+0x17d/0x490 [lustre] [1415909369.169600] [&lt;ffffffffa0afd1ef&gt;] cl_glimpse_size0+0x1af/0x1e0 [lustre] [1415909369.169600] [&lt;ffffffffa0af5aae&gt;] ll_agl_trigger+0x2ae/0x4d0 [lustre] [1415909369.169600] [&lt;ffffffffa0afbeac&gt;] ll_statahead_thread+0x40c/0xd10 [lustre] [1415909369.169600] [&lt;ffffffff8147ae64&gt;] kernel_thread_helper+0x4/0x10 </pre> </div></div> <p>I guess this is a reproduction of <a href="https://jira.whamcloud.com/browse/LU-4300" title="ptlrpcd threads deadlocked in cl_lock_mutex_get" class="issue-link" data-issue-key="LU-4300"><del>LU-4300</del></a>. If we can get the log from OST side, we can verify if this.</p> <p>Anyway, let's try patch at <a href="https://jira.whamcloud.com/browse/LU-4300" title="ptlrpcd threads deadlocked in cl_lock_mutex_get" class="issue-link" data-issue-key="LU-4300"><del>LU-4300</del></a>, master commit: 44cdf59 and see if it can fix the problem. </p> <p>Which logs would you like from the OST side?</p> <p>Lustre debug log.</p> <p><a href="https://jira.whamcloud.com/browse/LU-4300" title="ptlrpcd threads deadlocked in cl_lock_mutex_get" class="issue-link" data-issue-key="LU-4300"><del>LU-4300</del></a> looks has fixed the issue. Since applying the patch we have not hit the bug.</p> <p>Please close this</p> <p>Thanks Mahmoud</p> Development Rank 1|hzx0tr: Rank (Obsolete) 16510 Severity