https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4286 Lustre <p>We have hit this at least once a week for the last month. A seemingly random OST gets stuck refusing reconnection with a busy RPC after an ll_ost stack trace e.g.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Nov 15 07:23:21 boss4 kernel: LustreError: 28716:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Nov 15 07:23:36 boss4 kernel: LustreError: 28276:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Nov 15 07:23:36 boss4 kernel: LustreError: 28276:0:(ost_handler.c:1764:ost_blocking_ast()) Skipped 1 previous similar message Nov 15 07:43:31 boss4 kernel: LustreError: 16252:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Nov 15 07:43:31 boss4 kernel: LustreError: 16252:0:(ost_handler.c:1764:ost_blocking_ast()) Skipped 6 previous similar messages Nov 15 07:44:24 boss4 kernel: LNet: Service thread pid 5365 was inactive <span class="code-keyword">for</span> 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: Nov 15 07:44:24 boss4 kernel: Pid: 5365, comm: ll_ost02_083 Nov 15 07:44:24 boss4 kernel: Nov 15 07:44:24 boss4 kernel: Call Trace: Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa042b6fe&gt;] cfs_waitq_wait+0xe/0x10 [libcfs] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa05a6523&gt;] lu_object_find_at+0xb3/0x360 [obdclass] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa05a67e6&gt;] lu_object_find+0x16/0x20 [obdclass] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa0e015c5&gt;] ofd_object_find+0x35/0xf0 [ofd] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa05a73ee&gt;] ? lu_env_init+0x1e/0x30 [obdclass] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa0e11549&gt;] ofd_lvbo_update+0x6d9/0xea8 [ofd] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa0df9fe3&gt;] ofd_setattr+0x7f3/0xbd0 [ofd] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa0dc7c1c&gt;] ost_setattr+0x31c/0x990 [ost] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa0dcb746&gt;] ost_handle+0x21e6/0x48e0 [ost] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa0739beb&gt;] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa07423c8&gt;] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa042b5de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa043cd9f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa0739729&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffff81055ad3&gt;] ? __wake_up+0x53/0x70 Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa074375e&gt;] ptlrpc_main+0xace/0x1700 [ptlrpc] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa0742c90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa0742c90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffffa0742c90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Nov 15 07:44:24 boss4 kernel: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 Nov 15 07:44:24 boss4 kernel: Nov 15 07:44:24 boss4 kernel: LustreError: dumping log to /tmp/lustre-log.1384501464.5365 Nov 15 07:57:11 boss4 kernel: Lustre: 5360:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-367), not sending early reply Nov 15 07:57:11 boss4 kernel: req@ffff88324df9f400 x1450789664028660/t0(0) o2-&gt;bravo-MDT0000-mdtlov_UUID@10.21.22.50@tcp:0/0 lens 560/432 e 5 to 0 dl 1384502236 ref 2 fl Interpret:/0/0 rc 0/0 Nov 15 07:59:32 boss4 kernel: Lustre: bravo-OST0040: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Nov 15 07:59:32 boss4 kernel: Lustre: bravo-OST0040: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs Nov 15 07:59:57 boss4 kernel: Lustre: bravo-OST0040: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Nov 15 07:59:57 boss4 kernel: Lustre: bravo-OST0040: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs Nov 15 08:00:22 boss4 kernel: Lustre: bravo-OST0040: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Nov 15 08:00:22 boss4 kernel: Lustre: bravo-OST0040: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs Nov 15 08:00:47 boss4 kernel: Lustre: bravo-OST0040: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Nov 15 08:00:47 boss4 kernel: Lustre: bravo-OST0040: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs </pre> </div></div> <p>The filesystem grinds to a halt and after rebooting the OSS, recovery brings the OST back to the same place - 1 active RPC. We have to mount and abort_recov to bring things back to life. Another example (different OSS/OST):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Nov 9 01:16:37 boss1 kernel: LNet: Service thread pid 16551 was inactive <span class="code-keyword">for</span> 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: Nov 9 01:16:37 boss1 kernel: Pid: 16551, comm: ll_ost02_038 Nov 9 01:16:37 boss1 kernel: Nov 9 01:16:37 boss1 kernel: Call Trace: Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa03ca6fe&gt;] cfs_waitq_wait+0xe/0x10 [libcfs] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa0545523&gt;] lu_object_find_at+0xb3/0x360 [obdclass] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa05457e6&gt;] lu_object_find+0x16/0x20 [obdclass] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa0da05c5&gt;] ofd_object_find+0x35/0xf0 [ofd] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa05463ee&gt;] ? lu_env_init+0x1e/0x30 [obdclass] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa0db0549&gt;] ofd_lvbo_update+0x6d9/0xea8 [ofd] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa0d98fe3&gt;] ofd_setattr+0x7f3/0xbd0 [ofd] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa0d66c1c&gt;] ost_setattr+0x31c/0x990 [ost] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa0d6a746&gt;] ost_handle+0x21e6/0x48e0 [ost] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa06d8beb&gt;] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa06e13c8&gt;] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa03ca5de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa03dbd9f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa06d8729&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffff81055ad3&gt;] ? __wake_up+0x53/0x70 Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa06e275e&gt;] ptlrpc_main+0xace/0x1700 [ptlrpc] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa06e1c90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa06e1c90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffffa06e1c90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Nov 9 01:16:37 boss1 kernel: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 Nov 9 01:16:37 boss1 kernel: Nov 9 01:16:37 boss1 kernel: LustreError: dumping log to /tmp/lustre-log.1383959797.16551 Nov 9 01:26:19 boss1 kernel: INFO: task ll_ost02_038:16551 blocked <span class="code-keyword">for</span> more than 120 seconds. Nov 9 01:26:19 boss1 kernel: <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. Nov 9 01:26:19 boss1 kernel: ll_ost02_038 D 0000000000000007 0 16551 2 0x00000080 Nov 9 01:26:19 boss1 kernel: ffff883f44a259b0 0000000000000046 0000000000000000 ffff883f44a25974 Nov 9 01:26:19 boss1 kernel: ffff883f44a25960 ffffc901077e7030 0000000000000246 0000000000000246 Nov 9 01:26:19 boss1 kernel: ffff883f44491ab8 ffff883f44a25fd8 000000000000fb88 ffff883f44491ab8 Nov 9 01:26:19 boss1 kernel: Call Trace: Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa03ca6fe&gt;] cfs_waitq_wait+0xe/0x10 [libcfs] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa0545523&gt;] lu_object_find_at+0xb3/0x360 [obdclass] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa05457e6&gt;] lu_object_find+0x16/0x20 [obdclass] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa0da05c5&gt;] ofd_object_find+0x35/0xf0 [ofd] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa05463ee&gt;] ? lu_env_init+0x1e/0x30 [obdclass] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa0db0549&gt;] ofd_lvbo_update+0x6d9/0xea8 [ofd] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa0d98fe3&gt;] ofd_setattr+0x7f3/0xbd0 [ofd] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa0d66c1c&gt;] ost_setattr+0x31c/0x990 [ost] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa0d6a746&gt;] ost_handle+0x21e6/0x48e0 [ost] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa06d8beb&gt;] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa06e13c8&gt;] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa03ca5de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa03dbd9f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa06d8729&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffff81055ad3&gt;] ? __wake_up+0x53/0x70 Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa06e275e&gt;] ptlrpc_main+0xace/0x1700 [ptlrpc] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa06e1c90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa06e1c90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffffa06e1c90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Nov 9 01:26:19 boss1 kernel: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 Nov 9 01:26:44 boss1 kernel: Lustre: 16528:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-207), not sending early reply Nov 9 01:26:44 boss1 kernel: req@ffff883e8281f400 x1450784570036048/t0(0) o2-&gt;bravo-MDT0000-mdtlov_UUID@10.21.22.50@tcp:0/0 lens 560/432 e 5 to 0 dl 1383960409 ref 2 fl Interpret:/0/0 rc 0/0 Nov 9 01:28:49 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Nov 9 01:28:49 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs Nov 9 01:28:49 boss1 kernel: Lustre: Skipped 568 previous similar messages Nov 9 01:29:14 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Nov 9 01:29:14 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs Nov 9 01:29:39 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Nov 9 01:29:39 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs Nov 9 01:30:04 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Nov 9 01:30:04 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs Nov 9 01:30:29 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Nov 9 01:30:29 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs Nov 9 01:30:54 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Nov 9 01:30:54 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs Nov 9 01:31:19 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Nov 9 01:31:19 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs Nov 9 01:31:44 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Nov 9 01:31:44 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs Nov 9 01:32:34 boss1 kernel: Lustre: bravo-OST000d: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting </pre> </div></div> <p>If I can provide you with any more information next time it happens then let me know.</p> EL6.4 LU-4286

OST - still busy with 1 active RPCs

Bug Minor Resolved Duplicate Hongchao Zhang Daire Byrne Thu, 21 Nov 2013 12:32:14 +0000 Wed, 2 Apr 2014 02:51:12 +0000 Wed, 2 Apr 2014 02:51:12 +0000 Lustre 2.4.1 2 9 <p>We are still seeing this around once/twice a week. We have to reboot the OSS server once it happens. Is there anything else we can do to help you root cause the issue? It seems to occur more often when we run a large deletion across the filesystem. Our workload is extremely metadata heavy with millions of links/unlinks per day.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Jan 8 15:24:43 boss2 kernel: LustreError: 26897:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Jan 8 15:31:48 boss2 kernel: LustreError: 18797:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Jan 8 15:46:17 boss2 kernel: LNet: Service thread pid 22361 was inactive <span class="code-keyword">for</span> 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: Jan 8 15:46:17 boss2 kernel: Pid: 22361, comm: ll_ost02_039 Jan 8 15:46:17 boss2 kernel: Jan 8 15:46:17 boss2 kernel: Call Trace: Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa04af6fe&gt;] cfs_waitq_wait+0xe/0x10 [libcfs] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa060e523&gt;] lu_object_find_at+0xb3/0x360 [obdclass] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa060e7e6&gt;] lu_object_find+0x16/0x20 [obdclass] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa0db85c5&gt;] ofd_object_find+0x35/0xf0 [ofd] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa060f3ee&gt;] ? lu_env_init+0x1e/0x30 [obdclass] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa0dc8549&gt;] ofd_lvbo_update+0x6d9/0xea8 [ofd] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa0db0fe3&gt;] ofd_setattr+0x7f3/0xbd0 [ofd] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa0d7ec1c&gt;] ost_setattr+0x31c/0x990 [ost] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa0d82746&gt;] ost_handle+0x21e6/0x48e0 [ost] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa07a1beb&gt;] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa07aa3c8&gt;] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa04af5de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa04c0d9f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa07a1729&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa07ab75e&gt;] ptlrpc_main+0xace/0x1700 [ptlrpc] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa07aac90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa07aac90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffffa07aac90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jan 8 15:46:17 boss2 kernel: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 Jan 8 15:46:17 boss2 kernel: Jan 8 15:46:17 boss2 kernel: LustreError: dumping log to /tmp/lustre-log.1389195977.22361 Jan 8 15:51:58 boss2 kernel: INFO: task ll_ost02_039:22361 blocked <span class="code-keyword">for</span> more than 120 seconds. Jan 8 15:51:58 boss2 kernel: <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. Jan 8 15:51:58 boss2 kernel: ll_ost02_039 D 0000000000000015 0 22361 2 0x00000000 Jan 8 15:51:58 boss2 kernel: ffff882f016bf9b0 0000000000000046 0000000000000000 ffff882f016bf974 Jan 8 15:51:58 boss2 kernel: ffff882f016bf960 ffffc900cd15a030 0000000000000246 0000000000000246 Jan 8 15:51:58 boss2 kernel: ffff8837d33bd058 ffff882f016bffd8 000000000000fb88 ffff8837d33bd058 Jan 8 15:51:58 boss2 kernel: Call Trace: Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa04af6fe&gt;] cfs_waitq_wait+0xe/0x10 [libcfs] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa060e523&gt;] lu_object_find_at+0xb3/0x360 [obdclass] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa060e7e6&gt;] lu_object_find+0x16/0x20 [obdclass] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa0db85c5&gt;] ofd_object_find+0x35/0xf0 [ofd] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa060f3ee&gt;] ? lu_env_init+0x1e/0x30 [obdclass] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa0dc8549&gt;] ofd_lvbo_update+0x6d9/0xea8 [ofd] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa0db0fe3&gt;] ofd_setattr+0x7f3/0xbd0 [ofd] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa0d7ec1c&gt;] ost_setattr+0x31c/0x990 [ost] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa0d82746&gt;] ost_handle+0x21e6/0x48e0 [ost] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa07a1beb&gt;] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa07aa3c8&gt;] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa04af5de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa04c0d9f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa07a1729&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa07ab75e&gt;] ptlrpc_main+0xace/0x1700 [ptlrpc] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa07aac90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa07aac90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffffa07aac90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jan 8 15:51:58 boss2 kernel: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 Jan 8 15:53:58 boss2 kernel: INFO: task ll_ost02_039:22361 blocked <span class="code-keyword">for</span> more than 120 seconds. Jan 8 15:53:58 boss2 kernel: <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. Jan 8 15:53:58 boss2 kernel: ll_ost02_039 D 0000000000000015 0 22361 2 0x00000000 Jan 8 15:53:58 boss2 kernel: ffff882f016bf9b0 0000000000000046 0000000000000000 ffff882f016bf974 Jan 8 15:53:58 boss2 kernel: ffff882f016bf960 ffffc900cd15a030 0000000000000246 0000000000000246 Jan 8 15:53:58 boss2 kernel: ffff8837d33bd058 ffff882f016bffd8 000000000000fb88 ffff8837d33bd058 Jan 8 15:53:58 boss2 kernel: Call Trace: Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa04af6fe&gt;] cfs_waitq_wait+0xe/0x10 [libcfs] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa060e523&gt;] lu_object_find_at+0xb3/0x360 [obdclass] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa060e7e6&gt;] lu_object_find+0x16/0x20 [obdclass] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa0db85c5&gt;] ofd_object_find+0x35/0xf0 [ofd] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa060f3ee&gt;] ? lu_env_init+0x1e/0x30 [obdclass] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa0dc8549&gt;] ofd_lvbo_update+0x6d9/0xea8 [ofd] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa0db0fe3&gt;] ofd_setattr+0x7f3/0xbd0 [ofd] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa0d7ec1c&gt;] ost_setattr+0x31c/0x990 [ost] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa0d82746&gt;] ost_handle+0x21e6/0x48e0 [ost] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa07a1beb&gt;] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa07aa3c8&gt;] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa04af5de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa04c0d9f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa07a1729&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa07ab75e&gt;] ptlrpc_main+0xace/0x1700 [ptlrpc] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa07aac90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa07aac90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffffa07aac90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jan 8 15:53:58 boss2 kernel: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 Jan 8 15:55:58 boss2 kernel: INFO: task ll_ost02_039:22361 blocked <span class="code-keyword">for</span> more than 120 seconds. Jan 8 15:55:58 boss2 kernel: <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. Jan 8 15:55:58 boss2 kernel: ll_ost02_039 D 0000000000000015 0 22361 2 0x00000000 Jan 8 15:55:58 boss2 kernel: ffff882f016bf9b0 0000000000000046 0000000000000000 ffff882f016bf974 Jan 8 15:55:58 boss2 kernel: ffff882f016bf960 ffffc900cd15a030 0000000000000246 0000000000000246 Jan 8 15:55:58 boss2 kernel: ffff8837d33bd058 ffff882f016bffd8 000000000000fb88 ffff8837d33bd058 Jan 8 15:55:58 boss2 kernel: Call Trace: Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa04af6fe&gt;] cfs_waitq_wait+0xe/0x10 [libcfs] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa060e523&gt;] lu_object_find_at+0xb3/0x360 [obdclass] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa060e7e6&gt;] lu_object_find+0x16/0x20 [obdclass] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa0db85c5&gt;] ofd_object_find+0x35/0xf0 [ofd] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa060f3ee&gt;] ? lu_env_init+0x1e/0x30 [obdclass] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa0dc8549&gt;] ofd_lvbo_update+0x6d9/0xea8 [ofd] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa0db0fe3&gt;] ofd_setattr+0x7f3/0xbd0 [ofd] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa0d7ec1c&gt;] ost_setattr+0x31c/0x990 [ost] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa0d82746&gt;] ost_handle+0x21e6/0x48e0 [ost] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa07a1beb&gt;] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa07aa3c8&gt;] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa04af5de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa04c0d9f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa07a1729&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa07ab75e&gt;] ptlrpc_main+0xace/0x1700 [ptlrpc] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa07aac90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa07aac90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffffa07aac90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jan 8 15:55:58 boss2 kernel: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 Jan 8 15:56:24 boss2 kernel: Lustre: 22383:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-207), not sending early reply Jan 8 15:56:24 boss2 kernel: req@ffff883243230c00 x1456592052377504/t0(0) o2-&gt;bravo-MDT0000-mdtlov_UUID@10.21.22.50@tcp:0/0 lens 560/432 e 5 to 0 dl 1389196589 ref 2 fl Interpret:/0/0 rc 0/0 Jan 8 15:57:58 boss2 kernel: INFO: task ll_ost02_039:22361 blocked <span class="code-keyword">for</span> more than 120 seconds. Jan 8 15:57:58 boss2 kernel: <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. Jan 8 15:57:58 boss2 kernel: ll_ost02_039 D 0000000000000015 0 22361 2 0x00000000 Jan 8 15:57:58 boss2 kernel: ffff882f016bf9b0 0000000000000046 0000000000000000 ffff882f016bf974 Jan 8 15:57:58 boss2 kernel: ffff882f016bf960 ffffc900cd15a030 0000000000000246 0000000000000246 Jan 8 15:57:58 boss2 kernel: ffff8837d33bd058 ffff882f016bffd8 000000000000fb88 ffff8837d33bd058 Jan 8 15:57:58 boss2 kernel: Call Trace: Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa04af6fe&gt;] cfs_waitq_wait+0xe/0x10 [libcfs] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa060e523&gt;] lu_object_find_at+0xb3/0x360 [obdclass] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa060e7e6&gt;] lu_object_find+0x16/0x20 [obdclass] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa0db85c5&gt;] ofd_object_find+0x35/0xf0 [ofd] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa060f3ee&gt;] ? lu_env_init+0x1e/0x30 [obdclass] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa0dc8549&gt;] ofd_lvbo_update+0x6d9/0xea8 [ofd] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa0db0fe3&gt;] ofd_setattr+0x7f3/0xbd0 [ofd] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa0d7ec1c&gt;] ost_setattr+0x31c/0x990 [ost] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa0d82746&gt;] ost_handle+0x21e6/0x48e0 [ost] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa07a1beb&gt;] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa07aa3c8&gt;] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa04af5de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa04c0d9f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa07a1729&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa07ab75e&gt;] ptlrpc_main+0xace/0x1700 [ptlrpc] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa07aac90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa07aac90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffffa07aac90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jan 8 15:57:58 boss2 kernel: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 Jan 8 15:58:21 boss2 kernel: Lustre: bravo-OST001a: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Jan 8 15:58:21 boss2 kernel: Lustre: Skipped 1 previous similar message Jan 8 15:58:21 boss2 kernel: Lustre: bravo-OST001a: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs Jan 8 15:58:46 boss2 kernel: Lustre: bravo-OST001a: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Jan 8 15:58:46 boss2 kernel: Lustre: bravo-OST001a: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs Jan 8 15:59:12 boss2 kernel: Lustre: bravo-OST001a: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Jan 8 15:59:12 boss2 kernel: Lustre: bravo-OST001a: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs Jan 8 15:59:37 boss2 kernel: Lustre: bravo-OST001a: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) reconnecting Jan 8 15:59:37 boss2 kernel: Lustre: bravo-OST001a: Client bravo-MDT0000-mdtlov_UUID (at 10.21.22.50@tcp) refused reconnection, still busy with 1 active RPCs </pre> </div></div> <p>Hongchao</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>I think this is almost certainly a dup of <a href="https://jira.whamcloud.com/browse/LU-4019" title="today&#39;s master stick on shutdown on test == sanity test 132: on lu_object_find_at" class="issue-link" data-issue-key="LU-4019"><del>LU-4019</del></a>, there's a patch available in there.</p> <p>Yes, this issue should be the duplicate of <a href="https://jira.whamcloud.com/browse/LU-4019" title="today&#39;s master stick on shutdown on test == sanity test 132: on lu_object_find_at" class="issue-link" data-issue-key="LU-4019"><del>LU-4019</del></a>.</p> <p>Hi Daire,</p> <p>Could you please test with the patch in <a href="https://jira.whamcloud.com/browse/LU-4019" title="today&#39;s master stick on shutdown on test == sanity test 132: on lu_object_find_at" class="issue-link" data-issue-key="LU-4019"><del>LU-4019</del></a> (<a href="http://review.whamcloud.com/#/c/7795/)?" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7795/)?</a></p> <p>Thanks</p> <p>Okay. I will wait for us to hit this again (to make sure something else hasn't changed) and then apply the patch. Cheers.</p> <p>Hit the bug a couple of times overnight. I've patched in the fix. I will report back if we see this again.</p> <p>Hi Daire,<br/> how is the test going, does the issue occur again after patching the fix?<br/> Thanks</p> <p>Well there have been no further occurrences yet but it took ~3 weeks between instances last time so we will have to sit on this for a while and see what happens.</p> <p>Cheers.</p> <p>We are not seeing the ll_ost call traces, but are seeing frequent hits of this in our syslogs for oss nodes:</p> <p><span class="error">&#91;root@lustre-mgmt1 apps&#93;</span># grep "Mar 9" lustrekernel | grep "Error -2 syncing" | wc -l<br/> 508<br/> <span class="error">&#91;root@lustre-mgmt1 apps&#93;</span># grep "Mar 10" lustrekernel | grep "Error -2 syncing" | wc -l<br/> 3420<br/> <span class="error">&#91;root@lustre-mgmt1 apps&#93;</span># grep "Mar 11" lustrekernel | grep "Error -2 syncing" | wc -l<br/> 1076<br/> <span class="error">&#91;root@lustre-mgmt1 apps&#93;</span># grep "Mar 12" lustrekernel | grep "Error -2 syncing" | wc -l<br/> 154<br/> <span class="error">&#91;root@lustre-mgmt1 apps&#93;</span># grep "Mar 13" lustrekernel | grep "Error -2 syncing" | wc -l<br/> 2645</p> <p>what further debug can we provide? This is on the production filesystems here at ORNL.</p> <p>Error -2 syncing itself is a somewhat valid race, I would think, though I am nto exactly sure how would it happen.<br/> You might want to apply 4019 patch just in case since it does fix a real bug of course.</p> <p>this message in syslogs should be in "ost_blocking_ast", and it is not an error case,</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">int</span> ost_blocking_ast(struct ldlm_lock *lock, struct ldlm_lock_desc *desc, void *data, <span class="code-object">int</span> flag) { ... <span class="code-keyword">if</span> (rc == 0 &amp;&amp; flag == LDLM_CB_CANCELING &amp;&amp; (lock-&gt;l_granted_mode &amp; (LCK_PW|LCK_GROUP)) &amp;&amp; (sync_lock_cancel == ALWAYS_SYNC_ON_CANCEL || (sync_lock_cancel == BLOCKING_SYNC_ON_CANCEL &amp;&amp; lock-&gt;l_flags &amp; LDLM_FL_CBPENDING))) { ... rc = obd_sync(&amp;env, lock-&gt;l_export, oinfo, lock-&gt;l_policy_data.l_extent.start, lock-&gt;l_policy_data.l_extent.end, NULL); <span class="code-keyword">if</span> (rc) CERROR(<span class="code-quote">"Error %d syncing data on lock cancel\n"</span>, rc); OBDO_FREE(oa); OBD_FREE_PTR(oinfo); } ... } </pre> </div></div> <p>when the object is being deleted, "LU_OBJECT_HEARD_BANSHEE" flag will be set, and ofd_sync will return -ENOENT (-2)</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-object">int</span> ofd_sync(<span class="code-keyword">const</span> struct lu_env *env, struct obd_export *exp, struct obd_info *oinfo, obd_size start, obd_size end, struct ptlrpc_request_set *set) { ... <span class="code-keyword">if</span> (!ofd_object_exists(fo)) GOTO(put, rc = -ENOENT); ... } <span class="code-keyword">static</span> inline <span class="code-object">int</span> ofd_object_exists(struct ofd_object *obj) { LASSERT(obj != NULL); <span class="code-keyword">if</span> (lu_object_is_dying(obj-&gt;ofo_obj.do_lu.lo_header)) <span class="code-keyword">return</span> 0; <span class="code-keyword">return</span> lu_object_exists(&amp;obj-&gt;ofo_obj.do_lu); } <span class="code-keyword">static</span> inline <span class="code-object">int</span> lu_object_is_dying(<span class="code-keyword">const</span> struct lu_object_header *h) { <span class="code-keyword">return</span> test_bit(LU_OBJECT_HEARD_BANSHEE, &amp;h-&gt;loh_flags); } </pre> </div></div> <p>this issue should be marked as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-4019" title="today&#39;s master stick on shutdown on test == sanity test 132: on lu_object_find_at" class="issue-link" data-issue-key="LU-4019"><del>LU-4019</del></a></p> <p>Daire</p> <p>Have you had a chance to try out the <a href="https://jira.whamcloud.com/browse/LU-4019" title="today&#39;s master stick on shutdown on test == sanity test 132: on lu_object_find_at" class="issue-link" data-issue-key="LU-4019"><del>LU-4019</del></a> patch?</p> <p>Peter</p> <p>Peter,</p> <p>All our servers have been running with the patch from <a href="https://jira.whamcloud.com/browse/LU-4019" title="today&#39;s master stick on shutdown on test == sanity test 132: on lu_object_find_at" class="issue-link" data-issue-key="LU-4019"><del>LU-4019</del></a> for over a week now. Sometimes it took a couple of weeks to see this bug so we are waiting a bit longer before we resolve this one.</p> <p>Cheers,</p> <p>Fair enough - keep us posted!</p> <p>Well we normally would have seen an instance of this by now (~3 weeks since we patched). Feel free to close this ticket and I'll re-open if required.</p> <p>Cheers.</p> <p>ok - thanks Daire!</p> Development Rank 1|hzw9vb: Rank (Obsolete) 11764 Severity