https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6976 Lustre <p>Umounting client with oss down cased LBUG.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">(client.c:1940:ptlrpc_expire_one_request()) @@@ Request sent has timed out <span class="code-keyword">for</span> sent delay: [sent 1438873620/real 0] req@ffff880306604000 x1508264680679164/t0(0) o8-&gt;nbp5-OST0075-osc-ffff8802f3e12400@10.151.25.242@o2ib:28/4 lens 400/544 e 0 to 1 dl 1438873725 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [1438873725.449640] Lustre: 4378:0:(client.c:1940:ptlrpc_expire_one_request()) Skipped 74 previous similar messages [1438873850.511148] LustreError: 21772:0:(obd_config.c:1221:class_process_config()) no device <span class="code-keyword">for</span>: nbp5-OST0055-osc-ffff8802f3e12400 [1438873850.523149] LustreError: 21772:0:(obd_config.c:1221:class_process_config()) Skipped 14 previous similar messages [1438873850.535149] LustreError: 21772:0:(obd_config.c:1775:class_manual_cleanup()) cleanup failed -22: nbp5-OST0055-osc-ffff8802f3e12400 [1438873850.535149] LustreError: 21772:0:(obd_config.c:1775:class_manual_cleanup()) Skipped 14 previous similar messages [1438873850.535149] Lustre: Unmounted nbp5-client [1438876602.632140] LustreError: 4347:0:(osc_request.c:3597:osc_cleanup()) ASSERTION( atomic_read(&amp;cli-&gt;cl_cache-&gt;ccc_users) &gt; 0 ) failed: [1438876602.644140] LustreError: 4347:0:(osc_request.c:3597:osc_cleanup()) LBUG [1438876602.652140] Pid: 4347, comm: obd_zombid [1438876602.656141] [1438876602.656141] Call Trace: [1438876602.660141] [&lt;ffffffff81004b95&gt;] dump_trace+0x75/0x300 [1438876602.668141] [&lt;ffffffffa057b82a&gt;] libcfs_debug_dumpstack+0x4a/0x70 [libcfs] [1438876602.676142] [&lt;ffffffffa057bd5e&gt;] lbug_with_loc+0x3e/0xb0 [libcfs] [1438876602.680142] [&lt;ffffffffa096f8eb&gt;] osc_cleanup+0x16b/0x170 [osc] [1438876602.688142] [&lt;ffffffffa06b959f&gt;] class_decref+0x11f/0x550 [obdclass] [1438876602.696142] [&lt;ffffffffa06971ee&gt;] class_export_destroy+0xfe/0x480 [obdclass] [1438876602.704143] [&lt;ffffffffa069763d&gt;] obd_zombie_impexp_cull+0xcd/0x1e0 [obdclass] [1438876602.712143] [&lt;ffffffffa06977a5&gt;] obd_zombie_impexp_thread+0x55/0x1a0 [obdclass] [1438876602.720143] [&lt;ffffffff81083ae6&gt;] kthread+0x96/0xa0 [1438876602.720143] [&lt;ffffffff8147e8e4&gt;] kernel_thread_helper+0x4/0x10 [1438876602.732144] </pre> </div></div> LU-6976

LustreError: 4156:0:(osc_request.c:3597:osc_cleanup()) ASSERTION( atomic_read(&cli->cl_cache->ccc_users) > 0 ) failed:

Bug Major Resolved Incomplete Emoly Liu Mahmoud Hanafi Mon, 10 Aug 2015 15:44:44 +0000 Fri, 11 Mar 2016 01:28:08 +0000 Fri, 11 Mar 2016 01:28:08 +0000 Lustre 2.5.3 0 7 <p>Mahmoud, is this a one-time occurrence or is this happening regularly?</p> <p>It looks like the cl_cache reference count is bad. That might be due to the immediately-preceding errors, but it could also be due to any previous error that happened during the lifetime of the client, so it may be difficult to debug.</p> <p>One path forward is to improve the debugging in osc_cleanup() so that the LASSERT() is changed to LASSERTF() to actually print ccc_count, so that we can verify the value (either 0, a small negative value, or some garbage value caused by memory corruption). It may also be worthwhile in this case to print whether the cl_lru_osc list is empty or not.</p> <p>It is also worthwhile to check the places that change the refcount to see if it is possible to try to cleanup the OSC device multiple times in case of error.</p> <p>Last week we has hardware issues, with the journal ssd, on one of our OSS. We needed to umount clients, due to ib_cm getting overloaded during OSS bring up. 4 clients LBUG when umount the filesystem.</p> <p>Mahmoud, I think your LBUG issue is similar to <a href="https://jira.whamcloud.com/browse/LU-6173" title="CPU stalled with obd_zombid running" class="issue-link" data-issue-key="LU-6173"><del>LU-6173</del></a>. That patch has been included by master and some other branches.</p> <p>Could you please have a check if your tree has it? Thanks.</p> <p>This is not in the NASA 2.5.3 based release in production but is in the 2.5.5 FE based release under testing atm</p> <p>Thank you, Peter.</p> <p>Mahmoud, does the issue still exist? Or can we mark this ticket as resolved? Thanks.</p> <p>Mahmoud,</p> <p>Please let us know if you need any more work done on this ticket.</p> <p>Thanks,<br/> ~ jfc.</p> Development Rank 1|hzxk4f: Rank (Obsolete) 9223372036854775807 Severity