https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2615 Lustre <p>We have got 4 OSSes that crash at the same time, at umount, with the following bt :</p> <p>PID: 18173 TASK: ffff8803376dc040 CPU: 4 COMMAND: "umount"<br/> #0 <span class="error">&#91;ffff8802b115f8d0&#93;</span> machine_kexec at ffffffff8102895b<br/> 0000001 <span class="error">&#91;ffff8802b115f930&#93;</span> crash_kexec at ffffffff810a4622<br/> 0000002 <span class="error">&#91;ffff8802b115fa00&#93;</span> panic at ffffffff81484657<br/> 0000003 <span class="error">&#91;ffff8802b115fa80&#93;</span> lbug_with_loc at ffffffffa04ade5b <span class="error">&#91;libcfs&#93;</span><br/> 0000004 <span class="error">&#91;ffff8802b115faa0&#93;</span> llog_recov_thread_stop at ffffffffa072e55b <span class="error">&#91;ptlrpc&#93;</span><br/> 0000005 <span class="error">&#91;ffff8802b115fad0&#93;</span> llog_recov_thread_fini at ffffffffa072e593 <span class="error">&#91;ptlrpc&#93;</span><br/> 0000006 <span class="error">&#91;ffff8802b115faf0&#93;</span> filter_llog_finish at ffffffffa0c7d3dd <span class="error">&#91;obdfilter&#93;</span><br/> 0000007 <span class="error">&#91;ffff8802b115fb20&#93;</span> obd_llog_finish at ffffffffa057c2f8 <span class="error">&#91;obdclass&#93;</span><br/> 0000008 <span class="error">&#91;ffff8802b115fb40&#93;</span> filter_precleanup at ffffffffa0c7cdaf <span class="error">&#91;obdfilter&#93;</span><br/> 0000009 <span class="error">&#91;ffff8802b115fba0&#93;</span> class_cleanup at ffffffffa05a3ca7 <span class="error">&#91;obdclass&#93;</span><br/> 0000010 <span class="error">&#91;ffff8802b115fc20&#93;</span> class_process_config at ffffffffa05a5feb <span class="error">&#91;obdclass&#93;</span><br/> 0000011 <span class="error">&#91;ffff8802b115fcb0&#93;</span> class_manual_cleanup at ffffffffa05a6d29 <span class="error">&#91;obdclass&#93;</span><br/> 0000012 <span class="error">&#91;ffff8802b115fd70&#93;</span> server_put_super at ffffffffa05b2c0c <span class="error">&#91;obdclass&#93;</span><br/> 0000013 <span class="error">&#91;ffff8802b115fe40&#93;</span> generic_shutdown_super at ffffffff8116542b<br/> 0000014 <span class="error">&#91;ffff8802b115fe60&#93;</span> kill_anon_super at ffffffff81165546<br/> 0000015 <span class="error">&#91;ffff8802b115fe80&#93;</span> lustre_kill_super at ffffffffa05a8966 <span class="error">&#91;obdclass&#93;</span><br/> 0000016 <span class="error">&#91;ffff8802b115fea0&#93;</span> deactivate_super at ffffffff811664e0<br/> 0000017 <span class="error">&#91;ffff8802b115fec0&#93;</span> mntput_no_expire at ffffffff811826bf<br/> 0000018 <span class="error">&#91;ffff8802b115fef0&#93;</span> sys_umount at ffffffff81183188<br/> 0000019 <span class="error">&#91;ffff8802b115ff80&#93;</span> system_call_fastpath at ffffffff810030f2<br/> RIP: 00007f62ddfbdd67 RSP: 00007fffab738308 RFLAGS: 00010202<br/> RAX: 00000000000000a6 RBX: ffffffff810030f2 RCX: 0000000000000010<br/> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007f62deeb3bb0<br/> RBP: 00007f62deeb3b80 R8: 00007f62deeb3bd0 R9: 0000000000000000<br/> R10: 00007fffab738130 R11: 0000000000000246 R12: 0000000000000000<br/> R13: 0000000000000000 R14: 0000000000000000 R15: 00007f62deeb3c10<br/> ORIG_RAX: 00000000000000a6 CS: 0033 SS: 002b</p> <p>This bt is identical as the one shown <a href="https://jira.whamcloud.com/browse/LU-1194" title="llog_recov_thread_stop+0x1ae/0x1b0 asserting" class="issue-link" data-issue-key="LU-1194"><del>LU-1194</del></a> which is supposed to be fixed in 2.1.3.</p> <p>Site is classified so I can't upload the binary crash but I can export the content of some structures upon request.</p> LU-2615

group of OSS crashed at umount

Bug Major Resolved Done Hongchao Zhang Alexandre Louvet ptr Mon, 14 Jan 2013 09:16:20 +0000 Thu, 17 Dec 2015 23:57:24 +0000 Thu, 17 Dec 2015 23:57:24 +0000 Lustre 2.1.3 0 8 <p>Hongchao</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>is the debug log before the crash available? it will print the content of the remain llcd and the address of llog_ctxt and<br/> llog_commit_master will be printed also,</p> <p> CDEBUG(D_RPCTRACE, "Llcd (%p) at %s:%d:\n", llcd, func, line);<br/> CDEBUG(D_RPCTRACE, " size: %d\n", llcd-&gt;llcd_size);<br/> CDEBUG(D_RPCTRACE, " ctxt: %p\n", llcd-&gt;llcd_ctxt);<br/> CDEBUG(D_RPCTRACE, " lcm : %p\n", llcd-&gt;llcd_lcm);<br/> CDEBUG(D_RPCTRACE, " cookiebytes : %d\n", llcd-&gt;llcd_cookiebytes);</p> <p>could you please print the content of the two structures? Thanks!</p> <p>the other possible reason of this issue is the ptlrpc_request created in llog_send is not completed normally, and its rq_interpret_reply (llcd_interpret)<br/> is not called to free the llcd.</p> <p>Hi, what patches are you using with the 2.1.3?</p> <p>Here is the list of patches we have on our production machine.</p> <ul class="alternate" type="square"> <li><a href="https://jira.whamcloud.com/browse/LU-1039" title="data corruption in check_set" class="issue-link" data-issue-key="LU-1039"><del>LU-1039</del></a> handle bulk IO errors correctly</li> <li><a href="https://jira.whamcloud.com/browse/LU-1650" title="crash of lustre clients in osc_req_attr_set() routine" class="issue-link" data-issue-key="LU-1650"><del>LU-1650</del></a> find the lock by index of subpage: update with patch set 2</li> <li><a href="https://jira.whamcloud.com/browse/LU-2170" title="osc_extent_merge()) ASSERTION( cur-&gt;oe_osclock == victim-&gt;oe_osclock) while running racer" class="issue-link" data-issue-key="LU-2170"><del>LU-2170</del></a> osc: set osc_lock attribute only once</li> <li>ORNL-22 general ptlrpcd threads pool support</li> <li><a href="https://jira.whamcloud.com/browse/LU-1144" title="implement a NUMA aware ptlrpcd binding policy" class="issue-link" data-issue-key="LU-1144"><del>LU-1144</del></a> implement a NUMA aware ptlrpcd binding policy</li> <li><a href="https://jira.whamcloud.com/browse/LU-1110" title="MDS Oops in osd_xattr_get() during file open by FID" class="issue-link" data-issue-key="LU-1110"><del>LU-1110</del></a> MDS Oops in osd_xattr_get() during file open by FID</li> <li><a href="https://jira.whamcloud.com/browse/LU-1363" title="SELinux and stateahead hang" class="issue-link" data-issue-key="LU-1363"><del>LU-1363</del></a> llite: Not held lock when calling security_d_instantiate</li> <li><a href="https://jira.whamcloud.com/browse/LU-645" title="getcwd fails" class="issue-link" data-issue-key="LU-645"><del>LU-645</del></a>/BZ23978 getcwd failure</li> <li><a href="https://jira.whamcloud.com/browse/LU-1299" title="running truncated executable causes spewing of lock debug messages" class="issue-link" data-issue-key="LU-1299"><del>LU-1299</del></a> (patch set 11) loading large enough binary from lustre trigger OOM killer</li> <li><a href="https://jira.whamcloud.com/browse/LU-549" title="Add xattr list/value cache on client" class="issue-link" data-issue-key="LU-549"><del>LU-549</del></a> Improve statfs performance if selinux is disabled (complement)</li> <li><a href="https://jira.whamcloud.com/browse/LU-1749" title="llog_lvfs_create()) error looking up logfile" class="issue-link" data-issue-key="LU-1749"><del>LU-1749</del></a> mdt failing to open llog result in invalidating ost</li> <li><a href="https://jira.whamcloud.com/browse/LU-1331" title="changelogs: RNMTO record not always after RNMFRM" class="issue-link" data-issue-key="LU-1331"><del>LU-1331</del></a> changelogs: RNMTO record not always after RNMFRM</li> <li><a href="https://jira.whamcloud.com/browse/LU-1592" title="ASSERTION(cfs_atomic_read(&amp;imp-&gt;imp_refcount) == 0) failed: value: -1" class="issue-link" data-issue-key="LU-1592"><del>LU-1592</del></a> ldlm: protect obd_export:exp_imp_reverse's change</li> <li><a href="https://jira.whamcloud.com/browse/LU-1144" title="implement a NUMA aware ptlrpcd binding policy" class="issue-link" data-issue-key="LU-1144"><del>LU-1144</del></a> update patch to add ptlrpc option in modprobe.conf file</li> <li><a href="https://jira.whamcloud.com/browse/LU-1448" title="Disabled OSC can cause NULL pointer dereference when reading import" class="issue-link" data-issue-key="LU-1448"><del>LU-1448</del></a> prevent NULL pointer dereference on disabled OSC</li> <li><a href="https://jira.whamcloud.com/browse/LU-1057" title="low performance maybe related to quota" class="issue-link" data-issue-key="LU-1057"><del>LU-1057</del></a> quota speed up lookup in osc_quota_chkdq.patch</li> <li><a href="https://jira.whamcloud.com/browse/LU-1714" title="crash upon loading libcfs module" class="issue-link" data-issue-key="LU-1714"><del>LU-1714</del></a> properly initialyse sg_magic value.patch</li> </ul> <p>For CDEBUG log it will take a little more time. By default our production machine run with a debug filter set to 0 (none), so I do not have those trace in the dmesg log. I have to spend some time to extract them from the crash.</p> <p>what is the content of the two structures "llog_ctxt" and "llog_commit_master" of the llcd?</p> <p>Here is the content of llog_ctxt &amp; llog_commit_master extracted from the crash dump :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> crash&gt; llog_ctxt 0xffff8802327f8900 struct llog_ctxt { loc_idx = 3, loc_gen = { mnt_cnt = 0, conn_cnt = 0 }, loc_obd = 0xffff88032adbc038, loc_olg = 0xffff88032adbc2d0, loc_exp = 0xffff8802ed751400, loc_imp = 0x0, loc_logops = 0xffffffffa0cb99e0, loc_handle = 0x0, loc_lcm = 0xffff88033614b400, loc_llcd = 0x0, loc_sem = { lock = { raw_lock = { slock = 458759 } }, count = 0, wait_list = { next = 0xffff8802327f8960, prev = 0xffff8802327f8960 } }, loc_refcount = { counter = 2 }, llog_proc_cb = 0xffffffffa0c95220, loc_flags = 2 } crash&gt; llog_commit_master 0xffff88033614b400 struct llog_commit_master { lcm_flags = 4, lcm_count = { counter = 1 }, lcm_refcount = { counter = 3 }, lcm_pc = { pc_flags = 0, pc_lock = { raw_lock = { slock = 65537 } }, pc_starting = { done = 0, wait = { lock = { raw_lock = { slock = 196611 } }, task_list = { next = 0xffff88033614b430, prev = 0xffff88033614b430 } } }, pc_finishing = { done = 0, wait = { lock = { raw_lock = { slock = 196611 } }, task_list = { next = 0xffff88033614b450, prev = 0xffff88033614b450 } } }, pc_set = 0x0, pc_name = "lcm_xxxxx1-OST0", pc_env = { le_ctx = { lc_tags = 2415919112, lc_thread = 0x0, lc_value = 0x0, lc_state = LCS_FINALIZED, lc_remember = { next = 0xffff88033614b498, prev = 0xffff88033614b498 }, lc_version = 15, lc_cookie = 0 }, le_ses = 0x0 }, pc_index = -1, pc_npartners = 0, pc_partners = 0x0, pc_cursor = 0 }, lcm_lock = { raw_lock = { slock = 155453764 } }, lcm_llcds = { next = 0xffff88021b2c2050, prev = 0xffff88021b2c2050 }, lcm_name = "lcm_xxxxx1-OST0002\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000" } </pre> </div></div> <p>the lcm-&gt;lcm_llcds list is empty<br/> ...<br/> lcm_llcds = </p> { next = 0xffff88021b2c2050, prev = 0xffff88021b2c2050 } <p>,<br/> ...</p> <p>but the lcm-&gt;lcm_count is "1", which is very strange!<br/> could you please attach the "lustre/ptlrpc/ptlrpcd.c" and "lustre/ptlrpc/recov_thread.c" in your source tree, thanks very much!</p> <p>Hi, here are the source files requested by Hongchao.</p> <p>the list "lcm_llcds" is corrupted for its value of "next" and "prev" is wrong (it's not in the address region of "struct llog_commit_master").<br/> Could it be memory corrupt? there is no trace of the bug yet, sorry!</p> <p>does the issue occur again recently?</p> <p>&gt; Could it be memory corrupt?<br/> It's unexpected. The serveur is fully ECC protected and, as it's an OSS, almost only linux &amp; lustre are running on this node.</p> <p>&gt; does the issue occur again recently?<br/> It did occurs the last 4 times we did stop lustre on the node.</p> <p>Hi, <br/> Does the kernel dump referred in comment at 11/Feb/13 3:56 PM exist? if so, could you please print the content at 0xffff88021b2c2050<br/> as "struct llog_canceld_ctxt"? besides, can the console output(just the part related to Lustre) be attached here? Thanks a lot!</p> <p>the remaining "llcd" should have been sent over ptlrpc_request for llog_ctxt-&gt;loc_llcd == NULL, and the request could not finish, then "llcd_interpret" wasn't<br/> called to free the "llcd", there are 2 patches (ORNL-22 general ptlrpcd threads pool support; <a href="https://jira.whamcloud.com/browse/LU-1144" title="implement a NUMA aware ptlrpcd binding policy" class="issue-link" data-issue-key="LU-1144"><del>LU-1144</del></a> implement a NUMA aware ptlrpcd binding policy) among<br/> the patches applied currently is related to it, could you please help to revert the 2 patches and test it, Thanks!</p> <p>Hi,</p> <p>It might be difficult to have the opportunity to install packages with those 2 patches reverted at customer site.<br/> Instead, could we just set ptlrpcd_bind_policy=1 and max_ptlrpcds=2 as options for the ptlrpc kernel module, so that it behaves like if patch from ORNL-22 was not applied?<br/> Is it still a relevant test for you?</p> <p>Thanks,<br/> Sebastien.</p> <p>Hi,</p> <p>Yes, it will disable the ptlrpcd thread pools (although not shaking off the patch completely) and it should be still a relevant test.</p> <p>Thanks </p> <p>Hi, what is the output of the test? Thanks</p> <p>Hi,</p> <p>I have asked people on site for the results of the tests.</p> <p>Cheers,<br/> Sebastien.</p> <p>We hit the same problem on lustre-2.1.6 too.</p> <p>After reading a few codes, I am wondering whether it is possible for following race problem to happen. Please correct me if I am wrong.</p> <p>filter_llog_finish<br/> --llog_recov_thread_fini<br/> ----llog_sync<br/> ------llog_obd_repl_sync<br/> --------llog_cancel<br/> ----------llog_obd_repl_cancel<br/> ------------llcd_push<br/> --------------llcd_send<br/> ----------------Sending async<br/> ----llog_recov_thread_stop<br/> ------LBUG，because llcd_send is sending a llcd and llcd_interpret() is not called since no reply has been got now.</p> <p>Thanks!</p> <p>what are the two threads involved in the race?<br/> normally, the llog_recov_thread_stop is only called by llog_recov_thread_fini, and "llog_recov_thread_stop" is called in two places,<br/> one is the cleanup for the failed llog_recov_thread_init call, the other is the normal cleanup phase during device cleanup <br/> (called in filter_llog_finish). they can't be called simultaneously</p> <p>could you please attach some more info about this issue, and can it be reproduced on your site?</p> <p>Hi Hongchao,</p> <p>Sorry, may be 'race' is not the right word to express my thought.</p> <p>At the time llcd_send() returns, the completion handler llcd_interpret() might not be called yet, right? When the llcd is still under use by the RPC on flight, llog_recov_thread_stop() will hit a LBUG. I can't find any codes in filter_llog_finish() which waits for the RPC finishes, so I guess it is possible that when llog_recov_thread_stop() is called, the RPC is still on flight. Am I right?</p> <p>Thanks<br/> Li Xi</p> <p>in ptlrpcd_stop, cfs_wait_for_completion(&amp;pc-&gt;pc_finishing) will be called to wait the pending RPCs to complete!</p> <p>can the issue be reproduced in your site?</p> <p>Ah, I see. Thank you very much!</p> Development Rank 1|hzvf9j: Rank (Obsolete) 6118 Severity