https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8211 Lustre <p>enabling tbf then disabling without creating any rules triggers LBUG.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 9676:0:(nrs_tbf.c:89:nrs_tbf_rule_fini()) ASSERTION( list_empty(&amp;rule-&gt;tr_cli_list) ) failed: ^M LustreError: 9676:0:(nrs_tbf.c:89:nrs_tbf_rule_fini()) LBUG^M Pid: 9676, comm: lctl^M ^M Call Trace:^M [&lt;ffffffffa040e895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs]^M [&lt;ffffffffa040ee97&gt;] lbug_with_loc+0x47/0xb0 [libcfs]^M </pre> </div></div> 2.7.1-fe LU-8211

LustreError: 9676:0:(nrs_tbf.c:89:nrs_tbf_rule_fini()) ASSERTION( list_empty(&rule->tr_cli_list) ) failed: ^M

Bug Major Resolved Incomplete Emoly Liu Mahmoud Hanafi Fri, 27 May 2016 18:47:41 +0000 Thu, 22 Sep 2016 21:45:29 +0000 Thu, 22 Sep 2016 21:45:29 +0000 Lustre 2.7.0 0 7 <p>Emoly</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>I have tries the following commands:<br/> <span class="error">&#91;root@QYJ tests&#93;</span># lctl set_param ost.OSS.ost_io.nrs_policies="tbf nid"<br/> ost.OSS.ost_io.nrs_policies=tbf nid<br/> <span class="error">&#91;root@QYJ tests&#93;</span># lctl set_param ost.OSS.ost_io.nrs_policies="fifo"<br/> ost.OSS.ost_io.nrs_policies=fifo<br/> <span class="error">&#91;root@QYJ tests&#93;</span># lctl set_param ost.OSS.ost_io.nrs_policies="tbf nid"<br/> ost.OSS.ost_io.nrs_policies=tbf nid<br/> <span class="error">&#91;root@QYJ tests&#93;</span># lctl set_param ost.OSS.ost_io.nrs_tbf_rule="stop A"<br/> but it didn't trigger andy panic failure, could you please post your operation commands in details?</p> <p>Thanks,<br/> Qian</p> <p>Hi Mahmoud,</p> <p>Is following patch merged in your branch?</p> <p><a href="https://jira.whamcloud.com/browse/LU-6939" title="nrs_tbf.c:155:nrs_tbf_cli_reset()) ASSER TION( cli-&gt;tc_rule == ((void *)0) ) failed" class="issue-link" data-issue-key="LU-6939"><del>LU-6939</del></a> nrs: add lock to protect TBF rule linkage</p> <p>Mahmoud, as Qian&amp;LiXi said, could you please provide more information to show how to reproduce this LBUG and check your branch? Thanks.</p> <p>Hi Li Xi, we do not have the <a href="https://jira.whamcloud.com/browse/LU-6939" title="nrs_tbf.c:155:nrs_tbf_cli_reset()) ASSER TION( cli-&gt;tc_rule == ((void *)0) ) failed" class="issue-link" data-issue-key="LU-6939"><del>LU-6939</del></a> patch cherry-picked to our tree. We rebase to b2_7_fe periodically, but <a href="https://jira.whamcloud.com/browse/LU-6939" title="nrs_tbf.c:155:nrs_tbf_cli_reset()) ASSER TION( cli-&gt;tc_rule == ((void *)0) ) failed" class="issue-link" data-issue-key="LU-6939"><del>LU-6939</del></a> has not landed to b2_7_fe yet.</p> <p>Our git repo, since b2_7_fe is private, our nas-2.7.1 repo is a private repo as well:<br/> <a href="https://github.com/NASAEarthExchange/lustre-nas-fe/commits/nas-2.7.1" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/NASAEarthExchange/lustre-nas-fe/commits/nas-2.7.1</a></p> <p>Either you give me your github ID and I can add yours to the member list or you can use Peter Jones' ID to access the tree.</p> <p>Is there a way for us to get a complete list of landed NRS/TBF patches not in 2.7.1-fe? I don't want to keep opening LU on landed patches. For example I just hit where the server locks up. Backtrace showed this.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">PID: 32437 TASK: ffff881032c35520 CPU: 3 COMMAND: <span class="code-quote">"ldlm_cn02_023"</span> #0 [ffff88085c426e90] crash_nmi_callback at ffffffff81032256 #1 [ffff88085c426ea0] notifier_call_chain at ffffffff81568515 #2 [ffff88085c426ee0] atomic_notifier_call_chain at ffffffff8156857a #3 [ffff88085c426ef0] notify_die at ffffffff810a44fe #4 [ffff88085c426f20] do_nmi at ffffffff8156618f #5 [ffff88085c426f50] nmi at ffffffff815659f0 [exception RIP: _spin_lock+30] RIP: ffffffff8156525e RSP: ffff88085c423e78 RFLAGS: 00000097 RAX: 0000000000004603 RBX: ffff880f5ea080e0 RCX: 0000000000000000 RDX: 0000000000004602 RSI: ffff88085c430658 RDI: ffff880f5ea080e0 RBP: ffff88085c423e78 R8: 0000000000000000 R9: 0000000000000001 R10: 000000000000002c R11: 00000000000000b4 R12: ffff880f5ea08000 R13: ffff88085c430600 R14: ffff88085c423f28 R15: ffffffffa07e4b50 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- &lt;NMI exception stack&gt; --- #6 [ffff88085c423e78] _spin_lock at ffffffff8156525e #7 [ffff88085c423e80] nrs_tbf_timer_cb at ffffffffa07e4b7a [ptlrpc] #8 [ffff88085c423ea0] __run_hrtimer at ffffffff810a29ae #9 [ffff88085c423ef0] hrtimer_interrupt at ffffffff810a2d46 #10 [ffff88085c423f70] local_apic_timer_interrupt at ffffffff8103433d #11 [ffff88085c423f90] smp_apic_timer_interrupt at ffffffff8156ae25 #12 [ffff88085c423fb0] apic_timer_interrupt at ffffffff8100bc13 --- &lt;IRQ stack&gt; --- #13 [ffff880d87207c68] apic_timer_interrupt at ffffffff8100bc13 [exception RIP: nrs_resource_get_safe+88] RIP: ffffffffa07d7358 RSP: ffff880d87207d10 RFLAGS: 00000206 RAX: 0000000000004602 RBX: ffff880d87207d40 RCX: 0000000000000000 RDX: 0000000000004602 RSI: ffff880efb387778 RDI: ffff880fd2033140 RBP: ffffffff8100bc0e R8: ffff880f5ea080e0 R9: 0000000000000000 R10: 000000000000002c R11: 00000000000000b4 R12: ffff880d87207c90 R13: ffff880e4c2fc3d8 R14: ffffffffa086b640 R15: ffff881000000007 ORIG_RAX: ffffffffffffff10 CS: 0010 SS: 0018 #14 [ffff880d87207d48] ptlrpc_nrs_req_initialize at ffffffffa07d9dab [ptlrpc] #15 [ffff880d87207d68] ptlrpc_server_handle_req_in at ffffffffa079c647 [ptlrpc] #16 [ffff880d87207da8] ptlrpc_main at ffffffffa07a484c [ptlrpc] #17 [ffff880d87207ee8] kthread at ffffffff8109dc8e #18 [ffff880d87207f48] kernel_thread at ffffffff8100c28a PID: 32405 TASK: ffff880f2c04eab0 CPU: 5 COMMAND: <span class="code-quote">"ldlm_cn02_004"</span> #0 [ffff88085c446e90] crash_nmi_callback at ffffffff81032256 #1 [ffff88085c446ea0] notifier_call_chain at ffffffff81568515 #2 [ffff88085c446ee0] atomic_notifier_call_chain at ffffffff8156857a #3 [ffff88085c446ef0] notify_die at ffffffff810a44fe #4 [ffff88085c446f20] do_nmi at ffffffff8156618f #5 [ffff88085c446f50] nmi at ffffffff815659f0 [exception RIP: _spin_lock+33] RIP: ffffffff81565261 RSP: ffff880af029dd00 RFLAGS: 00000293 RAX: 0000000000004605 RBX: ffff880efb387478 RCX: 0000000000000000 RDX: 0000000000004602 RSI: ffff880efb387478 RDI: ffff880f5ea080e0 RBP: ffff880af029dd00 R8: ffff880f5ea080e0 R9: 0000000000000000 R10: 0000000000000031 R11: 00000000000000b4 R12: ffff880efb387478 R13: ffff880eb3ae53c0 R14: ffff880f5ea080e0 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- &lt;NMI exception stack&gt; --- #6 [ffff880af029dd00] _spin_lock at ffffffff81565261 #7 [ffff880af029dd08] nrs_resource_get_safe at ffffffffa07d7341 [ptlrpc] #8 [ffff880af029dd48] ptlrpc_nrs_req_initialize at ffffffffa07d9dab [ptlrpc] #9 [ffff880af029dd68] ptlrpc_server_handle_req_in at ffffffffa079c647 [ptlrpc] #10 [ffff880af029dda8] ptlrpc_main at ffffffffa07a484c [ptlrpc] #11 [ffff880af029dee8] kthread at ffffffff8109dc8e </pre> </div></div> <p>Hi, Mahmoud</p> <p>The dump stack looks like following issue. And it has an patch which has already been merged into master branch. Would you please check whether your branch has merged this patch?<br/> <a href="https://jira.hpdd.intel.com/browse/LU-5717" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-5717</a></p> <p>Also, please check whether following patches are merged too:</p> <p><a href="https://jira.whamcloud.com/browse/LU-6939" title="nrs_tbf.c:155:nrs_tbf_cli_reset()) ASSER TION( cli-&gt;tc_rule == ((void *)0) ) failed" class="issue-link" data-issue-key="LU-6939"><del>LU-6939</del></a> nrs: add lock to protect TBF rule linkage<br/> <a href="https://jira.whamcloud.com/browse/LU-5717" title="Dead lock of nrs_tbf_timer_cb" class="issue-link" data-issue-key="LU-5717"><del>LU-5717</del></a> ptlrpc: fix deadlock problem of nrs_tbf_timer_cb (this problem)<br/> <a href="https://jira.whamcloud.com/browse/LU-6921" title="sanityn 77f test failed Lustre: DEBUG MARKER: sanityn test_77f: @@@@@@ FAIL: failed to operate on TBF rules" class="issue-link" data-issue-key="LU-6921"><del>LU-6921</del></a> test: failed to operate on TBF rules (only test bug)<br/> <a href="https://jira.whamcloud.com/browse/LU-5580" title="Switch between &#39;JOBID&#39; and &#39;NID&#39; directly in NRS TBF" class="issue-link" data-issue-key="LU-5580"><del>LU-5580</del></a> ptlrpc: policy switch directly in tbf (feature improvement)<br/> <a href="https://jira.whamcloud.com/browse/LU-5379" title="Get error when has many rules in nrs tbf policy" class="issue-link" data-issue-key="LU-5379"><del>LU-5379</del></a> ptlprc: return 0 if buf in struct seq_file is overflow (when too many rules are defined, accessing nrs_tbf_rule would return error)<br/> <a href="https://jira.whamcloud.com/browse/LU-5320" title="Use after free, double lock found by coccinelle" class="issue-link" data-issue-key="LU-5320"><del>LU-5320</del></a>: fixes for errors found by coccinelle (An lock error of TBF, which might cause deadlock)</p> <p>And following patches are still under review, but worth to be merged for improvement:<br/> <a href="https://jira.whamcloud.com/browse/LU-7470" title="Extend TBF policy with NID/JobID expressions" class="issue-link" data-issue-key="LU-7470"><del>LU-7470</del></a> nrs: extend TBF with NID/JobID expression<br/> <a href="https://jira.whamcloud.com/browse/LU-8006" title="Specify ordering of TBF policy rules" class="issue-link" data-issue-key="LU-8006"><del>LU-8006</del></a> ptlrpc: specify ordering of TBF policy rules<br/> <a href="https://jira.whamcloud.com/browse/LU-8006" title="Specify ordering of TBF policy rules" class="issue-link" data-issue-key="LU-8006"><del>LU-8006</del></a> ptlrpc: cleanup codes of TBF command</p> <p>And also, if client side QoS is useful for your use case, please check:<br/> <a href="https://jira.hpdd.intel.com/browse/LU-7982" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-7982</a></p> <p>Hi Jay,</p> <p>Sorry for missing you message. My Github ID is ddn-lixi. Would you please add me to the group? Thanks!</p> <p>Our production systems are running 2.7.1-4.1nasS. That build does not have most of the NRS TBF patches.</p> <p>Our next images on deck to be installed is 2.7.1-5nasS. It has all landed patches except <a href="https://jira.whamcloud.com/browse/LU-6939" title="nrs_tbf.c:155:nrs_tbf_cli_reset()) ASSER TION( cli-&gt;tc_rule == ((void *)0) ) failed" class="issue-link" data-issue-key="LU-6939"><del>LU-6939</del></a>, but none of the unlanded.</p> <p>Our 2.7.2-1nas images for testing also includes <a href="https://jira.whamcloud.com/browse/LU-6939" title="nrs_tbf.c:155:nrs_tbf_cli_reset()) ASSER TION( cli-&gt;tc_rule == ((void *)0) ) failed" class="issue-link" data-issue-key="LU-6939"><del>LU-6939</del></a>, but none of the unlanded.</p> <p>Hi Li Xi,</p> <p>I am sure you has access privilege to b2_7_fe, but I just want to double check before I add you to our lustre-nas-fe repo access list. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Can be closed</p> Development End date Mon, 6 Jun 2016 18:47:41 +0000 Rank 1|hzyd0f: Rank (Obsolete) 9223372036854775807 Severity Start date Fri, 27 May 2016 18:47:41 +0000