https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16253 Lustre <p>This issue was created by maloo for Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/8e9b7081-bc95-4443-a2cb-32f33c6b9f54" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8e9b7081-bc95-4443-a2cb-32f33c6b9f54</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> [18399.419463] LustreError: 358716:0:(nrs_orr.c:481:nrs_trr_hop_exit()) ASSERTION( orro-&gt;oo_ref == 0 ) failed: Busy NRS TRR policy object for OST with index 3, with 1 refs [18399.422350] LustreError: 358716:0:(nrs_orr.c:481:nrs_trr_hop_exit()) LBUG [18399.423651] Pid: 358716, comm: lctl 4.18.0-372.26.1.el8_lustre.x86_64 #1 SMP Wed Oct 5 15:10:35 UTC 2022 [18399.425421] Call Trace TBD: [18399.426173] [&lt;0&gt;] libcfs_call_trace+0x6f/0x90 [libcfs] [18399.427258] [&lt;0&gt;] lbug_with_loc+0x3f/0x70 [libcfs] [18399.428199] [&lt;0&gt;] nrs_trr_hop_exit+0x11c/0x150 [ptlrpc] [18399.429791] [&lt;0&gt;] cfs_hash_putref+0x1c8/0x4b0 [libcfs] [18399.430814] [&lt;0&gt;] nrs_orr_stop+0x65/0x270 [ptlrpc] [18399.431868] [&lt;0&gt;] nrs_policy_stop0+0x38/0x1b0 [ptlrpc] [18399.432985] [&lt;0&gt;] nrs_policy_stop_primary.isra.10+0x181/0x1d0 [ptlrpc] [18399.434361] [&lt;0&gt;] nrs_policy_start_locked+0x467/0x660 [ptlrpc] [18399.435581] [&lt;0&gt;] nrs_policy_ctl+0x203/0x2d0 [ptlrpc] [18399.436678] [&lt;0&gt;] ptlrpc_nrs_policy_control+0x10f/0x2f0 [ptlrpc] [18399.437926] [&lt;0&gt;] ptlrpc_lprocfs_nrs_policies_seq_write+0x473/0x5e0 [ptlrpc] [18399.439365] [&lt;0&gt;] full_proxy_write+0x53/0x80 [18399.440266] [&lt;0&gt;] vfs_write+0xa5/0x1a0 [18399.441031] [&lt;0&gt;] ksys_write+0x4f/0xb0 [18399.441796] [&lt;0&gt;] do_syscall_64+0x5b/0x1a0 </pre> </div></div> LU-16253

sanityn: ASSERTION( orro->oo_ref == 0 ) in 77d

Bug Minor Resolved Fixed Feng Lei Maloo Wed, 19 Oct 2022 17:21:48 +0000 Sun, 20 Aug 2023 13:46:25 +0000 Sun, 20 Aug 2023 13:46:25 +0000 Lustre 2.16.0 0 7 <p>Etienne, is this related to your recently landed patch <a href="https://review.whamcloud.com/48494" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48494</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-16144" title="OST crash at umount in ptlrpc_nrs_req_stop_nolock (with TBF policy)." class="issue-link" data-issue-key="LU-16144"><del>LU-16144</del></a> nrs: implement force mode for nrs_tbf_req_get()</tt>"?</p> <p>Andreas, sorry for my late answer (I missed this comment).<br/> This does not to seem likely, <a href="https://review.whamcloud.com/48494" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48494</a> implements a force mode for TBF it does not change start/stop policies logics. And nrs_orr_req_get() does not implement "force" mode, so it should not be impacted by this patch.<br/> <a href="https://review.whamcloud.com/48523/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48523/</a> "<a href="https://jira.whamcloud.com/browse/LU-14976" title="Changing tbf policy induces high CPU load" class="issue-link" data-issue-key="LU-14976"><del>LU-14976</del></a> nrs: change nrs policies at run time" is more likely to provoke those kinds of crashes.</p> <p><tt>orro-&gt;oo_ref</tt> is a reference count but not an atomic type, so there should be a lock to protect it. Usually it is protected with the hash bucket lock if it is changed from the call path <tt>nrs_trr_hash_ops.hs_get()</tt> or <tt>nrs_trr_hash_ops.hs_put_locked()</tt>. So if it is changed from <tt>cfs_hash_ops.hs_put()</tt>, the function should hold the lock by itself.</p> <p><tt>nrs_trr_hash_ops.hs_put</tt> is filled with <tt>nrs_orr_hop_put()</tt>, which does not hold the bucket lock. It is a race condition.</p> <p>Make sense?</p> <p>"Feng Lei &lt;flei@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51260" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51260</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16253" title="sanityn: ASSERTION( orro-&gt;oo_ref == 0 ) in 77d" class="issue-link" data-issue-key="LU-16253"><del>LU-16253</del></a> ptlrpc: define nrs_orr_object.oo_ref atomic_t<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 28782c745172b65b2992ef1ab751bf79161382a0</p> <p>Patch <a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/40113" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/40113</a> for <a href="https://jira.whamcloud.com/browse/LU-8130" title="Migrate from libcfs hash to rhashtable" class="issue-link" data-issue-key="LU-8130">LU-8130</a> already address this issue.</p> <p>Patch 40113 landed which resolved this problem.</p> Duplicate LU-6849 Related LU-16144 LU-8130 Development Rank 1|i033af: Rank (Obsolete) 9223372036854775807 Severity