https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3936 Lustre <p>Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.844231&#93;</span> LustreError: 80225:0:(ldlm_lock.c:1792:ldlm_cancel_stale_locks()) ASSERTION( count &gt; 0 ) failed:<br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.855652&#93;</span> LustreError: 80225:0:(ldlm_lock.c:1792:ldlm_cancel_stale_locks()) LBUG<br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.864399&#93;</span> Pid: 80225, comm: mdt_rdpg_84<br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.869077&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.869078&#93;</span> Call Trace:<br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.873894&#93;</span> <span class="error">&#91;&lt;ffffffffa0467865&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.881895&#93;</span> <span class="error">&#91;&lt;ffffffffa0467e77&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.889058&#93;</span> <span class="error">&#91;&lt;ffffffffa06e8ecd&gt;&#93;</span> ldlm_cancel_stale_locks+0x17d/0x180 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.897439&#93;</span> <span class="error">&#91;&lt;ffffffffa0477884&gt;&#93;</span> ? cfs_hash_dual_bd_unlock+0x34/0x60 <span class="error">&#91;libcfs&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.905854&#93;</span> <span class="error">&#91;&lt;ffffffffa07195ec&gt;&#93;</span> ldlm_srv_pool_shrink+0x7c/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.913795&#93;</span> <span class="error">&#91;&lt;ffffffffa0718b97&gt;&#93;</span> ldlm_pool_shrink+0x37/0xf0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.921254&#93;</span> <span class="error">&#91;&lt;ffffffffa07198f3&gt;&#93;</span> ldlm_pools_shrink+0x283/0x330 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.928995&#93;</span> <span class="error">&#91;&lt;ffffffffa07199d3&gt;&#93;</span> ldlm_pools_srv_shrink+0x13/0x20 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.936901&#93;</span> <span class="error">&#91;&lt;ffffffff81125d5a&gt;&#93;</span> shrink_slab+0x13a/0x1a0<br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.943140&#93;</span> <span class="error">&#91;&lt;ffffffff81128934&gt;&#93;</span> zone_reclaim+0x284/0x410<br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.949469&#93;</span> <span class="error">&#91;&lt;ffffffff81129640&gt;&#93;</span> ? isolate_pages_global+0x0/0x380<br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.956582&#93;</span> <span class="error">&#91;&lt;ffffffff8111ef74&gt;&#93;</span> get_page_from_freelist+0x694/0x820<br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.963887&#93;</span> <span class="error">&#91;&lt;ffffffff8111fe81&gt;&#93;</span> __alloc_pages_nodemask+0x111/0x8b0<br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.971187&#93;</span> <span class="error">&#91;&lt;ffffffff8126c5a9&gt;&#93;</span> ? pointer+0xa9/0x900<br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.977130&#93;</span> <span class="error">&#91;&lt;ffffffffa08cfaea&gt;&#93;</span> ? kiblnd_queue_tx+0x4a/0x60 <span class="error">&#91;ko2iblnd&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.984824&#93;</span> <span class="error">&#91;&lt;ffffffff81159d72&gt;&#93;</span> kmem_getpages+0x62/0x170<br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.991152&#93;</span> <span class="error">&#91;&lt;ffffffff8115a3df&gt;&#93;</span> cache_grow+0x2cf/0x320<br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873893.997287&#93;</span> <span class="error">&#91;&lt;ffffffff8115a632&gt;&#93;</span> cache_alloc_refill+0x202/0x240<br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.004215&#93;</span> <span class="error">&#91;&lt;ffffffffa0468a03&gt;&#93;</span> ? cfs_alloc+0x63/0x90 <span class="error">&#91;libcfs&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.011138&#93;</span> <span class="error">&#91;&lt;ffffffff8115b339&gt;&#93;</span> __kmalloc+0x1b9/0x230<br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.017188&#93;</span> <span class="error">&#91;&lt;ffffffffa0468a03&gt;&#93;</span> cfs_alloc+0x63/0x90 <span class="error">&#91;libcfs&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.023969&#93;</span> <span class="error">&#91;&lt;ffffffffa0772e3f&gt;&#93;</span> null_alloc_rs+0x16f/0x3b0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.031314&#93;</span> <span class="error">&#91;&lt;ffffffffa0760544&gt;&#93;</span> sptlrpc_svc_alloc_rs+0x74/0x2d0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.039237&#93;</span> <span class="error">&#91;&lt;ffffffffa0732393&gt;&#93;</span> lustre_pack_reply_v2+0x93/0x280 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.047158&#93;</span> <span class="error">&#91;&lt;ffffffffa0734f20&gt;&#93;</span> ? lustre_swab_mdt_rec_reint+0x0/0xb0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.055667&#93;</span> <span class="error">&#91;&lt;ffffffffa0732636&gt;&#93;</span> lustre_pack_reply_flags+0xb6/0x210 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.063978&#93;</span> <span class="error">&#91;&lt;ffffffffa07327a1&gt;&#93;</span> lustre_pack_reply+0x11/0x20 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.071527&#93;</span> <span class="error">&#91;&lt;ffffffffa075df93&gt;&#93;</span> req_capsule_server_pack+0x53/0x120 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.079807&#93;</span> <span class="error">&#91;&lt;ffffffffa0c6fecb&gt;&#93;</span> mdt_close+0x10b/0x850 <span class="error">&#91;mdt&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.086494&#93;</span> <span class="error">&#91;&lt;ffffffffa07330ec&gt;&#93;</span> ? lustre_msg_get_version+0x8c/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.094879&#93;</span> <span class="error">&#91;&lt;ffffffffa0c48a02&gt;&#93;</span> mdt_handle_common+0x932/0x1770 <span class="error">&#91;mdt&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.102391&#93;</span> <span class="error">&#91;&lt;ffffffffa0c498f5&gt;&#93;</span> mdt_readpage_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.109838&#93;</span> <span class="error">&#91;&lt;ffffffffa0742b83&gt;&#93;</span> ptlrpc_main+0xf13/0x19e0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.117077&#93;</span> <span class="error">&#91;&lt;ffffffffa0741c70&gt;&#93;</span> ? ptlrpc_main+0x0/0x19e0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.124286&#93;</span> <span class="error">&#91;&lt;ffffffff8100c1ca&gt;&#93;</span> child_rip+0xa/0x20<br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.130075&#93;</span> <span class="error">&#91;&lt;ffffffffa0741c70&gt;&#93;</span> ? ptlrpc_main+0x0/0x19e0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.137310&#93;</span> <span class="error">&#91;&lt;ffffffffa0741c70&gt;&#93;</span> ? ptlrpc_main+0x0/0x19e0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 17 18:18:49 snx11003n003 kernel: <span class="error">&#91;873894.144517&#93;</span> <span class="error">&#91;&lt;ffffffff8100c1c0&gt;&#93;</span> ? child_rip+0x0/0x20</p> LU-3936

ldlm_cancel_stale_locks()) ASSERTION( count > 0 ) failed

Bug Critical Resolved Fixed Dmitry Eremin Andriy Skulysh patch Thu, 12 Sep 2013 07:07:29 +0000 Wed, 20 Nov 2013 09:21:28 +0000 Wed, 20 Nov 2013 09:21:28 +0000 Lustre 2.6.0 0 3 <p>patch: <a href="http://review.whamcloud.com/#/c/7626/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7626/</a></p> <p>I'm not sure this issue is related to 2.5 code. There is no ldlm_cancel_stale_locks() function at all. Could you specify the real version of Lustre which got this assertion please?</p> <p>It was caught on Lustre 2.1, but it doesn't matter because ldlm_pool_shrink() and others are called with negative number of locks to cancel</p> <p>Hmm. Could you provide a reproducer please? I agree the expression "1 + nr_locks * nr / total" can potentially overflow int32 but I try to understand why this cause a crash you referring to.</p> <p>Andriy,<br/> can you please explain further how it is possible for there to be more than 2^31 locks on any node? This would require 2TB of RAM to keep that many locks in memory and other parts of the code would completely explode trying to deal with that many locks. I think the approach of using __u64 for counting locks to be completely bogus. This is clearly a case of some underflow that is caused by a negative number, not an overflow. If the root of the problem is that "nr" is negative in some kernels, then changing to __u64 just means that the node will try to cancel 2^63 locks or something else bad, and not fix the root of the problem.</p> <p>Secondly, there is no ldlm_cancel_stale_locks() that I can find in either master or in 2.1, nor could I find the above LASSERT(count &gt; 0) in some other function. Could you please tell me which specific version of Lustre this is in, or is this in some patch in Gerrit that is not landed yet?</p> <p>I think I was incorrect in approving the original patch for this problem, because I didn't actually look closely enough at this bug when inspecting the code. I can't see how that patch actually fixes any problem.</p> <p>My bad. I see that there is an integer overflow if "nr" is large, so the original patch is not useless.</p> <p>I'm not yet sure what Dmitry's patch <a href="http://review.whamcloud.com/8075" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8075</a> is doing, but we shouldn't close this bug while it is still open.</p> <p>There are no more concerns, therefore I close the ticket.</p> Development Rank 1|hzw20v: Rank (Obsolete) 10409 Severity