https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3680 Lustre <p>Hit this running sanity in a loop:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;4&gt;[80900.195000] ldlm_cn01_000: page allocation failure. order:1, mode:0x40 &lt;4&gt;[80900.195002] Pid: 17587, comm: ldlm_cn01_000 Not tainted 2.6.32-rhe6.4-debug #2 &lt;4&gt;[80900.195003] Call Trace: &lt;4&gt;[80900.195008] [&lt;ffffffff8112a666&gt;] ? __alloc_pages_nodemask+0x7c6/0x980 &lt;4&gt;[80900.195011] [&lt;ffffffff811658f2&gt;] ? kmem_getpages+0x62/0x170 &lt;4&gt;[80900.195013] [&lt;ffffffff8116834a&gt;] ? fallback_alloc+0x1ba/0x270 &lt;4&gt;[80900.195015] [&lt;ffffffff81167bf7&gt;] ? cache_grow+0x4d7/0x520 &lt;4&gt;[80900.195017] [&lt;ffffffff81168038&gt;] ? ____cache_alloc_node+0xa8/0x200 &lt;4&gt;[80900.195018] [&lt;ffffffff81168943&gt;] ? kmem_cache_alloc_trace+0x1c3/0x250 &lt;4&gt;[80900.195029] [&lt;ffffffffa099cbc5&gt;] ? osd_key_init+0x25/0x4e0 [osd_ldiskfs] &lt;4&gt;[80900.195035] [&lt;ffffffffa099cbc5&gt;] ? osd_key_init+0x25/0x4e0 [osd_ldiskfs] &lt;4&gt;[80900.195060] [&lt;ffffffffa0bdd27f&gt;] ? keys_fill+0x6f/0x190 [obdclass] &lt;4&gt;[80900.195090] [&lt;ffffffffa0be132e&gt;] ? lu_context_init+0x4e/0x240 [obdclass] &lt;4&gt;[80900.195109] [&lt;ffffffffa0be1383&gt;] ? lu_context_init+0xa3/0x240 [obdclass] &lt;4&gt;[80900.195111] [&lt;ffffffff811665be&gt;] ? cache_free_debugcheck+0x2ae/0x360 &lt;4&gt;[80900.195130] [&lt;ffffffffa0be153e&gt;] ? lu_env_init+0x1e/0x30 [obdclass] &lt;4&gt;[80900.195140] [&lt;ffffffffa0e3d69a&gt;] ? ofd_lvbo_update+0x7a/0xea8 [ofd] &lt;4&gt;[80900.195164] [&lt;ffffffffa04ac434&gt;] ? ldlm_resource_putref+0x1d4/0x280 [ptlrpc] &lt;4&gt;[80900.195186] [&lt;ffffffffa04c97b7&gt;] ? ldlm_request_cancel+0x247/0x410 [ptlrpc] &lt;4&gt;[80900.195206] [&lt;ffffffffa04c9abd&gt;] ? ldlm_handle_cancel+0x13d/0x240 [ptlrpc] &lt;4&gt;[80900.195226] [&lt;ffffffffa04cefb9&gt;] ? ldlm_cancel_handler+0x1e9/0x500 [ptlrpc] &lt;4&gt;[80900.195250] [&lt;ffffffffa04ffad1&gt;] ? ptlrpc_server_handle_request+0x3b1/0xc70 [ptlrpc] &lt;4&gt;[80900.195260] [&lt;ffffffffa0a2355e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] &lt;4&gt;[80900.195270] [&lt;ffffffffa0a34b6f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] &lt;4&gt;[80900.195340] [&lt;ffffffffa04f6bb1&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] &lt;4&gt;[80900.195345] [&lt;ffffffff81054613&gt;] ? __wake_up+0x53/0x70 &lt;4&gt;[80900.195367] [&lt;ffffffffa0500db2&gt;] ? ptlrpc_main+0xa22/0x1650 [ptlrpc] &lt;4&gt;[80900.195437] [&lt;ffffffffa0500390&gt;] ? ptlrpc_main+0x0/0x1650 [ptlrpc] &lt;4&gt;[80900.195441] [&lt;ffffffff81094606&gt;] ? kthread+0x96/0xa0 &lt;4&gt;[80900.195444] [&lt;ffffffff8100c10a&gt;] ? child_rip+0xa/0x20 &lt;4&gt;[80900.195447] [&lt;ffffffff81094570&gt;] ? kthread+0x0/0xa0 &lt;4&gt;[80900.195448] [&lt;ffffffff8100c100&gt;] ? child_rip+0x0/0x20 &lt;6&gt;[80900.195449] Mem-Info: &lt;4&gt;[80900.195450] Node 0 DMA per-cpu: &lt;4&gt;[80900.195451] CPU 0: hi: 0, btch: 1 usd: 0 &lt;4&gt;[80900.195452] CPU 1: hi: 0, btch: 1 usd: 0 &lt;4&gt;[80900.195453] CPU 2: hi: 0, btch: 1 usd: 0 &lt;4&gt;[80900.195454] CPU 3: hi: 0, btch: 1 usd: 0 &lt;4&gt;[80900.195455] CPU 4: hi: 0, btch: 1 usd: 0 &lt;4&gt;[80900.195456] CPU 5: hi: 0, btch: 1 usd: 0 &lt;4&gt;[80900.195458] CPU 6: hi: 0, btch: 1 usd: 0 &lt;4&gt;[80900.195459] CPU 7: hi: 0, btch: 1 usd: 0 &lt;4&gt;[80900.195459] Node 0 DMA32 per-cpu: &lt;4&gt;[80900.195460] CPU 0: hi: 186, btch: 31 usd: 51 &lt;4&gt;[80900.195461] CPU 1: hi: 186, btch: 31 usd: 26 &lt;4&gt;[80900.195462] CPU 2: hi: 186, btch: 31 usd: 0 &lt;4&gt;[80900.195463] CPU 3: hi: 186, btch: 31 usd: 0 &lt;4&gt;[80900.195464] CPU 4: hi: 186, btch: 31 usd: 57 &lt;4&gt;[80900.195465] CPU 5: hi: 186, btch: 31 usd: 174 &lt;4&gt;[80900.195466] CPU 6: hi: 186, btch: 31 usd: 162 &lt;4&gt;[80900.195467] CPU 7: hi: 186, btch: 31 usd: 32 &lt;4&gt;[80900.195470] active_anon:61548 inactive_anon:61459 isolated_anon:0 &lt;4&gt;[80900.195470] active_file:94797 inactive_file:74222 isolated_file:0 &lt;4&gt;[80900.195471] unevictable:0 dirty:20 writeback:0 unstable:0 &lt;4&gt;[80900.195471] free:43025 slab_reclaimable:75111 slab_unreclaimable:271092 &lt;4&gt;[80900.195472] mapped:577 shmem:119300 pagetables:383 bounce:0 &lt;4&gt;[80900.195473] Node 0 DMA free:9692kB min:136kB low:168kB high:204kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:9296kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes &lt;4&gt;[80900.195478] lowmem_reserve[]: 0 2967 2967 2967 &lt;4&gt;[80900.195479] Node 0 DMA32 free:162408kB min:44916kB low:56144kB high:67372kB active_anon:246192kB inactive_anon:245836kB active_file:379188kB inactive_file:296888kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3039076kB mlocked:0kB dirty:80kB writeback:0kB mapped:2308kB shmem:477200kB slab_reclaimable:300444kB slab_unreclaimable:1084368kB kernel_stack:3296kB pagetables:1532kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no &lt;4&gt;[80900.195485] lowmem_reserve[]: 0 0 0 0 &lt;4&gt;[80900.195486] Node 0 DMA: 3*4kB 0*8kB 3*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9692kB &lt;4&gt;[80900.195490] Node 0 DMA32: 37378*4kB 1032*8kB 32*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 162408kB &lt;4&gt;[80900.195495] 129723 total pagecache pages &lt;4&gt;[80900.195496] 925 pages in swap cache &lt;4&gt;[80900.195497] Swap cache stats: add 376365, delete 375440, find 1323456/1328202 &lt;4&gt;[80900.195498] Free swap = 1869104kB &lt;4&gt;[80900.195499] Total swap = 2097144kB &lt;6&gt;[80900.198861] 774396 pages RAM &lt;6&gt;[80900.198861] 38583 pages reserved &lt;6&gt;[80900.198861] 11942 pages shared &lt;6&gt;[80900.198861] 675636 pages non-shared &lt;4&gt;[80900.226747] 129650 total pagecache pages &lt;4&gt;[80900.226747] 1136 pages in swap cache &lt;4&gt;[80900.226747] Swap cache stats: add 376650, delete 375514, find 1323456/1328202 &lt;4&gt;[80900.226747] Free swap = 1867964kB &lt;4&gt;[80900.226747] Total swap = 2097144kB &lt;6&gt;[80900.226747] 774396 pages RAM &lt;6&gt;[80900.226747] 38583 pages reserved &lt;6&gt;[80900.226747] 11963 pages shared &lt;6&gt;[80900.226747] 668761 pages non-shared &lt;0&gt;[80900.502883] LustreError: 17604:0:(sec_null.c:318:null_alloc_rs()) ASSERTION( rs-&gt;rs_size &gt;= rs_size ) failed: &lt;0&gt;[80900.504111] LustreError: 17604:0:(sec_null.c:318:null_alloc_rs()) LBUG &lt;4&gt;[80900.504782] Pid: 17604, comm: mdt01_002 &lt;4&gt;[80900.505352] &lt;4&gt;[80900.505353] Call Trace: &lt;4&gt;[80900.506312] [&lt;ffffffffa0a228a5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt;[80900.507011] [&lt;ffffffffa0a22ea7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt;[80900.507716] [&lt;ffffffffa052a382&gt;] null_alloc_rs+0x272/0x390 [ptlrpc] &lt;4&gt;[80900.508419] [&lt;ffffffffa0518f19&gt;] sptlrpc_svc_alloc_rs+0x1d9/0x2a0 [ptlrpc] &lt;4&gt;[80900.509166] [&lt;ffffffffa04ef218&gt;] lustre_pack_reply_v2+0x98/0x2a0 [ptlrpc] &lt;4&gt;[80900.509906] [&lt;ffffffffa04ef4ce&gt;] lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] &lt;4&gt;[80900.510935] [&lt;ffffffffa04ef621&gt;] lustre_pack_reply+0x11/0x20 [ptlrpc] &lt;4&gt;[80900.511642] [&lt;ffffffffa0516603&gt;] req_capsule_server_pack+0x53/0x100 [ptlrpc] &lt;4&gt;[80900.513248] [&lt;ffffffffa0d472e5&gt;] mdt_getxattr+0x585/0x13c0 [mdt] &lt;4&gt;[80900.514017] [&lt;ffffffffa0d2570e&gt;] mdt_intent_getxattr+0x9e/0x160 [mdt] &lt;4&gt;[80900.514572] [&lt;ffffffffa0d2265e&gt;] mdt_intent_policy+0x3ae/0x770 [mdt] &lt;4&gt;[80900.515391] [&lt;ffffffffa04a735a&gt;] ldlm_lock_enqueue+0x2ea/0x860 [ptlrpc] &lt;4&gt;[80900.516099] [&lt;ffffffffa04cfc7f&gt;] ldlm_handle_enqueue0+0x4ef/0x10c0 [ptlrpc] &lt;4&gt;[80900.518542] [&lt;ffffffffa0d22b26&gt;] mdt_enqueue+0x46/0xe0 [mdt] &lt;4&gt;[80900.519098] [&lt;ffffffffa0d28ca7&gt;] mdt_handle_common+0x647/0x16d0 [mdt] &lt;4&gt;[80900.519529] [&lt;ffffffffa0d63335&gt;] mds_regular_handle+0x15/0x20 [mdt] &lt;4&gt;[80900.519959] [&lt;ffffffffa04ffad1&gt;] ptlrpc_server_handle_request+0x3b1/0xc70 [ptlrpc] &lt;4&gt;[80900.520740] [&lt;ffffffffa0a2355e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] &lt;4&gt;[80900.521190] [&lt;ffffffffa0a34b6f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] &lt;4&gt;[80900.521644] [&lt;ffffffffa04f6bb1&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] &lt;4&gt;[80900.522146] [&lt;ffffffff81054613&gt;] ? __wake_up+0x53/0x70 &lt;4&gt;[80900.522670] [&lt;ffffffffa0500db2&gt;] ptlrpc_main+0xa22/0x1650 [ptlrpc] &lt;4&gt;[80900.523097] [&lt;ffffffffa0500390&gt;] ? ptlrpc_main+0x0/0x1650 [ptlrpc] &lt;4&gt;[80900.523532] [&lt;ffffffff81094606&gt;] kthread+0x96/0xa0 &lt;4&gt;[80900.526690] [&lt;ffffffff8100c10a&gt;] child_rip+0xa/0x20 &lt;4&gt;[80900.527224] [&lt;ffffffff81094570&gt;] ? kthread+0x0/0xa0 &lt;4&gt;[80900.527603] [&lt;ffffffff8100c100&gt;] ? child_rip+0x0/0x20 &lt;4&gt;[80900.528010] &lt;0&gt;[80900.528950] Kernel panic - not syncing: LBUG </pre> </div></div> LU-3680

OOM crash: null_alloc_rs()) ASSERTION( rs->rs_size >= rs_size ) failed

Bug Major Resolved Fixed Oleg Drokin Oleg Drokin Thu, 1 Aug 2013 01:25:57 +0000 Wed, 16 Mar 2016 05:22:47 +0000 Sun, 19 Jan 2014 04:09:44 +0000 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.5.1 Lustre 2.4.3 Lustre 2.8.0 0 13 <p>At the very minimum we need to fix the error handling for this allocation.</p> <p>Part of the issue is we never check the return state of the call and seem to just pass it along. </p> <p>This svc-&gt;srv_ops.so_req_handler(request) should be failing with ENOMEM, but we don't check and just call ptlrpc_rqphase_move like everything is fine. <br/> from ptlrpc_server_handle_request</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> rc = svc-&gt;srv_ops.so_req_handler(request); ptlrpc_rqphase_move(request, RQ_PHASE_COMPLETE); put_conn: </pre> </div></div> <p>The only mention of rc to the end of the function is an log message. I will make a patch to spurn conversation it seems we only return 0 or 1 out of this function. </p> <p>Hmm I am not sure these are 2 backtraces are 100% related as they are from different threads but there is an odd things I see on the lbug error call chain that confuses me a bit. </p> <p>In: </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>int lustre_pack_reply_v2(struct ptlrpc_request *req, int count, __u32 *lens, char **bufs, int flags) { struct ptlrpc_reply_state *rs; int msg_len, rc; ENTRY; LASSERT(req-&gt;rq_reply_state == NULL); &lt;==== LASSERT rq_reply_state is NULL if ((flags &amp; LPRFL_EARLY_REPLY) == 0) { spin_lock(&amp;req-&gt;rq_lock); req-&gt;rq_packed_final = 1; spin_unlock(&amp;req-&gt;rq_lock); } msg_len = lustre_msg_size_v2(count, lens); rc = sptlrpc_svc_alloc_rs(req, msg_len); &lt;==== pass that same req on. </pre> </div></div> <p>sptlrpc_svc_alloc_rs does a little safety check and calls this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static int null_alloc_rs(struct ptlrpc_request *req, int msgsize) { struct ptlrpc_reply_state *rs; int rs_size = sizeof(*rs) + msgsize; LASSERT(msgsize % 8 == 0); rs = req-&gt;rq_reply_state; &lt;=== LASSERTED as NULL just a little while ago. if (rs) { /* pre-allocated */ LASSERT(rs-&gt;rs_size &gt;= rs_size); &lt;==== we hit this and I am confused as to why. </pre> </div></div> <p>Is some separate thread acting on this request at the same time? </p> <p>You don't happen to have the a Lustre Debug log hanging around do you Oleg? </p> <p>I'm more concerned about why running sanity.sh in a loop is running out of memory?</p> <p>We hit the same LBUG on MDS too. Maybe service-&gt;srv_max_reply_size is not big enough to hold the reply message? When memory is under pressure and the buffer is returned by lustre_get_emerg_rs, this LBUG happens.</p> <p>here's a debug log from recent instance.</p> <p>The crash info is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;4&gt;[19273.725502] mdt01_000: page allocation failure. order:2, mode:0x40 &lt;4&gt;[19273.726180] Pid: 14141, comm: mdt01_000 Not tainted 2.6.32-rhe6.4-debug #2 &lt;4&gt;[19273.726873] Call Trace: &lt;4&gt;[19273.727380] [&lt;ffffffff8112a666&gt;] ? __alloc_pages_nodemask+0x7c6/0x980 &lt;4&gt;[19273.728063] [&lt;ffffffff8127f456&gt;] ? vsnprintf+0x336/0x5e0 &lt;4&gt;[19273.728697] [&lt;ffffffff811658f2&gt;] ? kmem_getpages+0x62/0x170 &lt;4&gt;[19273.729334] [&lt;ffffffff8116834a&gt;] ? fallback_alloc+0x1ba/0x270 &lt;4&gt;[19273.729334] [&lt;ffffffff81167bf7&gt;] ? cache_grow+0x4d7/0x520 &lt;4&gt;[19273.729334] [&lt;ffffffff81168038&gt;] ? ____cache_alloc_node+0xa8/0x200 &lt;4&gt;[19273.729334] [&lt;ffffffff811686e8&gt;] ? __kmalloc+0x208/0x2a0 &lt;4&gt;[19273.729334] [&lt;ffffffffa118d515&gt;] ? null_alloc_rs+0xc5/0x390 [ptlrpc] &lt;4&gt;[19273.729334] [&lt;ffffffffa118d515&gt;] ? null_alloc_rs+0xc5/0x390 [ptlrpc] &lt;4&gt;[19273.729334] [&lt;ffffffffa117c484&gt;] ? sptlrpc_svc_alloc_rs+0x74/0x2a0 [ptlrpc] &lt;4&gt;[19273.729334] [&lt;ffffffffa1153678&gt;] ? lustre_pack_reply_v2+0x98/0x2a0 [ptlrpc] &lt;4&gt;[19273.729334] [&lt;ffffffffa115392e&gt;] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] &lt;4&gt;[19273.738776] [&lt;ffffffffa1153a81&gt;] ? lustre_pack_reply+0x11/0x20 [ptlrpc] &lt;4&gt;[19273.738776] [&lt;ffffffffa1179c93&gt;] ? req_capsule_server_pack+0x53/0x100 [ptlrpc] &lt;4&gt;[19273.738776] [&lt;ffffffffa05a6765&gt;] ? mdt_getxattr+0x545/0x1490 [mdt] &lt;4&gt;[19273.738776] [&lt;ffffffffa05847be&gt;] ? mdt_intent_getxattr+0x9e/0x160 [mdt] &lt;4&gt;[19273.738776] [&lt;ffffffffa0581e2e&gt;] ? mdt_intent_policy+0x3ae/0x770 [mdt] &lt;4&gt;[19273.738776] [&lt;ffffffffa110b34a&gt;] ? ldlm_lock_enqueue+0x2ea/0x860 [ptlrpc] &lt;4&gt;[19273.738776] [&lt;ffffffffa1133c6f&gt;] ? ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] &lt;4&gt;[19273.738776] [&lt;ffffffffa05822f6&gt;] ? mdt_enqueue+0x46/0xe0 [mdt] &lt;4&gt;[19273.738776] [&lt;ffffffffa058897a&gt;] ? mdt_handle_common+0x52a/0x1470 [mdt] &lt;4&gt;[19273.738776] [&lt;ffffffffa05c2005&gt;] ? mds_regular_handle+0x15/0x20 [mdt] &lt;4&gt;[19273.738776] [&lt;ffffffffa11633f5&gt;] ? ptlrpc_server_handle_request+0x395/0xc20 [ptlrpc] &lt;4&gt;[19273.738776] [&lt;ffffffffa0eca35f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] &lt;4&gt;[19273.738776] [&lt;ffffffffa115ad61&gt;] ? ptlrpc_wait_event+0xc1/0x2e0 [ptlrpc] &lt;4&gt;[19273.738776] [&lt;ffffffffa11646da&gt;] ? ptlrpc_main+0xa5a/0x1690 [ptlrpc] &lt;4&gt;[19273.738776] [&lt;ffffffffa1163c80&gt;] ? ptlrpc_main+0x0/0x1690 [ptlrpc] &lt;4&gt;[19273.738776] [&lt;ffffffff81094606&gt;] ? kthread+0x96/0xa0 &lt;4&gt;[19273.738776] [&lt;ffffffff8100c10a&gt;] ? child_rip+0xa/0x20 &lt;4&gt;[19273.738776] [&lt;ffffffff81094570&gt;] ? kthread+0x0/0xa0 &lt;4&gt;[19273.738776] [&lt;ffffffff8100c100&gt;] ? child_rip+0x0/0x20 &lt;6&gt;[19273.738776] Mem-Info: &lt;4&gt;[19273.738776] Node 0 DMA per-cpu: &lt;4&gt;[19273.738776] CPU 0: hi: 0, btch: 1 usd: 0 &lt;4&gt;[19273.738776] CPU 1: hi: 0, btch: 1 usd: 0 &lt;4&gt;[19273.738776] CPU 2: hi: 0, btch: 1 usd: 0 &lt;4&gt;[19273.738776] CPU 3: hi: 0, btch: 1 usd: 0 &lt;4&gt;[19273.738776] CPU 4: hi: 0, btch: 1 usd: 0 &lt;4&gt;[19273.738776] CPU 5: hi: 0, btch: 1 usd: 0 &lt;4&gt;[19273.738776] CPU 6: hi: 0, btch: 1 usd: 0 &lt;4&gt;[19273.738776] CPU 7: hi: 0, btch: 1 usd: 0 &lt;4&gt;[19273.738776] Node 0 DMA32 per-cpu: &lt;4&gt;[19273.738776] CPU 0: hi: 186, btch: 31 usd: 33 &lt;4&gt;[19273.738776] CPU 1: hi: 186, btch: 31 usd: 172 &lt;4&gt;[19273.738776] CPU 2: hi: 186, btch: 31 usd: 147 &lt;4&gt;[19273.738776] CPU 3: hi: 186, btch: 31 usd: 154 &lt;4&gt;[19273.738776] CPU 4: hi: 186, btch: 31 usd: 126 &lt;4&gt;[19273.738776] CPU 5: hi: 186, btch: 31 usd: 33 &lt;4&gt;[19273.738776] CPU 6: hi: 186, btch: 31 usd: 155 &lt;4&gt;[19273.738776] CPU 7: hi: 186, btch: 31 usd: 123 &lt;4&gt;[19273.738776] active_anon:96976 inactive_anon:80133 isolated_anon:0 &lt;4&gt;[19273.738776] active_file:49462 inactive_file:38723 isolated_file:0 &lt;4&gt;[19273.738776] unevictable:0 dirty:1065 writeback:0 unstable:0 &lt;4&gt;[19273.738776] free:25017 slab_reclaimable:77546 slab_unreclaimable:312838 &lt;4&gt;[19273.738776] mapped:625 shmem:174432 pagetables:389 bounce:0 &lt;4&gt;[19273.738776] Node 0 DMA free:9692kB min:136kB low:168kB high:204kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:9296kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes &lt;4&gt;[19273.738776] lowmem_reserve[]: 0 2967 2967 2967 &lt;4&gt;[19273.738776] Node 0 DMA32 free:91616kB min:44916kB low:56144kB high:67372kB active_anon:387904kB inactive_anon:320532kB active_file:196476kB inactive_file:154892kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3039076kB mlocked:0kB dirty:4260kB writeback:0kB mapped:2500kB shmem:697728kB slab_reclaimable:310184kB slab_unreclaimable:1251744kB kernel_stack:3304kB pagetables:1556kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no &lt;4&gt;[19273.738776] lowmem_reserve[]: 0 0 0 0 &lt;4&gt;[19273.738776] Node 0 DMA: 3*4kB 0*8kB 3*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9692kB &lt;4&gt;[19273.738776] Node 0 DMA32: 10341*4kB 4432*8kB 23*16kB 19*32kB 10*64kB 2*128kB 6*256kB 2*512kB 2*1024kB 2*2048kB 1*4096kB = 91492kB &lt;4&gt;[19273.738776] 232691 total pagecache pages &lt;4&gt;[19273.738776] 53 pages in swap cache &lt;4&gt;[19273.738776] Swap cache stats: add 1502, delete 1449, find 18/20 &lt;4&gt;[19273.738776] Free swap = 2091200kB &lt;4&gt;[19273.738776] Total swap = 2097144kB &lt;6&gt;[19273.738776] 774396 pages RAM &lt;6&gt;[19273.738776] 38583 pages reserved &lt;6&gt;[19273.738776] 15526 pages shared &lt;6&gt;[19273.738776] 689850 pages non-shared &lt;0&gt;[19273.940246] LustreError: 14141:0:(sec_null.c:318:null_alloc_rs()) ASSERTION( rs-&gt;rs_size &gt;= rs_size ) failed: &lt;0&gt;[19273.941315] LustreError: 14141:0:(sec_null.c:318:null_alloc_rs()) LBUG &lt;4&gt;[19273.941915] Pid: 14141, comm: mdt01_000 &lt;4&gt;[19273.942396] &lt;4&gt;[19273.942396] Call Trace: &lt;4&gt;[19273.945170] [&lt;ffffffffa0eb88a5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt;[19273.945793] [&lt;ffffffffa0eb8ea7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt;[19273.950476] [&lt;ffffffffa118d6c2&gt;] null_alloc_rs+0x272/0x390 [ptlrpc] &lt;4&gt;[19273.951103] [&lt;ffffffffa117c5e9&gt;] sptlrpc_svc_alloc_rs+0x1d9/0x2a0 [ptlrpc] &lt;4&gt;[19273.955634] [&lt;ffffffffa1153678&gt;] lustre_pack_reply_v2+0x98/0x2a0 [ptlrpc] &lt;4&gt;[19273.956346] [&lt;ffffffffa115392e&gt;] lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] &lt;4&gt;[19273.957072] [&lt;ffffffffa1153a81&gt;] lustre_pack_reply+0x11/0x20 [ptlrpc] &lt;4&gt;[19273.957780] [&lt;ffffffffa1179c93&gt;] req_capsule_server_pack+0x53/0x100 [ptlrpc] &lt;4&gt;[19273.958495] [&lt;ffffffffa05a6765&gt;] mdt_getxattr+0x545/0x1490 [mdt] &lt;4&gt;[19273.970919] [&lt;ffffffffa05847be&gt;] mdt_intent_getxattr+0x9e/0x160 [mdt] &lt;4&gt;[19273.971619] [&lt;ffffffffa0581e2e&gt;] mdt_intent_policy+0x3ae/0x770 [mdt] &lt;4&gt;[19273.972322] [&lt;ffffffffa110b34a&gt;] ldlm_lock_enqueue+0x2ea/0x860 [ptlrpc] &lt;4&gt;[19273.973035] [&lt;ffffffffa1133c6f&gt;] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] &lt;4&gt;[19273.973881] [&lt;ffffffffa05822f6&gt;] mdt_enqueue+0x46/0xe0 [mdt] &lt;4&gt;[19273.995450] [&lt;ffffffffa058897a&gt;] mdt_handle_common+0x52a/0x1470 [mdt] &lt;4&gt;[19273.996181] [&lt;ffffffffa05c2005&gt;] mds_regular_handle+0x15/0x20 [mdt] &lt;4&gt;[19274.008857] [&lt;ffffffffa11633f5&gt;] ptlrpc_server_handle_request+0x395/0xc20 [ptlrpc] &lt;4&gt;[19274.011971] [&lt;ffffffffa0eca35f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] &lt;4&gt;[19274.012692] [&lt;ffffffffa115ad61&gt;] ? ptlrpc_wait_event+0xc1/0x2e0 [ptlrpc] &lt;4&gt;[19274.018425] [&lt;ffffffffa11646da&gt;] ptlrpc_main+0xa5a/0x1690 [ptlrpc] &lt;4&gt;[19274.019129] [&lt;ffffffffa1163c80&gt;] ? ptlrpc_main+0x0/0x1690 [ptlrpc] &lt;4&gt;[19274.019808] [&lt;ffffffff81094606&gt;] kthread+0x96/0xa0 &lt;4&gt;[19274.020415] [&lt;ffffffff8100c10a&gt;] child_rip+0xa/0x20 &lt;4&gt;[19274.021025] [&lt;ffffffff81094570&gt;] ? kthread+0x0/0xa0 &lt;4&gt;[19274.021639] [&lt;ffffffff8100c100&gt;] ? child_rip+0x0/0x20 &lt;4&gt;[19274.022265] &lt;0&gt;[19274.095286] Kernel panic - not syncing: LBUG </pre> </div></div> <p>I also have a crashdump.</p> <p>Hit this twice in the last few days on an MDS running master from Nov. 1st during testing for another bug. I've got dumps of both events if they'd be of any use. (Logging was set to default levels)</p> <p>The stack traces look the same for both of them, here's one:<br/> &lt;0&gt;LustreError: 2252:0:(sec_null.c:318:null_alloc_rs()) ASSERTION( rs-&gt;rs_size &gt;= rs_size ) failed:<br/> &lt;0&gt;LustreError: 2252:0:(sec_null.c:318:null_alloc_rs()) LBUG<br/> &lt;4&gt;Pid: 2252, comm: mdt00_001<br/> &lt;4&gt;<br/> &lt;4&gt;Call Trace:<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa032d895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa032de97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0627722&gt;&#93;</span> null_alloc_rs+0x272/0x390 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06161e9&gt;&#93;</span> sptlrpc_svc_alloc_rs+0x1d9/0x2a0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05ecd03&gt;&#93;</span> lustre_pack_reply_v2+0x93/0x280 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05ecf9e&gt;&#93;</span> lustre_pack_reply_flags+0xae/0x1f0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05ed0f1&gt;&#93;</span> lustre_pack_reply+0x11/0x20 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06138b3&gt;&#93;</span> req_capsule_server_pack+0x53/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c3eea5&gt;&#93;</span> mdt_getxattr+0x545/0x1490 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c1c5ee&gt;&#93;</span> mdt_intent_getxattr+0x9e/0x160 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c198ce&gt;&#93;</span> mdt_intent_policy+0x3ae/0x770 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05a4461&gt;&#93;</span> ldlm_lock_enqueue+0x361/0x8c0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05cd17f&gt;&#93;</span> ldlm_handle_enqueue0+0x4ef/0x10a0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c19d96&gt;&#93;</span> mdt_enqueue+0x46/0xe0 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c20a8a&gt;&#93;</span> mdt_handle_common+0x52a/0x1470 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c5ac55&gt;&#93;</span> mds_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05fce25&gt;&#93;</span> ptlrpc_server_handle_request+0x385/0xc00 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa032e4ce&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa033f27f&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x170 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05f44c9&gt;&#93;</span> ? ptlrpc_wait_event+0xa9/0x2d0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff81051439&gt;&#93;</span> ? __wake_up_common+0x59/0x90<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05fe18d&gt;&#93;</span> ptlrpc_main+0xaed/0x1740 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05fd6a0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1740 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff81096a36&gt;&#93;</span> kthread+0x96/0xa0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810969a0&gt;&#93;</span> ? kthread+0x0/0xa0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>Just occurred to me I should mention: I can reproduce this fairly consistently. If requested, I can try to do this with debugging turned up.</p> <p>Patrick. </p> <p> What is the memory usage like when you hit this error?</p> <p>Keith,</p> <p>Very high - Memory pressure, definitely. Not actually OOM, but as low as it can go (At the point where the kernel starts freeing memory and I see memory usage bounce around in the tens of MB free.).</p> <p>I've been working around it (still testing my other, unrelated bug) by adding more RAM to the virtual machine running the MDS and also dumping caches periodically to keep from going OOM. That's worked for me.</p> <p>Low memory is a tricky space in Lustre. Are you seeing other errors or just this one? </p> <p>This is the only one I've seen on the MDS. However, I've seen this one at least three times, and can reproduce it readily, so it may be obscuring others.</p> <p>On the client, we're chasing a recurrence of <a href="https://jira.whamcloud.com/browse/LU-3027" title="Failure on test suite parallel-scale test_write_disjoint: invalid file size 140329 instead of 160376 = 20047 * 8" class="issue-link" data-issue-key="LU-3027"><del>LU-3027</del></a>/<a href="https://jira.whamcloud.com/browse/LU-3889" title=" LBUG: (osc_lock.c:497:osc_lock_upcall()) ASSERTION( lock-&gt;cll_state &gt;= CLS_QUEUING ) " class="issue-link" data-issue-key="LU-3889"><del>LU-3889</del></a>, in which low memory can also play a role, but that's the client, not the MDS. (Oddly, the client testing for THAT was what led to our low memory issues on the MDS, but that's mostly due to an underspecced MDS in the testing system.)</p> <p>Hi Patrick,</p> <p>Would you please apply following patch and try to reproduce the problem again? I am not able to reproduce this consistently.<br/> <a href="http://review.whamcloud.com/#/c/8200/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8200/</a></p> <p>Thank you very much!</p> <p>Li,</p> <p>Sure. I'm doing that now - I've just tested to confirm I can still hit the problem, now I'm going to test the patch in the same situation.</p> <p>Li,</p> <p>Still hit the same crash with that patch in place. Stack trace is the same as before. I can make the dump available if you'd like. (Does Intel have an FTP site I could use? Otherwise I can find another way to make it available)</p> <p><span class="error">&#91;Edit&#93;</span><br/> It occurred to me it's past midnight in China, so I've uploaded the dump rather than potentially making you wait another day if you do need it.</p> <p>It's at:<br/> ftp.us.cray.com<br/> Username:<br/> anonymous<br/> Password:<br/> anonymous</p> <p>The file is at:<br/> outbound/<a href="https://jira.whamcloud.com/browse/LU-3680" title="OOM crash: null_alloc_rs()) ASSERTION( rs-&gt;rs_size &gt;= rs_size ) failed" class="issue-link" data-issue-key="LU-3680"><del>LU-3680</del></a>/<a href="https://jira.whamcloud.com/browse/LU-3680" title="OOM crash: null_alloc_rs()) ASSERTION( rs-&gt;rs_size &gt;= rs_size ) failed" class="issue-link" data-issue-key="LU-3680"><del>LU-3680</del></a>-mds-dump.tar.gz</p> <p>You cannot ls in outbound or <a href="https://jira.whamcloud.com/browse/LU-3680" title="OOM crash: null_alloc_rs()) ASSERTION( rs-&gt;rs_size &gt;= rs_size ) failed" class="issue-link" data-issue-key="LU-3680"><del>LU-3680</del></a>, but you should be able to grab the dump.</p> <p>Hi Patrick,</p> <p>Thank you very much for your test. It seems that the problem is complexer than I thought. I will investigate more.</p> <p>We can share the debug files by uploading it to <a href="ftp://ftp.whamcloud.com/uploads/LU-*" class="external-link" target="_blank" rel="nofollow noopener">ftp://ftp.whamcloud.com/uploads/LU-*</a>.</p> <p>Hi Patrick,</p> <p>It seems that there is a problem in lustre_get_emerg_rs(). The size of the buffer is set to zero, which will cause this LBUG when the memeory is under high pressure. I've updated the patch. Would you please test whether it works? Thank you very much!</p> <p>Li,</p> <p>Sure. I'll probably be testing this tomorrow. I expect you've noticed this, but you've got a small mistake in the current patch that's causing it not to build on RHEL6:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-comment">/* Just <span class="code-keyword">return</span> failure <span class="code-keyword">if</span> the size is too big */</span> CERROR(<span class="code-quote">"size of message is too big (%lu), %d allowed"</span>, msglen + sizeof(struct ptlrpc_reply_state), </pre> </div></div> <p>You're printing %lu, but msglen + sizeof(<span class="error">&#91;...&#93;</span>) is an int. I'd just do a cast to a long unsigned int, I suppose.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">(unsigned <span class="code-object">long</span>) (msglen + sizeof(struct ptlrpc_reply_state)), </pre> </div></div> <p>Should do it.</p> <ul class="alternate" type="square"> <li>Patrick</li> </ul> <p>Li,</p> <p>Just testing the FTP site...<br/> I'm connected to ftp.whamcloud.com as anonymous, but I can't create files in /upload, and I can't seem to cd to any LU directories. I tried <a href="https://jira.whamcloud.com/browse/LU-3680" title="OOM crash: null_alloc_rs()) ASSERTION( rs-&gt;rs_size &gt;= rs_size ) failed" class="issue-link" data-issue-key="LU-3680"><del>LU-3680</del></a>, <a href="https://jira.whamcloud.com/browse/LU-3027" title="Failure on test suite parallel-scale test_write_disjoint: invalid file size 140329 instead of 160376 = 20047 * 8" class="issue-link" data-issue-key="LU-3027"><del>LU-3027</del></a>, <a href="https://jira.whamcloud.com/browse/LU-4152" title=" layout locks can cause deadlock" class="issue-link" data-issue-key="LU-4152"><del>LU-4152</del></a> and a few others. Also can't make a new directory.</p> <p>Is there something I've missed here?</p> <ul class="alternate" type="square"> <li>Patrick</li> </ul> <p>Hi Patrick,</p> <p>I've just test the FTP. It works well. Following are the commands.</p> <p>lixitekiMacBook-Pro:~ lixi$ ftp ftp.whamcloud.com<br/> Connected to eric.whamcloud.com.<br/> 220 (vsFTPd 2.2.2)<br/> Name (ftp.whamcloud.com:lixi): anonymous<br/> 331 Please specify the password.<br/> Password: <br/> 230 Login successful.<br/> Remote system type is UNIX.<br/> Using binary mode to transfer files.<br/> ftp&gt; ls<br/> 229 Entering Extended Passive Mode (|||38409|).<br/> 150 Here comes the directory listing.<br/> d-wxrwx--- 68 123 840000001 4096 Nov 05 08:12 uploads<br/> 226 Directory send OK.<br/> ftp&gt; cd uploads<br/> 250 Directory successfully changed.<br/> ftp&gt; mkdir <a href="https://jira.whamcloud.com/browse/LU-3680" title="OOM crash: null_alloc_rs()) ASSERTION( rs-&gt;rs_size &gt;= rs_size ) failed" class="issue-link" data-issue-key="LU-3680"><del>LU-3680</del></a><br/> 257 "/uploads/<a href="https://jira.whamcloud.com/browse/LU-3680" title="OOM crash: null_alloc_rs()) ASSERTION( rs-&gt;rs_size &gt;= rs_size ) failed" class="issue-link" data-issue-key="LU-3680"><del>LU-3680</del></a>" created<br/> ftp&gt; cd <a href="https://jira.whamcloud.com/browse/LU-3680" title="OOM crash: null_alloc_rs()) ASSERTION( rs-&gt;rs_size &gt;= rs_size ) failed" class="issue-link" data-issue-key="LU-3680"><del>LU-3680</del></a><br/> 250 Directory successfully changed.<br/> ftp&gt; lpwd<br/> Local directory: /Users/lixi<br/> ftp&gt; put test.rtf <br/> local: test.rtf remote: test.rtf<br/> 229 Entering Extended Passive Mode (|||53942|).<br/> 150 Ok to send data.<br/> 100% |***********************************| 315 397.43 KiB/s 00:00 ETA<br/> 226 Transfer complete.<br/> 315 bytes sent in 00:00 (0.49 KiB/s)<br/> ftp&gt; ls<br/> 229 Entering Extended Passive Mode (|||40673|).<br/> 150 Here comes the directory listing.<br/> <del>rw-r</del><del>r</del>- 1 123 114 315 Nov 07 18:57 test.rtf<br/> 226 Directory send OK.</p> <p>Li,</p> <p>You're right - I was using a Windows FTP command line client, and now that I've switched to a Unix one, everything is fine.</p> <p>Thanks!</p> <p>Li,</p> <p>Different LBUG, same function:<br/> &lt;0&gt;LustreError: 12155:0:(sec_null.c:318:null_alloc_rs()) ASSERTION( rs-&gt;rs_size &gt;= rs_size ) failed:<br/> &lt;0&gt;LustreError: 12155:0:(sec_null.c:318:null_alloc_rs()) LBUG<br/> &lt;4&gt;Pid: 12155, comm: mdt00_000<br/> &lt;4&gt;<br/> &lt;4&gt;Call Trace:<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa032d895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa032de97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa06296a2&gt;&#93;</span> null_alloc_rs+0x272/0x390 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0617527&gt;&#93;</span> sptlrpc_svc_alloc_rs+0x1e7/0x350 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05edfb3&gt;&#93;</span> lustre_pack_reply_v2+0x93/0x280 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05ee24e&gt;&#93;</span> lustre_pack_reply_flags+0xae/0x1f0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05ee3a1&gt;&#93;</span> lustre_pack_reply+0x11/0x20 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0614be3&gt;&#93;</span> req_capsule_server_pack+0x53/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c3ef65&gt;&#93;</span> mdt_getxattr+0x545/0x1490 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c212ae&gt;&#93;</span> mdt_intent_getxattr+0x9e/0x160 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa046a356&gt;&#93;</span> ? lu_object_find+0x16/0x20 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0c1b659&gt;&#93;</span> mdt_intent_policy+0x499/0xca0 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05a5441&gt;&#93;</span> ldlm_lock_enqueue+0x361/0x8c0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05ce17f&gt;&#93;</span> ldlm_handle_enqueue0+0x4ef/0x10a0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0643b12&gt;&#93;</span> tgt_enqueue+0x62/0x1d0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa064203f&gt;&#93;</span> tgt_request_handle+0x5ff/0x1200 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05ef2bc&gt;&#93;</span> ? lustre_msg_get_transno+0x8c/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05fe0d5&gt;&#93;</span> ptlrpc_server_handle_request+0x385/0xc00 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa032e4ce&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa033f3df&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x170 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05f5779&gt;&#93;</span> ? ptlrpc_wait_event+0xa9/0x2d0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff81051439&gt;&#93;</span> ? __wake_up_common+0x59/0x90<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05ff43d&gt;&#93;</span> ptlrpc_main+0xaed/0x1740 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa05fe950&gt;&#93;</span> ? ptlrpc_main+0x0/0x1740 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff81096a36&gt;&#93;</span> kthread+0x96/0xa0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810969a0&gt;&#93;</span> ? kthread+0x0/0xa0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>Dump is going up on the WC FTP site momentarily.</p> <p>Dump is in /<a href="https://jira.whamcloud.com/browse/LU-3680" title="OOM crash: null_alloc_rs()) ASSERTION( rs-&gt;rs_size &gt;= rs_size ) failed" class="issue-link" data-issue-key="LU-3680"><del>LU-3680</del></a>, file is named:<br/> lu-3680-mds-dump 2.tar.gz</p> <p>Patrick,</p> <p>Thanks for you test. I am sorry that this LBUG happended again. You said that this is a different LBUG, but the trace looks the same to me. Is there anything I missed?<br/> It surprises me that the patch does not help at all. I though it fixes a problem and might eliminate the LBUG. Would you please doule check that the patch is applied properly?<br/> Since the problem happens when the memory is under heavy pressure, I think the buffer is likely to be allocaed by lustre_get_emerg_rs(). And that's why I am surpised that the patch didn't help...</p> <p>Thanks!</p> <p>Li,</p> <p>I'm sorry, you're right - I misread my own dump. I'm fairly sure I had the patch in place, but I'm happy to try again. It's always possible I made a mistake in setting things up.</p> <p>I should be able to test that tomorrow.</p> <p>By the way, if you're interested in trying to reproduce this, I'm just running this script from a client to generate activity:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">for</span> idx in $(seq 0 10000); <span class="code-keyword">do</span> time ls -laR &gt; /dev/<span class="code-keyword">null</span> touch somefile rm -f somefiles echo $idx: $(date +%T) $(grep MemFree /proc/meminfo) done </pre> </div></div> <p>Then running this code on the MDS to create memory pressure (you have to hold down enter, it consumes real memory much more slowly than the rate at which it's allocating memory). Once memory gets low, just keep running this code and you should see the bug within a minute or so:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#include &lt;stdio.h&gt; #include &lt;unistd.h&gt; <span class="code-object">int</span> main() { <span class="code-object">int</span> i; <span class="code-object">char</span>* junk; start: i = 0; <span class="code-keyword">while</span>(i &lt; 50) { printf(<span class="code-quote">"Malloc!\n"</span>); junk = malloc(1024*1024*1024); junk[0] = i; i++; } printf(<span class="code-quote">"Mallocced 50 GB. Press enter to malloc another 50.\n"</span>); printf(<span class="code-quote">"Note: This seems to use roughly 10 MB of real memory each time.\n"</span>); getchar(); <span class="code-keyword">goto</span> start; } </pre> </div></div> <p>Li,</p> <p>It looks like you're right - I must have made a mistake in my build/install process before. With your patch, I am no longer able to hit this bug.</p> <ul class="alternate" type="square"> <li>Patrick</li> </ul> <p>Patrick,</p> <p>Thank you very much for confirming this! I am glad that we've made some progress finally.</p> <p>Li Xi</p> <p>Re-pushed patch to get Maloo to test again. If tests pass, I'm planning to invite Oleg to review.</p> <p>Patch landed to master for 2.6.0, and to b2_4 for 2.4.3 (should it ever happen) and b2_5 for 2.5.1.</p> <p>Shows mode:0x40 == __GFP_IO, but missing __GFP_WAIT from <a href="https://jira.whamcloud.com/browse/LU-4357" title="page allocation failure. mode:0x40 caused by missing __GFP_WAIT flag" class="issue-link" data-issue-key="LU-4357"><del>LU-4357</del></a>.</p> <p><a href="http://review.whamcloud.com/#/c/9726/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9726/</a> - osd_thread_info fits 4K</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/9726/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9726/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-3680" title="OOM crash: null_alloc_rs()) ASSERTION( rs-&gt;rs_size &gt;= rs_size ) failed" class="issue-link" data-issue-key="LU-3680"><del>LU-3680</del></a> osd: reduce osd_thread_info in ldiskfs osd<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 56875fde8c66a4a937b173ccb065a9a6a3c67419</p> Duplicate LU-4357 LU-6472 Related LU-6324 Development Rank 1|hzvwyf: Rank (Obsolete) 9502 Severity