https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2790 Lustre <p>After recent landings of <a href="https://jira.whamcloud.com/browse/LU-1431" title="Support for larger than 1MB sequential I/O RPCs" class="issue-link" data-issue-key="LU-1431"><del>LU-1431</del></a> patch amnd corresponding crop up of allocation failures due to <a href="https://jira.whamcloud.com/browse/LU-2748" title="OSD uses kmalloc with high order to allocate a keys" class="issue-link" data-issue-key="LU-2748"><del>LU-2748</del></a>...</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[25420.342529] ll_ost00_005: page allocation failure. order:5, mode:0x50 [25420.342845] Pid: 22594, comm: ll_ost00_005 Not tainted 2.6.32-debug #6 [25420.343134] Call Trace: [25420.343356] [&lt;ffffffff81125bd6&gt;] ? __alloc_pages_nodemask+0x976/0x9e0 [25420.343652] [&lt;ffffffff81160a62&gt;] ? kmem_getpages+0x62/0x170 [25420.344062] [&lt;ffffffff8116349c&gt;] ? fallback_alloc+0x1bc/0x270 [25420.344431] [&lt;ffffffff81162db7&gt;] ? cache_grow+0x4d7/0x520 [25420.344748] [&lt;ffffffff81163188&gt;] ? ____cache_alloc_node+0xa8/0x200 [25420.345035] [&lt;ffffffff81163838&gt;] ? __kmalloc+0x208/0x2a0 [25420.345319] [&lt;ffffffffa09efc00&gt;] ? cfs_alloc+0x30/0x60 [libcfs] [25420.345614] [&lt;ffffffffa09efc00&gt;] ? cfs_alloc+0x30/0x60 [libcfs] [25420.345899] [&lt;ffffffffa048953e&gt;] ? osd_key_init+0x1e/0x5d0 [osd_ldiskfs] [25420.346231] [&lt;ffffffffa0eae3df&gt;] ? keys_fill+0x6f/0x190 [obdclass] [25420.346534] [&lt;ffffffffa0eb1e8b&gt;] ? lu_context_init+0xab/0x260 [obdclass] [25420.346842] [&lt;ffffffffa0eb205e&gt;] ? lu_env_init+0x1e/0x30 [obdclass] [25420.347134] [&lt;ffffffffa05bc90c&gt;] ? ost_blocking_ast+0x5c/0xca0 [ost] [25420.347443] [&lt;ffffffffa10ebded&gt;] ? ldlm_work_bl_ast_lock+0xdd/0x290 [ptlrpc] [25420.347770] [&lt;ffffffffa112c18f&gt;] ? ptlrpc_set_wait+0x6f/0x880 [ptlrpc] [25420.348102] [&lt;ffffffff81090154&gt;] ? __init_waitqueue_head+0x24/0x40 [25420.348548] [&lt;ffffffffa09ef8a5&gt;] ? cfs_waitq_init+0x15/0x20 [libcfs] [25420.348977] [&lt;ffffffffa112876e&gt;] ? ptlrpc_prep_set+0x11e/0x300 [ptlrpc] [25420.349293] [&lt;ffffffffa10ebd10&gt;] ? ldlm_work_bl_ast_lock+0x0/0x290 [ptlrpc] [25420.349796] [&lt;ffffffffa10ee19b&gt;] ? ldlm_run_ast_work+0x1db/0x460 [ptlrpc] [25420.350126] [&lt;ffffffffa110580f&gt;] ? ldlm_process_extent_lock+0x1af/0xa90 [ptlrpc] [25420.350606] [&lt;ffffffffa10ee7b4&gt;] ? ldlm_lock_enqueue+0x394/0x870 [ptlrpc] [25420.350923] [&lt;ffffffffa1114e87&gt;] ? ldlm_handle_enqueue0+0x4f7/0x1090 [ptlrpc] [25420.351417] [&lt;ffffffffa1115a86&gt;] ? ldlm_handle_enqueue+0x66/0x70 [ptlrpc] [25420.351749] [&lt;ffffffffa1115a90&gt;] ? ldlm_server_completion_ast+0x0/0x640 [ptlrpc] [25420.352248] [&lt;ffffffffa05bc8b0&gt;] ? ost_blocking_ast+0x0/0xca0 [ost] [25420.352574] [&lt;ffffffffa11123c0&gt;] ? ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] [25420.353124] [&lt;ffffffffa05c4807&gt;] ? ost_handle+0x1be7/0x4590 [ost] [25420.353543] [&lt;ffffffffa09fb204&gt;] ? libcfs_id2str+0x74/0xb0 [libcfs] [25420.353945] [&lt;ffffffffa1144e03&gt;] ? ptlrpc_server_handle_request+0x453/0xe50 [ptlrpc] [25420.354432] [&lt;ffffffffa09ef65e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [25420.354741] [&lt;ffffffffa113de91&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] [25420.355023] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [25420.355299] [&lt;ffffffffa11478cd&gt;] ? ptlrpc_main+0xafd/0x17f0 [ptlrpc] [25420.355606] [&lt;ffffffffa1146dd0&gt;] ? ptlrpc_main+0x0/0x17f0 [ptlrpc] [25420.355890] [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 [25420.356188] [&lt;ffffffffa1146dd0&gt;] ? ptlrpc_main+0x0/0x17f0 [ptlrpc] [25420.356495] [&lt;ffffffffa1146dd0&gt;] ? ptlrpc_main+0x0/0x17f0 [ptlrpc] [25420.356787] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 .... [25420.500609] LustreError: 22594:0:(ldlm_resource.c:1161:ldlm_resource_get()) lvbo_init failed for resource 114: rc -12 [25420.502383] LustreError: 18292:0:(ldlm_lock.c:1542:ldlm_fill_lvb()) ### Replied unexpected ost LVB size 0 ns: lustre-OST0000-osc-ffff88003f9d2bf0 lock: ffff880046658db0/0xd15dff8dc7742d63 lrc: 6/0,2 mode: --/PW res: 114/8589935616 rrc: 1 type: EXT [0-&gt;18446744073709551615] (req 0-&gt;4095) flags: 0x0 nid: local remote: 0xd15dff8dc77438ca expref: -99 pid: 20430 timeout: 0 lvb_type: 1 ... [25420.604693] LustreError: 22594:0:(ldlm_resource.c:1161:ldlm_resource_get()) lvbo_init failed for resource 116: rc -12 [25420.604777] LustreError: 18293:0:(ldlm_lock.c:1542:ldlm_fill_lvb()) ### Replied unexpected ost LVB size 0 ns: lustre-OST0000-osc-ffff880054e39bf0 lock: ffff880084978db0/0xd15dff8dc7744183 lrc: 6/0,2 mode: --/PW res: 116/8589935616 rrc: 1 type: EXT [0-&gt;18446744073709551615] (req 0-&gt;4095) flags: 0x0 nid: local remote: 0xd15dff8dc77442a9 expref: -99 pid: 20443 timeout: 0 lvb_type: 1 [25420.620838] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [25420.621142] [&lt;ffffffffa11478cd&gt;] ? ptlrpc_main+0xafd/0x17f0 [ptlrpc] [25420.621445] [&lt;ffffffffa1146dd0&gt;] ? ptlrpc_main+0x0/0x17f0 [ptlrpc] [25420.621760] [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 [25420.622044] [&lt;ffffffffa1146dd0&gt;] ? ptlrpc_main+0x0/0x17f0 [ptlrpc] [25420.622339] [&lt;ffffffffa1146dd0&gt;] ? ptlrpc_main+0x0/0x17f0 [ptlrpc] [25420.622648] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 ... [25420.702106] LustreError: 22594:0:(ofd_dlm.c:177:ofd_intent_policy()) ASSERTION( res_lvb != ((void *)0) ) failed: [25420.702490] LustreError: 22594:0:(ofd_dlm.c:177:ofd_intent_policy()) LBUG [25420.702705] Pid: 22594, comm: ll_ost00_005 [25420.702853] [25420.702853] Call Trace: [25420.703112] [&lt;ffffffffa09ee915&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [25420.703314] [&lt;ffffffffa09eef17&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [25420.703492] [&lt;ffffffffa0697c85&gt;] ofd_intent_policy+0x795/0x7c0 [ofd] [25420.703712] [&lt;ffffffffa10ee70a&gt;] ldlm_lock_enqueue+0x2ea/0x870 [ptlrpc] [25420.703906] [&lt;ffffffffa1114e87&gt;] ldlm_handle_enqueue0+0x4f7/0x1090 [ptlrpc] [25420.704121] [&lt;ffffffffa1115a86&gt;] ldlm_handle_enqueue+0x66/0x70 [ptlrpc] [25420.704332] [&lt;ffffffffa1115a90&gt;] ? ldlm_server_completion_ast+0x0/0x640 [ptlrpc] [25420.704667] [&lt;ffffffffa05bc8b0&gt;] ? ost_blocking_ast+0x0/0xca0 [ost] [25420.704926] [&lt;ffffffffa11123c0&gt;] ? ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc] [25420.705271] [&lt;ffffffffa05c4807&gt;] ost_handle+0x1be7/0x4590 [ost] [25420.705511] [&lt;ffffffffa09fb204&gt;] ? libcfs_id2str+0x74/0xb0 [libcfs] [25420.705715] [&lt;ffffffffa1144e03&gt;] ptlrpc_server_handle_request+0x453/0xe50 [ptlrpc] [25420.706015] [&lt;ffffffffa09ef65e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [25420.706213] [&lt;ffffffffa113de91&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] [25420.706397] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [25420.706585] [&lt;ffffffffa11478cd&gt;] ptlrpc_main+0xafd/0x17f0 [ptlrpc] [25420.706775] [&lt;ffffffffa1146dd0&gt;] ? ptlrpc_main+0x0/0x17f0 [ptlrpc] [25420.706956] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [25420.707232] [&lt;ffffffffa1146dd0&gt;] ? ptlrpc_main+0x0/0x17f0 [ptlrpc] [25420.707441] [&lt;ffffffffa1146dd0&gt;] ? ptlrpc_main+0x0/0x17f0 [ptlrpc] [25420.707634] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 [25420.707805] [25420.708112] Kernel panic - not syncing: LBUG </pre> </div></div> LU-2790

Failure to allocated osd keys leads to ofd_intent_policy()) ASSERTION( res_lvb != ((void *)0) ) failed

Bug Major Resolved Fixed nasf Oleg Drokin Sun, 10 Feb 2013 13:04:53 +0000 Sun, 31 Mar 2013 21:50:01 +0000 Sun, 31 Mar 2013 21:50:01 +0000 Lustre 2.4.0 Lustre 2.4.0 0 8 <p>probably it makes sense to allocate I/O-related data (like dr_pages, etc), separately and on-demand.</p> <p>The 2.1.4&lt;-&gt;2.4.0 interop testing is affected by the issue in this ticket. Please refer to the failure in <a href="https://jira.whamcloud.com/browse/LU-2867" title="2.1.4&lt;-&gt;2.4.0 interop: parallel-scale test_compilebench: IOError: [Errno 71] Protocol error" class="issue-link" data-issue-key="LU-2867"><del>LU-2867</del></a>.</p> <p>I think this will be solved by <a href="http://review.whamcloud.com/#change,5444" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5444</a></p> <p>Thanks Alex! Yu Jian could you please test this patch to confirm whether this is indeed the case?</p> <blockquote><p>Yu Jian could you please test this patch to confirm whether this is indeed the case?</p></blockquote> <p>Sure, I created <a href="http://review.whamcloud.com/5647" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5647</a> to test patch with Lustre b2_1 client. Let's wait for the test result.</p> <blockquote><p>Sure, I created <a href="http://review.whamcloud.com/5647" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5647</a> to test patch with Lustre b2_1 client. Let's wait for the test result.</p></blockquote> <p>The parallel-scale test passed: <a href="https://maloo.whamcloud.com/test_sessions/a8135b98-8a1e-11e2-b891-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/a8135b98-8a1e-11e2-b891-52540035b04c</a></p> <p>Closing as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-2748" title="OSD uses kmalloc with high order to allocate a keys" class="issue-link" data-issue-key="LU-2748"><del>LU-2748</del></a></p> <p>I disagree with Alex' assessment.</p> <p><a href="https://jira.whamcloud.com/browse/LU-2748" title="OSD uses kmalloc with high order to allocate a keys" class="issue-link" data-issue-key="LU-2748"><del>LU-2748</del></a> only masked the symptoms here by making the original allocation more robust.<br/> But should it fail for other reasons, this bug will still occur.</p> <p>this code was taken directly from obdfilter (which has the same assert) and it never was a problem. that said i don't mean the code is absolutely correct, but I don't think this will be a problem with ofd.</p> <p>The failure occurred in ofd_lvbo_init() as following:</p> <p>==========================<br/> OBD_ALLOC_PTR(lvb);<br/> if (lvb == NULL)<br/> GOTO(out, rc = -ENOMEM);<br/> ==========================</p> <p>The needed size for the LVB is just 56 bytes, very small.</p> <p>Well, now that hte failed caller is exposed, we just need to fix the caller to do something more sensible.</p> <p>But this is not a huge priority because it's not expected to really fit ever.</p> <p>This is the patch to handle lvbo_init() failure:</p> <p><a href="http://review.whamcloud.com/#change,5699" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5699</a></p> <p>Per discussions with Oleg, reducing priority to major.</p> <p>Landed for 2.4</p> Duplicate LU-2867 Related LU-1431 LU-2748 LU-2748 Development Rank 1|hzvixb: Rank (Obsolete) 6756 Severity