https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5282 Lustre <p>In ll_close_inode_openhandle() if mdc_close() is not called or if it fails to clear mod-&gt;mod_open_req-&gt;rq_replay then we have the following LBUG:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 2759.567725] LustreError: 23179:0:(mdc_request.c:791:mdc_free_open()) ASSERTION( mod-&gt;m\ od_open_req-&gt;rq_replay == 0 ) failed: [ 2759.568559] LustreError: 23203:0:(mdc_request.c:791:mdc_free_open()) ASSERTION( mod-&gt;m\ od_open_req-&gt;rq_replay == 0 ) failed: [ 2759.568563] LustreError: 23203:0:(mdc_request.c:791:mdc_free_open()) LBUG [ 2759.568564] Pid: 23203, comm: ls [ 2759.568566] [ 2759.568566] Call Trace: [ 2759.568584] [&lt;ffffffffa0ef18c5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [ 2759.568594] [&lt;ffffffffa0ef1ec7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [ 2759.568601] [&lt;ffffffffa055ce60&gt;] mdc_clear_open_replay_data+0x0/0x240 [mdc] [ 2759.568607] [&lt;ffffffffa055ceb8&gt;] mdc_clear_open_replay_data+0x58/0x240 [mdc] [ 2759.568630] [&lt;ffffffffa0523fce&gt;] lmv_clear_open_replay_data+0x20e/0x6f0 [lmv] [ 2759.568655] [&lt;ffffffffa0b13375&gt;] ll_close_inode_openhandle+0x485/0xfe0 [lustre] [ 2759.568670] [&lt;ffffffffa0b1463f&gt;] ll_md_real_close+0x1af/0x220 [lustre] [ 2759.568685] [&lt;ffffffffa0b14d15&gt;] ll_file_release+0x665/0xb90 [lustre] [ 2759.568698] [&lt;ffffffffa0afbaab&gt;] ll_dir_release+0xdb/0xf0 [lustre] [ 2759.568703] [&lt;ffffffff811a4318&gt;] __fput+0x108/0x230 [ 2759.568706] [&lt;ffffffff811a4465&gt;] fput+0x25/0x30 [ 2759.568708] [&lt;ffffffff8119f64d&gt;] filp_close+0x5d/0x90 </pre> </div></div> <p>This easiest way to see it is to inject a failure to allocate op_data in ll_close_inode_openhandle(). But early returns from lmv_close() or mdc_close() will likely trigger the same assertion.</p> LU-5282

(mdc_request.c:791:mdc_free_open()) ASSERTION( mod->mod_open_req->rq_replay == 0 ) failed

Bug Minor Resolved Fixed WC Triage John Hammond llite malloc mdc patch Tue, 1 Jul 2014 22:19:08 +0000 Fri, 4 Aug 2017 19:35:08 +0000 Fri, 8 Jul 2016 20:27:45 +0000 Lustre 2.6.0 Lustre 2.9.0 0 10 <p>John, how did you hit this originally, and how easily is it reproduced? Just trying to decide if this is a regression that needs to be addressed for 2.6.0?</p> <p>After <a href="https://jira.whamcloud.com/browse/LU-5274" title="ll_statahead_thread() may leak parent, uses parent after dput" class="issue-link" data-issue-key="LU-5274"><del>LU-5274</del></a> I dug up my OBD_ALLOC() fault injection patch and ran racer with it applied.</p> <p>A couple of Cray customers have hit this assertion using Lustre 2.5. One case suggests that the bug may be reproduced by running du against a large file system. The du causes Lustre to run the node out of memory (millions of unused ldlm_locks being canceled). An ldlm_bl thread calls into mdc_close but cannot allocate memory for the rpc. The mdc_close does not reset the rq_replay flag, and the subsequent call to mdc_free_open asserts. The ldlm_locks were held primarily by 2 processes, one of which had finished by the time of the LBUG. So there may have been at least 2 processes running du that triggered the LBUG.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&gt;c0-0c1s1n2 LustreError: 28818:0:(file.c:172:ll_close_inode_openhandle()) inode 4035227916085244137 mdc close failed: rc = -12 &gt;c0-0c1s1n2 LustreError: 28818:0:(mdc_request.c:816:mdc_free_open()) ASSERTION( mod-&gt;mod_open_req-&gt;rq_replay == 0 ) failed: &gt; c0-0c1s1n2 LustreError: 28818:0:(mdc_request.c:816:mdc_free_open()) LBUG &gt; c0-0c1s1n2 Pid: 28818, comm: ldlm_bl_51 &gt; c0-0c1s1n2 Call Trace: &gt; c0-0c1s1n2 [&lt;ffffffff81005f31&gt;] try_stack_unwind+0x191/0x1a0 &gt; c0-0c1s1n2 [&lt;ffffffff810047eb&gt;] dump_trace+0x8b/0x350 &gt; c0-0c1s1n2 [&lt;ffffffffa059e813&gt;] libcfs_debug_dumpstack+0x53/0x80 [libcfs] &gt; c0-0c1s1n2 [&lt;ffffffffa059eda5&gt;] lbug_with_loc+0x45/0xc0 [libcfs] &gt; c0-0c1s1n2 [&lt;ffffffffa0b5912a&gt;] mdc_free_open+0xda/0x110 [mdc] &gt; c0-0c1s1n2 [&lt;ffffffffa0b5b9e8&gt;] mdc_clear_open_replay_data+0x58/0x280 [mdc] &gt; c0-0c1s1n2 [&lt;ffffffffa0269c3d&gt;] lmv_clear_open_replay_data+0x17d/0x690 [lmv] &gt; c0-0c1s1n2 [&lt;ffffffffa0bb3a57&gt;] ll_close_inode_openhandle+0x437/0xf20 [lustre] &gt; c0-0c1s1n2 [&lt;ffffffffa0bb5d0a&gt;] ll_md_real_close+0x11a/0x210 [lustre] &gt; c0-0c1s1n2 [&lt;ffffffffa0be5635&gt;] ll_md_blocking_ast+0x665/0x780 [lustre] &gt; c0-0c1s1n2 [&lt;ffffffffa09355cd&gt;] ldlm_cancel_callback+0x6d/0x150 [ptlrpc] &gt; c0-0c1s1n2 [&lt;ffffffffa0940ca0&gt;] ldlm_cli_cancel_local+0xa0/0x420 [ptlrpc] &gt; c0-0c1s1n2 [&lt;ffffffffa094575a&gt;] ldlm_cli_cancel_list_local+0xea/0x280 [ptlrpc] &gt; c0-0c1s1n2 [&lt;ffffffffa094abe7&gt;] ldlm_bl_thread_main+0xd7/0x3f0 [ptlrpc] &gt; c0-0c1s1n2 [&lt;ffffffff81069a70&gt;] kthread+0xc0/0xd0 &gt; c0-0c1s1n2 [&lt;ffffffff815a0a88&gt;] ret_from_fork+0x58/0x90 &gt; PID: 11210 TASK: ffff880c5835f200 CPU: 4 COMMAND: "du" &gt; #0 [ffff880cfba2bbc8] schedule at ffffffff81595e45 &gt; #1 [ffff880cfba2bc48] cl_lock_state_wait at ffffffffa0825025 [obdclass] &gt; #2 [ffff880cfba2bcc0] cl_enqueue_locked at ffffffffa0829003 [obdclass] &gt; #3 [ffff880cfba2bcf8] cl_lock_request at ffffffffa0829d9b [obdclass] &gt; #4 [ffff880cfba2bd50] cl_glimpse_lock at ffffffffa0c03785 [lustre] &gt; #5 [ffff880cfba2bda0] cl_glimpse_size0 at ffffffffa0c03c25 [lustre] &gt; #6 [ffff880cfba2bde0] ll_inode_revalidate_it at ffffffffa0bbf416 [lustre] &gt; #7 [ffff880cfba2be10] ll_getattr_it at ffffffffa0bbf496 [lustre] &gt; #8 [ffff880cfba2be40] ll_getattr at ffffffffa0bbf5d4 [lustre] &gt; #9 [ffff880cfba2be98] vfs_getattr_nosec at ffffffff8117f4a9 &gt; #10 [ffff880cfba2bea8] vfs_fstatat at ffffffff8117f622 &gt; #11 [ffff880cfba2bef0] SYSC_newfstatat at ffffffff8117fb15 &gt; #12 [ffff880cfba2bf70] sys_newfstatat at ffffffff8117fcfe &gt; #13 [ffff880cfba2bf80] system_call_fastpath at ffffffff815a0b32 &gt; RIP: 00007f1ab40b383a RSP: 00007ffc0061e630 RFLAGS: 00000246 &gt; RAX: 0000000000000106 RBX: ffffffff815a0b32 RCX: 0000000000000000 &gt; RDX: 0000000000e695e8 RSI: 0000000000e69678 RDI: 0000000000000005 &gt; RBP: 0000000000e695e8 R8: 0000000000000100 R9: 0000000000000002 &gt; R10: 0000000000000100 R11: 0000000000000246 R12: ffffffff8117fcfe &gt; R13: ffff880cfba2bf78 R14: 0000000000e69570 R15: 0000000000000000 &gt; ORIG_RAX: 0000000000000106 CS: 0033 SS: 002b </pre> </div></div> <p>Alexander Boyko (alexander.boyko@seagate.com) uploaded a new patch: <a href="http://review.whamcloud.com/17442" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17442</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5282" title="(mdc_request.c:791:mdc_free_open()) ASSERTION( mod-&gt;mod_open_req-&gt;rq_replay == 0 ) failed" class="issue-link" data-issue-key="LU-5282"><del>LU-5282</del></a> test: check error during mdc_close<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5f71f93bac5472e2835ee1544770c243de1f95a0</p> <p>Fix pushed to gerrit - <a href="http://review.whamcloud.com/#/c/17495/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/17495/</a></p> <p>The patch change the assertion to warning. It is not a fatal error if client doesn`t send a close request to MDS. In worst case MDS evict client and client doesn`t send cookies to OST. With reproducer I see that mdc export was not fully released.<br/> 18 ST mdc lustre-MDT0000-mdc-ffff88005be2d800 18902755-d63e-aa43-5b3f-5e81f3be1b56 2</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17495/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17495/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5282" title="(mdc_request.c:791:mdc_free_open()) ASSERTION( mod-&gt;mod_open_req-&gt;rq_replay == 0 ) failed" class="issue-link" data-issue-key="LU-5282"><del>LU-5282</del></a> mdc: fix panic at mdc_free_open()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 62713a8530349a75b7202c9bfc6be121409a0203</p> <p>Landed to master for 2.9.0</p> Related Development Rank 1|hzwqen: Rank (Obsolete) 14743 Severity