https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2959 Lustre <p>Hit LBUG on one of the oss.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 16207:0:(mgc_request.c:1682:mgc_copy_llog()) ASSERTION( cli-&gt;cl_mgc_configs_dir ) failed: LustreError: 16207:0:(mgc_request.c:1682:mgc_copy_llog()) LBUG Pid: 16207, comm: ll_cfg_requeue Call Trace: [&lt;ffffffffa04e2895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa04e2e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0dbfe2b&gt;] mgc_process_cfg_log+0x134b/0x15c0 [mgc] [&lt;ffffffffa0dc2093&gt;] mgc_process_log+0x463/0x1390 [mgc] [&lt;ffffffff814ead1a&gt;] ? schedule_timeout+0x19a/0x2e0 [&lt;ffffffffa0dbca30&gt;] ? mgc_blocking_ast+0x0/0x770 [mgc] [&lt;ffffffffa07aed40&gt;] ? ldlm_completion_ast+0x0/0x960 [ptlrpc] [&lt;ffffffff81090d4c&gt;] ? remove_wait_queue+0x3c/0x50 [&lt;ffffffffa0dc3973&gt;] mgc_requeue_thread+0x1a3/0x750 [mgc] [&lt;ffffffff8105fa40&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0dc37d0&gt;] ? mgc_requeue_thread+0x0/0x750 [mgc] [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [&lt;ffffffffa0dc37d0&gt;] ? mgc_requeue_thread+0x0/0x750 [mgc] [&lt;ffffffffa0dc37d0&gt;] ? mgc_requeue_thread+0x0/0x750 [mgc] [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 LustreError: dumping log to /tmp/lustre-log.1363211399.16207 </pre> </div></div> while testing wide striping with 1mds, 2 oss. Each oss is creating 200 osts; and mount at the same time using a script. LU-2959

ASSERTION( cli->cl_mgc_configs_dir ) for 200 osts x 2 oss

Bug Major Closed Duplicate Nathaniel Clark Minh Diep LB Wed, 13 Mar 2013 18:33:49 +0000 Fri, 20 Jun 2014 07:19:16 +0000 Fri, 18 Oct 2013 16:51:29 +0000 Lustre 2.4.0 Lustre 2.5.0 0 9 <p>I haven't been able to reproduce this on latest master. We need to check with Alex to see if there's fix recently about this</p> <p>I used <a href="https://build.whamcloud.com/job/lustre-master/1337/" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-master/1337/</a></p> <p>sorry, i'm not aware of any fix in this area.</p> <p>I'm inclined to believe this is a race between lustre_fill_super() and LDLM_CB_CANCELING. I think it's a real bug, but I think it's very hard to get to. Many OST's on a single OSS helps exacerbate the race. I haven't nailed down the exact chain, yet.</p> <p>Dropping priority based on rarity</p> <p><a href="http://review.whamcloud.com/5860" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5860</a></p> <p>It seems the MGC "fs" stuff, like cl_mgc_configs_dir, are time-shared among different OSTs. Each OST sets "fs" and cleans "fs" up in server_start_targets(). mgc_requeue_thread() should set up the "fs" itself, but does not seem to be doing so.</p> <p>it doesn't but it checks that fs is proper one and don't process local log otherwise mgc_process_cfg_log():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (lctxt &amp;&amp; lsi &amp;&amp; IS_SERVER(lsi) &amp;&amp; (lsi-&gt;lsi_srv_mnt == cli-&gt;cl_mgc_vfsmnt) &amp;&amp; !IS_MGS(lsi) &amp;&amp; lsi-&gt;lsi_srv_mnt) { </pre> </div></div> <p>I think we do not update backups on a subsequent requeue (after initial during service startup). so, we don't need to setup/clear fs in requeue.<br/> but given lsi_srv_mnt is unique, we shouldn't be calling mgc_copy_llog() on requeue.</p> <p>interesting thing is that mgc_fs_setup() takes cl_mgc_sem to serialize setup procedues and llog processing for different services, then mgc_fs_cleanup() release it. but requeue doesn't do this.</p> <p>I have hit this assertion on Hyperion, testing 2.3.64 tag. Error occurred when attempting to mount a freshly formatted disk:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Apr 15 16:55:57 hyperion-dit31 kernel: Lustre: lustre-OST0032: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450 Apr 15 16:55:57 hyperion-dit31 kernel: LustreError: 5369:0:(mgc_request.c:1686:mgc_copy_llog()) ASSERTION( cli-&gt;cl_mgc_configs_dir ) failed: Apr 15 16:55:57 hyperion-dit31 kernel: LustreError: 5369:0:(mgc_request.c:1686:mgc_copy_llog()) ASSERTION( cli-&gt;cl_mgc_configs_dir ) failed: Apr 15 16:55:57 hyperion-dit31 kernel: LustreError: 5369:0:(mgc_request.c:1686:mgc_copy_llog()) LBUG Apr 15 16:55:57 hyperion-dit31 kernel: LustreError: 5369:0:(mgc_request.c:1686:mgc_copy_llog()) LBUG Apr 15 16:55:57 hyperion-dit31 kernel: Pid: 5369, comm: ll_cfg_requeue Apr 15 16:55:57 hyperion-dit31 kernel: Apr 15 16:55:57 hyperion-dit31 kernel: Call Trace: Apr 15 16:55:57 hyperion-dit31 kernel: [&lt;ffffffffa054c895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Apr 15 16:55:57 hyperion-dit31 kernel: [&lt;ffffffffa054ce97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] Apr 15 16:55:57 hyperion-dit31 kernel: [&lt;ffffffffa1075ecb&gt;] mgc_process_cfg_log+0x134b/0x15c0 [mgc] Apr 15 16:55:57 hyperion-dit31 kernel: [&lt;ffffffffa1078133&gt;] mgc_process_log+0x463/0x1390 [mgc] Apr 15 16:55:57 hyperion-dit31 kernel: [&lt;ffffffff8150eb5a&gt;] ? schedule_timeout+0x19a/0x2e0 Apr 15 16:55:57 hyperion-dit31 kernel: [&lt;ffffffffa1072a60&gt;] ? mgc_blocking_ast+0x0/0x7e0 [mgc] Apr 15 16:55:57 hyperion-dit31 kernel: [&lt;ffffffffa0acce00&gt;] ? ldlm_completion_ast+0x0/0x960 [ptlrpc] Apr 15 16:55:57 hyperion-dit31 kernel: [&lt;ffffffff8109705c&gt;] ? remove_wait_queue+0x3c/0x50 Apr 15 16:55:57 hyperion-dit31 kernel: [&lt;ffffffffa1079a13&gt;] mgc_requeue_thread+0x1a3/0x750 [mgc] Apr 15 16:55:57 hyperion-dit31 kernel: [&lt;ffffffff81063310&gt;] ? default_wake_function+0x0/0x20 Apr 15 16:55:57 hyperion-dit31 kernel: [&lt;ffffffffa1079870&gt;] ? mgc_requeue_thread+0x0/0x750 [mgc] Apr 15 16:55:57 hyperion-dit31 kernel: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 Apr 15 16:55:57 hyperion-dit31 kernel: [&lt;ffffffffa1079870&gt;] ? mgc_requeue_thread+0x0/0x750 [mgc] Apr 15 16:55:57 hyperion-dit31 kernel: [&lt;ffffffffa1079870&gt;] ? mgc_requeue_thread+0x0/0x750 [mgc] Apr 15 16:55:57 hyperion-dit31 kernel: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>The rest of the stack was lost.</p> <p>Fixed by <a href="http://review.whamcloud.com/5049" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5049</a></p> <p>This bug is hit on several 2.4 sites, could someone backport it to 2.4 ?</p> Duplicate LU-2059 LU-2059 Development Rank 1|hzvlbj: Rank (Obsolete) 7216 Severity