https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1644 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/e20033fc-d075-11e1-9002-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e20033fc-d075-11e1-9002-52540035b04c</a>.</p> <p>The sub-test lustre-initialization_1 failed with the following error:</p> <blockquote> <p>Test system failed to start single suite, so abandoning all hope and giving up</p></blockquote> <p>This is the console log from client-1</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>16:26:00:LustreError: 4164:0:(mgc_request.c:1297:mgc_apply_recover_logs()) ASSERTION( entry-&gt;mne_length &lt;= ((1UL) &lt;&lt; 12) ) failed: 16:26:00:LustreError: 4164:0:(mgc_request.c:1297:mgc_apply_recover_logs()) LBUG 16:26:00:Pid: 4164, comm: mount.lustre 16:26:00: 16:26:00:Call Trace: 16:26:00: [&lt;ffffffffa0428905&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 16:26:00: [&lt;ffffffffa0428f17&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 16:26:00: [&lt;ffffffffa053e438&gt;] mgc_apply_recover_logs+0x13e8/0x17e0 [mgc] 16:26:00: [&lt;ffffffffa0780426&gt;] ? __req_capsule_get+0x176/0x750 [ptlrpc] 16:26:00: [&lt;ffffffffa0429bae&gt;] ? cfs_free+0xe/0x10 [libcfs] 16:26:00: [&lt;ffffffffa0757cc0&gt;] ? lustre_swab_mgs_config_res+0x0/0x20 [ptlrpc] 16:26:01: [&lt;ffffffffa05413b4&gt;] mgc_process_log+0xe54/0x12f0 [mgc] 16:26:01: [&lt;ffffffffa053a980&gt;] ? mgc_blocking_ast+0x0/0x680 [mgc] 16:26:01: [&lt;ffffffffa0732380&gt;] ? ldlm_completion_ast+0x0/0x730 [ptlrpc] 16:26:01: [&lt;ffffffffa0542d76&gt;] mgc_process_config+0x5c6/0xee0 [mgc] 16:26:01: [&lt;ffffffffa05e80ec&gt;] lustre_process_log+0x25c/0xad0 [obdclass] 16:26:02: [&lt;ffffffff8127f332&gt;] ? __percpu_counter_init+0x62/0x70 16:26:02: [&lt;ffffffffa0a897e0&gt;] ll_fill_super+0xa70/0x1490 [lustre] 16:26:02: [&lt;ffffffffa05f342d&gt;] lustre_fill_super+0x11d/0xfd0 [obdclass] 16:26:02: [&lt;ffffffffa05f3310&gt;] ? lustre_fill_super+0x0/0xfd0 [obdclass] 16:26:02: [&lt;ffffffff8117989f&gt;] get_sb_nodev+0x5f/0xa0 16:26:02: [&lt;ffffffffa05e2cf5&gt;] lustre_get_sb+0x25/0x30 [obdclass] 16:26:02: [&lt;ffffffff811794fb&gt;] vfs_kern_mount+0x7b/0x1b0 16:26:02: [&lt;ffffffff811796a2&gt;] do_kern_mount+0x52/0x130 16:26:02: [&lt;ffffffff81197ce2&gt;] do_mount+0x2d2/0x8d0 16:26:02: [&lt;ffffffff81198370&gt;] sys_mount+0x90/0xe0 16:26:02: [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b 16:26:02: 16:26:02:Kernel panic - not syncing: LBUG 16:26:02:Pid: 4164, comm: mount.lustre Not tainted 2.6.32-220.17.1.el6.x86_64 #1 </pre> </div></div> server: lustre-b2_2<br/> client: master LU-1644

lustre b2_2<->master failure on lustre-initialization-1: ASSERTION( entry->mne_length <= ((1UL) << 12) )

Bug Blocker Resolved Fixed Jinshan Xiong Maloo Wed, 18 Jul 2012 13:56:54 +0000 Wed, 19 Dec 2018 16:24:22 +0000 Thu, 13 Sep 2012 01:37:23 +0000 Lustre 2.3.0 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.12.0 0 7 <p>Both master vs b2_2 and b2_2 vs master hit this issue:</p> <p><a href="https://maloo.whamcloud.com/test_sessions/611d3f18-d058-11e1-9002-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/611d3f18-d058-11e1-9002-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sessions/dc6a13ea-d075-11e1-9002-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/dc6a13ea-d075-11e1-9002-52540035b04c</a></p> <p>Jinshan,<br/> Would you be able to take a look at this one and assign to yourself if it's appropriate for you to fix?<br/> Thank you!</p> <p>Jinshan will look into this one</p> <p>We need to land commit 35a8ed2b2007d89c1f125f01f155232e7f511e98 to 2.2 to fix this problem.</p> <p>What do you know. I have the patch for <a href="https://jira.whamcloud.com/browse/LU-1252" title="Imperative recovery bugs go here" class="issue-link" data-issue-key="LU-1252"><del>LU-1252</del></a> for b2_2 already available at <a href="http://review.whamcloud.com/#change,3008" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3008</a></p> <p>Cool, thanks James.</p> <p>James</p> <p>Thanks! <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Peter</p> <p>Jinshan,<br/> Since we do have 2.2 servers, the 2.3 client should not crash. So we should create a patch on master to avoid the crash. Can you please look into this?<br/> Thank you!</p> <p>Hi Jodi,</p> <p>master has already had this patch. The problem was 2.2 was released during the time when we're working on this patch. After we commit this patch to 2.2, then we're done.</p> <p>Jinshan</p> <p>Apparently this patch being will crash the 2.2 client, so we cannot retroactively fix that release, and the chance of making a 2.2.1 release is small. We need to have some mechanism to detect if the client is handling this swabbing correctly. The best method is to use an OBD_CONNECT flag being sent from the 2.3+ clients and checked by the 2.3+ servers to decide how the swabbing needs to be done.</p> <p>I'm reluctant to use a separate flag for just fixing this rare bug. Oleg's suggestion is to re-use an existing OBD_CONNECT flag that is not currently being used for the MGS, which can be deprecated easily in the future. I would suggest OBD_CONNECT_GRANT, which is a flag that we can also soon deprecate for 2.x clients as well. Something like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">/* overload OBD_CONNECT_GRANT to fix rare 2.2/2.3 problem with mixed-endian * interop swabbing <span class="code-keyword">for</span> IR mne_length field. This can be removed in the * <span class="code-keyword">future</span> when we don't expect 2.2 clients running with 2.3+ servers. * See LU-1644 <span class="code-keyword">for</span> details */ #define OBD_CONNECT_MNE_SWAB OBD_CONNECT_GRANT </pre> </div></div> <p>A patch is pushed to: <a href="http://review.whamcloud.com/3548" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3548</a></p> <p>to fix the comparability problem between 2.2 client and 2.3+ servers.</p> <p>Landed for 2.3</p> <p>Stillhit this error on 2.2 client&lt;-&gt;2.3-tag2.2.94 interop testing<br/> <a href="https://maloo.whamcloud.com/test_sessions/1f9fde02-f42e-11e1-b3b2-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/1f9fde02-f42e-11e1-b3b2-52540035b04c</a></p> <p>I can't even mount 2.2 clients to master servers, with this error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@client-17 ~]# uname -r 2.6.32-220.4.2.el6_lustre.g45b2fe8.x86_64 [root@client-17 ~]# rpm -qa |grep lustre lustre-2.2.0-2.6.32_220.4.2.el6_lustre.g45b2fe8.x86_64_g25a1427.x86_64 kernel-2.6.32-220.4.2.el6_lustre.g45b2fe8.x86_64 lustre-modules-2.2.0-2.6.32_220.4.2.el6_lustre.g45b2fe8.x86_64_g25a1427.x86_64 lustre-ldiskfs-3.3.0-2.6.32_220.4.2.el6_lustre.g45b2fe8.x86_64_g25a1427.x86_64 [root@client-17 ~]# mount -t lustre client-18@tcp:/lustre /mnt/lustre mount.lustre: mount client-18@tcp:/lustre at /mnt/lustre failed: Invalid argument This may have multiple causes. Is 'lustre' the correct filesystem name? Are the mount options correct? Check the syslog for more info. [root@client-17 ~]# dmesg Lustre: MGC10.10.4.18@tcp: Reactivating import Lustre: 6833:0:(obd_config.c:1002:class_process_config()) Ignoring unknown param jobid_var=procname_uid LustreError: 6833:0:(obd_config.c:1362:class_config_llog_handler()) Err -22 on cfg command: Lustre: cmd=cf00f 0:(null) 1:sys.jobid_var=procname_uid 2:procname_uid LustreError: 15b-f: MGC10.10.4.18@tcp: The configuration from log 'lustre-client'failed from the MGS (-22). Make sure this client and the MGS are running compatible versions of Lustre. LustreError: 15c-8: MGC10.10.4.18@tcp: The configuration from log 'lustre-client' failed (-22). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. LustreError: 6821:0:(llite_lib.c:978:ll_fill_super()) Unable to process log: -22 LustreError: 6736:0:(lov_obd.c:928:lov_cleanup()) lov tgt 0 not cleaned! deathrow=0, lovrc=1 LustreError: 6736:0:(lov_obd.c:928:lov_cleanup()) Skipped 3 previous similar messages LustreError: 6821:0:(ldlm_request.c:1170:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway LustreError: 6821:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Lustre: client ffff880329563000 umount complete LustreError: 6821:0:(obd_mount.c:2349:lustre_fill_super()) Unable to mount (-22) </pre> </div></div> <p>any secret to do that successfully?</p> <p>Patch is at: <a href="http://review.whamcloud.com/3897" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3897</a></p> <p>Landed for 2.3 and 2.4</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32087" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32087</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1644" title="lustre b2_2&lt;-&gt;master failure on lustre-initialization-1: ASSERTION( entry-&gt;mne_length &lt;= ((1UL) &lt;&lt; 12) )" class="issue-link" data-issue-key="LU-1644"><del>LU-1644</del></a> mgc: remove obsolete IR swabbing workaround<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ee1e23c0b535f5726008b0de015cea813815d6de</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32088" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32088</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1644" title="lustre b2_2&lt;-&gt;master failure on lustre-initialization-1: ASSERTION( entry-&gt;mne_length &lt;= ((1UL) &lt;&lt; 12) )" class="issue-link" data-issue-key="LU-1644"><del>LU-1644</del></a> ptlrpc: fix return type of boolean functions<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 764161ac692e9ff99ba48d39032a1adb0ff5351e</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/32087/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32087/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1644" title="lustre b2_2&lt;-&gt;master failure on lustre-initialization-1: ASSERTION( entry-&gt;mne_length &lt;= ((1UL) &lt;&lt; 12) )" class="issue-link" data-issue-key="LU-1644"><del>LU-1644</del></a> mgc: remove obsolete IR swabbing workaround<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a0c644fde3405bba6752885481f0fdfe05da1bcd</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/32088/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32088/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1644" title="lustre b2_2&lt;-&gt;master failure on lustre-initialization-1: ASSERTION( entry-&gt;mne_length &lt;= ((1UL) &lt;&lt; 12) )" class="issue-link" data-issue-key="LU-1644"><del>LU-1644</del></a> ptlrpc: fix return type of boolean functions<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e2cac9fb9baf43f48eb58334fb5044cece5395c0</p> Duplicate LU-1701 Related LU-5329 Development Rank 1|hzv5nr: Rank (Obsolete) 4451 Severity