https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4260 Lustre <p>Servers and clients are 2.4.1 configured active-active failover</p> <p>one out of two clients is 1.8.9</p> <p>As soon as I wrote a file to remote directory, the second MDS crashed.</p> <p>Nov 15 09:00:06 lustre-mds-0-1 kernel: LustreError: 20726:0:(lod_lov.c:554:lod_generate_and_set_lovea()) rhino-MDT0001-mdtlov: Can not locate <span class="error">&#91;0x640000bd0:0x22:0x0&#93;</span>: rc = -5<br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: LustreError: 20726:0:(lod_object.c:704:lod_ah_init()) ASSERTION( lc-&gt;ldo_stripenr == 0 ) failed: <br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: LustreError: 20726:0:(lod_object.c:704:lod_ah_init()) LBUG<br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: Pid: 20726, comm: mdt03_000<br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: Call Trace:<br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0349895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0349e97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0e3f78f&gt;&#93;</span> lod_ah_init+0x57f/0x5c0 <span class="error">&#91;lod&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0b73a83&gt;&#93;</span> mdd_object_make_hint+0x83/0xa0 <span class="error">&#91;mdd&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0b7feb2&gt;&#93;</span> mdd_create_data+0x332/0x7d0 <span class="error">&#91;mdd&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0d9cc2c&gt;&#93;</span> mdt_finish_open+0x125c/0x18a0 <span class="error">&#91;mdt&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0d984f8&gt;&#93;</span> ? mdt_object_open_lock+0x1c8/0x510 <span class="error">&#91;mdt&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0d9ee8d&gt;&#93;</span> mdt_reint_open+0x115d/0x20c0 <span class="error">&#91;mdt&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa036682e&gt;&#93;</span> ? upcall_cache_get_entry+0x28e/0x860 <span class="error">&#91;libcfs&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa071fdcc&gt;&#93;</span> ? lustre_msg_add_version+0x6c/0xc0 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0d89911&gt;&#93;</span> mdt_reint_rec+0x41/0xe0 <span class="error">&#91;mdt&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0d6eae3&gt;&#93;</span> mdt_reint_internal+0x4c3/0x780 <span class="error">&#91;mdt&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0d6f06d&gt;&#93;</span> mdt_intent_reint+0x1ed/0x520 <span class="error">&#91;mdt&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0d6cf1e&gt;&#93;</span> mdt_intent_policy+0x39e/0x720 <span class="error">&#91;mdt&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa06d7831&gt;&#93;</span> ldlm_lock_enqueue+0x361/0x8d0 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa06fe1ef&gt;&#93;</span> ldlm_handle_enqueue0+0x4ef/0x10b0 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0d6d3a6&gt;&#93;</span> mdt_enqueue+0x46/0xe0 <span class="error">&#91;mdt&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0d73a97&gt;&#93;</span> mdt_handle_common+0x647/0x16d0 <span class="error">&#91;mdt&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0720bac&gt;&#93;</span> ? lustre_msg_get_transno+0x8c/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0dad3f5&gt;&#93;</span> mds_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa07303c8&gt;&#93;</span> ptlrpc_server_handle_request+0x398/0xc60 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa034a5de&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa035bd9f&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x170 <span class="error">&#91;libcfs&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0727729&gt;&#93;</span> ? ptlrpc_wait_event+0xa9/0x290 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffff81055ad3&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa073175e&gt;&#93;</span> ptlrpc_main+0xace/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0730c90&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0730c90&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffffa0730c90&gt;&#93;</span> ? ptlrpc_main+0x0/0x1700 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Nov 15 09:00:06 lustre-mds-0-1 kernel: </p> LU-4260

ASSERTION( lc->ldo_stripenr == 0 ) failed:

Bug Major Resolved Fixed Di Wang Minh Diep mn4 sdsc Fri, 15 Nov 2013 17:34:45 +0000 Tue, 29 Apr 2014 17:20:59 +0000 Tue, 11 Feb 2014 20:51:59 +0000 Lustre 2.4.1 Lustre 2.6.0 Lustre 2.5.1 0 11 <p>is this similar to <a href="https://jira.whamcloud.com/browse/LU-4226" title="MDS unable to locate swabbed FID SEQ in FLDB" class="issue-link" data-issue-key="LU-4226"><del>LU-4226</del></a>?</p> <p>Di</p> <p>Could you please assist with this one?</p> <p>Thanks</p> <p>Peter</p> <p>I suspect this is <a href="https://jira.whamcloud.com/browse/LU-2789" title="lod_load_striping()) ASSERTION( lo-&gt;ldo_stripenr == 0 ) failed" class="issue-link" data-issue-key="LU-2789"><del>LU-2789</del></a> or closely related.</p> <p>Or after another look: might be not, but certainly there are several crashes logged in there with different backtraces.</p> <p>Minh, could you please try the patches referenced by <a href="https://jira.whamcloud.com/browse/LU-2789" title="lod_load_striping()) ASSERTION( lo-&gt;ldo_stripenr == 0 ) failed" class="issue-link" data-issue-key="LU-2789"><del>LU-2789</del></a>. We aren't sure if this is a duplicate or not. That one relates to a race condition, which I don't think is the case here.</p> <p>Minh, it isn't possible to use clients &lt; 2.4.0 with multiple MDTs. There definitely shouldn't be an LASSERT() failure on the MDS, but it should return an error to the client.</p> <p>Hmm, I checked the debug log here (Minh collected for me), it seem the new created OST sequence is not being inserted into FLDB somehow, which caused some garbage stripe_info(of lod object) left in memory, then the LBUG is being hitted. So we need</p> <p>1. cleanup the stripe_info of lod object when some error happens.<br/> 2. Figure out why OST sequence is not being inserted into FLDB during upgrade process.</p> <p>Andreas, clients &lt; 2.4.0 still can be mounted and use the MDT0. Only 2.4.0+ clients can access all MDTs. This is hit when 2.4.1 client accessing the remote directory.</p> <p>Those OST sequence seems to be added accidentally during wrong strange upgrade process(though I do not know how to reproduce it <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>. But after we removed seq_srv on OST(i.e. those local sequence file on OST, and only used for DNE), and restart OST, and OST will re-acquire new meta sequence from MDT0. Everything works fine. </p> <p>Though we still need cleanup the stripe_info of lod object when some error happens, to avoid LBUG. Here is the patch </p> <p><a href="http://review.whamcloud.com/8325" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8325</a> (b2_4)<br/> <a href="http://review.whamcloud.com/8324" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8324</a> (master)</p> <p>Landed for 2.6</p> <p>Not sure if this is the same issue, but I'm still seeing the crash with exactly the same call stack on latest master.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 20580:0:(lod_object.c:1475:lod_ah_init()) ASSERTION( lc-&gt;ldo_stripenr == 0 ) failed: LustreError: 20580:0:(lod_object.c:1475:lod_ah_init()) LBUG Pid: 20580, comm: mdt01_008 Call Trace: [&lt;ffffffffa03a3895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa03a3e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0e098de&gt;] lod_ah_init+0xaae/0xb80 [lod] [&lt;ffffffffa0ce1b09&gt;] mdd_object_make_hint+0x139/0x180 [mdd] [&lt;ffffffffa0cd12f9&gt;] mdd_create_data+0x359/0x7f0 [mdd] [&lt;ffffffffa075c4e0&gt;] ? lustre_swab_mdt_body+0x0/0x140 [ptlrpc] [&lt;ffffffffa0d4afdb&gt;] mdt_mfd_open+0xc8b/0xf10 [mdt] [&lt;ffffffffa0e033a3&gt;] ? lod_xattr_get+0x153/0x420 [lod] [&lt;ffffffffa0d4c253&gt;] mdt_finish_open+0x553/0xc20 [mdt] [&lt;ffffffffa0d46383&gt;] ? mdt_object_open_lock+0x2f3/0x9c0 [mdt] [&lt;ffffffffa0d4e76f&gt;] mdt_reint_open+0x12af/0x2130 [mdt] [&lt;ffffffffa03c11c6&gt;] ? upcall_cache_get_entry+0x296/0x880 [libcfs] [&lt;ffffffffa0550310&gt;] ? lu_ucred+0x20/0x30 [obdclass] [&lt;ffffffffa0d36851&gt;] mdt_reint_rec+0x41/0xe0 [mdt] [&lt;ffffffffa0d1be13&gt;] mdt_reint_internal+0x4c3/0x7c0 [mdt] [&lt;ffffffffa0d1c308&gt;] mdt_intent_reint+0x1f8/0x520 [mdt] [&lt;ffffffffa0d1a9e9&gt;] mdt_intent_policy+0x499/0xca0 [mdt] [&lt;ffffffff81168742&gt;] ? kmem_cache_alloc+0x182/0x190 [&lt;ffffffffa070f809&gt;] ldlm_lock_enqueue+0x359/0x920 [ptlrpc] [&lt;ffffffffa0738c6f&gt;] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] [&lt;ffffffff8103c7d8&gt;] ? pvclock_clocksource_read+0x58/0xd0 [&lt;ffffffffa07bb022&gt;] tgt_enqueue+0x62/0x1d0 [ptlrpc] [&lt;ffffffffa07bb3cc&gt;] tgt_request_handle+0x23c/0xac0 [ptlrpc] [&lt;ffffffffa075a01c&gt;] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] [&lt;ffffffffa076a5ca&gt;] ptlrpc_main+0xd1a/0x1980 [ptlrpc] [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320 [&lt;ffffffff8150e600&gt;] ? thread_return+0x4e/0x76e [&lt;ffffffffa07698b0&gt;] ? ptlrpc_main+0x0/0x1980 [ptlrpc] [&lt;ffffffff81096a36&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [&lt;ffffffff810969a0&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>I have a core dump so feel free to ask for it.</p> Related LU-2789 LU-4623 Development Rank 1|hzw91j: Rank (Obsolete) 11624 Severity