https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15794 Lustre <p>Immediately on starting sanity, after downgrading the client - </p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[  368.034864] Lustre: DEBUG MARKER: trevis-86vm1.trevis.whamcloud.com: executing check_config_client /mnt/lustre^M [  370.048716] Lustre: DEBUG MARKER: Using TIMEOUT=100^M [  371.064948] LustreError: 11993:0:(lov_object.c:551:lov_init_dom()) ASSERTION( index == 0 ) failed: ^M [  371.066828] LustreError: 11993:0:(lov_object.c:551:lov_init_dom()) LBUG^M [  371.068127] Pid: 11993, comm: rm 3.10.0-1160.45.1.el7.x86_64 #1 SMP Wed Oct 13 17:20:51 UTC 2021^M [  371.069809] Call Trace:^M [  371.070360]  [&lt;ffffffffc08b57cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs]^M [  371.071728]  [&lt;ffffffffc08b587c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs]^M [  371.073141]  [&lt;ffffffffc0d3df99&gt;] lov_init_dom+0x8f9/0x990 [lov]^M [  371.074443]  [&lt;ffffffffc0d3e85c&gt;] lov_init_composite+0x82c/0xbe0 [lov]^M [  371.075827]  [&lt;ffffffffc0d3ad00&gt;] lov_object_init+0x130/0x300 [lov]^M [  371.077174]  [&lt;ffffffffc0a3454b&gt;] lu_object_start.isra.31+0x8b/0x120 [obdclass]^M [  371.078719]  [&lt;ffffffffc0a37a54&gt;] lu_object_find_at+0x234/0xab0 [obdclass]^M [  371.080157]  [&lt;ffffffffc0a3830f&gt;] lu_object_find_slice+0x1f/0x90 [obdclass]^M [  371.081625]  [&lt;ffffffffc0a3cba2&gt;] cl_object_find+0x32/0x60 [obdclass]^M [  371.082966]  [&lt;ffffffffc0e9e599&gt;] cl_file_inode_init+0x219/0x380 [lustre]^M [  371.084399]  [&lt;ffffffffc0e78595&gt;] ll_update_inode+0x2d5/0x5e0 [lustre]^M [  371.085759]  [&lt;ffffffffc0e78907&gt;] ll_read_inode2+0x67/0x420 [lustre]^M [  371.087094]  [&lt;ffffffffc0e8661b&gt;] ll_iget+0xdb/0x350 [lustre]^M [  371.088315]  [&lt;ffffffffc0e7a6f3&gt;] ll_prep_inode+0x253/0x970 [lustre]^M [  371.089648]  [&lt;ffffffffc0e87653&gt;] ll_lookup_it+0x523/0x1a20 [lustre]^M [  371.090989]  [&lt;ffffffffc0e89f9b&gt;] ll_lookup_nd+0xbb/0x190 [lustre]^M [  371.092301]  [&lt;ffffffffa0c591d3&gt;] lookup_real+0x23/0x60^M [  371.093424]  [&lt;ffffffffa0c59bf2&gt;] __lookup_hash+0x42/0x60^M [  371.094571]  [&lt;ffffffffa0c60adc&gt;] do_unlinkat+0x14c/0x2d0^M [  371.095717]  [&lt;ffffffffa0c61bbb&gt;] SyS_unlinkat+0x1b/0x40^M [  371.096862]  [&lt;ffffffffa1195f92&gt;] system_call_fastpath+0x25/0x2a^M [  371.098126]  [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff^M [  371.099214] Kernel panic - not syncing: LBUG^M [  371.100077] CPU: 0 PID: 11993 Comm: rm Kdump: loaded Tainted: G           OE  ------------   3.10.0-1160.45.1.el7.x86_64 #1^M [  371.102139] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011^M </pre> </div></div> trevis-86 - servers with version=2.15.0_RC3, client b2_12 build 150 version=2.12.8 LU-15794

Downgrade client fails: LustreError: 11993:0:(lov_object.c:551:lov_init_dom()) ASSERTION( index == 0 ) failed:

Bug Major Open Unresolved Zhenyu Xu Cliff White Wed, 27 Apr 2022 22:10:39 +0000 Wed, 15 Jun 2022 21:40:26 +0000 Lustre 2.15.0 0 4 <p>Bobijam</p> <p>What are you thoughts on this error? Could it be related to <a href="https://jira.whamcloud.com/browse/LU-13249" title="MDS getting: LustreError: 5095:0:(llog_osd.c:1244:llog_osd_open ()) ASSERTION( dt ) failed:" class="issue-link" data-issue-key="LU-13249">LU-13249</a></p> <p>Peter</p> <p>Cliff, presumably the servers were originally running 2.12.8 before upgrading to 2.15.0-RC3? Were the DOM files that are being removed originally created on 2.15.0 or 2.12.8?</p> <p>I think this looks like <a href="https://jira.whamcloud.com/browse/LU-15513" title="crash in lod_fill_mirrors() with sparse OSTs + PFL" class="issue-link" data-issue-key="LU-15513"><del>LU-15513</del></a> (something strange with the DoM layout on the client), and not related to llog at all (<a href="https://jira.whamcloud.com/browse/LU-13249" title="MDS getting: LustreError: 5095:0:(llog_osd.c:1244:llog_osd_open ()) ASSERTION( dt ) failed:" class="issue-link" data-issue-key="LU-13249">LU-13249</a> is on the server). I see that this LASSERT is not present on master, having been removed by patch <a href="https://review.whamcloud.com/35359" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35359</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-11421" title="DoM: manual migration OST-MDT, MDT-MDT" class="issue-link" data-issue-key="LU-11421"><del>LU-11421</del></a> dom: manual OST-to-DOM migration via mirroring</tt>":</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> - LASSERT(index == 0); + <span class="code-comment">/* DOM entry may be not zero index due to FLR but must start from 0 */</span> + <span class="code-keyword">if</span> (unlikely(lle-&gt;lle_extent-&gt;e_start != 0)) { + CERROR(<span class="code-quote">"%s: DOM entry must be the first stripe in a mirror\n"</span>, + lov2obd(dev-&gt;ld_lov)-&gt;obd_name); + dump_lsm(D_ERROR, lov-&gt;lo_lsm); + RETURN(-EINVAL); + } </pre> </div></div> <p>Cliff, was the problematic file mirrored or migrated while running 2.15.0-RC3? What does "<tt>lfs getstripe</tt>" (from a 2.15 client) show?</p> <p>I think this looks more like a bug in 2.12 rather than 2.15 (patch was landed as commit v2_12_58-12-g44a721b8c1). I <em>think</em> patch has other dependencies (at least <a href="https://review.whamcloud.com/45549" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45549</a>). At a minimum it would make sense to backport the LASSERT cleanups to b2_12 for 2.12.9?</p> <p>Sequence - </p> <p>Installed system w/2.12.8 (servers and client)  - </p> <p>Upgraded OSS then MDS then client to 2.15.0-RC3 </p> <p>Upon downgrade of client, LBUG is triggered</p> <p>LBUG continues after server downgrade. </p> <p>No errors at all on the server side, appears to be triggered by something in initial sanity.sh setup as we never get past the startup. </p> <p>MDS log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 2325.482110] Lustre: DEBUG MARKER: trevis-86vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 48 [ 2553.048674] Lustre: MGS: haven't heard from client 84d85de2-7af6-f334-55a2-6f84ca2afbe9 (at 10.240.43.40@tcp) in 230 seconds. I think it's dead, and I am evicting it. exp ffff89d48bab8c00, cur 1651103424 expire 1651103274 last 1651103194 </pre> </div></div> <p> Client suite log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> -----============= acceptance-small: sanity ============----- Wed Apr 27 23:46:31 UTC 2022 excepting tests: 103a 103b 103c 104a 160c 161a 161b 161c 208 220 225a 225b 228b 255a 255b 407 253 312 42a 42b 42c 77k skipping tests SLOW=no: 27m 64b 68 71 115 300o trevis-86vm1.trevis.whamcloud.com: executing check_config_client /mnt/lustre trevis-86vm1.trevis.whamcloud.com: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients trevis-86vm1.trevis.whamcloud.com environments Using TIMEOUT=100 osc.lustre-OST0000-osc-ffff974f7a0eb800.idle_timeout=debug disable quota as required trevis-86vm2: trevis-86vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 48 trevis-86vm3: trevis-86vm3.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 48 osd-ldiskfs.track_declares_assert=1 osd-ldiskfs.track_declares_assert=1 ~ </pre> </div></div> <p>tail of Client suite log, set -x </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>++ for var in LNETLND NETTYPE ++ '[' -n tcp ']' ++ echo -n ' NETTYPE=tcp' + pdsh -t 300 -S -w trevis-86vm2,trevis-86vm3 '(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre"  mds1_FSTYPE=ldiskfs ost1_FSTYPE=ldiskfs VERBOSE=false FSTYPE=ldiskfs NETTYPE=tcp sh -c "/usr/sbin/lctl set_param      osd-ldiskfs.track_declares_assert=1 || true")' osd-ldiskfs.track_declares_assert=1 </pre> </div></div> <p>Systems are still up (trevis-86vm<span class="error">&#91;1-3&#93;</span>)  </p> <p>I will start the 2.14 up/down testing tomorrow on these nodes unless otherwise advised. </p> Related LU-15219 LU-11421 Development Rank 1|i02oef: Rank (Obsolete) 9223372036854775807 Severity