https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2888 Lustre <p>Here are what I did:<br/> 1. format the system as 2.1.4 and then upgrade to 2.4, success.<br/> 2. showdown the filesystem and disable quota<br/> 3. downgrade the system to 2.1.4 again, when mount MDS, hit following errors</p> <p>Here is the console of MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == upgrade-downgrade End == 18:53:45 (1362020025) LDISKFS-fs warning (device sdb1): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs. LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: LDISKFS-fs warning (device sdb1): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs. LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: LDISKFS-fs warning (device sdb1): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs. LDISKFS-fs (sdb1): mounted filesystem with ordered data mode. Opts: Lustre: MGS MGS started Lustre: 7888:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from 7306ea48-8511-52b2-40cf-6424fc417e41@0@lo t0 exp (null) cur 1362020029 last 0 Lustre: MGC10.10.4.132@tcp: Reactivating import Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs. Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 Lustre: Setting parameter lustre-clilov.lov.stripesize in log lustre-client Lustre: Enabling ACL Lustre: Enabling user_xattr LustreError: 7901:0:(osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed: LustreError: 7901:0:(osd_handler.c:2343:osd_index_try()) LBUG Pid: 7901, comm: llog_process_th Message from Call Trace: syslogd@fat-amd [&lt;ffffffffa03797f5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] -1 at Feb 27 18: [&lt;ffffffffa0379e07&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 53:49 ... ker [&lt;ffffffffa0d6bd74&gt;] osd_index_try+0x84/0x540 [osd_ldiskfs] nel:LustreError: [&lt;ffffffffa04c1dfe&gt;] dt_try_as_dir+0x3e/0x60 [obdclass] 7901:0:(osd_han [&lt;ffffffffa0c5eb3a&gt;] orph_index_init+0x6a/0x1e0 [mdd] dler.c:2343:osd_ [&lt;ffffffffa0c6ec45&gt;] mdd_prepare+0x1d5/0x640 [mdd] index_try()) ASS [&lt;ffffffffa0ccd23c&gt;] ? mdt_process_config+0x6c/0x1030 [mdt] ERTION( dt_objec [&lt;ffffffffa0da0499&gt;] cmm_prepare+0x39/0xe0 [cmm] t_exists(dt) ) f [&lt;ffffffffa0ccfd7d&gt;] mdt_device_alloc+0xe0d/0x2190 [mdt] ailed: Me [&lt;ffffffffa04bdeff&gt;] ? keys_fill+0x6f/0x1a0 [obdclass] ssage from syslo [&lt;ffffffffa04a2c87&gt;] obd_setup+0x1d7/0x2f0 [obdclass] gd@fat-amd-1 at [&lt;ffffffffa048ef3b&gt;] ? class_new_export+0x72b/0x960 [obdclass] Feb 27 18:53:49 [&lt;ffffffffa04a2fa8&gt;] class_setup+0x208/0x890 [obdclass] ... kernel:Lu [&lt;ffffffffa04aac6c&gt;] class_process_config+0xc3c/0x1c30 [obdclass] streError: 7901: [&lt;ffffffffa037a993&gt;] ? cfs_alloc+0x63/0x90 [libcfs] 0:(osd_handler.c [&lt;ffffffffa04a5813&gt;] ? lustre_cfg_new+0x353/0x7e0 [obdclass] :2343:osd_index_ [&lt;ffffffffa04acd0b&gt;] class_config_llog_handler+0x9bb/0x1610 [obdclass] try()) LBUG [&lt;ffffffffa0637e3b&gt;] ? llog_client_next_block+0x1db/0x4b0 [ptlrpc] [&lt;ffffffffa0478098&gt;] llog_process_thread+0x888/0xd00 [obdclass] [&lt;ffffffffa0477810&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [&lt;ffffffffa0477810&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] [&lt;ffffffffa0477810&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 Kernel panic - not syncing: LBUG Pid: 7901, comm: llog_process_th Not tainted 2.6.32-279.14.1.el6_lustre.x86_64 #1 Call Trace: [&lt;ffffffff814fdcba&gt;] ? panic+0xa0/0x168 Message from sy [&lt;ffffffffa0379e5b&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] slogd@fat-amd-1 [&lt;ffffffffa0d6bd74&gt;] ? osd_index_try+0x84/0x540 [osd_ldiskfs] at Feb 27 18:53: [&lt;ffffffffa04c1dfe&gt;] ? dt_try_as_dir+0x3e/0x60 [obdclass] 49 ... kernel [&lt;ffffffffa0c5eb3a&gt;] ? orph_index_init+0x6a/0x1e0 [mdd] :Kernel panic - [&lt;ffffffffa0c6ec45&gt;] ? mdd_prepare+0x1d5/0x640 [mdd] not syncing: LBU [&lt;ffffffffa0ccd23c&gt;] ? mdt_process_config+0x6c/0x1030 [mdt] G [&lt;ffffffffa0da0499&gt;] ? cmm_prepare+0x39/0xe0 [cmm] [&lt;ffffffffa0ccfd7d&gt;] ? mdt_device_alloc+0xe0d/0x2190 [mdt] [&lt;ffffffffa04bdeff&gt;] ? keys_fill+0x6f/0x1a0 [obdclass] [&lt;ffffffffa04a2c87&gt;] ? obd_setup+0x1d7/0x2f0 [obdclass] [&lt;ffffffffa048ef3b&gt;] ? class_new_export+0x72b/0x960 [obdclass] [&lt;ffffffffa04a2fa8&gt;] ? class_setup+0x208/0x890 [obdclass] [&lt;ffffffffa04aac6c&gt;] ? class_process_config+0xc3c/0x1c30 [obdclass] [&lt;ffffffffa037a993&gt;] ? cfs_alloc+0x63/0x90 [libcfs] [&lt;ffffffffa04a5813&gt;] ? lustre_cfg_new+0x353/0x7e0 [obdclass] [&lt;ffffffffa04acd0b&gt;] ? class_config_llog_handler+0x9bb/0x1610 [obdclass] [&lt;ffffffffa0637e3b&gt;] ? llog_client_next_block+0x1db/0x4b0 [ptlrpc] [&lt;ffffffffa0478098&gt;] ? llog_process_thread+0x888/0xd00 [obdclass] [&lt;ffffffffa0477810&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 [&lt;ffffffffa0477810&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] [&lt;ffffffffa0477810&gt;] ? llog_process_thread+0x0/0xd00 [obdclass] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 Initializing cgroup subsys cpuset Initializing cgroup subsys cpu </pre> </div></div> before upgrade, server and client: 2.1.4 RHEL6<br/> after upgrade, server and client: lustre-master build# 1270 RHEL6 LU-2888

After downgrade from 2.4 to 2.1.4, hit (osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed

Bug Major Resolved Fixed Zhenyu Xu Sarah Liu Wed, 27 Feb 2013 22:02:35 +0000 Thu, 25 Jul 2013 20:07:10 +0000 Thu, 18 Apr 2013 06:28:51 +0000 Lustre 2.4.0 Lustre 2.1.4 Lustre 2.4.0 Lustre 2.1.6 0 14 <p>I doubt it's related to new OI index file changed in 2.4</p> <p>In 2.14, the calling path is mdd_prepare()-&gt; orph_index_init() -&gt; dt_store_open() // to open a dt_object named as "PENDING"<br/> -&gt;dt_reg_open() -&gt;dt_locate() -&gt;lu_object_find() -&gt;lu_object_find_at() -&gt;lu_object_find_try -&gt;lu_object_alloc -&gt;loo_object_init() // Initialize "PENDING" dt_object</p> <p>-&gt;osd_object_init() -&gt;osd_fid_lookup() // lookup "PENDING" dt_object</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>osd_oi_lookup()</b></div><div class="codeContent panelContent"> <pre class="code-java"> result = osd_oi_lookup(info, oi, fid, id); <span class="code-keyword">if</span> (result == 0) { ...... } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (result == -ENOENT) result = 0; </pre> </div></div> <p>If in OI index file, "PENDING" is not found there (2.4 changed OI index mechanism, could it be the reason?) osd_oi_lookup() returns 0, then osd_object_init() won't call osd_object_init0() and "PENDING" dt_object does not has LOHA_EXISTS in its common attributes.</p> <p>"PENDING" should definitely exist, and I'd think it would be in the special "lookup by name" list instead of needing the OI?</p> <p>Add some log, which partially fit my description. During orph_index_init(), it find out that PENDING object has been created.</p> <div class="panel" style="border-width: 1px;"><div class="panelContent"> <p>00000004:00000001:0.0:1362551173.779945:0:4818:0:(mdd_orphans.c:470:orph_index_init()) Process entered<br/> 00000020:00000001:0.0:1362551173.779945:0:4818:0:(dt_object.c:377:dt_store_open()) Process entered<br/> 00000020:00000001:0.0:1362551173.779952:0:4818:0:(dt_object.c:352:dt_reg_open()) Process entered<br/> 00000020:00000001:0.0:1362551173.779952:0:4818:0:(dt_object.c:222:dt_lookup()) Process entered<br/> 00000004:00000001:0.0:1362551173.779952:0:4818:0:(osd_handler.c:3804:osd_index_ea_lookup()) Process entered<br/> 00000004:00000001:0.0:1362551173.779953:0:4818:0:(osd_handler.c:2753:osd_get_fid_from_dentry()) Process leaving (rc=18446744073709551555 : -61 : ffffffffffffffc3)<br/> 00000004:00000001:0.0:1362551173.779954:0:4818:0:(osd_handler.c:1903:osd_ea_fid_get()) Process entered<br/> 00000004:00000001:0.0:1362551173.779955:0:4818:0:(osd_handler.c:1939:osd_ea_fid_get()) Process leaving (rc=0 : 0 : 0)<br/> 00000004:00000001:0.0:1362551173.779955:0:4818:0:(osd_handler.c:3099:osd_ea_lookup_rec()) Process leaving (rc=0 : 0 : 0)<br/> 00000004:00000001:0.0:1362551173.779956:0:4818:0:(osd_handler.c:3816:osd_index_ea_lookup()) Process leaving (rc=1 : 1 : 1)<br/> 00000020:00000001:0.0:1362551173.779956:0:4818:0:(dt_object.c:232:dt_lookup()) Process leaving (rc=0 : 0 : 0)<br/> 00000020:00000002:0.0:1362551173.779957:0:4818:0:(dt_object.c:355:dt_reg_open()) dt_locate <span class="error">&#91;0x200000001:0x7:0x0&#93;</span> <font color="red">==&gt; PENDING fid</font><br/> 00000020:00000001:0.0:1362551173.779957:0:4818:0:(dt_object.c:245:dt_locate()) Process entered<br/> 00000020:00000001:0.0:1362551173.779957:0:4818:0:(lu_object.c:621:lu_object_find_at()) Process entered<br/> 00000020:00000001:0.0:1362551173.779958:0:4818:0:(lu_object.c:550:lu_object_find_try()) Process entered<br/> 00000020:00000001:0.0:1362551173.779959:0:4818:0:(lu_object.c:573:lu_object_find_try()) Process leaving (rc=18446612133311027672 : -131940398523944 : ffff88003b6a19d8) <font color="red">==&gt; found it in the hash table</font><br/> 00000020:00000001:0.0:1362551173.779959:0:4818:0:(lu_object.c:625:lu_object_find_at()) Process leaving (rc=18446612133311027672 : -131940398523944 : ffff88003b6a19d8)<br/> 00000020:00000001:0.0:1362551173.779960:0:4818:0:(dt_object.c:253:dt_locate()) Process leaving (rc=18446612133330352192 : -131940379199424 : ffff88003c90f840)<br/> 00000020:00000001:0.0:1362551173.779961:0:4818:0:(dt_object.c:360:dt_reg_open()) Process leaving (rc=18446612133330352192 : -131940379199424 : ffff88003c90f840)<br/> 00000020:00000001:0.0:1362551173.779961:0:4818:0:(lustre_fid.h:375:fid_flatten32()) Process leaving (rc=4261420800 : 4261420800 : fe001f00)<br/> 00000020:00000001:0.0:1362551173.779962:0:4818:0:(dt_object.c:387:dt_store_open()) Process leaving (rc=18446612133330352192 : -131940379199424 : ffff88003c90f840)<br/> 00000004:00020000:0.0:1362551173.779963:0:4818:0:(osd_handler.c:2345:osd_index_try()) dt does not exists<br/> 00000004:00020000:0.0:1362551173.780032:0:4818:0:(mdd_orphans.c:478:orph_index_init()) "PENDING" is not an index! : rc = -20</p> </div></div> <p>so I look back to when this object was created, and I find that the PENDING dt_object was created in osd_prepare() =&gt; llo_local_objects_setup() =&gt; llo_store_create_index() =&gt; llo_create_obj() </p> <div class="panel" style="border-width: 1px;"><div class="panelContent"> <p>00000004:00000001:0.0:1362555707.896340:0:4934:0:(mdt_handler.c:4881:mdt_object_init()) Process leaving (rc=0 : 0 : 0)<br/> 00000004:00000001:0.0:1362555707.896341:0:4934:0:(cmm_object.c:187:cml_object_init()) Process entered<br/> 00000004:00000010:0.0:1362555707.896341:0:4934:0:(mdd_object.c:248:mdd_object_alloc()) kmalloced 'mdd_obj': 136 at ffff88003d75cb40.<br/> 00000004:00000001:0.0:1362555707.896341:0:4934:0:(cmm_object.c:209:cml_object_init()) Process leaving (rc=0 : 0 : 0)<br/> 00000004:00000001:0.0:1362555707.896342:0:4934:0:(mdd_object.c:271:mdd_object_init()) Process entered<br/> 00000004:00000010:0.0:1362555707.896342:0:4934:0:(osd_handler.c:298:osd_object_alloc()) kmalloced 'mo': 192 at ffff88003d75cc00.<br/> 00000004:00000001:0.0:1362555707.896343:0:4934:0:(mdd_object.c:282:mdd_object_init()) Process leaving (rc=0 : 0 : 0)<br/> 00000004:00000001:0.0:1362555707.896343:0:4934:0:(osd_handler.c:444:osd_object_init()) Process entered<br/> 00000004:00000001:0.0:1362555707.896344:0:4934:0:(osd_handler.c:386:osd_fid_lookup()) Process entered<br/> 00000004:00000002:0.0:1362555707.896344:0:4934:0:(osd_handler.c:397:osd_fid_lookup()) osd_oi_lookup <span class="error">&#91;0x200000001:0x7:0x0&#93;</span> returns -2 <font color="red"></font>oi lookup cannot find the obj<br/> 00000004:00000001:0.0:1362555707.896344:0:4934:0:(osd_handler.c:421:osd_fid_lookup()) Process leaving (rc=0 : 0 : 0)<br/> 00000004:00000001:0.0:1362555707.896345:0:4934:0:(osd_handler.c:453:osd_object_init()) Process leaving (rc=0 : 0 : 0)</p> </div></div> <p>check out b2_1 version of osd_oi_lookup</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>osd_oi_lookup</b></div><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (fid_seq(fid) == FID_SEQ_LOCAL_FILE) <span class="code-keyword">return</span> -ENOENT; </pre> </div></div> <p>2.1 does not use oi for local file lookup, that's the root cause.</p> <p>Try to remove the check "if (fid_seq(fid) == FID_SEQ_LOCAL_FILE)" from "osd_oi_lookup()".</p> <p>For the record, my test shows that removing the check works on this scenario, but panics on 2.1.4 formatted system.</p> <p>To clarify, in 2.4 the "PENDING" FID is stored in the OI file, but in 2.1 it is not? And 2.1 does not like this?</p> <p>Is it still possible to get a patch into 2.1.5 to fix this? Is this caused by the initial LFSCK scan added in 2.4 to fix the OI for local objects?</p> <p>In 2.4, the "PENDING" FID is added into local file, but it is NOT in 2.1. We can back port OI scrub related patches (including initial OI scrub) to b2_1, part of the back porting are as following:</p> <p><a href="http://review.whamcloud.com/#change,4620" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4620</a><br/> <a href="http://review.whamcloud.com/#change,4621" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4621</a><br/> <a href="http://review.whamcloud.com/#change,4623" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4623</a><br/> <a href="http://review.whamcloud.com/#change,4624" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4624</a><br/> <a href="http://review.whamcloud.com/#change,4626" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4626</a><br/> <a href="http://review.whamcloud.com/#change,4627" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4627</a><br/> <a href="http://review.whamcloud.com/#change,4628" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4628</a><br/> <a href="http://review.whamcloud.com/#change,4629" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4629</a><br/> <a href="http://review.whamcloud.com/#change,4630" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4630</a></p> <p>I do not think the back porting can be ready in a short time. So the better way is to fix b2_1 with removing some check. If there are other bugs after removing, we can try to resolve them one by one. I think it will be faster than the whole back porting.</p> <p>Well, we should not really crash unfixed 2.1 servers.<br/> and 2.1.5 should be released any moment now too so there's not tiem for any development to fit there.<br/> We need to think up something in 2.4, and fast I think</p> <p>just posted a 2.1 patch to port only necessary ldiskfs based OI implementation at <a href="http://review.whamcloud.com/5731" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5731</a> </p> <p>Will there also be a patch for master? </p> <p>Jodi,</p> <p>No, master does not need a patch, this patch is to make 2.1.4 server OI implementation align to that of master.</p> <p>Can yu please rebase this on top of current b2_1?</p> <p>ok, and done</p> <blockquote><p>just posted a 2.1 patch to port only necessary ldiskfs based OI implementation at <a href="http://review.whamcloud.com/5731" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5731</a></p></blockquote> <p>Here are the test configuration and result:</p> <p>Lustre b2_1 build: <a href="http://build.whamcloud.com/job/lustre-reviews/14375/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/14375/</a><br/> Lustre master build: <a href="http://build.whamcloud.com/job/lustre-master/1369/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1369/</a><br/> Distro/Arch: RHEL6.3/x86_64</p> <p>Clean upgrade and downgrade path: b2_1-&gt;master-&gt;b2_1</p> <p>After downgrading from master to b2_1, mounting the server targets and clients succeeded. However, on the MDS node, "lctl dl" showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@wtm-83 ~]# lctl dl 0 UP mgs MGS MGS 15 1 UP mgc MGC10.10.19.8@tcp ac763461-679d-82b7-e00e-7e3d7f5e6234 5 2 UP lov lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4 3 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 9 4 UP mds mdd_obd-lustre-MDT0000 mdd_obd_uuid-lustre-MDT0000 3 5 IN osc lustre-OST0000-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 6 IN osc lustre-OST0001-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 </pre> </div></div> <p>Dmesg on the MDS node showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: MGS MGS started Lustre: 10437:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from ac763461-679d-82b7-e00e-7e3d7f5e6234@0@lo t0 exp (null) cur 1365015074 last 0 Lustre: MGC10.10.19.8@tcp: Reactivating import Lustre: Enabling ACL Lustre: Enabling user_xattr Lustre: lustre-MDT0000: used disk, loading Lustre: 10441:0:(mdt_lproc.c:416:lprocfs_wr_identity_upcall()) lustre-MDT0000: identity upcall set to /usr/sbin/l_getidentity LustreError: 10441:0:(llog_lvfs.c:199:llog_lvfs_read_header()) bad log / header magic: 0x2e (expected 0x10645539) LustreError: 10441:0:(llog_obd.c:220:llog_setup_named()) obd lustre-OST0000-osc-MDT0000 ctxt 2 lop_setup=ffffffffa0562ca0 failed -5 LustreError: 10441:0:(osc_request.c:4231:__osc_llog_init()) failed LLOG_MDS_OST_ORIG_CTXT LustreError: 10441:0:(osc_request.c:4248:__osc_llog_init()) osc 'lustre-OST0000-osc-MDT0000' tgt 'mdd_obd-lustre-MDT0000' catid ffff880c24f6b8b0 rc=-5 LustreError: 10441:0:(osc_request.c:4250:__osc_llog_init()) logid 0x2:0x0 LustreError: 10441:0:(osc_request.c:4278:osc_llog_init()) rc: -5 LustreError: 10441:0:(lov_log.c:248:lov_llog_init()) error osc_llog_init idx 0 osc 'lustre-OST0000-osc-MDT0000' tgt 'mdd_obd-lustre-MDT0000' (rc=-5) LustreError: 10441:0:(llog_lvfs.c:616:llog_lvfs_create()) error looking up logfile 0x4:0x0: rc -116 LustreError: 10441:0:(llog_obd.c:220:llog_setup_named()) obd lustre-OST0001-osc-MDT0000 ctxt 2 lop_setup=ffffffffa0562ca0 failed -116 LustreError: 10441:0:(osc_request.c:4231:__osc_llog_init()) failed LLOG_MDS_OST_ORIG_CTXT LustreError: 10441:0:(osc_request.c:4248:__osc_llog_init()) osc 'lustre-OST0001-osc-MDT0000' tgt 'mdd_obd-lustre-MDT0000' catid ffff880c24f6b8b0 rc=-116 LustreError: 10441:0:(osc_request.c:4250:__osc_llog_init()) logid 0x4:0x0 LustreError: 10441:0:(osc_request.c:4278:osc_llog_init()) rc: -116 LustreError: 10441:0:(lov_log.c:248:lov_llog_init()) error osc_llog_init idx 1 osc 'lustre-OST0001-osc-MDT0000' tgt 'mdd_obd-lustre-MDT0000' (rc=-116) Lustre: 10566:0:(debug.c:326:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 10567:0:(debug.c:326:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. Lustre: 10437:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from 4447b0f0-2f2f-51f4-cd9f-aa011ef3eb77@10.10.19.17@tcp t0 exp (null) cur 1365015077 last 0 Lustre: 10209:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1365015074/real 1365015074] req@ffff8805fb180800 x1431322036797450/t0(0) o8-&gt;lustre-OST0000-osc-MDT0000@10.10.19.17@tcp:28/4 lens 368/512 e 0 to 1 dl 1365015079 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 10437:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from a8451e2f-8501-14bd-8ab3-88e0df1b7640@10.10.19.26@tcp t0 exp (null) cur 1365015079 last 0 Lustre: 10209:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1365015075/real 1365015075] req@ffff880626de8800 x1431322036797451/t0(0) o8-&gt;lustre-OST0001-osc-MDT0000@10.10.19.26@tcp:28/4 lens 368/512 e 0 to 1 dl 1365015080 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 10437:0:(ldlm_lib.c:952:target_handle_connect()) MGS: connection from a2dec252-dec3-5b33-4395-9b26a06a9dac@10.10.18.253@tcp t0 exp (null) cur 1365015081 last 0 Lustre: lustre-MDT0000: Temporarily refusing client connection from 10.10.18.253@tcp Lustre: lustre-MDT0000: Temporarily refusing client connection from 10.10.18.253@tcp Lustre: lustre-MDT0000: Temporarily refusing client connection from 10.10.18.253@tcp Lustre: lustre-MDT0000: Temporarily refusing client connection from 10.10.18.253@tcp LustreError: 10599:0:(lov_log.c:160:lov_llog_origin_connect()) error osc_llog_connect tgt 1 (-107) LustreError: 10598:0:(mds_lov.c:832:__mds_lov_synchronize()) lustre-OST0000_UUID failed at llog_origin_connect: -107 LustreError: 10598:0:(mds_lov.c:861:__mds_lov_synchronize()) sync lustre-OST0000_UUID failed -107 LustreError: 10598:0:(mds_lov.c:865:__mds_lov_synchronize()) deactivating lustre-OST0000_UUID LustreError: 10599:0:(lov_log.c:160:lov_llog_origin_connect()) Skipped 1 previous similar message LustreError: 10599:0:(mds_lov.c:832:__mds_lov_synchronize()) lustre-OST0001_UUID failed at llog_origin_connect: -107 LustreError: 10599:0:(mds_lov.c:861:__mds_lov_synchronize()) sync lustre-OST0001_UUID failed -107 LustreError: 10599:0:(mds_lov.c:865:__mds_lov_synchronize()) deactivating lustre-OST0001_UUID Lustre: 10454:0:(ldlm_lib.c:952:target_handle_connect()) lustre-MDT0000: connection from d03c55d9-e816-05ed-ff50-0ba89f2504bb@10.10.18.253@tcp t0 exp (null) cur 1365015101 last 0 Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: 2 OST are inactive after 20 seconds, give up </pre> </div></div> <p>Test report is still in Maloo import queue.</p> <p>Here is the test report: <a href="https://maloo.whamcloud.com/test_sessions/f77f5032-9cd5-11e2-802d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/f77f5032-9cd5-11e2-802d-52540035b04c</a></p> <p>So is my reaing right that the test actually failed because all OSCs are in inactive state, so it won't be possible to create any new files on such a filesystem? I do not see any attempts for the test to actually create anything post-downgrade so perhaps it's a case we are missing?</p> <p>I don't know about the test case, but the latest error has something about CATALOGS file changing in 2.4.</p> <p>the CATALOGS write by 2.1 is as follows (logid is i_ino+ __u64 0x0+i_generation)</p> <div class="panel" style="border-width: 1px;"><div class="panelContent"> <ol> <li>od -x /mnt/mds1/CATALOGS<br/> 0000000 0021 0000 0000 0000 0000 0000 0000 0000<br/> 0000020 2a9d 1d4f 0000 0000 0000 0000 0000 0000<br/> 0000040 0022 0000 0000 0000 0000 0000 0000 0000<br/> 0000060 2a9e 1d4f 0000 0000 0000 0000 0000 0000<br/> 0000100 0023 0000 0000 0000 0000 0000 0000 0000<br/> 0000120 2a9f 1d4f 0000 0000 0000 0000 0000 0000<br/> 0000140</li> </ol> </div></div> <p>after 2.4 mounted it, the CATALOGS logic arrays changes to </p> <div class="panel" style="border-width: 1px;"><div class="panelContent"> <ol> <li>od -x /mnt/mds1/CATALOGS<br/> 0000000 0002 0000 0000 0000 0001 0000 0000 0000<br/> 0000020 0000 0000 0000 0000 0000 0000 0000 0000<br/> 0000040 0004 0000 0000 0000 0001 0000 0000 0000<br/> 0000060 0000 0000 0000 0000 0000 0000 0000 0000<br/> 0000100 0006 0000 0000 0000 0001 0000 0000 0000<br/> 0000120 0000 0000 0000 0000 0000 0000 0000 0000<br/> 0000140</li> </ol> </div></div> <p>Without those patches having on-disk change porting backward, the upgrade then downgrade test would be a headache.</p> <blockquote><p>So is my reaing right that the test actually failed because all OSCs are in inactive state, so it won't be possible to create any new files on such a filesystem?</p></blockquote> <p>Right, touching a new file on Lustre client failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># touch /mnt/lustre/file touch: cannot touch `/mnt/lustre/file': Input/output error </pre> </div></div> <blockquote><p>I do not see any attempts for the test to actually create anything post-downgrade so perhaps it's a case we are missing?</p></blockquote> <p>The upgrade/downgrade testing needs to be performed as per wiki page <a href="https://wiki.hpdd.intel.com/display/ENG/Upgrade+and+Downgrade+Testing" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/ENG/Upgrade+and+Downgrade+Testing</a>. As we can see, the data creating/verifying steps are included in the post-downgrade phases. However, these test cases are not covered in one test script currently. The script which detected the issue in this ticket was upgrade-downgrade.sh, which covered extra quotas and OST pools testing along the upgrade/downgrade path.</p> <p>The issue is that while performing this script to do downgrade testing, the extra quotas testing was disabled (due to the new way of setting quotas on master branch), only OST pools testing was covered along the downgrade path, which only verified that the existing files/directories could be accessed and the striping info was correct, but did not create new files.</p> <p>So, in order to cover all of the test cases in the wiki page, we need improve upgrade-downgrade.sh to make the quotas codes work on master branch, and also need run the other two test scripts {clean,rolling}-upgrade-downgrade.sh before the test cases covered by them are added into upgrade-downgrade.sh.</p> <p>This log shows that 2.4 MDT start cannot find the old llog objects, and created new ones in 2.4 way (using llog_osd_ops), which 2.1 code (using llog_lvfs_ops) cannot recognise.</p> <p>Actually, I found that when 2.1 formatted disk upgraded 2.4, all files' size becomes 0, their content are lost.</p> <p>Bobijam, that sounds like a very critical problem. Does conf-sanity test 32 not detect this problem during 2.1 to 2.4 upgrade? Is that problem repeatable?</p> <p>Please file a separate bug for that problem and make it a 2.4 blocker until it is better understood and fixed. </p> <p>this is rather strange.. I do remember Li Wei improved conf-sanity/32 to verify actual data with md5sum.</p> <p>Is it possible that the disk2_1-ldiskfs.tar.bz2 was created with an outdated compatible b2_1 version? I just tested with current b2_1 and master branch, the procedure is </p> <p>1. create a lustre filesystem with b2_1 version<br/> 2. copy /etc/* to this filesystem<br/> 3. umount it<br/> 4. mount the file system with master version, succeeded<br/> 5. 'ls -l' the filesystem, all files are there but all sized 0 with no content.</p> <p>iirc, Li Wei did put a function to create such an image in conf-sanity.sh</p> <p>strangely, I don't even need master version, with b2_1 alone, format &amp; mount &amp; copy files &amp; umount &amp; mount it again, all files become 0.</p> <p>btw, I'm using current b2_1 lustre code with linux-2.6.32-279.22.1.el6 kernel code.</p> <p>Fired a ticket (<a href="https://jira.whamcloud.com/browse/LU-3141" title="umount lost all file contents" class="issue-link" data-issue-key="LU-3141"><del>LU-3141</del></a>) for the 'empty file' issue.</p> <p>it turns out that the 'empty file' issue is not a bug, I was using llmountcleanup.sh which calls 'umount -f' and it does not flush dirty data.</p> <p>I'll push another patch which can pass the upgrade &amp; downgrade test on my VM environment.</p> <p>Wang Di,</p> <p>The upgrade test shows these messages on client side </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> Maloo report: https://maloo.whamcloud.com/test_sessions/ef5c83fe-a2be-11e2-81ba-52540035b04c I found the following error messages in the console log of client wtm-81 after upgrading: LustreError: 33998:0:(lustre_idl.h:705:ostid_to_fid()) bad MDT0 id, 0x6dec:1024 ost_idx:0 LustreError: 33998:0:(lustre_idl.h:705:ostid_to_fid()) bad MDT0 id, 0x6dec:1024 ost_idx:0 LustreError: 33998:0:(lustre_idl.h:705:ostid_to_fid()) bad MDT0 id, 0x6d78:1024 ost_idx:0 LustreError: 33998:0:(lustre_idl.h:705:ostid_to_fid()) Skipped 31 previous similar messages LustreError: 33998:0:(lustre_idl.h:705:ostid_to_fid()) bad MDT0 id, 0x6d6d:1024 ost_idx:0 </pre> </div></div> <p>I noticed that it's from your recent patch, what the issue could it be? Is it potentially harmful?</p> <p>Just update what I found. My local test shows the error message is from lov_attr_get_raid0()</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 00020000:00000001:0.0:1365706828.914809:0:18347:0:(lov_object.c:420:lov_attr_get_raid0()) Process entered 00020000:00020000:0.0:1365706828.914809:0:18347:0:(lustre_idl.h:705:ostid_to_fid()) bad MDT0 id, 0x54:1024 ost_idx:0 00020000:00000002:0.0:1365706828.914810:0:18347:0:(lov_merge.c:76:lov_merge_lvb_kms()) MDT FID [0x0:0x0:0x0] initial value: s=0 m=9223372036854775808 a=9223372036854775808 c=9223372036854775808 b=0 00020000:00000001:0.0:1365706828.914811:0:18347:0:(lov_offset.c:59:lov_stripe_size()) Process entered 00020000:00000001:0.0:1365706828.914811:0:18347:0:(lov_offset.c:74:lov_stripe_size()) Process leaving (rc=370 : 370 : 172) 00020000:00000001:0.0:1365706828.914812:0:18347:0:(lov_offset.c:59:lov_stripe_size()) Process entered 00020000:00000001:0.0:1365706828.914812:0:18347:0:(lov_offset.c:74:lov_stripe_size()) Process leaving (rc=370 : 370 : 172) 00020000:00000002:0.0:1365706828.914813:0:18347:0:(lov_merge.c:110:lov_merge_lvb_kms()) MDT FID [0x0:0x0:0x0] on OST[1]: s=370 m=1365706747 a=1365706747 c=1365706747 b=8 00020000:00000001:0.0:1365706828.914814:0:18347:0:(lov_merge.c:119:lov_merge_lvb_kms()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1365706828.914814:0:18347:0:(cl_object.c:1019:cl_lvb2attr()) Process entered 00000020:00000001:0.0:1365706828.914814:0:18347:0:(cl_object.c:1025:cl_lvb2attr()) Process leaving 00020000:00000001:0.0:1365706828.914815:0:18347:0:(lov_object.c:476:lov_attr_get_raid0()) Process leaving (rc=0 : 0 : 0) </pre> </div></div> <p>while in lov_merge_lvb_kms(), stack variable 'fid' has no effect except for displaying debug messages.</p> <p>WangDi, need your input.</p> <p>Di, it appears that this might relate to <a href="http://review.whamcloud.com/5820" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5820</a> incorrectly converting struct llog_logid to use struct ost_id. The fields in that structure were already used a bit strangely, and I think that the ostid_to_fid() and fid_to_ostid() macros may be mangling the fields in this structure, since they were never really used as OST id/seq/, but rather MDS inode/0/generation.</p> <p>On 1.8 the CATALOGS file looks like oid=ino/ogr=0/ogen=gen:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>0000000 0000000000000021 0000000000000000 0000020 000000001d4f2a9d 0000000000000000 0000040 0000000000000022 0000000000000000 0000060 000000001d4f2a9e 0000000000000000 0000100 0000000000000023 0000000000000000 0000120 000000001d4f2a9f 0000000000000000 </pre> </div></div> <p>On 2.1 (upgraded from 1.8, if it matters) the CATALOGS file looks like oid=ino/oseq=1/ogen=gen:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>000000 0000000000020002 0000000000000001 000010 00000000ec467621 0000000000000000 000020 0000000000020003 0000000000000001 000030 00000000ec467622 0000000000000000 000040 0000000000020004 0000000000000001 000050 00000000ec467623 0000000000000000 000060 0000000000020005 0000000000000001 000070 0000000095595788 0000000000000000 000080 000000000002005b 0000000000000001 000090 000000001ecc9141 0000000000000000 </pre> </div></div> <p>On 2.4 (new) the CATALOGS file looks like (f_seq=1/f_oid=oid/ogen=0):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>000000 0000000000000001 0000000000000002 000010 0000000000000000 0000000000000000 000020 0000000000000001 0000000000000004 000030 0000000000000000 0000000000000000 000040 0000000000000001 0000000000000006 000050 0000000000000000 0000000000000000 </pre> </div></div> <p>I suspect that the latter change is unintentional, since I think it breaks the on-disk and on-wire llog_logid protocol. I'm just running another test with a pre 6794d7654b4c459519a9e6d85ed439c8c594c2e7 build to see what the CATALOGS file looks like.</p> <p>Di, could you please work up a patch that reverts the changes to llog_logid and its usage, and we can see if that fixes this latest issue.</p> <p>Only the latest issue may be related to <a href="https://jira.whamcloud.com/browse/LU-2684" title="convert ost_id to lu_fid for FID_SEQ_NORMAL objects" class="issue-link" data-issue-key="LU-2684"><del>LU-2684</del></a>, not the original one.</p> <p>The pre-<a href="https://jira.whamcloud.com/browse/LU-2684" title="convert ost_id to lu_fid for FID_SEQ_NORMAL objects" class="issue-link" data-issue-key="LU-2684"><del>LU-2684</del></a> CATALOGS file looks like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>000000 000000000000000a 0000000000000001 000010 0000000000000000 0000000000000000 000020 000000000000000c 0000000000000001 000030 0000000000000000 0000000000000000 000040 000000000000000e 0000000000000001 000050 0000000000000000 0000000000000000 </pre> </div></div> <p>so indeed this is an unintentional compatibility breakage with the new code.</p> <p>Bobi: Could you please try this patch? <a href="http://review.whamcloud.com/#change,6034" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,6034</a> It seems lmm_oi is being overwritten by some other threads. As Andreas said it might related with ostid_to_fid for llog object. So this patch will use oi_id/oi_seq directly to identify the log object, so to avoid ostid_to_fid conversion. Could you please try a few times, I guess this problem can not be reproduced often, at least I can not reproduce it locally. Thanks. </p> <p><a href="http://review.whamcloud.com/6034" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6034</a> does not solve the issue. The error message is not from llog handling, it's from stat/ls the file created by 2.1 system.</p> <p>attached is the -1 log.</p> <p>I have an example of ostid here extracted from lov_merge_lvb_kms()</p> <p>LustreError: 8277:0:(lustre_idl.h:705:ostid_to_fid()) bad MDT0 id, 0x51:1024 ost_idx:0<br/> LustreError: 8277:0:(lustre_idl.h:706:ostid_to_fid()) 0x51:0x200000400</p> <p>The second log message format is as follows</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> CERROR(<span class="code-quote">"bad MDT0 id, "</span>DOSTID<span class="code-quote">" ost_idx:%u\n"</span>, POSTID(ostid), ost_idx); CERROR(LPX64<span class="code-quote">":"</span>LPX64<span class="code-quote">"\n"</span>, ostid-&gt;oi.oi_id, ostid-&gt;oi.oi_seq); </pre> </div></div> <p>Hmm, the problem is that in 2.1, we define the lsm/lmm in this</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>struct lov_mds_md_v1 { /* LOV EA mds/wire data (little-endian) */ __u32 lmm_magic; /* magic number = LOV_MAGIC_V1 */ __u32 lmm_pattern; /* LOV_PATTERN_RAID0, LOV_PATTERN_RAID1 */ __u64 lmm_object_id; /* LOV object ID */ __u64 lmm_object_seq; /* LOV object seq number */ __u32 lmm_stripe_size; /* size of stripe in bytes */ __u32 lmm_stripe_count; /* num stripes in use for this object */ struct lov_ost_data_v1 lmm_objects[0]; /* per-stripe data */ }; </pre> </div></div> <p>But lmm_object_seq/lmm_object_id is actually normal MDT FIDS, i.e. lmm_object_id/lmm_object_seq will be f_oid/normal_seq, and when unpack lmm to lsm on 2.4,</p> <p>it will use ostid_le_to_cpu()</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static inline void ostid_le_to_cpu(struct ost_id *src_oi, struct ost_id *dst_oi) { if (fid_seq_is_mdt0(ostid_seq(src_oi))) { dst_oi-&gt;oi.oi_id = le64_to_cpu(src_oi-&gt;oi.oi_id); dst_oi-&gt;oi.oi_seq = le64_to_cpu(src_oi-&gt;oi.oi_seq); } else { fid_le_to_cpu(&amp;dst_oi-&gt;oi_fid, &amp;src_oi-&gt;oi_fid); } } </pre> </div></div> <p>And treat the ostid as normal FID, which cause the problem.</p> <p>Sigh, it seems we do not have better way to convert this special ostid to the real FID.</p> <p><a href="http://review.whamcloud.com/#change,6037" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,6037</a> Bobi: please check this one. Thanks!</p> <p>yes, with <a href="http://review.whamcloud.com/5731" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5731</a> on b2_1 and <a href="http://review.whamcloud.com/6034" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6034</a> and <a href="http://review.whamcloud.com/6037" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6037</a> on master, the downgrade and upgrade test passed with no noise.</p> <p>Change/6034 and 6037 have been merged into <a href="http://review.whamcloud.com/#change,6044" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,6044</a></p> <p>Landed for 2.4</p> <p><a href="http://review.whamcloud.com/#change,5731" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5731</a> hasn't landed on b2_1 yet, need that to be claimed to be fixed.</p> <p>Reducing 2.4 blocker, but keeping open until lands on b2_1.</p> <p>Please also make changes to b1_8, hit the same LBUG after downgrade from the latest tag-2.3.64 to 1.8.9</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: Enabling user_xattr LustreError: 27458:0:(osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed: LustreError: 27458:0:(osd_handler.c:2343:osd_index_try()) LBUG Pid: 27458, comm: llog_process_th </pre> </div></div> <p>Do we support 2.x to 1.8 downgrade?</p> <p>Sorry, I was downgraded the system to b2_1 by mistake, 1.8.9&lt;-&gt;2.4 doesn't have this issue.</p> <p>landed on b2_1 for 2.1.6</p> <p>Yes, I think that we should support 1.8.x-&gt;2.4-&gt;1.8.9-wc1 upgrade &amp; downgrade. Of course, w/o dirdata and large_xattr, etc. used.</p> <p>I also think that 2.3-&gt;2.4-&gt;2.3 upgrade &amp; downgrade should be supported. But this bug occurs there as well. Is the appropriate response to reopen this ticket and mark it as affecting 2.3 (and I suppose 2.2 too)? Or should we make a clone?</p> <p>Also, I think it is a pretty bad story that we'd have to get a fix onto b2_3 to make the upgrade/downgrade scenario work. Because even if we get the existing patch landed to b2_3 you can't revert to 2.3.0.</p> <p>I hit this issue when downgrade from 2.5 to 2.3.0, will create a new issue for that.</p> Duplicate LU-3574 Related LU-2684 LU-3267 Development Rank 1|hzvk1b: Rank (Obsolete) 6970 Severity