https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3139 Lustre <p>When starting lustre on Sequoia's MDS/MGS, it is hitting the following assertion:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2013-04-09 16:46:16 Lustre: lsv-MDT0000: Will be in recovery for at least 5:00, or until 2 clients reconnect. 2013-04-09 16:46:19 Lustre: lsv-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. 2013-04-09 16:46:58 LustreError: 11-0: lsv-OST000c-osc-MDT0000: Communicating with 172.20.20.12@o2ib500, operation ost_connect failed with -16. 2013-04-09 16:47:38 LustreError: 11-0: lsv-OST000b-osc-MDT0000: Communicating with 172.20.20.11@o2ib500, operation ost_connect failed with -16. 2013-04-09 16:47:38 LustreError: Skipped 9 previous similar messages 2013-04-09 16:48:03 LustreError: 11-0: lsv-OST0007-osc-MDT0000: Communicating with 172.20.20.7@o2ib500, operation ost_connect failed with -16. 2013-04-09 16:48:03 LustreError: Skipped 9 previous similar messages 2013-04-09 16:48:24 Lustre: lsv-OST0001-osc-MDT0000: Connection restored to lsv-OST0001 (at 172.20.20.1@o2ib500) 2013-04-09 16:48:24 Lustre: lsv-OST0003-osc-MDT0000: Connection restored to lsv-OST0003 (at 172.20.20.3@o2ib500) 2013-04-09 16:49:44 LustreError: 18017:0:(osp_precreate.c:496:osp_precreate_send()) ASSERTION( lu_fid_diff(fid, &amp;d-&gt;opd_pre_used_fid) &gt; 0 ) failed: reply fid [0x100090000:0x4c00:0x0] pre used fid [0x100090000:0x16bec0:0x0] 2013-04-09 16:49:44 LustreError: 18017:0:(osp_precreate.c:496:osp_precreate_send()) LBUG </pre> </div></div> <p>This is an x86_64 server with ppc64 clients. Lustre versions 2.3.63-3chaos and 2.3.63-4chaos.</p> <p>Seeing some vague similarity with <a href="https://jira.whamcloud.com/browse/LU-2895" title="recovery-small 24a: osp_precreate_get_fid(): ASSERTION( lu_fid_diff(&amp;d-&gt;opd_pre_used_fid, &amp;d-&gt;opd_pre_last_created_fid) &lt; 0 ) failed: next fid [0x0:0x1:0x0] last created fid [0x0:0x1:0x0]" class="issue-link" data-issue-key="LU-2895"><del>LU-2895</del></a>, we applited the patch from that issue with no improvement. But this assertion is in a different function so not necessarily surprising.</p> LU-3139

osp_precreate_send()) ASSERTION( lu_fid_diff(fid, &d->opd_pre_used_fid) > 0 ) failed

Bug Critical Resolved Fixed Niu Yawei Christopher Morrone LB sequoia topsequoia Wed, 10 Apr 2013 00:06:31 +0000 Thu, 24 Apr 2014 19:17:07 +0000 Mon, 22 Apr 2013 01:50:59 +0000 Lustre 2.4.0 0 9 <p>Niu</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Any suggestions? Sequoia's filesystem can't be started currently. We need a quick solution.</p> <p>Alex is looking into this.</p> <p>Christopher, would it be possible to start with full debug and attach the log here, please?</p> <p>I uploaded ftp.whamcloud.com/uploads/<a href="https://jira.whamcloud.com/browse/LU-3139" title="osp_precreate_send()) ASSERTION( lu_fid_diff(fid, &amp;d-&gt;opd_pre_used_fid) &gt; 0 ) failed" class="issue-link" data-issue-key="LU-3139"><del>LU-3139</del></a>.tar.gz which contains several debug logs from the MDS. We had panic_on_lbug disabled and hit the assertion multiple times.</p> <p>We see some suspicious-looking Lustre messages related to object precreation on the OSTs. When running 2.3.63, the OSTs all log messages like these:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Apr 9 16:50:06 vesta3 kernel: Lustre: lsv-OST0003: Slow creates, 2304/1482096 objects created at a rate of 46/s Apr 9 16:50:06 vesta1 kernel: Lustre: lsv-OST0001: Slow creates, 2176/1474192 objects created at a rate of 43/s Apr 9 16:50:06 vesta4 kernel: Lustre: lsv-OST0004: Slow creates, 2304/1478224 objects created at a rate of 46/s Apr 9 16:50:08 vesta2 kernel: Lustre: lsv-OST0002: Slow creates, 2304/1472688 objects created at a rate of 46/s Apr 9 16:50:10 vesta7 kernel: Lustre: lsv-OST0007: Slow creates, 2048/1482448 objects created at a rate of 40/s Apr 9 16:50:12 vesta6 kernel: Lustre: lsv-OST0006: Slow creates, 2176/1482448 objects created at a rate of 43/s Apr 9 16:50:12 vesta5 kernel: Lustre: lsv-OST0005: Slow creates, 2048/1482320 objects created at a rate of 40/s </pre> </div></div> <p>Which makes us wonder why they seem to be trying to precreate ~1.5 million objects. Perhaps the FID of the last created object is not being honored, causing it to restart from zero?</p> <p>Perhaps related, the OSTs also seem to be requesting very large grants:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Apr 9 16:49:14 vesta3 kernel: LustreError: 5369:0:(ofd_grant.c:605:ofd_grant()) lsv-OST0003: client lsv-OST0003_UUID/ffff880809acf000 requesting &gt; 2GB grant 3035332608 Apr 9 16:49:15 vesta1 kernel: LustreError: 5386:0:(ofd_grant.c:605:ofd_grant()) lsv-OST0001: client lsv-OST0001_UUID/ffff8807f6423400 requesting &gt; 2GB grant 3019145216 Apr 9 16:49:15 vesta4 kernel: LustreError: 5350:0:(ofd_grant.c:605:ofd_grant()) lsv-OST0004: client lsv-OST0004_UUID/ffff8807f90d1800 requesting &gt; 2GB grant 3027402752 Apr 9 16:49:16 vesta2 kernel: LustreError: 5375:0:(ofd_grant.c:605:ofd_grant()) lsv-OST0002: client lsv-OST0002_UUID/ffff8807f8a9ac00 requesting &gt; 2GB grant 3016065024 Apr 9 16:49:18 vesta7 kernel: LustreError: 5496:0:(ofd_grant.c:605:ofd_grant()) lsv-OST0007: client lsv-OST0007_UUID/ffff8807f7d64800 requesting &gt; 2GB grant 3036053504 Apr 9 16:49:21 vesta6 kernel: LustreError: 5487:0:(ofd_grant.c:605:ofd_grant()) lsv-OST0006: client lsv-OST0006_UUID/ffff8807f7404c00 requesting &gt; 2GB grant 3036053504 Apr 9 16:49:21 vesta5 kernel: LustreError: 5557:0:(ofd_grant.c:605:ofd_grant()) lsv-OST0005: client lsv-OST0005_UUID/ffff880808ccf800 requesting &gt; 2GB grant 3035791360 </pre> </div></div> <p>Finally, when we bring the OSTs back up running the old Lustre version 2.3.58, we see messages like these.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Apr 10 12:50:42 vesta44 kernel: Lustre: lsv-OST002c: deleting orphan objects from 1490242 to 1490307 Apr 10 12:50:42 vesta43 kernel: Lustre: lsv-OST002b: deleting orphan objects from 1489633 to 1489699 Apr 10 12:50:42 vesta72 kernel: Lustre: lsv-OST0048: deleting orphan objects from 1490721 to 1490787 Apr 10 12:50:42 vesta29 kernel: Lustre: lsv-OST001d: deleting orphan objects from 1490592 to 1490659 Apr 10 12:50:42 vesta78 kernel: Lustre: lsv-OST004e: deleting orphan objects from 1490080 to 1490147 Apr 10 12:50:42 vesta41 kernel: Lustre: lsv-OST0029: deleting orphan objects from 1490401 to 1490467 Apr 10 12:50:42 vesta25 kernel: Lustre: lsv-OST0019: deleting orphan objects from 1490337 to 1490403 </pre> </div></div> <p>Note that the ranges roughly coincide with the upper bound of the 'Slow create' messages quoted above.</p> <p>I wonder if this is related to the changes for fid on OST. In particular I am looking at <a href="http://review.whamcloud.com/#change,4324" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4324</a> which seems to have changed the on-disk representation of the last-used object id on the OST.</p> <p>Hi, Ned, the fs is upgraded from 2.3.58? Then I agree with you that the changes of fid-on-ost could be the culprit. It'll be helpful if there are full logs on OST.</p> <p>Hmm, with patch 4324, it will use </p> {seq, 0, 0} <p> as the fid of last_rcvd, I guess original OSD-ZFS does not map this zero oid FID to the last_rcvd file. We probably need another special handling here. Alex, Could you please confirm?</p> <p>If the systemm is upgraded from 2.3.58 to 2.3.63, I think the LASSERT was probably triggered as following:</p> <p>1.MDT read the correct last_used_fid 1490242 (or some very large number) from lov_objid, and use it to do orhpan cleanup;</p> <p>2.On OST side, OST got the incorrect last_oid 0 from disk (because of fid-on-ost changes, it failed to locate old last_rcvd), so orhpan cleanup will try to recreate million objects;</p> <p>3.The million objects re-creation breaks in the middle (see the "Slow creates.." message), and returned MDT with the created number 2304 (or some small value);</p> <p>4.MDT found that returned last_fid is smaller than current last_used_fid, so still keep using the last_used_fid (the larger one) to do precreate;</p> <p>5.The current last_oid is still a very small value, OST start million pre-creation again, and it should also break in the middle ("Slow creates..."), and return the last created fid (0x4c00 or something similar);</p> <p>6.The assert triggered on MDT, because the last fid returned from OST is still much smaller than opd_pre_used_fid (the correct one).</p> <p>Niu's description seem to be correct.. and we can do something like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">if</span> (zap lookup in OI failed) { <span class="code-keyword">if</span> (fid_idif() &amp;&amp; seq==FID_SEQ_OST_MDT0 &amp;&amp; oid==0) lookup {FID_SEQ_LOCAL_FILE; OFD_GROUP0_LAST_OID} in OI } </pre> </div></div> <p>though all the numbers/names above should be checked twice..</p> <p>the ideal "solution" would be to reformat, but I'm not sure this is possible.</p> <p>for a reference, we were using:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lu_local_obj_fid(&amp;info-&gt;fti_fid, OFD_GROUP0_LAST_OID + group); </pre> </div></div> <p>in OFD to access last_id for group0, where</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> inline void lu_local_obj_fid(struct lu_fid *fid, __u32 oid) 230 { 231 fid-&gt;f_seq = FID_SEQ_LOCAL_FILE; 232 fid-&gt;f_oid = oid; then 221 OFD_GROUP0_LAST_OID = 20UL, 422 FID_SEQ_LOCAL_FILE = 0x200000001ULL, </pre> </div></div> <p>so, it should be </p> {FID_SEQ_LOCAL_FILE; 20} <p> as OFD_GROUP0_LAST_OID is not defined now.</p> <p>hmm, this is not quite right as new object to track last_id with oid=0 has been created already.. I guess instead we should lookup OFD_GROUP0_LAST_OID first if osd_fid_lookup() is called for </p> {FID_SEQ_OST_MDT0; 0} <p> ?</p> <p>another (hopefully not fatal) issue is about lots of orphans we just created. luckily, creation rate wasn't great..</p> <blockquote> <p>hmm, this is not quite right as new object to track last_id with oid=0 has been created already.. I guess instead we should lookup OFD_GROUP0_LAST_OID first if osd_fid_lookup() is called for </p> <div class="error"><span class="error">Unknown macro: {FID_SEQ_OST_MDT0; 0}</span> </div> <p> </p></blockquote> <p>Indeed. I'm wondering if there are any tools for zfs which can copy the old LAST_ID into new </p> {seq, 0, 0} <p>? then we could probably avoid above extra checking (I assume there isn't any other system in the world needs such checking). Of course, it would be great if the system is possible to be reformat.</p> <p>AFAIU, yes, it <b>usually</b> can be mounted with ZPL. but.. this may not work for the old filesystem as compatibility with ZPL was implemented just before the landing in September, iirc.</p> <p>We can mount it with ZPL. There is some strange behavior like . or .. missing or showing up twice, or incorrect hard link counts. But we can read/write/open/close local objects like LAST_ID, last_rcvd, lov_objid, etc.</p> <p>Alex, BTW this filesystem will not be long-lived due to the risk of these on-disk incompatibilities. We will provide a newly-formatted filesystem for users that will coexist in the same zpools as this legacy one. We just need to be able to mount the legacy one under 2.3.63+ long enough for users to migrate their data.</p> <p>This is good news then. I'd suggest to: take a snapshot for safety, then ..</p> <p>mount with ZPL, check file /oi.1/0x200000001:0x14:0x0 and check its content, it should be 8byte length and contain a number close to 0x16bec0 (last id used on MDS).</p> <p>the new file should be in /O/0/d0/0 - it should be 8byte too and the number much less than 0x16bec0, close to the first number you saw last in<br/> a message like: Apr 9 16:50:12 vesta5 kernel: Lustre: lsv-OST0005: Slow creates, 2048/1482320 objects created at a rate of 40/s</p> <p>I think it should be enough to write the content from the old file (/oi.1/0x200000001:0x14:0x0) into the new one (/O/0/d0/0)</p> <p>Niu, Di, could you please confirm this suggestion is sane?</p> <p>Alex, the oi.* directories are mostly empty through the ZPL. /oi.1/0x200000001:0x14:0x0 doesn't exist , but /O/0/LAST_ID and /O/0/d0/0 are there with contents as you describe. So we should use the LAST_ID file instead, correct?</p> <p>Yes, LAST_ID should be used.</p> <p>Okay, we'll schedule a time to try out this fix. It will probably be sometime next week.</p> <p>Niu, Alex, Di, can you think of any other on-disk format changes that may bite us after this one? I don't want to get into a state where we can't mount the filesystem under any version of Lustre.</p> <p>Ned, there was a change with <a href="http://review.whamcloud.com/5820" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5820</a> (<a href="https://jira.whamcloud.com/browse/LU-2684" title="convert ost_id to lu_fid for FID_SEQ_NORMAL objects" class="issue-link" data-issue-key="LU-2684"><del>LU-2684</del></a>) that affects the MDS FID storage in the LOV EA. This <em>shouldn't</em> affect normal Lustre operation, but there is a bit of churn in that code right now (e.g. <a href="https://jira.whamcloud.com/browse/LU-3152" title="test_27z did not consider fid on OST." class="issue-link" data-issue-key="LU-3152"><del>LU-3152</del></a>, <a href="https://jira.whamcloud.com/browse/LU-2888" title="After downgrade from 2.4 to 2.1.4, hit (osd_handler.c:2343:osd_index_try()) ASSERTION( dt_object_exists(dt) ) failed" class="issue-link" data-issue-key="LU-2888"><del>LU-2888</del></a>) that may affect upgraded filesystems and it would probably be better to wait until that issue is resolved.</p> <p>We went ahead with the proposed workaround for one affected filesystem (lscratchv, used by vulcan). We were able to bring it up under Lustre 2.3.63 without hitting this bug.</p> <p>We will do the same for the legacy Sequoia filesystem (lscratch1) tomorrow. Sequoia is already mounting a new filesystem formatted using Lustre 2.3.63, but we want to mount the old one read-only to allow data migration. I'll report back on how it goes tomorrow.</p> <p>Thanks for the update Ned. I have dropped the priority slightly to reflect that this is still an important support issue but is not a general blocker for the release itself.</p> <p>The update of the sequoia filesystem was successful. If no work is planned for adding compatibility code, I think we can close this issue.</p> <p>The sequoia system is fixed, and no plan to add compatibility code for 2.3.58 &lt;-&gt; 2.3.63.</p> <p>I just hit this same bug on 2.5.1 branch. This happened on a newly formated file system.</p> <p>Actually this is more likely <a href="https://jira.whamcloud.com/browse/LU-4653" title="Hit LBUG ASSERTION( fid_seq(fid1) == fid_seq(fid2) ) failed after upgrade OST from 2.5.0 to 2.6" class="issue-link" data-issue-key="LU-4653"><del>LU-4653</del></a></p> Development Rank 1|hzvnin: Rank (Obsolete) 7623 Severity