https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5145 Lustre <p>Hi,</p> <p>CEA had 3 consecutive crashes of MDS server due to a corruption of kernel slab, and these crashes were concomitant to Robinhood startup.</p> <p>In fact in the Lustre Changelogs we can see a line like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>4281746500 01CREAT 17:47:33.690285184 2014.03.25 0x0 t=[0x22cb19e89:0x1f9c9:0x0] p=[0x22cb19e89:0x1f9c8:0x0] GRANDEURS_CENTREES </pre> </div></div> <p>And this is because Robinhood tried to process this Changelog entry that we crashed the MDS. Indeed, looking at the MDT with debugfs, we found out that this file was under the /OBJECTS directory (it used to be a regular file, that was moved here by Lustre for an unkown reason), and that its link EA was containing zero fid (link = "df f1 ea 11 00 00 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 " (24)).</p> <p>This issue can be reproduced by doing the following on a Lustre filesystem originally formatted with Lustre 2.1, and upgraded to Lustre 2.4:</p> <ul class="alternate" type="square"> <li>on a Lustre client :</li> </ul> <ol> <li>touch &lt;lustre dir&gt;/file</li> <li>lfs path2fid &lt;lustre dir&gt;/file</li> </ol> <ul class="alternate" type="square"> <li>stop the file system and mount the MDT with ldiskfs</li> <li>move the file from &lt;mdt ldiskfs&gt;/ROOT/file to &lt;mdt ldiskfs&gt;/OBJECTS directory</li> <li>with setfattr change the link EA to remove all links :</li> </ul> <ol> <li>setfattr -n trusted.link -v 0xdff1ea110000000018000000000000000000000000000000 &lt;mdt ldiskfs&gt;/OBJECTS/file</li> </ol> <ul class="alternate" type="square"> <li>umount ldiskfs MDT and restart the file system</li> <li>on a Lustre client :</li> </ul> <ol> <li>lfs fid2path &lt;lustre dir&gt; &lt;fid obtained at first step&gt;</li> </ol> <ul class="alternate" type="square"> <li>the MDS server crashes</li> </ul> <p>The crash can be avoided with the patch from <a href="https://jira.whamcloud.com/browse/LU-3474" title="MDS LBUG on unlink?" class="issue-link" data-issue-key="LU-3474"><del>LU-3474</del></a> at <a href="http://review.whamcloud.com/10464" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10464</a> .<br/> But the scenario at CEA is worst that the error case found by Andreas (lfs fid2path on old IGIF FIDs) because here it crashes the MDS.</p> <p>So at first we would need this patch to be landed to b2_4, or at least to know if this patch is suitable for use in production with Lustre 2.4.3.</p> <p>Secondly, given that a file system at CEA has more than 5,000 files in the OBJECTS directory, we are wondering:<br/> a) why some regular files are moved to OBJECTS directory? what is the Lustre mechanism leading to this?<br/> b) why link EA contains zero fid when files are moved to OBJECTS directory?<br/> c) why moving files to OBJECTS directory generates an entry in Lustre Changelogs?</p> <p>TIA,<br/> Sebastien.</p> LU-5145

kernel slab data memory coruption on MDS due to a file with truncated link extended attribute

Bug Critical Resolved Duplicate Bruno Faccini Sebastien Buisson Thu, 5 Jun 2014 12:48:35 +0000 Wed, 6 May 2020 23:25:25 +0000 Wed, 6 May 2020 23:25:25 +0000 Lustre 2.4.3 0 9 <p>Hello Sebastien,<br/> Thanks for this report, but I have more comments/questions in mind :<br/> a) I agree, we need to find this.<br/> b) what let's you think that link EA fid/name zeroing occurs during the "move" to OBJECTS directory ?<br/> c) similarly, why do you think this "move" is responsible of an entry in Lustre ChangeLogs ? Could it not be possible that the entry comes from the earlier file creation in the name-space and then it has been later used/referenced by RobinHood, finally causing the LBUG ?</p> <p>Last, since you identified that patch from <a href="https://jira.whamcloud.com/browse/LU-3474" title="MDS LBUG on unlink?" class="issue-link" data-issue-key="LU-3474"><del>LU-3474</del></a> at <a href="http://review.whamcloud.com/10464" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10464</a>, that unfortunately was missing to be merged in b2_4 until recently, can we decrease the priority of this ticket ?</p> <p>Sebastien, you speak about a corruption of Kernel slabs, can you better detail this ? Do you have at least a panic stack ?</p> <p>Fan Yong, just in case, do you have any idea why/how files can be "moved" to OBJECTS directory ? </p> <p>Hey Bruno!</p> <p>a) lfsck was run on the file system, but many files under /OBJECTS directory have a creation date later that the time when lfsck was run. So lfsck is not responsible for this.<br/> b) indeed, EA fid zeroing could have occurred before. But all files under /OBJECTS have a link EA containing zero fid, so both events must be related.<br/> c) I will ask people on Site if the entry in the Changelogs corresponds to the initial regular file creation, or if it matches its move to /OBJECTS. But as Robinhood consumes Changelog entries as they appear, it would be surprising if the entry corresponds to its initial creation, as it could have occurred a long time before its move to /OBJECTS.</p> <p>Concerning the ticket's priority, I would need to know if the patch at <a href="http://review.whamcloud.com/10464" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10464</a> can be used in production before lowering priority.</p> <p>Sebastien.</p> <p>Here is the information from the crash dump, get by Antoine.</p> <p>Hi Bruno <br/> Agree with Sebastien the patch <a href="http://review.whamcloud.com/10464" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10464</a> could resolve the issue <br/> But for me that's not enough and I attach my full crash analyze report with my proposal fix<br/> to fix definitivly all corupt strcpy risk ...<br/> I confirm also that the CREAT changelog entry is the entry when the file was created the 2014.03.25<br/> And I don't know when the file was moved on the OBJECTS directory<br/> Antoine</p> <p>I verified that the 10464 patch is matching the fix that was landed for 2.5.0 (in 2.4.52, just after b2_4 was branched off master). I believe the patch should be safe for your production use in 2.4 (and we are planning to land it for the next 2.4.x release).</p> <p>As for files in OBJECTS, that is confusing since this directory should only be used for nameless objects such as the quota admin files, and such. Regular files should not end up there. It appears almost as if the OBJECTS directory was being confused with the PENDING directory, for files that are open but unlinked. In that case, it would make sense that the "link" xattr has no links anymore, since there are no names for this file anymore. If possible, could you please check the FIDs on both the /OBJECTS and /PENDING to see if they conflict? You can do this while the MDT is mounted using:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>debugfs -c -f /dev/stdin /dev/\{mdsdev\} &lt;&lt;-EOF stat /OBJECTS stat /PENDING EOF </pre> </div></div> <p>I will check, I just can add that all files name in OBJECTS has <br/> the format &lt;hex inode file number&gt;:&lt;hex gen number of ?&gt;<br/> and some of them have empty link attr but some other have <br/> a good link attribute with good informations but each time I check<br/> the link parent fid does not exist anymore. and some other was symlink files.</p> <p>Antoine, thanks for your help+work on this !!<br/> Also, I will have a look to your enhanced fix proposal.</p> <p>Bruno, Here is a badly link example that could reproduce the corrupt with the patch 10464 </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>setfattr -n trusted.link -v 0xdff1ea11000000002f00000000000000000000000000000000000000000200002b10000000010000000066696c6531 &lt;mdt ldiskfs&gt;/OBJECTS/file </pre> </div></div> <p>as you know, never trust data on disk <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> </p> <p>From the file name format, it seems more like that those files have been "created" under b2_1, but not been "moved" to /OBJECTS. Have you ever downgraded to b2_1 after upgraded to b2_4?</p> <p>Generally, the files under /OBJECTS should not be visible to client, and they should not be found in the Changelog, then whether it has valid linkEA will not affect the Robinhood. So would you please to verify whether those files under the /OBJECTS are users' normal files or not?</p> <p>Sorry for the delay</p> <p>add debugfs trace for OBJECTS and PENDING stat information in debugfs.pdf </p> <p>answer about lustre downgrade : No, we never downgrade own lustre distribution <br/> between 2.4 and 2.&amp;. the fs was created with <br/> lustre 2.<span class="error">&#91;01&#93;</span> in Marsh 2012 and after wa have upgrade in 2.4.2 <br/> in Marsh 17th and after we have upgraded in 2.4.3.</p> <p>The crtime of all user file in /OBJECTS have a date between Mars 19th<br/> and April 6th. Around April 6th we decide to stop OI_scrub. Is it<br/> possible that OI-scrub could put file on /OBJECTS directories ?</p> <p>OI scrub will only update the OI mapping files, it may also move the orphans from /lost+found to /O/xxxx/ for OST device. It will not change other directory structure.</p> <p>Hi Bruno,</p> <p>Did you have the opportunity to look at the enhanced fix proposal from Antoine?</p> <p>Cheers,<br/> Sebastien.</p> <p>Hello Antoine and Sebastien, I spent some time to review the enhanced patch version (provided at the end of trace_debug_lascaux110_corrupt_slab_mdt_get_info.txt attachment for this ticket) from Antoine vs the current one for <a href="https://jira.whamcloud.com/browse/LU-3474" title="MDS LBUG on unlink?" class="issue-link" data-issue-key="LU-3474"><del>LU-3474</del></a> (ie, my patch at <a href="http://review.whamcloud.com/6772" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6772</a>).</p> <p>Basically and as far as I understand, Antoine's version has 2 additional hunks :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>@@ -5688,7 +5688,7 @@ static int mdt_path_current(struct mdt_t mdt_obj = mdt_object_find(info-&gt;mti_env, mdt, &amp;pli-&gt;pli_fids[pli-&gt;pli_fidcount]); if (IS_ERR(mdt_obj)) - GOTO(out, rc = PTR_ERR(mdt_obj)); + GOTO(out, rc = -ENOENT); if (mdt_object_remote(mdt_obj)) { mdt_object_put(info-&gt;mti_env, mdt_obj); GOTO(remote_out, rc = -EREMOTE); </pre> </div></div> <p>and</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>@@ -5726,6 +5726,8 @@ static int mdt_path_current(struct mdt_t ptr -= tmpname-&gt;ln_namelen; if (ptr - 1 &lt;= pli-&gt;pli_path) GOTO(out, rc = -EOVERFLOW); + if (tmpname-&gt;ln_namelen &lt;= 0) + GOTO(out, rc = -ENOENT); strncpy(ptr, tmpname-&gt;ln_name, tmpname-&gt;ln_namelen); *(--ptr) = '/'; </pre> </div></div> <p>both in mdt_path_current()/mdt_handler.c routine/file.</p> <p>My opinion, and you 2 may want to discuss it further, is that with the common patch part for our 2 changes, which now enables correct error from mdt_links_read()/linkea_init() return+handling, the second additional hunk is not needed since due to leh_reccount being 0 in a link-EA with no parent-fid, linkea_init() will return ENODATA that will now be correctly forwarded by mdt_links_read() that will be handled in mdt_path_current() to not continue further+wrongly in trying to decode non-existing Link-EA entry, which has leaded to the unexpected behavior you have encountered (memory corruption due to a negative length passed to strcpy() when trying to extract parent-FID from un-exiting Link-EA entry).</p> <p>Concerning the 1st additional hunk, and we may already have discussed about it off-line ..., but I don't understand the reason why you want to unconditionally return -ENOENT upon mdt_object_find() error return.</p> <p> I do this -ENOENT return because in this case, I want to be sure that<br/> the return will be not -EAGAIN, maybe that is not a good idea, My inspiration<br/> come from the mdt_object_find() call on mdt_fid2path() function :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>"lustre/mdt/mdt_handler.c" : 5814 obj = mdt_object_find(info-&gt;mti_env, mdt, &amp;fp-&gt;gf_fid); 5815 if (obj == NULL || IS_ERR(obj)) { 5816 CDEBUG(D_IOCTL, "no object "DFID": %ld\n", PFID(&amp;fp-&gt;gf_fid), 5817 PTR_ERR(obj)); 5818 RETURN(-EINVAL); 5819 } </pre> </div></div> <p> You can remove this part of my patch ...</p> <p> Did you test a fid2path on a file modified by my proposal command the 06/Jun/14 9:18 ?<br/> in that case the lee_reclen is zero ... and in this case the only way to protect the strcpy<br/> is to verify ln_namelen is no equal to zero</p> <p> You miss this part of my patch that also could useful</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>@@ -5701,7 +5701,7 @@ static int mdt_path_current(struct mdt_t rc = mdt_links_read(info, mdt_obj, &amp;ldata); mdt_object_put(info-&gt;mti_env, mdt_obj); if (rc != 0) - GOTO(out, rc = PTR_ERR(buf)); + GOTO(out, rc); </pre> </div></div> <p>Fixed with patch <a href="https://review.whamcloud.com/6772" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/6772</a> in 2.5.0 and 2.4.1.</p> Related LU-3474 Development End date Wed, 27 Aug 2014 12:48:35 +0000 Rank 1|hzwnr3: Rank (Obsolete) 14196 Severity Start date Thu, 5 Jun 2014 12:48:35 +0000