https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10248 Lustre <p>MDT should update PFID information to OST objects after layout swap, split, and merge. However, a protocol should be defined to make this correct. Fanyong proposed to use synchronous OUT setxattr. However, I tend to think it would be better to use llog as SETATTR and DESTROY RPC.</p> <p>This problem has been existing for really long time since Lustre 2.4. LFSCK could fix this problem but it's better to have a mechanism to solve this kind of problem because there will be more and more similar problems.</p> LU-10248

Need to update PFID of OST objects after layout change

Technical task LU-9771 Minor Resolved Fixed WC Triage Jinshan Xiong FLR Thu, 16 Nov 2017 07:09:18 +0000 Fri, 25 Jan 2019 19:11:06 +0000 Tue, 19 Dec 2017 16:42:50 +0000 Lustre 2.11.0 Lustre 2.11.0 0 10 <p>LFSCK can handle the inconsistent PFID EA. But it is impossible to run LFSCK every time when merge layout for new FLR.</p> <p>On the other hand, compared with other operations, the operation of merge layout for FLR is very rare, so the performance is not big issue, as long as we can guarantee the system consistency, it will be enough. And if such operation become quite common in the future, we can consider other better solution. So currently, follow the lod_replace_parent_fid() logic can resolve most of the issue.</p> <p>saying "to use llog" is not very informative as llog can be used (and already is) in many different ways.</p> <p>Alex - after adding the restrictions like MDT-&gt;OST communication plus SETATTR and DESTROY like RPC, it won't leave you much confusion.</p> <p>I think what we need is a mechanism here to perform it in a high efficient way. Using OUT SETXATTR is more like a workaround solution. The current way of handing SETATTR and DESTROY is a pretty good framework and we just need to extend it. Also this framework will be enhanced to pack multiple records in a single RPC, which will lead to better performance.</p> <p>I disagree, OUT is much more flexible interface. if you really want to find a good solution lets start from the requirements.<br/> probably you missed that but OUT <em>can</em> batch stuff from the beginning.</p> <p>When I looked at the code of OUT, I saw the operation of replacing PFID is already there as <tt>lod_replace_parent_fid()</tt>, probably I can just use it.</p> <p>Therefore, before swapping or merging layout, I will just declare and call xattr set XATTR_NAME_FID to change PFID. Is there any side effect of it? For example, local trans fails but remote operation succeeds, which will result in inconsistency, or similar fancy stuff like that.</p> <p>Alex - If I understand it correctly, OUT can only aggregate the updates that belong to the same transaction. Since the SETATTR and DESTROY mechanism is running outside of transaction, potentially it can pack whatever records from LLOG, as long as those records are sending to the same target device, the only limitation would be the size of RPC.</p> <p>I don't have any expertise on this area of the code, probably I'm seeing this issue on the surface.</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30137" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30137</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10248" title="Need to update PFID of OST objects after layout change" class="issue-link" data-issue-key="LU-10248"><del>LU-10248</del></a> mdd: set PFID for swap and merge layout<br/> Project: fs/lustre-release<br/> Branch: flr<br/> Current Patch Set: 1<br/> Commit: 9e7004ecc1a6ad9e8ec8b2259a6b91021d7c520c</p> <p>no, one can send few different transactions (each composed of many updates) within a single RPC.</p> <p>The patch in this ticket will be landed to 2.11 as a bug fix</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30292" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30292</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10248" title="Need to update PFID of OST objects after layout change" class="issue-link" data-issue-key="LU-10248"><del>LU-10248</del></a> mdd: set PFID for swap and merge layout<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 824b7b4d9ccfb6166a042ef06c340bef1ddfcd4f</p> <p>It looks like we hit this bug at NASA, triggered simply by running lfs_migrate. (possibly complicated by the migration of files with multiple hard links) If the issue is that easy to trigger, shouldn't this patch be pulled in to 2.10.x LTS branch?</p> <p>For those of us that hit the bug before the patch, precisely what lfsck args are needed to correct the problem? Something like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl lfsck_start -M lfstest-MDT0000 -c on -C on -o -r </pre> </div></div> <p>  (or is "-A" or "-t layout" or "-t namespace" also required?)</p> <p>Also, in the test-framework script for the patch in this LU, I see what seems to be another way to trigger the fix, but can't find any documentation on it:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$LCTL set_param -n obdfilter.${FSNAME}-OST*.lfsck_verify_pfid=1 </pre> </div></div> <p>Nathan, could you describe which problem you are hitting? Did you hit the problem in <a href="https://jira.whamcloud.com/browse/LU-9941" title="lsm_is_composite() isn&#39;t right" class="issue-link" data-issue-key="LU-9941"><del>LU-9941</del></a> when running "lfs migrate", or something else? This ticket doesn't really have any userspace visible symptoms unless there is already MDT corruption and you need to rebuild the filesystem with LFSCK.</p> <p>9941 is not familiar to me, did you mean a different LU?</p> <p>What we are doing is using a modified version of Lester (e2scan derivative) to quickly build lists of files on a set of OSTs to be drained. (faster than lfs find) Then running lfs_migrate for those files.  To parallelize the migrations, using xargs to run multiple copies of lfs_migrate, so there definitely could have been a race hit between two different instances working on the same file via different hard links.  This was also started with an older ~2.7 version of lfs_migrate.</p> <p>The PFID issue was not seen from userspace, but by one of my colleagues working on improving the OST scanning tool, who was looking at things with debugfs.  I'm afraid I did not save the details to give you the exact example though!</p> <p>Ah, if you are scanning the OST inodes directly, and looking at the <tt>filter_fid</tt> xattr of migrated OST objects, then indeed this is the correct issue. It wasn't clear from your initial comment.</p> <p>I'll let Fan Yong comment on what LFSCK command to use to fix the <tt>filter_fid</tt> xattrs on the OST objects, until such a time that this fix is implemented.</p> <p>The OST-object's PFID EA back references the MDT-object which references the OST-object via its LOV EA. The PFID has the following functionalities:</p> <p>1) When you enable the 'lfsck_verify_pfid' on the OST (disabled by default), the I/O logic on the OST will check whether the I/O target belong to the expected regular file or not. If not, deny I/O.<br/> 2) The layout LFSCK will use the PFID and LOV EA to verify whether the MDT-object and OST-object pairs consistently reference each other.<br/> 3) If the MDT-object crashed, then the layout LFSCK will use the PFID EA to re-geneated the MDT-object on the MDT to resolve the global orphan issues.</p> <p>Usually, the PFID EA will be set on the OST-object when the OST-object is modified for the first time (write/truncate/setattr). It also needs to be updated if the MDT-object's FID is changed, such as the migration case. But in the old implementation, we only swapped the LOV EA for the MDT-objects during migration, ignored the PFID EA for related OST-objects. Usually, such inconsistency is invisible/harmless to applications except above three cases or you make some special consistency check by yourself, such as your colleague did via debugfs.</p> <p>Jinshan's patch <a href="https://review.whamcloud.com/#/c/30292/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/30292/</a> will handle PFID during layout swap or merge (for FLR).</p> <p>Anyway, the layout LFSCK can repair the inconsistent PFID EA since Lustre-2.6</p> <p>To repair inconsistent PFID EA, you can run layout LFSCK, for example (run command on the MDT0000)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl lfsck_start -M ${fsname}-MDT0000 -A -t layout -r </pre> </div></div> <p>If just want to check whether there is inconsistency or not, then plus dryrun option "--dryrun".</p> <p>Thanks much for the clarification, nasf!</p> <p>I performed a dry-run lfsck as follows and did get one inconsistency.  Please let me know if there is any other debug info I should gather, or just go ahead and re-run without the dry-run flag.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl clear lctl debug_daemon start /var/log/lfsck-n.debug lctl lfsck_start -M nbp7-MDT0000 -A -t layout -r -n # wait for "lctl get_param -n osd-ldiskfs.*.oi_scrub | grep status" to show "completed" lctl debug_daemon stop lctl debug_file /var/log/lfsck-n.debug &gt; /var/log/lfsck-n.log egrep -v "(kiblnd_passive_connect|ping_evictor_main)" /var/log/lfsck-n.log </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00020000:15.0F:1512490067.127409:0:89302:0:(lod_dev.c:651:lod_sync()) nbp7-MDT0000-mdtlov: can't sync 79: -107 00000004:00020000:0.0F:1512490072.101529:0:89302:0:(lod_dev.c:651:lod_sync()) nbp7-MDT0000-mdtlov: can't sync 79: -107 00080000:12000000:4.0F:1512490467.544841:0:43141:0:(osd_handler.c:574:osd_check_lma()) nbp7-MDT0000-osd: FID [0x20016d1ea:0x627:0x0] != self_fid [0x20016d1ea:0x62f:0x0] Debug log: 12 lines, 12 kept, 0 dropped, 0 bad. </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>client ~ # lfs fid2path /nobackupp7/ 0x20016d1ea:0x627:0x0 fid2path: error on FID 0x20016d1ea:0x627:0x0: No such file or directory client ~# lfs fid2path /nobackupp7/ 0x20016d1ea:0x62f:0x0 /nobackupp7/somepath/esmf_field.xsd client ~ # lfs path2fid /nobackupp7/somepath/esmf_field.xsd [0x20016d1ea:0x62f:0x0] </pre> </div></div> <p>Note that the "can't sync 79" errors are almost certainly because we have finished running lfs_migrate on one of the OSTs (#79) already and taken it offline.</p> <p>Please show me the output:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl get_param -n osd-ldiskfs.*.oi_scrub lctl get_param -n mdd.*.lfsck_layout </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>repaired_unmatched_pair: 283 </pre> </div></div> <p>The layout LFSCK detected 283 unmatched MDT-object and OST-object pairs. Two choices:<br/> 1) Run layout LFSCK again without "dryrun" option, that will repair the inconsistency directly.<br/> 2) To be safe, enable "lfsck" debug log on the MDT, then re-run dryrun mode layout LFSCK again. After complete, dump the Lustre kernel debug logs on the MDT, that will contain the found inconsistencies. Please ONLY enable "lfsck" debug log to avoid log buffer full and overwritten.</p> <p>Redid the dry run and it reported a lot more than 283 errors! Commands I used are in the uploaded lfsck_2017-12-06-n.txt file, results in the other files should be evident. Please advise.</p> <p>There is known issue about the repaired_inconsistent_owner, there were also a lot of unexpected repaired_inconsistent_owner in your former logs. They are not the real inconsistency, please ignore them temporarily. On the other hand, as you can see, the Lustre debug logs were overwritten because of those fake repaired_inconsistent_owner information, as to the useful repaired_unmatched_pair logs were lost.</p> <p>There are two choices:<br/> 1) Run layout LFSCK again without "dryrun" option, that will repair the inconsistency directly. But because of too many repaired_inconsistent_owner information, we may cannot know what to be fixed.<br/> 2) Keep the inconsistency there. According to your former logs, there are about 283 unmatched MDT-object and OST-object pairs. These unmatched pairs will NOT affect the normal system access as described in the comment:<br/> <a href="https://jira.hpdd.intel.com/browse/LU-10248?focusedCommentId=215286&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-215286" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-10248?focusedCommentId=215286&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-215286</a></p> <p>I will work on the fake inconsistent owner issue, once done, you can apply related patch(es) and run layout LFSCK again.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ndauchy" class="user-hover" rel="ndauchy">ndauchy</a>, what is your Lustre version?</p> <p>It really seems like it would be better to transfer the NASA discussion to its own ticket rather than tacking on the end of this ticket to track part of the FLR implementation.</p> <p><a href="https://jira.whamcloud.com/browse/LU-10349" title="debug and cleanup of corrupted PFID, unmatched MDT-object and OST-object pairs" class="issue-link" data-issue-key="LU-10349"><del>LU-10349</del></a> created for NASA-specific debugging efforts</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30292/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30292/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10248" title="Need to update PFID of OST objects after layout change" class="issue-link" data-issue-key="LU-10248"><del>LU-10248</del></a> mdd: set PFID for swap and merge layout<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4d534365ab214e28452c54fd2e0d4781e2f290d6</p> <p>Is this task complete with the recent landing to master?</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30635" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30635</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10248" title="Need to update PFID of OST objects after layout change" class="issue-link" data-issue-key="LU-10248"><del>LU-10248</del></a> mdd: set PFID for swap and merge layout<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 8d034f5aa4794d4dcd761b3e5ba2537995cb4e5d</p> <p>If the work on #30635 is complete, can we land it to b2_10?</p> Related LU-2677 LU-3128 LU-9771 Development Rank 1|hzznxz: Rank (Obsolete) 9223372036854775807