https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6687 Lustre <p>After we reboot and mount mdt we see all osp-sync threads in D state and the following errors.</p> <p>Jun 3 14:38:03 nbp8-mds1 kernel: LustreError: 5838:0:(osp_sync.c:487:osp_sync_new_setattr_job()) nbp8-OST009e-osc-MDT0000: invalid setattr record, lsr_valid:100<br/> Jun 3 14:38:03 nbp8-mds1 kernel: LustreError: 5838:0:(osp_sync.c:487:osp_sync_new_setattr_job()) Skipped 1554778 previous similar messages<br/> Jun 3 14:40:11 nbp8-mds1 kernel: LustreError: 6043:0:(osp_sync.c:487:osp_sync_new_setattr_job()) nbp8-OST010e-osc-MDT0000: invalid setattr record, lsr_valid:8191<br/> Jun 3 14:40:11 nbp8-mds1 kernel: LustreError: 6043:0:(osp_sync.c:487:osp_sync_new_setattr_job()) Skipped 2608741 previous similar messages<br/> Jun 3 14:44:27 nbp8-mds1 kernel: LustreError: 6342:0:(osp_sync.c:487:osp_sync_new_setattr_job()) nbp8-OST0133-osc-MDT0000: invalid setattr record, lsr_valid:8191<br/> Jun 3 14:44:27 nbp8-mds1 kernel: LustreError: 6342:0:(osp_sync.c:487:osp_sync_new_setattr_job()) Skipped 10802201 previous similar messages<br/> Jun 3 14:47:16 nbp8-mds1 pcp-pmie<span class="error">&#91;4713&#93;</span>: High 1-minute load average 321load@nbp8-mds1<br/> Jun 3 14:52:59 nbp8-mds1 kernel: LustreError: 6220:0:(osp_sync.c:487:osp_sync_new_setattr_job()) nbp8-OST0045-osc-MDT0000: invalid setattr record, lsr_valid:68<br/> Jun 3 14:52:59 nbp8-mds1 kernel: LustreError: 6220:0:(osp_sync.c:487:osp_sync_new_setattr_job()) Skipped 58201221 previous similar messages</p> LU-6687

ALL osp-sync in D state

Bug Critical Resolved Won't Fix Niu Yawei Mahmoud Hanafi Wed, 3 Jun 2015 21:54:06 +0000 Fri, 16 Oct 2015 04:26:14 +0000 Fri, 16 Oct 2015 04:26:14 +0000 Lustre 2.5.3 0 4 <p>Niu</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>Looks the related patch was applied correctly in your 2.5.3 tree (<a href="https://github.com/jlan/lustre-nas/commit/fb970b342a7fac22a17b4932e11febb6963b3dff" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jlan/lustre-nas/commit/fb970b342a7fac22a17b4932e11febb6963b3dff</a>) </p> <p>Is this an upgraded system? and it's the first mount after upgrading? I'm wondering if these invalid records were some leftover from old system.</p> <p>BTW, because of <a href="https://jira.whamcloud.com/browse/LU-5297" title="osp_sync_thread can&#39;t handle invalid record gracefully" class="issue-link" data-issue-key="LU-5297"><del>LU-5297</del></a>, osp sync thread can't handle invalid records properly.</p> <p>This is a upgraded system to 2.5.3. This happens every time when the mdt is mounted. How do we go about fixing the invalid records?</p> <p>We are going and cleanup a lot of mismatch between object UID/GID and MDT records. These occurred most likely due to <a href="https://jira.whamcloud.com/browse/LU-6574" title="chown doesn&#39;t update object ownership on ost" class="issue-link" data-issue-key="LU-6574"><del>LU-6574</del></a>.</p> <p>Hmm, fixing these invalid records manually will be troublesome (there isn't any llog edit tool, so you have to use hex edit to modify the records...)</p> <p>Actually, if there are only few leftover records, we can just delete all of them by removing the llog files, then we can move on to mount the MDT.</p> <p>1. lctl --device $MDTDEV llog_catlist to show all the catalogs for the unlink/setattr records;<br/> 2. mount mdt as ldiskfs and find all the catalogs under /O;<br/> 3. use llog_reader to show plain logs belong to these catalogs;<br/> 4. use llog_reader to see how many leftover records in the plain logs;<br/> 5. remove all the plain logs and catalogs, they will be recreated on next mount;</p> <p>nbp8-mds1 ~ # lctl --device 6 llog_catlist<br/> OBD_IOC_CATLOGLIST failed: Operation not supported</p> <p>it looks like there are a lot of records. I am not sure If i understand item #4 and #5</p> <p>Hmm, looks llog_catlist is only available in master now.</p> <p>Ok, each chown &amp; unlink on MDT will generate a llog record in llog file, and this record will be used to sync the operations to OST objects, once the sync to OST done, the record will be removed from the llog file. Usually after a clean shutdown, there won't be any leftover records in the llog files. However, in your case, there are some invalid records which can't be processed and not removed at the end.</p> <p>Let's look at the on disk structure of llog files:</p> <ul class="alternate" type="square"> <li>There is a global CATLOGLIST file which storing all the catalog IDs for each OST; ("lctl llog_catlist" can print the content of this file)</li> <li>Each catalog is a plain log index, which stores plain log IDs;</li> <li>Each plain log stores the unlink/setattr records I mentioned above;</li> </ul> <p>I think that could make it easier for understanding the #4 &amp; #5 of my previous comment?</p> <p>Given the "lctl llog_catlist" isn't supported in 2.5, you can remove all the leftover records as following: mount mdt as ldiskfs, find all the files which name is numerical under /O/1/ and remove them all. (it's better to backup these files)</p> <p>Please close this case</p> <p>ok Mahmoud</p> Related LU-5297 Development Rank 1|hzxevr: Rank (Obsolete) 9223372036854775807 Severity