https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5265 Lustre <p>Context:<br/> OI_Scrub has been triggered after failover of the MDT on the failover MDS. (related to <a href="https://jira.whamcloud.com/browse/LU-4554" title="OI scrub always runs on ldiskfs MDS start up" class="issue-link" data-issue-key="LU-4554"><del>LU-4554</del></a>)</p> <p>---<del>8&lt;</del>---<br/> LustreError: 0-0: ptmp2-MDT0000: trigger OI scrub by RPC for <span class="error">&#91;0x22cb1aa25:0xfabf:0x0&#93;</span>, rc = 0 <span class="error">&#91;1&#93;</span><br/> LustreError: 0-0: spool2-MDT0000: trigger OI scrub by RPC for <span class="error">&#91;0x20cf1887f:0x92c:0x0&#93;</span>, rc = 0 <span class="error">&#91;1&#93;</span><br/> ---<del>8&lt;</del>---</p> <p>Issue:<br/> Lustre clients were hung while trying to read/write from/to the filesystem, getting an error EINPROGRESS from the server for each request until the completion of the OI_Scrub process.</p> <p>However, the following commands were still working: ls, cd, df</p> <p>Due to the number of inodes, the OI_Scrub took 3 hours to complete, hanging the production.</p> <p>OI_Scrub status once completed:<br/> ---<del>8&lt;</del>---</p> <ol> <li>cat /proc/fs/lustre/osd-ldiskfs/ptmp2-MDT0000/oi_scrub<br/> name: OI_scrub<br/> magic: 0x4c5fd252<br/> oi_files: 1<br/> status: completed<br/> flags:<br/> param:<br/> time_since_last_completed: 382 seconds<br/> time_since_latest_start: 11068 seconds<br/> time_since_last_checkpoint: 382 seconds<br/> latest_start_position: 12<br/> last_checkpoint_position: 499122177<br/> first_failure_position: N/A<br/> checked: 190095126<br/> updated: 2<br/> failed: 0<br/> prior_updated: 0<br/> noscrub: 1965<br/> igif: 239<br/> success_count: 3<br/> run_time: 10685 seconds<br/> average_speed: 17790 objects/sec<br/> real-time_speed: N/A<br/> current_position: N/A<br/> ---<del>8&lt;</del>---</li> </ol> <p>run_time/3600 = 10685/3600 ~= 2.97 hours.</p> <p>As a workaround, auto_scrub has been disabled (echo 0 &gt; /proc/fs/lustre/osd-ldiskfs/ptmp2-MDT0000/auto_scrub)</p> <p>We have since upgraded to Lustre 2.4.3 with the patch from <a href="https://jira.whamcloud.com/browse/LU-4554" title="OI scrub always runs on ldiskfs MDS start up" class="issue-link" data-issue-key="LU-4554"><del>LU-4554</del></a>. The customer would like to enable the auto_scrub feature in order to get a consistent OI table, but cannot accept such an impact on the production systems.</p> <p>Regarding the "OI Scrub and inode Iterator Solution Architecture", client can access the MDT while OI Scrub is running. Except the operations of FID-to-path or accessing parent from non-directory child, other operations behave as normal.</p> RHEL6 LU-5265

Lustre clients hang while OI_Scrub is running

Bug Major Resolved Not a Bug nasf Bruno Travouillon Fri, 27 Jun 2014 15:00:11 +0000 Wed, 27 Aug 2014 17:23:26 +0000 Wed, 27 Aug 2014 17:23:26 +0000 Lustre 2.4.2 0 8 <p>Top on the MDS while OI_Scrub was running</p> <p>Fan Yong</p> <p>Could you please advise on this issue?</p> <p>Thanks</p> <p>Peter</p> <p>During the OI scrub rebuilding the OI files, if the client accesses the system with name-based RPC, such as lookup, then it will not be affected. But if the client sends FID-based RPC to the MDS and related FID mapping has not been rebuilt yet, it will get -EINPROGRESS until related FID mapping has been rebuilt, the worst case is that the application has to wait until OI scrub finished. The case of FID-based RPC usually happens for old connected client that caches the FID on client-side before the upgrading or before the MDS file-level backup/restore. For the new connected client, the FID-based RPC will always be after name-based RPC (except for FID-to-path), so the new connected client will not be affected.</p> <p>So for your above case, it is normal. Since your system has already run OI scrub, the inconsistent cases should have been fixed already. So even though you enable the "auto_scrub", the OI scrub should not be triggered unless it finds some new inconsistency (very rare). On the other hand, even though the OI scrub is rebuilding the OI files, it is NOT all the FIDs will be affected. Means that if the application tries to open-read/write the file which FID is not cached on the client or its FID mapping has been rebuilt already, then the application should not be affected by the OI scrub.</p> <p>So please tell me whether your system often hits the OI mapping failures (and trigger OI scrub) or not. If not, then enable "auto_scrub" will be OK. Otherwise, means the OI scrub cannot build the OI files completely, there should be other hidden bugs.</p> <p>Thanks for you clear answer.</p> <p>However, can you tell me how to check if OI scrub is triggered while auto_scrub is off?</p> <p>In osd_fid_lookup(), the LCONSOLE message "trigger OI scrub by RPC for DFID" only displays when auto_scrub is on.</p> <p>Should I check on clients' consoles?</p> <p>If auto_scrub is disabled, then the OI scrub will NOT be triggered automatically even though some inconsistency is detected during the normal processing. So you can NOT find the message about OI scrub auto running on the MDS. But under such case, the administrator can trigger OI scrub manually via "lctl lfsck_start".</p> <p>The OI scrub is server-side work, in any cases, the client will NOT print any message.</p> <p>Understood. We should enable the auto_scrub by the beginning of September.</p> <p>We will then be able to monitor the OI mapping failures and open a new ticket if we hit some issue.</p> <p>Thanks.</p> Development Rank 1|hzwq3j: Rank (Obsolete) 14692 Severity