https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9952 Lustre <p>One of production file systems running lustre 2.8.1 experienced a soft lock up very similar to <a href="https://jira.whamcloud.com/browse/LU-9488" title="soft lockup in osd_inode_iteration()" class="issue-link" data-issue-key="LU-9488"><del>LU-9488</del></a>. I attempted to back port the patch but way to many changes have happened between 2.8.1 and lustre 2.10.0. Unsure if I would get the port right. I have attached the back trace.</p> Lustre ldiskfs server back end running version 2.8.1 with a few additional patches. The OS is RHEL6.9 LU-9952

soft lockup in osd_inode_iteration() for lustre 2.8.1

Bug Major Resolved Fixed nasf James A Simmons Wed, 6 Sep 2017 18:56:52 +0000 Tue, 5 Jun 2018 16:42:57 +0000 Tue, 5 Jun 2018 16:42:57 +0000 Lustre 2.8.0 0 4 <p>Fan Yong</p> <p>Can you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>The known patches on master that are related with the OI scrub soft lockup are back ported as following:</p> <p><a href="https://review.whamcloud.com/28903" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28903</a><br/> <a href="https://review.whamcloud.com/28904" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28904</a><br/> <a href="https://review.whamcloud.com/28905" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28905</a><br/> <a href="https://review.whamcloud.com/28906" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28906</a></p> <p>We are in the process of testing these patches. I attempted to recreate the problem with "lctl set_param fail_loc=0x1504" but that didn't work. What would you recommend to recreate this problem on a 2.8 system? Note we removed the offending files to make our production file system usable again.</p> <p>I think that we need some new fail_loc to simulate osd_inode_iteration() trouble. For example, inject the new failure stub in the osd_iit_next() to simulate kinds of bitmap layout cases.</p> <p>Could you create a test condition before the 30th of September? </p> <p>Let check whether this one <a href="https://review.whamcloud.com/#/c/29133/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/29133/</a> works or not.</p> <p>We wouldn't be running the test framework on our production system. It looks like I just need to create a bunch of files on the file system. </p> <p>lctl set_param -n osd*.<b>MDT</b>.force_sync=1<br/> lctl set_param fail_val=1 fail_loc=0x190<br/> lctl lfsck_start -M lustre-MDT0000<br/> lctl set_param fail_val=0 fail_loc=0x198</p> <p>While you check status:<br/> lctl get_param -n osd-ldiskfs.lustre-MDT000.oi_scrub | grep status</p> <p>Does this look right? What values do I use to reset it back to normal working conditions.</p> <p>We have been trying your reproducer by itself and we see it get stuck but no soft lock ups. Why is no soft lockups being reported?</p> <p>"fail_loc=0x190" will slow down the OI scrub scanning, then we can have time to inject other failures before the OI scrub complete.<br/> "fail_loc=0x198" will make the OI scrub iteration repeatedly scan the same bits for inode table. If without our former patches (28903/4/5/6), then the OI scrub will fall into soft lockup. But because we have such patches, then OI scrub can detect such dead repeat then move forward. So no soft lockup is the expected behavior.</p> <p>We have been trying patch 28903 by itself and have not seen the soft lockup <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/> BTW do we need to inject another failure?</p> <blockquote> <p>We have been trying patch 28903 by itself and have not seen the soft lockup BTW do we need to inject another failure?</p></blockquote> <p>You mean you can reproduce the soft lockup every time if without any of above patch, right? And the soft lockup will be disappear if only 28903 applied, right?</p> <p>Any further feedback?</p> <p>Thanks!</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a>,<br/> Any further feedback for this ticket?</p> <p>You can close this. With various patches applied we haven't seen problems in some time. Thanks.</p> <p>The issue has bee resolved via back porting the following patches:<br/> <a href="https://review.whamcloud.com/28903" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28903</a><br/> <a href="https://review.whamcloud.com/28904" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28904</a><br/> <a href="https://review.whamcloud.com/28905" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28905</a><br/> <a href="https://review.whamcloud.com/28906" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28906</a></p> Related LU-9040 LU-9488 Development Rank 1|hzzjo7: Rank (Obsolete) 9223372036854775807 Severity