https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5822 Lustre <p>Over the weekend we had an OST abort and get marked read-only:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[ 726.076561] LDISKFS-fs error (device dm-25): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 111692corrupted: 32768 blocks free in bitmap, 1024 - in gd [ 726.116663] [ 726.125085] Aborting journal on device dm-25-8. [ 726.133359] LustreError: 17032:0:(ofd_obd.c:1095:ofd_destroy()) f1-OST00ff: error destroying object [0x100000000:0x16546e7:0x0]: 0 [ 726.176268] LDISKFS-fs (dm-25): [ 726.179457] LDISKFS-fs error (device dm-25): ldiskfs_journal_start_sb: Detected aborted journal [ 726.179459] LDISKFS-fs (dm-25): Remounting filesystem read-only</pre> </div></div> <p>We rely on the /proc/fs/lustre/health_check file to notify us of these situations. Unfortunately, we never got a notification. I found a bug in the b2_5 implementation of the osd-ldiskfs osd_statfs() function. Code inspection leads me to believe it does not affect master, but I haven't tried it there. I will upload a patch momentarily.</p> LU-5822

health_check file not updating properly

Bug Major Resolved Fixed James Nunez Matt Ezell patch Tue, 28 Oct 2014 20:43:38 +0000 Fri, 24 Apr 2015 23:55:02 +0000 Thu, 4 Dec 2014 22:45:38 +0000 Lustre 2.5.3 Lustre 2.5.4 0 8 <p>I'd like to also add a test to make sure this doesn't break in the future, but I'm not sure of the best way to make the device go read only. I started with the read-only infrastructure that the test system uses, but that appears to set the underlying device to read-only, not the ldiskfs file system.</p> <p>First, I tried to implement a server_remount_fs() function so you could do 'mount -o remount,ro', but that gets a Lustre superblock that you would then need to call down into the osd-api to actually make it do anything to the underlying file system.<br/> I then looked at adding an IOCTL that lctl could call, but that also appears to require support from the osd-api.<br/> I have a prototype patch that adds a new osd-api method, dt_abort_device() that could be called from either remount or lctl, but I'm not sure if it makes sense to add a new method just for testing this. Thoughts?</p> <p>Is there an easier way to cause the underlying filesystem to abort or go read-only?</p> <p><a href="http://review.whamcloud.com/12463" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12463</a></p> <p>Matt, </p> <p>Thanks for the patch. I'll look into a test to check that health_check is being updated properly.</p> <p>James</p> <p>There was some work done in <a href="https://jira.whamcloud.com/browse/LU-137" title="ioctl passthrough mechanism for Lustre OST/MDT mountpoints" class="issue-link" data-issue-key="LU-137"><del>LU-137</del></a> to allow ioctl() pass through to the underlying filesystem, but this was complicated in the 2.4+ releases by the OSD API and the addition of ZFS. While I would be happy to see that work move forward, but it is probably overkill for this. </p> <p>It would be easier to use the existing fault injection method for Lustre and add a new FAIL_LOC for this case, like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (sb-&gt;s_flags &amp; MS_RDONLY || (OBD_FAIL_CHECK(OBD_FAIL_OSD_READONLY) &amp;&amp; osd-&gt;od_jndex == libcfs_fail_val)) osd-&gt;od_statfs.os_state = OS_STATE_READONLY; </pre> </div></div> <p>or similar (this is just from the top of my head so the syntax might not be quite correct).</p> <p>PS: I verified that this patch is not needed for master.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12463/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12463/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5822" title="health_check file not updating properly" class="issue-link" data-issue-key="LU-5822"><del>LU-5822</del></a> osd-ldiskfs: Correctly return OS_STATE_READONLY<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: <br/> Commit: 1ff49a78e443f935670daf0c84b5b989c02dca04</p> <p>Landed for 2.5.4</p> Related LU-137 Development Rank 1|hzwzp3: Rank (Obsolete) 16321 Severity