https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7169 Lustre <p><a href="https://testing.hpdd.intel.com/test_sessions/e089506c-5bf0-11e5-9dac-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/e089506c-5bf0-11e5-9dac-5254006e85c2</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 Lustre: Skipped 79 previous similar messages Lustre: ctl-lustre-MDT0000: <span class="code-keyword">super</span>-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400):0:mdt Lustre: Skipped 26 previous similar messages Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests<span class="code-comment">//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust </span>Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2&gt;/dev/<span class="code-keyword">null</span> Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 Lustre: DEBUG MARKER: sync; sync; sync Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 readonly LustreError: 14276:0:(osd_handler.c:1380:osd_ro()) *** setting lustre-MDT0000 read-only *** Turning device dm-0 (0xfd00000) read-only Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: lctl set_param fail_loc=0x20000709 fail_val=5 Lustre: DEBUG MARKER: grep -c /mnt/mds1<span class="code-quote">' '</span> /proc/mounts Lustre: DEBUG MARKER: umount -d /mnt/mds1 Lustre: Failing over lustre-MDT0000 Removing read-only on unknown block (0xfd00000) Lustre: DEBUG MARKER: lsmod | grep lnet &gt; /dev/<span class="code-keyword">null</span> &amp;&amp; lctl dl | grep <span class="code-quote">' ST '</span> Lustre: DEBUG MARKER: hostname Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 Lustre: DEBUG MARKER: mkdir -p /mnt/mds1; mount -t lustre -o recovery_time_hard=60,recovery_time_soft=60 /dev/lvm-Role_MDS/P1 /mnt/mds1 LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: LDISKFS-fs error (device dm-0): ldiskfs_lookup: deleted inode referenced: 75023 Aborting journal on device dm-0-8. LDISKFS-fs (dm-0): Remounting filesystem read-only LDISKFS-fs error (device dm-0): ldiskfs_put_super: Couldn't clean up the journal LustreError: 14732:0:(obd_config.c:575:class_setup()) setup lustre-MDT0000-osd failed (-30) LustreError: 14732:0:(obd_mount.c:203:lustre_start_simple()) lustre-MDT0000-osd setup error -30 LustreError: 14732:0:(obd_mount_server.c:1760:server_fill_super()) Unable to start osd on /dev/mapper/lvm--Role_MDS-P1: -30 LustreError: 14732:0:(obd_mount.c:1342:lustre_fill_super()) Unable to mount (-30) Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity test_84: @@@@@@ FAIL: Restart of mds1 failed! </pre> </div></div> <p>Looks the filesystem is corrupted somehow.</p> LU-7169

conf-sanity 84 restart mds1 failed

Bug Critical Resolved Duplicate nasf Niu Yawei Wed, 16 Sep 2015 03:58:01 +0000 Mon, 30 Nov 2015 18:49:42 +0000 Mon, 30 Nov 2015 18:49:42 +0000 Lustre 2.8.0 0 7 <p>The corruption here may be related to the problem seen in <a href="https://jira.whamcloud.com/browse/LU-6895" title="sanity-lfsck test 4 hung: bad entry in directory: rec_len is smaller than minimal - inode=3925999616" class="issue-link" data-issue-key="LU-6895"><del>LU-6895</del></a></p> <p>Another occurance<br/> 2015-09-15 14:42:13 - <a href="https://testing.hpdd.intel.com/test_sets/f3da22c2-5bf0-11e5-9dac-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f3da22c2-5bf0-11e5-9dac-5254006e85c2</a></p> <p>Fan Yong,<br/> Can you look at this issue and see if it is a dup of <a href="https://jira.whamcloud.com/browse/LU-6895" title="sanity-lfsck test 4 hung: bad entry in directory: rec_len is smaller than minimal - inode=3925999616" class="issue-link" data-issue-key="LU-6895"><del>LU-6895</del></a>?<br/> Thanks.<br/> Joe</p> <p>The MDS console log reports:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>18:38:45:LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 18:38:45:LDISKFS-fs error (device dm-0): ldiskfs_lookup: deleted inode referenced: 75023 18:38:45:Aborting journal on device dm-0-8. 18:38:45:LDISKFS-fs (dm-0): Remounting filesystem read-only </pre> </div></div> <p>On a later test when the MDS was remounted it crashed:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>18:38:45:LDISKFS-fs error (device dm-0): ldiskfs_lookup: deleted inode referenced: 75023 18:38:45:Aborting journal on device dm-0-8. 18:38:45:LDISKFS-fs (dm-0): Remounting filesystem read-only 18:38:45:BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 18:38:45:IP: [&lt;ffffffffa0b9833d&gt;] osd_scrub_refresh_mapping+0x39d/0x410 [osd_ldiskfs] 18:38:45:Oops: 0000 [#1] SMP 18:38:56:Pid: 15532, comm: mount.lustre Not tainted 2.6.32-573.3.1.el6_lustre.g3eb333f.x86_64 #1 Red Hat KVM 18:38:56:Call Trace: 18:38:56: [&lt;ffffffffa0b9c616&gt;] osd_scrub_setup+0xfb6/0x1170 [osd_ldiskfs] 18:38:56: [&lt;ffffffffa0b707e9&gt;] osd_device_alloc+0x6c9/0x960 [osd_ldiskfs] 18:38:56: [&lt;ffffffffa05bf8df&gt;] obd_setup+0x1bf/0x290 [obdclass] 18:38:56: [&lt;ffffffffa05bfc0f&gt;] class_setup+0x25f/0x940 [obdclass] 18:38:56: [&lt;ffffffffa05c6df1&gt;] class_process_config+0x1151/0x2840 [obdclass] 18:38:56: [&lt;ffffffffa05d0abb&gt;] do_lcfg+0x2cb/0x640 [obdclass] 18:38:56: [&lt;ffffffffa05d0ec4&gt;] lustre_start_simple+0x94/0x200 [obdclass] 18:38:56: [&lt;ffffffffa0605ce1&gt;] server_fill_super+0xfd1/0x1a70 [obdclass] 18:38:56: [&lt;ffffffffa05d33e8&gt;] lustre_fill_super+0x348/0x990 [obdclass] 18:38:56: [&lt;ffffffff8119532f&gt;] get_sb_nodev+0x5f/0xa0 18:38:56: [&lt;ffffffffa05cb8a5&gt;] lustre_get_sb+0x25/0x30 [obdclass] 18:38:56: [&lt;ffffffff8119496b&gt;] vfs_kern_mount+0x7b/0x1b0 18:38:56: [&lt;ffffffff81194b12&gt;] do_kern_mount+0x52/0x130 18:38:56: [&lt;ffffffff811b697b&gt;] do_mount+0x2fb/0x930 18:38:56: [&lt;ffffffff811b7040&gt;] sys_mount+0x90/0xe0 </pre> </div></div> <p>The MDS crash at "osd_scrub_refresh_mapping+0x39d" is a duplication of <a href="https://jira.whamcloud.com/browse/LU-7061" title="BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 IP: osd_scrub_refresh_mapping+0x39d/0x410" class="issue-link" data-issue-key="LU-7061"><del>LU-7061</del></a>.</p> <p>Fan Yong,<br/> Can you have a look at the corruption issue as well?<br/> Thanks.<br/> Joe</p> <p>It is NOT the same at <a href="https://jira.whamcloud.com/browse/LU-6895" title="sanity-lfsck test 4 hung: bad entry in directory: rec_len is smaller than minimal - inode=3925999616" class="issue-link" data-issue-key="LU-6895"><del>LU-6895</del></a>. The direct reason for the failure is as following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LDISKFS-fs error (device dm-0): ldiskfs_lookup: deleted inode referenced: 75023 </pre> </div></div> <p>ldiskfs_lookup() found a deleted inode, then its error handler set the system as read-only, and then caused cascaded failures for subsequent operations.</p> <p>In fact, before the ldiskfs_lookup() failure, the system already crashed. Here is the MDS debug log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00100000:00000001:0.0:1442342253.462893:0:14732:0:(osd_scrub.c:2423:osd_scrub_setup()) Process entered 00080000:00000001:0.0:1442342253.462936:0:14732:0:(osd_handler.c:2401:osd_ea_fid_set()) Process entered 00080000:00000001:0.0:1442342253.462944:0:14732:0:(osd_handler.c:2429:osd_ea_fid_set()) Process leaving (rc=0 : 0 : 0) 00100000:10000000:0.0:1442342253.463278:0:14732:0:(osd_scrub.c:277:osd_scrub_file_reset()) lustre-MDT0000: reset OI scrub file, old flags = 0x0, add flags = 0x2 </pre> </div></div> <p>Related source code is as following:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">int</span> osd_scrub_setup(<span class="code-keyword">const</span> struct lu_env *env, struct osd_device *dev) { ... } <span class="code-keyword">else</span> { <span class="code-keyword">if</span> (memcmp(sf-&gt;sf_uuid, es-&gt;s_uuid, 16) != 0) { osd_scrub_file_reset(scrub, es-&gt;s_uuid,SF_INCONSISTENT); dirty = 1; ... } </pre> </div></div> <p>The logs shows that during the MDT0000 mount, the osd_scrub_setup() found the super block's uuid has been changed. Usually, such case only happens when MDT file-level backup/restore. But in our conf-sanity test cases, there were no backup/restore operations. So the local file-system should have been crashed during the MDT failover. As for what caused the super block corruption, I have no idea yet.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16664" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16664</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7169" title="conf-sanity 84 restart mds1 failed" class="issue-link" data-issue-key="LU-7169"><del>LU-7169</del></a> tests: check disk corruption during failover<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: aa1a896b1fddadce8b95b2c9af6c4a8509535a19</p> <p>Fan Yong</p> <p>What are the next steps here? It looks like the debug patch did not trigger the failure. Should we land the debug patch or does it just need to be run with a higher number of runs to improve the chances of hitting it?</p> <p>Thanks</p> <p>Peter</p> <p>The issues cannot be reproduced any longer after the debug patch, even through several repeat the failed test case. So I suggest to land the patch to master and give the chance to normal Maloo run. I will update the patch to drop "fortestonly" and ask for landing permission.</p> <p>Excellent. Thanks Fan Yong.</p> <p>We hit the trouble with the patch applied, the log shows that there is really some disk inconsistency as following:<br/> <a href="https://testing.hpdd.intel.com/test_sets/e6f060ac-8707-11e5-bf92-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e6f060ac-8707-11e5-bf92-5254006e85c2</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: onyx-44vm3 e2fsck -d -v -t -t -f -n /dev/lvm-Role_MDS/P1 onyx-44vm3: e2fsck 1.42.13.wc3 (28-Aug-2015) Warning: skipping journal recovery because doing a read-only filesystem check. Pass 1: Checking inodes, blocks, and sizes Pass 1: Memory used: 292k/0k (87k/206k), time: 0.00/ 0.00/ 0.00 Pass 1: I/O read: 1MB, write: 0MB, rate: 553.40MB/s Pass 2: Checking directory structure Pass 2: Memory used: 292k/0k (98k/195k), time: 0.00/ 0.00/ 0.00 Pass 2: I/O read: 1MB, write: 0MB, rate: 655.74MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 292k/0k (98k/195k), time: 0.01/ 0.00/ 0.00 Pass 3: Memory used: 292k/0k (96k/197k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 4: Checking reference counts Pass 4: Memory used: 292k/0k (62k/231k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 5: Checking group summary information Free blocks count wrong (33296, counted=32947). Fix? no Free inodes count wrong (99987, counted=99750). Fix? no Pass 5: Memory used: 292k/0k (62k/231k), time: 0.00/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 333.78MB/s 13 inodes used (0.01%, out of 100000) 6 non-contiguous files (46.2%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 0/0/0 16704 blocks used (33.41%, out of 50000) 0 bad blocks 1 large file 125 regular files 116 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 241 files Memory used: 292k/0k (61k/232k), time: 0.01/ 0.01/ 0.00 I/O read: 1MB, write: 0MB, rate: 92.42MB/s reboot facets: mds1 </pre> </div></div> <p>Such inconsistency does not means the super block crashed. Because without disk checksum, the e2fsck cannot detect per-block based data corruption. Only with above logs, we cannot say it is the root reason for test_84 failure. I will update the patch with more debug information.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16664/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16664/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7169" title="conf-sanity 84 restart mds1 failed" class="issue-link" data-issue-key="LU-7169"><del>LU-7169</del></a> tests: check disk corruption during failover<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f84e06eead85de5cd7832855bab5ff72a542e971</p> <p>Landed for 2.8</p> <p>The landed patch was only for debugging. This issue is not resolved.</p> <p>Right, we are still waiting for more failure instances after landing the patch.</p> <p>There are many, many failures of this test, but unfortunately they have all been assigned different bugs because the error messages are different. </p> <p>In the tests I've seen, the e2fsck run is clean, except for the superblock inside and block counts, which is expected. </p> <p>I pushed a patch under <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a> that may fix the problem, which I think is caused by test_84() setting the MDS read-only right after mount, and that is causing some of the recently written data to be discarded (e.g. superblock label, llog records, etc). Unfortunately, it will take a few days to be tested. </p> <p>You are right. <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a> is the duplication of this ticket. According to the latest test logs (marked as <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a>)<br/> <a href="https://testing.hpdd.intel.com/sub_tests/15602932-9477-11e5-a5ac-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/15602932-9477-11e5-a5ac-5254006e85c2</a></p> <p>The former test_83 reformat the system after test_83 done, then move to test_84, after the replay_barrier(), the e2fsck found some inconsistency:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Warning: skipping journal recovery because doing a read-only filesystem check. Pass 1: Checking inodes, blocks, and sizes Pass 1: Memory used: 292k/0k (86k/207k), time: 0.01/ 0.00/ 0.00 Pass 1: I/O read: 1MB, write: 0MB, rate: 104.44MB/s Pass 2: Checking directory structure Pass 2: Memory used: 292k/0k (98k/195k), time: 0.02/ 0.00/ 0.01 Pass 2: I/O read: 1MB, write: 0MB, rate: 47.50MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 292k/0k (98k/195k), time: 0.04/ 0.01/ 0.01 Pass 3: Memory used: 292k/0k (96k/197k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 4: Checking reference counts Pass 4: Memory used: 292k/0k (62k/231k), time: 0.00/ 0.00/ 0.00 Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 5: Checking group summary information Free blocks count wrong (33296, counted=32960). Fix? no Free inodes count wrong (99987, counted=99754). Fix? no Pass 5: Memory used: 292k/0k (61k/232k), time: 0.01/ 0.01/ 0.00 ... </pre> </div></div> <p>It is reasonable to suspect that the replay_barrier() marked the system as readonly too early before super block synced to disk. But seems the replay_barrier() has considered that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>replay_barrier() { local facet=$1 do_facet $facet "sync; sync; sync" df $MOUNT .. </pre> </div></div> <p>So seems the "sync" does not work. As for your patch <a href="http://review.whamcloud.com/#/c/17371/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/17371/</a>, it should be helpful. But why not make the sync_all_data to be inside replay_barrier() to replace the 'do_facet $facet "sync; sync; sync"', then it can help others.</p> <p>The patch from Andreas:<br/> Gerrit Updater added a comment - Yesterday<br/> Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17371" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17371</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a> tests: write superblock in conf-sanity test_84<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d3e516e59697f7c48e5cb97054ee04cf97dc7132</p> <p>The <tt>Free blocks/inodes count wrong</tt> errors do not indicate any kind of problem. These fields in the superblock are not updated during normal usage, so after <tt>replay_barrier()</tt> they will be out of date.</p> <p>I thought about putting the <tt>sync_all_data(); sleep 5;</tt> call inside <tt>replay_barrier()</tt> but I didn't want to slow down any of the other tests that use this function that don't immediately follow reformatting the filesystem.</p> <p>We'll see if this patch actually fixes the problem.</p> <p>believed to be a duplicate of <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a></p> Related LU-6895 LU-7428 LU-6789 Development Rank 1|hzxnrj: Rank (Obsolete) 9223372036854775807 Severity