https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8729 Lustre <p>With patch <a href="http://review.whamcloud.com/7200" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7200</a> on master branch, conf-sanity test 84 failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: onyx-31vm7 e2label /dev/mapper/mds1_flakey 2&gt;/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' Update not seen after 90s: wanted '' got 'lustre:MDT0000' conf-sanity test_84: @@@@@@ FAIL: /dev/mapper/mds1_flakey failed to initialize! </pre> </div></div> <p><a href="https://testing.hpdd.intel.com/test_sets/e88a61c2-89bf-11e6-a8b7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e88a61c2-89bf-11e6-a8b7-5254006e85c2</a></p> LU-8729

conf-sanity test_84: FAIL: /dev/mapper/mds1_flakey failed to initialize!

Bug Critical Resolved Cannot Reproduce Hongchao Zhang Jian Yu Wed, 19 Oct 2016 04:02:31 +0000 Thu, 14 Sep 2017 22:05:14 +0000 Thu, 14 Sep 2017 22:05:14 +0000 Lustre 2.10.1 Lustre 2.11.0 0 12 <p>More failure instances:<br/> replay-single test 0a: <a href="https://testing.hpdd.intel.com/test_sets/5fadea7c-8a19-11e6-91aa-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5fadea7c-8a19-11e6-91aa-5254006e85c2</a><br/> recovery-mds-scale: <a href="https://testing.hpdd.intel.com/test_sets/ad25283e-8bb2-11e6-91aa-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ad25283e-8bb2-11e6-91aa-5254006e85c2</a></p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/23251" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23251</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8729" title="conf-sanity test_84: FAIL: /dev/mapper/mds1_flakey failed to initialize!" class="issue-link" data-issue-key="LU-8729"><del>LU-8729</del></a> utils: improve lctl to add freeze and unfreeze operations<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cb65ea32816a1df5ca23a11a681ff1db43d6cff9</p> <p>Hi Hongchao,</p> <p>I tested the dm-flakey patch on the latest master branch in <a href="https://review.whamcloud.com/24733" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24733</a> and found the following issues:</p> <p>On CentOS 6.8 distro, replay-single test 0c consistently hit the following errors on MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LDISKFS-fs (dm-6): ldiskfs_check_descriptors: Checksum for group 4 failed (61905!=6807) LDISKFS-fs (dm-6): group descriptors corrupted! LustreError: 6423:0:(osd_handler.c:6825:osd_mount()) lustre-MDT0000-osd: can't mount /dev/mapper/mds1_flakey: -22 </pre> </div></div> <p>On CentOS 7.3 distro, conf-sanity test 84, replay-single tests 0a, 4b, 5, 18, 19 and replay-ost-single test 9 consistently hit the following errors on MDS or OSS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: dmsetup load /dev/mapper/mds1_flakey --table "0 4194304 flakey 252:0 0 0 1800 1 drop_writes" Lustre: DEBUG MARKER: dmsetup resume /dev/mapper/mds1_flakey Buffer I/O error on dev dm-6, logical block 524272, async page read </pre> </div></div> <p>The "Buffer I/O error" didn't occur on CentOS 6.8 distro. It also didn't occur on CentOS 7.3 distro before patch set 21 for <a href="https://review.whamcloud.com/7200" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/7200</a>. The CentOS 7.3 kernel version for patch set 20 is <tt>3.10.0-327.36.3.el7</tt>, and the kernel version for patch set 21 is <tt>3.10.0-514.el7</tt>.</p> <p>Is there sth wrong with dm-flakey in different kernels?</p> <p>On my VM, I have setup dm-flakey manually on 7.3 (3.10.0-514.2.2.el7) and run some basic tests (say, creation, deletion,<br/> copy etc) on it, and it show no error. I aslo looked at the code line of dm-flakey.c and found no clue of the problem so far.<br/> Will update the status of the ticket once there is any progress.</p> <p>This whole 'dmsetup load /dev/mapper/mds1_flakey --table "0 4194304 flakey 252:0 0 0 1800 1 drop_writes"' this looks a bit suspicious to me.</p> <p>Looking into the kenrel documentation, we can see:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Table parameters ---------------- &lt;dev path&gt; &lt;offset&gt; &lt;up interval&gt; &lt;down interval&gt; \ [&lt;num_features&gt; [&lt;feature arguments&gt;]] Mandatory parameters: &lt;dev path&gt;: Full pathname to the underlying block-device, or a "major:minor" device-number. &lt;offset&gt;: Starting sector within the device. &lt;up interval&gt;: Number of seconds device is available. &lt;down interval&gt;: Number of seconds device returns errors. Optional feature parameters: If no feature parameters are present, during the periods of unreliability, all I/O returns errors. drop_writes: All write I/O is silently ignored. Read I/O is handled correctly. </pre> </div></div> <p>According to this, "1" at the end of the table above is spurious? And if a target is unknown then it's just makes io errors to be returned from the io - which seems to be matching what is seen on rhel 7.3?<br/> There's a huge diff in devicemapper code between rhell 7.2 and 7.3, though I don't think API ever had this extra number there.<br/> Anyway I submitted the same patch we have no with just this table changed in <a href="https://review.whamcloud.com/#/c/25387/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/25387/</a> to see how well would that work and might be that's our fix in the end for newer distros.</p> <p>oh, I guess 1 is the "num_features" that I missed</p> <p>Jian, when is the last time this was tested? <a href="https://jira.whamcloud.com/browse/LU-7481" title="Failover: recovery-mds-scale test_failover_mds: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7481"><del>LU-7481</del></a> had a patch that was landed 2017-01-27 that fixed a related issue. </p> <p>Hi Andreas,<br/> The last time was 2017-03-12 and the fix for <a href="https://jira.whamcloud.com/browse/LU-7481" title="Failover: recovery-mds-scale test_failover_mds: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7481"><del>LU-7481</del></a> was included.<br/> The consistent failures now are the errors in comment:<br/> <a href="https://jira.hpdd.intel.com/browse/LU-8729?focusedCommentId=180716&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-180716" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-8729?focusedCommentId=180716&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-180716</a></p> <p>Hi Andreas and Oleg,</p> <p>The "Buffer I/O error" on device-mapper flakey device is still blocking the testing on patch <a href="https://review.whamcloud.com/7200" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/7200</a>. Hongchao looked at the codes of dm-flakey.c in CentOS 7.3 (kernel version 3.10.0-514.2.2.el7) but didn't find any clue of the problem.</p> <p>Could you please advise what we need to do to move this forward? Thank you.</p> <p>While testing I started to ask myself why are using dm_flakey for ever test. It appears this is only needed for recovery testing to replace lct --device readonly. Should we look to reduce the scope of dm-flakey to only the test that it is needed for?</p> <p>After working on this patch today I believe I got a handle on what is going on. So the "Buffer I/O error" are correct. So in the test you have:</p> <p>mount_client $MOUNT1 || error "mount $MOUNT1 failed"<br/> mount_client $MOUNT2 || error "mount $MOUNT2 failed"</p> <ol> <li>make sure new superblock labels are sync'd before disabling writes<br/> sync_all_data<br/> sleep 5</li> </ol> <p>replay_barrier $SINGLEMDS<br/> createmany <del>o $DIR1/$tfile</del>%d 1000</p> <p>This mounts and syncs all the data. Then replay_barrier using the power of the flakey driver sets the disk /dev/mapper/mds1_flakey to read only mode.  The creation attempts by createmany on a read only file system is what is generated the "Buffer I/O errors". This is what is supposed to happen. Next is:</p> <p>do_facet $SINGLEMDS "lctl set_param fail_loc=0x20000709 fail_val=5"</p> <p>facet_failover --fsck $SINGLEMDS || error "failover: $?"</p> <p>The facet_failover part is what is important. Here we set a special flag to make sure e2fsck is run on the MDT. Looking at the Maloo logs:</p> <p><a href="https://testing.hpdd.intel.com/test_logs/2cb1e62c-1f0c-11e7-8920-5254006e85c2/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_logs/2cb1e62c-1f0c-11e7-8920-5254006e85c2/show_text</a></p> <p>I noticed the error:</p> <p><span class="error">&#91;26120.388883&#93;</span> Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity test_84: \<br/> @@@@@@ FAIL: e2fsck -d -v -t -t -f -n \/dev\/mapper\/mds1_flakey returned 8, \<br/> should be \&lt;= 4 <br/> <span class="error">&#91;26120.544691&#93;</span> Lustre: DEBUG MARKER: conf-sanity test_84: @@@@@@ FAIL: e2fsck \<br/> -d -v -t -t -f -n /dev/mapper/mds1_flakey returned 8, should be &lt;= 4</p> <p>Running e2fsck shouldn't work on a read only device, mds1_flakey. I fixed up<br/> shutdown_facet which is called right before run_e2fsck in the function <br/> failover_facet(). Now it should not only stop the device but remove the flakey<br/> device and then e2fsck will be run on the native device which is not read only.<br/> Now I see a different failure. It looks like MDT0000 is not suppose to recover<br/> but it does for. Will need to look into it.<br/>  </p> <p>Thank you very much James for helping debug this failure.</p> <blockquote><p>While testing I started to ask myself why are using dm_flakey for ever test. It appears this is only needed for recovery testing to replace lct --device readonly. Should we look to reduce the scope of dm-flakey to only the test that it is needed for?</p></blockquote> <p>Besides recovery test suites (replay-single, replay-dual, etc), recovery testing also exists in other test suites (sanity, insanity, conf-sanity, etc). The changes will be a lot. What's more, we would like to make mount and unmount transparent, so that developers do not need to know they have to use a special option to create flakey devices while creating a recovery testing related sub-test. </p> <p>After doing some experiments, I found out the commits that caused the "Buffer I/O error". They are related to RHEL 7.3 support:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit adc9592d1820d5086e52e387008263b4dace9b0e Author: Christopher J. Morrone &lt;morrone2@llnl.gov&gt; CommitDate: Fri Nov 18 15:35:18 2016 +0000 LU-8534 ldiskfs: Add patch series for RHEL7.3 Reviewed-on: http://review.whamcloud.com/22113 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 5763c175c9a1a330b664a01bc08f329b2bee54f7 Author: Bob Glossman &lt;bob.glossman@intel.com&gt; CommitDate: Fri Nov 18 15:35:26 2016 +0000 LU-8796 kernel: kernel upgrade RHEL7.3 [3.10.0-514.el7] Reviewed-on: http://review.whamcloud.com/23560 </pre> </div></div> <p>In <a href="https://jira.whamcloud.com/browse/LU-684" title="replace dev_rdonly kernel patch with dm-flakey" class="issue-link" data-issue-key="LU-684"><del>LU-684</del></a> patch <a href="https://review.whamcloud.com/7200" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/7200</a>, replay-single test passed on patch set 20:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/c01176f8-a5ef-11e6-b605-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/c01176f8-a5ef-11e6-b605-5254006e85c2</a><br/> but failed on patch set 21 because of the above commits:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/41fe0cb2-beeb-11e6-92c6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/41fe0cb2-beeb-11e6-92c6-5254006e85c2</a></p> <p>The error message on MDS console was:</p> <div class="panel" style="border-width: 1px;"><div class="panelContent"> <p>Lustre: DEBUG MARKER: dmsetup suspend --nolockfs --noflush /dev/mapper/mds1_flakey<br/> Lustre: DEBUG MARKER: dmsetup load /dev/mapper/mds1_flakey --table "0 4194304 flakey 252:0 0 0 1800 1 drop_writes"<br/> Lustre: DEBUG MARKER: dmsetup resume /dev/mapper/mds1_flakey<br/> <font color="#FF0000">Buffer I/O error on dev dm-6, logical block 524272, async page read</font></p> </div></div> <p>In RHEL 7.3 kernel 3.10.0-514.el7, the above error message came from fs/buffer.c:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> void end_buffer_async_read(struct buffer_head *bh, <span class="code-object">int</span> uptodate) { <span class="code-comment">//…… </span> page = bh-&gt;b_page; <span class="code-keyword">if</span> (uptodate) { set_buffer_uptodate(bh); } <span class="code-keyword">else</span> { clear_buffer_uptodate(bh); buffer_io_error(bh, <span class="code-quote">", async page read"</span>); SetPageError(page); } <span class="code-comment">//…… </span>} </pre> </div></div> <p>One option is to add a <tt>WARN_ON()</tt> in <tt>end_buffer_async_read()</tt>, so that this will dump the full stack. Note that the callpath for <tt>end_buffer_async_read()</tt> does not come from <tt>mark_buffer_async_read()</tt>, that is just setting the <tt>bh-&gt;b_end_io</tt> pointer so that the <tt>end_buffer_async_read()</tt> callback is called when the read from disk is actually finished.</p> <p>It is probably coming through:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>-&gt;dmio_complete() bio.bi_error = error ? -EIO : 0; or -&gt;bio_complete() -&gt;bio.bi_end_io = end_bio_bh_io_sync(); -&gt;bh-&gt;b_end_io(bh, !bio-&gt;bi_error); </pre> </div></div> <p>I do see some comments in <tt>dm-flakey.c</tt> that might indicate problems if the <tt>DROP_WRITES</tt> state is being changed:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (!test_bit(DROP_WRITES, &amp;fc-&gt;flags) &amp;&amp; !test_bit(ERROR_WRITES, &amp;fc-&gt;flags)) { /* * Error read during the down_interval <span class="code-keyword">if</span> drop_writes * and error_writes were not configured. */ <span class="code-keyword">return</span> -EIO; } </pre> </div></div> <p>so that may be a result of changing the <tt>drop_writes</tt> state in the middle of a read. I see just before configuring the <tt>dm-flakey</tt> device:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>dmsetup suspend --nolockfs --noflush /dev/mapper/mds1_flakey </pre> </div></div> <p>Does "--noflush" means there may still be partially-completed reads?</p> <p>At a higher level, have we considered using the <tt>dm-flakey</tt> device only for cases when <tt>lctl readonly</tt> is actually used? It appears that this is only called from <tt>replay_barrier()</tt>, <tt>replay_barrier_nosync()</tt>, and <tt>replay_barrier_nodf()</tt>. I see one call to <tt>replay_barrier</tt> in <tt>conf-sanity.sh::test_84()</tt>, but we might consider to move this test over to <tt>replay-single.sh</tt>. It is also worthwhile to note that <tt>dm-flakey</tt> is only needed for ldiskfs, not for ZFS filesystems.</p> <blockquote><p>One option is to add a WARN_ON() in end_buffer_async_read(), so that this will dump the full stack.</p></blockquote> <p>The full stack is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: dmsetup suspend --nolockfs --noflush /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: dmsetup load /dev/mapper/mds1_flakey --table "0 4194304 flakey 252:0 0 0 1800 1 drop_writes" Lustre: DEBUG MARKER: dmsetup resume /dev/mapper/mds1_flakey Buffer I/O error on dev dm-6, logical block 524272, async page read ------------[ cut here ]------------ WARNING: at fs/buffer.c:295 end_buffer_async_read+0x119/0x140() Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd pcspkr virtio_balloon i2c_piix4 parport_pc parport nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi virtio_blk cirrus 8139too drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea sysfillrect crc32c_intel serio_raw sysimgblt fb_sys_fops ttm ata_piix virtio_pci 8139cp virtio_ring mii virtio drm libata i2c_core floppy CPU: 1 PID: 14902 Comm: systemd-udevd Tainted: G W OE ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 0000000000000000 0000000081bc25b0 ffff880064f4b990 ffffffff81686d2f ffff880064f4b9c8 ffffffff81085cb0 ffff8800642a7e00 ffff8800631e5068 ffff8800642a7e00 ffffea00017e7640 ffff880077f24800 ffff880064f4b9d8 Call Trace: [&lt;ffffffff81686d2f&gt;] dump_stack+0x19/0x1b [&lt;ffffffff81085cb0&gt;] warn_slowpath_common+0x70/0xb0 [&lt;ffffffff81085dfa&gt;] warn_slowpath_null+0x1a/0x20 [&lt;ffffffff81234269&gt;] end_buffer_async_read+0x119/0x140 [&lt;ffffffff8123263c&gt;] end_bio_bh_io_sync+0x2c/0x60 [&lt;ffffffff81239454&gt;] bio_endio+0x64/0xa0 [&lt;ffffffffa063c8da&gt;] dec_pending+0x18a/0x2e0 [dm_mod] [&lt;ffffffffa063e359&gt;] __split_and_process_bio+0x2d9/0x500 [dm_mod] [&lt;ffffffff812e0000&gt;] ? aes_encrypt+0xb30/0xe00 [&lt;ffffffffa063e68e&gt;] dm_make_request+0x10e/0x170 [dm_mod] [&lt;ffffffff812eee19&gt;] generic_make_request+0x109/0x1e0 [&lt;ffffffff812eef61&gt;] submit_bio+0x71/0x150 [&lt;ffffffff812381dd&gt;] ? bio_alloc_bioset+0x1fd/0x350 [&lt;ffffffff81233283&gt;] _submit_bh+0x143/0x210 [&lt;ffffffff81235bd7&gt;] block_read_full_page+0x217/0x350 [&lt;ffffffff81239670&gt;] ? I_BDEV+0x10/0x10 [&lt;ffffffff8118e219&gt;] ? force_page_cache_readahead+0x99/0xe0 [&lt;ffffffff8123a0a8&gt;] blkdev_readpage+0x18/0x20 [&lt;ffffffff811829df&gt;] generic_file_aio_read+0x3cf/0x790 [&lt;ffffffff8123a46c&gt;] blkdev_aio_read+0x4c/0x70 [&lt;ffffffff811fde6d&gt;] do_sync_read+0x8d/0xd0 [&lt;ffffffff811fe61e&gt;] vfs_read+0x9e/0x170 [&lt;ffffffff811ff1ef&gt;] SyS_read+0x7f/0xe0 [&lt;ffffffff816973c9&gt;] system_call_fastpath+0x16/0x1b ---[ end trace 5808138721c1994b ]--- Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 </pre> </div></div> <p>I'm running the test again with "--noflush" removed.</p> <p>With "--noflush" removed, the full stack is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: dmsetup suspend --nolockfs /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: dmsetup load /dev/mapper/mds1_flakey --table "0 4194304 flakey 252:0 0 0 1800 1 drop_writes" Lustre: DEBUG MARKER: dmsetup resume /dev/mapper/mds1_flakey Buffer I/O error on dev dm-6, logical block 524272, async page read ------------[ cut here ]------------ WARNING: at fs/buffer.c:295 end_buffer_async_read+0x119/0x140() Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr virtio_balloon i2c_piix4 parport_pc parport nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul 8139too crct10dif_common ttm crc32c_intel serio_raw virtio_pci 8139cp virtio_ring mii virtio drm ata_piix i2c_core libata floppy CPU: 1 PID: 14904 Comm: systemd-udevd Tainted: G W OE ------------ 3.10.0-514.16.1.el7_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 0000000000000000 00000000cc24690a ffff88007b733990 ffffffff81686d2f ffff88007b7339c8 ffffffff81085cb0 ffff88007a994f00 ffff88006290fa90 ffff88007a994f00 ffffea0001ea3400 ffff880078178800 ffff88007b7339d8 Call Trace: [&lt;ffffffff81686d2f&gt;] dump_stack+0x19/0x1b [&lt;ffffffff81085cb0&gt;] warn_slowpath_common+0x70/0xb0 [&lt;ffffffff81085dfa&gt;] warn_slowpath_null+0x1a/0x20 [&lt;ffffffff81234269&gt;] end_buffer_async_read+0x119/0x140 [&lt;ffffffff8123263c&gt;] end_bio_bh_io_sync+0x2c/0x60 [&lt;ffffffff81239454&gt;] bio_endio+0x64/0xa0 [&lt;ffffffffa063b8da&gt;] dec_pending+0x18a/0x2e0 [dm_mod] [&lt;ffffffffa063d359&gt;] __split_and_process_bio+0x2d9/0x500 [dm_mod] [&lt;ffffffff812e0000&gt;] ? aes_encrypt+0xb30/0xe00 [&lt;ffffffffa063d68e&gt;] dm_make_request+0x10e/0x170 [dm_mod] [&lt;ffffffff812eee19&gt;] generic_make_request+0x109/0x1e0 [&lt;ffffffff812eef61&gt;] submit_bio+0x71/0x150 [&lt;ffffffff812381dd&gt;] ? bio_alloc_bioset+0x1fd/0x350 [&lt;ffffffff81233283&gt;] _submit_bh+0x143/0x210 [&lt;ffffffff81235bd7&gt;] block_read_full_page+0x217/0x350 [&lt;ffffffff81239670&gt;] ? I_BDEV+0x10/0x10 [&lt;ffffffff8118e219&gt;] ? force_page_cache_readahead+0x99/0xe0 [&lt;ffffffff8123a0a8&gt;] blkdev_readpage+0x18/0x20 [&lt;ffffffff811829df&gt;] generic_file_aio_read+0x3cf/0x790 [&lt;ffffffff8123a46c&gt;] blkdev_aio_read+0x4c/0x70 [&lt;ffffffff811fde6d&gt;] do_sync_read+0x8d/0xd0 [&lt;ffffffff811fe61e&gt;] vfs_read+0x9e/0x170 [&lt;ffffffff811ff1ef&gt;] SyS_read+0x7f/0xe0 [&lt;ffffffff816973c9&gt;] system_call_fastpath+0x16/0x1b ---[ end trace 0cce06cbf33ea2b1 ]--- Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000 LustreError: 12953:0:(osd_io.c:1460:osd_ldiskfs_read()) lustre-MDT0000: can't read 4096@0 on ino 265: rc = -5 LustreError: 12953:0:(osd_io.c:1460:osd_ldiskfs_read()) Skipped 6 previous similar messages LustreError: 12953:0:(llog_osd.c:262:llog_osd_read_header()) lustre-MDT0000-osd: bad log [0xa:0xb:0x0] header magic: 0x0 (expected 0x10645539) LustreError: 12953:0:(llog_osd.c:262:llog_osd_read_header()) Skipped 6 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 </pre> </div></div> <p>Hi Andreas,</p> <p>I also removed "&#45;&#45;nolockfs" and got the same failure and full stack. So, the failure is not related to "&#45;&#45;nolockfs" and/or "&#45;&#45;noflush" options.</p> <p>Hi Hongchao,</p> <p>Could you please proceed with this blocker? Thank you.</p> <p>Status Update:</p> <p>I have tested it in Maloo by disabling different patches contained in <a href="http://review.whamcloud.com/22113" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22113</a> and<br/> <a href="http://review.whamcloud.com/23560" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23560</a> (the debug patch is tracked at <a href="https://review.whamcloud.com/#/c/26788/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/26788/</a>, the patch set is 15 ~ 21).<br/> the error "Buffer I/O error" still occur.</p> <p>I looked at the test result in the original patch for <a href="https://jira.whamcloud.com/browse/LU-684" title="replace dev_rdonly kernel patch with dm-flakey" class="issue-link" data-issue-key="LU-684"><del>LU-684</del></a> (<a href="https://review.whamcloud.com/#/c/7200/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/7200/</a>), there is no such problem prior to<br/> patch set 20 and start to occur from patch set 21, the difference of Lustre itself between the two patch set is the two patches<br/> <a href="http://review.whamcloud.com/22113" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22113</a> and <a href="http://review.whamcloud.com/23560" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23560</a>, but the more big different is the change of kernel,<br/> patch set 20 still used RHEL7.2 to test, but patch set 21 began to use RHEL7.3.</p> <p>Patch Set 20: <a href="https://testing.hpdd.intel.com/test_sessions/c01176f8-a5ef-11e6-b605-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/c01176f8-a5ef-11e6-b605-5254006e85c2</a>
 (3.10.0-327.36.3.el7.x86_64)<br/> Patch Set 21: <a href="https://testing.hpdd.intel.com/test_sessions/41fe0cb2-beeb-11e6-92c6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/41fe0cb2-beeb-11e6-92c6-5254006e85c2</a> (3.10.0-514.el7.x86_64)</p> <p>Then the issue could be caused by the kernel RHEL7.3 (3.10.0-514.el7.x86_64)</p> <p>I have setup the RHEL7.3 (3.10.0-514.2.2.el7.x86_64) and reproduce the problem by using ext4 and dm-flakey module<br/> the reproducer is attached.</p> <p>Can you try linux-commit: 299f6230bc6d0ccd5f95bb0fb865d80a9c7d5ccc</p> <p>dm flakey: fix reads to be issued if drop_writes configured</p> <p>v4.8-rc3 commit 99f3c90d0d ("dm flakey: error READ bios during the<br/> down_interval") overlooked the 'drop_writes' feature, which is meant to<br/> allow reads to be issued rather than errored, during the down_interval.</p> <p>Fixes: 99f3c90d0d ("dm flakey: error READ bios during the down_interval")<br/> Reported-by: Qu Wenruo &lt;quwenruo@cn.fujitsu.com&gt;<br/> Signed-off-by: Mike Snitzer &lt;snitzer@redhat.com&gt;<br/> Cc: stable@vger.kernel.org</p> <p>Hi James,</p> <blockquote><p>Can you try linux-commit: 299f6230bc6d0ccd5f95bb0fb865d80a9c7d5ccc</p></blockquote> <p>I tried this in <a href="https://review.whamcloud.com/26788" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26788</a> (patch set 25). The same error still occurred:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Buffer I/O error on dev dm-6, logical block 524272, async page read </pre> </div></div> <p>I have tested the commit locally, the commit can fix the problem on 3.10.0-514.2.2.el7.x86_64,<br/> but can't on 3.10.0-514.21.1.el7.x86_64, which is being used in our Auotest system.</p> <p>The autotest system uses the build with the following kernel version supported by the latest master branch:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ head -n2 lustre/kernel_patches/targets/3.10-rhel7.target.in lnxmaj="3.10.0" lnxrel="514.21.1.el7" </pre> </div></div> <p>So, the commit doesn't work and the failure still exists.</p> <p>So the good news is this is fixed in RHEL7.4. If some one really wants to work on RHEL7.3 can compare the sources in RHEL7.4 to RHLE7.3 to see which change fixed this problem so we can include the patch fpr RHEL7.3 if so desired. I updated the patch for <a href="https://jira.whamcloud.com/browse/LU-684" title="replace dev_rdonly kernel patch with dm-flakey" class="issue-link" data-issue-key="LU-684"><del>LU-684</del></a> in the mean time to move this work forward.</p> <p>No - let's just focus on RHEL 7.4</p> Blocker LU-684 Duplicate LU-7481 Related LU-20 Development Rank 1|hzysgf: Rank (Obsolete) 9223372036854775807 Severity