Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-869] sanity.sh test_81b failed with "dd should fail for -ENOSPC, but succeed" https://jira.whamcloud.com/browse/LU-869 Lustre <p>This issue was created by maloo for sarah <sarah@whamcloud.com></p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/d3d44298-114c-11e1-ad46-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d3d44298-114c-11e1-ad46-52540025f9af</a>.<br/> Found similar error in ORI-403</p> LU-869

sanity.sh test_81b failed with "dd should fail for -ENOSPC, but succeed"

Bug Major Resolved Duplicate Niu Yawei Maloo Mon, 21 Nov 2011 16:27:21 +0000 Tue, 29 Sep 2015 08:12:23 +0000 Tue, 29 Sep 2015 08:12:22 +0000 Lustre 2.2.0 Lustre 2.1.1 Lustre 2.1.2 0 10 <p>Actually, the write failed with EIO instead of ENOSPC:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 81b: OST should return -ENOSPC when retry still fails ================================= 15:41:11 (1321486871) fail_loc=0x228 write: Input/output error sanity test_81b: @@@@@@ FAIL: dd should fail for -ENOSPC, but succeed. </pre> </div></div> <p>But the client still got ENOSPC:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:02020000:0.0:1321486871.819507:0:4759:0:(client.c:1141:ptlrpc_check_status()) 11-0: an error occurred while communicating with 10.10.4.77@tcp. The ost_write operation failed with -28 </pre> </div></div> <p>Looking closer, ENOSPC was returned on an async write and the following fsync got EIO:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000080:00200000:0.0:1321486871.744302:0:30812:0:(vvp_io.c:601:vvp_io_write_start()) write: [0, 4096) 00000080:00200000:0.0:1321486871.744305:0:30812:0:(xattr.c:419:ll_getxattr()) VFS Op:inode=144115238843711498/33554443(ffff8800751fe630), xattr security.capability 00000080:00200000:0.0:1321486871.744309:0:30812:0:(xattr.c:284:ll_getxattr_common()) VFS Op:inode=144115238843711498/33554443(ffff8800751fe630) 00000080:00200000:0.0:1321486871.744326:0:30812:0:(rw.c:218:ll_cl_init()) 0@[0x200000bd2:0xa:0x0] -> 0 ffff88007561e878 ffff880075153de8 00000080:00200000:0.0:1321486871.744344:0:30812:0:(vvp_io.c:1033:vvp_io_commit_write()) [0x200000bd2:0xa:0x0] updating i_size 4096 00000100:00100000:0.0:1321486871.744406:0:4759:0:(client.c:1411:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_1:cce1ed61-bd46-1d91-8ece-ef709b91e2ba:4759:1385676188533420:10.10.4.77@tcp:4 00000100:02020000:0.0:1321486871.819507:0:4759:0:(client.c:1141:ptlrpc_check_status()) 11-0: an error occurred while communicating with 10.10.4.77@tcp. The ost_write operation failed with -28 00000100:00100000:0.0:1321486871.819541:0:4759:0:(client.c:1745:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_1:cce1ed61-bd46-1d91-8ece-ef709b91e2ba:4759:1385676188533420:10.10.4.77@tcp:4 </pre> </div></div> <p>I think the problem could be probably be avoided if using synchronous write (like direct i/o) in the test.<br/> That said, i still don't understand why fsync did not fail with ENOSPC. We should end up in vvp_vmpage_error() and set AS_ENOSPC on the inode mapping and then filemap_fdatawait_range() should check for this bit and return ENOSPC instead of EIO ... strange.</p> <p>Niu</p> <p>Could you please investigate more?</p> <p>Thanks</p> <p>Peter</p> <p>Hi, Sarah</p> <p>Can this be reproduced on kernels other than 2.1.52--PRISTINE-2.6.32.36-0.5-default? I suspect it because some special kernel treat the ENOSPC differently in filemap_fdatawrite_range(), if that's true, I think we can just relax the test error condition a little bit, all kinds of write error can be regarded as pass.</p> <p>Where can we get the source of 2.6.32.36?</p> <p>No, this error only shown on SLES kernel</p> <p>I checked the source of 2.6.32.36, but didn't see why the ENOSPC isn't returned for the write syscall. I'll provide a patch changing the test from O_SYNC write to O_DIRECT write as Johann suggested.</p> <p><a href="http://review.whamcloud.com/1748" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1748</a></p> <p>Sarah, could you try if this works for the sles kernel?</p> <p>I rechecked the code, seems the culprit is that we don't use writeback bit for pages, which breaks the kernel wait page writeback APIs.</p> <p>Kernel 2.6.32.36 doesn't have writev, so it's write code path is different with other kernels having writev.</p> <p>For O_SYNC write, 2.6.32.36 calls vfs_fsync_range() to sync pages and wait, and the vfs_fsync_range() calls filemap_write_and_wait_range() to wait for io completion then check the AS_ENOSPC on mapping->flags, however, since we never set writeback bit for writing pages, this wait then check function might be finished before io completed! That's why the AS_ENOSPC set on io completion isn't detected.</p> <p>Other kernerls which have writev goes different code path, they call sync_page_range() to wait & check, the sync_page_range() always call generic_osync_inode() before calling the wait_on_page_writeback_range(), so io should probably has completed before calling the wait_on_page_writeback_range(), and the AS_ENOSPC is deteced.</p> <p>But I don't see why write returns -EIO for 2.6.32.36 so far. Hi Sarah, could you reproduce the bug with D_TRACE log enabled? Then I could probably figure out where the -EIO is come from. Thanks in advance.</p> <p>ok, will keep you updated</p> <blockquote> <p>I rechecked the code, seems the culprit is that we don't use writeback bit for pages, which breaks the kernel wait page writeback APIs.</p></blockquote> <p>Hmm, looks we do use writeback bit. Anyway, let's see what can we find in D_TRACE log.</p> <p>Here is the report with D_TRACE enabled.<br/> <a href="https://maloo.whamcloud.com/test_sets/ce3dbbf8-1a5b-11e1-abb1-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ce3dbbf8-1a5b-11e1-abb1-52540025f9af</a></p> <p>Thank you, Sarah.</p> <p>The log shows that ll_fsync() returned ENOSPC as well, but vvp_io_write_start() returned -EIO then.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">00000080:00000010:3.0:1322551344.059576:0:7829:0:(file.c:1992:ll_fsync()) slab-freed <span class="code-quote">'((oinfo->oi_oa))'</span>: 208 at ffff88031cbede50. 00000080:00000010:3.0:1322551344.059577:0:7829:0:(file.c:1993:ll_fsync()) kfreed <span class="code-quote">'oinfo'</span>: 104 at ffff88031ca61c40. 00000080:00000001:3.0:1322551344.059578:0:7829:0:(file.c:1997:ll_fsync()) <span class="code-object">Process</span> leaving (rc=18446744073709551588 : -28 : ffffffffffffffe4) 00000080:00000001:3.0:1322551344.059580:0:7829:0:(vvp_io.c:616:vvp_io_write_start()) <span class="code-object">Process</span> leaving (rc=18446744073709551611 : -5 : fffffffffffffffb) </pre> </div></div> <p>Look at the vfs_fsync_range()</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> err = fop->fsync(file, dentry, datasync); <span class="code-keyword">if</span> (!ret) ret = err; </pre> </div></div> <p>so if the 'ret' already been set as -EIO, the ENOSPC returned by fsync will be ignored, but I didn't figure out why the 'ret' is -EIO yet.</p> <p>Jinshan is just working on implementing proper PG_writeback support for CLIO. Possibly this will fix the problem as well. I've added him to this bug to get some advice on whether this is correct.</p> <p>If we know the problem is confined to this specific kernel version, and Jinshan will (I think) be delivering a fix for this soon, I would be OK with adding this test to ALWAYS_EXCEPT for this specific kernel version. That way, the test is still run on other kernels, and if we move to a new kernel and the bug is not fixed it will appear again, instead of being forgotten (like some other bugs in ALWAYS_EXCEPT today).</p> <blockquote> <p>Jinshan is just working on implementing proper PG_writeback support for CLIO. Possibly this will fix the problem as well. I've added him to this bug to get some advice on whether this is correct.</p></blockquote> <p>Sorry, my earlier analysis about writeback flag is wrong, we do set/clear PG_writeback for page transfer. </p> <p>Ah, I realized that the kernel version is 2.6.32.43-0.4-default but not 2.6.32.36-0.5, the versions: 2.1.52--PRISTINE-2.6.32.36-0.5-default (x86_64) displayed on maloo confused me...</p> <p>Hi, Sarah, where can I find the source of 2.6.32.43-0.4? Thanks.</p> <p>Sarah</p> <p>Are you able to answer Niu's question about the 2.6.32.43-0.4 source?</p> <p>If not, is there someone else who can point you in the right direction Niu? </p> <p>I tried to find the source on brent several weeks ago and failed. Niu knows that. </p> <p>Maybe Chris or Mike knows where we saved the kernel source?</p> <p>SLES kernel source is hard to find, if you don't know where to look.</p> <p>I have downloaded the requested SRPM:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">mjmac@brent:~$ ls -l /scratch/k_srpm/sles11/kernel-source-2.6.32.43-0.4.1.src.rpm total 82080 -rw-r--r-- 1 mjmac mjmac 84043443 2011-07-17 19:33 kernel-source-2.6.32.43-0.4.1.src.rpm </pre> </div></div> <p>For the future, others can download kernel sources directly from novell's update server using special mirror-only credentials (Note the single-quotes around the URL – necessary to protect $RCE from shell expansion):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># e.g. <span class="code-keyword">for</span> the kernel requested in <span class="code-keyword">this</span> ticket: wget <span class="code-quote">'https:<span class="code-comment">//UC1714458:a31da933d6@nu.novell.com/repo/$RCE/SLES11-SP1-Updates/sle-11-x86_64/rpm/src/kernel-source-2.6.32.43-0.4.1.src.rpm'</span></span> </pre> </div></div> <p>These credentials can be entered into a web browser and used for browsing the novell updates mirror. They can't be used for logging into the main novell partner portal or anything other than nu.novell.com.</p> <p>Thank you, Mike.</p> <p>I checked the source (2.6.32.43-0.4.1), but still don't see why the write returned -EIO. Sarah, how often did you see this bug on sles (2.6.32.43-0.4-default) client? did it appeared in every test?</p> <p>Niu,</p> <p>As I remembered, it was easy to reproduced.</p> <p>Sarah</p> <p>Have you repeated this test with 2.1.55 yet?</p> <p>Peter</p> <p>yes, hit the same error: <a href="https://maloo.whamcloud.com/test_sets/efca1288-51a8-11e1-8783-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/efca1288-51a8-11e1-8783-5254004bbbd3</a></p> <p>Hi, Sarah<br/> Could you apply this debug patch to client and repeat the test with D_TRACE & D_PAGE enabled?</p> <p>sure, will keep you updated. it may take some time due to the 2.2 testing, sorry about that.</p> <p>Lustre Tag: v2_1_1_0_RC1<br/> Distro/Arch: RHEL5.7 (server), SLES11SP1 (client)<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/37/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/37/</a><br/> Network: TCP</p> <p>The same issue occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/0ac86ad8-52ff-11e1-81a2-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0ac86ad8-52ff-11e1-81a2-5254004bbbd3</a></p> <blockquote> <p>Hi, Sarah<br/> Could you apply this debug patch to client and repeat the test with D_TRACE & D_PAGE enabled?</p></blockquote> <p>Hi Niu, do you mean this patch? <a href="http://review.whamcloud.com/#change,1748" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1748</a></p> <blockquote> <p>Hi Niu, do you mean this patch? <a href="http://review.whamcloud.com/#change,1748" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1748</a></p></blockquote> <p>I meant the patch attached here: <a href="https://jira.whamcloud.com/browse/LU-869" title="sanity.sh test_81b failed with "dd should fail for -ENOSPC, but succeed"" class="issue-link" data-issue-key="LU-869"><del>LU-869</del></a>-debug.patch.</p> <p>Hi Niu, could you please just submit the patch to review and tell me the build number? </p> <p>Thanks.</p> <p>ok, I uploaded the diagnostic patch in the patchset 2 of <a href="http://review.whamcloud.com/#change,1748" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1748</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/6e7dd01e-599d-11e1-a26b-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6e7dd01e-599d-11e1-a26b-5254004bbbd3</a></p> <p>For server log please the attached.</p> <p>Thanks, Sarah. The log shows that AS_ENOSPC was set on the proper i_mapping, and I can't tell why it's not transfered to -ENOSPCE in wait_on_page_writeback_range() from the kernel code.</p> <p>Niu, could you please run this test locally to try and debug it. I'm repeatedly seeing IO errors running sanity.sh test_27<span class="error">[opqv]</span> during file object creations running on a RHEL6 kernel.</p> <p>Hi, Andreas, did your tests pass or not?</p> <p>I tried the tests locally, and they all passed, I think the EIO error messages are what we expected, since these tests are trying to create file after exhausting the precreation, and that should get EIO error from the server. (see qos_prep_create()->alloc_idx_array(), when we fail to create any stripe, -EIO will be returned the client)</p> <p>another hit for 2.2 <a href="https://maloo.whamcloud.com/test_sets/9d70bce8-6ed7-11e1-b737-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9d70bce8-6ed7-11e1-b737-5254004bbbd3</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 81b: OST should return -ENOSPC when retry still fails ================================= 21:47:12 (1331786832) fail_loc=0x228 write: Input/output error sanity test_81b: @@@@@@ FAIL: dd should fail for -ENOSPC, but succeed. </pre> </div></div> <p>Got this issue on 2.2-RC1 test, RHEL6 server/SUSE client:<br/> <a href="https://maloo.whamcloud.com/test_sets/5c52111e-70f6-11e1-a89e-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/5c52111e-70f6-11e1-a89e-5254004bbbd3</a></p> <p>it looks like this issue can only be seen on suse clients. I don't know what's the difference between rhel and suse clients.</p> <p>tag-2.2.54 RHEL6 server/SLES client hit this issue again: <a href="https://maloo.whamcloud.com/test_sets/f0c1b98c-ad58-11e1-8152-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f0c1b98c-ad58-11e1-8152-52540035b04c</a></p> <p>Lustre Tag: v2_1_2_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/87/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/87/</a><br/> Distro/Arch: RHEL6.2/x86_64(server), SLES11SP1/x86_64(client)<br/> Network: TCP (1GigE)<br/> ENABLE_QUOTA=yes</p> <p>The same issue occurred: <a href="https://maloo.whamcloud.com/test_sets/a3243c4a-ae52-11e1-ae0d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a3243c4a-ae52-11e1-ae0d-52540035b04c</a></p> <p>This looks a duplicate of <a href="https://jira.whamcloud.com/browse/LU-2131" title="sanity test using ost0 which does not exist." class="issue-link" data-issue-key="LU-2131"><del>LU-2131</del></a>.</p> <p>Closing as a duplicate <a href="https://jira.whamcloud.com/browse/LU-2131" title="sanity test using ost0 which does not exist." class="issue-link" data-issue-key="LU-2131"><del>LU-2131</del></a>.</p> Duplicate LU-2131 Related LU-3801 LU-1228 Development Rank 1|hzv4f3: Rank (Obsolete) 4249 Severity