|
Actually, the write failed with EIO instead of ENOSPC:
But the client still got ENOSPC:
Looking closer, ENOSPC was returned on an async write and the following fsync got EIO:
I think the problem could be probably be avoided if using synchronous write (like direct i/o) in the test.
That said, i still don't understand why fsync did not fail with ENOSPC. We should end up in vvp_vmpage_error() and set AS_ENOSPC on the inode mapping and then filemap_fdatawait_range() should check for this bit and return ENOSPC instead of EIO ... strange.
|
|
Niu
Could you please investigate more?
Thanks
Peter
|
|
Hi, Sarah
Can this be reproduced on kernels other than 2.1.52--PRISTINE-2.6.32.36-0.5-default? I suspect it because some special kernel treat the ENOSPC differently in filemap_fdatawrite_range(), if that's true, I think we can just relax the test error condition a little bit, all kinds of write error can be regarded as pass.
Where can we get the source of 2.6.32.36?
|
|
No, this error only shown on SLES kernel
|
|
I checked the source of 2.6.32.36, but didn't see why the ENOSPC isn't returned for the write syscall. I'll provide a patch changing the test from O_SYNC write to O_DIRECT write as Johann suggested.
|
|
http://review.whamcloud.com/1748
Sarah, could you try if this works for the sles kernel?
|
|
I rechecked the code, seems the culprit is that we don't use writeback bit for pages, which breaks the kernel wait page writeback APIs.
Kernel 2.6.32.36 doesn't have writev, so it's write code path is different with other kernels having writev.
For O_SYNC write, 2.6.32.36 calls vfs_fsync_range() to sync pages and wait, and the vfs_fsync_range() calls filemap_write_and_wait_range() to wait for io completion then check the AS_ENOSPC on mapping->flags, however, since we never set writeback bit for writing pages, this wait then check function might be finished before io completed! That's why the AS_ENOSPC set on io completion isn't detected.
Other kernerls which have writev goes different code path, they call sync_page_range() to wait & check, the sync_page_range() always call generic_osync_inode() before calling the wait_on_page_writeback_range(), so io should probably has completed before calling the wait_on_page_writeback_range(), and the AS_ENOSPC is deteced.
But I don't see why write returns -EIO for 2.6.32.36 so far. Hi Sarah, could you reproduce the bug with D_TRACE log enabled? Then I could probably figure out where the -EIO is come from. Thanks in advance.
|
|
ok, will keep you updated
|
I rechecked the code, seems the culprit is that we don't use writeback bit for pages, which breaks the kernel wait page writeback APIs.
Hmm, looks we do use writeback bit. Anyway, let's see what can we find in D_TRACE log.
|
|
Here is the report with D_TRACE enabled.
https://maloo.whamcloud.com/test_sets/ce3dbbf8-1a5b-11e1-abb1-52540025f9af
|
|
Thank you, Sarah.
The log shows that ll_fsync() returned ENOSPC as well, but vvp_io_write_start() returned -EIO then.
00000080:00000010:3.0:1322551344.059576:0:7829:0:(file.c:1992:ll_fsync()) slab-freed '((oinfo->oi_oa))': 208 at ffff88031cbede50.
00000080:00000010:3.0:1322551344.059577:0:7829:0:(file.c:1993:ll_fsync()) kfreed 'oinfo': 104 at ffff88031ca61c40.
00000080:00000001:3.0:1322551344.059578:0:7829:0:(file.c:1997:ll_fsync()) Process leaving (rc=18446744073709551588 : -28 : ffffffffffffffe4)
00000080:00000001:3.0:1322551344.059580:0:7829:0:(vvp_io.c:616:vvp_io_write_start()) Process leaving (rc=18446744073709551611 : -5 : fffffffffffffffb)
Look at the vfs_fsync_range()
err = fop->fsync(file, dentry, datasync);
if (!ret)
ret = err;
so if the 'ret' already been set as -EIO, the ENOSPC returned by fsync will be ignored, but I didn't figure out why the 'ret' is -EIO yet.
|
|
Jinshan is just working on implementing proper PG_writeback support for CLIO. Possibly this will fix the problem as well. I've added him to this bug to get some advice on whether this is correct.
If we know the problem is confined to this specific kernel version, and Jinshan will (I think) be delivering a fix for this soon, I would be OK with adding this test to ALWAYS_EXCEPT for this specific kernel version. That way, the test is still run on other kernels, and if we move to a new kernel and the bug is not fixed it will appear again, instead of being forgotten (like some other bugs in ALWAYS_EXCEPT today).
|
Jinshan is just working on implementing proper PG_writeback support for CLIO. Possibly this will fix the problem as well. I've added him to this bug to get some advice on whether this is correct.
Sorry, my earlier analysis about writeback flag is wrong, we do set/clear PG_writeback for page transfer.
|
|
Ah, I realized that the kernel version is 2.6.32.43-0.4-default but not 2.6.32.36-0.5, the versions: 2.1.52--PRISTINE-2.6.32.36-0.5-default (x86_64) displayed on maloo confused me...
Hi, Sarah, where can I find the source of 2.6.32.43-0.4? Thanks.
|
|
Sarah
Are you able to answer Niu's question about the 2.6.32.43-0.4 source?
If not, is there someone else who can point you in the right direction Niu?
|
|
I tried to find the source on brent several weeks ago and failed. Niu knows that.
|
|
Maybe Chris or Mike knows where we saved the kernel source?
|
|
SLES kernel source is hard to find, if you don't know where to look.
I have downloaded the requested SRPM:
mjmac@brent:~$ ls -l /scratch/k_srpm/sles11/kernel-source-2.6.32.43-0.4.1.src.rpm
total 82080
-rw-r--r-- 1 mjmac mjmac 84043443 2011-07-17 19:33 kernel-source-2.6.32.43-0.4.1.src.rpm
For the future, others can download kernel sources directly from novell's update server using special mirror-only credentials (Note the single-quotes around the URL – necessary to protect $RCE from shell expansion):
# e.g. for the kernel requested in this ticket:
wget 'https:
These credentials can be entered into a web browser and used for browsing the novell updates mirror. They can't be used for logging into the main novell partner portal or anything other than nu.novell.com.
|
|
Thank you, Mike.
I checked the source (2.6.32.43-0.4.1), but still don't see why the write returned -EIO. Sarah, how often did you see this bug on sles (2.6.32.43-0.4-default) client? did it appeared in every test?
|
|
Niu,
As I remembered, it was easy to reproduced.
|
|
Sarah
Have you repeated this test with 2.1.55 yet?
Peter
|
|
yes, hit the same error: https://maloo.whamcloud.com/test_sets/efca1288-51a8-11e1-8783-5254004bbbd3
|
|
Hi, Sarah
Could you apply this debug patch to client and repeat the test with D_TRACE & D_PAGE enabled?
|
|
sure, will keep you updated. it may take some time due to the 2.2 testing, sorry about that.
|
|
Lustre Tag: v2_1_1_0_RC1
Distro/Arch: RHEL5.7 (server), SLES11SP1 (client)
Lustre Build: http://build.whamcloud.com/job/lustre-b2_1/37/
Network: TCP
The same issue occurred:
https://maloo.whamcloud.com/test_sets/0ac86ad8-52ff-11e1-81a2-5254004bbbd3
|
Hi, Sarah
Could you apply this debug patch to client and repeat the test with D_TRACE & D_PAGE enabled?
Hi Niu, do you mean this patch? http://review.whamcloud.com/#change,1748
|
Hi Niu, do you mean this patch? http://review.whamcloud.com/#change,1748
I meant the patch attached here: LU-869-debug.patch.
|
|
Hi Niu, could you please just submit the patch to review and tell me the build number?
Thanks.
|
|
ok, I uploaded the diagnostic patch in the patchset 2 of http://review.whamcloud.com/#change,1748
|
|
https://maloo.whamcloud.com/test_sets/6e7dd01e-599d-11e1-a26b-5254004bbbd3
For server log please the attached.
|
|
Thanks, Sarah. The log shows that AS_ENOSPC was set on the proper i_mapping, and I can't tell why it's not transfered to -ENOSPCE in wait_on_page_writeback_range() from the kernel code.
|
|
Niu, could you please run this test locally to try and debug it. I'm repeatedly seeing IO errors running sanity.sh test_27[opqv] during file object creations running on a RHEL6 kernel.
|
|
Hi, Andreas, did your tests pass or not?
I tried the tests locally, and they all passed, I think the EIO error messages are what we expected, since these tests are trying to create file after exhausting the precreation, and that should get EIO error from the server. (see qos_prep_create()->alloc_idx_array(), when we fail to create any stripe, -EIO will be returned the client)
|
|
another hit for 2.2 https://maloo.whamcloud.com/test_sets/9d70bce8-6ed7-11e1-b737-5254004bbbd3
|
|
Got this issue on 2.2-RC1 test, RHEL6 server/SUSE client:
https://maloo.whamcloud.com/test_sets/5c52111e-70f6-11e1-a89e-5254004bbbd3
|
|
it looks like this issue can only be seen on suse clients. I don't know what's the difference between rhel and suse clients.
|
|
tag-2.2.54 RHEL6 server/SLES client hit this issue again: https://maloo.whamcloud.com/test_sets/f0c1b98c-ad58-11e1-8152-52540035b04c
|
|
Lustre Tag: v2_1_2_RC2
Lustre Build: http://build.whamcloud.com/job/lustre-b2_1/87/
Distro/Arch: RHEL6.2/x86_64(server), SLES11SP1/x86_64(client)
Network: TCP (1GigE)
ENABLE_QUOTA=yes
The same issue occurred: https://maloo.whamcloud.com/test_sets/a3243c4a-ae52-11e1-ae0d-52540035b04c
|
|
This looks a duplicate of LU-2131.
|
|
Closing as a duplicate LU-2131.
|
Generated at Sat Feb 10 01:11:10 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
869.tar.gz
LU-869-debug.patch