https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5245 Lustre <p>This issue was created by maloo for wangdi &lt;di.wang@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/540fb674-fad7-11e3-b499-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/540fb674-fad7-11e3-b499-52540035b04c</a>.</p> <p>The sub-test test_1 failed with the following error:</p> <blockquote> <p>Total allocated inode limit: 0, total allocated block limit: 0<br/> Files for user (quota_usr):<br/> File: `/mnt/lustre/d1.sanity-quota/f1.sanity-quota-0'<br/> Size: 11534336 Blocks: 22528 IO Block: 4194304 regular file<br/> Device: 2c54f966h/743766374d Inode: 288230930219270255 Links: 1<br/> Access: (0644/<del>rw-r</del><del>r</del>-) Uid: (60000/quota_usr) Gid: (60000/quota_usr)<br/> Access: 2014-06-23 00:18:35.000000000 -0700<br/> Modify: 2014-06-23 00:18:36.000000000 -0700<br/> Change: 2014-06-23 00:18:36.000000000 -0700<br/> sanity-quota test_1: @@@@@@ FAIL: user write success, but expect EDQUOT <br/> Trace dump:<br/> = /usr/lib64/lustre/tests/test-framework.sh:4528:error_noexit()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4559:error()<br/> = /usr/lib64/lustre/tests/sanity-quota.sh:154:quota_error()<br/> = /usr/lib64/lustre/tests/sanity-quota.sh:440:test_1()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4820:run_one()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4855:run_one_logged()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4708:run_test()<br/> = /usr/lib64/lustre/tests/sanity-quota.sh:483:main()<br/> user write success, but expect EDQUOT</p></blockquote> <p>Info required for matching: sanity-quota 1</p> LU-5245

sanity-quota test_1: user write success, but expect EDQUOT

Bug Minor Resolved Incomplete Niu Yawei Maloo Mon, 23 Jun 2014 17:16:31 +0000 Tue, 20 Nov 2018 21:02:51 +0000 Tue, 20 Nov 2018 21:02:51 +0000 Lustre 2.6.0 Lustre 2.7.0 Lustre 2.8.0 0 12 <p>Niu,<br/> Can you please comment on this one?<br/> Thank you!</p> <blockquote> <p>01:18:26:== sanity-quota test 1: Block hard limit (normal use and out of quota) == 00:18:26 (1403507906)<br/> 01:18:28:Waiting for local destroys to complete<br/> 01:18:29:Creating test directory<br/> 01:18:29:CMD: shadow-47vm3,shadow-47vm4,shadow-47vm7 lctl set_param fail_val=0 fail_loc=0<br/> 01:18:29:fail_val=0<br/> 01:18:29:fail_loc=0<br/> 01:18:29:fail_val=0<br/> 01:18:29:fail_loc=0<br/> 01:18:29:fail_val=0<br/> 01:18:29:fail_loc=0<br/> 01:18:29:CMD: shadow-47vm3 /usr/sbin/lctl conf_param lustre.quota.ost=ug<br/> 01:18:29:CMD: shadow-47vm4 /usr/sbin/lctl dl<br/> 01:18:29:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0000.quota_slave.enabled<br/> 01:18:29:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0000.quota_slave.enabled<br/> 01:18:29:CMD: shadow-47vm4 /usr/sbin/lctl dl<br/> 01:18:30:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0001.quota_slave.enabled<br/> 01:18:30:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0001.quota_slave.enabled<br/> 01:18:30:CMD: shadow-47vm4 /usr/sbin/lctl dl<br/> 01:18:30:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0002.quota_slave.enabled<br/> 01:18:31:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0002.quota_slave.enabled<br/> 01:18:32:CMD: shadow-47vm4 /usr/sbin/lctl dl<br/> 01:18:32:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0003.quota_slave.enabled<br/> 01:18:32:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0003.quota_slave.enabled<br/> 01:18:32:CMD: shadow-47vm4 /usr/sbin/lctl dl<br/> 01:18:32:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0004.quota_slave.enabled<br/> 01:18:33:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0004.quota_slave.enabled<br/> 01:18:33:CMD: shadow-47vm4 /usr/sbin/lctl dl<br/> 01:18:33:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0005.quota_slave.enabled<br/> 01:18:33:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0005.quota_slave.enabled<br/> 01:18:34:CMD: shadow-47vm4 /usr/sbin/lctl dl<br/> 01:18:34:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0006.quota_slave.enabled<br/> 01:18:34:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0006.quota_slave.enabled<br/> 01:18:34:CMD: shadow-47vm4 /usr/sbin/lctl dl<br/> 01:18:34:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0007.quota_slave.enabled<br/> 01:18:35:CMD: shadow-47vm4 /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-OST0007.quota_slave.enabled<br/> 01:18:35:User quota (block hardlimit:10 MB)<br/> 01:18:35:Write...<br/> 01:18:36:running as uid/gid/euid/egid 60000/60000/60000/60000, groups:<br/> 01:18:36: <span class="error">&#91;dd&#93;</span> <span class="error">&#91;if=/dev/zero&#93;</span> <span class="error">&#91;bs=1M&#93;</span> <span class="error">&#91;of=/mnt/lustre/d1.sanity-quota/f1.sanity-quota-0&#93;</span> <span class="error">&#91;count=5&#93;</span><br/> 01:18:36:5+0 records in<br/> 01:18:36:5+0 records out<br/> 01:18:36:5242880 bytes (5.2 MB) copied, 0.117616 s, 44.6 MB/s<br/> 01:18:36:Write out of block quota ...<br/> 01:18:36:running as uid/gid/euid/egid 60000/60000/60000/60000, groups:<br/> 01:18:36: <span class="error">&#91;dd&#93;</span> <span class="error">&#91;if=/dev/zero&#93;</span> <span class="error">&#91;bs=1M&#93;</span> <span class="error">&#91;of=/mnt/lustre/d1.sanity-quota/f1.sanity-quota-0&#93;</span> <span class="error">&#91;count=5&#93;</span> <span class="error">&#91;seek=5&#93;</span><br/> 01:18:36:5+0 records in<br/> 01:18:36:5+0 records out<br/> 01:18:36:5242880 bytes (5.2 MB) copied, 0.0293611 s, 179 MB/s<br/> 01:18:36:running as uid/gid/euid/egid 60000/60000/60000/60000, groups:<br/> 01:18:36: <span class="error">&#91;dd&#93;</span> <span class="error">&#91;if=/dev/zero&#93;</span> <span class="error">&#91;bs=1M&#93;</span> <span class="error">&#91;of=/mnt/lustre/d1.sanity-quota/f1.sanity-quota-0&#93;</span> <span class="error">&#91;count=1&#93;</span> <span class="error">&#91;seek=10&#93;</span><br/> 01:18:36:1+0 records in<br/> 01:18:36:1+0 records out<br/> 01:18:36:1048576 bytes (1.0 MB) copied, 0.0343534 s, 30.5 MB/s<br/> 01:18:36:Disk quotas for user quota_usr (uid 60000):<br/> 01:18:36: Filesystem kbytes quota limit grace files quota limit grace<br/> 01:18:36: /mnt/lustre 11264* 0 10240 - 1 0 0 -<br/> 01:18:36:lustre-MDT0000_UUID<br/> 01:18:36: 0 - 0 - 1 - 0 -<br/> 01:18:36:lustre-MDT0001_UUID<br/> 01:18:36: 0 - 0 - 0 - 0 -<br/> 01:18:36:lustre-MDT0002_UUID<br/> 01:18:36: 0 - 0 - 0 - 0 -<br/> 01:18:36:lustre-MDT0003_UUID<br/> 01:18:36: 0 - 0 - 0 - 0 -<br/> 01:18:36:lustre-OST0000_UUID<br/> 01:18:36: 0 - 0 - - - - -<br/> 01:18:36:lustre-OST0001_UUID<br/> 01:18:36: 11264 - 0 - - - - -<br/> 01:18:36:lustre-OST0002_UUID<br/> 01:18:36: 0 - 0 - - - - -<br/> 01:18:36:lustre-OST0003_UUID<br/> 01:18:36: 0 - 0 - - - - -<br/> 01:18:36:lustre-OST0004_UUID<br/> 01:18:36: 0 - 0 - - - - -<br/> 01:18:36:lustre-OST0005_UUID<br/> 01:18:36: 0 - 0 - - - - -<br/> 01:18:36:lustre-OST0006_UUID<br/> 01:18:36: 0 - 0 - - - - -<br/> 01:18:36:lustre-OST0007_UUID<br/> 01:18:36: 0 - 0 - - - - -<br/> 01:18:36:Total allocated inode limit: 0, total allocated block limit: 0</p></blockquote> <p>quota wasn't enforced on OSTs, looks wait_update() was broken somehow? I didn't see the message like "Updated after..." or "Update not seen after..." on console message.<br/> The OSTs logs are missing in maloo too.</p> <p>I retriggered the test, unfortunately, this can not be reproduced. But it seems the failure rate of this test is pretty high</p> <p>Error: 'user write success, but expect EDQUOT' <br/> Failure Rate: 43.00% of last 100 executions <span class="error">&#91;all branches&#93;</span></p> <p>So could you please search the maloo test result? Thanks. </p> <p>We saw similar problem on a production system. A couple of users found their files are truncated when they copied new files to lustre. And finially, they reproduced the problem using following script:</p> <p>echo a &gt; testfile1 &amp;&amp; echo b &gt;&gt; testfile1 &amp;&amp; cat testfile1</p> <p>And the output of the script is always 'a\n' for these users. And the output of 'ls -l' shows that the file size is 2. However, we loged in another node, and run 'ls -l'. We found that the size of the 'testfile1' is actually 4 and the content of it is 'a\nb\n', which means the data has been written correctly. And please note root users do not have such kind of problem.</p> <p>We traced the operation, and found following logs.</p> <p>00000008:00000001:0.0:1407228580.717510:0:29229:0:(osc_cache.c:2274:osc_queue_async_io()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00000008:00000001:0.0:1407228580.717514:0:29229:0:(osc_page.c:224:osc_page_cache_add()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00000008:00000001:0.0:1407228580.717515:0:29229:0:(osc_io.c:313:osc_io_commit_async()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00000020:00000001:0.0:1407228580.717516:0:29229:0:(cl_io.c:801:cl_io_commit_async()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00020000:00000001:0.0:1407228580.717516:0:29229:0:(lov_io.c:665:lov_io_commit_async()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00000020:00000001:0.0:1407228580.717517:0:29229:0:(cl_io.c:801:cl_io_commit_async()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00000080:00000001:0.0:1407228580.742691:0:29229:0:(xattr.c:321:ll_getxattr_common()) Process leaving (rc=18446744073709551555 : -61 : ffffffffffffffc3)<br/> 00000008:00000001:0.0:1407228580.742728:0:29229:0:(osc_cache.c:2274:osc_queue_async_io()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00000008:00000001:0.0:1407228580.742731:0:29229:0:(osc_page.c:224:osc_page_cache_add()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00000008:00000001:0.0:1407228580.742732:0:29229:0:(osc_io.c:313:osc_io_commit_async()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00000020:00000001:0.0:1407228580.742733:0:29229:0:(cl_io.c:801:cl_io_commit_async()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00020000:00000001:0.0:1407228580.742734:0:29229:0:(lov_io.c:665:lov_io_commit_async()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00000020:00000001:0.0:1407228580.742734:0:29229:0:(cl_io.c:801:cl_io_commit_async()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00000008:00000001:0.0:1407228580.757374:0:29229:0:(osc_cache.c:2274:osc_queue_async_io()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00000008:00000001:0.0:1407228580.757377:0:29229:0:(osc_page.c:224:osc_page_cache_add()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00000008:00000001:0.0:1407228580.757378:0:29229:0:(osc_io.c:313:osc_io_commit_async()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00000020:00000001:0.0:1407228580.757379:0:29229:0:(cl_io.c:801:cl_io_commit_async()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00020000:00000001:0.0:1407228580.757379:0:29229:0:(lov_io.c:665:lov_io_commit_async()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)<br/> 00000020:00000001:0.0:1407228580.757380:0:29229:0:(cl_io.c:801:cl_io_commit_async()) Process leaving (rc=18446744073709551494 : -122 : ffffffffffffff86)</p> <p>It seems osc_quota_chkdq() returns NO_QUOTA. And +quota log is:</p> <p>00000001:04000000:0.0F:1407230285.413534:0:29229:0:(osc_quota.c:64:osc_quota_chkdq()) chkdq found noquota for user 5800<br/> 00000008:04000000:9.0F:1407230285.435331:0:1869:0:(osc_request.c:1528:osc_brw_fini_request()) setdq for <span class="error">&#91;5800 1090&#93;</span> with valid 0x6f184fb9, flags 2100<br/> 00000001:04000000:0.0:1407230285.435884:0:29229:0:(osc_quota.c:64:osc_quota_chkdq()) chkdq found noquota for user 5800<br/> 00000008:04000000:14.0F:1407230285.452187:0:1871:0:(osc_request.c:1528:osc_brw_fini_request()) setdq for <span class="error">&#91;5800 1090&#93;</span> with valid 0x6f184fb9, flags 2100<br/> 00000001:04000000:0.0:1407230285.455988:0:29229:0:(osc_quota.c:64:osc_quota_chkdq()) chkdq found noquota for user 5800<br/> 00000008:04000000:3.0F:1407230285.519352:0:1875:0:(osc_request.c:1528:osc_brw_fini_request()) setdq for <span class="error">&#91;5800 1090&#93;</span> with valid 0x6f184fb9, flags 2100</p> <p>It seems that Lustre thinks this user has no quota. However, first, the users definitely had not reached their space limits, and second 'echo &gt;&gt;' should return failure if the user's quota is exceeded.</p> <p>After we disable quota enforcement on OSTs, this problem is gone.</p> <p>Here is the trace log.</p> <blockquote> <p>It seems that Lustre thinks this user has no quota. However, first, the users definitely had not reached their space limits, and second 'echo &gt;&gt;' should return failure if the user's quota is exceeded.</p></blockquote> <p>osc_quota_chkdq() returns NO_QUOTA doesn't necessarily mean the user is already running out of quota, when the used space approaching quota limit, this function will return NO_QUOTA to inform client to turn the async write to sync write. Could you provide the output of 'lfs quota -v"? Thanks.</p> <blockquote> <p>And the output of the script is always 'a\n' for these users. And the output of 'ls -l' shows that the file size is 2. However, we loged in another node, and run 'ls -l'. We found that the size of the 'testfile1' is actually 4 and the content of it is 'a\nb\n', which means the data has been written correctly. And please note root users do not have such kind of problem.</p></blockquote> <p>I guess that's not a quota problem, we'd collect logs (on both clients and OST, better with D_TRACE enabled and some debug marker recording where the 'ls' start/end) of 'ls' to see why the file size is inconsistent on different client.</p> <p>Here it goes. Please note that we are using Lustre-2.4.x for servers and Lustre-2.5.x for client. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p><span class="error">&#91;12:26:53 root@r7:~&#93;</span> # lfs quota -v -u bjm900 /home<br/> Disk quotas for user bjm900 (uid 5800):<br/> Filesystem kbytes quota limit grace files quota limit grace<br/> /home 1599228 104857600 104857600 - 59224 1000000 1000000 -<br/> homsys-MDT0000_UUID<br/> 13428 - 0 - 59224 - 64206 -<br/> homsys-OST0000_UUID<br/> 21352 - 22376 - - - - -<br/> homsys-OST0001_UUID<br/> 16488 - 17512 - - - - -<br/> homsys-OST0002_UUID<br/> 12920 - 13576 - - - - -<br/> homsys-OST0003_UUID<br/> 23704 - 24220 - - - - -<br/> homsys-OST0004_UUID<br/> 17864 - 18888 - - - - -<br/> homsys-OST0005_UUID<br/> 27436 - 28160 - - - - -<br/> homsys-OST0006_UUID<br/> 12508 - 13532 - - - - -<br/> homsys-OST0007_UUID<br/> 20476 - 21496 - - - - -<br/> homsys-OST0008_UUID<br/> 11136 - 12156 - - - - -<br/> homsys-OST0009_UUID<br/> 21872 - 22896 - - - - -<br/> homsys-OST000a_UUID<br/> 13408 - 14432 - - - - -<br/> homsys-OST000b_UUID<br/> 15312 - 16336 - - - - -<br/> homsys-OST000c_UUID<br/> 39516 - 40536 - - - - -<br/> homsys-OST000d_UUID<br/> 21108 - 22132 - - - - -<br/> homsys-OST000e_UUID<br/> 17880 - 18904 - - - - -<br/> homsys-OST000f_UUID<br/> 24440 - 25464 - - - - -<br/> homsys-OST0010_UUID<br/> 18652 - 19676 - - - - -<br/> homsys-OST0011_UUID<br/> 36456 - 37476 - - - - -<br/> homsys-OST0012_UUID<br/> 17332 - 17864 - - - - -<br/> homsys-OST0013_UUID<br/> 28272 - 29296 - - - - -<br/> homsys-OST0014_UUID<br/> 32920 - 33944 - - - - -<br/> homsys-OST0015_UUID<br/> 21708 - 22728 - - - - -<br/> homsys-OST0016_UUID<br/> 21928 - 22952 - - - - -<br/> homsys-OST0017_UUID<br/> 15104 - 15872 - - - - -<br/> homsys-OST0018_UUID<br/> 18360 - 19384 - - - - -<br/> homsys-OST0019_UUID<br/> 22288 - 23304 - - - - -<br/> homsys-OST001a_UUID<br/> 11524 - 12548 - - - - -<br/> homsys-OST001b_UUID<br/> 23016 - 24040 - - - - -<br/> homsys-OST001c_UUID<br/> 14044 - 15068 - - - - -<br/> homsys-OST001d_UUID<br/> 16692 - 17716 - - - - -<br/> homsys-OST001e_UUID<br/> 39124 - 40148 - - - - -<br/> homsys-OST001f_UUID<br/> 13484 - 14012 - - - - -<br/> homsys-OST0020_UUID<br/> 11500 - 12524 - - - - -<br/> homsys-OST0021_UUID<br/> 12004 - 13028 - - - - -<br/> homsys-OST0022_UUID<br/> 26332 - 27356 - - - - -<br/> homsys-OST0023_UUID<br/> 13896 - 14920 - - - - -<br/> homsys-OST0024_UUID<br/> 17100 - 18120 - - - - -<br/> homsys-OST0025_UUID<br/> 27388 - 28412 - - - - -<br/> homsys-OST0026_UUID<br/> 10800 - 11824 - - - - -<br/> homsys-OST0027_UUID<br/> 25572 - 26596 - - - - -<br/> homsys-OST0028_UUID<br/> 23144 - 24064 - - - - -<br/> homsys-OST0029_UUID<br/> 13700 - 14552 - - - - -<br/> homsys-OST002a_UUID<br/> 21748 - 22772 - - - - -<br/> homsys-OST002b_UUID<br/> 21800 - 22824 - - - - -<br/> homsys-OST002c_UUID<br/> 16600 - 17624 - - - - -<br/> homsys-OST002d_UUID<br/> 12224 - 13248 - - - - -<br/> homsys-OST002e_UUID<br/> 12796 - 13820 - - - - -<br/> homsys-OST002f_UUID<br/> 10436 - 11460 - - - - -<br/> homsys-OST0030_UUID<br/> 24940 - 25960 - - - - -<br/> homsys-OST0031_UUID<br/> 13820 - 14844 - - - - -<br/> homsys-OST0032_UUID<br/> 10276 - 11296 - - - - -<br/> homsys-OST0033_UUID<br/> 14324 - 14856 - - - - -<br/> homsys-OST0034_UUID<br/> 11168 - 11776 - - - - -<br/> homsys-OST0035_UUID<br/> 17876 - 18900 - - - - -<br/> homsys-OST0036_UUID<br/> 14740 - 15764 - - - - -<br/> homsys-OST0037_UUID<br/> 24764 - 25788 - - - - -<br/> homsys-OST0038_UUID<br/> 17848 - 18868 - - - - -<br/> homsys-OST0039_UUID<br/> 15164 - 15720 - - - - -<br/> homsys-OST003a_UUID<br/> 18736 - 19760 - - - - -<br/> homsys-OST003b_UUID<br/> 14476 - 15500 - - - - -<br/> homsys-OST003c_UUID<br/> 4024 - 5048 - - - - -<br/> homsys-OST003d_UUID<br/> 13588 - 14612 - - - - -<br/> homsys-OST003e_UUID<br/> 13576 - 14600 - - - - -<br/> homsys-OST003f_UUID<br/> 26372 - 27396 - - - - -<br/> homsys-OST0040_UUID<br/> 50380 - 51404 - - - - -<br/> homsys-OST0041_UUID<br/> 24796 - 25816 - - - - -<br/> homsys-OST0042_UUID<br/> 24176 - 25196 - - - - -<br/> homsys-OST0043_UUID<br/> 12776 - 13800 - - - - -<br/> homsys-OST0044_UUID<br/> 13444 - 14468 - - - - -<br/> homsys-OST0045_UUID<br/> 23492 - 24476 - - - - -<br/> homsys-OST0046_UUID<br/> 11412 - 12436 - - - - -<br/> homsys-OST0047_UUID<br/> 14552 - 15576 - - - - -<br/> homsys-OST0048_UUID<br/> 19140 - 19664 - - - - -<br/> homsys-OST0049_UUID<br/> 12384 - 13408 - - - - -<br/> homsys-OST004a_UUID<br/> 29392 - 30416 - - - - -<br/> homsys-OST004b_UUID<br/> 40412 - 41436 - - - - -<br/> homsys-OST004c_UUID<br/> 52872 - 53896 - - - - -<br/> homsys-OST004d_UUID<br/> 29372 - 30396 - - - - -<br/> homsys-OST004e_UUID<br/> 13144 - 14164 - - - - -<br/> homsys-OST004f_UUID<br/> 13000 - 14024 - - - - -<br/> Total allocated inode limit: 64206, total allocated block limit: 1663052</p> <p>Looks the used space is far below the quota limit, so we'd figure out if it really write to OST first:</p> <ul class="alternate" type="square"> <li>If write on OST failed with EDQUOT either, this could be <a href="https://jira.whamcloud.com/browse/LU-4505" title="invalid &quot;Disk quota exceed&quot; error" class="issue-link" data-issue-key="LU-4505"><del>LU-4505</del></a>, collecting log on OST could be helpful. (for file write, with D_QUOTA enabled)</li> <li>If write on OST succeeded, we'd collect log to see why this client can't read it (as I mentioned in previous comment), and try the same test on another clean client to see if it can be reproduced.</li> </ul> <p>Anyway, I don't think this is related to current ticket, could you create a new ticket for this problem? Thanks.</p> <p>This test result looks somewhat similar:<br/> <a href="https://testing.hpdd.intel.com/test_sets/08918008-2437-11e5-b6b4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/08918008-2437-11e5-b6b4-5254006e85c2</a></p> <p>I hit the similar issue: <a href="https://testing.hpdd.intel.com/test_sets/f878df9a-2661-11e5-8b33-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f878df9a-2661-11e5-8b33-5254006e85c2</a></p> <p>All the failures were because quota slave didn't connect to master, so that quota wasn't enforced on slave.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">00000100:00100000:0.0:1436439856.246154:0:23692:0:(nrs_fifo.c:179:nrs_fifo_req_get()) NRS start fifo request from 12345-10.1.4.48@tcp, seq: 852 00000100:00100000:0.0:1436439856.246163:0:23692:0:(service.c:2076:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc mdt00_004:0+-99:3137:x1505240840316480:12345-10.1.4.48@tcp:38 00010000:02000400:0.0:1436439856.246191:0:23692:0:(ldlm_lib.c:1026:target_handle_connect()) lustre-MDT0000: Received LWP connection from 10.1.4.48@tcp, removing former export from 10.1.4.52@tcp 00000020:00080000:0.0:1436439856.246195:0:23692:0:(genops.c:1382:class_fail_export()) disconnecting export ffff8800798e9400/lustre-MDT0000-lwp-OST0003_UUID 00000020:00000080:0.0:1436439856.246204:0:23692:0:(genops.c:1215:class_disconnect()) disconnect: cookie 0x515596b0134eb7f7 00000100:00080000:0.0:1436439856.246211:0:23692:0:(<span class="code-keyword">import</span>.c:1601:ptlrpc_cleanup_imp()) ffff880063495000 PA#}: changing <span class="code-keyword">import</span> state from FULL to CLOSED 00000020:00080000:0.0:1436439856.246261:0:23692:0:(genops.c:1399:class_fail_export()) disconnected export ffff8800798e9400/lustre-MDT0000-lwp-OST0003_UUID 00000020:00000080:0.0:1436439856.246263:0:23692:0:(genops.c:814:class_export_put()) <span class="code-keyword">final</span> put ffff8800798e9400/lustre-MDT0000-lwp-OST0003_UUID 00010000:00080000:0.0:1436439856.246290:0:23692:0:(ldlm_lib.c:1092:target_handle_connect()) lustre-MDT0000: connection from lustre-MDT0000-lwp-OST0003_UUID@10.1.4.48@tcp t0 exp (<span class="code-keyword">null</span>) cur 1436439856 last 0 </pre> </div></div> <p>I see from the log that OST connected to master with different IP address during testing, do we failover OSTs while testing?</p> <p>I think we hit this issue again on PPC builds. Logs at:<br/> 2015-09-01 03:24:28 - <a href="https://testing.hpdd.intel.com/test_sets/b1d82b30-508d-11e5-95a9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b1d82b30-508d-11e5-95a9-5254006e85c2</a></p> <p>Another instance on master:<br/> 2015-11-15 14:01:26 - <a href="https://testing.hpdd.intel.com/test_sets/9e1632c2-8bbb-11e5-9933-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9e1632c2-8bbb-11e5-9933-5254006e85c2</a><br/> 2015-11-18 16:20:31 - <a href="https://testing.hpdd.intel.com/test_sets/cf90e830-8e25-11e5-8da8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cf90e830-8e25-11e5-8da8-5254006e85c2</a><br/> 2015-11-18 21:25:31 - <a href="https://testing.hpdd.intel.com/test_sets/1ce03240-8e52-11e5-8da8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1ce03240-8e52-11e5-8da8-5254006e85c2</a></p> <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/fedc1a0c-4219-11e7-b558-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fedc1a0c-4219-11e7-b558-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/a4bae786-61c9-11e7-9230-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a4bae786-61c9-11e7-9230-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/772bbf04-6dec-11e8-a522-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/772bbf04-6dec-11e8-a522-52540065bddc</a></p> <p>Closing this old issue, use <a href="https://jira.whamcloud.com/browse/LU-11678" title="sanity-quota test 1 fails with &#39;user write success, but expect EDQUOT&#39;" class="issue-link" data-issue-key="LU-11678"><del>LU-11678</del></a> for new<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/help_16.png" height="16" width="16" align="absmiddle" alt="" border="0"/> issue with same symptom.</p> Related LU-9326 LU-11678 Development Rank 1|hzwppr: Rank (Obsolete) 14627 Severity