https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7522 Lustre <p>While running conf-sanity test 32b with disk2_7-ldiskfs.tar.bz2 from <a href="http://review.whamcloud.com/16663" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16663</a>, the test failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Disk quotas for user 60000 (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /tmp/t32/mnt/lustre 30720* 0 20480 - 3* 0 2 - t32fs-MDT0000_UUID 0 - 0 - 1 - 0 - t32fs-MDT0001_UUID 0 - 0 - 2 - 0 - t32fs-OST0000_UUID 30720 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 0 bspace, act:30720, exp:10240 conf-sanity test_32b: @@@@@@ FAIL: verify quota failed </pre> </div></div> <p>Maloo report: <a href="https://testing.hpdd.intel.com/test_sets/3acf93b6-9bb5-11e5-9930-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3acf93b6-9bb5-11e5-9930-5254006e85c2</a></p> <p>The quota verification failure also occurred with disk2_7-zfs.tar.bz2:<br/> <a href="https://testing.hpdd.intel.com/test_sets/cf518162-9b78-11e5-b907-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cf518162-9b78-11e5-b907-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/409ef834-9b76-11e5-b7e5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/409ef834-9b76-11e5-b7e5-5254006e85c2</a></p> LU-7522

conf-sanity test_32b: FAIL: verify quota failed

Bug Critical Resolved Duplicate Niu Yawei Jian Yu Mon, 7 Dec 2015 08:26:53 +0000 Fri, 11 Dec 2015 06:50:44 +0000 Thu, 10 Dec 2015 07:03:08 +0000 Lustre 2.8.0 Lustre 2.8.0 0 5 <p>NIu</p> <p>Could you please look into this failure?</p> <p>Thanks</p> <p>Peter</p> <p>The same tests passed against disk2_4-*.tar.bz2.</p> <p>After upgrading, looks Lustre generated two files for quota_usr on MDT1 automatically.</p> <p>Before upgrading:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">t32fs-MDT0001_UUID 0 - 0 - 0 - 0 - </pre> </div></div> <p>After upgrading:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">t32fs-MDT0001_UUID 0 - 0 - 2 - 0 - </pre> </div></div> <p>Could you find out which files were generated? (mount the MDT1 as ldiskfs, then find the files owned by quota_usr) Thanks.</p> <p>Hi Niu,</p> <p>After upgrading, on MDT0000:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mount -t ldiskfs -o loop /tmp/t32/mdt /tmp/t32/mnt/mdt # cd /tmp/t32/mnt/mdt # ls -lR | grep quota_usr -rw-r--r-- 1 quota_usr quota_usr 0 Dec 8 02:25 t32_qf_old </pre> </div></div> <p>On MDT0001:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mount -t ldiskfs -o loop /tmp/t32/mdt2 /tmp/t32/mnt/mdt1 # cd /tmp/t32/mnt/mdt1 # ls -lR | grep quota_usr -rw-r--r-- 1 quota_usr quota_usr 0 Dec 8 02:25 t32_qf_old -rw-r--r-- 1 quota_usr quota_usr 0 Dec 8 02:25 t32_qf_old </pre> </div></div> <p>So, there are two more t32_qf_old files on MDT0001 after upgrading.</p> <p>I checked the original image from <a href="http://review.whamcloud.com/#/c/16663/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/16663/</a> , it does have 2 more quota_usr files on mdt1 and two more quota_usr objects on ost, looks the image is incorrect in first place.</p> <p>The objects on ost:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">/mnt/ost1/O/200000400/d21/55 /mnt/ost1/O/200000400/d10/1ca /mnt/ost1/O/0/d21/53 </pre> </div></div> <p>They are all 10M size.</p> <p>The files on mdt1:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">/mnt/mds1/REMOTE_PARENT_DIR/0x280000400:0x319:0x0/[0x280000401:0x1:0x0]:0/t32_qf_old /mnt/mds1/REMOTE_PARENT_DIR/0x280000400:0x1:0x0/t32_qf_old </pre> </div></div> <p>You're right, Niu.</p> <p>There is a defect in test_32newtarball() with MDSCOUNT=2. After gathering the quota data, t32_qf_old was copied to remote_dir and striped_dir_old directories separately. So, there are 3 t32_qf_old files with user quota_usr. I fixed that defect and generated the disk image again (please refer to the attached disk2_7-ldiskfs.tar.bz2).</p> <p>However, before upgrading:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Disk quotas for user 60000 (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /mnt/t32fs 30708 0 40960 - 3 0 4 - t32fs-MDT0000_UUID 0 - 0 - 1 - 0 - t32fs-MDT0001_UUID 0 - 0 - 2 - 0 - t32fs-OST0000_UUID 30708 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 0 </pre> </div></div> <p>After upgrading:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Disk quotas for user 60000 (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /tmp/t32/mnt/lustre 30724 0 40960 - 3 0 4 - t32fs-MDT0000_UUID 0 - 0 - 1 - 0 - t32fs-MDT0001_UUID 0 - 0 - 2 - 0 - t32fs-OST0000_UUID 30724 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 0 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>bspace, act:30724, exp:30708 conf-sanity test_32b: @@@@@@ FAIL: verify quota failed </pre> </div></div> <p>"ls -lR /tmp/t32/mnt/lustre | grep quota_usr" showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>-rw-r--r-- 1 quota_usr quota_usr 10485760 Dec 9 07:41 t32_qf_old -rw-r--r-- 1 quota_usr quota_usr 10485760 Dec 9 07:41 t32_qf_old -rw-r--r-- 1 quota_usr quota_usr 10485760 Dec 9 07:41 t32_qf_old </pre> </div></div> <p>The kbytes values didn't match.</p> <p>The original kbytes 30708 is less than actual size 30720, probably this value is obtained before all dirty flush back? Could you post the patch which fixing test_32newtarball()?</p> <p>While generating the disk image, after adding "sync" before getting the quota data, disk quotas for user quota_usr became:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Disk quotas for user 60000 (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /mnt/t32fs 30724 0 40960 - 3 0 4 - t32fs-MDT0000_UUID 0 - 0 - 1 - 0 - t32fs-MDT0001_UUID 0 - 0 - 2 - 0 - t32fs-OST0000_UUID 30724 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 0 </pre> </div></div> <p>I'm fixing test_32newtarball() in <a href="http://review.whamcloud.com/16663" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16663</a>. Let's close this ticket as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-7192" title="conf-sanity test_32c: ZFS test failure with DNE config" class="issue-link" data-issue-key="LU-7192"><del>LU-7192</del></a>.</p> <p>Could you check the ost image to see where the extra 4 bytes comes from?</p> <p>Hi Niu,</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mount -t ldiskfs -o loop ost /mnt/ost1 # find ./ -uid 60000 -gid 60000 -exec ls -l {} \; -rw-rw-rw- 1 quota_usr quota_usr 10485760 Dec 10 07:21 ./O/200000400/d10/1ca -rw-rw-rw- 1 quota_usr quota_usr 10485760 Dec 10 07:21 ./O/200000400/d21/55 -rw-rw-rw- 1 quota_usr quota_usr 10485760 Dec 10 07:21 ./O/0/d21/53 </pre> </div></div> <p>Not sure where the extra 4 bytes comes from.</p> <p>Ok, now we understand that the extra 4 bytes come from metadata of an object (in remote directory). When creating the image, sometimes metadata wasn't flushed, that's why we sometimes get 30724, sometimes get 30720.</p> Related LU-7192 Development Rank 1|hzxv27: Rank (Obsolete) 9223372036854775807 Severity