https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3326 Lustre <p>After running recovery-mds-scale test_failover_ost for 1.5 hours (OST failed over 6 times), client load on one of the clients failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;snip&gt; tar: etc/mail/submit.cf: Cannot open: No space left on device tar: etc/mail/trusted-users: Cannot open: No space left on device tar: etc/mail/virtusertable: Cannot open: No space left on device tar: etc/mail/access: Cannot open: No space left on device tar: etc/mail/aliasesdb-stamp: Cannot open: No space left on device tar: etc/gssapi_mech.conf: Cannot open: No space left on device tar: Exiting with failure status due to previous errors </pre> </div></div> <p>Console log on the client (client-32vm6) showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>19:40:31:INFO: task tar:2790 blocked for more than 120 seconds. 19:40:31:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 19:40:31:tar D 0000000000000000 0 2790 2788 0x00000080 19:40:31: ffff88004eb73a28 0000000000000082 ffff88004eb739d8 ffff88007c24fe50 19:40:31: 0000000000000286 0000000000000003 0000000000000001 0000000000000286 19:40:31: ffff88007bcb3ab8 ffff88004eb73fd8 000000000000fb88 ffff88007bcb3ab8 19:40:31:Call Trace: 19:40:31: [&lt;ffffffffa03d775a&gt;] ? cfs_waitq_signal+0x1a/0x20 [libcfs] 19:40:31: [&lt;ffffffff8150ea05&gt;] schedule_timeout+0x215/0x2e0 19:40:31: [&lt;ffffffffa068517c&gt;] ? ptlrpc_request_bufs_pack+0x5c/0x80 [ptlrpc] 19:40:31: [&lt;ffffffffa069a770&gt;] ? lustre_swab_ost_body+0x0/0x10 [ptlrpc] 19:40:31: [&lt;ffffffff8150e683&gt;] wait_for_common+0x123/0x180 19:40:31: [&lt;ffffffff81063310&gt;] ? default_wake_function+0x0/0x20 19:40:31: [&lt;ffffffff8150e79d&gt;] wait_for_completion+0x1d/0x20 19:40:31: [&lt;ffffffffa08cbf6c&gt;] osc_io_setattr_end+0xbc/0x190 [osc] 19:40:31: [&lt;ffffffffa095cde0&gt;] ? lov_io_end_wrapper+0x0/0x100 [lov] 19:40:31: [&lt;ffffffffa055cf30&gt;] cl_io_end+0x60/0x150 [obdclass] 19:40:31: [&lt;ffffffffa055d7e0&gt;] ? cl_io_start+0x0/0x140 [obdclass] 19:40:31: [&lt;ffffffffa095ced1&gt;] lov_io_end_wrapper+0xf1/0x100 [lov] 19:40:31: [&lt;ffffffffa095c86e&gt;] lov_io_call+0x8e/0x130 [lov] 19:40:31: [&lt;ffffffffa095e3bc&gt;] lov_io_end+0x4c/0xf0 [lov] 19:40:31: [&lt;ffffffffa055cf30&gt;] cl_io_end+0x60/0x150 [obdclass] 19:40:31: [&lt;ffffffffa0561f92&gt;] cl_io_loop+0xc2/0x1b0 [obdclass] 19:40:31: [&lt;ffffffffa0a2aa08&gt;] cl_setattr_ost+0x208/0x2c0 [lustre] 19:40:31: [&lt;ffffffffa09f8b0e&gt;] ll_setattr_raw+0x9ce/0x1000 [lustre] 19:40:31: [&lt;ffffffffa09f919b&gt;] ll_setattr+0x5b/0xf0 [lustre] 19:40:31: [&lt;ffffffff8119e708&gt;] notify_change+0x168/0x340 19:40:31: [&lt;ffffffff811b284c&gt;] utimes_common+0xdc/0x1b0 19:40:31: [&lt;ffffffff811828d1&gt;] ? __fput+0x1a1/0x210 19:40:31: [&lt;ffffffff811b29fe&gt;] do_utimes+0xde/0xf0 19:40:31: [&lt;ffffffff811b2b12&gt;] sys_utimensat+0x32/0x90 19:40:31: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/053120d2-bb19-11e2-8824-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/053120d2-bb19-11e2-8824-52540035b04c</a></p> <br/> Lustre Branch: master<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-master/1486">http://build.whamcloud.com/job/lustre-master/1486</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> Test Group: failover<br/> LU-3326

recovery-mds-scale test_failover_ost: tar: Cannot open: No space left on device

Bug Critical Resolved Fixed Hongchao Zhang Jian Yu mn1 mn4 Mon, 13 May 2013 05:09:25 +0000 Thu, 26 Feb 2015 21:58:13 +0000 Wed, 20 Aug 2014 15:19:10 +0000 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.6.0 Lustre 2.4.2 Lustre 2.5.1 Lustre 2.6.0 Lustre 2.5.3 0 8 <p>Lustre Branch: master<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-master/1481" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1481</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> Test Group: failover</p> <p>After running recovery-mds-scale test_failover_ost for more than 1 hour (OST failed over 5 times), dd operation on one of the clients failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2013-05-08 23:10:39: dd run starting + mkdir -p /mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com + /usr/bin/lfs setstripe -c -1 /mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com + cd /mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com ++ /usr/bin/lfs df /mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com + FREE_SPACE=4050128 + BLKS=911278 + echoerr 'Free disk space is 4050128, 4k blocks to dd is 911278' + echo 'Free disk space is 4050128, 4k blocks to dd is 911278' Free disk space is 4050128, 4k blocks to dd is 911278 + load_pid=27789 + wait 27789 + dd bs=4k count=911278 status=noxfer if=/dev/zero of=/mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com/dd-file dd: writing `/mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com/dd-file': No space left on device 409345+0 records in 409344+0 records out + '[' 1 -eq 0 ']' ++ date '+%F %H:%M:%S' + echoerr '2013-05-08 23:23:03: dd failed' </pre> </div></div> <p>Console log on client (client-26vm5) showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>23:23:08:LustreError: 27789:0:(vvp_io.c:1086:vvp_io_commit_write()) Write page 409344 of inode ffff880037eda1b8 failed -28 </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/52c5d99a-b8f9-11e2-891d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/52c5d99a-b8f9-11e2-891d-52540035b04c</a></p> <p>Hi Hongchao,</p> <p>It seems the above out of space issue on dd operation is related to <a href="https://jira.whamcloud.com/browse/LU-657" title="recovery-mds-scale (FLAVOR=MDS): client load failed" class="issue-link" data-issue-key="LU-657"><del>LU-657</del></a>. Could you please check whether the original out of space issue on tar operation in this ticket is also the same or not? Thanks.</p> <p>Yu Jian,<br/> is there an expectation that the OST would <em>not</em> run out of space after writing to it for an hour? Are the files written earlier in the test being deleted, or will the test fill up all of the OST space? Does it check the free space on the OST that will be written on, or is this the free space for the whole filesystem and the one file may run out of space?</p> <blockquote><p>is there an expectation that the OST would not run out of space after writing to it for an hour?</p></blockquote> <p>Yes, the OST should not run out of space during the whole test run, because in run_{dd,tar,dbench,iozone}.sh each application keeps running inside its own $TESTDIR, which will be removed and recreated for each successful loop.</p> <blockquote><p>Are the files written earlier in the test being deleted, or will the test fill up all of the OST space?</p></blockquote> <p>If the previous failover_mds test passed, then the applications on clients will just be terminated without deleting $TESTDIR and corresponding files. If the test failed, then the failed application will also leave the $TESTDIR and files undeleted. We need fix run_*.sh to delete test dirs for the two situations.</p> <p>However, it seems the above issue is not the cause of this ticket. Let's analyze the first Maloo report in this ticket, failover_mds test failed with <a href="https://jira.whamcloud.com/browse/LU-3325" title="recovery-mds-scale test_failover_mds: tar: Cannot write: Input/output error" class="issue-link" data-issue-key="LU-3325"><del>LU-3325</del></a> and left /mnt/lustre/d0.tar-client-32vm6.lab.whamcloud.com dir on client-32vm6. Then failover_ost test started, tar application was also run inside /mnt/lustre/d0.tar-client-32vm6.lab.whamcloud.com dir on client-32vm6. Since there were several successful loops, the /mnt/lustre/d0.tar-client-32vm6.lab.whamcloud.com dir has been removed and recreated for several times, so the dir and files left by failover_mds did not affect failover_ost.</p> <blockquote><p>Does it check the free space on the OST that will be written on, or is this the free space for the whole filesystem and the one file may run out of space?</p></blockquote> <p>Currently, only run_dd.sh checks free space before performing dd operation. We need improve other run_*.sh to add this check. Running out of space on OST is not the expectation for failover_ost test.</p> <p>Besides improving run_*.sh scripts, we still need figure out the real cause of this ticket.</p> <p>Reducing from blocker but want to continue digging into this to find the cause.</p> <p>One comment is that the previous data was in <tt>/mnt/lustre/d0.tar-client-32vm6.lab.whamcloud.com</tt> but the failed test was writing to <tt>/mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com/dd-file</tt> (i.e. <b>dd</b> vs <b>tar</b>), so it is possible that the old directory was not cleaned up?</p> <p>It might be useful to run <tt>lfs df</tt> output in the test script before each "dd" so that we know what the space distribution is.</p> <blockquote><p>One comment is that the previous data was in /mnt/lustre/d0.tar-client-32vm6.lab.whamcloud.com but the failed test was writing to /mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com/dd-file (i.e. dd vs tar), so it is possible that the old directory was not cleaned up?</p></blockquote> <p>For the first Maloo report (in the Description section above), both failover_mds and failover_ost tests failed at tar operation on client-32vm6, dd operation on client-32vm5 did not fail. Per the analysis in the above comment, the dir and files left by failover_mds did not affect failover_ost.</p> <p>For the second Maloo report above, failover_mds failed at tar operation on client-26vm6 and left data in /mnt/lustre/d0.tar-client-26vm6.lab.whamcloud.com, dd operation kept successful on client-26vm5 and cleaned up its dir before failover_mds stopped. Then failover_ost started, tar operation was still run on client-26vm6 in /mnt/lustre/d0.tar-client-26vm6.lab.whamcloud.com and did not fail, dd operation was still run on client-26vm5 in /mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com but failed. Again, since there was one successful dd operation loop before it failed, the /mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com dir has been removed and recreated, so the dir and files left by failover_mds did not affect failover_ost either.</p> <blockquote><p>It might be useful to run lfs df output in the test script before each "dd" so that we know what the space distribution is.</p></blockquote> <p>We can see this information in "run_dd_debug" log in the Maloo reports:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2013-05-08 23:10:39: dd run starting + mkdir -p /mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com + /usr/bin/lfs setstripe -c -1 /mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com + cd /mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com ++ /usr/bin/lfs df /mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com + FREE_SPACE=4050128 + BLKS=911278 + echoerr 'Free disk space is 4050128, 4k blocks to dd is 911278' + echo 'Free disk space is 4050128, 4k blocks to dd is 911278' Free disk space is 4050128, 4k blocks to dd is 911278 + load_pid=27789 + wait 27789 + dd bs=4k count=911278 status=noxfer if=/dev/zero of=/mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com/dd-file dd: writing `/mnt/lustre/d0.dd-client-26vm5.lab.whamcloud.com/dd-file': No space left on device 409345+0 records in 409344+0 records out </pre> </div></div> <p>The corresponding codes in run_dd.sh:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>while [ ! -e "$END_RUN_FILE" ] &amp;&amp; $CONTINUE; do # ...... FREE_SPACE=$($LFS df $TESTDIR|awk '/filesystem summary:/ {print $5}') BLKS=$((FREE_SPACE * 9 / 40)) echoerr "Free disk space is $FREE_SPACE, 4k blocks to dd is $BLKS" # ...... done </pre> </div></div> <p>It looks like dd operation keeps occupying too much free space (90%), which is likely to make the operations on other clients run out of space, or make itself run out of space since all of the operations compete for the free space concurrently.</p> <p>Since the goal for recovery-*-scale tests is not to fill up the space but to keep the applications running on different clients, I'd suggest to reduce the dd size (like just use 50% of the free space or dynamically calculate the percent according to the number of clients) and leave enough free space for other applications.</p> <p>the patch is tracked at <a href="http://review.whamcloud.com/#change,6376" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,6376</a></p> <p>Lustre Tag: v2_4_0_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/12/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/12/</a><br/> Distro/Arch: RHEL6.4/x86_64</p> <p>The issue occurred on recovery-mds-scale test_failover_ost:<br/> <a href="https://maloo.whamcloud.com/test_sets/2e35d81c-c6c5-11e2-ae4e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2e35d81c-c6c5-11e2-ae4e-52540035b04c</a></p> <p>This is blocking the recovery-mds-scale test running on Lustre b2_4 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/ed4a088e-02db-11e3-a4b4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ed4a088e-02db-11e3-a4b4-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/edec7172-fd14-11e2-b90c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/edec7172-fd14-11e2-b90c-52540035b04c</a></p> <p>the patch has been merged into master</p> <p>The patch was also cherry-picked to Lustre b2_4 branch.</p> <p>Lustre Tag: v2_4_1_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/45/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/45/</a><br/> Distro/Arch: RHEL6.4/x86_64</p> <p>The issue still occurred on recovery-mds-scale test_failover_ost:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>tar: etc/yum.repos.d/cobbler-config.repo: Cannot open: No space left on device tar: etc/yum.repos.d/lustre-build.repo: Cannot open: No space left on device tar: Exiting with failure status due to previous errors </pre> </div></div> <p>Console log on MDS showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>11:46:04:Lustre: lustre-OST0004-osc-MDT0000: slow creates, last=[0x100040000:0xbba1:0x0], next=[0x100040000:0xbba1:0x0], reserved=0, syn_changes=54, syn_rpc_in_progress=36, status=-28 11:46:26:LNet: Service thread pid 2233 was inactive for 40.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: 11:46:26:Pid: 2233, comm: mdt00_001 11:46:27: 11:46:27:Call Trace: 11:46:27: [&lt;ffffffff810810cc&gt;] ? lock_timer_base+0x3c/0x70 11:46:28: [&lt;ffffffff8150f362&gt;] schedule_timeout+0x192/0x2e0 11:46:28: [&lt;ffffffff810811e0&gt;] ? process_timeout+0x0/0x10 11:46:28: [&lt;ffffffffa04a56d1&gt;] cfs_waitq_timedwait+0x11/0x20 [libcfs] 11:46:28: [&lt;ffffffffa0fb2174&gt;] osp_precreate_reserve+0x5b4/0x1ed0 [osp] 11:46:28: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 11:46:28: [&lt;ffffffffa0fabc75&gt;] osp_declare_object_create+0x155/0x4f0 [osp] 11:46:29: [&lt;ffffffffa0f6b6ed&gt;] lod_qos_declare_object_on+0xed/0x480 [lod] 11:46:29: [&lt;ffffffffa0f6c569&gt;] lod_alloc_qos.clone.0+0xae9/0x1180 [lod] 11:46:30: [&lt;ffffffffa0cbff1f&gt;] ? qsd_op_begin+0x5f/0xb40 [lquota] 11:46:30: [&lt;ffffffffa0f6e65a&gt;] lod_qos_prep_create+0x74a/0x1b14 [lod] 11:46:30: [&lt;ffffffffa097ead2&gt;] ? fld_server_lookup+0x72/0x3d0 [fld] 11:46:30: [&lt;ffffffffa0f6902b&gt;] lod_declare_striped_object+0x14b/0x880 [lod] 11:46:31: [&lt;ffffffff81096d8f&gt;] ? wake_up_bit+0x2f/0x40 11:46:31: [&lt;ffffffffa0f69c71&gt;] lod_declare_object_create+0x511/0x7a0 [lod] 11:46:31: [&lt;ffffffffa0c198cf&gt;] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] 11:46:31: [&lt;ffffffffa0c28fee&gt;] mdd_declare_create+0x4e/0x870 [mdd] 11:46:31: [&lt;ffffffffa0c277ef&gt;] ? mdd_linkea_prepare+0x23f/0x430 [mdd] 11:46:31: [&lt;ffffffffa0c29fd5&gt;] mdd_create+0x7c5/0x1790 [mdd] 11:46:32: [&lt;ffffffffa0d9b927&gt;] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] 11:46:35: [&lt;ffffffffa0ec808f&gt;] mdt_reint_open+0x135f/0x20c0 [mdt] 11:46:35: [&lt;ffffffffa04c182e&gt;] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] 11:46:35: [&lt;ffffffffa07abdcc&gt;] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] 11:46:36: [&lt;ffffffffa063ff50&gt;] ? lu_ucred+0x20/0x30 [obdclass] 11:46:36: [&lt;ffffffffa0eb2911&gt;] mdt_reint_rec+0x41/0xe0 [mdt] 11:46:36: [&lt;ffffffffa0e97ae3&gt;] mdt_reint_internal+0x4c3/0x780 [mdt] 11:46:36: [&lt;ffffffffa0e9806d&gt;] mdt_intent_reint+0x1ed/0x520 [mdt] 11:46:38: [&lt;ffffffffa0e95f1e&gt;] mdt_intent_policy+0x39e/0x720 [mdt] 11:46:38: [&lt;ffffffffa0763831&gt;] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] 11:46:38: [&lt;ffffffffa078a1ef&gt;] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] 11:46:39: [&lt;ffffffffa0e963a6&gt;] mdt_enqueue+0x46/0xe0 [mdt] 11:46:39: [&lt;ffffffffa0e9ca97&gt;] mdt_handle_common+0x647/0x16d0 [mdt] 11:46:39: [&lt;ffffffffa0ed63f5&gt;] mds_regular_handle+0x15/0x20 [mdt] 11:46:41: [&lt;ffffffffa07bc3c8&gt;] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] 11:46:42: [&lt;ffffffffa04a55de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] 11:46:42: [&lt;ffffffffa04b6d9f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] 11:46:43: [&lt;ffffffffa07b3729&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 11:46:44: [&lt;ffffffff81055ad3&gt;] ? __wake_up+0x53/0x70 11:46:44: [&lt;ffffffffa07bd75e&gt;] ptlrpc_main+0xace/0x1700 [ptlrpc] 11:46:44: [&lt;ffffffffa07bcc90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 11:46:44: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 11:46:45: [&lt;ffffffffa07bcc90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 11:46:45: [&lt;ffffffffa07bcc90&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] 11:46:45: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 11:46:45: 11:46:46:LustreError: dumping log to /tmp/lustre-log.1378579577.2233 11:46:46:Lustre: lustre-OST0005-osc-MDT0000: slow creates, last=[0x100050000:0xbc41:0x0], next=[0x100050000:0xbc41:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=10, status=-28 11:46:48:LNet: Service thread pid 2233 completed after 40.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/cc263682-194c-11e3-bb73-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/cc263682-194c-11e3-bb73-52540035b04c</a></p> <p>More instances on Lustre b2_4 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/76525a9c-14fb-11e3-ac48-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/76525a9c-14fb-11e3-ac48-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/89a20084-14fb-11e3-ac48-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/89a20084-14fb-11e3-ac48-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/da5b6604-142f-11e3-80c8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/da5b6604-142f-11e3-80c8-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/ec68c8a0-142f-11e3-80c8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ec68c8a0-142f-11e3-80c8-52540035b04c</a></p> <p>the issue is reproduced locally, and it should be caused by the deletion of the large file created by "dd", which could contain many journal transactions<br/> preventing the freed disk space to be used by the following write operations.</p> <p>the patch is under creation&amp;test.</p> <p>the patch is tracked at <a href="http://review.whamcloud.com/#/c/7662/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7662/</a></p> <p>local reproducer will fail about 10 minutes without the patch, and run more than 6 hours successfully with the patch applied.</p> <p>Please complete the patch for Master before completing b2_4.</p> <p>Please submit the patch for master first.</p> <p>the patch for master is tracked at <a href="http://review.whamcloud.com/#/c/8071/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8071/</a></p> <p>there are two options to fix the issue,<br/> 1, modify the test script "run_dd.sh" to do some extra work to wait for the completion of the destroy of the object created by "dd" (preferred)<br/> 2, modify Lustre itself to wait for the destroy of the objects when it encounters the -ENOSPC during writing.</p> <p>Hi Yujian, <br/> could you please take care of it during my leave for vacation, thanks!</p> <blockquote><p>could you please take care of it during my leave for vacation, thanks!</p></blockquote> <p>Sure, I just updated the patch with the first method.</p> <p>Patch for Lustre b2_4 branch is in <a href="http://review.whamcloud.com/8447" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8447</a>.</p> <p>I don't see how fixing the test script avoids problems that users might also hit?</p> <p>I recall that there are some fixes in patch 8071 which are useful to fix the OST-side lock cancellation and client-side page discard, though I haven't looked at the latest patch version. It would be useful to verify if this is correctly causing unwritten pages to be discarded on the client.</p> <blockquote><p>I recall that there are some fixes in patch 8071 which are useful to fix the OST-side lock cancellation and client-side page discard</p></blockquote> <p>The non-script part of changes in <a href="http://review.whamcloud.com/8071" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8071</a> is on file lustre/include/lustre_dlm_flags.h:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>-#define LDLM_FL_AST_MASK 0x0000000080008000ULL +#define LDLM_FL_AST_MASK» » 0x0000000000018000ULL </pre> </div></div> <p>Should I remain this change and drop the script part changes?</p> <p>In addition, lustre/include/lustre_dlm_flags.h was added by <a href="http://review.whamcloud.com/5312" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5312</a>, which does not exist on Lustre b2_4 branch.</p> <p>Yu Jian, I think the code change in 8071 is potentially in the right area, but not necessarily the right fix. I <em>think</em> this will change the wire protocol to use a different flag (LDLM_FL_DISCARD_DATA vs. LDLM_FL_AST_DISCARD_DATA) in the wire protocol, but I'm not positive. It would be great to test this properly - write a large file in the background, delete it, and then check the kernel debug logs to see if more pages are being written or if they are being discarded.</p> <p>I applied the LDLM_FL_AST_MASK change from <a href="http://review.whamcloud.com/8071" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8071</a> to the latest master branch and created <a href="http://review.whamcloud.com/8495" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8495</a> to get a new build.</p> <p>The following recovery-mds-scale test_failover_mds was performed on that build:<br/> <a href="https://maloo.whamcloud.com/sub_tests/9752d892-5e8a-11e3-a925-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/9752d892-5e8a-11e3-a925-52540035b04c</a></p> <p>The "run_dd_debug" log on Client 2 (wtm-68) showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Total free disk space is 10589812, 4k blocks to dd is 2382707 + pdsh -t 300 -S -w 'wtm-[67,68,71,72]' 'export PATH=$PATH:/sbin:/usr/sbin; lctl set_param debug=-1' + pdsh -t 300 -S -w 'wtm-[67,68,71,72]' 'export PATH=$PATH:/sbin:/usr/sbin; lctl set_param debug_mb=150' + pdsh -t 300 -S -w 'wtm-[67,68,71,72]' 'export PATH=$PATH:/sbin:/usr/sbin; lctl dk &gt; /dev/null' + load_pid=42589 + wait 42589 + dd bs=4k count=2382707 status=noxfer if=/dev/zero of=/mnt/lustre/d0.dd-wtm-68/dd-file 2382707+0 records in 2382707+0 records out + '[' 0 -eq 0 ']' ++ date '+%F %H:%M:%S' + echoerr '2013-12-06 06:19:18: dd succeeded' + echo '2013-12-06 06:19:18: dd succeeded' 2013-12-06 06:19:18: dd succeeded + cd /tmp + rm -rf /mnt/lustre/d0.dd-wtm-68 + pdsh -t 300 -S -w 'wtm-[67,68,71,72]' 'export PATH=$PATH:/sbin:/usr/sbin; lctl dk &gt; /scratch/jianyu/test_logs/2013-12-06/060035/recovery-mds-scale.test_failover_mds.run_dd_dk.$(hostname -s)_$(date +%s).log' </pre> </div></div> <p>The lctl debug logs are in "run_dd_dk_1386339559" and "run_dd_dk_1386339667" (there are two separate dd runs) in the above Maloo report.</p> <p>I found ofd_destroy_by_fid() in the OSS (wtm-72) lctl debug logs. Need look into the logs deeply.</p> <p>In "run_dd_dk_1386339667" debug logs on OSS (wtm-72), I found that:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">00010000:00010000:18.0:1386339564.644518:0:33665:0:(ldlm_lockd.c:848:ldlm_server_blocking_ast()) ### server preparing blocking AST ns: filter-lustre-OST0003_UUID lock: ffff880416972cc0/0xe8fc4c150d5421b0 lrc: 3/0,0 mode: PW/PW res: [0x2:0x0:0x0].0 rrc: 2 type: EXT [0-&gt;18446744073709551615] (req 0-&gt;4095) flags: 0x50000000010020 nid: 10.10.18.127@tcp remote: 0xe0b36e7f4f2635f3 expref: 7 pid: 38018 timeout: 0 lvb_type: 0 </pre> </div></div> <p>A blocking AST RPC was sent to client. </p> <p>And on client (wtm-68), in "run_dd_dk_1386339667":</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">00000020:00000001:14.0:1386339585.987516:0:40723:0:(cl_page.c:420:cl_page_put()) <span class="code-object">Process</span> entered 00000020:00000001:14.0:1386339585.987517:0:40723:0:(cl_page.c:422:cl_page_put()) page@ffff880746ae8600[1 ffff8805e1f6ac30:397170 ^(<span class="code-keyword">null</span>)_(<span class="code-keyword">null</span>) 4 0 1 (<span class="code-keyword">null</span>) (<span class="code-keyword">null</span>) 0x0] 00000020:00000001:14.0:1386339585.987518:0:40723:0:(cl_page.c:422:cl_page_put()) 1 00000020:00000001:14.0:1386339585.987518:0:40723:0:(cl_page.c:160:cl_page_free()) <span class="code-object">Process</span> entered 00000020:00000001:14.0:1386339585.987519:0:40723:0:(lustre_fid.h:719:fid_flatten32()) <span class="code-object">Process</span> leaving (rc=788528898 : 788528898 : 2effff02) 00000020:00000010:14.0:1386339585.987519:0:40723:0:(cl_page.c:174:cl_page_free()) kfreed <span class="code-quote">'page'</span>: 440 at ffff880746ae8600. 00000020:00000001:14.0:1386339585.987520:0:40723:0:(cl_page.c:175:cl_page_free()) <span class="code-object">Process</span> leaving 00000020:00000001:14.0:1386339585.987520:0:40723:0:(cl_page.c:437:cl_page_put()) <span class="code-object">Process</span> leaving 00020000:00000001:14.0:1386339585.987520:0:40723:0:(lov_page.c:82:lov_page_fini()) <span class="code-object">Process</span> leaving 00000020:00000001:14.0:1386339585.987521:0:40723:0:(lustre_fid.h:719:fid_flatten32()) <span class="code-object">Process</span> leaving (rc=4194307 : 4194307 : 400003) 00000020:00000010:14.0:1386339585.987521:0:40723:0:(cl_page.c:174:cl_page_free()) kfreed <span class="code-quote">'page'</span>: 304 at ffff880746ae8800. 00000020:00000001:14.0:1386339585.987522:0:40723:0:(cl_page.c:175:cl_page_free()) <span class="code-object">Process</span> leaving 00000020:00000001:14.0:1386339585.987522:0:40723:0:(cl_page.c:437:cl_page_put()) <span class="code-object">Process</span> leaving 00000008:00000001:14.0:1386339585.987523:0:40723:0:(osc_cache.c:3137:osc_page_gang_lookup()) <span class="code-object">Process</span> leaving (rc=0 : 0 : 0) 00000008:00000001:14.0:1386339585.987534:0:40723:0:(osc_cache.c:3246:osc_lock_discard_pages()) <span class="code-object">Process</span> leaving (rc=0 : 0 : 0) 00000020:00001000:14.0:1386339585.987535:0:40723:0:(cl_object.c:971:cl_env_put()) 2@ffff88081d244a68 00000008:00000001:14.0:1386339585.987536:0:40723:0:(osc_lock.c:1376:osc_lock_flush()) <span class="code-object">Process</span> leaving (rc=0 : 0 : 0) 00010000:00000001:14.0:1386339585.987538:0:40723:0:(ldlm_request.c:1353:ldlm_cli_cancel()) <span class="code-object">Process</span> entered 00010000:00000001:14.0:1386339585.987543:0:40723:0:(ldlm_request.c:1122:ldlm_cli_cancel_local()) <span class="code-object">Process</span> entered 00010000:00010000:14.0:1386339585.987543:0:40723:0:(ldlm_request.c:1127:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0003-osc-ffff880435738000 lock: ffff88083821d980/0xe0b36e7f4f2635f3 lrc: 4/0,0 mode: PW/PW res: [0x2:0x0:0x0].0 rrc: 1 type: EXT [0-&gt;18446744073709551615] (req 0-&gt;4095) flags: 0x428400010000 nid: local remote: 0xe8fc4c150d5421b0 expref: -99 pid: 42589 timeout: 0 lvb_type: 1 </pre> </div></div> <p>The osc_lock_discard_pages() was called in osc_lock_flush(), however, the following code path was not covered in the debug logs:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (descr-&gt;cld_mode &gt;= CLM_WRITE) { result = osc_cache_writeback_range(env, obj, descr-&gt;cld_start, descr-&gt;cld_end, 1, discard); LDLM_DEBUG(ols-&gt;ols_lock, <span class="code-quote">"lock %p: %d pages were %s.\n"</span>, lock, result, discard ? <span class="code-quote">"discarded"</span> : <span class="code-quote">"written"</span>); <span class="code-keyword">if</span> (result &gt; 0) result = 0; } </pre> </div></div> <p>I just started the testing again to do the "write a large file in the background and delete it" operations for more times.</p> <p>I performed the test to dd a large file in the background and then delete the file from another client. Unfortunately, the debug logs did not show "pages were discarded" info on the clients, and there were also no "server preparing blocking AST" info on the OSS node. Still digging.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/70/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/70/</a> (2.4.2 RC2)<br/> <a href="https://maloo.whamcloud.com/test_sets/34a4c608-6be4-11e3-a73e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/34a4c608-6be4-11e3-a73e-52540035b04c</a></p> <p>local test with two nodes, the server node runs master and client runs b2_1, b2_3, b2_4 or master, the pages were not discarded after the lock's blocking AST<br/> is received, and will discard the pages after adding "LDLM_FL_DISCARD_DATA" instead of "LDLM_FL_AST_DISCARD_DATA" into "LDLM_FL_AST_MASK".</p> <p>LDLM_AST_DISCARD_DATA (changed to LDLM_FL_AST_DISCARD_DATA in master) is used to indicate ldlm to sent blocking AST with LDLM_FL_DISCARD_DATA flag,<br/> but it self is not on the wire, then the LDLM_FL_DISCARD_DATA will be ignored for it will be masked by "LDLM_FL_AST_MASK".</p> <p>code snippet in master</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">void ldlm_add_bl_work_item(struct ldlm_lock *lock, struct ldlm_lock *<span class="code-keyword">new</span>, cfs_list_t *work_list) { <span class="code-keyword">if</span> (!ldlm_is_ast_sent(lock)) { LDLM_DEBUG(lock, <span class="code-quote">"lock incompatible; sending blocking AST."</span>); ldlm_set_ast_sent(lock); /* If the enqueuing client said so, tell the AST recipient to * discard dirty data, rather than writing back. */ <span class="code-keyword">if</span> (ldlm_is_ast_discard_data(<span class="code-keyword">new</span>)) &lt;---- check LDLM_FL_AST_DISCARD_DATA ldlm_set_discard_data(lock); &lt;---- set LDLM_FL_DISCARD_DATA LASSERT(cfs_list_empty(&amp;lock-&gt;l_bl_ast)); cfs_list_add(&amp;lock-&gt;l_bl_ast, work_list); LDLM_LOCK_GET(lock); LASSERT(lock-&gt;l_blocking_lock == NULL); lock-&gt;l_blocking_lock = LDLM_LOCK_GET(<span class="code-keyword">new</span>); } } </pre> </div></div> <p>the corresponding patch is tracked at <a href="http://review.whamcloud.com/#/c/8671/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8671/</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/5/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/5/</a><br/> Test Group: failover</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/c86198c4-7505-11e3-95ae-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c86198c4-7505-11e3-95ae-52540035b04c</a></p> <p>More instances on Lustre b2_5 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/fcbcabd0-770e-11e3-b181-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fcbcabd0-770e-11e3-b181-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/e669c6a8-8643-11e3-9f3f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e669c6a8-8643-11e3-9f3f-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/fc5c9556-8505-11e3-8da9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fc5c9556-8505-11e3-8da9-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/f16b8720-9922-11e3-83d7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f16b8720-9922-11e3-83d7-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/7d4fc910-956b-11e3-936f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7d4fc910-956b-11e3-936f-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/39/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/39/</a> (2.5.1 RC1)<br/> Distro/Arch: RHEL6.5/x86_64<br/> Test Group: failover</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/36c8c4fe-a657-11e3-a191-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/36c8c4fe-a657-11e3-a191-52540035b04c</a></p> <p>Many sub-tests in replay-single in failover test group also hit the "No space left on device" failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/c55e55ee-a657-11e3-a191-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c55e55ee-a657-11e3-a191-52540035b04c</a></p> <p>This is blocking recovery-mds-scale testing.</p> <p>Hi Hongchao,</p> <p>Could you please drive patches of <a href="http://review.whamcloud.com/8671" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8671</a> (needs address the comment from Andreas) and <a href="http://review.whamcloud.com/8071" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8071</a> to be landed? Thanks. <a href="https://jira.whamcloud.com/browse/LU-3326" title="recovery-mds-scale test_failover_ost: tar: Cannot open: No space left on device" class="issue-link" data-issue-key="LU-3326"><del>LU-3326</del></a> has been blocking MDS/OST hard failover testing in Lustre release testing cycles.</p> <p>updated the two patches of this ticket</p> <p>(1) <a href="http://review.whamcloud.com/8671" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8671</a><br/> as per the comments in Lustre, the LDLM_FL_AST_DISCARD_DATA is also designed to indicate "FL_DISCARD" flag to the blocking ASTs</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>/** * These are flags that are mapped into the flags and ASTs of blocking * locks Add FL_DISCARD to blocking ASTs */ #define LDLM_FL_AST_DISCARD_DATA 0x0000000080000000ULL // bit 31 </pre> </div></div> <p>(2) <a href="http://review.whamcloud.com/8071" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8071</a><br/> wait the deletion of objects to complete before doing various tests at client node.</p> <p>The back-ported patch for <a href="http://review.whamcloud.com/8071" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8071</a> on Lustre b2_5 branch is in <a href="http://review.whamcloud.com/11425" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11425</a>.</p> <p>Test fixes landed for 2.5.3 and 2.7. Residual patch landing will be tracked under a new ticket.</p> <p>The new ticket is <a href="https://jira.whamcloud.com/browse/LU-5526" title="recovery-mds-scale test failover_mds: dd: No space left on device" class="issue-link" data-issue-key="LU-5526"><del>LU-5526</del></a>.</p> Duplicate LU-5235 Related LU-5526 Development Rank 1|hzvqov: Rank (Obsolete) 8208 Severity