https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7309 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;andreas.dilger@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/4f0d6d92-718c-11e5-bffb-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4f0d6d92-718c-11e5-bffb-5254006e85c2</a>.</p> <p>The sub-test test_70b failed with the following error in the client test log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>shadow-52vm5: [11429] open ./clients/client0/~dmtmp/PWRPNT/PPTC112.TMP failed for handle 12322 (No space left on device) shadow-52vm5: (11430) ERROR: handle 12322 was not found shadow-52vm5: Child failed with status 1 shadow-52vm1: [11429] open ./clients/client0/~dmtmp/PWRPNT/PPTC112.TMP failed for handle 12322 (No space left on device) shadow-52vm1: (11430) ERROR: handle 12322 was not found shadow-52vm1: Child failed with status 1 </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: replay-single 70b</p> LU-7309

replay-single test_70b: no space left on device

Bug Major Resolved Fixed Hongchao Zhang Maloo p4hc Fri, 16 Oct 2015 17:51:51 +0000 Mon, 28 Nov 2016 15:38:56 +0000 Thu, 28 Jan 2016 18:50:12 +0000 Lustre 2.8.0 Lustre 2.8.0 0 9 <p>James,<br/> Per triage discussion, can you take look to gather a bit more info?<br/> Thanks.<br/> Joe</p> <p>more instance: <br/> master tag-2.7.63 client/server: RHEL6.7 <br/> <a href="https://testing.hpdd.intel.com/test_sets/805b3852-947d-11e5-95f7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/805b3852-947d-11e5-95f7-5254006e85c2</a><br/> server: RHEL6.7, client SLES11 SP3<br/> <a href="https://testing.hpdd.intel.com/test_sessions/54c65392-9131-11e5-ad50-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/54c65392-9131-11e5-ad50-5254006e85c2</a><br/> client/server: RHEL7 zfs<br/> <a href="https://testing.hpdd.intel.com/test_sessions/e41920d2-945c-11e5-b268-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/e41920d2-945c-11e5-b268-5254006e85c2</a><br/> master tag-2.7.64 client/server: RHEL6.7<br/> <a href="https://testing.hpdd.intel.com/test_sets/80a20678-9edd-11e5-87a9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/80a20678-9edd-11e5-87a9-5254006e85c2</a><br/> server: RHEL7, client SLES11 SP3<br/> <a href="https://testing.hpdd.intel.com/test_sets/a8b3fb9e-a077-11e5-8d69-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a8b3fb9e-a077-11e5-8d69-5254006e85c2</a><br/> RHEL7 zfs<br/> <a href="https://testing.hpdd.intel.com/test_sets/5ba6d7bc-9e20-11e5-91b0-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5ba6d7bc-9e20-11e5-91b0-5254006e85c2</a></p> <p>Server: 2.5.5, b2_5_fe/62<br/> Client: Master, Build# 3266, Tag 2.7.64 <br/> <a href="https://testing.hpdd.intel.com/test_sets/a061b2ba-a04a-11e5-a33d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a061b2ba-a04a-11e5-a33d-5254006e85c2</a></p> <p>Hongchao,<br/> James reports that this problem is hitting master, 2.x, and 3.0.0 for the failover test group. This issue is not being seen in autotest. In most of these failure scenarios, several other tests will also fail with the same error. He strongly believes the issue is not directly related to test_70b, but rather is a larger issue. He also believes there is evidence to say that it is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-4846" title="Failover test failure on test suite replay-single test_26: No space left" class="issue-link" data-issue-key="LU-4846">LU-4846</a>.<br/> Can you have look please?<br/> Thanks.<br/> Joe</p> <p>I have analyzed several failed test, and some didn't contain obvious error logs related to "ENOSPC", but it did indicate the problem<br/> was caused by the object creation at MDT, which is smiler with the issue in <a href="https://jira.whamcloud.com/browse/LU-4846" title="Failover test failure on test suite replay-single test_26: No space left" class="issue-link" data-issue-key="LU-4846">LU-4846</a> and <a href="https://jira.whamcloud.com/browse/LU-5526" title="recovery-mds-scale test failover_mds: dd: No space left on device" class="issue-link" data-issue-key="LU-5526"><del>LU-5526</del></a>.</p> <p>in <a href="https://testing.hpdd.intel.com/test_logs/bdd52dbe-a080-11e5-85ed-5254006e85c2/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_logs/bdd52dbe-a080-11e5-85ed-5254006e85c2/show_text</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00020000:01000000:0.0:1449883124.461043:0:14270:0:(lod_qos.c:218:lod_statfs_and_check()) lustre-OST0000-osc-MDT0000: turns inactive 00020000:01000000:0.0:1449883124.461045:0:14270:0:(lod_qos.c:218:lod_statfs_and_check()) lustre-OST0001-osc-MDT0000: turns inactive 00020000:01000000:0.0:1449883124.461046:0:14270:0:(lod_qos.c:218:lod_statfs_and_check()) lustre-OST0002-osc-MDT0000: turns inactive 00020000:01000000:0.0:1449883124.461048:0:14270:0:(lod_qos.c:218:lod_statfs_and_check()) lustre-OST0003-osc-MDT0000: turns inactive 00020000:01000000:0.0:1449883124.461048:0:14270:0:(lod_qos.c:218:lod_statfs_and_check()) lustre-OST0004-osc-MDT0000: turns inactive 00020000:01000000:0.0:1449883124.461049:0:14270:0:(lod_qos.c:218:lod_statfs_and_check()) lustre-OST0005-osc-MDT0000: turns inactive </pre> </div></div> <p>the OSP devices were turned to inactive due to "-ENOTCONN" just between the failed request from client.<br/> the problem here is a little different from that in <a href="https://jira.whamcloud.com/browse/LU-5526" title="recovery-mds-scale test failover_mds: dd: No space left on device" class="issue-link" data-issue-key="LU-5526"><del>LU-5526</del></a>, but it's still better to let the client to retry the creation request<br/> just as that in <a href="https://jira.whamcloud.com/browse/LU-5526" title="recovery-mds-scale test failover_mds: dd: No space left on device" class="issue-link" data-issue-key="LU-5526"><del>LU-5526</del></a>/<a href="https://jira.whamcloud.com/browse/LU-4846" title="Failover test failure on test suite replay-single test_26: No space left" class="issue-link" data-issue-key="LU-4846">LU-4846</a>.</p> <p>I don't think the client should be retrying, since this is the MDS's job to handle the recovery properly. I think it should hold the create request until the connections to the OSTs are available after restarting the MDS.</p> <p>Okay, I'll try to create the corresponding patch to wait the connections to OSTs for the creation request.</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17839" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17839</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7309" title="replay-single test_70b: no space left on device" class="issue-link" data-issue-key="LU-7309"><del>LU-7309</del></a> lod: wait OSP to connect for creation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d4500300ad8d94d574fe9c87bc91910e54dd9828</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17839/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17839/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7309" title="replay-single test_70b: no space left on device" class="issue-link" data-issue-key="LU-7309"><del>LU-7309</del></a> lod: notify client retry creation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5ebc00ec79565ad62e978af65b023343ad360675</p> <p>Landed for 2.8</p> Related LU-5526 LU-6844 LU-4846 LU-7352 Development End date Wed, 23 Dec 2015 17:51:51 +0000 Rank 1|hzxqon: Rank (Obsolete) 9223372036854775807 Severity Start date Fri, 16 Oct 2015 17:51:51 +0000