https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4223 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/751ee23a-4106-11e3-a1e8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/751ee23a-4106-11e3-a1e8-52540035b04c</a>.</p> <p>The sub-test test_32c failed with the following error:</p> <blockquote> <p>test_32c failed with 1</p></blockquote> <p>Info required for matching: conf-sanity 32c</p> LU-4223

conf-sanity test_32c, test_32d: could not find any free loop device

Bug Critical Resolved Fixed Di Wang Maloo dne mn4 sdsc Thu, 7 Nov 2013 07:17:26 +0000 Wed, 16 Mar 2016 05:18:51 +0000 Wed, 18 Dec 2013 19:54:38 +0000 Lustre 2.6.0 Lustre 2.4.2 Lustre 2.5.1 Lustre 2.8.0 0 9 <p>test log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: wtm-32vm3 tunefs.lustre --quota /tmp/t32/ost wtm-32vm3: tunefs.lustre: out of loop devices! wtm-32vm3: wtm-32vm3: tunefs.lustre FATAL: Loop device setup for /tmp/t32/ost failed: Too many open files wtm-32vm3: tunefs.lustre: exiting with 24 (Too many open files) checking for existing Lustre data: found </pre> </div></div> <p>Oleg reports that he runs out of loop devices when running conf-sanity repeatedly on the same node. It seems there is a leak in the configuration of the loop devices, either by mount.lustre/unmount, or something else in conf-sanity. It is likely hitting DNE testing more than regular testing just due to DNE configurations using more MDT and OST devices.</p> <p>In the test output:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Upgrading from disk2_3-ldiskfs.tar.bz2, created with: Commit: 2.3.0 Kernel: 2.6.32-279.5.1.el6_lustre.gb16fe80.x86_64 Arch: x86_64 CMD: wtm-32vm3 tunefs.lustre --dryrun /tmp/t32/mdt : : loop: can't delete device /dev/loop3: Device or resource busy </pre> </div></div> <p>So it looks like the device is being referenced. I've noticed in recent DNE testing that the MDT device did not unmount cleanly, so this should be tested as the possible root cause of the problem.</p> <p>Hmm, it seems a few tests did not do "umount -d" when umount. <a href="http://review.whamcloud.com/8296" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8296</a> </p> <p>This appears to be the most common failure preventing review-dne from passing all of its tests (7 of 11 failures in the past three days if one adds <a href="https://jira.whamcloud.com/browse/LU-4261" title="Test failure on test suite sanity-quota, subtest test_36" class="issue-link" data-issue-key="LU-4261"><del>LU-4261</del></a> failures). Fixing this would probably allow review-dne to pass consistently enough to be enforced.</p> <p>Di, please use More Actions-&gt;Link to link duplicate bugs. Just closing a bug as a duplicate does not link it to the original bug (unlike with Bugzilla).</p> <p>This patch should be landed to b2_4 and b2_5. </p> <p>The patch was cherry-picked to Lustre b2_4 branch.</p> <p>In mount_utils_ldiskfs.c:is_feature_enabled(), popen() is used to invoke debugfs but fclose() is used to close the FILE * returned from popen(). Hence wait() is not called, debugfs may still be running (and holding the loop device open). This prevents losetup -d from detaching it.</p> <p>John: I think you are right, and it should use pclose, instead of fclose. Good catch!</p> <p> <a href="http://review.whamcloud.com/8409" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8409</a></p> <p>Hi Di,</p> <ol> <li>There are a few other places in lustre/utils/ where popen() if followed by fclose(), including one in is_e2fsprogs_feature_supp(). We should fix those too.</li> </ol> <ol> <li>This may not be enough. On my RHEL 6.4 kernel (2.6.32-358.18.1.el6.lustre.x86_64) if there are IOs still in flight to a loop device then it cannot be detached: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># dd if=/dev/zero of=/tmp/0 bs=1M count=100 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.195863 s, 535 MB/s # losetup -f /tmp/0 # losetup -a /dev/loop0: [fc01]:1055063 (/tmp/0) # dd if=/dev/zero of=/dev/loop0 bs=1M count=100; losetup -d /dev/loop0 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.15608 s, 672 MB/s loop: can't delete device /dev/loop0: Device or resource busy # losetup -a /dev/loop0: [fc01]:1055063 (/tmp/0) # losetup -d /dev/loop0 # # losetup /dev/loop0 /tmp/0; dd if=/dev/zero of=/dev/loop0 oflag=sync bs=1M count=100; losetup -d /dev/loop0 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 3.72073 s, 28.2 MB/s loop: can't delete device /dev/loop0: Device or resource busy </pre> </div></div> <p>As you see above, asking for sync IO to the loop device is not enough. So we probably need to add some wait retry logic to loop_cleanup().</p></li> </ol> <p>John, yes, this makes sense, I updated the patch, and please have a look.</p> <p>This patch was landed on 2013-12-09 but conf-sanity is still reporting this bug for failures:</p> <p><a href="https://maloo.whamcloud.com/test_sets/7d7a27aa-66ae-11e3-93e2-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7d7a27aa-66ae-11e3-93e2-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/202cbd9e-66c5-11e3-93e2-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/202cbd9e-66c5-11e3-93e2-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/868c4f72-66fd-11e3-a234-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/868c4f72-66fd-11e3-a234-52540035b04c</a></p> <p>and others.</p> <p>False alarm - it is actually <a href="https://jira.whamcloud.com/browse/LU-4358" title="conf-sanity test_32c: sha1sum verification failed" class="issue-link" data-issue-key="LU-4358"><del>LU-4358</del></a> that is being hit, but is misdiagnosed as <a href="https://jira.whamcloud.com/browse/LU-4223" title="conf-sanity test_32c, test_32d: could not find any free loop device" class="issue-link" data-issue-key="LU-4223"><del>LU-4223</del></a>.</p> <p>I'm closing this bug since it looks like it is not being hit in recent test runs.</p> <blockquote><p><a href="http://review.whamcloud.com/8409" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8409</a></p></blockquote> <p>The above patch was not cherry-picked to Lustre b2_5 branch.</p> <p>The same failure occurred on Lustre b2_5 build #5:<br/> <a href="https://maloo.whamcloud.com/test_sets/16cd0f30-7497-11e3-8b21-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/16cd0f30-7497-11e3-8b21-52540035b04c</a></p> <p>Here is the back-ported patch on Lustre b2_5 branch: <a href="http://review.whamcloud.com/8723" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8723</a></p> <p>Landed for Lustre 2.5.1.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13265" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13265</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4223" title="conf-sanity test_32c, test_32d: could not find any free loop device" class="issue-link" data-issue-key="LU-4223"><del>LU-4223</del></a> tests: fix conf-sanity test_32 typo<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 05a59163d8427f0c77d3aa6b31486b2219fb49d2</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13265/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13265/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4223" title="conf-sanity test_32c, test_32d: could not find any free loop device" class="issue-link" data-issue-key="LU-4223"><del>LU-4223</del></a> tests: fix conf-sanity test_32 typo<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5da53bcb1d8c38157325505f6619d6b6c3d4db6a</p> Duplicate LU-4261 Development Rank 1|hzw89r: Rank (Obsolete) 11491 Severity