[LU-4223] conf-sanity test_32c, test_32d: could not find any free loop device

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.6.0, Lustre 2.4.2, Lustre 2.5.1, Lustre 2.8.0
• Labels: dne mn4 sdsc
• Severity: 3
• Assignee: Di Wang
• Reporter: Maloo

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/751ee23a-4106-11e3-a1e8-52540035b04c.

The sub-test test_32c failed with the following error:

test_32c failed with 1

Info required for matching: conf-sanity 32c

Issue Links

• is duplicated by: LU-4261 Test failure on test suite sanity-quota, subtest test_36

Comments

[Andreas Dilger]

False alarm - it is actually LU-4358 that is being hit, but is misdiagnosed as LU-4223.

I'm closing this bug since it looks like it is not being hit in recent test runs.

[Andreas Dilger]

This patch was landed on 2013-12-09 but conf-sanity is still reporting this bug for failures:

https://maloo.whamcloud.com/test_sets/7d7a27aa-66ae-11e3-93e2-52540035b04c
https://maloo.whamcloud.com/test_sets/202cbd9e-66c5-11e3-93e2-52540035b04c
https://maloo.whamcloud.com/test_sets/868c4f72-66fd-11e3-a234-52540035b04c

and others.

[Di Wang]

John, yes, this makes sense, I updated the patch, and please have a look.

[John Hammond]

Hi Di,

1. There are a few other places in lustre/utils/ where popen() if followed by fclose(), including one in is_e2fsprogs_feature_supp(). We should fix those too.

1. This may not be enough. On my RHEL 6.4 kernel (2.6.32-358.18.1.el6.lustre.x86_64) if there are IOs still in flight to a loop device then it cannot be detached:

   # dd if=/dev/zero of=/tmp/0 bs=1M count=100
   100+0 records in
   100+0 records out
   104857600 bytes (105 MB) copied, 0.195863 s, 535 MB/s
   # losetup -f /tmp/0
   # losetup -a
   /dev/loop0: [fc01]:1055063 (/tmp/0)
   # dd if=/dev/zero of=/dev/loop0 bs=1M count=100; losetup -d /dev/loop0
   100+0 records in
   100+0 records out
   104857600 bytes (105 MB) copied, 0.15608 s, 672 MB/s
   loop: can't delete device /dev/loop0: Device or resource busy
   # losetup -a
   /dev/loop0: [fc01]:1055063 (/tmp/0)
   # losetup -d /dev/loop0
   #
   # losetup /dev/loop0 /tmp/0; dd if=/dev/zero of=/dev/loop0 oflag=sync bs=1M count=100; losetup -d /dev/loop0
   100+0 records in
   100+0 records out
   104857600 bytes (105 MB) copied, 3.72073 s, 28.2 MB/s
   loop: can't delete device /dev/loop0: Device or resource busy
   

   As you see above, asking for sync IO to the loop device is not enough. So we probably need to add some wait retry logic to loop_cleanup().

[Di Wang]

http://review.whamcloud.com/8409

[Di Wang]

John: I think you are right, and it should use pclose, instead of fclose. Good catch!

[John Hammond]

In mount_utils_ldiskfs.c:is_feature_enabled(), popen() is used to invoke debugfs but fclose() is used to close the FILE * returned from popen(). Hence wait() is not called, debugfs may still be running (and holding the loop device open). This prevents losetup -d from detaching it.

[Jian Yu]

The patch was cherry-picked to Lustre b2_4 branch.

[Andreas Dilger]

This patch should be landed to b2_4 and b2_5.

[Andreas Dilger]

Di, please use More Actions->Link to link duplicate bugs. Just closing a bug as a duplicate does not link it to the original bug (unlike with Bugzilla).

[Andreas Dilger]

This appears to be the most common failure preventing review-dne from passing all of its tests (7 of 11 failures in the past three days if one adds LU-4261 failures). Fixing this would probably allow review-dne to pass consistently enough to be enforced.