[LU-4223] conf-sanity test_32c, test_32d: could not find any free loop device

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.6.0, Lustre 2.4.2, Lustre 2.5.1, Lustre 2.8.0
• Labels: dne mn4 sdsc
• Severity: 3
• Assignee: Di Wang
• Reporter: Maloo

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/751ee23a-4106-11e3-a1e8-52540035b04c.

The sub-test test_32c failed with the following error:

test_32c failed with 1

Info required for matching: conf-sanity 32c

Issue Links

• is duplicated by: LU-4261 Test failure on test suite sanity-quota, subtest test_36

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13265/
Subject: LU-4223 tests: fix conf-sanity test_32 typo
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 5da53bcb1d8c38157325505f6619d6b6c3d4db6a

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13265
Subject: LU-4223 tests: fix conf-sanity test_32 typo
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 05a59163d8427f0c77d3aa6b31486b2219fb49d2

[Jian Yu]

Landed for Lustre 2.5.1.

[Jian Yu]

http://review.whamcloud.com/8409

The above patch was not cherry-picked to Lustre b2_5 branch.

The same failure occurred on Lustre b2_5 build #5:
https://maloo.whamcloud.com/test_sets/16cd0f30-7497-11e3-8b21-52540035b04c

Here is the back-ported patch on Lustre b2_5 branch: http://review.whamcloud.com/8723

[Andreas Dilger]

False alarm - it is actually LU-4358 that is being hit, but is misdiagnosed as LU-4223.

I'm closing this bug since it looks like it is not being hit in recent test runs.

[Andreas Dilger]

This patch was landed on 2013-12-09 but conf-sanity is still reporting this bug for failures:

https://maloo.whamcloud.com/test_sets/7d7a27aa-66ae-11e3-93e2-52540035b04c
https://maloo.whamcloud.com/test_sets/202cbd9e-66c5-11e3-93e2-52540035b04c
https://maloo.whamcloud.com/test_sets/868c4f72-66fd-11e3-a234-52540035b04c

and others.

[Di Wang]

John, yes, this makes sense, I updated the patch, and please have a look.

[John Hammond]

Hi Di,

1. There are a few other places in lustre/utils/ where popen() if followed by fclose(), including one in is_e2fsprogs_feature_supp(). We should fix those too.

1. This may not be enough. On my RHEL 6.4 kernel (2.6.32-358.18.1.el6.lustre.x86_64) if there are IOs still in flight to a loop device then it cannot be detached:

   # dd if=/dev/zero of=/tmp/0 bs=1M count=100
   100+0 records in
   100+0 records out
   104857600 bytes (105 MB) copied, 0.195863 s, 535 MB/s
   # losetup -f /tmp/0
   # losetup -a
   /dev/loop0: [fc01]:1055063 (/tmp/0)
   # dd if=/dev/zero of=/dev/loop0 bs=1M count=100; losetup -d /dev/loop0
   100+0 records in
   100+0 records out
   104857600 bytes (105 MB) copied, 0.15608 s, 672 MB/s
   loop: can't delete device /dev/loop0: Device or resource busy
   # losetup -a
   /dev/loop0: [fc01]:1055063 (/tmp/0)
   # losetup -d /dev/loop0
   #
   # losetup /dev/loop0 /tmp/0; dd if=/dev/zero of=/dev/loop0 oflag=sync bs=1M count=100; losetup -d /dev/loop0
   100+0 records in
   100+0 records out
   104857600 bytes (105 MB) copied, 3.72073 s, 28.2 MB/s
   loop: can't delete device /dev/loop0: Device or resource busy
   

   As you see above, asking for sync IO to the loop device is not enough. So we probably need to add some wait retry logic to loop_cleanup().

[Di Wang]

http://review.whamcloud.com/8409

[Di Wang]

John: I think you are right, and it should use pclose, instead of fclose. Good catch!