[LU-4846] Failover test failure on test suite replay-single test_26: No space left

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Affects Version/s: Lustre 2.6.0, Lustre 2.7.0, Lustre 2.5.3, Lustre 2.8.0
• Labels: mq115
• Environment: client and server: lustre-master build # 1945 RHEL6
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Maloo

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/17d5d848-b1c4-11e3-9a4b-52540035b04c.

The sub-test test_26 failed with the following error:

test_26 failed with 5

In the previous tag-2.5.56, we didn't hit this no space left error
test log:

== replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) == 03:14:43 (1395483283)
CMD: client-30vm3 sync; sync; sync
Filesystem                        1K-blocks   Used Available Use% Mounted on
client-30vm3:client-30vm7:/lustre  14449456 760996  12928224   6% /mnt/lustre
CMD: client-30vm1.lab.whamcloud.com,client-30vm5,client-30vm6 mcreate /mnt/lustre/fsa-\$(hostname); rm /mnt/lustre/fsa-\$(hostname)
CMD: client-30vm1.lab.whamcloud.com,client-30vm5,client-30vm6 if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-\$(hostname); rm /mnt/lustre2/fsa-\$(hostname); fi
CMD: client-30vm3 /usr/sbin/lctl --device lustre-MDT0000 notransno
CMD: client-30vm3 /usr/sbin/lctl --device lustre-MDT0000 readonly
CMD: client-30vm3 /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000
multiop /mnt/lustre/f26.replay-single-1 vO_tSc
TMPPIPE=/tmp/multiop_open_wait_pipe.19802
open(O_RDWR|O_CREAT): No space left on device

Issue Links

• is duplicated by: LU-5947 Failover replay-single test_15: No space left on device
• is duplicated by: LU-7294 Failover - replay-single test_53b: multiop_bg_pause /mnt/lustre/d53b.replay-single-1/f failed
• is related to: LU-10613 replay-single tests 20c, 21, 23, 24, 25, 26, 30, 48, 53f, 53g, 62, 70b, 70c, fails on open with ‘ No space left on device’
• is related to: LU-5526 recovery-mds-scale test failover_mds: dd: No space left on device
• is related to: LU-7309 replay-single test_70b: no space left on device

Comments

[Andreas Dilger]

Close ancient ticket.

[Saurabh Tandan (Inactive)]

Another instance found for hardfailover: EL6.7 Server/Client
https://testing.hpdd.intel.com/test_sets/439c1dda-bc93-11e5-8f65-5254006e85c2

Another instance found for hardfailover: EL6.7 Server/Client
https://testing.hpdd.intel.com/test_sets/48226896-bc93-11e5-8f65-5254006e85c2

[James Nunez (Inactive)]

LU-7309 seems to be tracking the same problem; multiple replay-single tests failing with 'No space left on device'

[Saurabh Tandan (Inactive)]

replay-single test_18, test_21, test_48 failed with same issue.
master, build# 3266, 2.7.64 tag
Hard Failover: EL7 Server/SLES11 SP3 Client
https://testing.hpdd.intel.com/test_sets/a8b3fb9e-a077-11e5-8d69-5254006e85c2

[Saurabh Tandan (Inactive)]

replay-dual test_26 failing with same issue.
master, build# 3266, 2.7.64 tag
Hard Failover: EL7 Server/SLES11 SP3 Client
https://testing.hpdd.intel.com/test_sets/a6c5740c-a077-11e5-8d69-5254006e85c2

[Saurabh Tandan (Inactive)]

master, build# 3266, 2.7.64 tag
Hard Failover: EL6.7 Server/SLES11 SP3 Clients
replay-single test_48 failed with same issue.
https://testing.hpdd.intel.com/test_sets/bcb62a46-a080-11e5-85ed-5254006e85c2

[Saurabh Tandan (Inactive)]

master, build# 3264, 2.7.64 tag
Hard Failover: EL6.7 Server/Client - ZFS
replay-single test_18 failed and test_20 timeout because of the same issue.
https://testing.hpdd.intel.com/test_sets/2f719028-9ebc-11e5-98a4-5254006e85c2

[Saurabh Tandan (Inactive)]

master, build# 3264, 2.7.64 tag
Hard Failover: EL6.7 Server/Client
Same issue blocking a series of tests tests_15,18,21 with the same issue.

https://testing.hpdd.intel.com/test_sets/80a20678-9edd-11e5-87a9-5254006e85c2

[Hongchao Zhang]

should be a duplicate

[Sarah Liu]

Hit this in current master:
https://testing.hpdd.intel.com/test_sets/9709e12a-e1b3-11e4-87fa-5254006e85c2