[LU-6443] Hard Failover mmp test_10: FAIL: test_10 failed with 22

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.8.0
• Fix Version/s: Lustre 2.8.0
• Type: Bug
• Priority: Minor
• Reporter: Maloo
• Assignee: WC Triage
• Resolution: Fixed

Environment

client and server: lustre-master build #2967 RHEL6

• Severity: 3

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/2b3d6d6a-d5ea-11e4-8a24-5254006e85c2.

The sub-test test_10 failed with the following error:

test_10 failed with 22

Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
lustre-MDT0000: 315/5242880 files (7.3% non-contiguous), 772621/2621440 blocks
Mounting /dev/lvm-Role_OSS/P1 on ost1...
CMD: onyx-43vm4 mkdir -p /mnt/ost1
CMD: onyx-43vm4 test -b /dev/lvm-Role_OSS/P1
Starting ost1:   /dev/lvm-Role_OSS/P1 /mnt/ost1
CMD: onyx-43vm4 mkdir -p /mnt/ost1; mount -t lustre   		                   /dev/lvm-Role_OSS/P1 /mnt/ost1
onyx-43vm4: mount.lustre: mount /dev/mapper/lvm--Role_OSS-P1 at /mnt/ost1 failed: Invalid argument
onyx-43vm4: This may have multiple causes.
onyx-43vm4: Are the mount options correct?
onyx-43vm4: Check the syslog for more info.
Start of /dev/lvm-Role_OSS/P1 on ost1 failed 22
CMD: onyx-43vm3 grep -c /mnt/mds1' ' /proc/mounts
Stopping /mnt/mds1 (opts:) on onyx-43vm3
CMD: onyx-43vm3 umount -d /mnt/mds1
CMD: onyx-43vm3 lsmod | grep lnet > /dev/null && lctl dl | grep ' ST '
 mmp test_10: @@@@@@ FAIL: test_10 failed with 22 

Comments

[Oleg Drokin]

All failed subtests so from 2 onward) report that the mount of ost failed
LDISKFS-fs warning (device dm-0): ldiskfs_multi_mount_protect: MMP failure info: last update time: 1427611301, last update node: onyx-43vm8.onyx.hpdd.intel.com, last update device: dm-0

So the onyx-43vm8 is listed in the nodes list but no logs from any of the sbtests so I have no idea what it is actually even doing.

mmp seems to be doing it's thing though, but onyx-43vm8 might be doing somethign weird at the same time

[Andreas Dilger]

No logs are available from vm8 for this failure, closing bug until it is hit again.

[Andreas Dilger]

Reopen to land a debugging patch.

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: http://review.whamcloud.com/14442
Subject: LU-6443 tests: add debugging to mmp.sh test
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 42a20462cadf78f00d04f61b423d6d63a2c2d72f

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/14442/
Subject: LU-6443 tests: add debugging to mmp.sh test
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 50f7e269307d28c87511615d3e40df8050129249

[Joseph Gmitter (Inactive)]

Landed for 2.8