[LU-9893] replay-single test_70c: test failed to respond and timed out

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.10.1, Lustre 2.11.0
• Type: Bug
• Priority: Minor
• Reporter: James Casper
• Assignee: WC Triage
• Resolution: Unresolved

Environment

Trevis2, failover
server: RHEL 7.3, ldiskfs, branch master, v2.10.51, b3620
client: RHEL 7.3, branch master, v2.10.51, b3620

• Severity: 3

Description

https://testing.hpdd.intel.com/test_sessions/9b7c7e8e-7b5a-4f4d-af09-400c586a8340

Looks like another mds hang on umount issue for this build. It may be related to these tickets:

LU-9791 (parallel-scale-nfsv3 & v4)
LU-9856 (racer)
LU-9469 (conf-sanity)

However, this one does not show this message in the MDS console log:

BUG: unable to handle kernel NULL pointer dereference at (null)

What this shares in common with the above three is they all have an mds umount at the end of the suite_log. No further activity is seen.

From suite_log:

test_70c fail mds1 1 times
Failing mds1 on trevis-41vm3
CMD: trevis-41vm3 grep -c /mnt/lustre-mds1' ' /proc/mounts
Stopping /mnt/lustre-mds1 (opts:) on trevis-41vm3
CMD: trevis-41vm3 umount -d /mnt/lustre-mds1  (end of log)

Comments

[John Hammond]

Jim, can you grab /kdumproot/scratch//dumps/trevis-41vm3.trevis.hpdd.intel.com/10.9.5.239-2017-08-04-12:28:28/vmcore-dmesg.txt.

See https://testing.hpdd.intel.com/test_logs/f0c4415a-799c-11e7-8e1f-5254006e85c2/show_text

[James Casper]

Looks like that directory no longer exists:

[root@trevis-41 trevis-41vm3.trevis.hpdd.intel.com]# pwd
/scratch/dumps/trevis-41vm3.trevis.hpdd.intel.com
[root@trevis-41 trevis-41vm3.trevis.hpdd.intel.com]# ls -al
total 100
drwxr-xr-x 7 root root 4096 Aug 21 23:05 .
drwxr-xr-x 1433 root root 73728 Aug 10 17:23 ..
drwxr-xr-x 2 root root 4096 Aug 5 03:29 10.9.5.239-2017-08-05-03:29:12
drwxr-xr-x 2 root root 4096 Aug 7 18:48 10.9.5.239-2017-08-07-18:48:49
drwxr-xr-x 2 root root 4096 Aug 7 21:50 10.9.5.239-2017-08-07-21:50:33
drwxr-xr-x 2 root root 4096 Aug 14 06:19 10.9.5.239-2017-08-14-06:19:45
drwxr-xr-x 2 root root 4096 Aug 15 03:22 10.9.5.239-2017-08-15-03:21:47
[root@trevis-41 trevis-41vm3.trevis.hpdd.intel.com]#

[James Nunez (Inactive)]

I looked in Maloo for all replay-single test 70c timeouts (hangs) this year. I found 14 occurrences of this test hanging, but none of them are hanging on umount.

If we see this issue again, we need to look for the vmcore-dmesg.txt file as early as possible.

[John Hammond]

James, maybe try something like

find /scratch/dumps -name vmcore-dmesg.txt -exec grep --with-filename test_70c {} \;

to find other instances of this crash.

[Andreas Dilger]

+1 on master: https://testing.whamcloud.com/test_sets/079de990-f2fe-47b8-a70f-cb455e084ec8

[Qian Yingjin]

+1 on master: https://testing.whamcloud.com/test_sets/102cd218-cda2-4092-b4f1-991fa8aeda2e