[LU-5420] Failure on test suite sanity test_17m: mount MDS failed, Input/output error

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.6.0, Lustre 2.7.0
• Labels: HB dne patch
• Environment: client and server: lustre-b2_6-rc2 RHEL6 ldiskfs DNE mode
• Severity: 3
• Assignee: Di Wang
• Reporter: Sarah Liu

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/16302020-14ed-11e4-bb6a-5254006e85c2.

The sub-test test_17m failed with the following error:

test failed to respond and timed out

Hit this bug in many tests, the env is configured as 1 MDS with 2 MDTs. Didn't hit this error when the configuration is 2 MDSs with 2 MDTs
client console:

CMD: onyx-46vm7 mkdir -p /mnt/mds1
CMD: onyx-46vm7 test -b /dev/lvm-Role_MDS/P1
Starting mds1:   /dev/lvm-Role_MDS/P1 /mnt/mds1
CMD: onyx-46vm7 mkdir -p /mnt/mds1; mount -t lustre   		                   /dev/lvm-Role_MDS/P1 /mnt/mds1
onyx-46vm7: mount.lustre: mount /dev/mapper/lvm--Role_MDS-P1 at /mnt/mds1 failed: Input/output error
onyx-46vm7: Is the MGS running?
Start of /dev/lvm-Role_MDS/P1 on mds1 failed 5

Issue Links

• is related to: LU-5404 sanity test_228b FAIL: Fail to start MDT.
• is related to: LU-5077 insanity test_1: out of memory on MDT in crypto_create_tfm()
• is related to: LU-5407 Failover failure on test suite replay-single test_58c: test_58c failed with 2
• is related to: LU-5130 Test failure sanity test_17n: destroy remote dir error 0
• is related to: LU-8206 ptlrpc_invalidate_import ( ASSERTION( imp->imp_invalid ) failed
• is related to: LU-4913 mgc import reconnect race

Comments

[Di Wang]

For now, you can use this http://review.whamcloud.com/13838/ , but that only reverts the patch (http://review.whamcloud.com/#/c/9967/)which cause the problem, and not real fix.

There are patches trying to fix this problem, but none of them are satisfied by everyone. So leave it to 2.8 for now.
http://review.whamcloud.com/13693
http://review.whamcloud.com/11258

[James A Simmons]

I see many patches for this. Which patches are valid?

[Jodi Levi (Inactive)]

Yes, my apologies.

[Peter Jones]

Actually Jodi the patches for master are still in flight. It is simply a workaround fix that has landed to b2_7

[Jodi Levi (Inactive)]

Patches landed to Master.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13838/
Subject: LU-5420 ptlrpc: revert ptlrpc_reconnect_import() changes
Project: fs/lustre-release
Branch: b2_7
Current Patch Set:
Commit: 02739a078f54b5ccdf49456fd0d1daea90472a8d

[Gerrit Updater]

Oleg Drokin ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13838
Subject: LU-5420 revert part of LU-4913
Project: fs/lustre-release
Branch: b2_7
Current Patch Set: 1
Commit: 77856caa2468dd69cfa5796bceb22c32aacf402f

[Gerrit Updater]

Oleg Drokin ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13832
Subject: LU-5420 revert part of LU-4913
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 260e150f98f07fa68fb124348ca9540e77fed100

[Jodi Levi (Inactive)]

http://review.whamcloud.com/#/c/12515/

[Gerrit Updater]

Alexey Lyashkov ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13693
Subject: LU-5420 mgc: fix reconnect
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ccfca18ad2ae9acb84dbfc4c0b2217bd10a0589d

[James A Simmons]

As a note I don't see this is my regular RHEL testing but I can constantly reproduce this problem with my 3.12 kernel setup. This is with the MGS and MDS each being on separate nodes.