[LU-5420] Failure on test suite sanity test_17m: mount MDS failed, Input/output error

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.6.0, Lustre 2.7.0
• Labels: HB dne patch
• Environment: client and server: lustre-b2_6-rc2 RHEL6 ldiskfs DNE mode
• Severity: 3
• Assignee: Di Wang
• Reporter: Sarah Liu

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/16302020-14ed-11e4-bb6a-5254006e85c2.

The sub-test test_17m failed with the following error:

test failed to respond and timed out

Hit this bug in many tests, the env is configured as 1 MDS with 2 MDTs. Didn't hit this error when the configuration is 2 MDSs with 2 MDTs
client console:

CMD: onyx-46vm7 mkdir -p /mnt/mds1
CMD: onyx-46vm7 test -b /dev/lvm-Role_MDS/P1
Starting mds1:   /dev/lvm-Role_MDS/P1 /mnt/mds1
CMD: onyx-46vm7 mkdir -p /mnt/mds1; mount -t lustre   		                   /dev/lvm-Role_MDS/P1 /mnt/mds1
onyx-46vm7: mount.lustre: mount /dev/mapper/lvm--Role_MDS-P1 at /mnt/mds1 failed: Input/output error
onyx-46vm7: Is the MGS running?
Start of /dev/lvm-Role_MDS/P1 on mds1 failed 5

Issue Links

• is related to: LU-5404 sanity test_228b FAIL: Fail to start MDT.
• is related to: LU-5077 insanity test_1: out of memory on MDT in crypto_create_tfm()
• is related to: LU-5407 Failover failure on test suite replay-single test_58c: test_58c failed with 2
• is related to: LU-5130 Test failure sanity test_17n: destroy remote dir error 0
• is related to: LU-8206 ptlrpc_invalidate_import ( ASSERTION( imp->imp_invalid ) failed
• is related to: LU-4913 mgc import reconnect race

Comments

[James A Simmons]

I see many patches for this. Which patches are valid?

[Jodi Levi (Inactive)]

Yes, my apologies.

[Peter Jones]

Actually Jodi the patches for master are still in flight. It is simply a workaround fix that has landed to b2_7

[Jodi Levi (Inactive)]

Patches landed to Master.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13838/
Subject: LU-5420 ptlrpc: revert ptlrpc_reconnect_import() changes
Project: fs/lustre-release
Branch: b2_7
Current Patch Set:
Commit: 02739a078f54b5ccdf49456fd0d1daea90472a8d

[Gerrit Updater]

Oleg Drokin ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13838
Subject: LU-5420 revert part of LU-4913
Project: fs/lustre-release
Branch: b2_7
Current Patch Set: 1
Commit: 77856caa2468dd69cfa5796bceb22c32aacf402f

[Gerrit Updater]

Oleg Drokin ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13832
Subject: LU-5420 revert part of LU-4913
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 260e150f98f07fa68fb124348ca9540e77fed100

[Jodi Levi (Inactive)]

http://review.whamcloud.com/#/c/12515/

[Gerrit Updater]

Alexey Lyashkov ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13693
Subject: LU-5420 mgc: fix reconnect
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ccfca18ad2ae9acb84dbfc4c0b2217bd10a0589d

[James A Simmons]

As a note I don't see this is my regular RHEL testing but I can constantly reproduce this problem with my 3.12 kernel setup. This is with the MGS and MDS each being on separate nodes.

[Di Wang]

Hmm, I thought there are two problems here
1. it is not just MGS and MDT are not share the same node, if several targets are sharing the same mgc, you will meet similar problem, because after http://review.whamcloud.com/#/c/9967 is landed, we can not make sure the import is FULL before mgc enqueue lock and retrieve logs, unless the MGC is new.
2. How can we make sure the local config log is stale or not. I think that is the reason we saw LU-5658, where the local config is stale.