Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
Lustre 2.6.0, Lustre 2.7.0
-
client and server: lustre-b2_6-rc2 RHEL6 ldiskfs DNE mode
-
3
-
15076
Description
This issue was created by maloo for sarah <sarah@whamcloud.com>
This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/16302020-14ed-11e4-bb6a-5254006e85c2.
The sub-test test_17m failed with the following error:
test failed to respond and timed out
Hit this bug in many tests, the env is configured as 1 MDS with 2 MDTs. Didn't hit this error when the configuration is 2 MDSs with 2 MDTs
client console:
CMD: onyx-46vm7 mkdir -p /mnt/mds1 CMD: onyx-46vm7 test -b /dev/lvm-Role_MDS/P1 Starting mds1: /dev/lvm-Role_MDS/P1 /mnt/mds1 CMD: onyx-46vm7 mkdir -p /mnt/mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/mds1 onyx-46vm7: mount.lustre: mount /dev/mapper/lvm--Role_MDS-P1 at /mnt/mds1 failed: Input/output error onyx-46vm7: Is the MGS running? Start of /dev/lvm-Role_MDS/P1 on mds1 failed 5
Attachments
Issue Links
- is related to
-
-
- Open
-
-
-
- Resolved
-
-
-
- Resolved
-
-
-
- Resolved
-
-
-
- Closed
-
- is related to
-
-
- Resolved
-
Activity
As a note I don't see this is my regular RHEL testing but I can constantly reproduce this problem with my 3.12 kernel setup. This is with the MGS and MDS each being on separate nodes.
Hmm, I thought there are two problems here
1. it is not just MGS and MDT are not share the same node, if several targets are sharing the same mgc, you will meet similar problem, because after http://review.whamcloud.com/#/c/9967 is landed, we can not make sure the import is FULL before mgc enqueue lock and retrieve logs, unless the MGC is new.
2. How can we make sure the local config log is stale or not. I think that is the reason we saw LU-5658, where the local config is stale.
In seagate these bug is occurred only when MDT and MGS are on same node, in case when mdt starts earlier than mgs.
When MDT is on separate node it uses LOCAL configuration if can't retrieve it from MGS.
But when MDT and MGS are on same node MDT can't use LOCAL configuration:
/* Copy the setup log locally if we can. Don't mess around if we're
* running an MGS though (logs are already local). */
if (lctxt && lsi && IS_SERVER(lsi) && !IS_MGS(lsi) &&
cli->cl_mgc_configs_dir != NULL &&
lu2dt_dev(cli->cl_mgc_configs_dir->do_lu.lo_dev) ==
lsi->lsi_dt_dev) {
....
} else {
if (local_only) /* no local log at client side */
GOTO(out_pop, rc = -EIO);
}
While running replay-dual tests on master branch with MDSCOUNT=4, the same failure occurred:
https://testing.hpdd.intel.com/test_sets/33dfc794-6dba-11e4-9d65-5254006e85c2
https://testing.hpdd.intel.com/test_sets/5cb7b7f8-6dba-11e4-9d65-5254006e85c2
Sergey Cheremencev (sergey_cheremencev@xyratex.com) uploaded a new patch: http://review.whamcloud.com/12515
Subject: LU-5420 mgc: process config logs only in mgc_requeue_thread()
Project: fs/lustre-release
Branch: master
Current Patch Set: 3
Commit: 1c3148dd8645cfa94bf3c36cfbe41176334ad4c5
Hello
We hit these problem in xyratex and have another solution http://review.whamcloud.com/#/c/12515/.
Hope it could be helpful.
The patch is still failing with a hang at unmount time (this failed in four separate conf-sanity runs in different subtests):
01:52:25:INFO: task umount:26263 blocked for more than 120 seconds. 01:52:25: Tainted: G W --------------- 2.6.32-431.23.3.el6_lustre.g9f5284f.x86_64 #1 01:52:26:umount D 0000000000000000 0 26263 26262 0x00000080 01:52:27:Call Trace: 01:52:27: [<ffffffff8152b6e5>] rwsem_down_failed_common+0x95/0x1d0 01:52:27: [<ffffffff8152b843>] rwsem_down_write_failed+0x23/0x30 01:52:28: [<ffffffff8128f7f3>] call_rwsem_down_write_failed+0x13/0x20 01:52:28: [<ffffffffa0b13cd1>] client_disconnect_export+0x61/0x460 [ptlrpc] 01:52:28: [<ffffffffa058975a>] lustre_common_put_super+0x28a/0xbf0 [obdclass] 01:52:28: [<ffffffffa05bc508>] server_put_super+0x198/0xe50 [obdclass] 01:52:29: [<ffffffff8118b23b>] generic_shutdown_super+0x5b/0xe0 01:52:29: [<ffffffff8118b326>] kill_anon_super+0x16/0x60 01:52:29: [<ffffffffa0580d06>] lustre_kill_super+0x36/0x60 [obdclass] 01:52:29: [<ffffffff8118bac7>] deactivate_super+0x57/0x80 01:52:29: [<ffffffff811ab4cf>] mntput_no_expire+0xbf/0x110 01:52:29: [<ffffffff811ac01b>] sys_umount+0x7b/0x3a0
Without this patch I'm also not able to test past sanity.sh test_17m and test_17n without a shared MGS+MDS failing to mount due to -EIO and causing testing to hang until I remount the MDS. I'm able to mount it manually after 2 or 3 tries, so there must be some kind of startup race between the MDS and the MGS. Once I applied this patch I made it through all of sanity.sh and sanityn.sh with multiple MDS remounts without problems until I hit a memory allocation deadlock running dbench that looks unrelated.
Test specific issues aside, we need to fix this as putting 2 MDTs from a FS on a single node will be a likely failover configuration.
t:lustre-release# export LUSTRE=$HOME/lustre-release/lustre t:lustre-release# export MDSCOUNT=2 t:lustre-release# llmount.sh ... Starting mds1: -o loop /tmp/lustre-mdt1 /mnt/mds1 Started lustre-MDT0000 Starting mds2: -o loop /tmp/lustre-mdt2 /mnt/mds2 Started lustre-MDT0001 Starting ost1: -o loop /tmp/lustre-ost1 /mnt/ost1 Started lustre-OST0000 Starting ost2: -o loop /tmp/lustre-ost2 /mnt/ost2 Started lustre-OST0001 Starting client: t: -o user_xattr,flock t@tcp:/lustre /mnt/lustre Using TIMEOUT=20 seting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90 secs for update Updated after 3s: wanted 'procname_uid' got 'procname_uid' disable quota as required t:lustre-release# umount /mnt/mds1 t:lustre-release# mount /tmp/lustre-mdt1 /mnt/mds1 -o loop -t lustre mount.lustre: mount /dev/loop0 at /mnt/mds1 failed: Input/output error Is the MGS running?
Alexey Lyashkov (alexey.lyashkov@seagate.com) uploaded a new patch: http://review.whamcloud.com/13693
Subject:
LU-5420mgc: fix reconnectProject: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ccfca18ad2ae9acb84dbfc4c0b2217bd10a0589d