|
According to the given log (lustre-log-lola-10-20160720_0136), the lola-10 was trying to connect MDT0003 for many many times:
As shown, at the beginning (timestamps: 1468994840), the MDT0002 on the lola-10 selected lola-11 (192.168.1.111@o2ib10) for the soaked-MDT0003-osp-MDT0002 connection. Unfortunately, the lola-11 returned -114 to the lola-10. That means another instance of soaked-MDT0003-osp-MDT0002 has already connected (or in connecting) to the MDT0003 on the lola-11. Because lola-10 and lola-11 were HA pairs, it was quite possible that the lola-11 took lola-10's role to server as MDT0002 during the lola-10 failover. Since MDT0002 had been mounted on the lola-11, the connection between MDT0002 and MDT0003 had been established at that time, then it is normal that the MDT0002 on the lola-10 got -114 failure.
Because lola-10 and lola-11 were HA pairs, when hit -114 failure, the MDT0002 on the lola-10 selected another peer for the soaked-MDT0003-osp-MDT0002 connection, it is lola-10 (192.168.1.111@o2ib10) itself. But it is obviously that MDT0003 was running on the lola-11, so the MDT0002 on the lola-10 got -19 failure.
Then, the MDT0002 has to find other peer for the soaked-MDT0003-osp-MDT0002 connection. There were only two choices, either lola-11 or lola-10, so it selected lola-11, the same as the first choice, but still hit -114 failure. And then try to connect lola-10 again, failed as -19 again.
So the MDT0002 on the lola-10 tried to connect to MDT0003 repeatedly for a long time (from the timestamps 1468994840 to 1469003821), 8981 seconds passed, but still cannot connect to the MDT0003. Such connection failure blocked the cross-MDT recovery. From the given logs, I cannot find when the soaked-MDT0003-osp-MDT0002 connection was established successfully. Because we have no logs on the lola-11, we do not know what happened for the lola-11. But it is suspected that it is related with another MDT0002 instance on the lola-11 because of the HA pairs configuration. Since the HA is controlled by some third-part HA software, such as heartbeat or pack-maker, can we check whether it works well or not under the failover case? Can we switch the resource by force under such case?
Anyway, offer me more logs on the lola-11 will be helpful. Thanks!
|
|
The HA configuration reads as follows:
- lola-8, lola-9 - failover cluster
- lola-8: MDT0000 (primary resource)
- lola-9: MDT0001 (primary resource)
- lola-10,lola-11 - failover cluster
- lola-10: MDT0002 (primary resource)
- lola-11: MDT0003 (primary resource)
MDTs are configured for failover during format. HA is controlled by the soak test framework. For the scenario above the following process flow were executed
- powercyle of (primary) MDS
- mount of MDT on secondary MDS
- Don't wait for recovery to complete on secondary; switch back to primary node immediately after MDT mount completes
- mount MDT on primary MDS again
The console and message files for lola-11 aren't available anymore. Because of capacity reason all files older than 6 weeks are deleted on the cluster's share FS.
For build '20160916' (see https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160916) this effect didn't occurred
till now although the effect might be eclipsed by LU-8580.
|
console-lola-10.log.bz2
lustre-log-lola-19-20160720_0136.bz2