https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8428 Lustre <p>The effect happens during soak testing of build '20160713' (see <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160713" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160713</a>)<br/> OSTs have been configured using <em>zfs</em>, MDTs using <em>ldiskfs</em><br/> Environment consist of 4 MDSes with 1 MDT each and 6 OSSes with 4 OSTs each.<br/> DNE is enabled.<br/> MDS and OSS nodes are configured in active-active HA configuration.</p> <p>The effect of very long recovery times occurred after the restart of two MDS nodes. <br/> <tt>lola-9</tt> : 89:38 min recovery time<br/> <tt>lola-10</tt> : 193:55 min recovery time</p> <p>An OSS failover/failback injected by the framework, while the MDTs of the restarted MDS nodes were in recovery, finished successful in 48 seconds. </p> <p>Sequence of events:</p> <ul> <li>2016-07-19 22:57:52 - lola-10 mds_restart</li> <li>2016-07-19 23:07:21 - lola-10 , recovery started</li> <li>2016-07-20 00:41:55 - lola-9 mds_restart</li> <li>2016-07-20 00:51:55 - lola-9 , recovery started</li> <li>2016-07-20 00:53:07 - lola-3 oss_failover</li> <li>2016-07-20 00:58:55 - lola-3 recovery started</li> <li>2016-07-20 00:59:43 - lola-3 kernel: Lustre: soaked-OST0007: Recovery over after 0:48, of 23 clients 23 recovered and 0 were evicted.</li> <li>2016-07-20 02:21:35 - lola-9 kernel: Lustre: soaked-MDT0001: Recovery over after 89:38, of 22 clients 21 recovered and 1 was evicted.</li> <li>2016-07-20 02:21:17 - lola-10 kernel: Lustre: soaked-MDT0002: Recovery over after 193:55, of 22 clients 20 recovered and 2 were evicted.</li> </ul> <p>Clients were in the following state during recovery:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@lola-16 ~]# pdsh -g clients 'lctl get_param *.*.state | grep -A 2 -E "MDT0001|MDT0002" ' | dshbak -c | less -i ---------------- lola-13 ---------------- mdc.soaked-MDT0001-mdc-ffff88028add7800.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff88028add7800.state= current_state: REPLAY_WAIT state_history: ---------------- lola-16 ---------------- mdc.soaked-MDT0001-mdc-ffff880415a56400.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff880415a56400.state= current_state: REPLAY_WAIT state_history: ---------------- lola-19 ---------------- mdc.soaked-MDT0001-mdc-ffff88021b0fa800.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff88021b0fa800.state= current_state: REPLAY_WAIT state_history: ---------------- lola-20 ---------------- mdc.soaked-MDT0001-mdc-ffff880c9b13f400.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff880c9b13f400.state= current_state: REPLAY_WAIT state_history: ---------------- lola-21 ---------------- mdc.soaked-MDT0001-mdc-ffff881033e32400.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff881033e32400.state= current_state: REPLAY_WAIT state_history: ---------------- ---------------- lola-22 ---------------- mdc.soaked-MDT0001-mdc-ffff88082e455000.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff88082e455000.state= current_state: REPLAY_WAIT state_history: ---------------- lola-23 ---------------- mdc.soaked-MDT0001-mdc-ffff8808e6af4400.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff8808e6af4400.state= current_state: REPLAY_WAIT state_history: ---------------- lola-24 ---------------- mdc.soaked-MDT0001-mdc-ffff88102e910800.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff88102e910800.state= current_state: REPLAY_WAIT state_history: ---------------- lola-25 ---------------- mdc.soaked-MDT0001-mdc-ffff88102f38a400.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff88102f38a400.state= current_state: REPLAY_WAIT state_history: ---------------- lola-26 ---------------- mdc.soaked-MDT0001-mdc-ffff880f7857fc00.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff880f7857fc00.state= current_state: REPLAY_WAIT state_history: ---------------- lola-27 ---------------- mdc.soaked-MDT0001-mdc-ffff88081a0eec00.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff88081a0eec00.state= current_state: REPLAY_WAIT state_history: ---------------- lola-29 ---------------- ---------------- lola-13 ---------------- mdc.soaked-MDT0001-mdc-ffff88028add7800.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff88028add7800.state= current_state: REPLAY_WAIT state_history: ---------------- lola-16 ---------------- mdc.soaked-MDT0001-mdc-ffff880415a56400.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff880415a56400.state= current_state: REPLAY_WAIT state_history: ---------------- lola-19 ---------------- mdc.soaked-MDT0001-mdc-ffff88021b0fa800.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff88021b0fa800.state= current_state: REPLAY_WAIT state_history: ---------------- lola-20 ---------------- mdc.soaked-MDT0001-mdc-ffff880c9b13f400.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff880c9b13f400.state= current_state: REPLAY_WAIT state_history: ---------------- lola-21 ---------------- mdc.soaked-MDT0001-mdc-ffff881033e32400.state= current_state: REPLAY_WAIT state_history: -- mdc.soaked-MDT0002-mdc-ffff881033e32400.state= current_state: REPLAY_WAIT state_history: ---------------- lola-22 ---------------- mdc.soaked-MDT0001-mdc-ffff88082e455000.state= current_state: REPLAY_WAIT state_history: </pre> </div></div> <p>Attached files:<br/> Console, messages logs of MDS nodes <tt>lola-<span class="error">&#91;9,10&#93;</span></tt><br/> debug kernel logs of MDS nodes <tt>lola-<span class="error">&#91;9,10&#93;</span></tt> and single Lustre client node <tt>lola-19</tt></p> lola<br/> build: <a href="https://build.hpdd.intel.com/job/lustre-master/3406">https://build.hpdd.intel.com/job/lustre-master/3406</a> LU-8428

Very long recovery times for MDTs after MDS restart

Bug Major Resolved Duplicate nasf Frank Heckes soak Thu, 21 Jul 2016 14:03:21 +0000 Wed, 5 Oct 2016 15:56:12 +0000 Wed, 5 Oct 2016 15:55:56 +0000 Lustre 2.9.0 0 9 <p>debug kernel files with suffix '_0136' had been created during recovery, those with suffix '_0225' after completion of recovery.</p> <p>We are still seeing the long recovery times, example:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lola-10 ---------------- mdt.soaked-MDT0002.recovery_status= status: COMPLETE recovery_start: 1471407102 recovery_duration: 1644 completed_clients: 21/21 replayed_requests: 0 last_transno: 150323855360 VBR: ENABLED IR: ENABLED </pre> </div></div> <p>According to the given log (lustre-log-lola-10-20160720_0136), the lola-10 was trying to connect MDT0003 for many many times:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:00000080:23.0:1468994840.970247:0:6167:0:(obd_config.c:1148:class_process_config()) processing cmd: cf003 00000004:00080000:23.0:1468994840.970280:0:6167:0:(osp_dev.c:1121:osp_init0()) soaked-MDT0003-osp-MDT0002: connect to soaked-MDT0002-osd (soaked-MDT0003-osp-MDT0002) 00080000:01000000:23.0:1468994840.970287:0:6167:0:(osd_handler.c:6820:osd_obd_connect()) connect #5 00000020:00000080:23.0:1468994840.970290:0:6167:0:(genops.c:1169:class_connect()) connect: client soaked-MDT0002-osd_UUID, cookie 0x24fc9dac120838ee 00010000:00080000:23.0:1468994840.970307:0:6167:0:(ldlm_lib.c:112:import_set_conn()) imp ffff8803fc39b800@soaked-MDT0003-osp-MDT0002: add connection 192.168.1.111@o2ib10 at head 00000020:00000080:23.0:1468994840.971931:0:6167:0:(obd_config.c:552:class_setup()) finished setup of obd soaked-MDT0003-osp-MDT0002 (uuid soaked-MDT0002-mdtlov_UUID) 00000020:00000080:23.0:1468994840.971939:0:6167:0:(obd_config.c:1148:class_process_config()) processing cmd: cf005 00000020:00000080:23.0:1468994840.971943:0:6167:0:(obd_config.c:1159:class_process_config()) adding mapping from uuid 192.168.1.111@o2ib10 to nid 0x5000ac0a8016f (192.168.1.111@o2ib10) 00000020:00000080:23.0:1468994840.971948:0:6167:0:(obd_config.c:1148:class_process_config()) processing cmd: cf00b 00010000:00080000:23.0:1468994840.971957:0:6167:0:(ldlm_lib.c:95:import_set_conn()) imp ffff8803fc39b800@soaked-MDT0003-osp-MDT0002: found existing conn 192.168.1.111@o2ib10 00000020:00000080:23.0:1468994840.971962:0:6167:0:(obd_config.c:1148:class_process_config()) processing cmd: cf005 00000020:00000080:23.0:1468994840.971967:0:6167:0:(obd_config.c:1159:class_process_config()) adding mapping from uuid 192.168.1.110@o2ib10 to nid 0x5000ac0a8016e (192.168.1.110@o2ib10) 00000020:00000080:23.0:1468994840.971971:0:6167:0:(obd_config.c:1148:class_process_config()) processing cmd: cf00b 00010000:00080000:23.0:1468994840.971976:0:6167:0:(ldlm_lib.c:112:import_set_conn()) imp ffff8803fc39b800@soaked-MDT0003-osp-MDT0002: add connection 192.168.1.110@o2ib10 at tail 00000020:00000080:23.0:1468994840.971980:0:6167:0:(obd_config.c:1148:class_process_config()) processing cmd: cf014 00000004:01000000:23.0:1468994840.971993:0:6167:0:(lod_lov.c:225:lod_add_device()) osp:soaked-MDT0003_UUID idx:3 gen:1 00000004:01000000:23.0:1468994840.971997:0:6167:0:(osp_dev.c:1411:osp_obd_connect()) connect #0 00000020:00000080:23.0:1468994840.972004:0:6167:0:(genops.c:1169:class_connect()) connect: client soaked-MDT0002-mdtlov_UUID, cookie 0x24fc9dac120838fc 00000100:00080000:23.0:1468994840.972006:0:6167:0:(import.c:679:ptlrpc_connect_import()) ffff8803fc39b800 soaked-MDT0003_UUID: changing import state from NEW to CONNECTING 00000100:00080000:23.0:1468994840.972010:0:6167:0:(import.c:523:import_select_connection()) soaked-MDT0003-osp-MDT0002: connect to NID 192.168.1.111@o2ib10 last attempt 0 00000100:00080000:23.0:1468994840.972014:0:6167:0:(import.c:601:import_select_connection()) soaked-MDT0003-osp-MDT0002: import ffff8803fc39b800 using connection 192.168.1.111@o2ib10/192.168.1.111@o2ib10 ... 00000100:02020000:1.0:1468994840.996289:0:4562:0:(client.c:1266:ptlrpc_check_status()) 11-0: soaked-MDT0003-osp-MDT0002: operation mds_connect to node 192.168.1.111@o2ib10 failed: rc = -114 00000100:00080000:1.0:1468994840.996295:0:4562:0:(import.c:1290:ptlrpc_connect_interpret()) ffff8803fc39b800 soaked-MDT0003_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1468994840.996298:0:4562:0:(import.c:1336:ptlrpc_connect_interpret()) recovery of soaked-MDT0003_UUID on 192.168.1.111@o2ib10 failed (-114) ... 00000100:00080000:17.0:1468994866.072408:0:4595:0:(recover.c:62:ptlrpc_initiate_recovery()) soaked-MDT0003_UUID: starting recovery 00000100:00080000:17.0:1468994866.072410:0:4595:0:(import.c:679:ptlrpc_connect_import()) ffff8803fc39b800 soaked-MDT0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:17.0:1468994866.072415:0:4595:0:(import.c:523:import_select_connection()) soaked-MDT0003-osp-MDT0002: connect to NID 192.168.1.111@o2ib10 last attempt 4295034989 00000100:00080000:17.0:1468994866.072418:0:4595:0:(import.c:523:import_select_connection()) soaked-MDT0003-osp-MDT0002: connect to NID 0@lo last attempt 0 00000100:00080000:17.0:1468994866.072428:0:4595:0:(import.c:593:import_select_connection()) soaked-MDT0003-osp-MDT0002: Connection changing to soaked-MDT0003 (at 0@lo) 00000100:00080000:17.0:1468994866.072431:0:4595:0:(import.c:601:import_select_connection()) soaked-MDT0003-osp-MDT0002: import ffff8803fc39b800 using connection 192.168.1.110@o2ib10/0@lo 00000100:00080000:17.0:1468994866.072578:0:4595:0:(pinger.c:221:ptlrpc_pinger_process_import()) soaked-MDT0000-lwp-MDT0002_UUID-&gt;soaked-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 0 suppress 0 00000100:00080000:17.0:1468994866.072584:0:4595:0:(recover.c:62:ptlrpc_initiate_recovery()) soaked-MDT0000_UUID: starting recovery 00000100:00080000:17.0:1468994866.072586:0:4595:0:(import.c:679:ptlrpc_connect_import()) ffff880832815000 soaked-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:17.0:1468994866.072592:0:4595:0:(import.c:523:import_select_connection()) soaked-MDT0000-lwp-MDT0002: connect to NID 192.168.1.108@o2ib10 last attempt 4295035090 00000100:00080000:17.0:1468994866.072597:0:4595:0:(import.c:523:import_select_connection()) soaked-MDT0000-lwp-MDT0002: connect to NID 192.168.1.109@o2ib10 last attempt 0 00000100:00080000:17.0:1468994866.072602:0:4595:0:(import.c:593:import_select_connection()) soaked-MDT0000-lwp-MDT0002: Connection changing to soaked-MDT0000 (at 192.168.1.109@o2ib10) 00000100:00080000:17.0:1468994866.072607:0:4595:0:(import.c:601:import_select_connection()) soaked-MDT0000-lwp-MDT0002: import ffff880832815000 using connection 192.168.1.109@o2ib10/192.168.1.109@o2ib10 00010000:02020000:1.0:1468994866.072759:0:6154:0:(ldlm_lib.c:962:target_handle_connect()) 137-5: soaked-MDT0003_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. 00000100:00080000:17.0:1468994866.089880:0:4562:0:(import.c:1290:ptlrpc_connect_interpret()) ffff8803fc39b800 soaked-MDT0003_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:17.0:1468994866.089885:0:4562:0:(import.c:1336:ptlrpc_connect_interpret()) recovery of soaked-MDT0003_UUID on 192.168.1.110@o2ib10 failed (-19) ... 00000100:00080000:17.0:1468994891.072459:0:4595:0:(recover.c:62:ptlrpc_initiate_recovery()) soaked-MDT0003_UUID: starting recovery 00000100:00080000:17.0:1468994891.072461:0:4595:0:(import.c:679:ptlrpc_connect_import()) ffff8803fc39b800 soaked-MDT0003_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:17.0:1468994891.072466:0:4595:0:(import.c:523:import_select_connection()) soaked-MDT0003-osp-MDT0002: connect to NID 192.168.1.111@o2ib10 last attempt 4295034989 00000100:00080000:17.0:1468994891.072470:0:4595:0:(import.c:523:import_select_connection()) soaked-MDT0003-osp-MDT0002: connect to NID 0@lo last attempt 4295060090 00000100:00080000:17.0:1468994891.072475:0:4595:0:(import.c:567:import_select_connection()) soaked-MDT0003-osp-MDT0002: tried all connections, increasing latency to 6s 00000100:00080000:17.0:1468994891.072488:0:4595:0:(import.c:593:import_select_connection()) soaked-MDT0003-osp-MDT0002: Connection changing to soaked-MDT0003 (at 192.168.1.111@o2ib10) 00000100:00080000:17.0:1468994891.072492:0:4595:0:(import.c:601:import_select_connection()) soaked-MDT0003-osp-MDT0002: import ffff8803fc39b800 using connection 192.168.1.111@o2ib10/192.168.1.111@o2ib10 00000100:02020000:2.0:1468994891.072768:0:4562:0:(client.c:1266:ptlrpc_check_status()) 11-0: soaked-MDT0000-osp-MDT0002: operation mds_connect to node 192.168.1.108@o2ib10 failed: rc = -114 00000100:00080000:2.0:1468994891.089071:0:4562:0:(import.c:1290:ptlrpc_connect_interpret()) ffff8804256c8000 soaked-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1468994891.089075:0:4562:0:(import.c:1336:ptlrpc_connect_interpret()) recovery of soaked-MDT0000_UUID on 192.168.1.108@o2ib10 failed (-114) 00000100:02020000:2.0:1468994891.089104:0:4562:0:(client.c:1266:ptlrpc_check_status()) 11-0: soaked-MDT0003-osp-MDT0002: operation mds_connect to node 192.168.1.111@o2ib10 failed: rc = -114 00000100:00080000:2.0:1468994891.089108:0:4562:0:(import.c:1290:ptlrpc_connect_interpret()) ffff8803fc39b800 soaked-MDT0003_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1468994891.089110:0:4562:0:(import.c:1336:ptlrpc_connect_interpret()) recovery of soaked-MDT0003_UUID on 192.168.1.111@o2ib10 failed (-114) ... </pre> </div></div> <p>As shown, at the beginning (timestamps: 1468994840), the MDT0002 on the lola-10 selected lola-11 (192.168.1.111@o2ib10) for the soaked-MDT0003-osp-MDT0002 connection. Unfortunately, the lola-11 returned -114 to the lola-10. That means another instance of soaked-MDT0003-osp-MDT0002 has already connected (or in connecting) to the MDT0003 on the lola-11. Because lola-10 and lola-11 were HA pairs, it was quite possible that the lola-11 took lola-10's role to server as MDT0002 during the lola-10 failover. Since MDT0002 had been mounted on the lola-11, the connection between MDT0002 and MDT0003 had been established at that time, then it is normal that the MDT0002 on the lola-10 got -114 failure.</p> <p>Because lola-10 and lola-11 were HA pairs, when hit -114 failure, the MDT0002 on the lola-10 selected another peer for the soaked-MDT0003-osp-MDT0002 connection, it is lola-10 (192.168.1.111@o2ib10) itself. But it is obviously that MDT0003 was running on the lola-11, so the MDT0002 on the lola-10 got -19 failure.</p> <p>Then, the MDT0002 has to find other peer for the soaked-MDT0003-osp-MDT0002 connection. There were only two choices, either lola-11 or lola-10, so it selected lola-11, the same as the first choice, but still hit -114 failure. And then try to connect lola-10 again, failed as -19 again.</p> <p>So the MDT0002 on the lola-10 tried to connect to MDT0003 repeatedly for a long time (from the timestamps 1468994840 to 1469003821), 8981 seconds passed, but still cannot connect to the MDT0003. Such connection failure blocked the cross-MDT recovery. From the given logs, I cannot find when the soaked-MDT0003-osp-MDT0002 connection was established successfully. Because we have no logs on the lola-11, we do not know what happened for the lola-11. But it is suspected that it is related with another MDT0002 instance on the lola-11 because of the HA pairs configuration. Since the HA is controlled by some third-part HA software, such as heartbeat or pack-maker, can we check whether it works well or not under the failover case? Can we switch the resource by force under such case?</p> <p>Anyway, offer me more logs on the lola-11 will be helpful. Thanks!</p> <p>The HA configuration reads as follows:</p> <ul> <li>lola-8, lola-9 - failover cluster <ul> <li>lola-8: MDT0000 (primary resource)</li> <li>lola-9: MDT0001 (primary resource)</li> </ul> </li> <li>lola-10,lola-11 - failover cluster <ul> <li>lola-10: MDT0002 (primary resource)</li> <li>lola-11: MDT0003 (primary resource)</li> </ul> </li> </ul> <p>MDTs are configured for failover during format. HA is controlled by the soak test framework. For the scenario above the following process flow were executed</p> <ul> <li>powercyle of (primary) MDS</li> <li>mount of MDT on secondary MDS</li> <li>Don't wait for recovery to complete on secondary; switch back to primary node immediately after MDT mount completes</li> <li>mount MDT on primary MDS again</li> </ul> <p>The console and message files for lola-11 aren't available anymore. Because of capacity reason all files older than 6 weeks are deleted on the cluster's share FS.</p> <p>For build '20160916' (see <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160916" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160916</a>) this effect didn't occurred <br/> till now although the effect might be eclipsed by <a href="https://jira.whamcloud.com/browse/LU-8580" title="general protection fault: osd_xattr_get+0x32c/0x5b0 [osd_ldiskfs]" class="issue-link" data-issue-key="LU-8580"><del>LU-8580</del></a>.</p> <blockquote> <p>Don't wait for recovery to complete on secondary; switch back to primary node immediately after MDT mount completes</p></blockquote> <p>How did you do that? umount the MDT just after the MDT mount complete? With or without "-f"? It seems that from the lola-11 view, the connection from the MDT0002 was not cleanup, as to refused the new connection from the MDT0002 on the lola-10. One possible solution is allow the new connection to replace the old. I wondered why the patch <a href="http://review.whamcloud.com/#/c/18800/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/18800/</a> did not take effect (or some side-effect?) under such case.</p> <p>Di believes this issue is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-8250" title="MDT recovery stalled on secondary node" class="issue-link" data-issue-key="LU-8250"><del>LU-8250</del></a></p> Duplicate LU-8250 Development Rank 1|hzyi53: Rank (Obsolete) 9223372036854775807 Severity