https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7938 Lustre <p>Description</p> <p>Error happens during soak testing of build '20160324' (see <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160324" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160324</a>). DNE is enabled. MDTs had been formatted with ldiskfs, OSTs using zfs. MDS and OSS nodes are configured in HA active-active failover configuration. MDS nodes operated wiht <b>1</b> MDT per MDS, while OSSes running 4 OST per node.<br/> Nodes lola-4 and lola-5 form a HA cluster.</p> <p>Event history</p> <ul> <li>2016-03-29 07:48:04,307:fsmgmt.fsmgmt:INFO triggering fault oss_failover of node <tt>lola-4</tt><br/> powercyle node</li> <li>2016-03-29 07:52:27,876:fsmgmt.fsmgmt:INFO lola-4 is up</li> <li>2016-03-29 07:53:09,557:fsmgmt.fsmgmt:INFO zpool import and mount of <tt>lola-4</tt>'s OSTs complete</li> <li>Recovery don't complete after recovery_time is zero</li> <li>2016-03-29 08:03 (approximately) Aborted recovery manually (<tt>lctl --device ... abort_recovery</tt>)</li> </ul> <p>Attached files:<br/> messages, console and debug log before (lustre-log-20160329-0759-recovery-stalled) and after recovery was aborted (lustre-log-20160329-0803-recovery-aborted)</p> lola<br/> build: 2.8 GA + patches LU-7938

Recovery on secondary OSS node stalled

Bug Blocker Resolved Not a Bug nasf Frank Heckes soak Tue, 29 Mar 2016 15:53:41 +0000 Wed, 27 Jul 2016 13:12:38 +0000 Wed, 27 Jul 2016 13:12:38 +0000 Lustre 2.8.0 Lustre 2.9.0 Lustre 2.9.0 0 6 <p>lustre-log-20160329-0759-recovery-stalled is referenced, but not really provided?</p> <p>Sorry, it took many attempts to upload the file.</p> <p>The error can be reproduced easily. Please let me know if debug logs with mask different than default are needed.</p> <p>The error occurred also in soak test for build '20160413' (see <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160413" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160413</a>). Test set-up is the same as the one described above, besides<br/> that the Lustre FS has been extended by 2 OSS nodes hosting 4 OSTs each.</p> <p>The recovery is stalled, but can be manually aborted, too. <br/> Sequence of events:</p> <ul> <li>2016-04-15 03:57:14,905:fsmgmt.fsmgmt:INFO triggering fault oss_failover (<tt>lola-5</tt> OSTs ----&gt; <tt>lola-4</tt>)</li> <li>2016-04-15 04:02:13,325:fsmgmt.fsmgmt:INFO Wait for recovery to complete (--&gt; all OSTs mounted on secondary node <tt>lola-4</tt>)</li> <li>2016-04-15 04:20 created debug log</li> <li>2016-04-15 04:28:29 Recovery aborted for ost- {3,9,15,21} <p> (decimal index)</p></li> <li>2016-04-15 04:30 created debug log<br/> Attached files:<br/> console, messages files of lola-4 and lola-4-lustre-log-20160415-0420 (before abort), lola-4-lustre-log-20160415-0430 (after abort)</li> </ul> <p>The error is occurring again on the tip of master plus the two patches.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># cat /proc/fs/lustre/mdt/soaked-MDT0002/recovery_status status: RECOVERING recovery_start: 1465232959 time_remaining: 0 connected_clients: 20/20 req_replay_clients: 0 lock_repay_clients: 1 completed_clients: 19 evicted_clients: 0 replayed_requests: 0 queued_requests: 0 next_transno: 442381632838 </pre> </div></div> <p>Log errors</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Lustre: Skipped 299 previous similar messages Lustre: soaked-MDT0002: Recovery already passed deadline 137:09, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. </pre> </div></div> <p>Does not appear possible to abort_recovery, results in timeouts.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">INFO: task lctl:41183 blocked <span class="code-keyword">for</span> more than 120 seconds. Tainted: P -- ------------ 2.6.32-573.26.1.el6_lustre.x86_64 #1 <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. lctl D 0000000000000019 0 41183 41083 0x00000080 ffff8806bdda7b68 0000000000000086 0000000000000000 ffff88082fe86ab0 ffff88082fe86ae8 ffff880439371150 0000ece58069441d 0000000000000000 ffff8806bdd8eae8 000000010f83d3bd ffff8806bdd8f068 ffff8806bdda7fd8 Call Trace: [&lt;ffffffff8153aa35&gt;] schedule_timeout+0x215/0x2e0 [&lt;ffffffff8153a693&gt;] wait_for_common+0x123/0x180 [&lt;ffffffff81067650&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffff8153a7cd&gt;] wait_for_completion+0x1d/0x20 [&lt;ffffffffa0992e80&gt;] target_stop_recovery_thread+0x50/0xe0 [ptlrpc] [&lt;ffffffffa112ac02&gt;] mdt_iocontrol+0x6b2/0x920 [mdt] [&lt;ffffffff812337cf&gt;] ? security_inode_permission+0x1f/0x30 [&lt;ffffffffa0783edc&gt;] class_handle_ioctl+0x15fc/0x20d0 [obdclass] [&lt;ffffffff811a5998&gt;] ? do_filp_open+0x798/0xd20 [&lt;ffffffffa07682ab&gt;] obd_class_ioctl+0x4b/0x190 [obdclass] [&lt;ffffffff811a7f82&gt;] vfs_ioctl+0x22/0xa0 [&lt;ffffffff811a8124&gt;] do_vfs_ioctl+0x84/0x580 [&lt;ffffffff811a03f6&gt;] ? final_putname+0x26/0x50 [&lt;ffffffff811a86a1&gt;] sys_ioctl+0x81/0xa0 [&lt;ffffffff810e905e&gt;] ? __audit_syscall_exit+0x25e/0x290 [&lt;ffffffff8100b0d2&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Comments from Cliff above seems to be related to <a href="https://jira.whamcloud.com/browse/LU-8089" title="MGT/MDT mount fails on secondary HA node" class="issue-link" data-issue-key="LU-8089"><del>LU-8089</del></a>. </p> <p>The error for for OST reported in the description above, also occurred for build <a href="https://build.hpdd.intel.com/job/lustre-master/3406" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-master/3406</a> (see <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160713" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160713</a>)<br/> and is reproducible for any failover initiated.<br/> Test environment configuration is<br/> 4 MDSes with 1 MDT each formatted with <em>ldiskfs</em> in active-active failover confguration <br/> 6 OSSes with 4 OSTs each formatted with <em>zfs</em> in active-active failover configuration<br/> 16 clients<br/> The stalled recovery process can be interrupted (aborted) at any time.</p> <p>I attached new message, console and debug log files (take during recovery, after recovery abort) of secondary node <tt>lola-5</tt>. <br/> Debug files of all other nodes taken at the same time, are available and can be uploaded on demand.</p> <p>Sequence of events:</p> <ul> <li>2016-07-22 04:21:54,665:fsmgmt.fsmgmt:INFO triggering fault oss_failover lola-4</li> <li>2016-07-22 04:26:37,971:fsmgmt.fsmgmt:INFO lola-4 is up</li> <li>2016-07-22 04:27:01,278:fsmgmt.fsmgmt:INFO ... soaked-OST0008 mounted successfully on lola-5 (NOTE: decimal id for OST)</li> <li>2016-07-22 04:27:12,348:fsmgmt.fsmgmt:INFO ... soaked-OST0020 mounted successfully on lola-5</li> <li>2016-07-22 04:27:23,515:fsmgmt.fsmgmt:INFO ... soaked-OST0002 mounted successfully on lola-5</li> <li>2016-07-22 04:27:36,047:fsmgmt.fsmgmt:INFO ... soaked-OST0014 mounted successfully on lola-5</li> <li>2016-07-22 04:27:36 Recovery started</li> <li>2016-07-22 04:37 Recovery aborted</li> </ul> <p>Status of recovery on OSS node <tt>lola-5</tt> and client status were:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@lola-5 ~]# lctl get_param *.*.recovery_status obdfilter.soaked-OST0002.recovery_status= status: RECOVERING recovery_start: 0 time_remaining: 0 connected_clients: 0/23 req_replay_clients: 0 lock_repay_clients: 0 completed_clients: 0 evicted_clients: 0 replayed_requests: 0 queued_requests: 0 next_transno: 68719725397 obdfilter.soaked-OST0003.recovery_status= status: COMPLETE recovery_start: 1468949759 recovery_duration: 30 completed_clients: 23/23 replayed_requests: 0 last_transno: 34368384861 VBR: DISABLED IR: ENABLED obdfilter.soaked-OST0008.recovery_status= status: RECOVERING recovery_start: 0 time_remaining: 0 connected_clients: 0/23 req_replay_clients: 0 lock_repay_clients: 0 completed_clients: 0 evicted_clients: 0 replayed_requests: 0 queued_requests: 0 next_transno: 68719723367 obdfilter.soaked-OST0009.recovery_status= status: COMPLETE recovery_start: 1468949751 recovery_duration: 38 completed_clients: 23/23 replayed_requests: 0 last_transno: 34368183131 VBR: DISABLED IR: ENABLED obdfilter.soaked-OST000e.recovery_status= status: RECOVERING recovery_start: 0 time_remaining: 0 connected_clients: 0/23 req_replay_clients: 0 lock_repay_clients: 0 completed_clients: 0 evicted_clients: 0 replayed_requests: 0 queued_requests: 0 next_transno: 68719751551 obdfilter.soaked-OST000f.recovery_status= status: COMPLETE recovery_start: 1468949763 recovery_duration: 26 completed_clients: 23/23 replayed_requests: 0 last_transno: 34368269722 VBR: DISABLED IR: ENABLED obdfilter.soaked-OST0014.recovery_status= status: RECOVERING recovery_start: 0 time_remaining: 0 connected_clients: 0/23 req_replay_clients: 0 lock_repay_clients: 0 completed_clients: 0 evicted_clients: 0 replayed_requests: 0 queued_requests: 0 next_transno: 68719756594 obdfilter.soaked-OST0015.recovery_status= status: COMPLETE recovery_start: 1468949751 recovery_duration: 38 completed_clients: 23/23 replayed_requests: 0 last_transno: 34368229218 VBR: DISABLED IR: ENABLED </pre> </div></div> <p>The recovery stalled in a previous session running during night time over several hours without completion. That's why I aborted the recovery after ten minutes; I wasn't sure debug logs would contain relevant information, but confident that the recovery would never end.</p> <p>According to the recovery status, no clients connected to the new mounted OSTs on the lola-5, and there are no client connection records in the lola-5 debug logs (I am sure that it is NOT because of logs overwritten). So it seems that the clients were not aware of the OST load switching from lola-4 to lola-5, as to no connection to the lola-5, then no recovery. We need the full debug logs on the MGS and the client (one active client is enough). Thanks!</p> <p>Resources of <tt>lola-2</tt> (<tt>OST0012 OST0000 OST0006 OST000c</tt>) failed over to <tt>lola-3</tt></p> <ul> <li>2016-07-25 08:49:12,572:fsmgmt.fsmgmt:INFO Wait for recovery to complete</li> <li>2016-07-25 08:55 created debug logs</li> <li>2016-07-25 08:57:51 aborted recovery</li> <li>2016-07-25 08:59 created debug logs</li> </ul> <p>Upload of debug logs has been started, but it might take some time.</p> <p>debug files have been uploaded. Roles:<br/> lola-8: MGS<br/> lola-3: secondary OSS<br/> lola-30: Lustre client</p> <p>Thanks Frank for the detailed logs. According to the client side log, it always tried to connect lola2 for the failed OSTs (OST0012 OST0000 OST0006 OST000c):</p> <blockquote> <p>00000100:00000001:30.0:1469462043.991030:0:125723:0:(import.c:508:import_select_connection()) Process entered<br/> 00000100:00080000:30.0:1469462043.991033:0:125723:0:(import.c:523:import_select_connection()) soaked-OST0000-osc-ffff8810717d2000: connect to NID 192.168.1.102@o2ib10 last attempt 5189761512<br/> 00000100:00080000:30.0:1469462043.991036:0:125723:0:(import.c:567:import_select_connection()) soaked-OST0000-osc-ffff8810717d2000: tried all connections, increasing latency to 50s<br/> 00000100:00080000:30.0:1469462043.991099:0:125723:0:(import.c:601:import_select_connection()) soaked-OST0000-osc-ffff8810717d2000: import ffff880ff6706000 using connection 192.168.1.102@o2ib10/192.168.1.102@o2ib10<br/> 00000100:00000001:30.0:1469462043.991104:0:125723:0:(import.c:605:import_select_connection()) Process leaving (rc=0 : 0 : 0)<br/> ...</p></blockquote> <p>That means there is only one candidate can be selected to connect the OST0000, but according to your description, there should be another candidate 192.168.1.103@o2ib for connection. So I am wondering how did you config the lola2 and lola3 as the failover pairs?</p> <p>Indeed it's a configuration error. Looks like NID of the secondary node (<tt>192.168.1.103@o2ib</tt>) weren't written during FS format. <br/> I set them explicitly again using <tt>zfs set lustre:failover.node</tt>, and check whether this resolves the issue.</p> <p>I set the parameter '<tt>lustre:failover.node</tt>' for on all OSTs to the appropriate values for the failover nodes (see attached file ost-failover-setttings-20160726_0702).<br/> Actually this didn't solve the problem. Recovery is stalled still on failover node.</p> <p>I also reformatted the soak FS. The parameter setting will be filled (e.g.) with the following content:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>192.168.1.102: mkfs_cmd = zfs create -o canmount=off -o xattr=sa soaked-ost18/ost18 192.168.1.102: Writing soaked-ost18/ost18 properties 192.168.1.102: lustre:version=1 192.168.1.102: lustre:flags=4194 192.168.1.102: lustre:index=18 192.168.1.102: lustre:fsname=soaked 192.168.1.102: lustre:svname=soaked:OST0012 192.168.1.102: lustre:mgsnode=192.168.1.108@o2ib10:192.168.1.109@o2ib10 192.168.1.102: lustre:failover.node=192.168.1.102@o2ib10:192.168.1.103@o2ib10 </pre> </div></div> <p>Additionally I used option '<tt>--force-nohostid</tt>' for mkfs.lustre - command. I think the patch enabling hostid haven't landed, yet and spl fails during reformat if /etc/hostid isn't available.</p> <p>reformat is onging. I'm going to start soak again, once it has completed.</p> <p>failover.node is specified for mkfs.lustre command via option <tt>--servicenode</tt></p> <p>Failover works after reformat of Lustre FS. Ticket can be closed.</p> Related Development Rank 1|hzy62v: Rank (Obsolete) 9223372036854775807 Severity