https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1166 Lustre <p>while testing we hit a situation when recovery never finished and recovery timer exceed a hard recovery timer.</p> <p>00010000:00080000:20.0:1330709620.108824:0:19858:0:(ldlm_lib.c:1361:reset_recovery_timer()) snxs4-MDT0000: recovery timer will expire in 70 seconds<br/> 00010000:00000400:20.0:1330709690.108951:0:19858:0:(ldlm_lib.c:1570:target_recovery_overseer()) recovery is timed out, evict stale exports<br/> 00010000:00080000:20.0:1330709690.120847:0:19858:0:(ldlm_lib.c:1361:reset_recovery_timer()) snxs4-MDT0000: recovery timer will expire in 70 seconds<br/> 00010000:00000400:1.0:1330709760.120868:0:19858:0:(ldlm_lib.c:1570:target_recovery_overseer()) recovery is timed out, evict stale exports<br/> 00010000:00080000:1.0:1330709760.132776:0:19858:0:(ldlm_lib.c:1361:reset_recovery_timer()) snxs4-MDT0000: recovery timer will expire in 70 seconds<br/> 00010000:00000400:1.0:1330709830.131858:0:19858:0:(ldlm_lib.c:1570:target_recovery_overseer()) recovery is timed out, evict stale exports<br/> 00010000:00080000:1.0:1330709830.143745:0:19858:0:(ldlm_lib.c:1361:reset_recovery_timer()) snxs4-MDT0000: recovery timer will expire in 70 seconds<br/> 00010000:00000400:1.0:1330709900.142871:0:19858:0:(ldlm_lib.c:1570:target_recovery_overseer()) recovery is timed out, evict stale exports<br/> 00010000:00080000:1.0:1330709900.154725:0:19858:0:(ldlm_lib.c:1361:reset_recovery_timer()) snxs4-MDT0000: recovery timer will expire in 40 seconds<br/> 00010000:00000400:1.0:1330709940.153865:0:19858:0:(ldlm_lib.c:1570:target_recovery_overseer()) recovery is timed out, evict stale exports<br/> 00010000:00080000:1.0:1330709940.165727:0:19858:0:(ldlm_lib.c:1361:reset_recovery_timer()) snxs4-MDT0000: recovery timer will expire in 0 seconds<br/> 00010000:00000400:13.0:1330709940.165827:0:19858:0:(ldlm_lib.c:1570:target_recovery_overseer()) recovery is timed out, evict stale exports<br/> 00010000:00080000:13.0:1330709940.177697:0:19858:0:(ldlm_lib.c:1361:reset_recovery_timer()) snxs4-MDT0000: recovery timer will expire in 0 seconds<br/> 00010000:00000400:1.0:1330709940.178088:0:19858:0:(ldlm_lib.c:1570:target_recovery_overseer()) recovery is timed out, evict stale exports<br/> 00010000:00080000:1.0:1330709940.189941:0:19858:0:(ldlm_lib.c:1361:reset_recovery_timer()) snxs4-MDT0000: recovery timer will expire in 0 seconds<br/> 00010000:00000400:13.0:1330709940.190014:0:19858:0:(ldlm_lib.c:1570:target_recovery_overseer()) recovery is timed out, evict stale exports<br/> 00010000:00080000:13.0:1330709940.201864:0:19858:0:(ldlm_lib.c:1361:reset_recovery_timer()) snxs4-MDT0000: recovery timer will expire in 0 seconds<br/> 00010000:00000400:1.0:1330709940.202082:0:19858:0:(ldlm_lib.c:1570:target_recovery_overseer()) recovery is timed out, evict stale exports<br/> 00010000:00080000:1.0:1330709940.213933:0:19858:0:(ldlm_lib.c:1361:reset_recovery_timer()) snxs4-MDT0000: recovery timer will expire in 0 seconds<br/> 00010000:00000400:1.0:1330709940.214821:0:19858:0:(ldlm_lib.c:1570:target_recovery_overseer()) recovery is timed out, evict stale exports</p> <p>...</p> <p>after analyzing a logs that hand looks addressed to waiting in target_recovery_overseer function with check_for_clients() argument.<br/> that hung looks a result of using a <br/>        if (obd-&gt;obd_no_conn == 0 &amp;&amp;<br/>            obd-&gt;obd_connected_clients + obd-&gt;obd_stale_clients ==<br/>            obd-&gt;obd_max_recoverable_clients)</p> <p>in case of MDT<br/> obd_no_conn set by post recovery if at least one ost connected and config llog processed.<br/> but mdt_postrecov can't called because recovery isn't finished.</p> <p>second issue in that area - reset_recovery_timer function.<br/> if we have a race and reset_recovery_timer function called in same time as recovery should be finished, but before timer a hit, we set a '0' (and negative number at next turn) as next timer time.</p> <p>00010000:00080000:1.0:1330709942.007696:0:19858:0:(ldlm_lib.c:1361:reset_recovery_timer()) snxs4-MDT0000: recovery timer will expire in 4294967294 seconds<br/> 00010000:00080000:1.1:1330709942.007794:0:9:0:(ldlm_lib.c:1887:target_recovery_expired()) snxs4-MDT0000: recovery timed out; 36 clients are still in recovery after 902s (49 clients connected)<br/> 00010000:00000400:13.0:1330709942.007802:0:19858:0:(ldlm_lib.c:1570:target_recovery_overseer()) recovery is timed out, evict stale exports</p> 2.1.0 + with minimal back porting from 2.2 LU-1166

recovery never finished

Bug Minor Resolved Fixed WC Triage Alexey Lyashkov Fri, 2 Mar 2012 19:26:38 +0000 Fri, 18 Nov 2016 21:36:48 +0000 Thu, 29 Mar 2012 08:49:53 +0000 Lustre 2.1.0 Lustre 2.3.0 Lustre 2.1.2 0 9 <p>obd_no_conn is set in mdt_obd_notify -&gt; mdt_allow_cli, morevoer that would be just useless to set it after recovery what means no any client can participate in recovery ever. I don't think this is related to the issue.</p> <p>About reset_recovery_timer(), can you show in details how it can become negative?</p> <p>Also note, that there were changes after 2.1.0 related to recovery timer changes:</p> <p>2012-02-16 Jinshan Xiong <a href="https://jira.whamcloud.com/browse/LU-889" title="rework extent_recovery_timer()" class="issue-link" data-issue-key="LU-889"><del>LU-889</del></a> recovery: rework extend_recovery_timer()<br/> 2011-11-03 Jinshan Xiong ORNL-28 recovery: rework extend_recovery_timer()<br/> 2011-10-21 Jinshan Xiong ORNL-28: Set recovery timeout correctly</p> <p>unfortunately first one broke recovery timer and only with last one it is restored. Please check these patches weren't ported to your 2.1.0 from 2.2 separately. And maybe it makes sense to port them all together, because I see that now this function differs from 2.1.0, probably issue is fixed already.</p> <p>Hm.. you are right. <br/> obd_no_conn == 0.<br/> I found a crash dump from a similar situation.<br/> $7 = {<br/> obd_type = 0xffff88043e9d4740, <br/> obd_magic = 2874988271, <br/> obd_name = "snxs4-MDT0000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00<br/> 0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\<br/> 000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000", <br/> obd_uuid = </p> { uuid = "snxs4-MDT0000_UUID\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000" } <p>, <br/> obd_lu_dev = 0xffff880441754000, <br/> obd_minor = 2, <br/> obd_attached = 1, <br/> obd_set_up = 1, <br/> obd_recovering = 1, <br/> obd_abort_recovery = 1, <br/> obd_version_recov = 1, <br/> obd_replayable = 1, <br/> obd_no_transno = 0, <br/> obd_no_recov = 0, <br/> obd_stopping = 0, <br/> obd_starting = 1, <br/> obd_force = 0, <br/> obd_fail = 0, <br/> obd_async_recov = 0, <br/> obd_no_conn = 0, <br/> obd_inactive = 0, <br/> obd_process_conf = 0, <br/> obd_recovery_expired = 0<br/> ...<br/> obd_refcount = </p> { counter = 4171 } <p>, <br/> ...</p> <p> obd_unlinked_exports = </p> { next = 0xffff88042b7fd4a0, prev = 0xffff88042b7fd4a0 } <p>, <br/> obd_delayed_exports = </p> { next = 0xffff88045033c1a8, prev = 0xffff88045033c1a8 } <p>, <br/> obd_num_exports = 2072,<br/> ..<br/> obd_exports_timed = </p> { next = 0xffff88042b98e4d0, prev = 0xffff88042b6f68d0 } <p>, <br/> obd_eviction_timer = 0, <br/> obd_max_recoverable_clients = 4471, <br/> obd_connected_clients = 2095, <br/> obd_stale_clients = 2400, <br/> obd_delayed_clients = 0,<br/> ...</p> <p> obd_recovery_start = 1330601299, <br/> obd_recovery_end = 1330602199, <br/> obd_recovery_time_hard = 900, <br/> obd_recovery_timeout = 900, <br/> obd_recovery_data = {<br/> trd_recovery_handler = 0xffffffffa09a18b0 &lt;mdt_recovery_handle&gt;, <br/> trd_processing_task = 15692, <br/> trd_starting = {<br/> done = 0, <br/> wait = {<br/> lock = {<br/> raw_lock = </p> { slock = 196611 } <p> }, <br/> task_list = </p> { next = 0xffff88045033c510, prev = 0xffff88045033c510 } <p> }<br/> }, <br/> trd_finishing = {<br/> done = 0, <br/> wait = {<br/> lock = {<br/> raw_lock = </p> { slock = 65537 } <p> }, <br/> task_list = </p> { next = 0xffff88042b94dbc0, prev = 0xffff88042b94dbc0 } <p> }<br/> }<br/> }, <br/> obd_replayed_locks = 0, <br/> obd_req_replay_clients = </p> { counter = 24 }, <br/> obd_lock_replay_clients = { counter = 24 } <p>, </p> <p>After discussion with Alex it is clean that endless recovery cycle caused by wrong number of stale clients and connected, so the check below never true:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">check_for_clients(struct obd_device *obd) { unsigned <span class="code-object">int</span> clnts = cfs_atomic_read(&amp;obd-&gt;obd_connected_clients); <span class="code-keyword">if</span> (obd-&gt;obd_abort_recovery || obd-&gt;obd_recovery_expired) <span class="code-keyword">return</span> 1; LASSERT(clnts &lt;= obd-&gt;obd_max_recoverable_clients); ---&gt; <span class="code-keyword">return</span> (clnts + obd-&gt;obd_stale_clients == obd-&gt;obd_max_recoverable_clients); } </pre> </div></div> <p>I suspect this is caused by class_disconnect_stale_exports() which moves stale clients from obd_exports list but they are still in hash, so connected client can find export from stale list and connect to it. Therefore that export will be 'connected' and 'stale' at the same time. Solution could be removal export from hash along with removal from obd_exports list, but more close investigation is needed to check that there is no other races.</p> <p>per additional discussion with Mike, we have a verdict - that is race between class_disconnect and target_handle_connect.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">int</span> class_disconnect(struct obd_export *export) { <span class="code-object">int</span> already_disconnected; ENTRY; <span class="code-keyword">if</span> (export == NULL) { fixme(); CDEBUG(D_IOCTL, <span class="code-quote">"attempting to free NULL export %p\n"</span>, export); RETURN(-EINVAL); } cfs_spin_lock(&amp;export-&gt;exp_lock); already_disconnected = export-&gt;exp_disconnected; export-&gt;exp_disconnected = 1; cfs_spin_unlock(&amp;export-&gt;exp_lock); /* class_cleanup(), abort_recovery(), and class_fail_export() * all end up in here, and <span class="code-keyword">if</span> any of them race we shouldn't * call extra class_export_puts(). */ <span class="code-keyword">if</span> (already_disconnected) { LASSERT(cfs_hlist_unhashed(&amp;export-&gt;exp_nid_hash)); GOTO(no_disconn, already_disconnected); } CDEBUG(D_IOCTL, <span class="code-quote">"disconnect: cookie "</span>LPX64<span class="code-quote">"\n"</span>, export-&gt;exp_handle.h_cookie); <span class="code-keyword">if</span> (!cfs_hlist_unhashed(&amp;export-&gt;exp_nid_hash)) cfs_hash_del(export-&gt;exp_obd-&gt;obd_nid_hash, &amp;export-&gt;exp_connection-&gt;c_peer.nid, &amp;export-&gt;exp_nid_hash); class_export_recovery_cleanup(export); &lt;&lt;&lt; wait where &gt;&gt;&gt; class_unlink_export(export); no_disconn: class_export_put(export); RETURN(0); } </pre> </div></div> <p>if target_handle_connect will raced with class_export_recovery_cleanup in waiting on <br/> obd_recovery_task_lock - we will count a one export twice.<br/> we will leak a obd_connected_clients, and other recovery counters...</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> obd_req_replay_clients = { counter = 24 }, obd_lock_replay_clients = { counter = 24 }, </pre> </div></div> <p>remote: New Changes:<br/> remote: <a href="http://review.whamcloud.com/2255" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2255</a></p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = FAILURE<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/obdclass/genops.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/obdclass/genops.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=ofa/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,ofa #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = FAILURE<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/obdclass/genops.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/obdclass/genops.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = FAILURE<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/obdclass/genops.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=ofa/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,ofa #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = FAILURE<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/obdclass/genops.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=ofa/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,ofa #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = FAILURE<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/obdclass/genops.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/obdclass/genops.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/obdclass/genops.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/obdclass/genops.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/obdclass/genops.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=ofa/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,ofa #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = FAILURE<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/obdclass/genops.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/obdclass/genops.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/obdclass/genops.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = FAILURE<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/obdclass/genops.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/obdclass/genops.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/obdclass/genops.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/531/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #531</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = FAILURE<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/obdclass/genops.c</li> </ul> <p>Landed for 2.3</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,client,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/obdclass/genops.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,client,el6,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/obdclass/genops.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,server,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/obdclass/genops.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,server,el6,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/obdclass/genops.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,client,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/obdclass/genops.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,server,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> <li>lustre/obdclass/genops.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,client,el6,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=737da0331e8407a704cd11c04f18c2cd3d437800" class="external-link" target="_blank" rel="nofollow noopener">737da0331e8407a704cd11c04f18c2cd3d437800</a><br/> Files : </p> <ul> <li>lustre/obdclass/genops.c</li> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p><a href="http://review.whamcloud.com/#change,2665" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2665</a><br/> back port to b2_1</p> <p><a href="http://review.whamcloud.com/#change,2874" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2874</a><br/> more changes missing from previous back port</p> <p>Hello Bob,</p> <p>I don't fully understand the portions of code impacted by this ticket, but could you explain why the additional change <a href="http://review.whamcloud.com/#change,2874" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2874</a> you submitted in b2_1 is not present in the master release.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/lustre/ldlm/ldlm_lib.c b/lustre/ldlm/ldlm_lib.c index a8433fc..0798ba7 100644 --- a/lustre/ldlm/ldlm_lib.c +++ b/lustre/ldlm/ldlm_lib.c @@ -1069,7 +1069,8 @@ dont_check_exports: class_disconnect-&gt;class_export_recovery_cleanup() race */ cfs_spin_lock(&amp;target-&gt;obd_recovery_task_lock); - <span class="code-keyword">if</span> (target-&gt;obd_recovering &amp;&amp; !export-&gt;exp_in_recovery) { + <span class="code-keyword">if</span> (target-&gt;obd_recovering &amp;&amp; !export-&gt;exp_in_recovery &amp;&amp; + !export-&gt;exp_disconnected) { cfs_spin_lock(&amp;export-&gt;exp_lock); export-&gt;exp_in_recovery = 1; export-&gt;exp_req_replay_needed = 1; </pre> </div></div> <p>thanks.</p> <p>Xyratex-bug-id: <a href="http://jira-nss.xy01.xyratex.com:8080/browse/MRP-451" class="external-link" target="_blank" rel="nofollow noopener">MRP-451</a></p> Related LU-1522 Development Rank 1|hzv707: Rank (Obsolete) 4669 Severity