obd_no_conn is set in mdt_obd_notify -> mdt_allow_cli, morevoer that would be just useless to set it after recovery what means no any client can participate in recovery ever. I don't think this is related to the issue.
About reset_recovery_timer(), can you show in details how it can become negative?
Also note, that there were changes after 2.1.0 related to recovery timer changes:
2012-02-16 Jinshan Xiong LU-889 recovery: rework extend_recovery_timer()
2011-11-03 Jinshan Xiong ORNL-28 recovery: rework extend_recovery_timer()
2011-10-21 Jinshan Xiong ORNL-28: Set recovery timeout correctly
unfortunately first one broke recovery timer and only with last one it is restored. Please check these patches weren't ported to your 2.1.0 from 2.2 separately. And maybe it makes sense to port them all together, because I see that now this function differs from 2.1.0, probably issue is fixed already.
Hm.. you are right.
obd_no_conn == 0.
I found a crash dump from a similar situation.
$7 = {
obd_type = 0xffff88043e9d4740,
obd_magic = 2874988271,
obd_name = "snxs4-MDT0000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\
000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000",
obd_uuid =
,
obd_lu_dev = 0xffff880441754000,
obd_minor = 2,
obd_attached = 1,
obd_set_up = 1,
obd_recovering = 1,
obd_abort_recovery = 1,
obd_version_recov = 1,
obd_replayable = 1,
obd_no_transno = 0,
obd_no_recov = 0,
obd_stopping = 0,
obd_starting = 1,
obd_force = 0,
obd_fail = 0,
obd_async_recov = 0,
obd_no_conn = 0,
obd_inactive = 0,
obd_process_conf = 0,
obd_recovery_expired = 0
...
obd_refcount =
,
...
obd_unlinked_exports =
{ next = 0xffff88042b7fd4a0, prev = 0xffff88042b7fd4a0 },
obd_delayed_exports =
,
obd_num_exports = 2072,
..
obd_exports_timed =
,
obd_eviction_timer = 0,
obd_max_recoverable_clients = 4471,
obd_connected_clients = 2095,
obd_stale_clients = 2400,
obd_delayed_clients = 0,
...
obd_recovery_start = 1330601299,
obd_recovery_end = 1330602199,
obd_recovery_time_hard = 900,
obd_recovery_timeout = 900,
obd_recovery_data = {
trd_recovery_handler = 0xffffffffa09a18b0 <mdt_recovery_handle>,
trd_processing_task = 15692,
trd_starting = {
done = 0,
wait = {
lock = {
raw_lock =
},
task_list =
}
},
trd_finishing = {
done = 0,
wait = {
lock = {
raw_lock =
},
task_list =
}
}
},
obd_replayed_locks = 0,
obd_req_replay_clients =
obd_lock_replay_clients = { counter = 24 }
,
After discussion with Alex it is clean that endless recovery cycle caused by wrong number of stale clients and connected, so the check below never true:
check_for_clients(struct obd_device *obd)
{
unsigned int clnts = cfs_atomic_read(&obd->obd_connected_clients);
if (obd->obd_abort_recovery || obd->obd_recovery_expired)
return 1;
LASSERT(clnts <= obd->obd_max_recoverable_clients);
---> return (clnts + obd->obd_stale_clients ==
obd->obd_max_recoverable_clients);
}
I suspect this is caused by class_disconnect_stale_exports() which moves stale clients from obd_exports list but they are still in hash, so connected client can find export from stale list and connect to it. Therefore that export will be 'connected' and 'stale' at the same time. Solution could be removal export from hash along with removal from obd_exports list, but more close investigation is needed to check that there is no other races.
per additional discussion with Mike, we have a verdict - that is race between class_disconnect and target_handle_connect.
int class_disconnect(struct obd_export *export) { int already_disconnected; ENTRY; if (export == NULL) { fixme(); CDEBUG(D_IOCTL, "attempting to free NULL export %p\n", export); RETURN(-EINVAL); } cfs_spin_lock(&export->exp_lock); already_disconnected = export->exp_disconnected; export->exp_disconnected = 1; cfs_spin_unlock(&export->exp_lock); /* class_cleanup(), abort_recovery(), and class_fail_export() * all end up in here, and if any of them race we shouldn't * call extra class_export_puts(). */ if (already_disconnected) { LASSERT(cfs_hlist_unhashed(&export->exp_nid_hash)); GOTO(no_disconn, already_disconnected); } CDEBUG(D_IOCTL, "disconnect: cookie "LPX64"\n", export->exp_handle.h_cookie); if (!cfs_hlist_unhashed(&export->exp_nid_hash)) cfs_hash_del(export->exp_obd->obd_nid_hash, &export->exp_connection->c_peer.nid, &export->exp_nid_hash); class_export_recovery_cleanup(export); <<< wait where >>> class_unlink_export(export); no_disconn: class_export_put(export); RETURN(0); }
if target_handle_connect will raced with class_export_recovery_cleanup in waiting on
obd_recovery_task_lock - we will count a one export twice.
we will leak a obd_connected_clients, and other recovery counters...
obd_req_replay_clients = {
counter = 24
},
obd_lock_replay_clients = {
counter = 24
},
remote: New Changes:
remote: http://review.whamcloud.com/2255
Integrated in 
lustre-master » i686,client,el5,ofa #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = FAILURE
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/obdclass/genops.c
- lustre/ldlm/ldlm_lib.c
Integrated in 
lustre-master » i686,server,el6,inkernel #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/obdclass/genops.c
- lustre/ldlm/ldlm_lib.c
Integrated in lustre-master » x86_64,client,el6,ofa #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = FAILURE
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/obdclass/genops.c
- lustre/ldlm/ldlm_lib.c
Integrated in lustre-master » i686,client,el6,inkernel #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/ldlm/ldlm_lib.c
- lustre/obdclass/genops.c
Integrated in lustre-master » i686,server,el5,ofa #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = FAILURE
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/ldlm/ldlm_lib.c
- lustre/obdclass/genops.c
Integrated in lustre-master » i686,client,el6,ofa #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = FAILURE
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/ldlm/ldlm_lib.c
- lustre/obdclass/genops.c
Integrated in lustre-master » i686,server,el6,ofa #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = FAILURE
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/ldlm/ldlm_lib.c
- lustre/obdclass/genops.c
Integrated in lustre-master » x86_64,client,sles11,inkernel #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/obdclass/genops.c
- lustre/ldlm/ldlm_lib.c
Integrated in lustre-master » x86_64,client,ubuntu1004,inkernel #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/obdclass/genops.c
- lustre/ldlm/ldlm_lib.c
Integrated in lustre-master » i686,client,el5,inkernel #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/ldlm/ldlm_lib.c
- lustre/obdclass/genops.c
Integrated in lustre-master » i686,server,el5,inkernel #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/obdclass/genops.c
- lustre/ldlm/ldlm_lib.c
Integrated in lustre-master » x86_64,server,el6,ofa #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = FAILURE
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/obdclass/genops.c
- lustre/ldlm/ldlm_lib.c
Integrated in lustre-master » x86_64,client,el6,inkernel #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/ldlm/ldlm_lib.c
- lustre/obdclass/genops.c
Integrated in lustre-master » x86_64,server,el6,inkernel #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/obdclass/genops.c
- lustre/ldlm/ldlm_lib.c
Integrated in lustre-master » x86_64,server,el5,ofa #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = FAILURE
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/obdclass/genops.c
- lustre/ldlm/ldlm_lib.c
Integrated in lustre-master » x86_64,server,el5,inkernel #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/ldlm/ldlm_lib.c
- lustre/obdclass/genops.c
Integrated in lustre-master » x86_64,client,el5,inkernel #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/obdclass/genops.c
- lustre/ldlm/ldlm_lib.c
Integrated in lustre-master » x86_64,client,el5,ofa #531
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = FAILURE
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/ldlm/ldlm_lib.c
- lustre/obdclass/genops.c
Landed for 2.3
Integrated in lustre-dev » x86_64,client,el5,inkernel #340
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/obdclass/genops.c
- lustre/ldlm/ldlm_lib.c
Integrated in lustre-dev » i686,client,el6,inkernel #340
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/ldlm/ldlm_lib.c
- lustre/obdclass/genops.c
Integrated in lustre-dev » i686,server,el5,inkernel #340
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/obdclass/genops.c
- lustre/ldlm/ldlm_lib.c
Integrated in lustre-dev » x86_64,server,el6,inkernel #340
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/obdclass/genops.c
- lustre/ldlm/ldlm_lib.c
Integrated in lustre-dev » i686,client,el5,inkernel #340
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/obdclass/genops.c
- lustre/ldlm/ldlm_lib.c
Integrated in lustre-dev » x86_64,server,el5,inkernel #340
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/ldlm/ldlm_lib.c
- lustre/obdclass/genops.c
Integrated in lustre-dev » x86_64,client,el6,inkernel #340
LU-1166 recovery: don't leak a connected client counter. (Revision 737da0331e8407a704cd11c04f18c2cd3d437800)
Result = SUCCESS
Oleg Drokin : 737da0331e8407a704cd11c04f18c2cd3d437800
Files :
- lustre/obdclass/genops.c
- lustre/ldlm/ldlm_lib.c
http://review.whamcloud.com/#change,2665
back port to b2_1
http://review.whamcloud.com/#change,2874
more changes missing from previous back port
Hello Bob,
I don't fully understand the portions of code impacted by this ticket, but could you explain why the additional change http://review.whamcloud.com/#change,2874 you submitted in b2_1 is not present in the master release.
diff --git a/lustre/ldlm/ldlm_lib.c b/lustre/ldlm/ldlm_lib.c
index a8433fc..0798ba7 100644
--- a/lustre/ldlm/ldlm_lib.c
+++ b/lustre/ldlm/ldlm_lib.c
@@ -1069,7 +1069,8 @@ dont_check_exports:
class_disconnect->class_export_recovery_cleanup() race
*/
cfs_spin_lock(&target->obd_recovery_task_lock);
- if (target->obd_recovering && !export->exp_in_recovery) {
+ if (target->obd_recovering && !export->exp_in_recovery &&
+ !export->exp_disconnected) {
cfs_spin_lock(&export->exp_lock);
export->exp_in_recovery = 1;
export->exp_req_replay_needed = 1;
thanks.
Xyratex-bug-id: MRP-451