[LU-328] OSS pseudo-hang due to (struct filter_obd *)->fo_llog_list_lock deadlock upon OSTs warm restart/recovery

Mon, 19 Nov 2012 17:15:49 +0000

During a "warm" (shine stop/start) of several OSTs, OSS became pseudo-hang (ping ok but no login possible).

Forced crash-dump analysis show that all 32 CPUs/Cores are running/spinning with 32 ll_ost_<xx> threads all with the following stack-trace:
=======================================================
_spin_lock()
filter_sync_llogs()
filter_set_info_async()
target_handle_connect()
ost_handle()
ptlrpc_server_handle_request()
ptlrpc_main()
kernel_thread()
=======================================================

when the thread owning the concerned (struct filter_obd *)->fo_llog_list_lock spin-lock is sleeping/waiting to be re-scheduled with the following stack-trace :
==================================================================================
schedule()
__cond_resched()
_cond_resched()
__kmalloc()
cfs_alloc()
filter_find_create_olg()
filter_set_info_async()
ost_set_info()
ost_handle()
ptlrpc_server_handle_request()
ptlrpc_main()
kernel_thread()
==================================================================================

So this definitelly points to a bug in filter_find_create_olg() where if filter_find_olg_internal() does not find a matching struct obd_llog_group and it needs to be kmem-allocated, this can and must be done with (struct filter_obd *)->fo_llog_list_lock freed to avoid this race+dead-lock scenario.

This means that the following source extract of filter_find_create_olg() routine :
==================================================================================
OBD_ALLOC_PTR(olg);
if (olg == NULL)
GOTO(out_unlock, olg = ERR_PTR(-ENOMEM));

llog_group_init(olg, group);
==================================================================================

should be replaced with :
=========================
spin_unlock(&filter->fo_llog_list_lock);
OBD_ALLOC_PTR(olg);
if (olg == NULL)
GOTO(out, olg = ERR_PTR(-ENOMEM));

llog_group_init(olg, group);
spin_lock(&filter->fo_llog_list_lock);
=========================

Whamcloud Community JIRA

[LU-328] OSS pseudo-hang due to (struct filter_obd *)->fo_llog_list_lock deadlock upon OSTs warm restart/recovery