https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1194 Lustre <p>1329761763787093,7345,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.418338&#93;</span> Call Trace:" <br/> 1329761763794294,7346,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.422654&#93;</span> <span class="error">&#91;&lt;ffffffff8105dc60&gt;&#93;</span> ? default_wake_function+0x0/0x20" <br/> 1329761763794316,7347,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.429093&#93;</span> <span class="error">&#91;&lt;ffffffffa0442855&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span>" <br/> 1329761763808114,7348,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.436284&#93;</span> <span class="error">&#91;&lt;ffffffffa0442e95&gt;&#93;</span> lbug_with_loc+0x75/0xe0 <span class="error">&#91;libcfs&#93;</span>" <br/> 1329761763815465,7349,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.442714&#93;</span> <span class="error">&#91;&lt;ffffffffa065805e&gt;&#93;</span> llog_recov_thread_stop+0x1ae/0x1b0 <span class="error">&#91;ptlrpc&#93;</span>" <br/> 1329761763815505,7350,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.450178&#93;</span> <span class="error">&#91;&lt;ffffffffa0658097&gt;&#93;</span> llog_recov_thread_fini+0x37/0x220 <span class="error">&#91;ptlrpc&#93;</span>" <br/> 1329761763820472,7351,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.457480&#93;</span> <span class="error">&#91;&lt;ffffffff81093fae&gt;&#93;</span> ? down+0x2e/0x50" <br/> 1329761763835467,7352,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.462464&#93;</span> <span class="error">&#91;&lt;ffffffffa0a56558&gt;&#93;</span> filter_llog_finish+0x148/0x2f0 <span class="error">&#91;obdfilter&#93;</span>" <br/> 1329761763842393,7353,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.469818&#93;</span> <span class="error">&#91;&lt;ffffffffa0609e5a&gt;&#93;</span> ? target_cleanup_recovery+0x4a/0x320 <span class="error">&#91;ptlrpc&#93;</span>" <br/> 1329761763842426,7354,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.477457&#93;</span> <span class="error">&#91;&lt;ffffffffa0512f58&gt;&#93;</span> obd_llog_finish+0x98/0x230 <span class="error">&#91;obdclass&#93;</span>" <br/> 1329761763857637,7355,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.484336&#93;</span> <span class="error">&#91;&lt;ffffffffa0a54baa&gt;&#93;</span> filter_precleanup+0x12a/0x440 <span class="error">&#91;obdfilter&#93;</span>" <br/> 1329761763871327,7356,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.491538&#93;</span> <span class="error">&#91;&lt;ffffffffa0523316&gt;&#93;</span> ? class_disconnect_exports+0x116/0x2b0 <span class="error">&#91;obdclass&#93;</span>" <br/> 1329761763871364,7357,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.499607&#93;</span> <span class="error">&#91;&lt;ffffffffa053b772&gt;&#93;</span> class_cleanup+0x192/0xe30 <span class="error">&#91;obdclass&#93;</span>" <br/> 1329761763878975,7358,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.506423&#93;</span> <span class="error">&#91;&lt;ffffffffa051bc76&gt;&#93;</span> ? class_name2dev+0x56/0xd0 <span class="error">&#91;obdclass&#93;</span>" <br/> 1329761763879013,7359,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.513330&#93;</span> <span class="error">&#91;&lt;ffffffffa053d603&gt;&#93;</span> class_process_config+0x11f3/0x1fd0 <span class="error">&#91;obdclass&#93;</span>" <br/> 1329761763892326,7360,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.520948&#93;</span> <span class="error">&#91;&lt;ffffffffa0443a13&gt;&#93;</span> ? cfs_alloc+0x63/0x90 <span class="error">&#91;libcfs&#93;</span>" <br/> 1329761763899616,7361,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.527229&#93;</span> <span class="error">&#91;&lt;ffffffffa0537fdb&gt;&#93;</span> ? lustre_cfg_new+0x33b/0x880 <span class="error">&#91;obdclass&#93;</span>" <br/> 1329761763899661,7362,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.534335&#93;</span> <span class="error">&#91;&lt;ffffffffa044d6b1&gt;&#93;</span> ? libcfs_debug_vmsg2+0x4d1/0xb50 <span class="error">&#91;libcfs&#93;</span>" <br/> 1329761763907044,7363,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.541589&#93;</span> <span class="error">&#91;&lt;ffffffffa053e545&gt;&#93;</span> class_manual_cleanup+0x165/0x790 <span class="error">&#91;obdclass&#93;</span>" <br/> 1329761763913937,7364,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.549022&#93;</span> <span class="error">&#91;&lt;ffffffffa051bc76&gt;&#93;</span> ? class_name2dev+0x56/0xd0 <span class="error">&#91;obdclass&#93;</span>" <br/> 1329761763920982,7365,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.555894&#93;</span> <span class="error">&#91;&lt;ffffffffa0549170&gt;&#93;</span> server_put_super+0xa50/0xf60 <span class="error">&#91;obdclass&#93;</span>" <br/> 1329761763927266,7366,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.562905&#93;</span> <span class="error">&#91;&lt;ffffffff8118d5b6&gt;&#93;</span> ? invalidate_inodes+0xf6/0x190" <br/> 1329761763933628,7367,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.569150&#93;</span> <span class="error">&#91;&lt;ffffffff811750ab&gt;&#93;</span> generic_shutdown_super+0x5b/0xe0" <br/> 1329761763939475,7368,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.575568&#93;</span> <span class="error">&#91;&lt;ffffffff81175196&gt;&#93;</span> kill_anon_super+0x16/0x60" <br/> 1329761763946424,7369,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.581419&#93;</span> <span class="error">&#91;&lt;ffffffffa05402c6&gt;&#93;</span> lustre_kill_super+0x36/0x60 <span class="error">&#91;obdclass&#93;</span>" <br/> 1329761763952386,7370,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.588352&#93;</span> <span class="error">&#91;&lt;ffffffff81176210&gt;&#93;</span> deactivate_super+0x70/0x90" <br/> 1329761763963859,7371,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.594278&#93;</span> <span class="error">&#91;&lt;ffffffff8119174f&gt;&#93;</span> mntput_no_expire+0xbf/0x110" <br/> 1329761763963903,7372,4,0,0,"snxs2n003","","kernel","<span class="error">&#91;272274.600238&#93;</span> <span class="error">&#91;&lt;ffffffff81191b7b&gt;&#93;</span> sys_umount+0x7b/0x3a0" </p> RHEL6 + 2.1.0 (xyratex 1.0). LU-1194

llog_recov_thread_stop+0x1ae/0x1b0 asserting

Bug Major Resolved Fixed WC Triage Alexey Lyashkov Wed, 7 Mar 2012 08:09:14 +0000 Mon, 14 Jan 2013 09:17:09 +0000 Wed, 25 Jul 2012 10:53:17 +0000 Lustre 2.1.0 Lustre 2.3.0 Lustre 2.1.3 0 8 <p>Hi Shadow,<br/> thanks for your bug report.</p> <p>Unfortunately, there isn't enough information in this bug to make any investigation on what is failing here. What kind of load is being run before hitting this problem? What did the CERROR() before the LBUG() report?</p> <p>1329761744817699,6366,4,0,0,"snxs2n003","Lustre","kernel","<span class="error">&#91;269602.953789&#93;</span> Lustre: 15857:0:(import.c:526:import_select_connection()) Skipped 8 previous similar messages"<br/> 1329761744817727,6368,4,0,0,"snxs2n003","Lustre","kernel","<span class="error">&#91;269658.938516&#93;</span> Lustre: 15856:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1394071067253911 sent from MGC10.10.100.5@o2ib<br/> to NID 10.10.100.5@o2ib has timed out for slow reply: <span class="error">&#91;sent 1329759091&#93;</span> <span class="error">&#91;real_sent 1329759091&#93;</span> <span class="error">&#91;current 1329759147&#93;</span> <span class="error">&#91;deadline 56s&#93;</span> <span class="error">&#91;delay 0s&#93;</span> req@ffff880364381000 x1394071067253911/t0(0) o-1-&gt;<br/> MGS@MGC10.10.100.5@o2ib_0:26/25 lens 368/512 e 0 to 1 dl 1329759147 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1"<br/> 1329761744817743,6370,4,0,0,"snxs2n003","Lustre","kernel","<span class="error">&#91;269658.977157&#93;</span> Lustre: 15856:0:(client.c:1778:ptlrpc_expire_one_request()) Skipped 8 previous similar messages"<br/> 1329761747023362,6378,4,0,0,"snxs2n002","Lustre","kernel","<span class="error">&#91;272256.552763&#93;</span> Lustre: server umount snxs2-OST0003 complete"<br/> 1329761747381152,6383,4,0,0,"snxs2n002","Lustre","kernel","<span class="error">&#91;272256.913120&#93;</span> Lustre: Failing over snxs2-OST0000"<br/> 1329761747414521,6385,4,0,0,"snxs2n003","Lustre","kernel","<span class="error">&#91;272257.995434&#93;</span> Lustre: Failing over snxs2-OST0007"<br/> 1329761748292808,6387,4,0,0,"snxs2n001","Lustre","kernel","[ 6339.830463] Lustre: 3868:0:(import.c:526:import_select_connection()) snxs2-OST000e-osc-MDT0000: tried all connections, increasing la<br/> tency to 11s"<br/> 1329761748292852,6389,4,0,0,"snxs2n001","Lustre","kernel","[ 6339.845109] Lustre: 3868:0:(import.c:526:import_select_connection()) Skipped 2 previous similar messages"<br/> 1329761748292981,6391,4,0,0,"snxs2n001","Lustre","kernel","[ 6339.853216] Lustre: snxs2-OST0000-osc-MDT0000: Connection to service snxs2-OST0000 via nid 10.10.100.1@o2ib was lost; in progress op<br/> erations using this service will wait for recovery to complete."<br/> 1329761748368977,6393,4,0,0,"snxs2n001","Lustre","kernel","[ 6339.895529] Lustre: snxs2-OST0003-osc-MDT0000: Connection to service snxs2-OST0003 via nid 10.10.100.1@o2ib was lost; in progress op<br/> erations using this service will wait for recovery to complete."<br/> 1329761748369012,6395,4,0,0,"snxs2n001","Lustre","kernel","[ 6339.896566] Lustre: 3867:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1394349653885019 sent from snxs2-OST000e-osc-MDT<br/> 0000 to NID 10.10.100.4@o2ib has failed due to network error: <span class="error">&#91;sent 1329761748&#93;</span> <span class="error">&#91;real_sent 1329761748&#93;</span> <span class="error">&#91;current 1329761748&#93;</span> <span class="error">&#91;deadline 16s&#93;</span> <span class="error">&#91;delay -16s&#93;</span> req@ffff88085bed0000 x1394349653885019/t0<br/> (0) o-1-&gt;snxs2-OST000e_UUID@10.10.100.4@o2ib:28/4 lens 368/512 e 0 to 1 dl 1329761764 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1"<br/> 1329761748369033,6397,4,0,0,"snxs2n001","Lustre","kernel","[ 6339.896574] Lustre: 3867:0:(client.c:1778:ptlrpc_expire_one_request()) Skipped 3 previous similar messages"<br/> 1329761748793789,6399,4,0,0,"snxs2n003","Lustre","kernel","<span class="error">&#91;272259.407949&#93;</span> Lustre: snxs2-OST0007: shutting down for failover; client state will be preserved."<br/> 1329761763772702,6400,3,0,0,"snxs2n003","LustreError","kernel","<span class="error">&#91;272274.395479&#93;</span> LustreError: 4998:0:(recov_thread.c:447:llog_recov_thread_stop()) Busy llcds found (1) on lcm ffff880419da7800"<br/> 1329761748844011,6402,6,0,0,"snxs2n003","Lustre","kernel","<span class="error">&#91;272259.437366&#93;</span> Lustre: OST snxs2-OST0007 has stopped."<br/> 1329761763772748,6403,0,0,0,"snxs2n003","LustreError","kernel","<span class="error">&#91;272274.406830&#93;</span> LustreError: 4998:0:(recov_thread.c:467:llog_recov_thread_stop()) LBUG"</p> <p>that is logs from a different hit's, but i think we have same root case.</p> <p>Probably root cause is window between llog_sync(ctxt, NULL) and class_import_put() at filter_llog_finish():</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> llog_sync(ctxt, NULL); /* * Balance class_import_get() in llog_receptor_accept(). * This is safe to do, as llog is already synchronized * and its import may go. */ cfs_mutex_down(&amp;ctxt-&gt;loc_sem); if (ctxt-&gt;loc_imp) { class_import_put(ctxt-&gt;loc_imp); ctxt-&gt;loc_imp = NULL; } cfs_mutex_up(&amp;ctxt-&gt;loc_sem); llog_ctxt_put(ctxt); </pre> </div></div> <p> If llog_obd_repl_cancel() happend at this window new llcd will be cached at ctx. And LBUG happend at llog_recov_thread_stop().<br/> I work to fix this issue.</p> <p>request <a href="http://review.whamcloud.com/2789" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2789</a></p> <p>Hi,</p> <p>At IFERC cluster, trying to stop all OSSes with some clients left, they got a bunch of the same LBUG in llog_recov_thread_stop() on multiple OSSes.<br/> On all the concerned OSSes, the panic stack and preceding messages transmitted by the Support team look like :<br/> ===============================================================================<br/> LustreError: 137-5: UUID 'work-OST006a_UUID' is not available for connect (stopping)<br/> Lustre: work-OST006e: shutting down for failover; client state will be preserved.<br/> LustreError: 137-5: UUID 'work-OST0069_UUID' is not available for connect (stopping)<br/> LustreError: Skipped 218 previous similar messages<br/> LustreError: 783:0:(recov_thread.c:447:llog_recov_thread_stop()) Busy llcds found (1) on lcm ffff8805a151d600<br/> LustreError: 783:0:(recov_thread.c:467:llog_recov_thread_stop()) LBUG<br/> Pid: 783, comm: umount</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffff8104c780&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> <span class="error">&#91;&lt;ffffffffa056a855&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa056ae95&gt;&#93;</span> lbug_with_loc+0x75/0xe0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa073bc7e&gt;&#93;</span> llog_recov_thread_stop+0x1ae/0x1b0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa073bcb7&gt;&#93;</span> llog_recov_thread_fini+0x37/0x220 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8108009e&gt;&#93;</span> ? down+0x2e/0x50<br/> <span class="error">&#91;&lt;ffffffffa09db5b8&gt;&#93;</span> filter_llog_finish+0x148/0x2f0 <span class="error">&#91;obdfilter&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05fcc85&gt;&#93;</span> ? obd_zombie_impexp_notify+0x15/0x20 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8107a5cb&gt;&#93;</span> ? remove_wait_queue+0x5b/0x70<br/> <span class="error">&#91;&lt;ffffffffa05f4ec8&gt;&#93;</span> obd_llog_finish+0x98/0x230 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8104c780&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> <span class="error">&#91;&lt;ffffffffa09d99d1&gt;&#93;</span> filter_precleanup+0x131/0x670 <span class="error">&#91;obdfilter&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0605316&gt;&#93;</span> ? class_disconnect_exports+0x116/0x2b0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05fe553&gt;&#93;</span> ? dump_exports+0x133/0x1a0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa061d7e5&gt;&#93;</span> class_cleanup+0x1b5/0xe20 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06d4c32&gt;&#93;</span> ? mgc_process_config+0x232/0xed0 <span class="error">&#91;mgc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05fe0a6&gt;&#93;</span> ? class_name2dev+0x56/0xd0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa061f643&gt;&#93;</span> class_process_config+0x11f3/0x1fd0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa056ba13&gt;&#93;</span> ? cfs_alloc+0x63/0x90 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa061a02b&gt;&#93;</span> ? lustre_cfg_new+0x33b/0x880 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0625d7b&gt;&#93;</span> ? lustre_cfg_new+0x33b/0x880 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0620585&gt;&#93;</span> class_manual_cleanup+0x165/0x790 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0626aa8&gt;&#93;</span> ? lustre_end_log+0x1c8/0x490 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81041f25&gt;&#93;</span> ? fair_select_idle_sibling+0x95/0x150<br/> <span class="error">&#91;&lt;ffffffffa05fe0a6&gt;&#93;</span> ? class_name2dev+0x56/0xd0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa062b1f0&gt;&#93;</span> server_put_super+0xa50/0xf60 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffff811782bc&gt;&#93;</span> ? dispose_list+0x11c/0x140<br/> <span class="error">&#91;&lt;ffffffff81178748&gt;&#93;</span> ? invalidate_inodes+0x158/0x1a0<br/> <span class="error">&#91;&lt;ffffffff8115feab&gt;&#93;</span> generic_shutdown_super+0x5b/0x110<br/> <span class="error">&#91;&lt;ffffffff8115ffc6&gt;&#93;</span> kill_anon_super+0x16/0x60<br/> <span class="error">&#91;&lt;ffffffffa0622306&gt;&#93;</span> lustre_kill_super+0x36/0x60 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81161060&gt;&#93;</span> deactivate_super+0x70/0x90<br/> <span class="error">&#91;&lt;ffffffff8117c76f&gt;&#93;</span> mntput_no_expire+0xbf/0x110<br/> <span class="error">&#91;&lt;ffffffff8117cba8&gt;&#93;</span> sys_umount+0x78/0x3c0<br/> <span class="error">&#91;&lt;ffffffff81003172&gt;&#93;</span> system_call_fastpath+0x16/0x1b</p> <p>Kernel panic - not syncing: LBUG<br/> Pid: 783, comm: umount Not tainted 2.6.32-131.17.1.bl6.Bull.27.0.x86_64 #1<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff8147d803&gt;&#93;</span> ? panic+0x78/0x143<br/> <span class="error">&#91;&lt;ffffffffa056aeeb&gt;&#93;</span> ? lbug_with_loc+0xcb/0xe0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa073bc7e&gt;&#93;</span> ? llog_recov_thread_stop+0x1ae/0x1b0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa073bcb7&gt;&#93;</span> ? llog_recov_thread_fini+0x37/0x220 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8108009e&gt;&#93;</span> ? down+0x2e/0x50<br/> <span class="error">&#91;&lt;ffffffffa09db5b8&gt;&#93;</span> ? filter_llog_finish+0x148/0x2f0 <span class="error">&#91;obdfilter&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05fcc85&gt;&#93;</span> ? obd_zombie_impexp_notify+0x15/0x20 <span class="error">&#91;obdclass&#93;</span><br/> Lustre: OST work-OST006e has stopped.<br/> Lustre: work-OST0069: shutting down for failover; client state will be preserved<br/> .<br/> <span class="error">&#91;&lt;ffffffff8107a5cb&gt;&#93;</span> ? remove_wait_queue+0x5b/0x70<br/> <span class="error">&#91;&lt;ffffffffa05f4ec8&gt;&#93;</span> ? obd_llog_finish+0x98/0x230 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8104c780&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> <span class="error">&#91;&lt;ffffffffa09d99d1&gt;&#93;</span> ? filter_precleanup+0x131/0x670 <span class="error">&#91;obdfilter&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0605316&gt;&#93;</span> ? class_disconnect_exports+0x116/0x2b0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05fe553&gt;&#93;</span> ? dump_exports+0x133/0x1a0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa061d7e5&gt;&#93;</span> ? class_cleanup+0x1b5/0xe20 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06d4c32&gt;&#93;</span> ? mgc_process_config+0x232/0xed0 <span class="error">&#91;mgc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05fe0a6&gt;&#93;</span> ? class_name2dev+0x56/0xd0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa061f643&gt;&#93;</span> ? class_process_config+0x11f3/0x1fd0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa056ba13&gt;&#93;</span> ? cfs_alloc+0x63/0x90 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa061a02b&gt;&#93;</span> ? lustre_cfg_new+0x33b/0x880 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0625d7b&gt;&#93;</span> ? lustre_cfg_new+0x33b/0x880 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0620585&gt;&#93;</span> ? class_manual_cleanup+0x165/0x790 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0626aa8&gt;&#93;</span> ? lustre_end_log+0x1c8/0x490 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81041f25&gt;&#93;</span> ? fair_select_idle_sibling+0x95/0x150<br/> <span class="error">&#91;&lt;ffffffffa05fe0a6&gt;&#93;</span> ? class_name2dev+0x56/0xd0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa062b1f0&gt;&#93;</span> ? server_put_super+0xa50/0xf60 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffff811782bc&gt;&#93;</span> ? dispose_list+0x11c/0x140<br/> <span class="error">&#91;&lt;ffffffff81178748&gt;&#93;</span> ? invalidate_inodes+0x158/0x1a0<br/> <span class="error">&#91;&lt;ffffffff8115feab&gt;&#93;</span> ? generic_shutdown_super+0x5b/0x110<br/> <span class="error">&#91;&lt;ffffffff8115ffc6&gt;&#93;</span> ? kill_anon_super+0x16/0x60<br/> <span class="error">&#91;&lt;ffffffffa0622306&gt;&#93;</span> ? lustre_kill_super+0x36/0x60 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81161060&gt;&#93;</span> ? deactivate_super+0x70/0x90<br/> <span class="error">&#91;&lt;ffffffff8117c76f&gt;&#93;</span> ? mntput_no_expire+0xbf/0x110<br/> <span class="error">&#91;&lt;ffffffff8117cba8&gt;&#93;</span> ? sys_umount+0x78/0x3c0<br/> <span class="error">&#91;&lt;ffffffff81003172&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b<br/> ===============================================================================</p> <p>It definitely looks like this issue.</p> <p>Landed for 2.3</p> <p><a href="http://review.whamcloud.com/#change,3480" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3480</a></p> <p>back port to b2_1</p> <p>Xyratex-bug-id: <a href="http://jira-nss.xy01.xyratex.com:8080/browse/MRP-456" class="external-link" target="_blank" rel="nofollow noopener">MRP-456</a></p> <p>Doesn't look enough.<br/> While we are running 2.1.3, we got 4 nodes that crash at umount with exactly the same bt.</p> <p>We are seeing this on 2.1.3 when we unmount OSTs.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; bt PID: 58311 TASK: ffff880285dbe040 CPU: 4 COMMAND: "umount" #0 [ffff880364a158c0] machine_kexec at ffffffff8103281b #1 [ffff880364a15920] crash_kexec at ffffffff810ba8e2 #2 [ffff880364a159f0] panic at ffffffff814fd6d1 #3 [ffff880364a15a70] lbug_with_loc at ffffffffa05bae5b [libcfs] #4 [ffff880364a15a90] llog_recov_thread_stop at ffffffffa0c23afb [ptlrpc] #5 [ffff880364a15ac0] llog_recov_thread_fini at ffffffffa0c23b33 [ptlrpc] #6 [ffff880364a15ae0] filter_llog_finish at ffffffffa06c132d [obdfilter] #7 [ffff880364a15b10] obd_llog_finish at ffffffffa0ac22d8 [obdclass] #8 [ffff880364a15b30] filter_precleanup at ffffffffa06c0d1f [obdfilter] #9 [ffff880364a15b90] class_cleanup at ffffffffa0ae9997 [obdclass] #10 [ffff880364a15c10] class_process_config at ffffffffa0aebcdb [obdclass] #11 [ffff880364a15ca0] class_manual_cleanup at ffffffffa0aeca19 [obdclass] #12 [ffff880364a15d60] server_put_super at ffffffffa0af88bc [obdclass] #13 [ffff880364a15e30] generic_shutdown_super at ffffffff8117d49b #14 [ffff880364a15e50] kill_anon_super at ffffffff8117d586 #15 [ffff880364a15e70] lustre_kill_super at ffffffffa0aee656 [obdclass] #16 [ffff880364a15e90] deactivate_super at ffffffff8117e600 #17 [ffff880364a15eb0] mntput_no_expire at ffffffff8119a64f #18 [ffff880364a15ee0] sys_umount at ffffffff8119b0eb #19 [ffff880364a15f80] system_call_fastpath at ffffffff8100b0f2 RIP: 00007fc434e45717 RSP: 00007fff3fb87248 RFLAGS: 00010206 RAX: 00000000000000a6 RBX: ffffffff8100b0f2 RCX: 0000000000000010 RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00007fc436843b60 RBP: 00007fc436843b40 R8: 00007fc436843b80 R9: 0000000000000001 R10: 00007fff3fb87070 R11: 0000000000000206 R12: 0000000000000000 R13: 0000000000000000 R14: 0000000000000000 R15: 00007fc436843bc0 ORIG_RAX: 00000000000000a6 CS: 0033 SS: 002b </pre> </div></div> <p>The logs leading up to this were:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[1733729.589047] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [1733729.600508] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Timed out RDMA with 192.168.66.124@o2ib (151): c: 7, oc: 0, rc: 8 [1733730.586896] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds [1733730.597628] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Skipped 1 previous similar message [1733730.608257] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Timed out RDMA with 192.168.66.129@o2ib (151): c: 7, oc: 0, rc: 8 [1733730.621140] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Skipped 1 previous similar message [1733733.580433] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [1733733.591675] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Skipped 4 previous similar messages [1733733.602613] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Timed out RDMA with 192.168.66.156@o2ib (155): c: 7, oc: 0, rc: 8 [1733733.615604] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Skipped 4 previous similar messages [1733742.561072] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [1733742.571750] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Skipped 4 previous similar messages [1733742.582543] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Timed out RDMA with 192.168.66.127@o2ib (162): c: 7, oc: 0, rc: 8 [1733742.595489] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Skipped 4 previous similar messages [1733971.067888] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [1733971.080940] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Skipped 3 previous similar messages [1733971.093679] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Timed out RDMA with 192.168.66.243@o2ib (150): c: 7, oc: 0, rc: 8 [1733971.109722] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Skipped 3 previous similar messages [1733976.057048] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [1733976.069835] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Skipped 8 previous similar messages [1733976.082962] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Timed out RDMA with 192.168.66.181@o2ib (153): c: 7, oc: 0, rc: 8 [1733976.097100] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Skipped 8 previous similar messages [1733985.037612] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds [1733985.049931] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Skipped 168 previous similar messages [1733985.062783] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Timed out RDMA with 192.168.66.249@o2ib (162): c: 7, oc: 0, rc: 8 [1733985.078215] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Skipped 168 previous similar messages [1735454.496518] LustreError: 48744:0:(sec.c:383:import_sec_validate_get()) import ffff88050b301800 (CLOSED) with no sec [1735454.508690] LustreError: 48744:0:(recov_thread.c:394:llcd_push()) Couldn't send llcd ffff880131a16038 (-13) [1735454.520599] LustreError: 48744:0:(filter.c:3062:filter_sync_llogs()) error flushing logs to MDS: rc -13 [1735604.539999] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [1735604.550820] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Skipped 42 previous similar messages [1735604.561625] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Timed out RDMA with 192.168.66.107@o2ib (150): c: 7, oc: 0, rc: 8 [1735604.576129] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Skipped 42 previous similar messages [1735608.531438] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [1735608.542183] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Skipped 12 previous similar messages [1735608.553255] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Timed out RDMA with 192.168.66.38@o2ib (152): c: 7, oc: 0, rc: 8 [1735608.566020] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Skipped 12 previous similar messages [1735613.520678] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds [1735613.531552] LustreError: 48595:0:(o2iblnd_cb.c:2991:kiblnd_check_txs_locked()) Skipped 15 previous similar messages [1735613.542474] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Timed out RDMA with 192.168.66.74@o2ib (159): c: 7, oc: 0, rc: 8 [1735613.557368] LustreError: 48595:0:(o2iblnd_cb.c:3054:kiblnd_check_conns()) Skipped 15 previous similar messages [1740454.405061] Lustre: 48598:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1355869330/real 1355869330] req@ffff88086b1f9c00 x1421268042665725/t0(0) o400-&gt;MGC192.168.65.5@o2ib@192.168.65.5@o2ib:26/25 lens 192/192 e 0 to 1 dl 1355869337 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [1740454.436739] LustreError: 166-1: MGC192.168.65.5@o2ib: Connection to MGS (at 192.168.65.5@o2ib) was lost; in progress operations using this service will fail [1740460.438928] Lustre: 48599:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1355869337/real 1355869337] req@ffff88086b1f9000 x1421268042665726/t0(0) o250-&gt;MGC192.168.65.5@o2ib@192.168.65.5@o2ib:26/25 lens 368/512 e 0 to 1 dl 1355869343 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [1740468.526512] LustreError: 58311:0:(recov_thread.c:447:llog_recov_thread_stop()) Busy llcds found (1) on lcm ffff880699307000 [1740468.538720] LustreError: 58311:0:(recov_thread.c:467:llog_recov_thread_stop()) LBUG </pre> </div></div> <p>may you put crashdump in some place ?</p> <p>I can't upload crashes, site is classified.<br/> We can manage to export some content of structures or backtraces of tid as text. In this case, let me know what you want to see.</p> <p>As per Intel request, I have open a new issue (<a href="https://jira.whamcloud.com/browse/LU-2615" title="group of OSS crashed at umount" class="issue-link" data-issue-key="LU-2615"><del>LU-2615</del></a>) to track this down.</p> Development Rank 1|hzv6gn: Rank (Obsolete) 4581 Severity