https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6273 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/0429703c-ba58-11e4-8053-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0429703c-ba58-11e4-8053-5254006e85c2</a>.</p> <p>The sub-test test_17 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test failed to respond and timed out </pre> </div></div> <p>ost dmesg</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: 3053:0:(client.c:1939:ptlrpc_expire_one_request()) Skipped 2 previous similar messages LustreError: 137-5: lustre-OST0002_UUID: not available for connect from 10.2.4.161@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 120 previous similar messages INFO: task mount.lustre:3630 blocked for more than 120 seconds. Tainted: P --------------- 2.6.32-431.29.2.el6_lustre.x86_64 #1 "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. mount.lustre D 0000000000000000 0 3630 3629 0x00000080 ffff88006edf9718 0000000000000082 0000000000000000 ffff88006ef82040 ffff88006edf9698 ffffffff81055783 ffff88007e4c2ad8 ffff880002216880 ffff88006ef825f8 ffff88006edf9fd8 000000000000fbc8 ffff88006ef825f8 Call Trace: [&lt;ffffffff81055783&gt;] ? set_next_buddy+0x43/0x50 [&lt;ffffffff8152a595&gt;] schedule_timeout+0x215/0x2e0 [&lt;ffffffff81069f15&gt;] ? enqueue_entity+0x125/0x450 [&lt;ffffffff8152a213&gt;] wait_for_common+0x123/0x180 [&lt;ffffffff81061d00&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa090cd00&gt;] ? client_lwp_config_process+0x0/0x1948 [obdclass] [&lt;ffffffff8152a32d&gt;] wait_for_completion+0x1d/0x20 [&lt;ffffffffa0898e14&gt;] llog_process_or_fork+0x354/0x540 [obdclass] [&lt;ffffffffa0899014&gt;] llog_process+0x14/0x30 [obdclass] [&lt;ffffffffa08c81d4&gt;] class_config_parse_llog+0x1e4/0x330 [obdclass] [&lt;ffffffffa10314f2&gt;] mgc_process_log+0xeb2/0x1970 [mgc] [&lt;ffffffffa102b1f0&gt;] ? mgc_blocking_ast+0x0/0x810 [mgc] [&lt;ffffffffa0ad0860&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] [&lt;ffffffffa1032ef8&gt;] mgc_process_config+0x658/0x1210 [mgc] [&lt;ffffffffa08d9383&gt;] lustre_process_log+0x7e3/0x1130 [obdclass] [&lt;ffffffffa07891c1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa08d514f&gt;] ? server_name2fsname+0x6f/0x90 [obdclass] [&lt;ffffffffa0907496&gt;] server_start_targets+0x12b6/0x1af0 [obdclass] [&lt;ffffffffa0783818&gt;] ? libcfs_log_return+0x28/0x40 [libcfs] [&lt;ffffffffa08dbfe6&gt;] ? lustre_start_mgc+0x4b6/0x1e00 [obdclass] [&lt;ffffffffa07891c1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa08d3390&gt;] ? class_config_llog_handler+0x0/0x1a70 [obdclass] [&lt;ffffffffa090c255&gt;] server_fill_super+0xbe5/0x1690 [obdclass] [&lt;ffffffffa0783818&gt;] ? libcfs_log_return+0x28/0x40 [libcfs] [&lt;ffffffffa08dde90&gt;] lustre_fill_super+0x560/0xa80 [obdclass] [&lt;ffffffffa08dd930&gt;] ? lustre_fill_super+0x0/0xa80 [obdclass] [&lt;ffffffff8118c56f&gt;] get_sb_nodev+0x5f/0xa0 [&lt;ffffffffa08d4ee5&gt;] lustre_get_sb+0x25/0x30 [obdclass] [&lt;ffffffff8118bbcb&gt;] vfs_kern_mount+0x7b/0x1b0 [&lt;ffffffff8118bd72&gt;] do_kern_mount+0x52/0x130 [&lt;ffffffff8119e972&gt;] ? vfs_ioctl+0x22/0xa0 [&lt;ffffffff811ad74b&gt;] do_mount+0x2fb/0x930 [&lt;ffffffff811ade10&gt;] sys_mount+0x90/0xe0 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b Lustre: 3053:0:(client.c:1939:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1424565647/real 1424565647] req@ffff880070449080 x1493765169611180/t0(0) o38-&gt;lustre-MDT0000-lwp-OST0001@10.2.4.158@tcp:12/10 lens 400/544 e 0 to 1 dl 1424565672 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3053:0:(client.c:1939:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Lustre: 3053:0:(client.c:1939:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1424565712/real 1424565712] req@ffff880070449680 x1493765169611316/t0(0) o38-&gt;lustre-MDT0000-lwp-OST0001@10.2.4.158@tcp:12/10 lens 400/544 e 0 to 1 dl 1424565737 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3053:0:(client.c:1939:ptlrpc_expire_one_request()) Skipped 7 previous similar messages INFO: task mount.lustre:3630 blocked for more than 120 seconds. Tainted: P --------------- 2.6.32-431.29.2.el6_lustre.x86_64 #1 "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. mount.lustre D 0000000000000000 0 3630 3629 0x00000080 ffff88006edf9718 0000000000000082 0000000000000000 ffff88006ef82040 ffff88006edf9698 ffffffff81055783 ffff88007e4c2ad8 ffff880002216880 ffff88006ef825f8 ffff88006edf9fd8 000000000000fbc8 ffff88006ef825f8 Call Trace: [&lt;ffffffff81055783&gt;] ? set_next_buddy+0x43/0x50 [&lt;ffffffff8152a595&gt;] schedule_timeout+0x215/0x2e0 [&lt;ffffffff81069f15&gt;] ? enqueue_entity+0x125/0x450 [&lt;ffffffff8152a213&gt;] wait_for_common+0x123/0x180 [&lt;ffffffff81061d00&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa090cd00&gt;] ? client_lwp_config_process+0x0/0x1948 [obdclass] [&lt;ffffffff8152a32d&gt;] wait_for_completion+0x1d/0x20 [&lt;ffffffffa0898e14&gt;] llog_process_or_fork+0x354/0x540 [obdclass] [&lt;ffffffffa0899014&gt;] llog_process+0x14/0x30 [obdclass] [&lt;ffffffffa08c81d4&gt;] class_config_parse_llog+0x1e4/0x330 [obdclass] [&lt;ffffffffa10314f2&gt;] mgc_process_log+0xeb2/0x1970 [mgc] [&lt;ffffffffa102b1f0&gt;] ? mgc_blocking_ast+0x0/0x810 [mgc] [&lt;ffffffffa0ad0860&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] [&lt;ffffffffa1032ef8&gt;] mgc_process_config+0x658/0x1210 [mgc] [&lt;ffffffffa08d9383&gt;] lustre_process_log+0x7e3/0x1130 [obdclass] [&lt;ffffffffa07891c1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa08d514f&gt;] ? server_name2fsname+0x6f/0x90 [obdclass] [&lt;ffffffffa0907496&gt;] server_start_targets+0x12b6/0x1af0 [obdclass] [&lt;ffffffffa0783818&gt;] ? libcfs_log_return+0x28/0x40 [libcfs] [&lt;ffffffffa08dbfe6&gt;] ? lustre_start_mgc+0x4b6/0x1e00 [obdclass] [&lt;ffffffffa07891c1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa08d3390&gt;] ? class_config_llog_handler+0x0/0x1a70 [obdclass] [&lt;ffffffffa090c255&gt;] server_fill_super+0xbe5/0x1690 [obdclass] [&lt;ffffffffa0783818&gt;] ? libcfs_log_return+0x28/0x40 [libcfs] [&lt;ffffffffa08dde90&gt;] lustre_fill_super+0x560/0xa80 [obdclass] [&lt;ffffffffa08dd930&gt;] ? lustre_fill_super+0x0/0xa80 [obdclass] [&lt;ffffffff8118c56f&gt;] get_sb_nodev+0x5f/0xa0 [&lt;ffffffffa08d4ee5&gt;] lustre_get_sb+0x25/0x30 [obdclass] [&lt;ffffffff8118bbcb&gt;] vfs_kern_mount+0x7b/0x1b0 [&lt;ffffffff8118bd72&gt;] do_kern_mount+0x52/0x130 [&lt;ffffffff8119e972&gt;] ? vfs_ioctl+0x22/0xa0 [&lt;ffffffff811ad74b&gt;] do_mount+0x2fb/0x930 [&lt;ffffffff811ade10&gt;] sys_mount+0x90/0xe0 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b LustreError: 137-5: lustre-OST0002_UUID: not available for connect from 10.2.4.156@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 304 previous similar messages Lustre: 3053:0:(client.c:1939:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1424565842/real 1424565842] req@ffff880070449c80 x1493765169611592/t0(0) o38-&gt;lustre-MDT0000-lwp-OST0001@10.2.4.158@tcp:12/10 lens 400/544 e 0 to 1 dl 1424565867 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3053:0:(client.c:1939:ptlrpc_expire_one_request()) Skipped 16 previous similar messages INFO: task mount.lustre:3630 blocked for more than 120 seconds. </pre> </div></div> client and server: lustre-master build # 2856<br/> zfs LU-6273

Hard Failover replay-dual test_17: Failover OST mount hang

Bug Blocker Resolved Fixed Mikhail Pershin Maloo zfs Tue, 24 Feb 2015 06:48:29 +0000 Wed, 23 Sep 2015 06:32:19 +0000 Tue, 22 Sep 2015 04:20:48 +0000 Lustre 2.6.0 Lustre 2.7.0 Lustre 2.8.0 Lustre 2.8.0 0 12 <p>So the real problem here is such that MDS has crashed and now OSTs cannot reconnect to it anymore.<br/> We need to find MDS crashdump and get a log our of there to see why did it crash.</p> <p>Mike,<br/> Could you please have a look and comment on this one?<br/> Thank you!</p> <p>Additionally - even with MDS/MGS node still down OSTs absolutely must be able to start as long as they have a cached copy of config.<br/> And yet in the OST logs we see OST0000 started and OST0002 hung - so we also need to see why did that one stack - I imagine Mike would want to look at this one.</p> <p>another instance seen in recovery-double-scale</p> <p><a href="https://testing.hpdd.intel.com/test_sets/85116226-c170-11e4-bef2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/85116226-c170-11e4-bef2-5254006e85c2</a></p> <p>OST</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: 27409:0:(client.c:1939:ptlrpc_expire_one_request()) Skipped 2 previous similar messages INFO: task mount.lustre:27983 blocked for more than 120 seconds. Tainted: P --------------- 2.6.32-504.8.1.el6_lustre.x86_64 #1 "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. mount.lustre D 0000000000000000 0 27983 27982 0x00000080 ffff88005ca1f718 0000000000000082 000000000000ebc8 ffff88005ed70640 ffff88007e4c2aa0 ffff88005ed70080 ffff88005ca1f698 ffffffff810588b3 ffff88007e4c2ad8 ffff880002215900 ffff88005ed70638 ffff88005ca1ffd8 Call Trace: [&lt;ffffffff810588b3&gt;] ? set_next_buddy+0x43/0x50 [&lt;ffffffff81041e98&gt;] ? pvclock_clocksource_read+0x58/0xd0 [&lt;ffffffff8152b185&gt;] schedule_timeout+0x215/0x2e0 [&lt;ffffffff8106d175&gt;] ? enqueue_entity+0x125/0x450 [&lt;ffffffff8105dea4&gt;] ? check_preempt_wakeup+0x1a4/0x260 [&lt;ffffffff8106d59b&gt;] ? enqueue_task_fair+0xfb/0x100 [&lt;ffffffff8152ae03&gt;] wait_for_common+0x123/0x180 [&lt;ffffffff81064b90&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa09127c0&gt;] ? client_lwp_config_process+0x0/0x1912 [obdclass] [&lt;ffffffff8152af1d&gt;] wait_for_completion+0x1d/0x20 [&lt;ffffffffa089ecd4&gt;] llog_process_or_fork+0x354/0x540 [obdclass] [&lt;ffffffffa089eed4&gt;] llog_process+0x14/0x30 [obdclass] [&lt;ffffffffa08cddf4&gt;] class_config_parse_llog+0x1e4/0x330 [obdclass] [&lt;ffffffffa10344f2&gt;] mgc_process_log+0xeb2/0x1970 [mgc] [&lt;ffffffffa102e1f0&gt;] ? mgc_blocking_ast+0x0/0x810 [mgc] [&lt;ffffffffa0ad5800&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] [&lt;ffffffffa1035ef8&gt;] mgc_process_config+0x658/0x1210 [mgc] [&lt;ffffffff81174d4c&gt;] ? __kmalloc+0x21c/0x230 [&lt;ffffffffa08defa3&gt;] lustre_process_log+0x7e3/0x1130 [obdclass] [&lt;ffffffffa0906e9c&gt;] ? server_find_mount+0xbc/0x160 [obdclass] [&lt;ffffffff811749f3&gt;] ? kmem_cache_alloc_trace+0x1b3/0x1c0 [&lt;ffffffffa08dad6f&gt;] ? server_name2fsname+0x6f/0x90 [obdclass] [&lt;ffffffffa090cf56&gt;] server_start_targets+0x12b6/0x1af0 [obdclass] [&lt;ffffffffa08e1bdb&gt;] ? lustre_start_mgc+0x48b/0x1e00 [obdclass] [&lt;ffffffffa08d8fb0&gt;] ? class_config_llog_handler+0x0/0x1a70 [obdclass] [&lt;ffffffffa0911d15&gt;] server_fill_super+0xbe5/0x1690 [obdclass] [&lt;ffffffffa08e3ab0&gt;] lustre_fill_super+0x560/0xa80 [obdclass] [&lt;ffffffffa08e3550&gt;] ? lustre_fill_super+0x0/0xa80 [obdclass] [&lt;ffffffff811917af&gt;] get_sb_nodev+0x5f/0xa0 [&lt;ffffffffa08dab05&gt;] lustre_get_sb+0x25/0x30 [obdclass] [&lt;ffffffff81190deb&gt;] vfs_kern_mount+0x7b/0x1b0 [&lt;ffffffff81190f92&gt;] do_kern_mount+0x52/0x130 [&lt;ffffffff811a3c12&gt;] ? vfs_ioctl+0x22/0xa0 [&lt;ffffffff811b2b9b&gt;] do_mount+0x2fb/0x930 [&lt;ffffffff811b3260&gt;] sys_mount+0x90/0xe0 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b Lustre: 27409:0:(client.c:1939:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1425299784/real 1425299784] req@ffff88005ee57380 x1494534964904508/t0(0) o38-&gt;lustre-MDT0000-lwp-OST0001@10.1.4.205@tcp:12/10 lens 400/544 e 0 to 1 dl 1425299809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 27409:0:(client.c:1939:ptlrpc_expire_one_request()) Skipped 6 previous similar messages LustreError: 137-5: lustre-OST0002_UUID: not available for connect from 10.1.4.198@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 40 previous similar messages Lustre: 27409:0:(client.c:1939:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1425299849/real 1425299849] req@ffff88005c0a69c0 x1494534964904636/t0(0) o38-&gt;lustre-MDT0000-lwp-OST0001@10.1.4.205@tcp:12/10 lens 400/544 e 0 to 1 dl 1425299874 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 27409:0:(client.c:1939:ptlrpc_expire_one_request()) Skipped 7 previous similar messages INFO: task mount.lustre:27983 blocked for more than 120 seconds. Tainted: P --------------- 2.6.32-504.8.1.el6_lustre.x86_64 #1 "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. mount.lustre D 0000000000000000 0 27983 27982 0x00000080 </pre> </div></div> <p>This is blocking ZFS hard failover testing on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/dbcec19e-c81e-11e4-be50-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/dbcec19e-c81e-11e4-be50-5254006e85c2</a></p> <p>Hi Mike,<br/> Can you take a look at this issue? It is a blocker for 2.8.<br/> Thanks.<br/> Joe</p> <p>yes, I am working on this</p> <p>this looks like bug with lw_client detection in target_handle_connect(). I think this patch should do the job:<br/> <a href="http://review.whamcloud.com/#/c/13726/3" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13726/3</a><br/> Meanwhile it is fixed in master by one of DNE2 patches, I will combine both patches to one for 2.7</p> <p>Looks like that patch needs to be rebased to move forward.</p> <p>in fact I think this is fixed already by commit f1d81db9376965e302ecc05e10be220d72b2f04a from <a href="https://jira.whamcloud.com/browse/LU-6629" title="sanity-benchmark test_bonnie: DQACQ failed with -22" class="issue-link" data-issue-key="LU-6629"><del>LU-6629</del></a>, I've just added fix for initial LWP connection logic from my old patch for <a href="https://jira.whamcloud.com/browse/LU-4214" title="Hyperion - OST never recovers on failover node" class="issue-link" data-issue-key="LU-4214"><del>LU-4214</del></a></p> <p>On master branch:</p> <p>RHEL 7.1 with ZFS:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/3d62dc24-48bc-11e5-a657-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/3d62dc24-48bc-11e5-a657-5254006e85c2</a></p> <p>RHEL 6.6 with ZFS:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/6a2783ea-4853-11e5-a4ad-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/6a2783ea-4853-11e5-a4ad-5254006e85c2</a></p> <p>RHEL 7.1 with ldiskfs:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/e8b22732-498a-11e5-8ada-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/e8b22732-498a-11e5-8ada-5254006e85c2</a></p> <p>RHEL 6.6 with ldiskfs:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/1220e9f8-470d-11e5-bfb6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/1220e9f8-470d-11e5-bfb6-5254006e85c2</a></p> <p>The failure also occurred on ZFS test sessions on master branch. Not on ldiskfs test sessions.</p> <p>It looks like OST is unable to switch to the failover MDS after all and keep trying to reach failed MDS. That means problem in its import connection logic or local config doesn't provide information about failover MDS</p> <p>Lustre Build: <a href="https://build.hpdd.intel.com/job/lustre-b2_6/2" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_6/2</a> (2.6.0)<br/> Distro/Arch: RHEL6.5/x86_64<br/> FSTYPE=zfs</p> <p>The failure also occurred on Lustre 2.6.0 release:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/05c8dc02-135f-11e4-92ae-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/05c8dc02-135f-11e4-92ae-5254006e85c2</a> </p> <p>The problem looks still related to the LWP device, it is not able to switch to the failover server for some reason. Meanwhile the local config llog exists on OST and contains failover nid, so problem might be inside LWP code. Also it is not clear why it exists only on ZFS setups.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Call Trace: [&lt;ffffffff81055783&gt;] ? set_next_buddy+0x43/0x50 [&lt;ffffffff8152a595&gt;] schedule_timeout+0x215/0x2e0 [&lt;ffffffff81069f15&gt;] ? enqueue_entity+0x125/0x450 [&lt;ffffffff8152a213&gt;] wait_for_common+0x123/0x180 [&lt;ffffffff81061d00&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa090cd00&gt;] ? client_lwp_config_process+0x0/0x1948 [obdclass] [&lt;ffffffff8152a32d&gt;] wait_for_completion+0x1d/0x20 [&lt;ffffffffa0898e14&gt;] llog_process_or_fork+0x354/0x540 [obdclass] [&lt;ffffffffa0899014&gt;] llog_process+0x14/0x30 [obdclass] [&lt;ffffffffa08c81d4&gt;] class_config_parse_llog+0x1e4/0x330 [obdclass] [&lt;ffffffffa10314f2&gt;] mgc_process_log+0xeb2/0x1970 [mgc] [&lt;ffffffffa102b1f0&gt;] ? mgc_blocking_ast+0x0/0x810 [mgc] [&lt;ffffffffa0ad0860&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] [&lt;ffffffffa1032ef8&gt;] mgc_process_config+0x658/0x1210 [mgc] [&lt;ffffffffa08d9383&gt;] lustre_process_log+0x7e3/0x1130 [obdclass] [&lt;ffffffffa07891c1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa08d514f&gt;] ? server_name2fsname+0x6f/0x90 [obdclass] [&lt;ffffffffa0907496&gt;] server_start_targets+0x12b6/0x1af0 [obdclass] [&lt;ffffffffa0783818&gt;] ? libcfs_log_return+0x28/0x40 [libcfs] [&lt;ffffffffa08dbfe6&gt;] ? lustre_start_mgc+0x4b6/0x1e00 [obdclass] [&lt;ffffffffa07891c1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa08d3390&gt;] ? class_config_llog_handler+0x0/0x1a70 [obdclass] [&lt;ffffffffa090c255&gt;] server_fill_super+0xbe5/0x1690 [obdclass] [&lt;ffffffffa0783818&gt;] ? libcfs_log_return+0x28/0x40 [libcfs] [&lt;ffffffffa08dde90&gt;] lustre_fill_super+0x560/0xa80 [obdclass] [&lt;ffffffffa08dd930&gt;] ? lustre_fill_super+0x0/0xa80 [obdclass] [&lt;ffffffff8118c56f&gt;] get_sb_nodev+0x5f/0xa0 [&lt;ffffffffa08d4ee5&gt;] lustre_get_sb+0x25/0x30 [obdclass] [&lt;ffffffff8118bbcb&gt;] vfs_kern_mount+0x7b/0x1b0 [&lt;ffffffff8118bd72&gt;] do_kern_mount+0x52/0x130 [&lt;ffffffff8119e972&gt;] ? vfs_ioctl+0x22/0xa0 [&lt;ffffffff811ad74b&gt;] do_mount+0x2fb/0x930 [&lt;ffffffff811ade10&gt;] sys_mount+0x90/0xe0 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>The mount thread is waiting on processing log records, let's see what the log process thread is doing:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">llog_process_ S 0000000000000001 0 28768 2 0x00000080 ffff880058535970 0000000000000046 ffff880058535960 ffffffffa0c04f4c ffff8800585359d0 000000007f823240 0000000000000000 ffff8800590f4511 0000004958535910 ffffffffa0cccece ffff880058f285f8 ffff880058535fd8 Call Trace: [&lt;ffffffffa0c04f4c&gt;] ? ptlrpc_unregister_reply+0x6c/0x810 [ptlrpc] [&lt;ffffffff8152b222&gt;] schedule_timeout+0x192/0x2e0 [&lt;ffffffff81087540&gt;] ? process_timeout+0x0/0x10 [&lt;ffffffffa0c07ba9&gt;] ptlrpc_set_wait+0x319/0xa20 [ptlrpc] [&lt;ffffffffa0bfd2c0&gt;] ? ptlrpc_interrupted_set+0x0/0x110 [ptlrpc] [&lt;ffffffff81064c00&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0c13da5&gt;] ? lustre_msg_set_jobid+0xf5/0x130 [ptlrpc] [&lt;ffffffffa0c08331&gt;] ptlrpc_queue_wait+0x81/0x220 [ptlrpc] [&lt;ffffffffa0e7455b&gt;] fld_client_rpc+0x15b/0x510 [fld] [&lt;ffffffff81297869&gt;] ? simple_strtoul+0x9/0x10 [&lt;ffffffffa0e7b024&gt;] fld_update_from_controller+0x1c4/0x570 [fld] [&lt;ffffffffa124f8e8&gt;] ofd_register_lwp_callback+0xa8/0x5a0 [ofd] [&lt;ffffffffa0a0c84f&gt;] lustre_lwp_connect+0xacf/0xd10 [obdclass] [&lt;ffffffffa0a0db25&gt;] lustre_lwp_setup+0x8c5/0xc60 [obdclass] [&lt;ffffffffa09dbc08&gt;] ? target_name2index+0x78/0xc0 [obdclass] [&lt;ffffffffa0a0fa07&gt;] client_lwp_config_process+0x1357/0x1da0 [obdclass] [&lt;ffffffffa099c35a&gt;] llog_process_thread+0x94a/0xfc0 [obdclass] [&lt;ffffffffa099d515&gt;] llog_process_thread_daemonize+0x45/0x70 [obdclass] [&lt;ffffffffa099d4d0&gt;] ? llog_process_thread_daemonize+0x0/0x70 [obdclass] [&lt;ffffffff8109e78e&gt;] kthread+0x9e/0xc0 [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 [&lt;ffffffff8109e6f0&gt;] ? kthread+0x0/0xc0 [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>It's trying to query FLDB from MDT0, but unfortunately the FLD RPC timedout somehow. Though I don't quite sure the timeout reason, I think we shouldn't do that in ofd_register_lwp_callback() from the beginning, because that'll block the OST mount process when the MDT0 isn't started.</p> <p>Di, could you take a look? Is there anyway to avert that problem?</p> <p>I see why the LWP on OST can't connect to the failover MDT now:</p> <p>On OST mount, we process the client log to setup LWP as following:<br/> 1&gt; Process LCFG_ADD_UUID record to setup LWP device, then connect to the MDT. (see lustre_lwp_setup());<br/> 2&gt; Process LCFG_ADD_CONN record to add failover connection;</p> <p>We can see if the mount process is blocked on step 1, then it will never have a chance to add failover connection, and LWP won't be able to switch to failover node forever.</p> <p>Unfortunately, the process could be blocked on step 1, see lustre_lwp_setup() -&gt; lustre_lwp_connect():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> rc = obd_connect(&amp;env, &amp;exp, lwp, uuid, data, NULL); <span class="code-keyword">if</span> (rc != 0) { CERROR(<span class="code-quote">"%s: connect failed: rc = %d\n"</span>, lwp-&gt;obd_name, rc); } <span class="code-keyword">else</span> { <span class="code-keyword">if</span> (unlikely(lwp-&gt;obd_lwp_export != NULL)) class_export_put(lwp-&gt;obd_lwp_export); lwp-&gt;obd_lwp_export = class_export_get(exp); lustre_notify_lwp_list(exp); } </pre> </div></div> <p>lustre_notify_lwp_list() is called after obd_connect() is called successfully (which means CONNECT can be sent successfully, but not connection established), lustre_notify_lwp_list calls ofd_register_lwp_callback() to send FLD RPC, please note that it's now try to send RPC to the primary MDT0, (connection of failover MDT0 hasn't been added yet).</p> <p>Let look at fld_client_rpc():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (rc != 0) { <span class="code-keyword">if</span> (imp-&gt;imp_state != LUSTRE_IMP_CLOSED &amp;&amp; !imp-&gt;imp_deactive) { /* Since LWP is not replayable, so it will keep * trying unless umount happens, otherwise it would * cause unecessary failure of the application. */ ptlrpc_req_finished(req); rc = 0; <span class="code-keyword">goto</span> again; } GOTO(out_req, rc); } </pre> </div></div> <p>It's trying to send the RPC to primary MDT0 again and again in a loop... At the end, the mount process is blocked here forever (before the failover connection being added).</p> <p>I'd think further on how to resolve this. Di, any thoughts? Thanks.</p> <p>Hmm, I actually met this problem in one of my patch test<br/> <a href="http://review.whamcloud.com/#/c/15275/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15275/</a> (patch set 7)</p> <p>See this failure <a href="https://testing.hpdd.intel.com/test_sets/e2a60dca-3ff5-11e5-afd4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e2a60dca-3ff5-11e5-afd4-5254006e85c2</a></p> <p>we probably should set no_delay and allow_replay flag for update FLD RPC. So I think this patch should fix the problem</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/lustre/fld/fld_request.c b/lustre/fld/fld_request.c index a59ab95..c82b098 100644 --- a/lustre/fld/fld_request.c +++ b/lustre/fld/fld_request.c @@ -399,6 +399,14 @@ again: req_capsule_set_size(&amp;req-&gt;rq_pill, &amp;RMF_GENERIC_DATA, RCL_SERVER, PAGE_CACHE_SIZE); + + /* This might happen before the import state becomes to FULL, + * let's set allow_replay for this request to avoid deadlock + * see LU-6273 */ + req-&gt;rq_allow_replay = 1; + /* This will always use LWP connection, let's send the req + * with no_delay flags, see above */ + req-&gt;rq_no_delay = 1; break; default: rc = -EINVAL; </pre> </div></div> <p>I will push the fix to <a href="http://review.whamcloud.com/#/c/15275/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15275/</a></p> <p>yes, that looks like the reason, but I still wonder why ldiskfs setups are not affected. Di, IIRC, you did local FLDB in past, shouldn't OST have a local copy of FLDB?</p> <p>I discussed with niu a bit, and this patch is probably not enough according to niu's comment above. We probably should move this fled_update_FLDB to a separate thread, otherwise the LWP config log analyze will not continue, and LWP will never be able to connect. Yes, there is FLDB locally on each target, this FLDB will only happens during first initialization (checking lsf_new flags) or upgrade from 2.4 (DNE env) to the current DNE version (2.8). And I do not know why it only happen on ZFS. It maybe because fld_index_init is slow on ZFS, so OST is restart before fld_index_init finish.</p> <p>I think we can just move the lustre_notify_lwp_list() to the lustre_start_lwp(), right after lsi_lwp_started is set to 1. At this point the llog is processed and lwp should be fully setup.</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16303" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16303</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6273" title="Hard Failover replay-dual test_17: Failover OST mount hang" class="issue-link" data-issue-key="LU-6273"><del>LU-6273</del></a> fld: update local FLDB in a separate thread<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 92feb9161586f00cc03c481c81598f3b77847fb3</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>I think we can just move the lustre_notify_lwp_list() to the lustre_start_lwp(), right after lsi_lwp_started is set to 1. At this point the llog is processed and lwp should be fully setup. </pre> </div></div> <p>No, lustre_start_lwp does not promise LWP will be setup accordingly, because mdt0 might not be added to the client config log yet, for example, OST might be started before MDT0.</p> <p>Mike Pershin (mike.pershin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16304" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16304</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6273" title="Hard Failover replay-dual test_17: Failover OST mount hang" class="issue-link" data-issue-key="LU-6273"><del>LU-6273</del></a> lwp: notify LWP is ready after llog processing<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1be148d882a209137e92e525bc69b601e114646c</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16303/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16303/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6273" title="Hard Failover replay-dual test_17: Failover OST mount hang" class="issue-link" data-issue-key="LU-6273"><del>LU-6273</del></a> lwp: notify LWP users in dedicated thread<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b1848aa5b23fd332362e9ae3d5aab31d8dd9d920</p> <p>Landed for 2.8</p> Related LU-4214 Development Rank 1|hzx6xb: Rank (Obsolete) 17590 Severity