https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10961 Lustre <p>We are seeing repeated hard hang on clients after server failover. <br/> 'df' on a client will hang, user tasks do no complete. So far no hard faults, the node just grinds to a halt. Yesterday this occurred on soak-17 and soak-23. I have dumped stacks on both nodes, and crash dumps are available on soak. <br/> We see: </p> <ul class="alternate" type="square"> <li>connections to one or more osts drop, and the client does not re-connect: <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Apr 27 03:28:42 soak-23 kernel: Lustre: 2084:0:(client.c:2099:ptlrpc_expire_one_request()) @@@ Request sent has timed out <span class="code-keyword">for</span> sent delay: [sent 1524799714/real 0] req@ffff8808fee67500 x1598738343197024/t0(0) o400-&gt;soaked-OST000b-osc-ffff8807f6ba0800@192.168.1.107@o2ib:28/4 lens 224/224 e 0 to 1 dl 1524799721 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Apr 27 03:28:42 soak-23 kernel: Lustre: soaked-OST0011-osc-ffff8807f6ba0800: Connection to soaked-OST0011 (at 192.168.1.107@o2ib) was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete Apr 27 03:28:42 soak-23 kernel: Lustre: Skipped 3 previous similar messages Apr 27 03:28:42 soak-23 kernel: Lustre: 2084:0:(client.c:2099:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Apr 27 03:28:42 soak-23 kernel: Lustre: soaked-OST000b-osc-ffff8807f6ba0800: Connection to soaked-OST000b (at 192.168.1.107@o2ib) was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete </pre> </div></div> <p>As of 1700 hours (14 hours after failover) the node still has not reconnected to this OST.</p></li> </ul> <p>We also see repeated errors referencing the MDT:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Apr 27 17:25:42 soak-23 kernel: LustreError: 223668:0:(file.c:4213:ll_inode_revalidate_fini()) soaked: revalidate FID [0x200000007:0x1:0x0] error: rc = -4 </pre> </div></div> <p>The error appears very repeatable. Logs and stack traces are attached. </p> soak cluster LU-10961

Clients hang after failovers. LustreError: 223668:0:(file.c:4213:ll_inode_revalidate_fini()) soaked: revalidate FID [0x200000007:0x1:0x0] error: rc = -4

Bug Blocker Resolved Fixed Mikhail Pershin Cliff White soak Fri, 27 Apr 2018 17:54:10 +0000 Mon, 10 Sep 2018 17:43:38 +0000 Mon, 10 Sep 2018 17:43:38 +0000 Lustre 2.12.0 Lustre 2.12.0 0 10 <p>During the weekend, most of the clients hung. Issued the command 'umount -f' all clients except soak-24 successfully un-mounted Lustre. <br/> Dumped stacks, lustre log and crash dumped soak-24, stacks and lustre-log attached. <br/> At this point this is seriously hampering soak testing. </p> <p>Some new errors in the latest hang:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[431310.219421] traps: lfs[157180] trap divide error ip:40dc1a sp:7ffda599e940 error:0 in lfs[400000+20000] [431346.590086] traps: lfs[157395] trap divide error ip:40dc1a sp:7ffee73a3900 error:0 in lfs[400000+20000] [431385.007477] Lustre: 154503:0:(client.c:2099:ptlrpc_expire_one_request()) @@@ Request sent has timed out <span class="code-keyword">for</span> sent delay: [sent 1525103886/real 0] req@ffff8805bcb47200 x1599185360888640/t0(0) o4-&gt;soaked-MDT0000-mdc-ffff880c502a0000@192.168.1.108@o2ib:13/10 lens 608/448 e 0 to 1 dl 1525103893 ref 3 fl Rpc:X/0/ffffffff rc 0/-1 </pre> </div></div> <p>Seeing this error again on current master. version=2.11.52_28_ge2d7e67</p> <p>More logs. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[12385.187946] LustreError: 22349:0:(osc_cache.c:955:osc_extent_wait()) extent ffff9997c6151550@{[0 -&gt; 0/255], [3|0|+|rpc|wihY|ffff9998b4fb5040], [28672|1|+|-|ffff999a22957980|256|ffff999b27449fa0]} soaked-MDT0000-mdc-ffff999f58e9f000: wait ext to 0 timedout, recovery in progress? [12385.187949] LustreError: 12910:0:(osc_cache.c:955:osc_extent_wait()) extent ffff9997c6151130@{[0 -&gt; 0/255], [2|0|+|cache|wihuY|ffff9998b4fb75c0], [28672|1|+|-|ffff999a22957bc0|256| (<span class="code-keyword">null</span>)]} soaked-MDT0000-mdc-ffff999f58e9f000: wait ext to 0 timedout, recovery in progress? [12385.187951] LustreError: 12901:0:(osc_cache.c:955:osc_extent_wait()) extent ffff9997c6150e70@{[0 -&gt; 0/255], [3|0|+|rpc|wihY|ffff9998b4fb5400], [28672|1|+|-|ffff99998672c140|256|ffff9997bfc20000]} soaked-MDT0000-mdc-ffff999f58e9f000: wait ext to 0 timedout, recovery in progress? [12385.187953] LustreError: 12749:0:(osc_cache.c:955:osc_extent_wait()) ### extent: ffff9997c61500b0 ns: soaked-MDT0000-mdc-ffff999f58e9f000 lock: ffff999986729b00/0x9ba73c642656dc6b lrc: 4/0,0 mode: PW/PW res: [0x200010e09:0x8ccb:0x0].0x0 bits 0x40/0x40 rrc: 2 type: IBT flags: 0x449400000000 nid: local remote: 0xaa9e212e88be10ed expref: -99 pid: 24926 timeout: 0 lvb_type: 0 [12385.187955] LustreError: 22349:0:(osc_cache.c:955:osc_extent_wait()) ### extent: ffff9997c6151550 ns: soaked-MDT0000-mdc-ffff999f58e9f000 lock: ffff999a22957980/0x9ba73c642656dd13 lrc: 4/0,0 mode: PW/PW res: [0x200010e09:0x8cd0:0x0].0x0 bits 0x40/0x40 rrc: 2 type: IBT flags: 0x449400000000 nid: local remote: 0xaa9e212e88be2a07 expref: -99 pid: 24926 timeout: 0 lvb_type: 0 [12385.187958] LustreError: 12910:0:(osc_cache.c:955:osc_extent_wait()) ### extent: ffff9997c6151130 ns: soaked-MDT0000-mdc-ffff999f58e9f000 lock: ffff999a22957bc0/0x9ba73c642656dd8a lrc: 4/0,0 mode: PW/PW res: [0x200010e09:0x8cd5:0x0].0x0 bits 0x40/0x40 rrc: 2 type: IBT flags: 0x449400000000 nid: local remote: 0xaa9e212e88be408f expref: -99 pid: 24926 timeout: 0 lvb_type: 0 [12385.187960] LustreError: 12901:0:(osc_cache.c:955:osc_extent_wait()) ### extent: ffff9997c6150e70 ns: soaked-MDT0000-mdc-ffff999f58e9f000 lock: ffff99998672c140/0x9ba73c642656dcaa lrc: 4/0,0 mode: PW/PW res: [0x200010e09:0x8ccd:0x0].0x0 bits 0x40/0x40 rrc: 2 type: IBT flags: 0x449400000000 nid: local remote: 0xaa9e212e88be18bf expref: -99 pid: 24926 timeout: 0 lvb_type: 0 [12385.187980] LustreError: 12751:0:(osc_cache.c:955:osc_extent_wait()) extent ffff9997c6150420@{[0 -&gt; 0/255], [2|0|+|cache|wihuY|ffff9998b4fb4000], [28672|1|+|-|ffff999a22954800|256| (<span class="code-keyword">null</span>)]} soaked-MDT0000-mdc-ffff999f58e9f000: wait ext to 0 timedout, recovery in progress? [12385.187986] LustreError: 12751:0:(osc_cache.c:955:osc_extent_wait()) ### extent: ffff9997c6150420 ns: soaked-MDT0000-mdc-ffff999f58e9f000 lock: ffff999a22954800/0x9ba73c642656dd6e lrc: 4/0,0 mode: PW/PW res: [0x200010e09:0x8cd4:0x0].0x0 bits 0x40/0x40 rrc: 2 type: IBT flags: 0x449400000000 nid: local remote: 0xaa9e212e88be3965 expref: -99 pid: 24926 timeout: 0 lvb_type: 0 [12385.188015] LustreError: 16607:0:(osc_cache.c:955:osc_extent_wait()) extent ffff9997c6150a50@{[0 -&gt; 0/255], [3|0|+|rpc|wihY|ffff9998b4fb7c00], [28672|1|+|-|ffff99998672ad00|256|ffff9997bfc26eb0]} soaked-MDT0000-mdc-ffff999f58e9f000: wait ext to 0 timedout, recovery in progress? [12385.188021] LustreError: 16607:0:(osc_cache.c:955:osc_extent_wait()) ### extent: ffff9997c6150a50 ns: soaked-MDT0000-mdc-ffff999f58e9f000 lock: ffff99998672ad00/0x9ba73c642656dc02 lrc: 4/0,0 mode: PW/PW res: [0x200010e09:0x8cc7:0x0].0x0 bits 0x40/0x40 rrc: 2 type: IBT flags: 0x449400000000 nid: local remote: 0xaa9e212e88be004d expref: -99 pid: 24926 timeout: 0 lvb_type: 0 [12385.188032] LustreError: 16605:0:(osc_cache.c:955:osc_extent_wait()) extent ffff9997c6151d90@{[0 -&gt; 0/255], [3|0|+|rpc|wihY|ffff9998b4fb4a00], [28672|1|+|-|ffff99998672e0c0|256|ffff999b6bb62f70]} soaked-MDT0000-mdc-ffff999f58e9f000: wait ext to 0 timedout, recovery in progress? [12385.188038] LustreError: 16605:0:(osc_cache.c:955:osc_extent_wait()) ### extent: ffff9997c6151d90 ns: soaked-MDT0000-mdc-ffff999f58e9f000 lock: ffff99998672e0c0/0x9ba73c642656dc48 lrc: 4/0,0 mode: PW/PW res: [0x200010e09:0x8cc9:0x0].0x0 bits 0x40/0x40 rrc: 2 type: IBT flags: 0x449400000000 nid: local remote: 0xaa9e212e88be0adb expref: -99 pid: 24926 timeout: 0 lvb_type: 0 [12385.188259] LustreError: 12677:0:(osc_cache.c:955:osc_extent_wait()) extent ffff9997c6150210@{[0 -&gt; 0/255], [3|0|+|rpc|wihY|ffff9998b4fb6800], [28672|1|+|-|ffff99998672d580|256|ffff999b6b272f70]} soaked-MDT0000-mdc-ffff999f58e9f000: wait ext to 0 timedout, recovery in progress? [12385.188264] LustreError: 12677:0:(osc_cache.c:955:osc_extent_wait()) ### extent: ffff9997c6150210 ns: soaked-MDT0000-mdc-ffff999f58e9f000 lock: ffff99998672d580/0x9ba73c642656dbd1 lrc: 4/0,0 mode: PW/PW res: [0x200010e09:0x8cc5:0x0].0x0 bits 0x40/0x40 rrc: 2 type: IBT flags: 0x449400000000 nid: local remote: 0xaa9e212e88bdf6b4 expref: -99 pid: 24926 timeout: 0 lvb_type: 0 [12385.188911] LustreError: 12746:0:(osc_cache.c:955:osc_extent_wait()) extent ffff9997c6151340@{[0 -&gt; 0/255], [3|0|+|rpc|wihY|ffff9998b4fb6300], [28672|1|+|-|ffff999a22956540|256|ffff999f5b654f10]} soaked-MDT0000-mdc-ffff999f58e9f000: wait ext to 0 timedout, recovery in progress? [12385.188918] LustreError: 12746:0:(osc_cache.c:955:osc_extent_wait()) ### extent: ffff9997c6151340 ns: soaked-MDT0000-mdc-ffff999f58e9f000 lock: ffff999a22956540/0x9ba73c642656ddd7 lrc: 4/0,0 mode: PW/PW res: [0x200010e09:0x8cd7:0x0].0x0 bits 0x40/0x40 rrc: 2 type: IBT flags: 0x449400000000 nid: local remote: 0xaa9e212e88be48d1 expref: -99 pid: 24926 timeout: 0 lvb_type: 0 [12388.654048] LustreError: 12904:0:(osc_cache.c:955:osc_extent_wait()) ### extent: ffff9997c61502c0 ns: soaked-MDT0000-mdc-ffff999f58e9f000 lock: ffff999986728d80/0x9ba73c642656dc25 lrc: 4/0,0 mode: PW/PW res: [0x200010e09:0x8cc8:0x0].0x0 bits 0x40/0x40 rrc: 2 type: IBT flags: 0x449400000000 nid: local remote: 0xaa9e212e88be05da expref: -99 pid: 24926 timeout: 0 lvb_type: 0 [12484.834610] INFO: task simul:25858 blocked <span class="code-keyword">for</span> more than 120 seconds. [12484.857231] <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. [12484.884393] simul D ffff999910485ee0 0 25858 25851 0x00000080 [12484.908994] Call Trace: [12484.918386] [&lt;ffffffffbb029e72&gt;] ? path_lookupat+0x122/0x8b0 [12484.938662] [&lt;ffffffffbb514e99&gt;] schedule_preempt_disabled+0x29/0x70 [12484.961219] [&lt;ffffffffbb512c57&gt;] __mutex_lock_slowpath+0xc7/0x1d0 [12484.982900] [&lt;ffffffffbb51203f&gt;] mutex_lock+0x1f/0x2f [12485.001116] [&lt;ffffffffbb0272d0&gt;] lock_rename+0xc0/0xe0 [12485.019600] [&lt;ffffffffbb02d39f&gt;] SYSC_renameat2+0x22f/0x5a0 [12485.039503] [&lt;ffffffffbaed849c&gt;] ? update_curr+0x14c/0x1e0 [12485.059063] [&lt;ffffffffbaed62dc&gt;] ? set_next_entity+0x3c/0xe0 [12485.079200] [&lt;ffffffffbb51367e&gt;] ? __schedule+0x14e/0xa20 [12485.098433] [&lt;ffffffffbb52077b&gt;] ? system_call_after_swapgs+0xc8/0x160 [12485.121395] [&lt;ffffffffbb52076f&gt;] ? system_call_after_swapgs+0xbc/0x160 [12485.144330] [&lt;ffffffffbb52077b&gt;] ? system_call_after_swapgs+0xc8/0x160 [12485.167214] [&lt;ffffffffbb52076f&gt;] ? system_call_after_swapgs+0xbc/0x160 [12485.190098] [&lt;ffffffffbb02e58e&gt;] SyS_renameat2+0xe/0x10 [12485.208651] [&lt;ffffffffbb02e5ce&gt;] SyS_rename+0x1e/0x20 [12485.226616] [&lt;ffffffffbb52082f&gt;] system_call_fastpath+0x1c/0x21 [12485.247435] [&lt;ffffffffbb52077b&gt;] ? system_call_after_swapgs+0xc8/0x160 [12605.265688] INFO: task simul:25858 blocked <span class="code-keyword">for</span> more than 120 seconds. [12605.343807] <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. [12605.438557] simul D ffff999910485ee0 0 25858 25851 0x00000080 [12605.524156] Call Trace: [12605.554383] [&lt;ffffffffbb029e72&gt;] ? path_lookupat+0x122/0x8b0 [12605.624122] [&lt;ffffffffbb514e99&gt;] schedule_preempt_disabled+0x29/0x70 [12605.702164] [&lt;ffffffffbb512c57&gt;] __mutex_lock_slowpath+0xc7/0x1d0 [12605.777070] [&lt;ffffffffbb51203f&gt;] mutex_lock+0x1f/0x2f [12605.839473] [&lt;ffffffffbb0272d0&gt;] lock_rename+0xc0/0xe0 [12605.902894] [&lt;ffffffffbb02d39f&gt;] SYSC_renameat2+0x22f/0x5a0 [12605.971484] [&lt;ffffffffbaed849c&gt;] ? update_curr+0x14c/0x1e0 [12606.039000] [&lt;ffffffffbaed62dc&gt;] ? set_next_entity+0x3c/0xe0 [12606.108587] [&lt;ffffffffbb51367e&gt;] ? __schedule+0x14e/0xa20 [12606.175047] [&lt;ffffffffbb52077b&gt;] ? system_call_after_swapgs+0xc8/0x160 [12606.254985] [&lt;ffffffffbb52076f&gt;] ? system_call_after_swapgs+0xbc/0x160 [12606.334925] [&lt;ffffffffbb52077b&gt;] ? system_call_after_swapgs+0xc8/0x160 [12606.414838] [&lt;ffffffffbb52076f&gt;] ? system_call_after_swapgs+0xbc/0x160 [12606.494700] [&lt;ffffffffbb02e58e&gt;] SyS_renameat2+0xe/0x10 [12606.558964] [&lt;ffffffffbb02e5ce&gt;] SyS_rename+0x1e/0x20 [12606.621092] [&lt;ffffffffbb52082f&gt;] system_call_fastpath+0x1c/0x21 [12606.693633] [&lt;ffffffffbb52077b&gt;] ? system_call_after_swapgs+0xc8/0x160 [12726.768761] INFO: task simul:25858 blocked <span class="code-keyword">for</span> more than 120 seconds. [12726.776720] <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. [12726.786099] simul D ffff999910485ee0 0 25858 25851 0x00000080 [12726.794629] Call Trace: [12726.794629] Call Trace: [12726.797974] [&lt;ffffffffbb029e72&gt;] ? path_lookupat+0x122/0x8b0 [12726.804992] [&lt;ffffffffbb514e99&gt;] schedule_preempt_disabled+0x29/0x70 [12726.812770] [&lt;ffffffffbb512c57&gt;] __mutex_lock_slowpath+0xc7/0x1d0 [12726.820246] [&lt;ffffffffbb51203f&gt;] mutex_lock+0x1f/0x2f [12726.826543] [&lt;ffffffffbb0272d0&gt;] lock_rename+0xc0/0xe0 [12726.832932] [&lt;ffffffffbb02d39f&gt;] SYSC_renameat2+0x22f/0x5a0 [12726.839803] [&lt;ffffffffbaed849c&gt;] ? update_curr+0x14c/0x1e0 [12726.846573] [&lt;ffffffffbaed62dc&gt;] ? set_next_entity+0x3c/0xe0 [12726.853530] [&lt;ffffffffbb51367e&gt;] ? __schedule+0x14e/0xa20 [12726.860202] [&lt;ffffffffbb52077b&gt;] ? system_call_after_swapgs+0xc8/0x160 [12726.868137] [&lt;ffffffffbb52076f&gt;] ? system_call_after_swapgs+0xbc/0x160 [12726.876068] [&lt;ffffffffbb52077b&gt;] ? system_call_after_swapgs+0xc8/0x160 [12726.883992] [&lt;ffffffffbb52076f&gt;] ? system_call_after_swapgs+0xbc/0x160 [12726.891915] [&lt;ffffffffbb02e58e&gt;] SyS_renameat2+0xe/0x10 [12726.898373] [&lt;ffffffffbb02e5ce&gt;] SyS_rename+0x1e/0x20 [12726.904628] [&lt;ffffffffbb52082f&gt;] system_call_fastpath+0x1c/0x21 [12726.911868] [&lt;ffffffffbb52077b&gt;] ? system_call_after_swapgs+0xc8/0x160 [12846.915800] INFO: task simul:25858 blocked <span class="code-keyword">for</span> more than 120 seconds. [12846.937671] <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. [12846.964035] simul D ffff999910485ee0 0 25858 25851 0x00000080 [12846.987832] Call Trace: [12846.996459] [&lt;ffffffffbb029e72&gt;] ? path_lookupat+0x122/0x8b0 [12847.015963] [&lt;ffffffffbb514e99&gt;] schedule_preempt_disabled+0x29/0x70 [12847.037750] [&lt;ffffffffbb512c57&gt;] __mutex_lock_slowpath+0xc7/0x1d0 [12847.058669] [&lt;ffffffffbb51203f&gt;] mutex_lock+0x1f/0x2f [12847.076143] [&lt;ffffffffbb0272d0&gt;] lock_rename+0xc0/0xe0 [12847.093894] [&lt;ffffffffbb02d39f&gt;] SYSC_renameat2+0x22f/0x5a0 [12847.113076] [&lt;ffffffffbaed849c&gt;] ? update_curr+0x14c/0x1e0 [12847.131964] [&lt;ffffffffbaed62dc&gt;] ? set_next_entity+0x3c/0xe0 [12847.151421] [&lt;ffffffffbb51367e&gt;] ? __schedule+0x14e/0xa20 [12847.170022] [&lt;ffffffffbb52077b&gt;] ? system_call_after_swapgs+0xc8/0x160 [12847.192344] [&lt;ffffffffbb52076f&gt;] ? system_call_after_swapgs+0xbc/0x160 [12847.214662] [&lt;ffffffffbb52077b&gt;] ? system_call_after_swapgs+0xc8/0x160 [12847.236971] [&lt;ffffffffbb52076f&gt;] ? system_call_after_swapgs+0xbc/0x160 [12847.259272] [&lt;ffffffffbb02e58e&gt;] SyS_renameat2+0xe/0x10 [12847.277283] [&lt;ffffffffbb02e5ce&gt;] SyS_rename+0x1e/0x20 [12847.294717] [&lt;ffffffffbb52082f&gt;] system_call_fastpath+0x1c/0x21 [12847.315019] [&lt;ffffffffbb52077b&gt;] ? system_call_after_swapgs+0xc8/0x160 12977.554496] LustreError: 12274:0:(osc_cache.c:955:osc_extent_wait()) extent ffff9997c6151290@{[0 -&gt; 0/255], [2|0|+|cache|wihuY|ffff9998b4fb6940], [28672|1|+|-|ffff99998672b180|256| (<span class="code-keyword">null</span>)]} soaked-MDT0000-mdc-ffff999f58e9f000: wait ext to 0 timedout, recovery in progress? [12977.587096] LustreError: 12274:0:(osc_cache.c:955:osc_extent_wait()) ### extent: ffff9997c6151290 ns: soaked-MDT0000-mdc-ffff999f58e9f000 lock: ffff99998672b180/0x9ba73c642656db99 lrc: 3/0,0 mode: PW/PW res: [0x200010e09:0x8cc4:0x0].0x0 bits 0x40/0x0 rrc: 2 type: IBT flags: 0x49400000000 nid: local remote: 0xaa9e212e88bdf0a2 expref: -99 pid: 24926 timeout: 0 lvb_type: 0 [24406.001241] LustreError: 22890:0:(vvp_io.c:1495:vvp_io_init()) soaked: refresh file layout [0x200010dc3:0x1d336:0x0] error -4. [25696.331238] LustreError: 24884:0:(file.c:4213:ll_inode_revalidate_fini()) soaked: revalidate FID [0x200000401:0x2:0x0] error: rc = -4 [27088.300797] perf: interrupt took too <span class="code-object">long</span> (5045 &gt; 4917), lowering kernel.perf_event_max_sample_rate to 39000 [62130.364898] perf: interrupt took too <span class="code-object">long</span> (6328 &gt; 6306), lowering kernel.perf_event_max_sample_rate to 31000 </pre> </div></div> <p>Dmitry</p> <p>Could you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>Working backwards to find a 'good' version. Seeing the error once in 2.10.58_139_g630cd49 with a different result code. (EIO instead of EINTR)<br/> [ 8145.869669] LustreError: 7130:0:(file.c:4213:ll_inode_revalidate_fini()) soaked: revalidate FID <span class="error">&#91;0x20002b381:0x341:0x0&#93;</span> error: rc = -5</p> <p>After several connection change during recovery it hangs with following states:<br/> MGS -&gt; CONNECTING<br/> MDT -&gt; REPLAY_LOCKS<br/> OST -&gt; CONNECTING</p> <p>The last recovery restart was for MGS but it looks REPLAY_LOCKS was not complete.</p> <p> </p> <p>So, is there a fix, or something to be done? Is the more data you require? This is halting soak testing ATM</p> <p>I cannot find out the root cause yet. I'm looking into logs and try to understand what was wrong.</p> <p>Mingwile could you check the value of following param:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lctl get_param osc.*.pinger_recov</pre> </div></div> <p>What activity was on this cluster during this hangs?</p> <p> </p> <p>Cluster was running a large mix of jobs for the stress tests. multiple failovers were occurring</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">]# lctl get_param osc.*.pinger_recov osc.soaked-OST0000-osc-ffff880823116800.pinger_recov=1 osc.soaked-OST0001-osc-ffff880823116800.pinger_recov=1 osc.soaked-OST0002-osc-ffff880823116800.pinger_recov=1 osc.soaked-OST0003-osc-ffff880823116800.pinger_recov=1 osc.soaked-OST0004-osc-ffff880823116800.pinger_recov=1 osc.soaked-OST0005-osc-ffff880823116800.pinger_recov=1 </pre> </div></div> <p>I can repeat the error and crash dump the systems, if that will help</p> <p>It would be good if you can increase the level of debug logs during this issue.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lctl set_param debug=<span class="code-quote">"+error +warning +info"</span> debug_mb=1024 </pre> </div></div> <p>Repeated the error with more debug, lustre log attached. There is a crash dump available on spirit - I dumped stack and forced a crash dump on a client. </p> <p>I sm testing today without MDS failures (OSS failover only) I believe this is triggered by MDS failover.</p> <p>Ran successfully for 24 without MDS failover - So, i believe the problem is only triggered by MDS failover. <br/> Next steps?</p> <p> I see the recovery process and locks replay.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">(<span class="code-keyword">import</span>.c:1498:ptlrpc_import_recovery_state_machine()) replay requested by soaked-MDT0000_UUID (recover.c:88:ptlrpc_replay_next()) <span class="code-keyword">import</span> ffff8804024da000 from soaked-MDT0000_UUID committed 25770229782 last 25770229782 (<span class="code-keyword">import</span>.c:1502:ptlrpc_import_recovery_state_machine()) ffff8804024da000 soaked-MDT0000_UUID: changing <span class="code-keyword">import</span> state from REPLAY to REPLAY_LOCKS </pre> </div></div> <p>But then it looks unexpected communication:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">(osc_cache.c:955:osc_extent_wait()) extent ffff880412a6be40@{[0 -&gt; 0/255], [3|0|+|rpc|wihY|ffff880058d16bc0], [28672|1|+|-|ffff8803ce09f500|256|ffff880827146eb0]} soaked-MDT0000-mdc-ffff88083f83d000: wait ext to 0 timedout, recovery in progress? (osc_cache.c:955:osc_extent_wait()) ### extent: ffff880412a6be40 ns: soaked-MDT0000-mdc-ffff88083f83d000 lock: ffff8803ce09f500/0xe64f32d5b6631b06 lrc: 3/0,0 mode: PW/PW res: [0x20000240e:0x1ba06:0x0].0x0 bits 0x40/0x0 rrc: 3 type: IBT flags: 0x49400000000 nid: local remote: 0xfcd90309686d82d6 expref: -99 pid: 28799 timeout: 0 lvb_type: 0 </pre> </div></div> <p>After this it hangs.</p> <p>Could you attach logs from MDSes?</p> <p>Yes, give me a bit to reproduce the error. Currently only one MDS</p> <p>Do you have DOM configured?</p> <p>Yes, a percentage of the tests uses DOM striping, basically striping is randomly chosen, either normal striping, a PFL layout or a DOM layout. </p> <p>It works like this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">coin=$((RANDOM %% 2)) toss=$((RANDOM %% 2)) # <span class="code-keyword">for</span> 2.8 testing # coin=1 # Use shift to get most random result (last bit) # Set directory striping <span class="code-keyword">if</span> [ $coin -eq 1 ];then lfs setstripe -c $((RANDOM %% (nrdt + 1))) $dirname <span class="code-keyword">else</span> <span class="code-keyword">if</span> [ $toss -eq 1 ];then lfs setstripe -E 32M -c 1 -S 1M -E 1G -c 4 -E -1 -c -1 -S 4m $dirname <span class="code-keyword">else</span> lfs setstripe -E 1M -L mdt -E EOF -c -1 $dirname fi fi </pre> </div></div> <p>Could you also check with DOM disabled? I tend to think this is issue with DOM.</p> <p>Yes, i can do this.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-object">int</span> osc_extent_wait(<span class="code-keyword">const</span> struct lu_env *env, struct osc_extent *ext, <span class="code-keyword">enum</span> osc_extent_state state) { struct osc_object *obj = ext-&gt;oe_obj; struct l_wait_info lwi = LWI_TIMEOUT_INTR(cfs_time_seconds(600), NULL, LWI_ON_SIGNAL_NOOP, NULL); <span class="code-object">int</span> rc = 0; ENTRY; osc_object_lock(obj); LASSERT(sanity_check_nolock(ext) == 0); /* `Kick' <span class="code-keyword">this</span> extent only <span class="code-keyword">if</span> the caller is waiting <span class="code-keyword">for</span> it to be * written out. */ <span class="code-keyword">if</span> (state == OES_INV &amp;&amp; !ext-&gt;oe_urgent &amp;&amp; !ext-&gt;oe_hp) { <span class="code-keyword">if</span> (ext-&gt;oe_state == OES_ACTIVE) { ext-&gt;oe_urgent = 1; } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (ext-&gt;oe_state == OES_CACHE) { ext-&gt;oe_urgent = 1; osc_extent_hold(ext); rc = 1; } } osc_object_unlock(obj); <span class="code-keyword">if</span> (rc == 1) osc_extent_release(env, ext); <span class="code-comment">/* wait <span class="code-keyword">for</span> the extent until its state becomes @state */</span> rc = l_wait_event(ext-&gt;oe_waitq, extent_wait_cb(ext, state), &amp;lwi); <span class="code-keyword">if</span> (rc == -ETIMEDOUT) { OSC_EXTENT_DUMP(D_ERROR, ext, <span class="code-quote">"%s: wait ext to %u timedout, recovery in progress?\n"</span>, cli_name(osc_cli(obj)), state); lwi = LWI_INTR(NULL, NULL); rc = l_wait_event(ext-&gt;oe_waitq, extent_wait_cb(ext, state), &lt;===== It waits here forever. &amp;lwi); } <span class="code-keyword">if</span> (rc == 0 &amp;&amp; ext-&gt;oe_rc &lt; 0) rc = ext-&gt;oe_rc; RETURN(rc); } </pre> </div></div> <p>i have started soak with only normal striping, no PFL or DOM. Will let that run for awhile, then add PFL back in.</p> <p>Ran through 9 MDT restarts/failovers ( + 6 hours) with no errors.<br/> I am enabling DOM striping and restarting</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lfs setstripe -E 1M -L mdt -E EOF -c -1 $dirname </pre> </div></div> <p>Test with only DOM, got the failure. Tested with only PFL, got the failure. Restarting now and will get some log dumps.</p> <p>Replicated error, but wasn't able to catch the log after the immediate failure, i fell asleep. However, crash-dumped the MDS with full debug. crash dumped the node with full debug. </p> <p>Lustre logs attached</p> <p>Where MDS logs and crash are placed? Can I see them?</p> <p>The crash dump and all logs are on the spirit cluster, do you have a login? /scratch/logs and /scratch/dumps. </p> <p>I have attached the console and system logs from the MDS (soak-8) to this bug. </p> <p>Sorry, I don't have login on spirit. Could you copy it to somewhere on onyx?</p> <p>You should be able to get a Spirit account quickly, file a DCO ticket and label it account-mgmnt. Usually happens in minutes - we've advised DCO, and they are ready to do it now. All they need is your public ssh key</p> <p>In the meantime, the crash dump is on onyx - /home/cliffwhi/lu-10961 - if you can't reach it, point me to a better directory and I'll put it there. </p> <p>Thanks. I submit the DCO ticket and now I'm coping core from onyx...</p> <p>Thanks, I forced the core dump several hours after the fault, it's difficult to catch as the fault normally occurs in the middle of my night <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Cliff, what type of load is used in this testing? Is it something particular, e.g. 'dd' or 'tar' or mix? I am thinking about possibility to reproduce that with simpler test.</p> <p>I though we had explained soak to you. The test running at the time of failure were:<br/> blogbench, iorssf, iorfpp, kcompile, mdtestfpp, mdtestssf, simul,<br/> fio( random, sequential, SAS simulation) <br/> The random mix is distributed across the clients, with the intent of seriously loading each client. It is difficult to tell exactly what is running on a specific node at the time of failure, but generally a node will have 2/3 different jobs running at any given time. </p> <p>yes, my question was about just particular time when error happened, so it is not possible to select any specific load causing that. In that case only logs at the moment of failure could make that clear. For this we can try to inject code which will cause error immediately instead of timeout.</p> <p>Dmitry showed already place where timeout happens, probably here we can output more debug info and return error immediately instead if waiting, so lustre logs will contain useful data.</p> <p>Mike - Would you please upload a patch with any necessary debug information that will cause an error so we can get fail instead of waiting for a timeout?</p> <p>Thanks.</p> <p>Mike Pershin (mike.pershin@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32710" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32710</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10961" title="Clients hang after failovers. LustreError: 223668:0:(file.c:4213:ll_inode_revalidate_fini()) soaked: revalidate FID [0x200000007:0x1:0x0] error: rc = -4" class="issue-link" data-issue-key="LU-10961"><del>LU-10961</del></a> osc: add debug code<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 523ac2ce306d182b2dc5db7c9a0f401b39124963</p> <p>Hit the bug right away, lustre logs for two clients attached. Also dumped stacks and crash dumped those nodes, bits are on Spirit. Restarting with full debug. </p> <p>In the same incident, soak-18/19 were hung, but not showing the _fini error. Lustre logs attached. Also lustre log from MDS. </p> <p>Console logs from soak-17,42 and MDS (soak-8) attached. </p> <p>Mike</p> <p>Has the information provided any insight?</p> <p>Peter</p> <p>Peter,</p> <p>yes, there are clues, I am working on reproducer and patch</p> <p>I was able to reproduce lock replay issue with DoM files, patch will be ready soon. The problem is the lock cancellation before lock replay, MDC cancels locks differently than OSC and that causes DoM locks to be handled improperly.</p> <p>Meanwhile that doesn't explain failover failure with PFL-only file because there is no MDC involved, so there can be another reasons for that. </p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/32791" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32791</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10961" title="Clients hang after failovers. LustreError: 223668:0:(file.c:4213:ll_inode_revalidate_fini()) soaked: revalidate FID [0x200000007:0x1:0x0] error: rc = -4" class="issue-link" data-issue-key="LU-10961"><del>LU-10961</del></a> ldlm: don't cancel DoM locks before replay<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5c5ef9be5904a7f16aa41a341c561b0c9c6beb42</p> <p>There is another problem with replay - if DOM write cause new component instantiation then replay of both layout change and write RPC failed. I am investigating that.</p> <p>It seems that problem is not in DOM but PFL. When a new component is instantiated during failover then that is not replayed properly. It is worth to create new ticket for that.<br/> That explains also why Cliff reported problems with PFL-only setup as well.</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/32889" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32889</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10961" title="Clients hang after failovers. LustreError: 223668:0:(file.c:4213:ll_inode_revalidate_fini()) soaked: revalidate FID [0x200000007:0x1:0x0] error: rc = -4" class="issue-link" data-issue-key="LU-10961"><del>LU-10961</del></a> ldlm: extended testing<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 43febdf398958c18590fc6a9d306a615b907305b</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32791/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32791/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10961" title="Clients hang after failovers. LustreError: 223668:0:(file.c:4213:ll_inode_revalidate_fini()) soaked: revalidate FID [0x200000007:0x1:0x0] error: rc = -4" class="issue-link" data-issue-key="LU-10961"><del>LU-10961</del></a> ldlm: don't cancel DoM locks before replay<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b44b1ff8c7fc50d5b0438926c89c69eb85817168</p> <p>Landed for 2.12</p> Blocker LU-11158 Development Rank 1|hzzwh3: Rank (Obsolete) 9223372036854775807 Severity