https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7067 Lustre <p>Running SWL clients timeout and LBUG after server (OST) failover</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Aug 31 11:22:37 iwc98 kernel: Lustre: 5465:0:(client.c:2014:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Aug 31 11:23:04 iwc98 kernel: Lustre: lustre-OST000a-osc-ffff8808713e3c00: Connection restored to lustre-OST000a (at 192.168.120.24@o2ib) Aug 31 11:33:32 iwc98 kernel: Lustre: 5457:0:(client.c:2014:ptlrpc_expire_one_request()) @@@ Request sent has timed out <span class="code-keyword">for</span> slow reply: [sent 1441045957/real 1441045957] req@ffff8806e1ec2080 x1510687803291588/t0(0) o8-&gt;stzfs-OST000a-osc-ffff880871117c00@192.168.120.23@o2ib:28/4 lens 520/544 e 0 to 1 dl 1441046012 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 31 11:33:32 iwc98 kernel: Lustre: 5457:0:(client.c:2014:ptlrpc_expire_one_request()) Skipped 32 previous similar messages Aug 31 11:34:55 iwc98 kernel: LustreError: 11-0: lustre-OST000a-osc-ffff8808713e3c00: operation ost_write to node 192.168.120.24@o2ib failed: rc = -107 Aug 31 11:34:55 iwc98 kernel: Lustre: lustre-OST000a-osc-ffff8808713e3c00: Connection to lustre-OST000a (at 192.168.120.24@o2ib) was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete Aug 31 11:34:55 iwc98 kernel: Lustre: Skipped 2 previous similar messages Aug 31 11:35:19 iwc98 kernel: LustreError: 167-0: lustre-OST000a-osc-ffff8808713e3c00: This client was evicted by lustre-OST000a; in progress operations using <span class="code-keyword">this</span> service will fail. Aug 31 11:35:22 iwc98 kernel: LustreError: 107401:0:(vvp_io.c:1076:vvp_io_write_start()) ASSERTION( vio-&gt;vui_iocb-&gt;ki_pos == pos ) failed: Aug 31 11:35:22 iwc98 kernel: LustreError: 107401:0:(vvp_io.c:1076:vvp_io_write_start()) ASSERTION( vio-&gt;vui_iocb-&gt;ki_pos == pos ) failed: Aug 31 11:35:22 iwc98 kernel: LustreError: 107399:0:(vvp_io.c:1076:vvp_io_write_start()) ASSERTION( vio-&gt;vui_iocb-&gt;ki_pos == pos ) failed: Aug 31 11:35:22 iwc98 kernel: LustreError: 107399:0:(vvp_io.c:1076:vvp_io_write_start()) ASSERTION( vio-&gt;vui_iocb-&gt;ki_pos == pos ) failed: Aug 31 11:35:22 iwc98 kernel: LustreError: 107399:0:(vvp_io.c:1076:vvp_io_write_start()) LBUG Aug 31 11:35:22 iwc98 kernel: LustreError: 107399:0:(vvp_io.c:1076:vvp_io_write_start()) LBUG Aug 31 11:35:22 iwc98 kernel: Pid: 107399, comm: ior Aug 31 11:35:22 iwc98 kernel: Aug 31 11:35:22 iwc98 kernel: Call Trace: Aug 31 11:35:22 iwc98 kernel: [&lt;ffffffffa0470875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Aug 31 11:35:22 iwc98 kernel: [&lt;ffffffffa0470e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] Aug 31 11:35:22 iwc98 kernel: [&lt;ffffffffa0a5f27c&gt;] vvp_io_write_start+0x5bc/0x5c0 [lustre] Aug 31 11:35:22 iwc98 kernel: [&lt;ffffffffa047cbf1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] Aug 31 11:35:22 iwc98 kernel: [&lt;ffffffffa060444a&gt;] cl_io_start+0x6a/0x140 [obdclass] Aug 31 11:35:22 iwc98 kernel: [&lt;ffffffffa0608994&gt;] cl_io_loop+0xb4/0x1b0 [obdclass] Aug 31 11:35:22 iwc98 kernel: [&lt;ffffffffa0a077b7&gt;] ll_file_io_generic+0x317/0xab0 [lustre] Aug 31 11:35:22 iwc98 kernel: [&lt;ffffffffa0a09c9b&gt;] ll_file_aio_write+0x20b/0x860 [lustre] Aug 31 11:35:22 iwc98 kernel: [&lt;ffffffffa0a0a41b&gt;] ll_file_write+0x12b/0x260 [lustre] Aug 31 11:35:22 iwc98 kernel: [&lt;ffffffff8118e3a8&gt;] vfs_write+0xb8/0x1a0 Aug 31 11:35:22 iwc98 kernel: [&lt;ffffffff8118ed71&gt;] sys_write+0x51/0x90 Aug 31 11:35:22 iwc98 kernel: [&lt;ffffffff8100b0d2&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> Hyperion/SWL LU-7067

vvp_io.c:1076:vvp_io_write_start()) ASSERTION( vio->vui_iocb->ki_pos == pos ) failed

Bug Blocker Resolved Fixed Zhenyu Xu Cliff White Mon, 31 Aug 2015 19:32:10 +0000 Thu, 7 Sep 2017 17:22:27 +0000 Thu, 22 Jun 2017 15:21:23 +0000 Lustre 2.8.0 Lustre 2.8.0 0 4 <p>Lustre log dump by LBUG</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ cat iwc98.lbug.txt Debug log: 431576 lines, 431576 kept, 0 dropped, 0 bad. </pre> </div></div> <p>lustre log does not have relevant info.</p> <p>Ran 48 hours SWL- did not recur</p> <p>Just started to see this bug in production with latest lustre 2.8. Its one particular app and the bug only manifest some times. I will see if I can collect some logs for you</p> <p>it looks like <a href="https://jira.whamcloud.com/browse/LU-9491" title="v2.9: silent data corruption with writev() and O_APPEND" class="issue-link" data-issue-key="LU-9491"><del>LU-9491</del></a>.</p> <p>Yes it is <a href="https://jira.whamcloud.com/browse/LU-9491" title="v2.9: silent data corruption with writev() and O_APPEND" class="issue-link" data-issue-key="LU-9491"><del>LU-9491</del></a>. Since running the patch in production this problem has gone away. I pushed the fix to <a href="https://review.whamcloud.com/#/c/27782" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/27782</a></p> Duplicate Related LU-7382 LU-9491 Development Rank 1|hzxluv: Rank (Obsolete) 9223372036854775807 Severity