https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7382 Lustre <p>kernel:LustreError: 29035:0:(vvp_io.c:573:vvp_io_update_iov()) ASSERTION( vio-&gt;vui_tot_nrsegs &gt;= vio-&gt;vui_iter-&gt;nr_segs ) failed: tot_nrsegs: 1, nrsegs: 2</p> <p>Message from syslogd@test1 at Nov 4 13:01:37 ...<br/> kernel:LustreError: 29035:0:(vvp_io.c:573:vvp_io_update_iov()) LBUG</p> LU-7382

(vvp_io.c:573:vvp_io_update_iov()) ASSERTION( vio->vui_tot_nrsegs >= vio->vui_iter->nr_segs ) failed

Bug Major Reopened Unresolved Zhenyu Xu Zhenyu Xu Wed, 4 Nov 2015 05:12:14 +0000 Fri, 8 Sep 2017 17:40:45 +0000 Lustre 2.7.0 Lustre 2.8.0 Lustre 2.5.4 Lustre 2.8.0 1 13 <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/17039" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17039</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7382" title="(vvp_io.c:573:vvp_io_update_iov()) ASSERTION( vio-&gt;vui_tot_nrsegs &gt;= vio-&gt;vui_iter-&gt;nr_segs ) failed" class="issue-link" data-issue-key="LU-7382">LU-7382</a> llite: vvp_io_update_iov() ASSERTION failure<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 564cc551579b105edea5e84a184fb71c015bf678</p> <p>Cray is seeing this bug regularly during system testing using Lustre 2.7 on a relatively small machine (1 MDT, 8 OSTs, 16 clients). I've uploaded a couple of dumps in case they may be useful in resolving the bug. Location:<br/> ftp.whamcloud.com:/uploads/<a href="https://jira.whamcloud.com/browse/LU-7382" title="(vvp_io.c:573:vvp_io_update_iov()) ASSERTION( vio-&gt;vui_tot_nrsegs &gt;= vio-&gt;vui_iter-&gt;nr_segs ) failed" class="issue-link" data-issue-key="LU-7382">LU-7382</a>/<a href="https://jira.whamcloud.com/browse/LU-7382" title="(vvp_io.c:573:vvp_io_update_iov()) ASSERTION( vio-&gt;vui_tot_nrsegs &gt;= vio-&gt;vui_iter-&gt;nr_segs ) failed" class="issue-link" data-issue-key="LU-7382">LU-7382</a>_dump.tgz</p> <p>Thank you Ann for the crash dump.</p> <p>Andriy Skulysh (andriy.skulysh@seagate.com) uploaded a new patch: <a href="http://review.whamcloud.com/17632" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17632</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7382" title="(vvp_io.c:573:vvp_io_update_iov()) ASSERTION( vio-&gt;vui_tot_nrsegs &gt;= vio-&gt;vui_iter-&gt;nr_segs ) failed" class="issue-link" data-issue-key="LU-7382">LU-7382</a> llite: vvp_io_update_iov() ASSERTION failure<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8431fec8ba5434b76aa994d93d5fa44b850be689</p> <p>I've uploaded a new set of four dumps with Andriy's patch to here:<br/> ftp.whamcloud.com:/uploads/<a href="https://jira.whamcloud.com/browse/LU-7382" title="(vvp_io.c:573:vvp_io_update_iov()) ASSERTION( vio-&gt;vui_tot_nrsegs &gt;= vio-&gt;vui_iter-&gt;nr_segs ) failed" class="issue-link" data-issue-key="LU-7382">LU-7382</a>/151223_dumps.tar.gz</p> <p>All four nodes which failed had debug enabled. I've included the extracted (and sorted) logs for one of them:<br/> c0-0c1s0n1-1512222203_log.sort</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17632/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17632/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7382" title="(vvp_io.c:573:vvp_io_update_iov()) ASSERTION( vio-&gt;vui_tot_nrsegs &gt;= vio-&gt;vui_iter-&gt;nr_segs ) failed" class="issue-link" data-issue-key="LU-7382">LU-7382</a> llite: Fix iovec references accounting in ll_file_aio_read/write<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 57f055f8d0df80e140724b00d1729f454222a83a</p> <p>Landed for 2.8.0</p> <p>This problem still exist on b2_8_fe branch. We have hit this problem twice while running in production.</p> <p>2016-07-01T20:09:21.362206-04:00 c11-3c1s2n3 LustreError:<br/> 32026:0:(vvp_io.c:573:vvp_io_update_iov()) ASSERTION( vio-&gt;vui_tot_nrsegs<br/> &gt;= vio-&gt;vui_iter-&gt;nr_segs ) failed: tot_nrsegs: 1,<br/> nrsegs: 2<br/> 2016-07-01T20:09:21.362293-04:00 c11-3c1s2n3 LustreError:<br/> 32026:0:(vvp_io.c:573:vvp_io_update_iov()) LBUG<br/> 2016-07-01T20:09:21.362336-04:00 c11-3c1s2n3 Pid: 32026, comm:<br/> bowtie2-build-s<br/> 2016-07-01T20:09:21.362344-04:00 c11-3c1s2n3 Call Trace:<br/> 2016-07-01T20:09:21.362359-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff81006651&gt;&#93;</span><br/> try_stack_unwind+0x161/0x1a0<br/> 2016-07-01T20:09:21.362367-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff81004eb9&gt;&#93;</span><br/> dump_trace+0x89/0x430<br/> 2016-07-01T20:09:21.391856-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffffa011aac0&gt;&#93;</span><br/> lbug_with_loc+0x90/0x1d0 <span class="error">&#91;libcfs&#93;</span><br/> 2016-07-01T20:09:21.391877-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffffa06cb3f8&gt;&#93;</span><br/> vvp_io_rw_lock+0x738/0x860 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.391893-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffffa06cb556&gt;&#93;</span><br/> vvp_io_write_lock+0x36/0x40 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.391902-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffffa030a514&gt;&#93;</span><br/> cl_io_lock+0x74/0x400 <span class="error">&#91;obdclass&#93;</span><br/> 2016-07-01T20:09:21.422510-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffffa030be27&gt;&#93;</span><br/> cl_io_loop+0x2b7/0x710 <span class="error">&#91;obdclass&#93;</span><br/> 2016-07-01T20:09:21.422529-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffffa0675574&gt;&#93;</span><br/> ll_file_io_generic+0x364/0xab0 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.422544-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffffa0676290&gt;&#93;</span><br/> ll_file_aio_write+0x5d0/0x6a0 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.422580-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff8114097b&gt;&#93;</span><br/> do_sync_readv_writev+0xdb/0x120<br/> 2016-07-01T20:09:21.422594-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff81141854&gt;&#93;</span><br/> do_readv_writev+0xd4/0x1e0<br/> 2016-07-01T20:09:21.422601-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff8114199e&gt;&#93;</span><br/> vfs_writev+0x3e/0x60<br/> 2016-07-01T20:09:21.422608-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff81141ae5&gt;&#93;</span><br/> sys_writev+0x55/0xc0<br/> 2016-07-01T20:09:21.503878-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff8133ac2b&gt;&#93;</span><br/> system_call_fastpath+0x16/0x1b<br/> 2016-07-01T20:09:21.503911-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;00002aaaab7581be&gt;&#93;</span><br/> 0x2aaaab7581be<br/> 2016-07-01T20:09:21.503920-04:00 c11-3c1s2n3 Kernel panic - not syncing:<br/> LBUG<br/> 2016-07-01T20:09:21.503944-04:00 c11-3c1s2n3 Pid: 32026, comm:<br/> bowtie2-build-s Tainted: P<br/> 3.0.101-0.46.1_1.0502.8871-cray_gem_c #1<br/> 2016-07-01T20:09:21.503961-04:00 c11-3c1s2n3 Call Trace:<br/> 2016-07-01T20:09:21.503970-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff81006651&gt;&#93;</span><br/> try_stack_unwind+0x161/0x1a0<br/> 2016-07-01T20:09:21.503977-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff81004eb9&gt;&#93;</span><br/> dump_trace+0x89/0x430<br/> 2016-07-01T20:09:21.503990-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff810060bc&gt;&#93;</span><br/> show_trace_log_lvl+0x5c/0x80<br/> 2016-07-01T20:09:21.503996-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff810060f5&gt;&#93;</span><br/> show_trace+0x15/0x20<br/> 2016-07-01T20:09:21.504011-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff81336d32&gt;&#93;</span><br/> dump_stack+0x79/0x84<br/> 2016-07-01T20:09:21.504018-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff81336dd1&gt;&#93;</span><br/> panic+0x94/0x1da<br/> 2016-07-01T20:09:21.504051-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffffa011abf1&gt;&#93;</span><br/> lbug_with_loc+0x1c1/0x1d0 <span class="error">&#91;libcfs&#93;</span><br/> 2016-07-01T20:09:21.504070-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffffa06cb3f8&gt;&#93;</span><br/> vvp_io_rw_lock+0x738/0x860 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.504079-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffffa06cb556&gt;&#93;</span><br/> vvp_io_write_lock+0x36/0x40 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.504121-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffffa030a514&gt;&#93;</span><br/> cl_io_lock+0x74/0x400 <span class="error">&#91;obdclass&#93;</span><br/> 2016-07-01T20:09:21.504139-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffffa030be27&gt;&#93;</span><br/> cl_io_loop+0x2b7/0x710 <span class="error">&#91;obdclass&#93;</span><br/> 2016-07-01T20:09:21.504157-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffffa0675574&gt;&#93;</span><br/> ll_file_io_generic+0x364/0xab0 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.504174-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffffa0676290&gt;&#93;</span><br/> ll_file_aio_write+0x5d0/0x6a0 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.504185-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff8114097b&gt;&#93;</span><br/> do_sync_readv_writev+0xdb/0x120<br/> 2016-07-01T20:09:21.532355-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff81141854&gt;&#93;</span><br/> do_readv_writev+0xd4/0x1e0<br/> 2016-07-01T20:09:21.532377-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff8114199e&gt;&#93;</span><br/> vfs_writev+0x3e/0x60<br/> 2016-07-01T20:09:21.532387-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff81141ae5&gt;&#93;</span><br/> sys_writev+0x55/0xc0<br/> 2016-07-01T20:09:21.532405-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;ffffffff8133ac2b&gt;&#93;</span><br/> system_call_fastpath+0x16/0x1b<br/> 2016-07-01T20:09:21.532417-04:00 c11-3c1s2n3 <span class="error">&#91;&lt;00002aaaab7581be&gt;&#93;</span><br/> 0x2aaaab7581bd<br/> 2016-07-01T20:09:21.662130-04:00 c6-0c2s1n2 LustreError:<br/> 24943:0:(vvp_io.c:573:vvp_io_update_iov()) ASSERTION( vio-&gt;vui_tot_nrsegs<br/> &gt;= vio-&gt;vui_iter-&gt;nr_segs ) failed: tot_nrsegs: 1,<br/> nrsegs: 2<br/> 2016-07-01T20:09:21.662163-04:00 c6-0c2s1n2 LustreError:<br/> 24943:0:(vvp_io.c:573:vvp_io_update_iov()) LBUG<br/> 2016-07-01T20:09:21.662205-04:00 c6-0c2s1n2 Pid: 24943, comm:<br/> bowtie2-build-s<br/> 2016-07-01T20:09:21.662214-04:00 c6-0c2s1n2 Call Trace:<br/> 2016-07-01T20:09:21.662222-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff81006651&gt;&#93;</span><br/> try_stack_unwind+0x161/0x1a0<br/> 2016-07-01T20:09:21.662229-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff81004eb9&gt;&#93;</span><br/> dump_trace+0x89/0x430<br/> 2016-07-01T20:09:21.662241-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffffa011aac0&gt;&#93;</span><br/> lbug_with_loc+0x90/0x1d0 <span class="error">&#91;libcfs&#93;</span><br/> 2016-07-01T20:09:21.692133-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffffa06cb3f8&gt;&#93;</span><br/> vvp_io_rw_lock+0x738/0x860 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.692165-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffffa06cb556&gt;&#93;</span><br/> vvp_io_write_lock+0x36/0x40 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.692204-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffffa030a514&gt;&#93;</span><br/> cl_io_lock+0x74/0x400 <span class="error">&#91;obdclass&#93;</span><br/> 2016-07-01T20:09:21.692212-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffffa030be27&gt;&#93;</span><br/> cl_io_loop+0x2b7/0x710 <span class="error">&#91;obdclass&#93;</span><br/> 2016-07-01T20:09:21.692236-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffffa0675574&gt;&#93;</span><br/> ll_file_io_generic+0x364/0xab0 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.692257-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffffa0676290&gt;&#93;</span><br/> ll_file_aio_write+0x5d0/0x6a0 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.742635-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff8114097b&gt;&#93;</span><br/> do_sync_readv_writev+0xdb/0x120<br/> 2016-07-01T20:09:21.742668-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff81141854&gt;&#93;</span><br/> do_readv_writev+0xd4/0x1e0<br/> 2016-07-01T20:09:21.742678-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff8114199e&gt;&#93;</span><br/> vfs_writev+0x3e/0x60<br/> 2016-07-01T20:09:21.742731-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff81141ae5&gt;&#93;</span><br/> sys_writev+0x55/0xc0<br/> 2016-07-01T20:09:21.742745-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff8133ac2b&gt;&#93;</span><br/> system_call_fastpath+0x16/0x1b<br/> 2016-07-01T20:09:21.742754-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;00002aaaab7581be&gt;&#93;</span><br/> 0x2aaaab7581be<br/> 2016-07-01T20:09:21.742800-04:00 c6-0c2s1n2 Kernel panic - not syncing:<br/> LBUG<br/> 2016-07-01T20:09:21.742827-04:00 c6-0c2s1n2 Pid: 24943, comm:<br/> bowtie2-build-s Tainted: P<br/> 3.0.101-0.46.1_1.0502.8871-cray_gem_c #1<br/> 2016-07-01T20:09:21.742849-04:00 c6-0c2s1n2 Call Trace:<br/> 2016-07-01T20:09:21.742904-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff81006651&gt;&#93;</span><br/> try_stack_unwind+0x161/0x1a0<br/> 2016-07-01T20:09:21.742940-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff81004eb9&gt;&#93;</span><br/> dump_trace+0x89/0x430<br/> 2016-07-01T20:09:21.742948-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff810060bc&gt;&#93;</span><br/> show_trace_log_lvl+0x5c/0x80<br/> 2016-07-01T20:09:21.772093-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff810060f5&gt;&#93;</span><br/> show_trace+0x15/0x20<br/> 2016-07-01T20:09:21.772123-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff81336d32&gt;&#93;</span><br/> dump_stack+0x79/0x84<br/> 2016-07-01T20:09:21.772132-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff81336dd1&gt;&#93;</span><br/> panic+0x94/0x1da<br/> 2016-07-01T20:09:21.772205-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffffa011abf1&gt;&#93;</span><br/> lbug_with_loc+0x1c1/0x1d0 <span class="error">&#91;libcfs&#93;</span><br/> 2016-07-01T20:09:21.772224-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffffa06cb3f8&gt;&#93;</span><br/> vvp_io_rw_lock+0x738/0x860 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.772241-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffffa06cb556&gt;&#93;</span><br/> vvp_io_write_lock+0x36/0x40 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.772256-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffffa030a514&gt;&#93;</span><br/> cl_io_lock+0x74/0x400 <span class="error">&#91;obdclass&#93;</span><br/> 2016-07-01T20:09:21.802196-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffffa030be27&gt;&#93;</span><br/> cl_io_loop+0x2b7/0x710 <span class="error">&#91;obdclass&#93;</span><br/> 2016-07-01T20:09:21.802260-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffffa0675574&gt;&#93;</span><br/> ll_file_io_generic+0x364/0xab0 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.802270-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffffa0676290&gt;&#93;</span><br/> ll_file_aio_write+0x5d0/0x6a0 <span class="error">&#91;lustre&#93;</span><br/> 2016-07-01T20:09:21.802279-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff8114097b&gt;&#93;</span><br/> do_sync_readv_writev+0xdb/0x120<br/> 2016-07-01T20:09:21.802299-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff81141854&gt;&#93;</span><br/> do_readv_writev+0xd4/0x1e0<br/> 2016-07-01T20:09:21.802317-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff8114199e&gt;&#93;</span><br/> vfs_writev+0x3e/0x60<br/> 2016-07-01T20:09:21.832209-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff81141ae5&gt;&#93;</span><br/> sys_writev+0x55/0xc0<br/> 2016-07-01T20:09:21.832229-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;ffffffff8133ac2b&gt;&#93;</span><br/> system_call_fastpath+0x16/0x1b<br/> 2016-07-01T20:09:21.832259-04:00 c6-0c2s1n2 <span class="error">&#91;&lt;00002aaaab7581be&gt;&#93;</span><br/> 0x2aaaab7581bd</p> <p>I have a patch to clean up this piece of code, please check out the patch with commit 1101120d3258509fa74f952cd8664bfdc17bd97d in the master branch and it would be worth porting that patch over and see if it can fix the problem.</p> <p>I pushed a b2_8_fe patch of <a href="https://jira.whamcloud.com/browse/LU-4257" title="parallel dds are slower than serial dds" class="issue-link" data-issue-key="LU-4257"><del>LU-4257</del></a> at <a href="http://review.whamcloud.com/#/c/21198" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/21198</a>. Do you have a good reproducer? We only see it once in awhile and haven't figured out what is causing the LBUG and if it can be done in a consistent way.</p> <p>Just as a "me too", we hit that same LBUG (trace below) with IEEL 3.0</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># cat /proc/fs/lustre/version lustre: 2.7.15.3 kernel: patchless_client build:  jenkins-arch=x86_64,build_type=client,distro=el6.7,ib_stack=inkernel-3843-ga11db72-PRISTINE-2.6.32-573.12.1.el6.x86_64 </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">2017-02-24 08:29:10 [3183317.732931] LustreError: 50671:0:(vvp_io.c:573:vvp_io_update_iov()) ASSERTION( vio-&gt;vui_tot_nrsegs &gt;= vio-&gt;vui_iter-&gt;nr_segs ) failed: tot_nrsegs: 1, nrsegs: 2 2017-02-24 08:29:10 [3183317.749876] LustreError: 50671:0:(vvp_io.c:573:vvp_io_update_iov()) LBUG 2017-02-24 08:29:10 [3183317.757841] Pid: 50671, comm: jellyfish 2017-02-24 08:29:11 [3183317.762581] 2017-02-24 08:29:11 [3183317.762581] Call Trace: 2017-02-24 08:29:11 [3183317.767941]  [&lt;ffffffffa030c895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 2017-02-24 08:29:11 [3183317.776222]  [&lt;ffffffffa030ce97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 2017-02-24 08:29:11 [3183317.783662]  [&lt;ffffffffa09e1d19&gt;] vvp_io_rw_lock+0x6f9/0x790 [lustre] 2017-02-24 08:29:11 [3183317.791356]  [&lt;ffffffffa09e1de5&gt;] vvp_io_write_lock+0x35/0x40 [lustre] 2017-02-24 08:29:11 [3183317.799184]  [&lt;ffffffffa0526893&gt;] cl_io_lock+0x63/0x3c0 [obdclass] 2017-02-24 08:29:11 [3183317.806589]  [&lt;ffffffffa0526c92&gt;] cl_io_loop+0xa2/0x1b0 [obdclass] 2017-02-24 08:29:11 [3183317.813985]  [&lt;ffffffffa097d470&gt;] ll_file_io_generic+0x5d0/0xae0 [lustre] 2017-02-24 08:29:11 [3183317.822061]  [&lt;ffffffff8105e173&gt;] ? __wake_up+0x53/0x70 2017-02-24 08:29:11 [3183317.828391]  [&lt;ffffffffa0987dbb&gt;] ll_file_aio_write+0x21b/0x9d0 [lustre] 2017-02-24 08:29:11 [3183317.836376]  [&lt;ffffffffa0987ba0&gt;] ? ll_file_aio_write+0x0/0x9d0 [lustre] 2017-02-24 08:29:11 [3183317.844351]  [&lt;ffffffff811917db&gt;] do_sync_readv_writev+0xfb/0x140 2017-02-24 08:29:11 [3183317.851649]  [&lt;ffffffff810a1460&gt;] ? autoremove_wake_function+0x0/0x40 2017-02-24 08:29:11 [3183317.859347]  [&lt;ffffffffa051ae0d&gt;] ? cl_env_put+0x16d/0x200 [obdclass] 2017-02-24 08:29:11 [3183317.867019]  [&lt;ffffffff81231a56&gt;] ? security_file_permission+0x16/0x20 2017-02-24 08:29:11 [3183317.874791]  [&lt;ffffffff81192886&gt;] do_readv_writev+0xd6/0x1f0 2017-02-24 08:29:11 [3183317.881616]  [&lt;ffffffffa098a4d3&gt;] ? ll_file_read+0x143/0x260 [lustre] 2017-02-24 08:29:11 [3183317.889288]  [&lt;ffffffff811929e6&gt;] vfs_writev+0x46/0x60 2017-02-24 08:29:11 [3183317.895509]  [&lt;ffffffff81192b11&gt;] sys_writev+0x51/0xd0 2017-02-24 08:29:11 [3183317.901747]  [&lt;ffffffff8153c64e&gt;] ? do_device_not_available+0xe/0x10 2017-02-24 08:29:11 [3183317.909337]  [&lt;ffffffff8100b0d2&gt;] system_call_fastpath+0x16/0x1b 2017-02-24 08:29:11 [3183317.916521] 2017-02-24 08:29:11 [3183317.919054] Kernel panic - not syncing: LBUG 2017-02-24 08:29:11 [3183317.924280] Pid: 50671, comm: jellyfish Not tainted 2.6.32-573.12.1.el6.noc0w.x86_64 #1 2017-02-24 08:29:11 [3183317.933932] Call Trace: 2017-02-24 08:29:11 [3183317.937145]  [&lt;ffffffff81538271&gt;] ? panic+0xa7/0x16f 2017-02-24 08:29:11 [3183317.943174]  [&lt;ffffffffa030ceeb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] 2017-02-24 08:29:11 [3183317.950774]  [&lt;ffffffffa09e1d19&gt;] ? vvp_io_rw_lock+0x6f9/0x790 [lustre] 2017-02-24 08:29:11 [3183317.958640]  [&lt;ffffffffa09e1de5&gt;] ? vvp_io_write_lock+0x35/0x40 [lustre] 2017-02-24 08:29:11 [3183317.966634]  [&lt;ffffffffa0526893&gt;] ? cl_io_lock+0x63/0x3c0 [obdclass] 2017-02-24 08:29:11 [3183317.974209]  [&lt;ffffffffa0526c92&gt;] ? cl_io_loop+0xa2/0x1b0 [obdclass] 2017-02-24 08:29:11 [3183317.981778]  [&lt;ffffffffa097d470&gt;] ? ll_file_io_generic+0x5d0/0xae0 [lustre] 2017-02-24 08:29:11 [3183317.990012]  [&lt;ffffffff8105e173&gt;] ? __wake_up+0x53/0x70 2017-02-24 08:29:11 [3183317.996318]  [&lt;ffffffffa0987dbb&gt;] ? ll_file_aio_write+0x21b/0x9d0 [lustre] 2017-02-24 08:29:11 [3183318.004469]  [&lt;ffffffffa0987ba0&gt;] ? ll_file_aio_write+0x0/0x9d0 [lustre] 2017-02-24 08:29:11 [3183318.012424]  [&lt;ffffffff811917db&gt;] ? do_sync_readv_writev+0xfb/0x140 2017-02-24 08:29:11 [3183318.019891]  [&lt;ffffffff810a1460&gt;] ? autoremove_wake_function+0x0/0x40 2017-02-24 08:29:11 [3183318.027561]  [&lt;ffffffffa051ae0d&gt;] ? cl_env_put+0x16d/0x200 [obdclass] 2017-02-24 08:29:11 [3183318.035212]  [&lt;ffffffff81231a56&gt;] ? security_file_permission+0x16/0x20 2017-02-24 08:29:11 [3183318.042963]  [&lt;ffffffff81192886&gt;] ? do_readv_writev+0xd6/0x1f0 2017-02-24 08:29:11 [3183318.049950]  [&lt;ffffffffa098a4d3&gt;] ? ll_file_read+0x143/0x260 [lustre] 2017-02-24 08:29:11 [3183318.057606]  [&lt;ffffffff811929e6&gt;] ? vfs_writev+0x46/0x60 2017-02-24 08:29:11 [3183318.063997]  [&lt;ffffffff81192b11&gt;] ? sys_writev+0x51/0xd0 2017-02-24 08:29:11 [3183318.070389]  [&lt;ffffffff8153c64e&gt;] ? do_device_not_available+0xe/0x10 2017-02-24 08:29:11 [3183318.077955]  [&lt;ffffffff8100b0d2&gt;] ? system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Cheers,<br/> -- <br/> Kilian</p> <p>The patch that landed to b2_8 seems to handle most of the cases but we recently have found one application on our systems that causes this problem at random times.</p> <p>Correction: It looks closed to the <a href="https://jira.whamcloud.com/browse/LU-7067" title="vvp_io.c:1076:vvp_io_write_start()) ASSERTION( vio-&gt;vui_iocb-&gt;ki_pos == pos ) failed" class="issue-link" data-issue-key="LU-7067"><del>LU-7067</del></a></p> Duplicate LU-9106 Related LU-7067 LU-4257 LU-6260 Development Rank 1|hzxs7j: Rank (Obsolete) 9223372036854775807 Severity