https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12503 Lustre <p>We are running our lustre file system on 1 mds and 8 oss nodes. we are running lustre 2.10.6 on the lustre servers and clients.</p> <p>On one of the clients, we are exporting lustre via NFS3 and smb, it has been working fine for more than a year, but recently the client which is exporting lustre as NFS and smb start to crash due to a lustre bug as following:</p> <p> </p> <p>2014.148312] LustreError: 19435:0:(vvp_io.c:1056:vvp_io_write_start()) ASSERTION( vio-&gt;vui_iocb-&gt;ki_pos == pos ) failed: ki_pos 1209601876 [1209597952, 1210056704)<br/> [ 2014.148338] LustreError: 19435:0:(vvp_io.c:1056:vvp_io_write_start()) LBUG<br/> [ 2014.148352] Pid: 19435, comm: nfsd 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019<br/> [ 2014.148353] Call Trace:<br/> [ 2014.148376] <span class="error">&#91;&lt;ffffffffc0a0d7cc&gt;&#93;</span> libcfs_call_trace+0x8c/0xc0 <span class="error">&#91;libcfs&#93;</span><br/> [ 2014.148389] <span class="error">&#91;&lt;ffffffffc0a0d87c&gt;&#93;</span> lbug_with_loc+0x4c/0xa0 <span class="error">&#91;libcfs&#93;</span><br/> [ 2014.148394] <span class="error">&#91;&lt;ffffffffc1061270&gt;&#93;</span> vvp_io_write_start+0x790/0x820 <span class="error">&#91;lustre&#93;</span><br/> [ 2014.148419] <span class="error">&#91;&lt;ffffffffc0cb5328&gt;&#93;</span> cl_io_start+0x68/0x130 <span class="error">&#91;obdclass&#93;</span><br/> [ 2014.148449] <span class="error">&#91;&lt;ffffffffc0cb74fc&gt;&#93;</span> cl_io_loop+0xcc/0x1c0 <span class="error">&#91;obdclass&#93;</span><br/> [ 2014.148462] <span class="error">&#91;&lt;ffffffffc101765b&gt;&#93;</span> ll_file_io_generic+0x63b/0xcb0 <span class="error">&#91;lustre&#93;</span><br/> [ 2014.148470] <span class="error">&#91;&lt;ffffffffc10182f2&gt;&#93;</span> ll_file_aio_write+0x442/0x590 <span class="error">&#91;lustre&#93;</span><br/> [ 2014.148476] <span class="error">&#91;&lt;ffffffff8d040e6b&gt;&#93;</span> do_sync_readv_writev+0x7b/0xd0<br/> [ 2014.148480] <span class="error">&#91;&lt;ffffffff8d042aae&gt;&#93;</span> do_readv_writev+0xce/0x260<br/> [ 2014.148482] <span class="error">&#91;&lt;ffffffff8d042cd5&gt;&#93;</span> vfs_writev+0x35/0x60<br/> [ 2014.148484] <span class="error">&#91;&lt;ffffffffc0699f90&gt;&#93;</span> nfsd_vfs_write+0xc0/0x3a0 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.148492] <span class="error">&#91;&lt;ffffffffc069c962&gt;&#93;</span> nfsd_write+0x112/0x2a0 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.148498] <span class="error">&#91;&lt;ffffffffc06a3070&gt;&#93;</span> nfsd3_proc_write+0xc0/0x160 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.148504] <span class="error">&#91;&lt;ffffffffc0694810&gt;&#93;</span> nfsd_dispatch+0xe0/0x290 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.148509] <span class="error">&#91;&lt;ffffffffc0610cf3&gt;&#93;</span> svc_process_common+0x493/0x760 <span class="error">&#91;sunrpc&#93;</span><br/> [ 2014.148523] <span class="error">&#91;&lt;ffffffffc06110c3&gt;&#93;</span> svc_process+0x103/0x190 <span class="error">&#91;sunrpc&#93;</span><br/> [ 2014.148531] <span class="error">&#91;&lt;ffffffffc069416f&gt;&#93;</span> nfsd+0xdf/0x150 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.148535] <span class="error">&#91;&lt;ffffffff8cec1da1&gt;&#93;</span> kthread+0xd1/0xe0<br/> [ 2014.148539] <span class="error">&#91;&lt;ffffffff8d575c1d&gt;&#93;</span> ret_from_fork_nospec_begin+0x7/0x21<br/> [ 2014.148543] <span class="error">&#91;&lt;ffffffffffffffff&gt;&#93;</span> 0xffffffffffffffff<br/> [ 2014.148551] Kernel panic - not syncing: LBUG<br/> [ 2014.148561] CPU: 2 PID: 19435 Comm: nfsd Kdump: loaded Tainted: G OE ------------ 3.10.0-957.21.3.el7.x86_64 #1<br/> [ 2014.148579] Hardware name: Dell Inc. PowerEdge R640/0W23H8, BIOS 1.4.8 05/21/2018<br/> [ 2014.148592] Call Trace:<br/> [ 2014.148603] <span class="error">&#91;&lt;ffffffff8d563107&gt;&#93;</span> dump_stack+0x19/0x1b<br/> [ 2014.148615] <span class="error">&#91;&lt;ffffffff8d55c810&gt;&#93;</span> panic+0xe8/0x21f<br/> [ 2014.148629] <span class="error">&#91;&lt;ffffffffc0a0d8cb&gt;&#93;</span> lbug_with_loc+0x9b/0xa0 <span class="error">&#91;libcfs&#93;</span><br/> [ 2014.148650] <span class="error">&#91;&lt;ffffffffc1061270&gt;&#93;</span> vvp_io_write_start+0x790/0x820 <span class="error">&#91;lustre&#93;</span><br/> [ 2014.148675] <span class="error">&#91;&lt;ffffffffc0cb3357&gt;&#93;</span> ? cl_lock_request+0x67/0x1f0 <span class="error">&#91;obdclass&#93;</span><br/> [ 2014.148699] <span class="error">&#91;&lt;ffffffffc0cb5328&gt;&#93;</span> cl_io_start+0x68/0x130 <span class="error">&#91;obdclass&#93;</span><br/> [ 2014.148722] <span class="error">&#91;&lt;ffffffffc0cb74fc&gt;&#93;</span> cl_io_loop+0xcc/0x1c0 <span class="error">&#91;obdclass&#93;</span><br/> [ 2014.148739] <span class="error">&#91;&lt;ffffffffc101765b&gt;&#93;</span> ll_file_io_generic+0x63b/0xcb0 <span class="error">&#91;lustre&#93;</span><br/> [ 2014.148753] <span class="error">&#91;&lt;ffffffff8ced3250&gt;&#93;</span> ? check_preempt_curr+0x80/0xa0<br/> [ 2014.148771] <span class="error">&#91;&lt;ffffffffc10182f2&gt;&#93;</span> ll_file_aio_write+0x442/0x590 <span class="error">&#91;lustre&#93;</span><br/> [ 2014.148784] <span class="error">&#91;&lt;ffffffff8d040e6b&gt;&#93;</span> do_sync_readv_writev+0x7b/0xd0<br/> [ 2014.148914] <span class="error">&#91;&lt;ffffffff8d042aae&gt;&#93;</span> do_readv_writev+0xce/0x260<br/> [ 2014.149049] <span class="error">&#91;&lt;ffffffffc1017eb0&gt;&#93;</span> ? ll_file_splice_read+0x1e0/0x1e0 <span class="error">&#91;lustre&#93;</span><br/> [ 2014.149185] <span class="error">&#91;&lt;ffffffffc1018440&gt;&#93;</span> ? ll_file_aio_write+0x590/0x590 <span class="error">&#91;lustre&#93;</span><br/> [ 2014.149318] <span class="error">&#91;&lt;ffffffff8d11e003&gt;&#93;</span> ? ima_get_action+0x23/0x30<br/> [ 2014.149447] <span class="error">&#91;&lt;ffffffff8d11d51e&gt;&#93;</span> ? process_measurement+0x8e/0x250<br/> [ 2014.149578] <span class="error">&#91;&lt;ffffffff8d03f087&gt;&#93;</span> ? do_dentry_open+0x1e7/0x2e0<br/> [ 2014.149708] <span class="error">&#91;&lt;ffffffff8d042cd5&gt;&#93;</span> vfs_writev+0x35/0x60<br/> [ 2014.149841] <span class="error">&#91;&lt;ffffffffc0699f90&gt;&#93;</span> nfsd_vfs_write+0xc0/0x3a0 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.149975] <span class="error">&#91;&lt;ffffffffc069c962&gt;&#93;</span> nfsd_write+0x112/0x2a0 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.150109] <span class="error">&#91;&lt;ffffffffc06a3070&gt;&#93;</span> nfsd3_proc_write+0xc0/0x160 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.150243] <span class="error">&#91;&lt;ffffffffc0694810&gt;&#93;</span> nfsd_dispatch+0xe0/0x290 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.150381] <span class="error">&#91;&lt;ffffffffc0610cf3&gt;&#93;</span> svc_process_common+0x493/0x760 <span class="error">&#91;sunrpc&#93;</span><br/> [ 2014.150489] LustreError: 19462:0:(vvp_io.c:1056:vvp_io_write_start()) ASSERTION( vio-&gt;vui_iocb-&gt;ki_pos == pos ) failed: ki_pos 1211699028 [1211695104, 1212153856)<br/> [ 2014.150491] LustreError: 19462:0:(vvp_io.c:1056:vvp_io_write_start()) LBUG<br/> [ 2014.150492] Pid: 19462, comm: nfsd 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019<br/> [ 2014.150492] Call Trace:<br/> [ 2014.150514] <span class="error">&#91;&lt;ffffffffc0a0d7cc&gt;&#93;</span> libcfs_call_trace+0x8c/0xc0 <span class="error">&#91;libcfs&#93;</span><br/> [ 2014.150519] <span class="error">&#91;&lt;ffffffffc0a0d87c&gt;&#93;</span> lbug_with_loc+0x4c/0xa0 <span class="error">&#91;libcfs&#93;</span><br/> [ 2014.150533] <span class="error">&#91;&lt;ffffffffc1061270&gt;&#93;</span> vvp_io_write_start+0x790/0x820 <span class="error">&#91;lustre&#93;</span><br/> [ 2014.150551] <span class="error">&#91;&lt;ffffffffc0cb5328&gt;&#93;</span> cl_io_start+0x68/0x130 <span class="error">&#91;obdclass&#93;</span><br/> [ 2014.150564] <span class="error">&#91;&lt;ffffffffc0cb74fc&gt;&#93;</span> cl_io_loop+0xcc/0x1c0 <span class="error">&#91;obdclass&#93;</span><br/> [ 2014.150571] <span class="error">&#91;&lt;ffffffffc101765b&gt;&#93;</span> ll_file_io_generic+0x63b/0xcb0 <span class="error">&#91;lustre&#93;</span><br/> [ 2014.150577] <span class="error">&#91;&lt;ffffffffc10182f2&gt;&#93;</span> ll_file_aio_write+0x442/0x590 <span class="error">&#91;lustre&#93;</span><br/> [ 2014.150580] <span class="error">&#91;&lt;ffffffff8d040e6b&gt;&#93;</span> do_sync_readv_writev+0x7b/0xd0<br/> [ 2014.150581] <span class="error">&#91;&lt;ffffffff8d042aae&gt;&#93;</span> do_readv_writev+0xce/0x260<br/> [ 2014.150583] <span class="error">&#91;&lt;ffffffff8d042cd5&gt;&#93;</span> vfs_writev+0x35/0x60<br/> [ 2014.150589] <span class="error">&#91;&lt;ffffffffc0699f90&gt;&#93;</span> nfsd_vfs_write+0xc0/0x3a0 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.150594] <span class="error">&#91;&lt;ffffffffc069c962&gt;&#93;</span> nfsd_write+0x112/0x2a0 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.150599] <span class="error">&#91;&lt;ffffffffc06a3070&gt;&#93;</span> nfsd3_proc_write+0xc0/0x160 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.150603] <span class="error">&#91;&lt;ffffffffc0694810&gt;&#93;</span> nfsd_dispatch+0xe0/0x290 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.150613] <span class="error">&#91;&lt;ffffffffc0610cf3&gt;&#93;</span> svc_process_common+0x493/0x760 <span class="error">&#91;sunrpc&#93;</span><br/> [ 2014.150621] <span class="error">&#91;&lt;ffffffffc06110c3&gt;&#93;</span> svc_process+0x103/0x190 <span class="error">&#91;sunrpc&#93;</span><br/> [ 2014.150625] <span class="error">&#91;&lt;ffffffffc069416f&gt;&#93;</span> nfsd+0xdf/0x150 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.150627] <span class="error">&#91;&lt;ffffffff8cec1da1&gt;&#93;</span> kthread+0xd1/0xe0<br/> [ 2014.150630] <span class="error">&#91;&lt;ffffffff8d575c1d&gt;&#93;</span> ret_from_fork_nospec_begin+0x7/0x21<br/> [ 2014.150634] <span class="error">&#91;&lt;ffffffffffffffff&gt;&#93;</span> 0xffffffffffffffff<br/> [ 2014.152515] LustreError: 19480:0:(vvp_io.c:1056:vvp_io_write_start()) ASSERTION( vio-&gt;vui_iocb-&gt;ki_pos == pos ) failed: ki_pos 1213796180 [1213792256, 1214251008)<br/> [ 2014.152517] LustreError: 19480:0:(vvp_io.c:1056:vvp_io_write_start()) LBUG<br/> [ 2014.152518] Pid: 19480, comm: nfsd 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019<br/> [ 2014.152519] Call Trace:<br/> [ 2014.152542] <span class="error">&#91;&lt;ffffffffc0a0d7cc&gt;&#93;</span> libcfs_call_trace+0x8c/0xc0 <span class="error">&#91;libcfs&#93;</span><br/> [ 2014.152548] <span class="error">&#91;&lt;ffffffffc0a0d87c&gt;&#93;</span> lbug_with_loc+0x4c/0xa0 <span class="error">&#91;libcfs&#93;</span><br/> [ 2014.152569] <span class="error">&#91;&lt;ffffffffc1061270&gt;&#93;</span> vvp_io_write_start+0x790/0x820 <span class="error">&#91;lustre&#93;</span><br/> [ 2014.152593] <span class="error">&#91;&lt;ffffffffc0cb5328&gt;&#93;</span> cl_io_start+0x68/0x130 <span class="error">&#91;obdclass&#93;</span><br/> [ 2014.152610] <span class="error">&#91;&lt;ffffffffc0cb74fc&gt;&#93;</span> cl_io_loop+0xcc/0x1c0 <span class="error">&#91;obdclass&#93;</span><br/> [ 2014.152620] <span class="error">&#91;&lt;ffffffffc101765b&gt;&#93;</span> ll_file_io_generic+0x63b/0xcb0 <span class="error">&#91;lustre&#93;</span><br/> [ 2014.152630] <span class="error">&#91;&lt;ffffffffc10182f2&gt;&#93;</span> ll_file_aio_write+0x442/0x590 <span class="error">&#91;lustre&#93;</span><br/> [ 2014.152632] <span class="error">&#91;&lt;ffffffff8d040e6b&gt;&#93;</span> do_sync_readv_writev+0x7b/0xd0<br/> [ 2014.152634] <span class="error">&#91;&lt;ffffffff8d042aae&gt;&#93;</span> do_readv_writev+0xce/0x260<br/> [ 2014.152635] <span class="error">&#91;&lt;ffffffff8d042cd5&gt;&#93;</span> vfs_writev+0x35/0x60<br/> [ 2014.152643] <span class="error">&#91;&lt;ffffffffc0699f90&gt;&#93;</span> nfsd_vfs_write+0xc0/0x3a0 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.152649] <span class="error">&#91;&lt;ffffffffc069c962&gt;&#93;</span> nfsd_write+0x112/0x2a0 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.152655] <span class="error">&#91;&lt;ffffffffc06a3070&gt;&#93;</span> nfsd3_proc_write+0xc0/0x160 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.152661] <span class="error">&#91;&lt;ffffffffc0694810&gt;&#93;</span> nfsd_dispatch+0xe0/0x290 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.152671] <span class="error">&#91;&lt;ffffffffc0610cf3&gt;&#93;</span> svc_process_common+0x493/0x760 <span class="error">&#91;sunrpc&#93;</span><br/> [ 2014.152679] <span class="error">&#91;&lt;ffffffffc06110c3&gt;&#93;</span> svc_process+0x103/0x190 <span class="error">&#91;sunrpc&#93;</span><br/> [ 2014.152685] <span class="error">&#91;&lt;ffffffffc069416f&gt;&#93;</span> nfsd+0xdf/0x150 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.152687] <span class="error">&#91;&lt;ffffffff8cec1da1&gt;&#93;</span> kthread+0xd1/0xe0<br/> [ 2014.152689] <span class="error">&#91;&lt;ffffffff8d575c1d&gt;&#93;</span> ret_from_fork_nospec_begin+0x7/0x21<br/> [ 2014.152693] <span class="error">&#91;&lt;ffffffffffffffff&gt;&#93;</span> 0xffffffffffffffff<br/> [ 2014.157437] <span class="error">&#91;&lt;ffffffffc06110c3&gt;&#93;</span> svc_process+0x103/0x190 <span class="error">&#91;sunrpc&#93;</span><br/> [ 2014.157572] <span class="error">&#91;&lt;ffffffffc069416f&gt;&#93;</span> nfsd+0xdf/0x150 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.157704] <span class="error">&#91;&lt;ffffffffc0694090&gt;&#93;</span> ? nfsd_destroy+0x80/0x80 <span class="error">&#91;nfsd&#93;</span><br/> [ 2014.157835] <span class="error">&#91;&lt;ffffffff8cec1da1&gt;&#93;</span> kthread+0xd1/0xe0<br/> [ 2014.157963] <span class="error">&#91;&lt;ffffffff8cec1cd0&gt;&#93;</span> ? insert_kthread_work+0x40/0x40<br/> [ 2014.158094] <span class="error">&#91;&lt;ffffffff8d575c1d&gt;&#93;</span> ret_from_fork_nospec_begin+0x7/0x21<br/> [ 2014.158224] <span class="error">&#91;&lt;ffffffff8cec1cd0&gt;&#93;</span> ? insert_kthread_work+0x40/0x40<br/> (END)</p> <p> </p> <p>We have updated that client to lustre 2.12.2, but it did not help </p> Server: PowerEdge R640 with 64 GB memory and Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz <br/> OS: CentOS 7.5.1804 <br/> Lustre client: 2.12.2 LU-12503

LustreError: 19435:0:(vvp_io.c:1056:vvp_io_write_start()) LBUG

Bug Critical Resolved Fixed Zhenyu Xu Saerda Halifu Tue, 2 Jul 2019 11:36:49 +0000 Mon, 1 Jun 2020 00:54:17 +0000 Sat, 14 Dec 2019 13:34:49 +0000 Lustre 2.10.6 Lustre 2.12.2 Lustre 2.14.0 Lustre 2.12.4 0 8 <p>Dear Zhenyu Xu, </p> <p> </p> <p>Thanks for looking into this issue, will it be possible to get a patch for this bug before 2.13.0 release?</p> <p>Is there any way to avoid it?  And I very much would like to know what is actually causing this bug?</p> <p> </p> <p>Best Regards</p> <p> </p> <p>Saerda  </p> <p>Observations so far:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>thread 19435 has iocb-&gt;ki_pos = 0x4819,0F54 while its io-&gt;u.ci_rw.crw_pos is 0x4819,0000 write count is 64K(0x1,000) thread 19462 0x4839,0F54 0x4839,0000 64K thread 19480 0x4859,0F54 0x4859,0000 64K </pre> </div></div> <p>I don't know how come the ki_pos is not updated by page alignment (it should be IMO).</p> <p>The iocb's ki_pos could get updated during __generic_file_write_iter() in vvp_io_write_start(), while later got updated again with crw_pos. This code was changed in commit 2b0a34fe43bf4fce5560af61a45e5393c96070a9, before the commit, ll_file_io_generic() uses its own iocb and pos, and only update outside kiocb's ki_pos by the bytes that have been written after finished the IO loop. While I'm not sure whether that change affects it or not (i.e. the root cause hasn't been found), but the hunch is that using variables local to the function should avoid the complexity.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=bobijam" class="user-hover" rel="bobijam">bobijam</a></p> <p>This issue was initially seen on 2.10.6 which pre-dates the <a href="https://jira.whamcloud.com/browse/LU-11825" title="Remove LU-8964/pio feature &amp; supporting framework" class="issue-link" data-issue-key="LU-11825"><del>LU-11825</del></a> change. If <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=halifu" class="user-hover" rel="halifu">halifu</a> has an easy reproducer could a diagnostic patch that could help spotlight the root cause?</p> <p>Peter</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/35516" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35516</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12503" title="LustreError: 19435:0:(vvp_io.c:1056:vvp_io_write_start()) LBUG" class="issue-link" data-issue-key="LU-12503"><del>LU-12503</del></a> llite: debug file pos mimatch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 53db2889ac057a493601e1fdec9677786e19fdf0</p> <p>H Saerda Halifu,</p> <p>The debug patch (<a href="https://review.whamcloud.com/35516" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35516</a>) is for client only, apply it to one client node and to see whether this issue hit again (hoping it's easy to reproduce) , if that's the case, collect the logs and upload here. Thank you.</p> <p>Hi Zhenyu Xu,</p> <p> </p> <p>Thanks, I will apply it, and will let you know.</p> <p>Best Regards</p> <p> </p> <p>Saerda</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=halifu" class="user-hover" rel="halifu">halifu</a> any change in the frequency of the occurrence of the crash with the debug patch applied?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=halifu" class="user-hover" rel="halifu">halifu</a> any news?</p> <p>Hi Peter, </p> <p> </p> <p>Sorry for the late answer, I was away for vacation.</p> <p>I have downloaded lustre-2.12.1-1.src.rpm, and unpacked it, replaced vvp_io.c file with the new one from the patch.</p> <p>I was able to create a new source rpm but I am not able to rebuild the Lustre-client rpm from this new source.</p> <p>when I run :</p> <p>rpmbuild --rebuild --without servers lustre-2.12.1-1.el7.src.rpm</p> <p>I got the following error message:</p> <p>Making all in .<br/> /root/rpmbuild/BUILD/lustre-2.12.1/lustre/llite/vvp_io.c: In function 'vvp_io_read_start':<br/> /root/rpmbuild/BUILD/lustre-2.12.1/lustre/llite/vvp_io.c:783:8: error: 'struct cl_io' has no member named 'ci_async_readahead'<br/> if (io-&gt;ci_async_readahead) {<br/> ^<br/> In file included from /root/rpmbuild/BUILD/lustre-2.12.1/libcfs/include/libcfs/libcfs.h:43:0,<br/> from /root/rpmbuild/BUILD/lustre-2.12.1/lustre/include/lustre_lib.h:50,<br/> from /root/rpmbuild/BUILD/lustre-2.12.1/lustre/include/obd.h:41,<br/> from /root/rpmbuild/BUILD/lustre-2.12.1/lustre/llite/vvp_io.c:41:<br/> /root/rpmbuild/BUILD/lustre-2.12.1/lustre/llite/vvp_io.c: In function 'vvp_io_write_start':<br/> /root/rpmbuild/BUILD/lustre-2.12.1/lustre/llite/vvp_io.c:1074:25: error: 'struct ll_sb_info' has no member named 'll_fsname'<br/> ll_i2sbi(inode)-&gt;ll_fsname,<br/> ^<br/> /root/rpmbuild/BUILD/lustre-2.12.1/libcfs/include/libcfs/libcfs_debug.h:158:55: note: in definition of macro '__CDEBUG'<br/> libcfs_debug_msg(&amp;msgdata, format, ## _<em>VA_ARGS</em>_); \<br/> ^<br/> /root/rpmbuild/BUILD/lustre-2.12.1/lustre/llite/vvp_io.c:1072:3: note: in expansion of macro 'CDEBUG'<br/> CDEBUG(D_INODE,<br/> ^<br/> /root/rpmbuild/BUILD/lustre-2.12.1/lustre/llite/vvp_io.c: In function 'vvp_io_init':<br/> /root/rpmbuild/BUILD/lustre-2.12.1/lustre/llite/vvp_io.c:1516:26: error: 'struct ll_sb_info' has no member named 'll_fsname'<br/> ll_i2sbi(inode)-&gt;ll_fsname,<br/> ^<br/> /root/rpmbuild/BUILD/lustre-2.12.1/libcfs/include/libcfs/libcfs_debug.h:158:55: note: in definition of macro '__CDEBUG'<br/> libcfs_debug_msg(&amp;msgdata, format, ## _<em>VA_ARGS</em>_); \<br/> ^<br/> /root/rpmbuild/BUILD/lustre-2.12.1/libcfs/include/libcfs/libcfs_debug.h:189:37: note: in expansion of macro 'CDEBUG_LIMIT'<br/> #define CERROR(format, ...) CDEBUG_LIMIT(D_ERROR, format, ## _<em>VA_ARGS</em>_)<br/> ^<br/> /root/rpmbuild/BUILD/lustre-2.12.1/lustre/llite/vvp_io.c:1515:4: note: in expansion of macro 'CERROR'<br/> CERROR("%s: refresh file layout " DFID " error %d.\n",<br/> ^<br/> /root/rpmbuild/BUILD/lustre-2.12.1/lustre/llite/vvp_io.c: At top level:<br/> cc1: error: unrecognized command line option "-Wno-format-truncation" <span class="error">&#91;-Werror&#93;</span><br/> cc1: all warnings being treated as errors<br/> make<span class="error">&#91;6&#93;</span>: *** <span class="error">&#91;/root/rpmbuild/BUILD/lustre-2.12.1/lustre/llite/vvp_io.o&#93;</span> Error 1<br/> make<span class="error">&#91;6&#93;</span>: *** Waiting for unfinished jobs....<br/> make<span class="error">&#91;5&#93;</span>: *** <span class="error">&#91;/root/rpmbuild/BUILD/lustre-2.12.1/lustre/llite&#93;</span> Error 2<br/> make<span class="error">&#91;5&#93;</span>: *** Waiting for unfinished jobs....<br/> make<span class="error">&#91;4&#93;</span>: *** <span class="error">&#91;/root/rpmbuild/BUILD/lustre-2.12.1/lustre&#93;</span> Error 2<br/> make<span class="error">&#91;3&#93;</span>: *** <span class="error">&#91;_module_/root/rpmbuild/BUILD/lustre-2.12.1&#93;</span> Error 2<br/> make<span class="error">&#91;2&#93;</span>: *** <span class="error">&#91;modules&#93;</span> Error 2<br/> make<span class="error">&#91;1&#93;</span>: *** <span class="error">&#91;all-recursive&#93;</span> Error 1<br/> make: *** <span class="error">&#91;all&#93;</span> Error 2<br/> error: Bad exit status from /var/tmp/rpm-tmp.MLq9qb (%build)</p> <p> </p> <p>For me it looks like it fails before it comes to the changes in the code?</p> <p>Am I doing something wrong ?</p> <p> </p> <p>Best Regards</p> <p>Saerda</p> <p>I would have suggested that it would be simpler to just use the build products from the Jenkins build- <a href="https://build.whamcloud.com/job/lustre-reviews/67091/arch=x86_64,build_type=client,distro=el7.6,ib_stack=inkernel/artifact/artifacts/" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/67091/arch=x86_64,build_type=client,distro=el7.6,ib_stack=inkernel/artifact/artifacts/</a> - but I see that you are using RHEL 7.5 so perhaps building from the SRPMs in the same location would be easier. That or temporarily move just one client to a kernel version that allow you to use the build products (then reset it back to what you want to use after running the test)</p> <p>Hi Peter, </p> <p>Thanks for the tips, I manage to get the right build and installed it. I have following rpms installed on this lustre client:</p> <p>rpm -qa |grep lustre<br/> kmod-lustre-client-2.12.56_85_g60392e6-1.el7.x86_64<br/> lustre-client-2.12.56_85_g60392e6-1.el7.x86_64<br/> kmod-lustre-client-tests-2.12.56_85_g60392e6-1.el7.x86_64</p> <p> </p> <p>Now I started NFS service on this lustre client, I will see when/if this will crash, if it does, I will upload the dump.</p> <p> </p> <p>Best Regards</p> <p>Saerda</p> <p>How frequently was the crash occurring before the debug patch was applied?</p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/35765" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35765</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12503" title="LustreError: 19435:0:(vvp_io.c:1056:vvp_io_write_start()) LBUG" class="issue-link" data-issue-key="LU-12503"><del>LU-12503</del></a> vvp_dev: increment *pos in .next<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d51de92e086e88c97d1e5d671154188bd9b36ffb</p> <p>Just wondering if this is a side effect of a bug found upstream that was fixed. Can you give it a try.</p> <p>Hi,</p> <p>My server manages to run without problem for the last 6 days after I have applied the debug patch.</p> <p>Before that, the server used to crash quite often, some times a couple of hours after I start NFS export, some times after a day or so.</p> <p>I also went through the logs, didn't see anything suspicious. </p> <p> </p> <p>I think this might have something to do with user activities, for example how users are reading and writing data to NFS exports.</p> <p> </p> <p>Best Regards</p> <p> </p> <p>Saerda</p> <p> </p> <p> </p> <p> </p> <p> </p> <p>ok, well, considering that this issue has been seen on older versions too I think that we should drop the prriority from Blocker to Critical.</p> <p>My server crashed today. I managed to upload vmcore-dmesg.txt file.</p> <p>Let me know if you need more information.</p> <p> </p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35765/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35765/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12503" title="LustreError: 19435:0:(vvp_io.c:1056:vvp_io_write_start()) LBUG" class="issue-link" data-issue-key="LU-12503"><del>LU-12503</del></a> vvp_dev: increment *pos in .next<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 02336a9a5d096dc9a603ed0e77e0c7cf7b41ffb3</p> <p>ok so James's patch has landed to master. Can we port it to b2_12 so that <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=halifu" class="user-hover" rel="halifu">halifu</a> can verify whether it is a fix? Or <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a> does the latest crash info confirm that this is indeed the issue?</p> <p>Ah, right.</p> <p>James, I'm almost certain the code your patch touches is only called in the dump page cache path, which is strictly a special, extreme debug path, and there's basically no way it would be invoked here.  Have I missed something?  Otherwise it can't be the fix for this.  (It's still correct and useful, it's just not a fix for this.)</p> <p>I agree with Patrick. It's a fix but not one to handle this problem. I was hoping it might address this issue <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/36021" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36021</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12503" title="LustreError: 19435:0:(vvp_io.c:1056:vvp_io_write_start()) LBUG" class="issue-link" data-issue-key="LU-12503"><del>LU-12503</del></a> llite: debug file pos mimatch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bf11eb0f6c4ead18897b14d3ff2b8ef09a72f97a</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=bobijam" class="user-hover" rel="bobijam">bobijam</a></p> <p>It looks like you have turned your original debug patch into a fix and now have added a new debug patch. Are you hoping for <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=halifu" class="user-hover" rel="halifu">halifu</a> to use both of these?</p> <p>Peter</p> <p>yes, I hope the fix patch can handle the issue, and add the debug patch to catch info if that's not the right fix for this issue.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36021/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36021/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12503" title="LustreError: 19435:0:(vvp_io.c:1056:vvp_io_write_start()) LBUG" class="issue-link" data-issue-key="LU-12503"><del>LU-12503</del></a> llite: file write pos mimatch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1d2aa1513dc4e65813ad0bea138966a55244dbde</p> <p>Landed for 2.14</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37034" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37034</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12503" title="LustreError: 19435:0:(vvp_io.c:1056:vvp_io_write_start()) LBUG" class="issue-link" data-issue-key="LU-12503"><del>LU-12503</del></a> llite: file write pos mimatch<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 7e79fc11ed73b291b8e7a4805b3f1144d71ff83f</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37035" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37035</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12503" title="LustreError: 19435:0:(vvp_io.c:1056:vvp_io_write_start()) LBUG" class="issue-link" data-issue-key="LU-12503"><del>LU-12503</del></a> vvp_dev: increment *pos in .next<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: cb2f58387f30271074dac0fcf021b5db157022e2</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37034/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37034/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12503" title="LustreError: 19435:0:(vvp_io.c:1056:vvp_io_write_start()) LBUG" class="issue-link" data-issue-key="LU-12503"><del>LU-12503</del></a> llite: file write pos mimatch<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 322cd140132e821c63b41b7da9ddb9f519b52194</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37035/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37035/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12503" title="LustreError: 19435:0:(vvp_io.c:1056:vvp_io_write_start()) LBUG" class="issue-link" data-issue-key="LU-12503"><del>LU-12503</del></a> vvp_dev: increment *pos in .next<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 589ba9b62c0e8b3a93145dd44bbbd92a26d6da8b</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/37921" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37921</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12503" title="LustreError: 19435:0:(vvp_io.c:1056:vvp_io_write_start()) LBUG" class="issue-link" data-issue-key="LU-12503"><del>LU-12503</del></a> llite: file write pos mimatch<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: d5a087e31b1e1cf6812640d476faae8774ba0d66</p> Related LU-11825 Development Epic/Theme NFS Rank 1|i00j33: Rank (Obsolete) 9223372036854775807 Severity