https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2910 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/3fa2a912-85b2-11e2-9f8d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/3fa2a912-85b2-11e2-9f8d-52540035b04c</a>.</p> <p>The sub-test test_1 failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>client console shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>18:47:35:Lustre: DEBUG MARKER: == racer test 1: racer on clients: client-22-ib,client-23-ib.lab.whamcloud.com DURATION=900 == 18:47:28 (1362451648) 18:47:35:Lustre: DEBUG MARKER: DURATION=900 MDSCOUNT=1 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre2/racer 18:47:35:Lustre: DEBUG MARKER: DURATION=900 MDSCOUNT=1 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre/racer 18:51:19:LustreError: 8649:0:(file.c:2601:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000bd0:0x9095:0x0] error: rc = -116 18:56:24:LustreError: 28058:0:(lcommon_cl.c:821:ccc_io_advance()) ASSERTION( cio-&gt;cui_tot_nrsegs &gt;= cio-&gt;cui_nrsegs ) failed: 18:56:24:LustreError: 28058:0:(lcommon_cl.c:821:ccc_io_advance()) LBUG 18:56:24:Pid: 28058, comm: cat 18:56:24: 18:56:24:Call Trace: 18:56:24: [&lt;ffffffffa03fe895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 18:56:24: [&lt;ffffffffa03fee97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 18:56:24: [&lt;ffffffffa1264c0e&gt;] ccc_io_advance+0x11e/0x190 [lustre] 18:56:24: [&lt;ffffffffa05a96d6&gt;] cl_io_rw_advance+0x66/0x150 [obdclass] 18:56:24: [&lt;ffffffffa05ae602&gt;] cl_io_loop+0xe2/0x1b0 [obdclass] 18:56:24: [&lt;ffffffffa1215323&gt;] ll_file_io_generic+0x223/0x570 [lustre] 18:56:24: [&lt;ffffffffa12157af&gt;] ll_file_aio_read+0x13f/0x2c0 [lustre] 18:56:25: [&lt;ffffffffa121635c&gt;] ll_file_read+0x16c/0x2a0 [lustre] 18:56:25: [&lt;ffffffff81176cb5&gt;] vfs_read+0xb5/0x1a0 18:56:25: [&lt;ffffffff81176df1&gt;] sys_read+0x51/0x90 18:56:25: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 18:56:25: 18:56:25:Kernel panic - not syncing: LBUG 18:56:25:Pid: 28058, comm: cat Not tainted 2.6.32-279.19.1.el6.x86_64 #1 18:56:25:Call Trace: 18:56:25: [&lt;ffffffff814e9541&gt;] ? panic+0xa0/0x168 18:56:25: [&lt;ffffffffa03feeeb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] 18:56:25: [&lt;ffffffffa1264c0e&gt;] ? ccc_io_advance+0x11e/0x190 [lustre] 18:56:25: [&lt;ffffffffa05a96d6&gt;] ? cl_io_rw_advance+0x66/0x150 [obdclass] 18:56:25: [&lt;ffffffffa05ae602&gt;] ? cl_io_loop+0xe2/0x1b0 [obdclass] 18:56:25: [&lt;ffffffffa1215323&gt;] ? ll_file_io_generic+0x223/0x570 [lustre] 18:56:25: [&lt;ffffffffa12157af&gt;] ? ll_file_aio_read+0x13f/0x2c0 [lustre] 18:56:25: [&lt;ffffffffa121635c&gt;] ? ll_file_read+0x16c/0x2a0 [lustre] 18:56:25: [&lt;ffffffff81176cb5&gt;] ? vfs_read+0xb5/0x1a0 18:56:25: [&lt;ffffffff81176df1&gt;] ? sys_read+0x51/0x90 18:56:26: [&lt;ffffffff8100b072&gt;] ? system_call_fastpath+0x16/0x1b 18:56:26:Initializing cgroup subsys cpuset 18:56:26:Initializing cgroup subsys cpu </pre> </div></div> LU-2910

racer test_1, sanity test_54c: ASSERTION( cio->cui_tot_nrsegs >= cio->cui_nrsegs ) failed

Bug Blocker Resolved Fixed Niu Yawei Maloo LB Tue, 5 Mar 2013 14:31:56 +0000 Mon, 18 Mar 2013 23:20:36 +0000 Mon, 18 Mar 2013 23:20:36 +0000 Lustre 2.4.0 Lustre 2.4.0 0 10 <p>The client debug log in above maloo link is unfortunately missed, Sarah, could you try to reproduce it and collect the client debug log (with D_VFSTRACE enabled)? Thanks in advance.</p> <p>Looks the LASSERT could be triggered when we restart io with not restored iovector: <a href="http://review.whamcloud.com/5652" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5652</a></p> <p>is file_swap.sh enabled in this run?</p> <p>Xiong, I think the file_swap.sh should be disabled by default. Sarah knows it better.</p> <p>I'm thinking in which situation it would make IO restart. Also it may be incorrect to restart whole IO in some circumstances.</p> <p>Now that the log is missed, I'd like to wait for a while to see if this issue can be reproduced. I'd like to make it clear before making a fix.</p> <p>Xiong, in the racer test, file size could be changed anytime (mostly from rename, I think), so it's possible trigger io restart when ccc_prep_size() thinking that the ppos is beyond of file size, isn't it?</p> <p>From the current code, seems we support only the whole io restart when there isn't any bit of read/write done yet, we don't support io restart when partial io done, right?</p> <p>I ran this test on Friday 4 times and cannot reproduce it on my test nodes. BTW Xiong, I cannot find file_swap.sh under tests/racer, could you please point me out? Thanks.</p> <p>Are you in master tree? Try to run the following command and see what you get:</p> <p><span class="error">&#91;jinxiong@intel lustre&#93;</span>$ git annotate tests/racer/file_swap.sh <br/> 4af3ab19 (Jinshan Xiong 2013-02-01 10:33:09 -0800 1)#!/bin/bash<br/> 4af3ab19 (Jinshan Xiong 2013-02-01 10:33:09 -0800 2)<br/> 4af3ab19 (Jinshan Xiong 2013-02-01 10:33:09 -0800 3)DIR=$1<br/> 4af3ab19 (Jinshan Xiong 2013-02-01 10:33:09 -0800 4)MAX=$2<br/> 4af3ab19 (Jinshan Xiong 2013-02-01 10:33:09 -0800 5)<br/> 4af3ab19 (Jinshan Xiong 2013-02-01 10:33:09 -0800 6)while : ; do</p> <p>Xiong, have you added the file into Makefile.am? If no, then it will not be built into the rpms</p> <p>ah no, i didn't</p> <p>Can this be reduced from a blocker since it doesn't sound like it can be reproduced?</p> <p>Didn't hit it in the latest tag testing, I think this can be dropped to major</p> <p>Though it's hard to be reproduced, from the code we can see if io restared, the iov_len could be messed up and this LASSERT will be triggered at the end.</p> <p>Xiong, if you think layout change in racer test is unexpected, then there could be something serious wrong in the layout code, because I can see lots of io restart in my local testing (without file_swap, most io restart come from vvp_io_fini(), some from can_populate_pages()). Is it possible caused by rename?</p> <p>Niu, you're right about this. There exists fake layout change after layout swapping is introduced, and rename can cause this because it revokes full ibits lock.</p> <p>patch landed for 2.4</p> <p>I'm hitting this reliably using 2.3.62-47-g3a0bb68 (which includes the patch here) on my home setup. The easiest way to reproduce for my is to run sanity 54c in a loop. Maybe this should be reopened.</p> <p>What I saw is another LASSERT which is newly added from the patch:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 31446:0:(lcommon_cl.c:794:ccc_io_update_iov()) ASSERTION( cio-&gt;cui_tot_nrsegs &gt;= cio-&gt;cui_nrsegs ) failed: tot_nrsegs: 0, nrsegs: 1 LustreError: 31446:0:(lcommon_cl.c:794:ccc_io_update_iov()) LBUG Kernel panic - not syncing: LBUG Pid: 31446, comm: loop3 Tainted: P --------------- 2.6.32 #1 Call Trace: [&lt;ffffffff814fe08e&gt;] ? panic+0xa0/0x168 [&lt;ffffffffa10b3f5b&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [&lt;ffffffffa0f6e569&gt;] ? ccc_io_update_iov+0xe9/0xf0 [lustre] [&lt;ffffffffa0f77d8c&gt;] ? vvp_io_rw_lock+0x8c/0x690 [lustre] [&lt;ffffffffa10c4301&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa0f783c5&gt;] ? vvp_io_write_lock+0x35/0x40 [lustre] [&lt;ffffffffa06e03d3&gt;] ? cl_io_lock+0x63/0x560 [obdclass] [&lt;ffffffffa10bedf8&gt;] ? libcfs_log_return+0x28/0x40 [libcfs] [&lt;ffffffffa0f4a2bf&gt;] ? ll_cl_init+0x38f/0x570 [lustre] [&lt;ffffffffa0f4a6e3&gt;] ? ll_prepare_write+0x53/0x1a0 [lustre] [&lt;ffffffffa0f62c0e&gt;] ? ll_write_begin+0x7e/0x1a0 [lustre] [&lt;ffffffff8111346c&gt;] ? pagecache_write_begin+0x1c/0x20 [&lt;ffffffff8135ef05&gt;] ? do_lo_send_aops+0x155/0x1a0 [&lt;ffffffff8135f15b&gt;] ? do_bio_filebacked+0x20b/0x2d0 [&lt;ffffffff8135edb0&gt;] ? do_lo_send_aops+0x0/0x1a0 [&lt;ffffffff8135f2f1&gt;] ? loop_thread+0xd1/0x270 [&lt;ffffffff81092180&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffff8135f220&gt;] ? loop_thread+0x0/0x270 [&lt;ffffffff81091e16&gt;] ? kthread+0x96/0xa0 [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 [&lt;ffffffff81091d80&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Actually, sometimes the tot_nrsegs could be zero (see ll_cl_init()), the newly added assert isn't correct in such case.</p> <p>Skip iov update when tot_nrsegs is zero: <a href="http://review.whamcloud.com/5747" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5747</a></p> <p>Increasing priority, since this is causing almost every test to fail.</p> <p>hit this <a href="https://maloo.whamcloud.com/test_sets/7e10a3f6-8e87-11e2-a173-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7e10a3f6-8e87-11e2-a173-52540035b04c</a></p> <p>Jinshan replied to my comment on Patch Set 1:</p> <blockquote> <p>"Failed tests always show "tot_nrsegs: 0" in the LASSERT() so I can imagine this will fix the problem."</p> <p>The root cause of this problem is lloop write</p></blockquote> <p>Is there something that should be fixed in the lloop code? There has been more interest in this recently from Robert and Richard for VM provisioning and other cloud applications.</p> <blockquote> <p>Is there something that should be fixed in the lloop code? There has been more interest in this recently from Robert and Richard for VM provisioning and other cloud applications.</p></blockquote> <p>Did I say lustre loop device? I meant to say linux kernel loop device, sorry.</p> <p>First of all, we should figure out what kind of IO has been received by lloop device then we can work out corresponding fix. In current implementation of lloop, we use direct IO to serve bio requests so an obvious way to improve performance is to do cache write - especially if we find out there are plenty of small IOs.</p> <p>Landed for 2.4</p> Development Rank 1|hzvk7r: Rank (Obsolete) 7001 Severity