https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4503 Lustre <p>== sanity-benchmark test dbench: dbench == 13:07:36 (1389949656)<br/> debug=0<br/> running as uid/gid/euid/egid 500/500/500/500, groups:<br/> <span class="error">&#91;touch&#93;</span> <span class="error">&#91;/mnt/lustre/d0_runas_test/f13997&#93;</span><br/> debug=0<br/> running as uid/gid/euid/egid 500/500/500/500, groups:<br/> <span class="error">&#91;bash&#93;</span> <span class="error">&#91;rundbench&#93;</span> <span class="error">&#91;-D&#93;</span> <span class="error">&#91;/mnt/lustre/d0.rhel6-64.shadowland&#93;</span> <span class="error">&#91;6&#93;</span> <span class="error">&#91;-t&#93;</span> <span class="error">&#91;120&#93;</span><br/> copying /usr/local/share/client.txt to /mnt/lustre/d0.rhel6-64.shadowland/client.txt</p> <p>Message from syslogd@rhel6-64 at Jan 17 13:07:37 ...<br/> kernel:LustreError: 16666:0:(lu_ref.c:265:lu_ref_del()) ASSERTION( 0 ) failed: </p> <p>Message from syslogd@rhel6-64 at Jan 17 13:07:37 ...<br/> kernel:LustreError: 16666:0:(lu_ref.c:265:lu_ref_del()) LBUG</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400):0:mdt LustreError: 16666:0:(lu_ref.c:105:lu_ref_print()) lu_ref: ffff88013997cee8 2 0 cl_page_alloc:197 LustreError: 16666:0:(lu_ref.c:107:lu_ref_print()) link: cl_io ffff880097136088 LustreError: 16666:0:(lu_ref.c:107:lu_ref_print()) link: transfer ffff88013997cde0 LustreError: 16666:0:(lu_ref.c:105:lu_ref_print()) lu_ref: ffff88008b045698 2 0 cl_page_alloc:197 LustreError: 16666:0:(lu_ref.c:265:lu_ref_del()) ASSERTION( 0 ) failed: LustreError: 16666:0:(lu_ref.c:265:lu_ref_del()) LBUG Pid: 16666, comm: cp Call Trace: [&lt;ffffffffa03ea8c5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa03eaec7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa05a78ed&gt;] lu_ref_del+0x23d/0x240 [obdclass] [&lt;ffffffffa0f9e826&gt;] write_commit_callback+0x86/0xb0 [lustre] [&lt;ffffffffa09fb26f&gt;] osc_io_commit_async+0xaf/0x3b0 [osc] [&lt;ffffffffa0f9e7a0&gt;] ? write_commit_callback+0x0/0xb0 [lustre] [&lt;ffffffffa0f9e7a0&gt;] ? write_commit_callback+0x0/0xb0 [lustre] [&lt;ffffffffa05a1dd6&gt;] cl_io_commit_async+0x76/0x130 [obdclass] [&lt;ffffffffa0a47917&gt;] lov_io_commit_async+0x2d7/0x500 [lov] [&lt;ffffffffa0f9e7a0&gt;] ? write_commit_callback+0x0/0xb0 [lustre] [&lt;ffffffffa0f9e7a0&gt;] ? write_commit_callback+0x0/0xb0 [lustre] [&lt;ffffffffa05a1dd6&gt;] cl_io_commit_async+0x76/0x130 [obdclass] [&lt;ffffffffa0fa0417&gt;] vvp_io_write_commit+0x267/0x8c0 [lustre] [&lt;ffffffffa059797b&gt;] ? cl_page_get+0x2b/0x100 [obdclass] [&lt;ffffffffa0f8ca3c&gt;] ll_write_end+0xbc/0x3e0 [lustre] [&lt;ffffffff81129e5a&gt;] generic_file_buffered_write+0x18a/0x300 [&lt;ffffffff8153481b&gt;] ? _spin_unlock+0x2b/0x40 [&lt;ffffffff8112bed0&gt;] __generic_file_aio_write+0x260/0x490 [&lt;ffffffff8112c173&gt;] ? generic_file_aio_write+0x73/0x100 [&lt;ffffffff8112c18a&gt;] generic_file_aio_write+0x8a/0x100 [&lt;ffffffffa0fa0b4b&gt;] vvp_io_write_start+0xdb/0x3d0 [lustre] [&lt;ffffffffa05a1efa&gt;] cl_io_start+0x6a/0x140 [obdclass] [&lt;ffffffffa05a6024&gt;] cl_io_loop+0xb4/0x1b0 [obdclass] [&lt;ffffffffa0f42326&gt;] ll_file_io_generic+0x2b6/0x710 [lustre] [&lt;ffffffffa05958a9&gt;] ? cl_env_get+0x29/0x350 [obdclass] [&lt;ffffffffa0f42ff2&gt;] ll_file_aio_write+0x142/0x2c0 [lustre] [&lt;ffffffffa0f432dc&gt;] ll_file_write+0x16c/0x2a0 [lustre] [&lt;ffffffff81196568&gt;] vfs_write+0xb8/0x1a0 [&lt;ffffffff815342e8&gt;] ? lockdep_sys_exit_thunk+0x35/0x67 [&lt;ffffffff81196e61&gt;] sys_write+0x51/0x90 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> rhel6 last master with lu_ref checks enabled LU-4503

Panic with lu_ref checks enabled

Bug Critical Resolved Fixed Zhenyu Xu Alexey Lyashkov MB Fri, 17 Jan 2014 09:10:04 +0000 Thu, 14 Aug 2014 07:34:17 +0000 Thu, 14 Aug 2014 07:34:17 +0000 Lustre 2.6.0 Lustre 2.7.0 0 5 <p>I am also hitting this.</p> <p>apparently the problem now stems from <a href="http://review.whamcloud.com/7893" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7893</a> that does not release the page to wroteback list, but aggregates them to be freed by later async callback that could be running from a different io context</p> <p>the io parameter for lu_ref_del() should be converted to top io and that's all.</p> <p>patch tracking at <a href="http://review.whamcloud.com/8970" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8970</a></p> <p>did you run acc-sm with lu_refcheck enabled ? of just fix that particular issue ?</p> <p>did you find more issues? I've run some test with enable-lu_ref.</p> <p>I just ask before i will start own stress testing. i hit it bug with simple run, and don't like to stop testing / find other clio bugs (like lack a mutex lock) after panic at next simple test. Also i know - Maloo don't run tests with recheck enabled - so any bugs in that area will don't found during automatic testing and acc-sm need to be run by hand if we need verification for a lu_refcheck.</p> <p>ok, thank you for the explanation, I've run some test and am still running other tests, haven't finished acc-sm yet for now.</p> <p>Thanks for testing.</p> <p>I hit more problems with lu_ref checking enabled in my testing but mostly in mdc now, unrelated to clio</p> <p>Oleg,</p> <p>what you think about enabling lu_ref checks and invariants for a Maloo testing ? I think it's good to have better code quality.</p> <p>Yes, I totally agre we need to ahve at least some portion of the runs with debug enabled.</p> <p>patch landed for 2.7 (master branch)</p> Development Rank 1|hzwd1b: Rank (Obsolete) 12316 Severity