https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5700 Lustre <p>LustreError: 58185:0:(rw26.c:474:ll_direct_IO_26()) ASSERTION( obj-&gt;cob_transient_pages == 0 ) failed:<br/> LustreError: 58184:0:(rw26.c:474:ll_direct_IO_26()) ASSERTION( obj-&gt;cob_transient_pages == 0 ) failed:<br/> LustreError: 58184:0:(rw26.c:474:ll_direct_IO_26()) LBUG<br/> Pid: 58184, comm: sio</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa057d895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa057de97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c03044&gt;&#93;</span> ll_direct_IO_26+0xb64/0x1140 <span class="error">&#91;lustre&#93;</span><br/> LustreError: 58186:0:(rw26.c:474:ll_direct_IO_26()) ASSERTION( obj-&gt;cob_transient_pages == 0 ) failed:<br/> LustreError: 58186:0:(rw26.c:474:ll_direct_IO_26()) LBUG<br/> Pid: 58186, comm: sio</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffff81058bd3&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> <span class="error">&#91;&lt;ffffffff8112158b&gt;&#93;</span> generic_file_aio_read+0x6bb/0x700<br/> <span class="error">&#91;&lt;ffffffff811b4b5f&gt;&#93;</span> ? list_move+0x1f/0x30<br/> <span class="error">&#91;&lt;ffffffff811b51ff&gt;&#93;</span> ? __mark_inode_dirty+0x13f/0x160<br/> <span class="error">&#91;&lt;ffffffff811a59cd&gt;&#93;</span> ? touch_atime+0x14d/0x1a0<br/> <span class="error">&#91;&lt;ffffffffa0c173be&gt;&#93;</span> vvp_io_read_start+0x22e/0x450 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa057d895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa057de97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> LustreError: 58183:0:(rw26.c:474:ll_direct_IO_26()) ASSERTION( obj-&gt;cob_transient_pages == 0 ) failed:<br/> LustreError: 58183:0:(rw26.c:474:ll_direct_IO_26()) LBUG<br/> Pid: 58183, comm: sio</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa0c03044&gt;&#93;</span> ll_direct_IO_26+0xb64/0x1140 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa057d895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81058bd3&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> <span class="error">&#91;&lt;ffffffffa057de97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8112158b&gt;&#93;</span> generic_file_aio_read+0x6bb/0x700<br/> <span class="error">&#91;&lt;ffffffffa06f14da&gt;&#93;</span> cl_io_start+0x6a/0x140 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81078fa7&gt;&#93;</span> ? current_fs_time+0x27/0x30<br/> <span class="error">&#91;&lt;ffffffffa0bce4d5&gt;&#93;</span> ? ll_inode_size_unlock+0x15/0x20 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06f5664&gt;&#93;</span> cl_io_loop+0xb4/0x1b0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0bb3f41&gt;&#93;</span> ll_file_io_generic+0x481/0xaa0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c0d6e5&gt;&#93;</span> ? ccc_object_size_unlock+0x35/0x40 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0bb4da0&gt;&#93;</span> ll_file_aio_read+0x130/0x2b0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff811a58f1&gt;&#93;</span> ? touch_atime+0x71/0x1a0<br/> <span class="error">&#91;&lt;ffffffffa0bb5079&gt;&#93;</span> ll_file_read+0x159/0x290 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c173be&gt;&#93;</span> vvp_io_read_start+0x22e/0x450 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c03044&gt;&#93;</span> ll_direct_IO_26+0xb64/0x1140 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff811894d5&gt;&#93;</span> vfs_read+0xb5/0x1a0<br/> <span class="error">&#91;&lt;ffffffff81189802&gt;&#93;</span> sys_pread64+0x82/0xa0<br/> <span class="error">&#91;&lt;ffffffff8100b072&gt;&#93;</span> system_call_fastpath+0x16/0x1b</p> <p> <span class="error">&#91;&lt;ffffffff81058bd3&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> <span class="error">&#91;&lt;ffffffff8112158b&gt;&#93;</span> generic_file_aio_read+0x6bb/0x700<br/> <span class="error">&#91;&lt;ffffffff81078fa7&gt;&#93;</span> ? current_fs_time+0x27/0x30<br/> <span class="error">&#91;&lt;ffffffffa0bce4d5&gt;&#93;</span> ? ll_inode_size_unlock+0x15/0x20 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c0d6e5&gt;&#93;</span> ? ccc_object_size_unlock+0x35/0x40 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff811a58f1&gt;&#93;</span> ? touch_atime+0x71/0x1a0<br/> <span class="error">&#91;&lt;ffffffffa0c173be&gt;&#93;</span> vvp_io_read_start+0x22e/0x450 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06f14da&gt;&#93;</span> cl_io_start+0x6a/0x140 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06f5664&gt;&#93;</span> cl_io_loop+0xb4/0x1b0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0bb3f41&gt;&#93;</span> ll_file_io_generic+0x481/0xaa0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0bb4da0&gt;&#93;</span> ll_file_aio_read+0x130/0x2b0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0bb5079&gt;&#93;</span> ll_file_read+0x159/0x290 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff811894d5&gt;&#93;</span> vfs_read+0xb5/0x1a0<br/> <span class="error">&#91;&lt;ffffffff81189802&gt;&#93;</span> sys_pread64+0x82/0xa0<br/> <span class="error">&#91;&lt;ffffffff8100b072&gt;&#93;</span> system_call_fastpath+0x16/0x1b</p> <p> <span class="error">&#91;&lt;ffffffffa06f14da&gt;&#93;</span> cl_io_start+0x6a/0x140 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06f5664&gt;&#93;</span> cl_io_loop+0xb4/0x1b0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0bb3f41&gt;&#93;</span> ll_file_io_generic+0x481/0xaa0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0bb4da0&gt;&#93;</span> ll_file_aio_read+0x130/0x2b0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0bb5079&gt;&#93;</span> ll_file_read+0x159/0x290 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff811894d5&gt;&#93;</span> vfs_read+0xb5/0x1a0<br/> <span class="error">&#91;&lt;ffffffff81189802&gt;&#93;</span> sys_pread64+0x82/0xa0<br/> <span class="error">&#91;&lt;ffffffff8100b072&gt;&#93;</span> system_call_fastpath+0x16/0x1b</p> <p>Kernel panic - not syncing: LBUG<br/> Pid: 58184, comm: sio Not tainted 2.6.32-431.20.3.hz100.el6.x86_64 #1<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff81528067&gt;&#93;</span> ? panic+0xa7/0x16f<br/> <span class="error">&#91;&lt;ffffffffa057deeb&gt;&#93;</span> ? lbug_with_loc+0x9b/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c03044&gt;&#93;</span> ? ll_direct_IO_26+0xb64/0x1140 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81058bd3&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> <span class="error">&#91;&lt;ffffffff8112158b&gt;&#93;</span> ? generic_file_aio_read+0x6bb/0x700<br/> <span class="error">&#91;&lt;ffffffff811b4b5f&gt;&#93;</span> ? list_move+0x1f/0x30<br/> <span class="error">&#91;&lt;ffffffff811b51ff&gt;&#93;</span> ? __mark_inode_dirty+0x13f/0x160<br/> <span class="error">&#91;&lt;ffffffff811a59cd&gt;&#93;</span> ? touch_atime+0x14d/0x1a0<br/> <span class="error">&#91;&lt;ffffffffa0c173be&gt;&#93;</span> ? vvp_io_read_start+0x22e/0x450 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06f14da&gt;&#93;</span> ? cl_io_start+0x6a/0x140 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06f5664&gt;&#93;</span> ? cl_io_loop+0xb4/0x1b0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0bb3f41&gt;&#93;</span> ? ll_file_io_generic+0x481/0xaa0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0bb4da0&gt;&#93;</span> ? ll_file_aio_read+0x130/0x2b0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0bb5079&gt;&#93;</span> ? ll_file_read+0x159/0x290 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff811894d5&gt;&#93;</span> ? vfs_read+0xb5/0x1a0<br/> <span class="error">&#91;&lt;ffffffff81189802&gt;&#93;</span> ? sys_pread64+0x82/0xa0<br/> <span class="error">&#91;&lt;ffffffff8100b072&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b</p> LU-5700

async IO LBUG obj->cob_transient_pages == 0

Bug Critical Resolved Fixed Lai Siyao Stephen Champion patch Thu, 2 Oct 2014 09:59:50 +0000 Mon, 20 Oct 2014 22:10:12 +0000 Mon, 20 Oct 2014 22:10:12 +0000 Lustre 2.7.0 Lustre 2.7.0 0 7 <p>This is easy to reproduce using sio on an 8 socket UV system with 160 threads and 4 tb of memory. I have been unable to reproduce this on a smaller configuration thus far.</p> <p>root@cy024-4-sys:/mnt/cy024/schamp # ./sio -tDcw -b 2048 -A 16 -s 83865632 -g /mnt/cy024/schamp/sio.1<br/> Warning: filesize adjusted to 83865600 blocks<br/> &lt;crash&gt;</p> <p>3 OSS with five OST each, stripe size -1 :</p> <ol> <li>lfs getstripe /mnt/cy024-mnt/schamp/sio.1<br/> /mnt/cy024-mnt/schamp/sio.1<br/> lmm_stripe_count: 15<br/> lmm_stripe_size: 1048576<br/> lmm_pattern: 1<br/> lmm_layout_gen: 0<br/> lmm_stripe_offset: 7<br/> obdidx objid objid group<br/> 7 6276 0x1884 0<br/> 11 6276 0x1884 0<br/> 15 6276 0x1884 0<br/> 2 6276 0x1884 0<br/> 13 6276 0x1884 0<br/> 9 6276 0x1884 0<br/> 6 6276 0x1884 0<br/> 10 6276 0x1884 0<br/> 12 6276 0x1884 0<br/> 4 6276 0x1884 0<br/> 1 6276 0x1884 0<br/> 0 6276 0x1884 0<br/> 14 6276 0x1884 0<br/> 5 6276 0x1884 0<br/> 8 6276 0x1884 0</li> </ol> <p>We have cores if they are useful.</p> <p>Olaf Weber's notes:</p> <p>There are four threads of interest, with stack traces like this showing they're doing direct IO.</p> <p>PID: 58183 TASK: ffff893de6382080 CPU: 151 COMMAND: "sio"<br/> #0 <span class="error">&#91;ffff893dd1fed9b0&#93;</span> panic at ffffffff81528001<br/> #1 <span class="error">&#91;ffff893dd1feda30&#93;</span> lbug_with_loc at ffffffffa057deeb <span class="error">&#91;libcfs&#93;</span><br/> #2 <span class="error">&#91;ffff893dd1feda50&#93;</span> ll_direct_IO_26 at ffffffffa0c03044 <span class="error">&#91;lustre&#93;</span><br/> #3 <span class="error">&#91;ffff893dd1fedb80&#93;</span> generic_file_aio_read at ffffffff8112158b<br/> #4 <span class="error">&#91;ffff893dd1fedc60&#93;</span> vvp_io_read_start at ffffffffa0c173be <span class="error">&#91;lustre&#93;</span><br/> #5 <span class="error">&#91;ffff893dd1fedcd0&#93;</span> cl_io_start at ffffffffa06f14da <span class="error">&#91;obdclass&#93;</span><br/> #6 <span class="error">&#91;ffff893dd1fedd00&#93;</span> cl_io_loop at ffffffffa06f5664 <span class="error">&#91;obdclass&#93;</span><br/> #7 <span class="error">&#91;ffff893dd1fedd30&#93;</span> ll_file_io_generic at ffffffffa0bb3f41 <span class="error">&#91;lustre&#93;</span><br/> #8 <span class="error">&#91;ffff893dd1fede20&#93;</span> ll_file_aio_read at ffffffffa0bb4da0 <span class="error">&#91;lustre&#93;</span><br/> #9 <span class="error">&#91;ffff893dd1fede80&#93;</span> ll_file_read at ffffffffa0bb5079 <span class="error">&#91;lustre&#93;</span><br/> #10 <span class="error">&#91;ffff893dd1fedef0&#93;</span> vfs_read at ffffffff811894d5<br/> #11 <span class="error">&#91;ffff893dd1fedf30&#93;</span> sys_pread64 at ffffffff81189802<br/> #12 <span class="error">&#91;ffff893dd1fedf80&#93;</span> system_call_fastpath at ffffffff8100b072<br/> RIP: 00007ffff7495e13 RSP: 00007ffff7ff8dd8 RFLAGS: 00010206<br/> RAX: 0000000000000011 RBX: ffffffff8100b072 RCX: 0000000000000000<br/> RDX: 0000000000800000 RSI: 00007ffff699c000 RDI: 0000000000000003<br/> RBP: 0000000000606240 R8: 00007ffff7ff9700 R9: 00007ffff7ff9700<br/> R10: 0000000000000000 R11: 0000000000000293 R12: 00007ffff7ff9700<br/> R13: 0000000000000003 R14: 000000000061f3e8 R15: 0000000000000003<br/> ORIG_RAX: 0000000000000011 CS: 0033 SS: 002b</p> <p>Approximately they're here:</p> <p>lustre/llite/rw26.c<br/> 373 static ssize_t ll_direct_IO_26(int rw, struct kiocb *iocb,<br/> 374 const struct iovec *iov, loff_t file_offset,<br/> 375 unsigned long nr_segs)<br/> 376 {<br/> 377 struct lu_env *env;<br/> 378 struct cl_io *io;<br/> 379 struct file *file = iocb-&gt;ki_filp;<br/> 380 struct inode *inode = file-&gt;f_mapping-&gt;host;<br/> 381 struct ccc_object *obj = cl_inode2ccc(inode);<br/> 382 long count = iov_length(iov, nr_segs);<br/> ...<br/> 433 page_count = ll_get_user_pages(rw, user_addr, bytes,<br/> 434 &amp;pages, &amp;max_pages);<br/> 435 if (likely(page_count &gt; 0)) {<br/> 436 if (unlikely(page_count &lt; max_pages))<br/> 437 bytes = page_count &lt;&lt; PAGE_CACHE_SHIFT;<br/> 438 result = ll_direct_IO_26_seg(env, io, rw, inode,<br/> 439 file-&gt;f_mapping,<br/> 440 bytes, file_offset,<br/> 441 pages, page_count);<br/> 442 ll_free_user_pages(pages, max_pages, rw==READ);<br/> ...<br/> 474 out:</p> <ul> <li>475 LASSERT(obj-&gt;cob_transient_pages == 0);</li> </ul> <p>They are all referencing the same obj</p> <p>crash&gt; ccc_object.cob_transient_pages ffff88bdd31dd078<br/> cob_transient_pages = 36</p> <p>Looking at how cob_transient_pages is manipulated:</p> <p>lustre/include/lclient.h:<br/> 194 /**<br/> 195 * ccc-private object state.<br/> 196 */<br/> 197 struct ccc_object {<br/> ...<br/> 210 /**<br/> 211 * Access this counter is protected by inode-&gt;i_sem. Now that<br/> 212 * the lifetime of transient pages must be covered by inode sem,<br/> 213 * we don't need to hold any lock..<br/> 214 */<br/> 215 int cob_transient_pages;</p> <p>Protected by i_sem? A very bad sign, as i_sem has been replaced by i_mutex for<br/> quite some time now, and the i_mutex is not held across a read operation. Lustre<br/> would have to do this itself.</p> <p>Which points to commit ed5ebb87bfc2b684958daac90c4369f395482a16, part of which is this:</p> <p>diff --git a/lustre/llite/rw26.c b/lustre/llite/rw26.c<br/> index b605fa4..b9c8293 100644<br/> &#8212; a/lustre/llite/rw26.c<br/> +++ b/lustre/llite/rw26.c<br/> @@ -412,13 +412,6 @@ static ssize_t ll_direct_IO_26(int rw, struct kiocb *iocb,<br/> io = ccc_env_io(env)-&gt;cui_cl.cis_io;<br/> LASSERT(io != NULL);</p> <ul class="alternate" type="square"> <li>/* 0. Need locking between buffered and direct access. and race with</li> <li>* size changing by concurrent truncates and writes.</li> <li>* 1. Need inode mutex to operate transient pages.</li> <li>*/</li> <li>if (rw == READ)</li> <li>mutex_lock(&amp;inode-&gt;i_mutex);<br/> -<br/> LASSERT(obj-&gt;cob_transient_pages == 0);<br/> for (seg = 0; seg &lt; nr_segs; seg++) { long iov_left = iov[seg].iov_len; @@ -480,8 +473,6 @@ static ssize_t ll_direct_IO_26(int rw, struct kiocb *iocb, } <p> out:<br/> LASSERT(obj-&gt;cob_transient_pages == 0);</p></li> <li>if (rw == READ)</li> <li>mutex_unlock(&amp;inode-&gt;i_mutex);</li> </ul> <p> if (tot_bytes &gt; 0) {<br/> struct ccc_io *cio = ccc_env_io(env);</p> <p>A bit more info from the commit header: the change was <a href="https://jira.whamcloud.com/browse/LU-1669" title="lli-&gt;lli_write_mutex (single shared file performance)" class="issue-link" data-issue-key="LU-1669"><del>LU-1669</del></a></p> <p>commit ed5ebb87bfc2b684958daac90c4369f395482a16<br/> Author: Prakash Surya &lt;surya1@llnl.gov&gt;<br/> Date: Wed Oct 2 17:16:51 2013 -0700</p> <p> <a href="https://jira.whamcloud.com/browse/LU-1669" title="lli-&gt;lli_write_mutex (single shared file performance)" class="issue-link" data-issue-key="LU-1669"><del>LU-1669</del></a> vvp: Use lockless __generic_file_aio_write<br/> ...<br/> Change-Id: I0023132b5d941b3304f39f015f95106542998072<br/> Reviewed-on: <a href="http://review.whamcloud.com/6672" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6672</a></p> <p>Lai</p> <p>Could you please advise on this issue?</p> <p>Thanks</p> <p>Peter</p> <p>Our test system had another crash with normal (not async) DIO last night. This led to </p> <p><a href="http://review.whamcloud.com/#/c/12179/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12179/</a></p> <p>Which lets us get on with the performance evaluation we were doing.</p> <p>The patch removes the assertions and makes the counter atomic. We are not convinced that this section is safe for multiple threads - it needs to be considered carefully. The patch should not introduce any new bugs, but may expose something by not crashing.</p> <p>Stephen,<br/> Your patch 12179 has been merged into master (October 10).</p> <p>Do you need any more work on this ticket, or can I mark it as resolved, fixed?</p> <p>Thanks,<br/> ~ jfc.</p> <p>Works for now.</p> <p>We're not entirely convinced that this section of code is safe for reentry, but this clearly fixed one problem, and we can open a new ticket if we identify another.</p> <p>ok thanks Steve</p> Related LU-1669 Development Rank 1|hzwxlz: Rank (Obsolete) 15957 Severity