https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-302 Lustre <p>While running runtests test, the ll_ost_io_* threads hung as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: copying files from /etc /bin to /mnt/lustre/runtest.5368/etc /bin at Tue May 10 02:14:07 PDT 2011 Lustre: Service thread pid 6575 was inactive for 40.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 6575, comm: ll_ost_io_03 Call Trace: [&lt;ffffffff8006466c&gt;] __down_read+0x7a/0x92 [&lt;ffffffff88bb2b0f&gt;] ldiskfs_ext_walk_space+0xdf/0x2d0 [ldiskfs] [&lt;ffffffff88c0bf10&gt;] ldiskfs_ext_new_extent_cb+0x0/0x650 [fsfilt_ldiskfs] [&lt;ffffffff8006456b&gt;] __down_write_nested+0x12/0x92 [&lt;ffffffff88c0846d&gt;] fsfilt_map_nblocks+0xfd/0x150 [fsfilt_ldiskfs] [&lt;ffffffff88c69a7d&gt;] filter_direct_io+0x46d/0xd50 [obdfilter] [&lt;ffffffff88c08be7&gt;] fsfilt_ldiskfs_setattr+0x1a7/0x250 [fsfilt_ldiskfs] [&lt;ffffffff88c6c840&gt;] filter_commitrw_write+0x1800/0x2be0 [obdfilter] [&lt;ffffffff8005c33c&gt;] cache_alloc_refill+0x106/0x186 [&lt;ffffffff88c24eed&gt;] ost_checksum_bulk+0x37d/0x5a0 [ost] [&lt;ffffffff88c2bd09&gt;] ost_brw_write+0x1c99/0x2480 [ost] [&lt;ffffffff8001aa2d&gt;] vsnprintf+0x5df/0x627 [&lt;ffffffff88945f25&gt;] lustre_msg_get_opc+0x35/0xf0 [ptlrpc] [&lt;ffffffff889460d8&gt;] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] [&lt;ffffffff88c2f09e&gt;] ost_handle+0x2bae/0x55b0 [ost] [&lt;ffffffff8890019a&gt;] lock_res_and_lock+0xba/0xd0 [ptlrpc] [&lt;ffffffff887b4a87&gt;] libcfs_next_nidstring+0x37/0x50 [libcfs] [&lt;ffffffff889556f9&gt;] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc] [&lt;ffffffff88955e55&gt;] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] [&lt;ffffffff8008c86f&gt;] __wake_up_common+0x3e/0x68 [&lt;ffffffff88956de6&gt;] ptlrpc_main+0xf66/0x1120 [ptlrpc] [&lt;ffffffff8005dfb1&gt;] child_rip+0xa/0x11 [&lt;ffffffff88955e80&gt;] ptlrpc_main+0x0/0x1120 [ptlrpc] [&lt;ffffffff8005dfa7&gt;] child_rip+0x0/0x11 LustreError: dumping log to /tmp/lustre-log.1305018888.6575 </pre> </div></div> <p>Please refer to the following Maloo report for more logs:<br/> <a href="https://maloo.whamcloud.com/test_sets/ecc7177c-7aec-11e0-b5bf-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ecc7177c-7aec-11e0-b5bf-52540025f9af</a></p> <p>The issue is blocking the testing on b1_8 branch.</p> Lustre Branch: b1_8<br/> Lustre Build: <a href="http://newbuild.whamcloud.com/job/lustre-reviews/363/arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/">http://newbuild.whamcloud.com/job/lustre-reviews/363/arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/</a><br/> Kernel Version: 2.6.18-238.9.1.el5_lustre.20110509050254 LU-302

ll_ost_io_* threads hung

Bug Blocker Closed Duplicate Yang Sheng Jian Yu Tue, 10 May 2011 04:11:50 +0000 Wed, 11 May 2011 08:33:02 +0000 Wed, 11 May 2011 08:33:02 +0000 Lustre 1.8.6 Lustre 1.8.6 0 4 <p>Johann</p> <p>Yu Jian suspects that this issue may be due to a recent landing. Could you please have a quick look and then assign as necessary? This is blocking all 1.8.x testing atm</p> <p>Regards</p> <p>Peter</p> <p>This is probably the ext_walk_space locking change that hit on some of the newer kernels also, probably because the RHEL 5 kernel backported some change.</p> <p>I think we need a better configure check to determine whether ext_walk_space needs to be locked by the caller or internally. I proposed a way to do this using "grep -A" in the original RHEL6 bug in bugzilla that Kalpak was working on. </p> <p>YangSheng</p> <p>It seems that this is something that you have been working on</p> <p>Regards</p> <p>Peter</p> <p>Looks REHL5.6 backport upstream commit fab3a549e204172236779f502eccb4f9bf0dc87d(ext4: Fix potential fiemap deadlock (mmap_sem vs. i_data_sem)). So we may need landed fix in <a href="https://bugzilla.lustre.org/show_bug.cgi?id=23780" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=23780</a>. Original patch against SLES11 SP1. But as further solution. How about change down_read() to down_write() in ext4_ext_walk_space:<br/> /* find extent for this block */<br/> down_read(&amp;EXT4_I(inode)-&gt;i_data_sem);<br/> path = ext4_ext_find_extent(inode, block, path);<br/> up_read(&amp;EXT4_I(inode)-&gt;i_data_sem);<br/> and trying to push this change to upstream. This change with WALK_SPACE_HAS_I_DATA_SEM config check. We can deal with this situation. </p> <p>This should be addressed by <a href="http://review.whamcloud.com/#change,491" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,491</a></p> <p>So just port lu-216 to b1_8 is enough to resolved this issue.</p> Development Rank 1|hzw09j: Rank (Obsolete) 10122 Severity