https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9479 Lustre <p>This issue was created by maloo for bobijam &lt;bobijam.xu@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/67af86be-2027-11e7-9073-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/67af86be-2027-11e7-9073-5254006e85c2</a>.</p> <p>The sub-test test_244 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test failed to respond and timed out </pre> </div></div> <p>Info required for matching: sanity 244</p> <p>sendfile_grouplock.c calls sendfile_copy(sourfile, 0, destfile, 98765)<br/> and sendfile_copy()&#45;&gt;llapi_group_lock(fd_out, dest_gid);</p> <p>which will call into lov_io_init() and atomic_inc(&amp;lov-&gt;lo_active_ios)</p> <p>and sendfile_copy() tries to write to the file, which will check to get layout, and ll_layout_refresh() finds there is an active ios (marked by ll_get_grouplock()), so the write hung there</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sendfile_grou S 0000000000000000 0 7394 7321 0x00000080 ffff88000eb3f618 0000000000000082 ffff88000eb3f5e0 ffff88000eb3f5dc 00001ce200000000 ffff88003f828400 0000005dce083b5f ffff880003436ac0 00000000000005ff 0000000100017a1d ffff88002b57fad0 ffff88000eb3ffd8 Call Trace: [&lt;ffffffffa0afa20b&gt;] lov_layout_wait+0x11b/0x220 [lov] [&lt;ffffffff810640e0&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0afc11e&gt;] lov_conf_set+0x37e/0xa30 [lov] [&lt;ffffffffa040f471&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa059d888&gt;] cl_conf_set+0x58/0x100 [obdclass] [&lt;ffffffffa0fa5dd4&gt;] ll_layout_conf+0x84/0x3f0 [lustre] [&lt;ffffffffa040f471&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa0fb0b9d&gt;] ll_layout_refresh+0x96d/0x1710 [lustre] [&lt;ffffffffa040f471&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa0ff7d6f&gt;] vvp_io_init+0x32f/0x450 [lustre] [&lt;ffffffffa040f471&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa05a5148&gt;] cl_io_init0+0x88/0x150 [obdclass] [&lt;ffffffffa05a7caa&gt;] cl_io_init+0x4a/0xa0 [obdclass] [&lt;ffffffffa05a7dbc&gt;] cl_io_rw_init+0xbc/0x200 [obdclass] [&lt;ffffffffa0fa7213&gt;] ll_file_io_generic+0x203/0xaf0 [lustre] [&lt;ffffffffa0fa941d&gt;] ll_file_aio_write+0x13d/0x280 [lustre] [&lt;ffffffffa0fa969a&gt;] ll_file_write+0x13a/0x270 [lustre] [&lt;ffffffff81189ef8&gt;] vfs_write+0xb8/0x1a0 [&lt;ffffffff811ba76d&gt;] kernel_write+0x3d/0x50 [&lt;ffffffff811ba7da&gt;] write_pipe_buf+0x5a/0x90 [&lt;ffffffff811b9342&gt;] splice_from_pipe_feed+0x72/0x120 [&lt;ffffffff811ba780&gt;] ? write_pipe_buf+0x0/0x90 [&lt;ffffffff811ba780&gt;] ? write_pipe_buf+0x0/0x90 [&lt;ffffffff811b9d9e&gt;] __splice_from_pipe+0x6e/0x80 [&lt;ffffffff811ba780&gt;] ? write_pipe_buf+0x0/0x90 [&lt;ffffffff811b9e01&gt;] splice_from_pipe+0x51/0x70 [&lt;ffffffff811b9e3d&gt;] default_file_splice_write+0x1d/0x30 [&lt;ffffffff811b9fca&gt;] do_splice_from+0xba/0xf0 [&lt;ffffffff811ba020&gt;] direct_splice_actor+0x20/0x30 [&lt;ffffffff811ba256&gt;] splice_direct_to_actor+0xc6/0x1c0 [&lt;ffffffff811ba000&gt;] ? direct_splice_actor+0x0/0x30 [&lt;ffffffff811ba39d&gt;] do_splice_direct+0x4d/0x60 [&lt;ffffffff8118a344&gt;] do_sendfile+0x184/0x1e0 [&lt;ffffffff8118a3d4&gt;] sys_sendfile64+0x34/0xb0 [&lt;ffffffff810e031e&gt;] ? __audit_syscall_exit+0x25e/0x290 [&lt;ffffffff8100b0d2&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> LU-9479

sanity test 184d 244: don't instantiate PFL component when taking group lock

Bug Major Open Unresolved Zhenyu Xu Andreas Dilger pfl Tue, 9 May 2017 20:40:37 +0000 Mon, 8 Jan 2024 20:13:13 +0000 Lustre 2.10.0 0 7 <p>This problem was worked around in patch <a href="https://review.whamcloud.com/26646" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26646</a> "<a href="https://jira.whamcloud.com/browse/LU-9344" title="sanity test_244: sendfile_grouplock test12() test hung" class="issue-link" data-issue-key="LU-9344"><del>LU-9344</del></a> test: hung with sendfile_grouplock test12()" by instantiating all of the file's components before taking the group lock. However, for operations like <tt>lfs swap_layouts</tt> in sanity.sh test_184d or <tt>lfs migrate</tt> since this will instantiate all of the components even if the file is empty. </p> <p>This is inefficient for the source file since it is about to be deleted, and for the destination file since it doesn't need all of those objects.</p> <p>Per comments in <a href="https://jira.whamcloud.com/browse/LU-9344" title="sanity test_244: sendfile_grouplock test12() test hung" class="issue-link" data-issue-key="LU-9344"><del>LU-9344</del></a> and <a href="https://review.whamcloud.com/26646" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26646</a> it makes sense to not instantiate the components of the source file, since it is not being modified. It is also known what the file size is for writes during migration/swap_layouts, so it is possible to instantiate the required components in advance and leave any later ones alone.</p> <p>Jinshan,</p> <p>from commit d25892863a30a385cdae2b1991ab0a9ca76fcd7d, the lov-&gt;lo_active_ios was created, and get grouplock also increases it.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> LU-1876 hsm: account for active IOs correctly We used to use refcount of lsm to account for active IOs but this turns out wrong because empty layout files don't have a lsm. In this patch, lov_object::lo_active_ios is invented to account for active IOs for both raid0 and empty layout; </pre> </div></div> <p>I'm think of not accounting get_grouplock as a active IOs so that process with the grouplock won't hung itself in the first place, do you think it's plausible?</p> <p>well, even lo_active_ios is resolved with my last comment, there still exists problem that the later instantiated component does not protected by the prior given group lock.</p> <p>Bobijam, can you discuss with Oleg some way we could get group locks for newly-instantiated components that is not racy. It seems that the workaround is not working properly, since there are now multiple problems with grouplocks being seen.</p> <p>Is it not possible for a group lock to prevent instantiation of further components by those not holding a group lock? It would imply a group lock also had to take a layout lock, though.</p> <p>Patrick, the problem here is that we don't want to prevent instantistion new component instantistion, but if new objects are instantisted they do not have a group lock on them (since that is handled between each OST and client separately for each object). </p> <p>Right, I understand - Note "instantiation of further components <b>by those not holding a group lock</b>". I think that's necessary because otherwise that problem you note occurs - The new segment doesn't get the old group lock applied to it. I don't see a way to solve that, so instead holding a group lock should prevent instantiation of a new component by those <b>not holding the group lock</b>, because otherwise they could get around the group lock and write to that new component.</p> <p>Those holding the group lock can do it, which gives them a chance to apply the group lock to the new component.</p> <p>This is still potentially problematic though, if the client who instantiated the new layout releases its group lock, other clients holding a group lock will not have that portion locked and so it could become unlocked.</p> <p>This is not fun.</p> <p>The issue is that group locks are handled by the OSS, but object instantiation is done by the MDS, so there is no way for the MDS to even know that there is a group lock on the file...</p> <p>Yup... No straightforward suggestions from me.</p> <p>No intention to work on this right now, but I wanted to write this down here so it doesn't get forgotten:</p> <p>As Andreas said, the issue is that group locks are handled by the OSS, but object instantiation is handled by the MDS.  As Oleg noted at LUG, given that, it seems like we need to move group locks to being layout locks.</p> <p>A few thoughts on this.</p> <ol> <li>There is no provision (now) for group lock IDs in IBITS locks.  There is enough room in the ldlm_wire_policy_data union to fit a gid in the IBITS case.  And I'm not sure how much of the rest of the mdc/mdt/IBITS LDLM code will need to be modified to support them correctly.</li> <li>We still need data locks to cover the data i/o.  So a group lock could start with locking the layout exclusively, then getting locks on all of the existing data components (ie just lock from 0 to EOF and not worry about uninstantiated components).  Unlocking one would go in reverse order - unlock data, unlock layout.</li> </ol> <p>There may also be some ... interesting ... interoperability issues with clients that do not request/expect group locks on the layout.  That might be challenging to get right.</p> <p>It is worthwhile to note that the "MDS holds IBITS group lock" issue is (partly) solved already via patch <a href="https://review.whamcloud.com/39406" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39406</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13645" title="Various data corruptions possible in lustre." class="issue-link" data-issue-key="LU-13645"><del>LU-13645</del></a> ldlm: group locks for DOM IBIT lock</tt>", which was commit v2_13_56-61-g0674044036 and as such is included in 2.14.0.</p> <p>That would potentially allow group locks to <b>always</b> lock on the MDT first, and then lock only initialized components. Older clients that don't understand the MDT group lock would (unfortunately) keep their current behavior of always instantiating and locking all components.</p> <p>The new MDS could potentially revoke the layout lock (and keep it) from older clients that do not have this functionality, to prevent non-group locks from instantiating uninitialized components, but there is unfortunately no <tt>OBD_CONNECT_&#42;</tt> flag for this. It might be enough in 2.15 for a group lock to hold the <tt>MDS_INODEBITS_LAYOUT</tt> lock exclusively, <b>if there are uninitialized components</b> to block older clients, but this wouldn't be needed if all components are already initialized, or all clients are new and understand MDS group locking (a new <tt>OBD_CONNECT_&#42;</tt> flag could be added for this).</p> Related LU-13645 LU-9341 LU-8998 LU-9344 LU-9349 LU-9429 LU-17403 LU-9756 LU-9793 LU-10070 LU-12738 Development Rank 1|hzzcaf: Rank (Obsolete) 9223372036854775807 Severity