https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9344 Lustre <p>This issue was created by maloo for bobijam &lt;bobijam.xu@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/67af86be-2027-11e7-9073-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/67af86be-2027-11e7-9073-5254006e85c2</a>.</p> <p>The sub-test test_244 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test failed to respond and timed out </pre> </div></div> <p>Info required for matching: sanity 244</p> <p>sendfile_grouplock.c calls sendfile_copy(sourfile, 0, destfile, 98765)<br/> and sendfile_copy()&#45;&gt;llapi_group_lock(fd_out, dest_gid);</p> <p>which will call into lov_io_init() and atomic_inc(&amp;lov-&gt;lo_active_ios)</p> <p>and sendfile_copy() tries to write to the file, which will check to get layout, and ll_layout_refresh() finds there is an active ios (marked by ll_get_grouplock()), so the write hung there</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sendfile_grou S 0000000000000000 0 7394 7321 0x00000080 ffff88000eb3f618 0000000000000082 ffff88000eb3f5e0 ffff88000eb3f5dc 00001ce200000000 ffff88003f828400 0000005dce083b5f ffff880003436ac0 00000000000005ff 0000000100017a1d ffff88002b57fad0 ffff88000eb3ffd8 Call Trace: [&lt;ffffffffa0afa20b&gt;] lov_layout_wait+0x11b/0x220 [lov] [&lt;ffffffff810640e0&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0afc11e&gt;] lov_conf_set+0x37e/0xa30 [lov] [&lt;ffffffffa040f471&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa059d888&gt;] cl_conf_set+0x58/0x100 [obdclass] [&lt;ffffffffa0fa5dd4&gt;] ll_layout_conf+0x84/0x3f0 [lustre] [&lt;ffffffffa040f471&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa0fb0b9d&gt;] ll_layout_refresh+0x96d/0x1710 [lustre] [&lt;ffffffffa040f471&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa0ff7d6f&gt;] vvp_io_init+0x32f/0x450 [lustre] [&lt;ffffffffa040f471&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa05a5148&gt;] cl_io_init0+0x88/0x150 [obdclass] [&lt;ffffffffa05a7caa&gt;] cl_io_init+0x4a/0xa0 [obdclass] [&lt;ffffffffa05a7dbc&gt;] cl_io_rw_init+0xbc/0x200 [obdclass] [&lt;ffffffffa0fa7213&gt;] ll_file_io_generic+0x203/0xaf0 [lustre] [&lt;ffffffffa0fa941d&gt;] ll_file_aio_write+0x13d/0x280 [lustre] [&lt;ffffffffa0fa969a&gt;] ll_file_write+0x13a/0x270 [lustre] [&lt;ffffffff81189ef8&gt;] vfs_write+0xb8/0x1a0 [&lt;ffffffff811ba76d&gt;] kernel_write+0x3d/0x50 [&lt;ffffffff811ba7da&gt;] write_pipe_buf+0x5a/0x90 [&lt;ffffffff811b9342&gt;] splice_from_pipe_feed+0x72/0x120 [&lt;ffffffff811ba780&gt;] ? write_pipe_buf+0x0/0x90 [&lt;ffffffff811ba780&gt;] ? write_pipe_buf+0x0/0x90 [&lt;ffffffff811b9d9e&gt;] __splice_from_pipe+0x6e/0x80 [&lt;ffffffff811ba780&gt;] ? write_pipe_buf+0x0/0x90 [&lt;ffffffff811b9e01&gt;] splice_from_pipe+0x51/0x70 [&lt;ffffffff811b9e3d&gt;] default_file_splice_write+0x1d/0x30 [&lt;ffffffff811b9fca&gt;] do_splice_from+0xba/0xf0 [&lt;ffffffff811ba020&gt;] direct_splice_actor+0x20/0x30 [&lt;ffffffff811ba256&gt;] splice_direct_to_actor+0xc6/0x1c0 [&lt;ffffffff811ba000&gt;] ? direct_splice_actor+0x0/0x30 [&lt;ffffffff811ba39d&gt;] do_splice_direct+0x4d/0x60 [&lt;ffffffff8118a344&gt;] do_sendfile+0x184/0x1e0 [&lt;ffffffff8118a3d4&gt;] sys_sendfile64+0x34/0xb0 [&lt;ffffffff810e031e&gt;] ? __audit_syscall_exit+0x25e/0x290 [&lt;ffffffff8100b0d2&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> LU-9344

sanity test_244: sendfile_grouplock test12() test hung

Technical task LU-9349 Critical Resolved Fixed Zhenyu Xu Maloo pfl Fri, 14 Apr 2017 08:58:02 +0000 Mon, 10 Jul 2017 20:08:35 +0000 Fri, 28 Apr 2017 22:21:12 +0000 Lustre 2.10.0 Lustre 2.10.0 0 5 <p>Jinshan,</p> <p>I think the sendfile_grouplock.c does not use group lock correctly. It holds a group lock while trying write data to it.</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/26646" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26646</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9344" title="sanity test_244: sendfile_grouplock test12() test hung" class="issue-link" data-issue-key="LU-9344"><del>LU-9344</del></a> test: hung with test12()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 92460697205d25e2de08f4c6d05e3dc8d8bc3387</p> <p>Bobijam and I discussed this problem a little bit. Group lock needs to acquire locks from all objects in the current layout, so that it has to increase active_ios in the LOV layer, therefore this layout won't disappear during the existence of group lock.</p> <p>When a write extends PFL layout with group lock held, it will result in deadlock because configuring new layout lock needs to wait for the active IOs to reach to zero.</p> <p>The current workaround solution is to instantiate all components before group lock is taken.</p> <p>This will hurt all file migration operations, since it will instantiate all layout components on both the source and target files. That wouldn't be so bad if it only instantiated the components on the source, but that doesn't really make sense to instantiate the components when getting the group lock on any file that is opened read-only.</p> <p>That said, I'm wondering if there is even a race when getting the group lock on the new objects? Since the client(s) writing to the file are already holding the group lock on the objects on the first part of the file, any other clients would be blocked from accessing the file if they are enqueuing the group locks in component order. The existing group lock holders could still group lock the newly allocated objects without dropping the locks on the existing objects (which <em>would</em> cause a deadlock).</p> <p>For migration, there is another option to use Lustre file lease. But really good point on acquiring group lock when a file is opened for read only.</p> <p>It seems like it's hard to maintain current semantics of group lock. Can we revise the semantics of the group lock? For example, group lock will fail if the file's layout change is changed.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/26646/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26646/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9344" title="sanity test_244: sendfile_grouplock test12() test hung" class="issue-link" data-issue-key="LU-9344"><del>LU-9344</del></a> test: hung with sendfile_grouplock test12()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c6b5df7644c245853b5dcf82b1c93614c5357f3f</p> <p>Landed for 2.10</p> Related LU-8998 LU-9429 LU-9479 LU-9756 Development Rank 1|hzza9b: Rank (Obsolete) 9223372036854775807