https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4727 Lustre <p>This is easy to reproduce. I hit this problem every time when I trying to run following commands.</p> <p>rm /mnt/lustre/XXXX -f;<br/> echo XXX &gt; /mnt/lustre/XXXX;<br/> cat /mnt/lustre/XXXX;<br/> lfs hsm_archive --archive=5 /mnt/lustre/XXXX;<br/> cat /mnt/lustre/XXXX;<br/> lfs hsm_release /mnt/lustre/XXXX;<br/> cat /mnt/lustre/XXXX; # This will restore automatically<br/> lfs hsm_release /mnt/lustre/XXXX;<br/> lfs hsm_restore /mnt/lustre/XXXX; # Lhsmtool_posix actually hang here<br/> cat /mnt/lustre/XXXX; # this will stuck</p> <p>And after some time, following messages shew up.</p> <p>INFO: task flush-lustre-1:4106 blocked for more than 120 seconds.<br/> "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> flush-lustre- D 0000000000000005 0 4106 2 0x00000080<br/> ffff8808165b7830 0000000000000046 0000000000000000 0000000000000000<br/> 0000000000013180 0000000000000000 ffff880851fc10f8 ffff88082d4e0c00<br/> ffff88082cb7fab8 ffff8808165b7fd8 000000000000fb88 ffff88082cb7fab8<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff814fc9fe&gt;&#93;</span> __mutex_lock_slowpath+0x13e/0x180<br/> <span class="error">&#91;&lt;ffffffff814fc89b&gt;&#93;</span> mutex_lock+0x2b/0x50<br/> <span class="error">&#91;&lt;ffffffffa0c2814c&gt;&#93;</span> ll_layout_refresh+0x26c/0x1080 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff813104bb&gt;&#93;</span> ? mix_pool_bytes_extract+0x16b/0x180<br/> <span class="error">&#91;&lt;ffffffff81135cf9&gt;&#93;</span> ? zone_statistics+0x99/0xc0<br/> <span class="error">&#91;&lt;ffffffffa059e007&gt;&#93;</span> ? cfs_hash_bd_lookup_intent+0x37/0x130 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c51230&gt;&#93;</span> ? ll_md_blocking_ast+0x0/0x7f0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08b7450&gt;&#93;</span> ? ldlm_completion_ast+0x0/0x930 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06dbba1&gt;&#93;</span> ? cl_io_slice_add+0xc1/0x190 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c78410&gt;&#93;</span> vvp_io_init+0x340/0x490 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05a11aa&gt;&#93;</span> ? cfs_hash_find_or_add+0x9a/0x190 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06daff8&gt;&#93;</span> cl_io_init0+0x98/0x160 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06ddc14&gt;&#93;</span> cl_io_init+0x64/0xe0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c1894d&gt;&#93;</span> cl_sync_file_range+0x12d/0x500 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c46cac&gt;&#93;</span> ll_writepages+0x9c/0x220 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81128d81&gt;&#93;</span> do_writepages+0x21/0x40<br/> <span class="error">&#91;&lt;ffffffff811a43bd&gt;&#93;</span> writeback_single_inode+0xdd/0x290<br/> <span class="error">&#91;&lt;ffffffff811a47ce&gt;&#93;</span> writeback_sb_inodes+0xce/0x180<br/> <span class="error">&#91;&lt;ffffffff811a492b&gt;&#93;</span> writeback_inodes_wb+0xab/0x1b0<br/> <span class="error">&#91;&lt;ffffffff811a4ccb&gt;&#93;</span> wb_writeback+0x29b/0x3f0<br/> <span class="error">&#91;&lt;ffffffff814fb3a0&gt;&#93;</span> ? thread_return+0x4e/0x76e<br/> <span class="error">&#91;&lt;ffffffff8107eb42&gt;&#93;</span> ? del_timer_sync+0x22/0x30<br/> <span class="error">&#91;&lt;ffffffff811a4fb9&gt;&#93;</span> wb_do_writeback+0x199/0x240<br/> <span class="error">&#91;&lt;ffffffff811a50c3&gt;&#93;</span> bdi_writeback_task+0x63/0x1b0<br/> <span class="error">&#91;&lt;ffffffff81091f97&gt;&#93;</span> ? bit_waitqueue+0x17/0xd0<br/> <span class="error">&#91;&lt;ffffffff811379e0&gt;&#93;</span> ? bdi_start_fn+0x0/0x100<br/> <span class="error">&#91;&lt;ffffffff81137a66&gt;&#93;</span> bdi_start_fn+0x86/0x100<br/> <span class="error">&#91;&lt;ffffffff811379e0&gt;&#93;</span> ? bdi_start_fn+0x0/0x100<br/> <span class="error">&#91;&lt;ffffffff81091d66&gt;&#93;</span> kthread+0x96/0xa0<br/> <span class="error">&#91;&lt;ffffffff8100c14a&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffff81091cd0&gt;&#93;</span> ? kthread+0x0/0xa0<br/> <span class="error">&#91;&lt;ffffffff8100c140&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>It seems copy tool is waiting for md_enqueue(MDS_INODELOCK_LAYOUT). Other processes who are trying to lock lli-&gt;lli_layout_mutex will be stuck. This problem won't recover until lock enque times out and client reconnects.</p> LU-4727

Lhsmtool_posix process stuck in ll_layout_refresh() when restoring

Bug Major Resolved Fixed Jinshan Xiong Li Xi HSM cea Fri, 7 Mar 2014 03:32:03 +0000 Tue, 25 Jan 2022 20:54:48 +0000 Thu, 23 Apr 2015 16:44:10 +0000 Lustre 2.6.0 Lustre 2.5.1 Lustre 2.8.0 0 19 <p>I'm seeing a similar hang here. My copytool stack:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Call Trace: [&lt;ffffffff81058d53&gt;] ? __wake_up+0x53/0x70 [&lt;ffffffffa0d2ec00&gt;] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc] [&lt;ffffffffa0d334a5&gt;] ldlm_completion_ast+0x545/0x920 [ptlrpc] [&lt;ffffffff81065df0&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0d2d9f6&gt;] ldlm_cli_enqueue_fini+0x936/0xe70 [ptlrpc] [&lt;ffffffffa0d2e2d5&gt;] ldlm_cli_enqueue+0x3a5/0x790 [ptlrpc] [&lt;ffffffffa0d32f60&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [&lt;ffffffffa0b30240&gt;] ? ll_md_blocking_ast+0x0/0x7d0 [lustre] [&lt;ffffffffa03bcd3e&gt;] mdc_enqueue+0x2be/0x1a10 [mdc] [&lt;ffffffffa037da84&gt;] lmv_enqueue+0x2f4/0xfc0 [lmv] [&lt;ffffffffa0b0b145&gt;] ll_layout_refresh+0x515/0xfe0 [lustre] [&lt;ffffffffa0b30240&gt;] ? ll_md_blocking_ast+0x0/0x7d0 [lustre] [&lt;ffffffffa0d32f60&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [&lt;ffffffffa0b57113&gt;] vvp_io_fini+0x173/0x260 [lustre] [&lt;ffffffffa0c46cc7&gt;] cl_io_fini+0x77/0x280 [obdclass] [&lt;ffffffffa0afad37&gt;] ll_file_io_generic+0xe7/0x610 [lustre] [&lt;ffffffffa0afbeb2&gt;] ll_file_aio_write+0x142/0x2c0 [lustre] [&lt;ffffffffa0afc19c&gt;] ll_file_write+0x16c/0x2a0 [lustre] [&lt;ffffffff81188f68&gt;] vfs_write+0xb8/0x1a0 [&lt;ffffffff81189861&gt;] sys_write+0x51/0x90 [&lt;ffffffff810e1e4e&gt;] ? __audit_syscall_exit+0x25e/0x290 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>I believe this is broken due to a recent optimization on the MD stack. Will create a workaround patch tomorrow.</p> <p>Hi Li Xi, I can only reproduce this if I run the copytool on the same mount point as used for the accesses. Can you confirm this? I strongly recommend using a dedicated mount point for the copytool. This should be somewhere in the HSM documentation.</p> <p>Hi John,</p> <p>Yeah, you are correct! I wan running the copytool on the same mount point as used for the accesses. Sorry for misusing it.</p> <p>However, I don't understand why we have to use a dedicated mount point for the copytool. Is there any special reason for it?</p> <p>I've seen this when running my copytool in a dedicated mount point. </p> <p>I suspect that running copy tool on dedicated mount point can not eliminate the problem but it can surely reduce the chance of it to occur.</p> <p>The root cause of this problem is as follows:</p> <p>in mdt_getattr_internal() it checks if the file is being restored. If this is the case, it will mark it as MS_RESTORE so that the client won't do glimpse.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* <span class="code-keyword">if</span> file is released, check <span class="code-keyword">if</span> a restore is running */</span> <span class="code-keyword">if</span> ((ma-&gt;ma_valid &amp; MA_HSM) &amp;&amp; (ma-&gt;ma_hsm.mh_flags &amp; HS_RELEASED) &amp;&amp; mdt_hsm_restore_is_running(info, mdt_object_fid(o))) { repbody-&gt;t_state = MS_RESTORE; repbody-&gt;valid |= OBD_MD_TSTATE; } </pre> </div></div> <p>However, when the restore request reaches the MDT, it doesn't flush UPDATE lock so if the client has UPDATE lock cached, it won't see this flag at all.</p> <p>The fix would be to adjust the sequence of adding RESTORE request into action list. Right now it does as follow:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> mdt_lock_reg_init(&amp;crh-&gt;crh_lh, LCK_EX); obj = mdt_object_find_lock(mti, &amp;crh-&gt;crh_fid, &amp;crh-&gt;crh_lh, MDS_INODELOCK_LAYOUT); <span class="code-keyword">if</span> (IS_ERR(obj)) { rc = PTR_ERR(obj); CERROR(<span class="code-quote">"%s: cannot take layout lock <span class="code-keyword">for</span> "</span> DFID<span class="code-quote">": rc = %d\n"</span>, mdt_obd_name(mdt), PFID(&amp;crh-&gt;crh_fid), rc); OBD_SLAB_FREE_PTR(crh, mdt_hsm_cdt_kmem); GOTO(out, rc); } /* we choose to not keep a keep a reference * on the object during the restore time which can be * very <span class="code-object">long</span> */ mdt_object_put(mti-&gt;mti_env, obj); mutex_lock(&amp;cdt-&gt;cdt_restore_lock); list_add_tail(&amp;crh-&gt;crh_list, &amp;cdt-&gt;cdt_restore_hdl); mutex_unlock(&amp;cdt-&gt;cdt_restore_lock); </pre> </div></div> <p>So it requests LAYOUT lock and then add the request into a global list, we should change it to:<br/> 1. add to global list<br/> 2. flush UPDATE lock<br/> 3. request LAYOUT lock</p> <p>In this way, we can make sure that all clients will see the MS_RESTORE flag.</p> <p>Another way to fix the problem is to not getattr in copy tool process. In that case, we can't call ct_stat_by_fid() in llapi_hsm_action_begin(). The stat fetched by ct_stat_by_fid() will be used for setting UID/GID and ATIME/MTIME to volatile file. I think we can do this in other ways.</p> <p>Robert, as a workaround fix, I think you can disable the function call ct_stat_by_fid() in llapi_hsm_action_begin() and do some corresponding cleanup - so that you can continue your work while we're fixing the problem.</p> <p>John, do you remember why we should set UID/GID to the volatile file?</p> <p>We do it to maintain quota. Specifically, layout swap will fail it the ownership of the two files are not equal.</p> <p>Yes it would be nice if the MDT sent what was needed to the copytool. I found that it's hard to fit that extra information into the current protocol. Otherwise a backdoor stat of something would be helpful.</p> <p>Another way is to save attribute in archive and restore it at restore time(copy tool may have already supported this, IIRC). The only problem is UID/GID setting because it may change when the file is in release state; mtime/atime won't be a problem at all.</p> <p>Or we can use a dedicated ioctl() to get the attribute in kernel but it won't try to glimpse the file size.</p> <p>If we assume that the copytool has a dedicated mount point (and we're feeling devious) then it may be enough to do:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static int ct_stat_by_fid(const struct hsm_copytool_private *ct, const struct lu_fid *fid, struct stat *buf) { char fid_name[FID_NOBRACE_LEN + 1]; snprintf(fid_name, sizeof(fid_name), DFID_NOBRACE, PFID(fid)); + unlinkat(ct-&gt;open_by_fid_fd, fid_name, 0); return fstatat(ct-&gt;open_by_fid_fd, fid_name, buf, 0); } </pre> </div></div> <p>But that's not something to be proud of.</p> <p>It turns out my issue was self-inflicted. (In my version of the copytool I had neglected to call flush() before calling llapi_hsm_action_end(), and this left data in the file pointer buffers that wasn't flushed until after the end call when I closed my file handle. So either you need to flush or just close the volatile file handle before calling end. But that is unrelated to this bug.)</p> <p>Robert, are there any restrictions on using this HSM API (e.g. capability needed, only on dedicated agent nodes set up by the sysadmin)? Otherwise, it seems like a potential problem for bad users to be able to lock up the system. Also, what is the extent of the problem? Is it only this one process that is hung (an acceptable loss for a self-inflicted problem) or does it affect the whole client, or even the MDS?</p> <p>I believe the copytool must be run as root, but it can be run on any client. In my case only the copytool process was hung and unkillable. It also prevents the file in question from being restored, at least until the coordinator times out the action request and sends it to another copytool.</p> <p>Hi all,</p> <p>Is there any progress in this issue? This issue is really annoying when I am testing HSM. Is there any easy way to walk around it at least? Using a dedicated mount point for the copytool is not helping....</p> <p>Thanks!</p> <p>Jinshan - Looking at your description of a possible solution...<br/> "So it requests LAYOUT lock and then add the request into a global list, we should change it to:<br/> 1. add to global list<br/> 2. flush UPDATE lock<br/> 3. request LAYOUT lock"</p> <p>When you say "flush UPDATE lock", how are you suggesting this be done? Take an update lock on the object, then take the layout lock? If so, when do we release the update lock? Before taking the layout lock, after getting the layout lock, or some other time?</p> <p>Also, this comment at the top of the function is confusing me:<br/> " * in case of restore, caller must hold layout lock"<br/> Is this comment in error? If not, what layout lock is it referring to/what sort of lock on layout? I ask because if it's a restore request, we take a layout lock, which seems to imply the caller did not have a layout lock already.</p> <p>I'm thinking about a solution for this problem.</p> <blockquote> <p>When you say "flush UPDATE lock", how are you suggesting this be done? Take an update lock on the object, then take the layout lock? If so, when do we release the update lock? Before taking the layout lock, after getting the layout lock, or some other time?</p></blockquote> <p>By "flush UPDATE lock", I meant to acquire the UPDATE lock and release it immediately.</p> <blockquote> <p>Is this comment in error? If not, what layout lock is it referring to/what sort of lock on layout? I ask because if it's a restore request, we take a layout lock, which seems to imply the caller did not have a layout lock already.</p></blockquote> <p>that means layout lock to take in the function, i.e., the code</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> mdt_lock_reg_init(&amp;crh-&gt;crh_lh, LCK_EX); obj = mdt_object_find_lock(mti, &amp;crh-&gt;crh_fid, &amp;crh-&gt;crh_lh, MDS_INODELOCK_LAYOUT); </pre> </div></div> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13138" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13138</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4727" title="Lhsmtool_posix process stuck in ll_layout_refresh() when restoring" class="issue-link" data-issue-key="LU-4727"><del>LU-4727</del></a> hsm: flush UPDATE lock for restore<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bf2e4b958f60cb7eda9303ad0c079fd23ff2d16b</p> <p>Please try patch 13138 and check if it can fix the problem.</p> <p> <a href="http://review.whamcloud.com/13138" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13138</a> solves the problem on single node setup on local vm. Thanks for the patch Jinshan</p> <p>the patch has been in Gerrit for a long time. Please let me know what I can do to move this forward, sigh.</p> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13750" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13750</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4727" title="Lhsmtool_posix process stuck in ll_layout_refresh() when restoring" class="issue-link" data-issue-key="LU-4727"><del>LU-4727</del></a> hsm: use IOC_MDC_GETFILEINFO in restore<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8ec6354ded37f3e1f39d6e0336c9e17b1a97785b</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13750/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13750/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4727" title="Lhsmtool_posix process stuck in ll_layout_refresh() when restoring" class="issue-link" data-issue-key="LU-4727"><del>LU-4727</del></a> hsm: use IOC_MDC_GETFILEINFO in restore<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 96dbac2eaef7a5d1090807bedc9951279c06d037</p> <p>Jinshan,</p> <p>I got this trace when using stat on a file I was restoring (or was restored). I haven't been able to reproduce it so far. Is it the error you mention in your patch? John's patch is applied on that tree, and works well otherwise.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Mar 20 15:00:55 tasclient01 kernel: INFO: task stat:951 blocked for more than 120 seconds. Mar 20 15:00:55 tasclient01 kernel: Tainted: P --------------- 2.6.32-431.17.1.el6.x86_64 #1 Mar 20 15:00:55 tasclient01 kernel: "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables thi s message. Mar 20 15:00:55 tasclient01 kernel: stat D 0000000000000000 0 951 29769 0x00000084 Mar 20 15:00:55 tasclient01 kernel: ffff8801d0471a58 0000000000000082 0000000000000000 000000000000000d Mar 20 15:00:55 tasclient01 kernel: 0000000000000004 ffff880237fee800 ffff880116db5610 0000000000000630 Mar 20 15:00:55 tasclient01 kernel: ffff8802143f1098 ffff8801d0471fd8 000000000000fbc8 ffff8802143f1098 Mar 20 15:00:55 tasclient01 kernel: Call Trace: Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffff8152935e&gt;] __mutex_lock_slowpath+0x13e/0x180 Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffff815291fb&gt;] mutex_lock+0x2b/0x50 Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffffa09e009a&gt;] ll_layout_refresh+0x1da/0xc60 [lustre] Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffff810f0e46&gt;] ? ftrace_test_stop_func+0x16/0x20 Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffff8100ad96&gt;] ? ftrace_call+0x5/0x2b Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffff810f0e46&gt;] ? ftrace_test_stop_func+0x16/0x20 Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffffa0a04ab0&gt;] ? ll_md_blocking_ast+0x0/0x7d0 [lustre] Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffffa06996a0&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffffa04dfe31&gt;] ? cl_io_slice_add+0xc1/0x190 [obdclass] Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffffa0a2d8f0&gt;] vvp_io_init+0x340/0x490 [lustre] Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffff810f0e46&gt;] ? ftrace_test_stop_func+0x16/0x20 Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffff8100ad96&gt;] ? ftrace_call+0x5/0x2b Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffffa04df278&gt;] cl_io_init0+0x98/0x160 [obdclass] Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffffa04e1ea4&gt;] cl_io_init+0x64/0xe0 [obdclass] Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffffa0a23161&gt;] cl_glimpse_size0+0x91/0x1d0 [lustre] Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffffa09d4c25&gt;] ll_inode_revalidate_it+0x1a5/0x1d0 [lustre] Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffffa09d4c99&gt;] ll_getattr_it+0x49/0x170 [lustre] Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffffa09d4df7&gt;] ll_getattr+0x37/0x40 [lustre] Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffff81227163&gt;] ? security_inode_getattr+0x23/0x30 Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffff8118e631&gt;] vfs_getattr+0x51/0x80 Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffff8118e6c4&gt;] vfs_fstatat+0x64/0xa0 Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffff810f0e46&gt;] ? ftrace_test_stop_func+0x16/0x20 Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffff8118e76e&gt;] vfs_lstat+0x1e/0x20 Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffff8118e794&gt;] sys_newlstat+0x24/0x50 Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffff810e1cc7&gt;] ? audit_syscall_entry+0x1d7/0x200 Mar 20 15:00:55 tasclient01 kernel: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>What's this process? From the name I guess it's not copy tool. John's patch can only fix copy tool case.</p> <p>You will need patch 13138 too to address this case if the restoring will take longer than 120 seconds.</p> <p>The process is stat, which does a stat() on a file.<br/> Is it the problem your patch should fix, or is it fixing something else?</p> <p>Yes, the patch can fix this problem</p> <p>This patch landed quite some while back. Please speak up (or open a new ticket) if there are still residual issues to address.</p> <p>Hmm, I still got following error on recent master branch with 13750 patch. </p> <p>May 6 19:13:44 server1 kernel: INFO: task test.py:32411 blocked for more than 120 seconds.<br/> May 6 19:13:44 server1 kernel: Not tainted 2.6.32-431.29.2.el6_lustre.2.5.29.ddnpf4.x86_64 #1<br/> May 6 19:13:44 server1 kernel: "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> May 6 19:13:44 server1 kernel: test.py D 0000000000000000 0 32411 23005 0x00000080<br/> May 6 19:13:44 server1 kernel: ffff88055238fb08 0000000000000082 0000000000000000 ffff88055351c5c0<br/> May 6 19:13:44 server1 kernel: ffff88055238fab8 ffffffffa0e0685e ffff88086d4a49d8 ffff880597c9b980<br/> May 6 19:13:44 server1 kernel: ffff880553ee85f8 ffff88055238ffd8 000000000000fbc8 ffff880553ee85f8<br/> May 6 19:13:44 server1 kernel: Call Trace:<br/> May 6 19:13:44 server1 kernel: <span class="error">&#91;&lt;ffffffffa0e0685e&gt;&#93;</span> ? mdc_set_open_replay_data+0x29e/0x500 <span class="error">&#91;mdc&#93;</span><br/> May 6 19:13:44 server1 kernel: <span class="error">&#91;&lt;ffffffff8152b9de&gt;&#93;</span> __mutex_lock_slowpath+0x13e/0x180<br/> May 6 19:13:44 server1 kernel: <span class="error">&#91;&lt;ffffffff8152b87b&gt;&#93;</span> mutex_lock+0x2b/0x50<br/> May 6 19:13:44 server1 kernel: <span class="error">&#91;&lt;ffffffffa0ec7ea1&gt;&#93;</span> ll_layout_refresh+0x191/0x300 <span class="error">&#91;lustre&#93;</span><br/> May 6 19:13:44 server1 kernel: <span class="error">&#91;&lt;ffffffffa0ed78ac&gt;&#93;</span> ll_setattr_raw+0x5cc/0xbe0 <span class="error">&#91;lustre&#93;</span><br/> May 6 19:13:44 server1 kernel: <span class="error">&#91;&lt;ffffffffa0ed7f25&gt;&#93;</span> ll_setattr+0x65/0xd0 <span class="error">&#91;lustre&#93;</span><br/> May 6 19:13:44 server1 kernel: <span class="error">&#91;&lt;ffffffff811a8158&gt;&#93;</span> notify_change+0x168/0x340<br/> May 6 19:13:44 server1 kernel: <span class="error">&#91;&lt;ffffffff81187374&gt;&#93;</span> do_truncate+0x64/0xa0<br/> May 6 19:13:44 server1 kernel: <span class="error">&#91;&lt;ffffffff8119c4b1&gt;&#93;</span> do_filp_open+0x861/0xd20<br/> May 6 19:13:44 server1 kernel: <span class="error">&#91;&lt;ffffffff81290c5a&gt;&#93;</span> ? strncpy_from_user+0x4a/0x90<br/> May 6 19:13:44 server1 kernel: <span class="error">&#91;&lt;ffffffff811a9032&gt;&#93;</span> ? alloc_fd+0x92/0x160<br/> May 6 19:13:44 server1 kernel: <span class="error">&#91;&lt;ffffffff81186039&gt;&#93;</span> do_sys_open+0x69/0x140<br/> May 6 19:13:44 server1 kernel: <span class="error">&#91;&lt;ffffffff81186150&gt;&#93;</span> sys_open+0x20/0x30<br/> May 6 19:13:44 server1 kernel: <span class="error">&#91;&lt;ffffffff8100b072&gt;&#93;</span> system_call_fastpath+0x16/0x1b</p> <p>&gt; Hmm, I still got following error on recent master branch with 13750 patch.<br/> &gt; ...</p> <p>Please post your reproducer.</p> <p>Hi John,</p> <p>Sorry, the problem I got today is a different issue. And I am not sure whether it is a Bug or not. It is confirmed that <a href="https://jira.whamcloud.com/browse/LU-4727" title="Lhsmtool_posix process stuck in ll_layout_refresh() when restoring" class="issue-link" data-issue-key="LU-4727"><del>LU-4727</del></a> is fixed for me too. </p> <p>I got the new problem when the HSM tool (which is a private tool, not standard lhsmtool_posix) exits right after 'lfs hsm_restore' completes. But the action status in "lctl get_param mdt.server1-MDT0000.hsm.actions" is still STARTED, not SUCCEED. As long as the tool keeps itself alive until the action status become SUCCEED, this problem is gone.</p> <p>However, this might be a issue too, because the HSM tool can exist at any time. And even it restarts, the action will never continue. I don't know any other way to recover HSM status except rebooting the machine. Do you have any advice? Thanks!</p> Duplicate LU-5196 Related LUDOC-252 LU-4728 LU-6460 LU-4002 Development Rank 1|hzwh1j: Rank (Obsolete) 13001 Severity