https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5196 Lustre <p>Internal Cray bug report (from Frank Zago):<br/> After trying to migrate a file back on Lustre, from the TAS backend, I noticed nothing was happening. I ctrl-C the TAS copytool (CMM), and it doesn't exit.</p> <p>The kernel has the following trace:</p> <p>Lustre: Layout lock feature supported.<br/> Lustre: Mounted tas01-client<br/> INFO: task tas_cmm:18139 blocked for more than 120 seconds.<br/> Tainted: P --------------- 2.6.32-431.17.1.el6.x86_64 #1<br/> "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> tas_cmm D 0000000000000000 0 18139 14601 0x00000080<br/> ffff8801052c5a58 0000000000000082 0000000000000000 0000000000000000<br/> ffff880106ef8580 0000000000000000 ffff8801313c70b8 ffff8801389e8800<br/> ffff88007e070638 ffff8801052c5fd8 000000000000fbc8 ffff88007e070638<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff8152935e&gt;&#93;</span> __mutex_lock_slowpath+0x13e/0x180<br/> <span class="error">&#91;&lt;ffffffff815291fb&gt;&#93;</span> mutex_lock+0x2b/0x50<br/> <span class="error">&#91;&lt;ffffffffa09dc04c&gt;&#93;</span> ll_layout_refresh+0x25c/0xfe0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa067bb1a&gt;&#93;</span> ? ldlm_lock_add_to_lru_nolock+0x4a/0x110 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa03830d7&gt;&#93;</span> ? cfs_hash_bd_lookup_intent+0x37/0x130 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0a01040&gt;&#93;</span> ? ll_md_blocking_ast+0x0/0x7d0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0694550&gt;&#93;</span> ? ldlm_completion_ast+0x0/0x920 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04db721&gt;&#93;</span> ? cl_io_slice_add+0xc1/0x190 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0a28240&gt;&#93;</span> vvp_io_init+0x340/0x490 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04dab68&gt;&#93;</span> cl_io_init0+0x98/0x160 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04ce995&gt;&#93;</span> ? cl_env_get+0x195/0x350 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04dd794&gt;&#93;</span> cl_io_init+0x64/0xe0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0a1f751&gt;&#93;</span> cl_glimpse_size0+0x91/0x1d0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa09d0855&gt;&#93;</span> ll_inode_revalidate_it+0x1a5/0x1d0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81196666&gt;&#93;</span> ? final_putname+0x26/0x50<br/> <span class="error">&#91;&lt;ffffffffa09d08c9&gt;&#93;</span> ll_getattr_it+0x49/0x170 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa09d0a27&gt;&#93;</span> ll_getattr+0x37/0x40 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81227163&gt;&#93;</span> ? security_inode_getattr+0x23/0x30<br/> <span class="error">&#91;&lt;ffffffff8118e631&gt;&#93;</span> vfs_getattr+0x51/0x80<br/> <span class="error">&#91;&lt;ffffffff8118e6c4&gt;&#93;</span> vfs_fstatat+0x64/0xa0<br/> <span class="error">&#91;&lt;ffffffff8118e82b&gt;&#93;</span> vfs_stat+0x1b/0x20<br/> <span class="error">&#91;&lt;ffffffff8118e854&gt;&#93;</span> sys_newstat+0x24/0x50<br/> <span class="error">&#91;&lt;ffffffff810e1cc7&gt;&#93;</span> ? audit_syscall_entry+0x1d7/0x200<br/> <span class="error">&#91;&lt;ffffffff810e1abe&gt;&#93;</span> ? __audit_syscall_exit+0x25e/0x290<br/> <span class="error">&#91;&lt;ffffffff8100b072&gt;&#93;</span> system_call_fastpath+0x16/0x1b</p> LU-5196

HSM: client task stuck waiting for mutex in ll_layout_refresh

Bug Critical Resolved Duplicate WC Triage Patrick Farrell hsm Fri, 13 Jun 2014 18:37:53 +0000 Wed, 25 Feb 2015 18:53:19 +0000 Wed, 25 Feb 2015 18:53:19 +0000 Lustre 2.5.1 0 6 <p>Dump uploading to ftp ftp.whamcloud.com:/uploads/<a href="https://jira.whamcloud.com/browse/LU-5196" title="HSM: client task stuck waiting for mutex in ll_layout_refresh" class="issue-link" data-issue-key="LU-5196"><del>LU-5196</del></a>/<a href="https://jira.whamcloud.com/browse/LU-5196" title="HSM: client task stuck waiting for mutex in ll_layout_refresh" class="issue-link" data-issue-key="LU-5196"><del>LU-5196</del></a>_ll_layout_refresh.tar.gz</p> <p>My apologies...</p> <p>After further investigation, I think we're not going to have information to debug this one. I hadn't looked closely enough. The other thread which is holding the mutex is waiting for a response from the MDS, and we do not have an MDS dump.</p> <p>Here's the stack trace for that other thread:</p> <p>PID: 18218 TASK: ffff88013c8b8aa0 CPU: 0 COMMAND: "md5sum"<br/> #0 <span class="error">&#91;ffff8801152396d8&#93;</span> schedule at ffffffff81527bb0<br/> #1 <span class="error">&#91;ffff8801152397a0&#93;</span> ldlm_completion_ast at ffffffffa0694a95 <span class="error">&#91;ptlrpc&#93;</span><br/> #2 <span class="error">&#91;ffff880115239830&#93;</span> ldlm_cli_enqueue_fini at ffffffffa068efe6 <span class="error">&#91;ptlrpc&#93;</span><br/> #3 <span class="error">&#91;ffff8801152398d0&#93;</span> ldlm_cli_enqueue at ffffffffa068f8c5 <span class="error">&#91;ptlrpc&#93;</span><br/> #4 <span class="error">&#91;ffff880115239980&#93;</span> mdc_enqueue at ffffffffa0876d1e <span class="error">&#91;mdc&#93;</span><br/> #5 <span class="error">&#91;ffff880115239ac0&#93;</span> lmv_enqueue at ffffffffa0b04a84 <span class="error">&#91;lmv&#93;</span><br/> #6 <span class="error">&#91;ffff880115239b90&#93;</span> ll_layout_refresh at ffffffffa09dc305 <span class="error">&#91;lustre&#93;</span><br/> #7 <span class="error">&#91;ffff880115239cd0&#93;</span> vvp_io_fini at ffffffffa0a27b33 <span class="error">&#91;lustre&#93;</span><br/> #8 <span class="error">&#91;ffff880115239d30&#93;</span> vvp_io_read_fini at ffffffffa0a27cec <span class="error">&#91;lustre&#93;</span><br/> #9 <span class="error">&#91;ffff880115239d60&#93;</span> cl_io_fini at ffffffffa04dd947 <span class="error">&#91;obdclass&#93;</span><br/> #10 <span class="error">&#91;ffff880115239d90&#93;</span> ll_file_io_generic at ffffffffa09cbd27 <span class="error">&#91;lustre&#93;</span><br/> #11 <span class="error">&#91;ffff880115239e20&#93;</span> ll_file_aio_read at ffffffffa09ccf8f <span class="error">&#91;lustre&#93;</span><br/> #12 <span class="error">&#91;ffff880115239e80&#93;</span> ll_file_read at ffffffffa09cd27c <span class="error">&#91;lustre&#93;</span><br/> #13 <span class="error">&#91;ffff880115239ef0&#93;</span> vfs_read at ffffffff81189365<br/> #14 <span class="error">&#91;ffff880115239f30&#93;</span> sys_read at ffffffff811894a1<br/> #15 <span class="error">&#91;ffff880115239f80&#93;</span> system_call_fastpath at ffffffff8100b072</p> <p>Looks like a dup of <a href="https://jira.whamcloud.com/browse/LU-4727" title="Lhsmtool_posix process stuck in ll_layout_refresh() when restoring" class="issue-link" data-issue-key="LU-4727"><del>LU-4727</del></a>.</p> <p>This issue should be resolved as a dup of <a href="https://jira.whamcloud.com/browse/LU-4727" title="Lhsmtool_posix process stuck in ll_layout_refresh() when restoring" class="issue-link" data-issue-key="LU-4727"><del>LU-4727</del></a>.</p> <p>ok. Thanks Cory</p> Duplicate LU-4727 Development Rank 1|hzwp27: Rank (Obsolete) 14514 Severity