https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6039 Lustre <p>cmd :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> # PDSH="pdsh -R ssh -S -w" ONLY=228 MDS_MOUNT_OPTS="-o rw,user_xattr" OST_MOUNT_OPTS="-o user_xattr" AGTCOUNT=1 agt1_HOST=vinayak lustre/tests/sanity-hsm.sh </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Dec 17 11:37:58 vinayak_centos kernel: INFO: task lhsmtool_posix:26565 blocked for more than 120 seconds. Dec 17 11:37:58 vinayak_centos kernel: Not tainted 2.6.32.431.20.3.el6_lustre #1 Dec 17 11:37:58 vinayak_centos kernel: "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 17 11:37:58 vinayak_centos kernel: lhsmtool_posi D 0000000000000001 0 26565 1 0x00000080 Dec 17 11:37:58 vinayak_centos kernel: ffff880000f4ba98 0000000000000082 0000000000000000 ffff88000df84000 Dec 17 11:37:58 vinayak_centos kernel: ffff880000f4ba98 ffffffff8128d1f6 0000000200000010 0000000affffffff Dec 17 11:37:58 vinayak_centos kernel: ffff88002d703098 ffff880000f4bfd8 000000000000fbc8 ffff88002d703098 Dec 17 11:37:58 vinayak_centos kernel: Call Trace: Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffff8128d1f6&gt;] ? vsnprintf+0x336/0x5e0 Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffff8152a40e&gt;] __mutex_lock_slowpath+0x13e/0x180 Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffff8152a2ab&gt;] mutex_lock+0x2b/0x50 Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffffa0f3219a&gt;] ll_layout_refresh+0x1da/0xcd0 [lustre] Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffffa07c4ab3&gt;] ? ldlm_lock_add_to_lru+0x43/0x120 [ptlrpc] Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffffa0f5cd80&gt;] ? ll_md_blocking_ast+0x0/0x7f0 [lustre] Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffffa07e87a0&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffffa0606011&gt;] ? cl_io_slice_add+0xc1/0x190 [obdclass] Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffffa0f82d70&gt;] vvp_io_init+0x340/0x490 [lustre] Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffffa0605088&gt;] cl_io_init0+0x88/0x150 [obdclass] Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffffa05fe505&gt;] ? cl_env_get+0x195/0x350 [obdclass] Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffffa0608004&gt;] cl_io_init+0x64/0xe0 [obdclass] Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffffa0f78ab1&gt;] cl_glimpse_size0+0x91/0x1d0 [lustre] Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffffa0f27966&gt;] ll_getattr+0x2e6/0x800 [lustre] Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffff8119b442&gt;] ? user_path_at+0x62/0xa0 Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffff8118e851&gt;] vfs_getattr+0x51/0x80 Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffff8118e8e4&gt;] vfs_fstatat+0x64/0xa0 Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffff8118e944&gt;] sys_newfstatat+0x24/0x50 Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffff810e1e87&gt;] ? audit_syscall_entry+0x1d7/0x200 Dec 17 11:37:58 vinayak_centos kernel: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> single node setup on local vm LU-6039

sanity-hsm test 228: INFO: task lhsmtool_posix:26565 blocked for more than 120 seconds.

Bug Critical Resolved Duplicate WC Triage Vinayak Hariharmath HSM Wed, 17 Dec 2014 06:47:39 +0000 Thu, 18 Dec 2014 23:18:01 +0000 Thu, 18 Dec 2014 23:18:01 +0000 Lustre 2.5.0 0 4 <p>If you can reproduce this problem, please use the following command:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>echo t &gt; /proc/sysrq-trigger dmesg &gt; log.txt </pre> </div></div> <p>and then attach the log.txt file here. I believe there is another process holding the mutex and waiting for response from the MDT.</p> <p>This bug is filed against 2.5.0. Are you really running that version, or is it 2.5.3 (which you probably should use, or even the tip of b2_5)?</p> <p>I ran it on master. Wrongly updated affect/versions.</p> <p>The test_228 is doing hsm_release 2 times. I commented out 2nd release statement and test case got passed. </p> <p>I mimicked the steps manually <br/> 1. cd /mnt/lustre<br/> 2. created file dd if=/dev/zero of=test/tfile bs=1k count=1k<br/> 3. lfs hsm_archive --archive=1 test/tfile<br/> 4. lfs hsm_release test/tfile<br/> 5. cp --sparse=auto test/tfile test/tfile.2<br/> 6. lfs hsm_release test/tfile<br/> 7. mkdir test/tdir<br/> 8. tar cf - --sparse test/tfile | tar xvf - -C test/tdir ==&gt; process hung on layout lock</p> <p>if we skip step 6, then everything goes fine.</p> <p>Even I checked with other operation (copy) instead of step 8<br/> cp --sparse=auto test/tfile test/tdir/tfile.2 ==&gt; this also got hung</p> <p>After this I got few of the questions in my mind<br/> 1. why file is released twice ?<br/> 2. Even if the file is released twice, why the operation getting hung ? <br/> releasing the archived file for "n" number of times should not affect.</p> <p>Jinshan Xiong : Yes, you are right. layout lock has been held by some one and getting stuck there. I will update the logs.</p> <p>logs file attached</p> <p>Got answer for <br/> 1. why file is released twice ?<br/> To perform step 5 ( cp --sparse=auto test/tfile test/tfile.2), file is restored first then copying is done.<br/> After this step hsm_state will be reverted to "exists" from "released". Thats why again the file is released for 2nd time.</p> <p>Below record proves that</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs hsm_state test/tfile test/tfile: (0x00000009) exists archived, archive_id:1 # lfs hsm_release test/tfile # lfs hsm_state test/tfile test/tfile: (0x0000000d) released exists archived, archive_id:1 # cp test/tfile test/tfile.2 lhsmtool_posix[13264]: copytool fs=lustre archive#=1 item_count=1 lhsmtool_posix[13264]: waiting for message from kernel lhsmtool_posix[13296]: '[0x200000400:0x2:0x0]' action RESTORE reclen 72, cookie=0x549282f9 lhsmtool_posix[13296]: processing file 'test/tfile' lhsmtool_posix[13296]: reading stripe rules from '/tmp/hsm//0002/0000/0400/0000/0002/0000/0x200000400:0x2:0x0.lov' for '/tmp/hsm//0002/0000/0400/0000/0002/0000/0x200000400:0x2:0x0' lhsmtool_posix[13296]: restoring data from '/tmp/hsm//0002/0000/0400/0000/0002/0000/0x200000400:0x2:0x0' to '{VOLATILE}=[0x200000400:0x5:0x0]' lhsmtool_posix[13296]: going to copy data from '/tmp/hsm//0002/0000/0400/0000/0002/0000/0x200000400:0x2:0x0' to '{VOLATILE}=[0x200000400:0x5:0x0]' lhsmtool_posix[13296]: data restore from '/tmp/hsm//0002/0000/0400/0000/0002/0000/0x200000400:0x2:0x0' to '{VOLATILE}=[0x200000400:0x5:0x0]' done lhsmtool_posix[13296]: Action completed, notifying coordinator cookie=0x549282f9, FID=[0x200000400:0x2:0x0], hp_flags=0 err=0 lhsmtool_posix[13296]: llapi_hsm_action_end() on '/mnt/lustre/.lustre/fid/0x200000400:0x2:0x0' ok (rc=0) # lfs hsm_state test/tfile test/tfile: (0x00000009) exists archived, archive_id:1 # lfs hsm_release test/tfile # lfs hsm_state test/tfile test/tfile: (0x0000000d) released exists archived, archive_id:1 </pre> </div></div> <p>is it related to <br/> <a href="https://jira.hpdd.intel.com/browse/LU-4727" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-4727</a> ?</p> <p>It looks like the same issue of <a href="https://jira.whamcloud.com/browse/LU-4727" title="Lhsmtool_posix process stuck in ll_layout_refresh() when restoring" class="issue-link" data-issue-key="LU-4727"><del>LU-4727</del></a>.</p> <p>duplication of <a href="https://jira.whamcloud.com/browse/LU-4727" title="Lhsmtool_posix process stuck in ll_layout_refresh() when restoring" class="issue-link" data-issue-key="LU-4727"><del>LU-4727</del></a>.</p> Development Epic client test Epic/Theme test Project Rank 1|hzx2of: Rank (Obsolete) 16831 Severity