https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2181 Lustre <p>This issue was created by maloo for yujian &lt;yujian@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/140674a6-16b2-11e2-962d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/140674a6-16b2-11e2-962d-52540035b04c</a>.</p> <p>Lustre Tag: v2_3_0_RC3<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_3/36" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/36</a><br/> Distro/Arch: RHEL6.3/x86_64(server), FC15/x86_64(client)<br/> Network: TCP<br/> ENABLE_QUOTA=yes</p> <p>The sub-test test_23a hung at unmounting the client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== conf-sanity test 23a: interrupt client during recovery mount delay ================================ 02:41:31 (1350294091) start mds service on fat-amd-2 Starting mds1: /dev/sdc5 /mnt/mds1 Started lustre-MDT0000 start ost1 service on fat-amd-3 Starting ost1: /dev/sdc5 /mnt/ost1 Started lustre-OST0000 mount lustre on /mnt/lustre..... Starting client: client-5: -o user_xattr,flock fat-amd-2@tcp:/lustre /mnt/lustre Stopping /mnt/mds1 (opts:) on fat-amd-2 Stopping client /mnt/lustre (opts: -f) </pre> </div></div> <p>Stack trace on client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 5526.947537] umount S ffff880316bb3170 0 7395 7009 0x00000080 [ 5526.954596] ffff8803136e57c8 0000000000000082 00000001004fdeea ffff88030af44560 [ 5526.962037] ffff8803136e5fd8 ffff8803136e5fd8 0000000000013840 0000000000013840 [ 5526.969479] ffff880323191720 ffff88030af44560 0000000000000000 0000000000000286 [ 5526.976921] Call Trace: [ 5526.979396] [&lt;ffffffffa054a570&gt;] ? ptlrpc_interrupted_set+0x0/0x120 [ptlrpc] [ 5526.986517] [&lt;ffffffff8147461a&gt;] schedule_timeout+0xa7/0xde [ 5526.992168] [&lt;ffffffff81060b58&gt;] ? process_timeout+0x0/0x10 [ 5526.997829] [&lt;ffffffffa02ae761&gt;] cfs_waitq_timedwait+0x11/0x20 [libcfs] [ 5527.004550] [&lt;ffffffffa0555a9c&gt;] ptlrpc_set_wait+0x2ec/0x8c0 [ptlrpc] [ 5527.011066] [&lt;ffffffff8104df76&gt;] ? default_wake_function+0x0/0x14 [ 5527.017270] [&lt;ffffffffa05560e8&gt;] ptlrpc_queue_wait+0x78/0x230 [ptlrpc] [ 5527.023900] [&lt;ffffffffa05386c5&gt;] ldlm_cli_enqueue+0x2f5/0x7b0 [ptlrpc] [ 5527.030528] [&lt;ffffffffa0536d90&gt;] ? ldlm_completion_ast+0x0/0x6f0 [ptlrpc] [ 5527.037408] [&lt;ffffffffa0905cc0&gt;] ? ll_md_blocking_ast+0x0/0x710 [lustre] [ 5527.044186] [&lt;ffffffffa0744e55&gt;] mdc_enqueue+0x505/0x1590 [mdc] [ 5527.050196] [&lt;ffffffffa02b9578&gt;] ? libcfs_log_return+0x28/0x40 [libcfs] [ 5527.056885] [&lt;ffffffffa074609e&gt;] ? mdc_revalidate_lock+0x1be/0x1d0 [mdc] [ 5527.063661] [&lt;ffffffffa0746270&gt;] mdc_intent_lock+0x1c0/0x5c0 [mdc] [ 5527.069932] [&lt;ffffffffa0905cc0&gt;] ? ll_md_blocking_ast+0x0/0x710 [lustre] [ 5527.076734] [&lt;ffffffffa0536d90&gt;] ? ldlm_completion_ast+0x0/0x6f0 [ptlrpc] [ 5527.083601] [&lt;ffffffffa09eed8b&gt;] lmv_intent_lookup+0x3bb/0x11c0 [lmv] [ 5527.090136] [&lt;ffffffffa0905cc0&gt;] ? ll_md_blocking_ast+0x0/0x710 [lustre] [ 5527.096913] [&lt;ffffffffa09f12f0&gt;] lmv_intent_lock+0x310/0x370 [lmv] [ 5527.103190] [&lt;ffffffffa0905cc0&gt;] ? ll_md_blocking_ast+0x0/0x710 [lustre] [ 5527.109982] [&lt;ffffffffa08e0944&gt;] __ll_inode_revalidate_it+0x214/0xd90 [lustre] [ 5527.117295] [&lt;ffffffffa0905cc0&gt;] ? ll_md_blocking_ast+0x0/0x710 [lustre] [ 5527.124084] [&lt;ffffffffa08e1764&gt;] ll_inode_revalidate_it+0x44/0x1a0 [lustre] [ 5527.131136] [&lt;ffffffffa08e1903&gt;] ll_getattr_it+0x43/0x170 [lustre] [ 5527.137408] [&lt;ffffffffa08e1a64&gt;] ll_getattr+0x34/0x40 [lustre] [ 5527.143317] [&lt;ffffffff81125113&gt;] vfs_getattr+0x45/0x63 [ 5527.148535] [&lt;ffffffff8112517e&gt;] vfs_fstatat+0x4d/0x63 [ 5527.153751] [&lt;ffffffff811251cf&gt;] vfs_stat+0x1b/0x1d [ 5527.158709] [&lt;ffffffff811252ce&gt;] sys_newstat+0x1a/0x33 [ 5527.163927] [&lt;ffffffff81129f89&gt;] ? path_put+0x1f/0x23 [ 5527.169059] [&lt;ffffffff8109fa08&gt;] ? audit_syscall_entry+0x145/0x171 [ 5527.175315] [&lt;ffffffff81009bc2&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Info required for matching: conf-sanity 23a</p> LU-2181

failure conf-sanity test_23a: umount -f client hung in stat() when MDS down

Bug Minor Resolved Fixed Bob Glossman Maloo Mon, 15 Oct 2012 06:40:47 +0000 Thu, 13 Apr 2023 22:24:21 +0000 Wed, 1 Oct 2014 18:04:45 +0000 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.4.0 0 10 <p>Lustre Tag: v2_3_0_RC3<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_3/36" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/36</a><br/> Distro/Arch: RHEL6.3/x86_64(server), FC15/x86_64(client)<br/> Network: TCP<br/> ENABLE_QUOTA=yes</p> <p>conf-sanity test 34b also hit the same issue:<br/> <a href="https://maloo.whamcloud.com/test_sets/b1d732d8-16da-11e2-afe1-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b1d732d8-16da-11e2-afe1-52540035b04c</a></p> <p>Lustre Tag: v2_3_0_RC3<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_3/36" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/36</a><br/> Distro/Arch: RHEL6.3/x86_64(server), FC15/x86_64(client)<br/> Network: TCP<br/> ENABLE_QUOTA=yes</p> <p>conf-sanity test 45 also hit the same issue:<br/> <a href="https://maloo.whamcloud.com/test_sets/c0834482-16e1-11e2-afe1-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c0834482-16e1-11e2-afe1-52540035b04c</a></p> <p>Yangsheng is looking into this one</p> <p>From first investigate, looks like client loop forever in ptlrpc_set_wait(). maybe relate to commit 7223b4746c71bf450c178ed21ddf99a0e3e26a1a. But i am still not understand why it just impact FC15 client. </p> <p>Found another failure in SLES11 SP2 client.<br/> <a href="https://maloo.whamcloud.com/test_sets/a9ac1d4a-2539-11e2-9e7c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a9ac1d4a-2539-11e2-9e7c-52540035b04c</a></p> <p>client trace shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[20995.079218] umount          S 0000000000000001     0  6319   5616 0x00000000 [20995.079218]  ffff880062751878 0000000000000082 0000000000000000 ffff880062750010 [20995.079218]  0000000000011180 0000000000011180 ffff880062751fd8 ffff880062751fd8 [20995.079218]  0000000000011180 ffff88006c426340 ffff880000000028 ffff880026852440 [20995.079218] Call Trace: [20995.079218]  [&lt;ffffffff81441570&gt;] schedule_timeout+0x1b0/0x2a0 [20995.079218]  [&lt;ffffffffa080bfbb&gt;] ptlrpc_set_wait+0x2eb/0x920 [ptlrpc] [20995.079218]  [&lt;ffffffffa080c666&gt;] ptlrpc_queue_wait+0x76/0x230 [ptlrpc] [20995.079218]  [&lt;ffffffffa07ef83b&gt;] ldlm_cli_enqueue+0x23b/0x790 [ptlrpc] [20995.079218]  [&lt;ffffffffa09928c8&gt;] mdc_enqueue+0x2b8/0xcb0 [mdc] [20995.079218]  [&lt;ffffffffa0993468&gt;] mdc_intent_lock+0x1a8/0xd40 [mdc] [20995.079218]  [&lt;ffffffffa09497a7&gt;] lmv_intent_lookup+0x3b7/0x11b0 [lmv] [20995.079218]  [&lt;ffffffffa094a8e2&gt;] lmv_intent_lock+0x342/0x3a0 [lmv] [20995.079218]  [&lt;ffffffffa0b17ea0&gt;] __ll_inode_revalidate_it+0x550/0xd70 [lustre] [20995.079218]  [&lt;ffffffffa0b18959&gt;] ll_inode_revalidate_it+0x49/0x1c0 [lustre] [20995.079218]  [&lt;ffffffffa0b18b0e&gt;] ll_getattr_it+0x3e/0x160 [lustre] [20995.079218]  [&lt;ffffffffa0b18c5f&gt;] ll_getattr+0x2f/0x40 [lustre] [20995.079218]  [&lt;ffffffff81154631&gt;] vfs_fstatat+0x81/0x90 [20995.079218]  [&lt;ffffffff8115478f&gt;] sys_newstat+0x1f/0x50 [20995.079218]  [&lt;ffffffff8144ac92&gt;] system_call_fastpath+0x16/0x1b [20995.079218]  [&lt;00007f60d1ef9335&gt;] 0x7f60d1ef9334 </pre> </div></div> <p>I have seen this repeatedly in sles11 sp2 clients too, so it's not just FC15 client.</p> <p>I'm very suspicious that this may be due to version skew in the umount command. In el6 which doesn't fail umount is found in util-linux-ng 2.17.2, in sles11 sp2 it is in util-linux 2.19.1, don't know what the version is in FC15.</p> <p>Running strace on a 'umount -f' with mds down or unreachable, in el6 I see the first significant syscall is umount():</p> <p>....<br/> getuid() = 0<br/> geteuid() = 0<br/> readlink("/mnt", 0x7fff08de1ad0, 4096) = -1 EINVAL (Invalid argument)<br/> readlink("/mnt/lustre", 0x7fff08de1ad0, 4096) = -1 EINVAL (Invalid argument)<br/> umask(077) = 022<br/> open("/etc/mtab", O_RDONLY) = 3<br/> umask(022) = 077<br/> fstat(3, </p> {st_mode=S_IFREG|0644, st_size=480, ...} <p>) = 0<br/> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4c55c00000<br/> read(3, "/dev/mapper/vg_centos1-lv_root /"..., 4096) = 480<br/> read(3, "", 4096) = 0<br/> close(3) = 0<br/> munmap(0x7f4c55c00000, 4096) = 0<br/> stat("/sbin/umount.lustre", 0x7fff08de2900) = -1 ENOENT (No such file or directory)<br/> rt_sigprocmask(SIG_BLOCK, ~<span class="error">&#91;TRAP SEGV RTMIN RT_1&#93;</span>, NULL, 8) = 0<br/> umount("/mnt/lustre", MNT_FORCE) = 0<br/> ....</p> <p>In sles11 sp2 the first significant syscall is stat() on the mount point:</p> <p> ....<br/> getuid() = 0<br/> geteuid() = 0<br/> readlink("/mnt", 0x7fff0e6455d0, 4096) = -1 EINVAL (Invalid argument)<br/> readlink("/mnt/lustre", 0x7fff0e6455d0, 4096) = -1 EINVAL (Invalid argument)<br/> stat("/mnt/lustre",<br/> &lt; hangs here &gt;</p> <p>It appears to be the stat() call of the mount point that gets into the permanent client loop YS has described and the umount command never gets to the umount() syscall. In earlier versions of the umount command, as seen in el6 clients, there is no stat() call and the umount() call succeeds even with the mds down.</p> <p>Bob, the util-linux sources are at <a href="https://github.com/karelzak/util-linux" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/karelzak/util-linux</a> and would be a good place to investigate where/why the <tt>stat()</tt> call was added to the code before <tt>umount()</tt> is called. As I was typing this, Oleg already checked this and found the following patch which fixes the problem:</p> <p> <a href="https://github.com/karelzak/util-linux/commit/6d5d2b5fd342308bb2db6b82d89ac22ef7374184" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/karelzak/util-linux/commit/6d5d2b5fd342308bb2db6b82d89ac22ef7374184</a></p> <p>We need to get this patch into the upstream SLES release if it isn't already. Peter will be contacting SuSE about this. I'll contact Red Hat about the same.</p> <p>In the meantime, can you please add this subtest to the ALWAYS_EXCEPT list for SLES11 SP2.</p> <p>Andreas, I don't think just adding this subtest to ALWAYS_EXCEPT is sufficient. I know of at least one other test that does umount with mds down, conf-sanity 34b, and there are probably others I don't know immediately.</p> <p>patch to disable the tests I know about for sles11 sp2:<br/> <a href="http://review.whamcloud.com/4639" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4639</a></p> <p>This bug needs to stay open for tracking until SLES11 has the fix to umount to remove the stat() call when -f is given.</p> <p>conf-sanity test_45 also hit this error on sles11 sp2 client:<br/> <a href="https://maloo.whamcloud.com/test_sets/3ed08c6c-46dc-11e2-b16f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/3ed08c6c-46dc-11e2-b16f-52540035b04c</a></p> <p>I suspect Sarah hit this problem due to running tests with SLOW=yes. By default SLOW=no and test 45 gets skipped. Probably needs to be fixed by adding 45 to the sles11 always skip list at the top of conf-sanity.sh</p> <p>added subtest 45 to the sles11 sp2 skip list<br/> <a href="http://review.whamcloud.com/4884" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4884</a></p> <p>SUSE pointed me to this patch:<br/> <a href="http://download.novell.com/Download?buildid=G4nSHdRyeOI~" class="external-link" target="_blank" rel="nofollow noopener">http://download.novell.com/Download?buildid=G4nSHdRyeOI~</a></p> <p>The patch consists of 7 patches:<br/> libblkid1-2.19.1-6.33.35.1.x86_64.rpm<br/> libblkid1-32bit-2.19.1-6.33.35.1.x86_64.rpm<br/> libuuid1-2.19.1-6.33.35.1.x86_64.rpm<br/> libuuid1-32bit-2.19.1-6.33.35.1.x86_64.rpm<br/> util-linux-2.19.1-6.33.35.1.x86_64.rpm<br/> util-linux-lang-2.19.1-6.33.35.1.x86_64.rpm<br/> uuid-runtime-2.19.1-6.33.35.1.x86_64.rpm</p> <p>I am not sure I need all of them, but installed them anyway. With these rpm set installed, the test passed!</p> <p>Thanks for the info, Jay. I suspect the important part of that patch is the util-linux rpm. When that patch becomes part of a regular sles11 sp2 update and no longer needs to be specially downloaded and applied, we can probably go ahead and take out the test skips added to work around the problem. We've been waiting for that to happen.</p> <p>Bob, any idea if this update has made it into SP2 or SP3 (whatever we are currently testing master on)? Then we could remove the exception from conf-sanity.sh.</p> <p>Andreas, the conf-sanity.sh exception as it is now only skips the test on SP2. it runs fine on SP3 and always has. That being the case there is no harm to leaving the exception in there forever. However it could indeed be removed as we no longer build or test SP2 in master.</p> <p>Closing bug, since the test exception is only enforced for SLES11SP2, and will naturally expire. </p> Related LU-9042 Development Rank 1|hzvae7: Rank (Obsolete) 5219 Severity