https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17301 Lustre <p>After patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51329," class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51329,</a> client mount hangs for several minutes when using `llmount.sh`. When this patch is reverted, the mount completes in a normal amount of time.</p> <p> </p> <p>Setup:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Kernel - 5.15.94 openZFS - 2.1.99-1735_gdc72c60ec Distro - CentOS Linux release 8.5.2111 </pre> </div></div> <p> </p> <p>Kernel messages:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [  480.629670] Lustre: mdt_rdpg01_001: service thread pid 52665 was inactive <span class="code-keyword">for</span> 40.946 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: [  480.632948] Pid: 52665, comm: mdt_rdpg01_001 5.15.94 #1 SMP Fri Feb 17 01:05:48 UTC 2023 [  480.634352] Call Trace TBD: [  500.084741] LustreError: 52665:0:(upcall_cache.c:251:upcall_cache_get_entry()) acquire <span class="code-keyword">for</span> key 0: error -110 [  500.086740] Lustre: Mounted lustre-client</pre> </div></div> <p>The mount hangs for several minutes even after the 'Mounted' message gets printed.</p> LU-17301

Client mount hangs for several minutes

Bug Minor Resolved Fixed Shaun Tancheff Tim Day Mon, 20 Nov 2023 16:49:56 +0000 Wed, 13 Dec 2023 15:22:26 +0000 Wed, 13 Dec 2023 15:22:26 +0000 Lustre 2.16.0 0 8 <p>Probably the immediate step should be to revert this patch from master until a fix can be worked out...</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ uname -r 3.10.0-1160.15.2.el7.x86_64 $ cat /etc/redhat-release  CentOS Linux release 7.9.2009 (Core)</pre> </div></div> <p>Also seen on 4.X series kernel. (el8). </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Nov 20 23:25:51 centos79 kernel: Lustre: mdt_rdpg00_001: service thread pid 8735 was inactive for 40.059 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 20 23:25:51 centos79 kernel: Pid: 8735, comm: mdt_rdpg00_001 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 Nov 20 23:25:51 centos79 kernel: Call Trace: Nov 20 23:25:51 centos79 kernel: [&lt;0&gt;] upcall_cache_get_entry+0x201/0x9d0 [obdclass] Nov 20 23:25:51 centos79 kernel: [&lt;0&gt;] mdt_identity_get+0x17/0x50 [mdt] Nov 20 23:25:51 centos79 kernel: [&lt;0&gt;] old_init_ucred_common+0xed/0x2c0 [mdt] Nov 20 23:25:51 centos79 kernel: [&lt;0&gt;] mdt_init_ucred+0x19c/0x320 [mdt] Nov 20 23:25:51 centos79 kernel: [&lt;0&gt;] mdt_getattr+0x156/0x640 [mdt] Nov 20 23:25:51 centos79 kernel: [&lt;0&gt;] tgt_request_handle+0x743/0x19a0 [ptlrpc] Nov 20 23:25:51 centos79 kernel: [&lt;0&gt;] ptlrpc_server_handle_request+0x261/0xcd0 [ptlrpc] Nov 20 23:25:51 centos79 kernel: [&lt;0&gt;] ptlrpc_main+0xc5c/0x16c0 [ptlrpc] Nov 20 23:25:51 centos79 kernel: [&lt;0&gt;] kthread+0xd1/0xe0 Nov 20 23:25:51 centos79 kernel: [&lt;0&gt;] ret_from_fork_nospec_begin+0xe/0x21 Nov 20 23:25:51 centos79 kernel: [&lt;0&gt;] 0xfffffffffffffffe Nov 20 23:26:11 centos79 kernel: LustreError: 8735:0:(upcall_cache.c:244:upcall_cache_get_entry()) acquire for key 0: error -110 Nov 20 23:26:11 centos79 kernel: Lustre: Mounted lustre-client Nov 20 23:26:52 centos79 kernel: Lustre: mdt00_002: service thread pid 8733 was inactive for 40.169 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 20 23:26:52 centos79 kernel: Pid: 8733, comm: mdt00_002 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 Nov 20 23:26:52 centos79 kernel: Call Trace: Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] upcall_cache_get_entry+0x201/0x9d0 [obdclass] Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] mdt_identity_get+0x17/0x50 [mdt] Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] old_init_ucred_common+0xed/0x2c0 [mdt] Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] mdt_init_ucred+0x19c/0x320 [mdt] Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] mdt_intent_getattr+0xc3/0x4b0 [mdt] Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] mdt_intent_opc+0x1c5/0xc40 [mdt] Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] mdt_intent_policy+0xfa/0x460 [mdt] Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] ldlm_lock_enqueue+0x3a4/0xb80 [ptlrpc] Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] ldlm_handle_enqueue+0x34d/0x17a0 [ptlrpc] Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] tgt_enqueue+0x68/0x240 [ptlrpc] Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] tgt_request_handle+0x743/0x19a0 [ptlrpc] Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] ptlrpc_server_handle_request+0x261/0xcd0 [ptlrpc] Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] ptlrpc_main+0xc5c/0x16c0 [ptlrpc] Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] kthread+0xd1/0xe0 Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] ret_from_fork_nospec_begin+0xe/0x21 Nov 20 23:26:52 centos79 kernel: [&lt;0&gt;] 0xfffffffffffffffe Nov 20 23:27:12 centos79 kernel: LustreError: 8733:0:(upcall_cache.c:244:upcall_cache_get_entry()) acquire for key 0: error -110</pre> </div></div> <p>Hello Arshad,</p> <p>can you please test the following patch:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> diff --git a/lustre/utils/l_getidentity.c b/lustre/utils/l_getidentity.c index 34c70d8d71..f99bdafc23 100644 --- a/lustre/utils/l_getidentity.c +++ b/lustre/utils/l_getidentity.c @@ -67,7 +67,7 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> nss_pw_buf_len; <span class="code-keyword">static</span> void *nss_pw_buf; <span class="code-keyword">static</span> <span class="code-object">int</span> nss_grent_buf_len; <span class="code-keyword">static</span> void *nss_grent_buf; -<span class="code-keyword">static</span> <span class="code-object">int</span> g_n_nss_modules; +<span class="code-keyword">static</span> <span class="code-object">int</span> g_n_nss_modules = 0; <span class="code-keyword">static</span> <span class="code-object">int</span> grent_mod_no = -1; struct nss_module { </pre> </div></div> <p>unfortunately I cannot reproduce the problem quickly.</p> <p>Thanks,<br/> Zam.</p> <p>"Alexander Zarochentsev &lt;alexander.zarochentsev@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53191" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53191</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17301" title="Client mount hangs for several minutes" class="issue-link" data-issue-key="LU-17301"><del>LU-17301</del></a> utils: l_getidentity nss init fix<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1794102c4b4b17f364715e3a9611191945bd5d77</p> <p>Hi Zam,</p> <p>Sorry, it did not work.  I am getting this by just doing llmount.sh</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Nov 21 07:15:56 centos79 kernel: Lustre: mdt_rdpg00_001: service thread pid 10702 was inactive for 40.000 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Nov 21 07:15:56 centos79 kernel: Pid: 10702, comm: mdt_rdpg00_001 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 Nov 21 07:15:56 centos79 kernel: Call Trace: Nov 21 07:15:56 centos79 kernel: [&lt;0&gt;] upcall_cache_get_entry+0x201/0xa00 [obdclass] Nov 21 07:15:56 centos79 kernel: [&lt;0&gt;] mdt_identity_get+0x17/0x50 [mdt] Nov 21 07:15:56 centos79 kernel: [&lt;0&gt;] old_init_ucred_common+0xed/0x2c0 [mdt] Nov 21 07:15:56 centos79 kernel: [&lt;0&gt;] mdt_init_ucred+0x19c/0x320 [mdt] Nov 21 07:15:56 centos79 kernel: [&lt;0&gt;] mdt_getattr+0x156/0x640 [mdt] Nov 21 07:15:56 centos79 kernel: [&lt;0&gt;] tgt_request_handle+0x743/0x19a0 [ptlrpc] Nov 21 07:15:56 centos79 kernel: [&lt;0&gt;] ptlrpc_server_handle_request+0x261/0xcd0 [ptlrpc] Nov 21 07:15:56 centos79 kernel: [&lt;0&gt;] ptlrpc_main+0xc5c/0x16c0 [ptlrpc] Nov 21 07:15:56 centos79 kernel: [&lt;0&gt;] kthread+0xd1/0xe0 Nov 21 07:15:56 centos79 kernel: [&lt;0&gt;] ret_from_fork_nospec_begin+0xe/0x21 Nov 21 07:15:56 centos79 kernel: [&lt;0&gt;] 0xfffffffffffffffe Nov 21 07:16:16 centos79 kernel: LustreError: 10702:0:(upcall_cache.c:251:upcall_cache_get_entry()) acquire for key 0: error -110</pre> </div></div> <p>Looks like the following hunk (except -ldl) in the original patch is not needed:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> diff --git a/lustre/utils/Makefile.am b/lustre/utils/Makefile.am index 56f67afd1d..bee65a1178 100644 --- a/lustre/utils/Makefile.am +++ b/lustre/utils/Makefile.am @@ -245,8 +245,9 @@ l_foreign_symlink_LDADD := $(top_builddir)/libcfs/libcfs/libcfs.la l_foreign_symlink_DEPENDENCIES := $(top_builddir)/libcfs/libcfs/libcfs.la l_getidentity_SOURCES = l_getidentity.c -l_getidentity_LDADD := $(top_builddir)/libcfs/libcfs/libcfs.la -l_getidentity_DEPENDENCIES := $(top_builddir)/libcfs/libcfs/libcfs.la +l_getidentity_LDADD := $(top_builddir)/libcfs/libcfs/libcfs.la liblustreapi.la $(LIBPTLCTL) +l_getidentity_LDFLAGS = -ldl +l_getidentity_DEPENDENCIES := $(top_builddir)/libcfs/libcfs/libcfs.la liblustreapi.la $(LIBPTLCTL) lhsmtool_posix_SOURCES = lhsmtool_posix.c pid_file.c pid_file.h lhsmtool_posix_LDADD := liblustreapi.la $(PTHREAD_LIBS) \ </pre> </div></div> <p>Moreover, it causes the shell wrapper lustre/utils/l_getidentity to fail silently when started by a kernel thread. The tool is not a binary file but a shell wrapper when I start Lustre from the source dir by llmount.sh:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>root@rocky lustre-release]# file lustre/utils/l_getidentity lustre/utils/l_getidentity: POSIX shell script, ASCII text executable, with very long lines [root@rocky lustre-release]# </pre> </div></div> <p>The failures cause lustre client mount delays.</p> <p>The reason for the shell wrapper is that allows running in the sandbox that people love to do. This way l_getindentity can find the proper libraries independent if its in source tree or installed into an image. If this application is important enough maybe we strip its need of any external libraries. If I remember correctly it only needs liblustreapi.so for mdt/%s/identity_info. Perhaps we can migrate to a udev rule that spins up l_getidentity instead. Another option is we use globs directly instead of cfs_get_param_paths(). BY the way why -ldl directly. Seems to want to work around the whole libtool handlng.</p> <p>Zam, your latest patch does work. Both on 7.9 and 8.4.</p> <p>&gt;The tool is not a binary file but a shell wrapper when I start Lustre from the source dir by llmount.sh:...</p> <p>The binaries are normally kept under lustre/utils/.libs/.</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># file lustre/utils/.libs/l_getidentity  lustre/utils/.libs/l_getidentity: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=b3cb5b448ee43365845f5d1fcb0a349bf91877c6, with debug_info, not stripped</pre> </div></div> <p>The targets are normally names lustre/utils/.libs/lt-* (example lfs and lctl) along with with real name</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ls lustre/utils/.libs/lt-* lustre/utils/.libs/lt-lctl  lustre/utils/.libs/lt-lfs   </pre> </div></div> <p>l_getidentity does not have lt-&lt;target&gt;. I am not sure if this is causing the issue.</p> <p> </p> <p> </p> <p>I intercepted the error stream output of l_getidentity being called by a Lustre server, and got something like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@rocky ~]# cat /tmp/identity.log PATH=/sbin:/usr/sbin 0 upcall start lustre-MDT0000 0 /mnt/nfs/git/lustre-release/lustre/utils/l_getidentity: line 150: ls: command not found /mnt/nfs/git/lustre-release/lustre/utils/l_getidentity: line 197: rm: command not found /mnt/nfs/git/lustre-release/lustre/utils/l_getidentity: line 211: rm: command not found /mnt/nfs/git/lustre-release/lustre/utils/l_getidentity: line 212: mv: command not found /mnt/nfs/git/lustre-release/lustre/utils/l_getidentity: line 213: rm: command not found upcall end lustre-MDT0000 0 0 [root@rocky ~]# </pre> </div></div> <p>I.e. the shell wrapper uses core linux utilities from /bin but PATH contains only "/sbin" and "/usr/sbin" , it is the reason for the libtool shell wrapper to function incorrectly. If I add "/bin:/usr/bin" to the PATH variable, the upcall (in its shell variant) starts to work.</p> <p>Thanks,<br/> Zam.</p> <p>I looked into an old vm and looks like lustre/utils/l_getidentity was always a binary and not a shell wrapper or launcher script.</p> <p>&gt;I looked into an old vm and looks like lustre/utils/l_getidentity was always a binary and not a shell wrapper or launcher script.</p> <p>yes, the patch converts it to a binary again.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53191/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53191/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17301" title="Client mount hangs for several minutes" class="issue-link" data-issue-key="LU-17301"><del>LU-17301</del></a> utils: l_getidentity build fix<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8fce6b40f245c438691800f6e239539a66e2395c</p> <p>Landed for 2.16</p> Related LU-16901 Development Rank 1|i0428n: Rank (Obsolete) 9223372036854775807 Severity