https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4498 Lustre <p>IU is running into an issue where running ls on a certain file causes clients to get evicted. It appears as if there is a hung MDT thread holding a lock on the file. After the MDT is rebooted, listing the directory and file works fine.</p> <p>We were able to capture client debug logs and a backtrace of all the threads from the running system, but due to an issue with STONITH, we were unable to get a good vmcore from the system. Also when we tried to get debug logs from the MDT, the log overflowed, even with a 20GB buffer. </p> <p>We are currently waiting for the issue to reappear and will get debug logs on a quiesced system, as well as a good vmcore. </p> <p>I'll upload the logs we have. Is there anything else we should be looking to get?</p> LU-4498

MDT thread hung, ls fails on directory

Bug Major Resolved Duplicate Zhenyu Xu Kit Westneat Thu, 16 Jan 2014 16:08:22 +0000 Fri, 21 Mar 2014 12:30:34 +0000 Fri, 21 Mar 2014 12:30:34 +0000 Lustre 2.1.6 0 8 <p>Here's an example of the directory listing. I forgot to mention it takes hours for the MDT to actually evict the client. </p> <p>ls -al<br/> /N/dc2/scratch/kmoriya/bggen_8.4_9.0-2013-12-11/0000/000100<br/> &#8212;<br/> Begin 2013-12-20_13:37:32<br/> total 164<br/> drwxr-xr-x 34 1012412 401 4096 Dec 11 20:59 .<br/> drwxr-xr-x 202 1012412 401 36864 Dec 16 09:32 ..<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:23 00003200<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003201<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003202<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003203<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003204<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003205<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003206<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003207<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003208<br/> ?--------- ? ? ? ? ? 00003209<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003210<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003211<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003212<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003213<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003214<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003215<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003216<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003217<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003218<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003219<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003220<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003221<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003222<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003223<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003224<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003225<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003226<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003227<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003228<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003229<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 13:24 00003230<br/> drwxr-xr-x 2 1012412 401 4096 Dec 12 07:42 00003231<br/> End 2013-12-20_15:44:49</p> <p>Is there anything else you can tell us about the 'certain file' ? <br/> Is it always the same file, or type of file?<br/> Have you run fsck on the MDT lately? </p> <p>It's happened three times on seemingly unrelated files. It looks like the last e2fsck was on June 12. FWIW there don't seem to be any ldisk errors in the logs anytime recently.</p> <p>Bobijam</p> <p>Could you please help with this one?</p> <p>Thanks</p> <p>Peter</p> <p>Fanyong,</p> <p>does it look like dir hash collision issue?</p> <p>According to the client side log, every getattr RPC is for different file, so it cannot to say that the "ls" fall into hash collision. On the other hand, the MDT side shows that during the "ls" processing, there are some "open-create" operations. What they are for?</p> <p>the client log shows that ls happened during 2014/01/09 23:46:11 to 2014/01/09 23:46:12, which took 1 seconds and does not match to the 1st comment report (from 2013-12-20_13:37:32 to 2013-12-20_15:44:49). Do you have logs which covers the issue time span?</p> <p>would you mind trying this patch <a href="http://review.whamcloud.com/8936" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8936</a> , it is a backport of dir hash collision fix patch.</p> <p>Bobijam</p> <p>Could this be related to <a href="https://jira.whamcloud.com/browse/LU-4616" title="ls or client eviction issues (ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 979@gni) returned -22 from blocking AST " class="issue-link" data-issue-key="LU-4616"><del>LU-4616</del></a>, the root cause of which has now been established?</p> <p>Peter</p> <p>yes, I think it's related to <a href="https://jira.whamcloud.com/browse/LU-4616" title="ls or client eviction issues (ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 979@gni) returned -22 from blocking AST " class="issue-link" data-issue-key="LU-4616"><del>LU-4616</del></a></p> <p>ok so then let's mark it as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-4616" title="ls or client eviction issues (ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 979@gni) returned -22 from blocking AST " class="issue-link" data-issue-key="LU-4616"><del>LU-4616</del></a> and reopen or open a new ticket if it manifests itself again now that <a href="https://jira.whamcloud.com/browse/LU-4616" title="ls or client eviction issues (ldlm_lockd.c:642:ldlm_handle_ast_error()) ### client (nid 979@gni) returned -22 from blocking AST " class="issue-link" data-issue-key="LU-4616"><del>LU-4616</del></a> seems to have been addressed.</p> Development Rank 1|hzwcyn: Rank (Obsolete) 12303 Severity