https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5514 Lustre <p>We have upgraded from 1.8.7 to 2.5.2</p> <p>After mounting the FS at approximately 17:29</p> <p>At 18:49 we see the following</p> <p>Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 15310 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:<br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: Pid: 15310, comm: mdt01_000<br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: Call Trace:<br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa057ffd5&gt;&#93;</span> ? cfs_hash_bd_lookup_intent+0x65/0x130 <span class="error">&#91;libcfs&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa06bdeab&gt;&#93;</span> lu_object_find_at+0xab/0x360 <span class="error">&#91;obdclass&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffff81061d00&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffff81282965&gt;&#93;</span> ? _atomic_dec_and_lock+0x55/0x80<br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa06be19f&gt;&#93;</span> lu_object_find_slice+0x1f/0x80 <span class="error">&#91;obdclass&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa0f99370&gt;&#93;</span> mdd_object_find+0x10/0x70 <span class="error">&#91;mdd&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa0fa97ba&gt;&#93;</span> mdd_is_parent+0xaa/0x3a0 <span class="error">&#91;mdd&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa0fa9c0c&gt;&#93;</span> mdd_is_subdir+0x15c/0x200 <span class="error">&#91;mdd&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa0e6d509&gt;&#93;</span> mdt_reint_rename+0x1049/0x1c20 <span class="error">&#91;mdt&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa0e48940&gt;&#93;</span> ? mdt_blocking_ast+0x0/0x2a0 <span class="error">&#91;mdt&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa0819f60&gt;&#93;</span> ? ldlm_completion_ast+0x0/0x920 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa0586f76&gt;&#93;</span> ? upcall_cache_get_entry+0x296/0x880 <span class="error">&#91;libcfs&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa06ddc00&gt;&#93;</span> ? lu_ucred+0x20/0x30 <span class="error">&#91;obdclass&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa0e684a1&gt;&#93;</span> mdt_reint_rec+0x41/0xe0 <span class="error">&#91;mdt&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa0e4dcb3&gt;&#93;</span> mdt_reint_internal+0x4c3/0x780 <span class="error">&#91;mdt&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa0e4dfb4&gt;&#93;</span> mdt_reint+0x44/0xe0 <span class="error">&#91;mdt&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa0e5158a&gt;&#93;</span> mdt_handle_common+0x52a/0x1470 <span class="error">&#91;mdt&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa0e8d755&gt;&#93;</span> mds_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa0853bc5&gt;&#93;</span> ptlrpc_server_handle_request+0x385/0xc00 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa056b4ce&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa057c3cf&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x170 <span class="error">&#91;libcfs&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa084b2a9&gt;&#93;</span> ? ptlrpc_wait_event+0xa9/0x2d0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffff810546b9&gt;&#93;</span> ? __wake_up_common+0x59/0x90<br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa0854f2d&gt;&#93;</span> ptlrpc_main+0xaed/0x1740 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffffa0854440&gt;&#93;</span> ? ptlrpc_main+0x0/0x1740 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffff8109ab56&gt;&#93;</span> kthread+0x96/0xa0<br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> child_rip+0xa/0x20<br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffff8109aac0&gt;&#93;</span> ? kthread+0x0/0xa0<br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: <br/> Aug 19 18:49:09 cs04r-sc-mds03-01 kernel: LustreError: dumping log to /tmp/lustre-log.1408470549.15310<br/> Aug 19 18:51:19 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 15470 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:<br/> Aug 19 18:51:19 cs04r-sc-mds03-01 kernel: Pid: 15470, comm: mdt03_004</p> LU-5514

After upgrade from 1.8.7 to 2.5.2 stack trace cfs_hash_bd_lookup_intent

Bug Major Resolved Duplicate Lai Siyao Dave Bond Wed, 20 Aug 2014 08:21:22 +0000 Wed, 13 Oct 2021 03:15:04 +0000 Wed, 13 Oct 2021 03:15:04 +0000 0 5 <p>When unmounting on MDS03-01 so I can mount on MDS03-02 I see the following on MDS03-01</p> <p>Message from syslogd@cs04r-sc-mds03-01 at Aug 20 09:41:30 ...<br/> kernel:LustreError: 15370:0:(osd_handler.c:2406:osd_object_destroy()) ASSERTION( !lu_object_is_dying(dt-&gt;do_lu.lo_header) ) failed: </p> <p>Message from syslogd@cs04r-sc-mds03-01 at Aug 20 09:41:30 ...<br/> kernel:LustreError: 15370:0:(osd_handler.c:2406:osd_object_destroy()) LBUG</p> <p>I won't elaborate too much further at the moment as I do not want to confuse the issue. </p> <p>Lai is looking into this ticket</p> <p>I don't see anything particular in the log. And there is a patch for a similar issue: <a href="http://review.whamcloud.com/#/c/10816/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10816/</a>, however the patch looks not included in 2.5.2, could you verify this?</p> <p>Besides, <a href="http://review.whamcloud.com/#/c/10816/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10816/</a> may not be able to fix this issue because that's in getattr context, but this in rename. There may be a similar error in rename.</p> <p>Lai,</p> <p>I'm not sure I fully understand your comment, did you want us (Diamond) to verify if that patch is included in 2.5.2? </p> <p>Also, after looking at the patch and issues referenced in the patch, I'm still not clear what might have happened? I assume it needs special circumstances during a rename/getattr to trigger this. Are there any more details know? Any hints where we should start looking for a reproducer? </p> <p>FYI, the file system is not yet seeing normal use as we're currently running tests before letting the users back on...</p> <p>Cheers,<br/> Frederik</p> <p>Frederik</p> <p>Lai is suggesting that you should apply the patch - which is missing from 2.5.2 but will be in the upcoming 2.5.3 release.</p> <p>Regards</p> <p>Peter</p> <p>racer.sh test script is the reproducer for <a href="http://review.whamcloud.com/#/c/10816/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10816/</a>, which you may use as well.</p> <p>This turns out to be a duplicate of <a href="https://jira.whamcloud.com/browse/LU-4725" title="wrong lock ordering in rename leads to deadlocks" class="issue-link" data-issue-key="LU-4725"><del>LU-4725</del></a>, and two patches are in review phase: <a href="http://review.whamcloud.com/#/c/10916/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10916/</a> <a href="http://review.whamcloud.com/#/c/10917/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10917/</a>.</p> <p>This is the same issue as <a href="http://review.whamcloud.com/#/c/10816/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10816/</a>, but this happens in rename, while that's in getattr. It's a race in lookup the same object twice while object is destroyed in the mean time. I'd suggest you apply these three patches to fully solve such issue.</p> <p>Peter, Lai,</p> <p>thanks for clarifying this.</p> <p>We'll look into compiling lustre 2.5.2 + those 3 patches, but if there is an easier way for us to get the correct compiled versions, let us know. </p> <p>I assume, initially it should be sufficient to run this patches version on the MDS, correct?</p> <p>Kind regards,<br/> Frederik</p> <p>Yes, it's sufficient with these 3 patches. Once those two patches for <a href="https://jira.whamcloud.com/browse/LU-4725" title="wrong lock ordering in rename leads to deadlocks" class="issue-link" data-issue-key="LU-4725"><del>LU-4725</del></a> are landed on 2.5.3, you can use 2.5.3 directly, but currently you have to apply these patches yourself.</p> <p>Frederik</p> <p>Is rename commonly used during regular production usage or is this an issue that is more likely to hit in a stressing test run?</p> <p>Peter</p> <p>There's a number of processes/applications that tend to create files with a temporary extension and than rename them to the proper name, to avoid any race conditions where processing starts before the data is there, so not that uncommon. And considering that we first saw it with basically only ior test runs, I'd say, something is doing it quite frequently...</p> <p>Cheers,<br/> Frederik</p> <p>ok fair enough then. I just did not want you to move forward with patches for something that was a rare edge case</p> Related LU-4725 Development End date Thu, 21 Aug 2014 08:21:22 +0000 Epic/Theme Lustre-2.5.2 Rank 1|hzwu4f: Rank (Obsolete) 15371 Severity Start date Wed, 20 Aug 2014 08:21:22 +0000