https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13980 Lustre <p>I removed some data stripes under '/O' folder on OST and started LFSCK. Then OST was forced to reboot because of kernel panic. By looking into vmcore, I found the specific error line is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 1057.367833] Lustre: lustre-OST0000: new disk, initializing [ 1057.367877] Lustre: srv-lustre-OST0000: No data found on store. Initialize space [ 1057.417121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 300-900 [ 1062.018722] Lustre: lustre-OST0000: Connection restored to lustre-MDT0000-mdtlov_UUID (at 10.0.0.122@tcp) [ 1089.010284] Lustre: lustre-OST0000: Connection restored to 89c68bff-12c8-9f48-f01e-f6306c666eb9 (at 10.0.0.98@tcp) [ 1281.516928] LustreError: 10410:0:(osd_handler.c:1982:osd_object_release()) LBUG [ 1281.516939] Pid: 10410, comm: ll_ost_out00_00 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Mon May 27 03:45:37 UTC 2019 [ 1281.516944] Call Trace: [ 1281.516960] [&lt;ffffffffc05fd7cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [ 1281.516986] [&lt;ffffffffc05fd87c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [ 1281.517004] [&lt;ffffffffc0b93820&gt;] osd_get_ldiskfs_dirent_param+0x0/0x130 [osd_ldiskfs] [ 1281.517173] [&lt;ffffffffc07442b0&gt;] lu_object_put+0x190/0x3e0 [obdclass] [ 1281.517244] [&lt;ffffffffc09d8bc3&gt;] out_handle+0x1503/0x1bc0 [ptlrpc] [ 1281.517369] [&lt;ffffffffc09ce7ca&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] [ 1281.517481] [&lt;ffffffffc097705b&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [ 1281.517582] [&lt;ffffffffc097a7a2&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] </pre> </div></div> <p>(The full dmesg log collected in vmcore is in the attachment)</p> <p>Then I found after removing files under '/O' on OST, even a simple write operation can result in the same kernel panic.</p> <p>I'm just curious about why 'osd_object_release' is evoked in above situations and the position of LFSCK functions in the error call trace.</p> <p>Thanks a lot! </p> CentOS Linux release 7.7.1908 (Core) with kernel 3.10.0-957.1.3.el7_lustre.x86_64 for Lustre 2.10.8 and CentOS Linux release 7.7.1908 (Core) with 3.10.0-1062.9.1.el7_lustre.x86_64 for Lustre 2.12.4. LU-13980

Kernel panic on OST after removing files under '/O' folder

Task Trivial Open Unresolved Andreas Dilger Runzhou Han Wed, 23 Sep 2020 02:40:43 +0000 Wed, 19 May 2021 17:35:47 +0000 Lustre 2.10.8 Lustre 2.12.4 0 5 <p>Can you please provide some more information about this problem.</p> <p>How did you remove the objects under the <tt>/O</tt> directory? Was the filesystem mounted as both type "<tt>lustre</tt>" and type "<tt>ldiskfs</tt>" at the same time, or was the "<tt>lustre</tt>" OST unmounted first?</p> <p>Which files were removed? The log messages makes it appear that the filesystem was sufficiently corrupted that the OST startup process wasn't able to detect the Lustre configuration files.</p> <p>If you are able to reproduce this, please enable full debugging with "<tt>lctl set_param debug=-1</tt>" on the OST before starting LFSCK, and then attach the debug log which should be written to <tt>/tmp/lustre_log.&lt;timestamp&gt;</tt> when the LBUG is triggered, or can be dumped manually like "<tt>lctl dk /tmp/lustre_log.txt</tt>".</p> <p>Thank you for you reply.  </p> <p>I mounted as both type "<tt>lustre</tt>" and type "<tt>ldiskfs</tt>" at the same time. </p> <p>The file removed is a data stripe of a client file. In my configuration my stripe setting is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs setstripe -i 0 -c -1 -S 64K /lustre </pre> </div></div> <p>For example, on client node I create a file with the following command:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>dd if=/dev/zero of=/lustre/10M bs=1M count=10 </pre> </div></div> <p>Then I use the "<tt>lfs getstripe /lustre/10M</tt>" to locate its data stripes on OSTs,</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@mds Desktop]# lfs getstripe 10M 10M lmm_stripe_count: 3 lmm_stripe_size: 65536 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2 0x2 0 1 2 0x2 0 2 2 0x2 0 </pre> </div></div> <p>Next I remove one of them under one OST "<tt>ldiskfs</tt>" mount point.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@oss0 osboxes]# rm -f /ost0_ldiskfs/O/0/d2/2 </pre> </div></div> <p>Then running LFSCK on MDT will trigger kernel panic caused by LBUG.</p> <p>I'm able to reproduce the LBUG. However, after kernel panic takes place, I'm not able to manipulate the VM any more (I was using a virtual machine cluster). The VM would either freezes or I configure the kernel to reboot in x seconds after kernel panic. In the next boot I'm not able to find <tt>/tmp/lustre_log.&lt;timestamp&gt;</tt>.</p> <p> </p> <p>Mounting the OST filesystem as both "<tt>lustre</tt>" and "<tt>ldiskfs</tt>" at the same time is not supported, since (as you can see with this assertion) the state is being changed from underneath the filesystem in an unexpected manner. It would be the same as if modifying the blocks underneath ext4 while it is mounted.</p> <p>I mistakenly thought that the "<tt>lustre-OST0000: new disk, initializing</tt>" was caused by a large number of files being deleted from the filesystem before startup, but I now can see from the low OST object numbers in your "<tt>lfs getstripe</tt>" output that this <b>is</b> a new filesystem, so this message is expected.</p> <p>I agree that it would be good to handle this error more gracefully (e.g. return an error instead of LBUG). Looking elsewhere in Jira, it seems that this LBUG is hit often enough that the error handling should really be more tolerant, since the design policy is that the server should not <tt>LASSERT()</tt> on bad values that come from the client or disk.</p> <p>I see. Maybe I should not mount them at the same time. </p> <p>Actually, I was trying to emulate some special cases in which the underneath file system is corrupted by accident while the system is still running. I want to see how Lustre reacts to these unexpected failures (especially LFSCK's reaction). </p> <p>In fact, to help develop more robust fsck for PFS/DFS, I'm also doing the same thing to other systems (e.g., BeeGFS, OrangeFS and Ceph). Since Lustre heavily relies on kernel modules, I observed more kernel crashes in Lustre when injecting faults. That's why I'm here to learn more about Lustre <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>. If possible, I'm willing to help with moderate solutions to unexpected crashes.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40058" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40058</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13980" title="Kernel panic on OST after removing files under &#39;/O&#39; folder" class="issue-link" data-issue-key="LU-13980">LU-13980</a> osd: remove osd_object_release LASSERT<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 50d7b40a457a733130f346515ab37ad3e1b54424</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40738" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40738</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13980" title="Kernel panic on OST after removing files under &#39;/O&#39; folder" class="issue-link" data-issue-key="LU-13980">LU-13980</a> osd-ldiskfs: print label instead of device<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1e1137a66c634288e40f3a2a017ce6d5c003fa2d</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40738/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40738/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13980" title="Kernel panic on OST after removing files under &#39;/O&#39; folder" class="issue-link" data-issue-key="LU-13980">LU-13980</a> osd-ldiskfs: print label instead of device<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8f793f14bf9928352623e61122f005252605b136</p> Related LU-10581 LU-8992 LU-12357 LU-12741 LU-12485 LU-13000 Development Rank 1|i01ajr: Rank (Obsolete) 9223372036854775807