https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11737 Lustre <p>This is related to <a href="https://jira.whamcloud.com/browse/LU-11584" title="kernel BUG at ldiskfs.h:1907!" class="issue-link" data-issue-key="LU-11584"><del>LU-11584</del></a>.</p> <ol> <li>umounted the targets</li> <li>mount mdt as ldiskfs</li> <li>deleted all the quarantined files</li> <li>reboot and remount targets</li> </ol> <p>Hit LBUG.</p> <p>[ 1781.324368] LustreError: 11060:0:(osd_handler.c:3985:osd_xattr_set()) ASSERTION( handle ) failed: <br/> [ 1781.351312] LustreError: 11060:0:(osd_handler.c:3985:osd_xattr_set()) LBUG</p> <p> </p> <p>I tried to mount with noscurb still hitting the lbug.</p> <p> </p> <p>Filesystem keeps crashing with lbug.</p> <p> </p> LU-11737

LustreError: 11060:0:(osd_handler.c:3985:osd_xattr_set()) ASSERTION( handle ) failed:

Bug Major Resolved Fixed Alex Zhuravlev Mahmoud Hanafi Thu, 6 Dec 2018 18:35:13 +0000 Tue, 19 Mar 2019 15:00:13 +0000 Thu, 10 Jan 2019 11:45:50 +0000 Lustre 2.10.5 Lustre 2.13.0 Lustre 2.10.7 Lustre 2.12.1 0 6 <p>please, show the full stack trace.</p> <p>I was running lfsck in dryrun</p> <p>lctl lfsc_start -r -n -o -t layout</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> PID: 11060 TASK: ffff882f2ed10fd0 CPU: 5 COMMAND: <span class="code-quote">"lfsck_layout"</span> #0 [ffff882bcca3f868] machine_kexec at ffffffff8105b64b #1 [ffff882bcca3f8c8] __crash_kexec at ffffffff81105342 #2 [ffff882bcca3f998] panic at ffffffff81689aad #3 [ffff882bcca3fa18] lbug_with_loc at ffffffffa06e08cb [libcfs] #4 [ffff882bcca3fa38] osd_xattr_set at ffffffffa10dfcdf [osd_ldiskfs] #5 [ffff882bcca3fab8] lfsck_layout_refill_lovea at ffffffffa11de3c2 [lfsck] #6 [ffff882bcca3fb40] lfsck_layout_recreate_lovea at ffffffffa11f20db [lfsck] #7 [ffff882bcca3fc88] lfsck_layout_assistant_handler_p2 at ffffffffa11f4ada [lfsck] #8 [ffff882bcca3fd80] lfsck_assistant_engine at ffffffffa11b77ce [lfsck] #9 [ffff882bcca3fec8] kthread at ffffffff810b1131 #10 [ffff882bcca3ff50] ret_from_fork at ffffffff816a14dd </pre> </div></div> <p> </p> <p>ok, thanks. working on this..</p> <p>please, try to mount MDS with skip_lfsck mount option.</p> <p>Do you want me to try re-running lfsck after remounting?</p> <p>my understanding is that we want to get the filesystem back first? in the mean time I'll try to get a fix for this issue.<br/> it seems a part of the code ignores read-only option.</p> <p>ok will do. </p> <p>some additional info. Not sure if the delete record is relevant.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.906822] Lustre: 16159:0:(lfsck_layout.c:1618:lfsck_layout_del_dangling_rec()) nbp13-MDT0000-osd: delete the dangling record <span class="code-keyword">for</span> [0x20000205c:0x15dc1:0x0], comp_id = 4, ea_off = 15 from the trace file: rc = -2 Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.906827] LustreError: 16159:0:(osd_handler.c:3985:osd_xattr_set()) ASSERTION( handle ) failed: Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.933775] LustreError: 16159:0:(osd_handler.c:3985:osd_xattr_set()) LBUG Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.954442] Pid: 16159, comm: lfsck_layout 3.10.0-693.21.1.el7.20180508.x86_64.lustre2105 #1 SMP Mon Aug 27 23:04:41 UTC 2018 Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.954442] Call Trace: Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.954458] [&lt;ffffffff8103a1f2&gt;] save_stack_trace_tsk+0x22/0x40 Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.954467] [&lt;ffffffffa06d17cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.954470] [&lt;ffffffffa06d187c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.954479] [&lt;ffffffffa110acdf&gt;] osd_xattr_set+0x95f/0xc10 [osd_ldiskfs] Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.954492] [&lt;ffffffffa12093c2&gt;] lfsck_layout_refill_lovea.isra.63+0x1c2/0x480 [lfsck] Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.954499] [&lt;ffffffffa121d0db&gt;] lfsck_layout_recreate_lovea+0xd8b/0x2370 [lfsck] Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.954505] [&lt;ffffffffa121fada&gt;] lfsck_layout_assistant_handler_p2+0x141a/0x1650 [lfsck] Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.954509] [&lt;ffffffffa11e27ce&gt;] lfsck_assistant_engine+0xfce/0x20b0 [lfsck] Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.954511] [&lt;ffffffff810b1131&gt;] kthread+0xd1/0xe0 Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.954513] [&lt;ffffffff816a14dd&gt;] ret_from_fork+0x5d/0xb0 Dec 6 12:09:17 nbp13-srv1 kernel: [ 5780.954528] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> <p>this is definitely useful, thanks.</p> <p>Here is the object info</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Inode: 1762289 Type: regular Mode: 07666 Flags: 0x80000 Generation: 2205332265 Version: 0x00000000:00000000 User: 30757 Group: 41548 Project: 0 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 <span class="code-object">Number</span>: 0 Size: 0 ctime: 0x5bd00854:00000000 -- Tue Oct 23 22:51:16 2018 atime: 0x00000000:00000000 -- Wed Dec 31 16:00:00 1969 mtime: 0x00000000:00000000 -- Wed Dec 31 16:00:00 1969 crtime: 0x5bd00837:b8168050 -- Tue Oct 23 22:50:47 2018 Size of extra inode fields: 32 Extended attributes: trusted.lma (24) = 08 00 00 00 00 00 00 00 00 00 14 00 01 00 00 00 60 1b 21 00 00 00 00 00 lma: fid=[0x100140000:0x211b60:0x0] compat=8 incompat=0 trusted.fid (44) fid: parent=[0x20000205c:0x15dc1:0x0] stripe=0 stripe_size=1048576 stripe_count=8 component_id=3 component_start=17179869184 component_end=68719476736 EXTENTS: </pre> </div></div> <p>\I scanned the MDT this 0x20000205c:0x15dc1:0x0 doesn't exist.</p> <p>We can lower the prio down to 2. </p> <p>Found 0x20000205c:0x15dc1:0x0<br/> it was in lost+found<br/> .lustre/lost+found/MDT0000/'<span class="error">&#91;0x20000205c:0x15dc1:0x0&#93;</span>-R-0'</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> tpfe2 /nobackupp13/.lustre/lost+found/MDT0000 # stat <span class="code-quote">'[0x20000205c:0x15dc1:0x0]-R-0'</span> File: <span class="code-quote">'[0x20000205c:0x15dc1:0x0]-R-0'</span> Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: d7d58d2eh/3621096750d Inode: 144115327058402753 Links: 1 Access: (0400/-r--------) Uid: (30757/ spocops) Gid: (41548/ s1548) Access: 2018-12-06 12:09:17.000000000 -0800 Modify: 2018-12-06 12:09:17.000000000 -0800 Change: 2018-12-06 12:09:17.000000000 -0800 Birth: - </pre> </div></div> <p>Mahmoud, I also filed <a href="https://jira.whamcloud.com/browse/LUDOC-421" title="document LFSCK filename types in lost+found" class="issue-link" data-issue-key="LUDOC-421">LUDOC-421</a> to add documentation for the files in <tt>.lustre/lost+found</tt>. In this case, the "<tt>&#45;R&#45;</tt>" means <em>"The orphan OST-object knows its parent MDT-object FID, but does not know the position (the file name) in the layout."</em></p> <p>Just to confirm that the file is what we expect, you could do "<tt>lfs getstripe -F /nobackupp13/.lustre/lost+found/MDT0000/<span class="error">&#91;0x20000205c:0x15dc1:0x0&#93;</span>-R-0</tt>" to get the FID from that file (it should probably be <tt><span class="error">&#91;0x20000205c:0x15dc1:0x0&#93;</span></tt>), and given this object has no data in it (size=0 and blocks=0, and no "filename" exists for it, you can probably just delete it.</p> <p>These are probably the objects of the corrupted files that I deleted. </p> <p>Any update on getting a patch to deal with the LBUG?</p> <p>I found one possible cause.. trying to reproduce locally.</p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33826" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33826</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11737" title="LustreError: 11060:0:(osd_handler.c:3985:osd_xattr_set()) ASSERTION( handle ) failed: " class="issue-link" data-issue-key="LU-11737"><del>LU-11737</del></a> lfsck: do not ignore dryrun<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e7a7d9836e8b3d51b917e84e425195130db7d2c5</p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33827" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33827</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11737" title="LustreError: 11060:0:(osd_handler.c:3985:osd_xattr_set()) ASSERTION( handle ) failed: " class="issue-link" data-issue-key="LU-11737"><del>LU-11737</del></a> lfsck: do not ignore dryrun<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 5ed85a4a03eb39f04d7b7379acc8b52f046eeb28</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33826/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33826/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11737" title="LustreError: 11060:0:(osd_handler.c:3985:osd_xattr_set()) ASSERTION( handle ) failed: " class="issue-link" data-issue-key="LU-11737"><del>LU-11737</del></a> lfsck: do not ignore dryrun<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 875f3fc03aa15049892fe19d6a4fc1132848fced</p> <p>Landed for 2.13</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33827/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33827/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11737" title="LustreError: 11060:0:(osd_handler.c:3985:osd_xattr_set()) ASSERTION( handle ) failed: " class="issue-link" data-issue-key="LU-11737"><del>LU-11737</del></a> lfsck: do not ignore dryrun<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 231649a4e93e5a630af8c0b5715da51a92cfb679</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34312" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34312</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11737" title="LustreError: 11060:0:(osd_handler.c:3985:osd_xattr_set()) ASSERTION( handle ) failed: " class="issue-link" data-issue-key="LU-11737"><del>LU-11737</del></a> lfsck: do not ignore dryrun<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: cf75e5bfd2661e3162355637e5ca1b2100b3a0ea</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34312/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34312/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11737" title="LustreError: 11060:0:(osd_handler.c:3985:osd_xattr_set()) ASSERTION( handle ) failed: " class="issue-link" data-issue-key="LU-11737"><del>LU-11737</del></a> lfsck: do not ignore dryrun<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 0b012bcab5426e9e1fd9f3570dd7704c8129d032</p> Related LU-11584 Development Rank 1|i007hr: Rank (Obsolete) 9223372036854775807 Severity