https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13189 Lustre <p>Seeing a crash fairly frequently on one of our oss</p> <p><span class="error">&#91;Feb 1 19:45&#93;</span> Lustre: work2-OST0002: Recovery over after 0:56, of 20 clients 20 recovered and 0 were evicted.<br/> [ +0.000279] Lustre: work2-OST0002: deleting orphan objects from 0x0:268076412 to 0x0:268081537<br/> [ +0.198454] LustreError: 14123:0:(osd_object.c:1345:osd_attr_set()) ASSERTION( obj-&gt;oo_with_projid ) failed: <br/> [ +0.000046] LustreError: 14123:0:(osd_object.c:1345:osd_attr_set()) LBUG<br/> [ +0.000064] Pid: 14123, comm: ll_ost_io01_013 3.10.0-1062.9.1.el7.x86_64 #1 SMP Mon Dec 2 08:31:54 EST 2019<br/> [ +0.000035] Call Trace:<br/> [ +0.000018] <span class="error">&#91;&lt;ffffffffc10e87cc&gt;&#93;</span> libcfs_call_trace+0x8c/0xc0 <span class="error">&#91;libcfs&#93;</span><br/> [ +0.001388] <span class="error">&#91;&lt;ffffffffc10e887c&gt;&#93;</span> lbug_with_loc+0x4c/0xa0 <span class="error">&#91;libcfs&#93;</span><br/> [ +0.001275] <span class="error">&#91;&lt;ffffffffc179b458&gt;&#93;</span> osd_attr_set+0xdd8/0xe50 <span class="error">&#91;osd_zfs&#93;</span></p> <p>Message from syslogd@rit-ost1.las.iastate.edu at Feb 1 19:45:04 ...<br/> kernel:LustreError: 14123:0:(osd_object.c:1345:osd_attr_set()) ASSERTION( obj-&gt;oo_with_projid ) failed: <br/> [ +0.001272] <span class="error">&#91;&lt;ffffffffc190e622&gt;&#93;</span> ofd_commitrw_write+0x13c2/0x1d40 <span class="error">&#91;ofd&#93;</span><br/> [ +0.001274] <span class="error">&#91;&lt;ffffffffc191212c&gt;&#93;</span> ofd_commitrw+0x48c/0x9e0 <span class="error">&#91;ofd&#93;</span><br/> [ +0.001255] <span class="error">&#91;&lt;ffffffffc15ad0fa&gt;&#93;</span> tgt_brw_write+0x10ba/0x1ce0 <span class="error">&#91;ptlrpc&#93;</span><br/> [ +0.001586] <span class="error">&#91;&lt;ffffffffc15ab2ea&gt;&#93;</span> tgt_request_handle+0xaea/0x1580 <span class="error">&#91;ptlrpc&#93;</span><br/> [ +0.001574] <span class="error">&#91;&lt;ffffffffc155029b&gt;&#93;</span> ptlrpc_server_handle_request+0x24b/0xab0 <span class="error">&#91;ptlrpc&#93;</span><br/> [ +0.001555] <span class="error">&#91;&lt;ffffffffc1553bfc&gt;&#93;</span> ptlrpc_main+0xb2c/0x1460 <span class="error">&#91;ptlrpc&#93;</span><br/> [ +0.001560] <span class="error">&#91;&lt;ffffffffb28c61f1&gt;&#93;</span> kthread+0xd1/0xe0<br/> [ +0.001500] <span class="error">&#91;&lt;ffffffffb2f8dd1d&gt;&#93;</span> ret_from_fork_nospec_begin+0x7/0x21<br/> [ +0.001487] <span class="error">&#91;&lt;ffffffffffffffff&gt;&#93;</span> 0xffffffffffffffff<br/> [ +0.001498] Kernel panic - not syncing: LBUG</p> <p> </p> <p>Not sure what exactly is causing it. Stack trace is from after the server reboots, as soon as recovery finishes and io starts again it happens. Originally I thought it was related to the recovery process and that aborting recovery would work around it, but it still occurs. I'm not really sure if it's a particular file or an io pattern that's leading to it, and I've not been able to narrow it down to a specific job in our environment.</p> <p> </p> <p> </p> rhel 7.7 zfs-0.8.2 kernel 3.10.0-1062.9.1.el7.x86_64 LU-13189

ASSERTION( obj->oo_with_projid ) failed with 2.12.3

Bug Major Resolved Fixed Dongyang Li Shane Nehring Sun, 2 Feb 2020 02:04:06 +0000 Wed, 5 Apr 2023 20:08:11 +0000 Mon, 11 Jul 2022 12:59:39 +0000 Lustre 2.12.3 Lustre 2.14.0 Lustre 2.15.0 Lustre 2.16.0 Lustre 2.15.1 1 9 <p>I ended up undefining ZFS_PROJINHERIT and recompiling so I could get the oss to stay up. It doesn't look like this code was touched in 2.12.4 (I had tried RC1 when I was running into this issue and it still occurred).</p> <p>Please let me know if you need any more information</p> <p>Had this start showing up on another oss/ost. Implemented the same work around there.</p> <p>Shane, we are running into this same issue with our lustre file system - we are running 2.14.  At what level did you do the #undef ZFS_PROJINHERIT?  Just in that source file or for the whole Lustre build?</p> <p>What I've done is add #undef ZFS_PROJINHERIT to lustre/osd-zfs/osd_internal.h</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/lustre/osd-zfs/osd_internal.h b/lustre/osd-zfs/osd_internal.h index ae21447..58ef131 100644 --- a/lustre/osd-zfs/osd_internal.h +++ b/lustre/osd-zfs/osd_internal.h @@ -55,6 +55,7 @@ #include &lt;sys/dbuf.h&gt; #include &lt;sys/dmu_objset.h&gt; #include &lt;lustre_scrub.h&gt; +#undef ZFS_PROJINHERIT /** * By design including kmem.h overrides the Linux slab interfaces to provide </pre> </div></div> <p> </p> <p>Which keeps things up. It will at the very least make project quotas non functional, I believe.</p> <p>We also ran into the same issue with 2.14:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> kernel:LustreError: 24469:0:(osd_object.c:1353:osd_attr_set()) ASSERTION( obj-&gt;oo_with_projid ) failed: kernel:LustreError: 24469:0:(osd_object.c:1353:osd_attr_set()) LBUG </pre> </div></div> <p>Is there any known workaround without recompiling?</p> <p>We never found another workaround besides recompiling.  I tried reaching out on the mailing list but didn't get any responses.  </p> <p> </p> <p><a href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2021-September/017791.html" class="external-link" target="_blank" rel="nofollow noopener">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2021-September/017791.html</a></p> <p><a href="http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2021-October/017794.html" class="external-link" target="_blank" rel="nofollow noopener">http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2021-October/017794.html</a></p> <p> </p> <p> </p> <p>Had either of you ever enabled project quotas?</p> <p>We migrated recently from 2.12.8 to 2.14.0. On this occasion we activated project quotas but have not actually used them yet. My currently solution is:</p> <ul> <li>project quotas disabled</li> <li>all clients evicted</li> </ul> <p>That broke the cycle of reboot, recovery, and kernel panic. During the last few hours everything worked fine. But we actually would like to use project quotas in future.</p> <p>Do you have the kernel patches for lustre applied on the oss?</p> <p>I believe, at least in my case, that this is the result of enabling project quotas with kernel version &lt; 4.5 without the lustre kernel patches.</p> <p>We are using the ZFS backend for MDT and OST. All servers are installed with the packages from the official repository. Versions:</p> <ul> <li>Kernel: 4.18.0-240.1.1.el8_lustre.x86_64</li> <li>Lustre: 2.14.0-1.el8 (kmod)</li> <li>ZFS: 2.0.0-1.el8 (kmod)</li> </ul> <p>On a second identical system project quotas are already in use since a few weeks without any problems.</p> <p>No, we are not patching the kernel.  I realized after the fact that project quota won't work without the patched kernel.  I'm still a little concerned as to why this would panic the OSS.  I would like to know how to clear the project ID's from our OSS so we could go back to the unmodified lustre source.  </p> <p>We are also using ZFS for our MDT and OST's.  Our servers are CentOS 7.9 with kernel 3.10.0-1160.31.1.el7.x86_64, lustre-2.14.0_1.el7, zfs 2.0.5.  </p> <p>Hmm, that you're seeing this on a rhel 8 kernel kinda shoots that idea down. Unless you're hitting it by some other means.</p> <p> </p> <p>I've recently explicitly disabled project quotas, I'll be doing an upgrade to 2.12.8 on Monday where I plan to not include my workaround to see if that resolves it. It's difficult for us to tell though, as it can take a day or two for someone to start hitting a file that has this problem.</p> <p>I actually just thought to look at the configuration logs for the osts, it doesn't look like I ever actually enabled project quotas. So this may be something waiting in the wings that an FS can hit regardless of whether those were ever enabled or not.</p> <p>Two new observations:</p> <ul> <li>project quotas work fine on the same server for a different OST.</li> <li>disabling project quotas does not help. It worked fine for the system with disabled project quotas for the last five days. But today again the same issue.</li> </ul> <p>The problem unfortunately persists with Lustre 2.15.0-RC5 and ZFS 2.0.7:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Jun 05 17:03:04 z-ha-oss02b kernel: LustreError: 809221:0:(osd_object.c:1300:osd_attr_set()) ASSERTION( obj-&gt;oo_with_projid ) failed: Jun 05 17:03:04 z-ha-oss02b kernel: LustreError: 809016:0:(osd_object.c:1300:osd_attr_set()) ASSERTION( obj-&gt;oo_with_projid ) failed: Jun 05 17:03:04 z-ha-oss02b kernel: LustreError: 808404:0:(osd_object.c:1300:osd_attr_set()) ASSERTION( obj-&gt;oo_with_projid ) failed: Jun 05 17:03:04 z-ha-oss02b kernel: LustreError: 808404:0:(osd_object.c:1300:osd_attr_set()) LBUG Jun 05 17:03:04 z-ha-oss02b kernel: Pid: 808404, comm: ll_ost02_001 4.18.0-372.9.1.el8.x86_64 #1 SMP Tue May 10 08:57:35 EDT 2022 Jun 05 17:03:04 z-ha-oss02b kernel: Call Trace TBD: Jun 05 17:03:04 z-ha-oss02b kernel: [&lt;0&gt;] libcfs_call_trace+0x6f/0x90 [libcfs] Jun 05 17:03:04 z-ha-oss02b kernel: [&lt;0&gt;] lbug_with_loc+0x3f/0x70 [libcfs] Jun 05 17:03:04 z-ha-oss02b kernel: [&lt;0&gt;] osd_attr_set+0xe3f/0xed0 [osd_zfs] Jun 05 17:03:04 z-ha-oss02b kernel: [&lt;0&gt;] ofd_attr_set+0x638/0x1080 [ofd] Jun 05 17:03:04 z-ha-oss02b kernel: [&lt;0&gt;] ofd_setattr_hdl+0x454/0x8d0 [ofd] Jun 05 17:03:04 z-ha-oss02b kernel: [&lt;0&gt;] tgt_request_handle+0xc93/0x1a40 [ptlrpc] Jun 05 17:03:04 z-ha-oss02b kernel: [&lt;0&gt;] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc] Jun 05 17:03:04 z-ha-oss02b kernel: [&lt;0&gt;] ptlrpc_main+0xc06/0x1560 [ptlrpc] Jun 05 17:03:04 z-ha-oss02b kernel: [&lt;0&gt;] kthread+0x10a/0x120 Jun 05 17:03:04 z-ha-oss02b kernel: [&lt;0&gt;] ret_from_fork+0x35/0x40 Jun 05 17:03:04 z-ha-oss02b kernel: LustreError: dumping log to /tmp/lustre-log.1654441384.808404 Jun 05 17:03:04 z-ha-oss02b kernel: LustreError: 809221:0:(osd_object.c:1300:osd_attr_set()) LBUG Jun 05 17:03:04 z-ha-oss02b kernel: LustreError: 822248:0:(osd_object.c:1300:osd_attr_set()) ASSERTION( obj-&gt;oo_with_projid ) failed: Jun 05 17:03:04 z-ha-oss02b kernel: LustreError: 822248:0:(osd_object.c:1300:osd_attr_set()) LBUG </pre> </div></div> <p>NSC hit this bug Yesterday. We have been running Rocky 8, Lustre 2.14 with ZFS 2.1.x on a few filesystems since March because we wanted dRAID support on the OSSs. We are not using project quota.</p> <p>When upgrading the remaining filesystems to 2.14 with OpenZFS 2.1.4 Yesterday, it ran for 6h before five of the newly upgraded OSSs PANICed, and then re-PANICed pretty quickly after reboot. After applying Shane's fix things remained stable over night. </p> <p>Today one OSS for another filesystem PANICed, so we decided to apply the fix to all remaining servers, including the MDSs. </p> <p>This is just to reaffirm that more people are seeing this and to offer thanks to Shane for sharing the workaround!</p> <p>"Li Dongyang &lt;dongyangli@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47709" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47709</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13189" title="ASSERTION( obj-&gt;oo_with_projid ) failed with 2.12.3" class="issue-link" data-issue-key="LU-13189"><del>LU-13189</del></a> osd-zfs: fix assert on oo_with_projid<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7d631c2f7f45caa36fbdc73b9d83bd98b43edd42</p> <p>Thanks a lot for the patch!</p> <p>Would this now only result in not hitting the LASSERT with an old object anymore, or would this now also result in the old object being updated on disk with an included project ID?</p> <p>Thanks for the patch.</p> <p>Running tests on a non production filesystem at NSC (Rocky 8.6 + ZFS 2.1.5 + Lustre 2.14.0).</p> <p>I've updated patch <a href="https://review.whamcloud.com/47709" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47709</a></p> <p>to add project id for old objects as well.</p> <p>Is the update of old objects also done for objects on a ZFS based MDT? Would <a href="https://jira.whamcloud.com/browse/LU-15640" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-15640</a> also be solved by this patch?</p> <p>Yes it's for both MDT and OST.</p> <p>I think it should let you set project id on old dirs now. if you would test and give some feedback it would be great.</p> <p>BTW was zpool upgrade used during the zfs upgrade? what does zpool status -v show?</p> <p>Thank you very much! I will report back after a test.</p> <p>About the zpool: it was created on new hardware on ZFS 2.0.0, so it did have project quotas enabled by default. But the datasets have been copied over from old hardware with zfs send/resv and did not have project quotas before. I tried to upgrade the datasets with zfs upgrade, but that did not have any effect.</p> <p>After applying the patch to MDTs and OSTs project quotas work es expected on a system that has been migrated with zfs send/recv. Setting the project ID on old directories and files that have been there before the migration is not failing anymore.</p> <p>Thanks a lot, @dongyang!</p> <p>Thanks for the feedback Robert.</p> <p>Good to know setting project ID is not failing. If you get the project id after setting on old files and dirs, it's showing the expected one right?</p> <p>Could you also verify after setting the project id on the old files/dirs, the project quota accounting is showing the expected numbers - they should reflect the old dirs/files?</p> <p>Cheers</p> <p>Dongyang</p> <p>Yes, I can confirm that after setting the project id it is also correctly shown by lfs project and also by lsattr -p. New files created in an old directory with with inherit flag set are also correctly inheriting the project id.</p> <p>The project quota also shows expected values.</p> <p>Great, thanks for the update Robert.</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47846" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47846</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13189" title="ASSERTION( obj-&gt;oo_with_projid ) failed with 2.12.3" class="issue-link" data-issue-key="LU-13189"><del>LU-13189</del></a> osd-zfs: add project id for old objects without ZFS_PROJID<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 26539cc3744155c6b6ad89fc0b5ef1413a8beb14</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47709/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47709/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13189" title="ASSERTION( obj-&gt;oo_with_projid ) failed with 2.12.3" class="issue-link" data-issue-key="LU-13189"><del>LU-13189</del></a> osd-zfs: add project id for old objects without ZFS_PROJID<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ec79791a7cda5b66649200b16a70167d86059e65</p> <p>Landed for 2.16</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47846/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47846/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13189" title="ASSERTION( obj-&gt;oo_with_projid ) failed with 2.12.3" class="issue-link" data-issue-key="LU-13189"><del>LU-13189</del></a> osd-zfs: add project id for old objects without ZFS_PROJID<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: 5a5dad1bc0147b63f377168dde3fe799156a5abd</p> <p>We just hit this today and Shane's</p> <p>#undef ZFS_PROJINHERIT</p> <p>patch seemed to fix it (thanks so much Shane!). Note we don't have project quotas enabled</p> <p>CentOS Linux release 7.9.2009 (Core)<br/> Kernel 3.10.0-1160.49.1.el7_lustre.x86_64<br/> Lustre 2.12.9<br/> MDT - ldiskfs<br/> OSTs - zfs-0.8.6</p> <p>We have only been running with these versions for ~ 3 weeks. The OSTs were upgraded from zfs 0.7.13 and we did run "zpool upgrade ostpool" </p> <p> </p> <p>-k</p> Related LU-15640 Development Rank 1|i00t13: Rank (Obsolete) 9223372036854775807 Severity