https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8967 Lustre <p>We have several directories with entries for non existing files. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@quartz2311:~]# ls -l /p/lscratchh/casses1/quartz-zinc_3/19519/dbench/quartz2322/clients/client0 ls: cannot access /p/lscratchh/casses1/quartz-zinc_3/19519/dbench/quartz2322/clients/client0/filler.003: No such file or directory total 3154 -rw------- 1 casses1 casses1 1048576 Dec 21 16:43 filler.000 -rw------- 1 casses1 casses1 1048576 Dec 21 16:43 filler.001 -rw------- 1 casses1 casses1 1048576 Dec 21 16:43 filler.002 -????????? ? ? ? ? ? filler.003 drwx------ 2 casses1 casses1 25600 Dec 21 16:43 ~dmtmp </pre> </div></div> <p>The directory itself is a remote directory on one MDT:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@quartz2311:~]# lfs getdirstripe -d /p/lscratchh/casses1/quartz-zinc_3/19519/dbench/quartz2322/clients/client0 lmv_stripe_count: 0 lmv_stripe_offset: 3 </pre> </div></div> <p>We are able to get striping information for this file:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@quartz2311:~]# lfs getstripe /p/lscratchh/casses1/quartz-zinc_3/19519/dbench/quartz2322/clients/client0/filler.003 /p/lscratchh/casses1/quartz-zinc_3/19519/dbench/quartz2322/clients/client0/filler.003 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 27 obdidx objid objid group 27 20538776 0x1396598 0xcc0000402 </pre> </div></div> <p>It looks like the OSS serving that OST was rebooted and the OST went through recovery around the time the missing file was created. In particular, we note that the object number falls in the range of orphan objects that were deleted:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@zinci:~]# grep 0xcc0000402 /var/log/conman/console.zinc* /var/log/conman/console.zinc43:2016-12-21 16:30:56 [189484.767900] Lustre: lsh-OST001b: deleting orphan objects from 0xcc0000402:20538706 to 0xcc0000402:20541649 /var/log/conman/console.zinc43:2016-12-21 16:33:30 [189639.110247] Lustre: lsh-OST001b: deleting orphan objects from 0xcc0000402:20538766 to 0xcc0000402:20541649 /var/log/conman/console.zinc43:2016-12-21 16:35:41 [189769.704490] Lustre: lsh-OST001b: deleting orphan objects from 0xcc0000402:20538766 to 0xcc0000402:20541649 /var/log/conman/console.zinc43:2016-12-21 16:40:19 [190047.449320] Lustre: lsh-OST001b: deleting orphan objects from 0xcc0000402:20538766 to 0xcc0000402:20541649 /var/log/conman/console.zinc43:2016-12-21 16:44:45 [190313.751155] Lustre: lsh-OST001b: deleting orphan objects from 0xcc0000402:20538820 to 0xcc0000402:20541649 /var/log/conman/console.zinc44:2016-12-21 16:49:27 [ 159.838420] Lustre: lsh-OST001b: deleting orphan objects from 0xcc0000402:20538820 to 0xcc0000402:20541649 </pre> </div></div> <p>I will attach server console logs separately.</p> <a href="ssh://review.whamcloud.com/fs/lustre-release-fe-llnl">ssh://review.whamcloud.com/fs/lustre-release-fe-llnl</a> LU-8967

directory entries for non existing files

Bug Blocker Resolved Fixed Mikhail Pershin Ned Bass llnl Fri, 23 Dec 2016 01:20:06 +0000 Thu, 10 Aug 2017 23:36:53 +0000 Mon, 27 Feb 2017 21:56:57 +0000 Lustre 2.8.0 0 9 <p>Also please note that we first observed this problem after our most recent Lustre update to 2.8.0_6chaos last Friday (December 16). The patches added in that update were:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>* 353716b (tag: 2.8.0_6.chaos, llnl/2.8.0-llnl) LU-8753 llog: add some debug patch * 17d469a LU-8936 llite: use percpu env correctly in ll_invalidatepage * a15b2ef LU-8361 lfsck: detect Lustre device automatically * 0220e0b LU-7648 man: new man pages for LFSCK commands * 1638a07 LU-7256 tests: wait current LFSCK to exit before next test * 1d8cfaa LU-8407 recovery: more clear message about recovery failure * fdea0d2 LU-7732 ldlm: silence verbose "waking for gap" log messages * 82e924c LU-8753 llog: remove lgh_write_offset * 3a8db9a LU-8493 osp: Do not set stale for new osp obj * 38c062b LU-7660 dne: support fs default stripe * bc3df36 Revert "LU-8422 update: add more debug info for the ticket" * 10170a0 Revert "LU-8422 llog: extended debug info" * 490414a Revert "LU-6635 lfsck: more debug message for sanity-lfsck test_18e" </pre> </div></div> <p>Mike</p> <p>Could you please assist with this issue?</p> <p>Thanks</p> <p>Peter</p> <p>I suspect this is related to <a href="https://jira.whamcloud.com/browse/LU-8562" title="osp_precreate_cleanup_orphans/osp_precreate_reserve race may cause data loss" class="issue-link" data-issue-key="LU-8562"><del>LU-8562</del></a>.</p> <p>Ned, are these entries occurred once when OST was failed over or still continue to occur? Is it possible to remove them?</p> <p>I am checking patches you've mentioned.</p> <p>Hi Mikhail, Each occurrence that I've investigated happened immediately after the OST completed recovery. The object numbers of the missing files all fall at the beginning of the range of deleted orphans. It does not continue to occur when all OSTs are up.</p> <p>I can remove the files as root. The rm command fails for an unprivileged user because stat() returns ENONENT and rm treats that as fatal unless you're root.</p> <p>I have confirmed that I can reproduce <a href="https://jira.whamcloud.com/browse/LU-8562" title="osp_precreate_cleanup_orphans/osp_precreate_reserve race may cause data loss" class="issue-link" data-issue-key="LU-8562"><del>LU-8562</del></a> on our system using the test case from that patch and it looks just like this issue. I tested <a href="https://review.whamcloud.com/#/c/22211/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/22211/</a> on a single node setup and wasn't able to reproduce the bug. However I ran into a defect with that patch that causes the osp_precreate thread to hang as I described in <a href="https://jira.whamcloud.com/browse/LU-8562" title="osp_precreate_cleanup_orphans/osp_precreate_reserve race may cause data loss" class="issue-link" data-issue-key="LU-8562"><del>LU-8562</del></a>.</p> <p> </p> <p>Ned, so this issue is solved by <a href="https://jira.whamcloud.com/browse/LU-8562" title="osp_precreate_cleanup_orphans/osp_precreate_reserve race may cause data loss" class="issue-link" data-issue-key="LU-8562"><del>LU-8562</del></a> in general, but patch itself contains defect. I checked your patch, does it solves your problem? Or more work is required in that area?</p> <p>Interesting that <a href="https://jira.whamcloud.com/browse/LU-8562" title="osp_precreate_cleanup_orphans/osp_precreate_reserve race may cause data loss" class="issue-link" data-issue-key="LU-8562"><del>LU-8562</del></a> itself is quite recent change and we did't observe a lot of issues similar to <a href="https://jira.whamcloud.com/browse/LU-8967" title="directory entries for non existing files" class="issue-link" data-issue-key="LU-8967"><del>LU-8967</del></a> without it. I wonder what was changed in your system when you start seeing it. Was it just a software update or hardware as well?</p> <blockquote> <p>Ned, so this issue is solved by <a href="https://jira.whamcloud.com/browse/LU-8562" title="osp_precreate_cleanup_orphans/osp_precreate_reserve race may cause data loss" class="issue-link" data-issue-key="LU-8562"><del>LU-8562</del></a> in general, but patch itself contains defect. I checked your patch, does it solves your problem? Or more work is required in that area?</p></blockquote> <p>I have tested the <a href="https://jira.whamcloud.com/browse/LU-8562" title="osp_precreate_cleanup_orphans/osp_precreate_reserve race may cause data loss" class="issue-link" data-issue-key="LU-8562"><del>LU-8562</del></a> patch and my one-line follow-on patch <a href="https://review.whamcloud.com/24758" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24758</a> on a single-node test setup. I am no longer able to reproduce the data loss bug with those patches applied. Without the patches I can reproduce it almost immediately using the <a href="https://jira.whamcloud.com/browse/LU-8562" title="osp_precreate_cleanup_orphans/osp_precreate_reserve race may cause data loss" class="issue-link" data-issue-key="LU-8562"><del>LU-8562</del></a> test case.</p> <p>The  remaining work to do in that area is as follows.</p> <ul> <li>An explanation is needed as to why conf-sanity test_101 is still failing as per <a href="https://jira.whamcloud.com/browse/LU-8972" title="conf-sanity test_101: File hasn&#39;t object on OST" class="issue-link" data-issue-key="LU-8972"><del>LU-8972</del></a>. The ongoing test case failure suggests the data loss bug is not completely resolved by the <a href="https://jira.whamcloud.com/browse/LU-8562" title="osp_precreate_cleanup_orphans/osp_precreate_reserve race may cause data loss" class="issue-link" data-issue-key="LU-8562"><del>LU-8562</del></a> patch. We need high confidence that this bug is resolved before putting user data on Lustre 2.8 FE.</li> <li>Patch <a href="https://review.whamcloud.com/24758" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24758</a> needs review by someone who understands the <tt>osp_precreate_thread</tt> state machine better than me.</li> </ul> <p> </p> <blockquote> <p>Interesting that <a href="https://jira.whamcloud.com/browse/LU-8562" title="osp_precreate_cleanup_orphans/osp_precreate_reserve race may cause data loss" class="issue-link" data-issue-key="LU-8562"><del>LU-8562</del></a> itself is quite recent change and we did't observe a lot of issues similar to <a href="https://jira.whamcloud.com/browse/LU-8967" title="directory entries for non existing files" class="issue-link" data-issue-key="LU-8967"><del>LU-8967</del></a> without it. I wonder what was changed in your system when you start seeing it. Was it just a software update or hardware as well?</p></blockquote> <p> </p> <p>My best guess as to why we started seeing <a href="https://jira.whamcloud.com/browse/LU-8562" title="osp_precreate_cleanup_orphans/osp_precreate_reserve race may cause data loss" class="issue-link" data-issue-key="LU-8562"><del>LU-8562</del></a> is that we made changes to our pacemaker/corosync HA system. We recently optimized the configuration so Lustre services are started with much less delay than before. This makes it is very likely that OST orphan cleanup will be interrupted by the HA partner coming up and failing back the OST. As I understand, that is the race window for <a href="https://jira.whamcloud.com/browse/LU-8562" title="osp_precreate_cleanup_orphans/osp_precreate_reserve race may cause data loss" class="issue-link" data-issue-key="LU-8562"><del>LU-8562</del></a> to occur. Before it took a long time for services to start up, so orphan cleanup was almost always done by the time the partner failed back the OST.</p> <p>with <a href="https://jira.whamcloud.com/browse/LU-8562" title="osp_precreate_cleanup_orphans/osp_precreate_reserve race may cause data loss" class="issue-link" data-issue-key="LU-8562"><del>LU-8562</del></a> I still see the same symptoms rarely.. there is another patch addressing the same issue, it's doing a bit better, but still possible to reproduce within few hours. I've been looking for the root cause.</p> <p>a prototype is under testing, I'm going to pass it through Maloo few more times..</p> <p>This is now confirmed as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-8562" title="osp_precreate_cleanup_orphans/osp_precreate_reserve race may cause data loss" class="issue-link" data-issue-key="LU-8562"><del>LU-8562</del></a> and so you should proceed with using Ned's ports of those patches to 2.8 FE. In addition it is recommended that you pick up the fix for <a href="https://jira.whamcloud.com/browse/LU-8367" title="delete orphan phase isn&#39;t stated for multistriped file" class="issue-link" data-issue-key="LU-8367"><del>LU-8367</del></a>.</p> <p>AFAIK items tracked under this ticket are complete</p> Duplicate LU-8562 Development Rank 1|hzyz7b: Rank (Obsolete) 9223372036854775807 Severity