[LU-3469] OSP dt_sync() operation should flush pending destroys and other updates

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.6.0
• Fix Version/s: Lustre 2.6.0
• Type: Technical task
• Priority: Blocker
• Reporter: Andreas Dilger
• Assignee: nasf (Inactive)
• Resolution: Fixed

Description

lod_sync() should be changed to allow flushing the pending OST object destroys so that LFSCK can ensure that the OST state is consistent after pass 1 is completed:

• call local/child OSD dt_sync() first, so that unlink name commit callbacks will be run and schedule all of the OST object destroys
• OSP dt_sync() will wait until all of the pending object destroys are at least sent to the OSTs (accessing their respective OST objects and removing them from the orphan list). They do not necessarily need to be committed.

Issue Links

• is blocking: LU-3336 LFSCK II: MDT-OST OST orphan handling
• is related to: LU-5387 Interop 2.5.2<->2.6 failure on test s...

Comments

[Andreas Dilger]

Alex, any comments on this issue?

[Andreas Dilger]

One proposal is to have LFSCK only do a sync on a specific FID (e.g. the orphan object FID) will cause the pending destroys to be flushed. There is a larger question of whether sync on the LOD in general should cause all the destroys to be flushed, so that users can force file deletions to be flushed to the OSTs and release space.

This issue is a fairly rare case of files being deleted that were traversed in the last 15 or so seconds of the phase II namespace scan. At worst this would result in previously-deleted files being linked into lost+found.

[Richard Henwood (Inactive)]

After discussion with Fan Yong: I would like this bug to be considered for Blocker status.

[nasf (Inactive)]

Recently, I hit some test failures for test_18d/test_18e. The root reason is that the async repair operations triggered during first cycle scanning are not executed until orphan handling phase.

So we need the dt_sync() for that.

[Alex Zhuravlev]

Yong, could you point out which specific lines will be calling dt_sync(), please?

[nasf (Inactive)]

Alex, the current LFSCK does not call it since osp_sync() is not implemented. But it is easy to add it as the following patch:

diff --git a/lustre/lfsck/lfsck_layout.c b/lustre/lfsck/lfsck_layout.c
index de96726..7be2346 100644
--- a/lustre/lfsck/lfsck_layout.c
+++ b/lustre/lfsck/lfsck_layout.c
@@ -3435,6 +3435,9 @@ static int lfsck_layout_assistant(void *args)
                                com->lc_time_last_checkpoint +
                                cfs_time_seconds(LFSCK_CHECKPOINT_INTERVAL);
 
+                       /* flush all async updating before handling orphan. */
+                       dt_sync(env, lfsck->li_next);
+
                        while (llmd->llmd_in_double_scan) {
                                struct lfsck_tgt_descs  *ltds =
                                                        &lfsck->li_ost_descs;

[Andreas Dilger]

Alex, Fan Yong, is there any reason not to just convert the previous comment into a patch? Is there more work to be done?

[Alex Zhuravlev]

like said before, osp_sync() is empty yet, I've been working on this.

[Alex Zhuravlev]

also, at the moment I'm not sure just a single dt_sync() is good enough. failed osp_sync() isn't any different from failed transaction? if we can't complete one phase on a specific OST we shouldn't continue with that OST in this LFSCK run? IOW, we'd prefer to track per-OST status rather than combined one which doesn't let us learn which OSTs are safe to proceed?

[nasf (Inactive)]

If osp_sync() is failed to flush all pending destroy to OSTs, and if the LFSCK goes ahead for orphan handling, then the LFSCK may found some OST-object)(s) which parent MDT-object have been destroyed on the MDT(s) by unlink, but the LFSCK cannot be aware of the race unlink, so it will re-create the MDT-object(s), that is unexpected.

One possible solution is that, if some osp_sync() failed, then skip orphan handling on related OST(s).

[Alex Zhuravlev]

http://review.whamcloud.com/10046 - this patch implements osp_sync() which is synchronous. give the discussion above I tend to think it'd be better to call into all OSPs/OSDs directly from LFSCK. in the long term, something like an empty (or almost empty async transactions with commit callbacks registered would serve better being concurrent.

[Andreas Dilger]

Alex, Fan Yong, what is needed next for this patch? Are further changes needed in the LFSCK code to start using this new functionality?

[nasf (Inactive)]

Generally, the patch looks good, but we still cannot land the patch because of Maloo test failure.

[Jodi Levi (Inactive)]

Assigned to Fan Yong as he has updated the patch

[Peter Jones]

Landed for 2.6