[LU-4215] Some expected improvements for OUT

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.6.0
• Labels: LMR
• Severity: 3
• Assignee: Alex Zhuravlev
• Reporter: nasf (Inactive)

Description

1. OUT RPC service threads on MDT and OST using different reply portals confused the OUT RPC user.

On MDT-side, it is:

                .psc_buf                = {
                        .bc_nbufs               = MDS_NBUFS,
                        .bc_buf_size            = OUT_BUFSIZE,
                        .bc_req_max_size        = OUT_MAXREQSIZE,
                        .bc_rep_max_size        = OUT_MAXREPSIZE,
                        .bc_req_portal          = OUT_PORTAL,
                        .bc_rep_portal          = MDC_REPLY_PORTAL,
                },

On OST-side, it is:

                .psc_buf                = {
                        .bc_nbufs               = OST_NBUFS,
                        .bc_buf_size            = OUT_BUFSIZE,
                        .bc_req_max_size        = OUT_MAXREQSIZE,
                        .bc_rep_max_size        = OUT_MAXREPSIZE,
                        .bc_req_portal          = OUT_PORTAL,
                        .bc_rep_portal          = OSC_REPLY_PORTAL,
                },

For the case that both MDT and OST runs on the same physical server node (especially for VM environment testing), when OSP wants to talk with OST via OUT_PORTAL, the OUT RPC maybe handled by MDT-side OUT RPC service thread unexpected, and replied via MDC_REPLY_PORTAL, instead of OSC_REPLY_PORTAL on which the OSP is waiting for the reply. Then caused the OSP-side OUT RPC timeout and resend again and again.

The bad case also can happen when OSP wants to talk with MDT via OUT_PORTAL.

Because NDE I has already used the OUT RPC for talking among MDTs. To be compatible with the old version, we cannot change the MDT-side OUT RPC reply portal. So we have to chance OST-side OUT RPC reply portal to "MDC_REPLY_PORTAL". But it is strange for OST-side to use MDT-side reply portal.

2. The OUT RPC version is fixed on "LUSTRE_MDS_VERSION", in spite of the RPC is to MDT or to OST. Also confused others. We can re-define "tgt_out_handlers". But it may break the policy of Unified Target.

3. Pack multiple idempotent sub-requests into single OUT RPC. In general, the OUT RPC should not assume that the sub-requests are related with each other. So even if one sub-request failed to be executed, the others should not be ignored. But in current implementation, it is not. If the other sub-requests are not related with the failed one, then such behavior is unexpected. Unfortunately, it is not easy to judge whether one sub-request is related with the others within current OUT request format, especially consider to be compatible with DNE I.

4. Iteration via OUT. I found some client-side iteration framework in osp_md_object.c, but seems no server side handler. Do we have any plan to support that?

Issue Links

• is blocked by: LU-7318 OUT: dynamic reply buffer
• is blocked by: LU-7319 OUT: continue updates processing upon an error
• is blocking: LU-4009 Add ZIL support to osd-zfs
• is related to: LU-17818 LMR: Lustre Metadata Redundancy
• is related to: LU-4690 sanity test_4: Expect error removing in-use dir /mnt/lustre/remote_dir
• is related to: LU-12310 MDT Device-level Replication/Mirroring
• is related to: LU-7426 DNE3: improve llog format for remote update llog
• is related to: LU-7427 DNE3: multiple entries for BATCHID
• is related to: LU-3467 Unified request handler on OST
• is related to: LU-3539 Change update RPC format
• is related to: LU-7426 DNE3: improve llog format for remote update llog

Comments

[Di Wang]

I just checked current master code, which seems not resolved yet, not sure in Nasf's patches. For DNE, it always fail immediately, which is good enough even for DNE2. For LFSCK, is this only for read-only updates like getattr? Hmm, there is padding in OSP update request

* Hold object_updates sending to the remote OUT in single RPC */
struct object_update_request {
        __u32                   ourq_magic;
        __u16                   ourq_count;     /* number of ourq_updates[] */
        __u16                   ourq_padding;
        struct object_update    ourq_updates[0];
};

We can add the flag there.

[Andreas Dilger]

Di, Nasf, what is the status on fixing this last issue? What is the proposed solution? Should the server mark all later batchids as failed, or should it try to execute them? What if they are dependent on each other? Is there a flag that could be set on the batch that indicates if it should be executed even if the previous batch failed?

[nasf (Inactive)]

The code for batched requests has worked since DNE 1. The trouble is that the handling for the batched requests within single OUT RPC will stop when it hits failure at some of the sub-request and the left sub-requests will be ignored even though they are not related with failed one. (that is the #3)

[Andreas Dilger]

It seems #3 is the only item still outstanding. Is the code to handle batched requests working?

[nasf (Inactive)]

Currently, LFSCK uses OUT RPC to talk with OST via OSP, it shares the interface out_prep_update_req() with the RPC to/from MDT. Inside such function, it always uses LUSTRE_MDS_VERSION in spite of whether it is for OST or MDT, which is confused.

[Di Wang]

Hmm, I think #2 means we also pack OUT RPC with LUSTRE_MDS_VERSION (see out_prep_update_req), no matter this OUT RPC will be sent to MDS or OST. Right now, DNE only send out RPC to another MDS, but for LFSCK, I assume some OUT RPC needs to be sent to OST. So I think this is the one needs to be fixed. Though I guess the request is from LFSCK project, probably Fan Yong can confirm.

[Alex Zhuravlev]

Di, could you clarify on #2 a bit please?

[Alex Zhuravlev]

Andreas, yes.

[Andreas Dilger]

Alex, is there a chance for you to work on patches for 2.6 for the #2 and #3 items? Di already has far too many 2.6 blocker bugs to work on this, so if we want these changes then you are the best candidate to do the work.

[Alex Zhuravlev]

yes, the ability to continue processing in case of error is important for batched DESTROY's, for example. as for efficiency, I think we should not use obdo - it's huge, instead we should probably be able to get/set just a subset of attributes like dt_attr_set() allows.

[nasf (Inactive)]

For requirement 3, if we want to support packaged attr_get/xattr_get on multiple OST-objects via single OUT RPC, then we need to continue the OUT RPC handling even if some sub-requests failed. For example, the MDT (via OSP) wants to attr_get/xattr_get on both OST-object1 and OST-object2 via single OUT RPC, and it does not whether the two targets exists or not; so on the OST-side, it should not skip the OST-object2 even if OST-object1 does not exist.

Currently, because we do not support that well yet, the MDT (via OSP) only pack the sub-requests which belong to the same target object is the OUT RPC. Once we improvement it, we can consider to make the OUT RPC to be more efficient.