Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20883

ptlrpc: GSS error reply to a resent RPC uses the old match bits and is dropped

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Problem

      Since LU-14594 (OBD_CONNECT2_REP_MBITS, 2.14.52), a resent RPC gets new reply match bits. The sender waits for the reply on the new bits (ptl_send_rpc(), niobuf.c). The receiver reads the new bits from the request into rq_rep_mbits and uses them for the reply (ptlrpc_server_handle_req_in(), service.c).

      But when the GSS layer rejects the request, ptlrpc_server_handle_req_in() sends the error reply as soon as sptlrpc_svc_unwrap_request() returns SECSVC_COMPLETE. That is before rq_rep_mbits is set. So the error reply uses rq_xid as match bits. For a resend these are the old bits. LNet on the sender finds no match and drops the reply. The sender then waits a full request timeout, instead of seeing the GSS error at once.

      The first send is not affected, because its match bits are equal to its XID.

      Where it was seen

      sanity-sec test_27d and test_27e, review-dne-selinux-ssk-part-2 (el8.10), Gerrit 67783 patch set 6 (LU-20542). There an OSS resends an LDLM completion AST over a reverse context that the client has already dropped.

      OSS debug log (27d):

      ptlrpc_set_mbits()) resend with new mbits old x1878292362931072 new x1878292362931200
      lnet_parse_put()) Dropping PUT from 12345-10.240.23.83@tcp portal 16 match 1878292362931072 offset 360 length 320: 4
      ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1791279480/real 1791279480] req@ff4b3aab007f56c0 x1878292362931072/t0(0) o105->lustre-OST0003@10.240.23.83@tcp:15/16 ...
      

      The client debug log shows that it got the resend and rejected it:

      ptlrpc_server_handle_req_in()) unwrap req x1878292362931072
      gss_svc_searchbyctx()) ctx hdl 0xb6b53fed0af81423 does not have mech ctx: rc = -2
      ptlrpc_server_handle_req_in()) finish req x1878292362931072
      

      27e shows the same sequence on OST0004. Logs: https://testing.whamcloud.com/test_sets/13406571-a9fa-4d90-ae90-819a9acf44be (the per-subtest debug_log files).

      Scope

      The same receive path handles forward RPCs. So a client resend that hits a GSS error on the server should lose the error reply in the same way. This is not tested.

      Possible fixes

      • On the receiver: send the error reply on the request's match bits. This only works if the receiver can read them. With the skpi flavor the request body is encrypted, and a rejected request cannot be decrypted. So this does not cover every case.
      • On the sender: keep the old match bits for a resend that can only get a GSS error. LU-20542 (Gerrit 67783) does this for its own reverse-context resend. In general the sender does not know in advance that a GSS error will come back.

      A general fix needs a decision on which side should change.

            wc-triage WC Triage
            hnishida Hiroshi Nishida
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: