Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20930

lst stop never resends the stop to a node marked down after an RPC timeout

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      lst stop can leave a batch running on a node and then wait for it indefinitely. In Autotest this is the lnet-selftest test_smoke timeout with "N batch in stopping" repeated until the session limit (LU-16876).

      Mechanism (master 4a7bddaf15):

      1. smoke.sh samples every node with lst stat --timeout 10. When a stat RPC to a slow node times out, lstcon_rpc_trans_abort() marks that node LST_NODE_DOWN (lnet/selftest/conrpc.c). A timeout under 6 s is treated as a cancel and does not.
      2. lst stop builds the stop transaction with lstcon_batrpc_condition(), which skips nodes that are not active for LST_TRANS_TSBSTOP (lnet/selftest/console.c). No stop RPC goes to that node, and nothing is printed.
      3. jt_lst_stop_batch() (lnet/utils/lst.c) then only queries. The query reply marks the node active again and reports the batch running, so the loop prints "N batch in stopping" on every query, but the stop is never resent. The node stops only when its test units finish all their loops, which for 1000 loops at a few MiB/s (or 100000 in full runs) is longer than the session.
      4. A variant: when the stop RPC is sent but times out (-110), lst stop returns at once ("stop batch RPC failed on ...") and the batch keeps running on that node; show_error to it then fails too.

      This matches the CI timeouts: no LNet or selftest errors in dmesg, lst waiting in the batch query, idle selftest threads in the stack dumps, and "Failed to stat" just before the stop. A slow node is what sets it off (most recent review failures had a VM on onyx-75, LU-20928), but the hang itself is in lst.

      Reproduction: a 2x2 brw batch with 100000 loops; delay only LNet traffic (TCP port 988) leaving the OSS by 15 s with netem; take one lst stat --timeout 10 sample; remove the delay; time lst stop. Unpatched lst hung to the 600 s cap in 2 of 2 runs (388 and 422 "batch in stopping" lines). It also happened once naturally in 7 local test_smoke runs; a second lst stop from another shell ended it.

      Fix: have lst stop resend the stop every ten polls while nodes still report the batch running (the query reply has marked them active again, so the resent stop reaches them), and report a failed stop RPC and keep waiting instead of returning. A node that does not answer the query still ends the loop, so a dead node cannot hang it. With the fix, the reproducer stopped in 16 s and 28 s. Sending the stop to LST_NODE_DOWN nodes in the kernel would not cover a stop RPC that times out, which the resend does.

            wc-triage WC Triage
            patrickbot Patrick Bot
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: