Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20892

nodemap: exp_banned write without exp_lock can lose exp_hashed

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

        1. Problem
          nodemap writes exp_banned without exp_lock. exp_banned is a 1-bit field in the same word as exp_hashed and other export flags. The struct comment says that exp_lock protects that word. A concurrent write to another bit in the word can lose the exp_banned update, or the exp_banned write can lose the other update.
        1. Environment
      • master at 550451c0e4 and earlier.
      • A server with nodemap active.
        1. Reproduction
          Source analysis only. We did not reproduce the failure.
          1. A client connects. target_handle_connect() calls obd_nid_add(), which sets exp_hashed.
          2. At the same time, a nodemap change calls nm_member_reclassify_nodemap(), which sets or clears exp_banned on the same export.
          3. Each side reads the word, changes its bit, and writes the word back. The later write removes the earlier update.
        1. Analysis
          Writers of exp_banned without exp_lock:
      • nodemap_add_member(), lustre/ptlrpc/nodemap_handler.c
      • nm_member_reclassify_nodemap(), the "banning client" and "un-banned client" branches, lustre/ptlrpc/nodemap_member.c

      The two results of a lost update:

      • exp_hashed goes back to 0 while the export is in obd_nid_hash. class_disconnect() then skips obd_nid_del(), and the hash keeps its export reference. The export leaks.
      • exp_banned goes back to its old value. The ban change does not apply.

      LU-20403 adds a runtime writer of exp_hashed: ptlrpc_refresh_export_primary() re-keys the export in obd_nid_hash under exp_lock when the client primary NID migrates. Before LU-20403, the main writer ran at connect.

        1. Proposed fix
          Take exp_lock around each exp_banned write in nodemap_handler.c and nodemap_member.c.

            wc-triage WC Triage
            hornc Chris Horn
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: