[LU-15491] rename deadlock

• Type: Bug
• Resolution: Unresolved
• Priority: Critical
• Fix Version/s: Lustre 2.16.0
• Affects Version/s: Lustre 2.15.0
• Severity: 3
• Assignee: Oleg Drokin
• Reporter: Oleg Drokin

Description

thread 1: mv dir1/10 dir2/10
thread 2: mv dir2/11 dir2/2
where dir1/10 is hardlink to dir2/2 and dir2/10 is hardlink to dir2/11
so thread 1 and thread2 try to take those locks in reverse order with the predictable result

This is reproduced with non-dne racer run here: http://testing.linuxhacker.ru:3333/lustre-reports/21289/testresults/racer-long-zfs-centos7_x86_64-centos7_x86_64-retry3/

deadlock report reports:

Deadlock!
ungranted lock
   -- Lock: 0xffff8800a93f5b00/0x539a6d335586e266 (pid: 6606)
       Node: local, ns lustre-MDT0000
       Resource: [0x200000401:0x9fd9:0x0].0
       Req mode: EX, grant mode: --, read: 0, write: 1
       Bits: 0x3
waits for granted lock
   -- Lock: 0xffff8800af61e1c0/0x539a6d335586e235 (pid: 14160)
       Node: local, ns lustre-MDT0000
       Resource: [0x200000401:0x9fd9:0x0].0
       Req mode: EX, grant mode: EX, read: 0, write: 1
       Bits: 0x21
but holder is blocked waiting on another lock
   -- Lock: 0xffff880114890900/0x539a6d335586e258 (pid: 14160)
       Node: local, ns lustre-MDT0000
       Resource: [0x200000401:0xa0b7:0x0].0
       Req mode: EX, grant mode: --, read: 0, write: 1
       Bits: 0x3
that is held by the first thread wanting the first lock in the chain
   -- Lock: 0xffff8800a93f61c0/0x539a6d335586e23c (pid: 6606)
       Node: local, ns lustre-MDT0000
       Resource: [0x200000401:0xa0b7:0x0].0
       Req mode: EX, grant mode: EX, read: 0, write: 1
       Bits: 0x21
Deadlock!
ungranted lock
   -- Lock: 0xffff880114890900/0x539a6d335586e258 (pid: 14160)
       Node: local, ns lustre-MDT0000
       Resource: [0x200000401:0xa0b7:0x0].0
       Req mode: EX, grant mode: --, read: 0, write: 1
       Bits: 0x3
waits for granted lock
   -- Lock: 0xffff8800a93f61c0/0x539a6d335586e23c (pid: 6606)
       Node: local, ns lustre-MDT0000
       Resource: [0x200000401:0xa0b7:0x0].0
       Req mode: EX, grant mode: EX, read: 0, write: 1
       Bits: 0x21
but holder is blocked waiting on another lock
   -- Lock: 0xffff8800a93f5b00/0x539a6d335586e266 (pid: 6606)
       Node: local, ns lustre-MDT0000
       Resource: [0x200000401:0x9fd9:0x0].0
       Req mode: EX, grant mode: --, read: 0, write: 1
       Bits: 0x3
that is held by the first thread wanting the first lock in the chain
   -- Lock: 0xffff8800af61e1c0/0x539a6d335586e235 (pid: 14160)
       Node: local, ns lustre-MDT0000
       Resource: [0x200000401:0x9fd9:0x0].0
       Req mode: EX, grant mode: EX, read: 0, write: 1
       Bits: 0x21

thread info for pid 6606 at 0xffff8800d3c6e000:

  mti_rr = {
    rr_opcode = REINT_RENAME,
    rr_open_handle = 0x0,
    rr_lease_handle = 0x0,
    rr_fid1 = 0xffff880116b15000,
    rr_fid2 = 0xffff880116b15010,
    rr_name = {
      ln_name = 0xffff880116b15060 "10",
      ln_namelen = 2
    },
    rr_tgt_name = {
      ln_name = 0xffff880116b15068 "10",
      ln_namelen = 2
    },
crash> p *(struct ll_fid *)0xffff880116b15000
$3 = {
  id = 8589935617,
  generation = 1,
  f_type = 0
}
crash> p *(struct ll_fid *)0xffff880116b15010
$4 = {
  id = 8589935618,
  generation = 41273,
  f_type = 0
}

thread info for pid 14160 at 0xffff88010f554000

  mti_rr = {
    rr_opcode = REINT_RENAME,
    rr_open_handle = 0x0,
    rr_lease_handle = 0x0,
    rr_fid1 = 0xffff880116b15c20,
    rr_fid2 = 0xffff880116b15c30,
    rr_name = {
      ln_name = 0xffff880116b15c80 "11",
      ln_namelen = 2
    },
    rr_tgt_name = {
      ln_name = 0xffff880116b15c88 "2",
      ln_namelen = 1
    },
crash> p *(struct ll_fid *)0xffff880116b15c20
$6 = {
  id = 8589935617,
  generation = 1,
  f_type = 0
}
crash> p *(struct ll_fid *)0xffff880116b15c30
$7 = {
  id = 8589935617,
  generation = 1,
  f_type = 0
}

 

And indeed in mdt_reint_rename() we see:

                lock_ibits = MDS_INODELOCK_LOOKUP | MDS_INODELOCK_XATTR;
                rc = mdt_rename_source_lock(info, msrcdir, mold, lh_oldp,
                                            lh_rmt, lock_ibits, cos_incompat);
 

aka 0x21 lock bit for mold

                lh_newp = &info->mti_lh[MDT_LH_NEW];
                mdt_lock_reg_init(lh_newp, LCK_EX);
                lock_ibits = MDS_INODELOCK_LOOKUP | MDS_INODELOCK_UPDATE;
... // skip remote check
                rc = mdt_reint_object_lock(info, mnew, lh_newp, lock_ibits,
                                           cos_incompat); 

aka 0x3 lock bit

vmcore is available in the report (the server one) if you want to marvel at the coincidence