-
Bug
-
Resolution: Fixed
-
Critical
-
Lustre 2.15.0
-
None
-
3
-
9223372036854775807
thread 1: mv dir1/10 dir2/10
thread 2: mv dir2/11 dir2/2
where dir1/10 is hardlink to dir2/2 and dir2/10 is hardlink to dir2/11
so thread 1 and thread2 try to take those locks in reverse order with the predictable result
This is reproduced with non-dne racer run here: http://testing.linuxhacker.ru:3333/lustre-reports/21289/testresults/racer-long-zfs-centos7_x86_64-centos7_x86_64-retry3/
deadlock report reports:
Deadlock!
ungranted lock
-- Lock: 0xffff8800a93f5b00/0x539a6d335586e266 (pid: 6606)
Node: local, ns lustre-MDT0000
Resource: [0x200000401:0x9fd9:0x0].0
Req mode: EX, grant mode: --, read: 0, write: 1
Bits: 0x3
waits for granted lock
-- Lock: 0xffff8800af61e1c0/0x539a6d335586e235 (pid: 14160)
Node: local, ns lustre-MDT0000
Resource: [0x200000401:0x9fd9:0x0].0
Req mode: EX, grant mode: EX, read: 0, write: 1
Bits: 0x21
but holder is blocked waiting on another lock
-- Lock: 0xffff880114890900/0x539a6d335586e258 (pid: 14160)
Node: local, ns lustre-MDT0000
Resource: [0x200000401:0xa0b7:0x0].0
Req mode: EX, grant mode: --, read: 0, write: 1
Bits: 0x3
that is held by the first thread wanting the first lock in the chain
-- Lock: 0xffff8800a93f61c0/0x539a6d335586e23c (pid: 6606)
Node: local, ns lustre-MDT0000
Resource: [0x200000401:0xa0b7:0x0].0
Req mode: EX, grant mode: EX, read: 0, write: 1
Bits: 0x21
Deadlock!
ungranted lock
-- Lock: 0xffff880114890900/0x539a6d335586e258 (pid: 14160)
Node: local, ns lustre-MDT0000
Resource: [0x200000401:0xa0b7:0x0].0
Req mode: EX, grant mode: --, read: 0, write: 1
Bits: 0x3
waits for granted lock
-- Lock: 0xffff8800a93f61c0/0x539a6d335586e23c (pid: 6606)
Node: local, ns lustre-MDT0000
Resource: [0x200000401:0xa0b7:0x0].0
Req mode: EX, grant mode: EX, read: 0, write: 1
Bits: 0x21
but holder is blocked waiting on another lock
-- Lock: 0xffff8800a93f5b00/0x539a6d335586e266 (pid: 6606)
Node: local, ns lustre-MDT0000
Resource: [0x200000401:0x9fd9:0x0].0
Req mode: EX, grant mode: --, read: 0, write: 1
Bits: 0x3
that is held by the first thread wanting the first lock in the chain
-- Lock: 0xffff8800af61e1c0/0x539a6d335586e235 (pid: 14160)
Node: local, ns lustre-MDT0000
Resource: [0x200000401:0x9fd9:0x0].0
Req mode: EX, grant mode: EX, read: 0, write: 1
Bits: 0x21
thread info for pid 6606 at 0xffff8800d3c6e000:
mti_rr = {
rr_opcode = REINT_RENAME,
rr_open_handle = 0x0,
rr_lease_handle = 0x0,
rr_fid1 = 0xffff880116b15000,
rr_fid2 = 0xffff880116b15010,
rr_name = {
ln_name = 0xffff880116b15060 "10",
ln_namelen = 2
},
rr_tgt_name = {
ln_name = 0xffff880116b15068 "10",
ln_namelen = 2
},
crash> p *(struct ll_fid *)0xffff880116b15000
$3 = {
id = 8589935617,
generation = 1,
f_type = 0
}
crash> p *(struct ll_fid *)0xffff880116b15010
$4 = {
id = 8589935618,
generation = 41273,
f_type = 0
}
thread info for pid 14160 at 0xffff88010f554000
mti_rr = {
rr_opcode = REINT_RENAME,
rr_open_handle = 0x0,
rr_lease_handle = 0x0,
rr_fid1 = 0xffff880116b15c20,
rr_fid2 = 0xffff880116b15c30,
rr_name = {
ln_name = 0xffff880116b15c80 "11",
ln_namelen = 2
},
rr_tgt_name = {
ln_name = 0xffff880116b15c88 "2",
ln_namelen = 1
},
crash> p *(struct ll_fid *)0xffff880116b15c20
$6 = {
id = 8589935617,
generation = 1,
f_type = 0
}
crash> p *(struct ll_fid *)0xffff880116b15c30
$7 = {
id = 8589935617,
generation = 1,
f_type = 0
}
And indeed in mdt_reint_rename() we see:
lock_ibits = MDS_INODELOCK_LOOKUP | MDS_INODELOCK_XATTR; rc = mdt_rename_source_lock(info, msrcdir, mold, lh_oldp, lh_rmt, lock_ibits, cos_incompat);
aka 0x21 lock bit for mold
lh_newp = &info->mti_lh[MDT_LH_NEW];
mdt_lock_reg_init(lh_newp, LCK_EX);
lock_ibits = MDS_INODELOCK_LOOKUP | MDS_INODELOCK_UPDATE;
... // skip remote check
rc = mdt_reint_object_lock(info, mnew, lh_newp, lock_ibits,
cos_incompat);
aka 0x3 lock bit
vmcore is available in the report (the server one) if you want to marvel at the coincidence
