Details
-
Improvement
-
Resolution: Fixed
-
Minor
-
None
-
Lustre 2.6.0
-
11321
Description
[15:32:02] John Hammond: Can I ask you something about LDLM_FL_LVB_READY and layout fetch?
[15:33:31] Jinshan Xiong: yes
[15:35:27] John Hammond: In ll_layout_refresh(), in the "requeue layout lock for file ..." case, I see that the layout is returned in the reply to LDLM_ENQUEUE but we still send a MDS_GETXATTR to fetch the layout.
[15:36:14] Jinshan Xiong: actually not. This is the tricky part.
[15:36:15] John Hammond: It seems because LDLM_FL_LVB_READY isn't set until ll_layout_conf() is called.
[15:37:07] Jinshan Xiong: ah LDLM_ENQUEUE, let me see.
[15:37:40] John Hammond: I'm looking at the debug log and I see two RPCs.
[15:38:26] Jinshan Xiong: right now, I believe the layout is returned by getxattr
[15:39:05] John Hammond: But also by the reply to LDLM_ENQUEUE:
[15:39:06] John Hammond: 00000002:00010000:3.0:1383074475.529239:0:4081:0:(mdc_locks.c:760:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56
[15:41:12] Jinshan Xiong: yes, it can be returned by LDLM_ENQUEUE, but not by completion_ast().
[15:42:10] Jinshan Xiong: did you see that the layout is returned but LVB_READY is not set
[15:42:12] Jinshan Xiong: ?
[15:42:23] John Hammond: Right.
[15:43:03] Jinshan Xiong: is that a LAYOUT lock enqueue?
[15:43:58] John Hammond: Yes, from ll_layout_refresh().
[15:46:02] John Hammond: Outside of OSC, only ldlm_lock_allow_match() will set LDLM_FL_LVB_READY and only ll_layout_conf() calls ldlm_lock_allow_match().
[15:46:36] Jinshan Xiong: This is not good.
[15:46:51] Jinshan Xiong: we can only set LVB_READY after layout is applied.
[15:47:54] Jinshan Xiong: seems that we need a new flag to mark that layout has been transferred by DLM ENQUEUE
[15:48:25] Jinshan Xiong: instead of using ldlm_is_lvb_ready() in ll_layout_fetch()
[15:55:58] John Hammond: Maybe I'm missing something but why can't the "layout lock returned by ..." block of mdc_finish_enqueue() set LDLM_FL_LVB_READY?
[15:56:40] Jinshan Xiong: because having a valid layout lock on the client means the layout is correct.
[15:57:02] Jinshan Xiong: So we have to apply the layout before setting the lvb ready flag
[15:57:58] John Hammond: > because having a valid layout lock on the client means the layout is correct.
[15:58:20] John Hammond: Which layout?
[16:00:11] John Hammond: Do you mean that we need separate flags for "LVB ready" and "allow match"?
[16:00:18] Jinshan Xiong: cl_conf_set().
[16:00:42] Jinshan Xiong: LVB ready implies allow match.
[16:00:58] Jinshan Xiong: I mean a new flag to mark layout is returned from DLM ENQ
[16:01:13] Jinshan Xiong: actually in current code, just an extra RPC is needed.
[16:01:30] Jinshan Xiong: based on the fact that layout is rarely changed, maybe not a big deal
[16:02:10] John Hammond: Altought it's rarely changed the lock can be cancelled frequently.
[16:02:47] John Hammond: I'm not changing layouts here.
[16:04:18] Jinshan Xiong: I see. In that case, we can define a new flag for this, or just use l_lvb_data to mark that it's already had a valid layout
[16:05:07] Jinshan Xiong: ah no, we can't use l_lvb_data, because it may be an empty file w/o layout
00000080:00010000:3.0:1383074475.528860:0:4081:0:(file.c:3826:ll_layout_refresh()) ### lustre: requeue layout lock for file ffff880216661180/[0x200000400:0x9:0x0]. 00010000:00010000:3.0:1383074475.528872:0:4081:0:(ldlm_lock.c:797:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8801f9de0c00 lock: ffff8802028618c0/0xb77a1fc2b7a52873 lrc: 3/1,0 mode: --/CR res: [0x200000400:0x9:0x0].0 bits 0x0 rrc: 1 type: IBT flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 4081 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1383074475.528877:0:4081:0:(ldlm_request.c:926:ldlm_cli_enqueue()) ### client-side enqueue START, flags 1000 00010000:00010000:3.0:1383074475.528881:0:4081:0:(ldlm_request.c:988:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8801f9de0c00 lock: ffff8802028618c0/0xb77a1fc2b7a52873 lrc: 3/1,0 mode: --/CR res: [0x200000400:0x9:0x0].0 bits 0x8 rrc: 1 type: IBT flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 4081 timeout: 0 lvb_type: 3 00000100:00100000:3.0:1383074475.528887:0:4081:0:(client.c:1469:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc sys_stat:6b228439-eb8c-afd9-ca76-05a18f32a6fe:4081:1450258649319240:0@lo:101 00000100:00100000:3.0:1383074475.528897:0:4081:0:(events.c:352:request_in_callback()) peer: 12345-0@lo 00000100:00100000:3.0:1383074475.528920:0:4081:0:(client.c:2117:ptlrpc_set_wait()) set ffff8801f5279cc0 going to sleep for 6 seconds 00000100:00100000:2.0:1383074475.528989:0:3435:0:(service.c:2011:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc mdt01_002:6b228439-eb8c-afd9-ca76-05a18f32a6fe+30:4081:x1450258649319240:12345-0@lo:101 00000100:00100000:2.0:1383074475.529103:0:3435:0:(service.c:2055:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc mdt01_002:6b228439-eb8c-afd9-ca76-05a18f32a6fe+32:4081:x1450258649319240:12345-0@lo:101 Request procesed in 116us (205us total) trans 0 rc 0/0 00010000:00080000:3.0:1383074475.529212:0:4081:0:(ldlm_request.c:1317:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 14166000000, Limit: 0) req@ffff8801f52d9800 x1450258649319240/t0(0) o101->lustre-MDT0000-mdc-ffff8801f9de0c00@0@lo:12/10 lens 376/368 e 0 to 0 dl 1383074482 ref 2 fl Rpc:R/0/0 rc 0/0 00000100:00100000:3.0:1383074475.529218:0:4081:0:(client.c:1834:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc sys_stat:6b228439-eb8c-afd9-ca76-05a18f32a6fe:4081:1450258649319240:0@lo:101 00010000:00010000:3.0:1383074475.529225:0:4081:0:(ldlm_lock.c:1091:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8801f9de0c00 lock: ffff8802028618c0/0xb77a1fc2b7a52873 lrc: 4/1,0 mode: CR/CR res: [0x200000400:0x9:0x0].0 bits 0x8 rrc: 1 type: IBT flags: 0x10000000000000 nid: local remote: 0xb77a1fc2b7a52881 expref: -99 pid: 4081 timeout: 0 lvb_type: 3 00010000:00010000:3.0:1383074475.529230:0:4081:0:(ldlm_request.c:700:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8801f9de0c00 lock: ffff8802028618c0/0xb77a1fc2b7a52873 lrc: 4/1,0 mode: CR/CR res: [0x200000400:0x9:0x0].0 bits 0x8 rrc: 1 type: IBT flags: 0x0 nid: local remote: 0xb77a1fc2b7a52881 expref: -99 pid: 4081 timeout: 0 lvb_type: 3 00000002:00100000:3.0:1383074475.529235:0:4081:0:(mdc_locks.c:640:mdc_finish_enqueue()) @@@ op: 1024 disposition: 0, status: 0 req@ffff8801f52d9800 x1450258649319240/t0(0) o101->lustre-MDT0000-mdc-ffff8801f9de0c00@0@lo:12/10 lens 376/368 e 0 to 0 dl 1383074482 ref 1 fl Complete:R/0/0 rc 0/0 00000002:00010000:3.0:1383074475.529239:0:4081:0:(mdc_locks.c:760:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 00000080:00010000:3.0:1383074475.529247:0:4081:0:(llite_internal.h:1569:ll_set_lock_data()) setting l_data to inode ffff880216661180 (144115205255725065/33554436) for lock 0xb77a1fc2b7a52873 00000080:00010000:3.0:1383074475.529250:0:4081:0:(file.c:3659:ll_layout_lock_set()) ### File ffff880216661180/[0x200000400:0x9:0x0] being reconfigured: 1. 00000100:00100000:3.0:1383074475.529267:0:4081:0:(client.c:1469:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc sys_stat:6b228439-eb8c-afd9-ca76-05a18f32a6fe:4081:1450258649319256:0@lo:49 00000100:00100000:3.0:1383074475.529289:0:4081:0:(events.c:352:request_in_callback()) peer: 12345-0@lo 00000100:00100000:3.0:1383074475.529295:0:4081:0:(client.c:2117:ptlrpc_set_wait()) set ffff8801f5279cc0 going to sleep for 6 seconds 00000100:00100000:3.0:1383074475.529329:0:3433:0:(service.c:2011:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc mdt01_000:6b228439-eb8c-afd9-ca76-05a18f32a6fe+32:4081:x1450258649319256:12345-0@lo:49 00000100:00100000:3.0:1383074481.529269:0:4081:0:(client.c:2117:ptlrpc_set_wait()) set ffff8801f5279cc0 going to sleep for 37 seconds