-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
Problem
A client mount fails with EEXIST (-17) when it starts soon after a umount on a file system with GSS (SSK or krb5). The new OSC cannot create its LDLM namespace in sysfs. The previous OSC with the same name still holds that sysfs entry. The failure occurred in sanity test_247f, in review testing of a change that is not public.
mount.lustre: mount 10.240.44.29@tcp:/lustre/d247f.sanity/striped/. at /mnt/lustre_d247f.sanity failed: File exists sysfs: cannot create duplicate filename '/fs/lustre/ldlm/namespaces/lustre-OST0000-osc-ff2ab407957f8000' LustreError: (ldlm_resource.c:839:ldlm_namespace_new()) lustre-OST0000-osc-ff2ab407957f8000: cannot initialize ns sysfs: rc = -17 LustreError: (ldlm_lib.c:637:client_obd_setup()) lustre-OST0000-osc-ff2ab407957f8000: unable to create client namespace: rc = -17
Environment
- Test group: review-dne-selinux-ssk-part-1 (SSK flavor ski), Rocky 9.7 x86_64, 2 or more MDTs.
- Master has the same code in gss_do_ctx_init_rpc() and ptlrpc_import_delay_req().
Reproduction
- Mount a client with a GSS flavor.
- Unmount the client before the GSS context negotiation of an OSC import completes.
- Mount the client again at once, so that the new super block gets the address of the old one.
sanity test_247f does steps 1 to 3 in a loop.
Analysis
The client debug log shows this sequence (epoch seconds 1793714xxx):
| Time | Thread | Event |
| ------ | -------- | ------- |
| 465.507 | mount | Creates the GSS ctx for OST0000 and starts the lgss_keyring upcall. The OSC import never connects. |
| 467.512 | lgss_keyring | gss_do_ctx_init_rpc() checks imp_deactive, takes an import reference and allocates the request. |
| 467.525 | umount | obd_cleanup_client_import() sets the import INVALID. ptlrpc_invalidate_import() sees imp_inflight == 0 and returns at once. |
| 467.530 | lgss_keyring | Sends SEC_CTX_INIT (o801) on the INVALID import. |
| 467.533 | umount | Detaches the OSC. The obd refcount stays at 2, because the RPC holds the import. |
| 467.759 | mount | Creates a new OSC with the same name. ldlm_namespace_new() fails with -17. |
| 468.131 | lgss_keyring | The reply arrives. The ctx, sec, import and old obd are freed. |
The race has two parts:
- gss_do_ctx_init_rpc() checks the import state before the send, not at the send. ptlrpc_invalidate_import() can run between the check and the send.
- ptlrpc_import_delay_req() always passes ctx init and ctx fini requests. Thus the ctx init request goes out on the import of a stopping obd, after the invalidate wait is complete.
The client OBD names contain the super block address. A new super block can get the address of the freed one, so the new OSC gets the name of the old OSC.
LU-20543 has the same symptom. Its fix (048bb8c49f) removes a permanent key leak. The fix is present in this build. Here the reference is transient and lasts for one RPC round trip.
Proposed fix
In ptlrpc_import_delay_req(), under imp_lock, fail a ctx init request with -ESHUTDOWN when the obd is stopping. class_cleanup() sets the stopping flag before obd_precleanup() invalidates the import. Thus the invalidate either sees the request in flight and waits for it, or the send sees the flag and fails. Ctx fini requests do not change.
- is related to
-
LU-21110 client device names reuse the super block address after umount
-
- Open
-