[LU-7736] lustre_rmmod does not remove all the Lustre modules

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.9.0
• Labels: patch
• Environment: Lustre 2.7.66
• Severity: 3
• Assignee: Bob Glossman (Inactive)
• Reporter: Gregoire Pichon

Description

lustre_rmmod does not remove all the Lustre modules. A second call to the command does.

[root@rio10 ~]# modprobe lustre
[root@rio10 ~]# lctl list_nids
10.1.0.64@o2ib
[root@rio10 ~]# lustre_rmmod
Modules still loaded: 
lnet/lnet/lnet.o libcfs/libcfs/libcfs.o
[root@rio10 ~]# lustre_rmmod

After analysing the problem, it appears that:

• the lnet module does not has a direct dependency on the lnd modules (ko2iblnd, ksocklnd, ...). This is because lnet uses function pointers to call the lnd routines (lnd_startup(), lnd_shutdown(), ...). This avoids a loop in module dependecy, because the lnd modules do also depends on lnet.

• the unloading of ptlrpc module triggers the shutdown of lnet network interfaces and thus the decrement of lnd refcount, which allows the lnd modules to be no more "in use".

• the lustre_rmmod script unloads the lustre modules recursively from libcfs, unloading each dependent module first.

By chance, in previous lustre versions (2.1, 2.4 or 2.5) the dependency order made the lustre_rmmod unload ptlrpc before ko2iblnd.

Unfortunately, since lustre version 2.7, ko2iblnd is still in use when trying to unload, which then prevents lnet to unload. Then lnet module unload is not attempted again, leading to lnet and libcfs still loaded at the end.

[root@rio10 ~]# modprobe lustre
[root@rio10 ~]# lustre_rmmod
DEBUG: rmmod lustre
DEBUG: rmmod mdc
DEBUG: rmmod fid
DEBUG: rmmod lmv
DEBUG: rmmod fld
DEBUG: rmmod lmv
rmmod: ERROR: Module lmv is not currently loaded
DEBUG: rmmod mdc
rmmod: ERROR: Module mdc is not currently loaded
DEBUG: rmmod lov
DEBUG: rmmod ko2iblnd
rmmod: ERROR: Module ko2iblnd is in use
DEBUG: rmmod ptlrpc
DEBUG: rmmod obdclass
DEBUG: rmmod ptlrpc
rmmod: ERROR: Module ptlrpc is not currently loaded
DEBUG: rmmod lnet
rmmod: ERROR: Module lnet is in use by: ko2iblnd
DEBUG: rmmod ko2iblnd
DEBUG: rmmod lustre
rmmod: ERROR: Module lustre is not currently loaded
DEBUG: rmmod obdclass
rmmod: ERROR: Module obdclass is not currently loaded
DEBUG: rmmod ptlrpc
rmmod: ERROR: Module ptlrpc is not currently loaded
DEBUG: rmmod libcfs
rmmod: ERROR: Module libcfs is in use by: lnet
Modules still loaded: 
lnet/lnet/lnet.o libcfs/libcfs/libcfs.o

Issue Links

• duplicates: LU-9439 Introduce an lnet systemd service