[LU-1279] failure trying to mount two targets at the same time after boot

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.7.0, Lustre 2.5.4
• Affects Version/s: Lustre 2.4.0, Lustre 2.1.2
• Labels: llnl
• Environment: lustre-modules-2.1.1-2.6.32_220.4.2.el6_lustre.gcbb4fad.x86_64_gae03fc8.x86_64
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Brian Murrell (Inactive)

Description

After booting an OSS, two OSTs are mounted simultaneously. The mounts fail due to module loading failure:

Lustre: OBD class driver, http://wiki.whamcloud.com/
Lustre:         Lustre Version: 2.1.1
Lustre:         Build Version: jenkins-gae03fc8-PRISTINE-2.6.32-220.4.2.el6_lustre.gcbb4fad.x86_64
Lustre: Lustre LU module (ffffffffa0578c60).
fld: gave up waiting for init of module ptlrpc.
fld: Unknown symbol RQF_FLD_QUERY
fld: gave up waiting for init of module ptlrpc.
fld: Unknown symbol req_capsule_server_pack
fld: gave up waiting for init of module ptlrpc.
fld: Unknown symbol req_capsule_client_get
fld: gave up waiting for init of module ptlrpc.
fld: Unknown symbol ptlrpc_queue_wait
fld: gave up waiting for init of module ptlrpc.
fld: Unknown symbol req_capsule_fini
fld: gave up waiting for init of module ptlrpc.
fld: Unknown symbol req_capsule_init
fld: gave up waiting for init of module ptlrpc.
fld: Unknown symbol req_capsule_set
INFO: task hydra-agent:1590 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
hydra-agent   D 0000000000000000     0  1590      1 0x00000080
 ffff88003db09d68 0000000000000082 ffff88003d740a88 ffff88003bad0250
 ffff88003db09d68 ffffffff8113fb78 800000002c760065 0000000000000086
 ffff880037c1c678 ffff88003db09fd8 000000000000f4e8 ffff880037c1c678
Call Trace:
 [<ffffffff8113fb78>] ? vma_adjust+0x128/0x590
 [<ffffffff814ee35e>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff814ee1fb>] mutex_lock+0x2b/0x50
 [<ffffffff810aaafd>] m_start+0x1d/0x40
 [<ffffffff81198cc0>] seq_read+0x90/0x3f0
 [<ffffffff811dae0e>] proc_reg_read+0x7e/0xc0
 [<ffffffff81176cb5>] vfs_read+0xb5/0x1a0
 [<ffffffff810d4582>] ? audit_syscall_entry+0x272/0x2a0
 [<ffffffff81176df1>] sys_read+0x51/0x90
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task modprobe:1679 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe      D 0000000000000000     0  1679   1651 0x00000080
 ffff88002ed35aa8 0000000000000082 ffff88002ed35a58 ffffffff810097cc
 ffff88003ef260f8 0000000000000000 0000000000d35a68 ffff880002213b00
 ffff880037415a78 ffff88002ed35fd8 000000000000f4e8 ffff880037415a78
Call Trace:
 [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
 [<ffffffff814ecd0e>] ? thread_return+0x4e/0x760
 [<ffffffff814edb75>] schedule_timeout+0x215/0x2e0
 [<ffffffff8104c9e9>] ? __wake_up_common+0x59/0x90
 [<ffffffff814ed7f3>] wait_for_common+0x123/0x180
 [<ffffffff8105e7f0>] ? default_wake_function+0x0/0x20
 [<ffffffff8108b741>] ? __queue_work+0x41/0x50
 [<ffffffff814ed90d>] wait_for_completion+0x1d/0x20
 [<ffffffff81089c90>] call_usermodehelper_exec+0xe0/0xf0
 [<ffffffffa04966d2>] ? lnet_startup_lndnis+0x262/0x6f0 [lnet]
 [<ffffffff81089feb>] __request_module+0x18b/0x210
 [<ffffffffa0498e00>] ? lnet_parse_networks+0x90/0x7e0 [lnet]
 [<ffffffffa041aa13>] ? cfs_alloc+0x63/0x90 [libcfs]
 [<ffffffffa04966d2>] lnet_startup_lndnis+0x262/0x6f0 [lnet]
 [<ffffffffa041aa13>] ? cfs_alloc+0x63/0x90 [libcfs]
 [<ffffffffa0496c85>] LNetNIInit+0x125/0x1f0 [lnet]
 [<ffffffffa06aa13a>] ? init_module+0x0/0x597 [ptlrpc]
 [<ffffffffa05f1c89>] ptlrpc_ni_init+0x29/0x170 [ptlrpc]
 [<ffffffff8105e7f0>] ? default_wake_function+0x0/0x20
 [<ffffffffa05f2053>] ptlrpc_init_portals+0x13/0xd0 [ptlrpc]
 [<ffffffffa06aa13a>] ? init_module+0x0/0x597 [ptlrpc]
 [<ffffffffa06aa21a>] init_module+0xe0/0x597 [ptlrpc]
 [<ffffffff81096d15>] ? __blocking_notifier_call_chain+0x65/0x80
 [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0
 [<ffffffff810af4e1>] sys_init_module+0xe1/0x250
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
INFO: task modprobe:1688 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
modprobe      D 0000000000000000     0  1688   1687 0x00000080
 ffff88003d6d3eb8 0000000000000086 ffff88003d6d3e18 0000000000000082
 ffff88003d6d1ab8 ffff88003d6d3fd8 000000000000f4e8 ffff88003d6d1ac0
 ffff88003d6d1ab8 ffff88003d6d3fd8 000000000000f4e8 ffff88003d6d1ab8
Call Trace:
 [<ffffffff814f39dd>] ? kprobes_module_callback+0xdd/0x170
 [<ffffffff814ee35e>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff81096d15>] ? __blocking_notifier_call_chain+0x65/0x80
 [<ffffffff814ee1fb>] mutex_lock+0x2b/0x50
 [<ffffffff810af533>] sys_init_module+0x133/0x250
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
fld: gave up waiting for init of module ptlrpc.
fld: Unknown symbol req_capsule_server_get
fld: gave up waiting for init of module ptlrpc.
fld: Unknown symbol ptlrpc_at_set_req_timeout
type=1305 audit(1333395888.750:31878): auid=4294967295 ses=4294967295 op="remove rule" key=(null) list=4 res=1
type=1305 audit(1333395888.750:31879): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 res=1
readahead-collector: starting delayed service auditd
readahead-collector: sorting
readahead-collector: finished
fld: gave up waiting for init of module ptlrpc.
fld: Unknown symbol ptlrpc_request_alloc_pack
fld: gave up waiting for init of module ptlrpc.
fld: Unknown symbol RMF_FLD_OPC
fld: gave up waiting for init of module ptlrpc.
fld: Unknown symbol ptlrpc_request_set_replen
fld: gave up waiting for init of module ptlrpc.
fld: Unknown symbol RMF_FLD_MDFLD
fld: gave up waiting for init of module ptlrpc.
fld: Unknown symbol ptlrpc_req_finished
LustreError: 1679:0:(socklnd.c:2420:ksocknal_base_startup()) Can't spawn socknal scheduler[0]: -513
LustreError: 105-4: Error -100 starting up LNI tcp
LustreError: 1679:0:(events.c:728:ptlrpc_init_portals()) network initialisation failed

Issue Links

• duplicates: LU-4311 Mount sometimes fails with EIO on OSS with several mounts in parallel
• is duplicated by: LU-3975 Race loading ldiskfs with parallel mounts
• is duplicated by: LU-5961 Concurrent mount of ZFS targets fails when modules are not loaded
• is related to: LU-5159 Lustre MGS/MDT fails to start using initscripts using 2.4.2 based packages
• is related to: LU-2456 Dynamic LNet Config Main Development Work
• is related to: LU-3975 Race loading ldiskfs with parallel mounts

Comments

[Hongchao Zhang]

the patch is updated

[Bryon Neitzel (Inactive)]

This is needed for https://jira.hpdd.intel.com/browse/HYD-2311. Could we try to get this landed for 2.5 please?

[Hongchao Zhang]

yes, it can be put into other initscripts, rc.local is just one of the possible ones.

[Brian Murrell (Inactive)]

If loading modules at boot time is the solution, it should be done it's own initscript (or integrating into the distro's native boot-time module loading infrastructure) but not rc.local. rc.local should be left alone for the sysadmin to use, remove, etc. as he wishes.

[Hongchao Zhang]

if modules can't be loaded after boot time(see above), it can be fixed by adding "modprobe lustre; lctl net load" to rc.local with the patch

[Hongchao Zhang]

the patch to load individual LND modules at mount time is tracked at http://review.whamcloud.com/#/c/7024/

[Hongchao Zhang]

the patch which checks and load the specified LNDs in mount.lustre is underway.

[Andreas Dilger]

The patch to revert ptlrpc loading is underway, but someone else needs to make the patch to load the individual LND modules at mount time.

[Andreas Dilger]

I've pushed http://review.whamcloud.com/4750 to revert the patch http://review.whamcloud.com/4292 and http://review.whamcloud.com/4449 fixup. Hopefully that will allow http://review.whamcloud.com/4727 to pass testing.

[Andreas Dilger]

I'm planning to revert the "modprobe ptlrpc" line to avoid the "mount" problem.

It might be possible to fix this by adding explicit modprobe lines for the LNDs based on the MGS NID type as it is parsed by mount_lustre.c, like:

convert_hostnames()
:
:
        (void)system("modprobe %s > /dev/null 2>&1",
                     libcfs_lnd2modname(libcfs_str2lnd(s1));

This isn't a perfect solution if there is routing involved since only the MGS LND module would be loaded and that may not match the LND type used by the client, but that is somewhat a corner case. It is likely to fix the problem for many environments, and will at least not break things like the current patch does.

The issue blocking a proper fix is that the lnet modules are loaded based on modprobe options, but the module options are not in a specific file, so mount would have to hunt and peck to fix this. It definitely makes sense to have the simplified LNET config fix this properly by loading the actual modules needed on the client.

[James A Simmons]

Agree. On cray computes the modules are loaded at boot time only. After the boot is complete the image then removes all the modules and turns off modprobe. You can't modprobe lustre during run time. In that case a compute node could only mount lustre at boot and not during run time.