[LU-9628] LBUG (niobuf.c:773:ptl_send_rpc()) ASSERTION( (at_max == 0) || imp->imp_state != LUSTRE_IMP_FULL || (imp->imp_msghdr_flags & 0x1) || !(imp->imp_connect_data.ocd_connect_flags & 0x1000000ULL) ) failed:

• Type: Bug
• Resolution: Duplicate
• Priority: Critical
• Affects Version/s: Lustre 2.10.0, Lustre 2.12.0
• Labels: soak
• Environment: Soak cluster
• Severity: 3
• Assignee: WC Triage
• Reporter: Cliff White (Inactive)

Description

Soak client running soak test

Jun  9 12:08:56 soak-16 systemd-logind: Removed session 1396.
Jun  9 12:09:24 soak-16 kernel: Lustre: soaked-MDT0000-mdc-ffff880828cb2000: Connection restored to 192.168.1.108@o2ib10 (at 192.168.1.108@o2ib10)
Jun  9 12:09:24 soak-16 kernel: LustreError: 11-0: soaked-OST0003-osc-ffff880828cb2000: operation ldlm_enqueue to node 192.168.1.104@o2ib10 failed: rc = -19
Jun  9 12:09:24 soak-16 kernel: LustreError: 2947:0:(import.c:671:ptlrpc_connect_import()) already connecting
Jun  9 12:09:25 soak-16 kernel: LustreError: 11-0: soaked-OST0003-osc-ffff880828cb2000: operation ldlm_enqueue to node 192.168.1.104@o2ib10 failed: rc = -107
Jun  9 12:09:25 soak-16 kernel: LustreError: Skipped 12826 previous similar messages
Jun  9 12:09:25 soak-16 kernel: LustreError: 2947:0:(import.c:671:ptlrpc_connect_import()) already connecting
Jun  9 12:09:25 soak-16 kernel: LustreError: 2947:0:(import.c:671:ptlrpc_connect_import()) Skipped 13048 previous similar messages
Jun  9 12:09:26 soak-16 kernel: LustreError: 167-0: soaked-OST0003-osc-ffff880828cb2000: This client was evicted by soaked-OST0003; in progress operations using this service will fail.
Jun  9 12:09:26 soak-16 kernel: LustreError: 2960:0:(client.c:1189:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff8805e1cf3900 x1569656575292352/t0(0) o101->soaked-OST0003-osc-ffff880828cb2000@192.168.1.104@o2ib10:28/4 lens 328/400 e 0 to 1 dl 1497009748 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Jun  9 12:09:26 soak-16 kernel: LustreError: 2947:0:(client.c:1176:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff880624b19800 x1569656617848704/t0(0) o8->soaked-OST0003-osc-ffff880828cb2000@192.168.1.104@o2ib10:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
Jun  9 12:09:56 soak-16 kernel: LustreError: 2960:0:(niobuf.c:773:ptl_send_rpc()) ASSERTION( (at_max == 0) || imp->imp_state != LUSTRE_IMP_FULL || (imp->imp_msghdr_flags & 0x1) || !(imp->imp_connect_data.ocd_connect_flags & 0x1000000ULL) ) failed:
Jun  9 12:09:56 soak-16 kernel: LustreError: 2960:0:(niobuf.c:773:ptl_send_rpc()) LBUG

[74364.267656] LustreError: 217188:0:(niobuf.c:773:ptl_send_rpc()) ASSERTION( (at_max == 0) || imp->imp_state != LUSTRE_IMP_FULL || (imp->imp_msghdr_flags & 0x1) || !(imp->imp_connect_data.ocd_connect_flags & 0x1000000ULL) ) failed:
[74364.267659] LustreError: 217188:0:(niobuf.c:773:ptl_send_rpc()) LBUG
[74364.267661] Pid: 217188, comm: df
[74364.267661]
Call Trace:
[74364.267685]  [<ffffffffa08287ee>] libcfs_call_trace+0x4e/0x60 [libcfs]
[74364.267695]  [<ffffffffa082887c>] lbug_with_loc+0x4c/0xb0 [libcfs]
[74364.267747]  [<ffffffffa0b61c4f>] ptl_send_rpc+0xb1f/0xe60 [ptlrpc]
[74364.267815]  [<ffffffffa0b95203>] ? sptlrpc_req_refresh_ctx+0x153/0x900 [ptlrpc]
[74364.267856]  [<ffffffffa0b570f0>] ptlrpc_send_new_req+0x460/0xa60 [ptlrpc]
[74364.267894]  [<ffffffffa0b5bcc1>] ptlrpc_set_wait+0x3d1/0x900 [ptlrpc]
[74364.267906]  [<ffffffffa0e1a45d>] ? osc_statfs_async+0xfd/0x1e0 [osc]
[74364.267919]  [<ffffffffa0cd5e67>] ? lov_statfs_async+0xe7/0x730 [lov]
[74364.267928]  [<ffffffff811dd065>] ? kmem_cache_alloc_node_trace+0x125/0x220
[74364.267955]  [<ffffffffa0d7800d>] ll_statfs_internal+0x35d/0xf30 [lustre]
[74364.267959]  [<ffffffff812094ac>] ? lookup_fast+0xcc/0x2e0 
[74364.267963]  [<ffffffff8120bd83>] ? path_lookupat+0x83/0x7a0
[74364.267966]  [<ffffffff8120be16>] ? path_lookupat+0x116/0x7a0
[74364.267979]  [<ffffffffa0ebb798>] ? _nfs4_proc_statfs+0xc8/0xf0 [nfsv4]
[74364.268023]  [<ffffffffa0968519>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
[74364.268044]  [<ffffffffa0d78c64>] ll_statfs+0x84/0x180 [lustre]
[74364.268047]  [<ffffffff8120ed4d>] ? putname+0x3d/0x60
[74364.268052]  [<ffffffff812312b1>] statfs_by_dentry+0xa1/0x140
[74364.268054]  [<ffffffff8123136b>] vfs_statfs+0x1b/0xb0
[74364.268056]  [<ffffffff81231455>] user_statfs+0x55/0xa0
[74364.268059]  [<ffffffff812314c7>] SYSC_statfs+0x27/0x60
[74364.268062]  [<ffffffff812316ce>] SyS_statfs+0xe/0x10
[74364.268068]  [<ffffffff81696b09>] system_call_fastpath+0x16/0x1b
[74364.268069]
[74364.268070] Kernel panic - not syncing: LBUG

Crash dump is available on soak at /scratch/dumps/soak-16
vmcore-dmesg attached

Issue Links

• duplicates: LU-7558 niobuf.c:721:ptl_send_rpc() LASSERT(AT_OFF || imp_state != LUSTRE_IMP_FULL || imp_msghdr_flags & MSGHDR_AT_SUPPORT ...)