[LU-11939] ASSERTION( tgd->tgd_tot_granted >= ted->ted_grant ) on OSS

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.14.0, Lustre 2.12.8
• Affects Version/s: Lustre 2.12.0
• Environment: CentOS 7.6, 3.10.0-957.1.3.el7_lustre.x86_64
• Severity: 3
• Assignee: Mikhail Pershin
• Reporter: Stephane Thiell

Description

We just hit the following LBUG with Lustre 2.12 on an OSS (Fir). All clients are running Lustre 2.12 also.

[1708550.581820] LustreError: 123124:0:(tgt_grant.c:1079:tgt_grant_discard()) ASSERTION( tgd->tgd_tot_granted >= ted->ted_grant ) failed: fir-OST001b: tot_granted 50041695803 cli d5e4b60f-fe33-b991-7d48-5b8db7e07ab0/ffff926b10975c00 ted_grant -49152
[1708550.603611] LustreError: 123124:0:(tgt_grant.c:1079:tgt_grant_discard()) LBUG
[1708550.610923] Pid: 123124, comm: ll_ost00_019 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
[1708550.621180] Call Trace:
[1708550.623814]  [<ffffffffc0aa37cc>] libcfs_call_trace+0x8c/0xc0 [libcfs]
[1708550.630548]  [<ffffffffc0aa387c>] lbug_with_loc+0x4c/0xa0 [libcfs]
[1708550.636935]  [<ffffffffc0f220bc>] tgt_grant_discard+0x1dc/0x1e0 [ptlrpc]
[1708550.643892]  [<ffffffffc14c81d4>] ofd_obd_disconnect+0x74/0x220 [ofd]
[1708550.650541]  [<ffffffffc0e60157>] target_handle_disconnect+0xd7/0x450 [ptlrpc]
[1708550.658005]  [<ffffffffc0efeb77>] tgt_disconnect+0x37/0x140 [ptlrpc]
[1708550.664609]  [<ffffffffc0f0635a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
[1708550.671734]  [<ffffffffc0eaa92b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[1708550.679628]  [<ffffffffc0eae25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
[1708550.686136]  [<ffffffff8dcc1c31>] kthread+0xd1/0xe0
[1708550.691224]  [<ffffffff8e374c24>] ret_from_fork_nospec_begin+0xe/0x21
[1708550.697873]  [<ffffffffffffffff>] 0xffffffffffffffff
[1708550.703065] Kernel panic - not syncing: LBUG
[1708550.707509] CPU: 20 PID: 123124 Comm: ll_ost00_019 Kdump: loaded Tainted: G           OE  ------------   3.10.0-957.1.3.el7_lustre.x86_64 #1
[1708550.720273] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018
[1708550.728015] Call Trace:
[1708550.730645]  [<ffffffff8e361e41>] dump_stack+0x19/0x1b
[1708550.735962]  [<ffffffff8e35b550>] panic+0xe8/0x21f
[1708550.740937]  [<ffffffffc0aa38cb>] lbug_with_loc+0x9b/0xa0 [libcfs]
[1708550.747346]  [<ffffffffc0f220bc>] tgt_grant_discard+0x1dc/0x1e0 [ptlrpc]
[1708550.754230]  [<ffffffffc14c81d4>] ofd_obd_disconnect+0x74/0x220 [ofd]
[1708550.760880]  [<ffffffffc0e9ed81>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
[1708550.767783]  [<ffffffffc0ec3933>] ? req_capsule_server_pack+0x43/0xf0 [ptlrpc]
[1708550.775207]  [<ffffffffc0e60157>] target_handle_disconnect+0xd7/0x450 [ptlrpc]
[1708550.782634]  [<ffffffffc0efeb77>] tgt_disconnect+0x37/0x140 [ptlrpc]
[1708550.789194]  [<ffffffffc0f0635a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
[1708550.796272]  [<ffffffffc0edfa51>] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc]
[1708550.804022]  [<ffffffffc0aa3bde>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
[1708550.811281]  [<ffffffffc0eaa92b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[1708550.819142]  [<ffffffffc0ea77b5>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
[1708550.826110]  [<ffffffff8dcd67c2>] ? default_wake_function+0x12/0x20
[1708550.832548]  [<ffffffff8dccba9b>] ? __wake_up_common+0x5b/0x90
[1708550.838589]  [<ffffffffc0eae25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
[1708550.845068]  [<ffffffffc0ead760>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
[1708550.852636]  [<ffffffff8dcc1c31>] kthread+0xd1/0xe0
[1708550.857688]  [<ffffffff8dcc1b60>] ? insert_kthread_work+0x40/0x40
[1708550.863956]  [<ffffffff8e374c24>] ret_from_fork_nospec_begin+0xe/0x21
[1708550.870567]  [<ffffffff8dcc1b60>] ? insert_kthread_work+0x40/0x40
 

Issue Links

• duplicates: LU-12120 LustreError: 15069:0:(tgt_grant.c:561:tgt_grant_incoming()) LBUG