[LU-15737] recovery-small: ll_ost00 - service thread hangs.

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0
• Affects Version/s: Lustre 2.15.0
• Severity: 3
• Assignee: Alexander Boyko
• Reporter: Maloo

Description

This issue was created by maloo for Cliff White <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/4784602c-4d77-42e2-919b-a194a0137d91

Test fails due to client timing out waiting on FULL state.
Appears to be due to thread hanging on one node:

[ 1927.612591] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == recovery-small test 26a: evict dead exports =========== 09:10:53 \(1649063453\)
[ 1928.007926] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 09:10:53 (1649063453)
[ 1974.360682] Lustre: ll_ost00_004: service thread pid 11008 was inactive for 43.187 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[ 1974.364718] Pid: 11008, comm: ll_ost00_004 4.18.0-348.2.1.el8_lustre.x86_64 #1 SMP Sun Apr 3 16:16:31 UTC 2022
[ 1974.366773] Call Trace TBD:
[ 1974.367500] [<0>] ldlm_completion_ast+0x7ac/0x900 [ptlrpc]
[ 1974.368739] [<0>] ldlm_cli_enqueue_local+0x307/0x860 [ptlrpc]
[ 1974.369924] [<0>] ofd_destroy_by_fid+0x235/0x4a0 [ofd]
[ 1974.370992] [<0>] ofd_destroy_hdl+0x263/0xa10 [ofd]
[ 1974.372045] [<0>] tgt_request_handle+0xc93/0x1a40 [ptlrpc]
[ 1974.373224] [<0>] ptlrpc_server_handle_request+0x323/0xbd0 [ptlrpc]
[ 1974.374523] [<0>] ptlrpc_main+0xc06/0x1560 [ptlrpc]
[ 1974.375548] [<0>] kthread+0x116/0x130
[ 1974.376336] [<0>] ret_from_fork+0x35/0x40
[ 1974.664781] Lustre: lustre-OST0005: haven't heard from client 0b624cdc-fcec-4fec-b859-486e2bb9b84b (at 10.240.40.108@tcp) in 47 seconds. I think it's dead, and I am evicting it. exp 000000003f573f19, cur 1649063501 expire 1649063471 last 1649063454
[ 2020.832883] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  recovery-small test_26a: @@@@@@ FAIL: lustre-OST0000-osc-ffff8f8645de7800 state is not FULL 
[ 2021.200277] Lustre: DEBUG MARKER: recovery-small test_26a: @@@@@@ FAIL: lustre-OST0000-osc-ffff8f8645de7800 state is not FULL

Issue Links

• is related to: LU-18392 recovery-small test_160: FAIL: destroying OST objects are blocked 8
• is related to: LU-10632 recovery-small test 26a fails with ‘client not evicted from OST’

Comments

[Alexander Boyko]

The question was about - sending/not sending blocking ast. Never mind, enqueue with LDLM_FL_BLOCK_NOWAIT will send blocking ast.

[Alex Zhuravlev]

not sure why did you mention LDLM_FL_SPECULATIVE, before the patch it was LDLM_FL_AST_DISCARD_DATA

[Alexander Boyko]

I think enqueue with LDLM_FL_SPECULATIVE does not send blocking ast, not a LDLM_FL_BLOCK_NOWAIT.

[Alex Zhuravlev]

hmm, will ldlm send blocking ast to client if LDLM_FL_BLOCK_NOWAIT is specified at enqueue? if not, then unlink-close will keep orphaned data in client's cache?

[Peter Jones]

Landed for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/55598/
Subject: LU-15737 ofd: don't block destroys
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 27f787daa7f25f1f14f8e041582ef969f87cd77a

[Gerrit Updater]

"Alexander Boyko <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/55598
Subject: LU-15737 ofd: don't block destroys
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f1dd736c3e4b0828aa7f932dbcba83284cf07472

[Alexander Boyko]

This hang looks close to issue which I'm working at. Probably the client opened unlink file, and wrote/red some bytes. Then it was evicted from MDT, and MDT sent destroy for a stripes, but OST had a lock on stripe, so destroy hanged until lock cancel or client eviction at OST.
The worst thing happens with group locks with similar case.

MDT sent 8 objects destroys to OST. All destroys hanged at

crash> p tk_core.timekeeper.xtime_sec
$1 = 1713871670
crash> bt 262904
PID: 262904   TASK: ffff9641d84ec740  CPU: 12   COMMAND: "ll_ost03_090"
 #0 [ffffb011e46cba68] __schedule at ffffffff9254e1d4
 #1 [ffffb011e46cbac8] schedule at ffffffff9254e648
 #2 [ffffb011e46cbad8] ldlm_completion_ast at ffffffffc14dc3ba [ptlrpc]
 #3 [ffffb011e46cbb68] ldlm_cli_enqueue_local at ffffffffc14da0f7 [ptlrpc]
 #4 [ffffb011e46cbbf0] ofd_destroy_by_fid at ffffffffc1a74de5 [ofd]
 #5 [ffffb011e46cbcd8] ofd_destroy_hdl at ffffffffc1a6bc33 [ofd]
 #6 [ffffb011e46cbd50] tgt_request_handle at ffffffffc1572453 [ptlrpc]
 #7 [ffffb011e46cbdd0] ptlrpc_server_handle_request at ffffffffc151e883 [ptlrpc]
 #8 [ffffb011e46cbe38] ptlrpc_main at ffffffffc15202f6 [ptlrpc]
 #9 [ffffb011e46cbf10] kthread at ffffffff91d043a6
#10 [ffffb011e46cbf50] ret_from_fork at ffffffff9260023f
crash> ps -m 262904
[7 13:42:01.163] [IN]  PID: 262904   TASK: ffff9641d84ec740  CPU: 12   COMMAND: "ll_ost03_090"
crash> 

MDT spent inflight limit and unable to send more.
OST processing req hanged because of GROUP_LOCK, clients did not release locks for 7 days. No destroys -> free space leakage at OST. 
A GROUP locking bases on application logic, and Lustre relies on it. Any way object destroying should not be blocked with such scenario.