[LU-14603] OST: no handlers for opcode 0x16 Created: 09/Apr/21  Updated: 15/Jul/21  Resolved: 19/May/21

Status: Resolved
Project: Lustre
Component/s: None
Affects Version/s: Lustre 2.12.6
Fix Version/s: Lustre 2.12.7, Lustre 2.15.0

Type: Bug Priority: Minor
Reporter: Stephane Thiell Assignee: Peter Jones
Resolution: Fixed Votes: 0
Labels: None
Environment:

CentOS 7; ldiskfs/mdraid backend, clients are running Lustre 2.12, 2.13 and 2.14


Severity: 3
Rank (Obsolete): 9223372036854775807

 Description   

After upgrading a few clients on our Sherlock cluster from Lustre 2.13 to 2.14, we are seeing more and more "no handlers for opcode 0x16" errors on the servers (running 2.12.6).

From lustre_idl.h, I can see that ost opcode 0x16 is:

        OST_FALLOCATE  = 22,

So it's probably related to new fallocate support for Lustre/ldiskfs (LU-3606). I guess that's a good sign that codes are using fallocate!  

However, we only have a few clients running 2.14 in production, and I'm now worried that after we upgrade the whole cluster (1,800+ nodes), this will lead to an extremely high rate of these messages on the servers. Would it be possible to update Lustre 2.12 LTS to print those when debugging only? for better compatibility between 2.12 servers and 2.14 clients.

Apr 09 09:26:39 oak-io6-s2 kernel: LustreError: 91410:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST012d: no handlers for opcode 0x16
Apr 09 09:26:39 oak-io6-s2 kernel: LustreError: 91410:0:(tgt_handler.c:617:tgt_handler_find_check()) Skipped 6 previous similar messages
Apr 09 09:26:39 oak-io5-s1 kernel: LustreError: 187145:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST00f2: no handlers for opcode 0x16
Apr 09 09:26:39 oak-io5-s1 kernel: LustreError: 187145:0:(tgt_handler.c:617:tgt_handler_find_check()) Skipped 15 previous similar messages
Apr 09 09:26:39 oak-io6-s1 kernel: LustreError: 46667:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST0124: no handlers for opcode 0x16
Apr 09 09:26:39 oak-io6-s1 kernel: LustreError: 46667:0:(tgt_handler.c:617:tgt_handler_find_check()) Skipped 14 previous similar messages
Apr 09 09:26:40 oak-io5-s2 kernel: LustreError: 272212:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST00ed: no handlers for opcode 0x16
Apr 09 09:26:40 oak-io5-s2 kernel: LustreError: 272212:0:(tgt_handler.c:617:tgt_handler_find_check()) Skipped 6 previous similar messages
Apr 09 09:26:40 oak-io6-s2 kernel: LustreError: 14583:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST012b: no handlers for opcode 0x16
Apr 09 09:26:40 oak-io6-s2 kernel: LustreError: 14583:0:(tgt_handler.c:617:tgt_handler_find_check()) Skipped 2 previous similar messages
Apr 09 09:26:40 oak-io5-s1 kernel: LustreError: 186541:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST00fc: no handlers for opcode 0x16
Apr 09 09:26:40 oak-io5-s1 kernel: LustreError: 186541:0:(tgt_handler.c:617:tgt_handler_find_check()) Skipped 1 previous similar message
Apr 09 09:26:41 oak-io5-s2 kernel: LustreError: 272212:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST00fb: no handlers for opcode 0x16
Apr 09 09:26:42 oak-io4-s1 kernel: LustreError: 259539:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST0096: no handlers for opcode 0x16
Apr 09 09:26:42 oak-io2-s1 kernel: LustreError: 55897:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST0048: no handlers for opcode 0x16
Apr 09 09:26:42 oak-io2-s1 kernel: LustreError: 55897:0:(tgt_handler.c:617:tgt_handler_find_check()) Skipped 1 previous similar message
Apr 09 09:26:43 oak-io3-s2 kernel: LustreError: 229599:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST0071: no handlers for opcode 0x16
Apr 09 09:26:43 oak-io6-s1 kernel: LustreError: 210680:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST0128: no handlers for opcode 0x16
Apr 09 09:26:43 oak-io4-s1 kernel: LustreError: 62400:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST00ac: no handlers for opcode 0x16
Apr 09 09:26:43 oak-io5-s1 kernel: LustreError: 55531:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST00d0: no handlers for opcode 0x16
Apr 09 09:26:43 oak-io5-s1 kernel: LustreError: 55531:0:(tgt_handler.c:617:tgt_handler_find_check()) Skipped 3 previous similar messages
Apr 09 09:26:44 oak-io5-s2 kernel: LustreError: 272148:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST00f1: no handlers for opcode 0x16
Apr 09 09:26:44 oak-io5-s2 kernel: LustreError: 272148:0:(tgt_handler.c:617:tgt_handler_find_check()) Skipped 1 previous similar message
Apr 09 09:26:44 oak-io3-s1 kernel: LustreError: 1095:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST0076: no handlers for opcode 0x16
Apr 09 09:26:45 oak-io6-s2 kernel: LustreError: 150751:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST012f: no handlers for opcode 0x16
Apr 09 09:26:45 oak-io6-s2 kernel: LustreError: 150751:0:(tgt_handler.c:617:tgt_handler_find_check()) Skipped 1 previous similar message
Apr 09 09:26:45 oak-io3-s2 kernel: LustreError: 233125:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST008f: no handlers for opcode 0x16
Apr 09 09:26:45 oak-io6-s1 kernel: LustreError: 149884:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST0132: no handlers for opcode 0x16
Apr 09 09:26:45 oak-io6-s1 kernel: LustreError: 149884:0:(tgt_handler.c:617:tgt_handler_find_check()) Skipped 4 previous similar messages
Apr 09 09:26:46 oak-io4-s2 kernel: LustreError: 51373:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST00ad: no handlers for opcode 0x16
Apr 09 09:26:47 oak-io4-s1 kernel: LustreError: 259545:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST00b0: no handlers for opcode 0x16
Apr 09 09:26:48 oak-io2-s1 kernel: LustreError: 65922:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST004c: no handlers for opcode 0x16
Apr 09 09:26:49 oak-io5-s2 kernel: LustreError: 272129:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST00e5: no handlers for opcode 0x16
Apr 09 09:26:49 oak-io5-s2 kernel: LustreError: 272129:0:(tgt_handler.c:617:tgt_handler_find_check()) Skipped 4 previous similar messages
Apr 09 09:26:49 oak-io5-s1 kernel: LustreError: 53823:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST00fe: no handlers for opcode 0x16
Apr 09 09:26:49 oak-io5-s1 kernel: LustreError: 53823:0:(tgt_handler.c:617:tgt_handler_find_check()) Skipped 2 previous similar messages
Apr 09 09:26:51 oak-io6-s2 kernel: LustreError: 183056:0:(tgt_handler.c:617:tgt_handler_find_check()) oak-OST012b: no handlers for opcode 0x16
Apr 09 09:26:51 oak-io6-s2 kernel: LustreError: 183056:0:(tgt_handler.c:617:tgt_handler_find_check()) Skipped 3 previous similar messages


 Comments   
Comment by Andreas Dilger [ 09/Apr/21 ]

I don't think there is any problem to quiet this error message, preferably to only quiet this opcode. Are you planning to submit a patch?

Comment by Andreas Dilger [ 09/Apr/21 ]

It looks like there may be a similar issue with OST_SEEK=23, but that has an OBD_CONNECT2_SEEK flag that may prevent its use on older servers. It wouldn't hurt to exclude that opcode anyway.

Comment by Stephane Thiell [ 09/Apr/21 ]

I am not planning to submit a patch at this time. It's just for reporting at this point.

Comment by Gerrit Updater [ 11/Apr/21 ]

Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/43257
Subject: LU-14603 ptlrpc: quiet messages for unsupported opcodes
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ed519374deddeacbeef9ae2e680bd9928ebe4d51

Comment by Gerrit Updater [ 11/Apr/21 ]

Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/43260
Subject: LU-14603 ptlrpc: quiet messages for unsupported opcodes
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 6aba58dca74f14cf982b1c94cc05e6793eb37806

Comment by Gerrit Updater [ 11/May/21 ]

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/43257/
Subject: LU-14603 ptlrpc: quiet messages for unsupported opcodes
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a23767580aebfab7f093df562ac7598e85b71b3e

Comment by Gerrit Updater [ 16/May/21 ]

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/43260/
Subject: LU-14603 ptlrpc: quiet messages for unsupported opcodes
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 2cbdbb65a6a40434191ef6c8a26acc2595cffb53

Comment by Peter Jones [ 19/May/21 ]

Seems to have landed to both master and b2_12

Generated at Sat Feb 10 03:11:10 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.