[LU-6664] (ost_handler.c:1765:ost_blocking_ast()) Error -2 syncing data on lock cancel

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Affects Version/s: Lustre 2.5.3
• Labels: llnl
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Christopher Morrone (Inactive)

Description

On all of our filesystems, the following error message is extremely common:

LustreError: 8746:0:(ost_handler.c:1776:ost_blocking_ast()) Error -2 syncing data on lock cancel

There is nothing else in the logs that gives any hint as to why this message is appearing.

Our filesystems all use osd-zfs, and we are currently running Lustre 2.5.3-5chaos (see github.com/chaos/lustre).

If this is a symptom of a bug, then please fix it. If this is not a symptom of a bug, then please stop scaring our system administrators with this message.

Issue Links

• is duplicated by: LU-7007 (ost_handler.c:1779:ost_blocking_ast()) Error -2 syncing data on lock cancel
• is related to: LU-7308 LustreError: 16956:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel
• is related to: LU-5805 tgt_recov blocked and "waking for gap in transno"

Comments

[Amit (Inactive)]

I too see ton's of these error messages: Any help in resolving them will be very helpful. I can provide any debug logs if required. It is consistently appearing in most of the OSS's.

LustreError: 7577:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel
LustreError: 25058:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel
LustreError: 1634:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel
LustreError: 1634:0:(ost_handler.c:1764:ost_blocking_ast()) Skipped 1 previous similar message
LustreError: 25058:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel
LustreError: 25058:0:(ost_handler.c:1764:ost_blocking_ast()) Skipped 2 previous similar messages
LustreError: 33552:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel

Thank you,
Amit

[Andreas Dilger]

Oleg and I looked into this issue more closely, and the current patch doesn't really solve the problem, since the race is when the two destroy threads are getting and dropping the DLM lock, and not when the actual destroy is happening. In master, the equivalent function tgt_blocking_ast() already has a check for dt_object_exists() and skips the call into ofd_sync() that generates this message completely.

I think the right fix (for 2.5.x only) is to just skip this message for rc == -ENOENT as is already done in master.

[Gerrit Updater]

Bobi Jam ([e-mail]) uploaded a new patch: http://review.whamcloud.com/15997
Subject: LU-6664 ofd: LDLM lock should cover object destroy
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4575d04887bfd2a78a5a0340841d2da6ef23c165

[Andreas Dilger]

Bobijam, can you please also make a version of your patch for master.

[Andreas Dilger]

The LFSCK in 2.5.x does not check MDT<->OST consistency. That feature ("lctl lfsck_start -t layout") wasn't added until 2.6.0.

[Oleg Drokin]

Just as I was reviewing this patch (the comments are n the patch), I just remembered that LLNL had suspicions of double referenced objects in the past (LU-5648) where same object was potentially referenced twice (that was never confirmed, though).
So having objects owned by two files would most likely lead to this message too.

I wonder if lfsck in 2.5.4 is already in a good enough shape to be able to detect that.

[Zhenyu Xu]

http://review.whamcloud.com/15167

commit message

LU-6664 ofd: LDLM lock should cover object destroy 

The exclusive PW lock protecting OST object destroy should be 
released after object destroy procedure. 

Quench error messages of object unavailability when trying to cancel 
a LDLM lock. 

[Christopher Morrone (Inactive)]

No, it has not just undergone recovery. There are no other messages in the logs surrounding these ost_blocking_ast() messages.

[Zhenyu Xu]

Chris,

Does your system just undergo recovery before these messages appears?

[Oleg Drokin]

Bobi: This message is ENOENT when we are supposedly trying to flush data on lock cancel. But technically if we have a lock, the object should be there (the only exception I can think of is the actual object destroy is happening under the lock, so at the end of destroy the lock is still there and the object is not, but then there should be nothing to flush).
So can you please examine server side code for lock cancel to see if there are any possible races that could lead to this message.

[Peter Jones]

Bobijam

Could you please look into this issue?

Thanks

Peter