[LU-6664] (ost_handler.c:1765:ost_blocking_ast()) Error -2 syncing data on lock cancel - Whamcloud Community JIRA

Details

Type: Bug
Resolution: Fixed
Priority: Minor
Fix Version/s: None
Affects Version/s: Lustre 2.5.3
Labels:
- llnl

Severity:
3
Rank (Obsolete):
9223372036854775807

Description

On all of our filesystems, the following error message is extremely common:

LustreError: 8746:0:(ost_handler.c:1776:ost_blocking_ast()) Error -2 syncing data on lock cancel

There is nothing else in the logs that gives any hint as to why this message is appearing.

Our filesystems all use osd-zfs, and we are currently running Lustre 2.5.3-5chaos (see github.com/chaos/lustre).

If this is a symptom of a bug, then please fix it. If this is not a symptom of a bug, then please stop scaring our system administrators with this message.

Attachments

Issue Links

is duplicated by

LU-7007 (ost_handler.c:1779:ost_blocking_ast()) Error -2 syncing data on lock cancel

Resolved

is related to

LU-7308 LustreError: 16956:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel

Resolved

is related to

LU-5805 tgt_recov blocked and "waking for gap in transno"

Resolved

Activity

[LU-6664] (ost_handler.c:1765:ost_blocking_ast()) Error -2 syncing data on lock cancel

Peter Jones added a comment - 31/Aug/15 9:14 PM

As per LLNL ok to close

Peter Jones added a comment - 31/Aug/15 9:14 PM As per LLNL ok to close

Amit (Inactive) added a comment - 24/Aug/15 4:37 PM

I too see ton's of these error messages: Any help in resolving them will be very helpful. I can provide any debug logs if required. It is consistently appearing in most of the OSS's.

LustreError: 7577:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel
LustreError: 25058:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel
LustreError: 1634:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel
LustreError: 1634:0:(ost_handler.c:1764:ost_blocking_ast()) Skipped 1 previous similar message
LustreError: 25058:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel
LustreError: 25058:0:(ost_handler.c:1764:ost_blocking_ast()) Skipped 2 previous similar messages
LustreError: 33552:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel

Thank you,
Amit

Amit (Inactive) added a comment - 24/Aug/15 4:37 PM I too see ton's of these error messages: Any help in resolving them will be very helpful. I can provide any debug logs if required. It is consistently appearing in most of the OSS's. LustreError: 7577:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel LustreError: 25058:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel LustreError: 1634:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel LustreError: 1634:0:(ost_handler.c:1764:ost_blocking_ast()) Skipped 1 previous similar message LustreError: 25058:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel LustreError: 25058:0:(ost_handler.c:1764:ost_blocking_ast()) Skipped 2 previous similar messages LustreError: 33552:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Thank you, Amit

Andreas Dilger added a comment - 21/Aug/15 6:00 PM

Oleg and I looked into this issue more closely, and the current patch doesn't really solve the problem, since the race is when the two destroy threads are getting and dropping the DLM lock, and not when the actual destroy is happening. In master, the equivalent function tgt_blocking_ast() already has a check for dt_object_exists() and skips the call into ofd_sync() that generates this message completely.

I think the right fix (for 2.5.x only) is to just skip this message for rc == -ENOENT as is already done in master.

Andreas Dilger added a comment - 21/Aug/15 6:00 PM Oleg and I looked into this issue more closely, and the current patch doesn't really solve the problem, since the race is when the two destroy threads are getting and dropping the DLM lock, and not when the actual destroy is happening. In master, the equivalent function tgt_blocking_ast() already has a check for dt_object_exists() and skips the call into ofd_sync() that generates this message completely. I think the right fix (for 2.5.x only) is to just skip this message for rc == -ENOENT as is already done in master.

Gerrit Updater added a comment - 15/Aug/15 2:32 AM

Bobi Jam (bobijam@hotmail.com) uploaded a new patch: http://review.whamcloud.com/15997
Subject: ~~LU-6664~~ ofd: LDLM lock should cover object destroy
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4575d04887bfd2a78a5a0340841d2da6ef23c165

Gerrit Updater added a comment - 15/Aug/15 2:32 AM Bobi Jam (bobijam@hotmail.com) uploaded a new patch: http://review.whamcloud.com/15997 Subject: LU-6664 ofd: LDLM lock should cover object destroy Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 4575d04887bfd2a78a5a0340841d2da6ef23c165

Andreas Dilger added a comment - 14/Aug/15 5:08 PM

Bobijam, can you please also make a version of your patch for master.

Andreas Dilger added a comment - 14/Aug/15 5:08 PM Bobijam, can you please also make a version of your patch for master.

Andreas Dilger added a comment - 06/Jul/15 8:55 PM

The LFSCK in 2.5.x does not check MDT<->OST consistency. That feature ("lctl lfsck_start -t layout") wasn't added until 2.6.0.

Andreas Dilger added a comment - 06/Jul/15 8:55 PM The LFSCK in 2.5.x does not check MDT<->OST consistency. That feature ("lctl lfsck_start -t layout") wasn't added until 2.6.0.

Oleg Drokin added a comment - 09/Jun/15 4:16 AM

Just as I was reviewing this patch (the comments are n the patch), I just remembered that LLNL had suspicions of double referenced objects in the past (~~LU-5648~~) where same object was potentially referenced twice (that was never confirmed, though).
So having objects owned by two files would most likely lead to this message too.

I wonder if lfsck in 2.5.4 is already in a good enough shape to be able to detect that.

Oleg Drokin added a comment - 09/Jun/15 4:16 AM Just as I was reviewing this patch (the comments are n the patch), I just remembered that LLNL had suspicions of double referenced objects in the past ( LU-5648 ) where same object was potentially referenced twice (that was never confirmed, though). So having objects owned by two files would most likely lead to this message too. I wonder if lfsck in 2.5.4 is already in a good enough shape to be able to detect that.

Zhenyu Xu added a comment - 06/Jun/15 4:17 AM

http://review.whamcloud.com/15167

commit message

LU-6664 ofd: LDLM lock should cover object destroy 

The exclusive PW lock protecting OST object destroy should be 
released after object destroy procedure. 

Quench error messages of object unavailability when trying to cancel 
a LDLM lock.

Zhenyu Xu added a comment - 06/Jun/15 4:17 AM http://review.whamcloud.com/15167 commit message LU-6664 ofd: LDLM lock should cover object destroy The exclusive PW lock protecting OST object destroy should be released after object destroy procedure. Quench error messages of object unavailability when trying to cancel a LDLM lock.

Christopher Morrone (Inactive) added a comment - 01/Jun/15 6:34 PM

No, it has not just undergone recovery. There are no other messages in the logs surrounding these ost_blocking_ast() messages.

Christopher Morrone (Inactive) added a comment - 01/Jun/15 6:34 PM No, it has not just undergone recovery. There are no other messages in the logs surrounding these ost_blocking_ast() messages.

Zhenyu Xu added a comment - 01/Jun/15 4:35 PM

Chris,

Does your system just undergo recovery before these messages appears?

Zhenyu Xu added a comment - 01/Jun/15 4:35 PM Chris, Does your system just undergo recovery before these messages appears?

People

Assignee:: Zhenyu Xu

Reporter:: Christopher Morrone (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 10 Start watching this issue

Dates

Created:: 29/May/15 7:20 PM

Updated:: 27/Jul/16 6:54 PM

Resolved:: 31/Aug/15 9:14 PM