[LU-7001] osp_sync.c: 1139: osp_sync_thread

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.4
• Labels: patch

Environment

https://build.hpdd.intel.com/job/lustre-reviews/34017/arch=x86_64,build_type=server,distro=el6.6,ib_stack=inkernel/
https://build.hpdd.intel.com/job/lustre-master/3137/arch=x86_64,build_type=server,distro=el6.6,ib_stack=inkernel/

• Severity: 3
• Assignee: Alexander Boyko
• Reporter: Alex (Inactive)

Description

The problem unfortunately is not solved, even with the patch http://review.whamcloud.com/#/c/15841/
I propose to raise the topic again LU-6944
The system restarts unexpectedly with errors
Message from syslogd @ hard at Aug 13 8:56:38 ...
  kernel: LustreError: 2796: 0: (osp_sync.c: 1139: osp_sync_thread ()) ASSERTION (thread-> t_flags! = SVC_RUNNING) failed: 684 changes, 1137 in progress, 7 in flight
Message from syslogd @ hard at Aug 13 8:56:38 ...
  kernel: LustreError: 2796: 0: (osp_sync.c: 1139: osp_sync_thread ()) LBUG

Issue Links

• is related to: LU-6944 LBUG: (osp_sync.c:1139:osp_sync_thread()) ASSERTION( thread->t_flags != SVC_RUNNING ) failed: 806 changes, 230 in progress, 7 in flight

Comments

[Gerrit Updater]

Alexander Boyko ([e-mail]) uploaded a new patch: https://review.whamcloud.com/26359
Subject: LU-7001 tests: check osp_sync_thread for wrapped llog
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8d779dab1025cd46ac44fc80f4dbd5ac85cba8a8

[Gerrit Updater]

Alexander Boyko ([e-mail]) uploaded a new patch: https://review.whamcloud.com/26132
Subject: LU-7001 osp: fix llog processing
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bcebaa04977761773d24fed0821c44fcbd5bef83

[Li Xi (Inactive)]

Thank you Andreas for the information.

Do you think it is possible to write a userspace tool to read as well as edit the llog files? I know that llog_reader is being changed, so hopefully, we will be able to at least dump the llog file. But since the llog files can be read locally from MDT/OST ldiskfs, maybe we can use a tool to remove wrong records mannually too?

[Andreas Dilger]

Li Xi, there are a couple of patches in flight that will repair or skip corrupted log records, but there may still be more types of corruption found on the future.

[Li Xi (Inactive)]

Finally, we walk around this problem by remove the CATALOGS file. I am wondering whether there is anyway to chack and recover broken llogs records...

[Li Xi (Inactive)]

We are hitting this issue repeatedly. I guess it will never recover unless we skip recovery or do something trick.

Can we just remove the assertion? It seems this assertion is not proper, since the running thread has no idea when it will be requested to stop. Also, in osp_init0(), if ptlrpc_init_import() function returns a failure (ptlrpc_init_import() will not return any failure at least currently), it seems the assertion will fail. So this assertion looks dangerous.

[Gerrit Updater]

Li Xi ([e-mail]) uploaded a new patch: http://review.whamcloud.com/16335
Subject: LU-7001 osp: remove improper assert of sync thread
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ec59112060a4182b2c9b028ab38cf5f3dc3f8e8d

[Alex (Inactive)]

How to enable logs to provide them? Operation delete files

[Andreas Dilger]

Please provide the console logs with stack trace from the failing node. What operations are being done to trigger these errors?