Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-7001

osp_sync.c: 1139: osp_sync_thread

Details

    • 3
    • 9223372036854775807

    Description

      The problem unfortunately is not solved, even with the patch http://review.whamcloud.com/#/c/15841/
      I propose to raise the topic again LU-6944
      The system restarts unexpectedly with errors
      Message from syslogd @ hard at Aug 13 8:56:38 ...
        kernel: LustreError: 2796: 0: (osp_sync.c: 1139: osp_sync_thread ()) ASSERTION (thread-> t_flags! = SVC_RUNNING) failed: 684 changes, 1137 in progress, 7 in flight
      Message from syslogd @ hard at Aug 13 8:56:38 ...
        kernel: LustreError: 2796: 0: (osp_sync.c: 1139: osp_sync_thread ()) LBUG

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-6944 LBUG: (osp_sync.c:1139:osp_sync_thread()) ASSERTION( thread->t_flags != SVC_RUNNING ) failed: 806 changes, 230 in progress, 7 in flight

          • Blocker - Blocks development and/or testing work, production could not run.
          • Resolved

          Activity

            [LU-7001] osp_sync.c: 1139: osp_sync_thread
            lixi Li Xi (Inactive) added a comment -

            Thank you Andreas for the information.

            Do you think it is possible to write a userspace tool to read as well as edit the llog files? I know that llog_reader is being changed, so hopefully, we will be able to at least dump the llog file. But since the llog files can be read locally from MDT/OST ldiskfs, maybe we can use a tool to remove wrong records mannually too?

            lixi Li Xi (Inactive) added a comment - Thank you Andreas for the information. Do you think it is possible to write a userspace tool to read as well as edit the llog files? I know that llog_reader is being changed, so hopefully, we will be able to at least dump the llog file. But since the llog files can be read locally from MDT/OST ldiskfs, maybe we can use a tool to remove wrong records mannually too?
            adilger Andreas Dilger added a comment -

            Li Xi, there are a couple of patches in flight that will repair or skip corrupted log records, but there may still be more types of corruption found on the future.

            adilger Andreas Dilger added a comment - Li Xi, there are a couple of patches in flight that will repair or skip corrupted log records, but there may still be more types of corruption found on the future.
            lixi Li Xi (Inactive) added a comment -

            Finally, we walk around this problem by remove the CATALOGS file. I am wondering whether there is anyway to chack and recover broken llogs records...

            lixi Li Xi (Inactive) added a comment - Finally, we walk around this problem by remove the CATALOGS file. I am wondering whether there is anyway to chack and recover broken llogs records...
            lixi Li Xi (Inactive) added a comment -

            We are hitting this issue repeatedly. I guess it will never recover unless we skip recovery or do something trick.

            Can we just remove the assertion? It seems this assertion is not proper, since the running thread has no idea when it will be requested to stop. Also, in osp_init0(), if ptlrpc_init_import() function returns a failure (ptlrpc_init_import() will not return any failure at least currently), it seems the assertion will fail. So this assertion looks dangerous.

            lixi Li Xi (Inactive) added a comment - We are hitting this issue repeatedly. I guess it will never recover unless we skip recovery or do something trick. Can we just remove the assertion? It seems this assertion is not proper, since the running thread has no idea when it will be requested to stop. Also, in osp_init0(), if ptlrpc_init_import() function returns a failure (ptlrpc_init_import() will not return any failure at least currently), it seems the assertion will fail. So this assertion looks dangerous.
            gerrit Gerrit Updater added a comment -

            Li Xi (lixi@ddn.com) uploaded a new patch: http://review.whamcloud.com/16335
            Subject: LU-7001 osp: remove improper assert of sync thread
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: ec59112060a4182b2c9b028ab38cf5f3dc3f8e8d

            gerrit Gerrit Updater added a comment - Li Xi (lixi@ddn.com) uploaded a new patch: http://review.whamcloud.com/16335 Subject: LU-7001 osp: remove improper assert of sync thread Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: ec59112060a4182b2c9b028ab38cf5f3dc3f8e8d
            Lexsoft Alex (Inactive) added a comment -

            How to enable logs to provide them? Operation delete files

            Lexsoft Alex (Inactive) added a comment - How to enable logs to provide them? Operation delete files
            adilger Andreas Dilger added a comment -

            Please provide the console logs with stack trace from the failing node. What operations are being done to trigger these errors?

            adilger Andreas Dilger added a comment - Please provide the console logs with stack trace from the failing node. What operations are being done to trigger these errors?

            People

              aboyko Alexander Boyko
              Lexsoft Alex (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: