Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-7640

stuck mdt thread required reboot of mds

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: Lustre 2.5.3
    • Fix Version/s: None
    • Labels:
      None
    • Severity:
      3
    • Rank (Obsolete):
      9223372036854775807

      Description

      MDS reported stuck mdt threads and dump stack trace

      <code>
      Jan 7 20:02:53 nbp8-mds1 kernel: LNet: Service thread pid 16286 was inactive for 464.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
      Jan 7 20:02:53 nbp8-mds1 kernel: LNet: Skipped 4 previous similar messages
      Jan 7 20:02:57 nbp8-mds1 kernel: Pid: 16286, comm: mdt02_020
      Jan 7 20:02:57 nbp8-mds1 kernel:
      Jan 7 20:02:57 nbp8-mds1 kernel: Call Trace:
      Jan 7 20:02:57 nbp8-mds1 kernel: [<ffffffffa04eee01>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
      Jan 7 20:02:57 nbp8-mds1 kernel: [<ffffffffa078af70>] ? ldlm_expired_completion_wait+0x0/0x360 [ptlrpc]
      Jan 7 20:02:57 nbp8-mds1 kernel: [<ffffffffa078f835>] ldlm_completion_ast+0x545/0x920 [ptlrpc]
      Jan 7 20:02:57 nbp8-mds1 kernel: [<ffffffff81061fe0>] ? default_wake_function+0x0/0x20
      Jan 7 20:02:57 nbp8-mds1 kernel: [<ffffffffa078ef00>] ldlm_cli_enqueue_local+0x1f0/0x5e0 [ptlrpc]
      Jan 7 20:02:57 nbp8-mds1 kernel: [<ffffffffa078f2f0>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
      Jan 7 20:02:57 nbp8-mds1 kernel: [<ffffffffa0e72de0>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa0e7cc06>] mdt_object_lock0+0x1b6/0xb30 [mdt]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa0e72de0>] ? mdt_blocking_ast+0x0/0x2a0 [mdt]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa078f2f0>] ? ldlm_completion_ast+0x0/0x920 [ptlrpc]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa0e7d644>] mdt_object_lock+0x14/0x20 [mdt]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa0e85b8e>] mdt_getattr_name_lock+0x8fe/0x19d0 [mdt]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa07df766>] ? __req_capsule_get+0x166/0x710 [ptlrpc]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa07ba7b4>] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa0e86ef9>] mdt_intent_getattr+0x299/0x480 [mdt]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa0e75c3e>] mdt_intent_policy+0x3ae/0x770 [mdt]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa076f2c5>] ldlm_lock_enqueue+0x135/0x980 [ptlrpc]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa0798ebb>] ldlm_handle_enqueue0+0x51b/0x10c0 [ptlrpc]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa0e76106>] mdt_enqueue+0x46/0xe0 [mdt]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa0e7aada>] mdt_handle_common+0x52a/0x1470 [mdt]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa0eb74a5>] mds_regular_handle+0x15/0x20 [mdt]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa07c80c5>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa04f08d5>] ? lc_watchdog_touch+0x65/0x170 [libcfs]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa07c0a69>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa07ca89d>] ptlrpc_main+0xafd/0x1780 [ptlrpc]
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
      Jan 7 20:03:00 nbp8-mds1 kernel: [<ffffffffa07c9da0>] ? ptlrpc_main+0x0/0x1780 [ptlrpc]
      Jan 7 20:03:01 nbp8-mds1 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
      Jan 7 20:03:01 nbp8-mds1 kernel:
      Jan 7 20:03:01 nbp8-mds1 kernel: LustreError: dumping log to /tmp/lustre-log.1452225773.16286
      <code>

      I am attaching /var/log/messages and lustre debug dump.

      The mds need to be rebooted to clear up the error state.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bobijam Zhenyu Xu
                Reporter:
                mhanafi Mahmoud Hanafi
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: