Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21245

OI scrub thread sleeps in D state and trips the hung task detector

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • Lustre 2.17.0
    • None
    • 3
    • 9223372036854775807

      The OI scrub kthread throttles itself and waits for the LFSCK consumer with wait_var_event(), which sleeps in TASK_UNINTERRUPTIBLE. When the consumer stalls for a while, e.g. while OSTs are unreachable after an MDS failover, the idle scrub thread stays in D state longer than hung_task_timeout_secs. khungtaskd then reports it, and a node with kernel.hung_task_panic set panics:

      INFO: task OI_scrub:2637226 blocked for more than 122 seconds.
      task:OI_scrub        state:D stack:0     pid:2637226 tgid:2637226 ppid:2      flags:0x00004000
      Call Trace:
       __schedule+0x231/0x540
       schedule+0x2c/0xb0
       osd_scrub_next.constprop.0+0x2fe/0x9e0 [osd_zfs]
       osd_scrub_main+0x738/0xc40 [osd_zfs]
       kthread+0x101/0x110
       ret_from_fork+0x28/0x50
      

      The trace is from osd-zfs during failover stress testing. osd-ldiskfs waits the same way in osd_scrub_exec(), osd_inode_iteration() and osd_scrub_main().

      These are idle waits: nothing is stuck while the consumer catches up. Sleeping in TASK_IDLE instead, with a TASK_IDLE variant of wait_var_event(), keeps khungtaskd quiet and leaves the wakeup logic as it is.

            panda Andrew Perepechko
            panda Andrew Perepechko
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: