Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-12017

Truncate vs setxattr deadlock with DoM

Details

    • 3
    • 9223372036854775807

    Description

      setxattr takes inode lock and sends reint to MDS.
      truncate takes MDS_INODELOCK_DOM lock and  wants to acquire inode lock.

      PID: 14942 TASK: ffff88007659cf10 CPU: 3 COMMAND: "truncate"
 #0 [ffff88011f397af8] __schedule at ffffffff816b3de4
 #1 [ffff88011f397b88] schedule_preempt_disabled at ffffffff816b5329
 #2 [ffff88011f397b98] __mutex_lock_slowpath at ffffffff816b30d7
 #3 [ffff88011f397bf0] mutex_lock at ffffffff816b24bf
 #4 [ffff88011f397c08] vvp_io_setattr_start at ffffffffc118993d [lustre]
 #5 [ffff88011f397c40] cl_io_start at ffffffffc06e7a25 [obdclass]
 #6 [ffff88011f397c68] cl_io_loop at ffffffffc06e9e01 [obdclass]
 #7 [ffff88011f397cd8] cl_setattr_ost at ffffffffc11847ef [lustre]
 #8 [ffff88011f397d28] ll_setattr_raw at ffffffffc11614d8 [lustre]
 #9 [ffff88011f397df0] ll_setattr at ffffffffc11617d3 [lustre]
#10 [ffff88011f397e00] notify_change at ffffffff81223bc4
#11 [ffff88011f397e48] do_truncate at ffffffff81203445
#12 [ffff88011f397ec0] vfs_truncate at ffffffff8120361c
#13 [ffff88011f397ef8] do_sys_truncate at ffffffff8120370c
#14 [ffff88011f397f40] sys_truncate at ffffffff812038de

      PID: 15194 TASK: ffff880077f18000 CPU: 1 COMMAND: "setfattr"
 #0 [ffff88011d33b8b8] __schedule at ffffffff816b3de4
 #1 [ffff88011d33b948] schedule at ffffffff816b4409
 #2 [ffff88011d33b958] schedule_timeout at ffffffff816b1ca4
 #3 [ffff88011d33ba00] ptlrpc_set_wait at ffffffffc09070a0 [ptlrpc]
 #4 [ffff88011d33baf0] ptlrpc_queue_wait at ffffffffc09074e3 [ptlrpc]
 #5 [ffff88011d33bb10] mdc_xattr_common at ffffffffc0b52186 [mdc]
 #6 [ffff88011d33bb90] mdc_setxattr at ffffffffc0b522de [mdc]
 #7 [ffff88011d33bbd0] lmv_setxattr at ffffffffc0872524 [lmv]
 #8 [ffff88011d33bc48] ll_xattr_set_common at ffffffffc1175b54 [lustre]
 #9 [ffff88011d33bcc8] ll_xattr_set_common_3_11 at ffffffffc11769ab [lustre]
#10 [ffff88011d33bcd8] generic_setxattr at ffffffff8122c2d8
#11 [ffff88011d33bd10] __vfs_setxattr_noperm at ffffffff8122cb45
#12 [ffff88011d33bd58] vfs_setxattr at ffffffff8122cd45
#13 [ffff88011d33bd98] setxattr at ffffffff8122ce7e
#14 [ffff88011d33bef0] sys_setxattr at ffffffff8122d177

      MDS locks are for different bits MDS_INODELOCK_UPDATE|MDS_INODELOCK_XATTR vs
      MDS_INODELOCK_DOM but they blocks each other if some blocking lock was present earlier because Lustre tries to grant only first lock in the waiting list.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-13456 Fix LU-12017

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-13467 truncate deadlock with DoM files

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-11285 don't stop on the first blocked lock in ldlm_reprocess_queue()

          • Major - Major loss of function.
          • Resolved

          Activity

            [LU-12017] Truncate vs setxattr deadlock with DoM
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-13456 [ LU-13456 ]
            spitzcor Cory Spitz made changes -
            Link New: This issue is related to LU-13467 [ LU-13467 ]
            spitzcor Cory Spitz added a comment - - edited

            This comment relates to LU-13467.

            spitzcor Cory Spitz added a comment - - edited This comment relates to LU-13467 .
            shadow Alexey Lyashkov added a comment - - edited

            Mike,

            i_mutex is useless for truncate case now. IO path uses an unlocked version with range lock for long time, truncate vs IO protected by separate mutex.

            shadow Alexey Lyashkov added a comment - - edited Mike, i_mutex is useless for truncate case now. IO path uses an unlocked version with range lock for long time, truncate vs IO protected by separate mutex.
            tappro Mikhail Pershin added a comment -

            Short update for everyone interested. After discussion with Vitaly Fertman there is conclusion about possibility of new deadlocks due to reverse locking for i_mutex and DOM LDLM lock in truncate path. That still can cause deadlocks in generic case like the following:

            th1: setattr took DOM lock, going to take inode lock
th2: wants some lock with DOM|XXX and stuck waiting the th1's DOM lock
th3: already have inode lock, enqueue some XXX bits and waits these bits on th2 waiting lock.

            Due to reverse order of taking inode lock and ldlm lock in th1 and th3, any th2 with DOM and XXX bits which are common with th3 bits can cause deadlock. Ideally that should be resolved by changing truncate for proper lock order, though that can be non-trivial. Until that we can just avoid combining DOM bit with other if they cannot be granted immediately. In most cases it is already so, the only exception is OPEN path, mdt_object_open_lock() particularly. In case of similar deadlocks it can be switched to 'trybits' mode, by setting mdt.*.dom_lock=trylock parameter

            tappro Mikhail Pershin added a comment - Short update for everyone interested. After discussion with Vitaly Fertman there is conclusion about possibility of new deadlocks due to reverse locking for i_mutex and DOM LDLM lock in truncate path. That still can cause deadlocks in generic case like the following: th1: setattr took DOM lock, going to take inode lock th2: wants some lock with DOM|XXX and stuck waiting the th1's DOM lock th3: already have inode lock, enqueue some XXX bits and waits these bits on th2 waiting lock. Due to reverse order of taking inode lock and ldlm lock in th1 and th3, any th2 with DOM and XXX bits which are common with th3 bits can cause deadlock. Ideally that should be resolved by changing truncate for proper lock order, though that can be non-trivial. Until that we can just avoid combining DOM bit with other if they cannot be granted immediately. In most cases it is already so, the only exception is OPEN path, mdt_object_open_lock() particularly. In case of similar deadlocks it can be switched to 'trybits' mode, by setting mdt.*.dom_lock=trylock parameter
            pjones Peter Jones made changes -
            Labels Original: LTS12 patch New: patch
            pjones Peter Jones made changes -
            Fix Version/s New: Lustre 2.12.3 [ 14418 ]
            gerrit Gerrit Updater added a comment -

            Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/35937/
            Subject: LU-12017 ldlm: DoM truncate deadlock
            Project: fs/lustre-release
            Branch: b2_12
            Current Patch Set:
            Commit: 671ece3104edd3342838e2c0eda350e72219dc97

            gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/35937/ Subject: LU-12017 ldlm: DoM truncate deadlock Project: fs/lustre-release Branch: b2_12 Current Patch Set: Commit: 671ece3104edd3342838e2c0eda350e72219dc97
            pjones Peter Jones added a comment -

            As long as there is some value to landing the original patch I would suggest moving remaining work to a distinct ticket.

            pjones Peter Jones added a comment - As long as there is some value to landing the original patch I would suggest moving remaining work to a distinct ticket.
            tappro Mikhail Pershin added a comment -

            The problem is not fixed fully, patch helps to avoid some situations with deadlock when IBITS of conflicting locks don't intersect but deadlock still exists and may happen in future when other IBITS combination will occur. The reason of deadlock on client side is not yet fixed.

            I'd like to keep this ticket opened with lowered severity and priority or create new ticket with link to this one to fix the reason of deadlock on client side

            tappro Mikhail Pershin added a comment - The problem is not fixed fully, patch helps to avoid some situations with deadlock when IBITS of conflicting locks don't intersect but deadlock still exists and may happen in future when other IBITS combination will occur. The reason of deadlock on client side is not yet fixed. I'd like to keep this ticket opened with lowered severity and priority or create new ticket with link to this one to fix the reason of deadlock on client side

            People

              tappro Mikhail Pershin
              askulysh Andriy Skulysh
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: