[LU-10859] Deadlock with heavy memory pressure

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.12.0, Lustre 2.10.4
• Type: Bug
• Priority: Minor
• Reporter: Wang Shilong (Inactive)
• Assignee: Wang Shilong (Inactive)
• Resolution: Fixed
• Labels: patch

Environment

RHEL7 Server
Lustre 2.7.x series

• Severity: 3

Description

On one Customer site, we hit following deadlock:

    Thread 1:

    ofd_object_punch

     osd_punch

      ldiskfs_truncate

       ldiskfs_inode_attach_jinode

         ...

         do_try_to_free_pages

          lu_cache_shrink

           mutex_lock -->try to hold @lu_sites_guard

    

    kswapd thread2:

    kthread

     shrink_slab

      lu_cache_shrink

        mutex_lock ---->hold already.

         ...

         dqget

          ldiskfs_acquire_dquot

           jbd2__journal_start-->blocked to wait for more credits.

    

    Thread3:

    kthread

     kjournald2

      jbd2_journal_commit_transaction-->blocked to wait Thread2 finished,

                                 since Thread1 add a handle into transaction.

    

    So deadlock happens because of Thread1 wait Thread2, Thread2 wait Thread3..

    but Thread3 wait Thread1....

    

    This problem still exists even we have switched @lu_sites_guard

    into a read/write lock, sine we hold write lock at lu_cahce_shrink().

    

    Fixed the problem by making ldiskfs_inode_attach_jinode() use

    GPF_NOFS.

 

Comments

[Gerrit Updater]

Wang Shilong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31806
Subject: LU-10859 ldiskfs: fix deadlock with heavy preassure
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 475c42e60f16281b9fd5f928559e2f0a4a4b6952

[Peter Jones]

Thanks wangshilong

 

[Bruno Faccini (Inactive)]

yes, thanks to create both this public ticket and Gerrit change in my place ...

[Wang Shilong (Inactive)]

Hi Bruno, sorry for that, we are a bit urgent for this issue.

[Andreas Dilger]

Bob, can you please add a follow-on patch for SLES, either using the same patch (if it applies cleanly) or new patches as needed, once this initial patch has passed review & testing.

[Wang Shilong (Inactive)]

Yup, Andreas, I should included SLES updates too, but it looks a bit hard for me to grab source codes for it, Ihara also reminded we need patch for ubuntu14+16 for master too.

 

[Peter Jones]

We only support ubuntu16 on master - not ubuntu14

[Chris Hunter (Inactive)]

Is this related to LU-9728 ?

[Wang Shilong (Inactive)]

Hello Chris,

  our ES3 have already included that patch, but still hit the problem, so it is different.

 

 

[Chris Hunter (Inactive)]

patch LU-9728 uses GFP_HIGHUSER for allocations instead of GFP_NOFS

kernel_patch filename is "GPF_NOFS" but alloc flag is GFP_NOFS
https://www.kernel.org/doc/gorman/html/understand/understand009.html

 

[Gerrit Updater]

Bob Glossman ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31825
Subject: LU-10859 ldiskfs: extend previous fix to SLES
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: c4956f2dff93c428b3040c1e03d08c14ec6232c8

[Bob Glossman (Inactive)]

Bob, can you please add a follow-on patch for SLES, either using the same patch (if it applies cleanly) or new patches as needed, once this initial patch has passed review & testing.

done

[Wang Shilong (Inactive)]

Hello chunteraa ,

Thanks for reminding, I refreshed the patch to fix that.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/31806/
Subject: LU-10859 ldiskfs: fix deadlock with heavy memory preassure
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0506e1bd6a6d5fafe7fc5e558aa1b75e456c2642

[Peter Jones]

Landed for 2.12

[Wang Shilong (Inactive)]

FYI, we'd better include this fix into b2_10 LTS branch.

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32058
Subject: LU-10859 ldiskfs: fix deadlock with heavy memory preassure
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: c5a9c83471aa5b6e0a593b7b99760e86c8311bee

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/32058/
Subject: LU-10859 ldiskfs: fix deadlock with heavy memory preassure
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 0595d92ad03ab9d975d599aad204d746aff991b3

[Chris Horn]

Bob Glossman ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31825
Subject: LU-10859 ldiskfs: extend previous fix to SLES
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: c4956f2dff93c428b3040c1e03d08c14ec6232c8

 

This patch was abandoned but its changes were never rolled into the primary patch for this ticket ( https://review.whamcloud.com/31806/). Should Bob's patch be revived?

 

Edit: Nevermind, I misread the patches