[LU-903] Race condition while get_attr after cancel_lru_locks and sysctl drop_caches

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.0, Lustre 1.8.6
• Fix Version/s: Lustre 2.4.0
• Type: Bug
• Priority: Minor
• Reporter: Artem Blagodarenko (Inactive)
• Assignee: Keith Mannthey (Inactive)
• Resolution: Fixed
• Labels: dentry, patch

Environment

SLES11

• Severity: 3
• Bugzilla ID: 24,555
• Epic: metadata

Description

Reproduction script is described in https://bugzilla.lustre.org/show_bug.cgi?id=24555
After some analysis the next bug picture drown:

1.
First thread makes lookup. Gets CR lock and terminates.
After that another thread a) make clear_lru cache b) sysctl that flush slab and another kernel
caches (dcache, icache, etc ...)

This results that sequence "shrink_dcache_memory -> "foreach_dentry_lru" -> prune_one_dentry ->
d_kill > d_iput()" is executed. After that ll_clear_inode executed that NULLed lock>l_ast_data.

2.
Some time after 1. another thread make get_attr on same inode. Gets another IBIT lock, but LOOKUP +
UPDATE.
Another client need cancel this lock, but 2 BL AST race arised. Second lock can't cancel first
lock because optimization, that was sown bellow and fist lock can't be canceled because its
inode == NULL.

Optimisation:

int ll_mdc_blocking_ast(struct ldlm_lock *lock, struct ldlm_lock_desc *desc,
                        void *data, int flag)
...
               if ((bits & MDS_INODELOCK_LOOKUP) &&
                    ll_have_md_lock(inode, MDS_INODELOCK_LOOKUP, LCK_MINMODE))
                        bits &= ~MDS_INODELOCK_LOOKUP;
                if ((bits & MDS_INODELOCK_UPDATE) &&
                    ll_have_md_lock(inode, MDS_INODELOCK_UPDATE, LCK_MINMODE))
                        bits &= ~MDS_INODELOCK_UPDATE;
                if ((bits & MDS_INODELOCK_OPEN) &&
                    ll_have_md_lock(inode, MDS_INODELOCK_OPEN, mode))
                        bits &= ~MDS_INODELOCK_OPEN;
...
if (inode->i_sb->s_root &&
        inode != inode->i_sb->s_root->d_inode &&
        (bits & MDS_INODELOCK_LOOKUP))
        ll_unhash_aliases(inode);
iput(inode);

Comments

[Artem Blagodarenko (Inactive)]

Patch is in review board.
http://review.whamcloud.com/#change,2054

[Artem Blagodarenko (Inactive)]

Second patch in set (2/2) in review board http://review.whamcloud.com/2409

[Build Master (Inactive)]

Integrated in lustre-reviews » x86_64,client,sles11,inkernel #4631
LU-903 ldlm: mdc lock inode reference moved to resource (Revision d94452871e7057e9bbaf1ba2025e2d54bef43ab0)

Result = SUCCESS
artem_blagodarenko : d94452871e7057e9bbaf1ba2025e2d54bef43ab0
Files :

• lustre/llite/llite_internal.h
• lustre/liblustre/super.c
• lustre/liblustre/llite_lib.h
• lustre/llite/llite_lib.c
• lustre/liblustre/dir.c
• lustre/liblustre/namei.c
• lustre/ldlm/ldlm_lock.c
• lustre/mdc/mdc_locks.c
• lustre/llite/dir.c
• lustre/ldlm/ldlm_lockd.c
• lustre/llite/namei.c
• lustre/mdc/mdc_request.c
• lustre/include/lustre_dlm.h

[Build Master (Inactive)]

Integrated in lustre-reviews » x86_64,client,el5,inkernel #4631
LU-903 ldlm: mdc lock inode reference moved to resource (Revision d94452871e7057e9bbaf1ba2025e2d54bef43ab0)

Result = SUCCESS
artem_blagodarenko : d94452871e7057e9bbaf1ba2025e2d54bef43ab0
Files :

• lustre/liblustre/super.c
• lustre/llite/llite_lib.c
• lustre/llite/namei.c
• lustre/ldlm/ldlm_lockd.c
• lustre/ldlm/ldlm_lock.c
• lustre/llite/llite_internal.h
• lustre/liblustre/namei.c
• lustre/include/lustre_dlm.h
• lustre/llite/dir.c
• lustre/liblustre/llite_lib.h
• lustre/mdc/mdc_request.c
• lustre/mdc/mdc_locks.c
• lustre/liblustre/dir.c

[Build Master (Inactive)]

Integrated in lustre-reviews » i686,server,el5,inkernel #4631
LU-903 ldlm: mdc lock inode reference moved to resource (Revision d94452871e7057e9bbaf1ba2025e2d54bef43ab0)

Result = SUCCESS
artem_blagodarenko : d94452871e7057e9bbaf1ba2025e2d54bef43ab0
Files :

• lustre/llite/llite_lib.c
• lustre/ldlm/ldlm_lock.c
• lustre/llite/namei.c
• lustre/include/lustre_dlm.h
• lustre/liblustre/dir.c
• lustre/mdc/mdc_locks.c
• lustre/ldlm/ldlm_lockd.c
• lustre/liblustre/super.c
• lustre/liblustre/llite_lib.h
• lustre/liblustre/namei.c
• lustre/llite/dir.c
• lustre/mdc/mdc_request.c
• lustre/llite/llite_internal.h

[Build Master (Inactive)]

Integrated in lustre-reviews » x86_64,client,el6,inkernel #4631
LU-903 ldlm: mdc lock inode reference moved to resource (Revision d94452871e7057e9bbaf1ba2025e2d54bef43ab0)

Result = SUCCESS
artem_blagodarenko : d94452871e7057e9bbaf1ba2025e2d54bef43ab0
Files :

• lustre/llite/llite_lib.c
• lustre/ldlm/ldlm_lockd.c
• lustre/mdc/mdc_request.c
• lustre/liblustre/llite_lib.h
• lustre/llite/dir.c
• lustre/liblustre/namei.c
• lustre/liblustre/super.c
• lustre/llite/llite_internal.h
• lustre/include/lustre_dlm.h
• lustre/llite/namei.c
• lustre/ldlm/ldlm_lock.c
• lustre/mdc/mdc_locks.c
• lustre/liblustre/dir.c

[Build Master (Inactive)]

Integrated in lustre-reviews » x86_64,server,el6,inkernel #4631
LU-903 ldlm: mdc lock inode reference moved to resource (Revision d94452871e7057e9bbaf1ba2025e2d54bef43ab0)

Result = SUCCESS
artem_blagodarenko : d94452871e7057e9bbaf1ba2025e2d54bef43ab0
Files :

• lustre/include/lustre_dlm.h
• lustre/liblustre/llite_lib.h
• lustre/llite/dir.c
• lustre/liblustre/namei.c
• lustre/liblustre/dir.c
• lustre/mdc/mdc_request.c
• lustre/liblustre/super.c
• lustre/llite/llite_internal.h
• lustre/ldlm/ldlm_lockd.c
• lustre/mdc/mdc_locks.c
• lustre/ldlm/ldlm_lock.c
• lustre/llite/llite_lib.c
• lustre/llite/namei.c

[Build Master (Inactive)]

Integrated in lustre-reviews » i686,client,el6,inkernel #4631
LU-903 ldlm: mdc lock inode reference moved to resource (Revision d94452871e7057e9bbaf1ba2025e2d54bef43ab0)

Result = SUCCESS
artem_blagodarenko : d94452871e7057e9bbaf1ba2025e2d54bef43ab0
Files :

• lustre/mdc/mdc_request.c
• lustre/mdc/mdc_locks.c
• lustre/liblustre/llite_lib.h
• lustre/llite/llite_internal.h
• lustre/llite/llite_lib.c
• lustre/liblustre/namei.c
• lustre/llite/namei.c
• lustre/liblustre/super.c
• lustre/ldlm/ldlm_lock.c
• lustre/ldlm/ldlm_lockd.c
• lustre/include/lustre_dlm.h
• lustre/llite/dir.c
• lustre/liblustre/dir.c

[Build Master (Inactive)]

Integrated in lustre-reviews » i686,server,el6,inkernel #4631
LU-903 ldlm: mdc lock inode reference moved to resource (Revision d94452871e7057e9bbaf1ba2025e2d54bef43ab0)

Result = SUCCESS
artem_blagodarenko : d94452871e7057e9bbaf1ba2025e2d54bef43ab0
Files :

• lustre/liblustre/dir.c
• lustre/mdc/mdc_request.c
• lustre/include/lustre_dlm.h
• lustre/llite/namei.c
• lustre/liblustre/namei.c
• lustre/liblustre/super.c
• lustre/liblustre/llite_lib.h
• lustre/llite/llite_lib.c
• lustre/llite/llite_internal.h
• lustre/mdc/mdc_locks.c
• lustre/ldlm/ldlm_lock.c
• lustre/llite/dir.c
• lustre/ldlm/ldlm_lockd.c

[Build Master (Inactive)]

Integrated in lustre-reviews » x86_64,server,el5,inkernel #4631
LU-903 ldlm: mdc lock inode reference moved to resource (Revision d94452871e7057e9bbaf1ba2025e2d54bef43ab0)

Result = SUCCESS
artem_blagodarenko : d94452871e7057e9bbaf1ba2025e2d54bef43ab0
Files :

• lustre/llite/namei.c
• lustre/ldlm/ldlm_lockd.c
• lustre/llite/llite_lib.c
• lustre/liblustre/llite_lib.h
• lustre/liblustre/namei.c
• lustre/llite/llite_internal.h
• lustre/ldlm/ldlm_lock.c
• lustre/liblustre/dir.c
• lustre/mdc/mdc_request.c
• lustre/liblustre/super.c
• lustre/include/lustre_dlm.h
• lustre/mdc/mdc_locks.c
• lustre/llite/dir.c

[Build Master (Inactive)]

Integrated in lustre-reviews » i686,client,el5,inkernel #4631
LU-903 ldlm: mdc lock inode reference moved to resource (Revision d94452871e7057e9bbaf1ba2025e2d54bef43ab0)

Result = SUCCESS
artem_blagodarenko : d94452871e7057e9bbaf1ba2025e2d54bef43ab0
Files :

• lustre/mdc/mdc_request.c
• lustre/mdc/mdc_locks.c
• lustre/ldlm/ldlm_lockd.c
• lustre/llite/namei.c
• lustre/include/lustre_dlm.h
• lustre/liblustre/namei.c
• lustre/llite/llite_internal.h
• lustre/liblustre/llite_lib.h
• lustre/ldlm/ldlm_lock.c
• lustre/llite/llite_lib.c
• lustre/llite/dir.c
• lustre/liblustre/super.c
• lustre/liblustre/dir.c

[Build Master (Inactive)]

Integrated in lustre-reviews » x86_64,client,sles11,inkernel #4632
LU-903 ldlm: inode references moved to resource for all locks (Revision 328ebce0e408ea08ca09265c36d216149e3aa92e)

Result = SUCCESS
artem_blagodarenko : 328ebce0e408ea08ca09265c36d216149e3aa92e
Files :

• lustre/mdc/mdc_locks.c
• lustre/include/obd.h
• lustre/liblustre/llite_lib.h
• lustre/lov/lov_obd.c
• lustre/llite/file.c
• lustre/mdc/mdc_request.c
• lustre/obdclass/lprocfs_status.c
• lustre/llite/llite_lib.c
• lustre/liblustre/namei.c
• lustre/liblustre/super.c
• lustre/liblustre/dir.c
• lustre/liblustre/rw.c
• lustre/osc/osc_request.c
• lustre/llite/llite_internal.h
• lustre/include/obd_class.h
• lustre/include/lustre_mds.h
• lustre/llite/dcache.c

[Build Master (Inactive)]

Integrated in lustre-reviews » x86_64,client,el5,inkernel #4632
LU-903 ldlm: inode references moved to resource for all locks (Revision 328ebce0e408ea08ca09265c36d216149e3aa92e)

Result = SUCCESS
artem_blagodarenko : 328ebce0e408ea08ca09265c36d216149e3aa92e
Files :

• lustre/mdc/mdc_request.c
• lustre/include/obd.h
• lustre/llite/file.c
• lustre/obdclass/lprocfs_status.c
• lustre/liblustre/super.c
• lustre/liblustre/namei.c
• lustre/liblustre/rw.c
• lustre/llite/llite_internal.h
• lustre/llite/llite_lib.c
• lustre/liblustre/llite_lib.h
• lustre/lov/lov_obd.c
• lustre/include/lustre_mds.h
• lustre/osc/osc_request.c
• lustre/llite/dcache.c
• lustre/mdc/mdc_locks.c
• lustre/include/obd_class.h
• lustre/liblustre/dir.c

[Build Master (Inactive)]

Integrated in lustre-reviews » x86_64,client,el6,inkernel #4632
LU-903 ldlm: inode references moved to resource for all locks (Revision 328ebce0e408ea08ca09265c36d216149e3aa92e)

Result = SUCCESS
artem_blagodarenko : 328ebce0e408ea08ca09265c36d216149e3aa92e
Files :

• lustre/obdclass/lprocfs_status.c
• lustre/include/obd.h
• lustre/liblustre/llite_lib.h
• lustre/llite/dcache.c
• lustre/mdc/mdc_request.c
• lustre/llite/file.c
• lustre/include/obd_class.h
• lustre/liblustre/dir.c
• lustre/include/lustre_mds.h
• lustre/mdc/mdc_locks.c
• lustre/liblustre/namei.c
• lustre/llite/llite_internal.h
• lustre/llite/llite_lib.c
• lustre/osc/osc_request.c
• lustre/liblustre/rw.c
• lustre/liblustre/super.c
• lustre/lov/lov_obd.c

[Build Master (Inactive)]

Integrated in lustre-reviews » x86_64,server,el6,inkernel #4632
LU-903 ldlm: inode references moved to resource for all locks (Revision 328ebce0e408ea08ca09265c36d216149e3aa92e)

Result = SUCCESS
artem_blagodarenko : 328ebce0e408ea08ca09265c36d216149e3aa92e
Files :

• lustre/llite/llite_internal.h
• lustre/mdc/mdc_locks.c
• lustre/llite/llite_lib.c
• lustre/include/lustre_mds.h
• lustre/liblustre/namei.c
• lustre/include/obd.h
• lustre/liblustre/super.c
• lustre/liblustre/llite_lib.h
• lustre/liblustre/dir.c
• lustre/llite/file.c
• lustre/include/obd_class.h
• lustre/osc/osc_request.c
• lustre/liblustre/rw.c
• lustre/obdclass/lprocfs_status.c
• lustre/llite/dcache.c
• lustre/lov/lov_obd.c
• lustre/mdc/mdc_request.c

[Build Master (Inactive)]

Integrated in lustre-reviews » i686,client,el6,inkernel #4632
LU-903 ldlm: inode references moved to resource for all locks (Revision 328ebce0e408ea08ca09265c36d216149e3aa92e)

Result = SUCCESS
artem_blagodarenko : 328ebce0e408ea08ca09265c36d216149e3aa92e
Files :

• lustre/mdc/mdc_request.c
• lustre/llite/dcache.c
• lustre/liblustre/namei.c
• lustre/llite/llite_internal.h
• lustre/obdclass/lprocfs_status.c
• lustre/include/obd.h
• lustre/liblustre/super.c
• lustre/llite/file.c
• lustre/include/lustre_mds.h
• lustre/llite/llite_lib.c
• lustre/lov/lov_obd.c
• lustre/liblustre/rw.c
• lustre/mdc/mdc_locks.c
• lustre/osc/osc_request.c
• lustre/liblustre/dir.c
• lustre/liblustre/llite_lib.h
• lustre/include/obd_class.h

[Build Master (Inactive)]

Integrated in lustre-reviews » i686,server,el6,inkernel #4632
LU-903 ldlm: inode references moved to resource for all locks (Revision 328ebce0e408ea08ca09265c36d216149e3aa92e)

Result = SUCCESS
artem_blagodarenko : 328ebce0e408ea08ca09265c36d216149e3aa92e
Files :

• lustre/osc/osc_request.c
• lustre/include/obd.h
• lustre/llite/file.c
• lustre/mdc/mdc_locks.c
• lustre/mdc/mdc_request.c
• lustre/liblustre/super.c
• lustre/llite/dcache.c
• lustre/lov/lov_obd.c
• lustre/llite/llite_lib.c
• lustre/include/obd_class.h
• lustre/llite/llite_internal.h
• lustre/liblustre/namei.c
• lustre/obdclass/lprocfs_status.c
• lustre/include/lustre_mds.h
• lustre/liblustre/rw.c
• lustre/liblustre/dir.c
• lustre/liblustre/llite_lib.h

[Build Master (Inactive)]

Integrated in lustre-reviews » i686,server,el5,inkernel #4632
LU-903 ldlm: inode references moved to resource for all locks (Revision 328ebce0e408ea08ca09265c36d216149e3aa92e)

Result = SUCCESS
artem_blagodarenko : 328ebce0e408ea08ca09265c36d216149e3aa92e
Files :

• lustre/osc/osc_request.c
• lustre/obdclass/lprocfs_status.c
• lustre/llite/llite_internal.h
• lustre/lov/lov_obd.c
• lustre/liblustre/dir.c
• lustre/liblustre/namei.c
• lustre/liblustre/super.c
• lustre/liblustre/llite_lib.h
• lustre/llite/dcache.c
• lustre/include/lustre_mds.h
• lustre/llite/file.c
• lustre/mdc/mdc_locks.c
• lustre/mdc/mdc_request.c
• lustre/include/obd.h
• lustre/include/obd_class.h
• lustre/llite/llite_lib.c
• lustre/liblustre/rw.c

[Build Master (Inactive)]

Integrated in lustre-reviews » x86_64,server,el5,inkernel #4632
LU-903 ldlm: inode references moved to resource for all locks (Revision 328ebce0e408ea08ca09265c36d216149e3aa92e)

Result = SUCCESS
artem_blagodarenko : 328ebce0e408ea08ca09265c36d216149e3aa92e
Files :

• lustre/obdclass/lprocfs_status.c
• lustre/include/obd.h
• lustre/liblustre/namei.c
• lustre/llite/dcache.c
• lustre/mdc/mdc_locks.c
• lustre/llite/file.c
• lustre/llite/llite_internal.h
• lustre/llite/llite_lib.c
• lustre/include/lustre_mds.h
• lustre/mdc/mdc_request.c
• lustre/osc/osc_request.c
• lustre/liblustre/dir.c
• lustre/liblustre/llite_lib.h
• lustre/include/obd_class.h
• lustre/lov/lov_obd.c
• lustre/liblustre/super.c
• lustre/liblustre/rw.c

[Build Master (Inactive)]

Integrated in lustre-reviews » i686,client,el5,inkernel #4632
LU-903 ldlm: inode references moved to resource for all locks (Revision 328ebce0e408ea08ca09265c36d216149e3aa92e)

Result = SUCCESS
artem_blagodarenko : 328ebce0e408ea08ca09265c36d216149e3aa92e
Files :

• lustre/mdc/mdc_locks.c
• lustre/include/lustre_mds.h
• lustre/liblustre/rw.c
• lustre/llite/file.c
• lustre/lov/lov_obd.c
• lustre/liblustre/dir.c
• lustre/llite/llite_internal.h
• lustre/liblustre/llite_lib.h
• lustre/obdclass/lprocfs_status.c
• lustre/include/obd.h
• lustre/liblustre/super.c
• lustre/liblustre/namei.c
• lustre/osc/osc_request.c
• lustre/llite/llite_lib.c
• lustre/include/obd_class.h
• lustre/llite/dcache.c
• lustre/mdc/mdc_request.c

[Andreas Dilger]

I'm all in favour of moving the inode reference onto the resource instead of on the lock. This patch should be landed to master first, not b1_8. Please submit a version of this patch against the master branch.

[Artem Blagodarenko (Inactive)]

Patch for master is uploaded http://review.whamcloud.com/2627

[Artem Blagodarenko (Inactive)]

The patch is reviewed. Is waiting for landing.

[Keith Mannthey (Inactive)]

I have reviewed the latest version of the patch.

[Nathan Rutman]

Xyratex-bug-id: MRP-269
Xyratex-bug-id: MRP-363

[Cory Spitz]

FYI, Cray has been using this patch for nearly a year.

[Peter Jones]

Landed for 2.4