[LU-6831] The ticket for tracking all DNE2 bugs

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.8.0, Lustre 2.9.0
• Labels: dne2 llnl
• Severity: 3
• Assignee: Di Wang (Inactive)
• Reporter: Di Wang (Inactive)

Description

This ticket is for tracking all of DNE2 bugs.

Issue Links

• is related to: LU-6826 sanity test_71: No space left on device
• is related to: LU-6984 Failure to delete over a million files in a DNE2 directory.
• is related to: LU-7107 lfs_migrate() may fail in striped directories
• is related to: LU-7013 replay-single test_110c: FAIL: create striped dir failed
• is related to: LU-6284 FLD read is not swabbed correctly
• is related to: LU-7293 DNE2 perfomance analysis
• is related to: LU-7357 Add layout lock for striped directories.
• is related to: LU-5222 Some operation should never happened on the sub-stripe.
• is related to: LU-6297 Move rename is_subdir check from MDD to MDT
• is related to: LU-6819 LBUG ASSERTION( tdtd->tdtd_last_update_transno <= transno ) failed
• is related to: LU-6834 idx_array is overwritten in lod_prep_md_striped_create()
• is related to: LU-6837 MDS panic during 24 hours failover test.
• is related to: LU-6840 update memory reply data in DNE update replay
• is related to: LU-6880 recovery timeout during 24 hours failover test
• is related to: LU-7039 llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec->lrh_index == tail->lrt_index ) failed:
• is related to: LU-7285 ASSERTION in check_for_next_transno
• is related to: LU-7318 OUT: dynamic reply buffer
• is related to: LU-5951 sanity test_39k: mtime is lost on close
• is related to: LU-6475 race between open and migration
• is related to: LU-6602 ASSERTION( rec->lrh_len <= 8192 ) failed
• is related to: LU-6741 OSP should use inline data for small requests
• is related to: LU-6896 update llog object is missing during recovery.
• is related to: LU-6916 unable to handle kernel NULL pointer dereference at (null) in tgt_client_free()
• is related to: LU-6977 lod_striped_it_next()) ASSERTION( next->do_index_ops != ((void *)0) )
• is related to: LU-6981 obd_last_committed is not updated in tgt_reply_data_init()
• is related to: LU-6998 Do not always lock stripe0 object in mdt_locks_slave()
• is related to: LU-6999 Disable xattr cache for remote mdt object
• is related to: LU-7091 refresh link after update linkea in mdd_migration_update_name()
• is related to: LU-5344 ldlm/ifind deadlock for striped directory
• is related to: LU-6586 "lctl conf_param testfs-MDT0001.mdc.active=0" does not deactivate MDC permanently
• is related to: LU-6838 update llog become too big before it is destroyed
• is related to: LU-6846 dt_record_write()) ASSERTION( dt->do_body_ops->dbo_write ) failed:
• is related to: LU-6852 MDS is evicted during 24-24 hours failover.
• is related to: LU-6859 osd_declare_object_destroy()) ASSERTION( inode )
• is related to: LU-6904 linkea prepare in mdt_reint_rename cause deadlock in 24 hours failover
• is related to: LU-6924 remote regular file are missing after recovery.
• is related to: LU-6928 Version mismatch during DNE replay
• is related to: LU-6997 migration should enqueue nonblock locks for objects in linkEA
• is related to: LU-7005 conf-sanity test_50i: lustre-MDT0001-osp-MDT0000:osp_attr_get update error
• is related to: LU-7008 osd_index_declare_ea_delete()) ASSERTION( inode )
• is related to: LU-6637 sanity-scrub test_9: Got speed 158, expected more than 168
• is related to: LU-6638 sanity-hsm test_37:request on 0x200000401:0x2d:0x0 is not SUCCEED on mds1
• is related to: LU-6639 sanity-lfsck test_2e: cpu softlock
• is related to: LU-6888 osp_update_request_create() should check return value
• is related to: LU-6906 During 24 hours DNE test, one of MDS can not be mounted after restarts.
• is related to: LU-6968 Update the whole header in llog_cancel_rec()
• is related to: LU-7009 Testing DNE on ZFS
• is related to: LU-7050 llog_skip_over skip the record by too little minimum record size.
• is related to: LU-7074 skip invalid linkEA entry in migration
• is related to: LU-7213 replay-single test_70d: mkdir: create stripe dir '/mnt/lustre/d70d.replay-single/test1' failed
• is related to: LU-6362 DNE2: add dt object and thandle check in OSP and OSD
• is related to: LU-6640 sanity-hsm test_21:Copytool failed to stop in 20s
• is related to: LU-3538 commit on share for cross-MDT operation.
• is related to: LU-6773 DNE2 Failover and recovery soak testing
• is related to: LU-7827 DNE3: automatically select MDT for lfs mkdir

Comments

[James A Simmons]

For my DNE2 testing here is the list of patches I running against:

http://review.whamcloud.com/#/c/14346
http://review.whamcloud.com/#/c/14747
http://review.whamcloud.com/#/c/15594
http://review.whamcloud.com/#/c/15720
http://review.whamcloud.com/#/c/15576
http://review.whamcloud.com/#/c/15730
http://review.whamcloud.com/#/c/15692
http://review.whamcloud.com/#/c/15691
http://review.whamcloud.com/#/c/15682
http://review.whamcloud.com/#/c/15690
http://review.whamcloud.com/#/c/15721
http://review.whamcloud.com/#/c/15724
http://review.whamcloud.com/#/c/15728
http://review.whamcloud.com/#/c/15770

[James A Simmons]

Yes LU-6831 helped with the revalidate FID bug.

[Di Wang (Inactive)]

James: Please try this patch to see if it works? thanks. Unfortunately, I can not reproduce this problem locally.

[Gerrit Updater]

wangdi ([e-mail]) uploaded a new patch: http://review.whamcloud.com/15720
Subject: LU-6831 lmv: revalidate the dentry for striped dir
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 26133ac386693647ef3ee4480a6ce0cbb2773adb

[James A Simmons]

I see what is triggering the client evictions. I'm getting these errors on the clients:

LustreError: 10306:0:(lmv_intent.c:234:lmv_revalidate_slaves()) sultan-clilmv-ffff8803ea284c00: nlink 1 < 2 corrupt stripe 1 [0x2800013ba:0x84ad:0x0]:[0x2400013c8:0x84ad:0x0]
LustreError: 10306:0:(file.c:3104:ll_inode_revalidate_fini()) sultan: revalidate FID [0x2400013ce:0x3:0x0] error: rc = -5
LustreError: 10316:0:(lmv_intent.c:234:lmv_revalidate_slaves()) sultan-clilmv-ffff8803ea284c00: nlink 1 < 2 corrupt stripe 0 [0x2400013b3:0x14262:0x0]:[0x2400013b3:0x14262:0x0]
LustreError: 10863:0:(lmv_intent.c:234:lmv_revalidate_slaves()) sultan-clilmv-ffff8803ea284c00: nlink 1 < 2 corrupt stripe 0 [0x2400013b3:0x14262:0x0]:[0x2400013b3:0x14262:0x0]
LustreError: 10894:0:(lmv_intent.c:234:lmv_revalidate_slaves()) sultan-clilmv-ffff8803ea284c00: nlink 1 < 2 corrupt stripe 1 [0x2800013ab:0x14262:0x0]:[0x2400013b3:0x14262:0x0]
LustreError: 10894:0:(file.c:3104:ll_inode_revalidate_fini()) sultan: revalidate FID [0x2400013a4:0x5:0x0] error: rc = -5

[Di Wang (Inactive)]

James: What test did you run? Do you have the trace? I am not sure if there are such fixes. Thanks.

[James A Simmons]

Now I'm seeing clients get evicted during heavy meta data operations. Di Wang have you seen this behavior and does a patch exist to address this?

[James A Simmons]

Doing more testing I found that the patch from LU-6341 was not the source of the bug but it exposed the issue reported in LU-6713. Once I applied patch 15269 the problem went away.

[James A Simmons]

Nope. I found the source of the problems. It was the patch from LU-6341. Once I removed it the problems went away.

[Di Wang (Inactive)]

Strange, does your build include this patch http://review.whamcloud.com/#/c/15269/ ? If it does, please remove this one. and retry? thanks

[James A Simmons]

http://review.whamcloud.com/#/c/13990
http://review.whamcloud.com/#/c/15572
http://review.whamcloud.com/#/c/15576