[LU-6831] The ticket for tracking all DNE2 bugs

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.8.0, Lustre 2.9.0
• Labels: dne2 llnl
• Severity: 3
• Assignee: Di Wang
• Reporter: Di Wang

Description

This ticket is for tracking all of DNE2 bugs.

Issue Links

• is related to: LU-6826 sanity test_71: No space left on device
• is related to: LU-6984 Failure to delete over a million files in a DNE2 directory.
• is related to: LU-7107 lfs_migrate() may fail in striped directories
• is related to: LU-7013 replay-single test_110c: FAIL: create striped dir failed
• is related to: LU-6284 FLD read is not swabbed correctly
• is related to: LU-7293 DNE2 perfomance analysis
• is related to: LU-7357 Add layout lock for striped directories.
• is related to: LU-5222 Some operation should never happened on the sub-stripe.
• is related to: LU-6297 Move rename is_subdir check from MDD to MDT
• is related to: LU-6819 LBUG ASSERTION( tdtd->tdtd_last_update_transno <= transno ) failed
• is related to: LU-6834 idx_array is overwritten in lod_prep_md_striped_create()
• is related to: LU-6837 MDS panic during 24 hours failover test.
• is related to: LU-6840 update memory reply data in DNE update replay
• is related to: LU-6880 recovery timeout during 24 hours failover test
• is related to: LU-7039 llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec->lrh_index == tail->lrt_index ) failed:
• is related to: LU-7285 ASSERTION in check_for_next_transno
• is related to: LU-7318 OUT: dynamic reply buffer
• is related to: LU-5951 sanity test_39k: mtime is lost on close
• is related to: LU-6475 race between open and migration
• is related to: LU-6602 ASSERTION( rec->lrh_len <= 8192 ) failed
• is related to: LU-6741 OSP should use inline data for small requests
• is related to: LU-6896 update llog object is missing during recovery.
• is related to: LU-6916 unable to handle kernel NULL pointer dereference at (null) in tgt_client_free()
• is related to: LU-6977 lod_striped_it_next()) ASSERTION( next->do_index_ops != ((void *)0) )
• is related to: LU-6981 obd_last_committed is not updated in tgt_reply_data_init()
• is related to: LU-6998 Do not always lock stripe0 object in mdt_locks_slave()
• is related to: LU-6999 Disable xattr cache for remote mdt object
• is related to: LU-7091 refresh link after update linkea in mdd_migration_update_name()
• is related to: LU-5344 ldlm/ifind deadlock for striped directory
• is related to: LU-6586 "lctl conf_param testfs-MDT0001.mdc.active=0" does not deactivate MDC permanently
• is related to: LU-6838 update llog become too big before it is destroyed
• is related to: LU-6846 dt_record_write()) ASSERTION( dt->do_body_ops->dbo_write ) failed:
• is related to: LU-6852 MDS is evicted during 24-24 hours failover.
• is related to: LU-6859 osd_declare_object_destroy()) ASSERTION( inode )
• is related to: LU-6904 linkea prepare in mdt_reint_rename cause deadlock in 24 hours failover
• is related to: LU-6924 remote regular file are missing after recovery.
• is related to: LU-6928 Version mismatch during DNE replay
• is related to: LU-6997 migration should enqueue nonblock locks for objects in linkEA
• is related to: LU-7005 conf-sanity test_50i: lustre-MDT0001-osp-MDT0000:osp_attr_get update error
• is related to: LU-7008 osd_index_declare_ea_delete()) ASSERTION( inode )
• is related to: LU-6637 sanity-scrub test_9: Got speed 158, expected more than 168
• is related to: LU-6638 sanity-hsm test_37:request on 0x200000401:0x2d:0x0 is not SUCCEED on mds1
• is related to: LU-6639 sanity-lfsck test_2e: cpu softlock
• is related to: LU-6888 osp_update_request_create() should check return value
• is related to: LU-6906 During 24 hours DNE test, one of MDS can not be mounted after restarts.
• is related to: LU-6968 Update the whole header in llog_cancel_rec()
• is related to: LU-7009 Testing DNE on ZFS
• is related to: LU-7050 llog_skip_over skip the record by too little minimum record size.
• is related to: LU-7074 skip invalid linkEA entry in migration
• is related to: LU-7213 replay-single test_70d: mkdir: create stripe dir '/mnt/lustre/d70d.replay-single/test1' failed
• is related to: LU-6362 DNE2: add dt object and thandle check in OSP and OSD
• is related to: LU-6640 sanity-hsm test_21:Copytool failed to stop in 20s
• is related to: LU-3538 commit on share for cross-MDT operation.
• is related to: LU-6773 DNE2 Failover and recovery soak testing
• is related to: LU-7827 DNE3: automatically select MDT for lfs mkdir

Comments

[James A Simmons]

Updated my software stack and I'm seeing a lot of these on the OSS servers:

[94725.339746] Lustre: sultan-OST0004: already connected client sultan-MDT0000-mdtlov_UUID (at 10.37.248.155@o2ib1) with handle 0xb4b2e32f66f3ee41. Rejecting client with the same UUID trying to reconnect with handle 0x157ffaac64917bbd

Its seems to be only MDS1 having this. On that MDS the error message is:

95881.016995] LustreError: 137-5: sultan-MDT0001_UUID: not available for connect from 10.37.248.130@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server.

[James A Simmons]

Here is the full log from the node that was crashing this morning

Just to let you know the IOC_LMV_SETSTRIPE is no longer a issue.

[Di Wang]

could you please get the debug log(-1 level) on MDT0? I assume jsimmons is on MDT0 ? Thanks.

[James A Simmons]

Due to the lose of some of my MDS servers I attempted to create new striped directories today but instead I get this error every time.

lfs setdirstripe -c 4 /lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test
error on LL_IOC_LMV_SETSTRIPE '/lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test' (3): Invalid argument
error: setdirstripe: create stripe dir '/lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test' failed

This happens even when I'm root.

[James A Simmons]

I attached my client logs to LU-6984.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/15720/
Subject: LU-6831 lmv: revalidate the dentry for striped dir
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a17909a92da74cb26fb9bf2824f968b2adf0897e

[James A Simmons]

Testing to see if the problem exist on directory striped across 8 MDS servers. Waiting for the results. I will push some log data soon for you.

[Di Wang]

James: Any news for this -2 problem? Thanks

[Di Wang]

James: no, I did not see these errors? Could you please collect -1 debug log on client side, when you remove one of these files? thanks

[James A Simmons]

An update in my latest testing. I'm still seeing problems when creating 1 million+ files per directory. Clearing out the debug logs I see the problem is only on the client side. When running a application I see:

command line used: /lustre/sultan/stf008/scratch/jsimmons/mdtest -I 100000 -i 5 -d /lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test/shared_1000k_10
Path: /lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test
FS: 21.8 TiB Used FS: 0.2% Inodes: 58.7 Mi Used Inodes: 4.6%

10 tasks, 1000000 files/directories
aprun: Apid 3172: Caught signal Window changed, sending to application
08/03/2015 10:34:45: Process 0(nid00028): FAILED in create_remove_directory_tree, Unable to remove directory: No such file or directory
Rank 0 [Mon Aug 3 10:34:45 2015] [c0-0c0s1n2] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
_pmiu_daemon(SIGCHLD): [NID 00028] [c0-0c0s1n2] [Mon Aug 3 10:34:45 2015] PE RANK 0 exit signal Aborted
aprun: Apid 3172: Caught signal Interrupt, sending to application
_pmiu_daemon(SIGCHLD): [NID 00012] [c0-0c0s6n0] [Mon Aug 3 10:50:50 2015] PE RANK 7 exit signal Interrupt
_pmiu_daemon(SIGCHLD): [NID 00018] [c0-0c0s6n2] [Mon Aug 3 10:50:50 2015] PE RANK 9 exit signal Interrupt
_pmiu_daemon(SIGCHLD): [NID 00013] [c0-0c0s6n1] [Mon Aug 3 10:50:50 2015] PE RANK 8 exit signal Interrupt

After the test failed any attempt to remove the files create by these test fail. When I attempt to remove the files I see the following errors in dmesg.

LustreError: 5430:0:(llite_lib.c:2286:ll_prep_inode()) new_inode -fatal: rc -2
LustreError: 5451:0:(llite_lib.c:2286:ll_prep_inode()) new_inode -fatal: rc -2
LustreError: 5451:0:(llite_lib.c:2286:ll_prep_inode()) Skipped 7 previous similar messages
LustreError: 5451:0:(llite_lib.c:2286:ll_prep_inode()) new_inode -fatal: rc -2

DiWang have you seen these errors during your testing?

[Di Wang]

Sorry, it might be a mistakes, even the patch on this ticket is not landed.