[LU-6831] The ticket for tracking all DNE2 bugs

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.8.0, Lustre 2.9.0
• Labels: dne2 llnl
• Severity: 3
• Assignee: Di Wang
• Reporter: Di Wang

Description

This ticket is for tracking all of DNE2 bugs.

Issue Links

• is related to: LU-6826 sanity test_71: No space left on device
• is related to: LU-6984 Failure to delete over a million files in a DNE2 directory.
• is related to: LU-7107 lfs_migrate() may fail in striped directories
• is related to: LU-7013 replay-single test_110c: FAIL: create striped dir failed
• is related to: LU-6284 FLD read is not swabbed correctly
• is related to: LU-7293 DNE2 perfomance analysis
• is related to: LU-7357 Add layout lock for striped directories.
• is related to: LU-5222 Some operation should never happened on the sub-stripe.
• is related to: LU-6297 Move rename is_subdir check from MDD to MDT
• is related to: LU-6819 LBUG ASSERTION( tdtd->tdtd_last_update_transno <= transno ) failed
• is related to: LU-6834 idx_array is overwritten in lod_prep_md_striped_create()
• is related to: LU-6837 MDS panic during 24 hours failover test.
• is related to: LU-6840 update memory reply data in DNE update replay
• is related to: LU-6880 recovery timeout during 24 hours failover test
• is related to: LU-7039 llog_osd.c:778:llog_osd_next_block()) ASSERTION( last_rec->lrh_index == tail->lrt_index ) failed:
• is related to: LU-7285 ASSERTION in check_for_next_transno
• is related to: LU-7318 OUT: dynamic reply buffer
• is related to: LU-5951 sanity test_39k: mtime is lost on close
• is related to: LU-6475 race between open and migration
• is related to: LU-6602 ASSERTION( rec->lrh_len <= 8192 ) failed
• is related to: LU-6741 OSP should use inline data for small requests
• is related to: LU-6896 update llog object is missing during recovery.
• is related to: LU-6916 unable to handle kernel NULL pointer dereference at (null) in tgt_client_free()
• is related to: LU-6977 lod_striped_it_next()) ASSERTION( next->do_index_ops != ((void *)0) )
• is related to: LU-6981 obd_last_committed is not updated in tgt_reply_data_init()
• is related to: LU-6998 Do not always lock stripe0 object in mdt_locks_slave()
• is related to: LU-6999 Disable xattr cache for remote mdt object
• is related to: LU-7091 refresh link after update linkea in mdd_migration_update_name()
• is related to: LU-5344 ldlm/ifind deadlock for striped directory
• is related to: LU-6586 "lctl conf_param testfs-MDT0001.mdc.active=0" does not deactivate MDC permanently
• is related to: LU-6838 update llog become too big before it is destroyed
• is related to: LU-6846 dt_record_write()) ASSERTION( dt->do_body_ops->dbo_write ) failed:
• is related to: LU-6852 MDS is evicted during 24-24 hours failover.
• is related to: LU-6859 osd_declare_object_destroy()) ASSERTION( inode )
• is related to: LU-6904 linkea prepare in mdt_reint_rename cause deadlock in 24 hours failover
• is related to: LU-6924 remote regular file are missing after recovery.
• is related to: LU-6928 Version mismatch during DNE replay
• is related to: LU-6997 migration should enqueue nonblock locks for objects in linkEA
• is related to: LU-7005 conf-sanity test_50i: lustre-MDT0001-osp-MDT0000:osp_attr_get update error
• is related to: LU-7008 osd_index_declare_ea_delete()) ASSERTION( inode )
• is related to: LU-6637 sanity-scrub test_9: Got speed 158, expected more than 168
• is related to: LU-6638 sanity-hsm test_37:request on 0x200000401:0x2d:0x0 is not SUCCEED on mds1
• is related to: LU-6639 sanity-lfsck test_2e: cpu softlock
• is related to: LU-6888 osp_update_request_create() should check return value
• is related to: LU-6906 During 24 hours DNE test, one of MDS can not be mounted after restarts.
• is related to: LU-6968 Update the whole header in llog_cancel_rec()
• is related to: LU-7009 Testing DNE on ZFS
• is related to: LU-7050 llog_skip_over skip the record by too little minimum record size.
• is related to: LU-7074 skip invalid linkEA entry in migration
• is related to: LU-7213 replay-single test_70d: mkdir: create stripe dir '/mnt/lustre/d70d.replay-single/test1' failed
• is related to: LU-6362 DNE2: add dt object and thandle check in OSP and OSD
• is related to: LU-6640 sanity-hsm test_21:Copytool failed to stop in 20s
• is related to: LU-3538 commit on share for cross-MDT operation.
• is related to: LU-6773 DNE2 Failover and recovery soak testing
• is related to: LU-7827 DNE3: automatically select MDT for lfs mkdir

Comments

[James A Simmons]

Yep. I'm testing it right now.

[Di Wang]

James: I just updated the patch http://review.whamcloud.com/#/c/16969/ please retry, thanks.

[James A Simmons]

a soft lockup happing on a spin lock -

Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.147894] BUG: soft lockup - CPU#0 stuck for 67s! [osp_up7-0:20904]
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.152894] BUG: soft lockup - CPU#1 stuck for 67s! [osp_up4-0:20901]

Dec 22 10:54:26 feral17.ccs.ornl.gItsov kernel: [ 793.152993] Pid: 20901, comm: osp_up4-0 Tainted: P --------------- 2.6.32
504.30.3.el6.head.x86_64 #1 Supermicro X8DT6/X8DT6
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.152996] RIP: 0010:[<ffffffff8152da2e>] [<ffffffff8152da2e>] _spin_lock+0x1e/0x30
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153004] RSP: 0018:ffff8817d793dda0 EFLAGS: 00000202
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153005] RAX: 0000000000000002 RBX: ffff8817d793dda0 RCX: 0000000000000000
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153007] RDX: 0000000000000003 RSI: ffff8817d793dea8 RDI: ffff880bcd9e8d50
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153009] RBP: ffffffff8100bc0e R08: ffff8817d793c000 R09: 00000000ffffffff
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153010] R10: 000000a7ad891c7a R11: 0000000000000001 R12: 0000000000000000
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153012] R13: 0000000000000000 R14: 0000000000015900 R15: 0000000000000000
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153014] FS: 0000000000000000(0000) GS:ffff880028220000(0000) knlGS:0000000000000000
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153016] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153017] CR2: 0000003d64205380 CR3: 0000000001a85000 CR4: 00000000000007e0
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153019] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153021] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153023] Process osp_up4-0 (pid: 20901, threadinfo ffff8817d793c000, task ffff881828342ab0)
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153024] Stack:
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153025] ffff8817d793ddf0 ffffffffa13ab4f9 0000000000000246 ffff8817d793dea8
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153027] <d> ffff8817d793ddf0 ffff881828342ab0 ffff880bcd9e8d40 ffff880bcd8d0800
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153030] <d> ffff8817d793de40 ffff8817d793dea8 ffff8817d793dee0 ffffffffa13b0074
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153032] Call Trace:
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153055] [<ffffffffa13ab4f9>] ? osp_get_next_request+0x29/0x1a0 [osp]
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153066] [<ffffffffa13b0074>] ? osp_send_update_thread+0x2f4/0x5b0 [osp]
Dec 22 10:54:26 feral17.ccs.ornl.gov kernel: [ 793.153071] [<ffffffff81064d00>] ? default_wake_function+0x0/0x20

[James A Simmons]

Updated my software stack and I'm seeing a lot of these on the OSS servers:

[94725.339746] Lustre: sultan-OST0004: already connected client sultan-MDT0000-mdtlov_UUID (at 10.37.248.155@o2ib1) with handle 0xb4b2e32f66f3ee41. Rejecting client with the same UUID trying to reconnect with handle 0x157ffaac64917bbd

Its seems to be only MDS1 having this. On that MDS the error message is:

95881.016995] LustreError: 137-5: sultan-MDT0001_UUID: not available for connect from 10.37.248.130@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server.

[James A Simmons]

Here is the full log from the node that was crashing this morning

Just to let you know the IOC_LMV_SETSTRIPE is no longer a issue.

[Di Wang]

could you please get the debug log(-1 level) on MDT0? I assume jsimmons is on MDT0 ? Thanks.

[James A Simmons]

Due to the lose of some of my MDS servers I attempted to create new striped directories today but instead I get this error every time.

lfs setdirstripe -c 4 /lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test
error on LL_IOC_LMV_SETSTRIPE '/lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test' (3): Invalid argument
error: setdirstripe: create stripe dir '/lustre/sultan/stf008/scratch/jsimmons/dne2_4_mds_md_test' failed

This happens even when I'm root.

[James A Simmons]

I attached my client logs to LU-6984.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/15720/
Subject: LU-6831 lmv: revalidate the dentry for striped dir
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a17909a92da74cb26fb9bf2824f968b2adf0897e

[James A Simmons]

Testing to see if the problem exist on directory striped across 8 MDS servers. Waiting for the results. I will push some log data soon for you.