[LU-4850] DNE Striped Directory - Changing default striping only works on MDT0

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.6.0
• Fix Version/s: Lustre 2.6.0
• Type: Bug
• Priority: Blocker
• Reporter: Patrick Farrell (Inactive)
• Assignee: Di Wang
• Resolution: Fixed
• Labels: dne2

Environment

Lustre master on CentOS clients and servers. Commit: Idcfb918e0d4e203ff0f9c6d838a68b1a204ee3bd (One commit behind current head.)

• Epic/Theme: dne
• Severity: 3

Description

I've been doing some simple testing of the new DNE striped directories, and I'm seeing some strangely inconsistent behavior. This system has 2 MDSes, and 2 MDTs per MDS. Clients and servers are all running master as described in "Environment".

Apologies in advance for the lengthy description, it takes a while to make the problem clear/show proof.

I'm seeing inconsistencies when I change the default stripe of a directory.

Create a striped directory, then set the default striping. Everything, at this point, works fine. All directories created get the correct striping:

[root@centclient02 centssm1]# lfs setdirstripe -c 4 striped_directory
[root@centclient02 centssm1]# lfs setdirstripe -c 4 -D striped_directory
[root@centclient02 centssm1]# mkdir -p striped_directory/{test1,test2,test3,test4,test5,test6,test7,test8,test9,test10,test11,test12}
[root@centclient02 centssm1]# cd striped_directory/; lfs getdirstripe test1 test2 test3 test4 test5 test6 test7 test8 test9 test10 test11 test12
test1
lmv_stripe_count: 4
lmv_stripe_offset: 0
mdtidx		 FID[seq:oid:ver]
     0		 [0x600000402:0x1d534:0x0]		
     1		 [0x640000400:0x1b:0x0]		
     2		 [0x680000400:0x1b:0x0]		
     3		 [0x6c0000400:0x1b:0x0]		

test2
lmv_stripe_count: 4
lmv_stripe_offset: 1
mdtidx		 FID[seq:oid:ver]
     1		 [0x640000401:0x21:0x0]		
     0		 [0x600000403:0x19:0x0]		
     2		 [0x680000403:0x19:0x0]		
     3		 [0x6c0000402:0x19:0x0]		

test3
lmv_stripe_count: 4
lmv_stripe_offset: 2
mdtidx		 FID[seq:oid:ver]
     2		 [0x680000402:0x1aa32:0x0]		
     0		 [0x600000404:0x39:0x0]		
     1		 [0x640000402:0x39:0x0]		
     3		 [0x6c0000403:0x39:0x0]		

test4
lmv_stripe_count: 4
lmv_stripe_offset: 3
mdtidx		 FID[seq:oid:ver]
     3		 [0x6c0000401:0x9e:0x0]		
     0		 [0x600000405:0x2d:0x0]		
     1		 [0x640000403:0x2d:0x0]		
     2		 [0x680000404:0x2d:0x0]		

All the directories are created with the correct striping information set.

The problems come in when I start changing the default striping. This only seems to work on MDT0.

Note the hash function (see output above) puts test1 on MDT0, test2 on MDT1, test3 on MDT2, and test4 on MDT3, which is handy for showing this bug.

If no subdirectories have been created before changing the default striping, everything works just fine:

[root@centclient02 centssm1]# lfs setdirstripe -c 4 striped_directory
[root@centclient02 centssm1]# lfs setdirstripe -c 4 -D striped_directory
[root@centclient02 centssm1]# lfs setdirstripe -c 1 -D striped_directory
[root@centclient02 centssm1]# mkdir -p striped_directory/{test1,test2,test3,test4}
[root@centclient02 centssm1]# cd striped_directory/; lfs getdirstripe test1 test2 test3 test4
test1
lmv_stripe_count: 1
lmv_stripe_offset: 0
mdtidx		 FID[seq:oid:ver]
     0		 [0x600000402:0x1d709:0x0]		

test2
lmv_stripe_count: 1
lmv_stripe_offset: 1
mdtidx		 FID[seq:oid:ver]
     1		 [0x640000401:0x95:0x0]		

test3
lmv_stripe_count: 1
lmv_stripe_offset: 2
mdtidx		 FID[seq:oid:ver]
     2		 [0x680000402:0x1aa35:0x0]		

test4
lmv_stripe_count: 1
lmv_stripe_offset: 3
mdtidx		 FID[seq:oid:ver]
     3		 [0x6c0000401:0xa1:0x0]		

[root@centclient02 striped_directory]# cd ..
[root@centclient02 centssm1]# rm -rf *

Changing the default striping after a subdirectory has been created only seems to work on MDT0.

Here's all four directories at once:

[root@centclient02 centssm1]# lfs setdirstripe -c 4 striped_directory
[root@centclient02 centssm1]# lfs setdirstripe -c 4 -D striped_directory
[root@centclient02 centssm1]# mkdir -p striped_directory/{test1,test2,test3,test4}
[root@centclient02 centssm1]# cd striped_directory/; lfs getdirstripe test1 test2 test3 test4
test1
lmv_stripe_count: 4
lmv_stripe_offset: 0
mdtidx		 FID[seq:oid:ver]
     0		 [0x600000402:0x1d735:0x0]		
     1		 [0x640000400:0x8f:0x0]		
     2		 [0x680000400:0x8f:0x0]		
     3		 [0x6c0000400:0x8f:0x0]		

test2
lmv_stripe_count: 4
lmv_stripe_offset: 1
mdtidx		 FID[seq:oid:ver]
     1		 [0x640000401:0xa5:0x0]		
     0		 [0x600000403:0x62:0x0]		
     2		 [0x680000403:0x62:0x0]		
     3		 [0x6c0000402:0x62:0x0]		

test3
lmv_stripe_count: 4
lmv_stripe_offset: 2
mdtidx		 FID[seq:oid:ver]
     2		 [0x680000402:0x1aa3c:0x0]		
     0		 [0x600000404:0x40:0x0]		
     1		 [0x640000402:0x40:0x0]		
     3		 [0x6c0000403:0x40:0x0]		

test4
lmv_stripe_count: 4
lmv_stripe_offset: 3
mdtidx		 FID[seq:oid:ver]
     3		 [0x6c0000401:0xa5:0x0]		
     0		 [0x600000405:0x33:0x0]		
     1		 [0x640000403:0x33:0x0]		
     2		 [0x680000404:0x33:0x0]		

[root@centclient02 striped_directory]# rm -rf *
[root@centclient02 striped_directory]# cd ..
[root@centclient02 centssm1]# lfs setdirstripe -c 1 -D striped_directory
[root@centclient02 centssm1]# mkdir -p striped_directory/{test1,test2,test3,test4}
[root@centclient02 centssm1]# cd striped_directory/; lfs getdirstripe test1 test2 test3 test4
test1
lmv_stripe_count: 1
lmv_stripe_offset: 0
mdtidx		 FID[seq:oid:ver]
     0		 [0x600000402:0x1d738:0x0]		

test2
lmv_stripe_count: 4
lmv_stripe_offset: 1
mdtidx		 FID[seq:oid:ver]
     1		 [0x640000401:0xa6:0x0]		
     0		 [0x600000403:0x63:0x0]		
     2		 [0x680000403:0x63:0x0]		
     3		 [0x6c0000402:0x63:0x0]		

test3
lmv_stripe_count: 4
lmv_stripe_offset: 2
mdtidx		 FID[seq:oid:ver]
     2		 [0x680000402:0x1aa3d:0x0]		
     0		 [0x600000404:0x41:0x0]		
     1		 [0x640000402:0x41:0x0]		
     3		 [0x6c0000403:0x41:0x0]		

test4
lmv_stripe_count: 4
lmv_stripe_offset: 3
mdtidx		 FID[seq:oid:ver]
     3		 [0x6c0000401:0xa6:0x0]		
     0		 [0x600000405:0x34:0x0]		
     1		 [0x640000403:0x34:0x0]		
     2		 [0x680000404:0x34:0x0]		

Here's the directories by themselves. Here's test1, on MDT0:

[root@centclient02 centssm1]# DIR=test1
[root@centclient02 centssm1]# lfs setdirstripe -c 4 striped_directory
[root@centclient02 centssm1]# lfs setdirstripe -c 4 -D striped_directory
[root@centclient02 centssm1]# mkdir -p striped_directory/$DIR
[root@centclient02 centssm1]# cd striped_directory/; lfs getdirstripe $DIR
test1
lmv_stripe_count: 4
lmv_stripe_offset: 0
mdtidx		 FID[seq:oid:ver]
     0		 [0x600000402:0x1d70d:0x0]		
     1		 [0x640000400:0x80:0x0]		
     2		 [0x680000400:0x80:0x0]		
     3		 [0x6c0000400:0x80:0x0]		

[root@centclient02 striped_directory]# rm -rf *
[root@centclient02 striped_directory]# cd ..
[root@centclient02 centssm1]# lfs setdirstripe -c 1 -D striped_directory
[root@centclient02 centssm1]# mkdir -p striped_directory/$DIR
[root@centclient02 centssm1]# cd striped_directory/; lfs getdirstripe $DIR
test1
lmv_stripe_count: 1
lmv_stripe_offset: 0
mdtidx		 FID[seq:oid:ver]
     0		 [0x600000402:0x1d70f:0x0]	

But here's test2, which goes on MDT1:

[root@centclient02 centssm1]# DIR=test2
[root@centclient02 centssm1]# lfs setdirstripe -c 4 striped_directory
[root@centclient02 centssm1]# lfs setdirstripe -c 4 -D striped_directory
[root@centclient02 centssm1]# mkdir -p striped_directory/$DIR
[root@centclient02 centssm1]# cd striped_directory/; lfs getdirstripe $DIR
test2
lmv_stripe_count: 4
lmv_stripe_offset: 1
mdtidx		 FID[seq:oid:ver]
     1		 [0x640000401:0x9d:0x0]		
     0		 [0x600000403:0x5a:0x0]		
     2		 [0x680000403:0x5a:0x0]		
     3		 [0x6c0000402:0x5a:0x0]		

[root@centclient02 striped_directory]# rm -rf *
[root@centclient02 striped_directory]# cd ..
[root@centclient02 centssm1]# lfs setdirstripe -c 1 -D striped_directory
[root@centclient02 centssm1]# mkdir -p striped_directory/$DIR
[root@centclient02 centssm1]# cd striped_directory/; lfs getdirstripe $DIR
test2
lmv_stripe_count: 4
lmv_stripe_offset: 1
mdtidx		 FID[seq:oid:ver]
     1		 [0x640000401:0x9e:0x0]		
     0		 [0x600000403:0x5b:0x0]		
     2		 [0x680000403:0x5b:0x0]		
     3		 [0x6c0000402:0x5b:0x0]		

We see the same results for test3 on MDT2, and test4 on MDT3.

Also note that it's creating any directory that goes on the particular MDT - It's not limited to being the same directory, as we can see from this example - test6 and test2 both go on MDT1:

[root@centclient02 centssm1]# DIR=test6
[root@centclient02 centssm1]# DIR2=test2
[root@centclient02 centssm1]# lfs setdirstripe -c 4 striped_directory
[root@centclient02 centssm1]# lfs setdirstripe -c 4 -D striped_directory
[root@centclient02 centssm1]# mkdir -p striped_directory/$DIR
[root@centclient02 centssm1]# cd striped_directory/; lfs getdirstripe $DIR
test6
lmv_stripe_count: 4
lmv_stripe_offset: 1
mdtidx		 FID[seq:oid:ver]
     1		 [0x640000401:0x9f:0x0]		
     0		 [0x600000403:0x5c:0x0]		
     2		 [0x680000403:0x5c:0x0]		
     3		 [0x6c0000402:0x5c:0x0]		

[root@centclient02 striped_directory]# rm -rf *
[root@centclient02 striped_directory]# cd ..
[root@centclient02 centssm1]# lfs setdirstripe -c 1 -D striped_directory
[root@centclient02 centssm1]# mkdir -p striped_directory/$DIR2
[root@centclient02 centssm1]# cd striped_directory/; lfs getdirstripe $DIR2
test2
lmv_stripe_count: 4
lmv_stripe_offset: 1
mdtidx		 FID[seq:oid:ver]
     1		 [0x640000401:0xa0:0x0]		
     0		 [0x600000403:0x5d:0x0]		
     2		 [0x680000403:0x5d:0x0]		
     3		 [0x6c0000402:0x5d:0x0]		

[root@centclient02 striped_directory]# cd ..
[root@centclient02 centssm1]# rm -rf *

And for additional confirmation, if I create test2 (which goes on MDT1) and then change default striping and create test3 (which goes on MDT2), we don't see the problem:

[root@centclient02 centssm1]# DIR=test2
[root@centclient02 centssm1]# DIR2=test3
[root@centclient02 centssm1]# lfs setdirstripe -c 4 striped_directory
[root@centclient02 centssm1]# lfs setdirstripe -c 4 -D striped_directory
[root@centclient02 centssm1]# mkdir -p striped_directory/$DIR
[root@centclient02 centssm1]# cd striped_directory/; lfs getdirstripe $DIR
test2
lmv_stripe_count: 4
lmv_stripe_offset: 1
mdtidx		 FID[seq:oid:ver]
     1		 [0x640000401:0xa1:0x0]		
     0		 [0x600000403:0x5e:0x0]		
     2		 [0x680000403:0x5e:0x0]		
     3		 [0x6c0000402:0x5e:0x0]		

[root@centclient02 striped_directory]# rm -rf *
[root@centclient02 striped_directory]# cd ..
[root@centclient02 centssm1]# lfs setdirstripe -c 1 -D striped_directory
[root@centclient02 centssm1]# mkdir -p striped_directory/$DIR2
[root@centclient02 centssm1]# cd striped_directory/; lfs getdirstripe $DIR2
test3
lmv_stripe_count: 1
lmv_stripe_offset: 2
mdtidx		 FID[seq:oid:ver]
     2		 [0x680000402:0x1aa3a:0x0]	

When the hash function is changed to "all_char", this behavior continues, but name to MDT mappings change, so we see it for different directories.

Issue Links

• is related to: LU-3531 DNE2: striped directory

Comments

[Di Wang]

Hello, Patrick, Thanks for testing, Could you please try http://review.whamcloud.com/#/c/9511, it may fix some of these issues here. Thanks.

[Patrick Farrell (Inactive)]

I built and installed that patch (git checkout of the patch) on the client and all four servers. After mounting on the client, I got the following crash when I "ls"'ed the root of the file system:
—
Message from syslogd@centclient02 at Apr 1 22:48:14 ...
kernel:LustreError: 15970:0:(mdc_locks.c:130:mdc_set_lock_data()) ASSERTION( old_inode->i_state & I_FREEING ) failed: Found existing inode ffff88011c301638/432345581441111874/0 state 0 in lock: setting data to ffff88011c301138/432345581441111874/100663300

Message from syslogd@centclient02 at Apr 1 22:48:14 ...
kernel:LustreError: 15970:0:(mdc_locks.c:130:mdc_set_lock_data()) LBUG
—

The system was not shut down correctly yesterday, so it's possible that's related. This happens every time I mount and do an ls on the patched client.

I took a crash dump with debug=-1 of the patched client, and also took debug logs on all four servers that cover the mount and attempted ls from the client.
That's on the WC FTP site here:

ftp.whamcloud.com/uploads/LU-4850/LU-4850_client_crash_server_logs_140402.tar.gz

Also:
With an unpatched client (running the version of master I described in the original problem report), I mounted the file system and was able to ls the root of the file system, and it worked until I tried to ls in the striped directory. Then, the primary MDS LBUGged.

Here's the stack trace of that MDS LBUG:
<0>LustreError: 8593:0:(osp_md_object.c:546:osp_it_load()) LBUG
<4>Pid: 8593, comm: mdt_rdpg00_001
<4>
<4>Call Trace:
<4> [<ffffffffa0b28895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
<4> [<ffffffffa0b28e97>] lbug_with_loc+0x47/0xb0 [libcfs]
<4> [<ffffffffa176ddcf>] osp_it_load+0x1f/0x20 [osp]
<4> [<ffffffffa169b950>] lod_striped_it_load+0xb0/0x1e0 [lod]
<4> [<ffffffffa0c8402d>] dt_index_walk+0xad/0x3d0 [obdclass]
<4> [<ffffffffa0b39581>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
<4> [<ffffffffa16ea110>] ? mdd_dir_page_build+0x0/0x210 [mdd]
<4> [<ffffffffa16ebefb>] mdd_readpage+0x38b/0x5a0 [mdd]
<4> [<ffffffffa15c8e2d>] mdt_readpage+0x47d/0x980 [mdt]
<4> [<ffffffffa0f189ac>] tgt_request_handle+0x23c/0xac0 [ptlrpc]
<4> [<ffffffffa0ec798a>] ptlrpc_main+0xd1a/0x1980 [ptlrpc]
<4> [<ffffffffa0ec6c70>] ? ptlrpc_main+0x0/0x1980 [ptlrpc]
<4> [<ffffffff8109aee6>] kthread+0x96/0xa0
<4> [<ffffffff8100c20a>] child_rip+0xa/0x20
<4> [<ffffffff8109ae50>] ? kthread+0x0/0xa0
<4> [<ffffffff8100c200>] ? child_rip+0x0/0x20

Let me know if you're interested in a crash dump from that occurrence, I could get debug logs there as well.

[Di Wang]

Patrick, Did you reformat the system before the test? I changed the disk-format of striped directory(according to our internal discussion) in the patch 9511. So you better reformat your system before the test. Sorry about the trouble. Thanks. I will check the debug log you uploaded.

[Patrick Farrell (Inactive)]

Ah, I was afraid of that. I will reformat before testing again. No need to look at the logs unless it interests you.

[Patrick Farrell (Inactive)]

OK, after reformatting and trying again (with patched client and servers), I've got a new crash for you.

This happened on the primary MDS when I did this sequence of commands:

lfs setdirstripe -c 4 striped_directory
lfs setdirstripe -c 4 -D striped_directory
mkdir -p striped_directory/test1
lfs getdirstripe striped_directory/test1
rm -rf striped_directory/*

lfs setdirstripe -c 1 -D striped_directory
mkdir -p striped_directory/test1

The crash on the first MDS(MDT0 & MDT1) (which is where test1 is created - lmv offset is 0) occurred when I did the last command in that sequence.

<1>BUG: unable to handle kernel NULL pointer dereference at (null)
<1>IP: [<ffffffffa0e8ca1a>] lod_dir_declare_xattr_set+0x25a/0x4b0 [lod]
[......]
<4>Call Trace:
<4> [<ffffffffa0e94072>] lod_dir_striping_create_internal+0x452/0x1450 [lod]
<4> [<ffffffffa0c516d8>] ? osd_declare_inode_qid+0x1e8/0x270 [osd_ldiskfs]
<4> [<ffffffffa030d4d8>] ? libcfs_log_return+0x28/0x40 [libcfs]
<4> [<ffffffffa0e95297>] lod_declare_object_create+0x227/0x390 [lod]
<4> [<ffffffffa0ed88c4>] mdd_declare_object_create_internal+0xb4/0x1e0 [mdd]
<4> [<ffffffffa0eea7a3>] mdd_create+0x813/0x18a0 [mdd]
<4> [<ffffffffa0dc0f83>] mdt_reint_create+0xac3/0xfa0 [mdt]
<4> [<ffffffffa0313581>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
<4> [<ffffffffa0dbc5e1>] mdt_reint_rec+0x41/0xe0 [mdt]
<4> [<ffffffffa0da1e13>] mdt_reint_internal+0x4c3/0x7c0 [mdt]
<4> [<ffffffffa0da269b>] mdt_reint+0x6b/0x120 [mdt]
<4> [<ffffffffa07059ac>] tgt_request_handle+0x23c/0xac0 [ptlrpc]
<4> [<ffffffffa06b498a>] ptlrpc_main+0xd1a/0x1980 [ptlrpc]
<4> [<ffffffffa06b3c70>] ? ptlrpc_main+0x0/0x1980 [ptlrpc]
<4> [<ffffffff8109aee6>] kthread+0x96/0xa0
<4> [<ffffffff8100c20a>] child_rip+0xa/0x20
<4> [<ffffffff8109ae50>] ? kthread+0x0/0xa0
<4> [<ffffffff8100c200>] ? child_rip+0x0/0x20

Dump of this with full debug on, as well as client logs and logs from the second MDS, is here:
ftp.whamcloud.com/uploads/LU-4850/LU-4850_mds_LBUG-140402.tar.gz

[Di Wang]

Hi, Patrick, I updated the patch again, please check. Btw: we do not support default stripe index yet, only default stripe count for now. Please try. Thanks.

[Patrick Farrell (Inactive)]

Di - Installed latest version from this afternoon, reformatted, started, mounted on client, then when I run:

lfs setdirstripe -c 4 striped_directory

The MDS crashes:

<0>LustreError: 2002:0:(fid_request.c:72:seq_client_rpc()) ASSERTION( exp != ((void *)0) && !IS_ERR(exp) ) failed:
<0>LustreError: 2002:0:(fid_request.c:72:seq_client_rpc()) LBUG
<4>Pid: 2002, comm: mdt00_004
<4>
<4>Call Trace:
<4> [<ffffffffa0303895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
<4> [<ffffffffa0303e97>] lbug_with_loc+0x47/0xb0 [libcfs]
<4> [<ffffffffa093a265>] seq_client_rpc+0x7a5/0x910 [fid]
<4> [<ffffffffa0314581>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
<4> [<ffffffffa093a7cf>] seq_client_alloc_seq+0x3ff/0x480 [fid]
<4> [<ffffffffa09399c3>] ? seq_fid_alloc_prep+0x43/0xc0 [fid]
<4> [<ffffffffa093accf>] seq_client_alloc_fid+0xdf/0x470 [fid]
<4> [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
<4> [<ffffffffa0f3d7de>] osp_fid_alloc+0xbe/0x100 [osp]
<4> [<ffffffffa0e943e7>] lod_declare_xattr_set_lmv+0x7c7/0x1ff0 [lod]
<4> [<ffffffffa0e95e50>] lod_dir_striping_create_internal+0x240/0x1460 [lod]
<4> [<ffffffffa0c536d8>] ? osd_declare_inode_qid+0x1e8/0x270 [osd_ldiskfs]
<4> [<ffffffffa030e4d8>] ? libcfs_log_return+0x28/0x40 [libcfs]
<4> [<ffffffffa0e97297>] lod_declare_object_create+0x227/0x390 [lod]
<4> [<ffffffffa0eda8c4>] mdd_declare_object_create_internal+0xb4/0x1e0 [mdd]
<4> [<ffffffffa0eec7f3>] mdd_create+0x813/0x18a0 [mdd]
<4> [<ffffffffa0dc2f83>] mdt_reint_create+0xac3/0xfa0 [mdt]
<4> [<ffffffffa0314581>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
<4> [<ffffffffa0dbe5e1>] mdt_reint_rec+0x41/0xe0 [mdt]
<4> [<ffffffffa0da3e13>] mdt_reint_internal+0x4c3/0x7c0 [mdt]
<4> [<ffffffffa0da469b>] mdt_reint+0x6b/0x120 [mdt]
<4> [<ffffffffa07069ac>] tgt_request_handle+0x23c/0xac0 [ptlrpc]
<4> [<ffffffffa06b598a>] ptlrpc_main+0xd1a/0x1980 [ptlrpc]
<4> [<ffffffffa06b4c70>] ? ptlrpc_main+0x0/0x1980 [ptlrpc]
<4> [<ffffffff8109aee6>] kthread+0x96/0xa0
<4> [<ffffffff8100c20a>] child_rip+0xa/0x20
<4> [<ffffffff8109ae50>] ? kthread+0x0/0xa0
<4> [<ffffffff8100c200>] ? child_rip+0x0/0x20

Dump with dk logs from all nodes involved is uploading here, should be up in a few minutes:

ftp.whamcloud.com
uploads/LU-4850/LU-4850-140403.tar.gz

[Di Wang]

Hmm, this seems because MDT did not check the intra connection (connections between MDTs) before it handle such requests. Sigh, we do this for the connection between MDT and OST. I will cook a patch, in the mean time, could you please wait a bit(say 2 or 3 mins) after setup system. Anyway I will cook a patch. Thanks again for testing.

[Di Wang]

Hello, Patrick, I create another patch based on 9511, http://review.whamcloud.com/#/c/9883/

[Patrick Farrell (Inactive)]

So, I applied 9883 on top of 9511, reinstalled everywhere, and reformatted.

For the first minute or two, setting the default striping on a directory didn't seem to affect its children, but after a minute or two, it did. [I was deleting and re-creating each time.] That now seems to be correct.

After that minute or two of time, the problem I originally reported with adjusting default striping seems to be gone and there were no crashes.

It sounds like fixing the delay on startup will be in a separate patch?

[Di Wang]

Hello, Patrick, 9883 suppose to fix "the delay", i.e. you should not see the lbug you mentioned above, even if you create striped dir immediately after startup. Yes, the default striping only works when you create new directory.

[Patrick Farrell (Inactive)]

Hmmm. Di, what I saw was when I created a striped directory with default striping set in it right after start up, and created directories in it, that didn't work. The child directories didn't get the default stripe setting from the parent.

I deleted all the directories and tried again, and got the same results. A minute or two later, I tried again, and it worked: The child directories got the striping settings from the parent.

I then removed the child directories and changed the striping default for the parent, and re-created the child directories. The new default striping pattern was applied correctly to the children.

However, I just tried to repeat this, to see if it was right, and the primary MDS crashed when trying to rm -rf * (deleting a striped directory and several children). I don't have logs from this, but I could grab a dump if needed:

LustreError: 2957:0:(osd_handler.c:2471:osd_object_destroy()) ASSERTION( osd_inode_unlinked(inode) || inode->i_nlink == 1 || inode->i_nlink == 2 ) failed:
<0>LustreError: 2957:0:(osd_handler.c:2471:osd_object_destroy()) LBUG
<4>Pid: 2957, comm: mdt_rdpg00_002
<4>
<4>Call Trace:
<4> [<ffffffffa0302895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
<4> [<ffffffffa0302e97>] lbug_with_loc+0x47/0xb0 [libcfs]
<4> [<ffffffffa0c29fd9>] osd_object_destroy+0x459/0x460 [osd_ldiskfs]
<4> [<ffffffffa0e90240>] lod_object_destroy+0x2c0/0x760 [lod]
<4> [<ffffffffa0edc6e0>] mdd_close+0x8e0/0xb80 [mdd]
<4> [<ffffffffa0dcf0b9>] mdt_mfd_close+0x4a9/0x1ba0 [mdt]
<4> [<ffffffffa0313581>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
<4> [<ffffffffa0dd2b73>] mdt_close+0x743/0xae0 [mdt]
<4> [<ffffffffa07059ac>] tgt_request_handle+0x23c/0xac0 [ptlrpc]
<4> [<ffffffffa06b498a>] ptlrpc_main+0xd1a/0x1980 [ptlrpc]
<4> [<ffffffffa06b3c70>] ? ptlrpc_main+0x0/0x1980 [ptlrpc]
<4> [<ffffffff8109aee6>] kthread+0x96/0xa0
<4> [<ffffffff8100c20a>] child_rip+0xa/0x20
<4> [<ffffffff8109ae50>] ? kthread+0x0/0xa0
<4> [<ffffffff8100c200>] ? child_rip+0x0/0x20
<4>
<0>Kernel panic - not syncing: LBUG
<4>Pid: 2957, comm: mdt_rdpg00_002 Not tainted 2.6.32.431.5.1.el6_lustre #1
<4>Call Trace:
<4> [<ffffffff81527983>] ? panic+0xa7/0x16f
<4> [<ffffffffa0302eeb>] ? lbug_with_loc+0x9b/0xb0 [libcfs]
<4> [<ffffffffa0c29fd9>] ? osd_object_destroy+0x459/0x460 [osd_ldiskfs]
<4> [<ffffffffa0e90240>] ? lod_object_destroy+0x2c0/0x760 [lod]
<4> [<ffffffffa0edc6e0>] ? mdd_close+0x8e0/0xb80 [mdd]
<4> [<ffffffffa0dcf0b9>] ? mdt_mfd_close+0x4a9/0x1ba0 [mdt]
<4> [<ffffffffa0313581>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
<4> [<ffffffffa0dd2b73>] ? mdt_close+0x743/0xae0 [mdt]
<4> [<ffffffffa07059ac>] ? tgt_request_handle+0x23c/0xac0 [ptlrpc]
<4> [<ffffffffa06b498a>] ? ptlrpc_main+0xd1a/0x1980 [ptlrpc]
<4> [<ffffffffa06b3c70>] ? ptlrpc_main+0x0/0x1980 [ptlrpc]
<4> [<ffffffff8109aee6>] ? kthread+0x96/0xa0
<4> [<ffffffff8100c20a>] ? child_rip+0xa/0x20
<4> [<ffffffff8109ae50>] ? kthread+0x0/0xa0
<4> [<ffffffff8100c200>] ? child_rip+0x0/0x20

[Di Wang]

Hmm, yes, please grub -1 dump log for me. Thanks. Just want to be clear, when you creating striped directory, no matter inherit stripes from default stripe or created by user specified stripes, MDT might adjust the stripe_count, in your case, it might be not enough MDTs have been fully started up yet. So you probably try this after startup

1. create a directory and set default stripe.
2. wait a few mins, then create sub-directories, to see whether they get correct stripes.

Thanks.

[Patrick Farrell (Inactive)]

I tried 1 and 2 above, and everything worked fine. (Waiting a few minutes before creating any directories.)

I was able to rm everything on the file system.

Then I stopped the file system and started it (our 'start' process is waiting for all the target mount commands to return), and mounted it on the client.

I immediately created some striped directories and set default striping and put some child directories in them. As expected, the default striping didn't work right away. I was able to delete those directories, then I created them again. Striping still didn't work.

I waited a few minutes, then created a separate striped directory and set default striping and created child directories. The child directories got the correct striping from the parent. When I tried to delete all of the file system contents, I got the LBUG I gave above on the primary MDS.

The -1 dk log dump for that is on the whamcloud FTP site here:

uploads/LU-4850/LU-4850-140403_nlink_lbug.tar.gz

Along with logs from the other MDS and client.

Just a heads up - I'm stopping work for the night, so I won't test anything else until at least tomorrow.

[Di Wang]

Hello, Patrick, Unfortunately, I can not reproduce the LBUG you mentioned here, and also the dump log information did not provide much information. Though I add another fixes based on 9511 and 9883, http://review.whamcloud.com/9890. Probably it can not fix the problem you saw, but I also add some debug information there. Could you please repeat these in your system.

Btw: before you start the test, could you please set

lctl set_param debug_mb=80
lctl set_param debug=-1

So provide more information in the dump. Thanks again for all of helps.

WangDi

[Patrick Farrell (Inactive)]

Di -

Sure. Just a heads up, I'm going to LUG this week, and I don't think I'll get to testing this before leaving for Miami. In that case, I might not test again until after LUG.

[Patrick Farrell (Inactive)]

With those three patches installed on client and server, I reformatted, set debug and debug_mb as requested (I used 1000 for debug_mb, rather than just 80) and ran the following commands about two-three minutes after startup:
lfs setdirstripe -c 4 striped_directory
lfs setdirstripe -c 4 -D striped_directory
mkdir -p striped_directory/

{test1,test2,test3,test4}
cd striped_directory/; lfs getdirstripe test1 test2 test3 test4
rm -rf *
cd ..
lfs setdirstripe -c 1 -D striped_directory
mkdir -p striped_directory/{test1,test2,test3,test4}

cd striped_directory/; lfs getdirstripe test1 test2 test3 test4
cd ..
rm -rf *

The primary MDS crashed when I ran the last command.

Here's the stack trace:
<0>LustreError: 14169:0:(osd_handler.c:2471:osd_object_destroy()) ASSERTION( osd_inode_unlinked(inode) || inode->i_nlink == 1 || inode->i_nlink == 2 ) failed:
<0>LustreError: 14169:0:(osd_handler.c:2471:osd_object_destroy()) LBUG
<4>Pid: 14169, comm: mdt_rdpg00_001
<4>
<4>Call Trace:
<4> [<ffffffffa089b895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
<4> [<ffffffffa089be97>] lbug_with_loc+0x47/0xb0 [libcfs]
<4> [<ffffffffa17290a9>] osd_object_destroy+0x459/0x460 [osd_ldiskfs]
<4> [<ffffffffa19c3490>] lod_object_destroy+0x360/0x800 [lod]
<4> [<ffffffffa1a0e6e0>] mdd_close+0x8e0/0xb80 [mdd]
<4> [<ffffffffa19000b9>] mdt_mfd_close+0x4a9/0x1ba0 [mdt]
<4> [<ffffffffa08ac581>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
<4> [<ffffffffa1903b73>] mdt_close+0x743/0xae0 [mdt]
<4> [<ffffffffa12b19ac>] tgt_request_handle+0x23c/0xac0 [ptlrpc]
<4> [<ffffffffa126098a>] ptlrpc_main+0xd1a/0x1980 [ptlrpc]
<4> [<ffffffffa125fc70>] ? ptlrpc_main+0x0/0x1980 [ptlrpc]
<4> [<ffffffff8109aee6>] kthread+0x96/0xa0
<4> [<ffffffff8100c20a>] child_rip+0xa/0x20
<4> [<ffffffff8109ae50>] ? kthread+0x0/0xa0
<4> [<ffffffff8100c200>] ? child_rip+0x0/0x20

Dump and logs from client and other servers will be here shortly:

ftp.whamcloud.com
uploads/LU-4850/LU-4850_140407.tar.gz

[Di Wang]

Patrick, thanks for testing. According to the debug log, it seems MDT mis-regarded this striped_directory as non-striped directory. I can not figure out the reason without client side debug log. Unfortunately, even I repeat what you did, I can not reproduce the problem here. Could you also please add patch http://review.whamcloud.com/#/c/9862/ in your build ? And collect the debug log(with -1 level) on client side as well? Thanks again for all of help.

My test

echo 0 > /proc/sys/lnet/panic_on_lbug 
+ echo 0
echo -1 > /proc/sys/lnet/debug
+ echo -1
echo 50 > /proc/sys/lnet/debug_mb
+ echo 50
cd /mnt/lustre
+ cd /mnt/lustre

/home/work/lustre-release/lustre/utils/lfs setdirstripe -i 1 -c4 striped_directory
+ /home/work/lustre-release/lustre/utils/lfs setdirstripe -i 1 -c4 striped_directory
 /home/work/lustre-release/lustre/utils/lfs setdirstripe -c 4 -D striped_directory
+ /home/work/lustre-release/lustre/utils/lfs setdirstripe -c 4 -D striped_directory
mkdir -p striped_directory/{test1,test2,test3,test4}
+ mkdir -p striped_directory/test1 striped_directory/test2 striped_directory/test3 striped_directory/test4
cd striped_directory
+ cd striped_directory
/home/work/lustre-release/lustre/utils/lfs getdirstripe test1 test2 test3 test4
+ /home/work/lustre-release/lustre/utils/lfs getdirstripe test1 test2 test3 test4
test1
lmv_stripe_count: 4
lmv_stripe_offset: 1
mdtidx		 FID[seq:oid:ver]
     1		 [0x280000401:0x2:0x0]		
     0		 [0x340000400:0x2:0x0]		
     2		 [0x2c0000400:0x2:0x0]		
     3		 [0x300000400:0x2:0x0]		
test2
lmv_stripe_count: 4
lmv_stripe_offset: 2
mdtidx		 FID[seq:oid:ver]
     2		 [0x2c0000402:0x1:0x0]		
     0		 [0x340000401:0x1:0x0]		
     1		 [0x280000402:0x1:0x0]		
     3		 [0x300000401:0x1:0x0]		
test3
lmv_stripe_count: 4
lmv_stripe_offset: 3
mdtidx		 FID[seq:oid:ver]
     3		 [0x300000403:0x1:0x0]		
     0		 [0x340000402:0x1:0x0]		
     1		 [0x280000403:0x1:0x0]		
     2		 [0x2c0000403:0x1:0x0]		
test4
lmv_stripe_count: 4
lmv_stripe_offset: 0
mdtidx		 FID[seq:oid:ver]
     0		 [0x340000404:0x1:0x0]		
     1		 [0x280000404:0x1:0x0]		
     2		 [0x2c0000404:0x1:0x0]		
     3		 [0x300000404:0x1:0x0]		
rm -rf *
+ rm -rf test1 test2 test3 test4
cd .. 
+ cd ..

/home/work/lustre-release/lustre/utils/lfs setdirstripe -c 1 -D striped_directory
+ /home/work/lustre-release/lustre/utils/lfs setdirstripe -c 1 -D striped_directory

mkdir -p striped_directory/{test1,test2,test3,test4}
+ mkdir -p striped_directory/test1 striped_directory/test2 striped_directory/test3 striped_directory/test4

cd striped_directory
+ cd striped_directory

/home/work/lustre-release/lustre/utils/lfs getdirstripe test1 test2 test3 test4
+ /home/work/lustre-release/lustre/utils/lfs getdirstripe test1 test2 test3 test4
test1
lmv_stripe_count: 1
lmv_stripe_offset: 1
mdtidx		 FID[seq:oid:ver]
     1		 [0x280000401:0x3:0x0]		
test2
lmv_stripe_count: 1
lmv_stripe_offset: 2
mdtidx		 FID[seq:oid:ver]
     2		 [0x2c0000402:0x2:0x0]		
test3
lmv_stripe_count: 1
lmv_stripe_offset: 3
mdtidx		 FID[seq:oid:ver]
     3		 [0x300000403:0x2:0x0]		
test4
lmv_stripe_count: 1
lmv_stripe_offset: 0
mdtidx		 FID[seq:oid:ver]
     0		 [0x340000404:0x2:0x0]		

cd ..
+ cd ..
rm -rf *
+ rm -rf striped_directory

[Di Wang]

Hi, Patrick, I had some trouble to access git right now, so I can not push patch to review until next monday. But I attached the fix above (patch.diff), which might fix the LBUG issue you met, please try. Thanks.

[Di Wang]

The later LBUG problem is actually brought in by another fix (http://review.whamcloud.com/#/c/9511/), which is not being landed to master yet. so I will add the fix to 9511. http://review.whamcloud.com/#/c/9862 and http://review.whamcloud.com/#/c/9883 will be used to track other fixes of this ticket.

[Di Wang]

The patch has been landed to master with 9511.