[LU-9766] DNE phase 2 - wrong directory inheritance

Tue, 9 Nov 2021 03:21:45 +0000

The inheritance of directory striping with "lfs setdirstripe" is not working as it should:

Setting a directory with default mdt striping to 2:
[root@vm4 test]# lfs setdirstripe -D -c 2 2-stripes

Creating directories (which should inherits from parent)
[root@vm4 test]# mkdir 2-stripes/foo.{0..9}

Some are correct but not all of them:
[root@vm4 test]#  lfs getdirstripe 2-stripes/
2-stripes/
lmv_stripe_count: 2 lmv_stripe_offset: 0
mdtidx           FID[seq:oid:ver]
    0           [0x300000401:0x60:0x0]
    1           [0x340000402:0x1:0x0]
2-stripes//foo.2
lmv_stripe_count: 2 lmv_stripe_offset: 1
mdtidx           FID[seq:oid:ver]
    1           [0x340000401:0xb4:0x0]
    0           [0x300000402:0x2:0x0]
2-stripes//foo.6
lmv_stripe_count: 2 lmv_stripe_offset: 1
mdtidx           FID[seq:oid:ver]
    1           [0x340000401:0xb6:0x0]
    0           [0x300000402:0x4:0x0]
2-stripes//foo.5
lmv_stripe_count: 1 lmv_stripe_offset: 0
mdtidx           FID[seq:oid:ver]
    0           [0x300000401:0x63:0x0]
2-stripes//foo.9
lmv_stripe_count: 1 lmv_stripe_offset: 0
mdtidx           FID[seq:oid:ver]
    0           [0x300000401:0x65:0x0]
2-stripes//foo.0
lmv_stripe_count: 2 lmv_stripe_offset: 1
mdtidx           FID[seq:oid:ver]
    1           [0x340000401:0xb3:0x0]
    0           [0x300000402:0x1:0x0]
2-stripes//foo.1
lmv_stripe_count: 1 lmv_stripe_offset: 0
mdtidx           FID[seq:oid:ver]
    0           [0x300000401:0x61:0x0]
2-stripes//foo.8
lmv_stripe_count: 2 lmv_stripe_offset: 1
mdtidx           FID[seq:oid:ver]
    1           [0x340000401:0xb7:0x0]
    0           [0x300000402:0x5:0x0]
2-stripes//foo.7
lmv_stripe_count: 1 lmv_stripe_offset: 0
mdtidx           FID[seq:oid:ver]
    0           [0x300000401:0x64:0x0]
2-stripes//foo.3
lmv_stripe_count: 1 lmv_stripe_offset: 0
mdtidx           FID[seq:oid:ver]
    0           [0x300000401:0x62:0x0]
2-stripes//foo.4
lmv_stripe_count: 2 lmv_stripe_offset: 1
mdtidx           FID[seq:oid:ver]
    1           [0x340000401:0xb5:0x0]
    0           [0x300000402:0x3:0x0]

On MDS, in logs I can see that lod_cache_parent_striping does not return the defined striping all the time but the default filesystem striping:

 57168 00000004:00000001:1.0:1499850278.062218:0:8981:0:(lod_object.c:3008:lod_cache_parent_lmv_striping()) Process leaving
 57169 00000004:00000001:1.0:1499850278.062219:0:8981:0:(lod_object.c:3053:lod_cache_parent_striping()) Process leaving (rc=0 : 0 : 0)
 57170 00000004:00000040:1.0:1499850278.062220:0:8981:0:(lod_object.c:3155:lod_ah_init()) inherit default EA nr:1 off:-1 t2
 57171 00000004:00000040:1.0:1499850278.062220:0:8981:0:(lod_object.c:3187:lod_ah_init()) inherit EA nr:1 off:-1
 57172 00000004:00000040:1.0:1499850278.062221:0:8981:0:(lod_object.c:3195:lod_ah_init()) final striping count:1, offset:-1
 57173 00000004:00000001:1.0:1499850278.062221:0:8981:0:(lod_object.c:3246:lod_ah_init()) Process leaving


581753 00000004:00000001:1.0:1499850525.180299:0:9133:0:(lod_object.c:3053:lod_cache_parent_striping()) Process leaving (rc=0 : 0 : 0)
581754 00000004:00000040:1.0:1499850525.180299:0:9133:0:(lod_object.c:3155:lod_ah_init()) inherit default EA nr:1 off:-1 t2
581755 00000004:00000040:1.0:1499850525.180300:0:9133:0:(lod_object.c:3175:lod_ah_init()) set stripe EA nr:2 off:0
581756 00000004:00000040:1.0:1499850525.180300:0:9133:0:(lod_object.c:3195:lod_ah_init()) final striping count:2, offset:0
581757 00000004:00000001:1.0:1499850525.180301:0:9133:0:(lod_object.c:3246:lod_ah_init()) Process leaving

This is a problem as when the stripe count is incorrect, the assigned resulting MDT is 0, so the MDT0 fills up faster than other MDTs.

Also "lfs mkdir -i 1" does not work, it creates a directory with a stripe count of 0 and one mdt index. A workaround is to do an "lfs setdirstripe -D -c 1" on the parent directory then create directories with mkdir.

When creating directories where default striping was specified, I have sometimes timeouts in 2.7 and panics on clients in 2.9

2.7:

[root@vm4]# mkdir 1-stripe-1/foo.0/foo.{0..9}
mkdir: cannot create directory `1-stripe-1/foo.0/foo.0': Input/output error
mkdir: cannot create directory `1-stripe-1/foo.0/foo.1': Cannot send after transport endpoint shutdown
mkdir: cannot create directory `1-stripe-1/foo.0/foo.2': Cannot send after transport endpoint shutdown
mkdir: cannot create directory `1-stripe-1/foo.0/foo.3': Cannot send after transport endpoint shutdown
mkdir: cannot create directory `1-stripe-1/foo.0/foo.4': Cannot send after transport endpoint shutdown
mkdir: cannot create directory `1-stripe-1/foo.0/foo.5': Cannot send after transport endpoint shutdown
mkdir: cannot create directory `1-stripe-1/foo.0/foo.6': Cannot send after transport endpoint shutdown
mkdir: cannot create directory `1-stripe-1/foo.0/foo.7': Cannot send after transport endpoint shutdown
mkdir: cannot create directory `1-stripe-1/foo.0/foo.8': Cannot send after transport endpoint shutdown
mkdir: cannot create directory `1-stripe-1/foo.0/foo.9': Cannot send after transport endpoint shutdown
[root@vm4]# mkdir 1-stripe-1/foo.0/foo.{0..9}
mkdir: cannot create directory `1-stripe-1/foo.0/foo.0': Input/output error

2.9:

crash> bt 2135
PID: 2135   TASK: ffff880035860000  CPU: 1   COMMAND: "mkdir"
 #0 [ffff880016c4b670] machine_kexec at ffffffff81059cdb
 #1 [ffff880016c4b6d0] __crash_kexec at ffffffff81105182
 #2 [ffff880016c4b7a0] crash_kexec at ffffffff81105270
 #3 [ffff880016c4b7b8] oops_end at ffffffff8168efc8
 #4 [ffff880016c4b7e0] no_context at ffffffff8167ebd3
 #5 [ffff880016c4b830] __bad_area_nosemaphore at ffffffff8167ec69
 #6 [ffff880016c4b878] bad_area at ffffffff8167ef8d
 #7 [ffff880016c4b8a0] __do_page_fault at ffffffff81691e5f
 #8 [ffff880016c4b900] do_page_fault at ffffffff81691f05
 #9 [ffff880016c4b930] page_fault at ffffffff8168e1c8
    [exception RIP: memcpy+22]
    RIP: ffffffff813269a6  RSP: ffff880016c4b9e0  RFLAGS: 00010283
    RAX: ffff8800395fb4c0  RBX: ffff880016c4baf8  RCX: ffff880016c4bfd8
    RDX: ffffffffffffffe5  RSI: 0000000000000000  RDI: ffff8800395fb4c0
    RBP: ffff880016c4bab8   R8: 0000000000019a80   R9: 0000000000000000
    R10: ffff8800395fb4c0  R11: 0000000000aaaaaa  R12: 0000000000000025
    R13: ffff880016c4bae8  R14: ffff8800358789a0  R15: 0000000000000025
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#10 [ffff880016c4b9e0] ll_lookup_it_finish at ffffffffa0ab5715 [lustre]
#11 [ffff880016c4bac0] ll_lookup_it at ffffffffa0ab70ae [lustre]
#12 [ffff880016c4bb78] ll_lookup_nd at ffffffffa0ab89dd [lustre]
#13 [ffff880016c4bc10] lookup_real at ffffffff812083dd
#14 [ffff880016c4bc30] __lookup_hash at ffffffff81208d52
#15 [ffff880016c4bc60] lookup_slow at ffffffff816833cb
#16 [ffff880016c4bc98] link_path_walk at ffffffff8120b96f
#17 [ffff880016c4bd48] path_lookupat at ffffffff8120bb6b
#18 [ffff880016c4bde0] filename_lookup at ffffffff8120c2cb
#19 [ffff880016c4be18] filename_create at ffffffff8120c3a2
#20 [ffff880016c4bee8] user_path_create at ffffffff8120eee1
#21 [ffff880016c4bf18] sys_mkdirat at ffffffff812101f6
#22 [ffff880016c4bf70] sys_mkdir at ffffffff812102a9
#23 [ffff880016c4bf80] system_call_fastpath at ffffffff81696709
    RIP: 00007f9ddb6d29a7  RSP: 00007ffcb2ec5690  RFLAGS: 00010246
    RAX: 0000000000000053  RBX: ffffffff81696709  RCX: 00007ffcb2ec57f0
    RDX: 00000000000001ff  RSI: 00000000000001ff  RDI: 00007ffcb2ec9790
    RBP: 00007ffcb2ec87d0   R8: 00000000000001ff   R9: 00000000004029f0
    R10: 000000000000000b  R11: 0000000000000206  R12: ffffffff812102a9
    R13: ffff880016c4bf78  R14: 00000000000001ff  R15: 00007ffcb2ec8820
    ORIG_RAX: 0000000000000053  CS: 0033  SS: 002b

Whamcloud Community JIRA

[LU-9766] DNE phase 2 - wrong directory inheritance