Details
-
Improvement
-
Resolution: Fixed
-
Minor
-
None
-
None
-
3
-
9223372036854775807
Description
a replica representation in LOV EA can be broken like this one:
lcm_layout_gen: 7
lcm_mirror_count: 2
lcm_entry_count: 2
lcme_id: 65538
lcme_mirror_id: 1
lcme_flags: init,stale
lcme_extent.e_start: 134217728
lcme_extent.e_end: 1073741824
lmm_stripe_count: 16
lmm_stripe_size: 16777216
lmm_pattern: 40000001
lmm_layout_gen: 1
lmm_stripe_offset: 4294967295
lmm_objects:
- 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
- 1: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
- 2: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
- 3: { l_ost_idx: 5, l_fid: [0xbc0000406:0x42fce0eb:0x0] }
- 4: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
- 5: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
- 6: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
- 7: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
- 8: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
- 9: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
- 10: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
- 11: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
- 12: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
- 13: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
- 14: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
- 15: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
lcme_id: 131073
lcme_mirror_id: 2
lcme_flags: init
lcme_extent.e_start: 0
lcme_extent.e_end: EOF
lmm_stripe_count: 16
lmm_stripe_size: 1048576
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 5
lmm_pool: hdd-pool
lmm_objects:
- 0: { l_ost_idx: 5, l_fid: [0xbc0000406:0x42feb0aa:0x0] }
- 1: { l_ost_idx: 8, l_fid: [0x8c0000402:0x3bf10cb:0x0] }
- 2: { l_ost_idx: 15, l_fid: [0x9c0000402:0x11f1d8f:0x0] }
- 3: { l_ost_idx: 13, l_fid: [0x900000402:0x77529c35:0x0] }
- 4: { l_ost_idx: 0, l_fid: [0x300000403:0x3beded4:0x0] }
- 5: { l_ost_idx: 7, l_fid: [0xa80000402:0x11e9898:0x0] }
- 6: { l_ost_idx: 12, l_fid: [0x880000402:0x3bef34f:0x0] }
- 7: { l_ost_idx: 10, l_fid: [0xa40000402:0x11d9d9d:0x0] }
- 8: { l_ost_idx: 14, l_fid: [0xa00000402:0x11e4d68:0x0] }
- 9: { l_ost_idx: 2, l_fid: [0xb80000402:0x11d545a:0x0] }
- 10: { l_ost_idx: 6, l_fid: [0xb40000400:0x11f22d9:0x0] }
- 11: { l_ost_idx: 4, l_fid: [0x2c0000403:0x4016eb6:0x0] }
- 12: { l_ost_idx: 9, l_fid: [0x940000402:0xaf7b184a:0x0] }
- 13: { l_ost_idx: 11, l_fid: [0x980000402:0x11dc273:0x0] }
- 14: { l_ost_idx: 1, l_fid: [0xac0000404:0x3015313c:0x0] }
- 15: { l_ost_idx: 3, l_fid: [0xb00000400:0x11dd9bb:0x0] }
but regular stat should use any valid replica, not return an error once a bogus one is met.
Attachments
Issue Links
- is related to
-
LU-17721 sanity-flr/210b fails: something went wrong with unlink
-
- Resolved
-
Activity
I did a simple test with unlink:
== sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:30:01 (1711179001) before dd lustre-OST0000_UUID 1818580 1524 1700672 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 1818580 1524 1700672 1% /mnt/lustre[OST:1] [ 28.527289] Lustre: DEBUG MARKER: == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:30:01 (1711179001) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.032342 s, 648 MB/s after dd lustre-OST0000_UUID 1818580 22004 1659592 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 1818580 1524 1700672 1% /mnt/lustre[OST:1] fail_loc=0x1428 [ 31.585672] Lustre: *** cfs_fail_loc=1428, val=0*** [ 31.670467] Lustre: 6901:0:(lov_ea.c:299:lsme_unpack()) lustre-clilov_UUID: FID 0x280000401:2 OST index -1 more than OST count 2 [ 31.670469] Lustre: lustre-clilov_UUID: wait 30s while client connects to new OST [ 61.791678] Lustre: 6901:0:(lov_pack.c:57:lov_dump_lmm_common()) objid 0x3:1026, magic 0x0bd10bd0, pattern 0x1 [ 61.791683] Lustre: 6901:0:(lov_pack.c:61:lov_dump_lmm_common()) stripe_size 4194304, stripe_count 1, layout_gen 0 [ 61.791685] Lustre: 6901:0:(lov_pack.c:81:lov_dump_lmm_objects()) stripe 0 idx 4294967295 subobj 0x280000401:2 after mirror extend [ 91.951586] Lustre: 6901:0:(lov_pack.c:57:lov_dump_lmm_common()) objid 0x3:1026, magic 0x0bd10bd0, pattern 0x1 [ 91.951591] Lustre: 6901:0:(lov_pack.c:61:lov_dump_lmm_common()) stripe_size 4194304, stripe_count 1, layout_gen 0 [ 91.951594] Lustre: 6901:0:(lov_pack.c:81:lov_dump_lmm_objects()) stripe 0 idx 4294967295 subobj 0x280000401:2 [ 121.071619] Lustre: 6905:0:(lov_pack.c:57:lov_dump_lmm_common()) objid 0x3:1026, magic 0x0bd10bd0, pattern 0x1 [ 121.071624] Lustre: 6905:0:(lov_pack.c:61:lov_dump_lmm_common()) stripe_size 4194304, stripe_count 1, layout_gen 0 [ 121.071627] Lustre: 6905:0:(lov_pack.c:81:lov_dump_lmm_objects()) stripe 0 idx 4294967295 subobj 0x280000401:2 now list directory total 0 === [ 151.231585] Lustre: 6905:0:(lov_pack.c:57:lov_dump_lmm_common()) objid 0x3:1026, magic 0x0bd10bd0, pattern 0x1 [ 151.231590] Lustre: 6905:0:(lov_pack.c:61:lov_dump_lmm_common()) stripe_size 4194304, stripe_count 1, layout_gen 0 [ 151.231592] Lustre: 6905:0:(lov_pack.c:81:lov_dump_lmm_objects()) stripe 0 idx 4294967295 subobj 0x280000401:2 Waiting for MDT destroys to complete after removal lustre-OST0000_UUID 1818580 1524 1700672 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 1818580 1524 1700672 1% /mnt/lustre[OST:1] PASS 210b (134s)
that seems to work (at least the space is back), but takes very long. not sure whether this is OK.
Alex, can we not just skip the bad stripes during unlink? I don't want to say "just skip the bad component and delete the file anyway, let LFSCK handle the orphan objects", since this is how this problem was caused in the first place - by LFSCK linking the orphan object(s) back into the file.
I'm sure we have the ability to delete files with bad objects in it, though maybe not when the OST index is "-1"? Could we pick a real OST index but assign a bogus FID that is ignored by the lower layers without generating a lot of error messages?
The existing patch here avoids crashing the MDS when accessing the bad layout, but does not actually allow deleting the bad file (or ideally the bad mirror.
yes, I tried to implement that, but this is not trivial - basically the problem is that MDS wants to initialize all in-core structures, including objects, but this is not possible (currently) as referenced OST(s) doesn't exist.
probably we may have a special OSP to redirect "invalid" objects to.
The existing patch here avoids crashing the MDS when accessing the bad layout, but does not actually allow deleting the bad file (or ideally the bad mirror.
I binary edited an existing layout to replace most of the stripes in a second mirror with "0000" for the FID and "0xffffffff" for the ost_idx using:
# lfs setstripe -N -E eof -c 2 -N -E eof -c 2 /mnt/testfs/mirror [ unmount and mount MDT as type ldiskfs] # getfattr -n trusted.lov /mnt/testfs/mirror > /tmp/mirror.new # vi /tmp/mirror.new (to replace the second-last FID with "0000" and index with "ffffffff" # setfattr --restore /tmp/mirror.new [ unmount and remount MDT as type lustre ]
This produced a layout like:
/mnt/testfs/mirror
lcm_layout_gen: 2
lcm_mirror_count: 2
lcm_entry_count: 2
lcme_id: 65537
lcme_mirror_id: 1
lcme_flags: init
lcme_extent.e_start: 0
lcme_extent.e_end: EOF
lmm_stripe_count: 2
lmm_stripe_size: 1048576
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 0
lmm_objects:
- 0: { l_ost_idx: 0, l_fid: [0x300000bd3:0x3142:0x0] }
- 1: { l_ost_idx: 1, l_fid: [0x340000407:0x32a1:0x0] }
lcme_id: 131074
lcme_mirror_id: 2
lcme_flags: init
lcme_extent.e_start: 0
lcme_extent.e_end: EOF
lmm_stripe_count: 2
lmm_stripe_size: 1048576
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 4294967295
lmm_objects:
- 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] }
- 1: { l_ost_idx: 3, l_fid: [0x3c0000407:0x3725:0x0] }
but this file cannot be deleted with rm or unlink. The debug.txt.gz
log is attached.
There is also an interaction with LU-17334 because it is now waiting 30s whenever a broken file is accessed, but it should just skip outrageously broken index numbers:
@@ -140,6 +140,10 @@ struct lu_tgt_desc *lmv_tgt_retry(struct lmv_obd *lmv, __u32 index)
unsigned int level;
int rc;
+ if (index >= LOV_V1_INSANE_STRIPE_COUNT) {
+ CERROR("%s: bad stripe index %u\n", obd->obd_name, index);
+ return NULL;
+ }
might_sleep();
retry:
tgt = lmv_tgt(lmv, index);
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52996/
Subject: LU-17261 lov: ignore broken components
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 902fe290e51dccdee89380fb725ae6e3c1802e2b
I tried to reproduce the exact situation seen in the broken files with an added test cases in patch https://review.whamcloud.com/53276 "LU-17261 tests: delete partial broken mirror is OK" with mixed success.
When LFSCK is re-attaching the orphan OST object to the mirrored file it created a different layout than seen in the original system. Originally the layout component looks like:
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_objects:
- 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3:0x0] }
- 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x4:0x0] }
lmm_pattern: 40000001
lmm_layout_gen: 1
lmm_objects:
- 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3:0x0] }
- 1: { l_ost_idx: 0, l_fid: [0x100000000:0x0:0x0] }
This used "LOV_PATTERN_F_HOLE = 0x40000000" and a different FID for the missing OST objects, instead of "l_ost_idx: -1, l_fid: [0:0x0:0x0]" seen on the other files. The usage of LOV_PATTERN_F_HOLE was added for PFL files in commit v2_9_55_0-27-ge2cdf469b0 and for plain files in commit v2_5_59_0-34-g754bf71c65.
So it isn't clear why the mirrors were recreated by LFSCK with "l_ost_idx: -1" in the original case?
"Alex Zhuravlev <bzzz@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52996
Subject: LU-17261 lov: ignore broken components
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 716fc5db86bebaee42c6c61ebc5d1098d741c453
this is what happens literally:
00020000:00000001:4.0:1698862926.643457:0:3746:0:(lov_ea.c:251:lsme_unpack()) Process leaving via out_lsme (rc=18446744073709551594 : -22 : 0xffffffffffffffea) 00020000:00000010:4.0:1698862926.643458:0:3746:0:(lov_ea.c:290:lsme_unpack()) slab-freed '(lsme->lsme_oinfo[i])': 112 at ffff9d87eab4c480. 00020000:00000010:4.0:1698862926.643459:0:3746:0:(lov_ea.c:292:lsme_unpack()) kfreed 'lsme': 128 at ffff9d87eab4ce80. 00020000:00000001:4.0:1698862926.643460:0:3746:0:(lov_ea.c:487:lsm_unpackmd_comp_md_v1()) Process leaving via out_lsm (rc=18446744073709551594 : -22 : 0xffffffffffffffea) 00020000:00000010:4.0:1698862926.643461:0:3746:0:(lov_ea.c:522:lsm_unpackmd_comp_md_v1()) kfreed 'lsm': 72 at ffff9d8e3015c360. 00020000:00000001:4.0:1698862926.643462:0:3746:0:(lov_ea.c:524:lsm_unpackmd_comp_md_v1()) Process leaving (rc=18446744073709551594 : -22 : ffffffffffffffea) 00020000:00000001:4.0:1698862926.643463:0:3746:0:(lov_pack.c:345:lov_unpackmd()) Process leaving (rc=18446744073709551594 : -22 : ffffffffffffffea) 00020000:00000001:4.0:1698862926.643464:0:3746:0:(lov_object.c:1299:lov_object_init()) Process leaving (rc=18446744073709551594 : -22 : ffffffffffffffea) 00020000:00000001:4.0:1698862926.643467:0:3746:0:(lov_object.c:1386:lov_object_delete()) Process entered 00020000:00000002:4.0:1698862926.643467:0:3746:0:(lov_object.c:1125:lov_conf_freeze()) To take share lov(ffff9d8c2ce92170) owner (null)/ffff9d82c6d80000 00020000:00000001:5.0:1698862926.643490:0:3746:0:(lov_object.c:1191:lov_layout_wait()) Process entered 00020000:00000001:5.0:1698862926.643491:0:3746:0:(lov_object.c:1201:lov_layout_wait()) Process leaving (rc=0 : 0 : 0) 00020000:00000002:5.0:1698862926.643492:0:3746:0:(lov_object.c:1133:lov_conf_thaw()) To release share lov(ffff9d8c2ce92170) owner (null)/ffff9d82c6d80000 00020000:00000001:5.0:1698862926.643493:0:3746:0:(lov_object.c:1388:lov_object_delete()) Process leaving 00020000:00000001:5.0:1698862926.643494:0:3746:0:(lov_object.c:1395:lov_object_free()) Process entered 00020000:00000002:5.0:1698862926.643494:0:3746:0:(lov_object.c:1125:lov_conf_freeze()) To take share lov(ffff9d8c2ce92170) owner (null)/ffff9d82c6d80000 00020000:00000002:5.0:1698862926.643495:0:3746:0:(lov_object.c:1133:lov_conf_thaw()) To release share lov(ffff9d8c2ce92170) owner (null)/ffff9d82c6d80000 00020000:00000010:5.0:1698862926.643497:0:3746:0:(lov_object.c:1398:lov_object_free()) slab-freed '(lov)': 184 at ffff9d8c2ce92170. 00020000:00000001:5.0:1698862926.643499:0:3746:0:(lov_object.c:1399:lov_object_free()) Process leaving 00000080:00000010:5.0:1698862926.643500:0:3746:0:(vvp_object.c:272:vvp_object_free()) slab-freed '(vob)': 176 at ffff9d92e4f2cdc0. 00000020:00000001:5.0:1698862926.643502:0:3746:0:(lu_object.c:814:lu_object_find_at()) Process leaving (rc=18446744073709551594 : -22 : ffffffffffffffea) 00000080:00020000:5.0:1698862926.643504:0:3746:0:(lcommon_cl.c:197:cl_file_inode_init()) nbp19: failed to initialize cl_object [0x3c005662a:0xc4f1:0x0]: rc = -22 00000020:00001000:5.0:1698862926.646200:0:3746:0:(cl_object.c:832:cl_env_put()) 1@ffff9d86304433f0 00000080:00000001:5.0:1698862926.646203:0:3746:0:(llite_lib.c:2260:ll_read_inode2()) Process leaving (rc=18446744073709551594 : -22 : ffffffffffffffea)
the very first error leads to an user visible error.
I think we prefer to stat both mirrors, so that the blocks count is updated. However, it is OK to stat only one copy of the other is broken.
The "wait 30s while client connects to new OST" delay is because of the OST index check in lmv_tgt_retry(), and it would be gone with the proposed change above.