FWIW the last little bit of the debug_file is
... 00100000:10000000:3.0:1530524123.223490:0:375744:0:(lfsck_namespace.c:576:lfsck_namespace_trace_update()) dagg-MDT0002-osd: namespace LFSCK add flags for [0x68002f9d1:0xa471:0x0] in the trace file, flags 1, old 0, new 1: rc = 0 00100000:10000000:9.0:1530524123.224700:0:377274:0:(lfsck_namespace.c:576:lfsck_namespace_trace_update()) dagg-MDT0002-osd: namespace LFSCK add flags for [0x68002f9d1:0xa471:0x0] in the trace file, flags 2, old 1, new 3: rc = 0 00100000:10000000:3.0:1530524123.224783:0:375744:0:(lfsck_namespace.c:576:lfsck_namespace_trace_update()) dagg-MDT0002-osd: namespace LFSCK add flags for [0x68002f9d1:0xa45c:0x0] in the trace file, flags 1, old 0, new 1: rc = 0 00100000:10000000:9.0:1530524123.225897:0:377274:0:(lfsck_namespace.c:576:lfsck_namespace_trace_update()) dagg-MDT0002-osd: namespace LFSCK add flags for [0x68002f9d1:0xa45c:0x0] in the trace file, flags 2, old 1, new 3: rc = 0 00100000:10000000:3.0:1530524123.225978:0:375744:0:(lfsck_namespace.c:576:lfsck_namespace_trace_update()) dagg-MDT0002-osd: namespace LFSCK add flags for [0x68002f9d1:0xa46a:0x0] in the trace file, flags 1, old 0, new 1: rc = 0 00100000:10000000:9.0:1530524123.227163:0:377274:0:(lfsck_namespace.c:576:lfsck_namespace_trace_update()) dagg-MDT0002-osd: namespace LFSCK add flags for [0x68002f9d1:0xa46a:0x0] in the trace file, flags 2, old 1, new 3: rc = 0 00100000:10000000:3.0:1530524123.227248:0:375744:0:(lfsck_namespace.c:576:lfsck_namespace_trace_update()) dagg-MDT0002-osd: namespace LFSCK add flags for [0x68002f9d1:0xa476:0x0] in the trace file, flags 1, old 0, new 1: rc = 0 00100000:10000000:9.0:1530524123.228443:0:377274:0:(lfsck_namespace.c:576:lfsck_namespace_trace_update()) dagg-MDT0002-osd: namespace LFSCK add flags for [0x68002f9d1:0xa476:0x0] in the trace file, flags 2, old 1, new 3: rc = 0 00100000:10000000:9.0:1530524123.231812:0:377274:0:(lfsck_striped_dir.c:505:lfsck_record_lmv()) dagg-MDT0002-osd: record slave LMV EA for the striped directory [0x68002fa24:0x172d1:0x0]: shard = [0x68002f9d1:0xa47c:0x0], index = 0, flags = 0, flags2 = 0, depth = 0 00100000:10000000:9.0:1530524123.231905:0:377274:0:(lfsck_striped_dir.c:505:lfsck_record_lmv()) dagg-MDT0002-osd: record slave LMV EA for the striped directory [0x68002fa24:0x172d1:0x0]: shard = [0x2800301a1:0xa489:0x0], index = 2, flags = 0, flags2 = 0, depth = 0 00100000:10000000:3.0:1530524123.231921:0:375744:0:(lfsck_engine.c:1092:lfsck_master_engine()) LFSCK exit: oit_flags = 0x60003, dir_flags = 0x8006, oit_cookie = 35184372088832, dir_cookie = 0x0, parent = [0x0:0x0:0x0], pid = 375744, rc = 1 00100000:10000000:3.0:1530524123.231926:0:375744:0:(lfsck_lib.c:2577:lfsck_post_generic()) dagg-MDT0002-osd: waiting for assistant to do lfsck_namespace post, rc = 1 00100000:10000000:9.0:1530524123.231951:0:377274:0:(lfsck_striped_dir.c:505:lfsck_record_lmv()) dagg-MDT0002-osd: record slave LMV EA for the striped directory [0x68002fa24:0x172d1:0x0]: shard = [0x20001f032:0xa4fa:0x0], index = 1, flags = 0, flags2 = 0, depth = 0 00100000:10000000:9.0:1530524123.231956:0:377274:0:(lfsck_striped_dir.c:2189:lfsck_namespace_striped_dir_rescan()) dagg-MDT0002-osd: namespace LFSCK repair the shard 0 [0x68002f9d1:0xa47c:0x0] of the striped directory [0x68002f9d1:0xa47c:0x0] with dangling no/no, rename no/no, llinkea no/no, repair_lmvea no/no: rc = 0 00100000:10000000:9.0:1530524123.231985:0:377274:0:(lfsck_striped_dir.c:2189:lfsck_namespace_striped_dir_rescan()) dagg-MDT0002-osd: namespace LFSCK repair the shard 1 [0x20001f032:0xa4fa:0x0] of the striped directory [0x20001f032:0xa4fa:0x0] with dangling no/no, rename no/no, llinkea no/no, repair_lmvea no/no: rc = 0 00100000:10000000:9.0:1530524123.232007:0:377274:0:(lfsck_striped_dir.c:2189:lfsck_namespace_striped_dir_rescan()) dagg-MDT0002-osd: namespace LFSCK repair the shard 2 [0x2800301a1:0xa489:0x0] of the striped directory [0x2800301a1:0xa489:0x0] with dangling no/no, rename no/no, llinkea no/no, repair_lmvea no/no: rc = 0 00100000:10000000:9.0:1530524123.232034:0:377274:0:(lfsck_engine.c:1650:lfsck_assistant_engine()) dagg-MDT0002-osd: lfsck_namespace LFSCK assistant thread post 00100000:10000000:3.0:1530524123.232035:0:375744:0:(lfsck_lib.c:2589:lfsck_post_generic()) dagg-MDT0002-osd: the assistant has done lfsck_namespace post, rc = 1 00100000:10000000:5.0F:1530524123.232052:0:17117:0:(lfsck_namespace.c:4721:lfsck_namespace_in_notify()) dagg-MDT0001-osd: namespace LFSCK handles notify 3 from MDT 2, status 1, flags 3 00100000:10000000:13.0F:1530524123.232053:0:24114:0:(lfsck_namespace.c:4721:lfsck_namespace_in_notify()) dagg-MDT0000-osd: namespace LFSCK handles notify 3 from MDT 2, status 1, flags 2 00100000:10000000:3.0:1530524123.232053:0:375744:0:(lfsck_namespace.c:4397:lfsck_namespace_post()) dagg-MDT0002-osd: namespace LFSCK post done: rc = 0 00100000:10000000:3.0:1530524123.232054:0:375744:0:(lfsck_lib.c:2607:lfsck_double_scan_generic()) dagg-MDT0002-osd: waiting for assistant to do lfsck_namespace double_scan, status 2 00100000:10000000:1.0F:1530524123.232061:0:375552:0:(lfsck_namespace.c:6229:lfsck_namespace_assistant_handler_p2()) dagg-MDT0000-osd: namespace LFSCK phase2 scan start 00100000:10000000:9.0:1530524123.232068:0:377274:0:(lfsck_engine.c:1669:lfsck_assistant_engine()) dagg-MDT0002-osd: LFSCK assistant notified others for lfsck_namespace post: rc = 0 00100000:10000000:9.0:1530524123.232070:0:377274:0:(lfsck_engine.c:1687:lfsck_assistant_engine()) dagg-MDT0002-osd: LFSCK assistant sync before the second-stage scaning 00100000:10000000:3.0:1530524123.232072:0:375744:0:(lfsck_lib.c:2617:lfsck_double_scan_generic()) dagg-MDT0002-osd: the assistant has done lfsck_namespace double_scan, status 0 00100000:10000000:11.0F:1530524123.232106:0:377219:0:(lfsck_namespace.c:6229:lfsck_namespace_assistant_handler_p2()) dagg-MDT0001-osd: namespace LFSCK phase2 scan start 00100000:10000000:1.0:1530524123.232204:0:375552:0:(lfsck_namespace.c:5944:lfsck_namespace_scan_local_lpf()) dagg-MDT0000-osd: stop to scan backend /lost+found: rc = 0 00100000:10000000:11.0:1530524123.232339:0:377219:0:(lfsck_namespace.c:5944:lfsck_namespace_scan_local_lpf()) dagg-MDT0001-osd: stop to scan backend /lost+found: rc = 0 00100000:10000000:11.0:1530524123.463692:0:377274:0:(lfsck_engine.c:1694:lfsck_assistant_engine()) dagg-MDT0002-osd: LFSCK assistant phase2 scan start, synced: rc = 0 00100000:10000000:11.0:1530524123.463694:0:377274:0:(lfsck_namespace.c:6229:lfsck_namespace_assistant_handler_p2()) dagg-MDT0002-osd: namespace LFSCK phase2 scan start 00100000:10000000:11.0:1530524123.463722:0:377274:0:(lfsck_namespace.c:5944:lfsck_namespace_scan_local_lpf()) dagg-MDT0002-osd: stop to scan backend /lost+found: rc = 0 00100000:10000000:3.0F:1530524329.684216:0:375552:0:(lfsck_namespace.c:2253:lfsck_namespace_repair_unmatched_pairs()) dagg-MDT0000-osd: namespace LFSCK rebuild dotdot name entry for the object [0x20001f034:0xb6c2:0x0], new parent [0x68002f9d2:0x866d:0x0]: rc = 1 00100000:10000000:2.0F:1530524869.309940:0:377274:0:(lfsck_namespace.c:1217:lfsck_namespace_insert_normal()) dagg-MDT0002-osd: namespace LFSCK insert object [0x68002c327:0xc8c9:0x0] with the name 00000001000002CB00000083 and type 100000 to the parent [0x2000182d2:0x2c6d:0x0]: rc = 1
cheers,
robin
Lai
Could you please assist?
Thanks
Peter
Can you post the stack trace of all processes in the crash dump?
sorry, the console log above is all we have. we don't have kernel crash dumps setup.
cheers,
robin
This may be because LFSCK added an orphan back to its original parent directory, and then it was unlinked again, which triggered this assertion, I'll commit a fix soon.
Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/32796
Subject: LU-11111 mdd: don't assert on ORPHAN flag
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 693fe452ec4349bb6ea12b8901b9ec1665d68e95
I just uploaded a fix, you can apply it and update MDS only because it's a fix for MDT code.
thanks.
I applied it last night and let the original lfsck continue.
it finished ok (output below).
because it's been a long time, and we've had many more crashes from LU-11082, I started another namespace lfsck.
this hung most of the way through phase2 on mdt0 and didn't complete. it completed on mdt1,2.
I waited about 15mins after all the counters stopped going up on mdt0 and then tried a lfsck_stop, but it just blocked. lfsck_namespace was using 100% of a core. looked hung. I had to power cycle the MDS.
MDTs mounted with -o skip_lfsck again now.
do you want me to open a new ticket for this hang or stick with this ticket?
there are a lot of problems repaired in both of the below. presumable because of LU-10887 and LU-11082 and maybe LU-10677.
output from continued lfsck from ages ago (that crashed in phase2) ->
[warble1]root: lctl get_param -n mdd.dagg-MDT000*.lfsck_namespace name: lfsck_namespace magic: 0xa0621a0b version: 2 status: completed flags: param: all_targets,create_mdtobj last_completed_time: 1532449023 time_since_last_completed: 22 seconds latest_start_time: 1532447977 time_since_latest_start: 1068 seconds last_checkpoint_time: 1532449023 time_since_last_checkpoint: 22 seconds latest_start_position: 266, N/A, N/A last_checkpoint_position: 35184372088832, N/A, N/A first_failure_position: N/A, N/A, N/A checked_phase1: 14945199 checked_phase2: 6638062 updated_phase1: 0 updated_phase2: 1 failed_phase1: 0 failed_phase2: 0 directories: 2205524 dirent_repaired: 0 linkea_repaired: 0 nlinks_repaired: 0 multiple_linked_checked: 324817 multiple_linked_repaired: 0 unknown_inconsistency: 0 unmatched_pairs_repaired: 1 dangling_repaired: 92 multiple_referenced_repaired: 0 bad_file_type_repaired: 0 lost_dirent_repaired: 0 local_lost_found_scanned: 0 local_lost_found_moved: 0 local_lost_found_skipped: 0 local_lost_found_failed: 0 striped_dirs_scanned: 0 striped_dirs_repaired: 0 striped_dirs_failed: 0 striped_dirs_disabled: 0 striped_dirs_skipped: 0 striped_shards_scanned: 1639566 striped_shards_repaired: 1 striped_shards_failed: 0 striped_shards_skipped: 3 name_hash_repaired: 0 linkea_overflow_cleared: 0 success_count: 10 run_time_phase1: 2357 seconds run_time_phase2: 1972 seconds average_speed_phase1: 6340 items/sec average_speed_phase2: 3366 objs/sec average_speed_total: 4985 items/sec real_time_speed_phase1: N/A real_time_speed_phase2: N/A current_position: N/A name: lfsck_namespace magic: 0xa0621a0b version: 2 status: completed flags: param: all_targets,create_mdtobj last_completed_time: 1532448851 time_since_last_completed: 194 seconds latest_start_time: 1532447985 time_since_latest_start: 1060 seconds last_checkpoint_time: 1532448851 time_since_last_checkpoint: 194 seconds latest_start_position: 266, N/A, N/A last_checkpoint_position: 35184372088832, N/A, N/A first_failure_position: N/A, N/A, N/A checked_phase1: 14356291 checked_phase2: 6221317 updated_phase1: 0 updated_phase2: 0 failed_phase1: 0 failed_phase2: 0 directories: 2185653 dirent_repaired: 0 linkea_repaired: 0 nlinks_repaired: 0 multiple_linked_checked: 323999 multiple_linked_repaired: 0 unknown_inconsistency: 0 unmatched_pairs_repaired: 0 dangling_repaired: 11 multiple_referenced_repaired: 0 bad_file_type_repaired: 0 lost_dirent_repaired: 0 local_lost_found_scanned: 0 local_lost_found_moved: 0 local_lost_found_skipped: 0 local_lost_found_failed: 0 striped_dirs_scanned: 0 striped_dirs_repaired: 0 striped_dirs_failed: 0 striped_dirs_disabled: 0 striped_dirs_skipped: 0 striped_shards_scanned: 1639415 striped_shards_repaired: 0 striped_shards_failed: 0 striped_shards_skipped: 5 name_hash_repaired: 0 linkea_overflow_cleared: 0 success_count: 11 run_time_phase1: 2374 seconds run_time_phase2: 1800 seconds average_speed_phase1: 6047 items/sec average_speed_phase2: 3456 objs/sec average_speed_total: 4929 items/sec real_time_speed_phase1: N/A real_time_speed_phase2: N/A current_position: N/A name: lfsck_namespace magic: 0xa0621a0b version: 2 status: completed flags: param: all_targets,create_mdtobj last_completed_time: 1532448851 time_since_last_completed: 194 seconds latest_start_time: 1532448011 time_since_latest_start: 1034 seconds last_checkpoint_time: 1532448851 time_since_last_checkpoint: 194 seconds latest_start_position: 266, N/A, N/A last_checkpoint_position: 35184372088832, N/A, N/A first_failure_position: N/A, N/A, N/A checked_phase1: 14368285 checked_phase2: 6254745 updated_phase1: 2 updated_phase2: 1 failed_phase1: 0 failed_phase2: 0 directories: 2187046 dirent_repaired: 0 linkea_repaired: 1 nlinks_repaired: 0 multiple_linked_checked: 324797 multiple_linked_repaired: 0 unknown_inconsistency: 0 unmatched_pairs_repaired: 0 dangling_repaired: 88 multiple_referenced_repaired: 0 bad_file_type_repaired: 0 lost_dirent_repaired: 1 local_lost_found_scanned: 0 local_lost_found_moved: 0 local_lost_found_skipped: 0 local_lost_found_failed: 0 striped_dirs_scanned: 0 striped_dirs_repaired: 0 striped_dirs_failed: 0 striped_dirs_disabled: 0 striped_dirs_skipped: 0 striped_shards_scanned: 1639666 striped_shards_repaired: 0 striped_shards_failed: 0 striped_shards_skipped: 4 name_hash_repaired: 0 linkea_overflow_cleared: 0 success_count: 11 run_time_phase1: 2387 seconds run_time_phase2: 1799 seconds average_speed_phase1: 6019 items/sec average_speed_phase2: 3476 objs/sec average_speed_total: 4926 items/sec real_time_speed_phase1: N/A real_time_speed_phase2: N/A current_position: N/A
output from next namespace lfsck that stopped progressing on mdt0 (before power-off of MDS) ->
[warble1]root: lctl get_param -n mdd.dagg-MDT000*.lfsck_namespace name: lfsck_namespace magic: 0xa0621a0b version: 2 status: scanning-phase2 flags: scanned-once,inconsistent param: all_targets,create_mdtobj last_completed_time: 1532449023 time_since_last_completed: 5691 seconds latest_start_time: 1532449114 time_since_latest_start: 5600 seconds last_checkpoint_time: 1532453018 time_since_last_checkpoint: 1696 seconds latest_start_position: 266, N/A, N/A last_checkpoint_position: 35184372088832, N/A, N/A first_failure_position: N/A, N/A, N/A checked_phase1: 18305111 checked_phase2: 5342520 updated_phase1: 44 updated_phase2: 0 failed_phase1: 0 failed_phase2: 0 directories: 2368594 dirent_repaired: 0 linkea_repaired: 0 nlinks_repaired: 0 multiple_linked_checked: 381414 multiple_linked_repaired: 0 unknown_inconsistency: 0 unmatched_pairs_repaired: 0 dangling_repaired: 18 multiple_referenced_repaired: 0 bad_file_type_repaired: 0 lost_dirent_repaired: 0 local_lost_found_scanned: 0 local_lost_found_moved: 0 local_lost_found_skipped: 0 local_lost_found_failed: 0 striped_dirs_scanned: 0 striped_dirs_repaired: 0 striped_dirs_failed: 0 striped_dirs_disabled: 0 striped_dirs_skipped: 0 striped_shards_scanned: 1761665 striped_shards_repaired: 20 striped_shards_failed: 0 striped_shards_skipped: 84 name_hash_repaired: 44 linkea_overflow_cleared: 0 success_count: 10 run_time_phase1: 2823 seconds run_time_phase2: 2776 seconds average_speed_phase1: 6484 items/sec average_speed_phase2: 1924 objs/sec average_speed_total: 4223 items/sec real_time_speed_phase1: N/A real_time_speed_phase2: 35 objs/sec current_position: [0x20001f8cf:0xcddd:0x0] name: lfsck_namespace magic: 0xa0621a0b version: 2 status: completed flags: param: all_targets,create_mdtobj last_completed_time: 1532453205 time_since_last_completed: 1509 seconds latest_start_time: 1532449114 time_since_latest_start: 5600 seconds last_checkpoint_time: 1532453205 time_since_last_checkpoint: 1509 seconds latest_start_position: 266, N/A, N/A last_checkpoint_position: 35184372088832, N/A, N/A first_failure_position: N/A, N/A, N/A checked_phase1: 17716791 checked_phase2: 6134353 updated_phase1: 155 updated_phase2: 0 failed_phase1: 0 failed_phase2: 0 directories: 2348707 dirent_repaired: 0 linkea_repaired: 0 nlinks_repaired: 0 multiple_linked_checked: 381110 multiple_linked_repaired: 0 unknown_inconsistency: 0 unmatched_pairs_repaired: 0 dangling_repaired: 0 multiple_referenced_repaired: 0 bad_file_type_repaired: 0 lost_dirent_repaired: 0 local_lost_found_scanned: 0 local_lost_found_moved: 0 local_lost_found_skipped: 0 local_lost_found_failed: 0 striped_dirs_scanned: 0 striped_dirs_repaired: 0 striped_dirs_failed: 0 striped_dirs_disabled: 0 striped_dirs_skipped: 0 striped_shards_scanned: 1761553 striped_shards_repaired: 78 striped_shards_failed: 0 striped_shards_skipped: 102 name_hash_repaired: 155 linkea_overflow_cleared: 0 success_count: 12 run_time_phase1: 2707 seconds run_time_phase2: 1267 seconds average_speed_phase1: 6544 items/sec average_speed_phase2: 4841 objs/sec average_speed_total: 6001 items/sec real_time_speed_phase1: N/A real_time_speed_phase2: N/A current_position: N/A name: lfsck_namespace magic: 0xa0621a0b version: 2 status: completed flags: param: all_targets,create_mdtobj last_completed_time: 1532453199 time_since_last_completed: 1515 seconds latest_start_time: 1532449114 time_since_latest_start: 5600 seconds last_checkpoint_time: 1532453199 time_since_last_checkpoint: 1515 seconds latest_start_position: 266, N/A, N/A last_checkpoint_position: 35184372088832, N/A, N/A first_failure_position: N/A, N/A, N/A checked_phase1: 17724868 checked_phase2: 6127858 updated_phase1: 1 updated_phase2: 0 failed_phase1: 0 failed_phase2: 0 directories: 2349705 dirent_repaired: 0 linkea_repaired: 0 nlinks_repaired: 0 multiple_linked_checked: 380857 multiple_linked_repaired: 0 unknown_inconsistency: 0 unmatched_pairs_repaired: 0 dangling_repaired: 36 multiple_referenced_repaired: 0 bad_file_type_repaired: 0 lost_dirent_repaired: 0 local_lost_found_scanned: 0 local_lost_found_moved: 0 local_lost_found_skipped: 0 local_lost_found_failed: 0 striped_dirs_scanned: 0 striped_dirs_repaired: 0 striped_dirs_failed: 0 striped_dirs_disabled: 0 striped_dirs_skipped: 0 striped_shards_scanned: 1761725 striped_shards_repaired: 0 striped_shards_failed: 0 striped_shards_skipped: 102 name_hash_repaired: 0 linkea_overflow_cleared: 0 success_count: 12 run_time_phase1: 2698 seconds run_time_phase2: 1261 seconds average_speed_phase1: 6569 items/sec average_speed_phase2: 4859 objs/sec average_speed_total: 6024 items/sec real_time_speed_phase1: N/A real_time_speed_phase2: N/A current_position: N/A
cheers,
robin
As for the hung issue, I have a ticket LU-10888, but it's a fairly big issue, I need some time to think it over and implement.
Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/32935
Subject: LU-11111 lfsck: print correct parent FID
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 1c38dbb03e64992de7483ca57e724d608d5ebfb0
I think that suppressing the LASSERT() when the ORPHAN_OBJ flag is seen again is potentially the wrong fix for this problem.
It seems to me that the root cause here is that the file was unlinked by the user while it was still open/in use (which can happen commonly for temporary files). This results in ORPHAN_OBJ being set at the MDD layer, and LUSTRE_ORPHAN_FL should also set at the OSD layer. In this case, LFSCK should not be reconnecting the file/directory back into the namespace which later triggers the LASSERT(), since the user wants the file to be deleted. Instead, I think the more correct fix is to keep the LASSERT() as-is, and instead modify LFSCK to not reconnect objects that are marked with the LUSTRE_ORPHAN_FL at the OSD layer.
Unlike the ORPHAN_OBJ flag at the MDD layer, LFSCK can access the LUSTRE_ORPHAN_FL from the OSD before it tries to reconnect the object.
There is a secondary question about why the orphan object is itself not linked into the PENDING directory when it is being unlinked? This is where open-unlinked files should be located, so that they are not lost (unlinked at mount time by the normal ldiskfs orphan object handling) if the MDS crashes while the file/directory is open and in use. The MDS needs to be able to re-open the file after the crash, which is why such files are linked into PENDING, rather than just being referenced in memory.
I think we have hit this or similar issue with lfsck background run against sanity 230h (lfs migrate) or unlink a directory.
In this case FS, assert was don't hit but FS is corrupted as lfsck have insert an directory based on linkEA data
[35018.890469] Lustre: 30550:0:(lfsck_namespace.c:1220:lfsck_namespace_insert_normal()) fs1-MDT0000-osd: namespace LFSCK insert object [0x200030975:0x3:0x0] with the name d230h.sanity and type 40000 to the parent [0x200000007:0x1:0x0]: rc = 1
PENDING have flushed after MDT remount and it caused to remove this entry from PENDING, but not from "/" (aka [0x200000007:0x1:0x0]
- so it cause a situation when dir entry have point to stale node
00080000:00000002:12.0:1532436667.487291:0:20602:0:(osd_handler.c:559:osd_iget_check()) no inode for FID: [0x200030975:0x3:0x0], ino = 422576130, rc = -116
I have replicated it with very simple test case.
console 1> mkdir /mnt/lustre/testdir && ./multiop /mnt/lustre/testdir D_c
console 2> rmdir /mnt/lustre/testdir && lctl lfsck_start -o -r -C on -c on -A -M lustre-MDT0000
lfsck moved an unlinked file to the normal namespace.
patch will be ready son.
Current namespace LFSCK logic does not check the ORPHAN flag, as to it may add the orphan object back to its original parent directory because of the stale misguided linkEA. That should be fixed. The fix should be done inside the LFSCK engine, instead of OSD. Because:
- The LFSCK is not only for namespace consistency, but also for layout consistency, and may be for more in the future. We can skip orphan object for namespace LFSCK, but still need to verify the layout consistency for orphan object to avoid accessing or destroying wrong OST-object(s).
- The trouble may happen for both ldiskfs and ZFS backend.
we would like to lfsck our filesystem. we have a dozen or so new unremovable directories from the MDS hangs. unfortunately a bunch of users have noticed these.
last time I ran a lfsck it hung and I had to reset the MDS, which risks causing more corruption.
is there an ETA for a fix so that lfsck won't hang?
alternatively, suggestions for ways that we can remove DNE from the filesystem would be appreciated.
thanks.
cheers,
robin
Hi Robin, it'll take a couple of weeks to finish patch for LU-10888.
What do you mean remove DNE? do you mean migrate all dir and files to MDT0 and remove other MDTs from the system? but 2.10 doesn't support striped dir migration yet, and patches to support it will land to master (check LU-4684) later.
Robin, so there are four issues:
1. MDT hung during recovery, which will be fixed by LU-10888.
2. lfsck hung, I'd suggest you create a new ticket, and post all process backtraces.
3. unremovable dirs/files after lfsck, LU-10677.
4. MDT hung and won't umount, LU-10301.
Which do you think has the top priority?
BTW I'm curious how you use DNE, normally the operations that involves multiple MDTs should be rare, such as create/unlink/setattr striped directory, because you only need to create striped directory for certain directories that may contain lots of files, and normal directories are just plain. I saw your comment in LU-11301 mentioned inherited dir striping across all MDTs, do you mean dir default stripe is set, so all dirs are created as striped directory? And do you mkdir/rmdir frequently?
Lai,
you are wrong. This bug caused a unremovable files (ESTALE error) after MDT remount. MDT was kill an PENDING object, but normal namespace have a name and fid of deleted file. It was my start point.
MDT hung during recovery is unfixed for now, and caused a situation when OSP start to take attributes from MDT/OSP when this target is not accessible. dt_sync is fixed, but osp_getattr not (i have hit it many times).
Hi Lai,
LU-11301 has by far the highest priority for us.
it hangs the entire cluster on average about once a week I guess. ie. cluster is down. also it's taken to evicting most clients recently too, which means a total reboot of clients, plus an unknown amount of jobs lost.
these incidents tend to happen in batches, so we can sometimes go a 2 or 3 weeks without incident and then 3 crashes in a row.
LU-11082 and LU-11301 have both had much the same symptoms.
we have inherited striping on all dirs across all 3 MDTs. this is the only DNE config that makes sense to us. how do we know where users are going to put 100's of millions of files across 10's of thousands of dirs? we cannot possibly know this. if we set inherited dir striping then the MDTs work like OSTs, and on average we will get good balance across them. multiple MDTs had some file stat() performance benefits, and we assume performance for the rest of ops will improve over time as DNE matures. AFAIK nobody does more than the usual amount of mkdir/rmdir, which yes, is ultra-slow.
so if this isn't a supported config or is something that's never going to work well, then a) please make that clear in docs b) there is a window of time while the filesystem isn't over half full yet, and we can take a few weeks or a month to rsync all files (probably <2 PiB) onto an unstriped top level dir on MDT0 (which we left there for this purpose), take a couple of days of downtime to tidy up, and then destroy MDT1,MDT2 and make the MDT0 pool bigger. ie. remove DNE entirely. that's the only other config that makes sense to us. will this approach work?
on your points ->
1. cool. thanks.
2. we applied patch from LU-11062 so stack traces are back. will re-try at some stage.
3. also being able to corrupt the filesystem with just some mkdir's
4. LU-11301
cheers,
robin
See lustre manual: https://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#dbdoclet.deactivating_mdt_ost
After you copy all dir/files to MDT0, remove old dirs/files(or you can keep them, but they are not accessible after deactivation, and consume space of MDT0 and OSTs), and permanently deactivate MDT1-2, then you can stop MDT1-2.
I'm working on LU-11301, will update you when I make progress.
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/32935/
Subject: LU-11111 lfsck: print correct parent FID
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3583e1fd998c56977adde9abe9c9a9dc9bd88d58
Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34174
Subject: LU-11111 lfsck: skip orphan processing
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5a7be2a5cd89b67b55daa42725efee62b8e3fb57
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/32959/
Subject: LU-11111 lfsck: skip orphan processing
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 077570483e75e0610fd45149b926097547c434b8
Since https://review.whamcloud.com/#/c/32959/ was landed after Andreas recommended, "to land this patch as-is, with the knowledge that there is a rare corner case that is missed," should we keep this ticket open or just track the remaining work somewhere else? Either way it seems that https://review.whamcloud.com/#/c/34174/ should be abandoned now.
Cory, that patch resolved the aforementioned race condition (LFSCK on an orphan object that also had a corrupt layout), but it was having problems in testing and I thought it prudent to fix the issue with the simpler (if slightly imperfect) version of the patch from Shadow, which we had eventually gotten working.
I don't want to abandon Fan yong's patch, since it shows how to fix the issue completely, but I don't think it is a priority to land at this point. I agree that it would probably make sense to open a different ticket and link the patch there, and close this one.
Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34528
Subject: LU-11111 lfsck: skip orphan processing
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 9847af4cbd7b2040e65decc589ecee2808eee521
Tracking remaining patch in LU-12127.
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34528/
Subject: LU-11111 lfsck: skip orphan processing
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 92912099c0ee982d11019b2c586e255654eb9f45