[LU-10036] lfsck stuck in init on multiple MDS

• Type: Bug
• Resolution: Duplicate
• Priority: Major
• Affects Version/s: Lustre 2.10.1
• Labels: soak
• Environment: soak stress cluster - b2.10.1 + LU-9983 diag patch
• Severity: 3
• Assignee: nasf (Inactive)
• Reporter: Cliff White (Inactive)

Description

soak starts lfsck:

2017-09-26 16:54:20,710:fsmgmt.fsmgmt:INFO     executing cmd:
                lctl lfsck_start -M soaked-MDT0000 -s 1000 -t all
                if [[ $? != 0 ]]; then
                        lctl lfsck_start -M soaked-MDT0000 -s 1000 -t namespace,layout
                        if [[ $? != 0 ]]; then
                                lctl lfsck_start -M soaked-MDT0000 -s 1000 -t namespace
                        fi
                fi

lfsck shows 'stopped' on soak-8, 'init' on all other MDS nodes. No apparent progress

----------------
soak-8
----------------
mdd.soaked-MDT0000.lfsck_namespace=
name: lfsck_namespace
magic: 0xa0621a0b
version: 2
status: stopped
flags:
param:
last_completed_time: 1506405886
time_since_last_completed: 41575 seconds
latest_start_time: 1506410669
time_since_latest_start: 36792 seconds
last_checkpoint_time: 1506411504
time_since_last_checkpoint: 35957 seconds
latest_start_position: 77, N/A, N/A
last_checkpoint_position: 500695836, N/A, N/A
first_failure_position: N/A, N/A, N/A
checked_phase1: 2156
.....
----------------
soak-9
----------------
mdd.soaked-MDT0001.lfsck_namespace=
name: lfsck_namespace
magic: 0xa0621a0b
version: 2
status: init
flags:
param:
last_completed_time: N/A
time_since_last_completed: N/A
latest_start_time: N/A
time_since_latest_start: N/A
last_checkpoint_time: N/A
....
----------------
soak-10
----------------
mdd.soaked-MDT0002.lfsck_namespace=
name: lfsck_namespace
magic: 0xa0621a0b
version: 2
status: init
flags:
param:
last_completed_time: N/A
time_since_last_completed: N/A
latest_start_time: N/A
time_since_latest_start: N/A
last_checkpoint_time: N/A
time_since_last_checkpoint: N/A
latest_start_position: N/A, N/A, N/A
last_checkpoint_position: N/A, N/A, N/A
first_failure_position: N/A, N/A, N/A
checked_phase1: 0
....
----------------
soak-11
----------------
mdd.soaked-MDT0003.lfsck_namespace=
name: lfsck_namespace
magic: 0xa0621a0b
version: 2
status: init
flags:
param:
last_completed_time: N/A
time_since_last_completed: N/A
latest_start_time: N/A
time_since_latest_start: N/A
last_checkpoint_time: N/A
time_since_last_checkpoint: N/A
latest_start_position: N/A, N/A, N/A
last_checkpoint_position: N/A, N/A, N/A
first_failure_position: N/A, N/A, N/A
checked_phase1: 0
checked_phase2: 0
updated_phase1: 0
updated_phase2: 0
failed_phase1: 0
failed_phase2: 0
directories: 0
dirent_repaired: 0
linkea_repaired: 0
nlinks_repaired: 0
multiple_linked_checked: 0
multiple_linked_repaired: 0
unknown_inconsistency: 0
unmatched_pairs_repaired: 0
dangling_repaired: 0
multiple_referenced_repaired: 0
bad_file_type_repaired: 0
lost_dirent_repaired: 0
local_lost_found_scanned: 0
local_lost_found_moved: 0
local_lost_found_skipped: 0
local_lost_found_failed: 0
striped_dirs_scanned: 0
striped_dirs_repaired: 0
striped_dirs_failed: 0
striped_dirs_disabled: 0
striped_dirs_skipped: 0
striped_shards_scanned: 0
striped_shards_repaired: 0
striped_shards_failed: 0
striped_shards_skipped: 0
name_hash_repaired: 0
linkea_overflow_cleared: 0
success_count: 0
run_time_phase1: 0 seconds
run_time_phase2: 0 seconds
average_speed_phase1: 0 items/sec
average_speed_phase2: 0 objs/sec
average_speed_total: 0 items/sec
real_time_speed_phase1: N/A
real_time_speed_phase2: N/A
current_position: N/A

soak is currently wedged with this failure.

Issue Links

• duplicates: LU-10419 LFSCK fails to start, hangs systems.