Details
-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
Lustre 2.5.3, Lustre 2.8.0
-
None
-
OLCF Atlas production system: clients running 2.8.0+ (with patches), server running 2.5.5+ (with patches)
-
3
-
9223372036854775807
Description
On atlas2 file system, we have a particular directory, any operations such as "ls" or "stat" will completely hang the process. This incurs no OS error or Lustre error from the client side. On server side, we did observe OI scrub message a few times, which may suggest there is some MDS data inconsistency, and it is "trying" to do the fix but no avail. We can't correlate the two yet.
Ops teams have collected traces on the client side by:
mount -t lustre 10.36.226.77@o2ib:/atlas2 /lustre/atlas2 -o rw,flock,nosuid,nodev
lctl set_param osc/*/checksums 0
echo “all” > /proc/sys/lnet/debug
echo “1024” > /proc/sys/lnet/debug_mb
Step2: cd /lustre/atlas2/path/to/offending_directory/
Step3: ls
Step1: lctl dk > /dev/null
Step4: Wait 30 seconds
Step5: lctl dk > atlas2-mds3_ls_for_fprof.out
the log is attached.
Attachments
Issue Links
- is cloned by
-
-
- Resolved
-
Activity
Since this is the case we will leave this open until the lfsck run.
Sorry, some misguide, "lctl set_param fail_loc=0x1505" is used for bypass FID-in-dirent when it is broken. So if your system hang before, but works well with "lctl set_param fail_loc=0x1505", that means it is quite possible that some FID-in-dirent is broken. Under such case, you need to run namespace LFSCK (with "lctl set_param fail_loc=0") to repair the FID-in-dirent. Otherwise, bypass FID-in-dirent will slowdown the lookup() performance.
The command "lctl set_param fail_loc=0x1505" was run on the MDS and it fixed the problem. Thanks nasf
Sorry for the delays Nasf! We will be having an outage on Feb. 07 to test lustre-2.8 servers and will hopefully leave it in production. After this outage we can run an online lfsck to see if this probably gets resolved.
Dustin, do you have more logs or any feedback about trying "lctl set_param fail_loc=0x1505" on the MDS? Thanks!
Another possible reason is that the FID-in-dirent is corrupted, that can explain why the OI scrub was triggered but no inconsistent OI mapping was found. It can be verified via "lctl set_param fail_loc=0x1505" on the MDS and try "ls" again after the setting. If it still hung there, then it is NOT the case; otherwise, we found the reason.
It is strange that the OI scrub has not found inconsistency. It should be some OI scrub issue.
Do you have the MDS side -1 level Lustre kernel debug logs when the "ls" hung? On the other hand, would you please to use "debugfs" to dump the directory and its sub-items that caused the system hung when "ls"? Thanks!
lfsck run seems to have fixed this.