https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8696 Lustre <p>On atlas2 file system, we have a particular directory, any operations such as "ls" or "stat" will completely hang the process. This incurs no OS error or Lustre error from the client side. On server side, we did observe OI scrub message a few times, which may suggest there is some MDS data inconsistency, and it is "trying" to do the fix but no avail. We can't correlate the two yet.</p> <p>Ops teams have collected traces on the client side by:</p> <p>mount -t lustre 10.36.226.77@o2ib:/atlas2 /lustre/atlas2 -o rw,flock,nosuid,nodev<br/> lctl set_param osc/*/checksums 0<br/> echo “all” &gt; /proc/sys/lnet/debug<br/> echo “1024” &gt; /proc/sys/lnet/debug_mb</p> <p>Step2: cd /lustre/atlas2/path/to/offending_directory/<br/> Step3: ls</p> <p>Step1: lctl dk &gt; /dev/null<br/> Step4: Wait 30 seconds<br/> Step5: lctl dk &gt; atlas2-mds3_ls_for_fprof.out</p> <p>the log is attached.</p> OLCF Atlas production system: clients running 2.8.0+ (with patches), server running 2.5.5+ (with patches) LU-8696

"ls" hangs on a particular directory on production system

Bug Critical Resolved Fixed nasf Feiyi Wang Wed, 12 Oct 2016 15:58:37 +0000 Mon, 13 Nov 2017 23:08:18 +0000 Thu, 9 Mar 2017 18:19:05 +0000 Lustre 2.5.3 Lustre 2.8.0 0 10 <p>from client side</p> <p>Hi Fan Yong,</p> <p>Could you please have a look at this issue?</p> <p>Thanks.<br/> Joe</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>... 00000002:00100000:1.0:1476224347.810367:0:14314:0:(mdc_locks.c:612:mdc_finish_enqueue()) @@@ op: 8 disposition: b, status: -115 req@ffff883f9d89a0c0 x1547933307679040/t0(0) o101-&gt;atlas2-MDT0000-mdc-ffff883fe0706c00@10.36.226.121@o2ib:12/10 lens 696/536 e 0 to 0 dl 1476224665 ref 1 fl Interpret:R/0/0 rc 301/301 00010000:00000001:1.0:1476224347.810373:0:14314:0:(ldlm_lock.c:598:__ldlm_handle2lock()) Process entered 00000020:00000001:1.0:1476224347.810373:0:14314:0:(lustre_handles.c:160:class_handle2object()) Process entered 00000020:00000001:1.0:1476224347.810374:0:14314:0:(lustre_handles.c:183:class_handle2object()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:1.0:1476224347.810376:0:14314:0:(ldlm_lock.c:604:__ldlm_handle2lock()) Process leaving (rc=0 : 0 : 0) 00000002:00000001:1.0:1476224347.810377:0:14314:0:(mdc_locks.c:741:mdc_finish_enqueue()) Process leaving (rc=0 : 0 : 0) 00000002:00000001:1.0:1476224347.810378:0:14314:0:(mdc_locks.c:922:mdc_finish_intent_lock()) Process entered 00000002:00000001:1.0:1476224347.810379:0:14314:0:(mdc_locks.c:946:mdc_finish_intent_lock()) Process leaving (rc=18446744073709551501 : -115 : ffffffffffffff8d) 00000002:00000001:1.0:1476224347.810380:0:14314:0:(mdc_locks.c:1195:mdc_intent_getattr_async_interpret()) Process leaving 00000080:00000001:1.0:1476224347.810382:0:14314:0:(statahead.c:663:ll_statahead_interpret()) Process entered 00000080:00400000:1.0:1476224347.810382:0:14314:0:(statahead.c:675:ll_statahead_interpret()) sa_entry DESIGN_3D_MAR2013 rc -115 00000080:00000001:1.0:1476224347.810384:0:14314:0:(dcache.c:269:ll_intent_release()) Process entered 00000080:00000040:1.0:1476224347.810385:0:14314:0:(dcache.c:271:ll_intent_release()) intent ffff883f928a0d38 released 00000080:00000001:1.0:1476224347.810386:0:14314:0:(dcache.c:282:ll_intent_release()) Process leaving 00000080:00000010:1.0:1476224347.810387:0:14314:0:(statahead.c:680:ll_statahead_interpret()) kfreed 'minfo': 480 at ffff883f928a0c00. 00000080:00000001:1.0:1476224347.810391:0:14314:0:(statahead.c:709:ll_statahead_interpret()) Process leaving (rc=18446744073709551501 : -115 : ffffffffffffff8d) </pre> </div></div> <p>As shown in the client side log, when the client wants to stat the object DESIGN_3D_MAR2013, it hits -115 failure. That means the MDT is doing OI scrub, and the target OI mapping for DESIGN_3D_MAR2013 is not ready yet. So the client has to retry again and again until related OI mapping is rebuilt, looks like hung there. As for what caused the OI scrub, we need the MDS side kernel debug log (-1 level).</p> <p>So there should be some OI inconsistency, I would suggest you to run OI scrub on the MDT with kernel debug log (please open lfsck debug: "lctl set_param debug=+lfsck") collected. Please show me the output of "lctl get_param -n osd-ldiskfs.$fsname-MDT0000.oi_scrub" before and after the OI scrub. Thanks!</p> <p>Nasf, <br/> Per Intel's recommendation, we ran an e2fsck during our last test shot to see if the problem gets fixed (despite the OI scrubber messages that we were seeing in the logs). We did find some non-critical issues, but we are still seeing the same hanging behavior with this directory. We have to take a downtime to temporarily upgrade to lustre-2.8 to use a functional LFSCK. I'm not 100% sure when we will get this opportunity, but I will keep it on our radar. For your reference, this is the IO scrub lfs get_param info you were wanting:</p> <p><span class="error">&#91;root@atlas2-mds1 mdt&#93;</span># lctl get_param -n osd-ldiskfs.atlas2-MDT0000.oi_scrub<br/> name: OI_scrub<br/> magic: 0x4c5fd252<br/> oi_files: 64<br/> status: completed<br/> flags:<br/> param:<br/> time_since_last_completed: 559 seconds<br/> time_since_latest_start: 5295 seconds<br/> time_since_last_checkpoint: 559 seconds<br/> latest_start_position: 12<br/> last_checkpoint_position: 1073741825<br/> first_failure_position: N/A<br/> checked: 406401957<br/> updated: 0<br/> failed: 0<br/> prior_updated: 0<br/> noscrub: 192023<br/> igif: 158502<br/> success_count: 1140<br/> run_time: 4736 seconds<br/> average_speed: 85811 objects/sec<br/> real-time_speed: N/A<br/> current_position: N/A<br/> lf_scanned: 0<br/> lf_reparied: 0<br/> lf_failed: 0</p> <p>Thanks,<br/> Dustin </p> <p>It is strange that the OI scrub has not found inconsistency. It should be some OI scrub issue.<br/> Do you have the MDS side -1 level Lustre kernel debug logs when the "ls" hung? On the other hand, would you please to use "debugfs" to dump the directory and its sub-items that caused the system hung when "ls"? Thanks!</p> <p>Another possible reason is that the FID-in-dirent is corrupted, that can explain why the OI scrub was triggered but no inconsistent OI mapping was found. It can be verified via "lctl set_param fail_loc=0x1505" on the MDS and try "ls" again after the setting. If it still hung there, then it is NOT the case; otherwise, we found the reason.</p> <p>Dustin, do you have more logs or any feedback about trying "lctl set_param fail_loc=0x1505" on the MDS? Thanks!</p> <p>Ping.</p> <p>Sorry for the delays Nasf! We will be having an outage on Feb. 07 to test lustre-2.8 servers and will hopefully leave it in production. After this outage we can run an online lfsck to see if this probably gets resolved. </p> <p>The command "lctl set_param fail_loc=0x1505" was run on the MDS and it fixed the problem. Thanks nasf</p> <p>Sorry, some misguide, "lctl set_param fail_loc=0x1505" is used for bypass FID-in-dirent when it is broken. So if your system hang before, but works well with "lctl set_param fail_loc=0x1505", that means it is quite possible that some FID-in-dirent is broken. Under such case, you need to run namespace LFSCK (with "lctl set_param fail_loc=0") to repair the FID-in-dirent. Otherwise, bypass FID-in-dirent will slowdown the lookup() performance.</p> <p>Since this is the case we will leave this open until the lfsck run.</p> <p>lfsck run seems to have fixed this.</p> Cloners LU-10237 Development Rank 1|hzyrjj: Rank (Obsolete) 9223372036854775807 Severity