Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-340] system hang when running sanity-quota on RHEL5-x86_64-OFED https://jira.whamcloud.com/browse/LU-340 Lustre <p>system hang when running sanity-quota on RHEL5-x86_64-ofa build. Please see the attachment for all the logs.</p> lustre-master/RHEL5-x86_64/#120/ofa build LU-340

system hang when running sanity-quota on RHEL5-x86_64-OFED

Bug Minor Resolved Duplicate Niu Yawei Sarah Liu Tue, 17 May 2011 22:19:56 +0000 Mon, 1 Apr 2013 03:07:12 +0000 Mon, 1 Apr 2013 03:07:12 +0000 Lustre 2.1.0 Lustre 2.1.1 0 6 <p>Niu</p> <p>Please look into this quotas issue when you get a chance</p> <p>Thanks</p> <p>Peter</p> <p>From the log we can see all pdflush threads on client were waiting on page lock, whereas the dd thread was holding the page lock to do synchronous IO, because of something wrong with group quota, the synchronous I/O can't finish in time, which caused the pdflush threads stalled.</p> <p>What confused me is that there were lots of "dqacq/dqrel failed! (rc:-5)" errors while setting group quota, but setting user quota was done successfully, and the user quota limit tests passed also. Looks there are only two possible cases that dqacq_handler() return -EIO, one is OBD_FAIL_OBD_DQACQ and another is ll_sb_has_quota_active() checking fails.</p> <p>Hi, Sarah</p> <p>Is it repeatable? What's the /proc/fs/lustre/fail_loc on mds? Thanks.</p> <blockquote> <p>Is it repeatable? What's the /proc/fs/lustre/fail_loc on mds? Thanks.</p></blockquote> <p>yes, it can be reproduced. <br/> <span class="error">[root@fat-intel-1 ~]</span># more /proc/sys/lustre/fail_loc <br/> 0</p> <p>Is the D_QUOTA enabled? can we get the debug log on MDS?</p> <blockquote> <p>Is the D_QUOTA enabled?</p></blockquote> <p>no. I can give you debug log tomorrow. please tell me the debug mask</p> <p>I think the default + D_QUOTA will be fine, thank you, Sarah.</p> <p>Thank you, Sarah. I think the debug_log confirmed that dqacq_handler failed for group quota not enabled or fail_loc set.</p> <p>Could you try the following commands on client-5 to see what will happen? (quotacheck then set group quota):<br/> lfs quotacheck -ug lustre_dir<br/> lfs setquota -g group_name -b 0 -B 0 -i 0 -I 0 lustre_dir</p> <p><span class="error">[root@client-15 ~]</span># lfs quotacheck -ug /mnt/lustre/<br/> <span class="error">[root@client-15 ~]</span># lfs setquota -g quota_usr -b 0 -B 0 -i 0 -I 0 /mnt/lustre/<br/> <span class="error">[root@client-15 ~]</span># mount<br/> /dev/sda1 on / type ext3 (rw)<br/> proc on /proc type proc (rw)<br/> sysfs on /sys type sysfs (rw)<br/> devpts on /dev/pts type devpts (rw,gid=5,mode=620)<br/> tmpfs on /dev/shm type tmpfs (rw)<br/> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)<br/> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)<br/> 192.168.4.128@o2ib:/lustre on /mnt/lustre type lustre (rw,flock)</p> <p>When I logon to the system, I found that "lfs quotaon -ug" can't turn on the local fs group quota on mds, though it can be successfully executed and no any abnormal messages in the debug log. </p> <p>The local fs group quota can be enabled by a "lfs quotaon -g", and after the "lfs quotaon -g" executed, the system returned back to normal status, the group quota can be enable/disabled by "lfs quotaon/off -ug" again.</p> <p>This bug appeared only on ofa build server, so I suspect it's ofa build related, will continue the investigation when I have time and spare nodes.</p> <p>Lustre Clients:<br/> Tag: 1.8.6-wc1<br/> Distro/Arch: RHEL5/x86_64 (kernel version: 2.6.18_238.12.1.el5.x86_64)<br/> Build: <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/100/arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/" class="external-link" target="_blank" rel="nofollow noopener">http://newbuild.whamcloud.com/job/lustre-b1_8/100/arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/</a><br/> Network: IB (OFED 1.5.3.1)</p> <p>Lustre Servers:<br/> Tag: v2_1_0_0_RC1<br/> Distro/Arch: RHEL5/x86_64 (kernel version: 2.6.18-238.19.1.el5_lustre.g65156ed.x86_64)<br/> Build: <a href="http://newbuild.whamcloud.com/job/lustre-master/273/arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/" class="external-link" target="_blank" rel="nofollow noopener">http://newbuild.whamcloud.com/job/lustre-master/273/arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/</a><br/> Network: IB (OFED 1.5.3.1)</p> <p>sanity-quota test 1 hung: <a href="https://maloo.whamcloud.com/test_sets/842c0928-cfc6-11e0-8d02-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/842c0928-cfc6-11e0-8d02-52540025f9af</a></p> <p>Dmesg on MDS (fat-amd-1-ib) showed:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == test 1: Block hard limit (normal use and out of quota) === == 01:51:35 Lustre: DEBUG MARKER: User quota (limit: 95511 kbytes) Lustre: DEBUG MARKER: Write ... Lustre: DEBUG MARKER: Done Lustre: DEBUG MARKER: Write out of block quota ... Lustre: DEBUG MARKER: -------------------------------------- Lustre: DEBUG MARKER: Group quota (limit: 95511 kbytes) LustreError: 8250:0:(ldlm_lib.c:2341:target_handle_dqacq_callback()) dqacq/dqrel failed! (rc:-5) LustreError: 8251:0:(ldlm_lib.c:2341:target_handle_dqacq_callback()) dqacq/dqrel failed! (rc:-5) LustreError: 6520:0:(quota_context.c:708:dqacq_completion()) acquire qunit got error! (rc:-5) LustreError: 6520:0:(quota_master.c:1263:mds_init_slave_blimits()) error mds adjust local block quota! (rc:-5) LustreError: 6520:0:(quota_master.c:1442:mds_set_dqblk()) init slave blimits failed! (rc:-5) <~snip~> </pre> </div></div> <p>Lustre Branch: master<br/> Lustre Build: <a href="http://newbuild.whamcloud.com/job/lustre-master/273/" class="external-link" target="_blank" rel="nofollow noopener">http://newbuild.whamcloud.com/job/lustre-master/273/</a><br/> Distro/Arch: RHEL5/x86_64<br/> Network: IB (OFED 1.5.3.1)</p> <p>The same failure occurred while running sanity-quota test: <a href="https://maloo.whamcloud.com/test_sets/4115f084-d2de-11e0-8d02-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4115f084-d2de-11e0-8d02-52540025f9af</a></p> <p>Lustre Tag: v2_1_1_0_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/41/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/41/</a><br/> Distro/Arch: RHEL5/x86_64 (kernel version: 2.6.18-274.12.1.el5)<br/> Network: IB (OFED 1.5.4)</p> <p>The same issue occurred: <a href="https://maloo.whamcloud.com/test_sets/f95cf180-584c-11e1-9df1-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f95cf180-584c-11e1-9df1-5254004bbbd3</a></p> <p>Fixed in <a href="https://jira.whamcloud.com/browse/LU-1782" title="Ignore sb_has_quota_active() in OFED's header" class="issue-link" data-issue-key="LU-1782"><del>LU-1782</del></a>.</p> Related LU-1782 Development Rank 1|hzvf5j: Rank (Obsolete) 6100 Severity