https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7459 Lustre <p>We have a process that periodically does a lfs quota throughout the day to monitor the usage on our file system. Recently we have started observing that while the usage numbers are being reported, the number of files that are reported for a given user went from a given number (e.g., 100,000 files) to 0 or 1.</p> <p>This has persisted through system reboots and a OS/Lustre update. Once the given user's file count is reported as 0 or 1 it never changes. However checking the quota for the user's GID will generate a file count. </p> <p>Anecdotally, it appears that only users that are affected are those that have files that have mixed user and group IDs.</p> <p>All the normal checks have been performed (i.e., consistency of UID/GID in the password files between the file servers and clients, checking the MDT for ownership, checking file count with ls -l, etc.). It almost appears that an error condition is being flagged that needs to be cleared.</p> Toss 2.3-4/2.4-2 LU-7459

Incorrect file count with lfs quota

Bug Minor Resolved Incomplete Niu Yawei Joe Mervini Sat, 21 Nov 2015 15:23:53 +0000 Fri, 21 Jul 2017 12:45:35 +0000 Fri, 21 Jul 2017 12:45:35 +0000 Lustre 2.5.4 0 4 <p>Niu</p> <p>Could you please advise</p> <p>Thanks</p> <p>Peter</p> <p>PS/ It was great to see you in person this week Joe!</p> <blockquote> <p> it appears that only users that are affected are those that have files that have mixed user and group IDs.</p></blockquote> <p>Could you give an example to illustrate such situation? I don't quite understand "mixed user and group IDs"</p> <p>Could you pick a problematic user and post the output of "lfs quota -u $UID -v $mount" here? And please check the numbers for this user in the "/proc/fs/lustre/osd-ldiskfs/$fsname-MDT0000/quota_slave/acct_user". Thanks.</p> <p>Here's a sample of the user's top level directory (I've X'ed out most of the file/dir names):</p> <p>total 266<br/> drwxr-xr-x 7 21371 33073 16384 Sep 14 09:36 3XXXXXXXXX<br/> drwxr-xr-x 3 21371 33073 4096 May 12 2015 AFXXXXXXXX<br/> drwxr-xr-x+ 5 21371 33073 4096 Jan 7 2013 ArXXXXXXXXX<br/> drwxr-xr-x 5 21371 33073 36864 Sep 14 09:30 BXXXXXXXXXX<br/> drwxr-xr-x 14 21371 33073 4096 Jan 6 2015 CXXXXXXXXXXX<br/> drwxr-xr-x 8 21371 33073 4096 Sep 14 11:17 From_gscratch2<br/> drwxr-xr-x 3 21371 33073 4096 Aug 19 2014 H2013<br/> drwxr-xr-x+ 2 21371 33073 4096 Mar 12 2013 HXXXXX<br/> drwxr-xr-x 11 21371 33073 4096 Aug 24 2012 hXXXXXX<br/> drwxr-xr-x 9 21371 33073 4096 Oct 8 2014 MXXXXXX<br/> <del>rw</del>------ 1 21371 21371 63681 Sep 14 14:30 MyDiskUsage_09142014<br/> drwxr-xr-x 3 21371 33073 16384 Sep 14 10:33 NXXXXXXX<br/> drwxr-xr-x 2 21371 33073 4096 Jun 17 10:53 P_XXXXXXX<br/> drwxr-xr-x 10 21371 33073 4096 Sep 14 11:45 RXXXXXXX<br/> drwxr-xr-x 4 21371 33073 4096 Jan 30 2012 Test_codes<br/> drwxr-xr-x 3 21371 33073 4096 Mar 10 2015 ValTests<br/> drwxr-xr-x 5 21371 33073 20480 Sep 14 14:27 ZXXXXXX</p> <p>Here is the output from the MDS:</p> <p><span class="error">&#91;root@fmds1 quota_slave&#93;</span># grep -A1 21371 acct_user</p> <ul class="alternate" type="square"> <li>id: 21371<br/> usage: { inodes: 1, kbytes: 0 } <p><span class="error">&#91;root@fmds1 quota_slave&#93;</span># grep -A1 21371 acct_group</p></li> <li>id: 21371<br/> usage: { inodes: 2238, kbytes: 180 } <p><span class="error">&#91;root@fmds1 quota_slave&#93;</span># grep -A1 33073 acct_group</p></li> <li>id: 33073<br/> usage: { inodes: 192927, kbytes: 20400 }</li> </ul> <p>I was going to get a word count on files in the directory using find on the UID but there were a sufficient number of files that after more than 1.5 hours it was still working and I terminated the process.</p> <p>I am also attaching the output from the lfs quota command as you requested. </p> <p>Looks the inode accounting on backend fs is broken somehow, I have no idea why it's broken so far, but I think a quotacheck may fix it. (disable then re-enable quota feature on MDT device by tune2fs, it requires MDT offline)</p> <blockquote> <p>However checking the quota for the user's GID will generate a file count.</p></blockquote> <p>Could you elaborate on this? What do you mean "checking the quota for the user's GID"?</p> <p>That is correct. When I run </p> <p>lfs quota - g &lt;GID&gt; -v &lt;file system&gt; </p> <p>I do get results for whatever GID exists in the directory that is having problems with the UID. So in the example of the top level directory above I get the following outputs: (Note that in our environment by default we assign the same value to both UID and GID for our users.)</p> <ol> <li>lfs quota -g 21371 /fscratch<br/> Disk quotas for group 21371 (gid 21371):<br/> Filesystem kbytes quota limit grace files quota limit grace<br/> /fscratch 275840572 0 0 - 2238 0 0 -</li> <li>lfs quota -g 33073 /fscratch<br/> Disk quotas for group 33073 (gid 33073):<br/> Filesystem kbytes quota limit grace files quota limit grace<br/> /fscratch 20500875399 0 0 - 192927 0 0 -</li> </ol> <p>And once again for illustration:</p> <ol> <li>lfs quota -u 21371 /fscratch<br/> Disk quotas for user 21371 (uid 21371):<br/> Filesystem kbytes quota limit grace files quota limit grace<br/> /fscratch 20773618792 0 0 - 1 0 0 -</li> </ol> <p>We have an scheduled outage that might permit us to do a quota check coming up the 2nd week of December. </p> <p>Just so I'm clear, what is the process for disabling and re-enabling quotas on the newer (&gt;2.4.0) lustre releases. The only references that I find in the manual for using tunefs.lustre is for enabling quotas after upgrading for a version of lustre older than 2.4. I'm guessing there's a --noquota parameter or something similar but there is no procedure defined. I don't know whether it's just a matter of running the tunefs.lustre back to back or if a mount is required so if you can provide that detail I'd appreciate it.</p> <p>The instructions of enable/disable quota is described in manual "21.2.1. Enabling Disk Quotas (Lustre Software Release 2.4 and later)":</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lctl conf_param fsname.quota.ost|mdt=u|g|ug|none </pre> </div></div> <p>Using tunefs.lustre (or tune2fs) is to disable/enable quota feature on the device, which is used to enable quota accounting on backend fs.</p> <p>Oh - sorry, I misread your comment above regarding tune2fs. I get that now.</p> <p>My question is: Is a mount required between the time you set mdt=none and when you set it back to mdt=ug? And when it is switched back on can I simply bring lustre back up and whatever process takes place is done in the background?</p> <p>No, enabling/disabling quota by "lctl conf_param" doesn't require umount/mount, but it do need some small amount of time to let the MGS spread the new configuration onto all servers. (It's same as other parameter settings done by "lctl conf_param"), you can verify if the new configuration is taking effect by checking the proc files on each server targets. (see the quota_slave.info)</p> <p>Joe, is it fixed by enabling/disabling quota feature?</p> <p>It does not seem as if this remains a concern</p> Related LU-7467 Development End date Wed, 2 Dec 2015 15:23:53 +0000 Epic/Theme Quota Rank 1|hzxtxr: Rank (Obsolete) 9223372036854775807 Severity Start date Mon, 23 Nov 2015 15:23:53 +0000