# Initial remarks: 
# We had seen wrong qouta information for a number of users on different 
# file systems, see SR 28763 / LU-4504. 
# We had seen that disabling and enabling quotas for some affected 
# OSTs has repaired the problem. 
# We also had seen that the group quota is sometimes much higher than 
# the sum of the quota of each user in that group. This is not possible 
# since normally only the users of the group create data which belongs 
# to that group. 
# Now, the idea is for each OST to sum quota information of each user 
# and each group and check if the results are similar. This information 
# is found below 
# /proc/fs/lustre/osd-ldiskfs/*OST*/quota_slave/acct_user
# and 
# /proc/fs/lustre/osd-ldiskfs/*OST*/quota_slave/acct_group

# We wrote a perl script which does this check. 

# Check it for file system pfs2dat1: 
[root@pfs2n2 ~]# pdsh -f1 -w pfs2n[4-5] 'for ost in $(ls /proc/fs/lustre/osd-ldiskfs/ | grep OST); do echo $ost:; ./bin/compare_user_group_ost_quotas.pl -d $ost -p 1; done'
pfs2n4: pfs2dat1-OST0000:
pfs2n4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (990.92)!
pfs2n4: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (984.05)!
pfs2n4: pfs2dat1-OST0001:
pfs2n4: pfs2dat1-OST0002:
pfs2n4: pfs2dat1-OST0003:
pfs2n4: pfs2dat1-OST0004:
pfs2n4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (988.47)!
pfs2n4: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (983.05)!
pfs2n4: pfs2dat1-OST0005:
pfs2n4: pfs2dat1-OST0006:
pfs2n4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (988.81)!
pfs2n4: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (984.65)!
pfs2n4: pfs2dat1-OST0007:
pfs2n4: pfs2dat1-OST0008:
pfs2n4: pfs2dat1-OST0009:
pfs2n5: pfs2dat1-OST000a:
pfs2n5: !!! Warning: Difference between user and group inodes is higher than  1 per mil (987.22)!
pfs2n5: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (980.84)!
pfs2n5: pfs2dat1-OST000b:
pfs2n5: pfs2dat1-OST000c:
pfs2n5: pfs2dat1-OST000d:
pfs2n5: pfs2dat1-OST000e:
pfs2n5: pfs2dat1-OST000f:
pfs2n5: pfs2dat1-OST0010:
pfs2n5: !!! Warning: Difference between user and group inodes is higher than  1 per mil (988.64)!
pfs2n5: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (976.58)!
pfs2n5: pfs2dat1-OST0011:
pfs2n5: !!! Warning: Difference between user and group inodes is higher than  1 per mil (988.59)!
pfs2n5: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (977.04)!
pfs2n5: pfs2dat1-OST0012:
pfs2n5: pfs2dat1-OST0013:
[root@pfs2n2 ~]# pdsh -f1 -w pfs2n[4-5] 'for ost in $(ls /proc/fs/lustre/osd-ldiskfs/ | grep OST); do echo $ost:; ./bin/compare_user_group_ost_quotas.pl -d $ost -p 1 -v; done'
pfs2n4: pfs2dat1-OST0000:
pfs2n4: Sum of inodes of users:      4738983
pfs2n4: Sum of inodes of groups:     4782414
pfs2n4: Sum of kbytes of users:   7080867064
pfs2n4: Sum of kbytes of groups:  7195645544
pfs2n4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (990.92)!
pfs2n4: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (984.05)!
pfs2n4: pfs2dat1-OST0001:
pfs2n4: Sum of inodes of users:      4782071
pfs2n4: Sum of inodes of groups:     4782068
pfs2n4: Sum of kbytes of users:   7815053104
pfs2n4: Sum of kbytes of groups:  7815053424
pfs2n4: pfs2dat1-OST0002:
pfs2n4: Sum of inodes of users:      4782710
pfs2n4: Sum of inodes of groups:     4782710
pfs2n4: Sum of kbytes of users:   7784000532
pfs2n4: Sum of kbytes of groups:  7784000532
pfs2n4: pfs2dat1-OST0003:
pfs2n4: Sum of inodes of users:      4783204
pfs2n4: Sum of inodes of groups:     4783204
pfs2n4: Sum of kbytes of users:   7530165124
pfs2n4: Sum of kbytes of groups:  7530165124
pfs2n4: pfs2dat1-OST0004:
pfs2n4: Sum of inodes of users:      4728519
pfs2n4: Sum of inodes of groups:     4783651
pfs2n4: Sum of kbytes of users:   6886328672
pfs2n4: Sum of kbytes of groups:  7005040248
pfs2n4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (988.47)!
pfs2n4: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (983.05)!
pfs2n4: pfs2dat1-OST0005:
pfs2n4: Sum of inodes of users:      4781933
pfs2n4: Sum of inodes of groups:     4781933
pfs2n4: Sum of kbytes of users:   7457294476
pfs2n4: Sum of kbytes of groups:  7457294480
pfs2n4: pfs2dat1-OST0006:
pfs2n4: Sum of inodes of users:      4728723
pfs2n4: Sum of inodes of groups:     4782233
pfs2n4: Sum of kbytes of users:   7572838324
pfs2n4: Sum of kbytes of groups:  7690909532
pfs2n4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (988.81)!
pfs2n4: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (984.65)!
pfs2n4: pfs2dat1-OST0007:
pfs2n4: Sum of inodes of users:      4782918
pfs2n4: Sum of inodes of groups:     4782914
pfs2n4: Sum of kbytes of users:   7208922868
pfs2n4: Sum of kbytes of groups:  7208923200
pfs2n4: pfs2dat1-OST0008:
pfs2n4: Sum of inodes of users:      4782844
pfs2n4: Sum of inodes of groups:     4782844
pfs2n4: Sum of kbytes of users:   7648361528
pfs2n4: Sum of kbytes of groups:  7648361528
pfs2n4: pfs2dat1-OST0009:
pfs2n4: Sum of inodes of users:      4783026
pfs2n4: Sum of inodes of groups:     4783026
pfs2n4: Sum of kbytes of users:   7558875288
pfs2n4: Sum of kbytes of groups:  7558875288
pfs2n5: pfs2dat1-OST000a:
pfs2n5: Sum of inodes of users:      4722091
pfs2n5: Sum of inodes of groups:     4783231
pfs2n5: Sum of kbytes of users:   6569763872
pfs2n5: Sum of kbytes of groups:  6698112704
pfs2n5: !!! Warning: Difference between user and group inodes is higher than  1 per mil (987.22)!
pfs2n5: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (980.84)!
pfs2n5: pfs2dat1-OST000b:
pfs2n5: Sum of inodes of users:      4781452
pfs2n5: Sum of inodes of groups:     4781452
pfs2n5: Sum of kbytes of users:   6618477916
pfs2n5: Sum of kbytes of groups:  6618477940
pfs2n5: pfs2dat1-OST000c:
pfs2n5: Sum of inodes of users:      4783166
pfs2n5: Sum of inodes of groups:     4783165
pfs2n5: Sum of kbytes of users:   7747738112
pfs2n5: Sum of kbytes of groups:  7747738624
pfs2n5: pfs2dat1-OST000d:
pfs2n5: Sum of inodes of users:      4781743
pfs2n5: Sum of inodes of groups:     4781743
pfs2n5: Sum of kbytes of users:   7399866668
pfs2n5: Sum of kbytes of groups:  7399866668
pfs2n5: pfs2dat1-OST000e:
pfs2n5: Sum of inodes of users:      4781055
pfs2n5: Sum of inodes of groups:     4781053
pfs2n5: Sum of kbytes of users:   6956855500
pfs2n5: Sum of kbytes of groups:  6956855508
pfs2n5: pfs2dat1-OST000f:
pfs2n5: Sum of inodes of users:      4781888
pfs2n5: Sum of inodes of groups:     4781888
pfs2n5: Sum of kbytes of users:   6696672124
pfs2n5: Sum of kbytes of groups:  6696673144
pfs2n5: pfs2dat1-OST0010:
pfs2n5: Sum of inodes of users:      4727148
pfs2n5: Sum of inodes of groups:     4781451
pfs2n5: Sum of kbytes of users:   7138312704
pfs2n5: Sum of kbytes of groups:  7309534740
pfs2n5: !!! Warning: Difference between user and group inodes is higher than  1 per mil (988.64)!
pfs2n5: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (976.58)!
pfs2n5: pfs2dat1-OST0011:
pfs2n5: Sum of inodes of users:      4727527
pfs2n5: Sum of inodes of groups:     4782103
pfs2n5: Sum of kbytes of users:   7825394764
pfs2n5: Sum of kbytes of groups:  8009272904
pfs2n5: !!! Warning: Difference between user and group inodes is higher than  1 per mil (988.59)!
pfs2n5: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (977.04)!
pfs2n5: pfs2dat1-OST0012:
pfs2n5: Sum of inodes of users:      4781497
pfs2n5: Sum of inodes of groups:     4781497
pfs2n5: Sum of kbytes of users:   6764213104
pfs2n5: Sum of kbytes of groups:  6764216476
pfs2n5: pfs2dat1-OST0013:
pfs2n5: Sum of inodes of users:      4783039
pfs2n5: Sum of inodes of groups:     4783039
pfs2n5: Sum of kbytes of users:   6750699688
pfs2n5: Sum of kbytes of groups:  6750699692

# Check it for file system pfs2wor1: 
[root@pfs2n2 ~]# pdsh -f1 -w pfs2n[8-11] 'for ost in $(ls /proc/fs/lustre/osd-ldiskfs/ | grep OST); do echo $ost:; ./bin/compare_user_group_ost_quotas.pl -d $ost -p 1; done'
pfs2n8: pfs2wor1-OST0000:
pfs2n8: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.79)!
pfs2n8: pfs2wor1-OST0001:
pfs2n8: pfs2wor1-OST0002:
pfs2n8: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.00)!
pfs2n8: pfs2wor1-OST0003:
pfs2n8: pfs2wor1-OST0004:
pfs2n8: pfs2wor1-OST0005:
pfs2n8: pfs2wor1-OST0006:
pfs2n8: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.27)!
pfs2n8: pfs2wor1-OST0007:
pfs2n8: pfs2wor1-OST0008:
pfs2n8: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (998.32)!
pfs2n8: pfs2wor1-OST0009:
pfs2n8: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (995.84)!
pfs2n9: pfs2wor1-OST000a:
pfs2n9: pfs2wor1-OST000b:
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.61)!
pfs2n9: pfs2wor1-OST000c:
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.18)!
pfs2n9: pfs2wor1-OST000d:
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (995.26)!
pfs2n9: pfs2wor1-OST000e:
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.95)!
pfs2n9: pfs2wor1-OST000f:
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.94)!
pfs2n9: pfs2wor1-OST0010:
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (994.95)!
pfs2n9: pfs2wor1-OST0011:
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.22)!
pfs2n9: pfs2wor1-OST0012:
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (995.17)!
pfs2n9: pfs2wor1-OST0013:
pfs2n10: pfs2wor1-OST0014:
pfs2n10: pfs2wor1-OST0015:
pfs2n10: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.93)!
pfs2n10: pfs2wor1-OST0016:
pfs2n10: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.74)!
pfs2n10: pfs2wor1-OST0017:
pfs2n10: pfs2wor1-OST0018:
pfs2n10: pfs2wor1-OST0019:
pfs2n10: pfs2wor1-OST001a:
pfs2n10: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.28)!
pfs2n10: pfs2wor1-OST001b:
pfs2n10: pfs2wor1-OST001c:
pfs2n10: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.61)!
pfs2n10: pfs2wor1-OST001d:
pfs2n11: pfs2wor1-OST001e:
pfs2n11: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.62)!
pfs2n11: pfs2wor1-OST001f:
pfs2n11: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (995.95)!
pfs2n11: pfs2wor1-OST0020:
pfs2n11: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.10)!
pfs2n11: pfs2wor1-OST0021:
pfs2n11: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (995.95)!
pfs2n11: pfs2wor1-OST0022:
pfs2n11: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (992.10)!
pfs2n11: pfs2wor1-OST0023:
pfs2n11: pfs2wor1-OST0024:
pfs2n11: pfs2wor1-OST0025:
pfs2n11: pfs2wor1-OST0026:
pfs2n11: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.25)!
pfs2n11: pfs2wor1-OST0027:
pfs2n11: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.93)!
[root@pfs2n2 ~]# pdsh -f1 -w pfs2n[8-11] 'for ost in $(ls /proc/fs/lustre/osd-ldiskfs/ | grep OST); do echo $ost:; ./bin/compare_user_group_ost_quotas.pl -d $ost -p 1 -v; done'
pfs2n8: pfs2wor1-OST0000:
pfs2n8: Sum of inodes of users:       188961
pfs2n8: Sum of inodes of groups:      189117
pfs2n8: Sum of kbytes of users:   7573120776
pfs2n8: Sum of kbytes of groups:  7597537108
pfs2n8: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.79)!
pfs2n8: pfs2wor1-OST0001:
pfs2n8: Sum of inodes of users:       188883
pfs2n8: Sum of inodes of groups:      188883
pfs2n8: Sum of kbytes of users:   7494002732
pfs2n8: Sum of kbytes of groups:  7494002732
pfs2n8: pfs2wor1-OST0002:
pfs2n8: Sum of inodes of users:       188965
pfs2n8: Sum of inodes of groups:      189098
pfs2n8: Sum of kbytes of users:   7514191808
pfs2n8: Sum of kbytes of groups:  7536792564
pfs2n8: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.00)!
pfs2n8: pfs2wor1-OST0003:
pfs2n8: Sum of inodes of users:       188944
pfs2n8: Sum of inodes of groups:      188944
pfs2n8: Sum of kbytes of users:   7407025560
pfs2n8: Sum of kbytes of groups:  7407025560
pfs2n8: pfs2wor1-OST0004:
pfs2n8: Sum of inodes of users:       188871
pfs2n8: Sum of inodes of groups:      188871
pfs2n8: Sum of kbytes of users:   7699952788
pfs2n8: Sum of kbytes of groups:  7699952788
pfs2n8: pfs2wor1-OST0005:
pfs2n8: Sum of inodes of users:       188793
pfs2n8: Sum of inodes of groups:      188793
pfs2n8: Sum of kbytes of users:   7415023752
pfs2n8: Sum of kbytes of groups:  7415023752
pfs2n8: pfs2wor1-OST0006:
pfs2n8: Sum of inodes of users:       188742
pfs2n8: Sum of inodes of groups:      188890
pfs2n8: Sum of kbytes of users:   7320942472
pfs2n8: Sum of kbytes of groups:  7348326500
pfs2n8: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.27)!
pfs2n8: pfs2wor1-OST0007:
pfs2n8: Sum of inodes of users:       188703
pfs2n8: Sum of inodes of groups:      188703
pfs2n8: Sum of kbytes of users:   7339432892
pfs2n8: Sum of kbytes of groups:  7339432892
pfs2n8: pfs2wor1-OST0008:
pfs2n8: Sum of inodes of users:       188648
pfs2n8: Sum of inodes of groups:      188759
pfs2n8: Sum of kbytes of users:   7752230680
pfs2n8: Sum of kbytes of groups:  7765281012
pfs2n8: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (998.32)!
pfs2n8: pfs2wor1-OST0009:
pfs2n8: Sum of inodes of users:       188764
pfs2n8: Sum of inodes of groups:      188926
pfs2n8: Sum of kbytes of users:   7435332804
pfs2n8: Sum of kbytes of groups:  7466410252
pfs2n8: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (995.84)!
pfs2n9: pfs2wor1-OST000a:
pfs2n9: Sum of inodes of users:       189219
pfs2n9: Sum of inodes of groups:      189219
pfs2n9: Sum of kbytes of users:   7091042576
pfs2n9: Sum of kbytes of groups:  7091042576
pfs2n9: pfs2wor1-OST000b:
pfs2n9: Sum of inodes of users:       188776
pfs2n9: Sum of inodes of groups:      188897
pfs2n9: Sum of kbytes of users:   7385800996
pfs2n9: Sum of kbytes of groups:  7410938704
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.61)!
pfs2n9: pfs2wor1-OST000c:
pfs2n9: Sum of inodes of users:       188808
pfs2n9: Sum of inodes of groups:      188947
pfs2n9: Sum of kbytes of users:   7361616392
pfs2n9: Sum of kbytes of groups:  7389816860
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.18)!
pfs2n9: pfs2wor1-OST000d:
pfs2n9: Sum of inodes of users:       188891
pfs2n9: Sum of inodes of groups:      189034
pfs2n9: Sum of kbytes of users:   7279004848
pfs2n9: Sum of kbytes of groups:  7313700184
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (995.26)!
pfs2n9: pfs2wor1-OST000e:
pfs2n9: Sum of inodes of users:       188810
pfs2n9: Sum of inodes of groups:      188926
pfs2n9: Sum of kbytes of users:   6983349328
pfs2n9: Sum of kbytes of groups:  7004714068
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.95)!
pfs2n9: pfs2wor1-OST000f:
pfs2n9: Sum of inodes of users:       188798
pfs2n9: Sum of inodes of groups:      188911
pfs2n9: Sum of kbytes of users:   7592973028
pfs2n9: Sum of kbytes of groups:  7616306784
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.94)!
pfs2n9: pfs2wor1-OST0010:
pfs2n9: Sum of inodes of users:       188524
pfs2n9: Sum of inodes of groups:      188663
pfs2n9: Sum of kbytes of users:   7367643076
pfs2n9: Sum of kbytes of groups:  7405072532
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (994.95)!
pfs2n9: pfs2wor1-OST0011:
pfs2n9: Sum of inodes of users:       188935
pfs2n9: Sum of inodes of groups:      189066
pfs2n9: Sum of kbytes of users:   7327613420
pfs2n9: Sum of kbytes of groups:  7348055548
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.22)!
pfs2n9: pfs2wor1-OST0012:
pfs2n9: Sum of inodes of users:       188817
pfs2n9: Sum of inodes of groups:      188919
pfs2n9: Sum of kbytes of users:   6706779228
pfs2n9: Sum of kbytes of groups:  6739330748
pfs2n9: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (995.17)!
pfs2n9: pfs2wor1-OST0013:
pfs2n9: Sum of inodes of users:       189003
pfs2n9: Sum of inodes of groups:      189003
pfs2n9: Sum of kbytes of users:   7284489208
pfs2n9: Sum of kbytes of groups:  7284489208
pfs2n10: pfs2wor1-OST0014:
pfs2n10: Sum of inodes of users:       188741
pfs2n10: Sum of inodes of groups:      188741
pfs2n10: Sum of kbytes of users:   7114763716
pfs2n10: Sum of kbytes of groups:  7114763716
pfs2n10: pfs2wor1-OST0015:
pfs2n10: Sum of inodes of users:       188662
pfs2n10: Sum of inodes of groups:      188794
pfs2n10: Sum of kbytes of users:   7378726380
pfs2n10: Sum of kbytes of groups:  7401436212
pfs2n10: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.93)!
pfs2n10: pfs2wor1-OST0016:
pfs2n10: Sum of inodes of users:       188506
pfs2n10: Sum of inodes of groups:      188653
pfs2n10: Sum of kbytes of users:   7171831584
pfs2n10: Sum of kbytes of groups:  7195280200
pfs2n10: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.74)!
pfs2n10: pfs2wor1-OST0017:
pfs2n10: Sum of inodes of users:       189043
pfs2n10: Sum of inodes of groups:      189043
pfs2n10: Sum of kbytes of users:   7294246088
pfs2n10: Sum of kbytes of groups:  7294246088
pfs2n10: pfs2wor1-OST0018:
pfs2n10: Sum of inodes of users:       188770
pfs2n10: Sum of inodes of groups:      188770
pfs2n10: Sum of kbytes of users:   7492211760
pfs2n10: Sum of kbytes of groups:  7492211760
pfs2n10: pfs2wor1-OST0019:
pfs2n10: Sum of inodes of users:       188865
pfs2n10: Sum of inodes of groups:      188865
pfs2n10: Sum of kbytes of users:   6850500140
pfs2n10: Sum of kbytes of groups:  6850500140
pfs2n10: pfs2wor1-OST001a:
pfs2n10: Sum of inodes of users:       188803
pfs2n10: Sum of inodes of groups:      188932
pfs2n10: Sum of kbytes of users:   7475016956
pfs2n10: Sum of kbytes of groups:  7495371504
pfs2n10: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.28)!
pfs2n10: pfs2wor1-OST001b:
pfs2n10: Sum of inodes of users:       188798
pfs2n10: Sum of inodes of groups:      188798
pfs2n10: Sum of kbytes of users:   7294196724
pfs2n10: Sum of kbytes of groups:  7294196724
pfs2n10: pfs2wor1-OST001c:
pfs2n10: Sum of inodes of users:       188913
pfs2n10: Sum of inodes of groups:      189055
pfs2n10: Sum of kbytes of users:   7217543424
pfs2n10: Sum of kbytes of groups:  7234839352
pfs2n10: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.61)!
pfs2n10: pfs2wor1-OST001d:
pfs2n10: Sum of inodes of users:       188880
pfs2n10: Sum of inodes of groups:      188880
pfs2n10: Sum of kbytes of users:   7688830608
pfs2n10: Sum of kbytes of groups:  7688830608
pfs2n11: pfs2wor1-OST001e:
pfs2n11: Sum of inodes of users:       188668
pfs2n11: Sum of inodes of groups:      188790
pfs2n11: Sum of kbytes of users:   7781906436
pfs2n11: Sum of kbytes of groups:  7808262600
pfs2n11: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.62)!
pfs2n11: pfs2wor1-OST001f:
pfs2n11: Sum of inodes of users:       188736
pfs2n11: Sum of inodes of groups:      188871
pfs2n11: Sum of kbytes of users:   7591940768
pfs2n11: Sum of kbytes of groups:  7622846980
pfs2n11: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (995.95)!
pfs2n11: pfs2wor1-OST0020:
pfs2n11: Sum of inodes of users:       188949
pfs2n11: Sum of inodes of groups:      189079
pfs2n11: Sum of kbytes of users:   7052805184
pfs2n11: Sum of kbytes of groups:  7080398540
pfs2n11: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (996.10)!
pfs2n11: pfs2wor1-OST0021:
pfs2n11: Sum of inodes of users:       189019
pfs2n11: Sum of inodes of groups:      189135
pfs2n11: Sum of kbytes of users:   7492171972
pfs2n11: Sum of kbytes of groups:  7522624544
pfs2n11: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (995.95)!
pfs2n11: pfs2wor1-OST0022:
pfs2n11: Sum of inodes of users:       188803
pfs2n11: Sum of inodes of groups:      188950
pfs2n11: Sum of kbytes of users:   7218623224
pfs2n11: Sum of kbytes of groups:  7276068416
pfs2n11: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (992.10)!
pfs2n11: pfs2wor1-OST0023:
pfs2n11: Sum of inodes of users:       189192
pfs2n11: Sum of inodes of groups:      189192
pfs2n11: Sum of kbytes of users:   7473360092
pfs2n11: Sum of kbytes of groups:  7473360092
pfs2n11: pfs2wor1-OST0024:
pfs2n11: Sum of inodes of users:       188886
pfs2n11: Sum of inodes of groups:      188886
pfs2n11: Sum of kbytes of users:   7379999844
pfs2n11: Sum of kbytes of groups:  7379999844
pfs2n11: pfs2wor1-OST0025:
pfs2n11: Sum of inodes of users:       188795
pfs2n11: Sum of inodes of groups:      188795
pfs2n11: Sum of kbytes of users:   7252287736
pfs2n11: Sum of kbytes of groups:  7252287736
pfs2n11: pfs2wor1-OST0026:
pfs2n11: Sum of inodes of users:       188631
pfs2n11: Sum of inodes of groups:      188758
pfs2n11: Sum of kbytes of users:   7000329580
pfs2n11: Sum of kbytes of groups:  7019618920
pfs2n11: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.25)!
pfs2n11: pfs2wor1-OST0027:
pfs2n11: Sum of inodes of users:       188815
pfs2n11: Sum of inodes of groups:      188954
pfs2n11: Sum of kbytes of users:   7564178220
pfs2n11: Sum of kbytes of groups:  7579843112
pfs2n11: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.93)!

# Check it for file system pfs2dat2: 
[root@pfs2n12 ~]# pdsh -f1 -w pfs2n[14-15] 'for ost in $(ls /proc/fs/lustre/osd-ldiskfs/ | grep OST); do echo $ost:; ./bin/compare_user_group_ost_quotas.pl -d $ost -p 1; done'
pfs2n14: pfs2dat2-OST0000:
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (990.66)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (981.41)!
pfs2n14: pfs2dat2-OST0001:
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (990.90)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (978.29)!
pfs2n14: pfs2dat2-OST0002:
pfs2n14: pfs2dat2-OST0003:
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (992.21)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (978.32)!
pfs2n14: pfs2dat2-OST0004:
pfs2n14: pfs2dat2-OST0005:
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (991.01)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (976.37)!
pfs2n14: pfs2dat2-OST0006:
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (989.75)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (981.77)!
pfs2n14: pfs2dat2-OST0007:
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (992.48)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (978.27)!
pfs2n14: pfs2dat2-OST0008:
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (991.60)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (984.29)!
pfs2n14: pfs2dat2-OST0009:
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (992.38)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (974.88)!
pfs2n15: pfs2dat2-OST000a:
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (991.50)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (982.48)!
pfs2n15: pfs2dat2-OST000b:
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1127.22)!
pfs2n15: pfs2dat2-OST000c:
pfs2n15: pfs2dat2-OST000d:
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (990.78)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (981.84)!
pfs2n15: pfs2dat2-OST000e:
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (991.50)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (982.92)!
pfs2n15: pfs2dat2-OST000f:
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (991.57)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (980.11)!
pfs2n15: pfs2dat2-OST0010:
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (989.90)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (970.88)!
pfs2n15: pfs2dat2-OST0011:
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (990.96)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (971.21)!
pfs2n15: pfs2dat2-OST0012:
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (992.48)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (980.61)!
pfs2n15: pfs2dat2-OST0013:
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (990.77)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (979.76)!
[root@pfs2n12 ~]# pdsh -f1 -w pfs2n[14-15] 'for ost in $(ls /proc/fs/lustre/osd-ldiskfs/ | grep OST); do echo $ost:; ./bin/compare_user_group_ost_quotas.pl -d $ost -p 1 -v; done'
pfs2n14: pfs2dat2-OST0000:
pfs2n14: Sum of inodes of users:      9974682
pfs2n14: Sum of inodes of groups:    10068737
pfs2n14: Sum of kbytes of users:  10782769240
pfs2n14: Sum of kbytes of groups: 10986969764
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (990.66)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (981.41)!
pfs2n14: pfs2dat2-OST0001:
pfs2n14: Sum of inodes of users:      9936007
pfs2n14: Sum of inodes of groups:    10027249
pfs2n14: Sum of kbytes of users:   9661257976
pfs2n14: Sum of kbytes of groups:  9875632280
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (990.90)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (978.29)!
pfs2n14: pfs2dat2-OST0002:
pfs2n14: Sum of inodes of users:     10066173
pfs2n14: Sum of inodes of groups:    10066173
pfs2n14: Sum of kbytes of users:  10942975596
pfs2n14: Sum of kbytes of groups: 10942975596
pfs2n14: pfs2dat2-OST0003:
pfs2n14: Sum of inodes of users:      9989373
pfs2n14: Sum of inodes of groups:    10067753
pfs2n14: Sum of kbytes of users:  10159540264
pfs2n14: Sum of kbytes of groups: 10384637168
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (992.21)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (978.32)!
pfs2n14: pfs2dat2-OST0004:
pfs2n14: Sum of inodes of users:     10036694
pfs2n14: Sum of inodes of groups:    10041757
pfs2n14: Sum of kbytes of users:  10269975416
pfs2n14: Sum of kbytes of groups: 10272424068
pfs2n14: pfs2dat2-OST0005:
pfs2n14: Sum of inodes of users:      9978007
pfs2n14: Sum of inodes of groups:    10068505
pfs2n14: Sum of kbytes of users:  10663288192
pfs2n14: Sum of kbytes of groups: 10921356128
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (991.01)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (976.37)!
pfs2n14: pfs2dat2-OST0006:
pfs2n14: Sum of inodes of users:      9929200
pfs2n14: Sum of inodes of groups:    10032056
pfs2n14: Sum of kbytes of users:  10356814844
pfs2n14: Sum of kbytes of groups: 10549069292
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (989.75)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (981.78)!
pfs2n14: pfs2dat2-OST0007:
pfs2n14: Sum of inodes of users:      9991570
pfs2n14: Sum of inodes of groups:    10067259
pfs2n14: Sum of kbytes of users:   9739260296
pfs2n14: Sum of kbytes of groups:  9955605924
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (992.48)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (978.27)!
pfs2n14: pfs2dat2-OST0008:
pfs2n14: Sum of inodes of users:      9983205
pfs2n14: Sum of inodes of groups:    10067804
pfs2n14: Sum of kbytes of users:  10465240440
pfs2n14: Sum of kbytes of groups: 10632287196
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (991.60)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (984.29)!
pfs2n14: pfs2dat2-OST0009:
pfs2n14: Sum of inodes of users:      9946518
pfs2n14: Sum of inodes of groups:    10022888
pfs2n14: Sum of kbytes of users:  10098546196
pfs2n14: Sum of kbytes of groups: 10358713324
pfs2n14: !!! Warning: Difference between user and group inodes is higher than  1 per mil (992.38)!
pfs2n14: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (974.88)!
pfs2n15: pfs2dat2-OST000a:
pfs2n15: Sum of inodes of users:      9979422
pfs2n15: Sum of inodes of groups:    10065022
pfs2n15: Sum of kbytes of users:   9977790840
pfs2n15: Sum of kbytes of groups: 10155729784
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (991.50)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (982.48)!
pfs2n15: pfs2dat2-OST000b:
pfs2n15: Sum of inodes of users:      9944733
pfs2n15: Sum of inodes of groups:     9942573
pfs2n15: Sum of kbytes of users:   9641609592
pfs2n15: Sum of kbytes of groups:  8553474664
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1127.22)!
pfs2n15: pfs2dat2-OST000c:
pfs2n15: Sum of inodes of users:     10067192
pfs2n15: Sum of inodes of groups:    10067192
pfs2n15: Sum of kbytes of users:  10114696000
pfs2n15: Sum of kbytes of groups: 10114696000
pfs2n15: pfs2dat2-OST000d:
pfs2n15: Sum of inodes of users:      9976176
pfs2n15: Sum of inodes of groups:    10069048
pfs2n15: Sum of kbytes of users:  10357118700
pfs2n15: Sum of kbytes of groups: 10548711344
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (990.78)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (981.84)!
pfs2n15: pfs2dat2-OST000e:
pfs2n15: Sum of inodes of users:      9981960
pfs2n15: Sum of inodes of groups:    10067534
pfs2n15: Sum of kbytes of users:  10486778508
pfs2n15: Sum of kbytes of groups: 10668968840
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (991.50)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (982.92)!
pfs2n15: pfs2dat2-OST000f:
pfs2n15: Sum of inodes of users:      9982026
pfs2n15: Sum of inodes of groups:    10066894
pfs2n15: Sum of kbytes of users:   9921680540
pfs2n15: Sum of kbytes of groups: 10123042704
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (991.57)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (980.11)!
pfs2n15: pfs2dat2-OST0010:
pfs2n15: Sum of inodes of users:      9948955
pfs2n15: Sum of inodes of groups:    10050486
pfs2n15: Sum of kbytes of users:   9836958212
pfs2n15: Sum of kbytes of groups: 10131956964
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (989.90)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (970.88)!
pfs2n15: pfs2dat2-OST0011:
pfs2n15: Sum of inodes of users:      9974364
pfs2n15: Sum of inodes of groups:    10065310
pfs2n15: Sum of kbytes of users:  10610953944
pfs2n15: Sum of kbytes of groups: 10925487580
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (990.96)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (971.21)!
pfs2n15: pfs2dat2-OST0012:
pfs2n15: Sum of inodes of users:      9991902
pfs2n15: Sum of inodes of groups:    10067597
pfs2n15: Sum of kbytes of users:  10293715872
pfs2n15: Sum of kbytes of groups: 10497265808
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (992.48)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (980.61)!
pfs2n15: pfs2dat2-OST0013:
pfs2n15: Sum of inodes of users:      9975685
pfs2n15: Sum of inodes of groups:    10068597
pfs2n15: Sum of kbytes of users:   9862071092
pfs2n15: Sum of kbytes of groups: 10065831992
pfs2n15: !!! Warning: Difference between user and group inodes is higher than  1 per mil (990.77)!
pfs2n15: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (979.76)!

# Check it for file system pfs2wor2: 
[root@pfs2n12 ~]# pdsh -f1 -w pfs2n[18-21] 'for ost in $(ls /proc/fs/lustre/osd-ldiskfs/ | grep OST); do echo $ost:; ./bin/compare_user_group_ost_quotas.pl -d $ost -p 1; done'
pfs2n18: pfs2wor2-OST0000:
pfs2n18: pfs2wor2-OST0001:
pfs2n18: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1002.91)!
pfs2n18: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1468.27)!
pfs2n18: pfs2wor2-OST0002:
pfs2n18: pfs2wor2-OST0003:
pfs2n18: pfs2wor2-OST0004:
pfs2n18: pfs2wor2-OST0005:
pfs2n18: pfs2wor2-OST0006:
pfs2n18: pfs2wor2-OST0007:
pfs2n18: pfs2wor2-OST0008:
pfs2n18: pfs2wor2-OST0009:
pfs2n19: pfs2wor2-OST000a:
pfs2n19: pfs2wor2-OST000b:
pfs2n19: pfs2wor2-OST000c:
pfs2n19: pfs2wor2-OST000d:
pfs2n19: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1481.32)!
pfs2n19: pfs2wor2-OST000e:
pfs2n19: pfs2wor2-OST000f:
pfs2n19: pfs2wor2-OST0010:
pfs2n19: pfs2wor2-OST0011:
pfs2n19: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1391.39)!
pfs2n19: pfs2wor2-OST0012:
pfs2n19: !!! Warning: Difference between user and group inodes is higher than  1 per mil (997.50)!
pfs2n19: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (994.52)!
pfs2n19: pfs2wor2-OST0013:
pfs2n20: pfs2wor2-OST0014:
pfs2n20: pfs2wor2-OST0015:
pfs2n20: pfs2wor2-OST0016:
pfs2n20: pfs2wor2-OST0017:
pfs2n20: pfs2wor2-OST0018:
pfs2n20: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1422.49)!
pfs2n20: pfs2wor2-OST0019:
pfs2n20: pfs2wor2-OST001a:
pfs2n20: pfs2wor2-OST001b:
pfs2n20: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1428.88)!
pfs2n20: pfs2wor2-OST001c:
pfs2n20: pfs2wor2-OST001d:
pfs2n21: pfs2wor2-OST001e:
pfs2n21: pfs2wor2-OST001f:
pfs2n21: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1436.51)!
pfs2n21: pfs2wor2-OST0020:
pfs2n21: pfs2wor2-OST0021:
pfs2n21: pfs2wor2-OST0022:
pfs2n21: pfs2wor2-OST0023:
pfs2n21: pfs2wor2-OST0024:
pfs2n21: pfs2wor2-OST0025:
pfs2n21: pfs2wor2-OST0026:
pfs2n21: pfs2wor2-OST0027:
[root@pfs2n12 ~]# pdsh -f1 -w pfs2n[18-21] 'for ost in $(ls /proc/fs/lustre/osd-ldiskfs/ | grep OST); do echo $ost:; ./bin/compare_user_group_ost_quotas.pl -d $ost -p 1 -v; done'
pfs2n18: pfs2wor2-OST0000:
pfs2n18: Sum of inodes of users:      3020866
pfs2n18: Sum of inodes of groups:     3020866
pfs2n18: Sum of kbytes of users:   2204918036
pfs2n18: Sum of kbytes of groups:  2204918036
pfs2n18: pfs2wor2-OST0001:
pfs2n18: Sum of inodes of users:      3023021
pfs2n18: Sum of inodes of groups:     3014243
pfs2n18: Sum of kbytes of users:   1806787456
pfs2n18: Sum of kbytes of groups:  1230629340
pfs2n18: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1002.91)!
pfs2n18: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1468.18)!
pfs2n18: pfs2wor2-OST0002:
pfs2n18: Sum of inodes of users:      3024043
pfs2n18: Sum of inodes of groups:     3024043
pfs2n18: Sum of kbytes of users:   2005611364
pfs2n18: Sum of kbytes of groups:  2005611364
pfs2n18: pfs2wor2-OST0003:
pfs2n18: Sum of inodes of users:      3023165
pfs2n18: Sum of inodes of groups:     3023165
pfs2n18: Sum of kbytes of users:   2083341200
pfs2n18: Sum of kbytes of groups:  2083341200
pfs2n18: pfs2wor2-OST0004:
pfs2n18: Sum of inodes of users:      3026460
pfs2n18: Sum of inodes of groups:     3026460
pfs2n18: Sum of kbytes of users:   2032110708
pfs2n18: Sum of kbytes of groups:  2032110708
pfs2n18: pfs2wor2-OST0005:
pfs2n18: Sum of inodes of users:      3024620
pfs2n18: Sum of inodes of groups:     3024620
pfs2n18: Sum of kbytes of users:   1857749216
pfs2n18: Sum of kbytes of groups:  1857749216
pfs2n18: pfs2wor2-OST0006:
pfs2n18: Sum of inodes of users:      3024754
pfs2n18: Sum of inodes of groups:     3024754
pfs2n18: Sum of kbytes of users:   2008456868
pfs2n18: Sum of kbytes of groups:  2008456692
pfs2n18: pfs2wor2-OST0007:
pfs2n18: Sum of inodes of users:      2541978
pfs2n18: Sum of inodes of groups:     2541978
pfs2n18: Sum of kbytes of users:   1778305836
pfs2n18: Sum of kbytes of groups:  1778305836
pfs2n18: pfs2wor2-OST0008:
pfs2n18: Sum of inodes of users:      3022778
pfs2n18: Sum of inodes of groups:     3022778
pfs2n18: Sum of kbytes of users:   4301396440
pfs2n18: Sum of kbytes of groups:  4301396440
pfs2n18: pfs2wor2-OST0009:
pfs2n18: Sum of inodes of users:      3027607
pfs2n18: Sum of inodes of groups:     3027607
pfs2n18: Sum of kbytes of users:   1972546412
pfs2n18: Sum of kbytes of groups:  1972546412
pfs2n19: pfs2wor2-OST000a:
pfs2n19: Sum of inodes of users:      3041337
pfs2n19: Sum of inodes of groups:     3041337
pfs2n19: Sum of kbytes of users:   1963265396
pfs2n19: Sum of kbytes of groups:  1963265396
pfs2n19: pfs2wor2-OST000b:
pfs2n19: Sum of inodes of users:      3035404
pfs2n19: Sum of inodes of groups:     3035404
pfs2n19: Sum of kbytes of users:   1858498140
pfs2n19: Sum of kbytes of groups:  1858498140
pfs2n19: pfs2wor2-OST000c:
pfs2n19: Sum of inodes of users:      3041213
pfs2n19: Sum of inodes of groups:     3041213
pfs2n19: Sum of kbytes of users:   2333606764
pfs2n19: Sum of kbytes of groups:  2333606764
pfs2n19: pfs2wor2-OST000d:
pfs2n19: Sum of inodes of users:      3042441
pfs2n19: Sum of inodes of groups:     3041208
pfs2n19: Sum of kbytes of users:   1785530804
pfs2n19: Sum of kbytes of groups:  1206008164
pfs2n19: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1480.53)!
pfs2n19: pfs2wor2-OST000e:
pfs2n19: Sum of inodes of users:      3041263
pfs2n19: Sum of inodes of groups:     3041263
pfs2n19: Sum of kbytes of users:   2031097876
pfs2n19: Sum of kbytes of groups:  2031097876
pfs2n19: pfs2wor2-OST000f:
pfs2n19: Sum of inodes of users:      3047400
pfs2n19: Sum of inodes of groups:     3047400
pfs2n19: Sum of kbytes of users:   2115274372
pfs2n19: Sum of kbytes of groups:  2115274372
pfs2n19: pfs2wor2-OST0010:
pfs2n19: Sum of inodes of users:      3043340
pfs2n19: Sum of inodes of groups:     3043339
pfs2n19: Sum of kbytes of users:   1978921900
pfs2n19: Sum of kbytes of groups:  1978921900
pfs2n19: pfs2wor2-OST0011:
pfs2n19: Sum of inodes of users:      3040677
pfs2n19: Sum of inodes of groups:     3039485
pfs2n19: Sum of kbytes of users:   1837323776
pfs2n19: Sum of kbytes of groups:  1320523068
pfs2n19: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1391.36)!
pfs2n19: pfs2wor2-OST0012:
pfs2n19: Sum of inodes of users:      3037042
pfs2n19: Sum of inodes of groups:     3044654
pfs2n19: Sum of kbytes of users:   4110444240
pfs2n19: Sum of kbytes of groups:  4133117052
pfs2n19: !!! Warning: Difference between user and group inodes is higher than  1 per mil (997.50)!
pfs2n19: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (994.51)!
pfs2n19: pfs2wor2-OST0013:
pfs2n19: Sum of inodes of users:      3040135
pfs2n19: Sum of inodes of groups:     3040135
pfs2n19: Sum of kbytes of users:   2090624972
pfs2n19: Sum of kbytes of groups:  2090624972
pfs2n20: pfs2wor2-OST0014:
pfs2n20: Sum of inodes of users:      3042717
pfs2n20: Sum of inodes of groups:     3042717
pfs2n20: Sum of kbytes of users:   1984352416
pfs2n20: Sum of kbytes of groups:  1984352416
pfs2n20: pfs2wor2-OST0015:
pfs2n20: Sum of inodes of users:      3042235
pfs2n20: Sum of inodes of groups:     3042221
pfs2n20: Sum of kbytes of users:   1742573876
pfs2n20: Sum of kbytes of groups:  1742498888
pfs2n20: pfs2wor2-OST0016:
pfs2n20: Sum of inodes of users:      3044045
pfs2n20: Sum of inodes of groups:     3044045
pfs2n20: Sum of kbytes of users:   2009813508
pfs2n20: Sum of kbytes of groups:  2009813508
pfs2n20: pfs2wor2-OST0017:
pfs2n20: Sum of inodes of users:      3043094
pfs2n20: Sum of inodes of groups:     3043094
pfs2n20: Sum of kbytes of users:   2019202528
pfs2n20: Sum of kbytes of groups:  2019202528
pfs2n20: pfs2wor2-OST0018:
pfs2n20: Sum of inodes of users:      3045542
pfs2n20: Sum of inodes of groups:     3044330
pfs2n20: Sum of kbytes of users:   1768477824
pfs2n20: Sum of kbytes of groups:  1243321780
pfs2n20: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1422.38)!
pfs2n20: pfs2wor2-OST0019:
pfs2n20: Sum of inodes of users:      3041616
pfs2n20: Sum of inodes of groups:     3041616
pfs2n20: Sum of kbytes of users:   2118816204
pfs2n20: Sum of kbytes of groups:  2118816204
pfs2n20: pfs2wor2-OST001a:
pfs2n20: Sum of inodes of users:      3040832
pfs2n20: Sum of inodes of groups:     3040832
pfs2n20: Sum of kbytes of users:   1987325476
pfs2n20: Sum of kbytes of groups:  1987325476
pfs2n20: pfs2wor2-OST001b:
pfs2n20: Sum of inodes of users:      3041386
pfs2n20: Sum of inodes of groups:     3040004
pfs2n20: Sum of kbytes of users:   1879241376
pfs2n20: Sum of kbytes of groups:  1315190652
pfs2n20: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1428.87)!
pfs2n20: pfs2wor2-OST001c:
pfs2n20: Sum of inodes of users:      3041051
pfs2n20: Sum of inodes of groups:     3041051
pfs2n20: Sum of kbytes of users:   2006749716
pfs2n20: Sum of kbytes of groups:  2006749716
pfs2n20: pfs2wor2-OST001d:
pfs2n20: Sum of inodes of users:      3043125
pfs2n20: Sum of inodes of groups:     3043125
pfs2n20: Sum of kbytes of users:   1916119124
pfs2n20: Sum of kbytes of groups:  1916119124
pfs2n21: pfs2wor2-OST001e:
pfs2n21: Sum of inodes of users:      3027372
pfs2n21: Sum of inodes of groups:     3027372
pfs2n21: Sum of kbytes of users:   1952990232
pfs2n21: Sum of kbytes of groups:  1952990232
pfs2n21: pfs2wor2-OST001f:
pfs2n21: Sum of inodes of users:      3017459
pfs2n21: Sum of inodes of groups:     3016264
pfs2n21: Sum of kbytes of users:   1809188444
pfs2n21: Sum of kbytes of groups:  1259411932
pfs2n21: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1436.53)!
pfs2n21: pfs2wor2-OST0020:
pfs2n21: Sum of inodes of users:      3026145
pfs2n21: Sum of inodes of groups:     3026145
pfs2n21: Sum of kbytes of users:   2208019740
pfs2n21: Sum of kbytes of groups:  2208019740
pfs2n21: pfs2wor2-OST0021:
pfs2n21: Sum of inodes of users:      3023624
pfs2n21: Sum of inodes of groups:     3023624
pfs2n21: Sum of kbytes of users:   2090782936
pfs2n21: Sum of kbytes of groups:  2090782936
pfs2n21: pfs2wor2-OST0022:
pfs2n21: Sum of inodes of users:      3021354
pfs2n21: Sum of inodes of groups:     3021354
pfs2n21: Sum of kbytes of users:   1946717832
pfs2n21: Sum of kbytes of groups:  1946717832
pfs2n21: pfs2wor2-OST0023:
pfs2n21: Sum of inodes of users:      3022725
pfs2n21: Sum of inodes of groups:     3022714
pfs2n21: Sum of kbytes of users:   1769923476
pfs2n21: Sum of kbytes of groups:  1769916140
pfs2n21: pfs2wor2-OST0024:
pfs2n21: Sum of inodes of users:      3023374
pfs2n21: Sum of inodes of groups:     3023374
pfs2n21: Sum of kbytes of users:   2047987956
pfs2n21: Sum of kbytes of groups:  2047987956
pfs2n21: pfs2wor2-OST0025:
pfs2n21: Sum of inodes of users:      3010879
pfs2n21: Sum of inodes of groups:     3010879
pfs2n21: Sum of kbytes of users:   2056885696
pfs2n21: Sum of kbytes of groups:  2056885696
pfs2n21: pfs2wor2-OST0026:
pfs2n21: Sum of inodes of users:      3025075
pfs2n21: Sum of inodes of groups:     3025075
pfs2n21: Sum of kbytes of users:   1971867100
pfs2n21: Sum of kbytes of groups:  1971867100
pfs2n21: pfs2wor2-OST0027:
pfs2n21: Sum of inodes of users:      3027348
pfs2n21: Sum of inodes of groups:     3027348
pfs2n21: Sum of kbytes of users:   1997209652
pfs2n21: Sum of kbytes of groups:  1997209652

# Check it for file system HC3WORK: 
[root@mds1 ~]# pdsh -f1 -w oss[1-4] 'for ost in $(ls /proc/fs/lustre/osd-ldiskfs/ | grep OST); do echo $ost:; ./bin/compare_user_group_ost_quotas.pl -d $ost -p 1; done'
oss1: HC3WORK-OST0000:
oss1: !!! Warning: Difference between user and group inodes is higher than  1 per mil (967.10)!
oss1: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (998.74)!
oss1: HC3WORK-OST0001:
oss1: HC3WORK-OST0002:
oss1: !!! Warning: Difference between user and group inodes is higher than  1 per mil (966.73)!
oss1: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.81)!
oss1: HC3WORK-OST0003:
oss1: !!! Warning: Difference between user and group inodes is higher than  1 per mil (967.52)!
oss1: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (998.74)!
oss1: HC3WORK-OST0004:
oss1: HC3WORK-OST0005:
oss1: HC3WORK-OST0006:
oss1: !!! Warning: Difference between user and group inodes is higher than  1 per mil (967.12)!
oss1: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (998.75)!
oss2: HC3WORK-OST0007:
oss2: HC3WORK-OST0008:
oss2: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1019.58)!
oss2: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1001.84)!
oss2: HC3WORK-OST0009:
oss2: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1018.05)!
oss2: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1001.01)!
oss2: HC3WORK-OST000a:
oss2: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1001.67)!
oss2: HC3WORK-OST000b:
oss2: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1020.26)!
oss2: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1001.23)!
oss2: HC3WORK-OST000c:
oss2: HC3WORK-OST000d:
oss2: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1019.84)!
oss2: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1001.44)!
oss3: HC3WORK-OST000e:
oss3: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1019.14)!
oss3: HC3WORK-OST000f:
oss3: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1019.20)!
oss3: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1001.09)!
oss3: HC3WORK-OST0010:
oss3: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1019.07)!
oss3: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1005.05)!
oss3: HC3WORK-OST0011:
oss3: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1018.89)!
oss3: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1001.29)!
oss3: HC3WORK-OST0012:
oss3: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1019.40)!
oss3: HC3WORK-OST0013:
oss3: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1018.26)!
oss3: HC3WORK-OST0014:
oss4: HC3WORK-OST0015:
oss4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (992.11)!
oss4: HC3WORK-OST0016:
oss4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (975.32)!
oss4: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.10)!
oss4: HC3WORK-OST0017:
oss4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (991.49)!
oss4: HC3WORK-OST0018:
oss4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (992.41)!
oss4: HC3WORK-OST0019:
oss4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (991.83)!
oss4: HC3WORK-OST001a:
oss4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (975.89)!
oss4: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.52)!
oss4: HC3WORK-OST001b:
oss4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (987.86)!
[root@mds1 ~]# pdsh -f1 -w oss[1-4] 'for ost in $(ls /proc/fs/lustre/osd-ldiskfs/ | grep OST); do echo $ost:; ./bin/compare_user_group_ost_quotas.pl -d $ost -p 1 -v; done'
oss1: HC3WORK-OST0000:
oss1: Sum of inodes of users:       611885
oss1: Sum of inodes of groups:      632698
oss1: Sum of kbytes of users:   4819512996
oss1: Sum of kbytes of groups:  4825598496
oss1: !!! Warning: Difference between user and group inodes is higher than  1 per mil (967.10)!
oss1: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (998.74)!
oss1: HC3WORK-OST0001:
oss1: Sum of inodes of users:       471247
oss1: Sum of inodes of groups:      471247
oss1: Sum of kbytes of users:   4647915908
oss1: Sum of kbytes of groups:  4647915908
oss1: HC3WORK-OST0002:
oss1: Sum of inodes of users:       647938
oss1: Sum of inodes of groups:      670239
oss1: Sum of kbytes of users:   4640578232
oss1: Sum of kbytes of groups:  4650754220
oss1: !!! Warning: Difference between user and group inodes is higher than  1 per mil (966.73)!
oss1: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.81)!
oss1: HC3WORK-OST0003:
oss1: Sum of inodes of users:       619645
oss1: Sum of inodes of groups:      640447
oss1: Sum of kbytes of users:   4763440176
oss1: Sum of kbytes of groups:  4769445040
oss1: !!! Warning: Difference between user and group inodes is higher than  1 per mil (967.52)!
oss1: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (998.74)!
oss1: HC3WORK-OST0004:
oss1: Sum of inodes of users:       605124
oss1: Sum of inodes of groups:      605133
oss1: Sum of kbytes of users:   5001864732
oss1: Sum of kbytes of groups:  5001572732
oss1: HC3WORK-OST0005:
oss1: Sum of inodes of users:       460658
oss1: Sum of inodes of groups:      460658
oss1: Sum of kbytes of users:   4791129448
oss1: Sum of kbytes of groups:  4791129448
oss1: HC3WORK-OST0006:
oss1: Sum of inodes of users:       623858
oss1: Sum of inodes of groups:      645068
oss1: Sum of kbytes of users:   4781295212
oss1: Sum of kbytes of groups:  4787281000
oss1: !!! Warning: Difference between user and group inodes is higher than  1 per mil (967.12)!
oss1: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (998.75)!
oss2: HC3WORK-OST0007:
oss2: Sum of inodes of users:       502313
oss2: Sum of inodes of groups:      502359
oss2: Sum of kbytes of users:   4982698540
oss2: Sum of kbytes of groups:  4983675624
oss2: HC3WORK-OST0008:
oss2: Sum of inodes of users:       524849
oss2: Sum of inodes of groups:      514770
oss2: Sum of kbytes of users:   4901041744
oss2: Sum of kbytes of groups:  4892051532
oss2: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1019.58)!
oss2: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1001.84)!
oss2: HC3WORK-OST0009:
oss2: Sum of inodes of users:       550756
oss2: Sum of inodes of groups:      540993
oss2: Sum of kbytes of users:   4770619916
oss2: Sum of kbytes of groups:  4765799764
oss2: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1018.05)!
oss2: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1001.01)!
oss2: HC3WORK-OST000a:
oss2: Sum of inodes of users:       535050
oss2: Sum of inodes of groups:      534158
oss2: Sum of kbytes of users:   4879926360
oss2: Sum of kbytes of groups:  4879295040
oss2: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1001.67)!
oss2: HC3WORK-OST000b:
oss2: Sum of inodes of users:       560820
oss2: Sum of inodes of groups:      549686
oss2: Sum of kbytes of users:   4640771508
oss2: Sum of kbytes of groups:  4635087576
oss2: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1020.26)!
oss2: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1001.23)!
oss2: HC3WORK-OST000c:
oss2: Sum of inodes of users:       469157
oss2: Sum of inodes of groups:      469157
oss2: Sum of kbytes of users:   4698253156
oss2: Sum of kbytes of groups:  4698253152
oss2: HC3WORK-OST000d:
oss2: Sum of inodes of users:       537846
oss2: Sum of inodes of groups:      527385
oss2: Sum of kbytes of users:   4811456228
oss2: Sum of kbytes of groups:  4804523296
oss2: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1019.84)!
oss2: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1001.44)!
oss3: HC3WORK-OST000e:
oss3: Sum of inodes of users:       511476
oss3: Sum of inodes of groups:      501869
oss3: Sum of kbytes of users:   4886517916
oss3: Sum of kbytes of groups:  4882088040
oss3: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1019.14)!
oss3: HC3WORK-OST000f:
oss3: Sum of inodes of users:       532406
oss3: Sum of inodes of groups:      522378
oss3: Sum of kbytes of users:   4736814636
oss3: Sum of kbytes of groups:  4731671052
oss3: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1019.20)!
oss3: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1001.09)!
oss3: HC3WORK-OST0010:
oss3: Sum of inodes of users:       518581
oss3: Sum of inodes of groups:      508878
oss3: Sum of kbytes of users:   4833552492
oss3: Sum of kbytes of groups:  4809244332
oss3: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1019.07)!
oss3: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1005.05)!
oss3: HC3WORK-OST0011:
oss3: Sum of inodes of users:       512591
oss3: Sum of inodes of groups:      503088
oss3: Sum of kbytes of users:   4896151680
oss3: Sum of kbytes of groups:  4889852476
oss3: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1018.89)!
oss3: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (1001.29)!
oss3: HC3WORK-OST0012:
oss3: Sum of inodes of users:       518484
oss3: Sum of inodes of groups:      508619
oss3: Sum of kbytes of users:   4839034028
oss3: Sum of kbytes of groups:  4834574108
oss3: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1019.40)!
oss3: HC3WORK-OST0013:
oss3: Sum of inodes of users:       508537
oss3: Sum of inodes of groups:      499417
oss3: Sum of kbytes of users:   4926212496
oss3: Sum of kbytes of groups:  4921585760
oss3: !!! Warning: Difference between user and group inodes is higher than  1 per mil (1018.26)!
oss3: HC3WORK-OST0014:
oss3: Sum of inodes of users:       466997
oss3: Sum of inodes of groups:      466997
oss3: Sum of kbytes of users:   4673592616
oss3: Sum of kbytes of groups:  4673592616
oss4: HC3WORK-OST0015:
oss4: Sum of inodes of users:       632032
oss4: Sum of inodes of groups:      637057
oss4: Sum of kbytes of users:   4729743972
oss4: Sum of kbytes of groups:  4730054384
oss4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (992.11)!
oss4: HC3WORK-OST0016:
oss4: Sum of inodes of users:       625185
oss4: Sum of inodes of groups:      641002
oss4: Sum of kbytes of users:   4804838656
oss4: Sum of kbytes of groups:  4818824376
oss4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (975.32)!
oss4: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.10)!
oss4: HC3WORK-OST0017:
oss4: Sum of inodes of users:       637641
oss4: Sum of inodes of groups:      643111
oss4: Sum of kbytes of users:   4726361064
oss4: Sum of kbytes of groups:  4726805584
oss4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (991.49)!
oss4: HC3WORK-OST0018:
oss4: Sum of inodes of users:       624360
oss4: Sum of inodes of groups:      629138
oss4: Sum of kbytes of users:   4811833904
oss4: Sum of kbytes of groups:  4813538920
oss4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (992.41)!
oss4: HC3WORK-OST0019:
oss4: Sum of inodes of users:       645434
oss4: Sum of inodes of groups:      650753
oss4: Sum of kbytes of users:   4704849528
oss4: Sum of kbytes of groups:  4704988612
oss4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (991.83)!
oss4: HC3WORK-OST001a:
oss4: Sum of inodes of users:       619930
oss4: Sum of inodes of groups:      635246
oss4: Sum of kbytes of users:   4850040456
oss4: Sum of kbytes of groups:  4862104404
oss4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (975.89)!
oss4: !!! Warning: Difference between user and group kbytes is higher than  1 per mil (997.52)!
oss4: HC3WORK-OST001b:
oss4: Sum of inodes of users:       577660
oss4: Sum of inodes of groups:      584758
oss4: Sum of kbytes of users:   5140550496
oss4: Sum of kbytes of groups:  5143800684
oss4: !!! Warning: Difference between user and group inodes is higher than  1 per mil (987.86)!

# Results: 
1. Some OSTs are affected and others are not affected, i.e. 
this is an easy way to find out which OSTs are affected. 
2. On the affected OSTs both inodes and kbytes are wrong. 
3. We have higher group values and higher user values, i.e. 
both user and group quotas are affected. 
4. On the different file systems in nearly all cases either 
group values or user values are higher. The reason for this 
behaviour is not clear.