<!-- 
RSS generated by JIRA (9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c) at Sat Feb 10 03:29:52 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>Whamcloud Community JIRA</title>
    <link>https://jira.whamcloud.com</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.4.14</version>
        <build-number>940014</build-number>
        <build-date>05-12-2023</build-date>
    </build-info>


<item>
            <title>[LU-16771] statfs_max_age not used with statfs() project quotas?</title>
                <link>https://jira.whamcloud.com/browse/LU-16771</link>
                <project id="10000" key="LU">Lustre</project>
                    <description>&lt;p&gt;We have noticed that smbd can do lots of&#160;&lt;tt&gt;statfs()&lt;/tt&gt; calls, which on our lustre filesystem with project quota enabled will gather project quota info and takes time.&lt;br/&gt;
I tried to play with lustre&apos;s statfs caching but it doesn&apos;t seem to help. Is that not used when project quotas are set?&lt;/p&gt;

&lt;p&gt;For example, on Oak we have 464 HDD-based OSTs and when they are loaded, gathering quota info might take a bit of time but statfs caching doesn&apos;t seem to help:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;[root@oak-h04v20 pmischel]# lctl set_param llite.oak-ffff999f20cf9800.statfs_max_age=600
llite.oak-ffff999f20cf9800.statfs_max_age=600
[root@oak-h04v20 pmischel]# time df .; sleep 1; time df .; sleep 1; time df .
Filesystem                              1K-blocks      Used    Available Use% Mounted on
10.0.2.51@o2ib5:10.0.2.52@o2ib5:/oak 250000000000 349118136 249650881864   1% /oak

real	0m1.786s
user	0m0.002s
sys	0m0.006s
Filesystem                              1K-blocks      Used    Available Use% Mounted on
10.0.2.51@o2ib5:10.0.2.52@o2ib5:/oak 250000000000 349118136 249650881864   1% /oak

real	0m1.253s
user	0m0.000s
sys	0m0.008s
Filesystem                              1K-blocks      Used    Available Use% Mounted on
10.0.2.51@o2ib5:10.0.2.52@o2ib5:/oak 250000000000 349118136 249650881864   1% /oak

real	0m0.272s
user	0m0.002s
sys	0m0.006s
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;We use the &quot;get quota command&quot; defined in smbd.conf to return filesystem quotas to SMB clients, which is a user-space program for us, that does some caching by itself, so we don&apos;t need quota info from &lt;tt&gt;statfs()&lt;/tt&gt; at all. Is there a way to either disable the project quota behavior with &lt;tt&gt;statfs()&lt;/tt&gt; or make &lt;tt&gt;statfs()&lt;/tt&gt; caching work with 2.15 clients and project quotas?&lt;/p&gt;</description>
                <environment>2.12 servers with 2.15 clients</environment>
        <key id="75793">LU-16771</key>
            <summary>statfs_max_age not used with statfs() project quotas?</summary>
                <type id="1" iconUrl="https://jira.whamcloud.com/secure/viewavatar?size=xsmall&amp;avatarId=11303&amp;avatarType=issuetype">Bug</type>
                                            <priority id="4" iconUrl="https://jira.whamcloud.com/images/icons/priorities/minor.svg">Minor</priority>
                        <status id="4" iconUrl="https://jira.whamcloud.com/images/icons/statuses/reopened.png" description="This issue was once resolved, but the resolution was deemed incorrect. From here issues are either marked assigned or resolved.">Reopened</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="sthiell">Stephane Thiell</assignee>
                                    <reporter username="sthiell">Stephane Thiell</reporter>
                        <labels>
                    </labels>
                <created>Tue, 25 Apr 2023 16:06:15 +0000</created>
                <updated>Thu, 8 Feb 2024 16:29:09 +0000</updated>
                                            <version>Lustre 2.15.2</version>
                                    <fixVersion>Lustre 2.16.0</fixVersion>
                                        <due></due>
                            <votes>0</votes>
                                    <watches>6</watches>
                                                                            <comments>
                            <comment id="370555" author="sthiell" created="Tue, 25 Apr 2023 16:13:24 +0000"  >&lt;p&gt;As performance comparison, this is what &lt;tt&gt;statfs()&lt;/tt&gt; gives on filesystem&apos;s root directory without project quotas (0):&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;[root@oak-h04v20 pmischel]# cd /oak
[root@oak-h04v20 oak]#&#160; time df .; sleep 1; time df .; sleep 1; time df .
Filesystem                                1K-blocks           Used      Available Use% Mounted on
10.0.2.51@o2ib5:10.0.2.52@o2ib5:/oak 50078673614848 32180974619712 17392664012044  65% /oak

real	0m0.001s
user	0m0.000s
sys	0m0.001s
Filesystem                                1K-blocks           Used      Available Use% Mounted on
10.0.2.51@o2ib5:10.0.2.52@o2ib5:/oak 50078673614848 32180974619712 17392664012044  65% /oak

real	0m0.001s
user	0m0.000s
sys	0m0.001s
Filesystem                                1K-blocks           Used      Available Use% Mounted on
10.0.2.51@o2ib5:10.0.2.52@o2ib5:/oak 50078673614848 32180974619712 17392664012044  65% /oak

real	0m0.001s
user	0m0.001s
sys	0m0.000s
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;</comment>
                            <comment id="370688" author="adilger" created="Wed, 26 Apr 2023 08:23:34 +0000"  >&lt;p&gt;Stephane, I had a brief look at the code and thought this might be easily fixed in &lt;tt&gt;ll_statfs()&lt;/tt&gt; just by adding a check for max_age.  However, I realized a complication is that the project quota is (as expected) project/directory-specific, so it isn&apos;t safe to cache the statfs data between calls for the whole filesystem.  The project quota usage would need to be cached in the &lt;tt&gt;ll_inode_info&lt;/tt&gt; directory union (which has unused space vs. the file union) or per-project basis (e.g. in a hash table).&lt;/p&gt;</comment>
                            <comment id="370772" author="sthiell" created="Wed, 26 Apr 2023 21:54:12 +0000"  >&lt;p&gt;It makes sense, thanks Andreas. Do you think it would make sense to add a client tunable to disable project-enabled &lt;tt&gt;statfs()&lt;/tt&gt; for such use cases (like our SMB gateways)? Maybe that would be the easiest thing to do and I could try to have a look. Don&apos;t get me wrong, we like the &lt;tt&gt;statfs()&lt;/tt&gt; feature in 2.15 on clients with actual users &#8211; we don&apos;t want to disable it everywhere.&lt;/p&gt;</comment>
                            <comment id="370789" author="adilger" created="Thu, 27 Apr 2023 02:46:09 +0000"  >&lt;p&gt;Yes, that would be reasonably straight forward to do, maybe &quot;&lt;tt&gt;llite.*.statfs_project&lt;/tt&gt;&quot;?&lt;/p&gt;

&lt;p&gt;Something like the following (totally untested) would enable the &lt;tt&gt;statfs_project&lt;/tt&gt; feature by default, but allow it to be turned off if needed:&lt;/p&gt;
&lt;div class=&quot;code panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;codeContent panelContent&quot;&gt;
&lt;pre class=&quot;code-java&quot;&gt;
        set_bit(LL_SBI_TINY_WRITE, sbi-&amp;gt;ll_flags);
        set_bit(LL_SBI_PARALLEL_DIO, sbi-&amp;gt;ll_flags);
+       set_bit(LL_SBI_STATFS_PROJECT, sbi-&amp;gt;ll_flags);

&lt;span class=&quot;code-object&quot;&gt;int&lt;/span&gt; ll_statfs(struct dentry *de, struct kstatfs *sfs)
{
        struct ll_sb_info *sbi = ll_s2sbi(sb);
        :
-       rc = ll_statfs_internal(ll_s2sbi(sb), &amp;amp;osfs, OBD_STATFS_SUM);
+       rc = ll_statfs_internal(sbi, &amp;amp;osfs, OBD_STATFS_SUM);
        :

        &lt;span class=&quot;code-keyword&quot;&gt;if&lt;/span&gt; (ll_i2info(de-&amp;gt;d_inode)-&amp;gt;lli_projid &amp;amp;&amp;amp;
+           test_bit(LL_SBI_STATFS_PROJECT, sbi-&amp;gt;ll_flags) &amp;amp;&amp;amp;
            test_bit(LLIF_PROJECT_INHERIT, &amp;amp;ll_i2info(de-&amp;gt;d_inode)-&amp;gt;lli_flags))
                &lt;span class=&quot;code-keyword&quot;&gt;return&lt;/span&gt; ll_statfs_project(de-&amp;gt;d_inode, sfs);
}

                &lt;span class=&quot;code-keyword&quot;&gt;case&lt;/span&gt; LL_SBI_LAZYSTATFS:
                &lt;span class=&quot;code-keyword&quot;&gt;case&lt;/span&gt; LL_SBI_VERBOSE:
+               &lt;span class=&quot;code-keyword&quot;&gt;case&lt;/span&gt; LL_SBI_STATFS_PROJECT:
                        &lt;span class=&quot;code-keyword&quot;&gt;if&lt;/span&gt; (turn_off)
                                clear_bit(token, sbi-&amp;gt;ll_flags);
                        &lt;span class=&quot;code-keyword&quot;&gt;else&lt;/span&gt;
                                set_bit(token, sbi-&amp;gt;ll_flags);
                        &lt;span class=&quot;code-keyword&quot;&gt;break&lt;/span&gt;;


  &#160; &#160; &#160; {LL_SBI_ENCRYPT, &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;&lt;span class=&quot;code-quote&quot;&gt;&quot;noencrypt&quot;&lt;/span&gt;},
  &#160; &#160; &#160; {LL_SBI_FOREIGN_SYMLINK, &#160; &#160; &#160; &#160;&lt;span class=&quot;code-quote&quot;&gt;&quot;foreign_symlink=%s&quot;&lt;/span&gt;},
+       {LL_SBI_STATFS_PROJECT,         &lt;span class=&quot;code-quote&quot;&gt;&quot;statfs_project&quot;&lt;/span&gt;},
+       {LL_SBI_STATFS_PROJECT,         &lt;span class=&quot;code-quote&quot;&gt;&quot;nostatfs_project&quot;&lt;/span&gt;},
 &#160; &#160; &#160; &#160;{LL_SBI_NUM_MOUNT_OPT, &#160; &#160; &#160; &#160; &#160;NULL},


  &#160; &#160; &#160; LL_SBI_FOREIGN_SYMLINK, &#160; &#160; &#160; &#160; &lt;span class=&quot;code-comment&quot;&gt;/* foreign fake-symlink support */&lt;/span&gt;
  &#160; &#160; &#160; LL_SBI_FOREIGN_SYMLINK_UPCALL, &#160;&lt;span class=&quot;code-comment&quot;&gt;/* foreign fake-symlink upcall set */&lt;/span&gt;
+       LL_SBI_STATFS_PROJECT,          &lt;span class=&quot;code-comment&quot;&gt;/* statfs returns project quota */&lt;/span&gt;
  &#160; &#160; &#160; LL_SBI_NUM_MOUNT_OPT,
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;along with suitable code copied in &lt;tt&gt;lustre/llite/lprocfs_llite.c&lt;/tt&gt; to add the &lt;tt&gt;statfs_project&lt;/tt&gt; parameter (e.g. from &lt;tt&gt;statahead_agl&lt;/tt&gt;).&lt;/p&gt;

&lt;p&gt;This should allow using &quot;&lt;tt&gt;mount &amp;#45;t lustre &amp;#45;o nostatfs_project&lt;/tt&gt;&quot; or &quot;&lt;tt&gt;lctl set_param llite.FSNAME&amp;#45;&amp;#42;.statfs_project=0&lt;/tt&gt;&quot; to disable it.&lt;/p&gt;</comment>
                            <comment id="390945" author="gerrit" created="Sat, 28 Oct 2023 02:25:48 +0000"  >&lt;p&gt;&quot;Stephane Thiell &amp;lt;sthiell@stanford.edu&amp;gt;&quot; uploaded a new patch: &lt;a href=&quot;https://review.whamcloud.com/c/fs/lustre-release/+/52872&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/c/fs/lustre-release/+/52872&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-16771&quot; title=&quot;statfs_max_age not used with statfs() project quotas?&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-16771&quot;&gt;LU-16771&lt;/a&gt; llite: add statfs_project tunable&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: 784280bc2682647cebd44e0c97fec65eb15a57d9&lt;/p&gt;</comment>
                            <comment id="390946" author="sthiell" created="Sat, 28 Oct 2023 02:57:10 +0000"  >&lt;p&gt;Thanks Andreas for the guidance! Sorry that I lost track of this but the problem resurfaced recently. In this case, we noticed a large amount of mds_quotactl Lustre RPCs coming from some MPI jobs, and this apparently was due to &lt;tt&gt;statfs()&lt;/tt&gt; being called by &lt;tt&gt;PMPI_File_delete()&lt;/tt&gt;:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;Thread 1 (Thread 0x7f9f899b3740 (LWP 24341)):
#0  0x00007f9f88046657 in statfs64 () from /lib64/libc.so.6
#1  0x00007f9f880466cb in statvfs64 () from /lib64/libc.so.6
#2  0x00007f9f87a15927 in opal_path_nfs () from /share/software/user/open/openmpi/4.1.2/lib/libopen-pal.so.40
#3  0x00007f9f88eed83d in mca_fs_base_get_fstype () from /share/software/user/open/openmpi/4.1.2/lib/libmpi.so.40
#4  0x00007f9f701ca3ef in mca_fs_lustre_component_file_query () from /share/software/user/open/openmpi/4.1.2/lib/openmpi/mca_fs_lustre.so
#5  0x00007f9f88eed1c6 in mca_fs_base_file_select () from /share/software/user/open/openmpi/4.1.2/lib/libmpi.so.40
#6  0x00007f9f7a46d24c in mca_common_ompio_file_delete () from /share/software/user/open/openmpi/4.1.2/lib/libmca_common_ompio.so.41
#7  0x00007f9f7b94d5de in delete_select () from /share/software/user/open/openmpi/4.1.2/lib/openmpi/mca_io_ompio.so
#8  0x00007f9f88eef1c5 in mca_io_base_delete () from /share/software/user/open/openmpi/4.1.2/lib/libmpi.so.40
#9  0x00007f9f88e99fd3 in PMPI_File_delete () from /share/software/user/open/openmpi/4.1.2/lib/libmpi.so.40
&amp;lt;private code&amp;gt;
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;The jobs in question were looping into thousands of files (of course) calling &lt;tt&gt;PMPI_File_delete()&lt;/tt&gt;. &lt;tt&gt;statfs()&lt;/tt&gt; is called by &lt;a href=&quot;https://github.com/open-mpi/ompi/blob/f6599061c55ff6fa6ab87b7bb38f70117296cbd4/opal/util/path.c#L503&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;opal_path_nfs()&lt;/a&gt; to check &quot;whether fname is on network file system&quot;... Having a tunable to turn statfs_project off should help until some kind of caching is implemented.&lt;/p&gt;</comment>
                            <comment id="390962" author="adilger" created="Sat, 28 Oct 2023 11:19:42 +0000"  >&lt;p&gt;Honestly, I would also file a bug on whomever implemented that for &lt;tt&gt;PMPI_File_delete()&lt;/tt&gt; to see if it can be changed/removed?  Why do they care if the file is on a network filesystem, and is sending thousands of extra RPCs on a network filesystem a good way to improve performance?  At a minimum it would make sense to do statfs on the parent directories and not the files, and cache this across calls if the files are being deleted in the same directory (or have the same parent, and statfs that once).  That will save everyone&apos;s time...&lt;/p&gt;

&lt;p&gt;Failing that, you &lt;em&gt;could&lt;/em&gt; add a per-projid statfs cache that is subject to the same 1 second expiry as with regular statfs?  That avoids the need to disable this feature just because of a misfeature in a userspace tool, and would solve the problem for everyone (who will likely not know about this tunable, or will not be able to turn it off themselves).&lt;/p&gt;</comment>
                            <comment id="390967" author="sthiell" created="Sat, 28 Oct 2023 16:11:47 +0000"  >&lt;p&gt;I will investigate where would be best to report this, you&apos;re totally right. However, I also fear it is not an isolated occurrence with &lt;tt&gt;PMPI_File_delete()&lt;/tt&gt; as from what I gather, MPI developers seem to &lt;em&gt;really&lt;/em&gt; like &lt;tt&gt;statfs()&lt;/tt&gt;!! See &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-15788&quot; title=&quot;lazystatfs + FOFB + mpich problems&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-15788&quot;&gt;&lt;del&gt;LU-15788&lt;/del&gt;&lt;/a&gt; for another example of &lt;tt&gt;statfs()&lt;/tt&gt; being used in some MPI-IO implementation when opening a file... I am actually a bit surprised this problem is not reported more often, but if you don&apos;t use project quotas, Lustre&apos;s &lt;tt&gt;statfs()&lt;/tt&gt; with the caching seems VERY fast. So OK, I will also start to look how to add a per-projid statfs cache. It would be ideal to have that available on compute nodes, to keep the nice per-project statfs feature while reducing overall RPCs.&lt;br/&gt;
For my original problem reported in this ticket, which was related to the use of Samba re-exporting Lustre on dedicated servers, &lt;tt&gt;statfs_project=0&lt;/tt&gt; will be sufficient.&lt;/p&gt;</comment>
                            <comment id="398367" author="gerrit" created="Wed, 3 Jan 2024 03:02:39 +0000"  >&lt;p&gt;&quot;Oleg Drokin &amp;lt;green@whamcloud.com&amp;gt;&quot; merged in patch &lt;a href=&quot;https://review.whamcloud.com/c/fs/lustre-release/+/52872/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/c/fs/lustre-release/+/52872/&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-16771&quot; title=&quot;statfs_max_age not used with statfs() project quotas?&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-16771&quot;&gt;LU-16771&lt;/a&gt; llite: add statfs_project tunable&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: &lt;br/&gt;
Commit: d0a968312aab883e4a0b3f40816e881b93b9dd2c&lt;/p&gt;</comment>
                            <comment id="398419" author="pjones" created="Wed, 3 Jan 2024 14:24:31 +0000"  >&lt;p&gt;Landed for 2.16&lt;/p&gt;</comment>
                            <comment id="398471" author="adilger" created="Wed, 3 Jan 2024 16:00:20 +0000"  >&lt;p&gt;Peter, I don&apos;t think that the landed patch really fixes the issue properly. It adds a knob to disable the project quota specific statfs, but that only helps if the admin changes the parameter. It would be better to have a cache on the client for this info so that it &quot;just works&quot;. &lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="79825">LU-17395</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="69584">LU-15721</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="78375">LU-17191</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                            <customfield id="customfield_10890" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10390" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i03jtz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10090" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10060" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Severity</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10022"><![CDATA[3]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        </customfields>
    </item>
</channel>
</rss>