https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5615 Lustre <p>Dear Team,</p> <p>We are trying to setup CGROUP in our lustre environment.<br/> In our setup lustre<br/> MDT<br/> OST</p> <p>Please share the guidance or simple setup example of cgroup with lustre .</p> <p>Thank You<br/> Atul Yadav</p> Lustre 2.5.2 CentOS 6.5 CGROUP LU-5615

Lustre 2.5.2 with CGROUP

Bug Major Resolved Won't Do WC Triage Atul Yadav Fri, 12 Sep 2014 09:59:47 +0000 Sat, 9 Oct 2021 06:32:47 +0000 Sat, 9 Oct 2021 06:32:47 +0000 Lustre 2.5.2 0 4 <p>Hi Atul,</p> <p>Cgroups allow you to allocate resources—such as CPU time, system memory, network bandwidth, or combinations of these resources—among user-defined groups of tasks (processes) running on a system <span class="error">&#91;1&#93;</span>. Given there is wide scope of Cgroups, please provide a use case for your setup to focus the discussion.</p> <p>thanks,<br/> Richard</p> <p>1. <a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/ch01.html" class="external-link" target="_blank" rel="nofollow noopener">https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/ch01.html</a></p> <p>Dear Team,</p> <p>Thanks for the info.<br/> But after going through document, we came to know cgroup require pid to bind with cpu.</p> <p>In lustre we are unable to locate the PID.</p> <p>Please guide us to identify the PID for the lustre service.</p> <p>Thank you<br/> Atul Yadav</p> <p>Hi Atul,</p> <p>To get an idea of the different threads that Lustre employs, I suggest searching the Operations Manual for 'thread':<br/> <a href="https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml</a></p> <p>You will see that Lustre uses multiple threads across multiple machines. Identifying the PID of the Lustre service is therefore different depending on which machine you are on.</p> <p>I suggest it will be more helpful for you to describe the system behaviour intending to satisfy by using cgroups. i.e. why are you looking into cgroups on Lustre?</p> <p>best regards,<br/> Richard</p> <p>Dear Team,</p> <p>We want to configure cgroup in such a way that mds and oss services should run on cpu0 and cpu1 exclusively<br/> For that we want to identify MDS and OSS service.</p> <p>Please guide to complete this activity .</p> <p>Regards,<br/> Atul Yadav</p> <p>There is no PID for an "MDS" or "OSS" service because all Lustre services are running in the kernel. As Richard pointed out, there are numerous kernel threads that implement aspects of the services, and we also have some support for binding some service threads to specific CPUs (<a href="https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#dbdoclet.mdsbinding" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#dbdoclet.mdsbinding</a>), though that is primarily to optimize rpc handling and not for isolation. I believe the only way to isolate multiple Lustre services on a single physical node is to run them in virtual machines. </p> <p>Dear Team,</p> <p>Thanks for the information and guidance.<br/> But Were we can set "mds_num_cpt" for binding the CPU0 with MDS.</p> <p>Thank You<br/> Atul Yadav</p> <p>I believe the section you need on configuring thread counts is immediately above in the Operations Manual:<br/> <a href="https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#dbdoclet.mdstuning" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#dbdoclet.mdstuning</a></p> <p>Please share your experiences.<br/> best regards,<br/> Richard</p> <p>Dear Admin,</p> <p>As per the info we added parameter under module file:<br/> options lnet networks=tcp0(eth0)<br/> options mdt mdt_num_cpts=<span class="error">&#91;0-1&#93;</span></p> <p>But when we load the lustre module, our mdt parameter is not coming like lnet we are getting.<br/> <span class="error">&#91;root@io1 ~&#93;</span># modprobe -v lustre<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/net/lustre/libcfs.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lvfs.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/net/lustre/lnet.ko networks=tcp0(eth0)<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/obdclass.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/ptlrpc.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fld.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fid.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/mdc.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/osc.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lov.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lustre.ko</p> <p>Please guide us.</p> <p>Thank YOu<br/> Atul Yadav</p> <p>in the documentation, the example given is::</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>options mds ... </pre> </div></div> <p>can you repeat, this time using 'mds' instead of 'mdt'?</p> <p>Dear Admin,</p> <p>Still same output, after changing "mdt" to "mds" .</p> <p>Thank You<br/> Atul Yadav</p> <p>Dear team,</p> <p>Thanks now its working fine .....</p> <p>We will check and update you.</p> <p>Thank You<br/> Atul Yadav</p> <p>Dear Team,</p> <p>The output of the commands are given below:-<br/> <span class="error">&#91;root@IO1 ~&#93;</span># modprobe -v mdt<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/net/lustre/libcfs.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lvfs.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/net/lustre/lnet.ko networks=tcp0(em1),o2ib0(ib0),o2ib1(ib1) forwarding="enabled"<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/obdclass.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/ptlrpc.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lquota.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fld.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/fid.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/mdt.ko mds_num_cpts="<span class="error">&#91;0-3&#93;</span>"<br/> <span class="error">&#91;root@IO1 ~&#93;</span># modprobe -v lustre<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/mdc.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/osc.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lov.ko<br/> insmod /lib/modules/2.6.32-431.17.1.el6_lustre.x86_64/extra/kernel/fs/lustre/lustre.ko</p> <p>Thank You<br/> Atul Yadav</p> <p>Thanks for working through this. I have couple of requests:</p> <p>1. Please share your progress and any learnings.</p> <p>2. Your work has identified an error in the manual. I have corrected this error. Can you please review my proposed change here:<br/> <a href="http://review.whamcloud.com/#/c/11938/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11938/</a></p> Development Rank 1|hzww3z: Rank (Obsolete) 15706 Severity