https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11234 Lustre <p><b>Choose lustre pool to place file’s objects by user-defined policies</b></p> <p>This feature fits to heterogeneous lustre environment where lustre ost targets consist of different devices（such as disk，ssd）and are maybe with different configures (Such as osd-zfs, osd-ldiskfs, RAID1, RAID6).  This feature can help users to make more fine-grained data placement.</p> <p>When using this feature, lustre ost targets are firstly divided into many pools. Every pool is composed of homogeneous targets. That means, targets in one pools are with the same type of devices and with the same configure. Then, users define file create rules to put files into different pools. When the rules are enabled, file create operations will search the users-defined rules, find the corresponding pool, and then allocate data objects on the choosed pool. </p> <p>We use lustre source compiling to illustrate this feature.</p> <p>When building the lustre source code, files are composed of the following types</p> <p>1. Source code with filename extension .c or .h and configure files to present build rules, which is usually without any filename extension.</p> <p>2. temporary files that with filename extension .o</p> <p>3. kernel modules and other final files.</p> <p>For the above 3 types, 1, 3 need more reliability than I/O performance，while 2 need more IO performance than reliability.</p> <p>Firstly, build two pools, one ( eg. named pool_src )use disk array with RAID1 configuration to provide high reliability. The other (eg. named pool_temp) use SSD with RAID0 to provide high IO performance.</p> <p>Secondly, define the following pool policy:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lfs setstripe -p pool_src lustre-source lctl set_param lod.*.policies=“add post=\“o\” pool_temp”  # file with post extortion .o will put in pool_temp </pre> </div></div> <p> </p> <p>When the policies are applied, filename with extortion .o will be put in pool_temp, while the others will be put in pool_src. </p> <p> ** </p> <p> </p> LU-11234

Choose lustre pool to place file’s objects by user-defined policies

New Feature Minor Open Unresolved Teddy Teddy TS Sun, 12 Aug 2018 04:38:39 +0000 Fri, 19 Jun 2020 19:27:45 +0000 1 11 <p>When you write "user" here, you really mean "administrator", since it typically is not possible for a regular user to login on the MGS to set a policy. Also, it appears that this policy would be global to all users. </p> <p>I think this proposal is a very useful one, we've been discussing it for years, but nobody has been working on it It would be nice if we could work out a mechanism for regular users to be able to create OST pools by themselves, and have per-user or per-directory policies for file creation. With the patch <a href="https://review.whamcloud.com/32814" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32814</a> "<a href="https://jira.whamcloud.com/browse/LU-11146" title="setstripe for specific osts are broken" class="issue-link" data-issue-key="LU-11146"><del>LU-11146</del></a> lustre: fix setstripe for specific osts upon dir" it is possible to set a directory default layout that selects specific OSTs, which is almost like a pool. The main difference is that the specific-OST layout currently requires the file to be striped over all OSTs, but it would be nice to allow striping overconly a subset of OSTs. </p> <p>The patch <a href="https://review.whamcloud.com/28972" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28972</a> "UL-9982 lustre: Clients striping from mapped FID in nodemap" allows specifying an arbitrary source FID for the default layout for files in a subdirectory mount.</p> <p>I think it would be possible to combine these ideas by having a <tt>$HOME/.lustre/defaults</tt> directory (nothing special there, just a regular directory) that users could create policies in or on (eg. different layouts for filename extensions), and then the $HOME/.lustre directory FID can be set as the default layout for all user directories if they want to apply this policy to their directories (it would be inherited by subdirectories automatically, or fetched from the fs root if needed. .</p> <p>Since this is not a configuration parameter, the user does not need to have any special login permisssion. Since thos layout is explicitly applied to user directories (there can be arbitrarily many different layouts for a user, they just need to supply the directory FID). </p> <p>In the description, ‘user’ means ‘administrator’: application users are only allowed to propose requirements for rules, but applying the rules to the system should be allowed to execute by authoritative administrator. Otherwise, if application users can define and apply rules by themselves, most will prefer to put their data on the faster devices. Besides, applying the rules by application users will increase the rate of misconfiguration.</p> <p>Indeed, I think, the rules of this feature deserves more careful designs. We can define rule conjunction, just like rules in TBF.  For example, rules with UID and filename extension will only allow control file’s layout of users with UID( in <a href="https://jira.whamcloud.com/browse/LU-9658" title="Add QoS for uid and gid in NRS-TBF" class="issue-link" data-issue-key="LU-9658"><del>LU-9658</del></a>, we have already add uid and gid in creat RPC).</p> <p> </p> <p>I'll upload the basic patch that only supports filename extension rule for review.</p> <p>There is nothing preventing users from allocating all of their files on the flash OSTs today, they can just use "<tt>lfs setstripe -o idx,idx,idx</tt>" directly if they want. We need to have OST quotas to prevent users from using these OSTs. </p> <p>In that respect, allowing users some mechanism to specify "pools" or filename extension policies doesn't affect this, but could make like a lot easier for some applications and users.</p> <p>One simple way to enable this would be to allow "<tt>lfs setstripe -o</tt>" to specify a smaller number of stripes than the OSTs listed. This would be treated the same way as a stripe count in a pool - the specified number of stripes is allocated to the file from the listed OSTs.</p> <p>After that point, it would be possible for users to create their own pools, which are just named lists of OSTs, and "<tt>lfs setstripe</tt>" can just get those lists from <tt>_{/.lustrerc</tt> or similar. It is a bit more tricky if the files are not being created via <tt>lfs</tt>, but eg. we could store the FID for the user <tt>}/.lustrerc</tt> in the LOV EA so that the MDS can access it. </p> <p>Teddy Zheng (jjkky@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/33126" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33126</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11234" title="Choose lustre pool to place file’s objects by user-defined policies" class="issue-link" data-issue-key="LU-11234">LU-11234</a> lod: Choose pools to place file's objects by filename extension policies<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5615ef1ee1db30e362e61ecc29add9907637a1e2</p> <p>As discussed with Teddy and Ihara, we want to implement something like:</p> <p>1) Use Jobid/UID/GID to define expressions of rules (Just like TBF rules).</p> <p>2) Each rule has an operation, the current operation is simple, just to create the file in a dedicated OST pool. That means, the rule is a tuple of (expression, OST pool name)</p> <p>3) Define a list of rules with order. The file creation process will walk through this list, trying to match the first ones of them. if matches, the file will be created on the defined OST pool.</p> <p>4) The rule can be defined through /proc entry, just like what did for TBF.</p> <p>A lot of codes can be shared by TBF and this new feature. Maybe we can put some the functions into shared library rather than copy and paste.</p> <p>Rather than make all of these policies global, they should be implemented as part of a nodemap, even if it is the" default" nodemap to start with. This will allow different policies to be implemented for different clients if needed (eg. prefer local OST pool for clients connected over WAN).</p> <p>Instead of specifying <em>just</em> the OST pool for files matching the policy, it would be much more useful to base this on patch <a href="https://review.whamcloud.com/28972" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28972</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-9982" title="Clients striping from mapped FID in nodemap" class="issue-link" data-issue-key="LU-9982">LU-9982</a> lustre: Clients striping from mapped FID in nodemap</tt>" and allow specifying the FID that is the source for the file layout. That allows specifying a pool, but more importantly allows specifying the rest of the layout (stripe count, PFL components with multiple different pools, FLR mirroring, etc.). Getting patch 28972 landed would itself be useful in any case for nodemap/fileset/isolation purposes. Once that patch lands, I think the changes to 33126 would be relatively straight forward, adding a <tt>fid=<span class="error">&#91;SEQ:OID:VER&#93;</span></tt> option to the matching rules.</p> <p>Another requirement collected from real user is:</p> <p>People would like to allocate the files to different pools according to the range of the IDs. For example, project ID from 0 to 1000, to pool 0, pool ID from 1000 to 2000, to pool 1. We need support rule like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param lod.*.layout_policy=\ "add rule_vip_users pool=pool1 uid={[0-000]}" </pre> </div></div> <p>This would benefit the setting of TBF rule too, since we can't set a rule that list all 1000 IDs.</p> <p>see also <a href="https://jira.whamcloud.com/browse/LU-10345" title="allow specifying file layout by filename extension" class="issue-link" data-issue-key="LU-10345">LU-10345</a></p> Blocker Related LU-9982 LU-10345 Development Rank 1|i000kn: Rank (Obsolete) 9223372036854775807