https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12530 Lustre <p>Lustre loads the zfs module whenever a block device is added or changed, if Lustre was built against zfs. It unintentionally does this on clients as well as servers.</p> <p>Commit 0d11a314787bc795797a016262e9bcfe86e2193e for <a href="https://jira.whamcloud.com/browse/LU-11563" title="99-lustre.rules on clients tries to execute non-existent /usr/sbin/l_tunedisk" class="issue-link" data-issue-key="LU-11563"><del>LU-11563</del></a> added a udev rule,<br/> lustre/conf/99-lustre-server.rules</p> <p>When a block device is added or changed:<br/> udev -&gt; l_tunedisk<br/> l_tunedisk -&gt; mount_utils.c:osd_init()<br/> osd_init() <del>&gt; backfs_ops[]</del>&gt;init() AKA zfs_init()<br/> zfs_init() -&gt; "/sbin/modprobe -q zfs"</p> <p>This occurs if the Lustre rpms installed are a full build of Lustre, instead of just a client build.</p> LU-12530

udev add/change rule loads zfs module on clients

Bug Minor Resolved Fixed Nathaniel Clark Olaf Faaland Tue, 9 Jul 2019 23:48:03 +0000 Tue, 14 Jul 2020 20:12:04 +0000 Tue, 12 Nov 2019 06:37:50 +0000 Lustre 2.10.5 Lustre 2.12.2 Lustre 2.14.0 Lustre 2.12.4 0 5 <p>For our tracking, our local ticket is TOSS-4559</p> <p>Nathaniel</p> <p>Can you please assist with this one?</p> <p>Thanks</p> <p>Peter</p> <p>You have ZFS installed on your clients? Interesting setup <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>&gt; You have ZFS installed on your clients? Interesting setup</p> <p>Yep, it's the kitchen sink approach. But of course we don't intend to actually load the kernel module <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ofaaland" class="user-hover" rel="ofaaland">ofaaland</a>,</p> <p>Are you using lustre-client or lustre rpms on your clients?  If you are using the lustre rpms, then yes, the modules will be loaded when a disk add is triggered in udev via that rule.</p> <p><a href="https://review.whamcloud.com/33466/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33466/</a> split out that udev rule to be server only so it should only affect your clients if they are using lustre instead of lustre-client.</p> <p>Nathaniel,</p> <p>Correct, It's lustre that is installed, not lustre-client, and yes I'm aware of that commit - see the Description <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>. I'd read the ticket about the kernel crashes due to differing device tunables, and I understand why the rule was created. But there are the reasons I think this is incorrect behavior. Maybe I'm mistaken.</p> <ol> <li>The "lustre" rpm has both server and client bits. It's not just server bits, and therefore not an indication, by itself, that the system on which it's installed is a lustre server - just that it might be.</li> <li>The rule seems overly broad. For example, blkid can tell us if the device is ldiskfs or zfs; if neither then no point in running tunedisk(). Can we use that?</li> <li>zfs_tune_lustre() doesn't do anything anyway, so no benefit to loading the zfs kernel module for that</li> <li>From what I read this seems to be working around a kernel bug, which exists only in some kernels and isn't specific to Lustre - so it seems like the wrong place for the fix. I guess some Lustre users are stuck (ie their distro didn't fix the issue or they can't upgrade, etc.) but then I think this should be available to them but not enabled by default.</li> </ol> <p>What do you think?</p> <p>&gt; 2. The rule seems overly broad.</p> <p>I agree the rule is probably overly broad, and we can use udev itself to limit it more than it is now.</p> <p>&gt; 3. zfs_tune_lustre() doesn't do anything</p> <p>But it might someday...</p> <p>&gt; 4. From what I read this seems to be working around a kernel bug</p> <p>It's not so much a kernel bug as kernel behavior.  There's a possible fix for this issue in device-mapper-multipath, but only in fairly recent releases (maybe only el7.6?) (the <tt>max_sectors_kb</tt> setting, but even that needs to be tuned per device, and I'm not positive it picks up the case that <a href="https://jira.whamcloud.com/browse/LU-11563" title="99-lustre.rules on clients tries to execute non-existent /usr/sbin/l_tunedisk" class="issue-link" data-issue-key="LU-11563"><del>LU-11563</del></a> is fixing, I only assume it does).</p> <p> </p> <p>Yes, I think making the udev rule more narrowly tailored would be a good thing.</p> <p>&gt;&gt; 3. zfs_tune_lustre() doesn't do anything<br/> &gt;<br/> &gt; But it might someday...</p> <p>That's fair, but it shouldn't do anything with the block devices in the pool in any case. If tunables need to be set at that level, that should be handled within ZFS so that non-lustre ZFS users get the benefit as well. ZFS already has to deal properly with block devices appearing and disappearing. For example, ZFS currently sets the elevator to "noop" for the block devices in the pool.</p> <p>zpool and dataset properties are persistent, so they don't need this treatment. One could also set module parameters upon module load, but this isn't the place for that.</p> <p>So I think this rule should match on </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ENV{ID_FS_TYPE}=="ext4" </pre> </div></div> <p>in addition to the ACTION and SUBSYSTEM already specified. That doesn't actually solve my problem, but I think it's a step in the right direction.</p> <p>I don't see another udev property that would help distinguish ext4-formatted devices from ldiskfs-formatted devices, but if you know of one that would be even better.</p> <p>For ldiskfs type, I think, the closest you can get is checking the FS_LABEL, that it matches one of these:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>MGS *-MDT* *-OST* </pre> </div></div> <p> If it doesn't match one of those, then it definitely isn't ldiskfs.</p> <p>One day ext4 will be what is ldiskfs today. So Nathaniel approach is the best idea.</p> <p>Olaf Faaland-LLNL (faaland1@llnl.gov) uploaded a new patch: <a href="https://review.whamcloud.com/36599" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36599</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12530" title="udev add/change rule loads zfs module on clients" class="issue-link" data-issue-key="LU-12530"><del>LU-12530</del></a> utils: narrow l_tunedisk udev rule<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: daa0f160640a6a3e5581e1f10ad33047a8a44ae6</p> <p>After this lands, please consider backporting to b2_12. Thanks.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36599/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36599/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12530" title="udev add/change rule loads zfs module on clients" class="issue-link" data-issue-key="LU-12530"><del>LU-12530</del></a> utils: narrow l_tunedisk udev rule<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7b2cb54858daa60d560fd6374c4ecba552a10d27</p> <p>Landed for 2.14</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36776" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36776</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12530" title="udev add/change rule loads zfs module on clients" class="issue-link" data-issue-key="LU-12530"><del>LU-12530</del></a> utils: narrow l_tunedisk udev rule<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: f5f3f9388aa9fa2c8eb7f2e83b3eb1912afea1fd</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36776/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36776/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12530" title="udev add/change rule loads zfs module on clients" class="issue-link" data-issue-key="LU-12530"><del>LU-12530</del></a> utils: narrow l_tunedisk udev rule<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: e365cf2b001af7ca0883d293b7c148bcaa8e1c61</p> Related LU-9551 LU-11736 LU-10599 Development Rank 1|i00jf3: Rank (Obsolete) 9223372036854775807 Severity