https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-137 Lustre <p>Implement an interface for sending IO Control (ioctl) commands from userspace through the Lustre mount point to the underlying ldiskfs filesystem to allow execution of filesystem-wide ioctl() commands, such as resize. This will allow user-space tools that operate via ioctl() commands on the filesystem mountpoint to be used on the Lustre MDT and OST filesystems while they are mounted and in use subject to any limitations of the original ioctl() commands themselves.</p> LU-137

ioctl passthrough mechanism for Lustre OST/MDT mountpoints

New Feature Minor Resolved Fixed Andreas Dilger Andreas Dilger patch Thu, 17 Mar 2011 01:24:38 +0000 Tue, 15 Aug 2023 07:36:29 +0000 Fri, 19 May 2023 12:30:22 +0000 Lustre 2.1.0 Lustre 2.5.0 Lustre 2.16.0 0 13 <p>Assigning back to Andreas until we find a new owner for this project.</p> <p>Prototype patch for ioctl passthrough. It is known to have problems (i.e. crash if used), but is close to what I think would work.</p> <p>There are a few avenues for investigation as to why it is not working:</p> <ul class="alternate" type="square"> <li>I had to make a change to the e2fsprogs library to allow resize2fs to work,<br/> because it was calling stat() on the mountpoint to verify that it was the<br/> same as the underlying block device, to determine if the filesystem is mounted.<br/> (first attachment)</li> <li>This in turn required a change to the lustre server mountpoint to ensure that<br/> it is initializing the device number correctly, so that st_rdev for the Lustre<br/> mountpoint is the same as st_dev of the underlying device node. This is also<br/> included with the server_ioctl() function that is supposed to be working.<br/> (second attachment)</li> </ul> <ul class="alternate" type="square"> <li>the pointer to the underlying ldiskfs device may be incorrect. The handling<br/> of the internal ldiskfs mountpoint is more complex than I'd like, so it isn't<br/> immediately clear that my code is pointing at the right device</li> <li>the size of the ioctl parameter that is being copied could be incorrect</li> </ul> <p>Patch for the e2fsprogs resize2fs tool to allow it to specify the directory mountpoint for resizing, instead of requiring one to specify a block device for the resize operation.</p> <p>I don't think this should be required for the ioctl passthrough to work, but it simplifies the usage of resize2fs, and I thought I needed it until I fixed the Lustre sb-&gt;s_dev on the mounted filesystem.</p> <p>Updated patch that includes sb-&gt;s_dev fix.</p> <p>Hi Andreas, I am working on this ticket. </p> <p>Unfortunately, I expect that the 1.8 version of the patch is completely useless for the current 2.1 and master code...</p> <p>You might want to experiment with some simple ioctl (e.g. EXT4_IOC_GETFLAGS or EXT4_IOC_GETVERSION) to get that working before you try to have resize2fs calling the online resizer. The end goal is that at least EXT4_IOC_GROUP_EXTEND/EXT4_IOC_GROUP_ADD (old resize), EXT4_IOC_RESIZE_FS (new resize), and FITRIM work.</p> <p>Hi Andeas,</p> <p>I have ported the ioctl_passthru-1_8.patch to the latest master.<br/> With this patch I tested EXT4_IOC_GETFLAGS and EXT4_IOC_GETVERSION ioctls on OST and MDT mountpoints. These ioctls work and return expected values.<br/> I have added this a testcase in sanity.sh<br/> The patch can be found here -&gt; <a href="http://review.whamcloud.com/#/c/7354/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7354/</a></p> <p>I tried the EXT4_IOC_SETVERSION ioctl but that resulted in a crash.<br/> Is that expected?</p> <p>It definitely shouldn't crash regardless if what ioctl is used, though I don't necessarily expect it to do anything useful. Presumably the GETVERSION and GETFLAGS ioctls return the correct values from the underlying root inode?</p> <p>Next step is to figure out why it crashed and fix that. </p> <p>Yes, GETVERSION and GETFLAGS ioctls return correct values which are as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>MDS: GETVERSION: 0 GETFLAGS: 0x0 OST: GETVERSION: 0 GETFLAGS: 0x80000 </pre> </div></div> <p>The crash occurred because of a NULL pointer dereference in mnt_want_write().<br/> The stack is as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;1&gt;BUG: unable to handle kernel NULL pointer dereference at 00000000000000e2 &lt;1&gt;IP: [&lt;ffffffff81193c34&gt;] mnt_want_write+0x14/0x80 &lt;4&gt;PGD 1512d067 PUD c0d4067 PMD 0 &lt;4&gt;Oops: 0000 [#1] SMP &lt;4&gt;last sysfs file: /sys/devices/pci0000:00/0000:00:11.0/0000:02:00.0/irq &lt;4&gt;CPU 1 &lt;4&gt;Modules linked in: lustre(U) ofd(U) osp(U) lod(U) ost(U) mdt(U) osd_ldiskfs(U) fsfilt_ldiskfs(U) ldiskfs(U) mdd(U) mgs(U) lquota(U) lfsck(U) obdecho(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) jbd2 sha512_generic sha256_generic crc32c_intel nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc ipv6 dm_mirror dm_region_hash dm_log uinput ppdev parport_pc parport e1000 sg vmware_balloon i2c_piix4 i2c_core shpchp ext3 jbd mbcache sd_mod crc_t10dif sr_mod cdrom mptspi mptscsih mptbase scsi_transport_spi pata_acpi ata_generic ata_piix dm_mod [last unloaded: libcfs] &lt;4&gt; &lt;4&gt;Pid: 26460, comm: ioctl_passthru Not tainted 2.6.32-279.19.1.el6_lustre.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform &lt;4&gt;RIP: 0010:[&lt;ffffffff81193c34&gt;] [&lt;ffffffff81193c34&gt;] mnt_want_write+0x14/0x80 &lt;4&gt;RSP: 0018:ffff880020525cc8 EFLAGS: 00010246 &lt;4&gt;RAX: 0000000000000000 RBX: ffff880020525d78 RCX: 0000000000000003 &lt;4&gt;RDX: 0000000000000001 RSI: 0000000040086604 RDI: 0000000000000002 &lt;4&gt;RBP: ffff880020525cc8 R08: ffffffffa0462c80 R09: 0000000000000000 &lt;4&gt;R10: 0000000000000000 R11: 0000000000000206 R12: 00007fff78321100 &lt;4&gt;R13: ffff88003c9be5b0 R14: 00007fff78321100 R15: 0000000000000000 &lt;4&gt;FS: 00007f94d8a74700(0000) GS:ffff880002280000(0000) knlGS:0000000000000000 &lt;4&gt;CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 &lt;4&gt;CR2: 00000000000000e2 CR3: 000000001dcfc000 CR4: 00000000000006e0 &lt;4&gt;DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 &lt;4&gt;DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 &lt;4&gt;Process ioctl_passthru (pid: 26460, threadinfo ffff880020524000, task ffff880020622aa0) &lt;4&gt;Stack: &lt;4&gt; ffff880020525d58 ffffffffa07198f4 0000000000000000 ffff88001dc499a8 &lt;4&gt;&lt;d&gt; 0000000000000286 ffff88003c9be5b0 ffff880020525d18 ffffffff811902c0 &lt;4&gt;&lt;d&gt; ffff880020525d18 ffff88003c9be5b0 ffff880020525d58 ffffffff8118ee10 &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffffa07198f4&gt;] ldiskfs_ioctl+0xe4/0x940 [ldiskfs] &lt;4&gt; [&lt;ffffffff811902c0&gt;] ? iput+0x30/0x70 &lt;4&gt; [&lt;ffffffff8118ee10&gt;] ? d_obtain_alias+0xc0/0x230 &lt;4&gt; [&lt;ffffffffa0451b2a&gt;] server_ioctl+0xba/0xf0 [obdclass] &lt;4&gt; [&lt;ffffffff81312cb3&gt;] ? pty_write+0x73/0x80 &lt;4&gt; [&lt;ffffffff8130c34e&gt;] ? do_output_char+0x1de/0x210 &lt;4&gt; [&lt;ffffffff81090d4c&gt;] ? remove_wait_queue+0x3c/0x50 &lt;4&gt; [&lt;ffffffff81052223&gt;] ? __wake_up+0x53/0x70 &lt;4&gt; [&lt;ffffffff81189012&gt;] vfs_ioctl+0x22/0xa0 &lt;4&gt; [&lt;ffffffff81310efe&gt;] ? tty_ldisc_deref+0xe/0x10 &lt;4&gt; [&lt;ffffffff81309e93&gt;] ? tty_write+0x233/0x2a0 &lt;4&gt; [&lt;ffffffff811891b4&gt;] do_vfs_ioctl+0x84/0x580 &lt;4&gt; [&lt;ffffffff81176602&gt;] ? vfs_write+0x132/0x1a0 &lt;4&gt; [&lt;ffffffff81189731&gt;] sys_ioctl+0x81/0xa0 &lt;4&gt; [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b &lt;4&gt;Code: 8b 40 58 83 e0 01 c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 65 8b 14 25 b8 e0 00 00 48 63 d2 &lt;48&gt; 8b 87 e0 00 00 00 48 03 04 d5 20 81 bf 81 83 00 01 0f ae f0 &lt;1&gt;RIP [&lt;ffffffff81193c34&gt;] mnt_want_write+0x14/0x80 &lt;4&gt; RSP &lt;ffff880020525cc8&gt; &lt;4&gt;CR2: 00000000000000e2 </pre> </div></div> <p>This was because the active_filp.f_path.mnt was not filled in before calling the ioctl. <br/> I have fixed this problem in the next patchset (<a href="http://review.whamcloud.com/#/c/7354/4" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7354/4</a>)<br/> With this fix, I am able to test the SETVERSION and SETFLAGS ioctl.</p> <p>I also tested the old online resizefs ioctl LDISKFS_IOC_GROUP_EXTEND (latest e2fsprogs) as follows and it seems to be working:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># df -kh Filesystem Size Used Avail Use% Mounted on /dev/sdb1 28G 20G 6.3G 76% / tmpfs 435M 88K 435M 1% /dev/shm /dev/sda1 486M 109M 352M 24% /boot /dev/loop0 147M 18M 120M 13% /mnt/mds1 /dev/loop1 184M 26M 149M 15% /mnt/ost1 /dev/loop2 184M 26M 149M 15% /mnt/ost2 TM2@tcp:/lustre 367M 51M 297M 15% /mnt/lustre # dd if=/dev/zero of=/tmp/lustre-ost-x bs=1M count=200 # mkfs.lustre --fsname=lustre --mgsnode=192.168.100.26@tcp0 --ost --index=2 --device-size=100000 /tmp/lustre-ost-x # mount -t lustre -o loop /tmp/lustre-ost-x /mnt/ost-x/ # df -kh Filesystem Size Used Avail Use% Mounted on /dev/sdb1 28G 20G 6.3G 76% / tmpfs 435M 88K 435M 1% /dev/shm /dev/sda1 486M 109M 352M 24% /boot /dev/loop0 147M 18M 120M 13% /mnt/mds1 /dev/loop1 184M 26M 149M 15% /mnt/ost1 /dev/loop2 184M 26M 149M 15% /mnt/ost2 TM2@tcp:/lustre 458M 56M 378M 13% /mnt/lustre /dev/loop3 92M 5.3M 82M 7% /mnt/ost-x ~/e2fsprogs# ./build/resize/resize2fs /dev/loop3 125M resize2fs 1.43-WIP (21-Jan-2013) Filesystem at /dev/loop3 is mounted on /mnt/ost-x; on-line resizing required old_desc_blocks = 1, new_desc_blocks = 1 Performing an on-line resize of /dev/loop3 to 32000 (4k) blocks. The filesystem on /dev/loop3 is now 32000 blocks long. # df -kh Filesystem Size Used Avail Use% Mounted on /dev/sdb1 28G 20G 6.3G 76% / tmpfs 435M 88K 435M 1% /dev/shm /dev/sda1 486M 109M 352M 24% /boot /dev/loop0 147M 18M 120M 13% /mnt/mds1 /dev/loop1 184M 26M 149M 15% /mnt/ost1 /dev/loop2 184M 26M 149M 15% /mnt/ost2 TM2@tcp:/lustre 486M 56M 406M 13% /mnt/lustre /dev/loop3 119M 5.3M 109M 5% /mnt/ost-x # umount /mnt/ost-x/ # mount -t lustre -o loop /tmp/lustre-ost-x /mnt/ost-x/ # df -kh Filesystem Size Used Avail Use% Mounted on /dev/sdb1 28G 20G 6.3G 76% / tmpfs 435M 88K 435M 1% /dev/shm /dev/sda1 486M 109M 352M 24% /boot /dev/loop0 147M 18M 120M 13% /mnt/mds1 /dev/loop1 184M 26M 149M 15% /mnt/ost1 /dev/loop2 184M 26M 149M 15% /mnt/ost2 TM2@tcp:/lustre 486M 56M 406M 13% /mnt/lustre /dev/loop3 119M 5.3M 109M 5% /mnt/ost-x </pre> </div></div> <p>Hi Alex,</p> <p>This patch: <a href="http://review.whamcloud.com/#/c/8286/2" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8286/2</a> removes lsi_srv_mnt, lmi_mnt and ddp_mnt. It is mentioned in the commit message that vfsmount has become redundant because of the introduction of local storage device.<br/> IOCTL passthrough patch needs a valid vfsmount.</p> <p>The following is per discussion with Andreas:<br/> Seems ext4_ioctl() needs filp for mnt_{want,drop}_write_file(), which uses file-&gt;f_path.mnt for a number of things, so it really needs a valid vfsmnt structure. It looks like the vfsmount is still available in osd_dt_dev(lsi-&gt;lsi_dt_dev)-&gt;od_mnt, but it looks like od_mnt is specific to the underlying OSD device (ldiskfs or ZFS). It looks like there are no direct ioctls for the DMU - they are all handled via /dev/zfs. there are two ioctls for ZFS files in ZPL, but we don't use that, so this only needs to work for ldiskfs mountpoints for now. But, it isn't safe to dig into the OSD structure directly, since this could crash on a ZFS-backed filesystem. It might make sense to add the vfsmnt into dt_device_param. We might need to add a -&gt;dt_ioctl() method to dt_device_operations, since accessing the superblock directly is also bad.</p> <p>Sadly, looking at this patch (<a href="https://jira.whamcloud.com/browse/LU-137" title="ioctl passthrough mechanism for Lustre OST/MDT mountpoints" class="issue-link" data-issue-key="LU-137"><del>LU-137</del></a>) in light of ZFS-backed devices (i.e. anything 2.4 and later), it seems that there are quite a few things that are not "right" about it. ZFS doesn't even have a superblock, so that makes much of the patch invalid. Some of the info can be fetched via the osd_conf_get() interface, in particular s_dev is important for the resize ioctl. Note that even accessing the osd_device outside of the OSD code isn't possible, because the osd_device structure is different for each OSD.</p> <p>What would be the best way to proceed in this case?</p> <p>I don't think you want to manage ZFS pools (or btrfs devices) directly. we can provide vfsmount or superblock via osd_conf_get(), but again that will work for ldiskfs only.</p> <p>Alex, what about adding a new dt_ioctl() method for the OSD API? It would of course be fine if the underlying OSD doesn't support the given ioctl (e.g. returns -ENOTTY) but gives a way to add features like this. I see that this was handled for FIEMAP by adding a dbo_fiemap_get() method (though it could return -EOPNOTSUPP a LOT earlier in the request processing), but that might result in an explosion of different methods if we add one for every ioctl. Other candidates that we might need in the future include FITRIM to trim unallocated space for SSD or thin-provisioned devices, EXT4_IOC_PRECACHE_EXTENTS to prefetch file extent metadata, EXT4_IOC_MOVE_EXT or EXT4_IOC_MIGRATE for data migration within the OST.</p> <p>It isn't yet clear to me if we want separate ioctl methods for the whole device and per file, or if it is OK to just do the "device" ioctls the root inode.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/20161" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20161</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-137" title="ioctl passthrough mechanism for Lustre OST/MDT mountpoints" class="issue-link" data-issue-key="LU-137"><del>LU-137</del></a> osd: better stat info for server mountpoints<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: db5b4aaa6177d9ab179b46a8d3e5d13c5d2c2883</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/23092" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23092</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-137" title="ioctl passthrough mechanism for Lustre OST/MDT mountpoints" class="issue-link" data-issue-key="LU-137"><del>LU-137</del></a> obdclass: add dt_object_put() and use it<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cf4f59a5a7253e18ac98983216791cb730511d18</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/23092/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/23092/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-137" title="ioctl passthrough mechanism for Lustre OST/MDT mountpoints" class="issue-link" data-issue-key="LU-137"><del>LU-137</del></a> obdclass: add dt_object_put() and use it<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5963af745b3aa14410d5ceb66f8a7b7d6aaf576a</p> <p>Linking this since I plan on fixing the ioctl direction issue which will provide a proper interface for this as well.</p> <p>It may be that this has been fixed via patch <a href="https://review.whamcloud.com/33131" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33131</a> "<tt>Subject: <a href="https://jira.whamcloud.com/browse/LU-11355" title="Add Lustre trim support" class="issue-link" data-issue-key="LU-11355"><del>LU-11355</del></a> lustre: enable fstrim on lustre device</tt>", which added a generic ioctl passthrough from userspace to ldiskfs.</p> <p>However, the last time I had tested this (several years ago) there were also some issues with e2fsprogs/resize2fs being unhappy that the block device (<tt>st_rdev</tt>) reported by the Lustre server stub mount did not match the underlying block device. That is what patch: <a href="http://review.whamcloud.com/20161" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20161</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-137" title="ioctl passthrough mechanism for Lustre OST/MDT mountpoints" class="issue-link" data-issue-key="LU-137"><del>LU-137</del></a> osd: better stat info for server mountpoints</tt>" was about, but I haven't updated it in several years.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/20161/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/20161/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-137" title="ioctl passthrough mechanism for Lustre OST/MDT mountpoints" class="issue-link" data-issue-key="LU-137"><del>LU-137</del></a> osd-ldiskfs: pass through resize ioctl<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ac0380dc519aa15310670d164e98453861ef332a</p> <p>Landed for 2.16. It's been a while since I closed a Jira ticket with a bugzilla id!</p> Related LU-6202 LU-11355 LU-16835 LU-5822 LU-4795 LU-8151 LU-4931 LU-17027 Bugzilla ID 14489.0 Development Rank 1|hzvrpb: Rank (Obsolete) 8383