https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9799 Lustre <p>When <tt>mount -t lustre ...</tt> has failed to actually mount a target, the exit code of <tt>mount</tt> does not reflect this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mount -t lustre zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/MGS /mnt/MGS e2label: No such file or directory while trying to open zfs_pool_scsi0QEMU_QEMU_HARDDISK_disk13/MGS Couldn't find valid filesystem superblock. # echo $? 0 </pre> </div></div> <p>This of course wreaks havoc on systems such as IML which rely on the exit code of one step in the process of starting a filesystem to decide if it should continue with subsequent steps.</p> Lustre: Build Version: 2.10.0_5_gbb3c407 LU-9799

mount doesn't return an error when failing

Bug Critical Resolved Fixed Nathaniel Clark Brian Murrell Wed, 26 Jul 2017 11:22:59 +0000 Fri, 18 Aug 2017 23:34:57 +0000 Thu, 17 Aug 2017 04:32:39 +0000 Lustre 2.10.0 Lustre 2.10.1 Lustre 2.11.0 0 5 <p>But what is interesting about this command and error is that clearly the user is trying to mount a ZFS target so why is <tt>ldiskfs_read_ldd()</tt> being called for it?</p> <p>Isn't <tt>osd_read_ldd()</tt> supposed to know that the target is (supposed to be formatted) ZFS at this point through <tt>ldd-&gt;ldd_mount_type</tt>?</p> <p>Could this be a symptom of a failed <tt>mkfs</tt> on the target perhaps?</p> <p>Any other suggestions of what this could be a symptom of would be welcome.</p> <p>Nathaniel</p> <p>Can you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>ldd_mount_type should be set by osd_is_lustre() or it would error out with "... has not been formatted with mkfs.lustre..."</p> <p>Are you sure it didn't mount it? The errors listed are ldiskfs errors, which is okay when mounting a ZFS device.</p> <blockquote> <p>ldd_mount_type should be set by osd_is_lustre() or it would error out with "... has not been formatted with mkfs.lustre..."</p></blockquote> <p>Indeed, that is my understanding from reading the code also.</p> <blockquote> <p>Are you sure it didn't mount it?</p></blockquote> <p>I got here actually because the error I was trying to find the source of was:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jun 27 15:59:53 lotus-55vm18.lotus.hpdd.lab.intel.com kernel: LustreError: 15f-b: testfs-OST0001: cannot register this server with the MGS: rc = -108. Is the MGS running? Jun 27 15:59:53 lotus-55vm18.lotus.hpdd.lab.intel.com kernel: LustreError: 12529:0:(obd_mount_server.c:1846:server_fill_super()) Unable to start targets: -108 Jun 27 15:59:53 lotus-55vm18.lotus.hpdd.lab.intel.com kernel: LustreError: 12529:0:(obd_mount_server.c:1560:server_put_super()) no obd testfs-OST0001 Jun 27 15:59:53 lotus-55vm18.lotus.hpdd.lab.intel.com kernel: LustreError: 12529:0:(obd_mount_server.c:135:server_deregister_mount()) testfs-OST0001 not registered Jun 27 15:59:53 lotus-55vm18.lotus.hpdd.lab.intel.com kernel: Lustre: server umount testfs-OST0001 complete Jun 27 15:59:53 lotus-55vm18.lotus.hpdd.lab.intel.com kernel: LustreError: 12529:0:(obd_mount.c:1505:lustre_fill_super()) Unable to mount (-108) </pre> </div></div> <p>So that would indicate that it didn't mount.</p> <blockquote> <p>The errors listed are ldiskfs errors, which is okay when mounting a ZFS device.</p></blockquote> <p>But why would one get any ldiskfs errors when trying to mount a ZFS device if <tt>osd_is_lustre()</tt> correctly set <tt>ldd_mount_type</tt>? If <tt>ldd_mount_type</tt> is correct then why would <tt>ldiskfs_read_ldd()</tt> (or <tt>ldiskfs_&lt;anything&gt;</tt> for that matter) even have any opportunity to produce any ldiskfs related errors?</p> <p>Can you give me exact versions of what you using, and maybe an sosreport from the host? I'm having trouble reproducing even the initial error messages.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=utopiabound" class="user-hover" rel="utopiabound">utopiabound</a>:</p> <blockquote> <p>Can you give me exact versions of what you using,</p></blockquote> <p>The Lustre build version is in the <em>Environment</em> field of this ticket.</p> <blockquote> <p>I'm having trouble reproducing even the initial error messages</p></blockquote> <p>Yes, it's not very reproducible. We only hit it occasionally. But since we do create Lustre filesystems so many times during our test runs, even only very intermittent issues can hit seemingly frequently.</p> <p>I think the first step is to agree on the expected behaviour. My position, from reading the code is that one should never be able to get the <tt>e2label: No such file or directory while trying to open...</tt> error if the target was properly formatted since the codepath that that error exists in should be only for <em>ldiskfs</em> and if the target was formatted ZFS then it should be an unreachable codepath. You seem to indicate that it could happen though. I explained why I don't think one should be able to hit that codepath but you must disagree with my explanation if you think it can be hit.</p> <p>Could you explain what I missed in my analysis?</p> <blockquote> <p>The Lustre build version is in the <em>Environment</em> field of this ticket.</p></blockquote> <p>But the ZFS, OS, e2fsprogs versions aren't.</p> <p>I can get sort of close, but not with your pool name:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ieel-mds03 tmp]# mount -t lustre zfs_pool_scsi0QEMU_Inode_HARDDISK_disk13/MGS /mnt/MGT; echo $? zfs_pool_scsi0QEMU_Inode_HARDDISK_disk13/MGS: No such file or directory while opening filesystem e2label: No such file or directory while trying to open zfs_pool_scsi0QEMU_Inode_HARDDISK_disk13/MGS Couldn't find valid filesystem superblock. mount.lustre: zfs_pool_scsi0QEMU_Inode_HARDDISK_disk13/MGS failed to read permanent mount data: 254 </pre> </div></div> <p>If <tt>e2label</tt> were to print anything (even a newline) to stdout on error, the final return code would be 0.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=utopiabound" class="user-hover" rel="utopiabound">utopiabound</a>:</p> <blockquote> <p>But the ZFS, OS, e2fsprogs versions aren't.</p></blockquote> <p>It's probably moot, but just for clarity, the ZFS version is whatever is built by Jenkins with b2_10. <em>e2fsprogs</em> is most recent GA and O/S is RHEL 7.4. I doubt any of these are particularly relevant though.</p> <blockquote> <p>I can get sort of close, but not with your pool name:</p></blockquote> <p>I think you got much more than just "sort of close".  I think you got an exact reproduction. The names of pools, etc. I think is quite irrelevant.</p> <p>The question is still though, when the <tt>e2label</tt> call is only in the <em>ldiskfs</em> OSD codepath, in <tt>ldiskfs_read_ldd()</tt>, why is that being hit for a ZFS formatted target?</p> <p>My reading of the code is that by the time <tt>osd_read_ldd()</tt> is supposed to call either <tt>zfs_read_ldd()</tt> or <tt>ldiskfs_read_ldd()</tt>, the format of the target is known and stored in <tt>ldd-&gt;ldd_mount_type</tt>, so only the relevant one of either <tt>zfs_read_ldd()</tt> or <tt>ldiskfs_read_ldd()</tt> should be called, not both and so why are we getting an error from the <tt>e2label</tt> that is only in <tt>ldiskfs_read_ldd()</tt>?</p> <p>The name of the pool is absolutely key for my reproduction:</p> <p>lustre/utils/libmount_utils_ldiskfs.c:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 448 /* Check whether the file exists in the device */ 449 static int file_in_dev(char *file_name, char *dev_name) 450 { 451 FILE *fp; 452 char debugfs_cmd[256]; 453 unsigned int inode_num; 454 int i; 455 456 /* Construct debugfs command line. */ 457 snprintf(debugfs_cmd, sizeof(debugfs_cmd), 458 "%s -c -R 'stat %s' '%s' 2&gt;&amp;1 | egrep '(Inode|unsupported)'", 459 DEBUGFS, file_name, dev_name); 460 </pre> </div></div> <p>Notice the <tt>...|egrep ...</tt> on line 458. That will report out report output text if the egrep matches the pool name on an error, since stderr is also redirected through the egrep.</p> <p>Ahhh.  So, relevant only to force that codepath.  But clearly that is not the problem in my case.</p> <p>Right. But something similar must be happening, though I am at a loss as to what. </p> <p>Okay, I was testing something totally different and ran into this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ieel-mds03 ~]# mount -t lustre MGS/MGT /mnt/MGT e2label: No such file or directory while trying to open MGS/MGT Couldn't find valid filesystem superblock. </pre> </div></div> <p>But on closer inspection:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ieel-mds03 ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/cl_ieel--mds03-root 6486016 1867296 4618720 29% / devtmpfs 496568 0 496568 0% /dev tmpfs 508324 39216 469108 8% /dev/shm tmpfs 508324 13188 495136 3% /run tmpfs 508324 0 508324 0% /sys/fs/cgroup /dev/sda1 1038336 193444 844892 19% /boot ieel-storage:/home 40572928 38486912 2086016 95% /home tmpfs 101668 0 101668 0% /run/user/0 MGS 5047168 0 5047168 0% /MGS MGS/MGT 5007744 0 5005696 0% /mnt/MGT </pre> </div></div> <p>The filesystem did mount, and each umount and remount, I get the same error message, but it succeeds in mounting.</p> <p>Nathaniel Clark (nathaniel.l.clark@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28456" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28456</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9799" title="mount doesn&#39;t return an error when failing " class="issue-link" data-issue-key="LU-9799"><del>LU-9799</del></a> mount: Call read_ldd with initialized mount type<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: aeffd031d67a99896657617f7acd6dafd7a7722c</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28456/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28456/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9799" title="mount doesn&#39;t return an error when failing " class="issue-link" data-issue-key="LU-9799"><del>LU-9799</del></a> mount: Call read_ldd with initialized mount type<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0108281c65545df169faaa0ce0690fb021680643</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28581" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28581</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9799" title="mount doesn&#39;t return an error when failing " class="issue-link" data-issue-key="LU-9799"><del>LU-9799</del></a> mount: Call read_ldd with initialized mount type<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 776cae93b762e819cf80eed04d57bfc4040f09f0</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/28581/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28581/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9799" title="mount doesn&#39;t return an error when failing " class="issue-link" data-issue-key="LU-9799"><del>LU-9799</del></a> mount: Call read_ldd with initialized mount type<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: f869b9da902fc305bfab8e902d0c1202aec6a7bc</p> Duplicate LU-9853 Development Rank 1|hzzhbb: Rank (Obsolete) 9223372036854775807 Severity