Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-9577] mount.lustre: mount /dev/sde at /mnt/testfs-MDT0001 failed: File exists https://jira.whamcloud.com/browse/LU-9577 Lustre <p> During a DNE test we got an error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mount -t lustre /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk11 /mnt/testfs-MDT0001 mount.lustre: increased /sys/block/sde/queue/max_sectors_kb from 512 to 16384 mount.lustre: mount /dev/sde at /mnt/testfs-MDT0001 failed: File exists # echo $? 17 </pre> </div></div> <p>The messages log reported:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>May 31 10:51:38 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro May 31 10:51:38 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdb): mounted filesystem with ordered data mode. Opts: errors=remount-ro May 31 10:51:38 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdc): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc May 31 10:51:38 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sdb): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc May 31 10:51:38 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: Lustre: ctl-testfs-MDT0000: No data found on store. Initialize space May 31 10:51:38 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: Lustre: testfs-MDT0000: new disk, initializing May 31 10:51:38 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: Lustre: testfs-MDT0000: Imperative Recovery not enabled, recovery window 300-900 May 31 10:51:38 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: Lustre: ctl-testfs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt May 31 10:51:38 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: Lustre: cli-ctl-testfs-MDT0002: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:2:mdt] May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: Lustre: Failing over testfs-MDT0002 May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: Lustre: Skipped 2 previous similar messages May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LustreError: 11-0: testfs-MDT0000-osp-MDT0002: operation mds_disconnect to node 0@lo failed: rc = -107 May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LustreError: Skipped 3 previous similar messages May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sde): mounted filesystem with ordered data mode. Opts: errors=remount-ro May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LDISKFS-fs (sde): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: Lustre: server umount testfs-MDT0002 complete May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LustreError: 28862:0:(genops.c:334:class_newdev()) Device MDS already exists at 5, won't add May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LustreError: 28862:0:(obd_config.c:366:class_attach()) Cannot create device MDS of type mds : -17 May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LustreError: 28862:0:(obd_mount.c:194:lustre_start_simple()) MDS attach error -17 May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LustreError: 28862:0:(obd_mount_server.c:1297:server_start_targets()) failed to start MDS: -17 May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LustreError: 28862:0:(obd_mount_server.c:1840:server_fill_super()) Unable to start targets: -17 May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LustreError: 28862:0:(obd_mount_server.c:1554:server_put_super()) no obd testfs-MDT0001 May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LustreError: 28862:0:(obd_mount_server.c:135:server_deregister_mount()) testfs-MDT0001 not registered May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: Lustre: Skipped 1 previous similar message May 31 10:51:39 lotus-45vm15.lotus.hpdd.lab.intel.com kernel: LustreError: 28862:0:(obd_mount.c:1501:lustre_fill_super()) Unable to mount (-17) </pre> </div></div> <p>Unfortunately this test harness is not really intended to test Lustre itself and therefore not very well tooled to have gathered any more useful information (i.e. no Lustre debugging) than what I have here.</p> Build Version: 2.9.58_22_gdb59ecb LU-9577 mount.lustre: mount /dev/sde at /mnt/testfs-MDT0001 failed: File exists Bug Blocker Resolved Duplicate Hongchao Zhang Brian Murrell Wed, 31 May 2017 19:01:50 +0000 Thu, 19 Oct 2017 12:20:58 +0000 Thu, 19 Oct 2017 12:20:58 +0000 Lustre 2.10.0 0 6 <p>We are seeing this in RC1, and ran into it today. </p> <p>Could this be triaged please and a remediation suggested (try again?).</p> <p>Latest was during a registration (i.e. initial for the target) mount where we got the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mount -t lustre /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk13 /mnt/testfs-OST0001 mount.lustre: increased /sys/block/sdc/queue/max_sectors_kb from 512 to 16384 mount.lustre: mount /dev/sdc at /mnt/testfs-OST0001 failed: File exists # echo $rc 17 </pre> </div></div> <p>The following was reported in syslog:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 2 19:46:23 localhost kernel: LustreError: 31962:0:(genops.c:334:class_newdev()) Device OSS already exists at 11, won't add Oct 2 19:46:23 localhost kernel: LustreError: 31962:0:(obd_config.c:400:class_attach()) Cannot create device OSS of type ost : -17 Oct 2 19:46:23 localhost kernel: LustreError: 31962:0:(obd_mount.c:198:lustre_start_simple()) OSS attach error -17 Oct 2 19:46:23 localhost kernel: LustreError: 31962:0:(obd_mount_server.c:1338:server_start_targets()) failed to start OSS: -17 Oct 2 19:46:23 localhost kernel: LustreError: 31962:0:(obd_mount_server.c:1866:server_fill_super()) Unable to start targets: -17 Oct 2 19:46:23 localhost kernel: LustreError: 31962:0:(obd_mount_server.c:1576:server_put_super()) no obd testfs-OST0001 Oct 2 19:46:23 localhost kernel: LustreError: 31962:0:(obd_mount_server.c:135:server_deregister_mount()) testfs-OST0001 not registered Oct 2 19:46:23 localhost kernel: LustreError: 31962:0:(obd_mount.c:1505:lustre_fill_super()) Unable to mount (-17) </pre> </div></div> <p>We don't have any lustre debug for this particular instance of the failure though.</p> <p>I wonder if the original report with MDS problems was due to <a href="https://jira.whamcloud.com/browse/LU-9034" title="Separate the config logs between different MGCs" class="issue-link" data-issue-key="LU-9034"><del>LU-9034</del></a> or similar. I imagine your code already has this patch though.<br/> Perhaps makes sense for Hongchao to take a look?</p> <p>This issue is not the same as <a href="https://jira.whamcloud.com/browse/LU-9034" title="Separate the config logs between different MGCs" class="issue-link" data-issue-key="LU-9034"><del>LU-9034</del></a>, which is caused by the unreleased MGC instance.<br/> It should be the same issue as <a href="https://jira.whamcloud.com/browse/LU-5020" title="OST can be all mounted successfully in parallel mount scenario" class="issue-link" data-issue-key="LU-5020"><del>LU-5020</del></a>, and the patch has landed on IEEL2_0 but not landed on master, <br/> the patch is updated and tracked at <a href="https://review.whamcloud.com/#/c/10229/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/10229/</a></p> <p>What action should a user take if he runs into this issue?  How does he rectify this to get to a working filesystem state?</p> <p>Fixed by <a href="https://jira.whamcloud.com/browse/LU-5020" title="OST can be all mounted successfully in parallel mount scenario" class="issue-link" data-issue-key="LU-5020"><del>LU-5020</del></a> which will be in the next Lustre LTS release (LU 2.10.2).</p> <p>The issue is unlikely to be seen in the field with Enterprise Edition / IML but it can be worked-around with retry/reboot or the patch can be provided.</p> Duplicate LU-5020 Development Rank 1|hzzdyv: Rank (Obsolete) 9223372036854775807 Severity