https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10881 Lustre <p>Downgraded cluster from 2.11 to 2.10.3. OSTs refuse to mount</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Apr 4 22:20:57 soak-2 sshd[5680]: pam_unix(sshd:session): session opened <span class="code-keyword">for</span> user root by (uid=0) Apr 4 22:20:58 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff8804147f6780[0x0, 1, [0x1:0x0:0x0] hash exist]{ Apr 4 22:20:58 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff8804147f67d0 Apr 4 22:20:58 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-zfs@ffff880403614618osd-zfs-object@ffff880403614618 Apr 4 22:20:58 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) } header@ffff8804147f6780 Apr 4 22:20:58 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff8800aeac7b00[0x0, 1, [0x200000003:0x0:0x0] hash exist]{ Apr 4 22:20:58 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff8800aeac7b50 Apr 4 22:20:58 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-zfs@ffff8803fed72970osd-zfs-object@ffff8803fed72970 Apr 4 22:20:58 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) } header@ffff8800aeac7b00 Apr 4 22:20:58 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff8804147f7a40[0x0, 1, [0x200000003:0x2:0x0] hash exist]{ Apr 4 22:20:58 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff8804147f7a90 Apr 4 22:20:58 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-zfs@ffff880403614750osd-zfs-object@ffff880403614750 Apr 4 22:20:58 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) } header@ffff8804147f7a40 Apr 4 22:20:59 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff88040a271ec0[0x0, 1, [0xa:0x0:0x0] hash exist]{ Apr 4 22:20:59 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff88040a271f10 Apr 4 22:20:59 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-zfs@ffff88041434c888osd-zfs-object@ffff88041434c888 Apr 4 22:20:59 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) } header@ffff88040a271ec0 Apr 4 22:20:59 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff88041722fa40[0x0, 1, [0xa:0x9:0x0] hash exist]{ Apr 4 22:20:59 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff88041722fa90 Apr 4 22:20:59 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-zfs@ffff880035d05998osd-zfs-object@ffff880035d05998 Apr 4 22:20:59 soak-2 kernel: LustreError: 5814:0:(ofd_dev.c:251:ofd_stack_fini()) } header@ffff88041722fa40 Apr 4 22:20:59 soak-2 kernel: LustreError: 5814:0:(obd_config.c:558:class_setup()) setup soaked-OST0000 failed (-17) Apr 4 22:20:59 soak-2 kernel: LustreError: 5814:0:(obd_config.c:1682:class_config_llog_handler()) MGC192.168.1.108@o2ib: cfg command failed: rc = -17 Apr 4 22:20:59 soak-2 kernel: Lustre: cmd=cf003 0:soaked-OST0000 1:dev 2:0 3:f Apr 4 22:20:59 soak-2 kernel: LustreError: 15c-8: MGC192.168.1.108@o2ib: The configuration from log <span class="code-quote">'soaked-OST0000'</span> failed (-17). This may be the result of communication errors between <span class="code-keyword">this</span> node and the MGS, a bad configuration, or other errors. See the syslog <span class="code-keyword">for</span> more information. Apr 4 22:20:59 soak-2 kernel: LustreError: 5705:0:(obd_mount_server.c:1386:server_start_targets()) failed to start server soaked-OST0000: -17 Apr 4 22:20:59 soak-2 kernel: LustreError: 5705:0:(obd_mount_server.c:1879:server_fill_super()) Unable to start targets: -17 </pre> </div></div> <p>Will try re-formatting the fs</p> Soak cluster, lustre-b2_10-ib build 33 LU-10881

OST fails to mount after installing 2.10.3

Bug Major Open Unresolved WC Triage Cliff White soak Wed, 4 Apr 2018 22:25:13 +0000 Fri, 11 May 2018 18:32:04 +0000 Lustre 2.10.4 0 4 <p>Finding the "<tt>hash exist</tt>" message was not very clear. This message is generated in two parts in <tt>lu&#95;object&#95;header&#95;print()</tt>, and it <em>appears</em> that this is from <tt>lu&#95;site&#95;print&#45;&gt;lu&#95;site&#95;obj&#95;print&#45;&gt;lu&#95;object&#95;print()</tt>. This is printed from <tt>ofd&#95;stack&#95;fini()</tt> during cleanup if <tt>ls&#95;obj&#95;hash()</tt> is not empty (apparently because <tt>local_oid_storage_fini()</tt> did not clean up properly), but that doesn't appear to be the reason <b>why</b> the startup failed.</p> <p>Are there earlier messages in the logs that indicate <b>why</b> the mount failed?</p> <p>Unfortunately no. We dumped lctl log after one failure, file attached. System has been re-formatted, which seems to have removed the problem</p> <p>Hit this again when downgrading to tip of 2.10. Will leave system in this state if further information desired. </p> <p>Console log. mount attempt was made after reboot.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"></pre> </div></div> <p>[ 102.088947] Lustre: Lustre: Build Version: 2.10.3_132_g6910400<br/> [ 102.282996] LNet: Added LNI 192.168.1.102@o2ib <span class="error">&#91;8/256/0/180&#93;</span><br/> [ 124.448446] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff8804046e0d80[0x0, 1, <span class="error">&#91;0x1:0x0:0x0&#93;</span> hash exist]</p> { [ 124.462670] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff8804046e0dd0 [ 124.474514] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-zfs@ffff8800ae6a49c0osd-zfs-object@ffff8800ae6a49c0 [ 124.488780] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) } <p> header@ffff8804046e0d80</p> <p>[ 124.499730] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff8801797dccc0[0x0, 1, <span class="error">&#91;0x200000003:0x0:0x0&#93;</span> hash exist]</p> { [ 124.514673] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff8801797dcd10 [ 124.526505] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-zfs@ffff8803ff7f1728osd-zfs-object@ffff8803ff7f1728 [ 124.540748] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) } <p> header@ffff8801797dccc0</p> <p>[ 124.551682] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff8804046e0540[0x0, 1, <span class="error">&#91;0x200000003:0x2:0x0&#93;</span> hash exist]</p> { [ 124.566619] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff8804046e0590 [ 124.578435] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-zfs@ffff8800ae6a4af8osd-zfs-object@ffff8800ae6a4af8 [ 124.592668] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) } <p> header@ffff8804046e0540</p> <p>[ 124.603600] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff8804046e0840[0x0, 1, <span class="error">&#91;0xa:0x0:0x0&#93;</span> hash exist]</p> { [ 124.617728] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff8804046e0890 [ 124.629531] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-zfs@ffff8800ae6a55f0osd-zfs-object@ffff8800ae6a55f0 [ 124.643755] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) } <p> header@ffff8804046e0840</p> <p>[ 124.654672] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff8800af39af00[0x0, 1, <span class="error">&#91;0xa:0x5:0x0&#93;</span> hash exist]</p> { [ 124.668796] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff8800af39af50 [ 124.514673] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff8801797dcd10 [ 124.526505] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-zfs@ffff8803ff7f1728osd-zfs-object@ffff8803ff7f1728 [ 124.540748] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) } <p> header@ffff8801797dccc0</p> <p>[ 124.551682] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff8804046e0540[0x0, 1, <span class="error">&#91;0x200000003:0x2:0x0&#93;</span> hash exist]</p> { [ 124.566619] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff8804046e0590 [ 124.578435] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-zfs@ffff8800ae6a4af8osd-zfs-object@ffff8800ae6a4af8 [ 124.592668] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) } <p> header@ffff8804046e0540</p> <p>[ 124.603600] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff8804046e0840[0x0, 1, <span class="error">&#91;0xa:0x0:0x0&#93;</span> hash exist]</p> { [ 124.617728] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff8804046e0890 [ 124.629531] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-zfs@ffff8800ae6a55f0osd-zfs-object@ffff8800ae6a55f0 [ 124.643755] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) } <p> header@ffff8804046e0840</p> <p>[ 124.654672] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff8800af39af00[0x0, 1, <span class="error">&#91;0xa:0x5:0x0&#93;</span> hash exist]</p> { [ 124.668796] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff8800af39af50 [ 124.681882] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-zfs@ffff88040c464138osd-zfs-object@ffff88040c464138 [ 124.698719] LustreError: 5487:0:(ofd_dev.c:251:ofd_stack_fini()) } <p> header@ffff8800af39af00</p> <p>[ 124.712451] LustreError: 5487:0:(obd_config.c:558:class_setup()) setup soaked-OST0000 failed (-17)<br/> [ 124.725499] LustreError: 5487:0:(obd_config.c:1682:class_config_llog_handler()) MGC192.168.1.108@o2ib: cfg command failed: rc = -17<br/> [ 124.744933] Lustre: cmd=cf003 0:soaked-OST0000 1:dev 2:0 3:f </p> <p>[ 124.759606] LustreError: 15c-8: MGC192.168.1.108@o2ib: The configuration from log 'soaked-OST0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br/> [ 124.792996] LustreError: 5358:0:(obd_mount_server.c:1386:server_start_targets()) failed to start server soaked-OST0000: -17<br/> [ 124.808813] LustreError: 5358:0:(obd_mount_server.c:1879:server_fill_super()) Unable to start targets: -17<br/> [ 124.822620] LustreError: 5358:0:(obd_config.c:609:class_cleanup()) Device 3 not setup<br/> [ 124.836307] Lustre: server umount soaked-OST0000 complete<br/> [ 124.844943] LustreError: 5358:0:(obd_mount.c:1582:lustre_fill_super()) Unable to mount soaked-ost0/ost0 (-17)</p> <p>I've also reproduced the failure on soak-3, so it's definitely a code issue. stack dumping/Crash dumping soak-3 now, crash dump will be available on Spirit</p> Development Rank 1|hzzv9z: Rank (Obsolete) 9223372036854775807 Severity