https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7378 Lustre <p>We are encountering an issue on our lustre-2.5.4+ servers where we are seeing an error destroying objects with RC115, then the OI scrubber is starting. We suspect that the OI scrubber may be causing some jobs to run over wallclock. </p> <p>I have attached some kernel logs, and below is the output from the OI scrubber:</p> <p><span class="error">&#91;root@f1-oss1d8 f1-OST00bf&#93;</span># cat /proc/fs/lustre/osd-ldiskfs/f1-OST00bf/oi_scrub<br/> name: OI_scrub<br/> magic: 0x4c5fd252<br/> oi_files: 64<br/> status: scanning<br/> flags: inconsistent,auto<br/> param:<br/> time_since_last_completed: 316 seconds<br/> time_since_latest_start: 17 seconds<br/> time_since_last_checkpoint: 17 seconds<br/> latest_start_position: 12<br/> last_checkpoint_position: N/A<br/> first_failure_position: 562883<br/> checked: 1900449<br/> updated: 0<br/> failed: 8<br/> prior_updated: 0<br/> noscrub: 119<br/> igif: 1<br/> success_count: 13873<br/> run_time: 18 seconds<br/> average_speed: 105580 objects/sec<br/> real-time_speed: 107345 objects/sec<br/> current_position: 2082404<br/> lf_scanned: 0<br/> lf_reparied: 0<br/> lf_failed: 0<br/> <span class="error">&#91;root@f1-oss1d8 f1-OST00bf&#93;</span>#</p> <p>Thanks,<br/> Dustin </p> LU-7378

Error destroying object with RC115

Bug Major Resolved Not a Bug nasf Dustin Leverman Tue, 3 Nov 2015 20:05:18 +0000 Wed, 16 Mar 2016 14:56:24 +0000 Thu, 3 Dec 2015 23:42:22 +0000 Lustre 2.5.3 0 9 <p>In kernel logs, the following error messages kept occurring:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 14 16:30:12 f1-oss1d8 kernel: [2604418.085966] LustreError: 10030:0:(ost_handler.c:1777:ost_blocking_ast()) Error -2 syncing data on lock cancel Oct 14 16:33:07 f1-oss1d8 kernel: [2604593.041236] Lustre: f1-OST00bf-osd: trigger OI scrub by RPC for [0x100000000:0x33d61e5:0x0], rc = 0 [1] Oct 14 16:33:07 f1-oss1d8 kernel: [2604593.065422] Lustre: Skipped 3 previous similar messages Oct 14 16:33:07 f1-oss1d8 kernel: [2604593.076384] LustreError: 99512:0:(ofd_obd.c:1096:ofd_destroy()) f1-OST00bf: error destroying object [0x100000000:0x33d61e5:0x0]: -115 Oct 14 16:33:07 f1-oss1d8 kernel: [2604593.125308] LustreError: 99512:0:(ofd_obd.c:1096:ofd_destroy()) Skipped 11 previous similar messages Oct 14 16:33:11 f1-oss1d8 kernel: [2604597.299596] LustreError: 38769:0:(osd_compat.c:598:osd_obj_update_entry()) f1-OST00bf-osd: the FID [0x100000000:0x2fef612:0x0] is used by two objects: 6037283/2343154970 562883/2343154970 Oct 14 16:33:11 f1-oss1d8 kernel: [2604597.348347] LustreError: 38769:0:(osd_compat.c:598:osd_obj_update_entry()) Skipped 15 previous similar messages </pre> </div></div> <p>For "-115" (-EINPROGRESS) error, it came from ofd_destroy -&gt; ofd_destroy_by_fid -&gt; ofd_object_find -&gt; … -&gt; osd_object_init -&gt; osd_fid_lookup():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> triggered = <span class="code-keyword">true</span>; <span class="code-keyword">if</span> (thread_is_running(&amp;scrub-&gt;os_thread)) { result = -EINPROGRESS; } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (!dev-&gt;od_noscrub) { result = osd_scrub_start(dev); LCONSOLE_WARN(<span class="code-quote">"%.16s: trigger OI scrub by RPC "</span> <span class="code-quote">"<span class="code-keyword">for</span> "</span>DFID<span class="code-quote">", rc = %d [1]\n"</span>, osd_name(dev), PFID(fid), result); <span class="code-keyword">if</span> (result == 0 || result == -EALREADY) result = -EINPROGRESS; <span class="code-keyword">else</span> result = -EREMCHG; } </pre> </div></div> <p>For "FID used by two objects" error, it came from osd_obj_update_entry():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (lu_fid_eq(fid, oi_fid)) { CERROR(<span class="code-quote">"%s: the FID "</span>DFID<span class="code-quote">" is used by two objects: "</span> <span class="code-quote">"%u/%u %u/%u\n"</span>, osd_name(osd), PFID(fid), oi_id-&gt;oii_ino, oi_id-&gt;oii_gen, id-&gt;oii_ino, id-&gt;oii_gen); GOTO(out, rc = -EEXIST); } </pre> </div></div> <p>Hi Dustin,</p> <p>Could you please run the following command on MDT device to see the pathnames of the two inodes that used the same FID?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>debugfs -c -R "ncheck 6037283 562883" /dev/{mdtdev} </pre> </div></div> <p>Hi Jian-</p> <p>I think those are OST inode numbers, so I ran the command there:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@f1-oss1d8 ~]# debugfs -c -R "ncheck 6037283 562883" /dev/mapper/f1-ddn1d-l53 debugfs 1.42.12.wc1 (15-Sep-2014) /dev/mapper/f1-ddn1d-l53: catastrophic mode - not reading inode or group bitmaps Inode Pathname 562883 /O/0/d4/54354404 6037283 /O/0/d18/50263570 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@f1-oss1d8 ~]# debugfs -c -R "stat /O/0/d4/54354404" /dev/mapper/f1-ddn1d-l53 | grep fid debugfs 1.42.12.wc1 (15-Sep-2014) /dev/mapper/f1-ddn1d-l53: catastrophic mode - not reading inode or group bitmaps lma: fid=[0x100000000:0x2fef612:0x0] compat=8 incompat=0 fid = "a8 6e 00 bc f6 02 00 02 00 00 00 00 02 00 00 00 " (16) fid: parent=[0x20002f6bc006ea8:0x0:0x0] stripe=2 [root@f1-oss1d8 ~]# debugfs -c -R "stat /O/0/d18/50263570" /dev/mapper/f1-ddn1d-l53 | grep fid debugfs 1.42.12.wc1 (15-Sep-2014) /dev/mapper/f1-ddn1d-l53: catastrophic mode - not reading inode or group bitmaps lma: fid=[0x100000000:0x2fef612:0x0] compat=8 incompat=0 fid = "a8 6e 00 bc f6 02 00 02 00 00 00 00 02 00 00 00 " (16) fid: parent=[0x20002f6bc006ea8:0x0:0x0] stripe=2 </pre> </div></div> <p>That's where I hit a dead end:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@lfs-mgmt01.ncrc.gov ~]# lfs fid2path /lustre/f1 '[0x100000000:0x2fef612:0x0]' ioctl err -22: Invalid argument (22) fid2path: error on FID [0x100000000:0x2fef612:0x0]: Invalid argument [root@lfs-mgmt01.ncrc.gov ~]# lfs fid2path /lustre/f1 '[0x20002f6bc006ea8:0x0:0x0]' ioctl err -22: Invalid argument (22) fid2path: error on FID [0x20002f6bc006ea8:0x0:0x0]: Invalid argument </pre> </div></div> <p>Hi Matt,</p> <p>Could you please refer to the steps in <a href="https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.html#dbdoclet.50438194_30872" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.html#dbdoclet.50438194_30872</a> ?</p> <p>Hi Jian-</p> <p>This file system was formatted with Lustre 2.4, so we shouldn't have any IGIF files. According to step 2, <b>fid2path</b> should have been able to find it, correct?</p> <p>~Matt</p> <p>Yes, Matt. Let me ask for help.</p> <p>Hi Nasf,</p> <p>Could you please advise?</p> <blockquote> <p><span class="error">&#91;root@lfs-mgmt01.ncrc.gov ~&#93;</span># lfs fid2path /lustre/f1 '<span class="error">&#91;0x20002f6bc006ea8:0x0:0x0&#93;</span>'<br/> ioctl err -22: Invalid argument (22)<br/> fid2path: error on FID <span class="error">&#91;0x20002f6bc006ea8:0x0:0x0&#93;</span>: Invalid argument</p></blockquote> <p>If the "/lustre/f1" is the right mount point, then that means the FID "<span class="error">&#91;0x20002f6bc006ea8:0x0:0x0&#93;</span>" is an invalid one. The first valid lu_fid::f_oid for MDT-object's FID is 1, not 0. So the PFID EA for the OST-object is wrong. So it is normal that you cannot locate related MDT-object via such invalid FID.</p> <p>On the other hand, your case happened during destroying OST-object. That means the MDT-object has been removed already. So even thought the PFID EA was right, you still could not locate the removed MDT-object.</p> <blockquote> <p><span class="error">&#91;root@f1-oss1d8 ~&#93;</span># debugfs -c -R "stat /O/0/d4/54354404" /dev/mapper/f1-ddn1d-l53 | grep fid<br/> debugfs 1.42.12.wc1 (15-Sep-2014)<br/> /dev/mapper/f1-ddn1d-l53: catastrophic mode - not reading inode or group bitmaps<br/> lma: fid=<span class="error">&#91;0x100000000:0x2fef612:0x0&#93;</span> compat=8 incompat=0<br/> fid = "a8 6e 00 bc f6 02 00 02 00 00 00 00 02 00 00 00 " (16)<br/> fid: parent=<span class="error">&#91;0x20002f6bc006ea8:0x0:0x0&#93;</span> stripe=2<br/> <span class="error">&#91;root@f1-oss1d8 ~&#93;</span># debugfs -c -R "stat /O/0/d18/50263570" /dev/mapper/f1-ddn1d-l53 | grep fid<br/> debugfs 1.42.12.wc1 (15-Sep-2014)<br/> /dev/mapper/f1-ddn1d-l53: catastrophic mode - not reading inode or group bitmaps<br/> lma: fid=<span class="error">&#91;0x100000000:0x2fef612:0x0&#93;</span> compat=8 incompat=0<br/> fid = "a8 6e 00 bc f6 02 00 02 00 00 00 00 02 00 00 00 " (16)<br/> fid: parent=<span class="error">&#91;0x20002f6bc006ea8:0x0:0x0&#93;</span> stripe=2</p></blockquote> <p>Searched the log, only the FID <span class="error">&#91;0x100000000:0x2fef612:0x0&#93;</span> was reported as conflict FID mapping. Such FID should be mapped to /O/0/d18/50263570 (6037283/2343154970). But for some unknown reason, the /O/0/d4/54354404 (562883/2343154970) also claims to be mapped to the FID <span class="error">&#91;0x100000000:0x2fef612:0x0&#93;</span>. The latter one is wrong.</p> <p>In fact, the OI entry /O/0/d4/54354404 is corresponding to the FID <span class="error">&#91;0x100000000:0x33d61e4:0x0&#93;</span>. Since we are destroying the OST-object that is corresponding to the FID <span class="error">&#91;0x100000000:0x33d61e4:0x0&#93;</span>. The most simple solution is that mount the OST as ldiskfs mode, then remove /O/0/d4/54354404 directly.</p> <p>Thank you for the information Nasf. This will be our plan, but we need to schedule an outage with the customer. We will let you know the result. </p> <p>Thanks,<br/> Dustin </p> <p>Nasf, <br/> We took a downtime this morning and removed the object you identified above. We are now getting new duplicate FID errors in the logs. It looks like 15 messages were being suppressed so we have more objects to destroy. The one that you helped with above is no longer generating messages. We will disable log suppression and identify the objects that need to be destroyed using the procedure we followed above. </p> <p>Thank you for your help, I think it is okay to close this ticket. </p> <p>Thanks,<br/> Dustin </p> <p>Thank you, Dustin. Let me close this ticket.</p> Related LU-7867 Development Rank 1|hzxs5r: Rank (Obsolete) 9223372036854775807 Severity