https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4554 Lustre <p>We are running Lustre 2.4.0-21chaos (see <a href="http://github/com/chaos/lustre" class="external-link" target="_blank" rel="nofollow noopener">http://github/com/chaos/lustre</a>), and most likely of particular interest are these two patches that we are carrying:</p> <ul> <li><a href="https://jira.whamcloud.com/browse/LU-3934" title="Directories gone missing after 2.4 update" class="issue-link" data-issue-key="LU-3934"><del>LU-3934</del></a> scrub: detect upgraded from 1.8 correctly</li> <li><a href="https://jira.whamcloud.com/browse/LU-3420" title="OI scrubbing could not automatically engage after restoring a secondary MDT from a (file-level) backup" class="issue-link" data-issue-key="LU-3420"><del>LU-3420</del></a> scrub: trigger OI scrub properly</li> </ul> <p>We now find that, at least on the ldiskfs MDS nodes, OI scrub runs on <em>every</em> start up of the MDS. The console message looks something like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2014-01-28 09:27:28 sumom-mds1 login: LustreError: 0-0: lsc-MDT0000: trigger OI scrub by RPC for [0x7e4d2310f09:0x2ddf:0x0], rc = 0 [1]</pre> </div></div> <p>Given the frequency of MDS reboots (i.e. often) required lately for other bugs, OI scrub is running far too much.</p> LU-4554

OI scrub always runs on ldiskfs MDS start up

Bug Major Resolved Fixed James Nunez Christopher Morrone mn4 Tue, 28 Jan 2014 18:40:56 +0000 Wed, 23 Apr 2014 15:08:34 +0000 Tue, 11 Feb 2014 22:21:50 +0000 Lustre 2.4.1 Lustre 2.6.0 Lustre 2.5.1 0 11 <p>Lai</p> <p>Could you please consult with Fan Yong and respond on this one?</p> <p>Thanks</p> <p>Peter</p> <p>We need to increase the priority on this ticket. The OI scrub is introducing a lot of production problems for us.</p> <p>First of all, while OI scrub is running, many client nodes hang while holding the mdc lock. Apparently they are doing an operation that triggers osd_fid_lookup() on the server. The FID lookup fails and the server decides to start OI scrub (or OI scrub is already running) and returns EINPROGRESS to the client.</p> <p>Unfortunately, the client is now pretty much unusable to anyone until OI scrub complete.</p> <p>Furthermore, it would appear that OI scrub will not respond to the stop command. I'm told that running "lctl lfsck_stop -M ls5-MDT0000" returns "operation already in progress".</p> <p>I'll note that in the <tt>oi_scrub</tt> file, the <tt>Updated</tt> count is 0 and the <tt>Success</tt> count is 5. From the current scan position relative to number of inodes used, I think it is nearing completion. So it seems the OI scrub is not finding anything to repair, yet certain client requests trigger the scrub. From server-side debug logs, it seems that the trigger is happening here in <tt>osd_fid_lookup()</tt>: </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 383 <span class="code-comment">/* Search order: 3. OI files. */</span> 384 result = osd_oi_lookup(info, dev, fid, id, <span class="code-keyword">true</span>); 385 <span class="code-keyword">if</span> (result == -ENOENT) { 386 <span class="code-keyword">if</span> (!fid_is_norm(fid) || fid_is_on_ost(info, dev, fid) || 387 !ldiskfs_test_bit(osd_oi_fid2idx(dev,fid), 388 sf-&gt;sf_oi_bitmap)) 389 GOTO(out, result = 0); 390 391 <span class="code-keyword">goto</span> trigger; 392 } </pre> </div></div> <p>There are no debug log entries from <tt>osd_iget()</tt> so it must be jumping past the <tt>iget:</tt> goto label, which means it must take the path above.</p> <p>The FID <span class="error">&#91;0x7e4d2310f09:0x2ddf:0x0&#93;</span> is an normal FID, according to above logic, the condition:</p> <p>"(ldiskfs_test_bit(osd_oi_fid2idx(dev,fid), sf-&gt;sf_oi_bitmap))" should true.</p> <p>That means the OSD thinks that it is in OI files re-creating, so it is not sure whether it is because the object does not non-exist or because related OI mapping is not re-inserted yet.</p> <p>So would you please to show the OI_scrub file under /proc to check whether the system is really in re-creating the OI files or not? Thanks!</p> <p>Here is the oi_scrub output from one of the MDS nodes that was rebooted on our open network this morning:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&gt; cat osd-ldiskfs/lsc-MDT0000/oi_scrub name: OI_scrub magic: 0x4c5fd252 oi_files: 1 status: completed flags: param: time_since_last_completed: 28001 seconds time_since_latest_start: 36680 seconds time_since_last_checkpoint: 28001 seconds latest_start_position: 12 last_checkpoint_position: 991133697 first_failure_position: N/A checked: 170974399 updated: 0 failed: 0 prior_updated: 0 noscrub: 2284 igif: 12381456 success_count: 12 run_time: 8678 seconds average_speed: 19702 objects/sec real-time_speed: N/A current_position: N/A </pre> </div></div> <p>Then it is strange, the code should not jump from such place. Is there any (-1 level) debug log when the OI scrub triggered?</p> <p>Fan Yong, the -1 debug log is from a classified system, so I can't send it, but if you have specific questions about it I can look for you. While the scrub was in progress, the flags field only had 'auto'. An example FID from that system that followed the "trigger" path was <span class="error">&#91;0x1a89082ad98:0x4d:0x0&#93;</span>.</p> <p>I also notice <tt>osd_fid_lookup()</tt> starts the scrub using <tt>osd_scrub_start(dev)</tt>, which only enables the flag <tt>SS_AUTO</tt>. So even though <tt>(ldiskfs_test_bit(osd_oi_fid2idx(dev,fid), sf-&gt;sf_oi_bitmap))</tt> is true, (unless I misunderstand something) we would not see the "recreated" flag in <tt>oi_scrub</tt>.</p> <p>I peeked at the /OI_scrub file while an auto-scrub was running. It showed that bit 0 was set in sf-&gt;sf_oi_bitmap. This is wrong, because the OI already exists and OI_scrub has already run to completion several times.</p> <p>I think I see the problem in <a href="https://github.com/chaos/lustre/blob/2.4.0-19chaos/lustre/osd-ldiskfs/osd_oi.c#L299" class="external-link" target="_blank" rel="nofollow noopener"><tt>osd_oi_table_open()</tt></a>. Note the format string assumes the OI containers have names like oi.16.0, oi.16.1, and so on. However, for our upgraded filesystems we have only one OI container named oi.16. So <tt>osd_oi_open()</tt> returns <tt>ENOENT</tt> and we proceed to set the "recreated" bit in the bitmap.</p> <p>Patch for master: <a href="http://review.whamcloud.com/#/c/9067/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9067/</a></p> <p>James, Lai is on holiday this week. Could you please cherry-pick this patch to b2_4 and b2_5 once it has landed to master. This can now be done directly in Gerrit. Please also add the "Lustre-change:" and "Lustre-commit:" tags to the commit messages as described on <a href="https://wiki.hpdd.intel.com/display/PUB/Commit+Comments" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/PUB/Commit+Comments</a></p> <p>This patch hasn't landed to master yet, but I created a b2_5 and b2_4 patch at:</p> <p>b2_4 - <a href="http://review.whamcloud.com/#/c/9140/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9140/</a><br/> b2_5 - <a href="http://review.whamcloud.com/#/c/9139/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9139/</a></p> <p>Landed for 2.5.1 and 2.6</p> Development Rank 1|hzwdsf: Rank (Obsolete) 12442 Severity