https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5812 Lustre <p>Intermittent occurrences of the following in console logs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2014-10-25 04:32:08.953847 {RMP22Oc150844422} [mmcs]{0}.7.0: LustreError: 3262:0:(osc_request.c:853:osc_announce_cached()) fsv-OST0045-osc-c0000003e09df1c0: dirty 129051 + 1573 - 0 &gt; system dirty_max 130608 2014-10-25 04:32:08.954403 {RMP22Oc150844422} [mmcs]{0}.7.0: LustreError: 3262:0:(osc_request.c:853:osc_announce_cached()) Skipped 152 previous similar messages </pre> </div></div> IBM BG/Q system vulcan<br/> Lustre client Build Version: 2.5.3-0.13morrone-0.13morrone--PRISTINE-2.6.32-431.1.1.bgq.3blueos.V1R2M2.bl2.2_4.ppc64<br/> Lustre server Build Version: 2.4.2-17chaos-17chaos--PRISTINE-2.6.32-431.29.2.1chaos.ch5.2.x86_64<br/> LU-5812

osc_request.c:853:osc_announce_cached() dirty 129051 + 1573 - 0 > system dirty_max 130608

Bug Major Resolved Won't Fix Jian Yu Olaf Faaland llnl Mon, 27 Oct 2014 18:23:25 +0000 Thu, 3 Jun 2021 07:33:13 +0000 Wed, 29 Nov 2017 17:42:40 +0000 Lustre 2.5.3 0 9 <p>The error is initially reported 33 times (not counting the "n previous similar messages"). In those 33 cases:</p> <p>345 &lt;= obd_dirty_pages &lt;= 5787<br/> obd_dirty_transit_pages = 0</p> <p>So obd_unstable_pages is high ( &gt;=124,821)</p> <p>console logs from I/O nodes</p> <p>Yu, Jian</p> <p>Could you please advise on this issue?</p> <p>Thanks</p> <p>Peter</p> <p>Hi Olaf,</p> <p>Could you please check whether the client build contained patch <a href="http://review.whamcloud.com/10937" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10937</a> or not? Thanks.</p> <p>It does not.</p> <p>Thank you Chris for the info.</p> <p>The "dirty 129051 + 1573 - 0 &gt; system dirty_max 130608" error message was printed by the following codes on 2.5.3-0.13morrone ( <a href="https://github.com/chaos/lustre" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/chaos/lustre</a> ):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> void osc_announce_cached(struct client_obd *cli, struct obdo *oa, <span class="code-object">long</span> writing_bytes) { <span class="code-comment">//…… </span> CERROR(<span class="code-quote">"%s: dirty %d + %d - %d &gt; system dirty_max %d\n"</span>, cli-&gt;cl_import-&gt;imp_obd-&gt;obd_name, cfs_atomic_read(&amp;obd_unstable_pages), cfs_atomic_read(&amp;obd_dirty_pages), cfs_atomic_read(&amp;obd_dirty_transit_pages), obd_max_dirty_pages); <span class="code-comment">//…… </span>} </pre> </div></div> <p>While on Lustre 2.5.3, the codes are:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> void osc_announce_cached(struct client_obd *cli, struct obdo *oa, <span class="code-object">long</span> writing_bytes) { <span class="code-comment">//…… </span> CERROR(<span class="code-quote">"dirty %d - %d &gt; system dirty_max %d\n"</span>, cfs_atomic_read(&amp;obd_dirty_pages), cfs_atomic_read(&amp;obd_dirty_transit_pages), obd_max_dirty_pages); <span class="code-comment">//...... </span>} </pre> </div></div> <p>The difference is that obd_unstable_pages is tracked and counted on 2.5.3-0.13morrone by the patch <a href="http://review.whamcloud.com/4245" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4245</a> for <a href="https://jira.whamcloud.com/browse/LU-2139" title="Tracking unstable pages" class="issue-link" data-issue-key="LU-2139"><del>LU-2139</del></a>, which was originally added into Lustre 2.5.3 but finally reverted due to regressions (<a href="https://jira.whamcloud.com/browse/LU-3274" title="osc_cache.c:1774:osc_dec_unstable_pages()) ASSERTION( atomic_read(&amp;cli-&gt;cl_cache-&gt;ccc_unstable_nr) &gt;= 0 ) failed" class="issue-link" data-issue-key="LU-3274"><del>LU-3274</del></a> and <a href="https://jira.whamcloud.com/browse/LU-3277" title="LU-2139 may cause the performance regression" class="issue-link" data-issue-key="LU-3277"><del>LU-3277</del></a>).</p> <p>If we do not want to revert the patch from 2.5.3-0.13morrone, we need figure out why the amount of unstable pages is large.</p> <p>On master branch, patch <a href="http://review.whamcloud.com/4245/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4245/</a> for <a href="https://jira.whamcloud.com/browse/LU-2139" title="Tracking unstable pages" class="issue-link" data-issue-key="LU-2139"><del>LU-2139</del></a> was reverted, and the revised version in <a href="http://review.whamcloud.com/6284" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6284</a> was landed later.<br/> After that, patch <a href="http://review.whamcloud.com/10003" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10003</a> for <a href="https://jira.whamcloud.com/browse/LU-4841" title="Performance regression in master for threads more than 2" class="issue-link" data-issue-key="LU-4841"><del>LU-4841</del></a> was landed to revise the counting of unstable pages, which can resolve the issue in this ticket.</p> <p>I'll back-port those patches to Lustre b2_5 branch.</p> <p>Here are the back-ported patches for Lustre b2_5 branch:</p> <ol> <li><a href="http://review.whamcloud.com/12604" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12604</a> (from <a href="http://review.whamcloud.com/6284" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6284</a>)</li> <li><a href="http://review.whamcloud.com/12605" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12605</a> (from <a href="http://review.whamcloud.com/4374" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4374</a>)</li> <li><a href="http://review.whamcloud.com/12606" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12606</a> (from <a href="http://review.whamcloud.com/4375" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4375</a>)</li> <li><a href="http://review.whamcloud.com/12612" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12612</a> (from <a href="http://review.whamcloud.com/5935" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5935</a>)</li> <li><a href="http://review.whamcloud.com/12613" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12613</a> (from <a href="http://review.whamcloud.com/8215" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8215</a>)</li> <li><a href="http://review.whamcloud.com/12615" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12615</a> (from <a href="http://review.whamcloud.com/10003" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10003</a>)</li> </ol> <p>With the above patches, the unstable pages tracking and counting issues were fixed.</p> <p>The first 5 patches are ready to land. The sixth patch needs to be re-implemented on Lustre b2_5 branch.</p> <p>The sixth patch from <a href="http://review.whamcloud.com/10003" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10003</a> heavily depends on the patches for <a href="https://jira.whamcloud.com/browse/LU-3321" title="2.x single thread/process throughput degraded from 1.8" class="issue-link" data-issue-key="LU-3321"><del>LU-3321</del></a>, which would not likely to be back-ported to Lustre b2_5 branch.</p> <p>Hi Jinshan,</p> <p>With applying the first 5 patches on Lustre b2_5 branch, for the sixth one, may I just made the following change so as to quiet the error message in this ticket?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/lustre/osc/osc_request.c b/lustre/osc/osc_request.c index 1c42033..dcfc660 100644 --- a/lustre/osc/osc_request.c +++ b/lustre/osc/osc_request.c @@ -839,16 +839,14 @@ <span class="code-keyword">static</span> void osc_announce_cached(struct client_obd *cli, struct obdo *oa, cli-&gt;cl_dirty_pages, cli-&gt;cl_dirty_transit, cli-&gt;cl_dirty_max_pages); oa-&gt;o_undirty = 0; - } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (unlikely(cfs_atomic_read(&amp;obd_unstable_pages) + - cfs_atomic_read(&amp;obd_dirty_pages) - + } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (unlikely(cfs_atomic_read(&amp;obd_dirty_pages) - cfs_atomic_read(&amp;obd_dirty_transit_pages) &gt; (<span class="code-object">long</span>)(obd_max_dirty_pages + 1))) { /* The cfs_atomic_read() allowing the cfs_atomic_inc() are * not covered by a lock thus they may safely race and trip * <span class="code-keyword">this</span> CERROR() unless we add in a small fudge factor (+1). */ - CERROR(<span class="code-quote">"%s: dirty %d + %d - %d &gt; system dirty_max %d\n"</span>, + CERROR(<span class="code-quote">"%s: dirty %d - %d &gt; system dirty_max %d\n"</span>, cli-&gt;cl_import-&gt;imp_obd-&gt;obd_name, - cfs_atomic_read(&amp;obd_unstable_pages), cfs_atomic_read(&amp;obd_dirty_pages), cfs_atomic_read(&amp;obd_dirty_transit_pages), obd_max_dirty_pages); </pre> </div></div> <p>Yes, that looks good.</p> <p>Thank you, Jinshan. The sixth patch <a href="http://review.whamcloud.com/12615" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12615</a> contains the above change.</p> <p>Olaf, is this in our local releases?</p> <p>Marc, Chris,</p> <p>It appears to me that some of the patches are not in our lustre 2.5.5-3chaos stack. Details:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>http://review.whamcloud.com/12604 partial * 9722ebf LU-2139 osc: Track and limit "unstable" pages http://review.whamcloud.com/12605 full * 666430c LU-2139 osc: Track number of "unstable" pages per osc http://review.whamcloud.com/12606 full * 534ef35 LU-2139 osc: Use SOFT_SYNC to urge server commit http://review.whamcloud.com/12612 full * 003f186 LU-2139 ofd: Do async commit if SOFT_SYNC is seen http://review.whamcloud.com/12613 not found http://review.whamcloud.com/12615 not found </pre> </div></div> <p>where "partial" means some of the files changed in the patch shown on gerritt were not changed by the patch in the 2.5.5-3chaos stack.</p> <p>-Olaf</p> <p>If the first one is not exact, it is likely because that is not the most recent version of the patch. You have no way to know that from the information presented on line, unfortunately.</p> <p>Although this was never resolved, the impact of this issue is low and we are moving our clients from Lustre 2.5 to 2.8. Closing.</p> <p>The 2.12.6 has the same issue.</p> Development End date Mon, 22 Feb 2016 18:23:25 +0000 Rank 1|hzwzk7: Rank (Obsolete) 16298 Severity Start date Mon, 27 Oct 2014 18:23:25 +0000