https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13854 Lustre <p>Trying to create a file after setting up changelogs on a fresh new FS yields an IO error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lustre/tests/llmount.sh (...) # lctl set_param mdd.*.changelog_mask=MTIME mdd.lustre-MDT0000.changelog_mask=MTIME # lctl --device lustre-MDT0000 changelog_register lustre-MDT0000: Registered changelog userid 'cl1' #touch /mnt/lustre/file touch: setting times of '/mnt/lustre/file': Input/output error </pre> </div></div> <p><tt>dmesg</tt> provides some interesting information:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># dmesg -H | tail -n 3 [Aug 5 11:23] Lustre: lustre-MDD0000: changelog on [Aug 5 11:25] LustreError: 23303:0:(llog_cat.c:544:llog_cat_current_log()) lustre-MDD0000: next log does not exist! [ +0.003231] LustreError: 24631:0:(llite_lib.c:1707:ll_md_setattr()) md_setattr fails: rc = -5 </pre> </div></div> <p>Happy hunting <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> LU-13854

lustre-MDD0000: next log does not exist!

Bug Major Open Unresolved WC Triage Quentin Bouget Wed, 5 Aug 2020 09:36:06 +0000 Thu, 14 Jul 2022 15:10:48 +0000 Upstream 0 4 <p>Is this something that you are planning to investigate?</p> <p>No, this was just a test system I setup to review a patch.<br/> This is not something we hit in production (yet).</p> <p>I believe I ran into this:</p> <p>root@node:/# chmod -R 777 /lustrefs<br/> chmod: changing permissions of '/lustrefs': Input/output error<br/> root@node:/# dmesg -T<br/> ...<br/> <span class="error">&#91;Mon Jun 27 17:25:18 2022&#93;</span> LustreError: 1259:0:(llite_lib.c:1712:ll_md_setattr()) md_setattr fails: rc = -5 </p> <p>Looking at the original error, I see the following at the exact time of chmod failure on the MDT:<br/> <span class="error">&#91;250875.733525&#93;</span> LustreError: 32002:0:(llog_cat.c:544:llog_cat_current_log()) lustrefs-MDD0000: next log does not exist!<br/>  <br/> There are no surrounding lines that suggest there weren't free slots or something else to indicate why this is happening.<br/>  <br/> The description seems to suggest the changelog mask may be involved.  On my clusters I use:<br/> mdd.lustrefs-MDT0000.changelog_mask=all-ATIME-FLRW-GXATR-MARK-MIGRT-NOPEN-OPEN-RESYNC-XATTR<br/>  <br/> <tt>Looking across the many clusters this bug is hit approximately once per day.  Are there any suggestions for ways I could help provide additional debugging information?  Also, is there a reason to not return EAGAIN here rather than EIO (I'm not intimately familiar with this code so I'm sure there is a good reason)?  Looking at this block it feels like just trying again would be reasonable (and certainly these nodes do recover – just that single command gets EIO back).  Failing a whole job for just a one-off retryable concurrency issue is unfortunate.</tt><br/>  <br/> <tt>   536   /* Sigh, the chd_next_log and chd_current_log is initialized</tt><tt>   537    * in declare phase, and we do not serialize the catlog</tt><br/> <tt>   538    * accessing, so it might be possible the llog creation</tt><br/> <tt>   539    * thread (see llog_cat_declare_add_rec()) did not create</tt><br/> <tt>   540    * llog successfully, then the following thread might</tt><br/> <tt>   541    * meet this situation. */</tt><br/> <tt>   542   if (IS_ERR_OR_NULL(cathandle-&gt;u.chd.chd_next_log)) {</tt><br/> <tt>   543     CERROR("%s: next log does not exist!\n",</tt><br/> <tt>   544            cathandle-&gt;lgh_ctxt-&gt;loc_obd-&gt;obd_name);</tt><br/> <tt>   545     loghandle = ERR_PTR(-EIO);</tt><br/> <tt>   546     if (cathandle-&gt;u.chd.chd_next_log == NULL) {</tt><br/> <tt>   547       /* Store the error in chd_next_log, so</tt><br/> <tt>   548        * the following process can get correct</tt><br/> <tt>   549        * failure value */</tt><br/> <tt>   550       cathandle-&gt;u.chd.chd_next_log = loghandle;</tt><br/> {{   551     }}}<br/> <tt>   552     GOTO(out_unlock, loghandle);</tt><br/>  <br/>  </p> Development Epic/Theme changelog Rank 1|i0171b: Rank (Obsolete) 9223372036854775807 Severity