https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4858 Lustre <p>This bug was discovered due to the fact we run a utility that monitors the OST stats from /proc/fs/lustre/obdfilter/fsname-OSTXXXX. These stats were still being accessed while the test file system was taken down.</p> <p>&lt;4&gt;<span class="error">&#91;85795.199509&#93;</span> Lustre: server umount a1_thin-OST0129 complete<br/> &lt;0&gt;<span class="error">&#91;85795.339106&#93;</span> LustreError: 9005:0:(dt_object.h:1257:dt_statfs()) ASSERTION( dev ) failed: <br/> &lt;0&gt;<span class="error">&#91;85795.357847&#93;</span> LustreError: 9005:0:(dt_object.h:1257:dt_statfs()) LBUG<br/> &lt;4&gt;<span class="error">&#91;85795.377690&#93;</span> Pid: 9005, comm: cerebrod<br/> &lt;4&gt;<span class="error">&#91;85795.387927&#93;</span> <br/> &lt;4&gt;<span class="error">&#91;85795.387928&#93;</span> Call Trace:<br/> &lt;4&gt;<span class="error">&#91;85795.406853&#93;</span> <span class="error">&#91;&lt;ffffffffa03d4895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt;<span class="error">&#91;85795.427104&#93;</span> <span class="error">&#91;&lt;ffffffffa03d4e97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt;<span class="error">&#91;85795.447295&#93;</span> <span class="error">&#91;&lt;ffffffffa0d8b3be&gt;&#93;</span> ofd_statfs_internal+0x23e/0x2a0 <span class="error">&#91;ofd&#93;</span><br/> &lt;4&gt;<span class="error">&#91;85795.467649&#93;</span> <span class="error">&#91;&lt;ffffffffa0d8b487&gt;&#93;</span> ofd_statfs+0x67/0x510 <span class="error">&#91;ofd&#93;</span><br/> &lt;4&gt;<span class="error">&#91;85795.487335&#93;</span> <span class="error">&#91;&lt;ffffffffa05496d0&gt;&#93;</span> lprocfs_rd_filesfree+0x170/0x4c0 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt;<span class="error">&#91;85795.508376&#93;</span> <span class="error">&#91;&lt;ffffffff811e9823&gt;&#93;</span> ? proc_reg_open+0xc3/0x160<br/> &lt;4&gt;<span class="error">&#91;85795.528056&#93;</span> <span class="error">&#91;&lt;ffffffff811e9760&gt;&#93;</span> ? proc_reg_open+0x0/0x160<br/> &lt;4&gt;<span class="error">&#91;85795.547148&#93;</span> <span class="error">&#91;&lt;ffffffffa0546563&gt;&#93;</span> lprocfs_fops_read+0xf3/0x1f0 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt;<span class="error">&#91;85795.567489&#93;</span> <span class="error">&#91;&lt;ffffffff811e9d9e&gt;&#93;</span> proc_reg_read+0x7e/0xc0<br/> &lt;4&gt;<span class="error">&#91;85795.587040&#93;</span> <span class="error">&#91;&lt;ffffffff81181cb5&gt;&#93;</span> vfs_read+0xb5/0x1a0<br/> &lt;4&gt;<span class="error">&#91;85795.597941&#93;</span> <span class="error">&#91;&lt;ffffffff81181df1&gt;&#93;</span> sys_read+0x51/0x90<br/> &lt;4&gt;<span class="error">&#91;85795.617517&#93;</span> <span class="error">&#91;&lt;ffffffff8100b072&gt;&#93;</span> system_call_fastpath+0x16/0x1b<br/> &lt;4&gt;<span class="error">&#91;85795.637694&#93;</span> <br/> &lt;0&gt;<span class="error">&#91;85795.647408&#93;</span> Kernel panic - not syncing: LBUG</p> <p>We have core dumps as well.</p> Affects any OSS with any environment. LU-4858

Race when reading ofd proc entries while unmounting OST

Bug Major Resolved Fixed Lai Siyao James A Simmons mn4 Thu, 3 Apr 2014 14:12:28 +0000 Wed, 28 May 2014 07:12:45 +0000 Thu, 8 May 2014 14:48:40 +0000 Lustre 2.6.0 Lustre 2.5.1 Lustre 2.4.3 Lustre 2.6.0 Lustre 2.5.2 0 9 <p>James, </p> <p>Are you planning on investigating this one and following this up with a patch?</p> <p>Thanks, <br/> James</p> <p>James, But why don't you stop the utility before you take the filesystem down ??!! <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>More seriously, at 1st look it seems not very logical that in ofd_<span class="error">&#91;init0,fini&#93;</span>() routines, the procfs stuff is initialized/stopped (via ofd_procfs_init()/ofd_profs_fini() routines) before/after others players (like the "stack" and its ofd_stack_init()/ofd_stack_fini() routines) it depends on. May be we can just change this order in the code ? If not easy/possible, then some protection will have to be added to prevent the race.</p> <p>Lai</p> <p>Could you please assist with this one?</p> <p>Thanks</p> <p>Peter</p> <p>Matt Ezell just came up with a patch <a href="http://review.whamcloud.com/#/c/9885" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9885</a> to expose the problem. No patch from me yet. Next week I will be at LUG so it might be a while before I have a patch to test.</p> <p>Bruno - the tool is LMT, which is always running in the background. In this specific case, we were only unmounting some of the OSTs from the OSS. So we wanted it to remain running.</p> <p>I had a good reproducer in my test environment based on 2.4, but I couldn't get it to fail in Maloo. It turns out that in master, some of the OBD methods (including statfs) were removed from OFD. So accessing /proc/fs/lustre/obdfilter/fsname-OSTXXXX/filesfree (for example) now returns EOPNOTSUPP. So for master we probably want to symlink the following entries to their corresponding ../../osd-*/fsname-OSTXXXX/&lt;name&gt; entry, similar to how we have brw_stats, read_cache_enable, readcache_max_filesize, and writethrough_cache_enable. If this doesn't happen, tools like LMT may break. The ones I found that don't work are:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>blocksize filesfree filestotal kbytesavail kbytesfree kbytestotal </pre> </div></div> <p>After that change, I'm not sure if any of the remaining code paths can hit this race or not. Certainly my old reproducer doesn't work anymore. And having a client do something like "df $MOUNT" just hangs while the unmount is happening.</p> <p>I haven't tried Bruno's suggestion of just reordering the OST tear-down procedure, but I would expect it to be safe.</p> <p>This also brings up another question of when to use a LASSERT versus handle the unexpected situation gracefully. That is, when should</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LASSERT( dev )</pre> </div></div> <p>be written as</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">if</span> ( dev == NULL ) <span class="code-keyword">return</span> -ENODEV;</pre> </div></div> <p>patches are ready:<br/> master: <a href="http://review.whamcloud.com/#/c/10082/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10082/</a><br/> 2.4: <a href="http://review.whamcloud.com/#/c/10083/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10083/</a><br/> 2.5: <a href="http://review.whamcloud.com/#/c/10084/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10084/</a></p> <p>Matt, could you help verify it works in your system? BTW I think you can revive <a href="http://review.whamcloud.com/#/c/9885" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9885</a> since my patch will fix the issue you mentioned above.</p> <p>Landed for 2.6. Will consider for b2_5 and b2_4 also.</p> <p>Patch landed to b2_5 for 2.5.2.</p> Development Rank 1|hzwj6n: Rank (Obsolete) 13397 Severity