https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7198 Lustre <p>We continue to see messages like this:<br/> &lt;ConMan&gt; Console <span class="error">&#91;sierra562&#93;</span> log at 2015-07-20 15:00:00 PDT.<br/> 2015-07-20 15:56:56 Lustre: 13097:0:(vvp_io.c:701:vvp_io_fault_start()) binary <span class="error">&#91;0x20004f83e:0x3bc8:0x0&#93;</span> changed while waiting for the page fault lock</p> <p>&lt;ConMan&gt; Console <span class="error">&#91;sierra562&#93;</span> log at 2015-07-20 16:00:00 PDT.<br/> 2015-07-20 16:13:50 Lustre: 13626:0:(vvp_io.c:701:vvp_io_fault_start()) binary <span class="error">&#91;0x20004f83e:0x3bc8:0x0&#93;</span> changed while waiting for the page fault lock<br/> 2015-07-20 16:30:41 Lustre: 13679:0:(vvp_io.c:701:vvp_io_fault_start()) binary <span class="error">&#91;0x20004f83e:0x3bc8:0x0&#93;</span> changed while waiting for the page fault lock</p> <p>&lt;ConMan&gt; Console <span class="error">&#91;sierra562&#93;</span> log at 2015-07-20 17:00:00 PDT.<br/> 2015-07-20 17:04:21 Lustre: 13807:0:(vvp_io.c:701:vvp_io_fault_start()) binary <span class="error">&#91;0x20004f83e:0x3bc8:0x0&#93;</span> changed while waiting for the page fault lock<br/> 2015-07-20 17:38:08 Lustre: 14373:0:(vvp_io.c:701:vvp_io_fault_start()) binary <span class="error">&#91;0x20004f83e:0x3bc8:0x0&#93;</span> changed while waiting for the page fault lock</p> <p>Creating a new issue rather than reopen <a href="https://jira.whamcloud.com/browse/LU-4308" title="MPI job causes errors &quot;binary changed while waiting for the page fault lock&quot;" class="issue-link" data-issue-key="LU-4308"><del>LU-4308</del></a>.</p> Livermore branch: lustre-2.5.4-4chaos_2.6.32_504.16.2.1chaos.ch5.3.x86_64.x86_64 LU-7198

vvp_io.c:701:vvp_io_fault_start()) binary changed while waiting for the page fault lock

Bug Major Resolved Fixed Zhenyu Xu D. Marc Stearman llnl llnlfixready Tue, 22 Sep 2015 20:41:02 +0000 Thu, 14 Jun 2018 21:41:18 +0000 Mon, 25 Apr 2016 20:25:34 +0000 Lustre 2.5.4 Lustre 2.9.0 0 16 <p>Hi Bobijam,<br/> Can you take this issue?<br/> Thanks.<br/> Joe</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/16622" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16622</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7198" title="vvp_io.c:701:vvp_io_fault_start()) binary changed while waiting for the page fault lock" class="issue-link" data-issue-key="LU-7198"><del>LU-7198</del></a> llite: take/release lli_trunc_sem in vvp_io layer<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: efcd5d8ea9b6a241d588959a7a4688998105fcc5</p> <p>Lustre 2.5.4-12chaos contains change 16650 targeting the b2_5_fe branch. That did not fix the stated problem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2015-10-19 18:34:53 Lustre: 87030:0:(vvp_io.c:716:vvp_io_fault_start()) binary [0x200053b04:0xee27:0x0] changed while waiting for the page fault lock &lt;ConMan&gt; Console [cab34] log at 2015-10-19 19:00:00 PDT. &lt;ConMan&gt; Console [cab34] log at 2015-10-19 20:00:00 PDT. 2015-10-19 20:21:21 Lustre: 88635:0:(vvp_io.c:716:vvp_io_fault_start()) binary [0x200053b04:0xee27:0x0] changed while waiting for the page fault lock </pre> </div></div> <p>Keep trying.</p> <p>And, even worse, we now have applications hanging in sys_ftruncate all over the place. I can't help but think that this patch that messes with locking around vvp_io_setattr_start() is to blame for our current major center-wide disruption.</p> <p>Here is the place where code is hanging:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2015-10-21 17:31:35 XXXXXX D 0000000000000006 0 28745 28734 0x00000004 2015-10-21 17:31:35 ffff8805f0befb98 0000000000000086 ffff88034ac159c0 ffff88033b277150 2015-10-21 17:31:35 00ff8805f0befc28 0000000000000000 ffff8805f0befb28 ffff88034ac159c0 2015-10-21 17:31:35 ffff8805eee3e0c0 0000000000000006 ffff8805f3043068 ffff8805f0beffd8 2015-10-21 17:31:35 Call Trace: 2015-10-21 17:31:35 [&lt;ffffffff8153e475&gt;] rwsem_down_failed_common+0x95/0x1d0 2015-10-21 17:31:35 [&lt;ffffffff8153e5d3&gt;] rwsem_down_write_failed+0x23/0x30 2015-10-21 17:31:35 [&lt;ffffffff8129fac3&gt;] call_rwsem_down_write_failed+0x13/0x20 2015-10-21 17:31:35 [&lt;ffffffff8153daa5&gt;] ? down_write+0x45/0x50 2015-10-21 17:31:35 [&lt;ffffffffa0b0c229&gt;] vvp_io_setattr_start+0x129/0x170 [lustre] 2015-10-21 17:31:35 [&lt;ffffffffa0638c7a&gt;] cl_io_start+0x6a/0x140 [obdclass] 2015-10-21 17:31:35 [&lt;ffffffffa063d384&gt;] cl_io_loop+0xb4/0x1b0 [obdclass] 2015-10-21 17:31:35 [&lt;ffffffffa0b04948&gt;] cl_setattr_ost+0x218/0x2f0 [lustre] 2015-10-21 17:31:35 [&lt;ffffffffa0ad06f1&gt;] ll_setattr_raw+0xa31/0x1060 [lustre] 2015-10-21 17:31:35 [&lt;ffffffffa0ad0d85&gt;] ll_setattr+0x65/0xd0 [lustre] 2015-10-21 17:31:35 [&lt;ffffffff811b1de8&gt;] notify_change+0x168/0x340 2015-10-21 17:31:35 [&lt;ffffffff81190bd4&gt;] do_truncate+0x64/0xa0 2015-10-21 17:31:35 [&lt;ffffffff8153c771&gt;] ? mutex_lock+0x31/0x60 2015-10-21 17:31:35 [&lt;ffffffff81190f80&gt;] sys_ftruncate+0x120/0x130 2015-10-21 17:31:35 [&lt;ffffffff8100b112&gt;] system_call_fastpath+0x16/0x1b 2015-10-21 17:31:35 XXXXXX D 000000000000000b 0 28750 28734 0x00000004 2015-10-21 17:31:35 ffff88054385bd08 0000000000000082 0000000000000000 ffff8805dce716e0 2015-10-21 17:31:35 ffff88054385bcc8 0000000000000286 00005ab6b25e0bdc ffff88054385bd64 2015-10-21 17:31:35 ffffffffa0b3bb00 0000000105ec1b77 ffff8806395a05f8 ffff88054385bfd8 2015-10-21 17:31:35 Call Trace: 2015-10-21 17:31:35 [&lt;ffffffff8153cc66&gt;] __mutex_lock_slowpath+0x96/0x210 2015-10-21 17:31:35 [&lt;ffffffff8153c77e&gt;] mutex_lock+0x3e/0x60 2015-10-21 17:31:35 [&lt;ffffffffa0acff9a&gt;] ll_setattr_raw+0x2da/0x1060 [lustre] 2015-10-21 17:31:35 [&lt;ffffffffa0ad0d85&gt;] ll_setattr+0x65/0xd0 [lustre] 2015-10-21 17:31:35 [&lt;ffffffff811b1de8&gt;] notify_change+0x168/0x340 2015-10-21 17:31:35 [&lt;ffffffff81190bd4&gt;] do_truncate+0x64/0xa0 2015-10-21 17:31:35 [&lt;ffffffff8153c771&gt;] ? mutex_lock+0x31/0x60 2015-10-21 17:31:35 [&lt;ffffffff81190f80&gt;] sys_ftruncate+0x120/0x130 2015-10-21 17:31:35 [&lt;ffffffff8100b112&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Would you mind getting the code line whereabout of ll_setattr_raw+0x2da? </p> <p>also please upload the reproduce program so that we can use it to check if the same problem exists in master.</p> <p>I like your optimism, but no, we don't have a reproducer to upload.</p> <p>I can get the line number later today when I'm free.</p> <p>ll_setattr_raw+0x2da looks like the obvious location, mutex_lock on line 1620 below:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1616 } 1617 ll_finish_md_op_data(op_data); 1618 } 1619 if (!S_ISDIR(inode-&gt;i_mode)) { 1620 mutex_lock(&amp;inode-&gt;i_mutex); 1621 if ((attr-&gt;ia_valid &amp; ATTR_SIZE) &amp;&amp; !hsm_import) 1622 inode_dio_wait(inode); 1623 } 1624 1625 ll_stats_ops_tally(ll_i2sbi(inode), (attr-&gt;ia_valid &amp; ATTR_SIZE) </pre> </div></div> <p>still investigating, haven't found the deallock root cause.</p> <p>Is there other kind of Dead threads besides these two? I think thread 28750 was waiting for the i_mutex which 28745 was holding, while 28745 seemed waiting for a write lock which is either lli-&gt;lli_trunc_sem (most likely) or i_alloc_sem (unlikely), and there should be other thread holding this lli_trunc_sem.</p> <p>Unfortunately, I do not have that information.</p> <p>Can you provide bt -a output and keep thread name masqueraded if necessary?</p> <p>I find out the deadlock reason, this patch need some prepositional patch to drop the i_mutex usage in write path, or else vvp_io_write_start() will held lli-&gt;lli_trunc_sem then call lustre_generic_file_write()-&gt;generic_file_aio_write() which will get i_mutex, and this order is different from other IO path (setattr, truncate, etc).</p> <p>So I need push some prepositional patch before this one.</p> <p>updated patch #16650 which depends on #17117 and #17116 (these two patches drops i_mutex usage in write path).</p> <p>We can't provide further information on the buggy patch 16500 (bt -a). We stopped using it in production as soon as we identified that patch as the problem.</p> <p>Keep in mind that not only did this patch introduce a deadlock, it also did not fix the error messages which this ticket is about. If you have only fixed the deadlock, then you should really be tracking your 16650 backport in some other ticket.</p> <p>If you think it will address our error message, we would appreciate an explanation of why the latest revision is expected to address the error message versus the previous one.</p> <p>After taking a further look, I tend to think timestamp checking in vvp_io_fault_start() is not atomic therefore it's normal to get this kind of mismatch.</p> <p>Taking a look at ll_merge_lvb() in b2_5, it first assigns mtime from MDS to inode's i_mtime, and then merge timestamp from OST objects. However, the timestamp from MDS could be stale because we maintain mtime mainly on OST objects(thinking about Lustre write implementation), and if the check in vvp_io_fault_start() happens before mtime on OST objects are merged, definitely it will get wrong timestamp from the inode, even the timestamp it fetched in vvp_io_fault_init() could be wrong in the first place.</p> <p>I tend to think we should remove the timestamp check in vvp_io_fault_start().</p> <p>Bobijam</p> <p>Do you think that Jinshan's suggestion is viable? If so please can you create a patch on that basis?</p> <p>Thanks</p> <p>Peter</p> <p><a href="http://review.whamcloud.com/#/c/17334/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/17334/</a> is the patch</p> <p>Bobi</p> <p>LLNL will need a fix targeted on b2_5_fe</p> <p>Peter</p> <p>port for b2_5_fe <a href="http://review.whamcloud.com/17624" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17624</a> </p> <p>Chris, can you pull in the latest patch please?</p> <p>There are no reviews on the backported patch, and it failed automated testing.</p> <p>Peter, please add this to the work queue. When a patch is ready for us, please add the "llnlfixready" lable.</p> <p>We need a master branch version of this patch as well.</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/19162" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19162</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7198" title="vvp_io.c:701:vvp_io_fault_start()) binary changed while waiting for the page fault lock" class="issue-link" data-issue-key="LU-7198"><del>LU-7198</del></a> clio: remove mtime check in vvp_io_fault_start()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f35f69b866566ce8b7852facf8c914da297e9be0</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19162/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19162/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7198" title="vvp_io.c:701:vvp_io_fault_start()) binary changed while waiting for the page fault lock" class="issue-link" data-issue-key="LU-7198"><del>LU-7198</del></a> clio: remove mtime check in vvp_io_fault_start()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d62f58c32074bc54055801ed0f919bfd5c277f6d</p> <p>Landed for 2.9</p> Duplicate Related LU-4308 Development Rank 1|hzxofj: Rank (Obsolete) 9223372036854775807 Severity