https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6271 Lustre <p>While performing failover testing:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 167-0: lustre-OST0000-osc-ffff88083b339000: This client was evicted by lustre-OST0000; in progress operations using <span class="code-keyword">this</span> service will fail. Feb 23 09:22:48 iwc260 kernel: LustreError: 51585:0:(ldlm_resource.c:777:ldlm_resource_complain()) lustre-OST0000-osc-ffff88083b339000: namespace resource [0x2d097bb:0x0:0x0].0 (ffff88106a992e40) refcount nonzero (1) after lock cleanup; forcing cleanup. Feb 23 09:22:48 iwc260 kernel: LustreError: 51585:0:(ldlm_resource.c:1374:ldlm_resource_dump()) --- Resource: [0x2d097bb:0x0:0x0].0 (ffff88106a992e40) refcount = 3 Feb 23 09:22:48 iwc260 kernel: LustreError: 51585:0:(ldlm_resource.c:1377:ldlm_resource_dump()) Granted locks (in reverse order): Feb 23 09:22:48 iwc260 kernel: LustreError: 51585:0:(ldlm_resource.c:1380:ldlm_resource_dump()) ### ### ns: lustre-OST0000-osc-ffff88083b339000 lock: ffff88106a60d540/0xa1f402a8812988f4 lrc: 3/0,1 mode: PW/PW res: [0x2d097bb:0x0:0x0].0 rrc: 3 type: EXT [0-&gt;18446744073709551615] (req 0-&gt;8191) flags: 0x126400020000 nid: local remote: 0x72a9f4c80e66e07b expref: -99 pid: 51570 timeout: 0 lvb_type: 1 Feb 23 09:22:48 iwc260 kernel: Lustre: lustre-OST0000-osc-ffff88083b339000: Connection restored to lustre-OST0000 (at 192.168.120.13@o2ib) Feb 23 09:22:49 iwc260 kernel: LustreError: 90252:0:(osc_cache.c:3150:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed: Feb 23 09:22:49 iwc260 kernel: LustreError: 90252:0:(osc_cache.c:3150:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed: Feb 23 09:22:49 iwc260 kernel: LustreError: 90252:0:(osc_cache.c:3150:discard_cb()) LBUG Feb 23 09:22:49 iwc260 kernel: LustreError: 90252:0:(osc_cache.c:3150:discard_cb()) LBUG Feb 23 09:22:49 iwc260 kernel: Pid: 90252, comm: ldlm_bl_36 Feb 23 09:22:49 iwc260 kernel: Feb 23 09:22:49 iwc260 kernel: Call Trace: Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa0435895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa0435e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa0b2ba56&gt;] discard_cb+0x156/0x190 [osc] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa0b2bdcc&gt;] osc_page_gang_lookup+0x1ac/0x330 [osc] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa0b2b900&gt;] ? discard_cb+0x0/0x190 [osc] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa0b2c094&gt;] osc_lock_discard_pages+0x144/0x240 [osc] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa04461c1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa0b2b900&gt;] ? discard_cb+0x0/0x190 [osc] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa0b2298b&gt;] osc_lock_flush+0x8b/0x260 [osc] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa0b22e08&gt;] osc_ldlm_blocking_ast+0x2a8/0x3c0 [osc] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa0761a6c&gt;] ldlm_cancel_callback+0x6c/0x170 [ptlrpc] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa077450a&gt;] ldlm_cli_cancel_local+0x8a/0x470 [ptlrpc] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa0779120&gt;] ldlm_cli_cancel+0x60/0x360 [ptlrpc] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa0b22c3b&gt;] osc_ldlm_blocking_ast+0xdb/0x3c0 [osc] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa04461c1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa077cb60&gt;] ldlm_handle_bl_callback+0x130/0x400 [ptlrpc] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa077d0c1&gt;] ldlm_bl_thread_main+0x291/0x3f0 [ptlrpc] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffff81061d00&gt;] ? default_wake_function+0x0/0x20 Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffffa077ce30&gt;] ? ldlm_bl_thread_main+0x0/0x3f0 [ptlrpc] Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 Feb 23 09:22:49 iwc260 kernel: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> Hyperon LU-6271

(osc_cache.c:3150:discard_cb()) ASSERTION( (!(page->cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed:

Bug Blocker Resolved Fixed Jinshan Xiong Cliff White patch Mon, 23 Feb 2015 17:48:52 +0000 Sun, 28 Jun 2020 17:16:31 +0000 Sun, 25 Oct 2015 12:45:50 +0000 Lustre 2.7.0 Lustre 2.8.0 0 25 <p>John,<br/> Could you please have a look and comment on this one?<br/> Thank you!</p> <p>Hi Cliff, could you please include the exact version you were testing along with a description of your failover testing?</p> <p>We are running 2.6.94 lustre-master build 2856. <br/> Failover testing consists of </p> <ul class="alternate" type="square"> <li>power an OSS node off.</li> <li>mount the devices from the dead OSS on the failover node.</li> <li>sleep for awhile</li> <li>power up the dead OSS</li> <li>failback the devices.<br/> This is done 'manually' via a script, as we do not have HA software installed. </li> </ul> <p>What were the clients running?</p> <p>They were running the client bits from the same build</p> <p>What application(s) were they running?</p> <p>They were running the SWL workload which is a mix of standard tests, including IOR, mutest, simul and others. </p> <p>Does the workload include Direct IO and how often can it be reproduced?</p> <p>I took a look at the log. From the log, it looks like the write back was not triggered at all. The client was discarding the pages starting from index 4608, and the assertion was hit when it tried to discard page 5120. The log is too short and some key information is missing because I want to know the lock range, type, and what pages were actually written back.</p> <p>Cliff, can you rerun the test case, with a little bit more debug buffer size, say 100M, and enable cache on the client and then reproduce this issue? As follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param debug_size=100M lctl set_param debug=+cache </pre> </div></div> <p>BTW, do the test programs use group lock?</p> <p>Our normal default is <br/> lctl set_param panic_on_lbug=0;lctl set_param debug_mb=512M</p> <p>So, no idea why the info you want is not in that log. I am past that build in the test cycle, I will repeat the test when I get the next build test cycle.</p> <p>We encountered the same problem at NCI, when the OSTs evicted the clients, many clients LBUGed with the same context above. Now we can reproduce this with a simple setup:<br/> on the client, have a script monitor and keep the number of write processes, e.g. IOR. on the OSTs, have another script does the eviction constantly with an interval, e.g. 10 seconds.</p> <p>The client always LBUG with the ldlm_resource_dump message. If message appears then it's close. sometimes it takes resource dump messages for a LBUG.</p> <p>Li Dongyang (dongyang.li@anu.edu.au) uploaded a new patch: <a href="http://review.whamcloud.com/14915" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14915</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6271" title="(osc_cache.c:3150:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed:" class="issue-link" data-issue-key="LU-6271"><del>LU-6271</del></a> ldlm: cleanup locks before activating import<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: aeff14823489ef3c53982d440a1df2c583c802d6</p> <p>The lock being canceled belongs to the previous instance of import. The dirty pages are actually protected by a new lock. They shouldn't be written back at this time at all. I'm working on a patch</p> <p>Hi Jinshan,<br/> IMHO the client should clean up everything in cleanup_resource() before reconnecting. Doing this can also avoid the annoying ldlm_resource_complain messages. On our login nodes where users are doing many IOs to their files, once it got evicted from the OST, the cpu will spin for a long time dumping the locks and the box is not responsive at all.</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/14989" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14989</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6271" title="(osc_cache.c:3150:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed:" class="issue-link" data-issue-key="LU-6271"><del>LU-6271</del></a> osc: handle osc eviction correctly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8c01bf6f3c689f40f5863b30478c1f61947e316f</p> <p>Hi Dongyang,</p> <p>Will you try the patch 14989 and see how it goes?</p> <p>Jinshan</p> <p>Hi Jinshan,</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;3&gt;LustreError: 31958:0:(ldlm_resource.c:776:ldlm_resource_complain()) testfs-OST0000-osc-ffff880139269c00: namespace resource [0x22:0x0:0x0].0 (ffff880101507840) refcount nonzero (1) after lock cleanup; forcing cleanup. &lt;3&gt;LustreError: 31958:0:(ldlm_resource.c:1369:ldlm_resource_dump()) --- Resource: [0x22:0x0:0x0].0 (ffff880101507840) refcount = 2 &lt;3&gt;LustreError: 31958:0:(ldlm_resource.c:1372:ldlm_resource_dump()) Granted locks (in reverse order): &lt;3&gt;LustreError: 31958:0:(ldlm_resource.c:1375:ldlm_resource_dump()) ### ### ns: testfs-OST0000-osc-ffff880139269c00 lock: ffff8800bbb524c0/0x51b0732653777408 lrc: 4/0,1 mode: PW/PW res: [0x22:0x0:0x0].0 rrc: 2 type: EXT [0-&gt;18446744073709551615] (req 0-&gt;1048575) flags: 0x126400000000 nid: local remote: 0xdc8209701f2ad6da expref: -99 pid: 31677 timeout: 0 lvb_type: 1 &lt;3&gt;LustreError: 28596:0:(osc_cache.c:3155:discard_cb()) page@ffff88010c35d400[3 ffff88003780db38 1 0 1 ffff8800be860220 (null)] &lt;3&gt; &lt;3&gt;LustreError: 28596:0:(osc_cache.c:3155:discard_cb()) vvp-page@ffff88010c35d468(0:0) vm@ffffea00044f7068 40000000000879 3:0 ffff88010c35d400 5888 lru &lt;3&gt; &lt;3&gt;LustreError: 28596:0:(osc_cache.c:3155:discard_cb()) lov-page@ffff88010c35d4a8, raid0 &lt;3&gt; &lt;3&gt;LustreError: 28596:0:(osc_cache.c:3155:discard_cb()) osc-page@ffff88010c35d510 5888: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 24117248 0 4096 0x0 0x520 | (null) ffff88013c46a568 ffff880037882d90 &gt; 3&lt; + ffff88010bcd7500 1 0 0 &gt; 4&lt; 0 1 8 18446744073665499136 - | + - + - &gt; 5&lt; + - + - | 0 - | 2560 - -&gt; &lt;3&gt; &lt;3&gt;LustreError: 28596:0:(osc_cache.c:3155:discard_cb()) end page@ffff88010c35d400 &lt;3&gt; &lt;3&gt;LustreError: 28596:0:(osc_cache.c:3155:discard_cb()) discard dirty page? &lt;3&gt;LustreError: 28596:0:(osc_cache.c:2466:osc_teardown_async_page()) extent ffff880112ed9288@{[5888 -&gt; 6143/6143], [2|0|+|locking|wi|ffff880037882d90], [1048576|256|+|-|ffff8801386b9880|256|ffff88013c193500]} trunc at 5888. &lt;3&gt;LustreError: 28596:0:(osc_cache.c:2466:osc_teardown_async_page()) ### extent: ffff880112ed9288 &lt;3&gt; ns: testfs-OST0000-osc-ffff880139269c00 lock: ffff8801386b9880/0x51b07326537775eb lrc: 35/0,3 mode: PW/PW res: [0x22:0x0:0x0].0 rrc: 2 type: EXT [0-&gt;18446744073709551615] (req 23068672-&gt;24117247) flags: 0x20000000000 nid: local remote: 0xdc8209701f2ad6fd expref: -99 pid: 31718 timeout: 0 lvb_type: 1 &lt;3&gt;LustreError: 28596:0:(osc_page.c:313:osc_page_delete()) page@ffff88010c35d400[3 ffff88003780db38 4 0 1 (null) (null)] &lt;3&gt; &lt;3&gt;LustreError: 28596:0:(osc_page.c:313:osc_page_delete()) vvp-page@ffff88010c35d468(0:0) vm@ffffea00044f7068 40000000000879 3:0 ffff88010c35d400 5888 lru &lt;3&gt; &lt;3&gt;LustreError: 28596:0:(osc_page.c:313:osc_page_delete()) lov-page@ffff88010c35d4a8, raid0 &lt;3&gt; &lt;3&gt;LustreError: 28596:0:(osc_page.c:313:osc_page_delete()) osc-page@ffff88010c35d510 5888: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 24117248 0 4096 0x0 0x520 | (null) ffff88013c46a568 ffff880037882d90 &gt; 3&lt; + ffff88010bcd7500 0 0 0 &gt; 4&lt; 0 1 8 18446744073644531712 + | + - + - &gt; 5&lt; + - + - | 0 - | 7680 - -&gt; &lt;3&gt; &lt;3&gt;LustreError: 28596:0:(osc_page.c:313:osc_page_delete()) end page@ffff88010c35d400 &lt;3&gt; &lt;3&gt;LustreError: 28596:0:(osc_page.c:313:osc_page_delete()) Trying to teardown failed: -16 &lt;0&gt;LustreError: 28596:0:(osc_page.c:314:osc_page_delete()) ASSERTION( 0 ) failed: &lt;0&gt;LustreError: 28596:0:(osc_page.c:314:osc_page_delete()) LBUG &lt;4&gt;Pid: 28596, comm: ldlm_bl_00 &lt;4&gt; &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffffa0251875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt; [&lt;ffffffffa0251e77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt; [&lt;ffffffffa09386a6&gt;] osc_page_delete+0x446/0x4e0 [osc] &lt;4&gt; [&lt;ffffffffa03b05bd&gt;] cl_page_delete0+0x7d/0x210 [obdclass] &lt;4&gt; [&lt;ffffffffa03b078d&gt;] cl_page_delete+0x3d/0x110 [obdclass] &lt;4&gt; [&lt;ffffffffa085f63d&gt;] ll_invalidatepage+0x8d/0x160 [lustre] &lt;4&gt; [&lt;ffffffffa086ee65&gt;] vvp_page_discard+0xc5/0x160 [lustre] &lt;4&gt; [&lt;ffffffffa03aef18&gt;] cl_page_invoid+0x68/0x160 [obdclass] &lt;4&gt; [&lt;ffffffffa03af023&gt;] cl_page_discard+0x13/0x20 [obdclass] &lt;4&gt; [&lt;ffffffffa0943cb8&gt;] discard_cb+0x88/0x1e0 [osc] &lt;4&gt; [&lt;ffffffffa094414c&gt;] osc_page_gang_lookup+0x1ac/0x330 [osc] &lt;4&gt; [&lt;ffffffffa0943c30&gt;] ? discard_cb+0x0/0x1e0 [osc] &lt;4&gt; [&lt;ffffffffa0943c30&gt;] ? discard_cb+0x0/0x1e0 [osc] &lt;4&gt; [&lt;ffffffffa0952108&gt;] osc_cache_flush+0x178/0x410 [osc] &lt;4&gt; [&lt;ffffffffa093aa68&gt;] osc_ldlm_blocking_ast+0x2a8/0x3c0 [osc] &lt;4&gt; [&lt;ffffffffa053a4cc&gt;] ldlm_cancel_callback+0x6c/0x170 [ptlrpc] &lt;4&gt; [&lt;ffffffffa054c35a&gt;] ldlm_cli_cancel_local+0x8a/0x470 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0550f80&gt;] ldlm_cli_cancel+0x60/0x360 [ptlrpc] &lt;4&gt; [&lt;ffffffffa093a89b&gt;] osc_ldlm_blocking_ast+0xdb/0x3c0 [osc] &lt;4&gt; [&lt;ffffffffa05549c0&gt;] ldlm_handle_bl_callback+0x130/0x400 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0554f21&gt;] ldlm_bl_thread_main+0x291/0x3f0 [ptlrpc] &lt;4&gt; [&lt;ffffffff81064b90&gt;] ? default_wake_function+0x0/0x20 &lt;4&gt; [&lt;ffffffffa0554c90&gt;] ? ldlm_bl_thread_main+0x0/0x3f0 [ptlrpc] &lt;4&gt; [&lt;ffffffff8109e66e&gt;] kthread+0x9e/0xc0 &lt;4&gt; [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 &lt;4&gt; [&lt;ffffffff8109e5d0&gt;] ? kthread+0x0/0xc0 &lt;4&gt; [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 &lt;4&gt; &lt;0&gt;Kernel panic - not syncing: LBUG &lt;4&gt;Pid: 28596, comm: ldlm_bl_00 Not tainted 2.6.32-504.12.2.el6.x86_64 #1 &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffff8152933c&gt;] ? panic+0xa7/0x16f &lt;4&gt; [&lt;ffffffffa0251ecb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] &lt;4&gt; [&lt;ffffffffa09386a6&gt;] ? osc_page_delete+0x446/0x4e0 [osc] &lt;4&gt; [&lt;ffffffffa03b05bd&gt;] ? cl_page_delete0+0x7d/0x210 [obdclass] &lt;4&gt; [&lt;ffffffffa03b078d&gt;] ? cl_page_delete+0x3d/0x110 [obdclass] &lt;4&gt; [&lt;ffffffffa085f63d&gt;] ? ll_invalidatepage+0x8d/0x160 [lustre] &lt;4&gt; [&lt;ffffffffa086ee65&gt;] ? vvp_page_discard+0xc5/0x160 [lustre] &lt;4&gt; [&lt;ffffffffa03aef18&gt;] ? cl_page_invoid+0x68/0x160 [obdclass] &lt;4&gt; [&lt;ffffffffa03af023&gt;] ? cl_page_discard+0x13/0x20 [obdclass] &lt;4&gt; [&lt;ffffffffa0943cb8&gt;] ? discard_cb+0x88/0x1e0 [osc] &lt;4&gt; [&lt;ffffffffa094414c&gt;] ? osc_page_gang_lookup+0x1ac/0x330 [osc] &lt;4&gt; [&lt;ffffffffa0943c30&gt;] ? discard_cb+0x0/0x1e0 [osc] &lt;4&gt; [&lt;ffffffffa0943c30&gt;] ? discard_cb+0x0/0x1e0 [osc] &lt;4&gt; [&lt;ffffffffa0952108&gt;] ? osc_cache_flush+0x178/0x410 [osc] &lt;4&gt; [&lt;ffffffffa093aa68&gt;] ? osc_ldlm_blocking_ast+0x2a8/0x3c0 [osc] &lt;4&gt; [&lt;ffffffffa053a4cc&gt;] ? ldlm_cancel_callback+0x6c/0x170 [ptlrpc] &lt;4&gt; [&lt;ffffffffa054c35a&gt;] ? ldlm_cli_cancel_local+0x8a/0x470 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0550f80&gt;] ? ldlm_cli_cancel+0x60/0x360 [ptlrpc] &lt;4&gt; [&lt;ffffffffa093a89b&gt;] ? osc_ldlm_blocking_ast+0xdb/0x3c0 [osc] &lt;4&gt; [&lt;ffffffffa05549c0&gt;] ? ldlm_handle_bl_callback+0x130/0x400 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0554f21&gt;] ? ldlm_bl_thread_main+0x291/0x3f0 [ptlrpc] &lt;4&gt; [&lt;ffffffff81064b90&gt;] ? default_wake_function+0x0/0x20 &lt;4&gt; [&lt;ffffffffa0554c90&gt;] ? ldlm_bl_thread_main+0x0/0x3f0 [ptlrpc] &lt;4&gt; [&lt;ffffffff8109e66e&gt;] ? kthread+0x9e/0xc0 &lt;4&gt; [&lt;ffffffff8100c20a&gt;] ? child_rip+0xa/0x20 &lt;4&gt; [&lt;ffffffff8109e5d0&gt;] ? kthread+0x0/0xc0 &lt;4&gt; [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Could you have a look at patch 14915 please? We have been testing it for a long time before submitting it.</p> <p>HI,<br/> At Cea with lustre 2.7, we meet the same LBUG <br/> (osc_cache.c:3150:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed:<br/> In my case the ldlm thread has the same stack and I have another user thread who are currently writing on the<br/> concerning file and the stack was :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> cl_sync_io_wait() cl_io_submit_sync() ll_page_sync-io() ll_write_begin() generic_file_buffered_write() __generic_file_aio_write() vvp_io_write_start() cl_io_start() cl_io_loop() ll_file_io_generic() ll_file_aio-write() ll_file_write() vfs_write() sys_write() system_call_fastpath() </pre> </div></div> <p> This thread do a write in 1024 (0x400) pages, and the page concerning by the LBug is a page<br/> who are in this write() <br/> The writing thread waiting the sync of the last page 1024 (0x400)<br/> If I understand well patch 14915 need some additional work, could you confirm ?<br/> Antoine</p> <p>Hi Antoine,</p> <p>Did you notice that this OSC had ever been evicted from your occurrence?</p> <p>Hello Jinshan,<br/> Talking about this problem with Antoine today, I think that the answer is yes, but it need to be double-checked by Antoine on-site.</p> <p>Hi,<br/> Yes, for this client, the 3 osts concerning by the current I/Os has evicted and reconnected just before the assert. And for me that is a "short eviction" due to huge I/O between these OSTs and this client.<br/> Antoine </p> <p>Here is a demo code to show how to reproduce the problem with grouplock:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#include &lt;sys/types.h&gt; #include &lt;sys/stat.h&gt; #include &lt;fcntl.h&gt; #include &lt;sys/ioctl.h&gt; #include &lt;stdlib.h&gt; #include &lt;string.h&gt; #include &lt;unistd.h&gt; #include &lt;stdio.h&gt; #include &lt;lustre/lustre_user.h&gt; <span class="code-object">int</span> main(<span class="code-object">int</span> argc, <span class="code-object">char</span> *argv[]) { <span class="code-object">int</span> fd; <span class="code-object">int</span> rc; <span class="code-object">int</span> gid; <span class="code-object">char</span> *buf; <span class="code-object">int</span> i; fd = open(&lt;file&gt;, O_RDWR|O_CREAT|O_TRUNC); gid = atoi(argv[1]); rc = ioctl(fd, LL_IOC_GROUP_LOCK, gid); printf(<span class="code-quote">"ioctl %d\n"</span>, rc); <span class="code-keyword">if</span> (rc) <span class="code-keyword">return</span> rc; buf = malloc(1&lt;&lt;20); memset(buf, 1, 1&lt;&lt;20); <span class="code-keyword">while</span> (1) write(fd, buf, 1&lt;&lt;20); <span class="code-keyword">return</span> 0; } </pre> </div></div> <p>Run this with a parameter as gid, then evict the client from the OST.</p> <p>I just update <a href="http://review.whamcloud.com/14989" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14989</a> it should be good for general IO but group lock is not yet supported. Please verify if this is the case on your side.</p> <p>We have hit this on lola, with 2.7.0</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">fff8807e5361e40) refcount = 3 LustreError: 118965:0:(ldlm_resource.c:1377:ldlm_resource_dump()) Granted locks (in reverse order): Lustre: soaked-OST000f-osc-ffff88082297e800: Connection restored to soaked-OST000f (at 192.168.1.105@o2ib10) LustreError: 73177:0:(osc_cache.c:3150:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed: LustreError: 73177:0:(osc_cache.c:3150:discard_cb()) LBUG Pid: 73177, comm: ldlm_bl_100 Call Trace: [&lt;ffffffffa046c895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa046ce97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0ba8046&gt;] discard_cb+0x156/0x190 [osc] [&lt;ffffffffa0ba83bc&gt;] osc_page_gang_lookup+0x1ac/0x330 [osc] [&lt;ffffffffa0ba7ef0&gt;] ? discard_cb+0x0/0x190 [osc] [&lt;ffffffffa0ba8684&gt;] osc_lock_discard_pages+0x144/0x240 [osc] [&lt;ffffffffa0ba7ef0&gt;] ? discard_cb+0x0/0x190 [osc] [&lt;ffffffffa0b9ef7b&gt;] osc_lock_flush+0x8b/0x260 [osc] [&lt;ffffffffa0b9f3f8&gt;] osc_ldlm_blocking_ast+0x2a8/0x3c0 [osc] [&lt;ffffffffa0764a6c&gt;] ldlm_cancel_callback+0x6c/0x170 [ptlrpc] [&lt;ffffffffa077732a&gt;] ldlm_cli_cancel_local+0x8a/0x470 [ptlrpc] [&lt;ffffffffa077bf40&gt;] ldlm_cli_cancel+0x60/0x360 [ptlrpc] [&lt;ffffffffa0b9f22b&gt;] osc_ldlm_blocking_ast+0xdb/0x3c0 [osc] [&lt;ffffffffa077f980&gt;] ldlm_handle_bl_callback+0x130/0x400 [ptlrpc] [&lt;ffffffffa077fee1&gt;] ldlm_bl_thread_main+0x291/0x3f0 [ptlrpc] [&lt;ffffffff81064be0&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa077fc50&gt;] ? ldlm_bl_thread_main+0x0/0x3f0 [ptlrpc] [&lt;ffffffff8109e78e&gt;] kthread+0x9e/0xc0 [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 [&lt;ffffffff8109e6f0&gt;] ? kthread+0x0/0xc0 [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Lustre Log from LBUG</p> <p>Hi Jinshan,<br/> The client is stuck after eviction with 14989 patched:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">PID: 15242 TASK: ffff880037cd3520 CPU: 0 COMMAND: <span class="code-quote">"ll_imp_inval"</span> #0 [ffff88003de53b60] schedule at ffffffff81529ab0 #1 [ffff88003de53c38] osc_object_invalidate at ffffffffa103a695 [osc] #2 [ffff88003de53cb8] osc_ldlm_resource_invalidate at ffffffffa102b769 [osc] #3 [ffff88003de53cf8] cfs_hash_for_each_relax at ffffffffa09726ab [libcfs] #4 [ffff88003de53d88] cfs_hash_for_each_nolock at ffffffffa09745ac [libcfs] #5 [ffff88003de53db8] osc_import_event at ffffffffa1036556 [osc] #6 [ffff88003de53e08] ptlrpc_invalidate_import at ffffffffa0c9d7a1 [ptlrpc] #7 [ffff88003de53ec8] ptlrpc_invalidate_import_thread at ffffffffa0ca0608 [ptlrpc] #8 [ffff88003de53ee8] kthread at ffffffff8109e78e #9 [ffff88003de53f48] kernel_thread at ffffffff8100c28a </pre> </div></div> <p>I've updated <a href="http://review.whamcloud.com/#/c/14915/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14915/</a> now we skip the grouplocks during cleanup so we can reconnect to the servers, purge the extents of the grouplock,<br/> and if we see any IO from the grouplock fd, just return -ENOLCK.<br/> The grouplock still needs to be released manually by the user. It's not ideal but at least the client is not crashing any more.</p> <p>In that case, let's land the patch 14915 at least it's working.</p> <p>Hi Dongyang,</p> <p>Do you have a program to reproduce the problem?</p> <p>Jinshan</p> <p>We also found this problem on hyperion test. Here is the debug log. </p> <p>Are the two patches here complimentary (i.e. both should land) or is only one of the two patches needed?</p> <p>Hi Andreas, the two patches are exclusive. I would rather land <a href="http://review.whamcloud.com/14989" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14989</a>, but it's fine for me to land <a href="http://review.whamcloud.com/#/c/14915" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14915</a> if you think it is a better solution.</p> <p>Hi Jinshan,<br/> was sick in the past week, sorry for the late reply.<br/> The reproducer is pretty simple:<br/> On the client, have some threads doing write to a file, ior would do, and start e.g. 20 of them.<br/> On the OSS evict the client over and over again, say 10 sec.<br/> Upon eviction on the client the processes will die as a result, just bump the number up back to 20.</p> <p>Keep it running for a while, and it will hit the problem pretty quick.<br/> BTW with patch set 4 from <a href="http://review.whamcloud.com/14989" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14989</a> the client still stuck at one point:<br/> crash&gt; bt 11413<br/> PID: 11413 TASK: ffff88003b1e0ab0 CPU: 0 COMMAND: "ll_imp_inval"<br/> #0 <span class="error">&#91;ffff88003bd59b60&#93;</span> schedule at ffffffff81529ab0<br/> #1 <span class="error">&#91;ffff88003bd59c38&#93;</span> osc_object_invalidate at ffffffffa090468c <span class="error">&#91;osc&#93;</span><br/> #2 <span class="error">&#91;ffff88003bd59cb8&#93;</span> osc_ldlm_resource_invalidate at ffffffffa08f5769 <span class="error">&#91;osc&#93;</span><br/> #3 <span class="error">&#91;ffff88003bd59cf8&#93;</span> cfs_hash_for_each_relax at ffffffffa022865b <span class="error">&#91;libcfs&#93;</span><br/> #4 <span class="error">&#91;ffff88003bd59d88&#93;</span> cfs_hash_for_each_nolock at ffffffffa022a53c <span class="error">&#91;libcfs&#93;</span><br/> #5 <span class="error">&#91;ffff88003bd59db8&#93;</span> osc_import_event at ffffffffa0900546 <span class="error">&#91;osc&#93;</span><br/> #6 <span class="error">&#91;ffff88003bd59e08&#93;</span> ptlrpc_invalidate_import at ffffffffa055fcb1 <span class="error">&#91;ptlrpc&#93;</span><br/> #7 <span class="error">&#91;ffff88003bd59ec8&#93;</span> ptlrpc_invalidate_import_thread at ffffffffa0562b18 <span class="error">&#91;ptlrpc&#93;</span><br/> #8 <span class="error">&#91;ffff88003bd59ee8&#93;</span> kthread at ffffffff8109e78e<br/> #9 <span class="error">&#91;ffff88003bd59f48&#93;</span> kernel_thread at ffffffff8100c28a</p> <p>Seems it won't reconnect to the OSS.</p> <p>Thanks</p> <p>Hi Dongyang,</p> <p>I have been running this test program for over 10 hours on my testing node but I didn't reproduce this problem. I did this test on my VM nodes so this may be one of reasons why I couldn't reproduce it. Will you please reproduce it again and collect lustre logs and output of dmesg when you see the problem? Also it will be helpful to take a crashdump and then upload it, along with lustre modules, to our ftp site at: ftp.whamcloud.com so that I can take a look.</p> <p>Thanks in advance,<br/> Jinshan</p> <p>Hi Jinshan,<br/> I finally got some time to generate the dump for you. I'm using patchset 5 of <a href="http://review.whamcloud.com/#/c/14989/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14989/</a> on top of master.<br/> While I was trying to reproduce the problem, I got a client crash. I don't know if it's related but I got the dump as well.</p> <p>I tried to upload them to ftp.whamcloud.com/uploads. the <a href="https://jira.whamcloud.com/browse/LU-6271" title="(osc_cache.c:3150:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed:" class="issue-link" data-issue-key="LU-6271"><del>LU-6271</del></a>_pkgs.tar.gz was uploaded fine, which has all the lustre modules and debuginfo I used to reproduce the problem.<br/> But when I tried to upload other 2 tarballs(crashdump of the crash and stuck), the transfer stalled despite multiple retries.<br/> Is there another way to upload those files to you?</p> <p>Thanks<br/> Dongyang</p> <p>Another way would be to upload it to dropbox or google drive and share it to me. Please encrypt it if the crash dump has sensitive data and email me the password.</p> <p>We just ran into this during our are large scale testing on titan today.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/14989/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14989/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6271" title="(osc_cache.c:3150:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed:" class="issue-link" data-issue-key="LU-6271"><del>LU-6271</del></a> osc: handle osc eviction correctly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8f01f8b51d114b0d2d54a5ab7db3161782e52447</p> <p>I reckon it is too soon to land <a href="http://review.whamcloud.com/14989/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14989/</a><br/> With the patch the client still crashes when I run the reproducer, for example:<br/> &lt;3&gt;LustreError: 15348:0:(ldlm_resource.c:835:ldlm_resource_complain()) testfs-OST0000-osc-ffff880037fea000: namespace resource <span class="error">&#91;0x302:0x0:0x0&#93;</span>.0 (ffff88003deae500) refcount no<br/> nzero (1) after lock cleanup; forcing cleanup.<br/> &lt;3&gt;LustreError: 15348:0:(ldlm_resource.c:1450:ldlm_resource_dump()) &#8212; Resource: <span class="error">&#91;0x302:0x0:0x0&#93;</span>.0 (ffff88003deae500) refcount = 2<br/> &lt;3&gt;LustreError: 15348:0:(ldlm_resource.c:1453:ldlm_resource_dump()) Granted locks (in reverse order):<br/> &lt;3&gt;LustreError: 15348:0:(ldlm_resource.c:1456:ldlm_resource_dump()) ### ### ns: testfs-OST0000-osc-ffff880037fea000 lock: ffff880037e29940/0xaa5a0b4e389da92d lrc: 4/0,1 mode: <br/> PW/PW res: <span class="error">&#91;0x302:0x0:0x0&#93;</span>.0 rrc: 2 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0-&gt;1048575) flags: 0x126400000000 nid: local remote: 0x59a587f3f1d1130e expref: -99 pid: 15181 tim<br/> eout: 0 lvb_type: 1<br/> &lt;3&gt;LustreError: 15348:0:(ldlm_resource.c:1450:ldlm_resource_dump()) &#8212; Resource: <span class="error">&#91;0x302:0x0:0x0&#93;</span>.0 (ffff88003deae500) refcount = 2<br/> &lt;3&gt;LustreError: 15348:0:(ldlm_resource.c:1453:ldlm_resource_dump()) Granted locks (in reverse order):<br/> &lt;3&gt;LustreError: 12670:0:(osc_cache.c:3141:discard_cb()) page@ffff880038c3fa00<span class="error">&#91;3 ffff88003ac29b38 1 0 1 ffff88003a78ebb8 (null)&#93;</span><br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 12670:0:(osc_cache.c:3141:discard_cb()) vvp-page@ffff880038c3fa68(0:0) vm@ffffea000071dfb8 2000000000087f 2:0 ffff880038c3fa00 21504 lru<br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 12670:0:(osc_cache.c:3141:discard_cb()) lov-page@ffff880038c3faa8, raid0<br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 12670:0:(osc_cache.c:3141:discard_cb()) osc-page@ffff880038c3fb10 21504: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 88080384 0 4096 0x0 0x520 | (null) ffff88003ac58500 ffff880<br/> 03cfdbe60 &gt; 3&lt; 1 0 0 &gt; 4&lt; 0 10 8 18446744073619427328 + | + - + - &gt; 5&lt; + - + - | 0 - | 6663 - -&gt;<br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 12670:0:(osc_cache.c:3141:discard_cb()) end page@ffff880038c3fa00<br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 12670:0:(osc_cache.c:3141:discard_cb()) discard dirty page?<br/> &lt;3&gt;LustreError: 12670:0:(osc_cache.c:2454:osc_teardown_async_page()) extent ffff88003da0eb10@</p> {[21504 -&gt; 21759/21759], [2|0|-|cache|wi|ffff88003cfdbe60], [1048576|256|+|-|ffff8 8003c75d680|256|(null)]} <p> trunc at 21504.<br/> &lt;3&gt;LustreError: 12670:0:(osc_cache.c:2454:osc_teardown_async_page()) ### extent: ffff88003da0eb10<br/> &lt;3&gt; ns: testfs-OST0000-osc-ffff880037fea000 lock: ffff88003c75d680/0xaa5a0b4e389daa3e lrc: 68/0,6 mode: PW/PW res: <span class="error">&#91;0x302:0x0:0x0&#93;</span>.0 rrc: 2 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span><br/> (req 70254592-&gt;71303167) flags: 0x20000000000 nid: local remote: 0x59a587f3f1d11323 expref: -99 pid: 15246 timeout: 0 lvb_type: 1<br/> &lt;3&gt;LustreError: 12670:0:(osc_page.c:307:osc_page_delete()) page@ffff880038c3fa00<span class="error">&#91;3 ffff88003ac29b38 4 0 1 (null) (null)&#93;</span><br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 12670:0:(osc_page.c:307:osc_page_delete()) vvp-page@ffff880038c3fa68(0:0) vm@ffffea000071dfb8 2000000000087f 2:0 ffff880038c3fa00 21504 lru<br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 12670:0:(osc_page.c:307:osc_page_delete()) lov-page@ffff880038c3faa8, raid0<br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 12670:0:(osc_page.c:307:osc_page_delete()) osc-page@ffff880038c3fb10 21504: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 88080384 0 4096 0x0 0x520 | (null) ffff88003ac58500 ffff<br/> 88003cfdbe60 &gt; 3&lt; 0 0 0 &gt; 4&lt; 0 9 8 18446744073619361792 + | + - + - &gt; 5&lt; + - + - | 0 - | 6679 - -&gt;<br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 12670:0:(osc_page.c:307:osc_page_delete()) end page@ffff880038c3fa00<br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 12670:0:(osc_page.c:307:osc_page_delete()) Trying to teardown failed: -16<br/> &lt;0&gt;LustreError: 12670:0:(osc_page.c:308:osc_page_delete()) ASSERTION( 0 ) failed: <br/> &lt;0&gt;LustreError: 12670:0:(osc_page.c:308:osc_page_delete()) LBUG<br/> &lt;4&gt;Pid: 12670, comm: ldlm_bl_00<br/> &lt;4&gt;<br/> &lt;4&gt;Call Trace:<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa021d875&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa021de77&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa090dbfe&gt;&#93;</span> osc_page_delete+0x46e/0x4e0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa037e9dd&gt;&#93;</span> cl_page_delete0+0x7d/0x210 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa037ebad&gt;&#93;</span> cl_page_delete+0x3d/0x110 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0837d0d&gt;&#93;</span> ll_invalidatepage+0x8d/0x160 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0846da5&gt;&#93;</span> vvp_page_discard+0xc5/0x160 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa037cfd8&gt;&#93;</span> cl_page_invoid+0x68/0x160 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa037d0e3&gt;&#93;</span> cl_page_discard+0x13/0x20 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0919678&gt;&#93;</span> discard_cb+0x88/0x1e0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa091946e&gt;&#93;</span> osc_page_gang_lookup+0x1ae/0x330 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa09195f0&gt;&#93;</span> ? discard_cb+0x0/0x1e0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0919914&gt;&#93;</span> osc_lock_discard_pages+0x144/0x240 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa09195f0&gt;&#93;</span> ? discard_cb+0x0/0x1e0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa090ff7b&gt;&#93;</span> osc_lock_flush+0x8b/0x260 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa09103f8&gt;&#93;</span> osc_ldlm_blocking_ast+0x2a8/0x3c0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa050e9dc&gt;&#93;</span> ldlm_cancel_callback+0x6c/0x170 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa052190a&gt;&#93;</span> ldlm_cli_cancel_local+0x8a/0x470 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0526540&gt;&#93;</span> ldlm_cli_cancel+0x60/0x360 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa091022b&gt;&#93;</span> osc_ldlm_blocking_ast+0xdb/0x3c0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa052a440&gt;&#93;</span> ldlm_handle_bl_callback+0x130/0x400 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa052af24&gt;&#93;</span> ldlm_bl_thread_main+0x484/0x700 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810672b0&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa052aaa0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x700 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810a101e&gt;&#93;</span> kthread+0x9e/0xc0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c28a&gt;&#93;</span> child_rip+0xa/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810a0f80&gt;&#93;</span> ? kthread+0x0/0xc0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c280&gt;&#93;</span> ? child_rip+0x0/0x20<br/> &lt;4&gt;<br/> &lt;0&gt;Kernel panic - not syncing: LBUG<br/> &lt;4&gt;Pid: 12670, comm: ldlm_bl_00 Not tainted 2.6.32-573.3.1.el6.x86_64 #1<br/> &lt;4&gt;Call Trace:<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff81537c54&gt;&#93;</span> ? panic+0xa7/0x16f<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa021decb&gt;&#93;</span> ? lbug_with_loc+0x9b/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa090dbfe&gt;&#93;</span> ? osc_page_delete+0x46e/0x4e0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa037e9dd&gt;&#93;</span> ? cl_page_delete0+0x7d/0x210 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa037ebad&gt;&#93;</span> ? cl_page_delete+0x3d/0x110 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0837d0d&gt;&#93;</span> ? ll_invalidatepage+0x8d/0x160 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0846da5&gt;&#93;</span> ? vvp_page_discard+0xc5/0x160 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa037cfd8&gt;&#93;</span> ? cl_page_invoid+0x68/0x160 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa037d0e3&gt;&#93;</span> ? cl_page_discard+0x13/0x20 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0919678&gt;&#93;</span> ? discard_cb+0x88/0x1e0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa091946e&gt;&#93;</span> ? osc_page_gang_lookup+0x1ae/0x330 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa09195f0&gt;&#93;</span> ? discard_cb+0x0/0x1e0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0919914&gt;&#93;</span> ? osc_lock_discard_pages+0x144/0x240 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa09195f0&gt;&#93;</span> ? discard_cb+0x0/0x1e0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa090ff7b&gt;&#93;</span> ? osc_lock_flush+0x8b/0x260 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa09103f8&gt;&#93;</span> ? osc_ldlm_blocking_ast+0x2a8/0x3c0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa050e9dc&gt;&#93;</span> ? ldlm_cancel_callback+0x6c/0x170 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa052190a&gt;&#93;</span> ? ldlm_cli_cancel_local+0x8a/0x470 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0526540&gt;&#93;</span> ? ldlm_cli_cancel+0x60/0x360 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa091022b&gt;&#93;</span> ? osc_ldlm_blocking_ast+0xdb/0x3c0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa052a440&gt;&#93;</span> ? ldlm_handle_bl_callback+0x130/0x400 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa052af24&gt;&#93;</span> ? ldlm_bl_thread_main+0x484/0x700 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810672b0&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa052aaa0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x700 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810a101e&gt;&#93;</span> ? kthread+0x9e/0xc0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c28a&gt;&#93;</span> ? child_rip+0xa/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810a0f80&gt;&#93;</span> ? kthread+0x0/0xc0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c280&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>and another one, however don't know if it's related.<br/> &lt;3&gt;LustreError: 4278:0:(ldlm_resource.c:835:ldlm_resource_complain()) testfs-OST0001-osc-ffff88003c715c00: namespace resource <span class="error">&#91;0x302:0x0:0x0&#93;</span>.0 (ffff880037015800) refcount non<br/> zero (1) after lock cleanup; forcing cleanup.<br/> &lt;3&gt;LustreError: 4278:0:(ldlm_resource.c:1450:ldlm_resource_dump()) &#8212; Resource: <span class="error">&#91;0x302:0x0:0x0&#93;</span>.0 (ffff880037015800) refcount = 2<br/> &lt;3&gt;LustreError: 4278:0:(ldlm_resource.c:1453:ldlm_resource_dump()) Granted locks (in reverse order):<br/> &lt;3&gt;LustreError: 4278:0:(ldlm_resource.c:1456:ldlm_resource_dump()) ### ### ns: testfs-OST0001-osc-ffff88003c715c00 lock: ffff880037269380/0x532f2161a177e283 lrc: 19/0,1 mode: <br/> PW/PW res: <span class="error">&#91;0x302:0x0:0x0&#93;</span>.0 rrc: 2 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0-&gt;1048575) flags: 0x126400000000 nid: local remote: 0x59a587f3f1d100b5 expref: -99 pid: 4040 time<br/> out: 0 lvb_type: 1<br/> &lt;3&gt;LustreError: 790:0:(osc_io.c:1010:osc_req_attr_set()) page@ffff880022f02600<span class="error">&#91;2 ffff88003be7db38 2 0 1 (null) ffff88003df849c0&#93;</span><br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 789:0:(osc_io.c:1010:osc_req_attr_set()) page@ffff88003dfc8000<span class="error">&#91;2 ffff88003be7db38 2 0 1 (null) ffff88003b3982c0&#93;</span><br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 789:0:(osc_io.c:1010:osc_req_attr_set()) vvp-page@ffff88003dfc8068(0:0) vm@ffffea0000785428 2000000000282c 2:0 ffff88003dfc8000 19200 lru<br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 789:0:(osc_io.c:1010:osc_req_attr_set()) lov-page@ffff88003dfc80a8, raid0<br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 789:0:(osc_io.c:1010:osc_req_attr_set()) osc-page@ffff88003dfc8110 19200: 1&lt; 0x845fed 258 0 + + &gt; 2&lt; 78643200 0 4096 0x5 0x520 | (null) ffff88003d970540 ffff88<br/> 003bc81e20 &gt; 3&lt; 1 12 0 &gt; 4&lt; 0 7 8 18446744073678077952 - | - - - - &gt; 5&lt; - - - - | 0 - | 0 - -&gt;<br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 789:0:(osc_io.c:1010:osc_req_attr_set()) end page@ffff88003dfc8000<br/> &lt;3&gt;<br/> &lt;3&gt;LustreError: 789:0:(osc_io.c:1010:osc_req_attr_set()) uncovered page!<br/> &lt;3&gt;LustreError: 789:0:(ldlm_resource.c:1450:ldlm_resource_dump()) &#8212; Resource: <span class="error">&#91;0x302:0x0:0x0&#93;</span>.0 (ffff880037015800) refcount = 3<br/> &lt;4&gt;Pid: 789, comm: ptlrpcd_00_00<br/> &lt;4&gt;<br/> &lt;4&gt;Call Trace:<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa021d875&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa09133fa&gt;&#93;</span> osc_req_attr_set+0x55a/0x720 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0383829&gt;&#93;</span> cl_req_attr_set+0xc9/0x220 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0904082&gt;&#93;</span> osc_build_rpc+0x882/0x12d0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa091f623&gt;&#93;</span> osc_io_unplug0+0x1133/0x1af0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0918428&gt;&#93;</span> ? osc_ap_completion+0x1a8/0x550 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0917a3e&gt;&#93;</span> ? osc_extent_put+0xbe/0x260 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0374f75&gt;&#93;</span> ? lu_object_put+0x135/0x3b0 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa09224b0&gt;&#93;</span> osc_io_unplug+0x10/0x20 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0905593&gt;&#93;</span> brw_interpret+0xac3/0x2320 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0546ee2&gt;&#93;</span> ? ptlrpc_unregister_bulk+0xa2/0xac0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa053f4bc&gt;&#93;</span> ? ptlrpc_unregister_reply+0x6c/0x810 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa053e2a4&gt;&#93;</span> ? ptlrpc_send_new_req+0x154/0x980 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0540551&gt;&#93;</span> ptlrpc_check_set+0x331/0x1be0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa056e443&gt;&#93;</span> ptlrpcd_check+0x3d3/0x610 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa056e8fa&gt;&#93;</span> ptlrpcd+0x27a/0x500 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810672b0&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa056e680&gt;&#93;</span> ? ptlrpcd+0x0/0x500 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810a101e&gt;&#93;</span> kthread+0x9e/0xc0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c28a&gt;&#93;</span> child_rip+0xa/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810a0f80&gt;&#93;</span> ? kthread+0x0/0xc0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c280&gt;&#93;</span> ? child_rip+0x0/0x20<br/> &lt;4&gt;<br/> &lt;0&gt;LustreError: 789:0:(osc_io.c:1020:osc_req_attr_set()) LBUG<br/> &lt;4&gt;Pid: 789, comm: ptlrpcd_00_00<br/> &lt;4&gt;<br/> &lt;4&gt;Call Trace:<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa021d875&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa021de77&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0913410&gt;&#93;</span> osc_req_attr_set+0x570/0x720 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0383829&gt;&#93;</span> cl_req_attr_set+0xc9/0x220 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0904082&gt;&#93;</span> osc_build_rpc+0x882/0x12d0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa091f623&gt;&#93;</span> osc_io_unplug0+0x1133/0x1af0 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0918428&gt;&#93;</span> ? osc_ap_completion+0x1a8/0x550 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0917a3e&gt;&#93;</span> ? osc_extent_put+0xbe/0x260 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0374f75&gt;&#93;</span> ? lu_object_put+0x135/0x3b0 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa09224b0&gt;&#93;</span> osc_io_unplug+0x10/0x20 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0905593&gt;&#93;</span> brw_interpret+0xac3/0x2320 <span class="error">&#91;osc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0546ee2&gt;&#93;</span> ? ptlrpc_unregister_bulk+0xa2/0xac0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa053f4bc&gt;&#93;</span> ? ptlrpc_unregister_reply+0x6c/0x810 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa053e2a4&gt;&#93;</span> ? ptlrpc_send_new_req+0x154/0x980 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa0540551&gt;&#93;</span> ptlrpc_check_set+0x331/0x1be0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa056e443&gt;&#93;</span> ptlrpcd_check+0x3d3/0x610 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa056e8fa&gt;&#93;</span> ptlrpcd+0x27a/0x500 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810672b0&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffffa056e680&gt;&#93;</span> ? ptlrpcd+0x0/0x500 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810a101e&gt;&#93;</span> kthread+0x9e/0xc0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c28a&gt;&#93;</span> child_rip+0xa/0x20<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff810a0f80&gt;&#93;</span> ? kthread+0x0/0xc0<br/> &lt;4&gt; <span class="error">&#91;&lt;ffffffff8100c280&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16456" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16456</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6271" title="(osc_cache.c:3150:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed:" class="issue-link" data-issue-key="LU-6271"><del>LU-6271</del></a> osc: further OSC cleanup after eviction<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9dd22234a098bb2bea26c4694d91edc928e027ac</p> <p>Hi Dongyang,</p> <p>Please try patch 16456 and see if it can fix the problem.</p> <p>Jinshan</p> <p>Hi Jinshan,<br/> I'm using master which has 14989 landed already, plus patch set 3 of 16456:<br/> The client crashed 5mins after I started the reproducer:<br/> Sep 18 13:29:00 client kernel: LustreError: 16824:0:(ldlm_resource.c:887:ldlm_resource_complain()) testfs-OST0000-osc-ffff8800375c4000: namespace resource <span class="error">&#91;0x303:0x0:0x0&#93;</span>.0 (ffff88003d2abcc0) refcount nonzero (1) after lock cleanup; forcing cleanup.<br/> Sep 18 13:29:00 client kernel: LustreError: 16824:0:(ldlm_resource.c:1502:ldlm_resource_dump()) &#8212; Resource: <span class="error">&#91;0x303:0x0:0x0&#93;</span>.0 (ffff88003d2abcc0) refcount = 3<br/> Sep 18 13:29:00 client kernel: LustreError: 16824:0:(ldlm_resource.c:1505:ldlm_resource_dump()) Granted locks (in reverse order):<br/> Sep 18 13:29:00 client kernel: LustreError: 16824:0:(ldlm_resource.c:1508:ldlm_resource_dump()) ### ### ns: testfs-OST0000-osc-ffff8800375c4000 lock: ffff88000b69d380/0x2104da78ff029aa9 lrc: 8/0,1 mode: PW/PW res: <span class="error">&#91;0x303:0x0:0x0&#93;</span>.0 rrc: 3 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0-&gt;1048575) flags: 0x126400000000 nid: local remote: 0x59a587f3f1d1d697 expref: -99 pid: 16588 timeout: 0 lvb_type: 1<br/> Sep 18 13:29:00 client kernel: LustreError: 16824:0:(ldlm_resource.c:1502:ldlm_resource_dump()) &#8212; Resource: <span class="error">&#91;0x303:0x0:0x0&#93;</span>.0 (ffff88003d2abcc0) refcount = 2<br/> Sep 18 13:29:00 client kernel: LustreError: 16824:0:(ldlm_resource.c:1505:ldlm_resource_dump()) Granted locks (in reverse order):<br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_cache.c:3134:discard_cb()) page@ffff88003d5a7600<span class="error">&#91;3 ffff88003a513b38 1 0 1 ffff88003a5d6b78&#93;</span><br/> Sep 18 13:29:01 client kernel: <br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_cache.c:3134:discard_cb()) vvp-page@ffff88003d5a7650(0:0) vm@ffffea00004439e0 2000000000087d 3:0 ffff88003d5a7600 25088 lru<br/> Sep 18 13:29:01 client kernel: <br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_cache.c:3134:discard_cb()) lov-page@ffff88003d5a7690, raid0<br/> Sep 18 13:29:01 client kernel: <br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_cache.c:3134:discard_cb()) osc-page@ffff88003d5a76f8 25088: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 102760448 0 4096 0x0 0x520 | (null) ffff88003c479500 ffff88003a517e20 &gt; 3&lt; 1 0 0 &gt; 4&lt; 0 9 8 18446744073642434560 + | + - + - &gt; 5&lt; + - + - | 0 - | 5635 - -&gt;<br/> Sep 18 13:29:01 client kernel: <br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_cache.c:3134:discard_cb()) end page@ffff88003d5a7600<br/> Sep 18 13:29:01 client kernel: <br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_cache.c:3134:discard_cb()) discard dirty page?<br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_cache.c:2447:osc_teardown_async_page()) extent ffff880039a18720@</p> {[25088 -&gt; 25341/25343], [2|0|+|locking|wi|ffff88003a517e20], [1040384|254|+|-|ffff88000b69d580|256|ffff8800372b3520]} <p> trunc at 25088.<br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_cache.c:2447:osc_teardown_async_page()) ### extent: ffff880039a18720<br/> Sep 18 13:29:01 client kernel: ns: testfs-OST0000-osc-ffff8800375c4000 lock: ffff88000b69d580/0x2104da78ff029c0e lrc: 41/0,3 mode: PW/PW res: <span class="error">&#91;0x303:0x0:0x0&#93;</span>.0 rrc: 2 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 102760448-&gt;103809023) flags: 0x20000000000 nid: local remote: 0x59a587f3f1d1d6ac expref: -99 pid: 16683 timeout: 0 lvb_type: 1<br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_page.c:307:osc_page_delete()) page@ffff88003d5a7600<span class="error">&#91;3 ffff88003a513b38 4 0 1 (null)&#93;</span><br/> Sep 18 13:29:01 client kernel: <br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_page.c:307:osc_page_delete()) vvp-page@ffff88003d5a7650(0:0) vm@ffffea00004439e0 2000000000087d 3:0 ffff88003d5a7600 25088 lru<br/> Sep 18 13:29:01 client kernel: <br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_page.c:307:osc_page_delete()) lov-page@ffff88003d5a7690, raid0<br/> Sep 18 13:29:01 client kernel: <br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_page.c:307:osc_page_delete()) osc-page@ffff88003d5a76f8 25088: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 102760448 0 4096 0x0 0x520 | (null) ffff88003c479500 ffff88003a517e20 &gt; 3&lt; 0 0 0 &gt; 4&lt; 0 9 8 18446744073642434560 + | + - + - &gt; 5&lt; + - + - | 0 - | 5635 - -&gt;<br/> Sep 18 13:29:01 client kernel: <br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_page.c:307:osc_page_delete()) end page@ffff88003d5a7600<br/> Sep 18 13:29:01 client kernel: <br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_page.c:307:osc_page_delete()) Trying to teardown failed: -16<br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_page.c:308:osc_page_delete()) ASSERTION( 0 ) failed: </p> <p>Message from syslogd@client at Sep 18 13:29:01 ...<br/> kernel:LustreError: 13392:0:(osc_page.c:308:osc_page_delete()) ASSERTION( 0 ) failed: <br/> Sep 18 13:29:01 client kernel: LustreError: 13392:0:(osc_page.c:308:osc_page_delete()) LBUG<br/> Sep 18 13:29:01 client kernel: Pid: 13392, comm: ldlm_bl_00<br/> Sep 18 13:29:01 client kernel: <br/> Sep 18 13:29:01 client kernel: Call Trace:<br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa10ad875&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa10ade77&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa178bf9e&gt;&#93;</span> osc_page_delete+0x46e/0x4e0 <span class="error">&#91;osc&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa12019cd&gt;&#93;</span> cl_page_delete0+0x7d/0x210 <span class="error">&#91;obdclass&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa1201b9d&gt;&#93;</span> cl_page_delete+0x3d/0x110 <span class="error">&#91;obdclass&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa16b6d2d&gt;&#93;</span> ll_invalidatepage+0x8d/0x160 <span class="error">&#91;lustre&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa16c5d85&gt;&#93;</span> vvp_page_discard+0xc5/0x160 <span class="error">&#91;lustre&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa11fffc8&gt;&#93;</span> cl_page_invoid+0x68/0x160 <span class="error">&#91;obdclass&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa12000d3&gt;&#93;</span> cl_page_discard+0x13/0x20 <span class="error">&#91;obdclass&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa1797158&gt;&#93;</span> discard_cb+0x88/0x1e0 <span class="error">&#91;osc&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa1796f4e&gt;&#93;</span> osc_page_gang_lookup+0x1ae/0x330 <span class="error">&#91;osc&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa17970d0&gt;&#93;</span> ? discard_cb+0x0/0x1e0 <span class="error">&#91;osc&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa17973f4&gt;&#93;</span> osc_lock_discard_pages+0x144/0x240 <span class="error">&#91;osc&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa17970d0&gt;&#93;</span> ? discard_cb+0x0/0x1e0 <span class="error">&#91;osc&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa178e45b&gt;&#93;</span> osc_lock_flush+0x8b/0x260 <span class="error">&#91;osc&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa178e8d8&gt;&#93;</span> osc_ldlm_blocking_ast+0x2a8/0x3c0 <span class="error">&#91;osc&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa138ea57&gt;&#93;</span> ldlm_cancel_callback+0x87/0x280 <span class="error">&#91;ptlrpc&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffff81060530&gt;&#93;</span> ? __dequeue_entity+0x30/0x50<br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffff8100969d&gt;&#93;</span> ? __switch_to+0x7d/0x340<br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa13a184a&gt;&#93;</span> ldlm_cli_cancel_local+0x8a/0x470 <span class="error">&#91;ptlrpc&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa13a6480&gt;&#93;</span> ldlm_cli_cancel+0x60/0x360 <span class="error">&#91;ptlrpc&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa178e70b&gt;&#93;</span> osc_ldlm_blocking_ast+0xdb/0x3c0 <span class="error">&#91;osc&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa13aa380&gt;&#93;</span> ldlm_handle_bl_callback+0x130/0x400 <span class="error">&#91;ptlrpc&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa13aae64&gt;&#93;</span> ldlm_bl_thread_main+0x484/0x700 <span class="error">&#91;ptlrpc&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffff810672b0&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffffa13aa9e0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x700 <span class="error">&#91;ptlrpc&#93;</span><br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffff810a101e&gt;&#93;</span> kthread+0x9e/0xc0<br/> Sep 18 13:29:01 client kernel: <span class="error">&#91;&lt;ffffffff8100c28a&gt;&#93;</span> child_rip+0xa/0x20</p> <p>Hi Dongyang,</p> <p>it certainly lasted longer on my VM node <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Please upload your reproducer and I will restart working on this issue once I get access to a real hardware.</p> <p>Thanks,</p> <p>Hi Jinshan,<br/> Hardware shouldn't matter as I'm reproducing it on my VMs.<br/> Here are the scripts I used.<br/> On client, I'm using IOR and when IOR dies because of eviction, just bring up new ones:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#!/bin/sh <span class="code-keyword">while</span> <span class="code-keyword">true</span> <span class="code-keyword">do</span> num=$(ps aux | grep IOR | wc -l) <span class="code-keyword">while</span> [ $num -lt 20 ] <span class="code-keyword">do</span> /root/IOR/src/C/IOR -b 8g -w -e -E -t 1m -v -k -o /mnt/testfile &amp; num=$(ps aux | grep IOR | wc -l) done sleep 1 done </pre> </div></div> <p>Here I got 20 of them running at the same time.</p> <p>On the OSS:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#!/bin/sh <span class="code-keyword">while</span> <span class="code-keyword">true</span> <span class="code-keyword">do</span> <span class="code-keyword">for</span> ost in /proc/fs/lustre/obdfilter/* <span class="code-keyword">do</span> <span class="code-keyword">if</span> [ -n $(cat $ost/exports/&lt;your client nid&gt;/uuid) ] then echo $(cat $ost/exports/&lt;your client nid&gt;/uuid) &gt; $ost/evict_client fi done sleep 10 done </pre> </div></div> <p>As you can see I'm evicting the client over and over, and I'm using 10secs as the interval. Again you can change this.</p> <p>Thanks<br/> Dongyang</p> <p>I did very similar thing with iozone but they wrote to different files, let me try your reproducer.</p> <p>Please check the new patch and see if it can fix the problem.</p> <p>OK I've been running the scripts with patchset 5 of 16456 for some time, seems the general IO is fine.<br/> But when I tried the group lock, the client crashed immediately after eviction:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">&lt;0&gt;LustreError: 1697:0:(osc_cache.c:2907:osc_cache_writeback_range()) ASSERTION( !ext-&gt;oe_hp ) failed: &lt;0&gt;LustreError: 1697:0:(osc_cache.c:2907:osc_cache_writeback_range()) LBUG &lt;4&gt;Pid: 1697, comm: ldlm_bl_11 &lt;4&gt; &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffffa021d875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt; [&lt;ffffffffa021de77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt; [&lt;ffffffffa0922085&gt;] osc_cache_writeback_range+0x1275/0x1280 [osc] &lt;4&gt; [&lt;ffffffffa090b545&gt;] osc_lock_flush+0x175/0x260 [osc] &lt;4&gt; [&lt;ffffffffa090b8d8&gt;] osc_ldlm_blocking_ast+0x2a8/0x3c0 [osc] &lt;4&gt; [&lt;ffffffffa050ba57&gt;] ldlm_cancel_callback+0x87/0x280 [ptlrpc] &lt;4&gt; [&lt;ffffffff81060530&gt;] ? __dequeue_entity+0x30/0x50 &lt;4&gt; [&lt;ffffffff8100969d&gt;] ? __switch_to+0x7d/0x340 &lt;4&gt; [&lt;ffffffffa051e84a&gt;] ldlm_cli_cancel_local+0x8a/0x470 [ptlrpc] &lt;4&gt; [&lt;ffffffffa05234bc&gt;] ldlm_cli_cancel+0x9c/0x3e0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa090b70b&gt;] osc_ldlm_blocking_ast+0xdb/0x3c0 [osc] &lt;4&gt; [&lt;ffffffff810672c2&gt;] ? default_wake_function+0x12/0x20 &lt;4&gt; [&lt;ffffffffa0527400&gt;] ldlm_handle_bl_callback+0x130/0x400 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0527ee4&gt;] ldlm_bl_thread_main+0x484/0x700 [ptlrpc] &lt;4&gt; [&lt;ffffffff810672b0&gt;] ? default_wake_function+0x0/0x20 &lt;4&gt; [&lt;ffffffffa0527a60&gt;] ? ldlm_bl_thread_main+0x0/0x700 [ptlrpc] &lt;4&gt; [&lt;ffffffff810a101e&gt;] kthread+0x9e/0xc0 &lt;4&gt; [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 &lt;4&gt; [&lt;ffffffff810a0f80&gt;] ? kthread+0x0/0xc0 &lt;4&gt; [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 &lt;4&gt; &lt;0&gt;Kernel panic - not syncing: LBUG &lt;4&gt;Pid: 1697, comm: ldlm_bl_11 Not tainted 2.6.32-573.3.1.el6.x86_64 #1 &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffff81537c54&gt;] ? panic+0xa7/0x16f &lt;4&gt; [&lt;ffffffffa021decb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] &lt;4&gt; [&lt;ffffffffa0922085&gt;] ? osc_cache_writeback_range+0x1275/0x1280 [osc] &lt;4&gt; [&lt;ffffffffa090b545&gt;] ? osc_lock_flush+0x175/0x260 [osc] &lt;4&gt; [&lt;ffffffffa090b8d8&gt;] ? osc_ldlm_blocking_ast+0x2a8/0x3c0 [osc] &lt;4&gt; [&lt;ffffffffa050ba57&gt;] ? ldlm_cancel_callback+0x87/0x280 [ptlrpc] &lt;4&gt; [&lt;ffffffff81060530&gt;] ? __dequeue_entity+0x30/0x50 &lt;4&gt; [&lt;ffffffff8100969d&gt;] ? __switch_to+0x7d/0x340 &lt;4&gt; [&lt;ffffffffa051e84a&gt;] ? ldlm_cli_cancel_local+0x8a/0x470 [ptlrpc] &lt;4&gt; [&lt;ffffffffa05234bc&gt;] ? ldlm_cli_cancel+0x9c/0x3e0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa090b70b&gt;] ? osc_ldlm_blocking_ast+0xdb/0x3c0 [osc] &lt;4&gt; [&lt;ffffffff810672c2&gt;] ? default_wake_function+0x12/0x20 &lt;4&gt; [&lt;ffffffffa0527400&gt;] ? ldlm_handle_bl_callback+0x130/0x400 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0527ee4&gt;] ? ldlm_bl_thread_main+0x484/0x700 [ptlrpc] &lt;4&gt; [&lt;ffffffff810672b0&gt;] ? default_wake_function+0x0/0x20 &lt;4&gt; [&lt;ffffffffa0527a60&gt;] ? ldlm_bl_thread_main+0x0/0x700 [ptlrpc] &lt;4&gt; [&lt;ffffffff810a101e&gt;] ? kthread+0x9e/0xc0 &lt;4&gt; [&lt;ffffffff8100c28a&gt;] ? child_rip+0xa/0x20 &lt;4&gt; [&lt;ffffffff810a0f80&gt;] ? kthread+0x0/0xc0 &lt;4&gt; [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Hi Dongyang, please upload your test cases.</p> <p>Hi Jinshan,<br/> See my comment earlier on 28/Jul/15 8:39 PM.</p> <p>Thanks. Please check if the latest patch can fix the problem</p> <p>With patch set 6 pf 16456, the client won't crash with the gplck reproducer and single eviction.<br/> However, the client crashes if:<br/> 1. run the gplck reproducer, evict the client once.<br/> 2. after client reconnects, don't kill the gplck, run another instance of gplck with a different gid.<br/> 3. evict the client again, the client crashes with the same stack trace ASSERTION( !ext-&gt;oe_hp ) failed</p> <p>This makes me think that the problem should be fixed in the ldlm layer like <a href="http://review.whamcloud.com/#/c/14915/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14915/</a><br/> not in the osc layer.<br/> Thanks<br/> Dongyang</p> <p>Hi Dongyang, the problem may be fixed in patch set 7 but feel free to use patch 14915 on your own site.</p> <p>this problem can only happen at eviction so even patch 14989 is good enough for daily use.</p> <p>Hi Dongyang, I took a look at your latest patch 14915 and I suggest you shouldn't apply this patch to your own site because it's not production ready yet.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16456/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16456/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6271" title="(osc_cache.c:3150:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed:" class="issue-link" data-issue-key="LU-6271"><del>LU-6271</del></a> osc: further OSC cleanup after eviction<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e8b421531c166b91ab5c1f417570c544bcdd050c</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16727" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16727</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6271" title="(osc_cache.c:3150:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed:" class="issue-link" data-issue-key="LU-6271"><del>LU-6271</del></a> osc: check oo_npages inside spin lock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e220fbbfa93069a783430d15a26dc3518006bc1c</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16727/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16727/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6271" title="(osc_cache.c:3150:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed:" class="issue-link" data-issue-key="LU-6271"><del>LU-6271</del></a> osc: faulty assertion in osc_object_prune()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c5e84d53198142919b526fd304c6782005e38bdb</p> <p>Landed for 2.8</p> <p>This bug is not resolved. At a customer site today, one of the issues reported with the patch series was reproduced:</p> <p>Here is the gathered information from one of the engineers:</p> <p>Nov 13 21:22:32 r2i1n4 kernel: <span class="error">&#91;41261.553520&#93;</span> LustreError: 11-0: work-OST0018-osc-ffff880f4a556c00: Communicating with 10.148.254.146@o2ib, operation obd_ping failed with -107.<br/> Nov 13 21:22:32 r2i1n4 kernel: <span class="error">&#91;41261.553537&#93;</span> Lustre: work-OST0018-osc-ffff880f4a556c00: Connection to work-OST0018 (at 10.148.254.146@o2ib) was lost; in progress operations using this service will wait for recovery to complete<br/> Nov 13 21:22:32 r2i1n4 kernel: <span class="error">&#91;41261.553744&#93;</span> LustreError: 167-0: work-OST0018-osc-ffff880f4a556c00: This client was evicted by work-OST0018; in progress operations using this service will fail.</p> <p>Nov 13 21:22:19 oss3 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 10.148.4.3@o2ib ns: filter-work-OST0018_UUID lock: ffff881bf46fa1c0/0x1b850f2869999e97 lrc: 3/0,0 mode: PW/PW res: <span class="error">&#91;0xcc6a24:0x0:0x0&#93;</span>.0 rrc: 31 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0-&gt;18446744073709551615) flags: 0x60000080010020 nid: 10.148.4.3@o2ib remote: 0x7a3c9c39e0502fcc expref: 5 pid: 20187 timeout: 4335795111 lvb_type: 0</p> <p>r2i1n4:~ # lctl dl<br/> 0 UP mgc MGC10.148.254.141@o2ib 2dc4c54f-dab6-1190-fca7-d7747093f936 5<br/> … <br/> 35 UP osc work-OST0017-osc-ffff880f4a556c00 9d007967-5de8-eb2e-eaf9-99d4b17d74e6 5<br/> 36 IN osc work-OST0018-osc-ffff880f4a556c00 9d007967-5de8-eb2e-eaf9-99d4b17d74e6 5<br/> 37 UP osc work-OST0019-osc-ffff880f4a556c00 9d007967-5de8-eb2e-eaf9-99d4b17d74e6 5</p> <p>r2i1n4:~ # lctl --device 36 activate<br/> error: activate: failed: Invalid argument</p> <p>r2i1n4:~ # cat /proc/fs/lustre/osc/work-OST0018-osc-ffff880f4a556c00/import<br/> import:<br/> name: work-OST0018-osc-ffff880f4a556c00<br/> target: work-OST0018_UUID<br/> state: EVICTED<br/> instance: 21<br/> connect_flags: [ write_grant, server_lock, version, request_portal, truncate_lock, max_byte_per_rpc, early_lock_cancel, adaptive_timeouts, alt_checksum_algorithm, fid_is_enabled, version_recovery, full20, layout_lock, 64bithash, object_max_bytes, jobstats, einprogress, lvb_type]<br/> import_flags: [ invalid, replayable, pingable, connect_tried ]<br/> connection:<br/> failover_nids: [ 10.148.254.146@o2ib, 10.148.254.147@o2ib ]<br/> current_connection: 10.148.254.146@o2ib<br/> connection_attempts: 2<br/> generation: 2<br/> in-progress_invalidations: 1<br/> rpcs:<br/> inflight: 0<br/> unregistering: 0<br/> timeouts: 0<br/> avg_waittime: 803 usec<br/> service_estimates:<br/> services: 1 sec<br/> network: 1 sec<br/> transactions:<br/> last_replay: 0<br/> peer_committed: 90194391109<br/> last_checked: 90194391109<br/> read_data_averages:<br/> bytes_per_rpc: 19550<br/> usec_per_rpc: 938<br/> MB_per_sec: 20.84<br/> write_data_averages:<br/> bytes_per_rpc: 508<br/> usec_per_rpc: 5386<br/> MB_per_sec: 0.09</p> <p>r2i1n4:~ # ps -ef | grep ll_imp_inval<br/> root 832 2 0 Nov13 ? 00:00:00 <span class="error">&#91;ll_imp_inval&#93;</span><br/> root 25849 25765 0 14:18 pts/0 00:00:00 grep ll_imp_inval</p> <p>Nov 16 14:22:19 r2i1n4 kernel: <span class="error">&#91;274907.717827&#93;</span> ll_imp_inval S ffff881ffc253b00 0 832 2 0x00000000<br/> Nov 16 14:22:19 r2i1n4 kernel: <span class="error">&#91;274907.717832&#93;</span> ffff881ffc34dc50 0000000000000046 ffff881ffc34c010 0000000000010900<br/> Nov 16 14:22:19 r2i1n4 kernel: <span class="error">&#91;274907.717838&#93;</span> 0000000000010900 0000000000010900 0000000000010900 ffff881ffc34dfd8<br/> Nov 16 14:22:19 r2i1n4 kernel: <span class="error">&#91;274907.717845&#93;</span> ffff881ffc34dfd8 0000000000010900 ffff881fd7ffa580 ffff881020f2c300<br/> Nov 16 14:22:19 r2i1n4 kernel: <span class="error">&#91;274907.717852&#93;</span> Call Trace:<br/> Nov 16 14:22:19 r2i1n4 kernel: <span class="error">&#91;274907.717879&#93;</span> <span class="error">&#91;&lt;ffffffffa0bcf361&gt;&#93;</span> osc_object_invalidate+0x1c1/0x280 <span class="error">&#91;osc&#93;</span><br/> Nov 16 14:22:19 r2i1n4 kernel: <span class="error">&#91;274907.717907&#93;</span> <span class="error">&#91;&lt;ffffffffa0bb5bae&gt;&#93;</span> osc_ldlm_resource_invalidate+0xae/0x170 <span class="error">&#91;osc&#93;</span><br/> Nov 16 14:22:19 r2i1n4 kernel: <span class="error">&#91;274907.717930&#93;</span> <span class="error">&#91;&lt;ffffffffa0716d68&gt;&#93;</span> cfs_hash_for_each_relax+0x178/0x340 <span class="error">&#91;libcfs&#93;</span><br/> Nov 16 14:22:19 r2i1n4 kernel: <span class="error">&#91;274907.717971&#93;</span> <span class="error">&#91;&lt;ffffffffa0718fc0&gt;&#93;</span> cfs_hash_for_each_nolock+0x70/0x1c0 <span class="error">&#91;libcfs&#93;</span><br/> Nov 16 14:22:19 r2i1n4 kernel: <span class="error">&#91;274907.718008&#93;</span> <span class="error">&#91;&lt;ffffffffa0bc989c&gt;&#93;</span> osc_import_event+0xfac/0x1420 <span class="error">&#91;osc&#93;</span><br/> Nov 16 14:22:19 r2i1n4 kernel: <span class="error">&#91;274907.718064&#93;</span> <span class="error">&#91;&lt;ffffffffa0a88592&gt;&#93;</span> ptlrpc_invalidate_import+0x292/0x890 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 16 14:22:19 r2i1n4 kernel: <span class="error">&#91;274907.718151&#93;</span> <span class="error">&#91;&lt;ffffffffa0a8a702&gt;&#93;</span> ptlrpc_invalidate_import_thread+0x42/0x2e0 <span class="error">&#91;ptlrpc&#93;</span><br/> Nov 16 14:22:19 r2i1n4 kernel: <span class="error">&#91;274907.718197&#93;</span> <span class="error">&#91;&lt;ffffffff810843f6&gt;&#93;</span> kthread+0x96/0xa0<br/> Nov 16 14:22:19 r2i1n4 kernel: <span class="error">&#91;274907.718205&#93;</span> <span class="error">&#91;&lt;ffffffff8146f364&gt;&#93;</span> kernel_thread_helper+0x4/0x10</p> <p>client messages</p> <p>server message</p> <p>uploaded 2 logs file detailing the problem, customer is running the ddn-ieel branch 2.5.39-ddn7.</p> <p>In short, clients were evicted suddenly and the OSTs stayed INACTIVE for those clients until the clients were restarted forcibly. </p> <p>NOTE: the 2.5.39-ddn7 has the patches:</p> <p>20401c7 <a href="https://jira.whamcloud.com/browse/LU-6271" title="(osc_cache.c:3150:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed:" class="issue-link" data-issue-key="LU-6271"><del>LU-6271</del></a> osc: faulty assertion in osc_object_prune()<br/> b08341e <a href="https://jira.whamcloud.com/browse/LU-6271" title="(osc_cache.c:3150:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed:" class="issue-link" data-issue-key="LU-6271"><del>LU-6271</del></a> osc: further OSC cleanup after eviction<br/> 42e9257 <a href="https://jira.whamcloud.com/browse/LU-6271" title="(osc_cache.c:3150:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(page)))) ) failed:" class="issue-link" data-issue-key="LU-6271"><del>LU-6271</del></a> osc: handle osc eviction correctly</p> <p>I think we should seriously consider using patch (written by Dongyang Li):<br/> <a href="http://review.whamcloud.com/#/c/14915/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14915/</a></p> <p>I am going to test the above patch at NCI, where we are able to reproduce the crash and inactive OST (due to client eviction)...</p> <p>From the log, obviously the root cause was a deadlock problem. It has nothing to do with these patches.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Nov 16 14:22:19 r2i1n4 kernel: [274907.716536] shald_dpd.out D ffff88201d872010 0 825 772 0x00000000 Nov 16 14:22:19 r2i1n4 kernel: [274907.716541] ffff88201d873c30 0000000000000082 ffff88201d872010 0000000000010900 Nov 16 14:22:19 r2i1n4 kernel: [274907.716548] 0000000000010900 0000000000010900 0000000000010900 ffff88201d873fd8 Nov 16 14:22:19 r2i1n4 kernel: [274907.716554] ffff88201d873fd8 0000000000010900 ffff881fbd5ac440 ffff881020932600 Nov 16 14:22:19 r2i1n4 kernel: [274907.716561] Call Trace: Nov 16 14:22:19 r2i1n4 kernel: [274907.716568] [&lt;ffffffff814651a8&gt;] __mutex_lock_slowpath+0xf8/0x150 Nov 16 14:22:19 r2i1n4 kernel: [274907.716575] [&lt;ffffffff81464c3a&gt;] mutex_lock+0x1a/0x40 Nov 16 14:22:19 r2i1n4 kernel: [274907.716585] [&lt;ffffffff810fd10a&gt;] generic_file_aio_write+0x3a/0xb0 Nov 16 14:22:19 r2i1n4 kernel: [274907.716622] [&lt;ffffffffa0d51e2e&gt;] vvp_io_write_start+0x12e/0x530 [lustre] Nov 16 14:22:19 r2i1n4 kernel: [274907.716707] [&lt;ffffffffa088d539&gt;] cl_io_start+0x69/0x140 [obdclass] Nov 16 14:22:19 r2i1n4 kernel: [274907.716790] [&lt;ffffffffa0890993&gt;] cl_io_loop+0xa3/0x190 [obdclass] Nov 16 14:22:19 r2i1n4 kernel: [274907.716860] [&lt;ffffffffa0cf0cc7&gt;] ll_file_io_generic+0x757/0x830 [lustre] Nov 16 14:22:19 r2i1n4 kernel: [274907.716892] [&lt;ffffffffa0d01e14&gt;] ll_file_aio_write+0x1b4/0x5d0 [lustre] Nov 16 14:22:19 r2i1n4 kernel: [274907.716926] [&lt;ffffffffa0d02415&gt;] ll_file_write+0x1e5/0x270 [lustre] Nov 16 14:22:19 r2i1n4 kernel: [274907.716942] [&lt;ffffffff8115ed4e&gt;] vfs_write+0xce/0x140 Nov 16 14:22:19 r2i1n4 kernel: [274907.716951] [&lt;ffffffff8115eec3&gt;] sys_write+0x53/0xa0 Nov 16 14:22:19 r2i1n4 kernel: [274907.716958] [&lt;ffffffff8146e1f2&gt;] system_call_fastpath+0x16/0x1b Nov 16 14:22:19 r2i1n4 kernel: [274907.716971] [&lt;00007fffec69579d&gt;] 0x7fffec69579c </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Nov 16 14:22:19 r2i1n4 kernel: [274907.717055] shald_dpd.out D ffff88201dc04010 0 826 772 0x00000000 Nov 16 14:22:19 r2i1n4 kernel: [274907.717060] ffff88201dc05b70 0000000000000086 ffff88201dc04010 0000000000010900 Nov 16 14:22:19 r2i1n4 kernel: [274907.717067] 0000000000010900 0000000000010900 0000000000010900 ffff88201dc05fd8 Nov 16 14:22:19 r2i1n4 kernel: [274907.717073] ffff88201dc05fd8 0000000000010900 ffff881fd7fce600 ffffffff81a11020 Nov 16 14:22:19 r2i1n4 kernel: [274907.717080] Call Trace: Nov 16 14:22:19 r2i1n4 kernel: [274907.717087] [&lt;ffffffff814651a8&gt;] __mutex_lock_slowpath+0xf8/0x150 Nov 16 14:22:19 r2i1n4 kernel: [274907.717094] [&lt;ffffffff81464c3a&gt;] mutex_lock+0x1a/0x40 Nov 16 14:22:19 r2i1n4 kernel: [274907.717121] [&lt;ffffffffa0d1adb0&gt;] ll_setattr_raw+0x2c0/0xb60 [lustre] Nov 16 14:22:19 r2i1n4 kernel: [274907.717143] [&lt;ffffffff81178faf&gt;] notify_change+0x19f/0x2f0 Nov 16 14:22:19 r2i1n4 kernel: [274907.717151] [&lt;ffffffff8115d517&gt;] do_truncate+0x57/0x80 Nov 16 14:22:19 r2i1n4 kernel: [274907.717160] [&lt;ffffffff8116c053&gt;] do_last+0x603/0x800 Nov 16 14:22:19 r2i1n4 kernel: [274907.717168] [&lt;ffffffff8116ceb9&gt;] path_openat+0xd9/0x420 Nov 16 14:22:19 r2i1n4 kernel: [274907.717175] [&lt;ffffffff8116d33c&gt;] do_filp_open+0x4c/0xc0 Nov 16 14:22:19 r2i1n4 kernel: [274907.717183] [&lt;ffffffff8115de5f&gt;] do_sys_open+0x17f/0x250 Nov 16 14:22:19 r2i1n4 kernel: [274907.717191] [&lt;ffffffff8146e1f2&gt;] system_call_fastpath+0x16/0x1b Nov 16 14:22:19 r2i1n4 kernel: [274907.717202] [&lt;00007fffeb9057bd&gt;] 0x7fffeb9057bc </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Nov 16 14:22:19 r2i1n4 kernel: [274907.717285] shald_dpd.out D 0000000000000000 0 827 772 0x00000000 Nov 16 14:22:19 r2i1n4 kernel: [274907.717290] ffff881fbd5afa60 0000000000000086 ffff881fbd5ae010 0000000000010900 Nov 16 14:22:19 r2i1n4 kernel: [274907.717297] 0000000000010900 0000000000010900 0000000000010900 ffff881fbd5affd8 Nov 16 14:22:19 r2i1n4 kernel: [274907.717303] ffff881fbd5affd8 0000000000010900 ffff881fa07c20c0 ffff881020fa2080 Nov 16 14:22:19 r2i1n4 kernel: [274907.717310] Call Trace: Nov 16 14:22:19 r2i1n4 kernel: [274907.717318] [&lt;ffffffff814661a5&gt;] rwsem_down_failed_common+0xb5/0x160 Nov 16 14:22:19 r2i1n4 kernel: [274907.717329] [&lt;ffffffff81264533&gt;] call_rwsem_down_write_failed+0x13/0x20 Nov 16 14:22:19 r2i1n4 kernel: [274907.717337] [&lt;ffffffff814654bc&gt;] down_write+0x1c/0x20 Nov 16 14:22:19 r2i1n4 kernel: [274907.717370] [&lt;ffffffffa0d51159&gt;] vvp_io_setattr_start+0x129/0x170 [lustre] Nov 16 14:22:19 r2i1n4 kernel: [274907.717447] [&lt;ffffffffa088d539&gt;] cl_io_start+0x69/0x140 [obdclass] Nov 16 14:22:19 r2i1n4 kernel: [274907.717528] [&lt;ffffffffa0890993&gt;] cl_io_loop+0xa3/0x190 [obdclass] Nov 16 14:22:19 r2i1n4 kernel: [274907.717605] [&lt;ffffffffa0d4905f&gt;] cl_setattr_ost+0x24f/0x2b0 [lustre] Nov 16 14:22:19 r2i1n4 kernel: [274907.717663] [&lt;ffffffffa0d1b2e4&gt;] ll_setattr_raw+0x7f4/0xb60 [lustre] Nov 16 14:22:19 r2i1n4 kernel: [274907.717683] [&lt;ffffffff81178faf&gt;] notify_change+0x19f/0x2f0 Nov 16 14:22:19 r2i1n4 kernel: [274907.717692] [&lt;ffffffff8115d517&gt;] do_truncate+0x57/0x80 Nov 16 14:22:19 r2i1n4 kernel: [274907.717699] [&lt;ffffffff8116c053&gt;] do_last+0x603/0x800 Nov 16 14:22:19 r2i1n4 kernel: [274907.717708] [&lt;ffffffff8116ceb9&gt;] path_openat+0xd9/0x420 Nov 16 14:22:19 r2i1n4 kernel: [274907.717715] [&lt;ffffffff8116d33c&gt;] do_filp_open+0x4c/0xc0 Nov 16 14:22:19 r2i1n4 kernel: [274907.717723] [&lt;ffffffff8115de5f&gt;] do_sys_open+0x17f/0x250 Nov 16 14:22:19 r2i1n4 kernel: [274907.717731] [&lt;ffffffff8146e1f2&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>You could try patch 14915 but most likely you're going to end up with the same problem.</p> <p>From the backtrace, the truncate process was waiting for i_alloc_sem while holding inode mutex but nobody should have held i_alloc_sem from the stack. Unfortunately I don't have a sle11-sp3 kernel source code on hand otherwise I could take a look.</p> <p>One alternative way is to get a kernel dump when you see this problem and search the variable address of alloc_sem probably you can find it on some proc's stack.</p> <p>To nip this in the bud - it seems like any ongoing work should move to a new ticket...</p> <blockquote> <p>To nip this in the bud - it seems like any ongoing work should move to a new ticket...</p></blockquote> <p>To be clear, the customer was using a build with their own patches, there is no evidence that this issue can be seen on master.</p> Blocker LU-6843 Duplicate LU-7066 Related LU-12142 Development Rank 1|hzx6vj: Rank (Obsolete) 17581 Severity