https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2531 Lustre <p>Hit this running racer on a single node</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[56349.656614] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff8800805c6f08[3 ffff88000bf11f80:256 ^(null)_ffff880014be0f08 0 0 1 (null) (null) 0x0] [56349.657354] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff880014be0f08[2 ffff88002a8ecf98:256 ^ffff8800805c6f08_(null) 0 0 1 (null) (null) 0x0] [56349.658145] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) vvp-page@ffff88006e65eb60(0:0:0) vm@ffffea00021b5a58 2000000000083c 2:0 ffff8800805c6f08 256 lru [56349.658890] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) lov-page@ffff88002510ee00 [56349.659360] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) osc-page@ffff880062170f08: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 1048576 0 4096 0x0 0x520 | (null) ffff88007d410cf0 ffff880078590eb0 &gt; 3&lt; + ffff88006fc50140 1 0 0 &gt; 4&lt; 0 0 8 20480 - | - - + - &gt; 5&lt; - - + - | 0 - | 251 - -&gt; [56349.660491] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) end page@ffff8800805c6f08 [56349.660996] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) exists 0/cp. [56349.661284] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff880014be1f08[3 ffff88000bf11f80:257 ^(null)_ffff88001333ef08 0 0 1 (null) (null) 0x0] [56349.662042] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff88001333ef08[2 ffff88002a8ecf98:257 ^ffff880014be1f08_(null) 0 0 1 (null) (null) 0x0] [56349.662802] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) vvp-page@ffff88006e65ebc8(0:0:0) vm@ffffea0001cef2d0 2000000000083c 2:0 ffff880014be1f08 257 lru [56349.663548] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) lov-page@ffff88002510e620 [56349.664016] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) osc-page@ffff88002b782f08: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 1052672 0 4096 0x0 0x520 | (null) ffff88007d410cf0 ffff880078590eb0 &gt; 3&lt; + ffff88006fc50140 1 0 0 &gt; 4&lt; 0 0 8 20480 - | - - + - &gt; 5&lt; - - + - | 0 - | 251 - -&gt; [56349.665336] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) end page@ffff880014be1f08 [56349.665805] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) exists 0/cp. [56349.666679] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff880093a26f08[3 ffff88000bf11f80:258 ^(null)_ffff88003c596f08 0 0 1 (null) (null) 0x0] [56349.667419] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff88003c596f08[2 ffff88002a8ecf98:258 ^ffff880093a26f08_(null) 0 0 1 (null) (null) 0x0] [56349.668166] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) vvp-page@ffff88006e65ec30(0:0:0) vm@ffffea0002285560 2000000000083c 2:0 ffff880093a26f08 258 lru [56349.668917] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) lov-page@ffff88002510ece0 [56349.669389] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) osc-page@ffff88005bf32f08: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 1056768 0 4096 0x0 0x520 | (null) ffff88007d410cf0 ffff880078590eb0 &gt; 3&lt; + ffff88006fc50140 1 0 0 &gt; 4&lt; 0 0 8 20480 - | - - + - &gt; 5&lt; - - + - | 0 - | 251 - -&gt; [56349.671402] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) end page@ffff880093a26f08 [56349.671883] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) exists 0/cp. [56349.672186] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff8800304a3f08[3 ffff88000bf11f80:259 ^(null)_ffff880032388f08 0 0 1 (null) (null) 0x0] [56349.672953] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff880032388f08[2 ffff88002a8ecf98:259 ^ffff8800304a3f08_(null) 0 0 1 (null) (null) 0x0] [56349.673667] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) vvp-page@ffff88006e65ec98(0:0:0) vm@ffffea0001d40e80 2000000000083c 2:0 ffff8800304a3f08 259 lru [56349.674403] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) lov-page@ffff8800740957d0 [56349.674898] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) osc-page@ffff8800922cff08: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 1060864 0 4096 0x0 0x520 | (null) ffff88007d410cf0 ffff880078590eb0 &gt; 3&lt; + ffff88006fc50140 1 0 0 &gt; 4&lt; 0 0 8 20480 - | - - + - &gt; 5&lt; - - + - | 0 - | 251 - -&gt; [56349.676000] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) end page@ffff8800304a3f08 [56349.676500] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) exists 0/cp. [56349.676872] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff8800304a2f08[3 ffff88000bf11f80:260 ^(null)_ffff8800639baf08 0 0 1 (null) (null) 0x0] [56349.677605] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff8800639baf08[2 ffff88002a8ecf98:260 ^ffff8800304a2f08_(null) 0 0 1 (null) (null) 0x0] [56349.678374] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) vvp-page@ffff88006e65ed00(0:0:0) vm@ffffea0002114048 2000000000083c 2:0 ffff8800304a2f08 260 lru [56349.679110] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) lov-page@ffff880074095c98 [56349.679560] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) osc-page@ffff880081cf0f08: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 1064960 0 4096 0x0 0x520 | (null) ffff88007d410cf0 ffff880078590eb0 &gt; 3&lt; + ffff88006fc50140 1 0 0 &gt; 4&lt; 0 0 8 20480 - | - - + - &gt; 5&lt; - - + - | 0 - | 251 - -&gt; [56349.680665] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) end page@ffff8800304a2f08 [56349.681134] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) exists 0/cp. [56349.681428] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff8800639bbf08[3 ffff88000bf11f80:261 ^(null)_ffff8800223a9f08 0 0 1 (null) (null) 0x0] [56349.682160] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff8800223a9f08[2 ffff88002a8ecf98:261 ^ffff8800639bbf08_(null) 0 0 1 (null) (null) 0x0] [56349.682893] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) vvp-page@ffff88006e65ed68(0:0:0) vm@ffffea000218f5a0 2000000000083c 2:0 ffff8800639bbf08 261 lru [56349.683625] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) lov-page@ffff880074095ce0 [56349.684092] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) osc-page@ffff88001309df08: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 1069056 0 4096 0x0 0x520 | (null) ffff88007d410cf0 ffff880078590eb0 &gt; 3&lt; + ffff88006fc50140 1 0 0 &gt; 4&lt; 0 0 8 20480 - | - - + - &gt; 5&lt; - - + - | 0 - | 251 - -&gt; [56349.685177] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) end page@ffff8800639bbf08 [56349.685623] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) exists 0/cp. [56349.685921] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff8800223a8f08[3 ffff88000bf11f80:262 ^(null)_ffff8800248a3f08 0 0 1 (null) (null) 0x0] [56349.686682] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff8800248a3f08[2 ffff88002a8ecf98:262 ^ffff8800223a8f08_(null) 0 0 1 (null) (null) 0x0] [56349.687427] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) vvp-page@ffff88002d0ee0d0(0:0:0) vm@ffffea0001f0b178 2000000000083c 2:0 ffff8800223a8f08 262 lru [56349.688184] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) lov-page@ffff880074095ae8 [56349.688665] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) osc-page@ffff88008e200f08: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 1073152 0 4096 0x0 0x520 | (null) ffff88007d410cf0 ffff880078590eb0 &gt; 3&lt; + ffff88006fc50140 1 0 0 &gt; 4&lt; 0 0 8 20480 - | - - + - &gt; 5&lt; - - + - | 0 - | 251 - -&gt; [56349.689748] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) end page@ffff8800223a8f08 [56349.690214] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) exists 0/cp. [56349.690501] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff880075c2ef08[3 ffff88000bf11f80:263 ^(null)_ffff880033ee2f08 0 0 1 (null) (null) 0x0] [56349.691237] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff880033ee2f08[2 ffff88002a8ecf98:263 ^ffff880075c2ef08_(null) 0 0 1 (null) (null) 0x0] [56349.691967] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) vvp-page@ffff88002d0ee4e0(0:0:0) vm@ffffea00020fec90 2000000000083c 2:0 ffff880075c2ef08 263 lru [56349.692706] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) lov-page@ffff880074095aa0 [56349.693172] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) osc-page@ffff880093ce0f08: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 1077248 0 4096 0x0 0x520 | (null) ffff88007d410cf0 ffff880078590eb0 &gt; 3&lt; + ffff88006fc50140 1 0 0 &gt; 4&lt; 0 0 8 20480 - | - - + - &gt; 5&lt; - - + - | 0 - | 251 - -&gt; [56349.694248] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) end page@ffff880075c2ef08 [56349.694717] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) exists 0/cp. [56349.695010] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff880033ee3f08[3 ffff88000bf11f80:264 ^(null)_ffff880061765f08 0 0 1 (null) (null) 0x0] [56349.696293] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff880061765f08[2 ffff88002a8ecf98:264 ^ffff880033ee3f08_(null) 0 0 1 (null) (null) 0x0] [56349.697945] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) vvp-page@ffff88002d0eea90(0:0:0) vm@ffffea0001f481f8 2000000000083c 2:0 ffff880033ee3f08 264 lru [56349.698697] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) lov-page@ffff880074095740 [56349.699176] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) osc-page@ffff8800794ccf08: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 1081344 0 4096 0x0 0x520 | (null) ffff88007d410cf0 ffff880078590eb0 &gt; 3&lt; + ffff88006fc50140 1 0 0 &gt; 4&lt; 0 0 8 20480 - | - - + - &gt; 5&lt; - - + - | 0 - | 251 - -&gt; [56349.700246] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) end page@ffff880033ee3f08 [56349.700732] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) exists 0/cp. [56349.701026] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff8800393f6f08[3 ffff88000bf11f80:265 ^(null)_ffff88008b6e9f08 0 0 1 (null) (null) 0x0] [56349.701758] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) page@ffff88008b6e9f08[2 ffff88002a8ecf98:265 ^ffff8800393f6f08_(null) 0 0 1 (null) (null) 0x0] [56349.702492] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) vvp-page@ffff88002d0eec30(0:0:0) vm@ffffea000277cc78 2000000000083c 2:0 ffff8800393f6f08 265 lru [56349.703850] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) lov-page@ffff880074095110 [56349.704321] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) osc-page@ffff88006b098f08: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 1085440 0 4096 0x0 0x520 | (null) ffff88007d410cf0 ffff880078590eb0 &gt; 3&lt; + ffff88006fc50140 1 0 0 &gt; 4&lt; 0 0 8 20480 - | - - + - &gt; 5&lt; - - + - | 0 - | 251 - -&gt; [56349.705396] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) end page@ffff8800393f6f08 [56349.706003] LustreError: 20323:0:(osc_io.c:364:trunc_check_cb()) exists 0/cp. [56349.758028] LustreError: 20323:0:(osc_cache.c:2370:osc_teardown_async_page()) extent ffff88006e1f1f38@{[256 -&gt; 506/511], [2|0|-|cache|wi|ffff880078590eb0], [1028096|251|+|-|ffff88002b60ced8|256|(null)]} trunc at 256. [56349.759172] LustreError: 20323:0:(osc_page.c:431:osc_page_delete()) page@ffff8800805c6f08[2 ffff88000bf11f80:256 ^(null)_ffff880014be0f08 4 0 1 (null) (null) 0x0] [56349.759981] LustreError: 20323:0:(osc_page.c:431:osc_page_delete()) page@ffff880014be0f08[2 ffff88002a8ecf98:256 ^ffff8800805c6f08_(null) 4 0 1 (null) (null) 0x0] [56349.760931] LustreError: 20323:0:(osc_page.c:431:osc_page_delete()) vvp-page@ffff88006e65eb60(0:0:0) vm@ffffea00021b5a58 20000000000035 3:0 0 256 lru [56349.761537] LustreError: 20323:0:(osc_page.c:431:osc_page_delete()) lov-page@ffff88002510ee00 [56349.762131] LustreError: 20323:0:(osc_page.c:431:osc_page_delete()) osc-page@ffff880062170f08: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 1048576 0 4096 0x0 0x520 | (null) ffff88007d410cf0 ffff880078590eb0 &gt; 3&lt; + ffff88006fc50140 0 0 0 &gt; 4&lt; 0 0 8 20480 - | - - + - &gt; 5&lt; - - + - | 0 - | 251 - -&gt; [56349.763222] LustreError: 20323:0:(osc_page.c:431:osc_page_delete()) end page@ffff8800805c6f08 [56349.763780] LustreError: 20323:0:(osc_page.c:431:osc_page_delete()) Trying to teardown failed: -16 [56349.764587] LustreError: 20323:0:(osc_page.c:432:osc_page_delete()) ASSERTION( 0 ) failed: [56349.765113] LustreError: 20323:0:(osc_page.c:432:osc_page_delete()) LBUG [56349.765406] Pid: 20323, comm: cp [56349.765648] [56349.765648] Call Trace: [56349.766258] [&lt;ffffffffa0405915&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [56349.766719] [&lt;ffffffffa0405f27&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [56349.767070] [&lt;ffffffffa09bd5dd&gt;] osc_page_delete+0x32d/0x330 [osc] [56349.767381] [&lt;ffffffffa05d6b15&gt;] cl_page_delete0+0xc5/0x510 [obdclass] [56349.767713] [&lt;ffffffffa05d6fa2&gt;] cl_page_delete+0x42/0x120 [obdclass] [56349.768192] [&lt;ffffffffa0e4025d&gt;] ll_invalidatepage+0x8d/0x180 [lustre] [56349.768623] [&lt;ffffffff81129ee8&gt;] do_invalidatepage+0x28/0x30 [56349.768950] [&lt;ffffffff8112a102&gt;] truncate_inode_page+0xa2/0xc0 [56349.769216] [&lt;ffffffff8112a42a&gt;] truncate_inode_pages_range+0x15a/0x460 [56349.769515] [&lt;ffffffff8112a745&gt;] truncate_inode_pages+0x15/0x20 [56349.769877] [&lt;ffffffff8112a797&gt;] truncate_pagecache+0x47/0x70 [56349.770297] [&lt;ffffffff8112a7d9&gt;] truncate_setsize+0x19/0x20 [56349.770657] [&lt;ffffffff8112a81e&gt;] vmtruncate+0x3e/0x70 [56349.771010] [&lt;ffffffffa0e53a53&gt;] vvp_io_setattr_end+0x63/0x80 [lustre] [56349.771349] [&lt;ffffffffa05dec60&gt;] cl_io_end+0x60/0x150 [obdclass] [56349.771769] [&lt;ffffffffa05e3d82&gt;] cl_io_loop+0xc2/0x1b0 [obdclass] [56349.772084] [&lt;ffffffffa0e4c448&gt;] cl_setattr_ost+0x208/0x2d0 [lustre] [56349.772389] [&lt;ffffffffa0e1c122&gt;] ll_setattr_raw+0x792/0xfb0 [lustre] [56349.772706] [&lt;ffffffffa0e1c99b&gt;] ll_setattr+0x5b/0xf0 [lustre] [56349.772987] [&lt;ffffffff81197688&gt;] notify_change+0x168/0x340 [56349.773252] [&lt;ffffffff81179af4&gt;] do_truncate+0x64/0xa0 [56349.773530] [&lt;ffffffff8118c309&gt;] do_filp_open+0x829/0xd90 [56349.773854] [&lt;ffffffff8118827b&gt;] ? getname+0x3b/0x250 [56349.774277] [&lt;ffffffff81044f4e&gt;] ? kernel_map_pages+0xfe/0x110 [56349.774708] [&lt;ffffffff8119861b&gt;] ? alloc_fd+0xab/0x160 [56349.775166] [&lt;ffffffff81178839&gt;] do_sys_open+0x69/0x140 [56349.775569] [&lt;ffffffff81178950&gt;] sys_open+0x20/0x30 [56349.775968] [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b [56349.776371] [56349.777741] Kernel panic - not syncing: LBUG </pre> </div></div> <p>vmcore and modules are in /exports/crashdumps/192.168.10.218-2012-12-23-08:15:29</p> LU-2531

osc_page.c:432:osc_page_delete()) ASSERTION( 0 ) failed running racer

Bug Blocker Resolved Fixed Jinshan Xiong Oleg Drokin MB Thu, 27 Dec 2012 00:11:22 +0000 Mon, 1 Jul 2013 20:20:32 +0000 Mon, 1 Jul 2013 20:20:20 +0000 Lustre 2.4.0 Lustre 2.4.0 0 4 <p>Just hit it again</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[37032.954552] LustreError: 25132:0:(osc_io.c:364:trunc_check_cb()) page@ffff880010eaef08[3 ffff880022de2f80:256 ^(null)_ffff880010eaff08 2 0 1 (null) ffff880014409458 0x0] [37032.956370] LustreError: 25132:0:(osc_io.c:364:trunc_check_cb()) page@ffff880010eaff08[2 ffff880084a92f98:256 ^ffff880010eaef08_(null) 0 0 1 (null) (null) 0x0] [37032.957249] LustreError: 25132:0:(osc_io.c:364:trunc_check_cb()) vvp-page@ffff880010ea54e0(0:0:0) vm@ffffea00022004f0 2000000000082e 2:0 ffff880010eaef08 256 lru [37032.958143] LustreError: 25132:0:(osc_io.c:364:trunc_check_cb()) lov-page@ffff88005a0487d0 [37032.958671] LustreError: 25132:0:(osc_io.c:364:trunc_check_cb()) osc-page@ffff8800571e0f08: 1&lt; 0x845fed 258 0 - - &gt; 2&lt; 1048576 0 4096 0x0 0x520 | (null) ffff88003723caf0 ffff88008a4deeb0 &gt; 3&lt; + (null) 0 0 0 &gt; 4&lt; 0 0 8 4997120 - | - - + - &gt; 5&lt; - - + - | 0 - | 1 - -&gt; [37032.959823] LustreError: 25132:0:(osc_io.c:364:trunc_check_cb()) end page@ffff880010eaef08 [37032.960336] LustreError: 25132:0:(osc_io.c:364:trunc_check_cb()) exists 0/dd. [37032.960749] LustreError: 25132:0:(osc_io.c:364:trunc_check_cb()) page@ffff880071bdef08[3 ffff880022de2f80:512 ^(null)_ffff880071bdff08 0 0 1 (null) (null) 0x0] [37032.961551] LustreError: 25132:0:(osc_io.c:364:trunc_check_cb()) page@ffff880071bdff08[2 ffff880084a92f98:512 ^ffff880071bdef08_(null) 0 0 1 (null) (null) 0x0] [37032.962441] LustreError: 25132:0:(osc_io.c:364:trunc_check_cb()) vvp-page@ffff8800824c4a90(0:0:0) vm@ffffea0001ce55f8 2000000000081c 3:0 ffff880071bdef08 512 lru [37032.969323] LustreError: 25132:0:(osc_io.c:364:trunc_check_cb()) lov-page@ffff88005390e5d8 [37032.969898] LustreError: 25132:0:(osc_io.c:364:trunc_check_cb()) osc-page@ffff880071bc8f08: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 2097152 0 4096 0x0 0x520 | (null) ffff88003723caf0 ffff88008a4deeb0 &gt; 3&lt; + ffff88006d186400 1 0 0 &gt; 4&lt; 0 0 8 4997120 - | - - + - &gt; 5&lt; - - + - | 0 - | 1 - -&gt; [37032.971071] LustreError: 25132:0:(osc_io.c:364:trunc_check_cb()) end page@ffff880071bdef08 [37032.971577] LustreError: 25132:0:(osc_io.c:364:trunc_check_cb()) exists 0/dd. [37032.983825] LustreError: 25132:0:(osc_cache.c:2370:osc_teardown_async_page()) extent ffff88004b3eaf38@{[512 -&gt; 512/767], [2|0|-|cache|wi|ffff88008a4deeb0], [4096|1|+|-|ffff880051415ed8|256|(null)]} trunc at 512. [37032.984734] LustreError: 25132:0:(osc_page.c:431:osc_page_delete()) page@ffff880071bdef08[2 ffff880022de2f80:512 ^(null)_ffff880071bdff08 4 0 1 (null) (null) 0x0] [37032.985539] LustreError: 25132:0:(osc_page.c:431:osc_page_delete()) page@ffff880071bdff08[2 ffff880084a92f98:512 ^ffff880071bdef08_(null) 4 0 1 (null) (null) 0x0] [37032.986366] LustreError: 25132:0:(osc_page.c:431:osc_page_delete()) vvp-page@ffff8800824c4a90(0:0:0) vm@ffffea0001ce55f8 20000000000035 3:0 0 512 lru [37032.986978] LustreError: 25132:0:(osc_page.c:431:osc_page_delete()) lov-page@ffff88005390e5d8 [37032.987521] LustreError: 25132:0:(osc_page.c:431:osc_page_delete()) osc-page@ffff880071bc8f08: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 2097152 0 4096 0x0 0x520 | (null) ffff88003723caf0 ffff88008a4deeb0 &gt; 3&lt; + ffff88006d186400 0 0 0 &gt; 4&lt; 0 0 8 4997120 - | - - + - &gt; 5&lt; - - + - | 0 - | 1 - -&gt; [37032.988857] LustreError: 25132:0:(osc_page.c:431:osc_page_delete()) end page@ffff880071bdef08 [37032.989387] LustreError: 25132:0:(osc_page.c:431:osc_page_delete()) Trying to teardown failed: -16 [37032.989919] LustreError: 25132:0:(osc_page.c:432:osc_page_delete()) ASSERTION( 0 ) failed: [37032.990433] LustreError: 25132:0:(osc_page.c:432:osc_page_delete()) LBUG [37032.990736] Pid: 25132, comm: dd [37032.990974] [37032.990975] Call Trace: [37032.991411] [&lt;ffffffffa0c09915&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [37032.991739] [&lt;ffffffffa0c09f27&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [37032.992094] [&lt;ffffffffa049a5dd&gt;] osc_page_delete+0x32d/0x330 [osc] [37032.992466] [&lt;ffffffffa0edcb15&gt;] cl_page_delete0+0xc5/0x510 [obdclass] [37032.992808] [&lt;ffffffffa0edcfa2&gt;] cl_page_delete+0x42/0x120 [obdclass] [37032.993141] [&lt;ffffffffa0a5025d&gt;] ll_invalidatepage+0x8d/0x180 [lustre] [37032.993469] [&lt;ffffffff81129ee8&gt;] do_invalidatepage+0x28/0x30 [37032.993889] [&lt;ffffffff8112a102&gt;] truncate_inode_page+0xa2/0xc0 [37032.994200] [&lt;ffffffff8112a42a&gt;] truncate_inode_pages_range+0x15a/0x460 [37032.994517] [&lt;ffffffff8112a745&gt;] truncate_inode_pages+0x15/0x20 [37032.994857] [&lt;ffffffff8112a797&gt;] truncate_pagecache+0x47/0x70 [37032.995175] [&lt;ffffffff8112a7d9&gt;] truncate_setsize+0x19/0x20 [37032.995461] [&lt;ffffffff8112a81e&gt;] vmtruncate+0x3e/0x70 [37032.995750] [&lt;ffffffffa0a63a53&gt;] vvp_io_setattr_end+0x63/0x80 [lustre] [37032.996091] [&lt;ffffffffa0ee4c60&gt;] cl_io_end+0x60/0x150 [obdclass] [37032.996417] [&lt;ffffffffa0ee9d82&gt;] cl_io_loop+0xc2/0x1b0 [obdclass] [37032.996726] [&lt;ffffffffa0a5c448&gt;] cl_setattr_ost+0x208/0x2d0 [lustre] [37032.997040] [&lt;ffffffffa0a2c122&gt;] ll_setattr_raw+0x792/0xfb0 [lustre] [37032.997384] [&lt;ffffffffa0a2c99b&gt;] ll_setattr+0x5b/0xf0 [lustre] [37032.997676] [&lt;ffffffff81197688&gt;] notify_change+0x168/0x340 [37032.998008] [&lt;ffffffff81179af4&gt;] do_truncate+0x64/0xa0 [37032.998286] [&lt;ffffffff8118c309&gt;] do_filp_open+0x829/0xd90 [37032.998563] [&lt;ffffffff8119861b&gt;] ? alloc_fd+0xab/0x160 [37032.998839] [&lt;ffffffff81178839&gt;] do_sys_open+0x69/0x140 [37032.999168] [&lt;ffffffff8117b666&gt;] ? sys_lseek+0x66/0x80 [37032.999446] [&lt;ffffffff81178950&gt;] sys_open+0x20/0x30 [37032.999719] [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b [37033.000016] [37033.014558] Kernel panic - not syncing: LBUG </pre> </div></div> <p>Dump and modules in /exports/crashdumps/192.168.10.210-2012-12-27-10:28:56</p> <p>I found the root cause of this problem. It turns out that the one write RPC was failed with -28 so that truncate process stopped to evicting OSC extents and caused dirty pages still in cache. This problem is easy to fix but it seems like the grant algorithm is broken.</p> <p>patch is at: <a href="http://review.whamcloud.com/4896" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4896</a>, please give it a try</p> <p>Patch landed. Please reopen if more work is needed.</p> <p>Hit this issue again on build lustre-master #1532</p> <p><a href="https://maloo.whamcloud.com/test_sets/480b1f6e-d993-11e2-8ecf-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/480b1f6e-d993-11e2-8ecf-52540035b04c</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == racer test 1: racer on clients: client-16vm1,client-16vm2.lab.whamcloud.com DURATION=900 == 07:11:36 (1371651096) 07:11:38:Lustre: DEBUG MARKER: DURATION=900 MDSCOUNT=1 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre/racer 07:11:38:Lustre: DEBUG MARKER: DURATION=900 MDSCOUNT=1 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre2/racer 07:14:02:LustreError: 11841:0:(osc_cache.c:2381:osc_teardown_async_page()) extent ffff88007b768e58@{[1418 -&gt; 1422/1535], [2|0|-|cache|wi|ffff88005a321180], [20480|5|+|-|ffff8800583e6ed0|256|(null)]} trunc at 1418. 07:14:02:LustreError: 11841:0:(osc_page.c:430:osc_page_delete()) page@ffff8800530bde00[2 ffff880057212808:1418 ^(null)_ffff8800530bc200 4 0 1 (null) (null) 0x0] 07:14:02:LustreError: 11841:0:(osc_page.c:430:osc_page_delete()) page@ffff8800530bc200[2 ffff88005b73db08:1418 ^ffff8800530bde00_(null) 4 0 1 (null) (null) 0x0] 07:14:05:LustreError: 11841:0:(osc_page.c:430:osc_page_delete()) vvp-page@ffff8800530bdeb0(0:0:0) vm@ffffea0000ddc238 20000000000015 4:0 0 1418 lru 07:14:05:LustreError: 11841:0:(osc_page.c:430:osc_page_delete()) lov-page@ffff8800530bdf00 07:14:05:LustreError: 11841:0:(osc_page.c:430:osc_page_delete()) osc-page@ffff8800530bc2d8: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 5808128 0 4096 0x0 0x520 | (null) ffff8800693187c0 ffff88005a321180 &gt; 3&lt; + ffff88005809eaa0 0 0 0 &gt; 4&lt; 0 0 8 39362560 - | - - + - &gt; 5&lt; - - + - | 0 - | 5 - -&gt; 07:14:06:LustreError: 11841:0:(osc_page.c:430:osc_page_delete()) end page@ffff8800530bde00 07:14:06:LustreError: 11841:0:(osc_page.c:430:osc_page_delete()) Trying to teardown failed: -16 07:14:06:LustreError: 11841:0:(osc_page.c:431:osc_page_delete()) ASSERTION( 0 ) failed: 07:14:06:LustreError: 11841:0:(osc_page.c:431:osc_page_delete()) LBUG 07:14:06:Pid: 11841, comm: cp 07:14:06: 07:14:06:Call Trace: 07:14:06: [&lt;ffffffffa058e895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 07:14:06: [&lt;ffffffffa058ee97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 07:14:06: [&lt;ffffffffa0a3b701&gt;] osc_page_delete+0x311/0x320 [osc] 07:14:06: [&lt;ffffffffa06f5125&gt;] cl_page_delete0+0xc5/0x4e0 [obdclass] 07:14:06: [&lt;ffffffffa06f5582&gt;] cl_page_delete+0x42/0x120 [obdclass] 07:14:06: [&lt;ffffffffa0b7c7ad&gt;] ll_invalidatepage+0x8d/0x160 [lustre] 07:14:06: [&lt;ffffffff811304e5&gt;] do_invalidatepage+0x25/0x30 07:14:06: [&lt;ffffffff81130802&gt;] truncate_inode_page+0xa2/0xc0 07:14:06: [&lt;ffffffff81130baf&gt;] truncate_inode_pages_range+0x16f/0x500 07:14:06: [&lt;ffffffff81142562&gt;] ? unmap_mapping_range+0x72/0x140 07:14:06: [&lt;ffffffff81130fd5&gt;] truncate_inode_pages+0x15/0x20 07:14:06: [&lt;ffffffff8113102f&gt;] truncate_pagecache+0x4f/0x70 07:14:06: [&lt;ffffffff811a918a&gt;] simple_setsize+0x3a/0x50 07:14:06: [&lt;ffffffff811a91e0&gt;] simple_setattr+0x40/0x70 07:14:06: [&lt;ffffffffa0b56556&gt;] ll_setattr_raw+0x2a6/0x1090 [lustre] 07:14:06: [&lt;ffffffffa0b5739b&gt;] ll_setattr+0x5b/0xf0 [lustre] 07:14:06: [&lt;ffffffff8119e708&gt;] notify_change+0x168/0x340 07:14:06: [&lt;ffffffff8117f1e4&gt;] do_truncate+0x64/0xa0 07:14:06: [&lt;ffffffff8121cd9f&gt;] ? security_inode_permission+0x1f/0x30 07:14:06: [&lt;ffffffff81193084&gt;] do_filp_open+0x844/0xdd0 07:14:06: [&lt;ffffffff8104757c&gt;] ? __do_page_fault+0x1ec/0x480 07:14:06: [&lt;ffffffff8119f5e2&gt;] ? alloc_fd+0x92/0x160 07:14:06: [&lt;ffffffff8117df59&gt;] do_sys_open+0x69/0x140 07:14:06: [&lt;ffffffff8117e070&gt;] sys_open+0x20/0x30 07:14:06: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 07:14:06: 07:14:06:Kernel panic - not syncing: LBUG </pre> </div></div> <p>This should be a duplication of <a href="https://jira.whamcloud.com/browse/LU-3448" title="osc_page_delete()) ASSERTION(0) failed running racer" class="issue-link" data-issue-key="LU-3448"><del>LU-3448</del></a>.</p> Related LU-3448 Development Rank 1|hzve9r: Rank (Obsolete) 5951 Severity