https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2208 Lustre <p>Hit this in sanity 118k</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[19970.051046] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =========== 22:02:17 (1350525737) [20160.612156] INFO: task flush-lustre-5:18802 blocked for more than 120 seconds. [20160.612995] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. [20160.613824] flush-lustre- D 0000000000000000 3504 18802 2 0x00000000 [20160.614395] ffff880047a39958 0000000000000046 0000000000000000 ffff880047a39900 [20160.615163] ffff880047a399a0 0000000100000000 0000000000000020 ffff88005f620c20 [20160.615529] ffff880020730738 ffff880047a39fd8 000000000000fba8 ffff880020730738 [20160.615893] Call Trace: [20160.616057] [&lt;ffffffff814fabbd&gt;] rwsem_down_failed_common+0x8d/0x1d0 [20160.616272] [&lt;ffffffff814fad56&gt;] rwsem_down_read_failed+0x26/0x30 [20160.616483] [&lt;ffffffff8127c104&gt;] call_rwsem_down_read_failed+0x14/0x30 [20160.616718] [&lt;ffffffff814f9ec7&gt;] ? down_read+0x37/0x40 [20160.616954] [&lt;ffffffffa1527b24&gt;] lov_lsm_addref+0x34/0x150 [lov] [20160.617186] [&lt;ffffffffa1528043&gt;] lov_io_init+0x73/0x160 [lov] [20160.617417] [&lt;ffffffffa10eb4e8&gt;] cl_io_init0+0x98/0x160 [obdclass] [20160.617647] [&lt;ffffffffa10ee2a4&gt;] cl_io_init+0x64/0x100 [obdclass] [20160.617869] [&lt;ffffffffa095e9ce&gt;] cl_sync_file_range+0x11e/0x560 [lustre] [20160.618793] [&lt;ffffffffa0984342&gt;] ll_writepages+0x72/0x1b0 [lustre] [20160.619027] [&lt;ffffffff81127f44&gt;] do_writepages+0x24/0x40 [20160.619228] [&lt;ffffffff811a5594&gt;] writeback_single_inode+0xe4/0x2d0 [20160.619439] [&lt;ffffffff811a5a13&gt;] writeback_sb_inodes+0xd3/0x190 [20160.619646] [&lt;ffffffff811a5b4b&gt;] writeback_inodes_wb+0x7b/0x1a0 [20160.619854] [&lt;ffffffff811a5efb&gt;] wb_writeback+0x28b/0x3d0 [20160.620060] [&lt;ffffffff8107c6e2&gt;] ? del_timer_sync+0x22/0x30 [20160.620266] [&lt;ffffffff811a61e5&gt;] wb_do_writeback+0x1a5/0x250 [20160.620474] [&lt;ffffffff811a62f3&gt;] bdi_writeback_task+0x63/0x1b0 [20160.620714] [&lt;ffffffff8108fc27&gt;] ? bit_waitqueue+0x17/0xd0 [20160.620928] [&lt;ffffffff81136dc0&gt;] ? bdi_start_fn+0x0/0x100 [20160.621172] [&lt;ffffffff81136e46&gt;] bdi_start_fn+0x86/0x100 [20160.621396] [&lt;ffffffff81136dc0&gt;] ? bdi_start_fn+0x0/0x100 [20160.621598] [&lt;ffffffff8108fa16&gt;] kthread+0x96/0xa0 [20160.621790] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [20160.621985] [&lt;ffffffff8108f980&gt;] ? kthread+0x0/0xa0 [20160.622191] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 [20160.622411] INFO: task dd:26845 blocked for more than 120 seconds. [20160.622631] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. [20160.622987] dd D 0000000000000007 2608 26845 1 0x00000000 [20160.623220] ffff88005bea3470 0000000000000086 0000000000000000 0000000000000286 [20160.623585] 0000000000000092 ffff88003b6a8548 000000015bea3458 ffffffff81f61808 [20160.623949] ffff880046f00b78 ffff88005bea3fd8 000000000000fba8 ffff880046f00b78 [20160.624318] Call Trace: [20160.624476] [&lt;ffffffff81044f4e&gt;] ? kernel_map_pages+0xfe/0x110 [20160.624700] [&lt;ffffffff814fabbd&gt;] rwsem_down_failed_common+0x8d/0x1d0 [20160.624978] [&lt;ffffffffa08d1afe&gt;] ? cfs_mem_cache_free+0xe/0x10 [libcfs] [20160.625319] [&lt;ffffffff814fad56&gt;] rwsem_down_read_failed+0x26/0x30 [20160.625557] [&lt;ffffffff8127c104&gt;] call_rwsem_down_read_failed+0x14/0x30 [20160.625774] [&lt;ffffffff814f9ec7&gt;] ? down_read+0x37/0x40 [20160.625992] [&lt;ffffffffa1527b24&gt;] lov_lsm_addref+0x34/0x150 [lov] [20160.626238] [&lt;ffffffffa1528043&gt;] lov_io_init+0x73/0x160 [lov] [20160.626467] [&lt;ffffffffa10eb4e8&gt;] cl_io_init0+0x98/0x160 [obdclass] [20160.626690] [&lt;ffffffffa10ee2a4&gt;] cl_io_init+0x64/0x100 [obdclass] [20160.626922] [&lt;ffffffffa149b6d3&gt;] osc_lru_shrink+0x4a3/0x8c0 [osc] [20160.627143] [&lt;ffffffff8116133a&gt;] ? cache_alloc_debugcheck_after+0x14a/0x210 [20160.627378] [&lt;ffffffffa149bfa8&gt;] osc_page_init+0x4b8/0xb40 [osc] [20160.627610] [&lt;ffffffffa10de7b5&gt;] ? cl_page_slice_add+0x55/0x140 [obdclass] [20160.627856] [&lt;ffffffffa10e2e7b&gt;] cl_page_find0+0x2db/0x900 [obdclass] [20160.628081] [&lt;ffffffff8116133a&gt;] ? cache_alloc_debugcheck_after+0x14a/0x210 [20160.628327] [&lt;ffffffffa10e34b8&gt;] cl_page_find_sub+0x18/0x20 [obdclass] [20160.628575] [&lt;ffffffffa152a503&gt;] lov_page_init_raid0+0x1a3/0x780 [lov] [20160.628806] [&lt;ffffffffa1527f58&gt;] lov_page_init+0x68/0xe0 [lov] [20160.629038] [&lt;ffffffffa10e2e7b&gt;] cl_page_find0+0x2db/0x900 [obdclass] [20160.629268] [&lt;ffffffffa08e83e2&gt;] ? cfs_hash_lookup+0x82/0xa0 [libcfs] [20160.629492] [&lt;ffffffff811704f5&gt;] ? mem_cgroup_charge_common+0xa5/0xd0 [20160.629728] [&lt;ffffffffa10e34d1&gt;] cl_page_find+0x11/0x20 [obdclass] [20160.629962] [&lt;ffffffffa0985ab4&gt;] ll_cl_init+0x154/0x5b0 [lustre] [20160.630190] [&lt;ffffffff814faeee&gt;] ? _spin_unlock_irq+0xe/0x20 [20160.630430] [&lt;ffffffffa0986163&gt;] ll_prepare_write+0x53/0x1a0 [lustre] [20160.630672] [&lt;ffffffffa099dfbe&gt;] ll_write_begin+0x7e/0x1a0 [lustre] [20160.630898] [&lt;ffffffff81112d23&gt;] generic_file_buffered_write+0x123/0x300 [20160.631130] [&lt;ffffffff8106fea7&gt;] ? current_fs_time+0x27/0x30 [20160.631341] [&lt;ffffffff811147e0&gt;] __generic_file_aio_write+0x250/0x480 [20160.631563] [&lt;ffffffff81114a7f&gt;] generic_file_aio_write+0x6f/0xe0 [20160.631790] [&lt;ffffffffa09b171c&gt;] vvp_io_write_start+0x9c/0x240 [lustre] [20160.632034] [&lt;ffffffffa10eb26a&gt;] cl_io_start+0x6a/0x140 [obdclass] [20160.632285] [&lt;ffffffffa10efa54&gt;] cl_io_loop+0xb4/0x1b0 [obdclass] [20160.632529] [&lt;ffffffffa095e05b&gt;] ll_file_io_generic+0x42b/0x550 [lustre] [20160.632763] [&lt;ffffffffa095ef4c&gt;] ll_file_aio_write+0x13c/0x2c0 [lustre] [20160.632994] [&lt;ffffffffa095f239&gt;] ll_file_write+0x169/0x2a0 [lustre] [20160.633222] [&lt;ffffffff8117b2e8&gt;] vfs_write+0xb8/0x1a0 [20160.633423] [&lt;ffffffff8117bbb1&gt;] sys_write+0x51/0x90 [20160.633624] [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b [20160.633837] INFO: task dd:26851 blocked for more than 120 seconds. </pre> </div></div> <p>There's a bunch of other dd processes hung in there.<br/> Jinshan, I am leaving centos6-1 node in this state for you to look when you have time. Feel free to crash it with echo c &gt;/proc/sysrq-trigger if you feel you would benefit from using crash to further investigate it that way.</p> LU-2208

deadlock in add_lsmref

Bug Blocker Resolved Duplicate Jinshan Xiong Oleg Drokin Thu, 18 Oct 2012 03:51:33 +0000 Tue, 13 Nov 2012 15:09:27 +0000 Tue, 13 Nov 2012 15:09:27 +0000 Lustre 2.4.0 0 3 <p>Might be related to <a href="https://jira.whamcloud.com/browse/LU-1612" title="Test failure on test suite parallel-scale-nfsv3, subtest test_iorssf" class="issue-link" data-issue-key="LU-1612"><del>LU-1612</del></a></p> <p>Oh, and test output to date (hanging at this stage for many hours already):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 118k: bio alloc -ENOMEM and IO TERM handling =========== 22:02:17 (1350525737) fail_loc=0x20e 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0580369 s, 181 MB/s sanity.sh: line 6428: kill: (26840) - No such process sanity.sh: line 6429: 26843 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0606453 s, 173 MB/s sanity.sh: line 6428: kill: (26846) - No such process sanity.sh: line 6429: 26849 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0595693 s, 176 MB/s sanity.sh: line 6428: kill: (26852) - No such process sanity.sh: line 6429: 26855 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) sanity.sh: line 6429: 26858 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) sanity.sh: line 6429: 26861 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) sanity.sh: line 6429: 26864 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) sanity.sh: line 6429: 26867 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) fail_loc=0 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 2.63154 s, 4.0 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.63022 s, 6.4 MB/s </pre> </div></div> <p>Hit this again, here's the debug log</p> <p>patch is at <a href="http://review.whamcloud.com/4416" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4416</a></p> <p>This problem was imported by layout lock client implementation. Set this as duplicate of <a href="https://jira.whamcloud.com/browse/LU-1876" title="Layout Lock Server Patch Landings to Master" class="issue-link" data-issue-key="LU-1876"><del>LU-1876</del></a> as it will be fixed over there.</p> Related LU-1612 Development Rank 1|hzvalr: Rank (Obsolete) 5253 Severity