https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5970 Lustre <p>sanity-benchmark test bonnie hung.</p> <p>On client node:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>bonnie++ D 0000000000000001 0 12696 12355 0x00000080 ffff88006cd17ae8 0000000000000082 00000000ffffffff 000027e0a71a87f8 ffff88006d38bec0 ffff8800398b1830 0000000000aebe00 ffffffffab83d239 ffff880069133ab8 ffff88006cd17fd8 000000000000fbc8 ffff880069133ab8 Call Trace: [&lt;ffffffff810a6d31&gt;] ? ktime_get_ts+0xb1/0xf0 [&lt;ffffffff8111f750&gt;] ? sync_page_killable+0x0/0x40 [&lt;ffffffff815295e3&gt;] io_schedule+0x73/0xc0 [&lt;ffffffff8111f73d&gt;] sync_page+0x3d/0x50 [&lt;ffffffff8111f75e&gt;] sync_page_killable+0xe/0x40 [&lt;ffffffff81529e7a&gt;] __wait_on_bit_lock+0x5a/0xc0 [&lt;ffffffff8111f667&gt;] __lock_page_killable+0x67/0x70 [&lt;ffffffff8109b020&gt;] ? wake_bit_function+0x0/0x50 [&lt;ffffffff8111f50e&gt;] ? find_get_page+0x1e/0xa0 [&lt;ffffffff81121394&gt;] generic_file_aio_read+0x4b4/0x700 [&lt;ffffffffa06ec41b&gt;] vvp_io_read_start+0x22b/0x410 [lustre] [&lt;ffffffffa0c46d2a&gt;] cl_io_start+0x6a/0x140 [obdclass] [&lt;ffffffffa0c4b434&gt;] cl_io_loop+0xb4/0x1b0 [obdclass] [&lt;ffffffffa068e326&gt;] ll_file_io_generic+0x2a6/0x610 [lustre] [&lt;ffffffffa068e7cf&gt;] ll_file_aio_read+0x13f/0x2c0 [lustre] [&lt;ffffffffa068ef3c&gt;] ll_file_read+0x16c/0x2a0 [lustre] [&lt;ffffffff81189645&gt;] vfs_read+0xb5/0x1a0 [&lt;ffffffff81189781&gt;] sys_read+0x51/0x90 [&lt;ffffffff810e1bfe&gt;] ? __audit_syscall_exit+0x25e/0x290 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>On OSS node:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ll_ost_io00_0 D 0000000000000000 0 9922 2 0x00000080 ffff88007aa67790 0000000000000046 ffff88007aa67758 ffff88007aa67754 ffff880079f02378 ffff88007f823040 ffff880002316740 0000000000000400 ffff88006b02daf8 ffff88007aa67fd8 000000000000fb88 ffff88006b02daf8 Call Trace: [&lt;ffffffffa0ce06cb&gt;] osd_do_bio+0x68b/0x800 [osd_ldiskfs] [&lt;ffffffffa0d4402c&gt;] ? fsfilt_map_nblocks+0xcc/0xf0 [fsfilt_ldiskfs] [&lt;ffffffff81096da0&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffffa0d442d5&gt;] ? fsfilt_ldiskfs_map_inode_pages+0x85/0x90 [fsfilt_ldiskfs] [&lt;ffffffffa0ce29d8&gt;] osd_read_prep+0x338/0x3b0 [osd_ldiskfs] [&lt;ffffffffa0db2bd3&gt;] ofd_preprw_read+0x253/0x7f0 [ofd] [&lt;ffffffffa0db38ea&gt;] ofd_preprw+0x77a/0x1480 [ofd] [&lt;ffffffffa077dff1&gt;] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] [&lt;ffffffffa0d68e0c&gt;] obd_preprw+0x12c/0x3d0 [ost] [&lt;ffffffffa0d707c0&gt;] ost_brw_read+0xd60/0x1340 [ost] [&lt;ffffffff81281876&gt;] ? vsnprintf+0x336/0x5e0 [&lt;ffffffffa0486beb&gt;] ? libcfs_debug_vmsg2+0x50b/0xbb0 [libcfs] [&lt;ffffffffa077ee9c&gt;] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] [&lt;ffffffffa077eff8&gt;] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] [&lt;ffffffffa0d78028&gt;] ost_handle+0x2ac8/0x48e0 [ost] [&lt;ffffffffa04830f4&gt;] ? libcfs_id2str+0x74/0xb0 [libcfs] [&lt;ffffffffa078e3b8&gt;] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [&lt;ffffffffa04775de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [&lt;ffffffffa0488d9f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [&lt;ffffffffa0785719&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [&lt;ffffffff81055ad3&gt;] ? __wake_up+0x53/0x70 [&lt;ffffffffa078f74e&gt;] ptlrpc_main+0xace/0x1700 [ptlrpc] [&lt;ffffffffa078ec80&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [&lt;ffffffffa078ec80&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [&lt;ffffffffa078ec80&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Maloo report:<br/> <a href="https://testing.hpdd.intel.com/test_sets/dfee42f4-7a67-11e4-be53-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/dfee42f4-7a67-11e4-be53-5254006e85c2</a></p> Lustre client build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/104">https://build.hpdd.intel.com/job/lustre-b2_5/104</a><br/> Lustre server build: <a href="https://build.hpdd.intel.com/job/lustre-b2_4/73">https://build.hpdd.intel.com/job/lustre-b2_4/73</a> (2.4.3) LU-5970

2.4.3<->2.5.4 interop: sanity-benchmark test bonnie hung

Bug Blocker Closed Fixed WC Triage Jian Yu Tue, 2 Dec 2014 21:31:50 +0000 Sun, 14 Aug 2016 17:29:05 +0000 Sun, 14 Aug 2016 17:29:05 +0000 Lustre 2.4.3 0 6 <p>This is a regression failure introduced by Lustre b2_5 build #104.<br/> Here is a for-test-only patch to reproduce the failure: <a href="http://review.whamcloud.com/12908" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12908</a></p> <p>Lustre client build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/104" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_5/104</a><br/> Lustre server build: <a href="https://build.hpdd.intel.com/job/lustre-b2_4/73" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_4/73</a> (2.4.3)</p> <p>replay-ost-single test 5 also hit the same failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LNet: Service thread pid 9041 was inactive for 40.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Pid: 9041, comm: ll_ost_io00_022 Call Trace: [&lt;ffffffff810a2431&gt;] ? ktime_get_ts+0xb1/0xf0 [&lt;ffffffff8150ed93&gt;] io_schedule+0x73/0xc0 [&lt;ffffffff8125ed08&gt;] get_request_wait+0x108/0x1d0 [&lt;ffffffff81096da0&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffff81255fbb&gt;] ? elv_merge+0x1cb/0x200 [&lt;ffffffff8125ee6b&gt;] blk_queue_bio+0x9b/0x5d0 [&lt;ffffffff8125d51e&gt;] generic_make_request+0x25e/0x520 [&lt;ffffffffa0002f77&gt;] ? dm_merge_bvec+0xc7/0x100 [dm_mod] [&lt;ffffffff8125d86d&gt;] submit_bio+0x8d/0x120 [&lt;ffffffffa05bd41e&gt;] ? lprocfs_oh_tally+0x2e/0x50 [obdclass] [&lt;ffffffffa0cdfffc&gt;] osd_submit_bio+0x1c/0x60 [osd_ldiskfs] [&lt;ffffffffa0ce041c&gt;] osd_do_bio+0x3dc/0x800 [osd_ldiskfs] [&lt;ffffffffa0d4402c&gt;] ? fsfilt_map_nblocks+0xcc/0xf0 [fsfilt_ldiskfs] [&lt;ffffffffa0d442d5&gt;] ? fsfilt_ldiskfs_map_inode_pages+0x85/0x90 [fsfilt_ldiskfs] [&lt;ffffffffa0ce31a8&gt;] osd_write_commit+0x328/0x610 [osd_ldiskfs] [&lt;ffffffffa0db1e54&gt;] ofd_commitrw_write+0x684/0x11b0 [ofd] [&lt;ffffffffa0db4bbd&gt;] ofd_commitrw+0x5cd/0xbb0 [ofd] [&lt;ffffffffa05287e5&gt;] ? lprocfs_counter_add+0x125/0x182 [lvfs] [&lt;ffffffffa0d691d8&gt;] obd_commitrw+0x128/0x3d0 [ost] [&lt;ffffffffa0d731d1&gt;] ost_brw_write+0xea1/0x15d0 [ost] [&lt;ffffffff81281876&gt;] ? vsnprintf+0x336/0x5e0 [&lt;ffffffffa073e1a0&gt;] ? target_bulk_timeout+0x0/0xc0 [ptlrpc] [&lt;ffffffffa0d7942b&gt;] ost_handle+0x3ecb/0x48e0 [ost] [&lt;ffffffffa04830f4&gt;] ? libcfs_id2str+0x74/0xb0 [libcfs] [&lt;ffffffffa078e3b8&gt;] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] [&lt;ffffffffa04775de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [&lt;ffffffffa0488d9f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [&lt;ffffffffa0785719&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] [&lt;ffffffff81055ad3&gt;] ? __wake_up+0x53/0x70 [&lt;ffffffffa078f74e&gt;] ptlrpc_main+0xace/0x1700 [ptlrpc] [&lt;ffffffffa078ec80&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [&lt;ffffffffa078ec80&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [&lt;ffffffffa078ec80&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Maloo report: <a href="https://testing.hpdd.intel.com/test_sets/32c28f34-7a69-11e4-be53-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/32c28f34-7a69-11e4-be53-5254006e85c2</a></p> <p>I'm moving the affects version to 2.4.3 since the problem appears to be totally on the server and unrelated to the client. </p> <p>at least in the second occurence we can see that OST is excessively slow but requests do complete after 210 seconds.</p> <blockquote><p>Here is a for-test-only patch to reproduce the failure: <a href="http://review.whamcloud.com/12908" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12908</a></p></blockquote> <p>sanity-benchmark was run for 6 times and all passed.</p> <p>Last comments suggest this is fixed.</p> Development Rank 1|hzx1qn: Rank (Obsolete) 16673 Severity