https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6075 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/a1aa0448-7e7b-11e4-8b8b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a1aa0448-7e7b-11e4-8b8b-5254006e85c2</a>.</p> <p>The sub-test test_7 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test failed to respond and timed out </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>01:03:59:LustreError: 17421:0:(qmt_handler.c:427:qmt_dqacq0()) $$$ Release too much! uuid:lustre-MDT0000-lwp-OST0004_UUID release:262188 granted:0, total:0 qmt:lustre-QMT0000 pool:0-dt id:500 enforced:1 hard:394749 soft:375952 granted:0 time:0 qunit:16384 edquot:0 may_rel:0 revoke:0 01:03:59:LustreError: 17421:0:(qmt_handler.c:427:qmt_dqacq0()) Skipped 1 previous similar message 01:03:59:Lustre: *** cfs_fail_loc=191, val=0*** 01:03:59:Lustre: lustre-MDT0000: trigger OI scrub by RPC for [0x200000bd0:0x100:0x0], rc = 0 [2] 01:03:59:INFO: task jbd2/dm-0-8:17400 blocked for more than 120 seconds. 01:03:59: Not tainted 2.6.32-431.29.2.el6_lustre.x86_64 #1 01:03:59:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 01:03:59:jbd2/dm-0-8 D 0000000000000001 0 17400 2 0x00000080 01:03:59: ffff88005b709d20 0000000000000046 ffff880065334090 ffff880065334080 01:03:59: ffff8800606a7680 ffff8800023168e8 0000000000017997 ffff88006d06d540 01:03:59: ffff88006d06daf8 ffff88005b709fd8 000000000000fbc8 ffff88006d06daf8 01:03:59:Call Trace: 01:03:59: [&lt;ffffffff8109b2ce&gt;] ? prepare_to_wait+0x4e/0x80 01:03:59: [&lt;ffffffffa03df80f&gt;] jbd2_journal_commit_transaction+0x19f/0x1500 [jbd2] 01:03:59: [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320 01:03:59: [&lt;ffffffff81083e1c&gt;] ? lock_timer_base+0x3c/0x70 01:03:59: [&lt;ffffffff8109afa0&gt;] ? autoremove_wake_function+0x0/0x40 01:03:59: [&lt;ffffffffa03e5a58&gt;] kjournald2+0xb8/0x220 [jbd2] 01:03:59: [&lt;ffffffff8109afa0&gt;] ? autoremove_wake_function+0x0/0x40 01:03:59: [&lt;ffffffffa03e59a0&gt;] ? kjournald2+0x0/0x220 [jbd2] 01:03:59: [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 01:03:59: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 01:03:59: [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 01:03:59: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 01:03:59:INFO: task osp-syn-0-0:17440 blocked for more than 120 seconds. 01:03:59: Not tainted 2.6.32-431.29.2.el6_lustre.x86_64 #1 01:03:59:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 01:03:59:osp-syn-0-0 D 0000000000000000 0 17440 2 0x00000080 01:03:59: ffff8800605c1700 0000000000000046 0000000000000000 ffffffff8128d756 01:03:59: 0000000000000008 ffff88006a811a80 ffff8800605c2f68 ffff88005e305600 01:03:59: ffff88006aeb9ab8 ffff8800605c1fd8 000000000000fbc8 ffff88006aeb9ab8 01:03:59:Call Trace: 01:03:59: [&lt;ffffffff8128d756&gt;] ? vsnprintf+0x336/0x5e0 01:03:59: [&lt;ffffffffa03de08a&gt;] start_this_handle+0x25a/0x480 [jbd2] 01:03:59: [&lt;ffffffff8109afa0&gt;] ? autoremove_wake_function+0x0/0x40 01:03:59: [&lt;ffffffffa03de495&gt;] jbd2_journal_start+0xb5/0x100 [jbd2] 01:03:59: [&lt;ffffffffa05b55d4&gt;] ? llog_osd_declare_write_rec+0xd4/0x3f0 [obdclass] 01:03:59: [&lt;ffffffffa04367b6&gt;] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] 01:03:59: [&lt;ffffffffa0d0c7cf&gt;] osd_trans_start+0x1df/0x660 [osd_ldiskfs] 01:03:59: [&lt;ffffffffa05a4b7c&gt;] llog_write+0x22c/0x420 [obdclass] 01:03:59: [&lt;ffffffffa05a4e3f&gt;] llog_cancel_rec+0xaf/0x690 [obdclass] 01:03:59: [&lt;ffffffffa05aadf3&gt;] llog_cat_cancel_records+0x133/0x2f0 [obdclass] 01:03:59: [&lt;ffffffffa10b13f2&gt;] osp_sync_process_committed+0x2b2/0x820 [osp] 01:03:59: [&lt;ffffffffa10b19d3&gt;] osp_sync_process_queues+0x73/0x700 [osp] 01:03:59: [&lt;ffffffff81061d00&gt;] ? default_wake_function+0x0/0x20 01:03:59: [&lt;ffffffffa05a5ec9&gt;] llog_process_thread+0xaa9/0xe80 [obdclass] 01:03:59: [&lt;ffffffffa10b1960&gt;] ? osp_sync_process_queues+0x0/0x700 [osp] 01:03:59: [&lt;ffffffffa05a7abf&gt;] llog_process_or_fork+0x13f/0x540 [obdclass] 01:03:59: [&lt;ffffffffa05aab28&gt;] llog_cat_process_cb+0x448/0x5e0 [obdclass] 01:03:59: [&lt;ffffffffa05a5ec9&gt;] llog_process_thread+0xaa9/0xe80 [obdclass] 01:03:59: [&lt;ffffffffa05aa6e0&gt;] ? llog_cat_process_cb+0x0/0x5e0 [obdclass] 01:03:59: [&lt;ffffffffa05a7abf&gt;] llog_process_or_fork+0x13f/0x540 [obdclass] 01:03:59: [&lt;ffffffffa05a8dfd&gt;] llog_cat_process_or_fork+0x1ad/0x300 [obdclass] 01:03:59: [&lt;ffffffff81061d12&gt;] ? default_wake_function+0x12/0x20 01:03:59: [&lt;ffffffff810546b9&gt;] ? __wake_up_common+0x59/0x90 01:03:59: [&lt;ffffffffa10b1960&gt;] ? osp_sync_process_queues+0x0/0x700 [osp] 01:03:59: [&lt;ffffffffa05a8f69&gt;] llog_cat_process+0x19/0x20 [obdclass] 01:03:59: [&lt;ffffffffa10b2673&gt;] osp_sync_thread+0x243/0x7d0 [osp] 01:03:59: [&lt;ffffffff815296ee&gt;] ? thread_return+0x4e/0x770 01:03:59: [&lt;ffffffffa10b2430&gt;] ? osp_sync_thread+0x0/0x7d0 [osp] 01:03:59: [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 01:03:59: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 01:03:59: [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 01:03:59: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 01:03:59:INFO: task osp-syn-1-0:17442 blocked for more than 120 seconds. 01:03:59: Not tainted 2.6.32-431.29.2.el6_lustre.x86_64 #1 01:03:59:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 01:03:59:osp-syn-1-0 D 0000000000000000 0 17442 2 0x00000080 01:03:59: ffff880057c99700 0000000000000046 0000000000000000 ffffffff8128d756 01:03:59: 0000000000000008 ffff88006a811b40 ffff88005d7d0f68 ffff88005e305600 01:03:59: ffff88006cf95098 ffff880057c99fd8 000000000000fbc8 ffff88006cf95098 01:03:59:Call Trace: 01:03:59: [&lt;ffffffff8128d756&gt;] ? vsnprintf+0x336/0x5e0 01:03:59: [&lt;ffffffffa03de08a&gt;] start_this_handle+0x25a/0x480 [jbd2] 01:03:59: [&lt;ffffffff8109afa0&gt;] ? autoremove_wake_function+0x0/0x40 01:03:59: [&lt;ffffffffa03de495&gt;] jbd2_journal_start+0xb5/0x100 [jbd2] 01:03:59: [&lt;ffffffffa05b55d4&gt;] ? llog_osd_declare_write_rec+0xd4/0x3f0 [obdclass] 01:03:59: [&lt;ffffffffa04367b6&gt;] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] 01:03:59: [&lt;ffffffffa0d0c7cf&gt;] osd_trans_start+0x1df/0x660 [osd_ldiskfs] 01:03:59: [&lt;ffffffffa05a4b7c&gt;] llog_write+0x22c/0x420 [obdclass] 01:03:59: [&lt;ffffffffa05a4e3f&gt;] llog_cancel_rec+0xaf/0x690 [obdclass] 01:03:59: [&lt;ffffffffa05aadf3&gt;] llog_cat_cancel_records+0x133/0x2f0 [obdclass] 01:03:59: [&lt;ffffffffa10b13f2&gt;] osp_sync_process_committed+0x2b2/0x820 [osp] 01:03:59: [&lt;ffffffffa10b19d3&gt;] osp_sync_process_queues+0x73/0x700 [osp] 01:03:59: [&lt;ffffffff81061d00&gt;] ? default_wake_function+0x0/0x20 01:03:59: [&lt;ffffffffa05a5ec9&gt;] llog_process_thread+0xaa9/0xe80 [obdclass] 01:03:59: [&lt;ffffffffa10b1960&gt;] ? osp_sync_process_queues+0x0/0x700 [osp] 01:03:59: [&lt;ffffffffa05a7abf&gt;] llog_process_or_fork+0x13f/0x540 [obdclass] 01:03:59: [&lt;ffffffffa05aab28&gt;] llog_cat_process_cb+0x448/0x5e0 [obdclass] 01:03:59: [&lt;ffffffffa05a5ec9&gt;] llog_process_thread+0xaa9/0xe80 [obdclass] 01:03:59: [&lt;ffffffffa05aa6e0&gt;] ? llog_cat_process_cb+0x0/0x5e0 [obdclass] 01:03:59: [&lt;ffffffffa05a7abf&gt;] llog_process_or_fork+0x13f/0x540 [obdclass] 01:03:59: [&lt;ffffffffa05a8dfd&gt;] llog_cat_process_or_fork+0x1ad/0x300 [obdclass] 01:03:59: [&lt;ffffffff81061d12&gt;] ? default_wake_function+0x12/0x20 01:03:59: [&lt;ffffffff810546b9&gt;] ? __wake_up_common+0x59/0x90 01:03:59: [&lt;ffffffffa10b1960&gt;] ? osp_sync_process_queues+0x0/0x700 [osp] 01:03:59: [&lt;ffffffffa05a8f69&gt;] llog_cat_process+0x19/0x20 [obdclass] 01:03:59: [&lt;ffffffffa10b2673&gt;] osp_sync_thread+0x243/0x7d0 [osp] 01:03:59: [&lt;ffffffff815296ee&gt;] ? thread_return+0x4e/0x770 01:03:59: [&lt;ffffffffa10b2430&gt;] ? osp_sync_thread+0x0/0x7d0 [osp] 01:03:59: [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 01:03:59: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 01:03:59: [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 01:03:59: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 01:03:59:INFO: task osp-syn-2-0:17444 blocked for more than 120 seconds. 01:03:59: Not tainted 2.6.32-431.29.2.el6_lustre.x86_64 #1 01:03:59:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 01:03:59:osp-syn-2-0 D 0000000000000001 0 17444 2 0x00000080 01:03:59: ffff880079239700 0000000000000046 0000000000000000 ffffffff8128d756 01:03:59: 0000000000000008 ffff88006ca29d80 ffff88007923cf68 ffff88005e305600 01:03:59: ffff880057dbd098 ffff880079239fd8 000000000000fbc8 ffff880057dbd098 01:03:59:Call Trace: 01:03:59: [&lt;ffffffff8128d756&gt;] ? vsnprintf+0x336/0x5e0 01:03:59: [&lt;ffffffffa03de08a&gt;] start_this_handle+0x25a/0x480 [jbd2] 01:03:59: [&lt;ffffffff8109afa0&gt;] ? autoremove_wake_function+0x0/0x40 01:03:59: [&lt;ffffffffa03de495&gt;] jbd2_journal_start+0xb5/0x100 [jbd2] 01:03:59: [&lt;ffffffffa05b55d4&gt;] ? llog_osd_declare_write_rec+0xd4/0x3f0 [obdclass] 01:03:59: [&lt;ffffffffa04367b6&gt;] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] 01:03:59: [&lt;ffffffffa0d0c7cf&gt;] osd_trans_start+0x1df/0x660 [osd_ldiskfs] 01:03:59: [&lt;ffffffffa05a4b7c&gt;] llog_write+0x22c/0x420 [obdclass] 01:03:59: [&lt;ffffffffa05a4e3f&gt;] llog_cancel_rec+0xaf/0x690 [obdclass] 01:03:59: [&lt;ffffffffa05aadf3&gt;] llog_cat_cancel_records+0x133/0x2f0 [obdclass] 01:03:59: [&lt;ffffffffa10b13f2&gt;] osp_sync_process_committed+0x2b2/0x820 [osp] 01:03:59: [&lt;ffffffffa10b19d3&gt;] osp_sync_process_queues+0x73/0x700 [osp] 01:03:59: [&lt;ffffffff81061d00&gt;] ? default_wake_function+0x0/0x20 01:03:59: [&lt;ffffffffa05a5ec9&gt;] llog_process_thread+0xaa9/0xe80 [obdclass] 01:03:59: [&lt;ffffffff81060aa3&gt;] ? perf_event_task_sched_out+0x33/0x70 01:03:59: [&lt;ffffffffa10b1960&gt;] ? osp_sync_process_queues+0x0/0x700 [osp] 01:03:59: [&lt;ffffffffa05a7abf&gt;] llog_process_or_fork+0x13f/0x540 [obdclass] 01:03:59: [&lt;ffffffffa05aab28&gt;] llog_cat_process_cb+0x448/0x5e0 [obdclass] 01:03:59: [&lt;ffffffffa05a5ec9&gt;] llog_process_thread+0xaa9/0xe80 [obdclass] 01:03:59: [&lt;ffffffffa05aa6e0&gt;] ? llog_cat_process_cb+0x0/0x5e0 [obdclass] 01:03:59: [&lt;ffffffffa05a7abf&gt;] llog_process_or_fork+0x13f/0x540 [obdclass] 01:03:59: [&lt;ffffffffa05a8dfd&gt;] llog_cat_process_or_fork+0x1ad/0x300 [obdclass] 01:03:59: [&lt;ffffffff81061d12&gt;] ? default_wake_function+0x12/0x20 01:03:59: [&lt;ffffffff810546b9&gt;] ? __wake_up_common+0x59/0x90 01:03:59: [&lt;ffffffffa10b1960&gt;] ? osp_sync_process_queues+0x0/0x700 [osp] 01:03:59: [&lt;ffffffffa05a8f69&gt;] llog_cat_process+0x19/0x20 [obdclass] 01:03:59: [&lt;ffffffffa10b2673&gt;] osp_sync_thread+0x243/0x7d0 [osp] 01:03:59: [&lt;ffffffff815296ee&gt;] ? thread_return+0x4e/0x770 01:03:59: [&lt;ffffffffa10b2430&gt;] ? osp_sync_thread+0x0/0x7d0 [osp] 01:03:59: [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 01:03:59: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 01:03:59: [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 01:03:59: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 01:03:59:INFO: task osp-syn-3-0:17446 blocked for more than 120 seconds. 01:03:59: Not tainted 2.6.32-431.29.2.el6_lustre.x86_64 #1 01:03:59:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 01:03:59:osp-syn-3-0 D 0000000000000001 0 17446 2 0x00000080 01:03:59: ffff8800581b5700 0000000000000046 0000000000000000 ffffffff8128d756 01:03:59: 0000000000000008 ffff880065334680 ffff8800581b6f68 ffff88005e305600 01:03:59: ffff880066fde5f8 ffff8800581b5fd8 000000000000fbc8 ffff880066fde5f8 01:03:59:Call Trace: 01:03:59: [&lt;ffffffff8128d756&gt;] ? vsnprintf+0x336/0x5e0 01:03:59: [&lt;ffffffffa03de08a&gt;] start_this_handle+0x25a/0x480 [jbd2] 01:03:59: [&lt;ffffffff8109afa0&gt;] ? autoremove_wake_function+0x0/0x40 01:03:59: [&lt;ffffffffa03de495&gt;] jbd2_journal_start+0xb5/0x100 [jbd2] 01:03:59: [&lt;ffffffffa05b55d4&gt;] ? llog_osd_declare_write_rec+0xd4/0x3f0 [obdclass] 01:03:59: [&lt;ffffffffa04367b6&gt;] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs] 01:03:59: [&lt;ffffffffa0d0c7cf&gt;] osd_trans_start+0x1df/0x660 [osd_ldiskfs] 01:03:59: [&lt;ffffffffa05a4b7c&gt;] llog_write+0x22c/0x420 [obdclass] 01:03:59: [&lt;ffffffffa05a4e3f&gt;] llog_cancel_rec+0xaf/0x690 [obdclass] 01:03:59: [&lt;ffffffffa05aadf3&gt;] llog_cat_cancel_records+0x133/0x2f0 [obdclass] 01:03:59: [&lt;ffffffffa10b13f2&gt;] osp_sync_process_committed+0x2b2/0x820 [osp] 01:03:59: [&lt;ffffffffa10b19d3&gt;] osp_sync_process_queues+0x73/0x700 [osp] 01:03:59: [&lt;ffffffff81061d00&gt;] ? default_wake_function+0x0/0x20 01:03:59: [&lt;ffffffffa05a5ec9&gt;] llog_process_thread+0xaa9/0xe80 [obdclass] 01:03:59: [&lt;ffffffffa10b1960&gt;] ? osp_sync_process_queues+0x0/0x700 [osp] 01:03:59: [&lt;ffffffffa05a7abf&gt;] llog_process_or_fork+0x13f/0x540 [obdclass] 01:03:59: [&lt;ffffffffa05aab28&gt;] llog_cat_process_cb+0x448/0x5e0 [obdclass] 01:03:59: [&lt;ffffffffa05a5ec9&gt;] llog_process_thread+0xaa9/0xe80 [obdclass] 01:03:59: [&lt;ffffffffa05aa6e0&gt;] ? llog_cat_process_cb+0x0/0x5e0 [obdclass] 01:03:59: [&lt;ffffffffa05a7abf&gt;] llog_process_or_fork+0x13f/0x540 [obdclass] 01:03:59: [&lt;ffffffffa05a8dfd&gt;] llog_cat_process_or_fork+0x1ad/0x300 [obdclass] 01:03:59: [&lt;ffffffff81061d12&gt;] ? default_wake_function+0x12/0x20 01:03:59: [&lt;ffffffff810546b9&gt;] ? __wake_up_common+0x59/0x90 01:03:59: [&lt;ffffffffa10b1960&gt;] ? osp_sync_process_queues+0x0/0x700 [osp] 01:03:59: [&lt;ffffffffa05a8f69&gt;] llog_cat_process+0x19/0x20 [obdclass] 01:03:59: [&lt;ffffffffa10b2673&gt;] osp_sync_thread+0x243/0x7d0 [osp] 01:03:59: [&lt;ffffffff815296ee&gt;] ? thread_return+0x4e/0x770 01:03:59: [&lt;ffffffffa10b2430&gt;] ? osp_sync_thread+0x0/0x7d0 [osp] 01:03:59: [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 01:03:59: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 01:03:59: [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 01:03:59: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 01:03:59:INFO: task mdt00_004:18960 blocked for more than 120 seconds. 01:03:59: Not tainted 2.6.32-431.29.2.el6_lustre.x86_64 #1 01:03:59:"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. 01:03:59:mdt00_004 D 0000000000000001 0 18960 2 0x00000080 01:03:59: ffff88007942f7b0 0000000000000046 0000000000000000 ffffffff8116f933 01:03:59: ffff88007942f760 ffffffff8116f933 ffff88007942f7d0 ffffffffa0cb5d0f 01:03:59: ffff88006c955ab8 ffff88007942ffd8 000000000000fbc8 ffff88006c955ab8 </pre> </div></div> server: lustre-master build #2770<br/> client: 2.6.0 LU-6075

Interop 2.6.0<->2.7 sanity-scrub test_7: D process in MDS

Bug Minor Resolved Fixed WC Triage Maloo Mon, 29 Dec 2014 22:16:19 +0000 Fri, 20 Feb 2015 16:15:51 +0000 Fri, 20 Feb 2015 16:15:51 +0000 Lustre 2.7.0 Lustre 2.7.0 0 8 <p>In the test:<br/> <a href="https://testing.hpdd.intel.com/test_sets/a1aa0448-7e7b-11e4-8b8b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a1aa0448-7e7b-11e4-8b8b-5254006e85c2</a>.</p> <p>sanity-scrub test_6 failed for <a href="https://jira.whamcloud.com/browse/LU-6012" title="sanity-scrub test_6 test_7 test_8 test_9 test_10a: expected &#39;inconsistent&#39; but got &#39;inconsistent,auto&#39;" class="issue-link" data-issue-key="LU-6012"><del>LU-6012</del></a>, which has been fixed by the patch: <a href="http://review.whamcloud.com/#/c/13020/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13020/</a>. The build #2770 does not contains such patch. So please retry the latest master.</p> <p>About sanity-scrub test_7 failure: the OI scrub thread was blocked at jbd2_journal_start() because the thread jbd2/dm-0-8 was making journal flush at jbd2_journal_commit_transaction(); but thread jbd2/dm-0-8 was waiting the thread mdt_007 to stop its current transaction. Unfortunately, the thread mdt_007 was hung inside mdd_unlink for OSD PDO lock:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">01:03:59: [&lt;ffffffff811bf647&gt;] ? __find_get_block+0x97/0xe0 01:03:59: [&lt;ffffffffa0441a8a&gt;] htree_node_lock_try+0x5aa/0x720 [ldiskfs] 01:03:59: [&lt;ffffffffa04234ae&gt;] ldiskfs_htree_node_lock+0x9e/0xc0 [ldiskfs] 01:03:59: [&lt;ffffffffa042524f&gt;] dx_probe+0x3cf/0x6f0 [ldiskfs] 01:03:59: [&lt;ffffffffa042580e&gt;] ldiskfs_dx_find_entry+0x29e/0x2e0 [ldiskfs] 01:03:59: [&lt;ffffffffa0cb5d0f&gt;] ? qsd_op_begin+0x5f/0xb40 [lquota] 01:03:59: [&lt;ffffffffa0425b6b&gt;] ldiskfs_find_entry+0x31b/0x4a0 [ldiskfs] 01:03:59: [&lt;ffffffffa0d1543a&gt;] osd_index_ea_delete+0x2ba/0xc90 [osd_ldiskfs] 01:03:59: [&lt;ffffffffa0fd0117&gt;] lod_index_delete+0xa7/0x140 [lod] 01:03:59: [&lt;ffffffffa103758c&gt;] __mdd_index_delete_only+0xdc/0x280 [mdd] 01:03:59: [&lt;ffffffffa1038500&gt;] __mdd_index_delete+0x50/0x2c0 [mdd] 01:03:59: [&lt;ffffffffa103df5c&gt;] mdd_unlink+0x81c/0xea0 [mdd] 01:03:59: [&lt;ffffffffa0efe008&gt;] mdo_unlink+0x18/0x50 [mdt] 01:03:59: [&lt;ffffffffa0f080b6&gt;] mdt_reint_unlink+0x9c6/0x10c0 [mdt] 01:03:59: [&lt;ffffffffa060e810&gt;] ? lu_ucred+0x20/0x30 [obdclass] 01:03:59: [&lt;ffffffffa0efe09d&gt;] mdt_reint_rec+0x5d/0x200 [mdt] 01:03:59: [&lt;ffffffffa0ee218b&gt;] mdt_reint_internal+0x4cb/0x7a0 [mdt] 01:03:59: [&lt;ffffffffa0ee29eb&gt;] mdt_reint+0x6b/0x120 [mdt] 01:03:59: [&lt;ffffffffa0873ade&gt;] tgt_request_handle+0x6fe/0xaf0 [ptlrpc] 01:03:59: [&lt;ffffffffa0823411&gt;] ptlrpc_main+0xe41/0x1950 [ptlrpc] 01:03:59: [&lt;ffffffffa08225d0&gt;] ? ptlrpc_main+0x0/0x1950 [ptlrpc] 01:03:59: [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 01:03:59: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 01:03:59: [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 01:03:59: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>There were no more logs to indicate why mdt_007 was blocked by the PDO lock.</p> <p>this test passed on the latest master version</p> <p><a href="https://testing.hpdd.intel.com/test_sessions/98b0df84-8fcd-11e4-9293-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/98b0df84-8fcd-11e4-9293-5254006e85c2</a></p> <blockquote> <p>this test passed on the latest master version<br/> <a href="https://testing.hpdd.intel.com/test_sessions/98b0df84-8fcd-11e4-9293-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/98b0df84-8fcd-11e4-9293-5254006e85c2</a></p></blockquote> <p>That's fine.</p> <p>The left issue is why sanity-scrub test_7 hung. According to the test scripts, there is no unlink operation during the test. So there should be some other non-test threads running during the sanity-scrub. What they were? It may be that another competitive thread held the PDO lock as to block the unlink operation. Unfortunately, there were no logs for that.</p> <p>Liang, do you have any idea about the PDO lock?</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13311" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13311</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6075" title="Interop 2.6.0&lt;-&gt;2.7 sanity-scrub test_7: D process in MDS" class="issue-link" data-issue-key="LU-6075"><del>LU-6075</del></a> osd: race for check/chance od_dirent_journal<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b559a828a4318882a5c619d84ec836d21ecbe987</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13311/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13311/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6075" title="Interop 2.6.0&lt;-&gt;2.7 sanity-scrub test_7: D process in MDS" class="issue-link" data-issue-key="LU-6075"><del>LU-6075</del></a> osd: race for check/chance od_dirent_journal<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: fe486c9894aa1d2275b66b3b3b0baee69a016afa</p> <p>Patch landed to Master.</p> Development Rank 1|hzx33j: Rank (Obsolete) 16902 Severity