https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3870 Lustre <p>while testing a own patch i hit a deadlock with journal</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============================== 21:09:39 (1377968979) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop INFO: task jbd2/loop0-8:11882 blocked for more than 120 seconds. "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. jbd2/loop0-8 D ffff8801225702b0 0 11882 2 0x00000080 ffff88010811bcf0 0000000000000046 0000000000000002 ffff8801225702b0 ffff88010811bc90 0000000000000046 ffff88010811bdd8 0000000000000282 ffff88010b0845f8 ffff88010811bfd8 000000000000fc88 ffff88010b0845f8 Call Trace: [&lt;ffffffffa0041aed&gt;] jbd2_journal_commit_transaction+0x19d/0x16e0 [jbd2] [&lt;ffffffff81081d5c&gt;] ? lock_timer_base+0x3c/0x70 [&lt;ffffffff815323f1&gt;] ? _spin_lock_irqsave+0x81/0xa0 [&lt;ffffffff8108381b&gt;] ? try_to_del_timer_sync+0x7b/0xe0 [&lt;ffffffff81099690&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffff8108390a&gt;] ? del_timer_sync+0x8a/0xc0 [&lt;ffffffff81083880&gt;] ? del_timer_sync+0x0/0xc0 [&lt;ffffffffa00481e1&gt;] kjournald2+0xc1/0x220 [jbd2] [&lt;ffffffff81099690&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffffa0048120&gt;] ? kjournald2+0x0/0x220 [jbd2] [&lt;ffffffff81099346&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c1ca&gt;] child_rip+0xa/0x20 [&lt;ffffffff81531fc0&gt;] ? _spin_unlock_irq+0x30/0x40 [&lt;ffffffff8100bb10&gt;] ? restore_args+0x0/0x30 [&lt;ffffffff810992b0&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c1c0&gt;] ? child_rip+0x0/0x20 INFO: lockdep is turned off. INFO: task mdt01_003:12211 blocked for more than 120 seconds. "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. mdt01_003 D ffff88013fc26d00 0 12211 2 0x00000080 ffff88010aa11870 0000000000000046 0000000000000000 ffffffff810b262d ffff88010aa117f0 0000000000000286 ffff88002cbd6cd8 0000000000000286 ffff880106ac8cf8 ffff88010aa11fd8 000000000000fc88 ffff880106ac8cf8 Call Trace: [&lt;ffffffff810b262d&gt;] ? trace_hardirqs_on+0xd/0x10 [&lt;ffffffffa0f456be&gt;] cfs_waitq_wait+0xe/0x10 [libcfs] [&lt;ffffffffa04ef057&gt;] lu_object_find_at+0xb7/0x370 [obdclass] [&lt;ffffffffa014d7e2&gt;] ? fld_server_lookup+0x72/0x440 [fld] [&lt;ffffffff810642f0&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0bdc6f4&gt;] osd_object_find+0x34/0x270 [osd_ldiskfs] [&lt;ffffffffa0be168e&gt;] osd_index_ea_insert+0x4ae/0xdf0 [osd_ldiskfs] [&lt;ffffffffa0be8f46&gt;] ? osd_index_ea_delete+0x906/0xe60 [osd_ldiskfs] [&lt;ffffffffa0b8a818&gt;] ? __ldiskfs_journal_stop+0x68/0xa0 [ldiskfs] [&lt;ffffffffa0449b84&gt;] lod_index_insert+0x44/0xe0 [lod] [&lt;ffffffffa04499d8&gt;] ? lod_index_delete+0x38/0xd0 [lod] [&lt;ffffffffa0b04d5e&gt;] __mdd_orphan_add+0x33e/0x3b0 [mdd] [&lt;ffffffffa0bdf43d&gt;] ? osd_attr_get+0x13d/0x180 [osd_ldiskfs] [&lt;ffffffffa0b0cb51&gt;] mdd_finish_unlink+0xe1/0x380 [mdd] [&lt;ffffffffa0b0f7c8&gt;] mdd_unlink+0x9d8/0xe30 [mdd] [&lt;ffffffffa0c7eb98&gt;] mdo_unlink+0x18/0x50 [mdt] [&lt;ffffffffa0c81f80&gt;] mdt_reint_unlink+0x820/0x1010 [mdt] [&lt;ffffffffa0c7e891&gt;] mdt_reint_rec+0x41/0xe0 [mdt] [&lt;ffffffffa0c66573&gt;] mdt_reint_internal+0x4c3/0x780 [mdt] [&lt;ffffffffa0c66874&gt;] mdt_reint+0x44/0xe0 [mdt] [&lt;ffffffffa0c693da&gt;] mdt_handle_common+0x53a/0x1480 [mdt] [&lt;ffffffffa0ca2b95&gt;] mds_regular_handle+0x15/0x20 [mdt] [&lt;ffffffffa072b6e1&gt;] ptlrpc_server_handle_request+0x3b1/0xc80 [ptlrpc] [&lt;ffffffffa0f4557e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [&lt;ffffffffa0f568af&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [&lt;ffffffffa0722fe7&gt;] ? ptlrpc_wait_event+0xb7/0x2c0 [ptlrpc] [&lt;ffffffff810b262d&gt;] ? trace_hardirqs_on+0xd/0x10 [&lt;ffffffff81055d73&gt;] ? __wake_up+0x53/0x70 [&lt;ffffffffa072c9f2&gt;] ptlrpc_main+0xa42/0x1670 [ptlrpc] [&lt;ffffffffa072bfb0&gt;] ? ptlrpc_main+0x0/0x1670 [ptlrpc] [&lt;ffffffff81099346&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c1ca&gt;] child_rip+0xa/0x20 [&lt;ffffffff81531fc0&gt;] ? _spin_unlock_irq+0x30/0x40 [&lt;ffffffff8100bb10&gt;] ? restore_args+0x0/0x30 [&lt;ffffffff810992b0&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c1c0&gt;] ? child_rip+0x0/0x20 INFO: lockdep is turned off. INFO: task jbd2/loop0-8:11882 blocked for more than 120 seconds. "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. jbd2/loop0-8 D ffff8801225702b0 0 11882 2 0x00000080 ffff88010811bcf0 0000000000000046 0000000000000002 ffff8801225702b0 ffff88010811bc90 0000000000000046 ffff88010811bdd8 0000000000000282 ffff88010b0845f8 ffff88010811bfd8 000000000000fc88 ffff88010b0845f8 Call Trace: [&lt;ffffffffa0041aed&gt;] jbd2_journal_commit_transaction+0x19d/0x16e0 [jbd2] [&lt;ffffffff81081d5c&gt;] ? lock_timer_base+0x3c/0x70 [&lt;ffffffff815323f1&gt;] ? _spin_lock_irqsave+0x81/0xa0 [&lt;ffffffff8108381b&gt;] ? try_to_del_timer_sync+0x7b/0xe0 [&lt;ffffffff81099690&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffff8108390a&gt;] ? del_timer_sync+0x8a/0xc0 [&lt;ffffffff81083880&gt;] ? del_timer_sync+0x0/0xc0 [&lt;ffffffffa00481e1&gt;] kjournald2+0xc1/0x220 [jbd2] [&lt;ffffffff81099690&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffffa0048120&gt;] ? kjournald2+0x0/0x220 [jbd2] [&lt;ffffffff81099346&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c1ca&gt;] child_rip+0xa/0x20 [&lt;ffffffff81531fc0&gt;] ? _spin_unlock_irq+0x30/0x40 [&lt;ffffffff8100bb10&gt;] ? restore_args+0x0/0x30 [&lt;ffffffff810992b0&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c1c0&gt;] ? child_rip+0x0/0x20 INFO: lockdep is turned off. INFO: task mdt01_003:12211 blocked for more than 120 seconds. "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. mdt01_003 D ffff88013fc26d00 0 12211 2 0x00000080 ffff88010aa11870 0000000000000046 0000000000000000 ffffffff810b262d ffff88010aa117f0 0000000000000286 ffff88002cbd6cd8 0000000000000286 ffff880106ac8cf8 ffff88010aa11fd8 000000000000fc88 ffff880106ac8cf8 Call Trace: [&lt;ffffffff810b262d&gt;] ? trace_hardirqs_on+0xd/0x10 [&lt;ffffffffa0f456be&gt;] cfs_waitq_wait+0xe/0x10 [libcfs] [&lt;ffffffffa04ef057&gt;] lu_object_find_at+0xb7/0x370 [obdclass] [&lt;ffffffffa014d7e2&gt;] ? fld_server_lookup+0x72/0x440 [fld] [&lt;ffffffff810642f0&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0bdc6f4&gt;] osd_object_find+0x34/0x270 [osd_ldiskfs] [&lt;ffffffffa0be168e&gt;] osd_index_ea_insert+0x4ae/0xdf0 [osd_ldiskfs] [&lt;ffffffffa0be8f46&gt;] ? osd_index_ea_delete+0x906/0xe60 [osd_ldiskfs] [&lt;ffffffffa0b8a818&gt;] ? __ldiskfs_journal_stop+0x68/0xa0 [ldiskfs] [&lt;ffffffffa0449b84&gt;] lod_index_insert+0x44/0xe0 [lod] [&lt;ffffffffa04499d8&gt;] ? lod_index_delete+0x38/0xd0 [lod] [&lt;ffffffffa0b04d5e&gt;] __mdd_orphan_add+0x33e/0x3b0 [mdd] [&lt;ffffffffa0bdf43d&gt;] ? osd_attr_get+0x13d/0x180 [osd_ldiskfs] [&lt;ffffffffa0b0cb51&gt;] mdd_finish_unlink+0xe1/0x380 [mdd] [&lt;ffffffffa0b0f7c8&gt;] mdd_unlink+0x9d8/0xe30 [mdd] [&lt;ffffffffa0c7eb98&gt;] mdo_unlink+0x18/0x50 [mdt] [&lt;ffffffffa0c81f80&gt;] mdt_reint_unlink+0x820/0x1010 [mdt] [&lt;ffffffffa0c7e891&gt;] mdt_reint_rec+0x41/0xe0 [mdt] [&lt;ffffffffa0c66573&gt;] mdt_reint_internal+0x4c3/0x780 [mdt] [&lt;ffffffffa0c66874&gt;] mdt_reint+0x44/0xe0 [mdt] [&lt;ffffffffa0c693da&gt;] mdt_handle_common+0x53a/0x1480 [mdt] [&lt;ffffffffa0ca2b95&gt;] mds_regular_handle+0x15/0x20 [mdt] [&lt;ffffffffa072b6e1&gt;] ptlrpc_server_handle_request+0x3b1/0xc80 [ptlrpc] [&lt;ffffffffa0f4557e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [&lt;ffffffffa0f568af&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [&lt;ffffffffa0722fe7&gt;] ? ptlrpc_wait_event+0xb7/0x2c0 [ptlrpc] [&lt;ffffffff810b262d&gt;] ? trace_hardirqs_on+0xd/0x10 [&lt;ffffffff81055d73&gt;] ? __wake_up+0x53/0x70 [&lt;ffffffffa072c9f2&gt;] ptlrpc_main+0xa42/0x1670 [ptlrpc] [&lt;ffffffffa072bfb0&gt;] ? ptlrpc_main+0x0/0x1670 [ptlrpc] [&lt;ffffffff81099346&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c1ca&gt;] child_rip+0xa/0x20 [&lt;ffffffff81531fc0&gt;] ? _spin_unlock_irq+0x30/0x40 [&lt;ffffffff8100bb10&gt;] ? restore_args+0x0/0x30 [&lt;ffffffff810992b0&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c1c0&gt;] ? child_rip+0x0/0x20 INFO: lockdep is turned off. ... </pre> </div></div> <p>i have a crashdump - but it's triggered too later to get a lustre log.</p> RHEL6 + master branch (two days ago) LU-3870

deadlock with journal

Bug Major Closed Duplicate Alex Zhuravlev Alexey Lyashkov Sun, 1 Sep 2013 18:05:07 +0000 Mon, 2 Jun 2014 16:54:11 +0000 Mon, 2 Jun 2014 16:54:11 +0000 Lustre 2.5.0 0 6 <p>logs from panic. I may upload a crashdump if it's need.</p> <p>node isn't continue work after ~8h after first soft lookup so looks dead.</p> <p>Shadow, can you please add the git commit hash you are testing in the Environment section.</p> <p>Andreas,</p> <p>look to last patch in <a href="https://jira.whamcloud.com/browse/LU-3240" title="The link count is not updated after the mkdir" class="issue-link" data-issue-key="LU-3240"><del>LU-3240</del></a>. If crashdump with vmlinux + lustre modules is need - i may upload. but i don't able to replicate it.</p> <p>Bobijam will look into this one</p> <p>Shadow,</p> <p>Would you mind uploading the crashdump packages please? Thanks.</p> <p>Hi Bobijam.</p> <p><a href="https://jira.whamcloud.com/browse/LU-3870" title="deadlock with journal" class="issue-link" data-issue-key="LU-3870"><del>LU-3870</del></a>.tar.bz2 finished to upload in next 10-15 minutes. it's have vmlinux, system.map, crashdump with all logs and copy of my lustre tree.<br/> but crash triggered too later :/ let me know if you need something else.</p> <p>I didn't find anything suspicious in the dump so far.</p> <p>Moved over to 2.6.0 since we haven't been able to make any progress with understanding, reproducing, or fixing this problem. Can increase priority again if this is hit again.</p> <p>from my point view - it's bug similar to the <a href="https://jira.whamcloud.com/browse/LU-4019" title="today&#39;s master stick on shutdown on test == sanity test 132: on lu_object_find_at" class="issue-link" data-issue-key="LU-4019"><del>LU-4019</del></a>.<br/> mdt thread stick in</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">PID: 12211 TASK: ffff880106ac8740 CPU: 6 COMMAND: <span class="code-quote">"mdt01_003"</span> #0 [ffff88010aa117b0] schedule at ffffffff8152eebd #1 [ffff88010aa11878] cfs_waitq_wait at ffffffffa0f456be [libcfs] #2 [ffff88010aa11888] lu_object_find_at at ffffffffa04ef057 [obdclass] #3 [ffff88010aa11948] osd_object_find at ffffffffa0bdc6f4 [osd_ldiskfs] #4 [ffff88010aa11978] osd_index_ea_insert at ffffffffa0be168e [osd_ldiskfs] #5 [ffff88010aa11a18] lod_index_insert at ffffffffa0449b84 [lod] #6 [ffff88010aa11a38] __mdd_orphan_add at ffffffffa0b04d5e [mdd] #7 [ffff88010aa11aa8] mdd_finish_unlink at ffffffffa0b0cb51 [mdd] </pre> </div></div> <p>and transaction open. that transaction block any other IO on MDT.</p> <p>Bobijam - did you test a status of the object? may it's race between create and unlink ? or other possibilities to be similar for <a href="https://jira.whamcloud.com/browse/LU-4019" title="today&#39;s master stick on shutdown on test == sanity test 132: on lu_object_find_at" class="issue-link" data-issue-key="LU-4019"><del>LU-4019</del></a> ?</p> <p>Bobijam is away on holiday this week. Alex, could you please take a look?</p> <p>wait in lu_object_find_at() means that the object was marked as dying which is set in -&gt;do_destroy(). if I checked correctly, all calls to -&gt;destroy() are done with the lock held. can you tell what test was running?</p> <p>Alex,</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============================== 21:09:39 (1377968979) Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop Lustre: DEBUG MARKER: cancel_lru_locks osc start Lustre: DEBUG MARKER: cancel_lru_locks osc stop INFO: task jbd2/loop0-8:11882 blocked for more than 120 seconds. </pre> </div></div> <p>Reopening to remove fix version since this is a duplicate.</p> Duplicate LU-2492 Development Rank 1|hzvzsv: Rank (Obsolete) 10042 Severity