https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12509 Lustre <p>[ 141.023079]<br/> ============================================<br/> [ 141.023750] WARNING: possible recursive locking detected<br/> [ 141.024443] 4.18.0-debug #8 Tainted: G O --------- - -<br/> [ 141.025226] --------------------------------------------<br/> [ 141.025967] ll_ost00_002/7298 is trying to acquire lock:<br/> [ 141.030372] 000000001989e622 (&amp;mo-&gt;oo_sem){++++}, at: osd_write_lock+0x8a/0xb0 <span class="error">&#91;osd_zfs&#93;</span><br/> [ 141.031190] <br/> [ 141.031190] but task is already holding lock:<br/> [ 141.032872] 0000000086a83aaf (&amp;mo-&gt;oo_sem){++++}, at: osd_write_lock+0x8a/0xb0 <span class="error">&#91;osd_zfs&#93;</span><br/> [ 141.033676] <br/> [ 141.033676] other info that might help us debug this:<br/> [ 141.034304] Possible unsafe locking scenario:<br/> [ 141.034304] <br/> [ 141.034875] CPU0<br/> [ 141.035121] ----<br/> [ 141.035366] lock(&amp;mo-&gt;oo_sem);<br/> [ 141.035678] lock(&amp;mo-&gt;oo_sem);<br/> [ 141.035993] <br/> [ 141.035993] *** DEADLOCK ***<br/> [ 141.035993] <br/> [ 141.037086] May be due to missing lock nesting notation<br/> [ 141.037086] <br/> [ 141.037727] 3 locks held by ll_ost00_002/7298:<br/> [ 141.038186] #0: 000000003540612a (&amp;m-&gt;ofd_lastid_rwsem)</p> {.+.+} <p>, at: ofd_create_hdl+0x1a9/0x2270 <span class="error">&#91;ofd&#93;</span><br/> [ 141.039081] #1: 00000000ee55375f (&amp;oseq-&gt;os_create_lock){<ins>.</ins>.}, at: ofd_create_hdl+0x364/0x2270 <span class="error">&#91;ofd&#93;</span><br/> [ 141.039969] #2: 0000000086a83aaf (&amp;mo-&gt;oo_sem){++++}, at: osd_write_lock+0x8a/0xb0 <span class="error">&#91;osd_zfs&#93;</span><br/> [ 141.041147] <br/> [ 141.041147] stack backtrace:<br/> [ 141.041568] CPU: 0 PID: 7298 Comm: ll_ost00_002 Tainted: G O --------- - - 4.18.0-debug #8<br/> [ 141.042465] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011<br/> [ 141.043004] Call Trace:<br/> [ 141.043256] dump_stack+0x106/0x175<br/> [ 141.043600] validate_chain.isra.26.cold.44+0x224/0x2da<br/> [ 141.044103] __lock_acquire+0x3df/0xa70<br/> [ 141.044535] lock_acquire+0x13a/0x370<br/> [ 141.044908] ? osd_write_lock+0x8a/0xb0 <span class="error">&#91;osd_zfs&#93;</span><br/> [ 141.045372] down_write_nested+0x6f/0x120<br/> [ 141.045770] ? osd_write_lock+0x8a/0xb0 <span class="error">&#91;osd_zfs&#93;</span><br/> [ 141.046236] osd_write_lock+0x8a/0xb0 <span class="error">&#91;osd_zfs&#93;</span><br/> [ 141.046694] ofd_precreate_objects+0x1764/0x2480 <span class="error">&#91;ofd&#93;</span><br/> [ 141.047201] ofd_create_hdl+0xbec/0x2270 <span class="error">&#91;ofd&#93;</span><br/> [ 141.047722] tgt_handle_request0+0xdf/0x890 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 141.048274] tgt_request_handle+0x3ca/0x1aa0 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 141.048789] ? libcfs_nid2str_r+0x12e/0x160 <span class="error">&#91;lnet&#93;</span><br/> [ 141.049332] ptlrpc_server_handle_request+0x634/0x1180 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 141.049898] ? __wake_up+0x17/0x20<br/> [ 141.050299] ptlrpc_main+0xd7f/0x1470 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 141.050794] ? ptlrpc_register_service+0x14c0/0x14c0 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 141.051353] kthread+0x190/0x1c0<br/> [ 141.051668] ? kthread_create_worker+0x90/0x90<br/> [ 141.052102] ret_from_fork+0x3a/0x50</p> <p>we should use different subclass to silent the lockdep false warning.</p> LU-12509

lockdep warning in ofd_precreate_objects()

Improvement Minor Resolved Fixed Dongyang Li Dongyang Li Fri, 5 Jul 2019 07:14:03 +0000 Fri, 9 Aug 2019 15:14:38 +0000 Fri, 9 Aug 2019 15:14:38 +0000 Lustre 2.13.0 0 5 <p>Li Dongyang (dongyangli@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/35420" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35420</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12509" title="lockdep warning in ofd_precreate_objects()" class="issue-link" data-issue-key="LU-12509"><del>LU-12509</del></a> ofd: ofd_precreate_objects lockdep warning<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: eb02aa76b5a91d5e1f0e0fb5e472ca6cf14e9a75</p> <p>With the patch , still got error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 282.413781] ====================================================== [ 282.414728] WARNING: possible circular locking dependency detected [ 282.415585] 4.18.0-debug #8 Tainted: G O --------- - - [ 282.416258] ------------------------------------------------------ [ 282.416893] ll_ost00_002/11461 is trying to acquire lock: [ 282.417510] 000000002a45f69d (&amp;mo-&gt;oo_sem/1){+.+.}, at: osd_write_lock+0x8a/0xb0 [osd_zfs] [ 282.418461] [ 282.418461] but task is already holding lock: [ 282.419094] 000000004fe6dea8 (&amp;mo-&gt;oo_sem){++++}, at: osd_write_lock+0x8a/0xb0 [osd_zfs] [ 282.420475] [ 282.420475] which lock already depends on the new lock. [ 282.420475] [ 282.421953] [ 282.421953] the existing dependency chain (in reverse order) is: [ 282.423119] [ 282.423119] -&gt; #1 (&amp;mo-&gt;oo_sem){++++}: [ 282.423898] __lock_acquire+0x3df/0xa70 [ 282.424291] lock_acquire+0x13a/0x370 [ 282.424820] down_write_nested+0x6f/0x120 [ 282.425497] osd_write_lock+0x8a/0xb0 [osd_zfs] [ 282.426504] __local_file_create+0xb37/0x2360 [obdclass] [ 282.427635] local_file_find_or_create+0x17a/0x190 [obdclass] [ 282.429418] lquota_disk_dir_find_create+0x18a/0x840 [lquota] [ 282.432498] qmt_device_prepare+0xd0/0x280 [lquota] [ 282.433505] mdt_quota_init+0x18ba/0x1c60 [mdt] [ 282.434509] mdt_init0+0x1239/0x1490 [mdt] [ 282.435114] mdt_device_alloc+0x115/0x170 [mdt] [ 282.435944] obd_setup+0x150/0x300 [obdclass] [ 282.436867] class_setup+0x4cd/0xa40 [obdclass] [ 282.437786] class_process_config+0x194c/0x2ea0 [obdclass] [ 282.438892] class_config_llog_handler+0xafa/0x1bd0 [obdclass] [ 282.440104] llog_process_thread+0xc82/0x1f80 [obdclass] [ 282.441045] llog_process_thread_daemonize+0xe4/0x140 [obdclass] [ 282.441787] kthread+0x190/0x1c0 [ 282.442178] ret_from_fork+0x3a/0x50 [ 282.442667] [ 282.442667] -&gt; #0 (&amp;mo-&gt;oo_sem/1){+.+.}: [ 282.443508] validate_chain.isra.26+0x9a5/0xce0 [ 282.444331] __lock_acquire+0x3df/0xa70 [ 282.445106] lock_acquire+0x13a/0x370 [ 282.445775] down_write_nested+0x6f/0x120 [ 282.446536] osd_write_lock+0x8a/0xb0 [osd_zfs] [ 282.447515] ofd_precreate_objects+0x1767/0x2480 [ofd] [ 282.448677] ofd_create_hdl+0x952/0x2150 [ofd] [ 282.449838] tgt_handle_request0+0xdf/0x890 [ptlrpc] [ 282.450870] tgt_request_handle+0x3ca/0x1aa0 [ptlrpc] [ 282.452084] ptlrpc_server_handle_request+0x634/0x1180 [ptlrpc] [ 282.453411] ptlrpc_main+0xd7f/0x1470 [ptlrpc] [ 282.454434] kthread+0x190/0x1c0 [ 282.455099] ret_from_fork+0x3a/0x50 [ 282.455816] [ 282.455816] other info that might help us debug this: [ 282.455816] [ 282.457096] Possible unsafe locking scenario: [ 282.457096] [ 282.458055] CPU0 CPU1 [ 282.458836] ---- ---- [ 282.459634] lock(&amp;mo-&gt;oo_sem); [ 282.460271] lock(&amp;mo-&gt;oo_sem/1); [ 282.461337] lock(&amp;mo-&gt;oo_sem); [ 282.462291] lock(&amp;mo-&gt;oo_sem/1); [ 282.462910] [ 282.462910] *** DEADLOCK *** [ 282.462910] [ 282.463974] 3 locks held by ll_ost00_002/11461: [ 282.464726] #0: 00000000e1797232 (&amp;m-&gt;ofd_lastid_rwsem){.+.+}, at: ofd_create_hdl+0x1a9/0x2150 [ofd] [ 282.466266] #1: 00000000f9816a9f (&amp;oseq-&gt;os_create_lock){+.+.}, at: ofd_create_hdl+0x364/0x2150 [ofd] [ 282.467782] #2: 000000004fe6dea8 (&amp;mo-&gt;oo_sem){++++}, at: osd_write_lock+0x8a/0xb0 [osd_zfs] [ 282.469186] [ 282.469186] stack backtrace: [ 282.470030] CPU: 2 PID: 11461 Comm: ll_ost00_002 Kdump: loaded Tainted: G O --------- - - 4.18.0-debug #8 [ 282.471895] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 282.473009] Call Trace: [ 282.473563] dump_stack+0x106/0x175 [ 282.474224] print_circular_bug.isra.25.cold.36+0x238/0x252 [ 282.475345] check_prev_add.constprop.28+0x607/0x6f0 [ 282.476472] ? __lock_acquire+0x410/0xa70 [ 282.477316] validate_chain.isra.26+0x9a5/0xce0 [ 282.478188] __lock_acquire+0x3df/0xa70 [ 282.478922] lock_acquire+0x13a/0x370 [ 282.479711] ? osd_write_lock+0x8a/0xb0 [osd_zfs] [ 282.480625] down_write_nested+0x6f/0x120 [ 282.481385] ? osd_write_lock+0x8a/0xb0 [osd_zfs] [ 282.482245] osd_write_lock+0x8a/0xb0 [osd_zfs] [ 282.483065] ofd_precreate_objects+0x1767/0x2480 [ofd] [ 282.484082] ofd_create_hdl+0x952/0x2150 [ofd] [ 282.485115] tgt_handle_request0+0xdf/0x890 [ptlrpc] [ 282.486177] tgt_request_handle+0x3ca/0x1aa0 [ptlrpc] [ 282.487285] ptlrpc_server_handle_request+0x634/0x1180 [ptlrpc] [ 282.488390] ? __wake_up+0x17/0x20 [ 282.489200] ptlrpc_main+0xd7f/0x1470 [ptlrpc] [ 282.490286] ? ptlrpc_register_service+0x14c0/0x14c0 [ptlrpc] [ 282.491460] kthread+0x190/0x1c0 [ 282.492007] ? kthread_create_worker+0x90/0x90 [ 282.492805] ret_from_fork+0x3a/0x50 [ 288.692309] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 192.168.200.130@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. </pre> </div></div> <p><a href="http://testing.linuxhacker.ru:3333/lustre-reports/1076/testresults/runtests-zfs-rhel8.0_x86_64-rhel8.0_x86_64/oleg30-server-console.txt" class="external-link" target="_blank" rel="nofollow noopener">http://testing.linuxhacker.ru:3333/lustre-reports/1076/testresults/runtests-zfs-rhel8.0_x86_64-rhel8.0_x86_64/oleg30-server-console.txt</a></p> <p>Alex, does this relate to your patch <a href="https://review.whamcloud.com/31293" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31293</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> ofd: take local locks within transaction</tt>"?</p> <p>Looks like 31293 changed the dt_write_lock order, so if we have 31293 landed, we don't need this patch anymore.</p> <p>what exact kernel options do you use to cause the warning?<br/> I'm using linux-4.18.0-32.el8 with the following options:<br/> CONFIG_LOCKDEP_SUPPORT=y<br/> CONFIG_LOCKDEP=y<br/> CONFIG_DEBUG_LOCKDEP=y<br/> but can't reproduce the warning.</p> <p>This is actually found by Oleg, see here:<br/> <a href="http://testing.linuxhacker.ru:3333/lustre-reports/dev/7/testresults/racer-zfs-rhel8.0_x86_64-rhel8.0_x86_64/oleg103-server-console.txt" class="external-link" target="_blank" rel="nofollow noopener">http://testing.linuxhacker.ru:3333/lustre-reports/dev/7/testresults/racer-zfs-rhel8.0_x86_64-rhel8.0_x86_64/oleg103-server-console.txt</a></p> <p>and here:<br/> <a href="http://testing.linuxhacker.ru:3333/lustre-reports/dev/7/" class="external-link" target="_blank" rel="nofollow noopener">http://testing.linuxhacker.ru:3333/lustre-reports/dev/7/</a></p> <p>hmm, took Oleg's .config for rhel8, but still can't get the warning.. keep trying.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35420/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35420/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12509" title="lockdep warning in ofd_precreate_objects()" class="issue-link" data-issue-key="LU-12509"><del>LU-12509</del></a> ofd: ofd_precreate_objects lockdep warning<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 697dcf4e87f2dbebe57f3ccb9c0b0962b89cf1b4</p> <p>Landed for 2.13</p> Related LU-12269 Development Rank 1|i00j7r: Rank (Obsolete) 9223372036854775807