https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10067 Lustre <p><a href="https://testing.hpdd.intel.com/test_sessions/00511056-db38-4995-a4f8-b35856fa09f7" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/00511056-db38-4995-a4f8-b35856fa09f7</a></p> <p>racer, test_1: test failed to respond and timed out</p> <p>From MDS console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>06:18:07:[19681.903127] LustreError: 22680:0:(mdt_handler.c:222:mdt_lock_pdo_mode()) ASSERTION( lh-&gt;mlh_pdo_mode == LCK_MINMODE ) failed: 06:18:07:[19681.908979] LustreError: 22680:0:(mdt_handler.c:222:mdt_lock_pdo_mode()) LBUG 06:18:07:[19681.912061] Pid: 22680, comm: mdt00_037 06:18:07:[19681.914860] 06:18:07:[19681.914860] Call Trace: 06:18:07:[19681.920079] [&lt;ffffffffc06817ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] 06:18:07:[19681.923062] [&lt;ffffffffc068183c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] 06:18:07:[19681.926146] [&lt;ffffffffc11b59db&gt;] mdt_object_local_lock+0xa4b/0xad0 [mdt] 06:18:07:[19681.928946] [&lt;ffffffffc11b56db&gt;] ? mdt_object_local_lock+0x74b/0xad0 [mdt] 06:18:07:[19681.931741] [&lt;ffffffffc11a5f10&gt;] ? mdt_blocking_ast+0x0/0x2e0 [mdt] 06:18:07:[19681.934380] [&lt;ffffffffc0e33770&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] 06:18:07:[19681.937367] [&lt;ffffffffc11b5ad0&gt;] mdt_object_lock_internal+0x70/0x330 [mdt] 06:18:07:[19681.940203] [&lt;ffffffffc11b5ad0&gt;] ? mdt_object_lock_internal+0x70/0x330 [mdt] 06:18:07:[19681.943129] [&lt;ffffffffc11b5db0&gt;] mdt_object_lock+0x20/0x30 [mdt] 06:18:07:[19681.945893] [&lt;ffffffffc11fa3f4&gt;] mdt_lock_objects_in_linkea+0x748/0xa68 [mdt] 06:18:07:[19681.948705] [&lt;ffffffffc11ca148&gt;] mdt_reint_migrate_internal.isra.38+0x8c8/0x16d0 [mdt] 06:18:07:[19681.951559] [&lt;ffffffffc0692002&gt;] ? cfs_hash_bd_from_key+0x32/0xb0 [libcfs] 06:18:07:[19681.954250] [&lt;ffffffffc0c41788&gt;] ? lu_object_put+0x148/0x3d0 [obdclass] 06:18:07:[19681.956946] [&lt;ffffffffc11cb1b5&gt;] mdt_reint_rename_or_migrate.isra.39+0x265/0x860 [mdt] 06:18:07:[19681.959450] [&lt;ffffffffc11c0181&gt;] ? mdt_root_squash+0x21/0x430 [mdt] 06:18:07:[19681.961948] [&lt;ffffffff8132c212&gt;] ? strlcpy+0x42/0x60 06:18:07:[19681.964254] [&lt;ffffffffc11cb7c0&gt;] mdt_reint_migrate+0x10/0x20 [mdt] 06:18:07:[19681.966606] [&lt;ffffffffc11cf790&gt;] mdt_reint_rec+0x80/0x210 [mdt] 06:18:07:[19681.969134] [&lt;ffffffffc11b131b&gt;] mdt_reint_internal+0x5fb/0x9c0 [mdt] 06:18:07:[19681.971704] [&lt;ffffffffc11bcda7&gt;] mdt_reint+0x67/0x140 [mdt] 06:18:07:[19681.974204] [&lt;ffffffffc0ecc225&gt;] tgt_request_handle+0x925/0x1370 [ptlrpc] 06:18:07:[19681.976507] [&lt;ffffffffc0e750c6&gt;] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] 06:18:07:[19681.979113] [&lt;ffffffff810ba588&gt;] ? __wake_up_common+0x58/0x90 06:18:07:[19681.981580] [&lt;ffffffffc0e78862&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] 06:18:07:[19681.983929] [&lt;ffffffff81029557&gt;] ? __switch_to+0xd7/0x510 06:18:07:[19681.986432] [&lt;ffffffff816a8f00&gt;] ? __schedule+0x330/0x8b0 06:18:07:[19681.988691] [&lt;ffffffffc0e77dd0&gt;] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] 06:18:07:[19681.990751] [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 06:18:07:[19681.993085] [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 06:18:07:[19681.995015] [&lt;ffffffff816b4f18&gt;] ret_from_fork+0x58/0x90 06:18:07:[19681.997259] [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 06:18:07:[19681.999174] 06:18:07:[19682.000888] Kernel panic - not syncing: LBUG 06:18:07:[19682.001870] CPU: 1 PID: 22680 Comm: mdt00_037 Tainted: P OE ------------ 3.10.0-693.1.1.el7_lustre.x86_64 #1 06:18:07:[19682.001870] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 06:18:07:[19682.001870] ffff8800664d1f00 0000000068d88604 ffff88004cf8b898 ffffffff816a3d6d 06:18:07:[19682.001870] ffff88004cf8b918 ffffffff8169dc54 ffffffff00000008 ffff88004cf8b928 06:18:07:[19682.001870] ffff88004cf8b8c8 0000000068d88604 0000000068d88604 ffff88007fd0f8b8 06:18:07:[19682.001870] Call Trace: 06:18:07:[19682.001870] [&lt;ffffffff816a3d6d&gt;] dump_stack+0x19/0x1b 06:18:07:[19682.001870] [&lt;ffffffff8169dc54&gt;] panic+0xe8/0x20d 06:18:07:[19682.001870] [&lt;ffffffffc0681854&gt;] lbug_with_loc+0x64/0xb0 [libcfs] 06:18:07:[19682.001870] [&lt;ffffffffc11b59db&gt;] mdt_object_local_lock+0xa4b/0xad0 [mdt] 06:18:07:[19682.001870] [&lt;ffffffffc11b56db&gt;] ? mdt_object_local_lock+0x74b/0xad0 [mdt] 06:18:07:[19682.001870] [&lt;ffffffffc11a5f10&gt;] ? mdt_obd_reconnect+0x1d0/0x1d0 [mdt] 06:18:07:[19682.001870] [&lt;ffffffffc0e33770&gt;] ? ldlm_expired_completion_wait+0x240/0x240 [ptlrpc] 06:18:07:[19682.001870] [&lt;ffffffffc11b5ad0&gt;] mdt_object_lock_internal+0x70/0x330 [mdt] 06:18:07:[19682.001870] [&lt;ffffffffc11b5ad0&gt;] ? mdt_object_lock_internal+0x70/0x330 [mdt] 06:18:07:[19682.001870] [&lt;ffffffffc11b5db0&gt;] mdt_object_lock+0x20/0x30 [mdt] 06:18:07:[19682.001870] [&lt;ffffffffc11fa3f4&gt;] mdt_lock_objects_in_linkea+0x748/0xa68 [mdt] 06:18:07:[19682.001870] [&lt;ffffffffc11ca148&gt;] mdt_reint_migrate_internal.isra.38+0x8c8/0x16d0 [mdt] 06:18:07:[19682.001870] [&lt;ffffffffc0692002&gt;] ? cfs_hash_bd_from_key+0x32/0xb0 [libcfs] 06:18:07:[19682.001870] [&lt;ffffffffc0c41788&gt;] ? lu_object_put+0x148/0x3d0 [obdclass] 06:18:07:[19682.001870] [&lt;ffffffffc11cb1b5&gt;] mdt_reint_rename_or_migrate.isra.39+0x265/0x860 [mdt] 06:18:07:[19682.001870] [&lt;ffffffffc11c0181&gt;] ? mdt_root_squash+0x21/0x430 [mdt] 06:18:07:[19682.001870] [&lt;ffffffff8132c212&gt;] ? strlcpy+0x42/0x60 06:18:07:[19682.001870] [&lt;ffffffffc11cb7c0&gt;] mdt_reint_migrate+0x10/0x20 [mdt] 06:18:07:[19682.001870] [&lt;ffffffffc11cf790&gt;] mdt_reint_rec+0x80/0x210 [mdt] 06:18:07:[19682.001870] [&lt;ffffffffc11b131b&gt;] mdt_reint_internal+0x5fb/0x9c0 [mdt] 06:18:07:[19682.001870] [&lt;ffffffffc11bcda7&gt;] mdt_reint+0x67/0x140 [mdt] 06:18:07:[19682.001870] [&lt;ffffffffc0ecc225&gt;] tgt_request_handle+0x925/0x1370 [ptlrpc] 06:18:07:[19682.001870] [&lt;ffffffffc0e750c6&gt;] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] 06:18:07:[19682.001870] [&lt;ffffffff810ba588&gt;] ? __wake_up_common+0x58/0x90 06:18:07:[19682.001870] [&lt;ffffffffc0e78862&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] 06:18:07:[19682.001870] [&lt;ffffffff81029557&gt;] ? __switch_to+0xd7/0x510 06:18:07:[19682.001870] [&lt;ffffffff816a8f00&gt;] ? __schedule+0x330/0x8b0 06:18:07:[19682.001870] [&lt;ffffffffc0e77dd0&gt;] ? ptlrpc_register_service+0xe80/0xe80 [ptlrpc] 06:18:07:[19682.001870] [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 06:18:07:[19682.001870] [&lt;ffffffff810b08c0&gt;] ? insert_kthread_work+0x40/0x40 06:18:07:[19682.001870] [&lt;ffffffff816b4f18&gt;] ret_from_fork+0x58/0x90 06:18:07:[19682.001870] [&lt;ffffffff810b08c0&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> trevis, full DNE<br/> servers: el7.4, zfs, branch master, v2.10.53.1, b3642<br/> clients: el7.4, branch master, v2.10.53.1, b3642<br/> LU-10067

LBUG mdt_handler.c:222:mdt_lock_pdo_mode()

Bug Major Resolved Fixed Hongchao Zhang James Casper Tue, 3 Oct 2017 19:53:37 +0000 Thu, 9 Nov 2017 20:34:01 +0000 Thu, 9 Nov 2017 20:33:53 +0000 Lustre 2.11.0 Lustre 2.11.0 0 4 <p>In the test_log, we see</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: trevis-6vm4 /usr/sbin/lctl snapshot_create -F lustre -n lss_27457 trevis-6vm4: Warning: Permanently added 'trevis-6vm4' (ECDSA) to the list of known hosts. trevis-6vm4: Warning: Permanently added 'trevis-6vm3' (ECDSA) to the list of known hosts. CMD: trevis-6vm4 /usr/sbin/lctl snapshot_list -F lustre 2&gt;/dev/null CMD: trevis-6vm4 /usr/sbin/lctl snapshot_list -F lustre 2&gt;/dev/null trevis-6vm4: Warning: Permanently added 'trevis-6vm8' (ECDSA) to the list of known hosts. CMD: trevis-6vm4 /usr/sbin/lctl snapshot_create -F lustre -n lss_29957 CMD: trevis-6vm4 /usr/sbin/lctl snapshot_list -F lustre 2&gt;/dev/null CMD: trevis-6vm4 /usr/sbin/lctl snapshot_destroy -F lustre -n lss_29957 -f CMD: trevis-6vm4 /usr/sbin/lctl snapshot_create -F lustre -n lss_12954 trevis-6vm4: Can't lock the snapshot config file /etc/ldev.conf (2): Resource temporarily unavailable CMD: trevis-6vm4 /usr/sbin/lctl snapshot_create -F lustre -n lss_2287 trevis-6vm4: Can't lock the snapshot config file /etc/ldev.conf (2): Resource temporarily unavailable CMD: trevis-6vm4 /usr/sbin/lctl snapshot_create -F lustre -n lss_14963 trevis-6vm4: Can't lock the snapshot config file /etc/ldev.conf (2): Resource temporarily unavailable CMD: trevis-6vm4 /usr/sbin/lctl snapshot_create -F lustre -n lss_12585 trevis-6vm4: Can't lock the snapshot config file /etc/ldev.conf (2): Resource temporarily unavailable trevis-6vm4: Connection closed by UNKNOWN port 65535 … </pre> </div></div> <p>Although racer test_1 hangs frequently during autotesting and does fail with “Resource temporarily unavailable” messages in the test log, we don’t see the LBUG in the MDS logs in previous runs of racer.</p> <p>Hongchao, </p> <p>Would you please comment on this issue?</p> <p>James</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29597" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29597</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10067" title="LBUG mdt_handler.c:222:mdt_lock_pdo_mode()" class="issue-link" data-issue-key="LU-10067"><del>LU-10067</del></a> mdt: reinit lock when fail to try lock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 24bd3ed58b9fd6a80f5e9c000119c15076d38053</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/29597/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29597/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10067" title="LBUG mdt_handler.c:222:mdt_lock_pdo_mode()" class="issue-link" data-issue-key="LU-10067"><del>LU-10067</del></a> mdt: reinit lock when fail to try lock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 37e4bcaad4b4cd1f539c257f7424850e51d685c1</p> <p>Landed for 2.11</p> Development Rank 1|hzzl7b: Rank (Obsolete) 9223372036854775807 Severity