https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4840 Lustre <p>While migrating a file with "lfs migrate", if a process tries to truncate the file, both lfs migrate and truncating processes will deadlock.</p> <p>This will result in both processes never finishing (unless it is killed) and watchdog messages saying that the processes did not progress for the last XXX seconds.</p> <p>Here is a reproducer:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@lustre24cli ~]# cat reproducer.sh #!/bin/sh FS=/test FILE=${FS}/file rm -f ${FILE} # Create a file on OST 1 of size 512M lfs setstripe -o 1 -c 1 ${FILE} dd if=/dev/zero of=${FILE} bs=1M count=512 echo 3 &gt; /proc/sys/vm/drop_caches # Launch a migrate to OST 0 and a bit later open it for write lfs migrate -i 0 --block ${FILE} &amp; sleep 2 dd if=/dev/zero of=${FILE} bs=1M count=512 </pre> </div></div> <p>Once the last dd tries to open the file, both lfs and dd processes stay forever with this stack:</p> <p>lfs stack:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[&lt;ffffffff8128e864&gt;] call_rwsem_down_read_failed+0x14/0x30 [&lt;ffffffffa08d98dd&gt;] ll_file_io_generic+0x29d/0x600 [lustre] [&lt;ffffffffa08d9d7f&gt;] ll_file_aio_read+0x13f/0x2c0 [lustre] [&lt;ffffffffa08da61c&gt;] ll_file_read+0x16c/0x2a0 [lustre] [&lt;ffffffff811896b5&gt;] vfs_read+0xb5/0x1a0 [&lt;ffffffff811897f1&gt;] sys_read+0x51/0x90 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> <p>dd stack:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[&lt;ffffffffa03436fe&gt;] cfs_waitq_wait+0xe/0x10 [libcfs] [&lt;ffffffffa04779fa&gt;] cl_lock_state_wait+0x1aa/0x320 [obdclass] [&lt;ffffffffa04781eb&gt;] cl_enqueue_locked+0x15b/0x1f0 [obdclass] [&lt;ffffffffa0478d6e&gt;] cl_lock_request+0x7e/0x270 [obdclass] [&lt;ffffffffa047e00c&gt;] cl_io_lock+0x3cc/0x560 [obdclass] [&lt;ffffffffa047e242&gt;] cl_io_loop+0xa2/0x1b0 [obdclass] [&lt;ffffffffa092a8c8&gt;] cl_setattr_ost+0x208/0x2c0 [lustre] [&lt;ffffffffa08f8a0e&gt;] ll_setattr_raw+0x9ce/0x1000 [lustre] [&lt;ffffffffa08f909b&gt;] ll_setattr+0x5b/0xf0 [lustre] [&lt;ffffffff811a7348&gt;] notify_change+0x168/0x340 [&lt;ffffffff81187074&gt;] do_truncate+0x64/0xa0 [&lt;ffffffff8119bcc1&gt;] do_filp_open+0x861/0xd20 [&lt;ffffffff81185d39&gt;] do_sys_open+0x69/0x140 [&lt;ffffffff81185e50&gt;] sys_open+0x20/0x30 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> LU-4840

Deadlock when truncating file during lfs migrate

Bug Critical Resolved Fixed Zhenyu Xu Patrick Valentin cea Mon, 31 Mar 2014 13:41:52 +0000 Sun, 9 Oct 2016 13:52:48 +0000 Mon, 14 Sep 2015 17:32:22 +0000 Lustre 2.4.2 Lustre 2.8.0 0 20 <p>Bobijam</p> <p>Could you please comment?</p> <p>thanks</p> <p>Peter</p> <p>So I tried the script on master and it also happens there.</p> <p>I can also reproduce it on master. The exact details of the lfs and dd task stacks are a little different but the deadlock is still there.</p> <p>It relates to the lfs migrate implementation (<a href="https://jira.whamcloud.com/browse/LU-2445" title="add &quot;lfs migrate&quot; support" class="issue-link" data-issue-key="LU-2445"><del>LU-2445</del></a>), lfs migrate first takes a group lock to limite concurrent OST access from other clients, then vfs reads and writes the file to migrate data from OSTs to OSTs, at last drops the group lock.</p> <p>In this case, lfs migrate gets the group lock, then the other client from the same node tries to truncate the file which takes the inode truncate semaphore (lli_trunc_sem) and enqueues OST lock and waits for it to be granted.</p> <p>The lfs migrate comes to the read phase, it tries get the truncate semaphore as well.</p> <p>The other client cannot get its OST lock granted, since OST cannot revoke it from the lfs migrate process. Deadlock happens.</p> <p>It's a good chance to reimplement migration with open lease.</p> <p>Aurelien and JC, do you have any inputs on this?</p> <p>I see no objection to replacing grouplock-based code with an open-lease mechanism. Current code was using this lock because open lease did not exist as that time and we were advised to do this this way as they was no mechanism to protect from concurrent access. I think it will be cleaner.</p> <p>That's true, Aurelien. As long as the migration was implemented by CEA, would it be possible for CEA to pick it up again to reimplement it with open lease?</p> <p>Jinshan</p> <p>OK, CEA will do it. <br/> Could you confirm some behaviour of open lease to be sure we are in line. There is only exclusive open lease for now? When lease are revoked by concurrent access?</p> <p>Here's a patch aimed to solve it:</p> <p><a href="http://review.whamcloud.com/10013" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10013</a></p> <p>Use file lease to implement migration.</p> <p>Please check the attachment for the implementation of migration.</p> <p>The procedure is a little bit like HSM release where close and swap layout should be an atomic operation. Also, you need to check if the lease is valid in the middle of data copying periodically therefore data copying can abort if the file is being opened by others.</p> <p>Please take a look at it and I'll be happy to answer questions.</p> <p>Thanks Jinshan.</p> <p>I still have a question regarding when to check data version. In which case could it fail if we both get and check it under the file lease? Also swap layout will perform a dataversion check. Is the following sequence flawed?</p> <ul> <li>open source file</li> <li>get lease</li> <li>open volatile</li> <li>get data version</li> <li>copy file content</li> <li>put lease</li> <li>swap layout (pass in data version for check)</li> <li>close volatile</li> <li>close source file</li> </ul> <p>We have to make `put lease', `swap layout', and `close source file' in one atomic operation. Otherwise, if the source is opened for writing after `put lease' and generate some dirty pages, it will produce data corruption.</p> <p>Cf. the new patchset.</p> <p>Indeed, I do see the race window between llapi_lease_put() and llapi_fswap_layouts() but I can't see any userland API that would allow us to get rid of it. Am I missing something? You're stressing the need to do operations atomically, do you have something in mind, like making the SWAP_LAYOUT ioctl lease-aware?</p> <p>No, there is no API in user space ready to use.</p> <blockquote> <p>You're stressing the need to do operations atomically, do you have something in mind, like making the SWAP_LAYOUT ioctl lease-aware?</p></blockquote> <p>Yes, similar. Instead of making SWAP_LAYOUT lease-aware, what we need is to make lease SWAP-LAYOUT aware.</p> <p>When we release a lease, a CLOSE RPC will be sent to MDT. We're going to pack the FID of volatile file into the RPC, with a special bias (similar to MDS_HSM_RELEASE, please check ll_close_inode_openhandle() for 'op_data-&gt;op_bias |= MDS_HSM_RELEASE', and mdt_hsm_release()), for example, MDS_CLOSE_SWAP_LAYOUT to tell MDT that we want to unlock the release and swap the layout. We're going to extend `struct close_data' to include those information. And the ioctl() of LL_IOC_SET_LEASE with F_UNLCK will be extended as well.</p> <p>It's totally fine for me to write a design but I think you have understood the problem and are capable of doing it yourself, based on your questions above.</p> <p>It took me a little while but I buckled down and just pushed a patch that follows the guidelines you gave me. In userland I've extended the swap layouts ioctl to leverage the existing API and preserve compatibility (I was reluctant to change the parameters of the set lease ioctl). In kernel land it's very close to hsm_release. Hope this is fine conceptually.</p> <p>Any update on this?</p> <p>I'll look at the patch this week, really sorry for delay.</p> <p>I tested rev 12 of this patch on top of head of tree. Nothing fancy:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>./llmount.sh cd /mnt/lustre cp /bin/ls . ~/lustre-release/lustre/utils/lfs migrate -o 0 ls </pre> </div></div> <p>After lfs decided to output the content of ls (???) to stdout, the node proceeded to not like me and crash.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; bt PID: 6463 TASK: ffff88007ac1b500 CPU: 1 COMMAND: "lfs" #0 [ffff880012b89820] machine_kexec at ffffffff81038f3b #1 [ffff880012b89880] crash_kexec at ffffffff810c5b62 #2 [ffff880012b89950] oops_end at ffffffff8152c8a0 #3 [ffff880012b89980] no_context at ffffffff8104a00b #4 [ffff880012b899d0] __bad_area_nosemaphore at ffffffff8104a295 #5 [ffff880012b89a20] bad_area at ffffffff8104a3be #6 [ffff880012b89a50] __do_page_fault at ffffffff8104ab6f #7 [ffff880012b89b70] do_page_fault at ffffffff8152e7ee #8 [ffff880012b89ba0] page_fault at ffffffff8152bba5 [exception RIP: ll_mdscapa_get+65] RIP: ffffffffa08cdc91 RSP: ffff880012b89c58 RFLAGS: 00010286 RAX: 0000000000000000 RBX: ffff88000ccf9200 RCX: 0000000000000000 RDX: 0000000000000001 RSI: ffff88001fb0e138 RDI: ffff88007c8303d8 RBP: ffff880012b89c68 R8: 0000000000000000 R9: 0000000000000000 R10: ffff88000ccf9200 R11: 0000000000000200 R12: ffff88007c8303d8 R13: ffff88007c8303d8 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #9 [ffff880012b89c70] ll_prep_md_op_data at ffffffffa08a0e45 [lustre] #10 [ffff880012b89ce0] ll_prepare_close at ffffffffa0881098 [lustre] #11 [ffff880012b89d30] ll_close_inode_openhandle at ffffffffa088a6f2 [lustre] #12 [ffff880012b89db0] ll_file_ioctl at ffffffffa0894fc8 [lustre] #13 [ffff880012b89e60] vfs_ioctl at ffffffff8119e422 #14 [ffff880012b89ea0] do_vfs_ioctl at ffffffff8119e8ea #15 [ffff880012b89f30] sys_ioctl at ffffffff8119eb41 #16 [ffff880012b89f80] system_call_fastpath at ffffffff8100b072 RIP: 0000003a522e0b37 RSP: 00007fff083e48d0 RFLAGS: 00010292 RAX: 0000000000000010 RBX: ffffffff8100b072 RCX: 00000000545ab5fe RDX: 00007fff083e48c0 RSI: 00000000402066db RDI: 0000000000000003 RBP: 0000000000000001 R8: 000000002e1aa2e5 R9: 0000000000000010 R10: 0000000000000000 R11: 0000000000000246 R12: 000000002e1aa2e5 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003 ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b </pre> </div></div> <p>Without this patch, I still get the junk output, but not the crash.</p> <p>Thanks Frank. Null pointer (sbi) dereference in ll_mdscapa_get(). Fixed in patchset #13. The file content ending up in the console remains unexplained to me so far. You said it was present before, is there an open ticket for that?</p> <p>Thanks. The fix works, and I can migrate a file between osts now.</p> <p>Regarding the junk output, I found the bug in llapi_file_open_param(). I'll submit a patch soon.</p> <p>Follow-up patch, fixes numerous issues with the first one: <a href="http://review.whamcloud.com/#/c/12616/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12616/</a><br/> Both patches can be merged if need be, just let me know what's preferred.</p> <p>Henri, I agree with Frank that we should not be landing a patch with significant known defects,since this would break the code for anyone testing this. Please merge the patches. </p> <p>One advantage to the old approach of using group locks for migration was that it was theoretically possible to create a version of lfs migrate that could migrate a file in parallel using multiple clients. Is this still possible with the new approach?</p> <p>Yes, it is still possible. Though an early version of the patch removed grouplock-protected migration, it has now been re-introduced. Migration can be either grouplock-protected and blocking (as before), or based on exclusive open and non-blocking (would safely abort if a concurrent process opens the file). We would need file leases to provide a notion of "group" to be able to implement non-blocking parallel migration too.</p> <p>Thanks for the response, Henri. I'm glad to hear the group lock option was retained, and I see the deadlock with truncate was resolved as well.</p> <p>Just to draw attention to my comment in gerrit.<br/> The latest patch still deadlocks in racer on mds, also seems to be leaking ost locks at times?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>8.832781] LNet: Service thread pid 26108 was inactive for 62.00s. The thread mig 8.833657] Pid: 26108, comm: mdt00_007 8.833906] 8.833907] Call Trace: 8.834350] [&lt;ffffffffa13be503&gt;] ? _ldlm_lock_debug+0x2e3/0x670 [ptlrpc] 8.834649] [&lt;ffffffff81516894&gt;] ? _spin_lock_irqsave+0x24/0x30 8.834934] [&lt;ffffffff81514231&gt;] schedule_timeout+0x191/0x2e0 8.835310] [&lt;ffffffff81081e50&gt;] ? process_timeout+0x0/0x10 8.835629] [&lt;ffffffffa13decf0&gt;] ? ldlm_expired_completion_wait+0x0/0x370 [ptlrpc 8.836070] [&lt;ffffffffa13e3841&gt;] ldlm_completion_ast+0x5e1/0x9b0 [ptlrpc] 8.836267] [&lt;ffffffff8105de00&gt;] ? default_wake_function+0x0/0x20 8.836475] [&lt;ffffffffa13e2c8e&gt;] ldlm_cli_enqueue_local+0x21e/0x7f0 [ptlrpc] 8.836735] [&lt;ffffffffa13e3260&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] 8.836950] [&lt;ffffffffa056bbc0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] 8.837190] [&lt;ffffffffa0574805&gt;] mdt_object_local_lock+0x3c5/0xa80 [mdt] 8.837391] [&lt;ffffffffa056bbc0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] 8.837638] [&lt;ffffffffa13e3260&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] 8.837852] [&lt;ffffffffa0575245&gt;] mdt_object_lock_internal+0x65/0x360 [mdt] 8.838074] [&lt;ffffffffa0575604&gt;] mdt_object_lock+0x14/0x20 [mdt] 8.838265] [&lt;ffffffffa059328a&gt;] mdt_reint_unlink+0x20a/0x10c0 [mdt] 8.838485] [&lt;ffffffffa120fa80&gt;] ? lu_ucred+0x20/0x30 [obdclass] 8.838676] [&lt;ffffffffa056ad25&gt;] ? mdt_ucred+0x15/0x20 [mdt] 8.838898] [&lt;ffffffffa05858bc&gt;] ? mdt_root_squash+0x2c/0x3f0 [mdt] 8.839232] [&lt;ffffffffa1434e02&gt;] ? __req_capsule_get+0x162/0x6d0 [ptlrpc] 8.839566] [&lt;ffffffffa0589aad&gt;] mdt_reint_rec+0x5d/0x200 [mdt] 8.839881] [&lt;ffffffffa056f5ab&gt;] mdt_reint_internal+0x4cb/0x7a0 [mdt] 8.840205] [&lt;ffffffffa056fe0b&gt;] mdt_reint+0x6b/0x120 [mdt] 8.840550] [&lt;ffffffffa146e85e&gt;] tgt_request_handle+0x8be/0x1000 [ptlrpc] 8.840915] [&lt;ffffffffa141fd64&gt;] ptlrpc_main+0xdf4/0x1940 [ptlrpc] 8.841304] [&lt;ffffffffa141ef70&gt;] ? ptlrpc_main+0x0/0x1940 [ptlrpc] 8.841624] [&lt;ffffffff81098c06&gt;] kthread+0x96/0xa0 8.841917] [&lt;ffffffff8100c24a&gt;] child_rip+0xa/0x20 8.842199] [&lt;ffffffff81098b70&gt;] ? kthread+0x0/0xa0 8.842492] [&lt;ffffffff8100c240&gt;] ? child_rip+0x0/0x20 8.842780] 8.842997] LustreError: dumping log to /tmp/lustre-log.1420492523.26108 9.015282] Pid: 9643, comm: mdt00_006 9.015565] 9.015566] Call Trace: 9.016033] [&lt;ffffffffa13be503&gt;] ? _ldlm_lock_debug+0x2e3/0x670 [ptlrpc] 9.017088] [&lt;ffffffff81516894&gt;] ? _spin_lock_irqsave+0x24/0x30 9.017352] [&lt;ffffffff81514231&gt;] schedule_timeout+0x191/0x2e0 9.017687] [&lt;ffffffff81081e50&gt;] ? process_timeout+0x0/0x10 9.018074] [&lt;ffffffffa13decf0&gt;] ? ldlm_expired_completion_wait+0x0/0x370 [ptlrpc 9.018593] [&lt;ffffffffa13e3841&gt;] ldlm_completion_ast+0x5e1/0x9b0 [ptlrpc] 9.018880] [&lt;ffffffff8105de00&gt;] ? default_wake_function+0x0/0x20 9.019196] [&lt;ffffffffa13e2c8e&gt;] ldlm_cli_enqueue_local+0x21e/0x7f0 [ptlrpc] 9.019615] [&lt;ffffffffa13e3260&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] 9.019940] [&lt;ffffffffa056bbc0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] 9.020220] [&lt;ffffffffa05745fb&gt;] mdt_object_local_lock+0x1bb/0xa80 [mdt] 9.020558] [&lt;ffffffffa056bbc0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] 9.020860] [&lt;ffffffffa13e3260&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] 9.021150] [&lt;ffffffffa0575245&gt;] mdt_object_lock_internal+0x65/0x360 [mdt] 9.021477] [&lt;ffffffffa0575604&gt;] mdt_object_lock+0x14/0x20 [mdt] 9.022987] [&lt;ffffffffa05806fc&gt;] mdt_getattr_name_lock+0x103c/0x1ab0 [mdt] 9.023297] [&lt;ffffffff8128863a&gt;] ? strlcpy+0x4a/0x60 9.023573] [&lt;ffffffffa140ff84&gt;] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc] 9.023888] [&lt;ffffffffa14116d0&gt;] ? lustre_swab_ldlm_reply+0x0/0x40 [ptlrpc] 9.024182] [&lt;ffffffffa0581692&gt;] mdt_intent_getattr+0x292/0x470 [mdt] 9.024493] [&lt;ffffffffa056e064&gt;] mdt_intent_policy+0x494/0xce0 [mdt] 9.024789] [&lt;ffffffffa13c305f&gt;] ldlm_lock_enqueue+0x12f/0x950 [ptlrpc] 9.025140] [&lt;ffffffffa10b9201&gt;] ? cfs_hash_for_each_enter+0x1/0xa0 [libcfs] 9.025454] [&lt;ffffffffa13eedeb&gt;] ldlm_handle_enqueue0+0x51b/0x13e0 [ptlrpc] 9.025747] [&lt;ffffffffa146dc72&gt;] tgt_enqueue+0x62/0x1d0 [ptlrpc] 9.026616] [&lt;ffffffffa146e85e&gt;] tgt_request_handle+0x8be/0x1000 [ptlrpc] 9.026970] [&lt;ffffffffa141fd64&gt;] ptlrpc_main+0xdf4/0x1940 [ptlrpc] 9.027313] [&lt;ffffffffa141ef70&gt;] ? ptlrpc_main+0x0/0x1940 [ptlrpc] 9.027622] [&lt;ffffffff81098c06&gt;] kthread+0x96/0xa0 9.028084] [&lt;ffffffff8100c24a&gt;] child_rip+0xa/0x20 9.029247] Pid: 6818, comm: mdt01_002 9.029453] 9.029453] Call Trace: 9.029739] [&lt;ffffffffa13be503&gt;] ? _ldlm_lock_debug+0x2e3/0x670 [ptlrpc] 9.029980] [&lt;ffffffff81516894&gt;] ? _spin_lock_irqsave+0x24/0x30 9.030164] [&lt;ffffffff81514231&gt;] schedule_timeout+0x191/0x2e0 9.030341] [&lt;ffffffff81081e50&gt;] ? process_timeout+0x0/0x10 9.030579] [&lt;ffffffffa13decf0&gt;] ? ldlm_expired_completion_wait+0x0/0x370 [ptlrpc 9.031058] [&lt;ffffffffa13e3841&gt;] ldlm_completion_ast+0x5e1/0x9b0 [ptlrpc] 9.031335] [&lt;ffffffff8105de00&gt;] ? default_wake_function+0x0/0x20 9.031630] [&lt;ffffffffa13e2c8e&gt;] ldlm_cli_enqueue_local+0x21e/0x7f0 [ptlrpc] 9.032198] [&lt;ffffffffa13e3260&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] 9.032491] [&lt;ffffffffa056bbc0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] 9.032773] [&lt;ffffffffa05745fb&gt;] mdt_object_local_lock+0x1bb/0xa80 [mdt] 9.033094] [&lt;ffffffffa056bbc0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] 9.033387] [&lt;ffffffffa13e3260&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] 9.033675] [&lt;ffffffffa0575245&gt;] mdt_object_lock_internal+0x65/0x360 [mdt] 9.033988] [&lt;ffffffffa0575604&gt;] mdt_object_lock+0x14/0x20 [mdt] 9.034264] [&lt;ffffffffa05806fc&gt;] mdt_getattr_name_lock+0x103c/0x1ab0 [mdt] 9.034544] [&lt;ffffffff8128863a&gt;] ? strlcpy+0x4a/0x60 9.034891] [&lt;ffffffffa140ff84&gt;] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc] 9.035203] [&lt;ffffffffa14116d0&gt;] ? lustre_swab_ldlm_reply+0x0/0x40 [ptlrpc] 9.035495] [&lt;ffffffffa0581692&gt;] mdt_intent_getattr+0x292/0x470 [mdt] 9.035774] [&lt;ffffffffa056e064&gt;] mdt_intent_policy+0x494/0xce0 [mdt] 9.036192] [&lt;ffffffffa13c305f&gt;] ldlm_lock_enqueue+0x12f/0x950 [ptlrpc] 9.036479] [&lt;ffffffffa10b9201&gt;] ? cfs_hash_for_each_enter+0x1/0xa0 [libcfs] 9.036788] [&lt;ffffffffa13eedeb&gt;] ldlm_handle_enqueue0+0x51b/0x13e0 [ptlrpc] 9.037136] [&lt;ffffffffa146dc72&gt;] tgt_enqueue+0x62/0x1d0 [ptlrpc] 9.037429] [&lt;ffffffffa146e85e&gt;] tgt_request_handle+0x8be/0x1000 [ptlrpc] 9.037735] [&lt;ffffffffa141fd64&gt;] ptlrpc_main+0xdf4/0x1940 [ptlrpc] 9.039938] [&lt;ffffffffa141ef70&gt;] ? ptlrpc_main+0x0/0x1940 [ptlrpc] 9.040213] [&lt;ffffffff81098c06&gt;] kthread+0x96/0xa0 9.040456] [&lt;ffffffff8100c24a&gt;] child_rip+0xa/0x20 9.040707] [&lt;ffffffff81098b70&gt;] ? kthread+0x0/0xa0 9.040980] [&lt;ffffffff8100c240&gt;] ? child_rip+0x0/0x20 9.041230] 9.041415] Pid: 6815, comm: mdt00_002 9.041637] 9.041638] Call Trace: 9.042070] [&lt;ffffffffa13be503&gt;] ? _ldlm_lock_debug+0x2e3/0x670 [ptlrpc] 9.042351] [&lt;ffffffff81516894&gt;] ? _spin_lock_irqsave+0x24/0x30 9.042615] [&lt;ffffffff81514231&gt;] schedule_timeout+0x191/0x2e0 9.042890] [&lt;ffffffff81081e50&gt;] ? process_timeout+0x0/0x10 9.043183] [&lt;ffffffffa13decf0&gt;] ? ldlm_expired_completion_wait+0x0/0x370 [ptlrpc 9.043656] [&lt;ffffffffa13e3841&gt;] ldlm_completion_ast+0x5e1/0x9b0 [ptlrpc] 9.044075] [&lt;ffffffff8105de00&gt;] ? default_wake_function+0x0/0x20 9.044366] [&lt;ffffffffa13e2c8e&gt;] ldlm_cli_enqueue_local+0x21e/0x7f0 [ptlrpc] 9.044677] [&lt;ffffffffa13e3260&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] 9.045145] [&lt;ffffffffa056bbc0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] 9.045420] [&lt;ffffffffa05745fb&gt;] mdt_object_local_lock+0x1bb/0xa80 [mdt] 9.045705] [&lt;ffffffffa056bbc0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] 9.046019] [&lt;ffffffffa13e3260&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] 9.046309] [&lt;ffffffffa0575245&gt;] mdt_object_lock_internal+0x65/0x360 [mdt] 9.046596] [&lt;ffffffffa0575604&gt;] mdt_object_lock+0x14/0x20 [mdt] 9.046873] [&lt;ffffffffa05806fc&gt;] mdt_getattr_name_lock+0x103c/0x1ab0 [mdt] 9.047152] [&lt;ffffffff8128863a&gt;] ? strlcpy+0x4a/0x60 9.047427] [&lt;ffffffffa140ff84&gt;] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc] 9.047736] [&lt;ffffffffa14116d0&gt;] ? lustre_swab_ldlm_reply+0x0/0x40 [ptlrpc] 9.048031] [&lt;ffffffffa0581692&gt;] mdt_intent_getattr+0x292/0x470 [mdt] 9.048308] [&lt;ffffffffa056e064&gt;] mdt_intent_policy+0x494/0xce0 [mdt] 9.048604] [&lt;ffffffffa13c305f&gt;] ldlm_lock_enqueue+0x12f/0x950 [ptlrpc] 9.048918] [&lt;ffffffffa10b9201&gt;] ? cfs_hash_for_each_enter+0x1/0xa0 [libcfs] 9.049232] [&lt;ffffffffa13eedeb&gt;] ldlm_handle_enqueue0+0x51b/0x13e0 [ptlrpc] 9.049544] [&lt;ffffffffa146dc72&gt;] tgt_enqueue+0x62/0x1d0 [ptlrpc] 9.049852] [&lt;ffffffffa146e85e&gt;] tgt_request_handle+0x8be/0x1000 [ptlrpc] 9.050160] [&lt;ffffffffa141fd64&gt;] ptlrpc_main+0xdf4/0x1940 [ptlrpc] 9.050453] [&lt;ffffffffa141ef70&gt;] ? ptlrpc_main+0x0/0x1940 [ptlrpc] 9.050720] [&lt;ffffffff81098c06&gt;] kthread+0x96/0xa0 9.050970] [&lt;ffffffff8100c24a&gt;] child_rip+0xa/0x20 9.051895] Pid: 6817, comm: mdt01_001 9.052114] 9.052114] Call Trace: 9.052817] [&lt;ffffffffa13be503&gt;] ? _ldlm_lock_debug+0x2e3/0x670 [ptlrpc] 9.053127] [&lt;ffffffff81516894&gt;] ? _spin_lock_irqsave+0x24/0x30 9.053391] [&lt;ffffffff81514231&gt;] schedule_timeout+0x191/0x2e0 9.053653] [&lt;ffffffff81081e50&gt;] ? process_timeout+0x0/0x10 9.053952] [&lt;ffffffffa13decf0&gt;] ? ldlm_expired_completion_wait+0x0/0x370 [ptlrpc 9.054484] [&lt;ffffffffa13e3841&gt;] ldlm_completion_ast+0x5e1/0x9b0 [ptlrpc] 9.054768] [&lt;ffffffff8105de00&gt;] ? default_wake_function+0x0/0x20 9.055087] [&lt;ffffffffa13e2c8e&gt;] ldlm_cli_enqueue_local+0x21e/0x7f0 [ptlrpc] 9.055394] [&lt;ffffffffa13e3260&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] 9.055697] [&lt;ffffffffa056bbc0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] 9.056558] [&lt;ffffffffa05745fb&gt;] mdt_object_local_lock+0x1bb/0xa80 [mdt] 9.056893] [&lt;ffffffffa056bbc0&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] 9.057247] [&lt;ffffffffa13e3260&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] 9.057488] [&lt;ffffffffa0575245&gt;] mdt_object_lock_internal+0x65/0x360 [mdt] 9.057687] [&lt;ffffffffa0575604&gt;] mdt_object_lock+0x14/0x20 [mdt] 9.057949] [&lt;ffffffffa05806fc&gt;] mdt_getattr_name_lock+0x103c/0x1ab0 [mdt] 9.058223] [&lt;ffffffff8128863a&gt;] ? strlcpy+0x4a/0x60 9.058530] [&lt;ffffffffa140ff84&gt;] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc] 9.058752] [&lt;ffffffffa14116d0&gt;] ? lustre_swab_ldlm_reply+0x0/0x40 [ptlrpc] 9.058958] [&lt;ffffffffa0581692&gt;] mdt_intent_getattr+0x292/0x470 [mdt] 9.059174] [&lt;ffffffffa056e064&gt;] mdt_intent_policy+0x494/0xce0 [mdt] 9.059398] [&lt;ffffffffa13c305f&gt;] ldlm_lock_enqueue+0x12f/0x950 [ptlrpc] 9.059629] [&lt;ffffffffa10b9201&gt;] ? cfs_hash_for_each_enter+0x1/0xa0 [libcfs] 9.059858] [&lt;ffffffffa13eedeb&gt;] ldlm_handle_enqueue0+0x51b/0x13e0 [ptlrpc] 9.060079] [&lt;ffffffffa146dc72&gt;] tgt_enqueue+0x62/0x1d0 [ptlrpc] 9.060286] [&lt;ffffffffa146e85e&gt;] tgt_request_handle+0x8be/0x1000 [ptlrpc] 9.060501] [&lt;ffffffffa141fd64&gt;] ptlrpc_main+0xdf4/0x1940 [ptlrpc] 9.060710] [&lt;ffffffffa141ef70&gt;] ? ptlrpc_main+0x0/0x1940 [ptlrpc] 9.060914] [&lt;ffffffff81098c06&gt;] kthread+0x96/0xa0 9.061085] [&lt;ffffffff8100c24a&gt;] child_rip+0xa/0x20 9.061275] [&lt;ffffffff81098b70&gt;] ? kthread+0x0/0xa0 9.061443] [&lt;ffffffff8100c240&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Dropping this from Blocker to Critical, since it is not a new issue for 2.7.0 (it exists since migrate was added in 2.4.0), and only affects a subset of users of the migrate functionality, and not anyone else.</p> <p>I'm investigating this issue.</p> <p>please apply <a href="http://review.whamcloud.com/13344" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13344</a> to your tree. It worked well after that patch was applied in my test.</p> <p>Patch that adds some tests for the new API: <a href="http://review.whamcloud.com/13441/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13441/</a><br/> It has a couple questions left (see BUG?? in source) but otherwise is complete.<br/> It has to be applied on top on Henri's patch.</p> <p>for the record: using Jinshan's patch did not help all that much and I was still seeing deadlocks on mdt</p> <p>I couldn't reproduce the deadlock problem on MDT. Please collect a core dump when you see the deadlock issue again.</p> <p>Frank, Henri, Jinshan,<br/> according to Oleg's last comments, he was still able to hit this deadlock even when the patch was applied, which raises a concern whether the risk of landing this complex patch is worth the risk at this late stage.</p> <p>Could you please confirm that the current patch has resolved the deadlock in your testing? It may be that Oleg is hitting a second issue that is not directly related. </p> <p>The second question is whether you are currently running with this patch in your other testing and can confirm that it doesn't introduce other problems?</p> <p>1. I can confirm that the original problem is fixed by this patch;<br/> 2. I tried to reproduce the problem mentioned by Oleg, i.e., running racer, for over 10 hours but I couldn't reproduce the problem. I'm not saying that this issue doesn't exist but is hard to reproduce. </p> <p>Hello,</p> <p>same here, I have not been able to reproduce the issues aforementioned... I can try investigating from crash dumps if Oleg has any that he can share, though it's indeed harder than with a reproducer.</p> <p>This patch introduces no regression I'm aware of, and it fixes the original problem. If its size/complexity makes it "unlandable" I can try to split it into two subpatches (one fixing the blocking mode, another one adding the non-blocking mode). I was actually not expecting it to grow that much <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Edit: I have just triggered deadlocks with racer, as descibed by Oleg, <b>with all rpms installed</b>, not simply running from the source tree. There was concurrent activity going on, but I'll try to reproduce in a non-disturbed environment.</p> <p>Henri, that's great. Please let me know if you need any help from me.</p> <p>So in order to move things forward, and based on my understanding that this patch actually helps some user-ereported problems (and also on the assumption thatthe racer problems are now possibly a separate bug)- let's split the patch into two parts: the actual fix and the racer test patch.</p> <p>We can then land the code fix (if it otherwise does not introduce any more failures) and we can wait with the racer test patch until we better understand why it fails and fix those. I am a bit nervous about landing tests that tend to fail as that invalidates our testing strategy quite a bit though.</p> <p>Henri Doreau (henri.doreau@cea.fr) uploaded a new patch: <a href="http://review.whamcloud.com/13669" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13669</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4840" title=" Deadlock when truncating file during lfs migrate" class="issue-link" data-issue-key="LU-4840"><del>LU-4840</del></a> tests: Add file migration to racer<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 739d73d58b2877711893e2df0b5f487e6b141060</p> <p>Racer patch: <a href="http://review.whamcloud.com/#/c/13669" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13669</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/10013/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10013/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4840" title=" Deadlock when truncating file during lfs migrate" class="issue-link" data-issue-key="LU-4840"><del>LU-4840</del></a> lfs: Use file lease to implement migration<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 85bd36cc69563d7a79e3ed34f8fadb4ed1a72b7c</p> <p>Hi,</p> <p>Now that patch at <a href="http://review.whamcloud.com/10013" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10013</a> has been merged into master, is it possible to have a backport to b2_5?<br/> Indeed, the original issue was met on one of our customer clusters running Lustre 2.4, that was updated afterwards in Lustre 2.5.</p> <p>Thanks,<br/> Sebastien.</p> <p>Closing <a href="https://jira.whamcloud.com/browse/LU-4840" title=" Deadlock when truncating file during lfs migrate" class="issue-link" data-issue-key="LU-4840"><del>LU-4840</del></a> since the patches here are landed to fix the problems described here, except <a href="http://review.whamcloud.com/13669" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13669</a> to enable racer with object migrate. <a href="https://jira.whamcloud.com/browse/LU-7073" title="racer with OST object migration hangs on cleanup" class="issue-link" data-issue-key="LU-7073"><del>LU-7073</del></a> has been opened to track the new racer + object migrate failure.</p> <p>Backport to b2_7_fe <a href="http://review.whamcloud.com/#/c/21513/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/21513/</a></p> Related LU-6903 LU-6785 LU-5915 LU-7073 Development Rank 1|hzwiv3: Rank (Obsolete) 13336 Severity