https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2665 Lustre <p>Hi,</p> <p>When trying to unmount a Lustre client, we got the following problem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: Wed Nov 21 06:25:01 2012 LustreError: 11559:0:(ldlm_lock.c:1697:ldlm_lock_cancel()) ### lock still has references ns: ptmp-MDT0000-mdc-ffff88030871bc00 lock: ffff88060dbd2d80/0x4618f3ec8d79d8be lrc: 4/0,1 mode: PW/PW res: 8590405073/266 rrc: 2 type: FLK pid: 4414 [0-&gt;551] flags: 0x22002890 remote: 0xc8980c051f8f6afd expref: -99 pid: 4414 timeout: 0 LustreError: 11559:0:(ldlm_lock.c:1698:ldlm_lock_cancel()) LBUG Pid: 11559, comm: umount Call Trace: [&lt;ffffffffa040d7f5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa040de07&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa063343d&gt;] ldlm_lock_cancel+0x1ad/0x1b0 [ptlrpc] [&lt;ffffffffa064d245&gt;] ldlm_cli_cancel_local+0xb5/0x380 [ptlrpc] [&lt;ffffffffa06510b8&gt;] ldlm_cli_cancel+0x58/0x3b0 [ptlrpc] [&lt;ffffffffa063ae18&gt;] cleanup_resource+0x168/0x300 [ptlrpc] [&lt;ffffffffa063afda&gt;] ldlm_resource_clean+0x2a/0x50 [ptlrpc] [&lt;ffffffffa041e28f&gt;] cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [&lt;ffffffffa063afb0&gt;] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [&lt;ffffffffa063afb0&gt;] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [&lt;ffffffffa041fcaf&gt;] cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [&lt;ffffffffa0637a69&gt;] ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [&lt;ffffffffa0638adb&gt;] __ldlm_namespace_free+0x4b/0x540 [ptlrpc] [&lt;ffffffffa06502d0&gt;] ? ldlm_cli_hash_cancel_unused+0x0/0xa0 [ptlrpc] [&lt;ffffffffa06502d0&gt;] ? ldlm_cli_hash_cancel_unused+0x0/0xa0 [ptlrpc] [&lt;ffffffffa06502d0&gt;] ? ldlm_cli_hash_cancel_unused+0x0/0xa0 [ptlrpc] [&lt;ffffffffa041fcb7&gt;] ? cfs_hash_for_each_nolock+0x87/0x1c0 [libcfs] [&lt;ffffffffa063903f&gt;] ldlm_namespace_free_prior+0x6f/0x230 [ptlrpc] [&lt;ffffffffa063fc4c&gt;] client_disconnect_export+0x23c/0x460 [ptlrpc] [&lt;ffffffffa0b42a44&gt;] lmv_disconnect+0x644/0xc70 [lmv] [&lt;ffffffffa0a470bc&gt;] client_common_put_super+0x46c/0xe80 [lustre] [&lt;ffffffffa0a47ba0&gt;] ll_put_super+0xd0/0x360 [lustre] [&lt;ffffffff8117e01c&gt;] ? dispose_list+0x11c/0x140 [&lt;ffffffff8117e4a8&gt;] ? invalidate_inodes+0x158/0x1a0 [&lt;ffffffff8116542b&gt;] generic_shutdown_super+0x5b/0x110 [&lt;ffffffff81165546&gt;] kill_anon_super+0x16/0x60 [&lt;ffffffffa050897a&gt;] lustre_kill_super+0x4a/0x60 [obdclass] [&lt;ffffffff811664e0&gt;] deactivate_super+0x70/0x90 [&lt;ffffffff811826bf&gt;] mntput_no_expire+0xbf/0x110 [&lt;ffffffff81183188&gt;] sys_umount+0x78/0x3c0 [&lt;ffffffff810030f2&gt;] system_call_fastpath+0x16/0x1b Kernel panic - not syncing: LBUG Pid: 11559, comm: umount Not tainted 2.6.32-220.23.1.bl6.Bull.28.8.x86_64 #1 Call Trace: [&lt;ffffffff81484650&gt;] ? panic+0x78/0x143 [&lt;ffffffffa040de5b&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [&lt;ffffffffa063343d&gt;] ? ldlm_lock_cancel+0x1ad/0x1b0 [ptlrpc] [&lt;ffffffffa064d245&gt;] ? ldlm_cli_cancel_local+0xb5/0x380 [ptlrpc] [&lt;ffffffffa06510b8&gt;] ? ldlm_cli_cancel+0x58/0x3b0 [ptlrpc] [&lt;ffffffffa063ae18&gt;] ? cleanup_resource+0x168/0x300 [ptlrpc] [&lt;ffffffffa063afda&gt;] ? ldlm_resource_clean+0x2a/0x50 [ptlrpc] [&lt;ffffffffa041e28f&gt;] ? cfs_hash_for_each_relax+0x17f/0x380 [libcfs] [&lt;ffffffffa063afb0&gt;] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [&lt;ffffffffa063afb0&gt;] ? ldlm_resource_clean+0x0/0x50 [ptlrpc] [&lt;ffffffffa041fcaf&gt;] ? cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] [&lt;ffffffffa0637a69&gt;] ? ldlm_namespace_cleanup+0x29/0xb0 [ptlrpc] [&lt;ffffffffa0638adb&gt;] ? __ldlm_namespace_free+0x4b/0x540 [ptlrpc] [&lt;ffffffffa06502d0&gt;] ? ldlm_cli_hash_cancel_unused+0x0/0xa0 [ptlrpc] [&lt;ffffffffa06502d0&gt;] ? ldlm_cli_hash_cancel_unused+0x0/0xa0 [ptlrpc] [&lt;ffffffffa06502d0&gt;] ? ldlm_cli_hash_cancel_unused+0x0/0xa0 [ptlrpc] [&lt;ffffffffa041fcb7&gt;] ? cfs_hash_for_each_nolock+0x87/0x1c0 [libcfs] [&lt;ffffffffa063903f&gt;] ? ldlm_namespace_free_prior+0x6f/0x230 [ptlrpc] [&lt;ffffffffa063fc4c&gt;] ? client_disconnect_export+0x23c/0x460 [ptlrpc] [&lt;ffffffffa0b42a44&gt;] ? lmv_disconnect+0x644/0xc70 [lmv] [&lt;ffffffffa0a470bc&gt;] ? client_common_put_super+0x46c/0xe80 [lustre] [&lt;ffffffffa0a47ba0&gt;] ? ll_put_super+0xd0/0x360 [lustre] [&lt;ffffffff8117e01c&gt;] ? dispose_list+0x11c/0x140 [&lt;ffffffff8117e4a8&gt;] ? invalidate_inodes+0x158/0x1a0 [&lt;ffffffff8116542b&gt;] ? generic_shutdown_super+0x5b/0x110 [&lt;ffffffff81165546&gt;] ? kill_anon_super+0x16/0x60 [&lt;ffffffffa050897a&gt;] ? lustre_kill_super+0x4a/0x60 [obdclass] [&lt;ffffffff811664e0&gt;] ? deactivate_super+0x70/0x90 [&lt;ffffffff811826bf&gt;] ? mntput_no_expire+0xbf/0x110 [&lt;ffffffff81183188&gt;] ? sys_umount+0x78/0x3c0 [&lt;ffffffff810030f2&gt;] ? system_call_fastpath+0x16/0x1b </pre> </div></div> <p>This issue is exactly the same as the one described in <a href="https://jira.whamcloud.com/browse/LU-1429" title="LBUG while unmounting client" class="issue-link" data-issue-key="LU-1429"><del>LU-1429</del></a>, which is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-1328" title="Failing customer&#39;s file creation test" class="issue-link" data-issue-key="LU-1328"><del>LU-1328</del></a>, which itself seems to be related to <a href="https://jira.whamcloud.com/browse/LU-1421" title="Client LBUG in ll_file_write after filesystem expansion" class="issue-link" data-issue-key="LU-1421"><del>LU-1421</del></a>.<br/> The issue seems to be resolved, but it is very unclear to me which patches are needed in order to completely fix the issue.<br/> I add that we need of fix for b2_1.</p> <p>Can you please advise?</p> <p>TIA,<br/> Sebastien.</p> LU-2665

LBUG while unmounting client

Bug Major Resolved Fixed Bruno Faccini Sebastien Buisson mn1 ptr Tue, 22 Jan 2013 11:29:16 +0000 Fri, 22 Nov 2013 08:21:38 +0000 Wed, 18 Sep 2013 10:20:57 +0000 Lustre 2.4.0 Lustre 2.1.3 Lustre 2.5.0 Lustre 2.4.2 0 12 <p>Hello Seb !!<br/> Is there a crash-dump available ?? If Yes, can you get us with a "foreach bt" output and the ldlm_lock content ??<br/> On the other hand, I will try to understand the link between this-one/<a href="https://jira.whamcloud.com/browse/LU-1429" title="LBUG while unmounting client" class="issue-link" data-issue-key="LU-1429"><del>LU-1429</del></a> and <a href="https://jira.whamcloud.com/browse/LU-1328" title="Failing customer&#39;s file creation test" class="issue-link" data-issue-key="LU-1328"><del>LU-1328</del></a>/<a href="https://jira.whamcloud.com/browse/LU-1421" title="Client LBUG in ll_file_write after filesystem expansion" class="issue-link" data-issue-key="LU-1421"><del>LU-1421</del></a>.</p> <p>Thank's in advance.</p> <p>Looks like this is an flock lock, and there were several patches landed to master about fixing those. Maybe one needs to be back-ported?</p> <p>Hi Bruno!!</p> <p>Antoine already sent us the content of the ldlm_lock structure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; p *(struct ldlm_lock *)0xffff88060dbd2d80 $3 = { l_handle = { h_link = { next = 0xffffc90016ae91d8, prev = 0xffffc90016ae91d8 }, h_cookie = 5051055179407415486, h_addref = 0xffffffffa062f130 &lt;lock_handle_addref&gt;, h_lock = { raw_lock = { slock = 196611 } }, h_ptr = 0x0, h_free_cb = 0, h_rcu = { next = 0x0, func = 0 }, h_size = 0, h_in = 1 '\001', h_unused = "\000\000" }, l_refc = { counter = 4 }, l_lock = { raw_lock = { slock = 851980 } }, l_resource = 0xffff8802f8a14300, l_lru = { next = 0xffff88060dbd2de0, prev = 0xffff88060dbd2de0 }, l_res_link = { next = 0xffff8802f8a14320, prev = 0xffff8802f8a14320 }, l_tree_node = 0x0, l_exp_hash = { next = 0x0, pprev = 0x0 }, l_req_mode = LCK_PW, l_granted_mode = LCK_PW, l_completion_ast = 0xffffffffa065ec20 &lt;ldlm_flock_completion_ast&gt;, l_blocking_ast = 0, l_glimpse_ast = 0, l_weigh_ast = 0, l_export = 0x0, l_conn_export = 0xffff88030871b800, l_remote_handle = { cookie = 14454316220189469437 }, l_policy_data = { l_extent = { start = 0, end = 551, gid = 18446612146206140608 }, l_flock = { start = 0, end = 551, owner = 18446612146206140608, blocking_owner = 0, blocking_export = 0x0, pid = 4414 }, l_inodebits = { bits = 0 } }, l_flags = 570435728, l_readers = 0, l_writers = 1, l_destroyed = 0 '\000', l_ns_srv = 0 '\000', l_waitq = { lock = { raw_lock = { slock = 196611 } }, task_list = { next = 0xffff88060dbd2ea8, prev = 0xffff88060dbd2ea8 prev = 0xffff88060dbd2ea8 } }, l_last_activity = 1353415544, l_last_used = 0, l_req_extent = { start = 0, end = 0, gid = 0 }, l_lvb_len = 0, l_lvb_data = 0x0, l_ast_data = 0xffff8805e2161c38, l_client_cookie = 0, l_pending_chain = { next = 0xffff88060dbd2f00, prev = 0xffff88060dbd2f00 }, l_callback_timeout = 0, l_pid = 4414, l_bl_ast = { next = 0xffff88060dbd2f20, prev = 0xffff88060dbd2f20 }, l_cp_ast = { next = 0xffff88060dbd2f30, prev = 0xffff88060dbd2f30 }, l_rk_ast = { next = 0xffff88060dbd2f40, prev = 0xffff88060dbd2f40 }, l_blocking_lock = 0x0, l_bl_ast_run = 0, l_sl_mode = { next = 0xffff88060dbd2f60, prev = 0xffff88060dbd2f60 }, l_sl_policy = { next = 0xffff88060dbd2f70, prev = 0xffff88060dbd2f70 }, l_reference = {&lt;No data fields&gt;}, l_exp_refs_nr = 0, l_exp_refs_link = { next = 0xffff88060dbd2f88, prev = 0xffff88060dbd2f88 }, l_exp_refs_target = 0x0, l_exp_list = { next = 0xffff88060dbd2fa0, prev = 0xffff88060dbd2fa0 } } </pre> </div></div> <p>I will ask him for the 'foreach bt' output.</p> <p>Cheers,<br/> Sebastien.</p> <p>Thank's Sebastien already.</p> <p>BTW, the fact a flock is involved seems to indicate to Andreas and others teammates that it could be a new situation than the one in <a href="https://jira.whamcloud.com/browse/LU-1429" title="LBUG while unmounting client" class="issue-link" data-issue-key="LU-1429"><del>LU-1429</del></a> finally.</p> <p>Also, do we know what has been running on this particular node ? I mean is it a compute node or a service (login/batch/...) node ??</p> <p>Sebastien, still no news about the "foreach bt" required infos ??<br/> We need this to find who is 18446612146206140608/0xFFFF88033C05E8C0 and where it could be stuck.</p> <p>Hi Bruno,</p> <p>Still no news, I will ping the Support team!</p> <p>Sebastien.</p> <p>Hi,</p> <p>Here is the requested information. Inside the tarball, you have:</p> <ul class="alternate" type="square"> <li>M1013_bt.txt, containing the 'foreach bt' output;</li> <li>M1013_lctl_debug_file_allcpu_sort_1133.txt, for the last Lustre debug traces.</li> </ul> <p>Sebastien.</p> <p>Sorry for the delay Seb.<br/> Requestor of the concerned FLock was no longer here at the time of the crash.</p> <p>As the last infos I may need for this, can you or somebody from the site (I am sure Antoine kept this ...) double-check in node's syslog/dmesg if there was any previous messages with references to the lock/0xffff88060dbd2d80, the PID/4414 or the owner/task/0xffff88033c05e8c0 ??</p> <p>Talking directly with Antoine, as I expected there may be earlier related msgs in the log he will provide/comment soon. Using these new infos we may go faster to the root cause now.</p> <p>Also he confirmed that there has been a few more occurrences since January with available crash-dumps.</p> <p>Yes unfortunatly, i have lost the crash dump who produce the LBUG so<br/> I search new occurence and I have found 2 others crash with the same<br/> LBUG and on one I found maybe interesting messages :</p> <p>LustreError: 28052:0:(file.c:158:ll_close_inode_openhandle()) inode 147840184874990214 mdc close failed: rc = -4</p> <p>I can't link inode to the LBUG but that is the PID who have the lock issue</p> <p>LustreError: 12577:0:(ldlm_lock.c:1697:ldlm_lock_cancel()) ### lock still has references ns: ptmp2-MDT0000-mdc-ffff8804739f5c00 lock: ffff88024b7496c0/0x141b572a020ffcdd lrc: 4/0,1 mode: PW/PW res: 8811961703/31366 rrc: 2 type: FLK pid: 28052 <span class="error">&#91;0-&gt;9223372036854775807&#93;</span> flags: 0x22002890 remote: 0xdf8e5c6f462dccb6 expref: -99 pid: 28052 timeout: 0</p> <p>So maybe lustre miss to release the lock when there are an issue during <br/> the close (due to an oops on the network for this case) ...</p> <p>I have attach the all trace in attachment ... and sorry for the delay</p> <p>Hi Bruno,</p> <p>Now that Antoine provided you with additional information, are you able to investigate on this issue?</p> <p>Thanks,<br/> Sebastien.</p> <p>Yes and my current focus is to try determine how an FLock reference from a dead process can remain ...</p> <p>There are known quick and dirty fix like the very simple one from <a href="https://jira.whamcloud.com/browse/LU-736" title="LBUG and kernel panic on client unmount" class="issue-link" data-issue-key="LU-736"><del>LU-736</del></a> but I will try to find the root cause of this situation ...</p> <p>I will call and work from remote with Antoine in order to identify how we can fall in this situation.</p> <p>I setup what I think is a reproducer. <br/> It is a version of flocks_test that takes a lock and then sleeps, after this I force evict of client and kills process ...<br/> It does not fail under 2.4 but shows the same Antoine identified. I am setting up a 2.1 test platform and will run it there. I may also call the site tomorrow and ask for a local test too ...</p> <p>But in case of evict event, ldlm_namespace_cleanup()/ldlm_resource_clean()/cleanup_resource() are called with LDLM_FL_LOCAL_ONLY unconditionally and then locks are treated as local which cause ldlm_cli_cancel()/ldlm_cli_cancel_local()/ldlm_lock_cancel() are not called to let the Server know of the cancel since it is assumed it did it on its own.</p> <p>This is also what happen in case of "umount -f", which could already be used as a work-around on-site !!... Or when mount localflock option (instead of flock) is used.</p> <p>From the crash-dumps available, all l_flags of involved ldlm_lock are set to 0x22002890 which means LDLM_FL_CBPENDING/LDLM_FL_CANCEL/LDLM_FL_FAILED/LDLM_FL_CANCELING/LDLM_FL_CLEANED/LDLM_FL_BL_DONE. All these flags should have been set as part of current cleanup_resource()/ldlm_cli_cancel<span class="error">&#91;_local&#93;</span>() call sequence.</p> <p>The l_last_activity time-stamp is a long time ago but only a few minutes before the ll_close_inode_openhandle() error-message for pid owning lock, pointed by Antoine, when present.</p> <p>Working more with Antoine to understand if something could have been wrong on Server/MDS side, we found that at least for the last 2 occurrences of problem, MDS had crashed/rebooted at the same period than l_last_activity and ll_close_inode_openhandle() error-message.</p> <p>Thus, problem could only be the cause of wrong handling of MDS reboot condition during FLock unlock/clean-up.</p> <p>Will try to reproduce it now locally with a MDS re-boot and full traces enabled.</p> <p>Antoine just found a new occurrence that happen during March. But for this new case, MDS crash/reboot did not happen around l_last_activity timing, but much more later (in fact this Client was not unmounted/stopped before Servers during a full Cluster shutdown ...). And the concerned FLock was from an application that has been killed -9 ...</p> <p>Thus I requested that they check if the FLock clean-up really happen during Application exit/kill, by writing a small reproducer to grant a FLock and then sleep, to kill/signal it and with full Lustre debug traces enabled, see if the automatic Unlock occurs.</p> <p>Good news, I am able to get the crash with a reproducer that mimics what we identified to be the possible scenario in our last findings/comments :</p> <p> _ run a cmd/bin doing/granting flock and then going to sleep for ever. I modified flocks_test for this.</p> <p> _ force MDS to crash.</p> <p> _ wait a bit and then kill/^C to abort cmd. We get the "(file.c:158:ll_close_inode_openhandle()) inode &lt;INO&gt; mdc close failed: rc = -4" msg in Client log.</p> <p> _ reboot MDS, re-mount MDT.</p> <p> _ umount lustre-FS on Client. --&gt;&gt; LBUG!</p> <p>more to come.</p> <p>Humm, I took a full Lustre trace during a new reproducer run and here is what I discovered :</p> <p> _ when the application is killed/signaled and exits, the Kernel try to do some automatic cleanup/close on <br/> the opened file-descs and for any FLock still around this is done via locks_remove_flock().</p> <p> _ locks_remove_flock() to terminate any current FLock will then force-create a request for an UnLock on the whole<br/> file, and also calls the FS associated+specific routine to do so, in our/Lustre case ll_file_flock().</p> <p> _ unfortunately, if MDS is down at this time, the UnLock request will be failed and leave the current granted <br/> FLocks as orphans.</p> <p> _ they will even be replayed when MDS/MDT is back.</p> <p> _ and on even much later umount, they can still be found with references and cause the LBUG.</p> <p>So, how can we fix this to handle the MDS in-between crash case ? Anyway, the fix has to be as generic as possible and it may not be obvious.</p> <p>Having a better/detailed look of the full lustre debug traces I may have found the reason why the UNLCK request is failed+thrown, it is because it is generated automatically by the Kernel during a fatal signal processing but then, if any problem occurs during communications with MDS, Lustre interrupts and trashes RPCs/requests when it detects a signal is pending/current in ptlrpc_set_wait()/ptlrpc_check_set() !!....</p> <p>Problem is that this behavior is incompatible in case of an FLock/F_UNLCK because, in any case (MDS crash/reboot + replay, or temporary comms/LNet problem) since Server/MDS will never know that covered locks can be released and also we finally end-up with the LBUG during a later umount due to these orphan granted Lustre-FLocks left.</p> <p>Thus, we need to find a way (keep track or retry) of FLock requests (at least F_UNLCKs) in such cases.</p> <p>I have a patch that fixes LBUG vs reproducer (crash of MDS when FLock set, kill/exit in-between, re-start MDT, umount on Client) and also passes auto-tests.</p> <p>Since problem is also in master (I ran reproducer against latest master build, ie 2.4.50 !), I will submit the same change to master and see how it runs against auto-tests full set.</p> <p>Ok, patch for b2_1 is at <a href="http://review.whamcloud.com/6407" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6407</a>. Master version is at <a href="http://review.whamcloud.com/6415" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6415</a>.</p> <p>What's the status of #6415? Can it land?</p> <p>Hello Cory,<br/> Change #6415 has just land. But do you also experienced this issue ?</p> <p>Yes, we did. We've adopted change #6415 and it appears to be the solution to our problem.</p> <p>Thank's for your feedback !! Nothing can be better for a patch than real "life"/production exposure.</p> <p>Sebastien</p> <p>Any word on the b2_1 version of the patch? Does this address your original concern? Can we close the ticket?</p> <p>Peter</p> <p>Hi Bruno,</p> <p>We have retrieved the patch for b2_1. It will be rolled out at CEA's next maintenance, planned for early September.</p> <p>Cheers,<br/> Sebastien.</p> <p>Related to <a href="https://jira.whamcloud.com/browse/LU-3701" title="Failure on test suite posix subtest test_1: fcntl.18/fcntl.35 Unresolved" class="issue-link" data-issue-key="LU-3701"><del>LU-3701</del></a> and <a href="http://review.whamcloud.com/#/c/7453" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7453</a>.</p> <p>I don't like an idea of resending flocks. There is no guarantee that request will succeed.<br/> Client has cleanup mechanism already. <br/> ldlm_flock_completion_ast() should call ldlm_lock_decref_internal() correctly.<br/> The patch should be combined with similar patch from <a href="https://jira.whamcloud.com/browse/LU-2177" title="ldlm_flock_completion_ast causes LBUG because of a race" class="issue-link" data-issue-key="LU-2177"><del>LU-2177</del></a> .</p> <p>Cory: Thanks to add a reference to <a href="https://jira.whamcloud.com/browse/LU-3701" title="Failure on test suite posix subtest test_1: fcntl.18/fcntl.35 Unresolved" class="issue-link" data-issue-key="LU-3701"><del>LU-3701</del></a>. This is the follow-on to this ticket as its change introduced some regressions vs the POSIX test suite.</p> <p>Andriy: If you read carefully history/comments for this/<a href="https://jira.whamcloud.com/browse/LU-2665" title="LBUG while unmounting client" class="issue-link" data-issue-key="LU-2665"><del>LU-2665</del></a> ticket, you will find it addresses a very particular scenario which can be simplified as "FLock/F_UNLCK requests can be trashed upon MDS crash or communications problem, leaving orphaned FLocks". This is definitelly not the race problem described in <a href="https://jira.whamcloud.com/browse/LU-2177" title="ldlm_flock_completion_ast causes LBUG because of a race" class="issue-link" data-issue-key="LU-2177"><del>LU-2177</del></a>. And concerning the "retry" mechanism in <a href="https://jira.whamcloud.com/browse/LU-3701" title="Failure on test suite posix subtest test_1: fcntl.18/fcntl.35 Unresolved" class="issue-link" data-issue-key="LU-3701"><del>LU-3701</del></a> change it becomes fully limited to FLock/F_UNLCK requests.</p> <p>But "retry" mechanism doesn't guarantee that lock will reach MDS. cleanup_resource() should deal with orphaned locks also.</p> <p>&gt; But "retry" mechanism doesn't guarantee that lock will reach MDS<br/> Yes, but we need to do our best! And particularly for FLock/F_UNLCKs (where Server must know, unless other Clients/processes will stay stuck forever) which can not be trashed. At least with retries we now also cover MDS crashes and communications problems, if reboot/restart/failover/fix/... occurs finally and if not Lustre is dead on this Client and who cares that we retry forever ?</p> <p>&gt; cleanup_resource() should deal with orphaned locks also<br/> Only during evict or "umount -f".</p> <p>Patches are landed to b2_1 and master for 2.5.0. Can this bug be closed?</p> <p>Sure, this bug can be closed, as additional work was carried out by Bruno in <a href="https://jira.whamcloud.com/browse/LU-3701" title="Failure on test suite posix subtest test_1: fcntl.18/fcntl.35 Unresolved" class="issue-link" data-issue-key="LU-3701"><del>LU-3701</del></a>.</p> <p>Thanks,<br/> Sebastien.</p> <p>ok - thanks Sebastien!</p> <p>Patch <a href="http://review.whamcloud.com/6415" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6415</a> was cherry-picked to Lustre b2_4 branch.</p> Related LU-3701 Development Rank 1|hzvfvb: Rank (Obsolete) 6217 Severity