|
From the MDS (vm12) console log, we see two different call traces
There are several other test sessions that time out with the same call traces.
https://testing.whamcloud.com/test_sets/5dd05488-b401-11e8-b86b-52540065bddc (The stack call with ‘mdt_dom_discard_data’ is not hung )
https://testing.whamcloud.com/test_sets/abf52952-b2a5-11e8-b86b-52540065bddc
|
|
Lai
Could you please advise?
Thanks
Peter
|
|
I am taking this ticket, it looks related to DOM data discard and Oleg have reported similar issues also. After several local runs and investigations I have found that issue is related to mdt_dom_discard_data() function. While it is ran after all local objects are unlocked, it is still in the middle of unlink RPC processing, so until mdt_dom_discard_data() is finished no reply is being sent for unlink RPC. The problem is that parent directory is taking PW lock during that and lock will not be canceled until REP ACK will be received or local commit happens, so all other operations in the same directory will wait for dom discard lock completion, which is turn may wait for client lock to be canceled. Meanwhile we don't need to wait here at all, the only thing that discard should do is the bl_ast sending to the client. That can be fixed by avoiding waiting in completion callback for that lock or, per discussion with Vitaly, with LDLM_FL_CANCEL_ON_BLOCK flag. The discard lock will issue BL AST to the client and cancel server lock immediately without waiting for the client response.
|
|
Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/33522
Subject: LU-11359 mdt: fix mdt_dom_discard_data() timeouts
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0a947c24473666c9f6511ad786f147f8fc90c1ec
|
|
Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34071
Subject: LU-11359 mdt: fix mdt_dom_discard_data() timeouts
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e5b2a1998bb845a02ef788eda526221898bfec96
|
|
seeing similar error on soak which is running b2_10-ib build #98
on MDS 0, ldiskfs
|
|
Attached is a log of the execution of mdt_dom_discard_data with this patch. Patch does not achieve intended effect of getting another context. We're waiting for the lock request to complete, rather than putting it on the waiting queue, so we're sleeping and executing the completion AST in the original thread context.
|
|
Patrick, it is not quite so, there is no wait for lock request completion waiting, it only takes lock in the same context and sends blocking ASTs but don't wait for them to complete, that was an idea. Problem was in generic ldlm_completion_ast() which waits for all blocking lock to cancel in execution context.
|
|
Mike,
Take a look at the log if you haven't. It waits for them to complete. I added a few more comments in the patch.
|
|
Patrick, I don't see in log you've sent that it 'waits', it executes lock in the same context, yes, process lock policy, finds conflicting locks, sends blocking ASTs and exits. With generic ldlm_completion_ast() it would also 'waits' for all blocking locks to cancel. That is the main problem because if waits for other client locks to cancel in the current context, but with the patch it doesn't. Often that happens so that client cancels blocking locks right away and server gets cancels earlier than calling completion AST, in that case all is finished in the single context but that is not problem
|
|
In case anyone is not following Gerrit, the details are complicated, but Mike is correct. My comments can be disregarded. 
|
|
This is a follow-up to LU-12037. We're using this patch and I wanted to provide a first update before the weekend. The patch above has simply changed our life. While we still have 3 unpatched clients on Sherlock (~1400 node cluster), that are running long jobs, I can already say that this has been a tremendous change, in a very good way!!! After having patched about 1200 nodes (about ~200 nodes remaining), we started to see a significant drop in the number of
ldlm_completion_ast() call traces on the MDS (after at least one with mdt_dom_discard_data()). This morning, Robinhood was still blocked (perhaps I missed this part but we couldn't run Robinhood, or at least not for long, the lustre client there kept disconnecting/reconnecting from the MDTs)... So this morning I restarted all Fir's 4 MDTs and now logs are so pristine clean I'm even wondering if Lustre is running  ...). If someone is doing the same, I recommend: (1)patching servers, (2)patching clients, (3)restarting all MDTs to clean dead locks. Perhaps this should be added to the 2.12.1 changelogs. Anyway, I'll back report again next week to confirm that everything is alright in that aspect. Thanks so much!!!
|
|
Thanks for the update Stephane - this is really good news!
|
|
Hmm. While we haven't seen new call traces with mdt_dom_discard_data(), we just had an event with a lot of ldlm_completion_ast() and Robinhood was stuck. Filesystem was still usable overall but I noticed a few nodes that couldn't access a specific set of files. I restarted the Robinhood server and after some time the system recover by itself. This may be related to LU-11358 so I'll attach MDS logs there.
|
|
Stephane, this also can be result of LU-11751
|
|
Mike, ok thanks. I'll follow this one too. I have seen a few more traces on MDS last night, from time to time (see an example of one of these 'events' below). Not sure about the impact on users yet, but I assume some file access must be blocked, at least temporarily. We stopped Robinhood (which is not critical here) so it's not coming from it I think. Our plan is not to touch anything unless we see a real user-facing problem at this point, but do you think https://review.whamcloud.com/#/c/33077 would be worth trying if ever we see that it has an impact on production?
|
|
We've completed the patching of all our clients (~1,400) last weekend. We haven't seen any new {{mdt_dom_discard_data }} call traces, so I consider this issue closed. However, we likely hit LU-11751 last weekend and I did already follow-up there. Thanks!
|
|
Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34804
Subject: LU-11359 mdt: more debug for lu_object_put() issue
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a67945fc08744fa70c7b56a31adbc35ffa412dce
|
|
We are also seeing this issue for our production system. I want to add it can be reproduced with running the io500 script.
|
|
James
Have you tested the current patch (and LU-11285)?
Peter
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34071/
Subject: LU-11359 mdt: fix mdt_dom_discard_data() timeouts
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9c028e74c2202a8a481557c4cb22225734aaf19f
|
|
Landed for 2.13
|
|
Sorry missed the message. I have tested the LU-11359 patch but not LU-11285.
|
|
Latest patchset of https://review.whamcloud.com/34071/ introduced the use of lu_env_find (due to aa82cc83 LU-12034 obdclass: put all service's env on the list ) and thus it doesn't compile on b2_12 anymore. I wanted to rebase our branch against 2.12.2 and also use the patch that landed master (patchset 20) instead of patchset 12 that we are currently using in production. What is your recommendation? Or could you please backport it to b2_12? Thanks much!
|
|
LU-12034 is being ported to b2_12, Stephane - Note the LTS12 tag there.
|
|
Ah, cool. Thanks!
|
|
Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/35197
Subject: LU-11359 mdt: fix mdt_dom_discard_data() timeouts
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 69bc9c24f5ea8e9a952ad5ab76dc5e36f6b24b66
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/35197/
Subject: LU-11359 mdt: fix mdt_dom_discard_data() timeouts
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: e5810126b3fb488a3fed37e085e3ca4ae585324c
|
Generated at Sat Feb 10 02:43:11 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
mdt_dom_discard_data_thread.log