https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10993 Lustre <p>I think aptch <a href="https://review.whamcloud.com/#/c/31690/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/31690/</a> for <a href="https://jira.whamcloud.com/browse/LU-10826" title="Regression in LU-9372 on OPA enviroment and no recovery triggered" class="issue-link" data-issue-key="LU-10826"><del>LU-10826</del></a> is more problematic.<br/> after apply patch <a href="https://review.whamcloud.com/#/c/31690/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/31690/</a> and test_req_buffer_pressure=1, it prevents OOM, but they are skipping some recvoery clients. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@voss05 ~]# lctl get_param obdfilter.*.recovery_status obdfilter.scratch-OST0024.recovery_status= status: COMPLETE recovery_start: 1525317355 recovery_duration: 54 completed_clients: 7249/7249 replayed_requests: 0 last_transno: 98784247808 VBR: DISABLED IR: ENABLED obdfilter.scratch-OST0025.recovery_status= status: COMPLETE recovery_start: 1525317353 recovery_duration: 56 completed_clients: 7031/7031 replayed_requests: 0 last_transno: 98784247808 VBR: DISABLED IR: ENABLED obdfilter.scratch-OST0026.recovery_status= status: COMPLETE recovery_start: 1525317352 recovery_duration: 57 completed_clients: 8168/8168 replayed_requests: 0 last_transno: 98784247808 VBR: DISABLED IR: ENABLED obdfilter.scratch-OST0027.recovery_status= status: COMPLETE recovery_start: 1525317350 recovery_duration: 59 completed_clients: 8195/8195 replayed_requests: 0 last_transno: 98784247808 VBR: DISABLED IR: ENABLED obdfilter.scratch-OST0028.recovery_status= status: COMPLETE recovery_start: 1525317355 recovery_duration: 54 completed_clients: 7984/7984 replayed_requests: 0 last_transno: 98784247808 VBR: DISABLED IR: ENABLED obdfilter.scratch-OST0029.recovery_status= status: COMPLETE recovery_start: 1525317352 recovery_duration: 57 completed_clients: 7985/7985 replayed_requests: 0 last_transno: 98784247808 VBR: DISABLED IR: ENABLED obdfilter.scratch-OST002a.recovery_status= status: COMPLETE recovery_start: 1525317354 recovery_duration: 55 completed_clients: 8329/8329 replayed_requests: 0 last_transno: 98784247808 VBR: DISABLED IR: ENABLED obdfilter.scratch-OST002b.recovery_status= status: COMPLETE recovery_start: 1525317351 recovery_duration: 58 completed_clients: 8291/8291 replayed_requests: 0 last_transno: 98784247808 VBR: DISABLED IR: ENABLED obdfilter.scratch-OST002c.recovery_status= status: COMPLETE recovery_start: 1525317350 recovery_duration: 59 completed_clients: 8286/8286 replayed_requests: 0 last_transno: 94489280512 VBR: DISABLED IR: ENABLED </pre> </div></div> <p>And, aslo sometimes, recovery still never triggered. e.g failover situation. <br/> I see the messages after restart OSTs</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 9169.158440] Lustre: 14598:0:(events.c:368:request_in_callback()) All ost request buffers busy [ 9169.158447] Lustre: 14598:0:(events.c:368:request_in_callback()) Skipped 3508 previous similar messages </pre> </div></div> LU-10993

Fix for LU-10826 is problematic and skips recvoery

Bug Minor Closed Cannot Reproduce Mikhail Pershin Shuichi Ihara Thu, 3 May 2018 03:47:26 +0000 Sun, 16 Jan 2022 08:34:13 +0000 Sun, 16 Jan 2022 08:34:13 +0000 Lustre 2.12.0 0 5 <p>Hello Shuichi, <br/> Why do you think that some recovery clients are being missed ? Because you expect the number of completed_clients to be the same for each OST ?</p> <p>Also, the "(events.c:368:request_in_callback()) All ost request buffers busy" is expected to occur when running when test_req_buffer_pressure=1.</p> <p>Yes, and I've checked client side, but they didn't connect to OST even reveroy stat is completed.<br/> Another prolbem. there are 40 x OSS here and some of OSS triggered recovery, but still many OSS didn't trigger recovery.<br/> Actually, if we do umounted OSTs and remount them again on those OSS, recoery retriggered, but not all clients to recover.<br/> might imcomplete patch of <a href="https://review.whamcloud.com/#/c/31690/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/31690/</a></p> <p>I know it is not a simple task, but as you seem to be able to reproduce easily, can you try to reduce the test to a minimal sub-set of OSS's OSTs and connected Clients and then take a full Lustre debug log on OSS and Clients ? I would like to get at least the trace from OSS and from both a successful and failed Clients.<br/> In the mean time I will try to reproduce on a test platform.</p> <p>ok, let me know what exact information do you need.<br/> at least, non re-trigger recvoery situation could be easy possible to reproduce.</p> <p>&gt; ok, let me know what exact information do you need.<br/> Well, like what I have already indicated in my previous comment! : "can you try to reduce the test to a minimal sub-set of OSS's OSTs and connected Clients and then take a full Lustre debug log on OSS and Clients ? I would like to get at least the trace from OSS and from both a successful and failed Clients."</p> <p>Hello Shuichi,<br/> Just a small update to let you know that the attempts to reproduce this problem have all been unsuccessful until now.<br/> BTW, did you find sometime to reproduce again on your side and in order to provide the infos I have requested before?</p> <p>Mike</p> <p>Could you please assess this situation?</p> <p>Thanks</p> <p>Peter</p> <p>Descoping from 2.12 for now as there is not enough to work on. We can certainly continue to work this as soon as there is some more data available</p> Related LU-10826 Development Rank 1|hzzwqn: Rank (Obsolete) 9223372036854775807 Severity