https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1466 Lustre <p>Running IOR, single-shared file, a single client is always evicted for a blocking callback by OSS. OSS and client debug logs attached.</p> LU-1466

Hyperion DAT - IOR ssf - client eviction

Bug Blocker Resolved Fixed Oleg Drokin Cliff White Fri, 1 Jun 2012 15:41:31 +0000 Fri, 22 Jun 2012 10:04:13 +0000 Fri, 22 Jun 2012 10:04:13 +0000 Lustre 2.1.2 0 3 <p>Debug logs are on FTP, uploads<br/> dat.fail.client617.gz<br/> dat.fail.oss32.log.gz</p> <p>Repeated tests, repeated error, full debug and msgs uploaded to uploads<br/> oss.dit29.debug.log.gz oss.dit29.msg.gz hy34.debug.log.gz hy34.msg.gz</p> <p>Oleg</p> <p>What do you advise here?</p> <p>Thanks</p> <p>Peter</p> <p>I have repeated the test with the 2.2.54 tag, same errors.</p> <p>the second test log set contains the culprit:<br/> The client got the cancel request:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00010000:00010000:1.0:1338581263.050526:0:22810:0:(ldlm_lockd.c:1503:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST002a-osc-ffff8101a4928800 lock: ffff8101c7918b40/0xc2c2e1a2df0fb82d lrc: 3/0,0 mode: PW/PW res: 36588/0 rrc: 6 type: EXT [4706009088-&gt;4739563519] (req 4706009088-&gt;4707057663) flags: 0x10100000 remote: 0x1272af3896302f40 expref: -99 pid: 25237 timeout 0 00010000:00010000:1.0:1338581263.050531:0:22810:0:(ldlm_lockd.c:1516:ldlm_handle_bl_callback()) Lock ffff8101c7918b40 already unused, calling callback (ffffffff88837f80) </pre> </div></div> <p>Now I assume the bl callback just blocked on the cl_lock_mutex_get(env, lock); in osc_dlm_blocking_ast0() as there is basically nothing else to block on.</p> <p>Then there's no activity with this lock until finally the lock is cancelled:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00010000:00010000:1.0:1338581374.583277:0:22810:0:(ldlm_request.c:1030:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST002a-osc-ffff8101a4928800 lock: ffff8101c7918b40/0xc2c2e1a2df0fb82d lrc: 4/0,0 mode: PW/PW res: 36588/0 rrc: 6 type: EXT [4706009088-&gt;4739563519] (req 4706009088-&gt;4707057663) flags: 0x10102010 remote: 0x1272af3896302f40 expref: -99 pid: 25237 timeout 0 </pre> </div></div> <p>Now this only happens after the client was already evicted, in fact - right after the eviction and cancellation of a bunch of RPCs, but none of them seems to be too old, all of them certainly have appeared way after we got the BL ast.<br/> But something was certainly holding the lock mutex and I am not really sure what might have been it.</p> <p>This seems to be at least marginally related to lu1274 which was dealing with a similar issue in glimpse callback and at the time it was decided that it was the server slowly progressing through IO, but in fact it might be the client that hogs the lock after all.</p> <p>The cancel process is not blocked at acquiring a lock mutex because I saw this line in the same log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:00010000:1.0:1338581263.050536:0:22810:0:(cl_lock.c:143:cl_lock_trace0()) cancel lock: ffff810137d5f4b0@(1 ffff810228bc4080 1 5 0 0 0 0)(ffff81016091ea30/1/1) at cl_lock_cancel():1830 </pre> </div></div> <p>this means it has already grabbed lock mutex to call cl_lock_cancel().</p> <p>However, I don't know why it can;t go through the canceling process. Maybe it was blocked at a page based on the situation that there is no RPC sent at all. Can you please show me the backtrace with a higher level debug information?</p> <p>That is likely to be difficult short term, may be possible after next test cycle on Hyperion.</p> <p>Current testing with 2.1.2 fails to reproduce this issue. <a href="https://maloo.whamcloud.com/test_sets/652c72e0-b9b5-11e1-9392-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/652c72e0-b9b5-11e1-9392-52540035b04c</a> Closing.</p> Development Rank 1|hzvgwn: Rank (Obsolete) 6388 Severity