https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-180 Lustre <p>Hi,</p> <p>On our benchmarking cluster we observe a very strange behavior of Lustre: sometimes clients that are evicted by an OST never reconnect. The consequence is that we have a lot of Defunct processes on the impacted compute nodes, so we have to reboot them.</p> <p>For instance on a compute node we can see that one OST connection is missing:</p> <p><span class="error">&#91;root@kay297 ~&#93;</span># lfs df<br/> UUID 1K-blocks Used Available Use% Mounted on<br/> scratch-MDT0000_UUID 983490512 887336 982603176 0% /scratch_lustre<span class="error">&#91;MDT:0&#93;</span><br/> scratch-OST0000_UUID 7691221300 547721748 7143497248 7% /scratch_lustre<span class="error">&#91;OST:0&#93;</span><br/> scratch-OST0001_UUID 7691221300 548640356 7142579792 7% /scratch_lustre<span class="error">&#91;OST:1&#93;</span><br/> scratch-OST0002_UUID 7691221300 555585984 7135634108 7% /scratch_lustre<span class="error">&#91;OST:2&#93;</span><br/> scratch-OST0003_UUID 7691221300 551102404 7140118212 7% /scratch_lustre<span class="error">&#91;OST:3&#93;</span><br/> scratch-OST0004_UUID 7691221300 569235872 7121985356 7% /scratch_lustre<span class="error">&#91;OST:4&#93;</span><br/> scratch-OST0005_UUID 7691221300 565971400 7125210680 7% /scratch_lustre<span class="error">&#91;OST:5&#93;</span><br/> scratch-OST0006_UUID 7691221300 551380176 7139839904 7% /scratch_lustre<span class="error">&#91;OST:6&#93;</span><br/> scratch-OST0007_UUID 7691221300 552060560 7139160472 7% /scratch_lustre<span class="error">&#91;OST:7&#93;</span><br/> scratch-OST0008_UUID 7691221300 540060736 7151160480 7% /scratch_lustre<span class="error">&#91;OST:8&#93;</span><br/> scratch-OST0009_UUID 7691221300 542803928 7148417244 7% /scratch_lustre<span class="error">&#91;OST:9&#93;</span><br/> scratch-OST000a_UUID 7691221300 549910932 7141309212 7% /scratch_lustre<span class="error">&#91;OST:10&#93;</span><br/> scratch-OST000b_UUID 7691221300 553465732 7137754416 7% /scratch_lustre<span class="error">&#91;OST:11&#93;</span><br/> scratch-OST000c_UUID 7691221300 547134756 7144086380 7% /scratch_lustre<span class="error">&#91;OST:12&#93;</span><br/> scratch-OST000d_UUID 7691221300 542512828 7148708296 7% /scratch_lustre<span class="error">&#91;OST:13&#93;</span><br/> scratch-OST000e_UUID 7691221300 540940940 7150278116 7% /scratch_lustre<span class="error">&#91;OST:14&#93;</span><br/> scratch-OST000f_UUID 7691221300 552187304 7139031756 7% /scratch_lustre<span class="error">&#91;OST:15&#93;</span><br/> scratch-OST0010_UUID 7691221300 553010540 7138207368 7% /scratch_lustre<span class="error">&#91;OST:16&#93;</span><br/> scratch-OST0011_UUID 7691221300 549111608 7142109332 7% /scratch_lustre<span class="error">&#91;OST:17&#93;</span><br/> OST0012 : inactive device<br/> scratch-OST0013_UUID 7691221300 545678392 7145547784 7% /scratch_lustre<span class="error">&#91;OST:19&#93;</span><br/> scratch-OST0014_UUID 7691221300 545673392 7145547784 7% /scratch_lustre<span class="error">&#91;OST:20&#93;</span><br/> scratch-OST0015_UUID 7691221300 553029372 7138191732 7% /scratch_lustre<span class="error">&#91;OST:21&#93;</span><br/> scratch-OST0016_UUID 7691221300 578557784 7112659292 7% /scratch_lustre<span class="error">&#91;OST:22&#93;</span><br/> scratch-OST0017_UUID 7691221300 553574948 7137640080 7% /scratch_lustre<span class="error">&#91;OST:23&#93;</span><br/> scratch-OST0018_UUID 7691221300 593382232 7097838936 7% /scratch_lustre<span class="error">&#91;OST:24&#93;</span><br/> scratch-OST0019_UUID 7691221300 550952100 7140232336 7% /scratch_lustre<span class="error">&#91;OST:25&#93;</span><br/> scratch-OST001a_UUID 7691221300 604897244 7086322904 7% /scratch_lustre<span class="error">&#91;OST:26&#93;</span><br/> scratch-OST001b_UUID 7691221300 545086976 7146133184 7% /scratch_lustre<span class="error">&#91;OST:27&#93;</span><br/> scratch-OST001c_UUID 7691221300 550491540 7140725472 7% /scratch_lustre<span class="error">&#91;OST:28&#93;</span><br/> scratch-OST001d_UUID 7691221300 542008368 7149211712 7% /scratch_lustre<span class="error">&#91;OST:29&#93;</span></p> <p>filesystem summary: 215354196400 16076170152 199823591804 7% /scratch_lustre</p> <p>The phenomenon is erratic, it does not always impact the same clients or the same OSTs.</p> <p>I attach the syslogs of kay297 (client) and kay3 (OSS).</p> LU-180

Sometimes evicted clients never reconnect

Bug Critical Resolved Duplicate Niu Yawei Sebastien Buisson Wed, 30 Mar 2011 06:07:32 +0000 Tue, 19 Apr 2011 08:57:21 +0000 Tue, 19 Apr 2011 08:57:21 +0000 Lustre 2.0.0 0 5 <p>Hi,</p> <p>After less than 24 hours we have a new occurrence of this issue. The client node is kay310 (10.17.1.211@o2ib) and the OST is scratch-OST000b hosted by OSS kay2.</p> <p><span class="error">&#91;root@kay310 ~&#93;</span># lctl dl<br/> 0 UP mgc MGC10.17.0.2@o2ib 1d6fa677-53e6-5478-7b63-13c135edb2cd 5<br/> 1 UP lov scratch-clilov-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 4<br/> 2 UP lmv scratch-clilmv-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 4<br/> 3 UP mdc scratch-MDT0000-mdc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 4 UP osc scratch-OST0016-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 5 UP osc scratch-OST0017-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 6 UP osc scratch-OST001d-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 7 UP osc scratch-OST0010-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 8 UP osc scratch-OST0003-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 9 UP osc scratch-OST0015-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 10 UP osc scratch-OST0005-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 11 UP osc scratch-OST0009-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 12 UP osc scratch-OST0007-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 13 UP osc scratch-OST0006-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 14 UP osc scratch-OST000d-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 15 UP osc scratch-OST0002-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 16 UP osc scratch-OST000e-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 17 UP osc scratch-OST0004-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 18 UP osc scratch-OST000f-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 19 UP osc scratch-OST0008-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 20 UP osc scratch-OST001b-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 21 IN osc scratch-OST000b-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 22 UP osc scratch-OST0019-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 23 UP osc scratch-OST0012-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 24 UP osc scratch-OST000a-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 25 UP osc scratch-OST0011-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 26 UP osc scratch-OST000c-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 27 UP osc scratch-OST0001-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 28 UP osc scratch-OST001c-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 29 UP osc scratch-OST0014-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 30 UP osc scratch-OST0018-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 31 UP osc scratch-OST001a-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 32 UP osc scratch-OST0000-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5<br/> 33 UP osc scratch-OST0013-osc-ffff8803313ff000 0d84de33-3370-ded5-56b3-420997b67d2c 5</p> <p><span class="error">&#91;root@kay310 ~&#93;</span># cat /proc/fs/lustre/osc/scratch-OST000b-osc-ffff8803313ff000/import <br/> import:<br/> name: scratch-OST000b-osc-ffff8803313ff000<br/> target: scratch-OST000b_UUID<br/> state: EVICTED<br/> connect_flags: <span class="error">&#91;write_grant, server_lock, version, request_portal, truncate_lock, max_byte_per_rpc, early_lock_cancel, adaptive_timeouts, lru_resize, alt_checksum_algorithm, version_recovery, full20&#93;</span><br/> import_flags: <span class="error">&#91;invalid, replayable, pingable&#93;</span><br/> connection:<br/> failover_nids: <span class="error">&#91;10.17.0.3@o2ib&#93;</span><br/> current_connection: 10.17.0.3@o2ib<br/> connection_attempts: 2<br/> generation: 2<br/> in-progress_invalidations: 1<br/> rpcs:<br/> inflight: 0<br/> unregistering: 0<br/> timeouts: 0<br/> avg_waittime: 167693 usec<br/> service_estimates:<br/> services: 1 sec<br/> network: 1 sec<br/> transactions:<br/> last_replay: 0<br/> peer_committed: 4514305<br/> last_checked: 4514305<br/> read_data_averages:<br/> bytes_per_rpc: 1046287<br/> usec_per_rpc: 57030<br/> MB_per_sec: 18.34<br/> write_data_averages:<br/> bytes_per_rpc: 1046268<br/> usec_per_rpc: 585258<br/> MB_per_sec: 1.78</p> <p><span class="error">&#91;buissons@kay2 ~&#93;</span>$ lctl dl<br/> 0 UP mgc MGC10.17.0.2@o2ib b612ff68-0ab5-593c-d7ca-0fc4e5087a57 5<br/> 1 UP ost OSS OSS_uuid 3<br/> 2 UP obdfilter scratch-OST0006 scratch-OST0006_UUID 157<br/> 3 UP obdfilter scratch-OST000b scratch-OST000b_UUID 153<br/> 4 UP obdfilter scratch-OST0017 scratch-OST0017_UUID 157<br/> 5 UP obdfilter scratch-OST0007 scratch-OST0007_UUID 157<br/> 6 UP obdfilter scratch-OST0008 scratch-OST0008_UUID 157<br/> 7 UP obdfilter scratch-OST001d scratch-OST001d_UUID 157<br/> 8 UP obdfilter scratch-OST000c scratch-OST000c_UUID 157<br/> 9 UP obdfilter scratch-OST0003 scratch-OST0003_UUID 157<br/> 10 UP obdfilter scratch-OST0004 scratch-OST0004_UUID 157<br/> 11 UP obdfilter scratch-OST0016 scratch-OST0016_UUID 157<br/> 12 UP obdfilter scratch-OST0005 scratch-OST0005_UUID 155<br/> 13 UP obdfilter scratch-OST0010 scratch-OST0010_UUID 157<br/> 14 UP obdfilter scratch-OST0002 scratch-OST0002_UUID 157<br/> 15 UP obdfilter scratch-OST0015 scratch-OST0015_UUID 157<br/> 16 UP obdfilter scratch-OST000a scratch-OST000a_UUID 157</p> <p><span class="error">&#91;buissons@kay2 ~&#93;</span>$ cat /proc/fs/lustre/obdfilter/scratch-OST000b/exports/10.17.1.211&#64;o2ib/uuid <br/> <span class="error">&#91;buissons@kay2 ~&#93;</span>$ cat /proc/fs/lustre/obdfilter/scratch-OST000b/exports/10.17.1.211&#64;o2ib/stats<br/> snapshot_time 1301562458.151546 secs.usecs<br/> read_bytes 4109 samples <span class="error">&#91;bytes&#93;</span> 4096 1048576 4299194368<br/> write_bytes 4089 samples <span class="error">&#91;bytes&#93;</span> 57344 1048576 4278190080<br/> get_info 41 samples <span class="error">&#91;reqs&#93;</span><br/> set_info_async 3 samples <span class="error">&#91;reqs&#93;</span><br/> disconnect 6 samples <span class="error">&#91;reqs&#93;</span><br/> sync 6 samples <span class="error">&#91;reqs&#93;</span><br/> preprw 8198 samples <span class="error">&#91;reqs&#93;</span><br/> commitrw 8198 samples <span class="error">&#91;reqs&#93;</span><br/> ping 24659 samples <span class="error">&#91;reqs&#93;</span></p> <p>This is very annoying because I tried to reactivate the import on the client, but with no success.</p> <p><span class="error">&#91;root@kay310 ~&#93;</span># lctl --device 21 activate<br/> error: activate: failed: Invalid argument</p> <p>The only option we have is to reboot the impacted Lustre clients, which is not acceptable on a benchmarking cluster.</p> <p>Sebastien.</p> <p>Niu</p> <p>Could you look into this one please?</p> <p>Thanks</p> <p>Peter</p> <p>Hi,</p> <p>This problem seems to be similar to bugzilla 21636. The initial description in this bugzilla is not really the same, but if you look at comment 82 you will see that the errors are exactly the same as what we are suffering from on our benchmarking cluster.</p> <p>HTH<br/> Sebastien.</p> <p>Looks the invalidate thread did never finish the import invalidation job, I suspect that there is something deadlock.</p> <p>Hi, Sebastien</p> <p>Could you also collect all the threads' stack trace of the abnormal client when this issue happened? (echo t &gt; /proc/sysrq-trigger) I want to see where the invalidate thread was blocked.</p> <p>This bug is probably caused by a deadlock in clio, which could cause the client being evicted, and because of the deadlock, the invalidation thread (ll_imp_inval) would be blocked on a semaphore, then the import can not be reactived anymore.</p> <p>The deadlock issue is <a href="https://jira.whamcloud.com/browse/LU-122" title="Revert bug 21122 since it causes deadlock" class="issue-link" data-issue-key="LU-122"><del>LU-122</del></a>, and Jay is working on it.</p> <p>Hi, Sebastien, when you hit the issue next time, could you please check the stack trace to see if it's similar to the trace shown in the attached file? Thanks. </p> <p>There is a new occurence of the bug.</p> <p>In attachment is the trace log of the client</p> <ul class="alternate" type="square"> <li>at the time the client was evicted by the OST</li> <li>with the threads' stack trace generated by 'sysrq-trigger'</li> </ul> <p>Tell me if you need other information from the client node.</p> <p>The stack trace of the new occurence looks similar to the one in deadlock-trace.log.</p> <p>We are going to install the fix described by <a href="https://jira.whamcloud.com/browse/LU-122" title="Revert bug 21122 since it causes deadlock" class="issue-link" data-issue-key="LU-122"><del>LU-122</del></a> and see if it reproduces.</p> <p>Hi, Gregoire</p> <p>Yes, I checked the log and I believe it's the deadlock problem in <a href="https://jira.whamcloud.com/browse/LU-122" title="Revert bug 21122 since it causes deadlock" class="issue-link" data-issue-key="LU-122"><del>LU-122</del></a>, mark it as duplicated.</p> Development Rank 1|hzvslz: Rank (Obsolete) 8539 Severity