https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5532 Lustre <p>Lustre OSS reports lustre is unhealthy after issuing the following sequence of messages. Second occurrence in the last 24 hours. </p> <p>Aug 21 14:00:23 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3794666.155556&#93;</span> Lustre: atlas1-OST0016: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting<br/> Aug 21 14:00:23 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3794666.185134&#93;</span> Lustre: Skipped 6 previous similar messages<br/> Aug 21 14:02:53 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3794816.201589&#93;</span> LustreError: 137-5: atlas1-OST00a2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target)<br/> Aug 21 14:12:20 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3795383.544447&#93;</span> Lustre: atlas1-OST0016: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting<br/> Aug 21 14:12:20 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3795383.567821&#93;</span> Lustre: Skipped 5 previous similar messages<br/> Aug 21 14:19:20 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3795803.534475&#93;</span> LustreError: 137-5: atlas1-OST00a2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target)<br/> Aug 21 14:19:20 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3795803.566134&#93;</span> LustreError: Skipped 3 previous similar messages<br/> Aug 21 14:26:17 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3796220.710761&#93;</span> Lustre: atlas1-OST01c6: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting<br/> Aug 21 14:26:17 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3796220.734900&#93;</span> Lustre: Skipped 11 previous similar messages<br/> Aug 21 14:31:17 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3796520.850175&#93;</span> LustreError: 137-5: atlas1-OST00a2_UUID: not available for connect from 10.38.145.2@o2ib4 (no target)<br/> Aug 21 14:31:17 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3796520.878544&#93;</span> LustreError: Skipped 4 previous similar messages<br/> Aug 21 14:38:14 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3796937.996148&#93;</span> Lustre: atlas1-OST0136: Client 5d5389e1-62ad-c671-5318-48ff669e4a6e (at 10.38.145.2@o2ib4) reconnecting<br/> Aug 21 14:38:14 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3796938.026732&#93;</span> Lustre: Skipped 6 previous similar messages<br/> Aug 21 14:44:39 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3797323.165332&#93;</span> LustreError: 33287:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 603, more than at_max 600<br/> Aug 21 14:44:39 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3797323.165334&#93;</span> ns: filter-atlas1-OST0376_UUID lock: ffff88034b20a900/0xecd0a12120f17d4c lrc: 4/0,0 mode: PW/PW res: <span class="error">&#91;0x5c2a0f:0x0:0x0&#93;</span>.0 rrc: 2 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0-&gt;16383) flags: 0x10020 nid: 10.36.205.208@o2ib remote: 0xc39a22a84c361c61 expref: 26 pid: 14426 timeout: 8090929608 lvb_type: 0<br/> Aug 21 14:44:39 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3797323.292254&#93;</span> LustreError: 33287:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) Skipped 3 previous similar messages<br/> Aug 21 14:44:57 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3797340.909561&#93;</span> Lustre: atlas1-OST0136: Slow creates, 128/256 objects created at a rate of 2/s<br/> Aug 21 14:45:53 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3797397.137591&#93;</span> LustreError: 137-5: atlas1-OST0372_UUID: not available for connect from 10.38.145.2@o2ib4 (no target)<br/> Aug 21 14:45:53 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3797397.161035&#93;</span> LustreError: Skipped 2 previous similar messages<br/> Aug 21 14:48:24 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3797548.207840&#93;</span> Lustre: atlas1-OST01c6: Client 1942a1b8-14c2-1c85-f1cc-f5a627755ef9 (at 10.38.145.2@o2ib4) reconnecting<br/> Aug 21 14:48:24 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3797548.238152&#93;</span> Lustre: Skipped 6 previous similar messages<br/> Aug 21 14:52:26 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3797790.627280&#93;</span> LustreError: 45633:0:(service.c:3216:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 610s</p> <p>Then followed by several messages like this:</p> <p>Aug 21 14:54:27 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3797911.436996&#93;</span> Lustre: 33219:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/3), not sending early reply<br/> Aug 21 14:54:27 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3797911.436998&#93;</span> req@ffff8802f640e400 x1476723491621465/t0(0) o3-&gt;6ae34883-e4de-08ed-e3e9-ccdd41e9934d@9719@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647272 ref 2 fl Interpret:/0/0 rc 0/0<br/> Aug 21 14:54:32 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3797916.390933&#93;</span> Lustre: atlas1-OST0136: Bulk IO read error with 711e8a57-ae3b-7204-47a9-6a996887d00c (at 7253@gni108), client will retry: rc -110<br/> Aug 21 14:54:32 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3797916.392844&#93;</span> LustreError: 33261:0:(ldlm_lib.c:2702:target_bulk_io()) @@@ timeout on bulk PUT after 0+0s req@ffff8803bab27000 x1476723483171409/t0(0) o3-&gt;711e8a57-ae3b-7204-47a9-6a996887d00c@7253@gni108:0/0 lens 448/432 e 0 to 0 dl 1408647272 ref 1 fl Interpret:/0/0 rc 0/0</p> <p>Full syslog to follow. </p> Lustre 2.4.3, RHEL 6.4, kernel 2.6.32-358.23.2.el6.atlas.x86_64 LU-5532

LustreError: 77234:0:(ldlm_lockd.c:460:__ldlm_add_waiting_lock()) ### requested timeout 755, more than at_max 600

Bug Major Resolved Not a Bug Oleg Drokin Jason Hill Thu, 21 Aug 2014 19:28:35 +0000 Thu, 28 Aug 2014 18:50:48 +0000 Thu, 28 Aug 2014 18:50:48 +0000 Lustre 2.4.3 0 7 <p>Lustre logs from the affected OSS and the MDS for that filesystem for the 2 occurences of this particular issue.</p> <p>also of significance (I think), there is very little back-end IO happening; system load is over 300 in all categories, and memory utilization is extremely high; &lt; 650MB/64GB free.</p> <p><span class="error">&#91;root@atlas-oss1c7 ~&#93;</span># free -m<br/> total used free shared buffers cached<br/> Mem: 64306 64038 268 0 761 51748<br/> -/+ buffers/cache: 11527 52778<br/> Swap: 0 0 0</p> <p>Yesterday I successfully unmounted all the OST's from this OSS, removed all lustre kernel modules and restarted Lustre to a positive outcome. </p> <p>clients are both from the same cluster, both are running :</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=" class="user-hover" rel=""></a># rpm -qa | grep lustre<br/> lustre-client-2.4.2-2.6.32_358.23.2.el6.x86_64_g89cc68b.x86_64<br/> lustre-client-modules-2.4.2-2.6.32_358.23.2.el6.x86_64_g89cc68b.x86_64</p> <p>please drop severity; that was my mistake I tabbed through the field and had not intended to hit sev 2.</p> <p>Oleg is looking into this one</p> <p>So in the logs it seems there's a severe disk backend slowness going on.<br/> We see things like " Aug 21 14:52:26 atlas-oss1c7.ccs.ornl.gov kernel: <span class="error">&#91;3797790.627280&#93;</span> LustreError: 45633:0:(service.c:3216:ptlrpc_svcpt_health_check()) ost_io: unhealthy - request has been waiting 610s" and slow creates at the rate of 2/sec.</p> <p>I remember one of teh times we hit something like this before was soon after you restarted an OST. Was this restart you are talking about also before these problems occurred?</p> <p>Any interesting data from the DDN side about load on the array?</p> <p>Oleg,</p> <p>No interesting data at the DDN level, but we were able to see some errors on the infiniband interface between the OSS and the DDN. We replaced the IB cable and that seems to have quelled the issue. This was just unexpected because I thought the only way to clear an "unhealthy" state in /proc/fs/lustre/health_check was to reboot or at least unmount the deviced and unload the lustre modules. </p> <p>I'm good if we close this issue; thanks for the response and I apologize for my delay in getting back to you.</p> <p>&#8211;<br/> -Jason</p> <p>If "unhealthy" state was set due to slowness of the disk, and then the disk performance improves (ie requests no longer take 10 minutes to complete), unhealthy state will clear.</p> <p>Per ORNL, the issue is now understood and we can close the ticket.</p> Development Rank 1|hzwuaf: Rank (Obsolete) 15399 Severity