https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1455 Lustre <p>Dear support, during a normal usage of the file system, we noticed an major drop down of the I/O performance (ex. from 5Gb/s to 0.9Gb/s ) and into the log a bunch of:</p> <p>May 31 10:17:39 weisshorn06 kernel: Lustre: scratch-OST001d: Client scratch-MDT0000-mdtlov_UUID (at 148.187.7.101@o2ib2) reconnecting<br/> May 31 10:17:39 weisshorn06 kernel: Lustre: 7445:0:(filter.c:2699:filter_connect_internal()) scratch-OST001d: Received MDS connection for group 0<br/> May 31 10:17:39 weisshorn06 kernel: Lustre: scratch-OST001d: received MDS connection from 148.187.7.101@o2ib2<br/> May 31 10:17:39 weisshorn06 kernel: Lustre: 7445:0:(llog_net.c:168:llog_receptor_accept()) changing the import ffff880f8b248800 - ffff880edff5b800<br/> May 31 10:17:39 weisshorn06 kernel: Lustre: 7445:0:(llog_net.c:168:llog_receptor_accept()) Skipped 1 previous similar message<br/> May 31 10:17:39 weisshorn06 kernel: Lustre: 7445:0:(filter.c:2555:filter_llog_connect()) scratch-OST001d: Recovery from log 0x140001f/0x0:b3e57b20</p> <p>May 31 10:21:32 weisshorn01 kernel: LustreError: 7733:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x5ef sub-object on OST idx 49/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 9518:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x2560 sub-object on OST idx 12/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 4004:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x225b sub-object on OST idx 12/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 3986:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x1257 sub-object on OST idx 12/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 7357:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x76b0 sub-object on OST idx 49/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 9539:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x879 sub-object on OST idx 12/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 7324:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0xf3ad sub-object on OST idx 12/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 3988:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x2d00 sub-object on OST idx 49/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 7746:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x1220 sub-object on OST idx 49/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 9640:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x230f sub-object on OST idx 12/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 7328:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x5b4 sub-object on OST idx 12/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 9476:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x2591 sub-object on OST idx 12/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 4062:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x28f3 sub-object on OST idx 12/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 9654:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x2251 sub-object on OST idx 12/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 7737:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x251c sub-object on OST idx 12/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 7347:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x2bd9 sub-object on OST idx 12/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 7337:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x2560 sub-object on OST idx 49/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 7725:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x23d8 sub-object on OST idx 12/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 3908:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x8d2 sub-object on OST idx 12/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 9484:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x57b sub-object on OST idx 12/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 7403:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0xc7e sub-object on OST idx 49/38: rc = -5<br/> May 31 10:21:33 weisshorn01 kernel: LustreError: 9689:0:(lov_request.c:560:lov_update_create_set()) error creating fid 0x16c9b sub-object on OST idx 12/38: rc = -5</p> <p>May 31 10:21:49 weisshorn08 kernel: Lustre: scratch-OST0017: Client scratch-MDT0000-mdtlov_UUID (at 148.187.7.101@o2ib2) reconnecting<br/> May 31 10:21:49 weisshorn08 kernel: Lustre: Skipped 1 previous similar message<br/> May 31 10:21:49 weisshorn08 kernel: Lustre: 7332:0:(filter.c:2699:filter_connect_internal()) scratch-OST0017: Received MDS connection for group 0<br/> May 31 10:21:49 weisshorn08 kernel: Lustre: 7332:0:(filter.c:2699:filter_connect_internal()) Skipped 1 previous similar message<br/> May 31 10:21:49 weisshorn08 kernel: Lustre: scratch-OST0017: received MDS connection from 148.187.7.101@o2ib2<br/> May 31 10:21:49 weisshorn08 kernel: Lustre: Skipped 1 previous similar message<br/> May 31 10:21:49 weisshorn08 kernel: Lustre: 7332:0:(llog_net.c:168:llog_receptor_accept()) changing the import ffff880fc7844800 - ffff880fc68b5000<br/> May 31 10:21:49 weisshorn08 kernel: Lustre: 7332:0:(llog_net.c:168:llog_receptor_accept()) Skipped 3 previous similar messages<br/> May 31 10:21:49 weisshorn08 kernel: Lustre: 7332:0:(filter.c:2555:filter_llog_connect()) scratch-OST0017: Recovery from log 0x1400019/0x0:b3e57b1a<br/> May 31 10:21:49 weisshorn08 kernel: Lustre: 7332:0:(filter.c:2555:filter_llog_connect()) Skipped 1 previous similar message<br/> May 31 10:21:49 weisshorn04 kernel: Lustre: scratch-OST0025: Client scratch-MDT0000-mdtlov_UUID (at 148.187.7.101@o2ib2) reconnecting<br/> May 31 10:21:49 weisshorn04 kernel: Lustre: 7284:0:(filter.c:2699:filter_connect_internal()) scratch-OST0025: Received MDS connection for group 0<br/> May 31 10:21:49 weisshorn04 kernel: Lustre: scratch-OST0025: received MDS connection from 148.187.7.101@o2ib2<br/> May 31 10:21:49 weisshorn04 kernel: Lustre: 7284:0:(llog_net.c:168:llog_receptor_accept()) changing the import ffff8802d1d4c800 - ffff880fbe6d3800<br/> May 31 10:21:49 weisshorn04 kernel: Lustre: 7284:0:(llog_net.c:168:llog_receptor_accept()) Skipped 1 previous similar message<br/> May 31 10:21:49 weisshorn04 kernel: Lustre: 7603:0:(filter.c:2555:filter_llog_connect()) scratch-OST0025: Recovery from log 0x1400027/0x0:b3e57b28</p> <p>We are running an IOR benchmark (writing an aggregate file of 512Gb) and the first run was fine, but now we monitoring with LTOP and the output of the job that the performance are drop down. <br/> Also few users complain about a behavior like that running hdf5 jobs, the first runs was fine but the following were really slow.</p> <p>The load on the machine regarding CPU is really low.</p> MDS HW <br/> ---------------------------------------------------------------------------------------------------- <br/> Linux XXXX.admin.cscs.ch 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 <br/> Architecture: x86_64 <br/> CPU op-mode(s): 32-bit, 64-bit <br/> Byte Order: Little Endian <br/> CPU(s): 16 <br/> Vendor ID: AuthenticAMD <br/> CPU family: 16 <br/> 64Gb RAM <br/> Interconnect IB 40Gb/s <br/> <br/> MDT LSI 5480 Pikes Peak <br/> SSDs SLC <br/> ---------------------------------------------------------------------------------------------------- <br/> <br/> OSS HW <br/> ---------------------------------------------------------------------------------------------------- <br/> Architecture: x86_64 <br/> CPU op-mode(s): 32-bit, 64-bit <br/> Byte Order: Little Endian <br/> CPU(s): 32 <br/> Vendor ID: GenuineIntel <br/> CPU family: 6 <br/> 64Gb RAM <br/> Interconnect IB 40Gb/s <br/> <br/> OST LSI 7900 <br/> ---------------------------------------------------------------------------------------------------- <br/> <br/> Router nodes<br/> -------------------<br/> 12 router nodes - IB 40Gb/s<br/> <br/> Clients<br/> ---------<br/> Cray XE6 - Lustre 1.8.6<br/> <br/> <br/> <br/> 1 MDS + 1 fail over <br/> 12 OSS - 6 OST per OSS LU-1455

Drop of the performance and 'scratch-OSTXXXX: Recovery'

Bug Critical Resolved Cannot Reproduce Oleg Drokin Fabio Verzelloni Thu, 31 May 2012 04:26:59 +0000 Mon, 29 May 2017 03:53:38 +0000 Mon, 29 May 2017 03:53:38 +0000 Lustre 2.2.0 0 1 <p>Error -5 is IO error. Could also be returned when local osc enters recovery mode (i.e. disconnected from server).</p> <p>Now, in your logs I see that MDS got disconnected from a lot of OSTs (possibly all of them?)because they did not respond to MDS requests.<br/> There is not really much messages from the OSSes to indicate they are terribly unhappy with something, so I wonder if it's just the network that gets overloaded and starts to drop messages?</p> <p>After some debugging we changed our FDR switch with a QDR switch, and after that we didn't see anymore the disconnecting and reconnecting of the OSTs. We are monitoring and running benchmark since yesterday and no error so far. <br/> I'll keep you up to date with the situation.</p> <p>Fabio </p> <p>Fabio</p> <p>Anything new to report?</p> <p>Peter</p> <p>Hi Peter, <br/> so far seems that this error did not appear anymore, we have some drop of performance but my colleague Nicola has already open another ticket 1503 and I've just added few more info. </p> <p>Thanks<br/> Fabio</p> <p>Close old ticket.</p> Development Rank 1|hzv32n: Rank (Obsolete) 3997 Severity