osd-ldiskfs.MGS.brw_stats= snapshot_time: 1693236421.166662708 secs.nsecs start_time: 1693184110.466449623 secs.nsecs elapsed_time: 52310.700213085 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % read | write discontiguous pages rpcs % cum % | rpcs % cum % read | write discontiguous blocks rpcs % cum % | rpcs % cum % read | write disk fragmented I/Os ios % cum % | ios % cum % read | write disk I/Os in flight ios % cum % | ios % cum % read | write I/O time (1/1000s) ios % cum % | ios % cum % read | write disk I/O size ios % cum % | ios % cum % read | write block maps msec maps % cum % | maps % cum % osd-ldiskfs.nbp12-MDT0000.brw_stats= snapshot_time: 1693236421.166914854 secs.nsecs start_time: 1693184155.822593199 secs.nsecs elapsed_time: 52265.344321655 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % read | write discontiguous pages rpcs % cum % | rpcs % cum % read | write discontiguous blocks rpcs % cum % | rpcs % cum % read | write disk fragmented I/Os ios % cum % | ios % cum % read | write disk I/Os in flight ios % cum % | ios % cum % read | write I/O time (1/1000s) ios % cum % | ios % cum % read | write disk I/O size ios % cum % | ios % cum % read | write block maps msec maps % cum % | maps % cum % osd-ldiskfs.nbp12-OST0000.brw_stats= snapshot_time: 1693236421.167500085 secs.nsecs start_time: 1693184202.864430046 secs.nsecs elapsed_time: 52218.303070039 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 1: 5128 34 34 | 3585 11 11 2: 237 1 35 | 259 0 11 4: 130 0 36 | 164 0 12 8: 96 0 37 | 172 0 12 16: 139 0 38 | 1107 3 16 32: 122 0 39 | 230 0 17 64: 244 1 40 | 186 0 17 128: 525 3 44 | 230 0 18 256: 7086 47 91 | 615 1 20 512: 634 4 95 | 276 0 21 1K: 619 4 100 | 25373 78 100 read | write discontiguous pages rpcs % cum % | rpcs % cum % 0: 14502 96 96 | 31338 97 97 1: 355 2 99 | 221 0 98 2: 88 0 99 | 134 0 98 3: 2 0 99 | 487 1 99 4: 0 0 99 | 1 0 99 5: 0 0 99 | 2 0 99 6: 1 0 99 | 1 0 99 7: 1 0 99 | 1 0 99 8: 0 0 99 | 0 0 99 9: 0 0 99 | 0 0 99 10: 0 0 99 | 2 0 99 11: 1 0 99 | 0 0 99 12: 0 0 99 | 0 0 99 13: 0 0 99 | 0 0 99 14: 0 0 99 | 0 0 99 15: 0 0 99 | 1 0 99 16: 0 0 99 | 2 0 99 17: 0 0 99 | 1 0 99 18: 0 0 99 | 1 0 99 19: 0 0 99 | 1 0 99 20: 0 0 99 | 1 0 99 21: 0 0 99 | 2 0 99 22: 0 0 99 | 0 0 99 23: 1 0 99 | 0 0 99 24: 0 0 99 | 0 0 99 25: 0 0 99 | 0 0 99 26: 0 0 99 | 0 0 99 27: 0 0 99 | 0 0 99 28: 0 0 99 | 0 0 99 29: 0 0 99 | 0 0 99 30: 0 0 99 | 1 0 100 31: 9 0 100 | 0 0 100 read | write discontiguous blocks rpcs % cum % | rpcs % cum % 0: 13967 93 93 | 27943 86 86 1: 799 5 98 | 2598 8 94 2: 124 0 99 | 581 1 96 3: 8 0 99 | 688 2 98 4: 3 0 99 | 111 0 99 5: 7 0 99 | 84 0 99 6: 5 0 99 | 37 0 99 7: 1 0 99 | 48 0 99 8: 18 0 99 | 26 0 99 9: 2 0 99 | 25 0 99 10: 3 0 99 | 21 0 99 11: 1 0 99 | 11 0 99 12: 0 0 99 | 7 0 99 13: 1 0 99 | 4 0 99 14: 0 0 99 | 2 0 99 15: 0 0 99 | 3 0 99 16: 0 0 99 | 3 0 99 17: 1 0 99 | 3 0 99 18: 0 0 99 | 0 0 99 19: 0 0 99 | 0 0 99 20: 0 0 99 | 1 0 99 21: 0 0 99 | 0 0 99 22: 0 0 99 | 1 0 100 23: 1 0 99 | 0 0 100 24: 0 0 99 | 0 0 100 25: 0 0 99 | 0 0 100 26: 1 0 99 | 0 0 100 27: 0 0 99 | 0 0 100 28: 0 0 99 | 0 0 100 29: 0 0 99 | 0 0 100 30: 0 0 99 | 0 0 100 31: 18 0 100 | 0 0 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 1: 11556 92 92 | 6547 20 20 2: 785 6 98 | 22983 71 91 3: 169 1 99 | 1790 5 97 4: 14 0 99 | 681 2 99 5: 2 0 99 | 122 0 99 6: 1 0 99 | 29 0 99 7: 3 0 99 | 19 0 99 8: 1 0 99 | 12 0 99 9: 0 0 99 | 4 0 99 10: 1 0 99 | 1 0 99 11: 0 0 99 | 1 0 99 12: 1 0 99 | 1 0 99 13: 0 0 99 | 0 0 99 14: 0 0 99 | 1 0 99 15: 0 0 99 | 0 0 99 16: 0 0 99 | 1 0 99 17: 0 0 99 | 2 0 99 18: 1 0 99 | 1 0 99 19: 0 0 99 | 0 0 99 20: 0 0 99 | 1 0 99 21: 0 0 99 | 0 0 99 22: 0 0 99 | 1 0 100 23: 0 0 99 | 0 0 100 24: 1 0 99 | 0 0 100 25: 0 0 99 | 0 0 100 26: 0 0 99 | 0 0 100 27: 0 0 99 | 0 0 100 28: 0 0 99 | 0 0 100 29: 0 0 99 | 0 0 100 30: 0 0 99 | 0 0 100 31: 2 0 100 | 0 0 100 read | write disk I/Os in flight ios % cum % | ios % cum % 1: 6875 48 48 | 19601 31 31 2: 820 5 54 | 14570 23 55 3: 439 3 57 | 5395 8 64 4: 343 2 59 | 4387 7 71 5: 314 2 62 | 2574 4 75 6: 325 2 64 | 2307 3 79 7: 325 2 66 | 1839 2 81 8: 318 2 68 | 1813 2 84 9: 363 2 71 | 1692 2 87 10: 352 2 73 | 1563 2 90 11: 290 2 75 | 1311 2 92 12: 270 1 77 | 1159 1 94 13: 263 1 79 | 936 1 95 14: 218 1 81 | 783 1 96 15: 186 1 82 | 607 0 97 16: 153 1 83 | 534 0 98 17: 148 1 84 | 378 0 99 18: 141 0 85 | 212 0 99 19: 120 0 86 | 76 0 99 20: 90 0 87 | 39 0 99 21: 86 0 87 | 16 0 99 22: 73 0 88 | 8 0 99 23: 67 0 88 | 4 0 99 24: 77 0 89 | 4 0 99 25: 58 0 89 | 2 0 99 26: 79 0 90 | 1 0 99 27: 57 0 90 | 1 0 99 28: 46 0 91 | 1 0 100 29: 40 0 91 | 0 0 100 30: 46 0 91 | 0 0 100 31: 1185 8 100 | 0 0 100 read | write I/O time (1/1000s) ios % cum % | ios % cum % 1: 5946 47 47 | 11246 34 34 2: 36 0 47 | 942 2 37 4: 29 0 47 | 1454 4 42 8: 130 1 48 | 1879 5 48 16: 791 6 55 | 2441 7 55 32: 1407 11 66 | 5600 17 73 64: 1877 14 81 | 5572 17 90 128: 1340 10 92 | 2282 7 97 256: 658 5 97 | 601 1 99 512: 302 2 99 | 162 0 99 1K: 20 0 99 | 16 0 99 2K: 1 0 100 | 2 0 100 read | write disk I/O size ios % cum % | ios % cum % 4K: 3321 23 23 | 4644 7 7 8K: 127 0 24 | 258 0 7 16K: 120 0 25 | 214 0 8 32K: 113 0 25 | 331 0 8 64K: 174 1 27 | 982 1 10 128K: 172 1 28 | 643 1 11 256K: 348 2 30 | 679 1 12 512K: 644 4 35 | 703 1 13 1M: 7753 54 90 | 4372 7 20 2M: 1395 9 100 | 48987 79 100 read | write block maps msec maps % cum % | maps % cum % 1: 24673 99 99 | 33313 99 99 2: 3 0 99 | 3 0 99 4: 0 0 99 | 5 0 99 8: 0 0 99 | 8 0 99 16: 12 0 99 | 16 0 99 32: 19 0 99 | 31 0 99 64: 50 0 99 | 75 0 99 128: 68 0 99 | 71 0 99 256: 0 0 99 | 27 0 99 512: 1 0 100 | 77 0 99 1K: 0 0 100 | 11 0 100 osd-ldiskfs.nbp12-OST0001.brw_stats= snapshot_time: 1693236421.168460582 secs.nsecs start_time: 1693184202.861978805 secs.nsecs elapsed_time: 52218.306481777 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 1: 8204 45 45 | 5054 14 14 2: 173 0 45 | 708 2 16 4: 36 0 46 | 224 0 17 8: 57 0 46 | 178 0 18 16: 118 0 47 | 179 0 18 32: 79 0 47 | 255 0 19 64: 138 0 48 | 166 0 19 128: 311 1 50 | 135 0 20 256: 5079 27 77 | 451 1 21 512: 1787 9 87 | 567 1 23 1K: 2244 12 100 | 26153 76 100 read | write discontiguous pages rpcs % cum % | rpcs % cum % 0: 17964 98 98 | 32105 94 94 1: 239 1 99 | 664 1 96 2: 10 0 99 | 390 1 97 3: 2 0 99 | 824 2 99 4: 1 0 99 | 52 0 99 5: 0 0 99 | 13 0 99 6: 0 0 99 | 4 0 99 7: 4 0 99 | 6 0 99 8: 1 0 99 | 2 0 99 9: 1 0 99 | 0 0 99 10: 0 0 99 | 1 0 99 11: 1 0 99 | 0 0 99 12: 0 0 99 | 0 0 99 13: 0 0 99 | 0 0 99 14: 0 0 99 | 0 0 99 15: 0 0 99 | 0 0 99 16: 0 0 99 | 0 0 99 17: 0 0 99 | 2 0 99 18: 0 0 99 | 2 0 99 19: 0 0 99 | 0 0 99 20: 0 0 99 | 1 0 99 21: 0 0 99 | 0 0 99 22: 0 0 99 | 3 0 99 23: 0 0 99 | 0 0 99 24: 0 0 99 | 0 0 99 25: 0 0 99 | 0 0 99 26: 0 0 99 | 0 0 99 27: 0 0 99 | 0 0 99 28: 0 0 99 | 0 0 99 29: 0 0 99 | 0 0 99 30: 0 0 99 | 0 0 99 31: 3 0 100 | 1 0 100 read | write discontiguous blocks rpcs % cum % | rpcs % cum % 0: 16072 88 88 | 28594 83 83 1: 1596 8 96 | 3762 11 94 2: 326 1 98 | 606 1 96 3: 57 0 99 | 780 2 99 4: 24 0 99 | 218 0 99 5: 9 0 99 | 48 0 99 6: 4 0 99 | 26 0 99 7: 17 0 99 | 14 0 99 8: 42 0 99 | 4 0 99 9: 4 0 99 | 1 0 99 10: 5 0 99 | 0 0 99 11: 8 0 99 | 0 0 99 12: 6 0 99 | 0 0 99 13: 3 0 99 | 0 0 99 14: 1 0 99 | 0 0 99 15: 10 0 99 | 0 0 99 16: 1 0 99 | 1 0 99 17: 2 0 99 | 6 0 99 18: 3 0 99 | 2 0 99 19: 2 0 99 | 1 0 99 20: 1 0 99 | 1 0 99 21: 2 0 99 | 1 0 99 22: 0 0 99 | 1 0 99 23: 1 0 99 | 1 0 99 24: 0 0 99 | 0 0 99 25: 2 0 99 | 1 0 99 26: 0 0 99 | 0 0 99 27: 0 0 99 | 0 0 99 28: 0 0 99 | 0 0 99 29: 0 0 99 | 1 0 99 30: 0 0 99 | 0 0 99 31: 28 0 100 | 1 0 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 1: 11607 79 79 | 7258 21 21 2: 1975 13 92 | 23247 68 89 3: 853 5 98 | 2574 7 97 4: 135 0 99 | 776 2 99 5: 20 0 99 | 171 0 99 6: 4 0 99 | 25 0 99 7: 1 0 99 | 11 0 99 8: 6 0 99 | 2 0 99 9: 3 0 99 | 1 0 99 10: 2 0 99 | 1 0 99 11: 2 0 99 | 0 0 99 12: 6 0 99 | 0 0 99 13: 3 0 99 | 0 0 99 14: 2 0 99 | 0 0 99 15: 1 0 99 | 0 0 99 16: 3 0 99 | 0 0 99 17: 2 0 99 | 0 0 99 18: 3 0 99 | 2 0 99 19: 1 0 99 | 2 0 100 20: 1 0 99 | 0 0 100 21: 1 0 99 | 0 0 100 22: 1 0 99 | 0 0 100 23: 0 0 99 | 0 0 100 24: 0 0 99 | 0 0 100 25: 0 0 99 | 0 0 100 26: 2 0 99 | 0 0 100 27: 1 0 99 | 0 0 100 28: 0 0 99 | 0 0 100 29: 0 0 99 | 0 0 100 30: 0 0 99 | 0 0 100 31: 1 0 100 | 0 0 100 read | write disk I/Os in flight ios % cum % | ios % cum % 1: 8191 42 42 | 19870 30 30 2: 2552 13 55 | 16179 24 54 3: 1179 6 61 | 5078 7 62 4: 629 3 64 | 4023 6 68 5: 452 2 67 | 3006 4 73 6: 363 1 68 | 2716 4 77 7: 352 1 70 | 2219 3 80 8: 319 1 72 | 2088 3 83 9: 247 1 73 | 1840 2 86 10: 270 1 75 | 1670 2 89 11: 240 1 76 | 1336 2 91 12: 214 1 77 | 1164 1 93 13: 174 0 78 | 981 1 94 14: 155 0 79 | 859 1 95 15: 130 0 79 | 753 1 96 16: 126 0 80 | 687 1 98 17: 127 0 81 | 573 0 98 18: 124 0 81 | 383 0 99 19: 113 0 82 | 183 0 99 20: 114 0 82 | 105 0 99 21: 99 0 83 | 28 0 99 22: 94 0 83 | 11 0 99 23: 92 0 84 | 6 0 99 24: 110 0 84 | 4 0 99 25: 97 0 85 | 1 0 99 26: 101 0 85 | 1 0 99 27: 113 0 86 | 1 0 99 28: 125 0 87 | 1 0 99 29: 124 0 87 | 1 0 99 30: 123 0 88 | 1 0 99 31: 2237 11 100 | 1 0 100 read | write I/O time (1/1000s) ios % cum % | ios % cum % 1: 6938 47 47 | 12668 37 37 2: 56 0 47 | 645 1 39 4: 42 0 48 | 859 2 41 8: 185 1 49 | 932 2 44 16: 898 6 55 | 3205 9 53 32: 1521 10 65 | 6772 19 73 64: 1830 12 78 | 5767 16 90 128: 1343 9 87 | 2302 6 97 256: 1107 7 95 | 699 2 99 512: 578 3 99 | 203 0 99 1K: 136 0 99 | 18 0 100 2K: 2 0 100 | 0 0 100 read | write disk I/O size ios % cum % | ios % cum % 4K: 4887 25 25 | 6151 9 9 8K: 143 0 25 | 459 0 10 16K: 84 0 26 | 195 0 10 32K: 146 0 27 | 239 0 10 64K: 258 1 28 | 304 0 11 128K: 224 1 29 | 482 0 11 256K: 475 2 32 | 540 0 12 512K: 668 3 35 | 902 1 14 1M: 7348 37 73 | 6992 10 24 2M: 5153 26 100 | 49505 75 100 read | write block maps msec maps % cum % | maps % cum % 1: 34600 99 99 | 35319 98 98 2: 2 0 99 | 6 0 98 4: 5 0 99 | 7 0 98 8: 4 0 99 | 13 0 98 16: 45 0 99 | 43 0 98 32: 40 0 99 | 84 0 98 64: 42 0 99 | 163 0 99 128: 20 0 99 | 143 0 99 256: 12 0 99 | 82 0 99 512: 4 0 99 | 115 0 99 1K: 5 0 100 | 10 0 100 osd-ldiskfs.nbp12-OST0002.brw_stats= snapshot_time: 1693236421.169422151 secs.nsecs start_time: 1693184205.848706424 secs.nsecs elapsed_time: 52215.320715727 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 1: 3624 26 26 | 756 2 2 2: 74 0 27 | 461 1 3 4: 68 0 28 | 324 0 4 8: 113 0 28 | 213 0 5 16: 202 1 30 | 1145 3 8 32: 121 0 31 | 329 0 9 64: 258 1 33 | 591 1 11 128: 500 3 36 | 326 0 11 256: 7162 53 90 | 451 1 13 512: 634 4 94 | 523 1 14 1K: 674 5 100 | 29574 85 100 read | write discontiguous pages rpcs % cum % | rpcs % cum % 0: 13203 98 98 | 32875 94 94 1: 202 1 99 | 575 1 96 2: 16 0 99 | 363 1 97 3: 3 0 99 | 792 2 99 4: 1 0 99 | 53 0 99 5: 1 0 99 | 10 0 99 6: 0 0 99 | 3 0 99 7: 2 0 99 | 1 0 99 8: 0 0 99 | 0 0 99 9: 0 0 99 | 0 0 99 10: 0 0 99 | 0 0 99 11: 0 0 99 | 1 0 99 12: 0 0 99 | 1 0 99 13: 0 0 99 | 1 0 99 14: 0 0 99 | 2 0 99 15: 0 0 99 | 0 0 99 16: 0 0 99 | 1 0 99 17: 0 0 99 | 1 0 99 18: 0 0 99 | 5 0 99 19: 0 0 99 | 1 0 99 20: 0 0 99 | 1 0 99 21: 0 0 99 | 2 0 99 22: 0 0 99 | 3 0 99 23: 0 0 99 | 0 0 99 24: 0 0 99 | 0 0 99 25: 0 0 99 | 1 0 99 26: 0 0 99 | 0 0 99 27: 0 0 99 | 0 0 99 28: 0 0 99 | 0 0 99 29: 0 0 99 | 1 0 100 30: 0 0 99 | 0 0 100 31: 2 0 100 | 0 0 100 read | write discontiguous blocks rpcs % cum % | rpcs % cum % 0: 12245 91 91 | 25910 74 74 1: 901 6 97 | 6321 18 92 2: 97 0 98 | 1109 3 96 3: 31 0 98 | 962 2 98 4: 9 0 98 | 250 0 99 5: 15 0 99 | 77 0 99 6: 11 0 99 | 28 0 99 7: 10 0 99 | 8 0 99 8: 73 0 99 | 3 0 99 9: 3 0 99 | 6 0 99 10: 8 0 99 | 4 0 99 11: 2 0 99 | 2 0 99 12: 1 0 99 | 2 0 99 13: 3 0 99 | 2 0 99 14: 1 0 99 | 2 0 99 15: 1 0 99 | 0 0 99 16: 0 0 99 | 1 0 99 17: 0 0 99 | 0 0 99 18: 1 0 99 | 4 0 99 19: 0 0 99 | 1 0 99 20: 0 0 99 | 0 0 99 21: 1 0 99 | 0 0 99 22: 1 0 99 | 0 0 99 23: 0 0 99 | 0 0 99 24: 0 0 99 | 0 0 99 25: 1 0 99 | 1 0 100 26: 0 0 99 | 0 0 100 27: 0 0 99 | 0 0 100 28: 0 0 99 | 0 0 100 29: 0 0 99 | 0 0 100 30: 0 0 99 | 0 0 100 31: 15 0 100 | 0 0 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 1: 8843 87 87 | 4607 13 13 2: 1038 10 97 | 24169 69 82 3: 207 2 99 | 4658 13 96 4: 34 0 99 | 990 2 99 5: 4 0 99 | 197 0 99 6: 8 0 99 | 48 0 99 7: 3 0 99 | 9 0 99 8: 6 0 99 | 3 0 99 9: 0 0 99 | 0 0 99 10: 1 0 99 | 0 0 99 11: 0 0 99 | 1 0 99 12: 2 0 99 | 1 0 99 13: 0 0 99 | 1 0 99 14: 0 0 99 | 1 0 99 15: 0 0 99 | 1 0 99 16: 0 0 99 | 0 0 99 17: 0 0 99 | 1 0 99 18: 0 0 99 | 0 0 99 19: 0 0 99 | 4 0 99 20: 0 0 99 | 1 0 99 21: 0 0 99 | 0 0 99 22: 0 0 99 | 0 0 99 23: 0 0 99 | 0 0 99 24: 0 0 99 | 0 0 99 25: 0 0 99 | 0 0 99 26: 0 0 99 | 1 0 100 27: 0 0 99 | 0 0 100 28: 0 0 99 | 0 0 100 29: 0 0 99 | 0 0 100 30: 0 0 99 | 0 0 100 31: 1 0 100 | 0 0 100 read | write disk I/Os in flight ios % cum % | ios % cum % 1: 3969 33 33 | 20222 27 27 2: 904 7 40 | 17623 24 52 3: 393 3 43 | 8021 11 63 4: 227 1 45 | 5381 7 70 5: 186 1 47 | 3116 4 75 6: 143 1 48 | 2510 3 78 7: 130 1 49 | 2033 2 81 8: 105 0 50 | 1809 2 83 9: 86 0 51 | 1617 2 86 10: 113 0 52 | 1516 2 88 11: 82 0 52 | 1225 1 89 12: 77 0 53 | 1126 1 91 13: 67 0 53 | 1032 1 92 14: 64 0 54 | 1038 1 94 15: 54 0 54 | 1102 1 95 16: 43 0 55 | 1058 1 97 17: 42 0 55 | 908 1 98 18: 32 0 55 | 600 0 99 19: 35 0 56 | 273 0 99 20: 42 0 56 | 145 0 99 21: 43 0 56 | 48 0 99 22: 27 0 57 | 21 0 99 23: 38 0 57 | 10 0 99 24: 39 0 57 | 6 0 99 25: 35 0 58 | 3 0 100 26: 28 0 58 | 0 0 100 27: 37 0 58 | 0 0 100 28: 34 0 58 | 0 0 100 29: 30 0 59 | 0 0 100 30: 29 0 59 | 0 0 100 31: 4879 40 100 | 0 0 100 read | write I/O time (1/1000s) ios % cum % | ios % cum % 1: 2571 25 25 | 11448 32 32 2: 28 0 25 | 1137 3 36 4: 26 0 25 | 1194 3 39 8: 124 1 27 | 1876 5 45 16: 742 7 34 | 3011 8 53 32: 1107 10 45 | 6176 17 71 64: 893 8 54 | 5830 16 88 128: 720 7 61 | 2687 7 96 256: 1065 10 71 | 993 2 99 512: 1657 16 88 | 250 0 99 1K: 1065 10 98 | 59 0 99 2K: 139 1 99 | 27 0 99 4K: 10 0 100 | 2 0 99 8K: 0 0 100 | 3 0 100 read | write disk I/O size ios % cum % | ios % cum % 4K: 683 5 5 | 2538 3 3 8K: 102 0 6 | 483 0 4 16K: 129 1 7 | 291 0 4 32K: 163 1 8 | 410 0 5 64K: 167 1 10 | 933 1 6 128K: 211 1 12 | 752 1 7 256K: 407 3 15 | 1156 1 9 512K: 779 6 21 | 1458 2 11 1M: 7868 65 87 | 8299 11 22 2M: 1504 12 100 | 56123 77 100 read | write block maps msec maps % cum % | maps % cum % 1: 20346 99 99 | 42511 98 98 2: 0 0 99 | 3 0 98 4: 0 0 99 | 7 0 98 8: 1 0 99 | 14 0 98 16: 11 0 99 | 39 0 98 32: 24 0 99 | 86 0 98 64: 18 0 99 | 161 0 99 128: 21 0 99 | 158 0 99 256: 9 0 99 | 106 0 99 512: 3 0 99 | 130 0 99 1K: 4 0 99 | 8 0 100 2K: 1 0 100 | 0 0 100 osd-ldiskfs.nbp12-OST0003.brw_stats= snapshot_time: 1693236421.170382085 secs.nsecs start_time: 1693184205.922309766 secs.nsecs elapsed_time: 52215.248072319 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 1: 7018 52 52 | 5835 15 15 2: 123 0 53 | 627 1 17 4: 44 0 53 | 393 1 18 8: 57 0 54 | 263 0 18 16: 112 0 54 | 250 0 19 32: 78 0 55 | 385 1 20 64: 84 0 56 | 791 2 22 128: 215 1 57 | 613 1 24 256: 4973 37 94 | 1345 3 27 512: 115 0 95 | 876 2 30 1K: 563 4 100 | 26218 69 100 read | write discontiguous pages rpcs % cum % | rpcs % cum % 0: 13174 98 98 | 33724 89 89 1: 168 1 99 | 846 2 91 2: 15 0 99 | 548 1 93 3: 2 0 99 | 1962 5 98 4: 1 0 99 | 112 0 98 5: 1 0 99 | 69 0 99 6: 2 0 99 | 51 0 99 7: 11 0 99 | 30 0 99 8: 0 0 99 | 27 0 99 9: 0 0 99 | 10 0 99 10: 0 0 99 | 13 0 99 11: 0 0 99 | 10 0 99 12: 0 0 99 | 8 0 99 13: 0 0 99 | 8 0 99 14: 0 0 99 | 11 0 99 15: 0 0 99 | 4 0 99 16: 0 0 99 | 7 0 99 17: 0 0 99 | 6 0 99 18: 0 0 99 | 7 0 99 19: 0 0 99 | 5 0 99 20: 0 0 99 | 6 0 99 21: 0 0 99 | 4 0 99 22: 0 0 99 | 6 0 99 23: 0 0 99 | 9 0 99 24: 0 0 99 | 3 0 99 25: 0 0 99 | 4 0 99 26: 0 0 99 | 5 0 99 27: 0 0 99 | 3 0 99 28: 0 0 99 | 2 0 99 29: 0 0 99 | 5 0 99 30: 0 0 99 | 7 0 99 31: 8 0 100 | 84 0 100 read | write discontiguous blocks rpcs % cum % | rpcs % cum % 0: 12734 95 95 | 31323 83 83 1: 413 3 98 | 2369 6 89 2: 57 0 98 | 992 2 92 3: 22 0 98 | 1901 5 97 4: 14 0 98 | 393 1 98 5: 11 0 99 | 232 0 98 6: 10 0 99 | 105 0 99 7: 16 0 99 | 79 0 99 8: 35 0 99 | 76 0 99 9: 2 0 99 | 39 0 99 10: 3 0 99 | 19 0 99 11: 3 0 99 | 14 0 99 12: 1 0 99 | 3 0 99 13: 0 0 99 | 5 0 99 14: 1 0 99 | 4 0 99 15: 0 0 99 | 9 0 99 16: 3 0 99 | 4 0 99 17: 1 0 99 | 1 0 99 18: 2 0 99 | 6 0 99 19: 0 0 99 | 4 0 99 20: 0 0 99 | 0 0 99 21: 2 0 99 | 0 0 99 22: 2 0 99 | 0 0 99 23: 1 0 99 | 3 0 99 24: 1 0 99 | 0 0 99 25: 0 0 99 | 0 0 99 26: 1 0 99 | 1 0 99 27: 0 0 99 | 0 0 99 28: 0 0 99 | 0 0 99 29: 0 0 99 | 1 0 99 30: 0 0 99 | 2 0 99 31: 47 0 100 | 11 0 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 1: 10491 93 93 | 10351 27 27 2: 518 4 98 | 23586 62 90 3: 162 1 99 | 1177 3 93 4: 16 0 99 | 1832 4 98 5: 7 0 99 | 357 0 99 6: 3 0 99 | 171 0 99 7: 4 0 99 | 53 0 99 8: 12 0 99 | 28 0 99 9: 0 0 99 | 15 0 99 10: 1 0 99 | 2 0 99 11: 1 0 99 | 2 0 99 12: 0 0 99 | 1 0 99 13: 0 0 99 | 0 0 99 14: 1 0 99 | 0 0 99 15: 0 0 99 | 0 0 99 16: 0 0 99 | 0 0 99 17: 0 0 99 | 1 0 99 18: 1 0 99 | 0 0 99 19: 1 0 99 | 2 0 99 20: 1 0 99 | 1 0 99 21: 0 0 99 | 0 0 99 22: 0 0 99 | 0 0 99 23: 0 0 99 | 0 0 99 24: 1 0 99 | 3 0 99 25: 0 0 99 | 0 0 99 26: 0 0 99 | 0 0 99 27: 2 0 99 | 1 0 99 28: 0 0 99 | 0 0 99 29: 0 0 99 | 0 0 99 30: 1 0 99 | 1 0 99 31: 3 0 100 | 12 0 100 read | write disk I/Os in flight ios % cum % | ios % cum % 1: 8714 69 69 | 24576 33 33 2: 807 6 75 | 15602 21 55 3: 406 3 79 | 5832 8 63 4: 258 2 81 | 5092 7 70 5: 203 1 82 | 3300 4 74 6: 179 1 84 | 2965 4 78 7: 164 1 85 | 2297 3 82 8: 155 1 86 | 2094 2 85 9: 139 1 87 | 1725 2 87 10: 131 1 88 | 1615 2 89 11: 129 1 89 | 1287 1 91 12: 128 1 90 | 1156 1 92 13: 121 0 91 | 967 1 94 14: 134 1 92 | 900 1 95 15: 121 0 93 | 837 1 96 16: 126 1 94 | 780 1 97 17: 122 0 95 | 604 0 98 18: 94 0 96 | 406 0 99 19: 91 0 97 | 189 0 99 20: 61 0 97 | 118 0 99 21: 50 0 98 | 46 0 99 22: 43 0 98 | 30 0 99 23: 31 0 98 | 24 0 99 24: 23 0 98 | 22 0 99 25: 14 0 99 | 15 0 99 26: 9 0 99 | 16 0 99 27: 13 0 99 | 20 0 99 28: 9 0 99 | 12 0 99 29: 10 0 99 | 12 0 99 30: 11 0 99 | 12 0 99 31: 67 0 100 | 86 0 100 read | write I/O time (1/1000s) ios % cum % | ios % cum % 1: 7184 63 63 | 15607 41 41 2: 41 0 64 | 761 2 43 4: 43 0 64 | 894 2 45 8: 153 1 66 | 1065 2 48 16: 776 6 73 | 3242 8 57 32: 1077 9 82 | 6278 16 74 64: 1004 8 91 | 6679 17 91 128: 587 5 96 | 2299 6 97 256: 171 1 98 | 603 1 99 512: 176 1 99 | 146 0 99 1K: 14 0 100 | 19 0 99 2K: 0 0 100 | 3 0 100 read | write disk I/O size ios % cum % | ios % cum % 4K: 5172 41 41 | 7856 10 10 8K: 75 0 41 | 1659 2 13 16K: 58 0 42 | 321 0 13 32K: 119 0 43 | 352 0 14 64K: 265 2 45 | 460 0 14 128K: 101 0 46 | 803 1 15 256K: 148 1 47 | 1198 1 17 512K: 294 2 49 | 1252 1 19 1M: 5270 41 91 | 11563 15 35 2M: 1061 8 100 | 47173 64 100 read | write block maps msec maps % cum % | maps % cum % 1: 31390 99 99 | 39466 99 99 2: 0 0 99 | 2 0 99 4: 2 0 99 | 2 0 99 8: 0 0 99 | 11 0 99 16: 8 0 99 | 15 0 99 32: 11 0 99 | 33 0 99 64: 17 0 99 | 44 0 99 128: 13 0 99 | 38 0 99 256: 2 0 99 | 17 0 99 512: 0 0 99 | 41 0 99 1K: 1 0 100 | 3 0 100 osd-ldiskfs.nbp12-OST0004.brw_stats= snapshot_time: 1693236421.171318010 secs.nsecs start_time: 1693184207.603433088 secs.nsecs elapsed_time: 52213.567884922 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 1: 9074 27 27 | 3399 10 10 2: 1766 5 32 | 723 2 12 4: 1743 5 38 | 401 1 14 8: 2096 6 44 | 190 0 14 16: 2373 7 51 | 164 0 15 32: 2574 7 59 | 217 0 15 64: 5384 16 75 | 107 0 16 128: 1325 4 79 | 83 0 16 256: 5528 16 96 | 639 1 18 512: 495 1 97 | 567 1 20 1K: 696 2 100 | 25787 79 100 read | write discontiguous pages rpcs % cum % | rpcs % cum % 0: 29196 88 88 | 29420 91 91 1: 2004 6 94 | 621 1 93 2: 859 2 96 | 493 1 94 3: 384 1 98 | 1719 5 99 4: 230 0 98 | 4 0 99 5: 153 0 99 | 0 0 99 6: 82 0 99 | 1 0 99 7: 64 0 99 | 0 0 99 8: 24 0 99 | 0 0 99 9: 15 0 99 | 0 0 99 10: 12 0 99 | 0 0 99 11: 5 0 99 | 1 0 99 12: 6 0 99 | 0 0 99 13: 6 0 99 | 0 0 99 14: 4 0 99 | 2 0 99 15: 1 0 99 | 2 0 99 16: 2 0 99 | 1 0 99 17: 2 0 99 | 1 0 99 18: 1 0 99 | 2 0 99 19: 0 0 99 | 0 0 99 20: 0 0 99 | 2 0 99 21: 0 0 99 | 4 0 99 22: 1 0 99 | 1 0 99 23: 0 0 99 | 1 0 99 24: 0 0 99 | 1 0 99 25: 0 0 99 | 0 0 99 26: 1 0 99 | 0 0 99 27: 0 0 99 | 0 0 99 28: 0 0 99 | 0 0 99 29: 1 0 99 | 0 0 99 30: 0 0 99 | 0 0 99 31: 1 0 100 | 1 0 100 read | write discontiguous blocks rpcs % cum % | rpcs % cum % 0: 28756 86 86 | 27694 85 85 1: 2324 7 94 | 1885 5 91 2: 896 2 96 | 758 2 93 3: 398 1 97 | 1715 5 99 4: 233 0 98 | 147 0 99 5: 159 0 99 | 24 0 99 6: 89 0 99 | 20 0 99 7: 66 0 99 | 8 0 99 8: 56 0 99 | 8 0 99 9: 14 0 99 | 5 0 99 10: 18 0 99 | 1 0 99 11: 7 0 99 | 0 0 99 12: 6 0 99 | 1 0 99 13: 6 0 99 | 1 0 99 14: 5 0 99 | 3 0 99 15: 1 0 99 | 2 0 99 16: 3 0 99 | 1 0 99 17: 2 0 99 | 1 0 99 18: 1 0 99 | 1 0 99 19: 0 0 99 | 0 0 99 20: 0 0 99 | 0 0 99 21: 1 0 99 | 1 0 99 22: 1 0 99 | 0 0 99 23: 0 0 99 | 0 0 99 24: 0 0 99 | 0 0 99 25: 0 0 99 | 0 0 99 26: 1 0 99 | 0 0 99 27: 1 0 99 | 0 0 99 28: 0 0 99 | 0 0 99 29: 1 0 99 | 0 0 99 30: 0 0 99 | 0 0 99 31: 9 0 100 | 1 0 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 1: 26615 86 86 | 5982 18 18 2: 2378 7 93 | 23345 72 90 3: 901 2 96 | 1053 3 94 4: 403 1 97 | 1700 5 99 5: 237 0 98 | 145 0 99 6: 150 0 99 | 31 0 99 7: 87 0 99 | 8 0 99 8: 67 0 99 | 3 0 99 9: 27 0 99 | 0 0 99 10: 14 0 99 | 0 0 99 11: 17 0 99 | 0 0 99 12: 6 0 99 | 0 0 99 13: 6 0 99 | 0 0 99 14: 6 0 99 | 1 0 99 15: 4 0 99 | 2 0 99 16: 1 0 99 | 2 0 99 17: 2 0 99 | 1 0 99 18: 2 0 99 | 1 0 99 19: 1 0 99 | 1 0 99 20: 0 0 99 | 0 0 99 21: 0 0 99 | 0 0 99 22: 0 0 99 | 1 0 99 23: 1 0 99 | 0 0 99 24: 0 0 99 | 0 0 99 25: 0 0 99 | 0 0 99 26: 0 0 99 | 0 0 99 27: 0 0 99 | 0 0 99 28: 1 0 99 | 0 0 99 29: 0 0 99 | 0 0 99 30: 1 0 99 | 0 0 99 31: 2 0 100 | 1 0 100 read | write disk I/Os in flight ios % cum % | ios % cum % 1: 9621 23 23 | 19590 30 30 2: 3328 8 32 | 17415 27 57 3: 1565 3 36 | 4910 7 65 4: 1387 3 39 | 4265 6 72 5: 1026 2 42 | 2609 4 76 6: 896 2 44 | 2276 3 80 7: 825 2 46 | 1691 2 82 8: 920 2 48 | 1617 2 85 9: 748 1 50 | 1403 2 87 10: 718 1 52 | 1352 2 89 11: 726 1 54 | 1103 1 91 12: 664 1 55 | 1007 1 92 13: 592 1 57 | 895 1 94 14: 596 1 58 | 847 1 95 15: 597 1 60 | 750 1 96 16: 573 1 61 | 679 1 97 17: 494 1 62 | 568 0 98 18: 508 1 64 | 415 0 99 19: 468 1 65 | 196 0 99 20: 479 1 66 | 119 0 99 21: 557 1 67 | 40 0 99 22: 495 1 69 | 25 0 99 23: 469 1 70 | 15 0 99 24: 438 1 71 | 8 0 99 25: 381 0 72 | 6 0 99 26: 335 0 73 | 3 0 99 27: 295 0 74 | 2 0 99 28: 316 0 74 | 1 0 100 29: 261 0 75 | 0 0 100 30: 230 0 76 | 0 0 100 31: 9624 23 100 | 0 0 100 read | write I/O time (1/1000s) ios % cum % | ios % cum % 1: 19766 63 63 | 10950 33 33 2: 153 0 64 | 965 2 36 4: 263 0 65 | 1431 4 41 8: 653 2 67 | 1978 6 47 16: 1795 5 73 | 3453 10 58 32: 2134 6 80 | 6138 19 77 64: 1826 5 85 | 5163 15 93 128: 1092 3 89 | 1704 5 98 256: 854 2 92 | 380 1 99 512: 2106 6 99 | 90 0 99 1K: 267 0 99 | 14 0 99 2K: 20 0 100 | 11 0 100 read | write disk I/O size ios % cum % | ios % cum % 4K: 8147 20 20 | 4627 7 7 8K: 2184 5 25 | 451 0 7 16K: 2417 6 31 | 143 0 8 32K: 3071 7 39 | 186 0 8 64K: 3408 8 47 | 225 0 8 128K: 3916 9 57 | 294 0 9 256K: 8549 21 78 | 229 0 9 512K: 1563 3 82 | 333 0 10 1M: 5643 14 96 | 10443 16 26 2M: 1234 3 100 | 46876 73 100 read | write block maps msec maps % cum % | maps % cum % 1: 50646 99 99 | 36313 99 99 2: 1 0 99 | 1 0 99 4: 2 0 99 | 4 0 99 8: 0 0 99 | 8 0 99 16: 18 0 99 | 14 0 99 32: 15 0 99 | 30 0 99 64: 16 0 99 | 36 0 99 128: 1 0 99 | 58 0 99 256: 1 0 99 | 40 0 99 512: 4 0 100 | 74 0 99 1K: 0 0 100 | 6 0 100 osd-ldiskfs.nbp12-OST0005.brw_stats= snapshot_time: 1693236421.172682857 secs.nsecs start_time: 1693184207.920602633 secs.nsecs elapsed_time: 52213.252080224 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 1: 3249 24 24 | 2050 6 6 2: 126 0 25 | 630 1 7 4: 36 0 25 | 258 0 8 8: 61 0 25 | 272 0 9 16: 78 0 26 | 1224 3 13 32: 72 0 26 | 433 1 14 64: 151 1 28 | 701 2 16 128: 293 2 30 | 483 1 18 256: 6921 51 81 | 869 2 20 512: 1815 13 95 | 676 2 22 1K: 650 4 100 | 26003 77 100 read | write discontiguous pages rpcs % cum % | rpcs % cum % 0: 13184 98 98 | 31098 92 92 1: 238 1 99 | 704 2 94 2: 16 0 99 | 415 1 95 3: 1 0 99 | 1146 3 99 4: 0 0 99 | 48 0 99 5: 0 0 99 | 28 0 99 6: 1 0 99 | 18 0 99 7: 3 0 99 | 8 0 99 8: 0 0 99 | 4 0 99 9: 1 0 99 | 5 0 99 10: 0 0 99 | 1 0 99 11: 0 0 99 | 6 0 99 12: 0 0 99 | 2 0 99 13: 0 0 99 | 0 0 99 14: 0 0 99 | 3 0 99 15: 0 0 99 | 6 0 99 16: 0 0 99 | 5 0 99 17: 0 0 99 | 3 0 99 18: 0 0 99 | 4 0 99 19: 0 0 99 | 2 0 99 20: 0 0 99 | 0 0 99 21: 0 0 99 | 5 0 99 22: 0 0 99 | 2 0 99 23: 0 0 99 | 1 0 99 24: 0 0 99 | 3 0 99 25: 0 0 99 | 0 0 99 26: 0 0 99 | 2 0 99 27: 0 0 99 | 2 0 99 28: 0 0 99 | 0 0 99 29: 0 0 99 | 1 0 99 30: 0 0 99 | 1 0 99 31: 8 0 100 | 76 0 100 read | write discontiguous blocks rpcs % cum % | rpcs % cum % 0: 12574 93 93 | 27164 80 80 1: 721 5 98 | 3548 10 91 2: 59 0 99 | 952 2 94 3: 9 0 99 | 1209 3 97 4: 8 0 99 | 318 0 98 5: 12 0 99 | 124 0 99 6: 11 0 99 | 55 0 99 7: 3 0 99 | 45 0 99 8: 25 0 99 | 29 0 99 9: 4 0 99 | 23 0 99 10: 2 0 99 | 16 0 99 11: 2 0 99 | 11 0 99 12: 2 0 99 | 11 0 99 13: 1 0 99 | 4 0 99 14: 1 0 99 | 3 0 99 15: 1 0 99 | 8 0 99 16: 0 0 99 | 7 0 99 17: 0 0 99 | 28 0 99 18: 0 0 99 | 14 0 99 19: 1 0 99 | 1 0 99 20: 0 0 99 | 2 0 99 21: 0 0 99 | 1 0 99 22: 0 0 99 | 2 0 99 23: 0 0 99 | 0 0 99 24: 0 0 99 | 8 0 99 25: 0 0 99 | 1 0 99 26: 0 0 99 | 1 0 99 27: 1 0 99 | 0 0 99 28: 1 0 99 | 5 0 99 29: 0 0 99 | 2 0 99 30: 1 0 99 | 0 0 99 31: 13 0 100 | 7 0 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 1: 10335 90 90 | 6811 20 20 2: 875 7 98 | 23325 69 89 3: 159 1 99 | 1994 5 95 4: 8 0 99 | 1083 3 98 5: 3 0 99 | 243 0 99 6: 4 0 99 | 77 0 99 7: 3 0 99 | 32 0 99 8: 2 0 99 | 4 0 99 9: 1 0 99 | 1 0 99 10: 3 0 99 | 2 0 99 11: 0 0 99 | 1 0 99 12: 1 0 99 | 3 0 99 13: 1 0 99 | 1 0 99 14: 1 0 99 | 0 0 99 15: 1 0 99 | 3 0 99 16: 1 0 99 | 3 0 99 17: 0 0 99 | 3 0 99 18: 0 0 99 | 3 0 99 19: 0 0 99 | 2 0 99 20: 0 0 99 | 1 0 99 21: 0 0 99 | 0 0 99 22: 0 0 99 | 1 0 99 23: 0 0 99 | 0 0 99 24: 0 0 99 | 0 0 99 25: 0 0 99 | 1 0 99 26: 0 0 99 | 0 0 99 27: 0 0 99 | 1 0 99 28: 0 0 99 | 0 0 99 29: 0 0 99 | 0 0 99 30: 0 0 99 | 0 0 99 31: 9 0 100 | 4 0 100 read | write disk I/Os in flight ios % cum % | ios % cum % 1: 5002 26 26 | 19228 29 29 2: 724 3 29 | 13725 20 49 3: 369 1 31 | 6596 9 59 4: 306 1 33 | 5471 8 67 5: 288 1 34 | 3295 4 72 6: 259 1 36 | 2821 4 77 7: 261 1 37 | 2207 3 80 8: 238 1 38 | 2036 3 83 9: 226 1 39 | 1653 2 86 10: 203 1 41 | 1514 2 88 11: 183 0 41 | 1228 1 90 12: 162 0 42 | 1121 1 91 13: 160 0 43 | 942 1 93 14: 145 0 44 | 907 1 94 15: 104 0 44 | 925 1 96 16: 78 0 45 | 876 1 97 17: 69 0 45 | 750 1 98 18: 67 0 46 | 514 0 99 19: 55 0 46 | 239 0 99 20: 47 0 46 | 131 0 99 21: 45 0 46 | 25 0 99 22: 66 0 47 | 19 0 99 23: 57 0 47 | 8 0 99 24: 58 0 47 | 9 0 99 25: 73 0 48 | 7 0 99 26: 75 0 48 | 2 0 99 27: 59 0 48 | 2 0 99 28: 54 0 49 | 2 0 99 29: 48 0 49 | 2 0 99 30: 44 0 49 | 2 0 99 31: 9664 50 100 | 4 0 100 read | write I/O time (1/1000s) ios % cum % | ios % cum % 1: 3861 33 33 | 10793 32 32 2: 26 0 34 | 566 1 33 4: 21 0 34 | 1051 3 36 8: 106 0 35 | 903 2 39 16: 678 5 41 | 3176 9 49 32: 1084 9 50 | 7092 21 70 64: 1453 12 63 | 6773 20 90 128: 1135 9 73 | 2455 7 97 256: 1335 11 85 | 609 1 99 512: 1357 11 96 | 172 0 99 1K: 320 2 99 | 9 0 100 2K: 25 0 99 | 0 0 100 4K: 0 0 99 | 0 0 100 8K: 1 0 99 | 0 0 100 16K: 5 0 100 | 0 0 100 read | write disk I/O size ios % cum % | ios % cum % 4K: 7710 40 40 | 4203 6 6 8K: 173 0 41 | 910 1 7 16K: 123 0 41 | 303 0 8 32K: 60 0 42 | 372 0 8 64K: 93 0 42 | 894 1 10 128K: 96 0 43 | 652 0 11 256K: 196 1 44 | 950 1 12 512K: 387 2 46 | 1059 1 14 1M: 7780 40 86 | 8212 12 26 2M: 2571 13 100 | 48706 73 100 read | write block maps msec maps % cum % | maps % cum % 1: 26924 99 99 | 37220 98 98 2: 2 0 99 | 8 0 98 4: 3 0 99 | 11 0 98 8: 6 0 99 | 27 0 98 16: 22 0 99 | 36 0 98 32: 33 0 99 | 102 0 99 64: 27 0 99 | 93 0 99 128: 17 0 99 | 91 0 99 256: 11 0 99 | 42 0 99 512: 8 0 99 | 91 0 99 1K: 10 0 99 | 14 0 99 2K: 1 0 99 | 3 0 99 4K: 1 0 100 | 1 0 100 osd-ldiskfs.nbp12-OST0006.brw_stats= snapshot_time: 1693236421.173492779 secs.nsecs start_time: 1693184209.301523743 secs.nsecs elapsed_time: 52211.871969036 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 1: 6763 50 50 | 2484 7 7 2: 207 1 51 | 858 2 10 4: 72 0 52 | 245 0 11 8: 49 0 52 | 163 0 11 16: 184 1 54 | 164 0 12 32: 54 0 54 | 170 0 13 64: 88 0 55 | 150 0 13 128: 190 1 56 | 155 0 14 256: 3271 24 80 | 526 1 15 512: 120 0 81 | 435 1 17 1K: 2467 18 100 | 25993 82 100 read | write discontiguous pages rpcs % cum % | rpcs % cum % 0: 13171 97 97 | 29888 95 95 1: 231 1 99 | 454 1 96 2: 47 0 99 | 269 0 97 3: 0 0 99 | 564 1 99 4: 1 0 99 | 63 0 99 5: 0 0 99 | 23 0 99 6: 0 0 99 | 21 0 99 7: 9 0 99 | 13 0 99 8: 0 0 99 | 10 0 99 9: 0 0 99 | 4 0 99 10: 0 0 99 | 7 0 99 11: 0 0 99 | 2 0 99 12: 0 0 99 | 3 0 99 13: 0 0 99 | 2 0 99 14: 0 0 99 | 3 0 99 15: 0 0 99 | 0 0 99 16: 0 0 99 | 2 0 99 17: 0 0 99 | 2 0 99 18: 0 0 99 | 3 0 99 19: 0 0 99 | 0 0 99 20: 0 0 99 | 4 0 99 21: 0 0 99 | 2 0 99 22: 0 0 99 | 1 0 99 23: 0 0 99 | 0 0 99 24: 0 0 99 | 0 0 99 25: 1 0 99 | 1 0 99 26: 0 0 99 | 0 0 99 27: 0 0 99 | 0 0 99 28: 0 0 99 | 0 0 99 29: 0 0 99 | 0 0 99 30: 0 0 99 | 1 0 99 31: 5 0 100 | 1 0 100 read | write discontiguous blocks rpcs % cum % | rpcs % cum % 0: 12095 89 89 | 26739 85 85 1: 991 7 97 | 3068 9 95 2: 120 0 98 | 529 1 96 3: 23 0 98 | 580 1 98 4: 17 0 98 | 160 0 99 5: 25 0 98 | 63 0 99 6: 9 0 98 | 45 0 99 7: 16 0 98 | 21 0 99 8: 85 0 99 | 35 0 99 9: 3 0 99 | 21 0 99 10: 8 0 99 | 8 0 99 11: 3 0 99 | 17 0 99 12: 2 0 99 | 12 0 99 13: 1 0 99 | 11 0 99 14: 0 0 99 | 8 0 99 15: 8 0 99 | 5 0 99 16: 1 0 99 | 7 0 99 17: 3 0 99 | 1 0 99 18: 3 0 99 | 5 0 99 19: 1 0 99 | 3 0 99 20: 2 0 99 | 1 0 99 21: 1 0 99 | 0 0 99 22: 0 0 99 | 0 0 99 23: 2 0 99 | 0 0 99 24: 0 0 99 | 0 0 99 25: 1 0 99 | 0 0 99 26: 0 0 99 | 1 0 99 27: 1 0 99 | 1 0 99 28: 0 0 99 | 0 0 99 29: 0 0 99 | 0 0 99 30: 1 0 99 | 0 0 99 31: 43 0 100 | 2 0 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 1: 6390 70 70 | 4877 15 15 2: 1896 20 91 | 23718 75 91 3: 649 7 98 | 1884 6 97 4: 29 0 99 | 600 1 99 5: 10 0 99 | 174 0 99 6: 6 0 99 | 37 0 99 7: 3 0 99 | 15 0 99 8: 12 0 99 | 9 0 99 9: 4 0 99 | 9 0 99 10: 2 0 99 | 2 0 99 11: 4 0 99 | 3 0 99 12: 2 0 99 | 6 0 99 13: 0 0 99 | 3 0 99 14: 0 0 99 | 1 0 99 15: 0 0 99 | 1 0 99 16: 1 0 99 | 0 0 99 17: 3 0 99 | 1 0 99 18: 3 0 99 | 1 0 99 19: 2 0 99 | 1 0 99 20: 2 0 99 | 0 0 99 21: 4 0 99 | 0 0 99 22: 1 0 99 | 0 0 99 23: 0 0 99 | 0 0 99 24: 0 0 99 | 0 0 99 25: 0 0 99 | 0 0 99 26: 4 0 99 | 0 0 99 27: 1 0 99 | 0 0 99 28: 1 0 99 | 0 0 99 29: 0 0 99 | 0 0 99 30: 0 0 99 | 0 0 99 31: 1 0 100 | 1 0 100 read | write disk I/Os in flight ios % cum % | ios % cum % 1: 7334 56 56 | 19997 32 32 2: 1556 11 68 | 16503 26 58 3: 621 4 72 | 5171 8 67 4: 280 2 74 | 3985 6 73 5: 233 1 76 | 2627 4 77 6: 214 1 78 | 2285 3 81 7: 160 1 79 | 1852 2 84 8: 142 1 80 | 1717 2 87 9: 122 0 81 | 1349 2 89 10: 98 0 82 | 1190 1 91 11: 83 0 82 | 951 1 92 12: 86 0 83 | 857 1 94 13: 80 0 84 | 663 1 95 14: 80 0 84 | 649 1 96 15: 75 0 85 | 592 0 97 16: 70 0 85 | 569 0 98 17: 65 0 86 | 471 0 99 18: 61 0 86 | 317 0 99 19: 46 0 87 | 132 0 99 20: 42 0 87 | 75 0 99 21: 40 0 87 | 32 0 99 22: 34 0 88 | 7 0 99 23: 33 0 88 | 3 0 99 24: 29 0 88 | 2 0 100 25: 27 0 88 | 0 0 100 26: 30 0 89 | 0 0 100 27: 36 0 89 | 0 0 100 28: 43 0 89 | 0 0 100 29: 41 0 89 | 0 0 100 30: 39 0 90 | 0 0 100 31: 1270 9 100 | 0 0 100 read | write I/O time (1/1000s) ios % cum % | ios % cum % 1: 5411 59 59 | 11326 36 36 2: 126 1 61 | 973 3 39 4: 28 0 61 | 1495 4 44 8: 129 1 63 | 1916 6 50 16: 714 7 70 | 2818 8 59 32: 823 9 80 | 5609 17 77 64: 719 7 88 | 4935 15 92 128: 355 3 91 | 1732 5 98 256: 277 3 95 | 408 1 99 512: 365 4 99 | 110 0 99 1K: 81 0 99 | 21 0 100 2K: 2 0 100 | 0 0 100 read | write disk I/O size ios % cum % | ios % cum % 4K: 2811 21 21 | 3895 6 6 8K: 119 0 22 | 421 0 6 16K: 115 0 23 | 238 0 7 32K: 196 1 24 | 341 0 7 64K: 370 2 27 | 377 0 8 128K: 131 1 28 | 472 0 9 256K: 174 1 29 | 491 0 10 512K: 309 2 32 | 727 1 11 1M: 4383 33 65 | 5201 8 19 2M: 4462 34 100 | 49833 80 100 read | write block maps msec maps % cum % | maps % cum % 1: 24377 99 99 | 34221 99 99 2: 5 0 99 | 5 0 99 4: 3 0 99 | 7 0 99 8: 3 0 99 | 7 0 99 16: 35 0 99 | 21 0 99 32: 31 0 99 | 42 0 99 64: 37 0 99 | 67 0 99 128: 24 0 99 | 57 0 99 256: 0 0 99 | 26 0 99 512: 21 0 99 | 67 0 99 1K: 9 0 100 | 6 0 100 osd-ldiskfs.nbp12-OST0007.brw_stats= snapshot_time: 1693236421.174443559 secs.nsecs start_time: 1693184214.837810688 secs.nsecs elapsed_time: 52206.336632871 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 1: 9522 11 11 | 425 1 1 2: 5206 6 18 | 213 0 1 4: 4181 5 23 | 138 0 2 8: 5303 6 30 | 143 0 2 16: 6719 8 38 | 149 0 3 32: 8635 10 49 | 202 0 3 64: 12724 15 65 | 233 0 4 128: 12447 15 80 | 299 0 5 256: 9194 11 92 | 1408 4 9 512: 4070 5 97 | 1130 3 12 1K: 2317 2 100 | 29783 87 100 read | write discontiguous pages rpcs % cum % | rpcs % cum % 0: 64394 80 80 | 29640 86 86 1: 7775 9 89 | 878 2 89 2: 3197 3 93 | 821 2 91 3: 1855 2 96 | 2693 7 99 4: 1198 1 97 | 54 0 99 5: 761 0 98 | 15 0 99 6: 471 0 99 | 4 0 99 7: 260 0 99 | 3 0 99 8: 158 0 99 | 2 0 99 9: 84 0 99 | 1 0 99 10: 56 0 99 | 1 0 99 11: 25 0 99 | 0 0 99 12: 24 0 99 | 0 0 99 13: 8 0 99 | 2 0 99 14: 11 0 99 | 0 0 99 15: 7 0 99 | 0 0 99 16: 9 0 99 | 1 0 99 17: 11 0 99 | 1 0 99 18: 4 0 99 | 1 0 99 19: 1 0 99 | 0 0 99 20: 3 0 99 | 1 0 99 21: 1 0 99 | 1 0 99 22: 0 0 99 | 1 0 99 23: 0 0 99 | 0 0 99 24: 1 0 99 | 0 0 99 25: 0 0 99 | 0 0 99 26: 0 0 99 | 1 0 99 27: 2 0 99 | 1 0 99 28: 0 0 99 | 0 0 99 29: 0 0 99 | 0 0 99 30: 0 0 99 | 0 0 99 31: 2 0 100 | 1 0 100 read | write discontiguous blocks rpcs % cum % | rpcs % cum % 0: 63388 78 78 | 25439 74 74 1: 8507 10 89 | 3384 9 84 2: 3250 4 93 | 1228 3 88 3: 1891 2 95 | 2630 7 95 4: 1206 1 97 | 499 1 97 5: 780 0 98 | 310 0 98 6: 507 0 99 | 135 0 98 7: 278 0 99 | 95 0 98 8: 207 0 99 | 83 0 99 9: 92 0 99 | 67 0 99 10: 57 0 99 | 55 0 99 11: 25 0 99 | 41 0 99 12: 36 0 99 | 18 0 99 13: 9 0 99 | 20 0 99 14: 11 0 99 | 30 0 99 15: 8 0 99 | 23 0 99 16: 9 0 99 | 17 0 99 17: 11 0 99 | 11 0 99 18: 7 0 99 | 4 0 99 19: 2 0 99 | 7 0 99 20: 6 0 99 | 5 0 99 21: 1 0 99 | 5 0 99 22: 0 0 99 | 5 0 99 23: 2 0 99 | 4 0 99 24: 1 0 99 | 3 0 99 25: 0 0 99 | 2 0 99 26: 0 0 99 | 0 0 99 27: 2 0 99 | 1 0 99 28: 0 0 99 | 1 0 99 29: 0 0 99 | 0 0 99 30: 0 0 99 | 0 0 99 31: 25 0 100 | 1 0 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 1: 61331 78 78 | 3142 9 9 2: 8696 11 89 | 24250 71 80 3: 3433 4 93 | 3024 8 89 4: 1892 2 95 | 2731 8 97 5: 1206 1 97 | 618 1 98 6: 779 0 98 | 174 0 99 7: 506 0 99 | 69 0 99 8: 275 0 99 | 37 0 99 9: 169 0 99 | 17 0 99 10: 92 0 99 | 14 0 99 11: 55 0 99 | 5 0 99 12: 23 0 99 | 5 0 99 13: 28 0 99 | 7 0 99 14: 8 0 99 | 9 0 99 15: 11 0 99 | 5 0 99 16: 7 0 99 | 2 0 99 17: 9 0 99 | 3 0 99 18: 11 0 99 | 4 0 99 19: 5 0 99 | 0 0 99 20: 1 0 99 | 1 0 99 21: 3 0 99 | 2 0 99 22: 1 0 99 | 2 0 99 23: 0 0 99 | 1 0 99 24: 0 0 99 | 0 0 99 25: 1 0 99 | 0 0 99 26: 0 0 99 | 0 0 99 27: 0 0 99 | 0 0 99 28: 2 0 99 | 0 0 99 29: 0 0 99 | 0 0 99 30: 0 0 99 | 0 0 99 31: 8 0 100 | 1 0 100 read | write disk I/Os in flight ios % cum % | ios % cum % 1: 3999 3 3 | 17428 22 22 2: 1287 1 4 | 16532 21 43 3: 917 0 5 | 8946 11 55 4: 800 0 5 | 7607 9 65 5: 817 0 6 | 4413 5 70 6: 851 0 7 | 3586 4 75 7: 948 0 8 | 2805 3 79 8: 890 0 8 | 2520 3 82 9: 936 0 9 | 1982 2 84 10: 1049 0 10 | 1779 2 87 11: 1057 0 11 | 1458 1 89 12: 1009 0 12 | 1328 1 90 13: 962 0 13 | 1210 1 92 14: 966 0 13 | 1183 1 93 15: 1031 0 14 | 1107 1 95 16: 1011 0 15 | 1084 1 96 17: 1017 0 16 | 897 1 97 18: 1093 0 17 | 595 0 98 19: 1115 0 18 | 307 0 98 20: 1136 0 19 | 193 0 99 21: 1033 0 20 | 94 0 99 22: 1018 0 20 | 67 0 99 23: 1086 0 21 | 51 0 99 24: 1117 0 22 | 34 0 99 25: 1055 0 23 | 29 0 99 26: 1065 0 24 | 22 0 99 27: 1095 0 25 | 18 0 99 28: 1222 1 26 | 13 0 99 29: 1181 0 27 | 15 0 99 30: 1137 0 28 | 13 0 99 31: 85001 71 100 | 253 0 100 read | write I/O time (1/1000s) ios % cum % | ios % cum % 1: 30310 38 38 | 7605 22 22 2: 635 0 39 | 679 1 24 4: 866 1 40 | 940 2 27 8: 1865 2 42 | 1073 3 30 16: 3403 4 47 | 3251 9 39 32: 5457 6 54 | 8692 25 65 64: 7768 9 64 | 7725 22 87 128: 8173 10 74 | 2871 8 96 256: 7560 9 84 | 866 2 98 512: 10560 13 97 | 285 0 99 1K: 1921 2 99 | 68 0 99 2K: 34 0 100 | 42 0 99 4K: 0 0 100 | 12 0 99 8K: 0 0 100 | 13 0 99 16K: 0 0 100 | 1 0 100 read | write disk I/O size ios % cum % | ios % cum % 4K: 13955 11 11 | 1187 1 1 8K: 8414 7 18 | 369 0 2 16K: 6494 5 24 | 430 0 2 32K: 8155 6 31 | 1021 1 3 64K: 10303 8 39 | 1194 1 5 128K: 13476 11 51 | 1297 1 7 256K: 20581 17 68 | 1248 1 8 512K: 22531 18 87 | 1743 2 10 1M: 11320 9 96 | 17749 22 33 2M: 3672 3 100 | 51331 66 100 read | write block maps msec maps % cum % | maps % cum % 1: 125486 99 99 | 51123 99 99 2: 2 0 99 | 3 0 99 4: 0 0 99 | 4 0 99 8: 6 0 99 | 7 0 99 16: 16 0 99 | 23 0 99 32: 45 0 99 | 47 0 99 64: 39 0 99 | 97 0 99 128: 60 0 99 | 68 0 99 256: 19 0 99 | 22 0 99 512: 19 0 99 | 66 0 99 1K: 6 0 99 | 12 0 99 2K: 1 0 100 | 5 0 100 osd-ldiskfs.nbp12-OST0008.brw_stats= snapshot_time: 1693236421.175410651 secs.nsecs start_time: 1693184211.272834340 secs.nsecs elapsed_time: 52209.902576311 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 1: 12056 64 64 | 5261 9 9 2: 365 1 66 | 1606 2 12 4: 129 0 67 | 789 1 13 8: 79 0 67 | 746 1 14 16: 153 0 68 | 1743 3 18 32: 60 0 68 | 779 1 19 64: 118 0 69 | 926 1 21 128: 163 0 70 | 1061 1 22 256: 5085 27 97 | 2955 5 28 512: 106 0 98 | 2437 4 32 1K: 331 1 100 | 37981 67 100 read | write discontiguous pages rpcs % cum % | rpcs % cum % 0: 18194 97 97 | 43057 76 76 1: 389 2 99 | 2658 4 81 2: 48 0 99 | 1496 2 83 3: 5 0 99 | 6194 11 94 4: 3 0 99 | 469 0 95 5: 0 0 99 | 291 0 96 6: 0 0 99 | 217 0 96 7: 4 0 99 | 169 0 96 8: 0 0 99 | 119 0 97 9: 1 0 99 | 88 0 97 10: 1 0 100 | 68 0 97 11: 0 0 100 | 55 0 97 12: 0 0 100 | 57 0 97 13: 0 0 100 | 47 0 97 14: 0 0 100 | 48 0 97 15: 0 0 100 | 49 0 97 16: 0 0 100 | 43 0 97 17: 0 0 100 | 36 0 98 18: 0 0 100 | 40 0 98 19: 0 0 100 | 37 0 98 20: 0 0 100 | 38 0 98 21: 0 0 100 | 31 0 98 22: 0 0 100 | 40 0 98 23: 0 0 100 | 30 0 98 24: 0 0 100 | 31 0 98 25: 0 0 100 | 31 0 98 26: 0 0 100 | 36 0 98 27: 0 0 100 | 30 0 98 28: 0 0 100 | 29 0 98 29: 0 0 100 | 30 0 98 30: 0 0 100 | 29 0 98 31: 0 0 100 | 691 1 100 read | write discontiguous blocks rpcs % cum % | rpcs % cum % 0: 17649 94 94 | 35828 63 63 1: 599 3 97 | 8981 15 79 2: 108 0 98 | 2437 4 83 3: 42 0 98 | 5088 9 92 4: 24 0 98 | 1822 3 96 5: 23 0 98 | 827 1 97 6: 16 0 99 | 321 0 98 7: 10 0 99 | 205 0 98 8: 71 0 99 | 209 0 98 9: 9 0 99 | 151 0 99 10: 5 0 99 | 61 0 99 11: 1 0 99 | 38 0 99 12: 20 0 99 | 33 0 99 13: 3 0 99 | 38 0 99 14: 3 0 99 | 19 0 99 15: 0 0 99 | 17 0 99 16: 1 0 99 | 12 0 99 17: 2 0 99 | 20 0 99 18: 1 0 99 | 10 0 99 19: 2 0 99 | 11 0 99 20: 2 0 99 | 6 0 99 21: 2 0 99 | 6 0 99 22: 0 0 99 | 10 0 99 23: 0 0 99 | 22 0 99 24: 2 0 99 | 17 0 99 25: 0 0 99 | 2 0 99 26: 1 0 99 | 3 0 99 27: 1 0 99 | 2 0 99 28: 3 0 99 | 3 0 99 29: 0 0 99 | 0 0 99 30: 0 0 99 | 0 0 99 31: 45 0 100 | 85 0 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 1: 11079 95 95 | 14318 25 25 2: 340 2 98 | 29527 52 77 3: 114 0 99 | 5135 9 87 4: 17 0 99 | 4739 8 95 5: 8 0 99 | 1512 2 98 6: 4 0 99 | 619 1 99 7: 4 0 99 | 176 0 99 8: 3 0 99 | 87 0 99 9: 2 0 99 | 39 0 99 10: 4 0 99 | 37 0 99 11: 1 0 99 | 9 0 99 12: 0 0 99 | 9 0 99 13: 2 0 99 | 7 0 99 14: 0 0 99 | 8 0 99 15: 1 0 99 | 9 0 99 16: 1 0 99 | 9 0 99 17: 0 0 99 | 1 0 99 18: 0 0 99 | 2 0 99 19: 1 0 99 | 7 0 99 20: 0 0 99 | 5 0 99 21: 0 0 99 | 0 0 99 22: 0 0 99 | 2 0 99 23: 0 0 99 | 3 0 99 24: 0 0 99 | 1 0 99 25: 3 0 99 | 1 0 99 26: 0 0 99 | 1 0 99 27: 0 0 99 | 2 0 99 28: 1 0 99 | 1 0 99 29: 0 0 99 | 2 0 99 30: 1 0 100 | 0 0 99 31: 0 0 100 | 16 0 100 read | write disk I/Os in flight ios % cum % | ios % cum % 1: 8990 71 71 | 39803 32 32 2: 1276 10 81 | 30051 24 56 3: 427 3 85 | 15091 12 68 4: 250 1 87 | 11186 9 77 5: 202 1 88 | 6079 4 82 6: 188 1 90 | 4442 3 86 7: 163 1 91 | 3132 2 88 8: 170 1 92 | 2637 2 90 9: 130 1 93 | 2048 1 92 10: 131 1 94 | 1696 1 93 11: 85 0 95 | 1290 1 94 12: 81 0 96 | 1112 0 95 13: 78 0 96 | 899 0 96 14: 73 0 97 | 815 0 97 15: 61 0 97 | 743 0 97 16: 45 0 98 | 653 0 98 17: 39 0 98 | 542 0 98 18: 28 0 98 | 401 0 99 19: 35 0 99 | 221 0 99 20: 22 0 99 | 146 0 99 21: 14 0 99 | 80 0 99 22: 8 0 99 | 65 0 99 23: 6 0 99 | 55 0 99 24: 8 0 99 | 47 0 99 25: 7 0 99 | 43 0 99 26: 11 0 99 | 42 0 99 27: 9 0 99 | 37 0 99 28: 8 0 99 | 34 0 99 29: 6 0 99 | 29 0 99 30: 6 0 99 | 23 0 99 31: 6 0 100 | 240 0 100 read | write I/O time (1/1000s) ios % cum % | ios % cum % 1: 7372 63 63 | 28746 51 51 2: 23 0 63 | 1202 2 53 4: 58 0 64 | 1527 2 55 8: 192 1 65 | 2256 4 59 16: 962 8 74 | 4888 8 68 32: 1324 11 85 | 8034 14 82 64: 1100 9 95 | 6440 11 94 128: 425 3 98 | 2269 4 98 256: 63 0 99 | 698 1 99 512: 58 0 99 | 210 0 99 1K: 9 0 100 | 14 0 100 read | write disk I/O size ios % cum % | ios % cum % 4K: 5765 45 45 | 10892 8 8 8K: 98 0 46 | 6561 5 14 16K: 80 0 47 | 976 0 14 32K: 79 0 47 | 736 0 15 64K: 158 1 49 | 1383 1 16 128K: 67 0 49 | 1724 1 18 256K: 157 1 50 | 2424 1 19 512K: 231 1 52 | 3729 3 22 1M: 5333 42 95 | 34635 28 50 2M: 595 4 100 | 60622 49 100 read | write block maps msec maps % cum % | maps % cum % 1: 46829 99 99 | 73522 99 99 2: 1 0 99 | 2 0 99 4: 0 0 99 | 15 0 99 8: 1 0 99 | 11 0 99 16: 21 0 99 | 27 0 99 32: 21 0 99 | 70 0 99 64: 21 0 99 | 127 0 99 128: 6 0 99 | 118 0 99 256: 1 0 100 | 66 0 99 512: 0 0 100 | 100 0 99 1K: 0 0 100 | 29 0 99 2K: 0 0 100 | 3 0 100 osd-ldiskfs.nbp12-OST0009.brw_stats= snapshot_time: 1693236421.176390541 secs.nsecs start_time: 1693184212.129161441 secs.nsecs elapsed_time: 52209.047229100 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 1: 7084 12 12 | 1300 3 3 2: 2628 4 16 | 1087 3 7 4: 2269 3 20 | 535 1 8 8: 2914 5 25 | 213 0 9 16: 4267 7 32 | 279 0 10 32: 6088 10 43 | 282 0 10 64: 7556 12 56 | 296 0 11 128: 11272 19 75 | 281 0 12 256: 8677 14 90 | 916 2 15 512: 2138 3 94 | 670 1 17 1K: 3283 5 100 | 28044 82 100 read | write discontiguous pages rpcs % cum % | rpcs % cum % 0: 46970 80 80 | 30307 89 89 1: 5241 9 89 | 775 2 91 2: 2443 4 93 | 497 1 93 3: 1346 2 96 | 2040 6 99 4: 838 1 97 | 40 0 99 5: 523 0 98 | 32 0 99 6: 343 0 99 | 29 0 99 7: 203 0 99 | 9 0 99 8: 111 0 99 | 12 0 99 9: 65 0 99 | 6 0 99 10: 24 0 99 | 8 0 99 11: 21 0 99 | 8 0 99 12: 13 0 99 | 4 0 99 13: 8 0 99 | 4 0 99 14: 4 0 99 | 5 0 99 15: 1 0 99 | 4 0 99 16: 4 0 99 | 6 0 99 17: 2 0 99 | 9 0 99 18: 1 0 99 | 6 0 99 19: 3 0 99 | 4 0 99 20: 0 0 99 | 3 0 99 21: 1 0 99 | 5 0 99 22: 1 0 99 | 7 0 99 23: 0 0 99 | 3 0 99 24: 1 0 99 | 3 0 99 25: 0 0 99 | 3 0 99 26: 0 0 99 | 3 0 99 27: 0 0 99 | 2 0 99 28: 0 0 99 | 5 0 99 29: 0 0 99 | 3 0 99 30: 0 0 99 | 3 0 99 31: 9 0 100 | 58 0 100 read | write discontiguous blocks rpcs % cum % | rpcs % cum % 0: 46214 79 79 | 23208 68 68 1: 5827 10 89 | 6568 19 87 2: 2491 4 93 | 1441 4 92 3: 1349 2 96 | 1967 5 97 4: 848 1 97 | 326 0 98 5: 548 0 98 | 193 0 99 6: 356 0 99 | 76 0 99 7: 206 0 99 | 24 0 99 8: 140 0 99 | 17 0 99 9: 75 0 99 | 10 0 99 10: 27 0 99 | 14 0 99 11: 21 0 99 | 12 0 99 12: 13 0 99 | 7 0 99 13: 8 0 99 | 2 0 99 14: 4 0 99 | 3 0 99 15: 2 0 99 | 3 0 99 16: 5 0 99 | 3 0 99 17: 2 0 99 | 5 0 99 18: 2 0 99 | 1 0 99 19: 3 0 99 | 1 0 99 20: 0 0 99 | 0 0 99 21: 1 0 99 | 0 0 99 22: 1 0 99 | 2 0 99 23: 0 0 99 | 2 0 99 24: 1 0 99 | 0 0 99 25: 0 0 99 | 0 0 99 26: 1 0 99 | 0 0 99 27: 0 0 99 | 0 0 99 28: 1 0 99 | 1 0 99 29: 0 0 99 | 0 0 99 30: 0 0 99 | 0 0 99 31: 30 0 100 | 17 0 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 1: 42886 76 76 | 5006 14 14 2: 6750 12 88 | 21350 62 77 3: 2862 5 93 | 4875 14 92 4: 1359 2 95 | 2138 6 98 5: 854 1 97 | 356 1 99 6: 543 0 98 | 107 0 99 7: 357 0 99 | 30 0 99 8: 207 0 99 | 7 0 99 9: 118 0 99 | 6 0 99 10: 71 0 99 | 5 0 99 11: 25 0 99 | 2 0 99 12: 21 0 99 | 2 0 99 13: 14 0 99 | 1 0 99 14: 8 0 99 | 1 0 99 15: 5 0 99 | 1 0 99 16: 2 0 99 | 1 0 99 17: 6 0 99 | 2 0 99 18: 2 0 99 | 4 0 99 19: 1 0 99 | 1 0 99 20: 4 0 99 | 1 0 99 21: 0 0 99 | 0 0 99 22: 2 0 99 | 0 0 99 23: 1 0 99 | 2 0 99 24: 1 0 99 | 0 0 99 25: 2 0 99 | 0 0 99 26: 1 0 99 | 0 0 99 27: 0 0 99 | 0 0 99 28: 0 0 99 | 0 0 99 29: 0 0 99 | 1 0 99 30: 0 0 99 | 0 0 99 31: 3 0 100 | 4 0 100 read | write disk I/Os in flight ios % cum % | ios % cum % 1: 6159 7 7 | 18567 25 25 2: 2238 2 9 | 15459 20 45 3: 1412 1 11 | 7657 10 56 4: 1109 1 12 | 5808 7 64 5: 1108 1 14 | 3719 5 69 6: 1132 1 15 | 3029 4 73 7: 1263 1 16 | 2489 3 76 8: 1192 1 18 | 2317 3 79 9: 1154 1 19 | 2171 2 82 10: 1095 1 20 | 2014 2 85 11: 1145 1 22 | 1720 2 87 12: 1171 1 23 | 1505 2 89 13: 1108 1 24 | 1344 1 91 14: 1094 1 26 | 1291 1 93 15: 1106 1 27 | 1272 1 94 16: 957 1 28 | 1251 1 96 17: 927 1 29 | 1056 1 98 18: 967 1 30 | 655 0 98 19: 912 1 31 | 325 0 99 20: 1017 1 33 | 174 0 99 21: 1104 1 34 | 91 0 99 22: 1037 1 35 | 61 0 99 23: 998 1 36 | 38 0 99 24: 875 1 37 | 25 0 99 25: 866 1 38 | 20 0 99 26: 850 0 39 | 13 0 99 27: 817 0 40 | 10 0 99 28: 739 0 41 | 9 0 99 29: 708 0 42 | 8 0 99 30: 717 0 43 | 8 0 99 31: 48472 56 100 | 46 0 100 read | write I/O time (1/1000s) ios % cum % | ios % cum % 1: 24770 44 44 | 9853 29 29 2: 348 0 44 | 745 2 31 4: 525 0 45 | 890 2 33 8: 973 1 47 | 908 2 36 16: 2508 4 51 | 3003 8 45 32: 3549 6 58 | 6999 20 66 64: 4641 8 66 | 5936 17 83 128: 5001 8 75 | 3450 10 93 256: 5605 9 85 | 1501 4 98 512: 6477 11 96 | 525 1 99 1K: 1635 2 99 | 62 0 99 2K: 73 0 100 | 29 0 99 4K: 0 0 100 | 2 0 100 read | write disk I/O size ios % cum % | ios % cum % 4K: 8539 9 9 | 3272 4 4 8K: 4343 5 15 | 1275 1 6 16K: 3382 3 19 | 398 0 6 32K: 4443 5 24 | 433 0 7 64K: 6778 7 32 | 562 0 8 128K: 9462 11 43 | 879 1 9 256K: 12486 14 57 | 1098 1 10 512K: 21148 24 82 | 1706 2 12 1M: 10446 12 94 | 14254 19 32 2M: 4422 5 100 | 50275 67 100 read | write block maps msec maps % cum % | maps % cum % 1: 93740 99 99 | 44933 98 98 2: 1 0 99 | 4 0 98 4: 2 0 99 | 5 0 98 8: 7 0 99 | 26 0 98 16: 48 0 99 | 22 0 98 32: 45 0 99 | 90 0 98 64: 40 0 99 | 129 0 98 128: 24 0 99 | 152 0 99 256: 7 0 99 | 92 0 99 512: 17 0 99 | 212 0 99 1K: 8 0 99 | 20 0 100 2K: 2 0 100 | 0 0 100 osd-ldiskfs.nbp12-OST000a.brw_stats= snapshot_time: 1693236421.177757280 secs.nsecs start_time: 1693184205.163688401 secs.nsecs elapsed_time: 52216.014068879 secs.nsecs read | write pages per bulk r/w rpcs % cum % | rpcs % cum % 1: 9874 58 58 | 6913 18 18 2: 123 0 58 | 492 1 19 4: 69 0 59 | 166 0 19 8: 42 0 59 | 148 0 20 16: 126 0 60 | 1070 2 23 32: 65 0 60 | 298 0 23 64: 78 0 61 | 68 0 24 128: 196 1 62 | 75 0 24 256: 5258 31 93 | 484 1 25 512: 129 0 94 | 399 1 26 1K: 992 5 100 | 27847 73 100 read | write discontiguous pages rpcs % cum % | rpcs % cum % 0: 16743 98 98 | 36770 96 96 1: 148 0 99 | 326 0 97 2: 42 0 99 | 216 0 98 3: 1 0 99 | 558 1 99 4: 0 0 99 | 53 0 99 5: 1 0 99 | 12 0 99 6: 0 0 99 | 6 0 99 7: 6 0 99 | 4 0 99 8: 1 0 99 | 0 0 99 9: 1 0 99 | 1 0 99 10: 0 0 99 | 2 0 99 11: 1 0 99 | 1 0 99 12: 0 0 99 | 0 0 99 13: 0 0 99 | 1 0 99 14: 0 0 99 | 2 0 99 15: 0 0 99 | 1 0 99 16: 0 0 99 | 0 0 99 17: 0 0 99 | 1 0 99 18: 0 0 99 | 0 0 99 19: 0 0 99 | 0 0 99 20: 0 0 99 | 1 0 99 21: 0 0 99 | 2 0 99 22: 0 0 99 | 0 0 99 23: 0 0 99 | 1 0 99 24: 0 0 99 | 0 0 99 25: 0 0 99 | 0 0 99 26: 0 0 99 | 0 0 99 27: 0 0 99 | 0 0 99 28: 0 0 99 | 0 0 99 29: 0 0 99 | 0 0 99 30: 0 0 99 | 0 0 99 31: 8 0 100 | 2 0 100 read | write discontiguous blocks rpcs % cum % | rpcs % cum % 0: 16246 95 95 | 35240 92 92 1: 463 2 98 | 1426 3 96 2: 77 0 99 | 429 1 97 3: 12 0 99 | 623 1 99 4: 7 0 99 | 137 0 99 5: 9 0 99 | 51 0 99 6: 6 0 99 | 14 0 99 7: 10 0 99 | 11 0 99 8: 77 0 99 | 10 0 99 9: 2 0 99 | 4 0 99 10: 3 0 99 | 8 0 99 11: 1 0 99 | 1 0 99 12: 0 0 99 | 0 0 99 13: 0 0 99 | 1 0 99 14: 1 0 99 | 2 0 99 15: 0 0 99 | 0 0 99 16: 0 0 99 | 0 0 99 17: 0 0 99 | 1 0 99 18: 1 0 99 | 0 0 99 19: 0 0 99 | 0 0 99 20: 1 0 99 | 0 0 99 21: 1 0 99 | 0 0 99 22: 2 0 99 | 0 0 99 23: 1 0 99 | 0 0 99 24: 0 0 99 | 0 0 99 25: 1 0 99 | 0 0 99 26: 0 0 99 | 0 0 99 27: 0 0 99 | 0 0 99 28: 0 0 99 | 0 0 99 29: 0 0 99 | 1 0 99 30: 0 0 99 | 0 0 99 31: 31 0 100 | 1 0 100 read | write disk fragmented I/Os ios % cum % | ios % cum % 1: 11981 91 91 | 9819 25 25 2: 827 6 97 | 26854 70 96 3: 228 1 99 | 585 1 98 4: 12 0 99 | 548 1 99 5: 3 0 99 | 112 0 99 6: 4 0 99 | 20 0 99 7: 2 0 99 | 9 0 99 8: 7 0 99 | 5 0 99 9: 0 0 99 | 0 0 99 10: 2 0 99 | 0 0 99 11: 0 0 99 | 3 0 99 12: 1 0 99 | 1 0 99 13: 0 0 99 | 0 0 99 14: 0 0 99 | 0 0 99 15: 0 0 99 | 2 0 99 16: 0 0 99 | 0 0 99 17: 0 0 99 | 0 0 99 18: 0 0 99 | 1 0 99 19: 0 0 99 | 0 0 99 20: 0 0 99 | 0 0 99 21: 1 0 99 | 0 0 99 22: 0 0 99 | 0 0 99 23: 1 0 99 | 0 0 99 24: 1 0 99 | 0 0 99 25: 0 0 99 | 0 0 99 26: 0 0 99 | 0 0 99 27: 0 0 99 | 0 0 99 28: 0 0 99 | 0 0 99 29: 0 0 99 | 0 0 99 30: 0 0 99 | 0 0 99 31: 7 0 100 | 1 0 100 read | write disk I/Os in flight ios % cum % | ios % cum % 1: 9278 45 45 | 23900 34 34 2: 1718 8 53 | 17584 25 60 3: 420 2 55 | 5282 7 68 4: 302 1 57 | 4798 7 75 5: 236 1 58 | 2798 4 79 6: 214 1 59 | 2502 3 83 7: 187 0 60 | 1857 2 85 8: 152 0 60 | 1796 2 88 9: 148 0 61 | 1464 2 90 10: 144 0 62 | 1276 1 92 11: 133 0 62 | 980 1 93 12: 108 0 63 | 864 1 95 13: 111 0 64 | 710 1 96 14: 92 0 64 | 662 0 97 15: 75 0 64 | 600 0 98 16: 59 0 65 | 531 0 98 17: 47 0 65 | 395 0 99 18: 40 0 65 | 255 0 99 19: 36 0 65 | 85 0 99 20: 37 0 65 | 46 0 99 21: 33 0 66 | 2 0 100 22: 31 0 66 | 0 0 100 23: 30 0 66 | 0 0 100 24: 31 0 66 | 0 0 100 25: 34 0 66 | 0 0 100 26: 33 0 66 | 0 0 100 27: 28 0 66 | 0 0 100 28: 31 0 67 | 0 0 100 29: 28 0 67 | 0 0 100 30: 23 0 67 | 0 0 100 31: 6702 32 100 | 0 0 100 read | write I/O time (1/1000s) ios % cum % | ios % cum % 1: 8310 63 63 | 14817 39 39 2: 27 0 63 | 1098 2 41 4: 57 0 64 | 1432 3 45 8: 200 1 65 | 1825 4 50 16: 1018 7 73 | 3202 8 58 32: 1377 10 84 | 7035 18 77 64: 1093 8 92 | 6002 15 93 128: 419 3 95 | 1977 5 98 256: 219 1 97 | 397 1 99 512: 291 2 99 | 140 0 99 1K: 41 0 99 | 32 0 99 2K: 18 0 99 | 3 0 100 4K: 1 0 99 | 0 0 100 8K: 0 0 99 | 0 0 100 16K: 6 0 100 | 0 0 100 read | write disk I/O size ios % cum % | ios % cum % 4K: 12215 59 59 | 8163 11 11 8K: 69 0 59 | 211 0 12 16K: 35 0 59 | 131 0 12 32K: 39 0 60 | 162 0 12 64K: 138 0 60 | 661 0 13 128K: 100 0 61 | 414 0 14 256K: 157 0 62 | 255 0 14 512K: 260 1 63 | 374 0 15 1M: 5710 27 91 | 3967 5 20 2M: 1818 8 100 | 54049 79 100 read | write block maps msec maps % cum % | maps % cum % 1: 36699 99 99 | 33543 99 99 2: 0 0 99 | 2 0 99 4: 0 0 99 | 2 0 99 8: 6 0 99 | 6 0 99 16: 16 0 99 | 20 0 99 32: 16 0 99 | 40 0 99 64: 27 0 99 | 45 0 99 128: 7 0 99 | 43 0 99 256: 3 0 99 | 21 0 99 512: 3 0 99 | 51 0 99 1K: 11 0 99 | 2 0 99 2K: 3 0 100 | 2 0 99 4K: 0 0 100 | 1 0 100