catalyst263@dinatale:bash /tmp/scripts/run_mdtest.sh -r lustre_28_testing -f lustre === TEST: MDSCount 1 === -- started at 03/25/2016 16:07:42 -- mdtest-1.8.3 was launched with 144 total task(s) on 6 nodes Command line used: /opt/mdtest-1.8.3/bin/mdtest -u -i 4 -n 2048 -p 60 -v -d /p/lustre/dinatale/count1_index0 Path: /p/lustre/dinatale FS: 2.8 TiB Used FS: 0.0% Inodes: 22.5 Mi Used Inodes: 0.2% 144 tasks, 294912 files/directories Operation Duration Rate --------- -------- ---- * iteration 1 * Tree creation : 0.017 sec, 60.226 ops/sec delaying 60 seconds . . . Directory creation: 66.702 sec, 4421.330 ops/sec Directory stat : 5.715 sec, 51603.132 ops/sec Directory removal : 133.353 sec, 2211.514 ops/sec delaying 60 seconds . . . File creation : 54.413 sec, 5419.870 ops/sec File stat : 8.491 sec, 34730.759 ops/sec File removal : 100.288 sec, 2940.656 ops/sec Tree removal : 0.037 sec, 26.890 ops/sec * iteration 2 * Tree creation : 0.016 sec, 61.546 ops/sec delaying 60 seconds . . . Directory creation: 54.345 sec, 5426.620 ops/sec Directory stat : 5.601 sec, 52649.234 ops/sec Directory removal : 138.659 sec, 2126.894 ops/sec delaying 60 seconds . . . File creation : 53.873 sec, 5474.210 ops/sec File stat : 8.784 sec, 33575.093 ops/sec File removal : 108.617 sec, 2715.166 ops/sec Tree removal : 0.038 sec, 26.239 ops/sec * iteration 3 * Tree creation : 0.020 sec, 49.588 ops/sec delaying 60 seconds . . . Directory creation: 60.381 sec, 4884.166 ops/sec Directory stat : 5.710 sec, 51645.035 ops/sec Directory removal : 110.546 sec, 2667.776 ops/sec delaying 60 seconds . . . File creation : 70.885 sec, 4160.440 ops/sec File stat : 5.594 sec, 52720.944 ops/sec File removal : 102.743 sec, 2870.393 ops/sec Tree removal : 0.030 sec, 33.411 ops/sec * iteration 4 * Tree creation : 0.016 sec, 64.284 ops/sec delaying 60 seconds . . . Directory creation: 80.507 sec, 3663.177 ops/sec Directory stat : 5.623 sec, 52450.293 ops/sec Directory removal : 128.143 sec, 2301.421 ops/sec delaying 60 seconds . . . File creation : 55.649 sec, 5299.462 ops/sec File stat : 6.644 sec, 44385.694 ops/sec File removal : 114.896 sec, 2566.773 ops/sec Tree removal : 0.080 sec, 12.483 ops/sec SUMMARY: (of 4 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 5430.605 3665.080 4601.181 647.529 Directory stat : 53112.492 52023.002 52549.901 483.261 Directory removal : 2667.779 2126.894 2326.902 206.254 File creation : 5475.449 4161.539 5091.093 540.337 File stat : 53069.512 33780.824 41593.824 7854.316 File removal : 2940.678 2566.776 2773.256 144.458 Tree creation : 64.284 49.588 58.911 5.578 Tree removal : 33.411 12.483 24.756 7.620 -- finished at 03/25/2016 16:40:29 -- catalyst255: Linux 2.6.32-573.18.1.1chaos.ch5.4.x86_64 (catalyst255) 03/25/16 _x86_64_ (48 CPU) catalyst255: catalyst255: avg-cpu: %user %nice %system %iowait %steal %idle catalyst255: 0.00 0.00 0.04 0.00 0.00 99.96 catalyst255: catalyst255: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util catalyst255: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: catalyst255: avg-cpu: %user %nice %system %iowait %steal %idle catalyst255: 0.00 0.00 0.04 0.00 0.00 99.96 catalyst255: catalyst255: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util catalyst255: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: catalyst255: avg-cpu: %user %nice %system %iowait %steal %idle catalyst255: 0.00 0.00 0.00 0.00 0.00 100.00 catalyst255: catalyst255: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util catalyst255: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: === TEST: MDSCount 2 === -- started at 03/25/2016 16:41:39 -- mdtest-1.8.3 was launched with 144 total task(s) on 6 nodes Command line used: /opt/mdtest-1.8.3/bin/mdtest -u -i 4 -n 2048 -p 60 -v -d /p/lustre/dinatale/count2_index0 Path: /p/lustre/dinatale FS: 2.8 TiB Used FS: 0.0% Inodes: 22.5 Mi Used Inodes: 0.2% 144 tasks, 294912 files/directories Operation Duration Rate --------- -------- ---- * iteration 1 * Tree creation : 2.102 sec, 0.476 ops/sec delaying 60 seconds . . . Directory creation: 2717.550 sec, 108.521 ops/sec Directory stat : 14.686 sec, 20080.619 ops/sec Directory removal : 3421.642 sec, 86.190 ops/sec delaying 60 seconds . . . File creation : 42.427 sec, 6951.100 ops/sec File stat : 4.876 sec, 60477.662 ops/sec File removal : 57.757 sec, 5106.041 ops/sec Tree removal : 3.655 sec, 0.274 ops/sec * iteration 2 * Tree creation : 3.620 sec, 0.276 ops/sec delaying 60 seconds . . . Directory creation: 2101.941 sec, 140.305 ops/sec Directory stat : 14.622 sec, 20168.422 ops/sec Directory removal : 3232.880 sec, 91.223 ops/sec delaying 60 seconds . . . File creation : 38.507 sec, 7658.588 ops/sec File stat : 3.675 sec, 80253.557 ops/sec File removal : 51.417 sec, 5735.729 ops/sec Tree removal : 9.866 sec, 0.101 ops/sec * iteration 3 * Tree creation : 0.680 sec, 1.472 ops/sec delaying 60 seconds . . . Directory creation: 2290.300 sec, 128.766 ops/sec Directory stat : 14.696 sec, 20067.119 ops/sec Directory removal : 3317.432 sec, 88.898 ops/sec delaying 60 seconds . . . File creation : 212.553 sec, 1387.477 ops/sec File stat : 3.161 sec, 93305.707 ops/sec File removal : 53.685 sec, 5493.361 ops/sec Tree removal : 3.141 sec, 0.318 ops/sec * iteration 4 * Tree creation : 0.993 sec, 1.007 ops/sec delaying 60 seconds . . . Directory creation: 2262.722 sec, 130.335 ops/sec Directory stat : 14.634 sec, 20152.388 ops/sec Directory removal : 1807.437 sec, 163.166 ops/sec delaying 60 seconds . . . File creation : 38.656 sec, 7629.103 ops/sec File stat : 4.094 sec, 72038.464 ops/sec File removal : 95.072 sec, 3101.972 ops/sec Tree removal : 1.783 sec, 0.561 ops/sec SUMMARY: (of 4 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 140.306 108.522 126.983 11.541 Directory stat : 20169.405 20068.291 20118.116 44.012 Directory removal : 163.166 86.190 107.369 32.263 File creation : 7675.876 1387.942 5916.152 2629.838 File stat : 93324.604 60479.125 76524.687 11976.144 File removal : 5735.745 3101.974 4859.285 1039.144 Tree creation : 1.472 0.276 0.808 0.467 Tree removal : 0.561 0.101 0.314 0.164 -- finished at 03/25/2016 22:53:43 -- catalyst255: Linux 2.6.32-573.18.1.1chaos.ch5.4.x86_64 (catalyst255) 03/25/16 _x86_64_ (48 CPU) catalyst255: catalyst255: avg-cpu: %user %nice %system %iowait %steal %idle catalyst255: 0.00 0.00 0.04 0.00 0.00 99.96 catalyst255: catalyst255: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util catalyst255: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: catalyst255: avg-cpu: %user %nice %system %iowait %steal %idle catalyst255: 0.00 0.00 0.00 0.00 0.00 100.00 catalyst255: catalyst255: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util catalyst255: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: catalyst255: avg-cpu: %user %nice %system %iowait %steal %idle catalyst255: 0.00 0.00 0.00 0.00 0.00 100.00 catalyst255: catalyst255: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util catalyst255: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: === TEST: MDSCount 3 === -- started at 03/25/2016 22:54:53 -- mdtest-1.8.3 was launched with 144 total task(s) on 6 nodes Command line used: /opt/mdtest-1.8.3/bin/mdtest -u -i 4 -n 2048 -p 60 -v -d /p/lustre/dinatale/count3_index0 Path: /p/lustre/dinatale FS: 2.8 TiB Used FS: 0.0% Inodes: 22.5 Mi Used Inodes: 0.2% 144 tasks, 294912 files/directories Operation Duration Rate --------- -------- ---- * iteration 1 * Tree creation : 5.184 sec, 0.193 ops/sec delaying 60 seconds . . . Directory creation: 3064.275 sec, 96.242 ops/sec Directory stat : 15.917 sec, 18527.801 ops/sec Directory removal : 4844.058 sec, 60.881 ops/sec delaying 60 seconds . . . File creation : 177.710 sec, 1659.515 ops/sec File stat : 5.588 sec, 52773.059 ops/sec File removal : 168.051 sec, 1754.897 ops/sec Tree removal : 4.980 sec, 0.201 ops/sec * iteration 2 * Tree creation : 2.288 sec, 0.437 ops/sec delaying 60 seconds . . . Directory creation: 2787.200 sec, 105.809 ops/sec Directory stat : 15.872 sec, 18580.148 ops/sec Directory removal : 5091.972 sec, 57.917 ops/sec delaying 60 seconds . . . File creation : 36.253 sec, 8134.735 ops/sec File stat : 2.771 sec, 106422.512 ops/sec File removal : 40.752 sec, 7236.670 ops/sec Tree removal : 8.704 sec, 0.115 ops/sec * iteration 3 * Tree creation : 5.590 sec, 0.179 ops/sec delaying 60 seconds . . . Directory creation: 2862.225 sec, 103.036 ops/sec Directory stat : 16.736 sec, 17620.979 ops/sec Directory removal : 5031.486 sec, 58.613 ops/sec delaying 60 seconds . . . File creation : 179.593 sec, 1642.110 ops/sec File stat : 5.764 sec, 51160.670 ops/sec File removal : 170.305 sec, 1731.670 ops/sec Tree removal : 5.315 sec, 0.188 ops/sec * iteration 4 * Tree creation : 3.326 sec, 0.301 ops/sec delaying 60 seconds . . . Directory creation: 2812.340 sec, 104.864 ops/sec Directory stat : 15.939 sec, 18502.766 ops/sec Directory removal : 4811.927 sec, 61.288 ops/sec delaying 60 seconds . . . File creation : 197.057 sec, 1496.584 ops/sec File stat : 5.935 sec, 49686.277 ops/sec File removal : 156.915 sec, 1879.440 ops/sec Tree removal : 4.881 sec, 0.205 ops/sec SUMMARY: (of 4 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 105.811 96.243 102.489 3.741 Directory stat : 18580.815 17621.591 18308.662 397.660 Directory removal : 61.288 57.917 59.675 1.438 File creation : 8144.679 1497.023 3236.109 2834.671 File stat : 106422.896 49691.752 65014.456 23932.091 File removal : 7236.697 1731.673 3150.678 2359.733 Tree creation : 0.437 0.179 0.277 0.104 Tree removal : 0.205 0.115 0.177 0.036 -- finished at 03/26/2016 08:05:32 -- catalyst255: Linux 2.6.32-573.18.1.1chaos.ch5.4.x86_64 (catalyst255) 03/26/16 _x86_64_ (48 CPU) catalyst255: catalyst255: avg-cpu: %user %nice %system %iowait %steal %idle catalyst255: 0.00 0.00 0.04 0.00 0.00 99.96 catalyst255: catalyst255: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util catalyst255: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: catalyst255: avg-cpu: %user %nice %system %iowait %steal %idle catalyst255: 0.00 0.00 0.00 0.00 0.00 100.00 catalyst255: catalyst255: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util catalyst255: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: catalyst255: avg-cpu: %user %nice %system %iowait %steal %idle catalyst255: 0.00 0.00 0.00 0.00 0.00 100.00 catalyst255: catalyst255: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util catalyst255: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: === TEST: MDSCount 4 === -- started at 03/26/2016 08:06:43 -- mdtest-1.8.3 was launched with 144 total task(s) on 6 nodes Command line used: /opt/mdtest-1.8.3/bin/mdtest -u -i 4 -n 2048 -p 60 -v -d /p/lustre/dinatale/count4_index0 Path: /p/lustre/dinatale FS: 2.8 TiB Used FS: 0.0% Inodes: 22.5 Mi Used Inodes: 0.2% 144 tasks, 294912 files/directories Operation Duration Rate --------- -------- ---- * iteration 1 * Tree creation : 5.044 sec, 0.198 ops/sec delaying 60 seconds . . . Directory creation: 2861.194 sec, 103.073 ops/sec Directory stat : 17.517 sec, 16835.936 ops/sec Directory removal : 4733.765 sec, 62.300 ops/sec delaying 60 seconds . . . File creation : 145.114 sec, 2032.276 ops/sec File stat : 4.526 sec, 65166.564 ops/sec File removal : 145.897 sec, 2021.376 ops/sec Tree removal : 8.499 sec, 0.118 ops/sec * iteration 2 * Tree creation : 6.104 sec, 0.164 ops/sec delaying 60 seconds . . . Directory creation: 2691.911 sec, 109.555 ops/sec Directory stat : 17.495 sec, 16856.491 ops/sec Directory removal : 4701.870 sec, 62.722 ops/sec delaying 60 seconds . . . File creation : 126.763 sec, 2326.491 ops/sec File stat : 4.457 sec, 66167.933 ops/sec File removal : 81.308 sec, 3627.090 ops/sec Tree removal : 7.756 sec, 0.129 ops/sec * iteration 3 * Tree creation : 4.358 sec, 0.229 ops/sec delaying 60 seconds . . . Directory creation: 2715.645 sec, 108.597 ops/sec Directory stat : 17.495 sec, 16857.322 ops/sec Directory removal : 3617.251 sec, 81.529 ops/sec delaying 60 seconds . . . File creation : 20.782 sec, 14190.785 ops/sec File stat : 4.622 sec, 63801.437 ops/sec File removal : 31.738 sec, 9292.188 ops/sec Tree removal : 6.946 sec, 0.144 ops/sec * iteration 4 * Tree creation : 5.969 sec, 0.168 ops/sec delaying 60 seconds . . . Directory creation: 2765.994 sec, 106.621 ops/sec Directory stat : 17.447 sec, 16903.588 ops/sec Directory removal : 4205.088 sec, 70.132 ops/sec delaying 60 seconds . . . File creation : 20.128 sec, 14651.758 ops/sec File stat : 2.796 sec, 105474.999 ops/sec File removal : 35.653 sec, 8271.672 ops/sec Tree removal : 6.397 sec, 0.156 ops/sec SUMMARY: (of 4 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation: 109.556 103.074 106.963 2.482 Directory stat : 16904.242 16836.548 16863.872 24.812 Directory removal : 81.529 62.300 69.171 7.785 File creation : 14711.490 2033.056 8329.171 6151.561 File stat : 105478.809 63802.763 75158.724 17525.599 File removal : 9292.211 2021.377 5803.090 3053.858 Tree creation : 0.229 0.164 0.190 0.027 Tree removal : 0.156 0.118 0.137 0.015 -- finished at 03/26/2016 16:18:42 -- catalyst255: Linux 2.6.32-573.18.1.1chaos.ch5.4.x86_64 (catalyst255) 03/26/16 _x86_64_ (48 CPU) catalyst255: catalyst255: avg-cpu: %user %nice %system %iowait %steal %idle catalyst255: 0.00 0.00 0.00 0.00 0.00 100.00 catalyst255: catalyst255: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util catalyst255: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: catalyst255: avg-cpu: %user %nice %system %iowait %steal %idle catalyst255: 0.00 0.00 0.00 0.00 0.00 100.00 catalyst255: catalyst255: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util catalyst255: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: catalyst255: avg-cpu: %user %nice %system %iowait %steal %idle catalyst255: 0.00 0.00 0.04 0.00 0.00 99.96 catalyst255: catalyst255: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util catalyst255: sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 catalyst255: