Overview The Lustre lock ahead feature is meant to improve shared file performance of MPI‐IO collective buffering. This method aggregates IO to a specific set of nodes, each of which handles parts of a file. Writes are strided and non‐overlapping. In this pattern, the lock count per OST is equal to the file size in MB divided by the numbers of OSTs, assuming 1MB striping. Lock ahead requests are non‐blocking. If client B holds a lock on part of the file, and client A requests a lock ahead on the same part of the file, client A’s request will fail. Lock ahead requests are asynchronous. Write or read lock requests are blocking lock requests, with one exception which is group lock. Group locks are using the library to clearing out any other locks before requesting group locks. They’re exclusive, blocking, and absolute. Test cases were developed based on a test review of the feature and documented in: https://connect.us.cray.com/confluence/display/LUSTRE/Lock+Ahead Usage Cray’s message passing interface (MPI) is derived from Argonne National Laboratory MPICH. Cray‐MPICH adds support for a MPI I/O hint, cray_cb_write_lock, that specifies the file locking mode for accessing Lustre files and another hint, cray_cb_nodes_multiplier, to specify the number of collective buffering aggregators per OST for Lustre files. With Cray’s message passing library, lock ahead is enabled by setting cray_cb_write_lock_mode=2 in the MPICH_MPIIO_HINTS environment variable. cray_cb_write_lock_mode=0 is standard Lustre file locking cray_cb_write_lock_mode=1 is a Lustre group lock cray_cb_write_lock_mode=2 is lock ahead Test Environments saturn‐p2 is a partition of an XC30 cabinet with 46 compute nodes. It is the main Lustre testing and development system for the SDM I/O Filesystems group. saturn‐p2 runs a Rhine Redwood development branch. The system has three storage systems available, a DAL file system, external Lustre file system (CLFS) called husk and a RAM‐backed Lustre fs. For more information: https://connect.us.cray.com/confluence/display/LUSTRE/saturn‐p2 pollux‐p6 is a partition of an XC cabinet with 116 compute nodes and is running 5.2.UP04. The system had a Sonexion 1600 (snx11011) with SU 16 installed for the storage system. Manual Test Cases The manual test cases were run on saturn-p2 using a DAL and/or a CLFS Lustre file system. If run on another system the test scripts should be modified for sizing of the IOR runs and path to the Lustre fs. MPI-IO and HDF5 with IOR. Several IOR tests using MPI-IO and HDF5 were run to demonstrate collective writes on a shared file with lock ahead, group lock and standard Lustre file locking enabled. For each test a separate aprun was used to read and verify the IOR collective write data. Sizing of the IOR runs on this test system, saturn-p2, can be seen in the results below. One MPI rank was placed per compute node. The Cray MPI-IO hint, cray_cb_nodes_multiplier, was set to vary the number of aggregators per OST. Tests Executed: 75 Tests Passed: 75 Tests Failed: 0 Test scripts: lock_mode1.sh, lock_mode2.sh, posix.sh Stat and Read Disruption. With lock ahead enabled, several IOR tests using MPI-IO and HDF5 were run while another program was running stat or reading the file as the shared file is written. The purpose of this test is to simulate a user watching/reading files while an application is running. Tests Executed: 12 Tests Passed: 12 Tests Failed: 0 Test scripts: stat_loop.sh, read_loop.sh Mix lock modes. A mix of IOR tests using MPI-IO and HDF5, defined above, were run from different clients to the same file system and executing with a mix of the three Lustre locking methods. Tests Executed: 33 Tests Passed: 33 Tests Failed: 0 Test scripts: mix_locks.sh Read after write. Following a collective write this test sends a message, with the write offset, to a higher MPI rank so another node can read and verify the data just written. Tests Executed: 15 Tests Passed: 15 Tests Failed: 0 Test scripts: read_after_write.sh Automated Test Cases To understand and compare performance between the various file locking modes and for automated performance regression testing in the future, a number of IOR MPPIO, HDF5 collective and POSIX i/o using the various Lustre file locking modes were added to RTS. Test tag names are limited to 32 characters in RTS so the following naming conventions were used: sl = Standard Lustre Locking gl = Group Lock la = Lock Ahead fpp = POSIX i/o to a file per process shr = POSIX i/o to a single shared file dir = Uses O_DIRECT for POSIX, bypassing i/o buffers buf = Standard POSIX buffered i/o APIs for I/O = mpiio, hdf5 or posix cb = collective i/o For example, CL_IOR_all_hdf5_cb_la_w_1m, is an IOR collective i/o write test with the HDF5 api and lock ahead enabled. The 1m in the tag name is the IOR transfer size in bytes. Each write test is followed by separate test to read/verify the data, e.g. CL_IOR_all_hdf5_cb_la_r_1m. All these IOR tests are sized to run with 64 clients (2 per node), an 8m block size (contingous bytes to write per task) and an aggregate filesize of 512 GiB. CL_IOR_all_mpiio_cb_la_w_1m CL_IOR_all_mpiio_cb_la_r_1m CL_IOR_all_mpiio_cb_gl_w_1m CL_IOR_all_mpiio_cb_gl_r_1m CL_IOR_all_hdf5_cb_sl_w_1m CL_IOR_all_hdf5_cb_sl_r_1m CL_IOR_all_hdf5_cb_la_w_1m CL_IOR_all_hdf5_cb_la_r_1m CL_IOR_all_mpiio_cb_gl_w_32m CL_IOR_all_mpiio_cb_gl_r_32m CL_IOR_all_mpiio_cb_la_w_32m CL_IOR_all_mpiio_cb_la_r_32m CL_IOR_all_hdf5_cb_sl_w_32m CL_IOR_all_hdf5_cb_sl_r_32m CL_IOR_all_hdf5_cb_la_w_32m CL_IOR_all_hdf5_cb_la_r_32m CL_IOR_all_mpiio_cb_gl_w_64m CL_IOR_all_mpiio_cb_gl_r_64m CL_IOR_all_mpiio_cb_la_w_64m CL_IOR_all_mpiio_cb_la_r_64m CL_IOR_all_hdf5_cb_sl_w_64m CL_IOR_all_hdf5_cb_sl_r_64m CL_IOR_all_hdf5_cb_la_w_64m CL_IOR_all_hdf5_cb_la_r_64m CL_IOR_all_posix_buf_fpp_w_1m CL_IOR_all_posix_buf_fpp_r_1m CL_IOR_all_posix_buf_fpp_w_32m CL_IOR_all_posix_buf_fpp_r_32m CL_IOR_all_posix_buf_fpp_w_64m CL_IOR_all_posix_buf_fpp_r_64m CL_IOR_all_posix_dir_fpp_w_1m CL_IOR_all_posix_dir_fpp_r_1m CL_IOR_all_posix_dir_fpp_w_32m CL_IOR_all_posix_dir_fpp_r_32m CL_IOR_all_posix_dir_fpp_w_64m CL_IOR_all_posix_dir_fpp_r_64m CL_IOR_all_posix_buf_shr_w_1m CL_IOR_all_posix_buf_shr_r_1m CL_IOR_all_posix_buf_shr_w_32m CL_IOR_all_posix_buf_shr_r_32m CL_IOR_all_posix_buf_shr_w_64m CL_IOR_all_posix_buf_shr_r_64m The automated tests were run on three test environments: saturn‐ p2 using CLFS, saturn‐p2 using a RAM‐backed Lustre file system and pollux‐p6 using snx11011 with SU 16. In addition to the lock ahead tests, the RTS regression write_verify and relrun io test suites were run on the pollux‐p6 test environment. There was one (1) downed compute node while running RTS suite (relrun io). Tests Executed: 118 Tests Passed: 106 Tests Failed: 12 Six (6) failures were the POSIX, buffered, file per process tests running on the RAM‐backed fs; out of disk space. On pollux‐p6, two (2) read tests failed due to timeouts, slow reads. These tests pass, written data is good, if given more time to complete. Four (4) HDF5 tests failed on pollux‐p6 because the Cray PE environment was not installed correctly (the Cray mpich shared library was not available). Critical Bugs: None Open Bugs: None Fixed Bugs: Bug 838695 ‐ ldlm_extent_shift_kms does linear search of granted extent lock list Test results from saturn‐DAL‐manual‐Feb25 Test Environment dal-MDT0000_UUID 39314720 789736 35834012 2% /lus/dal[MDT:0] dal-MDT0001_UUID 39314720 472840 36150288 1% /lus/dal[MDT:1] dal-OST0000_UUID 459900672 490824 435475744 0% /lus/dal[OST:0] dal-OST0001_UUID 459900672 490828 435475740 0% /lus/dal[OST:1] dal-OST0002_UUID 459900672 490780 435475788 0% /lus/dal[OST:2] dal-OST0003_UUID 459900672 490796 435475772 0% /lus/dal[OST:3] lustre: 2.7.1 kernel: patchless_client build: 2.7.0-trunk-1.0000.d21f5f5.0.0-jmiller-lustre-filesystem.git@d21f5f5-2016-01-29-14:58 build: 2.7.0-trunk-1.0000.d21f5f5.0.0-jmiller-lustre-filesystem.git@d21f5f5-2016-02-04-20:10 build: 2.7.0-trunk-1.0000.70d26c7.0.0-jmiller-lustre-filesystem.git@70d26c7-2016-02-05-20:43 build: 2.7.0-trunk-1.0000.f527c24.0.0-jmiller-lustre-filesystem.git@f527c24-2016-02-12-15:41 build: 2.7.0-trunk-1.0000.f5e96c5.0.0-jmiller-lustre-filesystem.git@f5e96c5-2016-02-18-17:29 cray-mpich/7.2.4 cray-hdf5-parallel/1.8.14 IOR test cases -------------- Clients = 8 (1 per node) Blocksize = 8 GiB Aggregate filesize = 64 GiB Xfersize = 64 MiB Max Write: 76.17 MiB/sec (79.88 MB/sec) - mppio, collective io, lock_mode=1, nodes_multiplier=2 Max Write: 74.34 MiB/sec (77.95 MB/sec) - mppio, collective io, lock_mode=1, nodes_multiplier=4 Max Write: 88.58 MiB/sec (92.88 MB/sec) - mppio, collective io, lock_mode=2, nodes_multiplier=2 Max Write: 81.53 MiB/sec (85.49 MB/sec) - mppio, collective io, lock_mode=2, nodes_multiplier=4 Max Write: 83.55 MiB/sec (87.61 MB/sec) - mppio, collective io, lock_mode=2, nodes_multiplier=8 Max Write: 80.14 MiB/sec (84.03 MB/sec) - mppio, collective io, lock_mode=2, nodes_multiplier=2, read loop Max Write: 96.42 MiB/sec (101.10 MB/sec) - mppio, collective io, lock_mode=2, nodes_multiplier=8, read loop Max Write: 89.19 MiB/sec (93.52 MB/sec) - hdf5, collective io, lock_mode=0, nodes_multiplier=1 Max Write: 90.70 MiB/sec (95.11 MB/sec) - hdf5, collective io, lock_mode=2, nodes_multiplier=2 Max Write: 84.04 MiB/sec (88.12 MB/sec) - hdf5, collective io, lock_mode=2, nodes_multiplier=4 Max Write: 81.03 MiB/sec (84.97 MB/sec) - hdf5, collective io, lock_mode=2, nodes_multiplier=8 Max Write: 92.02 MiB/sec (96.49 MB/sec) - hdf5, collective io, lock_mode=2, nodes_multiplier=2, stat loop Max Write: 78.38 MiB/sec (92.19 MB/sec) - hdf5, collective io, lock_mode=2, nodes_multiplier=4, stat loop Max Write: 74.14 MiB/sec (77.75 MB/sec) - posix, buffered io Max Write: 71.16 MiB/sec (74.62 MB/sec) - posix, direct io Xfersize = 32 MiB Max Write: 80.54 MiB/sec (84.45 MB/sec) - mpiio, collective io, lock_mode=1, nodes_multiplier=2 Max Write: 71.95 MiB/sec (75.45 MB/sec) - mpiio, collective io, lock_mode=1, nodes_multiplier=4 Max Write: 95.89 MiB/sec (100.55 MB/sec) - mpiio, collective io, lock_mode=2, nodes_multiplier=2 Max Write: 90.53 MiB/sec (94.93 MB/sec) - mpiio, collective io, lock_mode=2, nodes_multiplier=4 Max Write: 92.02 MiB/sec (96.50 MB/sec) - mpiio, collective io, lock_mode=2, nodes_multiplier=8 Max Write: 91.37 MiB/sec (95.81 MB/sec) - mpiio, collective io, lock_mode=2, nodes_multiplier=2, read loop Max Write: 81.72 MiB/sec (85.69 MB/sec) - mpiio, collective io, lock_mode=2, nodes_multiplier=8, read loop Max Write: 91.83 MiB/sec (96.29 MB/sec) - hdf5, collective io, lock_mode=0, nodes_multiplier=1 Max Write: 100.69 MiB/sec (105.58 MB/sec) - hdf5, collective io, lock_mode=2, nodes_multiplier=2 Max Write: 86.20 MiB/sec (90.39 MB/sec) - hdf5, collective io, lock_mode=2, nodes_multiplier=4 Max Write: 89.10 MiB/sec (93.43 MB/sec) - hdf5, collective io, lock_mode=2, nodes_multiplier=8 Max Write: 89.77 MiB/sec (94.13 MB/sec) - hdf5, collective io, lock_mode=2, nodes_multiplier=2, stat loop Max Write: 85.76 MiB/sec (89.93 MB/sec) - hdf5, collective io, lock_mode=2, nodes_multiplier=4, stat loop Max Write: 71.54 MiB/sec (75.02 MB/sec) - posix, buffered io Max Write: 72.40 MiB/sec (75.92 MB/sec) - posix, direct io Xfersize = 1 MiB Max Write: 68.46 MiB/sec (71.79 MB/sec) - mpiio, collective io, lock_mode=1, nodes_multiplier=2 Max Write: 68.99 MiB/sec (72.34 MB/sec) - mppio, collective io, lock_mode=1, nodes_multiplier=4 Max Write: 76.58 MiB/sec (80.30 MB/sec) - mpiio, collective io, lock_mode=2, nodes_multiplier=2 Max Write: 70.66 MiB/sec (74.10 MB/sec) - mpiio, collective io, lock_mode=2, nodes_multiplier=4 Max Write: 73.13 MiB/sec (76.68 MB/sec) - mpiio, collective io, lock_mode=2, nodes_multiplier=8 Max Write: 95.90 MiB/sec (100.56 MB/sec) - mpiio, collective io, lock_mode=2, nodes_multiplier=2, read loop Max Write: 92.29 MiB/sec (96.78 MB/sec) - mpiio, collective io, lock_mode=2, nodes_multiplier=8, read loop Max Write: 71.55 MiB/sec (75.02 MB/sec) - hdf5, collective io, lock_mode=0, nodes_multiplier=1 Max Write: 76.49 MiB/sec (80.21 MB/sec) - hdf5, collective io, lock_mode=2, nodes_multiplier=2 Max Write: 71.14 MiB/sec (74.60 MB/sec) - hdf5, collective io, lock_mode=2, nodes_multiplier=4 Max Write: 70.92 MiB/sec (74.37 MB/sec) - hdf5, collective io, lock_mode=2, nodes_multiplier=8 Max Write: 90.42 MiB/sec (94.81 MB/sec) - hdf5, collective io, lock_mode=2, nodes_multiplier=2, stat loop Max Write: 92.21 MiB/sec (96.69 MB/sec) - hdf5, collective io, lock_mode=2, nodes_multiplier=4, stat loop Max Write: 72.71 MiB/sec (76.25 MB/sec) - posix, buffered io Max Write: 73.67 MiB/sec (77.24 MB/sec) - posix, direct io Read after write ---------------- Clients = 8 (1 per node) Aggregate filesize = 64 GiB Xfersize = 64 MiB Write: 124.724315 Mbytes/sec - mpiio, collective io, lock_mode=0, nodes_multiplier=1 Write: 90.160743 Mbytes/sec - mpiio, collective io, lock_mode=2, nodes_multiplier=2 Write: 85.452484 Mbytes/sec - mpiio, collective io, lock_mode=2, nodes_multiplier=4 Xfersize = 32 MiB Write: 171.911069 Mbytes/sec - mpiio, collective io, lock_mode=0, nodes_multiplier=1 Write: 89.495822 Mbytes/sec - mpiio, collective io, lock_mode=2, nodes_multiplier=2 Write: 83.572242 Mbytes/sec - mpiio, collective io, lock_mode=2, nodes_multiplier=4 Xfersize = 1 MiB Write: 2560.072459 Mbytes/sec - mpiio, collective io, lock_mode=0, nodes_multiplier=1 Write: 279.242361 Mbytes/sec - mpiio, collective io, lock_mode=2, nodes_multiplier=2 Write: 132.264581 Mbytes/sec - mpiio, collective io, lock_mode=2, nodes_multiplier=4 Candidates for RTS automation ----------------------------- For automated performance regression and comparison testing, add mppio, hdf5 collective, posix i/o and stat disruption tests cases with the Lustre file locking modes. mppio, collective io, lock_mode=1 mppio, collective io, lock_mode=2 hdf5, collective io, lock_mode=0 hdf5, collective io, lock_mode=2 posix Add the write visibility test to one of the RTS test suites. Test run example ---------------- For more results see: /cray/css/users/dmoen/lockahead/results + RANKS=8 + RANKS_PER_NODE=1 + FS=/lus/dal/dmoen + MPIIO=./IOR_mpiio.cfg + HDF5=./IOR_hdf5.cfg + BIN=../bin + RUN_IOR=../bin/run_IOR.pl + JOBID=8849 + TESTDIR=/lus/dal/dmoen/testdir.8849 + mkdir /lus/dal/dmoen/testdir.8849 + lfs setstripe -S 1m -c -1 /lus/dal/dmoen/testdir.8849 + module list ++ /opt/cray/pe/modules/3.2.10.4/bin/modulecmd bash list Currently Loaded Modulefiles: 1) modules/3.2.10.4 2) alps/6.1.1-19.2 3) nodestat/2.2-1.0000.58434.41.17.ari 4) sdb/2.2.0-4.21 5) udreg/2.3.2-3.41 6) ugni/6.0.8-2.1 7) gni-headers/5.0.6-2.1 8) dmapp/7.1.0-19.1 9) xpmem/0.1-2.0000.62384.86.34.ari 10) llm/20.2.0-2.7 11) nodehealth/5.2.0-8.10 12) sysadm/2.2.0-4.17 13) Base-opts/2.1.1-2.4 14) cce/8.4.4 15) craype-network-aries 16) craype/2.5.2.3 17) cray-libsci/13.3.0 18) pmi/5.0.10-1.0000.11050.0.0.ari 19) rca/1.0.0-2.0000.63804.154.19.ari 20) atp/1.8.4.6 21) PrgEnv-cray/6.0.1 22) cray-mpich/7.3.2.3 23) craype-sandybridge 24) craype-hugepages512M 25) cray-hdf5-parallel/1.8.14 + eval + cat /proc/fs/lustre/version lustre: 2.7.1 kernel: patchless_client build: 2.7.0-trunk-1.0000.f527c24.0.0-jmiller-lustre-filesystem.git@f527c24-2016-02-12-15:41 + lfs df UUID 1K-blocks Used Available Use% Mounted on dal-MDT0000_UUID 39314720 867992 35755908 2% /lus/dal[MDT:0] dal-MDT0001_UUID 39314720 472840 36150288 1% /lus/dal[MDT:1] dal-OST0000_UUID 459900672 490892 435474652 0% /lus/dal[OST:0] dal-OST0001_UUID 459900672 490884 435475684 0% /lus/dal[OST:1] dal-OST0002_UUID 459900672 490824 435474720 0% /lus/dal[OST:2] dal-OST0003_UUID 459900672 490860 435475708 0% /lus/dal[OST:3] filesystem summary: 1839602688 1963460 1741900764 0% /lus/dal + export MPICH_VERSION_DISPLAY=1 + MPICH_VERSION_DISPLAY=1 + export MPICH_MPIIO_HINTS_DISPLAY=1 + MPICH_MPIIO_HINTS_DISPLAY=1 + export MPICH_MPIIO_STATS=1 + MPICH_MPIIO_STATS=1 + export MPICH_MPIIO_ABORT_ON_RW_ERROR=enable + MPICH_MPIIO_ABORT_ON_RW_ERROR=enable + export MPICH_CPUMASK_DISPLAY=1 + MPICH_CPUMASK_DISPLAY=1 + export IOR_HINT__MPI__romio_no_indep_rw=true + IOR_HINT__MPI__romio_no_indep_rw=true + export IOR_HINT__MPI__romio_cb_write=enable + IOR_HINT__MPI__romio_cb_write=enable + export IOR_HINT__MPI__cray_cb_write_lock_mode=2 + IOR_HINT__MPI__cray_cb_write_lock_mode=2 + export IOR_HINT__MPI__cray_cb_nodes_multiplier=8 + IOR_HINT__MPI__cray_cb_nodes_multiplier=8 + ../bin/run_IOR.pl -n 8 -N 1 /lus/dal/dmoen/testdir.8849 ./IOR_mpiio.cfg ALPS Node Information total 0 /lus/dal/dmoen/testdir.8849 stripe_count: -1 stripe_size: 1048576 stripe_offset: -1 Start IOR test mpiio_collective_32m aprun -n 8 -N 1 -j 1 -T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -w -a MPIIO -H -C -k -c -t 32m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/mpiio_collective_32m [PE_6]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.2.4 (ANL base 3.1.2) BUILD INFO : Built Mon Jul 27 12:57:53 2015 (svn rev 10827) MT-G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR-2.10.3: MPI Coordinated Test of Parallel I/O Run began: Mon Feb 15 06:47:44 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -w -a MPIIO -H -C -k -c -t 32m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/mpiio_collective_32m Machine: Linux nid00032 Summary: api = MPIIO (version=3, subversion=0) test filename = /lus/dal/dmoen/testdir.8849/mpiio_collective_32m access = single-shared-file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 8 (1 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 64 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 2 cray_cb_nodes_multiplier = 8 romio_cb_write = enable } PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/mpiio_collective_32m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +--------------------------------------------------------+ | MPIIO write access patterns for /lus/dal/dmoen/testdir.8849/mpiio_collective_32m | independent writes = 0 | collective writes = 2048 | independent writers = 0 | aggregators = 8 | stripe count = 4 | stripe size = 1048576 | system writes = 65536 | stripe sized writes = 65536 | total bytes for writes = 68719476736 = 65536 MiB = 64 GiB | ave system write size = 1048576 | read-modify-write count = 0 | read-modify-write bytes = 0 | number of write gaps = 16376 | ave write gap size = 14979151478 | See "Optimizing MPI I/O on Cray XE Systems" S-0013-20 for explanations. +--------------------------------------------------------+ PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/mpiio_collective_32m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 4 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) --------- --------- --------- ---------- ------- --------- --------- ---------- ------- -------- write 92.02 92.02 92.02 0.00 2.88 2.88 2.88 0.00 712.15567 EXCEL Max Write: 92.02 MiB/sec (96.50 MB/sec) Run finished: Mon Feb 15 06:59:36 2016 Application 177418 resources: utime ~2745s, stime ~55s, Rss ~4540, inblocks ~0, outblocks ~134217728 aprun -n 8 -N 1 -j 1 -T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -r -a MPIIO -H -C -k -c -t 32m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/mpiio_collective_32m [PE_1]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.2.4 (ANL base 3.1.2) BUILD INFO : Built Mon Jul 27 12:57:53 2015 (svn rev 10827) MT-G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR-2.10.3: MPI Coordinated Test of Parallel I/O Run began: Mon Feb 15 06:59:43 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -r -a MPIIO -H -C -k -c -t 32m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/mpiio_collective_32m Machine: Linux nid00064 Summary: api = MPIIO (version=3, subversion=0) test filename = /lus/dal/dmoen/testdir.8849/mpiio_collective_32m access = single-shared-file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 8 (1 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 64 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 2 cray_cb_nodes_multiplier = 8 romio_cb_write = enable } PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/mpiio_collective_32m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +--------------------------------------------------------+ | MPIIO read access patterns for /lus/dal/dmoen/testdir.8849/mpiio_collective_32m | independent reads = 0 | collective reads = 2048 | independent readers = 0 | aggregators = 8 | stripe count = 4 | stripe size = 1048576 | system reads = 65536 | stripe sized reads = 65536 | total bytes for reads = 68719476736 = 65536 MiB = 64 GiB | ave system read size = 1048576 | number of read gaps = 16376 | ave read gap size = 4179960 | See "Optimizing MPI I/O on Cray XE Systems" S-0013-20 for explanations. +--------------------------------------------------------+ PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/mpiio_collective_32m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 4 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) --------- --------- --------- ---------- ------- --------- --------- ---------- ------- -------- read 73.19 73.19 73.19 0.00 2.29 2.29 2.29 0.00 895.43896 EXCEL Max Read: 73.19 MiB/sec (76.74 MB/sec) Run finished: Mon Feb 15 07:14:38 2016 Application 177419 resources: utime ~1214s, stime ~25s, Rss ~4460, inblocks ~0, outblocks ~0 Done IOR mpiio_collective_32m 122 0x7a c0-0c1s14n2 compute down interactive Down compute nodes mpiio_collective_32m, down= 1 total 65G -rw-r--r-- 1 dmoen criemp 64G Feb 15 06:59 mpiio_collective_32m /lus/dal/dmoen/testdir.8849 stripe_count: -1 stripe_size: 1048576 stripe_offset: -1 /lus/dal/dmoen/testdir.8849/mpiio_collective_32m lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 10434864 0x9f3930 0 0 10610544 0xa1e770 0 2 10565824 0xa138c0 0 1 10306270 0x9d42de 0 Start IOR test mpiio_collective_64m aprun -n 8 -N 1 -j 1 -T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -w -a MPIIO -H -C -k -c -t 64m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/mpiio_collective_64m [PE_1]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.2.4 (ANL base 3.1.2) BUILD INFO : Built Mon Jul 27 12:57:53 2015 (svn rev 10827) MT-G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR-2.10.3: MPI Coordinated Test of Parallel I/O Run began: Mon Feb 15 07:14:49 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -w -a MPIIO -H -C -k -c -t 64m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/mpiio_collective_64m Machine: Linux nid00032 Summary: api = MPIIO (version=3, subversion=0) test filename = /lus/dal/dmoen/testdir.8849/mpiio_collective_64m access = single-shared-file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 8 (1 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 64 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 2 cray_cb_nodes_multiplier = 8 romio_cb_write = enable } PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/mpiio_collective_64m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +--------------------------------------------------------+ | MPIIO write access patterns for /lus/dal/dmoen/testdir.8849/mpiio_collective_64m | independent writes = 0 | collective writes = 1024 | independent writers = 0 | aggregators = 8 | stripe count = 4 | stripe size = 1048576 | system writes = 65536 | stripe sized writes = 65536 | total bytes for writes = 68719476736 = 65536 MiB = 64 GiB | ave system write size = 1048576 | read-modify-write count = 0 | read-modify-write bytes = 0 | number of write gaps = 8184 | ave write gap size = 14927702186 | See "Optimizing MPI I/O on Cray XE Systems" S-0013-20 for explanations. +--------------------------------------------------------+ PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/mpiio_collective_64m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 4 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) --------- --------- --------- ---------- ------- --------- --------- ---------- ------- -------- write 83.55 83.55 83.55 0.00 1.31 1.31 1.31 0.00 784.41915 EXCEL Max Write: 83.55 MiB/sec (87.61 MB/sec) Run finished: Mon Feb 15 07:27:54 2016 Application 177420 resources: utime ~2886s, stime ~58s, Rss ~4540, inblocks ~0, outblocks ~134217728 aprun -n 8 -N 1 -j 1 -T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -r -a MPIIO -H -C -k -c -t 64m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/mpiio_collective_64m [PE_4]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.2.4 (ANL base 3.1.2) BUILD INFO : Built Mon Jul 27 12:57:53 2015 (svn rev 10827) MT-G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR-2.10.3: MPI Coordinated Test of Parallel I/O Run began: Mon Feb 15 07:28:00 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -r -a MPIIO -H -C -k -c -t 64m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/mpiio_collective_64m Machine: Linux nid00064 Summary: api = MPIIO (version=3, subversion=0) test filename = /lus/dal/dmoen/testdir.8849/mpiio_collective_64m access = single-shared-file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 8 (1 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 64 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 2 cray_cb_nodes_multiplier = 8 romio_cb_write = enable } PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/mpiio_collective_64m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +--------------------------------------------------------+ | MPIIO read access patterns for /lus/dal/dmoen/testdir.8849/mpiio_collective_64m | independent reads = 0 | collective reads = 1024 | independent readers = 0 | aggregators = 8 | stripe count = 4 | stripe size = 1048576 | system reads = 65536 | stripe sized reads = 65536 | total bytes for reads = 68719476736 = 65536 MiB = 64 GiB | ave system read size = 1048576 | number of read gaps = 8184 | ave read gap size = 8331207 | See "Optimizing MPI I/O on Cray XE Systems" S-0013-20 for explanations. +--------------------------------------------------------+ PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/mpiio_collective_64m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 4 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) --------- --------- --------- ---------- ------- --------- --------- ---------- ------- -------- read 71.94 71.94 71.94 0.00 1.12 1.12 1.12 0.00 910.96511 EXCEL Max Read: 71.94 MiB/sec (75.44 MB/sec) Run finished: Mon Feb 15 07:43:10 2016 Application 177421 resources: utime ~1372s, stime ~28s, Rss ~4460, inblocks ~0, outblocks ~0 Done IOR mpiio_collective_64m 122 0x7a c0-0c1s14n2 compute down interactive Down compute nodes mpiio_collective_64m, down= 1 total 129G -rw-r--r-- 1 dmoen criemp 64G Feb 15 06:59 mpiio_collective_32m -rw-r--r-- 1 dmoen criemp 64G Feb 15 07:27 mpiio_collective_64m /lus/dal/dmoen/testdir.8849 stripe_count: -1 stripe_size: 1048576 stripe_offset: -1 /lus/dal/dmoen/testdir.8849/mpiio_collective_32m lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 10434864 0x9f3930 0 0 10610544 0xa1e770 0 2 10565824 0xa138c0 0 1 10306270 0x9d42de 0 /lus/dal/dmoen/testdir.8849/mpiio_collective_64m lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 10610545 0xa1e771 0 2 10565825 0xa138c1 0 1 10306271 0x9d42df 0 3 10434865 0x9f3931 0 Start IOR test mpiio_collective_1m aprun -n 8 -N 1 -j 1 -T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -w -a MPIIO -H -C -k -c -t 1m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/mpiio_collective_1m [PE_1]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.2.4 (ANL base 3.1.2) BUILD INFO : Built Mon Jul 27 12:57:53 2015 (svn rev 10827) MT-G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR-2.10.3: MPI Coordinated Test of Parallel I/O Run began: Mon Feb 15 07:43:19 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -w -a MPIIO -H -C -k -c -t 1m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/mpiio_collective_1m Machine: Linux nid00032 Summary: api = MPIIO (version=3, subversion=0) test filename = /lus/dal/dmoen/testdir.8849/mpiio_collective_1m access = single-shared-file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 8 (1 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 64 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 2 cray_cb_nodes_multiplier = 8 romio_cb_write = enable } PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/mpiio_collective_1m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +--------------------------------------------------------+ | MPIIO write access patterns for /lus/dal/dmoen/testdir.8849/mpiio_collective_1m | independent writes = 0 | collective writes = 65536 | independent writers = 0 | aggregators = 8 | stripe count = 4 | stripe size = 1048576 | system writes = 65536 | stripe sized writes = 65536 | total bytes for writes = 68719476736 = 65536 MiB = 64 GiB | ave system write size = 1048576 | read-modify-write count = 0 | read-modify-write bytes = 0 | number of write gaps = 65528 | ave write gap size = 15017705472 | See "Optimizing MPI I/O on Cray XE Systems" S-0013-20 for explanations. +--------------------------------------------------------+ PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/mpiio_collective_1m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 4 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) --------- --------- --------- ---------- ------- --------- --------- ---------- ------- -------- write 73.13 73.13 73.13 0.00 73.13 73.13 73.13 0.00 896.17743 EXCEL Max Write: 73.13 MiB/sec (76.68 MB/sec) Run finished: Mon Feb 15 07:58:16 2016 Application 177422 resources: utime ~6264s, stime ~67s, Rss ~4540, inblocks ~0, outblocks ~134217728 aprun -n 8 -N 1 -j 1 -T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -r -a MPIIO -H -C -k -c -t 1m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/mpiio_collective_1m [PE_3]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.2.4 (ANL base 3.1.2) BUILD INFO : Built Mon Jul 27 12:57:53 2015 (svn rev 10827) MT-G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR-2.10.3: MPI Coordinated Test of Parallel I/O Run began: Mon Feb 15 07:58:19 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -r -a MPIIO -H -C -k -c -t 1m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/mpiio_collective_1m Machine: Linux nid00064 Summary: api = MPIIO (version=3, subversion=0) test filename = /lus/dal/dmoen/testdir.8849/mpiio_collective_1m access = single-shared-file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 8 (1 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 64 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 2 cray_cb_nodes_multiplier = 8 romio_cb_write = enable } PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/mpiio_collective_1m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +--------------------------------------------------------+ | MPIIO read access patterns for /lus/dal/dmoen/testdir.8849/mpiio_collective_1m | independent reads = 0 | collective reads = 65536 | independent readers = 0 | aggregators = 8 | stripe count = 4 | stripe size = 1048576 | system reads = 65536 | stripe sized reads = 65536 | total bytes for reads = 68719476736 = 65536 MiB = 64 GiB | ave system read size = 1048576 | number of read gaps = 65528 | ave read gap size = 1047679 | See "Optimizing MPI I/O on Cray XE Systems" S-0013-20 for explanations. +--------------------------------------------------------+ PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/mpiio_collective_1m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 4 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) --------- --------- --------- ---------- ------- --------- --------- ---------- ------- -------- read 51.30 51.30 51.30 0.00 51.30 51.30 51.30 0.001277.58331 EXCEL Max Read: 51.30 MiB/sec (53.79 MB/sec) Run finished: Mon Feb 15 08:19:36 2016 Application 177423 resources: utime ~5590s, stime ~38s, Rss ~4460, inblocks ~0, outblocks ~0 Done IOR mpiio_collective_1m 122 0x7a c0-0c1s14n2 compute down interactive Down compute nodes mpiio_collective_1m, down= 1 + ../bin/run_IOR.pl -n 8 -N 1 /lus/dal/dmoen/testdir.8849 ./IOR_hdf5.cfg ALPS Node Information total 193G -rw-r--r-- 1 dmoen criemp 64G Feb 15 07:58 mpiio_collective_1m -rw-r--r-- 1 dmoen criemp 64G Feb 15 06:59 mpiio_collective_32m -rw-r--r-- 1 dmoen criemp 64G Feb 15 07:27 mpiio_collective_64m /lus/dal/dmoen/testdir.8849 stripe_count: -1 stripe_size: 1048576 stripe_offset: -1 /lus/dal/dmoen/testdir.8849/mpiio_collective_32m lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 10434864 0x9f3930 0 0 10610544 0xa1e770 0 2 10565824 0xa138c0 0 1 10306270 0x9d42de 0 /lus/dal/dmoen/testdir.8849/mpiio_collective_1m lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 10565826 0xa138c2 0 1 10306272 0x9d42e0 0 3 10434866 0x9f3932 0 0 10610546 0xa1e772 0 /lus/dal/dmoen/testdir.8849/mpiio_collective_64m lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 10610545 0xa1e771 0 2 10565825 0xa138c1 0 1 10306271 0x9d42df 0 3 10434865 0x9f3931 0 Start IOR test hdf5_collective_32m aprun -n 8 -N 1 -j 1 -T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -w -a HDF5 -H -C -k -c -t 32m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/hdf5_collective_32m [PE_1]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.2.4 (ANL base 3.1.2) BUILD INFO : Built Mon Jul 27 12:57:53 2015 (svn rev 10827) MT-G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR-2.10.3: MPI Coordinated Test of Parallel I/O Run began: Mon Feb 15 08:19:46 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -w -a HDF5 -H -C -k -c -t 32m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/hdf5_collective_32m Machine: Linux nid00032 Summary: api = HDF5-1.8.14 (Parallel) test filename = /lus/dal/dmoen/testdir.8849/hdf5_collective_32m access = single-shared-file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 8 (1 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 64 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 2 cray_cb_nodes_multiplier = 8 romio_cb_write = enable } PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/hdf5_collective_32m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +--------------------------------------------------------+ | MPIIO write access patterns for /lus/dal/dmoen/testdir.8849/hdf5_collective_32m | independent writes = 6 | collective writes = 2048 | independent writers = 6 | aggregators = 8 | stripe count = 4 | stripe size = 1048576 | system writes = 67590 | stripe sized writes = 63488 | total bytes for writes = 68719478136 = 65536 MiB = 64 GiB | ave system write size = 1016710 | read-modify-write count = 0 | read-modify-write bytes = 0 | number of write gaps = 16382 | ave write gap size = 14998141547 | See "Optimizing MPI I/O on Cray XE Systems" S-0013-20 for explanations. +--------------------------------------------------------+ PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/hdf5_collective_32m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 4 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) --------- --------- --------- ---------- ------- --------- --------- ---------- ------- -------- write 89.10 89.10 89.10 0.00 2.78 2.78 2.78 0.00 735.54412 EXCEL Max Write: 89.10 MiB/sec (93.43 MB/sec) Run finished: Mon Feb 15 08:32:01 2016 Application 177424 resources: utime ~3000s, stime ~57s, Rss ~4704, inblocks ~0, outblocks ~134233536 aprun -n 8 -N 1 -j 1 -T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -r -a HDF5 -H -C -k -c -t 32m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/hdf5_collective_32m [PE_1]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.2.4 (ANL base 3.1.2) BUILD INFO : Built Mon Jul 27 12:57:53 2015 (svn rev 10827) MT-G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR-2.10.3: MPI Coordinated Test of Parallel I/O Run began: Mon Feb 15 08:32:09 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -r -a HDF5 -H -C -k -c -t 32m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/hdf5_collective_32m Machine: Linux nid00064 Summary: api = HDF5-1.8.14 (Parallel) test filename = /lus/dal/dmoen/testdir.8849/hdf5_collective_32m access = single-shared-file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 8 (1 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 64 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 2 cray_cb_nodes_multiplier = 8 romio_cb_write = enable } PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/hdf5_collective_32m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +--------------------------------------------------------+ | MPIIO read access patterns for /lus/dal/dmoen/testdir.8849/hdf5_collective_32m | independent reads = 64 | collective reads = 2048 | independent readers = 8 | aggregators = 8 | stripe count = 4 | stripe size = 1048576 | system reads = 67648 | stripe sized reads = 63488 | total bytes for reads = 68719496832 = 65536 MiB = 64 GiB | ave system read size = 1015839 | number of read gaps = 16393 | ave read gap size = 4175178 | See "Optimizing MPI I/O on Cray XE Systems" S-0013-20 for explanations. +--------------------------------------------------------+ PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/hdf5_collective_32m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 4 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) --------- --------- --------- ---------- ------- --------- --------- ---------- ------- -------- read 69.70 69.70 69.70 0.00 2.18 2.18 2.18 0.00 940.23724 EXCEL Max Read: 69.70 MiB/sec (73.09 MB/sec) Run finished: Mon Feb 15 08:47:49 2016 Application 177425 resources: utime ~1496s, stime ~27s, Rss ~4556, inblocks ~0, outblocks ~0 Done IOR hdf5_collective_32m 122 0x7a c0-0c1s14n2 compute down interactive Down compute nodes hdf5_collective_32m, down= 1 total 257G -rw-r--r-- 1 dmoen criemp 65G Feb 15 08:32 hdf5_collective_32m -rw-r--r-- 1 dmoen criemp 64G Feb 15 07:58 mpiio_collective_1m -rw-r--r-- 1 dmoen criemp 64G Feb 15 06:59 mpiio_collective_32m -rw-r--r-- 1 dmoen criemp 64G Feb 15 07:27 mpiio_collective_64m /lus/dal/dmoen/testdir.8849 stripe_count: -1 stripe_size: 1048576 stripe_offset: -1 /lus/dal/dmoen/testdir.8849/mpiio_collective_32m lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 10434864 0x9f3930 0 0 10610544 0xa1e770 0 2 10565824 0xa138c0 0 1 10306270 0x9d42de 0 /lus/dal/dmoen/testdir.8849/mpiio_collective_1m lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 10565826 0xa138c2 0 1 10306272 0x9d42e0 0 3 10434866 0x9f3932 0 0 10610546 0xa1e772 0 /lus/dal/dmoen/testdir.8849/mpiio_collective_64m lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 10610545 0xa1e771 0 2 10565825 0xa138c1 0 1 10306271 0x9d42df 0 3 10434865 0x9f3931 0 /lus/dal/dmoen/testdir.8849/hdf5_collective_32m lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 10306273 0x9d42e1 0 3 10434867 0x9f3933 0 0 10610547 0xa1e773 0 2 10565827 0xa138c3 0 Start IOR test hdf5_collective_1m aprun -n 8 -N 1 -j 1 -T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -w -a HDF5 -H -C -k -c -t 1m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/hdf5_collective_1m [PE_4]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.2.4 (ANL base 3.1.2) BUILD INFO : Built Mon Jul 27 12:57:53 2015 (svn rev 10827) MT-G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR-2.10.3: MPI Coordinated Test of Parallel I/O Run began: Mon Feb 15 08:48:00 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -w -a HDF5 -H -C -k -c -t 1m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/hdf5_collective_1m Machine: Linux nid00032 Summary: api = HDF5-1.8.14 (Parallel) test filename = /lus/dal/dmoen/testdir.8849/hdf5_collective_1m access = single-shared-file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 8 (1 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 64 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 2 cray_cb_nodes_multiplier = 8 romio_cb_write = enable } PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/hdf5_collective_1m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +--------------------------------------------------------+ | MPIIO write access patterns for /lus/dal/dmoen/testdir.8849/hdf5_collective_1m | independent writes = 6 | collective writes = 65536 | independent writers = 6 | aggregators = 8 | stripe count = 4 | stripe size = 1048576 | system writes = 131078 | stripe sized writes = 0 | total bytes for writes = 68719478136 = 65536 MiB = 64 GiB | ave system write size = 524264 | read-modify-write count = 0 | read-modify-write bytes = 0 | number of write gaps = 131070 | ave write gap size = 15023833301 | See "Optimizing MPI I/O on Cray XE Systems" S-0013-20 for explanations. +--------------------------------------------------------+ PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/hdf5_collective_1m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 4 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) --------- --------- --------- ---------- ------- --------- --------- ---------- ------- -------- write 70.92 70.92 70.92 0.00 70.92 70.92 70.92 0.00 924.03787 EXCEL Max Write: 70.92 MiB/sec (74.37 MB/sec) Run finished: Mon Feb 15 09:03:24 2016 Application 177426 resources: utime ~6425s, stime ~68s, Rss ~4740, inblocks ~0, outblocks ~134218984 aprun -n 8 -N 1 -j 1 -T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -r -a HDF5 -H -C -k -c -t 1m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/hdf5_collective_1m [PE_2]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.2.4 (ANL base 3.1.2) BUILD INFO : Built Mon Jul 27 12:57:53 2015 (svn rev 10827) MT-G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR-2.10.3: MPI Coordinated Test of Parallel I/O Run began: Mon Feb 15 09:03:26 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -r -a HDF5 -H -C -k -c -t 1m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/hdf5_collective_1m Machine: Linux nid00064 Summary: api = HDF5-1.8.14 (Parallel) test filename = /lus/dal/dmoen/testdir.8849/hdf5_collective_1m access = single-shared-file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 8 (1 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 64 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 2 cray_cb_nodes_multiplier = 8 romio_cb_write = enable } PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/hdf5_collective_1m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +--------------------------------------------------------+ | MPIIO read access patterns for /lus/dal/dmoen/testdir.8849/hdf5_collective_1m | independent reads = 64 | collective reads = 65536 | independent readers = 8 | aggregators = 8 | stripe count = 4 | stripe size = 1048576 | system reads = 131136 | stripe sized reads = 0 | total bytes for reads = 68719496832 = 65536 MiB = 64 GiB | ave system read size = 524032 | number of read gaps = 131081 | ave read gap size = 523748 | See "Optimizing MPI I/O on Cray XE Systems" S-0013-20 for explanations. +--------------------------------------------------------+ PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/hdf5_collective_1m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 4 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) --------- --------- --------- ---------- ------- --------- --------- ---------- ------- -------- read 44.85 44.85 44.85 0.00 44.85 44.85 44.85 0.001461.06738 EXCEL Max Read: 44.85 MiB/sec (47.03 MB/sec) Run finished: Mon Feb 15 09:27:47 2016 Application 177427 resources: utime ~5920s, stime ~42s, Rss ~4588, inblocks ~0, outblocks ~0 Done IOR hdf5_collective_1m 122 0x7a c0-0c1s14n2 compute down interactive Down compute nodes hdf5_collective_1m, down= 1 total 321G -rw-r--r-- 1 dmoen criemp 65G Feb 15 09:03 hdf5_collective_1m -rw-r--r-- 1 dmoen criemp 65G Feb 15 08:32 hdf5_collective_32m -rw-r--r-- 1 dmoen criemp 64G Feb 15 07:58 mpiio_collective_1m -rw-r--r-- 1 dmoen criemp 64G Feb 15 06:59 mpiio_collective_32m -rw-r--r-- 1 dmoen criemp 64G Feb 15 07:27 mpiio_collective_64m /lus/dal/dmoen/testdir.8849 stripe_count: -1 stripe_size: 1048576 stripe_offset: -1 /lus/dal/dmoen/testdir.8849/mpiio_collective_32m lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 10434864 0x9f3930 0 0 10610544 0xa1e770 0 2 10565824 0xa138c0 0 1 10306270 0x9d42de 0 /lus/dal/dmoen/testdir.8849/mpiio_collective_1m lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 10565826 0xa138c2 0 1 10306272 0x9d42e0 0 3 10434866 0x9f3932 0 0 10610546 0xa1e772 0 /lus/dal/dmoen/testdir.8849/mpiio_collective_64m lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 10610545 0xa1e771 0 2 10565825 0xa138c1 0 1 10306271 0x9d42df 0 3 10434865 0x9f3931 0 /lus/dal/dmoen/testdir.8849/hdf5_collective_32m lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 10306273 0x9d42e1 0 3 10434867 0x9f3933 0 0 10610547 0xa1e773 0 2 10565827 0xa138c3 0 /lus/dal/dmoen/testdir.8849/hdf5_collective_1m lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 10434868 0x9f3934 0 0 10610548 0xa1e774 0 2 10565828 0xa138c4 0 1 10306274 0x9d42e2 0 Start IOR test hdf5_collective_64m aprun -n 8 -N 1 -j 1 -T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -w -a HDF5 -H -C -k -c -t 64m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/hdf5_collective_64m [PE_4]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.2.4 (ANL base 3.1.2) BUILD INFO : Built Mon Jul 27 12:57:53 2015 (svn rev 10827) MT-G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR-2.10.3: MPI Coordinated Test of Parallel I/O Run began: Mon Feb 15 09:27:57 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -w -a HDF5 -H -C -k -c -t 64m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/hdf5_collective_64m Machine: Linux nid00032 Summary: api = HDF5-1.8.14 (Parallel) test filename = /lus/dal/dmoen/testdir.8849/hdf5_collective_64m access = single-shared-file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 8 (1 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 64 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 2 cray_cb_nodes_multiplier = 8 romio_cb_write = enable } PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/hdf5_collective_64m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +--------------------------------------------------------+ | MPIIO write access patterns for /lus/dal/dmoen/testdir.8849/hdf5_collective_64m | independent writes = 6 | collective writes = 1024 | independent writers = 6 | aggregators = 8 | stripe count = 4 | stripe size = 1048576 | system writes = 66566 | stripe sized writes = 64512 | total bytes for writes = 68719478136 = 65536 MiB = 64 GiB | ave system write size = 1032351 | read-modify-write count = 0 | read-modify-write bytes = 0 | number of write gaps = 8190 | ave write gap size = 14966412949 | See "Optimizing MPI I/O on Cray XE Systems" S-0013-20 for explanations. +--------------------------------------------------------+ PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/hdf5_collective_64m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 4 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) --------- --------- --------- ---------- ------- --------- --------- ---------- ------- -------- write 81.03 81.03 81.03 0.00 1.27 1.27 1.27 0.00 808.79373 EXCEL Max Write: 81.03 MiB/sec (84.97 MB/sec) Run finished: Mon Feb 15 09:41:26 2016 Application 177428 resources: utime ~3069s, stime ~58s, Rss ~4704, inblocks ~0, outblocks ~134225656 aprun -n 8 -N 1 -j 1 -T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -r -a HDF5 -H -C -k -c -t 64m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/hdf5_collective_64m [PE_1]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.2.4 (ANL base 3.1.2) BUILD INFO : Built Mon Jul 27 12:57:53 2015 (svn rev 10827) MT-G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR-2.10.3: MPI Coordinated Test of Parallel I/O Run began: Mon Feb 15 09:41:33 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR -r -a HDF5 -H -C -k -c -t 64m -s 1 -b 8g -o /lus/dal/dmoen/testdir.8849/hdf5_collective_64m Machine: Linux nid00064 Summary: api = HDF5-1.8.14 (Parallel) test filename = /lus/dal/dmoen/testdir.8849/hdf5_collective_64m access = single-shared-file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 8 (1 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 64 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 2 cray_cb_nodes_multiplier = 8 romio_cb_write = enable } PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/hdf5_collective_64m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 8 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 8 cray_cb_write_lock_mode = 2 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +--------------------------------------------------------+ | MPIIO read access patterns for /lus/dal/dmoen/testdir.8849/hdf5_collective_64m | independent reads = 64 | collective reads = 1024 | independent readers = 8 | aggregators = 8 | stripe count = 4 | stripe size = 1048576 | system reads = 66624 | stripe sized reads = 64512 | total bytes for reads = 68719496832 = 65536 MiB = 64 GiB | ave system read size = 1031452 | number of read gaps = 8201 | ave read gap size = 8313043 | See "Optimizing MPI I/O on Cray XE Systems" S-0013-20 for explanations. +--------------------------------------------------------+ PE 0: MPIIO hints for /lus/dal/dmoen/testdir.8849/hdf5_collective_64m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 4 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 4 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = -1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) --------- --------- --------- ---------- ------- --------- --------- ---------- ------- -------- read 69.54 69.54 69.54 0.00 1.09 1.09 1.09 0.00 942.42781 EXCEL Max Read: 69.54 MiB/sec (72.92 MB/sec) Run finished: Mon Feb 15 09:57:15 2016 Application 177429 resources: utime ~1400s, stime ~27s, Rss ~4556, inblocks ~0, outblocks ~0 Done IOR hdf5_collective_64m 122 0x7a c0-0c1s14n2 compute down interactive Down compute nodes hdf5_collective_64m, down= 1 + ls -lh /lus/dal/dmoen/testdir.8849 total 385G -rw-r--r-- 1 dmoen criemp 65G Feb 15 09:03 hdf5_collective_1m -rw-r--r-- 1 dmoen criemp 65G Feb 15 08:32 hdf5_collective_32m -rw-r--r-- 1 dmoen criemp 65G Feb 15 09:41 hdf5_collective_64m -rw-r--r-- 1 dmoen criemp 64G Feb 15 07:58 mpiio_collective_1m -rw-r--r-- 1 dmoen criemp 64G Feb 15 06:59 mpiio_collective_32m -rw-r--r-- 1 dmoen criemp 64G Feb 15 07:27 mpiio_collective_64m + lfs df UUID 1K-blocks Used Available Use% Mounted on dal-MDT0000_UUID 39314720 867992 35755908 2% /lus/dal[MDT:0] dal-MDT0001_UUID 39314720 472840 36150288 1% /lus/dal[MDT:1] dal-OST0000_UUID 459900672 101154448 335000536 23% /lus/dal[OST:0] dal-OST0001_UUID 459900672 101154436 334994404 23% /lus/dal[OST:1] dal-OST0002_UUID 459900672 101154368 334979112 23% /lus/dal[OST:2] dal-OST0003_UUID 459900672 101154412 335001596 23% /lus/dal[OST:3] filesystem summary: 1839602688 404617664 1339975648 23% /lus/dal + rm -rf /lus/dal/dmoen/testdir.8849 Test results from saturn‐CLFS‐manual‐Apr21 + RANKS=64 + RANKS_PER_NODE=2 + FS=/lus/husk/dmoen + MPIIO=./IOR_mpiio.cfg + HDF5=./IOR_hdf5.cfg + BIN=../bin + RUN_IOR=../bin/run_IOR.pl + JOBID=12507 + TESTDIR=/lus/husk/dmoen/testdir.12507 + mkdir /lus/husk/dmoen/testdir.12507 + lfs setstripe ‐S 1m ‐c ‐1 /lus/husk/dmoen/testdir.12507 + module list ++ /opt/cray/pe/modules/3.2.10.4/bin/modulecmd bash list Currently Loaded Modulefiles: 1) modules/3.2.10.4 2) alps/6.1.3‐52.7 3) nodestat/2.2‐1.0000.58434.41.84.ari 4) sdb/2.2.0‐2.1256 5) udreg/2.3.2‐6.1 6) ugni/6.0.12‐2.3 7) gni‐headers/5.0.7‐2.3 8) dmapp/7.1.0‐28.15 9) xpmem/0.1‐2.0000.66820.89.2.ari 10) llm/20.2.1‐2.12 11) nodehealth/5.2.0‐14.17 12) sysadm/2.2.1‐2.118 13) Base‐opts/2.1.1‐2.4 14) cce/8.5.0.4664 15) craype‐network‐aries 16) craype/2.5.5.2 17) cray‐libsci/16.03.1 18) pmi/5.0.10‐1.0000.11050.0.0.ari 19) rca/1.0.0‐2.0000.66809.157.31.ari 20) atp/2.0.1 21) PrgEnv‐cray/6.0.2 22) cray‐mpich/7.3.2 23) craype‐sandybridge 24) craype‐hugepages512M 25) cray‐hdf5‐parallel/1.8.14 + eval + cat /proc/fs/lustre/version lustre: 2.7.1 kernel: patchless_client build: 2.7.0‐trunk‐1.0000.f114a17.0.0‐jmiller‐lustre‐filesystem.git@f114a17‐2016‐04‐19‐14:45 + lfs df UUID 1K‐blocks Used Available Use% Mounted on husk‐MDT0000_UUID 392737536 871608 364937240 0% /lus/husk[MDT:0] husk‐MDT0001_UUID 392737536 79816 365727484 0% /lus/husk[MDT:1] husk‐MDT0002_UUID 392737536 80020 365727280 0% /lus/husk[MDT:2] husk‐MDT0003_UUID 392737536 79996 365727304 0% /lus/husk[MDT:3] husk‐OST0000_UUID 7805162980 638444952 6762966716 9% /lus/husk[OST:0] husk‐OST0001_UUID 7805162980 638537800 6762883224 9% /lus/husk[OST:1] husk‐OST0002_UUID 7805148880 643560684 6757801780 9% /lus/husk[OST:2] husk‐OST0003_UUID 7805148880 638344976 6763062860 9% /lus/husk[OST:3] husk‐OST0004_UUID 7805148880 638340644 6763063580 9% /lus/husk[OST:4] husk‐OST0005_UUID 7805148880 638309196 6763088156 9% /lus/husk[OST:5] filesystem summary: 46830921480 3835538252 40572866316 9% /lus/husk + export MPICH_VERSION_DISPLAY=1 + MPICH_VERSION_DISPLAY=1 + export MPICH_MPIIO_HINTS_DISPLAY=1 + MPICH_MPIIO_HINTS_DISPLAY=1 + export MPICH_MPIIO_STATS=1 + MPICH_MPIIO_STATS=1 + export MPICH_MPIIO_ABORT_ON_RW_ERROR=enable + MPICH_MPIIO_ABORT_ON_RW_ERROR=enable + export MPICH_CPUMASK_DISPLAY=1 + MPICH_CPUMASK_DISPLAY=1 + export IOR_HINT__MPI__romio_no_indep_rw=true + IOR_HINT__MPI__romio_no_indep_rw=true + export IOR_HINT__MPI__romio_cb_write=enable + IOR_HINT__MPI__romio_cb_write=enable + export IOR_HINT__MPI__cray_cb_write_lock_mode=1 + IOR_HINT__MPI__cray_cb_write_lock_mode=1 ++ lfs df /lus/husk/dmoen/testdir.12507 ++ grep OST ++ wc ‐l + export IOR_HINT__MPI__cray_cb_nodes_multiplier=6 + IOR_HINT__MPI__cray_cb_nodes_multiplier=6 + ../bin/run_IOR.pl ‐n 64 ‐N 2 /lus/husk/dmoen/testdir.12507 ./IOR_mpiio.cfg ALPS Node Information total 0 /lus/husk/dmoen/testdir.12507 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 Start IOR test mpiio_collective_32m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/mpiio_collective_32m [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 19:35:59 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/mpiio_collective_32mMachine: Linux nid00032 Summary: api = MPIIO (version=3, subversion=1) test filename = /lus/husk/dmoen/testdir.12507/mpiio_collective_32m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 1 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/mpiio_collective_32m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 1 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 1 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/testdir.12507/mpiio_collective_32m | independent writes = 0 | collective writes = 16384 | independent writers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system writes = 524288 | stripe sized writes = 524288 | total bytes for writes = 549755813888 = 524288 MiB = 512 GiB | ave system write size = 1048576 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 524252 | ave write gap size = 18884242326 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/mpiio_collective_32m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 2446.07 2446.07 2446.07 0.00 76.44 76.44 76.44 0.00 214.33889 64 2 1 0 1 1 0 0 1 0 33554432 549755813888 ‐1 MPIIO EXCEL Max Write: 2446.07 MiB/sec (2564.89 MB/sec) Perf_Data IOR 33554432 write 2446.070 MiB/sec Run finished: Wed Apr 20 19:39:33 2016 Application 183660 resources: utime ~3595s, stime ~184s, Rss ~4744, inblocks ~0, outblocks ~298268672 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/mpiio_collective_32m [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 19:39:47 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/mpiio_collective_32mMachine: Linux nid00032 Summary: api = MPIIO (version=3, subversion=1) test filename = /lus/husk/dmoen/testdir.12507/mpiio_collective_32m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 1 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/mpiio_collective_32m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 1 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 1 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/testdir.12507/mpiio_collective_32m | independent reads = 0 | collective reads = 16384 | independent readers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system reads = 524288 | stripe sized reads = 524288 | total bytes for reads = 549755813888 = 524288 MiB = 512 GiB | ave system read size = 1048576 | number of read gaps = 524252 | ave read gap size = 652940 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/mpiio_collective_32m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 1429.04 1429.04 1429.04 0.00 44.66 44.66 44.66 0.00 366.88005 64 2 1 0 1 1 0 0 1 0 33554432 549755813888 ‐1 MPIIO EXCEL Max Read: 1429.04 MiB/sec (1498.46 MB/sec) Perf_Data IOR 33554432 read 1429.045 MiB/sec Run finished: Wed Apr 20 19:45:54 2016 Application 183661 resources: utime ~5665s, stime ~140s, Rss ~4744, inblocks ~0, outblocks ~0 Done IOR mpiio_collective_32m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes mpiio_collective_32m, down= 3 total 513G ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 19:39 mpiio_collective_32m /lus/husk/dmoen/testdir.12507 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir.12507/mpiio_collective_32m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202754 0x45cfc42 0 3 72732436 0x455cf14 0 1 71221383 0x43ec087 0 4 73313915 0x45eae7b 0 5 72496231 0x4523467 0 0 74076620 0x46a51cc 0 Start IOR test mpiio_collective_1m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/mpiio_collective_1m [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 19:46:10 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/mpiio_collective_1mMachine: Linux nid00032 Summary: api = MPIIO (version=3, subversion=1) test filename = /lus/husk/dmoen/testdir.12507/mpiio_collective_1m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 1 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/mpiio_collective_1m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 1 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 1 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/testdir.12507/mpiio_collective_1m | independent writes = 0 | collective writes = 524288 | independent writers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system writes = 524288 | stripe sized writes = 524288 | total bytes for writes = 549755813888 = 524288 MiB = 512 GiB | ave system write size = 1048576 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 524252 | ave write gap size = 132814189114 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/mpiio_collective_1m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 2312.58 2312.58 2312.58 0.00 2312.58 2312.58 2312.58 0.00 226.71139 64 2 1 0 1 1 0 0 1 0 1048576 549755813888 ‐1 MPIIO EXCEL Max Write: 2312.58 MiB/sec (2424.91 MB/sec) Perf_Data IOR 1048576 write 2312.579 MiB/sec Run finished: Wed Apr 20 19:49:57 2016 Application 183662 resources: utime ~4345s, stime ~191s, Rss ~4744, inblocks ~0, outblocks ~357920768 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/mpiio_collective_1m [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 19:50:11 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/mpiio_collective_1mMachine: Linux nid00032 Summary: api = MPIIO (version=3, subversion=1) test filename = /lus/husk/dmoen/testdir.12507/mpiio_collective_1m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 1 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/mpiio_collective_1m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 1 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 1 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/testdir.12507/mpiio_collective_1m | independent reads = 0 | collective reads = 524288 | independent readers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system reads = 524288 | stripe sized reads = 524288 | total bytes for reads = 549755813888 = 524288 MiB = 512 GiB | ave system read size = 1048576 | number of read gaps = 524252 | ave read gap size = 5306188 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/mpiio_collective_1m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 472.24 472.24 472.24 0.00 472.24 472.24 472.24 0.001110.21557 64 2 1 0 1 1 0 0 1 0 1048576 549755813888 ‐1 MPIIO EXCEL Max Read: 472.24 MiB/sec (495.18 MB/sec) Perf_Data IOR 1048576 read 472.240 MiB/sec Run finished: Wed Apr 20 20:08:41 2016 Application 183663 resources: utime ~23682s, stime ~187s, Rss ~4744, inblocks ~0, outblocks ~0 Done IOR mpiio_collective_1m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes mpiio_collective_1m, down= 3 total 1.1T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 19:49 mpiio_collective_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 19:39 mpiio_collective_32m /lus/husk/dmoen/testdir.12507 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir.12507/mpiio_collective_1m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732437 0x455cf15 0 1 71221384 0x43ec088 0 4 73313916 0x45eae7c 0 5 72496232 0x4523468 0 0 74076621 0x46a51cd 0 2 73202755 0x45cfc43 0 /lus/husk/dmoen/testdir.12507/mpiio_collective_32m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202754 0x45cfc42 0 3 72732436 0x455cf14 0 1 71221383 0x43ec087 0 4 73313915 0x45eae7b 0 5 72496231 0x4523467 0 0 74076620 0x46a51cc 0 Start IOR test mpiio_collective_64m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/mpiio_collective_64m [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 20:08:57 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/mpiio_collective_64mMachine: Linux nid00032 Summary: api = MPIIO (version=3, subversion=1) test filename = /lus/husk/dmoen/testdir.12507/mpiio_collective_64m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 1 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/mpiio_collective_64m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 1 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 1 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/testdir.12507/mpiio_collective_64m | independent writes = 0 | collective writes = 8192 | independent writers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system writes = 524288 | stripe sized writes = 524288 | total bytes for writes = 549755813888 = 524288 MiB = 512 GiB | ave system write size = 1048576 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 294876 | ave write gap size = 16780223599 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/mpiio_collective_64m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 2528.01 2528.01 2528.01 0.00 39.50 39.50 39.50 0.00 207.39198 64 2 1 0 1 1 0 0 1 0 67108864 549755813888 ‐1 MPIIO EXCEL Max Write: 2528.01 MiB/sec (2650.81 MB/sec) Perf_Data IOR 67108864 write 2528.005 MiB/sec Run finished: Wed Apr 20 20:12:25 2016 Application 183664 resources: utime ~3889s, stime ~180s, Rss ~4744, inblocks ~0, outblocks ~357920768 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/mpiio_collective_64m [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 20:12:40 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/mpiio_collective_64mMachine: Linux nid00032 Summary: api = MPIIO (version=3, subversion=1) test filename = /lus/husk/dmoen/testdir.12507/mpiio_collective_64m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 1 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/mpiio_collective_64m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 1 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 1 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/testdir.12507/mpiio_collective_64m | independent reads = 0 | collective reads = 8192 | independent readers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system reads = 524288 | stripe sized reads = 524288 | total bytes for reads = 549755813888 = 524288 MiB = 512 GiB | ave system read size = 1048576 | number of read gaps = 294876 | ave read gap size = 1040511 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/mpiio_collective_64m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 1832.66 1832.66 1832.66 0.00 28.64 28.64 28.64 0.00 286.08063 64 2 1 0 1 1 0 0 1 0 67108864 549755813888 ‐1 MPIIO EXCEL Max Read: 1832.66 MiB/sec (1921.68 MB/sec) Perf_Data IOR 67108864 read 1832.658 MiB/sec Run finished: Wed Apr 20 20:17:26 2016 Application 183665 resources: utime ~4638s, stime ~139s, Rss ~4744, inblocks ~0, outblocks ~0 Done IOR mpiio_collective_64m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes mpiio_collective_64m, down= 3 + export IOR_HINT__MPI__cray_cb_write_lock_mode=0 + IOR_HINT__MPI__cray_cb_write_lock_mode=0 + export IOR_HINT__MPI__cray_cb_nodes_multiplier=0 + IOR_HINT__MPI__cray_cb_nodes_multiplier=0 + ../bin/run_IOR.pl ‐n 64 ‐N 2 /lus/husk/dmoen/testdir.12507 ./IOR_hdf5.cfg ALPS Node Information total 1.6T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 19:49 mpiio_collective_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 19:39 mpiio_collective_32m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 20:12 mpiio_collective_64m /lus/husk/dmoen/testdir.12507 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir.12507/mpiio_collective_1m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732437 0x455cf15 0 1 71221384 0x43ec088 0 4 73313916 0x45eae7c 0 5 72496232 0x4523468 0 0 74076621 0x46a51cd 0 2 73202755 0x45cfc43 0 /lus/husk/dmoen/testdir.12507/mpiio_collective_32m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202754 0x45cfc42 0 3 72732436 0x455cf14 0 1 71221383 0x43ec087 0 4 73313915 0x45eae7b 0 5 72496231 0x4523467 0 0 74076620 0x46a51cc 0 /lus/husk/dmoen/testdir.12507/mpiio_collective_64m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221385 0x43ec089 0 4 73313917 0x45eae7d 0 5 72496233 0x4523469 0 0 74076622 0x46a51ce 0 2 73202756 0x45cfc44 0 3 72732438 0x455cf16 0 Start IOR test hdf5_collective_64m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/hdf5_collective_64m [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 20:17:44 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/hdf5_collective_64mMachine: Linux nid00032 Summary: api = HDF5‐1.8.14 (Parallel) test filename = /lus/husk/dmoen/testdir.12507/hdf5_collective_64m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 0 cray_cb_nodes_multiplier = 0 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/hdf5_collective_64m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 6 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/testdir.12507/hdf5_collective_64m | independent writes = 6 | collective writes = 8192 | independent writers = 6 | aggregators = 6 | stripe count = 6 | stripe size = 1048576 | system writes = 532486 | stripe sized writes = 516096 | total bytes for writes = 549755815288 = 524288 MiB = 512 GiB | ave system write size = 1032432 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 49147 | ave write gap size = 16791196750 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/hdf5_collective_64m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 1666.58 1666.58 1666.58 0.00 26.04 26.04 26.04 0.00 314.58875 64 2 1 0 1 1 0 0 1 0 67108864 549755813888 ‐1 HDF5 EXCEL Max Write: 1666.58 MiB/sec (1747.54 MB/sec) Perf_Data IOR 67108864 write 1666.582 MiB/sec Run finished: Wed Apr 20 20:22:59 2016 Application 183666 resources: utime ~7127s, stime ~203s, Rss ~5404, inblocks ~0, outblocks ~536894568 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/hdf5_collective_64m [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 20:23:07 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/hdf5_collective_64mMachine: Linux nid00032 Summary: api = HDF5‐1.8.14 (Parallel) test filename = /lus/husk/dmoen/testdir.12507/hdf5_collective_64m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 0 cray_cb_nodes_multiplier = 0 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/hdf5_collective_64m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 6 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/testdir.12507/hdf5_collective_64m | independent reads = 512 | collective reads = 8192 | independent readers = 64 | aggregators = 6 | stripe count = 6 | stripe size = 1048576 | system reads = 532992 | stripe sized reads = 516096 | total bytes for reads = 549755974656 = 524288 MiB = 512 GiB | ave system read size = 1031452 | number of read gaps = 49159 | ave read gap size = 1040129 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/hdf5_collective_64m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 786.34 786.34 786.34 0.00 12.29 12.29 12.29 0.00 666.74278 64 2 1 0 1 1 0 0 1 0 67108864 549755813888 ‐1 HDF5 EXCEL Max Read: 786.34 MiB/sec (824.54 MB/sec) Perf_Data IOR 67108864 read 786.342 MiB/sec Run finished: Wed Apr 20 20:34:14 2016 Application 183667 resources: utime ~14974s, stime ~193s, Rss ~5228, inblocks ~0, outblocks ~0 Done IOR hdf5_collective_64m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes hdf5_collective_64m, down= 3 total 2.1T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 513G Apr 20 20:21 hdf5_collective_64m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 19:49 mpiio_collective_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 19:39 mpiio_collective_32m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 20:12 mpiio_collective_64m /lus/husk/dmoen/testdir.12507 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir.12507/mpiio_collective_1m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732437 0x455cf15 0 1 71221384 0x43ec088 0 4 73313916 0x45eae7c 0 5 72496232 0x4523468 0 0 74076621 0x46a51cd 0 2 73202755 0x45cfc43 0 /lus/husk/dmoen/testdir.12507/mpiio_collective_32m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202754 0x45cfc42 0 3 72732436 0x455cf14 0 1 71221383 0x43ec087 0 4 73313915 0x45eae7b 0 5 72496231 0x4523467 0 0 74076620 0x46a51cc 0 /lus/husk/dmoen/testdir.12507/mpiio_collective_64m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221385 0x43ec089 0 4 73313917 0x45eae7d 0 5 72496233 0x4523469 0 0 74076622 0x46a51ce 0 2 73202756 0x45cfc44 0 3 72732438 0x455cf16 0 /lus/husk/dmoen/testdir.12507/hdf5_collective_64m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313918 0x45eae7e 0 5 72496234 0x452346a 0 0 74076623 0x46a51cf 0 2 73202757 0x45cfc45 0 3 72732439 0x455cf17 0 1 71221386 0x43ec08a 0 Start IOR test hdf5_collective_1m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/hdf5_collective_1m [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 20:34:40 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/hdf5_collective_1mMachine: Linux nid00032 Summary: api = HDF5‐1.8.14 (Parallel) test filename = /lus/husk/dmoen/testdir.12507/hdf5_collective_1m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 0 cray_cb_nodes_multiplier = 0 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/hdf5_collective_1m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 6 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/testdir.12507/hdf5_collective_1m | independent writes = 6 | collective writes = 524288 | independent writers = 6 | aggregators = 6 | stripe count = 6 | stripe size = 1048576 | system writes = 1048582 | stripe sized writes = 0 | total bytes for writes = 549755815288 = 524288 MiB = 512 GiB | ave system write size = 524285 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 1048571 | ave write gap size = 49115673333 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/hdf5_collective_1m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 689.29 689.29 689.29 0.00 689.29 689.29 689.29 0.00 760.62236 64 2 1 0 1 1 0 0 1 0 1048576 549755813888 ‐1 HDF5 EXCEL Max Write: 689.29 MiB/sec (722.77 MB/sec) Perf_Data IOR 1048576 write 689.288 MiB/sec Run finished: Wed Apr 20 20:47:21 2016 Application 183668 resources: utime ~17763s, stime ~209s, Rss ~5444, inblocks ~0, outblocks ~536903960 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/hdf5_collective_1m [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 20:47:28 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/hdf5_collective_1mMachine: Linux nid00032 Summary: api = HDF5‐1.8.14 (Parallel) test filename = /lus/husk/dmoen/testdir.12507/hdf5_collective_1m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 0 cray_cb_nodes_multiplier = 0 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/hdf5_collective_1m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 6 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/testdir.12507/hdf5_collective_1m | independent reads = 512 | collective reads = 524288 | independent readers = 64 | aggregators = 6 | stripe count = 6 | stripe size = 1048576 | system reads = 1049088 | stripe sized reads = 0 | total bytes for reads = 549755974656 = 524288 MiB = 512 GiB | ave system read size = 524032 | number of read gaps = 1048587 | ave read gap size = 147421 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/hdf5_collective_1m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 178.09 178.09 178.09 0.00 178.09 178.09 178.09 0.002943.99336 64 2 1 0 1 1 0 0 1 0 1048576 549755813888 ‐1 HDF5 EXCEL Max Read: 178.09 MiB/sec (186.74 MB/sec) Perf_Data IOR 1048576 read 178.087 MiB/sec Run finished: Wed Apr 20 21:36:32 2016 Application 183669 resources: utime ~65966s, stime ~264s, Rss ~5260, inblocks ~0, outblocks ~0 Done IOR hdf5_collective_1m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes hdf5_collective_1m, down= 3 total 2.6T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 513G Apr 20 20:46 hdf5_collective_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 513G Apr 20 20:21 hdf5_collective_64m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 19:49 mpiio_collective_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 19:39 mpiio_collective_32m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 20:12 mpiio_collective_64m /lus/husk/dmoen/testdir.12507 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir.12507/mpiio_collective_1m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732437 0x455cf15 0 1 71221384 0x43ec088 0 4 73313916 0x45eae7c 0 5 72496232 0x4523468 0 0 74076621 0x46a51cd 0 2 73202755 0x45cfc43 0 /lus/husk/dmoen/testdir.12507/hdf5_collective_1m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496235 0x452346b 0 0 74076624 0x46a51d0 0 2 73202758 0x45cfc46 0 3 72732440 0x455cf18 0 1 71221387 0x43ec08b 0 4 73313919 0x45eae7f 0 /lus/husk/dmoen/testdir.12507/mpiio_collective_32m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202754 0x45cfc42 0 3 72732436 0x455cf14 0 1 71221383 0x43ec087 0 4 73313915 0x45eae7b 0 5 72496231 0x4523467 0 0 74076620 0x46a51cc 0 /lus/husk/dmoen/testdir.12507/mpiio_collective_64m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221385 0x43ec089 0 4 73313917 0x45eae7d 0 5 72496233 0x4523469 0 0 74076622 0x46a51ce 0 2 73202756 0x45cfc44 0 3 72732438 0x455cf16 0 /lus/husk/dmoen/testdir.12507/hdf5_collective_64m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313918 0x45eae7e 0 5 72496234 0x452346a 0 0 74076623 0x46a51cf 0 2 73202757 0x45cfc45 0 3 72732439 0x455cf17 0 1 71221386 0x43ec08a 0 Start IOR test hdf5_collective_32m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/hdf5_collective_32m [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 21:37:03 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/hdf5_collective_32mMachine: Linux nid00032 Summary: api = HDF5‐1.8.14 (Parallel) test filename = /lus/husk/dmoen/testdir.12507/hdf5_collective_32m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 0 cray_cb_nodes_multiplier = 0 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/hdf5_collective_32m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 6 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/testdir.12507/hdf5_collective_32m | independent writes = 6 | collective writes = 16384 | independent writers = 6 | aggregators = 6 | stripe count = 6 | stripe size = 1048576 | system writes = 540678 | stripe sized writes = 507904 | total bytes for writes = 549755815288 = 524288 MiB = 512 GiB | ave system write size = 1016789 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 98299 | ave write gap size = 16850813098 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/hdf5_collective_32m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 1730.57 1730.57 1730.57 0.00 54.08 54.08 54.08 0.00 302.95630 64 2 1 0 1 1 0 0 1 0 33554432 549755813888 ‐1 HDF5 EXCEL Max Write: 1730.57 MiB/sec (1814.64 MB/sec) Perf_Data IOR 33554432 write 1730.573 MiB/sec Run finished: Wed Apr 20 21:42:06 2016 Application 183670 resources: utime ~6850s, stime ~207s, Rss ~5404, inblocks ~0, outblocks ~536916000 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/hdf5_collective_32m [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 21:42:14 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.12507/hdf5_collective_32mMachine: Linux nid00032 Summary: api = HDF5‐1.8.14 (Parallel) test filename = /lus/husk/dmoen/testdir.12507/hdf5_collective_32m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 0 cray_cb_nodes_multiplier = 0 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/hdf5_collective_32m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 6 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/testdir.12507/hdf5_collective_32m | independent reads = 512 | collective reads = 16384 | independent readers = 64 | aggregators = 6 | stripe count = 6 | stripe size = 1048576 | system reads = 541184 | stripe sized reads = 507904 | total bytes for reads = 549755974656 = 524288 MiB = 512 GiB | ave system read size = 1015839 | number of read gaps = 98311 | ave read gap size = 522149 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.12507/hdf5_collective_32m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 638.33 638.33 638.33 0.00 19.95 19.95 19.95 0.00 821.34536 64 2 1 0 1 1 0 0 1 0 33554432 549755813888 ‐1 HDF5 EXCEL Max Read: 638.33 MiB/sec (669.34 MB/sec) Perf_Data IOR 33554432 read 638.328 MiB/sec Run finished: Wed Apr 20 21:55:55 2016 Application 183671 resources: utime ~18312s, stime ~194s, Rss ~5228, inblocks ~0, outblocks ~0 Done IOR hdf5_collective_32m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes hdf5_collective_32m, down= 3 + ls ‐lh /lus/husk/dmoen/testdir.12507 total 3.1T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 513G Apr 20 20:46 hdf5_collective_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 513G Apr 20 21:41 hdf5_collective_32m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 513G Apr 20 20:21 hdf5_collective_64m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 19:49 mpiio_collective_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 19:39 mpiio_collective_32m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 20:12 mpiio_collective_64m + lfs df UUID 1K‐blocks Used Available Use% Mounted on husk‐MDT0000_UUID 392737536 871612 364937236 0% /lus/husk[MDT:0] husk‐MDT0001_UUID 392737536 79816 365727484 0% /lus/husk[MDT:1] husk‐MDT0002_UUID 392737536 80020 365727280 0% /lus/husk[MDT:2] husk‐MDT0003_UUID 392737536 79996 365727304 0% /lus/husk[MDT:3] husk‐OST0000_UUID 7805162980 638444936 6762966732 9% /lus/husk[OST:0] husk‐OST0001_UUID 7805162980 638537764 6762883260 9% /lus/husk[OST:1] husk‐OST0002_UUID 7805148880 643560652 6757801808 9% /lus/husk[OST:2] husk‐OST0003_UUID 7805148880 638344924 6763062912 9% /lus/husk[OST:3] husk‐OST0004_UUID 7805148880 638340576 6763063648 9% /lus/husk[OST:4] husk‐OST0005_UUID 7805148880 638309180 6763088172 9% /lus/husk[OST:5] filesystem summary: 46830921480 3835538032 40572866532 9% /lus/husk + rm ‐rf /lus/husk/dmoen/testdir.12507 + RANKS=64 + RANKS_PER_NODE=2 + FS=/lus/husk/dmoen + MPIIO=./IOR_mpiio.cfg + HDF5=./IOR_hdf5.cfg + BIN=../bin + RUN_IOR=../bin/run_IOR.pl + JOBID=10606 + TESTDIR=/lus/husk/dmoen/testdir.10606 + mkdir /lus/husk/dmoen/testdir.10606 + lfs setstripe ‐S 1m ‐c ‐1 /lus/husk/dmoen/testdir.10606 + module list ++ /opt/cray/pe/modules/3.2.10.4/bin/modulecmd bash list Currently Loaded Modulefiles: 1) modules/3.2.10.4 2) alps/6.1.3‐52.7 3) nodestat/2.2‐1.0000.58434.41.84.ari 4) sdb/2.2.0‐2.1256 5) udreg/2.3.2‐6.1 6) ugni/6.0.12‐2.3 7) gni‐headers/5.0.7‐2.3 8) dmapp/7.1.0‐28.15 9) xpmem/0.1‐2.0000.66820.89.2.ari 10) llm/20.2.1‐2.12 11) nodehealth/5.2.0‐14.17 12) sysadm/2.2.1‐2.118 13) Base‐opts/2.1.1‐2.4 14) cce/8.5.0.4664 15) craype‐network‐aries 16) craype/2.5.5.2 17) cray‐libsci/16.03.1 18) pmi/5.0.10‐1.0000.11050.0.0.ari 19) rca/1.0.0‐2.0000.66809.157.31.ari 20) atp/2.0.1 21) PrgEnv‐cray/6.0.2 22) cray‐mpich/7.3.2 23) craype‐sandybridge 24) craype‐hugepages512M 25) cray‐hdf5‐parallel/1.8.14 + eval + cat /proc/fs/lustre/version lustre: 2.7.1 kernel: patchless_client build: 2.7.0‐trunk‐1.0000.f114a17.0.0‐jmiller‐lustre‐filesystem.git@f114a17‐2016‐04‐19‐14:45 + lfs df UUID 1K‐blocks Used Available Use% Mounted on husk‐MDT0000_UUID 392737536 871608 364937240 0% /lus/husk[MDT:0] husk‐MDT0001_UUID 392737536 79816 365727484 0% /lus/husk[MDT:1] husk‐MDT0002_UUID 392737536 80020 365727280 0% /lus/husk[MDT:2] husk‐MDT0003_UUID 392737536 79996 365727304 0% /lus/husk[MDT:3] husk‐OST0000_UUID 7805162980 101572620 7298723908 1% /lus/husk[OST:0] husk‐OST0001_UUID 7805162980 101665384 7298674296 1% /lus/husk[OST:1] husk‐OST0002_UUID 7805148880 106688280 7293591812 1% /lus/husk[OST:2] husk‐OST0003_UUID 7805148880 101472548 7298820148 1% /lus/husk[OST:3] husk‐OST0004_UUID 7805148880 101468208 7298841356 1% /lus/husk[OST:4] husk‐OST0005_UUID 7805148880 101436808 7298864864 1% /lus/husk[OST:5] filesystem summary: 46830921480 614303848 43787516384 1% /lus/husk + export MPICH_VERSION_DISPLAY=1 + MPICH_VERSION_DISPLAY=1 + export MPICH_MPIIO_HINTS_DISPLAY=1 + MPICH_MPIIO_HINTS_DISPLAY=1 + export MPICH_MPIIO_STATS=1 + MPICH_MPIIO_STATS=1 + export MPICH_MPIIO_ABORT_ON_RW_ERROR=enable + MPICH_MPIIO_ABORT_ON_RW_ERROR=enable + export MPICH_CPUMASK_DISPLAY=1 + MPICH_CPUMASK_DISPLAY=1 + export IOR_HINT__MPI__romio_no_indep_rw=true + IOR_HINT__MPI__romio_no_indep_rw=true + export IOR_HINT__MPI__romio_cb_write=enable + IOR_HINT__MPI__romio_cb_write=enable + export IOR_HINT__MPI__cray_cb_write_lock_mode=4 + IOR_HINT__MPI__cray_cb_write_lock_mode=4 ++ lfs df /lus/husk/dmoen/testdir.10606 ++ grep OST ++ wc ‐l + export IOR_HINT__MPI__cray_cb_nodes_multiplier=6 + IOR_HINT__MPI__cray_cb_nodes_multiplier=6 + ../bin/run_IOR.pl ‐n 64 ‐N 2 /lus/husk/dmoen/testdir.10606 ./IOR_mpiio.cfg ALPS Node Information total 0 /lus/husk/dmoen/testdir.10606 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 Start IOR test mpiio_collective_64m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/mpiio_collective_64m [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 18:00:17 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/mpiio_collective_64mMachine: Linux nid00032 Summary: api = MPIIO (version=3, subversion=1) test filename = /lus/husk/dmoen/testdir.10606/mpiio_collective_64m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 4 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/mpiio_collective_64m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/testdir.10606/mpiio_collective_64m | independent writes = 0 | collective writes = 8192 | independent writers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system writes = 524288 | stripe sized writes = 524288 | total bytes for writes = 549755813888 = 524288 MiB = 512 GiB | ave system write size = 1048576 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 294876 | ave write gap size = 16780223599 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/mpiio_collective_64m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 2530.04 2530.04 2530.04 0.00 39.53 39.53 39.53 0.00 207.22540 64 2 1 0 1 1 0 0 1 0 67108864 549755813888 ‐1 MPIIO EXCEL Max Write: 2530.04 MiB/sec (2652.94 MB/sec) Perf_Data IOR 67108864 write 2530.037 MiB/sec Run finished: Wed Apr 20 18:03:44 2016 Application 183648 resources: utime ~4585s, stime ~184s, Rss ~4744, inblocks ~0, outblocks ~477224960 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/mpiio_collective_64m [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 18:04:00 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/mpiio_collective_64mMachine: Linux nid00032 Summary: api = MPIIO (version=3, subversion=1) test filename = /lus/husk/dmoen/testdir.10606/mpiio_collective_64m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 4 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/mpiio_collective_64m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/testdir.10606/mpiio_collective_64m | independent reads = 0 | collective reads = 8192 | independent readers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system reads = 524288 | stripe sized reads = 524288 | total bytes for reads = 549755813888 = 524288 MiB = 512 GiB | ave system read size = 1048576 | number of read gaps = 294876 | ave read gap size = 1040511 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/mpiio_collective_64m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 1879.76 1879.76 1879.76 0.00 29.37 29.37 29.37 0.00 278.91223 64 2 1 0 1 1 0 0 1 0 67108864 549755813888 ‐1 MPIIO EXCEL Max Read: 1879.76 MiB/sec (1971.07 MB/sec) Perf_Data IOR 67108864 read 1879.760 MiB/sec Run finished: Wed Apr 20 18:08:39 2016 Application 183649 resources: utime ~4497s, stime ~150s, Rss ~4744, inblocks ~0, outblocks ~0 Done IOR mpiio_collective_64m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes mpiio_collective_64m, down= 3 total 513G ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 18:03 mpiio_collective_64m /lus/husk/dmoen/testdir.10606 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir.10606/mpiio_collective_64m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202748 0x45cfc3c 0 3 72732430 0x455cf0e 0 1 71221377 0x43ec081 0 4 73313909 0x45eae75 0 5 72496225 0x4523461 0 0 74076614 0x46a51c6 0 Start IOR test mpiio_collective_1m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/mpiio_collective_1m [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 18:08:56 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/mpiio_collective_1mMachine: Linux nid00032 Summary: api = MPIIO (version=3, subversion=1) test filename = /lus/husk/dmoen/testdir.10606/mpiio_collective_1m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 4 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/mpiio_collective_1m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/testdir.10606/mpiio_collective_1m | independent writes = 0 | collective writes = 524288 | independent writers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system writes = 524288 | stripe sized writes = 524288 | total bytes for writes = 549755813888 = 524288 MiB = 512 GiB | ave system write size = 1048576 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 524252 | ave write gap size = 132814189114 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/mpiio_collective_1m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 2286.72 2286.72 2286.72 0.00 2286.72 2286.72 2286.72 0.00 229.27538 64 2 1 0 1 1 0 0 1 0 1048576 549755813888 ‐1 MPIIO EXCEL Max Write: 2286.72 MiB/sec (2397.80 MB/sec) Perf_Data IOR 1048576 write 2286.717 MiB/sec Run finished: Wed Apr 20 18:12:45 2016 Application 183650 resources: utime ~5225s, stime ~175s, Rss ~4744, inblocks ~0, outblocks ~477224960 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/mpiio_collective_1m [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 18:13:02 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/mpiio_collective_1mMachine: Linux nid00032 Summary: api = MPIIO (version=3, subversion=1) test filename = /lus/husk/dmoen/testdir.10606/mpiio_collective_1m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 4 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/mpiio_collective_1m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/testdir.10606/mpiio_collective_1m | independent reads = 0 | collective reads = 524288 | independent readers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system reads = 524288 | stripe sized reads = 524288 | total bytes for reads = 549755813888 = 524288 MiB = 512 GiB | ave system read size = 1048576 | number of read gaps = 524252 | ave read gap size = 5306188 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/mpiio_collective_1m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 380.16 380.16 380.16 0.00 380.16 380.16 380.16 0.001379.11196 64 2 1 0 1 1 0 0 1 0 1048576 549755813888 ‐1 MPIIO EXCEL Max Read: 380.16 MiB/sec (398.63 MB/sec) Perf_Data IOR 1048576 read 380.163 MiB/sec Run finished: Wed Apr 20 18:36:01 2016 Application 183651 resources: utime ~29191s, stime ~221s, Rss ~4744, inblocks ~0, outblocks ~0 Done IOR mpiio_collective_1m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes mpiio_collective_1m, down= 3 total 1.1T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 18:12 mpiio_collective_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 18:03 mpiio_collective_64m /lus/husk/dmoen/testdir.10606 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir.10606/mpiio_collective_1m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732431 0x455cf0f 0 1 71221378 0x43ec082 0 4 73313910 0x45eae76 0 5 72496226 0x4523462 0 0 74076615 0x46a51c7 0 2 73202749 0x45cfc3d 0 /lus/husk/dmoen/testdir.10606/mpiio_collective_64m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202748 0x45cfc3c 0 3 72732430 0x455cf0e 0 1 71221377 0x43ec081 0 4 73313909 0x45eae75 0 5 72496225 0x4523461 0 0 74076614 0x46a51c6 0 Start IOR test mpiio_collective_32m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/mpiio_collective_32m [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 18:36:19 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/mpiio_collective_32mMachine: Linux nid00032 Summary: api = MPIIO (version=3, subversion=1) test filename = /lus/husk/dmoen/testdir.10606/mpiio_collective_32m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 4 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/mpiio_collective_32m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/testdir.10606/mpiio_collective_32m | independent writes = 0 | collective writes = 16384 | independent writers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system writes = 524288 | stripe sized writes = 524288 | total bytes for writes = 549755813888 = 524288 MiB = 512 GiB | ave system write size = 1048576 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 524252 | ave write gap size = 18884242326 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/mpiio_collective_32m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 2447.36 2447.36 2447.36 0.00 76.48 76.48 76.48 0.00 214.22554 64 2 1 0 1 1 0 0 1 0 33554432 549755813888 ‐1 MPIIO EXCEL Max Write: 2447.36 MiB/sec (2566.25 MB/sec) Perf_Data IOR 33554432 write 2447.365 MiB/sec Run finished: Wed Apr 20 18:39:53 2016 Application 183652 resources: utime ~4762s, stime ~179s, Rss ~4744, inblocks ~0, outblocks ~477224960 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/mpiio_collective_32m [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 18:40:09 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a MPIIO ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/mpiio_collective_32mMachine: Linux nid00032 Summary: api = MPIIO (version=3, subversion=1) test filename = /lus/husk/dmoen/testdir.10606/mpiio_collective_32m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to MPI_File_open() { romio_no_indep_rw = true cray_cb_write_lock_mode = 4 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/mpiio_collective_32m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: hints returned from opened file { cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: } +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/testdir.10606/mpiio_collective_32m | independent reads = 0 | collective reads = 16384 | independent readers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system reads = 524288 | stripe sized reads = 524288 | total bytes for reads = 549755813888 = 524288 MiB = 512 GiB | ave system read size = 1048576 | number of read gaps = 524252 | ave read gap size = 652940 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/mpiio_collective_32m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 1404.22 1404.22 1404.22 0.00 43.88 43.88 43.88 0.00 373.36602 64 2 1 0 1 1 0 0 1 0 33554432 549755813888 ‐1 MPIIO EXCEL Max Read: 1404.22 MiB/sec (1472.43 MB/sec) Perf_Data IOR 33554432 read 1404.220 MiB/sec Run finished: Wed Apr 20 18:46:22 2016 Application 183653 resources: utime ~5844s, stime ~153s, Rss ~4744, inblocks ~0, outblocks ~0 Done IOR mpiio_collective_32m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes mpiio_collective_32m, down= 3 + ../bin/run_IOR.pl ‐n 64 ‐N 2 /lus/husk/dmoen/testdir.10606 ./IOR_hdf5.cfg ALPS Node Information total 1.6T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 18:12 mpiio_collective_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 18:39 mpiio_collective_32m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 18:03 mpiio_collective_64m /lus/husk/dmoen/testdir.10606 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir.10606/mpiio_collective_1m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732431 0x455cf0f 0 1 71221378 0x43ec082 0 4 73313910 0x45eae76 0 5 72496226 0x4523462 0 0 74076615 0x46a51c7 0 2 73202749 0x45cfc3d 0 /lus/husk/dmoen/testdir.10606/mpiio_collective_32m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221379 0x43ec083 0 4 73313911 0x45eae77 0 5 72496227 0x4523463 0 0 74076616 0x46a51c8 0 2 73202750 0x45cfc3e 0 3 72732432 0x455cf10 0 /lus/husk/dmoen/testdir.10606/mpiio_collective_64m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202748 0x45cfc3c 0 3 72732430 0x455cf0e 0 1 71221377 0x43ec081 0 4 73313909 0x45eae75 0 5 72496225 0x4523461 0 0 74076614 0x46a51c6 0 Start IOR test hdf5_collective_1m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/hdf5_collective_1m [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 18:46:38 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/hdf5_collective_1mMachine: Linux nid00032 Summary: api = HDF5‐1.8.14 (Parallel) test filename = /lus/husk/dmoen/testdir.10606/hdf5_collective_1m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 4 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/hdf5_collective_1m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/testdir.10606/hdf5_collective_1m | independent writes = 6 | collective writes = 524288 | independent writers = 6 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system writes = 1048582 | stripe sized writes = 0 | total bytes for writes = 549755815288 = 524288 MiB = 512 GiB | ave system write size = 524285 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 1048546 | ave write gap size = 132828725545 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/hdf5_collective_1m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 1781.70 1781.70 1781.70 0.00 1781.70 1781.70 1781.70 0.00 294.26212 64 2 1 0 1 1 0 0 1 0 1048576 549755813888 ‐1 HDF5 EXCEL Max Write: 1781.70 MiB/sec (1868.25 MB/sec) Perf_Data IOR 1048576 write 1781.704 MiB/sec Run finished: Wed Apr 20 18:51:33 2016 Application 183654 resources: utime ~6481s, stime ~196s, Rss ~5440, inblocks ~0, outblocks ~478265760 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/hdf5_collective_1m [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 18:51:48 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/hdf5_collective_1mMachine: Linux nid00032 Summary: api = HDF5‐1.8.14 (Parallel) test filename = /lus/husk/dmoen/testdir.10606/hdf5_collective_1m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 4 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/hdf5_collective_1m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/testdir.10606/hdf5_collective_1m | independent reads = 512 | collective reads = 524288 | independent readers = 64 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system reads = 1049088 | stripe sized reads = 0 | total bytes for reads = 549755974656 = 524288 MiB = 512 GiB | ave system read size = 524032 | number of read gaps = 1048645 | ave read gap size = 2652746 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/hdf5_collective_1m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 364.69 364.69 364.69 0.00 364.69 364.69 364.69 0.001437.60992 64 2 1 0 1 1 0 0 1 0 1048576 549755813888 ‐1 HDF5 EXCEL Max Read: 364.69 MiB/sec (382.41 MB/sec) Perf_Data IOR 1048576 read 364.694 MiB/sec Run finished: Wed Apr 20 19:15:46 2016 Application 183655 resources: utime ~28845s, stime ~208s, Rss ~5268, inblocks ~0, outblocks ~0 Done IOR hdf5_collective_1m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes hdf5_collective_1m, down= 3 total 2.1T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 513G Apr 20 18:51 hdf5_collective_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 18:12 mpiio_collective_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 18:39 mpiio_collective_32m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 18:03 mpiio_collective_64m /lus/husk/dmoen/testdir.10606 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir.10606/mpiio_collective_1m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732431 0x455cf0f 0 1 71221378 0x43ec082 0 4 73313910 0x45eae76 0 5 72496226 0x4523462 0 0 74076615 0x46a51c7 0 2 73202749 0x45cfc3d 0 /lus/husk/dmoen/testdir.10606/hdf5_collective_1m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313912 0x45eae78 0 5 72496228 0x4523464 0 0 74076617 0x46a51c9 0 2 73202751 0x45cfc3f 0 3 72732433 0x455cf11 0 1 71221380 0x43ec084 0 /lus/husk/dmoen/testdir.10606/mpiio_collective_32m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221379 0x43ec083 0 4 73313911 0x45eae77 0 5 72496227 0x4523463 0 0 74076616 0x46a51c8 0 2 73202750 0x45cfc3e 0 3 72732432 0x455cf10 0 /lus/husk/dmoen/testdir.10606/mpiio_collective_64m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202748 0x45cfc3c 0 3 72732430 0x455cf0e 0 1 71221377 0x43ec081 0 4 73313909 0x45eae75 0 5 72496225 0x4523461 0 0 74076614 0x46a51c6 0 Start IOR test hdf5_collective_32m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/hdf5_collective_32m [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 19:16:04 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/hdf5_collective_32mMachine: Linux nid00032 Summary: api = HDF5‐1.8.14 (Parallel) test filename = /lus/husk/dmoen/testdir.10606/hdf5_collective_32m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 4 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/hdf5_collective_32m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/testdir.10606/hdf5_collective_32m | independent writes = 6 | collective writes = 16384 | independent writers = 6 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system writes = 540678 | stripe sized writes = 507904 | total bytes for writes = 549755815288 = 524288 MiB = 512 GiB | ave system write size = 1016789 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 540642 | ave write gap size = 18332972531 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/hdf5_collective_32m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 2650.62 2650.62 2650.62 0.00 82.83 82.83 82.83 0.00 197.79826 64 2 1 0 1 1 0 0 1 0 33554432 549755813888 ‐1 HDF5 EXCEL Max Write: 2650.62 MiB/sec (2779.38 MB/sec) Perf_Data IOR 33554432 write 2650.620 MiB/sec Run finished: Wed Apr 20 19:19:22 2016 Application 183656 resources: utime ~4362s, stime ~179s, Rss ~5408, inblocks ~0, outblocks ~477282184 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/hdf5_collective_32m [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 19:19:40 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/hdf5_collective_32mMachine: Linux nid00032 Summary: api = HDF5‐1.8.14 (Parallel) test filename = /lus/husk/dmoen/testdir.10606/hdf5_collective_32m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 4 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/hdf5_collective_32m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/testdir.10606/hdf5_collective_32m | independent reads = 512 | collective reads = 16384 | independent readers = 64 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system reads = 541184 | stripe sized reads = 507904 | total bytes for reads = 549755974656 = 524288 MiB = 512 GiB | ave system read size = 1015839 | number of read gaps = 540712 | ave read gap size = 617149 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/hdf5_collective_32m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 1368.03 1368.03 1368.03 0.00 42.75 42.75 42.75 0.00 383.24399 64 2 1 0 1 1 0 0 1 0 33554432 549755813888 ‐1 HDF5 EXCEL Max Read: 1368.03 MiB/sec (1434.48 MB/sec) Perf_Data IOR 33554432 read 1368.027 MiB/sec Run finished: Wed Apr 20 19:26:03 2016 Application 183657 resources: utime ~6165s, stime ~150s, Rss ~5236, inblocks ~0, outblocks ~0 Done IOR hdf5_collective_32m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes hdf5_collective_32m, down= 3 total 2.6T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 513G Apr 20 18:51 hdf5_collective_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 513G Apr 20 19:19 hdf5_collective_32m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 18:12 mpiio_collective_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 18:39 mpiio_collective_32m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 18:03 mpiio_collective_64m /lus/husk/dmoen/testdir.10606 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir.10606/mpiio_collective_1m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732431 0x455cf0f 0 1 71221378 0x43ec082 0 4 73313910 0x45eae76 0 5 72496226 0x4523462 0 0 74076615 0x46a51c7 0 2 73202749 0x45cfc3d 0 /lus/husk/dmoen/testdir.10606/hdf5_collective_1m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313912 0x45eae78 0 5 72496228 0x4523464 0 0 74076617 0x46a51c9 0 2 73202751 0x45cfc3f 0 3 72732433 0x455cf11 0 1 71221380 0x43ec084 0 /lus/husk/dmoen/testdir.10606/mpiio_collective_32m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221379 0x43ec083 0 4 73313911 0x45eae77 0 5 72496227 0x4523463 0 0 74076616 0x46a51c8 0 2 73202750 0x45cfc3e 0 3 72732432 0x455cf10 0 /lus/husk/dmoen/testdir.10606/mpiio_collective_64m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202748 0x45cfc3c 0 3 72732430 0x455cf0e 0 1 71221377 0x43ec081 0 4 73313909 0x45eae75 0 5 72496225 0x4523461 0 0 74076614 0x46a51c6 0 /lus/husk/dmoen/testdir.10606/hdf5_collective_32m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496229 0x4523465 0 0 74076618 0x46a51ca 0 2 73202752 0x45cfc40 0 3 72732434 0x455cf12 0 1 71221381 0x43ec085 0 4 73313913 0x45eae79 0 Start IOR test hdf5_collective_64m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/hdf5_collective_64m [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 19:26:19 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/hdf5_collective_64mMachine: Linux nid00032 Summary: api = HDF5‐1.8.14 (Parallel) test filename = /lus/husk/dmoen/testdir.10606/hdf5_collective_64m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 4 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/hdf5_collective_64m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/testdir.10606/hdf5_collective_64m | independent writes = 6 | collective writes = 8192 | independent writers = 6 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system writes = 532486 | stripe sized writes = 516096 | total bytes for writes = 549755815288 = 524288 MiB = 512 GiB | ave system write size = 1032432 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 294882 | ave write gap size = 16790079128 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/hdf5_collective_64m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 2541.63 2541.63 2541.63 0.00 39.71 39.71 39.71 0.00 206.28021 64 2 1 0 1 1 0 0 1 0 67108864 549755813888 ‐1 HDF5 EXCEL Max Write: 2541.63 MiB/sec (2665.09 MB/sec) Perf_Data IOR 67108864 write 2541.630 MiB/sec Run finished: Wed Apr 20 19:29:45 2016 Application 183658 resources: utime ~4633s, stime ~177s, Rss ~5408, inblocks ~0, outblocks ~477253608 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/hdf5_collective_64m [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 19:30:03 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a HDF5 ‐H ‐C ‐k ‐c ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir.10606/hdf5_collective_64mMachine: Linux nid00032 Summary: api = HDF5‐1.8.14 (Parallel) test filename = /lus/husk/dmoen/testdir.10606/hdf5_collective_64m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 512 GiB hints passed to access property list { romio_no_indep_rw = true cray_cb_write_lock_mode = 4 cray_cb_nodes_multiplier = 6 romio_cb_write = enable } PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/hdf5_collective_64m: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: WARNING: showHints not working for HDF5. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/testdir.10606/hdf5_collective_64m | independent reads = 512 | collective reads = 8192 | independent readers = 64 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system reads = 532992 | stripe sized reads = 516096 | total bytes for reads = 549755974656 = 524288 MiB = 512 GiB | ave system read size = 1031452 | number of read gaps = 294949 | ave read gap size = 1040129 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/testdir.10606/hdf5_collective_64m: cb_buffer_size = 16777216 romio_cb_read = automatic romio_cb_write = automatic cb_nodes = 6 cb_align = 2 romio_no_indep_rw = false romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cb_config_list = *:* romio_filesystem_type = CRAY ADIO: Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 1547.08 1547.08 1547.08 0.00 24.17 24.17 24.17 0.00 338.88942 64 2 1 0 1 1 0 0 1 0 67108864 549755813888 ‐1 HDF5 EXCEL Max Read: 1547.08 MiB/sec (1622.23 MB/sec) Perf_Data IOR 67108864 read 1547.077 MiB/sec Run finished: Wed Apr 20 19:35:42 2016 Application 183659 resources: utime ~5843s, stime ~155s, Rss ~5228, inblocks ~0, outblocks ~0 Done IOR hdf5_collective_64m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes hdf5_collective_64m, down= 3 + ls ‐lh /lus/husk/dmoen/testdir.10606 total 3.1T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 513G Apr 20 18:51 hdf5_collective_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 513G Apr 20 19:19 hdf5_collective_32m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 513G Apr 20 19:29 hdf5_collective_64m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 18:12 mpiio_collective_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 18:39 mpiio_collective_32m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 18:03 mpiio_collective_64m + lfs df UUID 1K‐blocks Used Available Use% Mounted on husk‐MDT0000_UUID 392737536 871608 364937240 0% /lus/husk[MDT:0] husk‐MDT0001_UUID 392737536 79816 365727484 0% /lus/husk[MDT:1] husk‐MDT0002_UUID 392737536 80020 365727280 0% /lus/husk[MDT:2] husk‐MDT0003_UUID 392737536 79996 365727304 0% /lus/husk[MDT:3] husk‐OST0000_UUID 7805162980 638444952 6762966716 9% /lus/husk[OST:0] husk‐OST0001_UUID 7805162980 638537800 6762883224 9% /lus/husk[OST:1] husk‐OST0002_UUID 7805148880 643560684 6757801780 9% /lus/husk[OST:2] husk‐OST0003_UUID 7805148880 638344976 6763062860 9% /lus/husk[OST:3] husk‐OST0004_UUID 7805148880 638340644 6763063580 9% /lus/husk[OST:4] husk‐OST0005_UUID 7805148880 638309196 6763088156 9% /lus/husk[OST:5] filesystem summary: 46830921480 3835538252 40572866316 9% /lus/husk + rm ‐rf /lus/husk/dmoen/testdir.10606 + RANKS=64 + RANKS_PER_NODE=2 + FS=/lus/husk/dmoen + CFGFILE=./IOR_posix_fpp.cfg + BIN=../bin + RUN_IOR=../bin/run_IOR.pl + JOBID=15099 + TESTDIR1=/lus/husk/dmoen/testdir1.15099 + mkdir /lus/husk/dmoen/testdir1.15099 + lfs setstripe ‐S 1m ‐c ‐1 /lus/husk/dmoen/testdir1.15099 + module list ++ /opt/cray/pe/modules/3.2.10.4/bin/modulecmd bash list Currently Loaded Modulefiles: 1) modules/3.2.10.4 2) alps/6.1.3‐52.7 3) nodestat/2.2‐1.0000.58434.41.84.ari 4) sdb/2.2.0‐2.1256 5) udreg/2.3.2‐6.1 6) ugni/6.0.12‐2.3 7) gni‐headers/5.0.7‐2.3 8) dmapp/7.1.0‐28.15 9) xpmem/0.1‐2.0000.66820.89.2.ari 10) llm/20.2.1‐2.12 11) nodehealth/5.2.0‐14.17 12) sysadm/2.2.1‐2.118 13) Base‐opts/2.1.1‐2.4 14) cce/8.5.0.4664 15) craype‐network‐aries 16) craype/2.5.5.2 17) cray‐libsci/16.03.1 18) pmi/5.0.10‐1.0000.11050.0.0.ari 19) rca/1.0.0‐2.0000.66809.157.31.ari 20) atp/2.0.1 21) PrgEnv‐cray/6.0.2 22) cray‐mpich/7.3.2 23) craype‐sandybridge 24) craype‐hugepages512M 25) cray‐hdf5‐parallel/1.8.14 + eval + cat /proc/fs/lustre/version lustre: 2.7.1 kernel: patchless_client build: 2.7.0‐trunk‐1.0000.f114a17.0.0‐jmiller‐lustre‐filesystem.git@f114a17‐2016‐04‐19‐14:45 + lfs df UUID 1K‐blocks Used Available Use% Mounted on husk‐MDT0000_UUID 392737536 871620 364937228 0% /lus/husk[MDT:0] husk‐MDT0001_UUID 392737536 79816 365727484 0% /lus/husk[MDT:1] husk‐MDT0002_UUID 392737536 80020 365727280 0% /lus/husk[MDT:2] husk‐MDT0003_UUID 392737536 79996 365727304 0% /lus/husk[MDT:3] husk‐OST0000_UUID 7805162980 638444936 6762966732 9% /lus/husk[OST:0] husk‐OST0001_UUID 7805162980 638537764 6762883260 9% /lus/husk[OST:1] husk‐OST0002_UUID 7805148880 643560652 6757801808 9% /lus/husk[OST:2] husk‐OST0003_UUID 7805148880 638344924 6763062912 9% /lus/husk[OST:3] husk‐OST0004_UUID 7805148880 638340576 6763063648 9% /lus/husk[OST:4] husk‐OST0005_UUID 7805148880 638309180 6763088172 9% /lus/husk[OST:5] filesystem summary: 46830921480 3835538032 40572866532 9% /lus/husk + export MPICH_VERSION_DISPLAY=1 + MPICH_VERSION_DISPLAY=1 + export MPICH_MPIIO_HINTS_DISPLAY=1 + MPICH_MPIIO_HINTS_DISPLAY=1 + export MPICH_MPIIO_STATS=1 + MPICH_MPIIO_STATS=1 + export MPICH_MPIIO_ABORT_ON_RW_ERROR=enable + MPICH_MPIIO_ABORT_ON_RW_ERROR=enable + export MPICH_CPUMASK_DISPLAY=1 + MPICH_CPUMASK_DISPLAY=1 + export IOR_HINT__MPI__romio_no_indep_rw=true + IOR_HINT__MPI__romio_no_indep_rw=true + export IOR_HINT__MPI__romio_cb_write=enable + IOR_HINT__MPI__romio_cb_write=enable + ../bin/run_IOR.pl ‐n 64 ‐N 2 /lus/husk/dmoen/testdir1.15099 ./IOR_posix_fpp.cfg ALPS Node Information total 0 /lus/husk/dmoen/testdir1.15099 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 Start IOR test posix_directio_64m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐B ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_directio_64m [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 21:56:24 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐B ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_directio_64mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.15099/posix_directio_64m access = file‐per‐process ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 1329.66 1329.66 1329.66 0.00 20.78 20.78 20.78 0.00 394.30166 64 2 1 1 1 1 0 0 1 0 67108864 549755813888 ‐1 POSIX EXCEL Max Write: 1329.66 MiB/sec (1394.25 MB/sec) Perf_Data IOR 67108864 write 1329.662 MiB/sec Run finished: Wed Apr 20 22:02:59 2016 Application 183672 resources: utime ~93s, stime ~49s, Rss ~4744, inblocks ~0, outblocks ~0 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐B ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_directio_64m [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 22:03:07 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐B ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_directio_64mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.15099/posix_directio_64m access = file‐per‐process ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 1792.41 1792.41 1792.41 0.00 28.01 28.01 28.01 0.00 292.50504 64 2 1 1 1 1 0 0 1 0 67108864 549755813888 ‐1 POSIX EXCEL Max Read: 1792.41 MiB/sec (1879.47 MB/sec) Perf_Data IOR 67108864 read 1792.407 MiB/sec Run finished: Wed Apr 20 22:08:00 2016 Application 183673 resources: utime ~492s, stime ~50s, Rss ~4744, inblocks ~0, outblocks ~0 Done IOR posix_directio_64m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes posix_directio_64m, down= 3 total 513G ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000063 /lus/husk/dmoen/testdir1.15099 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000047 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732445 0x455cf1d 0 1 71221392 0x43ec090 0 4 73313925 0x45eae85 0 5 72496241 0x4523471 0 0 74076630 0x46a51d6 0 2 73202764 0x45cfc4c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000001 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202767 0x45cfc4f 0 3 72732450 0x455cf22 0 1 71221398 0x43ec096 0 4 73313931 0x45eae8b 0 5 72496248 0x4523478 0 0 74076636 0x46a51dc 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000032 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496287 0x452349f 0 0 74076677 0x46a5205 0 2 73202811 0x45cfc7b 0 3 72732494 0x455cf4e 0 1 71221441 0x43ec0c1 0 4 73313973 0x45eaeb5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000025 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076640 0x46a51e0 0 2 73202775 0x45cfc57 0 3 72732457 0x455cf29 0 1 71221405 0x43ec09d 0 4 73313938 0x45eae92 0 5 72496255 0x452347f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000057 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313936 0x45eae90 0 5 72496254 0x452347e 0 0 74076644 0x46a51e4 0 2 73202779 0x45cfc5b 0 3 72732461 0x455cf2d 0 1 71221409 0x43ec0a1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000026 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202800 0x45cfc70 0 3 72732482 0x455cf42 0 1 71221429 0x43ec0b5 0 4 73313961 0x45eaea9 0 5 72496277 0x4523495 0 0 74076666 0x46a51fa 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000041 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221431 0x43ec0b7 0 4 73313963 0x45eaeab 0 5 72496279 0x4523497 0 0 74076668 0x46a51fc 0 2 73202802 0x45cfc72 0 3 72732484 0x455cf44 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000031 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202789 0x45cfc65 0 3 72732472 0x455cf38 0 1 71221422 0x43ec0ae 0 4 73313953 0x45eaea1 0 5 72496270 0x452348e 0 0 74076660 0x46a51f4 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000043 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313947 0x45eae9b 0 5 72496264 0x4523488 0 0 74076653 0x46a51ed 0 2 73202787 0x45cfc63 0 3 72732471 0x455cf37 0 1 71221420 0x43ec0ac 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000039 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496256 0x4523480 0 0 74076645 0x46a51e5 0 2 73202780 0x45cfc5c 0 3 72732463 0x455cf2f 0 1 71221411 0x43ec0a3 0 4 73313943 0x45eae97 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000029 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076670 0x46a51fe 0 2 73202804 0x45cfc74 0 3 72732486 0x455cf46 0 1 71221433 0x43ec0b9 0 4 73313965 0x45eaead 0 5 72496281 0x4523499 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000011 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076678 0x46a5206 0 2 73202812 0x45cfc7c 0 3 72732495 0x455cf4f 0 1 71221442 0x43ec0c2 0 4 73313974 0x45eaeb6 0 5 72496290 0x45234a2 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000036 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496282 0x452349a 0 0 74076671 0x46a51ff 0 2 73202805 0x45cfc75 0 3 72732487 0x455cf47 0 1 71221434 0x43ec0ba 0 4 73313966 0x45eaeae 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000049 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076680 0x46a5208 0 2 73202815 0x45cfc7f 0 3 72732497 0x455cf51 0 1 71221446 0x43ec0c6 0 4 73313978 0x45eaeba 0 5 72496294 0x45234a6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000062 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732493 0x455cf4d 0 1 71221440 0x43ec0c0 0 4 73313972 0x45eaeb4 0 5 72496289 0x45234a1 0 0 74076679 0x46a5207 0 2 73202813 0x45cfc7d 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000002 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202760 0x45cfc48 0 3 72732442 0x455cf1a 0 1 71221389 0x43ec08d 0 4 73313921 0x45eae81 0 5 72496237 0x452346d 0 0 74076626 0x46a51d2 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000060 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202784 0x45cfc60 0 3 72732466 0x455cf32 0 1 71221413 0x43ec0a5 0 4 73313945 0x45eae99 0 5 72496262 0x4523486 0 0 74076651 0x46a51eb 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000040 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732449 0x455cf21 0 1 71221397 0x43ec095 0 4 73313930 0x45eae8a 0 5 72496247 0x4523477 0 0 74076637 0x46a51dd 0 2 73202772 0x45cfc54 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000046 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076659 0x46a51f3 0 2 73202795 0x45cfc6b 0 3 72732478 0x455cf3e 0 1 71221425 0x43ec0b1 0 4 73313957 0x45eaea5 0 5 72496273 0x4523491 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000009 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313958 0x45eaea6 0 5 72496274 0x4523492 0 0 74076663 0x46a51f7 0 2 73202797 0x45cfc6d 0 3 72732479 0x455cf3f 0 1 71221426 0x43ec0b2 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000034 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202778 0x45cfc5a 0 3 72732462 0x455cf2e 0 1 71221410 0x43ec0a2 0 4 73313942 0x45eae96 0 5 72496259 0x4523483 0 0 74076648 0x46a51e8 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000058 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076685 0x46a520d 0 2 73202819 0x45cfc83 0 3 72732502 0x455cf56 0 1 71221451 0x43ec0cb 0 4 73313984 0x45eaec0 0 5 72496300 0x45234ac 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000000 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076665 0x46a51f9 0 2 73202799 0x45cfc6f 0 3 72732481 0x455cf41 0 1 71221428 0x43ec0b4 0 4 73313960 0x45eaea8 0 5 72496276 0x4523494 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000035 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221437 0x43ec0bd 0 4 73313969 0x45eaeb1 0 5 72496285 0x452349d 0 0 74076674 0x46a5202 0 2 73202808 0x45cfc78 0 3 72732490 0x455cf4a 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000017 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221417 0x43ec0a9 0 4 73313952 0x45eaea0 0 5 72496268 0x452348c 0 0 74076657 0x46a51f1 0 2 73202792 0x45cfc68 0 3 72732474 0x455cf3a 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000037 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221402 0x43ec09a 0 4 73313935 0x45eae8f 0 5 72496252 0x452347c 0 0 74076642 0x46a51e2 0 2 73202776 0x45cfc58 0 3 72732458 0x455cf2a 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000004 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496275 0x4523493 0 0 74076664 0x46a51f8 0 2 73202798 0x45cfc6e 0 3 72732480 0x455cf40 0 1 71221427 0x43ec0b3 0 4 73313959 0x45eaea7 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000018 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313970 0x45eaeb2 0 5 72496286 0x452349e 0 0 74076676 0x46a5204 0 2 73202810 0x45cfc7a 0 3 72732492 0x455cf4c 0 1 71221439 0x43ec0bf 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000061 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221445 0x43ec0c5 0 4 73313977 0x45eaeb9 0 5 72496293 0x45234a5 0 0 74076684 0x46a520c 0 2 73202817 0x45cfc81 0 3 72732499 0x455cf53 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000020 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202768 0x45cfc50 0 3 72732452 0x455cf24 0 1 71221401 0x43ec099 0 4 73313933 0x45eae8d 0 5 72496250 0x452347a 0 0 74076639 0x46a51df 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000027 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496244 0x4523474 0 0 74076633 0x46a51d9 0 2 73202769 0x45cfc51 0 3 72732451 0x455cf23 0 1 71221399 0x43ec097 0 4 73313932 0x45eae8c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000005 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202766 0x45cfc4e 0 3 72732448 0x455cf20 0 1 71221395 0x43ec093 0 4 73313927 0x45eae87 0 5 72496243 0x4523473 0 0 74076632 0x46a51d8 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000051 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076675 0x46a5203 0 2 73202809 0x45cfc79 0 3 72732491 0x455cf4b 0 1 71221438 0x43ec0be 0 4 73313971 0x45eaeb3 0 5 72496288 0x45234a0 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000053 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221414 0x43ec0a6 0 4 73313948 0x45eae9c 0 5 72496265 0x4523489 0 0 74076655 0x46a51ef 0 2 73202791 0x45cfc67 0 3 72732475 0x455cf3b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000022 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202781 0x45cfc5d 0 3 72732464 0x455cf30 0 1 71221412 0x43ec0a4 0 4 73313944 0x45eae98 0 5 72496260 0x4523484 0 0 74076649 0x46a51e9 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000014 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221449 0x43ec0c9 0 4 73313982 0x45eaebe 0 5 72496298 0x45234aa 0 0 74076688 0x46a5210 0 2 73202821 0x45cfc85 0 3 72732503 0x455cf57 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000024 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202807 0x45cfc77 0 3 72732489 0x455cf49 0 1 71221436 0x43ec0bc 0 4 73313968 0x45eaeb0 0 5 72496284 0x452349c 0 0 74076673 0x46a5201 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000055 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313946 0x45eae9a 0 5 72496263 0x4523487 0 0 74076652 0x46a51ec 0 2 73202786 0x45cfc62 0 3 72732469 0x455cf35 0 1 71221418 0x43ec0aa 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000008 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076631 0x46a51d7 0 2 73202765 0x45cfc4d 0 3 72732447 0x455cf1f 0 1 71221394 0x43ec092 0 4 73313926 0x45eae86 0 5 72496242 0x4523472 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000021 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313964 0x45eaeac 0 5 72496280 0x4523498 0 0 74076669 0x46a51fd 0 2 73202803 0x45cfc73 0 3 72732485 0x455cf45 0 1 71221432 0x43ec0b8 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000007 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202762 0x45cfc4a 0 3 72732444 0x455cf1c 0 1 71221391 0x43ec08f 0 4 73313924 0x45eae84 0 5 72496240 0x4523470 0 0 74076629 0x46a51d5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000042 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313981 0x45eaebd 0 5 72496297 0x45234a9 0 0 74076687 0x46a520f 0 2 73202822 0x45cfc86 0 3 72732504 0x455cf58 0 1 71221452 0x43ec0cc 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000052 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732500 0x455cf54 0 1 71221448 0x43ec0c8 0 4 73313980 0x45eaebc 0 5 72496296 0x45234a8 0 0 74076686 0x46a520e 0 2 73202820 0x45cfc84 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000012 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313928 0x45eae88 0 5 72496245 0x4523475 0 0 74076634 0x46a51da 0 2 73202770 0x45cfc52 0 3 72732453 0x455cf25 0 1 71221403 0x43ec09b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000054 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221443 0x43ec0c3 0 4 73313975 0x45eaeb7 0 5 72496291 0x45234a3 0 0 74076681 0x46a5209 0 2 73202816 0x45cfc80 0 3 72732498 0x455cf52 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000050 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221400 0x43ec098 0 4 73313934 0x45eae8e 0 5 72496251 0x452347b 0 0 74076641 0x46a51e1 0 2 73202774 0x45cfc56 0 3 72732456 0x455cf28 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000006 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732473 0x455cf39 0 1 71221421 0x43ec0ad 0 4 73313954 0x45eaea2 0 5 72496271 0x452348f 0 0 74076661 0x46a51f5 0 2 73202796 0x45cfc6c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000015 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313949 0x45eae9d 0 5 72496266 0x452348a 0 0 74076654 0x46a51ee 0 2 73202788 0x45cfc64 0 3 72732470 0x455cf36 0 1 71221419 0x43ec0ab 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000016 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076682 0x46a520a 0 2 73202818 0x45cfc82 0 3 72732501 0x455cf55 0 1 71221447 0x43ec0c7 0 4 73313979 0x45eaebb 0 5 72496295 0x45234a7 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000048 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732467 0x455cf33 0 1 71221415 0x43ec0a7 0 4 73313951 0x45eae9f 0 5 72496267 0x452348b 0 0 74076656 0x46a51f0 0 2 73202790 0x45cfc66 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000059 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732459 0x455cf2b 0 1 71221407 0x43ec09f 0 4 73313941 0x45eae95 0 5 72496258 0x4523482 0 0 74076647 0x46a51e7 0 2 73202783 0x45cfc5f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000028 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221396 0x43ec094 0 4 73313929 0x45eae89 0 5 72496246 0x4523476 0 0 74076635 0x46a51db 0 2 73202771 0x45cfc53 0 3 72732454 0x455cf26 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000023 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221450 0x43ec0ca 0 4 73313983 0x45eaebf 0 5 72496299 0x45234ab 0 0 74076689 0x46a5211 0 2 73202823 0x45cfc87 0 3 72732505 0x455cf59 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000056 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496283 0x452349b 0 0 74076672 0x46a5200 0 2 73202806 0x45cfc76 0 3 72732488 0x455cf48 0 1 71221435 0x43ec0bb 0 4 73313967 0x45eaeaf 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000019 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221406 0x43ec09e 0 4 73313939 0x45eae93 0 5 72496257 0x4523481 0 0 74076646 0x46a51e6 0 2 73202782 0x45cfc5e 0 3 72732465 0x455cf31 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000045 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202793 0x45cfc69 0 3 72732476 0x455cf3c 0 1 71221423 0x43ec0af 0 4 73313955 0x45eaea3 0 5 72496272 0x4523490 0 0 74076662 0x46a51f6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000010 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496253 0x452347d 0 0 74076643 0x46a51e3 0 2 73202777 0x45cfc59 0 3 72732460 0x455cf2c 0 1 71221408 0x43ec0a0 0 4 73313940 0x45eae94 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000063 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496249 0x4523479 0 0 74076638 0x46a51de 0 2 73202773 0x45cfc55 0 3 72732455 0x455cf27 0 1 71221404 0x43ec09c 0 4 73313937 0x45eae91 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000030 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076627 0x46a51d3 0 2 73202761 0x45cfc49 0 3 72732443 0x455cf1b 0 1 71221390 0x43ec08e 0 4 73313922 0x45eae82 0 5 72496238 0x452346e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000013 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732483 0x455cf43 0 1 71221430 0x43ec0b6 0 4 73313962 0x45eaeaa 0 5 72496278 0x4523496 0 0 74076667 0x46a51fb 0 2 73202801 0x45cfc71 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000033 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496261 0x4523485 0 0 74076650 0x46a51ea 0 2 73202785 0x45cfc61 0 3 72732468 0x455cf34 0 1 71221416 0x43ec0a8 0 4 73313950 0x45eae9e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000044 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313923 0x45eae83 0 5 72496239 0x452346f 0 0 74076628 0x46a51d4 0 2 73202763 0x45cfc4b 0 3 72732446 0x455cf1e 0 1 71221393 0x43ec091 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000038 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202814 0x45cfc7e 0 3 72732496 0x455cf50 0 1 71221444 0x43ec0c4 0 4 73313976 0x45eaeb8 0 5 72496292 0x45234a4 0 0 74076683 0x46a520b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000003 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496269 0x452348d 0 0 74076658 0x46a51f2 0 2 73202794 0x45cfc6a 0 3 72732477 0x455cf3d 0 1 71221424 0x43ec0b0 0 4 73313956 0x45eaea4 0 Start IOR test posix_buffered_32m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_buffered_32m [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 22:08:08 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_buffered_32mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.15099/posix_buffered_32m access = file‐per‐process ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 2683.55 2683.55 2683.55 0.00 83.86 83.86 83.86 0.00 195.37093 64 2 1 1 1 1 0 0 1 0 33554432 549755813888 ‐1 POSIX EXCEL Max Write: 2683.55 MiB/sec (2813.91 MB/sec) Perf_Data IOR 33554432 write 2683.552 MiB/sec Run finished: Wed Apr 20 22:11:23 2016 Application 183674 resources: utime ~451s, stime ~181s, Rss ~4744, inblocks ~0, outblocks ~436207616 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_buffered_32m [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 22:11:35 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_buffered_32mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.15099/posix_buffered_32m access = file‐per‐process ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 2458.29 2458.29 2458.29 0.00 76.82 76.82 76.82 0.00 213.27379 64 2 1 1 1 1 0 0 1 0 33554432 549755813888 ‐1 POSIX EXCEL Max Read: 2458.29 MiB/sec (2577.70 MB/sec) Perf_Data IOR 33554432 read 2458.286 MiB/sec Run finished: Wed Apr 20 22:15:08 2016 Application 183675 resources: utime ~377s, stime ~136s, Rss ~4744, inblocks ~0, outblocks ~0 Done IOR posix_buffered_32m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes posix_buffered_32m, down= 3 total 1.1T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000063 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000063 /lus/husk/dmoen/testdir1.15099 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000047 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732445 0x455cf1d 0 1 71221392 0x43ec090 0 4 73313925 0x45eae85 0 5 72496241 0x4523471 0 0 74076630 0x46a51d6 0 2 73202764 0x45cfc4c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000001 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202767 0x45cfc4f 0 3 72732450 0x455cf22 0 1 71221398 0x43ec096 0 4 73313931 0x45eae8b 0 5 72496248 0x4523478 0 0 74076636 0x46a51dc 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000016 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314018 0x45eaee2 0 5 72496334 0x45234ce 0 0 74076723 0x46a5233 0 2 73202857 0x45cfca9 0 3 72732539 0x455cf7b 0 1 71221486 0x43ec0ee 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000032 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496287 0x452349f 0 0 74076677 0x46a5205 0 2 73202811 0x45cfc7b 0 3 72732494 0x455cf4e 0 1 71221441 0x43ec0c1 0 4 73313973 0x45eaeb5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000025 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076640 0x46a51e0 0 2 73202775 0x45cfc57 0 3 72732457 0x455cf29 0 1 71221405 0x43ec09d 0 4 73313938 0x45eae92 0 5 72496255 0x452347f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000057 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313936 0x45eae90 0 5 72496254 0x452347e 0 0 74076644 0x46a51e4 0 2 73202779 0x45cfc5b 0 3 72732461 0x455cf2d 0 1 71221409 0x43ec0a1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000026 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202800 0x45cfc70 0 3 72732482 0x455cf42 0 1 71221429 0x43ec0b5 0 4 73313961 0x45eaea9 0 5 72496277 0x4523495 0 0 74076666 0x46a51fa 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000040 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202866 0x45cfcb2 0 3 72732548 0x455cf84 0 1 71221495 0x43ec0f7 0 4 73314027 0x45eaeeb 0 5 72496343 0x45234d7 0 0 74076732 0x46a523c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000041 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221431 0x43ec0b7 0 4 73313963 0x45eaeab 0 5 72496279 0x4523497 0 0 74076668 0x46a51fc 0 2 73202802 0x45cfc72 0 3 72732484 0x455cf44 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000005 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221491 0x43ec0f3 0 4 73314023 0x45eaee7 0 5 72496339 0x45234d3 0 0 74076728 0x46a5238 0 2 73202862 0x45cfcae 0 3 72732544 0x455cf80 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000017 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313986 0x45eaec2 0 5 72496303 0x45234af 0 0 74076692 0x46a5214 0 2 73202827 0x45cfc8b 0 3 72732507 0x455cf5b 0 1 71221454 0x43ec0ce 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000031 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202789 0x45cfc65 0 3 72732472 0x455cf38 0 1 71221422 0x43ec0ae 0 4 73313953 0x45eaea1 0 5 72496270 0x452348e 0 0 74076660 0x46a51f4 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000043 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313947 0x45eae9b 0 5 72496264 0x4523488 0 0 74076653 0x46a51ed 0 2 73202787 0x45cfc63 0 3 72732471 0x455cf37 0 1 71221420 0x43ec0ac 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000057 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732524 0x455cf6c 0 1 71221473 0x43ec0e1 0 4 73314005 0x45eaed5 0 5 72496321 0x45234c1 0 0 74076711 0x46a5227 0 2 73202845 0x45cfc9d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000054 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221514 0x43ec10a 0 4 73314047 0x45eaeff 0 5 72496363 0x45234eb 0 0 74076752 0x46a5250 0 2 73202887 0x45cfcc7 0 3 72732569 0x455cf99 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000039 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496256 0x4523480 0 0 74076645 0x46a51e5 0 2 73202780 0x45cfc5c 0 3 72732463 0x455cf2f 0 1 71221411 0x43ec0a3 0 4 73313943 0x45eae97 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000029 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076670 0x46a51fe 0 2 73202804 0x45cfc74 0 3 72732486 0x455cf46 0 1 71221433 0x43ec0b9 0 4 73313965 0x45eaead 0 5 72496281 0x4523499 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000011 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076678 0x46a5206 0 2 73202812 0x45cfc7c 0 3 72732495 0x455cf4f 0 1 71221442 0x43ec0c2 0 4 73313974 0x45eaeb6 0 5 72496290 0x45234a2 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000036 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496282 0x452349a 0 0 74076671 0x46a51ff 0 2 73202805 0x45cfc75 0 3 72732487 0x455cf47 0 1 71221434 0x43ec0ba 0 4 73313966 0x45eaeae 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000051 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732531 0x455cf73 0 1 71221478 0x43ec0e6 0 4 73314010 0x45eaeda 0 5 72496326 0x45234c6 0 0 74076716 0x46a522c 0 2 73202849 0x45cfca1 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000044 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202863 0x45cfcaf 0 3 72732545 0x455cf81 0 1 71221492 0x43ec0f4 0 4 73314025 0x45eaee9 0 5 72496341 0x45234d5 0 0 74076731 0x46a523b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000049 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076680 0x46a5208 0 2 73202815 0x45cfc7f 0 3 72732497 0x455cf51 0 1 71221446 0x43ec0c6 0 4 73313978 0x45eaeba 0 5 72496294 0x45234a6 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000006 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496335 0x45234cf 0 0 74076724 0x46a5234 0 2 73202858 0x45cfcaa 0 3 72732540 0x455cf7c 0 1 71221487 0x43ec0ef 0 4 73314019 0x45eaee3 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000062 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732493 0x455cf4d 0 1 71221440 0x43ec0c0 0 4 73313972 0x45eaeb4 0 5 72496289 0x45234a1 0 0 74076679 0x46a5207 0 2 73202813 0x45cfc7d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000001 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202860 0x45cfcac 0 3 72732542 0x455cf7e 0 1 71221489 0x43ec0f1 0 4 73314021 0x45eaee5 0 5 72496337 0x45234d1 0 0 74076726 0x46a5236 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000002 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076725 0x46a5235 0 2 73202859 0x45cfcab 0 3 72732541 0x455cf7d 0 1 71221488 0x43ec0f0 0 4 73314020 0x45eaee4 0 5 72496336 0x45234d0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000062 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496353 0x45234e1 0 0 74076743 0x46a5247 0 2 73202877 0x45cfcbd 0 3 72732559 0x455cf8f 0 1 71221505 0x43ec101 0 4 73314037 0x45eaef5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000002 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202760 0x45cfc48 0 3 72732442 0x455cf1a 0 1 71221389 0x43ec08d 0 4 73313921 0x45eae81 0 5 72496237 0x452346d 0 0 74076626 0x46a51d2 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000050 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496354 0x45234e2 0 0 74076744 0x46a5248 0 2 73202878 0x45cfcbe 0 3 72732560 0x455cf90 0 1 71221508 0x43ec104 0 4 73314039 0x45eaef7 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000038 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496340 0x45234d4 0 0 74076729 0x46a5239 0 2 73202864 0x45cfcb0 0 3 72732546 0x455cf82 0 1 71221493 0x43ec0f5 0 4 73314024 0x45eaee8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000036 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076730 0x46a523a 0 2 73202865 0x45cfcb1 0 3 72732547 0x455cf83 0 1 71221494 0x43ec0f6 0 4 73314026 0x45eaeea 0 5 72496342 0x45234d6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000060 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202784 0x45cfc60 0 3 72732466 0x455cf32 0 1 71221413 0x43ec0a5 0 4 73313945 0x45eae99 0 5 72496262 0x4523486 0 0 74076651 0x46a51eb 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000015 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313995 0x45eaecb 0 5 72496312 0x45234b8 0 0 74076703 0x46a521f 0 2 73202840 0x45cfc98 0 3 72732522 0x455cf6a 0 1 71221470 0x43ec0de 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000040 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732449 0x455cf21 0 1 71221397 0x43ec095 0 4 73313930 0x45eae8a 0 5 72496247 0x4523477 0 0 74076637 0x46a51dd 0 2 73202772 0x45cfc54 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000046 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076659 0x46a51f3 0 2 73202795 0x45cfc6b 0 3 72732478 0x455cf3e 0 1 71221425 0x43ec0b1 0 4 73313957 0x45eaea5 0 5 72496273 0x4523491 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000018 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202868 0x45cfcb4 0 3 72732550 0x455cf86 0 1 71221498 0x43ec0fa 0 4 73314031 0x45eaeef 0 5 72496347 0x45234db 0 0 74076737 0x46a5241 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000031 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314046 0x45eaefe 0 5 72496362 0x45234ea 0 0 74076751 0x46a524f 0 2 73202886 0x45cfcc6 0 3 72732568 0x455cf98 0 1 71221516 0x43ec10c 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000061 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221479 0x43ec0e7 0 4 73314011 0x45eaedb 0 5 72496327 0x45234c7 0 0 74076718 0x46a522e 0 2 73202852 0x45cfca4 0 3 72732535 0x455cf77 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000009 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313958 0x45eaea6 0 5 72496274 0x4523492 0 0 74076663 0x46a51f7 0 2 73202797 0x45cfc6d 0 3 72732479 0x455cf3f 0 1 71221426 0x43ec0b2 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000034 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202885 0x45cfcc5 0 3 72732567 0x455cf97 0 1 71221515 0x43ec10b 0 4 73314048 0x45eaf00 0 5 72496364 0x45234ec 0 0 74076753 0x46a5251 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000048 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221482 0x43ec0ea 0 4 73314014 0x45eaede 0 5 72496331 0x45234cb 0 0 74076720 0x46a5230 0 2 73202854 0x45cfca6 0 3 72732537 0x455cf79 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000034 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202778 0x45cfc5a 0 3 72732462 0x455cf2e 0 1 71221410 0x43ec0a2 0 4 73313942 0x45eae96 0 5 72496259 0x4523483 0 0 74076648 0x46a51e8 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000058 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076685 0x46a520d 0 2 73202819 0x45cfc83 0 3 72732502 0x455cf56 0 1 71221451 0x43ec0cb 0 4 73313984 0x45eaec0 0 5 72496300 0x45234ac 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000014 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732549 0x455cf85 0 1 71221496 0x43ec0f8 0 4 73314028 0x45eaeec 0 5 72496344 0x45234d8 0 0 74076733 0x46a523d 0 2 73202867 0x45cfcb3 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000000 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076665 0x46a51f9 0 2 73202799 0x45cfc6f 0 3 72732481 0x455cf41 0 1 71221428 0x43ec0b4 0 4 73313960 0x45eaea8 0 5 72496276 0x4523494 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000060 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496359 0x45234e7 0 0 74076748 0x46a524c 0 2 73202882 0x45cfcc2 0 3 72732564 0x455cf94 0 1 71221511 0x43ec107 0 4 73314043 0x45eaefb 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000035 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221437 0x43ec0bd 0 4 73313969 0x45eaeb1 0 5 72496285 0x452349d 0 0 74076674 0x46a5202 0 2 73202808 0x45cfc78 0 3 72732490 0x455cf4a 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000017 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221417 0x43ec0a9 0 4 73313952 0x45eaea0 0 5 72496268 0x452348c 0 0 74076657 0x46a51f1 0 2 73202792 0x45cfc68 0 3 72732474 0x455cf3a 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000008 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496328 0x45234c8 0 0 74076719 0x46a522f 0 2 73202853 0x45cfca5 0 3 72732536 0x455cf78 0 1 71221483 0x43ec0eb 0 4 73314016 0x45eaee0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000028 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202829 0x45cfc8d 0 3 72732512 0x455cf60 0 1 71221461 0x43ec0d5 0 4 73313996 0x45eaecc 0 5 72496314 0x45234ba 0 0 74076704 0x46a5220 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000035 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076715 0x46a522b 0 2 73202850 0x45cfca2 0 3 72732533 0x455cf75 0 1 71221481 0x43ec0e9 0 4 73314013 0x45eaedd 0 5 72496330 0x45234ca 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000037 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221402 0x43ec09a 0 4 73313935 0x45eae8f 0 5 72496252 0x452347c 0 0 74076642 0x46a51e2 0 2 73202776 0x45cfc58 0 3 72732458 0x455cf2a 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000004 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496275 0x4523493 0 0 74076664 0x46a51f8 0 2 73202798 0x45cfc6e 0 3 72732480 0x455cf40 0 1 71221427 0x43ec0b3 0 4 73313959 0x45eaea7 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000018 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313970 0x45eaeb2 0 5 72496286 0x452349e 0 0 74076676 0x46a5204 0 2 73202810 0x45cfc7a 0 3 72732492 0x455cf4c 0 1 71221439 0x43ec0bf 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000055 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732534 0x455cf76 0 1 71221484 0x43ec0ec 0 4 73314017 0x45eaee1 0 5 72496333 0x45234cd 0 0 74076722 0x46a5232 0 2 73202856 0x45cfca8 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000061 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221445 0x43ec0c5 0 4 73313977 0x45eaeb9 0 5 72496293 0x45234a5 0 0 74076684 0x46a520c 0 2 73202817 0x45cfc81 0 3 72732499 0x455cf53 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000020 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202768 0x45cfc50 0 3 72732452 0x455cf24 0 1 71221401 0x43ec099 0 4 73313933 0x45eae8d 0 5 72496250 0x452347a 0 0 74076639 0x46a51df 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000029 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076749 0x46a524d 0 2 73202883 0x45cfcc3 0 3 72732566 0x455cf96 0 1 71221513 0x43ec109 0 4 73314045 0x45eaefd 0 5 72496361 0x45234e9 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000027 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496244 0x4523474 0 0 74076633 0x46a51d9 0 2 73202769 0x45cfc51 0 3 72732451 0x455cf23 0 1 71221399 0x43ec097 0 4 73313932 0x45eae8c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000005 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202766 0x45cfc4e 0 3 72732448 0x455cf20 0 1 71221395 0x43ec093 0 4 73313927 0x45eae87 0 5 72496243 0x4523473 0 0 74076632 0x46a51d8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000049 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202824 0x45cfc88 0 3 72732506 0x455cf5a 0 1 71221453 0x43ec0cd 0 4 73313985 0x45eaec1 0 5 72496301 0x45234ad 0 0 74076690 0x46a5212 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000051 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076675 0x46a5203 0 2 73202809 0x45cfc79 0 3 72732491 0x455cf4b 0 1 71221438 0x43ec0be 0 4 73313971 0x45eaeb3 0 5 72496288 0x45234a0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000033 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496318 0x45234be 0 0 74076708 0x46a5224 0 2 73202843 0x45cfc9b 0 3 72732526 0x455cf6e 0 1 71221472 0x43ec0e0 0 4 73314004 0x45eaed4 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000047 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221507 0x43ec103 0 4 73314040 0x45eaef8 0 5 72496356 0x45234e4 0 0 74076746 0x46a524a 0 2 73202880 0x45cfcc0 0 3 72732562 0x455cf92 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000053 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221414 0x43ec0a6 0 4 73313948 0x45eae9c 0 5 72496265 0x4523489 0 0 74076655 0x46a51ef 0 2 73202791 0x45cfc67 0 3 72732475 0x455cf3b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000022 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202781 0x45cfc5d 0 3 72732464 0x455cf30 0 1 71221412 0x43ec0a4 0 4 73313944 0x45eae98 0 5 72496260 0x4523484 0 0 74076649 0x46a51e9 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000021 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732510 0x455cf5e 0 1 71221459 0x43ec0d3 0 4 73313992 0x45eaec8 0 5 72496308 0x45234b4 0 0 74076697 0x46a5219 0 2 73202834 0x45cfc92 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000014 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221449 0x43ec0c9 0 4 73313982 0x45eaebe 0 5 72496298 0x45234aa 0 0 74076688 0x46a5210 0 2 73202821 0x45cfc85 0 3 72732503 0x455cf57 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000024 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202807 0x45cfc77 0 3 72732489 0x455cf49 0 1 71221436 0x43ec0bc 0 4 73313968 0x45eaeb0 0 5 72496284 0x452349c 0 0 74076673 0x46a5201 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000055 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313946 0x45eae9a 0 5 72496263 0x4523487 0 0 74076652 0x46a51ec 0 2 73202786 0x45cfc62 0 3 72732469 0x455cf35 0 1 71221418 0x43ec0aa 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000008 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076631 0x46a51d7 0 2 73202765 0x45cfc4d 0 3 72732447 0x455cf1f 0 1 71221394 0x43ec092 0 4 73313926 0x45eae86 0 5 72496242 0x4523472 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000007 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496302 0x45234ae 0 0 74076691 0x46a5213 0 2 73202825 0x45cfc89 0 3 72732508 0x455cf5c 0 1 71221457 0x43ec0d1 0 4 73313989 0x45eaec5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000021 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313964 0x45eaeac 0 5 72496280 0x4523498 0 0 74076669 0x46a51fd 0 2 73202803 0x45cfc73 0 3 72732485 0x455cf45 0 1 71221432 0x43ec0b8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000004 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313990 0x45eaec6 0 5 72496311 0x45234b7 0 0 74076701 0x46a521d 0 2 73202835 0x45cfc93 0 3 72732516 0x455cf64 0 1 71221463 0x43ec0d7 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000024 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313987 0x45eaec3 0 5 72496304 0x45234b0 0 0 74076695 0x46a5217 0 2 73202833 0x45cfc91 0 3 72732518 0x455cf66 0 1 71221465 0x43ec0d9 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000007 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202762 0x45cfc4a 0 3 72732444 0x455cf1c 0 1 71221391 0x43ec08f 0 4 73313924 0x45eae84 0 5 72496240 0x4523470 0 0 74076629 0x46a51d5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000042 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313981 0x45eaebd 0 5 72496297 0x45234a9 0 0 74076687 0x46a520f 0 2 73202822 0x45cfc86 0 3 72732504 0x455cf58 0 1 71221452 0x43ec0cc 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000010 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202839 0x45cfc97 0 3 72732521 0x455cf69 0 1 71221468 0x43ec0dc 0 4 73314001 0x45eaed1 0 5 72496317 0x45234bd 0 0 74076706 0x46a5222 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000052 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732500 0x455cf54 0 1 71221448 0x43ec0c8 0 4 73313980 0x45eaebc 0 5 72496296 0x45234a8 0 0 74076686 0x46a520e 0 2 73202820 0x45cfc84 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000023 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221460 0x43ec0d4 0 4 73313994 0x45eaeca 0 5 72496310 0x45234b6 0 0 74076698 0x46a521a 0 2 73202831 0x45cfc8f 0 3 72732514 0x455cf62 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000009 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732565 0x455cf95 0 1 71221512 0x43ec108 0 4 73314044 0x45eaefc 0 5 72496360 0x45234e8 0 0 74076750 0x46a524e 0 2 73202884 0x45cfcc4 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000052 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221499 0x43ec0fb 0 4 73314032 0x45eaef0 0 5 72496349 0x45234dd 0 0 74076738 0x46a5242 0 2 73202872 0x45cfcb8 0 3 72732554 0x455cf8a 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000022 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732543 0x455cf7f 0 1 71221490 0x43ec0f2 0 4 73314022 0x45eaee6 0 5 72496338 0x45234d2 0 0 74076727 0x46a5237 0 2 73202861 0x45cfcad 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000043 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732529 0x455cf71 0 1 71221476 0x43ec0e4 0 4 73314008 0x45eaed8 0 5 72496324 0x45234c4 0 0 74076713 0x46a5229 0 2 73202847 0x45cfc9f 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000003 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221455 0x43ec0cf 0 4 73313988 0x45eaec4 0 5 72496305 0x45234b1 0 0 74076694 0x46a5216 0 2 73202830 0x45cfc8e 0 3 72732513 0x455cf61 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000039 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496313 0x45234b9 0 0 74076702 0x46a521e 0 2 73202838 0x45cfc96 0 3 72732519 0x455cf67 0 1 71221466 0x43ec0da 0 4 73313999 0x45eaecf 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000012 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313928 0x45eae88 0 5 72496245 0x4523475 0 0 74076634 0x46a51da 0 2 73202770 0x45cfc52 0 3 72732453 0x455cf25 0 1 71221403 0x43ec09b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000054 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221443 0x43ec0c3 0 4 73313975 0x45eaeb7 0 5 72496291 0x45234a3 0 0 74076681 0x46a5209 0 2 73202816 0x45cfc80 0 3 72732498 0x455cf52 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000050 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221400 0x43ec098 0 4 73313934 0x45eae8e 0 5 72496251 0x452347b 0 0 74076641 0x46a51e1 0 2 73202774 0x45cfc56 0 3 72732456 0x455cf28 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000019 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202841 0x45cfc99 0 3 72732523 0x455cf6b 0 1 71221471 0x43ec0df 0 4 73314003 0x45eaed3 0 5 72496320 0x45234c0 0 0 74076709 0x46a5225 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000020 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221497 0x43ec0f9 0 4 73314029 0x45eaeed 0 5 72496345 0x45234d9 0 0 74076735 0x46a523f 0 2 73202870 0x45cfcb6 0 3 72732552 0x455cf88 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000006 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732473 0x455cf39 0 1 71221421 0x43ec0ad 0 4 73313954 0x45eaea2 0 5 72496271 0x452348f 0 0 74076661 0x46a51f5 0 2 73202796 0x45cfc6c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000015 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313949 0x45eae9d 0 5 72496266 0x452348a 0 0 74076654 0x46a51ee 0 2 73202788 0x45cfc64 0 3 72732470 0x455cf36 0 1 71221419 0x43ec0ab 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000016 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076682 0x46a520a 0 2 73202818 0x45cfc82 0 3 72732501 0x455cf55 0 1 71221447 0x43ec0c7 0 4 73313979 0x45eaebb 0 5 72496295 0x45234a7 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000027 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732532 0x455cf74 0 1 71221480 0x43ec0e8 0 4 73314012 0x45eaedc 0 5 72496329 0x45234c9 0 0 74076717 0x46a522d 0 2 73202851 0x45cfca3 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000048 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732467 0x455cf33 0 1 71221415 0x43ec0a7 0 4 73313951 0x45eae9f 0 5 72496267 0x452348b 0 0 74076656 0x46a51f0 0 2 73202790 0x45cfc66 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000059 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732459 0x455cf2b 0 1 71221407 0x43ec09f 0 4 73313941 0x45eae95 0 5 72496258 0x4523482 0 0 74076647 0x46a51e7 0 2 73202783 0x45cfc5f 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000042 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314042 0x45eaefa 0 5 72496358 0x45234e6 0 0 74076747 0x46a524b 0 2 73202881 0x45cfcc1 0 3 72732563 0x455cf93 0 1 71221510 0x43ec106 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000025 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314035 0x45eaef3 0 5 72496351 0x45234df 0 0 74076740 0x46a5244 0 2 73202875 0x45cfcbb 0 3 72732556 0x455cf8c 0 1 71221504 0x43ec100 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000028 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221396 0x43ec094 0 4 73313929 0x45eae89 0 5 72496246 0x4523476 0 0 74076635 0x46a51db 0 2 73202771 0x45cfc53 0 3 72732454 0x455cf26 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000023 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221450 0x43ec0ca 0 4 73313983 0x45eaebf 0 5 72496299 0x45234ab 0 0 74076689 0x46a5211 0 2 73202823 0x45cfc87 0 3 72732505 0x455cf59 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000037 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202837 0x45cfc95 0 3 72732520 0x455cf68 0 1 71221467 0x43ec0db 0 4 73314000 0x45eaed0 0 5 72496316 0x45234bc 0 0 74076705 0x46a5221 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000056 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496283 0x452349b 0 0 74076672 0x46a5200 0 2 73202806 0x45cfc76 0 3 72732488 0x455cf48 0 1 71221435 0x43ec0bb 0 4 73313967 0x45eaeaf 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000019 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221406 0x43ec09e 0 4 73313939 0x45eae93 0 5 72496257 0x4523481 0 0 74076646 0x46a51e6 0 2 73202782 0x45cfc5e 0 3 72732465 0x455cf31 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000045 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202793 0x45cfc69 0 3 72732476 0x455cf3c 0 1 71221423 0x43ec0af 0 4 73313955 0x45eaea3 0 5 72496272 0x4523490 0 0 74076662 0x46a51f6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000010 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496253 0x452347d 0 0 74076643 0x46a51e3 0 2 73202777 0x45cfc59 0 3 72732460 0x455cf2c 0 1 71221408 0x43ec0a0 0 4 73313940 0x45eae94 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000063 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496249 0x4523479 0 0 74076638 0x46a51de 0 2 73202773 0x45cfc55 0 3 72732455 0x455cf27 0 1 71221404 0x43ec09c 0 4 73313937 0x45eae91 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000030 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076627 0x46a51d3 0 2 73202761 0x45cfc49 0 3 72732443 0x455cf1b 0 1 71221390 0x43ec08e 0 4 73313922 0x45eae82 0 5 72496238 0x452346e 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000046 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221474 0x43ec0e2 0 4 73314006 0x45eaed6 0 5 72496322 0x45234c2 0 0 74076710 0x46a5226 0 2 73202844 0x45cfc9c 0 3 72732528 0x455cf70 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000011 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314030 0x45eaeee 0 5 72496346 0x45234da 0 0 74076736 0x46a5240 0 2 73202871 0x45cfcb7 0 3 72732553 0x455cf89 0 1 71221501 0x43ec0fd 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000058 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496306 0x45234b2 0 0 74076696 0x46a5218 0 2 73202836 0x45cfc94 0 3 72732517 0x455cf65 0 1 71221464 0x43ec0d8 0 4 73313998 0x45eaece 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000013 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732483 0x455cf43 0 1 71221430 0x43ec0b6 0 4 73313962 0x45eaeaa 0 5 72496278 0x4523496 0 0 74076667 0x46a51fb 0 2 73202801 0x45cfc71 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000056 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076742 0x46a5246 0 2 73202876 0x45cfcbc 0 3 72732558 0x455cf8e 0 1 71221506 0x43ec102 0 4 73314038 0x45eaef6 0 5 72496355 0x45234e3 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000013 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202826 0x45cfc8a 0 3 72732509 0x455cf5d 0 1 71221458 0x43ec0d2 0 4 73313991 0x45eaec7 0 5 72496307 0x45234b3 0 0 74076699 0x46a521b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000032 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202873 0x45cfcb9 0 3 72732555 0x455cf8b 0 1 71221503 0x43ec0ff 0 4 73314036 0x45eaef4 0 5 72496352 0x45234e0 0 0 74076741 0x46a5245 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000033 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496261 0x4523485 0 0 74076650 0x46a51ea 0 2 73202785 0x45cfc61 0 3 72732468 0x455cf34 0 1 71221416 0x43ec0a8 0 4 73313950 0x45eae9e 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000063 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076714 0x46a522a 0 2 73202848 0x45cfca0 0 3 72732530 0x455cf72 0 1 71221477 0x43ec0e5 0 4 73314009 0x45eaed9 0 5 72496325 0x45234c5 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000012 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076734 0x46a523e 0 2 73202869 0x45cfcb5 0 3 72732551 0x455cf87 0 1 71221500 0x43ec0fc 0 4 73314033 0x45eaef1 0 5 72496348 0x45234dc 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000000 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221456 0x43ec0d0 0 4 73313993 0x45eaec9 0 5 72496309 0x45234b5 0 0 74076700 0x46a521c 0 2 73202832 0x45cfc90 0 3 72732515 0x455cf63 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000045 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076693 0x46a5215 0 2 73202828 0x45cfc8c 0 3 72732511 0x455cf5f 0 1 71221462 0x43ec0d6 0 4 73313997 0x45eaecd 0 5 72496315 0x45234bb 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000026 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076745 0x46a5249 0 2 73202879 0x45cfcbf 0 3 72732561 0x455cf91 0 1 71221509 0x43ec105 0 4 73314041 0x45eaef9 0 5 72496357 0x45234e5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000044 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313923 0x45eae83 0 5 72496239 0x452346f 0 0 74076628 0x46a51d4 0 2 73202763 0x45cfc4b 0 3 72732446 0x455cf1e 0 1 71221393 0x43ec091 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000038 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202814 0x45cfc7e 0 3 72732496 0x455cf50 0 1 71221444 0x43ec0c4 0 4 73313976 0x45eaeb8 0 5 72496292 0x45234a4 0 0 74076683 0x46a520b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000041 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221469 0x43ec0dd 0 4 73314002 0x45eaed2 0 5 72496319 0x45234bf 0 0 74076707 0x46a5223 0 2 73202842 0x45cfc9a 0 3 72732525 0x455cf6d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000059 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221502 0x43ec0fe 0 4 73314034 0x45eaef2 0 5 72496350 0x45234de 0 0 74076739 0x46a5243 0 2 73202874 0x45cfcba 0 3 72732557 0x455cf8d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000053 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732527 0x455cf6f 0 1 71221475 0x43ec0e3 0 4 73314007 0x45eaed7 0 5 72496323 0x45234c3 0 0 74076712 0x46a5228 0 2 73202846 0x45cfc9e 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000030 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314015 0x45eaedf 0 5 72496332 0x45234cc 0 0 74076721 0x46a5231 0 2 73202855 0x45cfca7 0 3 72732538 0x455cf7a 0 1 71221485 0x43ec0ed 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000003 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496269 0x452348d 0 0 74076658 0x46a51f2 0 2 73202794 0x45cfc6a 0 3 72732477 0x455cf3d 0 1 71221424 0x43ec0b0 0 4 73313956 0x45eaea4 0 Start IOR test posix_buffered_1m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_buffered_1m [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 22:15:20 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_buffered_1mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.15099/posix_buffered_1m access = file‐per‐process ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 2362.19 2362.19 2362.19 0.00 2362.19 2362.19 2362.19 0.00 221.94972 64 2 1 1 1 1 0 0 1 0 1048576 549755813888 ‐1 POSIX EXCEL Max Write: 2362.19 MiB/sec (2476.94 MB/sec) Perf_Data IOR 1048576 write 2362.193 MiB/sec Run finished: Wed Apr 20 22:19:02 2016 Application 183676 resources: utime ~405s, stime ~159s, Rss ~4744, inblocks ~0, outblocks ~402653184 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_buffered_1m [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 22:19:13 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_buffered_1mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.15099/posix_buffered_1m access = file‐per‐process ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 2480.20 2480.20 2480.20 0.00 2480.20 2480.20 2480.20 0.00 211.38914 64 2 1 1 1 1 0 0 1 0 1048576 549755813888 ‐1 POSIX EXCEL Max Read: 2480.20 MiB/sec (2600.68 MB/sec) Perf_Data IOR 1048576 read 2480.203 MiB/sec Run finished: Wed Apr 20 22:22:44 2016 Application 183677 resources: utime ~341s, stime ~118s, Rss ~4744, inblocks ~0, outblocks ~0 Done IOR posix_buffered_1m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes posix_buffered_1m, down= 3 total 1.6T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000063 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000063 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000063 /lus/husk/dmoen/testdir1.15099 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000047 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732445 0x455cf1d 0 1 71221392 0x43ec090 0 4 73313925 0x45eae85 0 5 72496241 0x4523471 0 0 74076630 0x46a51d6 0 2 73202764 0x45cfc4c 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000012 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496411 0x452351b 0 0 74076800 0x46a5280 0 2 73202935 0x45cfcf7 0 3 72732617 0x455cfc9 0 1 71221565 0x43ec13d 0 4 73314098 0x45eaf32 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000042 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732620 0x455cfcc 0 1 71221567 0x43ec13f 0 4 73314099 0x45eaf33 0 5 72496416 0x4523520 0 0 74076805 0x46a5285 0 2 73202939 0x45cfcfb 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000001 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202767 0x45cfc4f 0 3 72732450 0x455cf22 0 1 71221398 0x43ec096 0 4 73313931 0x45eae8b 0 5 72496248 0x4523478 0 0 74076636 0x46a51dc 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000054 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076796 0x46a527c 0 2 73202930 0x45cfcf2 0 3 72732612 0x455cfc4 0 1 71221559 0x43ec137 0 4 73314091 0x45eaf2b 0 5 72496407 0x4523517 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000016 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314018 0x45eaee2 0 5 72496334 0x45234ce 0 0 74076723 0x46a5233 0 2 73202857 0x45cfca9 0 3 72732539 0x455cf7b 0 1 71221486 0x43ec0ee 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000032 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496287 0x452349f 0 0 74076677 0x46a5205 0 2 73202811 0x45cfc7b 0 3 72732494 0x455cf4e 0 1 71221441 0x43ec0c1 0 4 73313973 0x45eaeb5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000025 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076640 0x46a51e0 0 2 73202775 0x45cfc57 0 3 72732457 0x455cf29 0 1 71221405 0x43ec09d 0 4 73313938 0x45eae92 0 5 72496255 0x452347f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000057 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313936 0x45eae90 0 5 72496254 0x452347e 0 0 74076644 0x46a51e4 0 2 73202779 0x45cfc5b 0 3 72732461 0x455cf2d 0 1 71221409 0x43ec0a1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000026 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202800 0x45cfc70 0 3 72732482 0x455cf42 0 1 71221429 0x43ec0b5 0 4 73313961 0x45eaea9 0 5 72496277 0x4523495 0 0 74076666 0x46a51fa 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000013 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202904 0x45cfcd8 0 3 72732586 0x455cfaa 0 1 71221533 0x43ec11d 0 4 73314065 0x45eaf11 0 5 72496381 0x45234fd 0 0 74076770 0x46a5262 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000011 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076754 0x46a5252 0 2 73202889 0x45cfcc9 0 3 72732571 0x455cf9b 0 1 71221518 0x43ec10e 0 4 73314050 0x45eaf02 0 5 72496366 0x45234ee 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000052 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314096 0x45eaf30 0 5 72496413 0x452351d 0 0 74076802 0x46a5282 0 2 73202937 0x45cfcf9 0 3 72732619 0x455cfcb 0 1 71221566 0x43ec13e 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000040 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202866 0x45cfcb2 0 3 72732548 0x455cf84 0 1 71221495 0x43ec0f7 0 4 73314027 0x45eaeeb 0 5 72496343 0x45234d7 0 0 74076732 0x46a523c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000041 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221431 0x43ec0b7 0 4 73313963 0x45eaeab 0 5 72496279 0x4523497 0 0 74076668 0x46a51fc 0 2 73202802 0x45cfc72 0 3 72732484 0x455cf44 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000005 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221491 0x43ec0f3 0 4 73314023 0x45eaee7 0 5 72496339 0x45234d3 0 0 74076728 0x46a5238 0 2 73202862 0x45cfcae 0 3 72732544 0x455cf80 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000017 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313986 0x45eaec2 0 5 72496303 0x45234af 0 0 74076692 0x46a5214 0 2 73202827 0x45cfc8b 0 3 72732507 0x455cf5b 0 1 71221454 0x43ec0ce 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000033 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732587 0x455cfab 0 1 71221534 0x43ec11e 0 4 73314066 0x45eaf12 0 5 72496382 0x45234fe 0 0 74076771 0x46a5263 0 2 73202906 0x45cfcda 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000059 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314107 0x45eaf3b 0 5 72496423 0x4523527 0 0 74076813 0x46a528d 0 2 73202948 0x45cfd04 0 3 72732630 0x455cfd6 0 1 71221577 0x43ec149 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000031 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202789 0x45cfc65 0 3 72732472 0x455cf38 0 1 71221422 0x43ec0ae 0 4 73313953 0x45eaea1 0 5 72496270 0x452348e 0 0 74076660 0x46a51f4 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000043 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313947 0x45eae9b 0 5 72496264 0x4523488 0 0 74076653 0x46a51ed 0 2 73202787 0x45cfc63 0 3 72732471 0x455cf37 0 1 71221420 0x43ec0ac 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000057 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732524 0x455cf6c 0 1 71221473 0x43ec0e1 0 4 73314005 0x45eaed5 0 5 72496321 0x45234c1 0 0 74076711 0x46a5227 0 2 73202845 0x45cfc9d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000054 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221514 0x43ec10a 0 4 73314047 0x45eaeff 0 5 72496363 0x45234eb 0 0 74076752 0x46a5250 0 2 73202887 0x45cfcc7 0 3 72732569 0x455cf99 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000039 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496256 0x4523480 0 0 74076645 0x46a51e5 0 2 73202780 0x45cfc5c 0 3 72732463 0x455cf2f 0 1 71221411 0x43ec0a3 0 4 73313943 0x45eae97 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000029 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076670 0x46a51fe 0 2 73202804 0x45cfc74 0 3 72732486 0x455cf46 0 1 71221433 0x43ec0b9 0 4 73313965 0x45eaead 0 5 72496281 0x4523499 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000011 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076678 0x46a5206 0 2 73202812 0x45cfc7c 0 3 72732495 0x455cf4f 0 1 71221442 0x43ec0c2 0 4 73313974 0x45eaeb6 0 5 72496290 0x45234a2 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000036 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202919 0x45cfce7 0 3 72732601 0x455cfb9 0 1 71221548 0x43ec12c 0 4 73314080 0x45eaf20 0 5 72496396 0x452350c 0 0 74076785 0x46a5271 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000014 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314082 0x45eaf22 0 5 72496399 0x452350f 0 0 74076788 0x46a5274 0 2 73202922 0x45cfcea 0 3 72732604 0x455cfbc 0 1 71221551 0x43ec12f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000036 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496282 0x452349a 0 0 74076671 0x46a51ff 0 2 73202805 0x45cfc75 0 3 72732487 0x455cf47 0 1 71221434 0x43ec0ba 0 4 73313966 0x45eaeae 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000051 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732531 0x455cf73 0 1 71221478 0x43ec0e6 0 4 73314010 0x45eaeda 0 5 72496326 0x45234c6 0 0 74076716 0x46a522c 0 2 73202849 0x45cfca1 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000005 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314092 0x45eaf2c 0 5 72496408 0x4523518 0 0 74076797 0x46a527d 0 2 73202931 0x45cfcf3 0 3 72732613 0x455cfc5 0 1 71221560 0x43ec138 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000044 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202863 0x45cfcaf 0 3 72732545 0x455cf81 0 1 71221492 0x43ec0f4 0 4 73314025 0x45eaee9 0 5 72496341 0x45234d5 0 0 74076731 0x46a523b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000049 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076680 0x46a5208 0 2 73202815 0x45cfc7f 0 3 72732497 0x455cf51 0 1 71221446 0x43ec0c6 0 4 73313978 0x45eaeba 0 5 72496294 0x45234a6 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000025 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496391 0x4523507 0 0 74076780 0x46a526c 0 2 73202914 0x45cfce2 0 3 72732596 0x455cfb4 0 1 71221544 0x43ec128 0 4 73314076 0x45eaf1c 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000006 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496335 0x45234cf 0 0 74076724 0x46a5234 0 2 73202858 0x45cfcaa 0 3 72732540 0x455cf7c 0 1 71221487 0x43ec0ef 0 4 73314019 0x45eaee3 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000062 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732493 0x455cf4d 0 1 71221440 0x43ec0c0 0 4 73313972 0x45eaeb4 0 5 72496289 0x45234a1 0 0 74076679 0x46a5207 0 2 73202813 0x45cfc7d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000001 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202860 0x45cfcac 0 3 72732542 0x455cf7e 0 1 71221489 0x43ec0f1 0 4 73314021 0x45eaee5 0 5 72496337 0x45234d1 0 0 74076726 0x46a5236 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000002 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076725 0x46a5235 0 2 73202859 0x45cfcab 0 3 72732541 0x455cf7d 0 1 71221488 0x43ec0f0 0 4 73314020 0x45eaee4 0 5 72496336 0x45234d0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000062 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496353 0x45234e1 0 0 74076743 0x46a5247 0 2 73202877 0x45cfcbd 0 3 72732559 0x455cf8f 0 1 71221505 0x43ec101 0 4 73314037 0x45eaef5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000002 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202760 0x45cfc48 0 3 72732442 0x455cf1a 0 1 71221389 0x43ec08d 0 4 73313921 0x45eae81 0 5 72496237 0x452346d 0 0 74076626 0x46a51d2 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000050 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496354 0x45234e2 0 0 74076744 0x46a5248 0 2 73202878 0x45cfcbe 0 3 72732560 0x455cf90 0 1 71221508 0x43ec104 0 4 73314039 0x45eaef7 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000038 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496340 0x45234d4 0 0 74076729 0x46a5239 0 2 73202864 0x45cfcb0 0 3 72732546 0x455cf82 0 1 71221493 0x43ec0f5 0 4 73314024 0x45eaee8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000036 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076730 0x46a523a 0 2 73202865 0x45cfcb1 0 3 72732547 0x455cf83 0 1 71221494 0x43ec0f6 0 4 73314026 0x45eaeea 0 5 72496342 0x45234d6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000060 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202784 0x45cfc60 0 3 72732466 0x455cf32 0 1 71221413 0x43ec0a5 0 4 73313945 0x45eae99 0 5 72496262 0x4523486 0 0 74076651 0x46a51eb 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000015 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313995 0x45eaecb 0 5 72496312 0x45234b8 0 0 74076703 0x46a521f 0 2 73202840 0x45cfc98 0 3 72732522 0x455cf6a 0 1 71221470 0x43ec0de 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000044 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076810 0x46a528a 0 2 73202944 0x45cfd00 0 3 72732628 0x455cfd4 0 1 71221575 0x43ec147 0 4 73314108 0x45eaf3c 0 5 72496424 0x4523528 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000057 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732626 0x455cfd2 0 1 71221574 0x43ec146 0 4 73314106 0x45eaf3a 0 5 72496422 0x4523526 0 0 74076812 0x46a528c 0 2 73202947 0x45cfd03 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000040 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732449 0x455cf21 0 1 71221397 0x43ec095 0 4 73313930 0x45eae8a 0 5 72496247 0x4523477 0 0 74076637 0x46a51dd 0 2 73202772 0x45cfc54 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000046 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076659 0x46a51f3 0 2 73202795 0x45cfc6b 0 3 72732478 0x455cf3e 0 1 71221425 0x43ec0b1 0 4 73313957 0x45eaea5 0 5 72496273 0x4523491 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000018 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202868 0x45cfcb4 0 3 72732550 0x455cf86 0 1 71221498 0x43ec0fa 0 4 73314031 0x45eaeef 0 5 72496347 0x45234db 0 0 74076737 0x46a5241 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000047 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732598 0x455cfb6 0 1 71221545 0x43ec129 0 4 73314077 0x45eaf1d 0 5 72496394 0x452350a 0 0 74076783 0x46a526f 0 2 73202917 0x45cfce5 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000031 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314046 0x45eaefe 0 5 72496362 0x45234ea 0 0 74076751 0x46a524f 0 2 73202886 0x45cfcc6 0 3 72732568 0x455cf98 0 1 71221516 0x43ec10c 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000061 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221479 0x43ec0e7 0 4 73314011 0x45eaedb 0 5 72496327 0x45234c7 0 0 74076718 0x46a522e 0 2 73202852 0x45cfca4 0 3 72732535 0x455cf77 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000009 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313958 0x45eaea6 0 5 72496274 0x4523492 0 0 74076663 0x46a51f7 0 2 73202797 0x45cfc6d 0 3 72732479 0x455cf3f 0 1 71221426 0x43ec0b2 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000053 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314059 0x45eaf0b 0 5 72496375 0x45234f7 0 0 74076764 0x46a525c 0 2 73202898 0x45cfcd2 0 3 72732580 0x455cfa4 0 1 71221528 0x43ec118 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000034 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202885 0x45cfcc5 0 3 72732567 0x455cf97 0 1 71221515 0x43ec10b 0 4 73314048 0x45eaf00 0 5 72496364 0x45234ec 0 0 74076753 0x46a5251 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000048 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221482 0x43ec0ea 0 4 73314014 0x45eaede 0 5 72496331 0x45234cb 0 0 74076720 0x46a5230 0 2 73202854 0x45cfca6 0 3 72732537 0x455cf79 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000024 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202949 0x45cfd05 0 3 72732632 0x455cfd8 0 1 71221579 0x43ec14b 0 4 73314111 0x45eaf3f 0 5 72496426 0x452352a 0 0 74076815 0x46a528f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000034 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202778 0x45cfc5a 0 3 72732462 0x455cf2e 0 1 71221410 0x43ec0a2 0 4 73313942 0x45eae96 0 5 72496259 0x4523483 0 0 74076648 0x46a51e8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000019 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221542 0x43ec126 0 4 73314074 0x45eaf1a 0 5 72496390 0x4523506 0 0 74076779 0x46a526b 0 2 73202913 0x45cfce1 0 3 72732595 0x455cfb3 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000058 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496392 0x4523508 0 0 74076781 0x46a526d 0 2 73202915 0x45cfce3 0 3 72732597 0x455cfb5 0 1 71221543 0x43ec127 0 4 73314075 0x45eaf1b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000034 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202946 0x45cfd02 0 3 72732629 0x455cfd5 0 1 71221576 0x43ec148 0 4 73314109 0x45eaf3d 0 5 72496425 0x4523529 0 0 74076814 0x46a528e 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000003 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202888 0x45cfcc8 0 3 72732570 0x455cf9a 0 1 71221517 0x43ec10d 0 4 73314049 0x45eaf01 0 5 72496365 0x45234ed 0 0 74076755 0x46a5253 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000035 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732599 0x455cfb7 0 1 71221547 0x43ec12b 0 4 73314079 0x45eaf1f 0 5 72496395 0x452350b 0 0 74076784 0x46a5270 0 2 73202918 0x45cfce6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000058 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076685 0x46a520d 0 2 73202819 0x45cfc83 0 3 72732502 0x455cf56 0 1 71221451 0x43ec0cb 0 4 73313984 0x45eaec0 0 5 72496300 0x45234ac 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000014 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732549 0x455cf85 0 1 71221496 0x43ec0f8 0 4 73314028 0x45eaeec 0 5 72496344 0x45234d8 0 0 74076733 0x46a523d 0 2 73202867 0x45cfcb3 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000000 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076665 0x46a51f9 0 2 73202799 0x45cfc6f 0 3 72732481 0x455cf41 0 1 71221428 0x43ec0b4 0 4 73313960 0x45eaea8 0 5 72496276 0x4523494 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000060 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496359 0x45234e7 0 0 74076748 0x46a524c 0 2 73202882 0x45cfcc2 0 3 72732564 0x455cf94 0 1 71221511 0x43ec107 0 4 73314043 0x45eaefb 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000020 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202933 0x45cfcf5 0 3 72732615 0x455cfc7 0 1 71221562 0x43ec13a 0 4 73314094 0x45eaf2e 0 5 72496410 0x452351a 0 0 74076799 0x46a527f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000035 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221437 0x43ec0bd 0 4 73313969 0x45eaeb1 0 5 72496285 0x452349d 0 0 74076674 0x46a5202 0 2 73202808 0x45cfc78 0 3 72732490 0x455cf4a 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000029 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202896 0x45cfcd0 0 3 72732578 0x455cfa2 0 1 71221525 0x43ec115 0 4 73314057 0x45eaf09 0 5 72496373 0x45234f5 0 0 74076762 0x46a525a 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000017 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221417 0x43ec0a9 0 4 73313952 0x45eaea0 0 5 72496268 0x452348c 0 0 74076657 0x46a51f1 0 2 73202792 0x45cfc68 0 3 72732474 0x455cf3a 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000008 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496328 0x45234c8 0 0 74076719 0x46a522f 0 2 73202853 0x45cfca5 0 3 72732536 0x455cf78 0 1 71221483 0x43ec0eb 0 4 73314016 0x45eaee0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000030 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496406 0x4523516 0 0 74076795 0x46a527b 0 2 73202929 0x45cfcf1 0 3 72732611 0x455cfc3 0 1 71221558 0x43ec136 0 4 73314090 0x45eaf2a 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000063 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314068 0x45eaf14 0 5 72496384 0x4523500 0 0 74076773 0x46a5265 0 2 73202908 0x45cfcdc 0 3 72732590 0x455cfae 0 1 71221537 0x43ec121 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000028 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202829 0x45cfc8d 0 3 72732512 0x455cf60 0 1 71221461 0x43ec0d5 0 4 73313996 0x45eaecc 0 5 72496314 0x45234ba 0 0 74076704 0x46a5220 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000041 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496383 0x45234ff 0 0 74076775 0x46a5267 0 2 73202909 0x45cfcdd 0 3 72732593 0x455cfb1 0 1 71221541 0x43ec125 0 4 73314073 0x45eaf19 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000035 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076715 0x46a522b 0 2 73202850 0x45cfca2 0 3 72732533 0x455cf75 0 1 71221481 0x43ec0e9 0 4 73314013 0x45eaedd 0 5 72496330 0x45234ca 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000037 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221402 0x43ec09a 0 4 73313935 0x45eae8f 0 5 72496252 0x452347c 0 0 74076642 0x46a51e2 0 2 73202776 0x45cfc58 0 3 72732458 0x455cf2a 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000004 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496275 0x4523493 0 0 74076664 0x46a51f8 0 2 73202798 0x45cfc6e 0 3 72732480 0x455cf40 0 1 71221427 0x43ec0b3 0 4 73313959 0x45eaea7 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000006 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076808 0x46a5288 0 2 73202942 0x45cfcfe 0 3 72732624 0x455cfd0 0 1 71221571 0x43ec143 0 4 73314103 0x45eaf37 0 5 72496419 0x4523523 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000027 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221520 0x43ec110 0 4 73314052 0x45eaf04 0 5 72496368 0x45234f0 0 0 74076757 0x46a5255 0 2 73202892 0x45cfccc 0 3 72732574 0x455cf9e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000018 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313970 0x45eaeb2 0 5 72496286 0x452349e 0 0 74076676 0x46a5204 0 2 73202810 0x45cfc7a 0 3 72732492 0x455cf4c 0 1 71221439 0x43ec0bf 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000055 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732534 0x455cf76 0 1 71221484 0x43ec0ec 0 4 73314017 0x45eaee1 0 5 72496333 0x45234cd 0 0 74076722 0x46a5232 0 2 73202856 0x45cfca8 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000061 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221445 0x43ec0c5 0 4 73313977 0x45eaeb9 0 5 72496293 0x45234a5 0 0 74076684 0x46a520c 0 2 73202817 0x45cfc81 0 3 72732499 0x455cf53 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000050 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496415 0x452351f 0 0 74076804 0x46a5284 0 2 73202938 0x45cfcfa 0 3 72732621 0x455cfcd 0 1 71221568 0x43ec140 0 4 73314100 0x45eaf34 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000046 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732631 0x455cfd7 0 1 71221578 0x43ec14a 0 4 73314110 0x45eaf3e 0 5 72496427 0x452352b 0 0 74076816 0x46a5290 0 2 73202950 0x45cfd06 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000020 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202768 0x45cfc50 0 3 72732452 0x455cf24 0 1 71221401 0x43ec099 0 4 73313933 0x45eae8d 0 5 72496250 0x452347a 0 0 74076639 0x46a51df 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000007 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202905 0x45cfcd9 0 3 72732589 0x455cfad 0 1 71221535 0x43ec11f 0 4 73314070 0x45eaf16 0 5 72496387 0x4523503 0 0 74076777 0x46a5269 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000015 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202890 0x45cfcca 0 3 72732572 0x455cf9c 0 1 71221521 0x43ec111 0 4 73314053 0x45eaf05 0 5 72496369 0x45234f1 0 0 74076758 0x46a5256 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000023 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076759 0x46a5257 0 2 73202894 0x45cfcce 0 3 72732576 0x455cfa0 0 1 71221523 0x43ec113 0 4 73314056 0x45eaf08 0 5 72496372 0x45234f4 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000029 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076749 0x46a524d 0 2 73202883 0x45cfcc3 0 3 72732566 0x455cf96 0 1 71221513 0x43ec109 0 4 73314045 0x45eaefd 0 5 72496361 0x45234e9 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000027 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496244 0x4523474 0 0 74076633 0x46a51d9 0 2 73202769 0x45cfc51 0 3 72732451 0x455cf23 0 1 71221399 0x43ec097 0 4 73313932 0x45eae8c 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000018 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221573 0x43ec145 0 4 73314105 0x45eaf39 0 5 72496421 0x4523525 0 0 74076811 0x46a528b 0 2 73202945 0x45cfd01 0 3 72732627 0x455cfd3 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000008 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221556 0x43ec134 0 4 73314088 0x45eaf28 0 5 72496404 0x4523514 0 0 74076793 0x46a5279 0 2 73202927 0x45cfcef 0 3 72732609 0x455cfc1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000005 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202766 0x45cfc4e 0 3 72732448 0x455cf20 0 1 71221395 0x43ec093 0 4 73313927 0x45eae87 0 5 72496243 0x4523473 0 0 74076632 0x46a51d8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000039 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732585 0x455cfa9 0 1 71221532 0x43ec11c 0 4 73314064 0x45eaf10 0 5 72496380 0x45234fc 0 0 74076769 0x46a5261 0 2 73202903 0x45cfcd7 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000049 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202824 0x45cfc88 0 3 72732506 0x455cf5a 0 1 71221453 0x43ec0cd 0 4 73313985 0x45eaec1 0 5 72496301 0x45234ad 0 0 74076690 0x46a5212 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000051 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076675 0x46a5203 0 2 73202809 0x45cfc79 0 3 72732491 0x455cf4b 0 1 71221438 0x43ec0be 0 4 73313971 0x45eaeb3 0 5 72496288 0x45234a0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000033 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496318 0x45234be 0 0 74076708 0x46a5224 0 2 73202843 0x45cfc9b 0 3 72732526 0x455cf6e 0 1 71221472 0x43ec0e0 0 4 73314004 0x45eaed4 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000009 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314055 0x45eaf07 0 5 72496371 0x45234f3 0 0 74076761 0x46a5259 0 2 73202895 0x45cfccf 0 3 72732577 0x455cfa1 0 1 71221524 0x43ec114 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000047 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221507 0x43ec103 0 4 73314040 0x45eaef8 0 5 72496356 0x45234e4 0 0 74076746 0x46a524a 0 2 73202880 0x45cfcc0 0 3 72732562 0x455cf92 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000001 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202943 0x45cfcff 0 3 72732625 0x455cfd1 0 1 71221572 0x43ec144 0 4 73314104 0x45eaf38 0 5 72496420 0x4523524 0 0 74076809 0x46a5289 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000053 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221414 0x43ec0a6 0 4 73313948 0x45eae9c 0 5 72496265 0x4523489 0 0 74076655 0x46a51ef 0 2 73202791 0x45cfc67 0 3 72732475 0x455cf3b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000022 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202781 0x45cfc5d 0 3 72732464 0x455cf30 0 1 71221412 0x43ec0a4 0 4 73313944 0x45eae98 0 5 72496260 0x4523484 0 0 74076649 0x46a51e9 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000026 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076790 0x46a5276 0 2 73202925 0x45cfced 0 3 72732607 0x455cfbf 0 1 71221554 0x43ec132 0 4 73314086 0x45eaf26 0 5 72496402 0x4523512 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000056 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314067 0x45eaf13 0 5 72496386 0x4523502 0 0 74076776 0x46a5268 0 2 73202911 0x45cfcdf 0 3 72732594 0x455cfb2 0 1 71221540 0x43ec124 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000031 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732579 0x455cfa3 0 1 71221526 0x43ec116 0 4 73314058 0x45eaf0a 0 5 72496374 0x45234f6 0 0 74076763 0x46a525b 0 2 73202897 0x45cfcd1 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000021 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732510 0x455cf5e 0 1 71221459 0x43ec0d3 0 4 73313992 0x45eaec8 0 5 72496308 0x45234b4 0 0 74076697 0x46a5219 0 2 73202834 0x45cfc92 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000014 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221449 0x43ec0c9 0 4 73313982 0x45eaebe 0 5 72496298 0x45234aa 0 0 74076688 0x46a5210 0 2 73202821 0x45cfc85 0 3 72732503 0x455cf57 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000017 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732606 0x455cfbe 0 1 71221553 0x43ec131 0 4 73314085 0x45eaf25 0 5 72496401 0x4523511 0 0 74076791 0x46a5277 0 2 73202924 0x45cfcec 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000024 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202807 0x45cfc77 0 3 72732489 0x455cf49 0 1 71221436 0x43ec0bc 0 4 73313968 0x45eaeb0 0 5 72496284 0x452349c 0 0 74076673 0x46a5201 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000055 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313946 0x45eae9a 0 5 72496263 0x4523487 0 0 74076652 0x46a51ec 0 2 73202786 0x45cfc62 0 3 72732469 0x455cf35 0 1 71221418 0x43ec0aa 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000008 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076631 0x46a51d7 0 2 73202765 0x45cfc4d 0 3 72732447 0x455cf1f 0 1 71221394 0x43ec092 0 4 73313926 0x45eae86 0 5 72496242 0x4523472 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000007 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496302 0x45234ae 0 0 74076691 0x46a5213 0 2 73202825 0x45cfc89 0 3 72732508 0x455cf5c 0 1 71221457 0x43ec0d1 0 4 73313989 0x45eaec5 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000055 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076766 0x46a525e 0 2 73202900 0x45cfcd4 0 3 72732582 0x455cfa6 0 1 71221529 0x43ec119 0 4 73314061 0x45eaf0d 0 5 72496377 0x45234f9 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000021 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313964 0x45eaeac 0 5 72496280 0x4523498 0 0 74076669 0x46a51fd 0 2 73202803 0x45cfc73 0 3 72732485 0x455cf45 0 1 71221432 0x43ec0b8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000004 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313990 0x45eaec6 0 5 72496311 0x45234b7 0 0 74076701 0x46a521d 0 2 73202835 0x45cfc93 0 3 72732516 0x455cf64 0 1 71221463 0x43ec0d7 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000024 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313987 0x45eaec3 0 5 72496304 0x45234b0 0 0 74076695 0x46a5217 0 2 73202833 0x45cfc91 0 3 72732518 0x455cf66 0 1 71221465 0x43ec0d9 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000007 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202762 0x45cfc4a 0 3 72732444 0x455cf1c 0 1 71221391 0x43ec08f 0 4 73313924 0x45eae84 0 5 72496240 0x4523470 0 0 74076629 0x46a51d5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000042 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313981 0x45eaebd 0 5 72496297 0x45234a9 0 0 74076687 0x46a520f 0 2 73202822 0x45cfc86 0 3 72732504 0x455cf58 0 1 71221452 0x43ec0cc 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000010 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202839 0x45cfc97 0 3 72732521 0x455cf69 0 1 71221468 0x43ec0dc 0 4 73314001 0x45eaed1 0 5 72496317 0x45234bd 0 0 74076706 0x46a5222 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000038 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314093 0x45eaf2d 0 5 72496409 0x4523519 0 0 74076798 0x46a527e 0 2 73202932 0x45cfcf4 0 3 72732614 0x455cfc6 0 1 71221561 0x43ec139 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000052 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732500 0x455cf54 0 1 71221448 0x43ec0c8 0 4 73313980 0x45eaebc 0 5 72496296 0x45234a8 0 0 74076686 0x46a520e 0 2 73202820 0x45cfc84 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000032 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202934 0x45cfcf6 0 3 72732616 0x455cfc8 0 1 71221564 0x43ec13c 0 4 73314097 0x45eaf31 0 5 72496414 0x452351e 0 0 74076803 0x46a5283 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000023 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221460 0x43ec0d4 0 4 73313994 0x45eaeca 0 5 72496310 0x45234b6 0 0 74076698 0x46a521a 0 2 73202831 0x45cfc8f 0 3 72732514 0x455cf62 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000045 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496393 0x4523509 0 0 74076782 0x46a526e 0 2 73202916 0x45cfce4 0 3 72732600 0x455cfb8 0 1 71221546 0x43ec12a 0 4 73314078 0x45eaf1e 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000009 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732565 0x455cf95 0 1 71221512 0x43ec108 0 4 73314044 0x45eaefc 0 5 72496360 0x45234e8 0 0 74076750 0x46a524e 0 2 73202884 0x45cfcc4 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000052 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221499 0x43ec0fb 0 4 73314032 0x45eaef0 0 5 72496349 0x45234dd 0 0 74076738 0x46a5242 0 2 73202872 0x45cfcb8 0 3 72732554 0x455cf8a 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000022 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732543 0x455cf7f 0 1 71221490 0x43ec0f2 0 4 73314022 0x45eaee6 0 5 72496338 0x45234d2 0 0 74076727 0x46a5237 0 2 73202861 0x45cfcad 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000043 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732529 0x455cf71 0 1 71221476 0x43ec0e4 0 4 73314008 0x45eaed8 0 5 72496324 0x45234c4 0 0 74076713 0x46a5229 0 2 73202847 0x45cfc9f 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000004 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221527 0x43ec117 0 4 73314060 0x45eaf0c 0 5 72496376 0x45234f8 0 0 74076765 0x46a525d 0 2 73202899 0x45cfcd3 0 3 72732581 0x455cfa5 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000003 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221455 0x43ec0cf 0 4 73313988 0x45eaec4 0 5 72496305 0x45234b1 0 0 74076694 0x46a5216 0 2 73202830 0x45cfc8e 0 3 72732513 0x455cf61 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000039 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496313 0x45234b9 0 0 74076702 0x46a521e 0 2 73202838 0x45cfc96 0 3 72732519 0x455cf67 0 1 71221466 0x43ec0da 0 4 73313999 0x45eaecf 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000060 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496400 0x4523510 0 0 74076789 0x46a5275 0 2 73202923 0x45cfceb 0 3 72732605 0x455cfbd 0 1 71221552 0x43ec130 0 4 73314084 0x45eaf24 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000012 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313928 0x45eae88 0 5 72496245 0x4523475 0 0 74076634 0x46a51da 0 2 73202770 0x45cfc52 0 3 72732453 0x455cf25 0 1 71221403 0x43ec09b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000049 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732584 0x455cfa8 0 1 71221531 0x43ec11b 0 4 73314063 0x45eaf0f 0 5 72496379 0x45234fb 0 0 74076768 0x46a5260 0 2 73202902 0x45cfcd6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000054 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221443 0x43ec0c3 0 4 73313975 0x45eaeb7 0 5 72496291 0x45234a3 0 0 74076681 0x46a5209 0 2 73202816 0x45cfc80 0 3 72732498 0x455cf52 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000050 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221400 0x43ec098 0 4 73313934 0x45eae8e 0 5 72496251 0x452347b 0 0 74076641 0x46a51e1 0 2 73202774 0x45cfc56 0 3 72732456 0x455cf28 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000010 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496398 0x452350e 0 0 74076787 0x46a5273 0 2 73202921 0x45cfce9 0 3 72732603 0x455cfbb 0 1 71221550 0x43ec12e 0 4 73314083 0x45eaf23 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000019 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202841 0x45cfc99 0 3 72732523 0x455cf6b 0 1 71221471 0x43ec0df 0 4 73314003 0x45eaed3 0 5 72496320 0x45234c0 0 0 74076709 0x46a5225 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000020 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221497 0x43ec0f9 0 4 73314029 0x45eaeed 0 5 72496345 0x45234d9 0 0 74076735 0x46a523f 0 2 73202870 0x45cfcb6 0 3 72732552 0x455cf88 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000006 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732473 0x455cf39 0 1 71221421 0x43ec0ad 0 4 73313954 0x45eaea2 0 5 72496271 0x452348f 0 0 74076661 0x46a51f5 0 2 73202796 0x45cfc6c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000015 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313949 0x45eae9d 0 5 72496266 0x452348a 0 0 74076654 0x46a51ee 0 2 73202788 0x45cfc64 0 3 72732470 0x455cf36 0 1 71221419 0x43ec0ab 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000016 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076682 0x46a520a 0 2 73202818 0x45cfc82 0 3 72732501 0x455cf55 0 1 71221447 0x43ec0c7 0 4 73313979 0x45eaebb 0 5 72496295 0x45234a7 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000037 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221580 0x43ec14c 0 4 73314112 0x45eaf40 0 5 72496428 0x452352c 0 0 74076817 0x46a5291 0 2 73202951 0x45cfd07 0 3 72732633 0x455cfd9 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000027 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732532 0x455cf74 0 1 71221480 0x43ec0e8 0 4 73314012 0x45eaedc 0 5 72496329 0x45234c9 0 0 74076717 0x46a522d 0 2 73202851 0x45cfca3 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000016 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314054 0x45eaf06 0 5 72496370 0x45234f2 0 0 74076760 0x46a5258 0 2 73202893 0x45cfccd 0 3 72732575 0x455cf9f 0 1 71221522 0x43ec112 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000048 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732467 0x455cf33 0 1 71221415 0x43ec0a7 0 4 73313951 0x45eae9f 0 5 72496267 0x452348b 0 0 74076656 0x46a51f0 0 2 73202790 0x45cfc66 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000059 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732459 0x455cf2b 0 1 71221407 0x43ec09f 0 4 73313941 0x45eae95 0 5 72496258 0x4523482 0 0 74076647 0x46a51e7 0 2 73202783 0x45cfc5f 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000042 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314042 0x45eaefa 0 5 72496358 0x45234e6 0 0 74076747 0x46a524b 0 2 73202881 0x45cfcc1 0 3 72732563 0x455cf93 0 1 71221510 0x43ec106 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000025 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314035 0x45eaef3 0 5 72496351 0x45234df 0 0 74076740 0x46a5244 0 2 73202875 0x45cfcbb 0 3 72732556 0x455cf8c 0 1 71221504 0x43ec100 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000022 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732608 0x455cfc0 0 1 71221555 0x43ec133 0 4 73314087 0x45eaf27 0 5 72496403 0x4523513 0 0 74076792 0x46a5278 0 2 73202926 0x45cfcee 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000028 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221396 0x43ec094 0 4 73313929 0x45eae89 0 5 72496246 0x4523476 0 0 74076635 0x46a51db 0 2 73202771 0x45cfc53 0 3 72732454 0x455cf26 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000023 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221450 0x43ec0ca 0 4 73313983 0x45eaebf 0 5 72496299 0x45234ab 0 0 74076689 0x46a5211 0 2 73202823 0x45cfc87 0 3 72732505 0x455cf59 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000037 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202837 0x45cfc95 0 3 72732520 0x455cf68 0 1 71221467 0x43ec0db 0 4 73314000 0x45eaed0 0 5 72496316 0x45234bc 0 0 74076705 0x46a5221 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000056 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496283 0x452349b 0 0 74076672 0x46a5200 0 2 73202806 0x45cfc76 0 3 72732488 0x455cf48 0 1 71221435 0x43ec0bb 0 4 73313967 0x45eaeaf 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000019 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221406 0x43ec09e 0 4 73313939 0x45eae93 0 5 72496257 0x4523481 0 0 74076646 0x46a51e6 0 2 73202782 0x45cfc5e 0 3 72732465 0x455cf31 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000045 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202793 0x45cfc69 0 3 72732476 0x455cf3c 0 1 71221423 0x43ec0af 0 4 73313955 0x45eaea3 0 5 72496272 0x4523490 0 0 74076662 0x46a51f6 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000021 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496378 0x45234fa 0 0 74076767 0x46a525f 0 2 73202901 0x45cfcd5 0 3 72732583 0x455cfa7 0 1 71221530 0x43ec11a 0 4 73314062 0x45eaf0e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000010 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496253 0x452347d 0 0 74076643 0x46a51e3 0 2 73202777 0x45cfc59 0 3 72732460 0x455cf2c 0 1 71221408 0x43ec0a0 0 4 73313940 0x45eae94 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000063 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496249 0x4523479 0 0 74076638 0x46a51de 0 2 73202773 0x45cfc55 0 3 72732455 0x455cf27 0 1 71221404 0x43ec09c 0 4 73313937 0x45eae91 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000030 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076627 0x46a51d3 0 2 73202761 0x45cfc49 0 3 72732443 0x455cf1b 0 1 71221390 0x43ec08e 0 4 73313922 0x45eae82 0 5 72496238 0x452346e 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000028 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314089 0x45eaf29 0 5 72496405 0x4523515 0 0 74076794 0x46a527a 0 2 73202928 0x45cfcf0 0 3 72732610 0x455cfc2 0 1 71221557 0x43ec135 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000046 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221474 0x43ec0e2 0 4 73314006 0x45eaed6 0 5 72496322 0x45234c2 0 0 74076710 0x46a5226 0 2 73202844 0x45cfc9c 0 3 72732528 0x455cf70 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000011 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314030 0x45eaeee 0 5 72496346 0x45234da 0 0 74076736 0x46a5240 0 2 73202871 0x45cfcb7 0 3 72732553 0x455cf89 0 1 71221501 0x43ec0fd 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000061 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221519 0x43ec10f 0 4 73314051 0x45eaf03 0 5 72496367 0x45234ef 0 0 74076756 0x46a5254 0 2 73202891 0x45cfccb 0 3 72732573 0x455cf9d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000058 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496306 0x45234b2 0 0 74076696 0x46a5218 0 2 73202836 0x45cfc94 0 3 72732517 0x455cf65 0 1 71221464 0x43ec0d8 0 4 73313998 0x45eaece 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000051 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076772 0x46a5264 0 2 73202910 0x45cfcde 0 3 72732591 0x455cfaf 0 1 71221539 0x43ec123 0 4 73314072 0x45eaf18 0 5 72496389 0x4523505 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000013 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732483 0x455cf43 0 1 71221430 0x43ec0b6 0 4 73313962 0x45eaeaa 0 5 72496278 0x4523496 0 0 74076667 0x46a51fb 0 2 73202801 0x45cfc71 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000040 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076806 0x46a5286 0 2 73202940 0x45cfcfc 0 3 72732622 0x455cfce 0 1 71221570 0x43ec142 0 4 73314102 0x45eaf36 0 5 72496418 0x4523522 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000056 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076742 0x46a5246 0 2 73202876 0x45cfcbc 0 3 72732558 0x455cf8e 0 1 71221506 0x43ec102 0 4 73314038 0x45eaef6 0 5 72496355 0x45234e3 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000013 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202826 0x45cfc8a 0 3 72732509 0x455cf5d 0 1 71221458 0x43ec0d2 0 4 73313991 0x45eaec7 0 5 72496307 0x45234b3 0 0 74076699 0x46a521b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000032 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202873 0x45cfcb9 0 3 72732555 0x455cf8b 0 1 71221503 0x43ec0ff 0 4 73314036 0x45eaef4 0 5 72496352 0x45234e0 0 0 74076741 0x46a5245 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000033 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496261 0x4523485 0 0 74076650 0x46a51ea 0 2 73202785 0x45cfc61 0 3 72732468 0x455cf34 0 1 71221416 0x43ec0a8 0 4 73313950 0x45eae9e 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000063 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076714 0x46a522a 0 2 73202848 0x45cfca0 0 3 72732530 0x455cf72 0 1 71221477 0x43ec0e5 0 4 73314009 0x45eaed9 0 5 72496325 0x45234c5 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000000 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732588 0x455cfac 0 1 71221536 0x43ec120 0 4 73314071 0x45eaf17 0 5 72496388 0x4523504 0 0 74076778 0x46a526a 0 2 73202912 0x45cfce0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000043 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314069 0x45eaf15 0 5 72496385 0x4523501 0 0 74076774 0x46a5266 0 2 73202907 0x45cfcdb 0 3 72732592 0x455cfb0 0 1 71221538 0x43ec122 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000012 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076734 0x46a523e 0 2 73202869 0x45cfcb5 0 3 72732551 0x455cf87 0 1 71221500 0x43ec0fc 0 4 73314033 0x45eaef1 0 5 72496348 0x45234dc 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000002 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732602 0x455cfba 0 1 71221549 0x43ec12d 0 4 73314081 0x45eaf21 0 5 72496397 0x452350d 0 0 74076786 0x46a5272 0 2 73202920 0x45cfce8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000000 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221456 0x43ec0d0 0 4 73313993 0x45eaec9 0 5 72496309 0x45234b5 0 0 74076700 0x46a521c 0 2 73202832 0x45cfc90 0 3 72732515 0x455cf63 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000045 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076693 0x46a5215 0 2 73202828 0x45cfc8c 0 3 72732511 0x455cf5f 0 1 71221462 0x43ec0d6 0 4 73313997 0x45eaecd 0 5 72496315 0x45234bb 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000062 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221569 0x43ec141 0 4 73314101 0x45eaf35 0 5 72496417 0x4523521 0 0 74076807 0x46a5287 0 2 73202941 0x45cfcfd 0 3 72732623 0x455cfcf 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000026 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076745 0x46a5249 0 2 73202879 0x45cfcbf 0 3 72732561 0x455cf91 0 1 71221509 0x43ec105 0 4 73314041 0x45eaef9 0 5 72496357 0x45234e5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000044 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313923 0x45eae83 0 5 72496239 0x452346f 0 0 74076628 0x46a51d4 0 2 73202763 0x45cfc4b 0 3 72732446 0x455cf1e 0 1 71221393 0x43ec091 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000038 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202814 0x45cfc7e 0 3 72732496 0x455cf50 0 1 71221444 0x43ec0c4 0 4 73313976 0x45eaeb8 0 5 72496292 0x45234a4 0 0 74076683 0x46a520b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000048 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221563 0x43ec13b 0 4 73314095 0x45eaf2f 0 5 72496412 0x452351c 0 0 74076801 0x46a5281 0 2 73202936 0x45cfcf8 0 3 72732618 0x455cfca 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000041 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221469 0x43ec0dd 0 4 73314002 0x45eaed2 0 5 72496319 0x45234bf 0 0 74076707 0x46a5223 0 2 73202842 0x45cfc9a 0 3 72732525 0x455cf6d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000059 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221502 0x43ec0fe 0 4 73314034 0x45eaef2 0 5 72496350 0x45234de 0 0 74076739 0x46a5243 0 2 73202874 0x45cfcba 0 3 72732557 0x455cf8d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000053 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732527 0x455cf6f 0 1 71221475 0x43ec0e3 0 4 73314007 0x45eaed7 0 5 72496323 0x45234c3 0 0 74076712 0x46a5228 0 2 73202846 0x45cfc9e 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000030 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314015 0x45eaedf 0 5 72496332 0x45234cc 0 0 74076721 0x46a5231 0 2 73202855 0x45cfca7 0 3 72732538 0x455cf7a 0 1 71221485 0x43ec0ed 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000003 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496269 0x452348d 0 0 74076658 0x46a51f2 0 2 73202794 0x45cfc6a 0 3 72732477 0x455cf3d 0 1 71221424 0x43ec0b0 0 4 73313956 0x45eaea4 0 Start IOR test posix_directio_1m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐B ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_directio_1m [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 22:22:56 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐B ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_directio_1mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.15099/posix_directio_1m access = file‐per‐process ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 1551.68 1551.68 1551.68 0.00 1551.68 1551.68 1551.68 0.00 337.88450 64 2 1 1 1 1 0 0 1 0 1048576 549755813888 ‐1 POSIX EXCEL Max Write: 1551.68 MiB/sec (1627.05 MB/sec) Perf_Data IOR 1048576 write 1551.678 MiB/sec Run finished: Wed Apr 20 22:28:34 2016 Application 183678 resources: utime ~26s, stime ~49s, Rss ~4744, inblocks ~0, outblocks ~0 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐B ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_directio_1m [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 22:28:42 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐B ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_directio_1mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.15099/posix_directio_1m access = file‐per‐process ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 1824.85 1824.85 1824.85 0.00 1824.85 1824.85 1824.85 0.00 287.30452 64 2 1 1 1 1 0 0 1 0 1048576 549755813888 ‐1 POSIX EXCEL Max Read: 1824.85 MiB/sec (1913.50 MB/sec) Perf_Data IOR 1048576 read 1824.851 MiB/sec Run finished: Wed Apr 20 22:33:29 2016 Application 183679 resources: utime ~501s, stime ~53s, Rss ~4744, inblocks ~0, outblocks ~0 Done IOR posix_directio_1m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes posix_directio_1m, down= 3 total 2.1T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000063 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000063 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000063 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000063 /lus/husk/dmoen/testdir1.15099 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000047 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732445 0x455cf1d 0 1 71221392 0x43ec090 0 4 73313925 0x45eae85 0 5 72496241 0x4523471 0 0 74076630 0x46a51d6 0 2 73202764 0x45cfc4c 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000012 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496411 0x452351b 0 0 74076800 0x46a5280 0 2 73202935 0x45cfcf7 0 3 72732617 0x455cfc9 0 1 71221565 0x43ec13d 0 4 73314098 0x45eaf32 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000042 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732620 0x455cfcc 0 1 71221567 0x43ec13f 0 4 73314099 0x45eaf33 0 5 72496416 0x4523520 0 0 74076805 0x46a5285 0 2 73202939 0x45cfcfb 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000001 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202767 0x45cfc4f 0 3 72732450 0x455cf22 0 1 71221398 0x43ec096 0 4 73313931 0x45eae8b 0 5 72496248 0x4523478 0 0 74076636 0x46a51dc 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000054 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076796 0x46a527c 0 2 73202930 0x45cfcf2 0 3 72732612 0x455cfc4 0 1 71221559 0x43ec137 0 4 73314091 0x45eaf2b 0 5 72496407 0x4523517 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000060 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221640 0x43ec188 0 4 73314173 0x45eaf7d 0 5 72496490 0x452356a 0 0 74076880 0x46a52d0 0 2 73203015 0x45cfd47 0 3 72732697 0x455d019 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000016 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314018 0x45eaee2 0 5 72496334 0x45234ce 0 0 74076723 0x46a5233 0 2 73202857 0x45cfca9 0 3 72732539 0x455cf7b 0 1 71221486 0x43ec0ee 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000032 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496287 0x452349f 0 0 74076677 0x46a5205 0 2 73202811 0x45cfc7b 0 3 72732494 0x455cf4e 0 1 71221441 0x43ec0c1 0 4 73313973 0x45eaeb5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000037 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496457 0x4523549 0 0 74076846 0x46a52ae 0 2 73202980 0x45cfd24 0 3 72732662 0x455cff6 0 1 71221609 0x43ec169 0 4 73314141 0x45eaf5d 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000013 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076869 0x46a52c5 0 2 73203003 0x45cfd3b 0 3 72732685 0x455d00d 0 1 71221632 0x43ec180 0 4 73314164 0x45eaf74 0 5 72496480 0x4523560 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000025 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076640 0x46a51e0 0 2 73202775 0x45cfc57 0 3 72732457 0x455cf29 0 1 71221405 0x43ec09d 0 4 73313938 0x45eae92 0 5 72496255 0x452347f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000057 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313936 0x45eae90 0 5 72496254 0x452347e 0 0 74076644 0x46a51e4 0 2 73202779 0x45cfc5b 0 3 72732461 0x455cf2d 0 1 71221409 0x43ec0a1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000026 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202800 0x45cfc70 0 3 72732482 0x455cf42 0 1 71221429 0x43ec0b5 0 4 73313961 0x45eaea9 0 5 72496277 0x4523495 0 0 74076666 0x46a51fa 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000013 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202904 0x45cfcd8 0 3 72732586 0x455cfaa 0 1 71221533 0x43ec11d 0 4 73314065 0x45eaf11 0 5 72496381 0x45234fd 0 0 74076770 0x46a5262 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000011 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076754 0x46a5252 0 2 73202889 0x45cfcc9 0 3 72732571 0x455cf9b 0 1 71221518 0x43ec10e 0 4 73314050 0x45eaf02 0 5 72496366 0x45234ee 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000028 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076819 0x46a5293 0 2 73202954 0x45cfd0a 0 3 72732637 0x455cfdd 0 1 71221583 0x43ec14f 0 4 73314115 0x45eaf43 0 5 72496431 0x452352f 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000052 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314096 0x45eaf30 0 5 72496413 0x452351d 0 0 74076802 0x46a5282 0 2 73202937 0x45cfcf9 0 3 72732619 0x455cfcb 0 1 71221566 0x43ec13e 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000040 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202866 0x45cfcb2 0 3 72732548 0x455cf84 0 1 71221495 0x43ec0f7 0 4 73314027 0x45eaeeb 0 5 72496343 0x45234d7 0 0 74076732 0x46a523c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000041 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221431 0x43ec0b7 0 4 73313963 0x45eaeab 0 5 72496279 0x4523497 0 0 74076668 0x46a51fc 0 2 73202802 0x45cfc72 0 3 72732484 0x455cf44 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000012 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496449 0x4523541 0 0 74076838 0x46a52a6 0 2 73202972 0x45cfd1c 0 3 72732654 0x455cfee 0 1 71221601 0x43ec161 0 4 73314134 0x45eaf56 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000005 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221491 0x43ec0f3 0 4 73314023 0x45eaee7 0 5 72496339 0x45234d3 0 0 74076728 0x46a5238 0 2 73202862 0x45cfcae 0 3 72732544 0x455cf80 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000001 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202985 0x45cfd29 0 3 72732668 0x455cffc 0 1 71221615 0x43ec16f 0 4 73314148 0x45eaf64 0 5 72496464 0x4523550 0 0 74076853 0x46a52b5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000000 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076820 0x46a5294 0 2 73202953 0x45cfd09 0 3 72732636 0x455cfdc 0 1 71221584 0x43ec150 0 4 73314116 0x45eaf44 0 5 72496432 0x4523530 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000051 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221611 0x43ec16b 0 4 73314143 0x45eaf5f 0 5 72496459 0x452354b 0 0 74076848 0x46a52b0 0 2 73202982 0x45cfd26 0 3 72732664 0x455cff8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000017 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313986 0x45eaec2 0 5 72496303 0x45234af 0 0 74076692 0x46a5214 0 2 73202827 0x45cfc8b 0 3 72732507 0x455cf5b 0 1 71221454 0x43ec0ce 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000033 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732587 0x455cfab 0 1 71221534 0x43ec11e 0 4 73314066 0x45eaf12 0 5 72496382 0x45234fe 0 0 74076771 0x46a5263 0 2 73202906 0x45cfcda 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000057 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314165 0x45eaf75 0 5 72496481 0x4523561 0 0 74076870 0x46a52c6 0 2 73203005 0x45cfd3d 0 3 72732686 0x455d00e 0 1 71221634 0x43ec182 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000059 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314107 0x45eaf3b 0 5 72496423 0x4523527 0 0 74076813 0x46a528d 0 2 73202948 0x45cfd04 0 3 72732630 0x455cfd6 0 1 71221577 0x43ec149 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000031 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202789 0x45cfc65 0 3 72732472 0x455cf38 0 1 71221422 0x43ec0ae 0 4 73313953 0x45eaea1 0 5 72496270 0x452348e 0 0 74076660 0x46a51f4 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000043 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313947 0x45eae9b 0 5 72496264 0x4523488 0 0 74076653 0x46a51ed 0 2 73202787 0x45cfc63 0 3 72732471 0x455cf37 0 1 71221420 0x43ec0ac 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000005 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496442 0x452353a 0 0 74076831 0x46a529f 0 2 73202965 0x45cfd15 0 3 72732647 0x455cfe7 0 1 71221594 0x43ec15a 0 4 73314126 0x45eaf4e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000003 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496444 0x452353c 0 0 74076834 0x46a52a2 0 2 73202968 0x45cfd18 0 3 72732650 0x455cfea 0 1 71221597 0x43ec15d 0 4 73314129 0x45eaf51 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000057 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732524 0x455cf6c 0 1 71221473 0x43ec0e1 0 4 73314005 0x45eaed5 0 5 72496321 0x45234c1 0 0 74076711 0x46a5227 0 2 73202845 0x45cfc9d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000054 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221514 0x43ec10a 0 4 73314047 0x45eaeff 0 5 72496363 0x45234eb 0 0 74076752 0x46a5250 0 2 73202887 0x45cfcc7 0 3 72732569 0x455cf99 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000034 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221589 0x43ec155 0 4 73314122 0x45eaf4a 0 5 72496438 0x4523536 0 0 74076829 0x46a529d 0 2 73202963 0x45cfd13 0 3 72732645 0x455cfe5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000039 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496256 0x4523480 0 0 74076645 0x46a51e5 0 2 73202780 0x45cfc5c 0 3 72732463 0x455cf2f 0 1 71221411 0x43ec0a3 0 4 73313943 0x45eae97 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000029 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076670 0x46a51fe 0 2 73202804 0x45cfc74 0 3 72732486 0x455cf46 0 1 71221433 0x43ec0b9 0 4 73313965 0x45eaead 0 5 72496281 0x4523499 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000011 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076678 0x46a5206 0 2 73202812 0x45cfc7c 0 3 72732495 0x455cf4f 0 1 71221442 0x43ec0c2 0 4 73313974 0x45eaeb6 0 5 72496290 0x45234a2 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000036 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202919 0x45cfce7 0 3 72732601 0x455cfb9 0 1 71221548 0x43ec12c 0 4 73314080 0x45eaf20 0 5 72496396 0x452350c 0 0 74076785 0x46a5271 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000014 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314082 0x45eaf22 0 5 72496399 0x452350f 0 0 74076788 0x46a5274 0 2 73202922 0x45cfcea 0 3 72732604 0x455cfbc 0 1 71221551 0x43ec12f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000024 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314113 0x45eaf41 0 5 72496429 0x452352d 0 0 74076818 0x46a5292 0 2 73202952 0x45cfd08 0 3 72732634 0x455cfda 0 1 71221581 0x43ec14d 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000036 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496282 0x452349a 0 0 74076671 0x46a51ff 0 2 73202805 0x45cfc75 0 3 72732487 0x455cf47 0 1 71221434 0x43ec0ba 0 4 73313966 0x45eaeae 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000051 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732531 0x455cf73 0 1 71221478 0x43ec0e6 0 4 73314010 0x45eaeda 0 5 72496326 0x45234c6 0 0 74076716 0x46a522c 0 2 73202849 0x45cfca1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000059 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076849 0x46a52b1 0 2 73202983 0x45cfd27 0 3 72732665 0x455cff9 0 1 71221612 0x43ec16c 0 4 73314144 0x45eaf60 0 5 72496460 0x452354c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000007 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496430 0x452352e 0 0 74076821 0x46a5295 0 2 73202955 0x45cfd0b 0 3 72732635 0x455cfdb 0 1 71221582 0x43ec14e 0 4 73314114 0x45eaf42 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000005 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314092 0x45eaf2c 0 5 72496408 0x4523518 0 0 74076797 0x46a527d 0 2 73202931 0x45cfcf3 0 3 72732613 0x455cfc5 0 1 71221560 0x43ec138 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000044 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202863 0x45cfcaf 0 3 72732545 0x455cf81 0 1 71221492 0x43ec0f4 0 4 73314025 0x45eaee9 0 5 72496341 0x45234d5 0 0 74076731 0x46a523b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000020 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076827 0x46a529b 0 2 73202961 0x45cfd11 0 3 72732643 0x455cfe3 0 1 71221592 0x43ec158 0 4 73314124 0x45eaf4c 0 5 72496440 0x4523538 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000049 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076680 0x46a5208 0 2 73202815 0x45cfc7f 0 3 72732497 0x455cf51 0 1 71221446 0x43ec0c6 0 4 73313978 0x45eaeba 0 5 72496294 0x45234a6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000054 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732691 0x455d013 0 1 71221639 0x43ec187 0 4 73314172 0x45eaf7c 0 5 72496489 0x4523569 0 0 74076878 0x46a52ce 0 2 73203013 0x45cfd45 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000025 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496391 0x4523507 0 0 74076780 0x46a526c 0 2 73202914 0x45cfce2 0 3 72732596 0x455cfb4 0 1 71221544 0x43ec128 0 4 73314076 0x45eaf1c 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000006 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496335 0x45234cf 0 0 74076724 0x46a5234 0 2 73202858 0x45cfcaa 0 3 72732540 0x455cf7c 0 1 71221487 0x43ec0ef 0 4 73314019 0x45eaee3 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000056 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221602 0x43ec162 0 4 73314137 0x45eaf59 0 5 72496453 0x4523545 0 0 74076842 0x46a52aa 0 2 73202978 0x45cfd22 0 3 72732660 0x455cff4 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000062 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732493 0x455cf4d 0 1 71221440 0x43ec0c0 0 4 73313972 0x45eaeb4 0 5 72496289 0x45234a1 0 0 74076679 0x46a5207 0 2 73202813 0x45cfc7d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000001 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202860 0x45cfcac 0 3 72732542 0x455cf7e 0 1 71221489 0x43ec0f1 0 4 73314021 0x45eaee5 0 5 72496337 0x45234d1 0 0 74076726 0x46a5236 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000002 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076725 0x46a5235 0 2 73202859 0x45cfcab 0 3 72732541 0x455cf7d 0 1 71221488 0x43ec0f0 0 4 73314020 0x45eaee4 0 5 72496336 0x45234d0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000062 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496353 0x45234e1 0 0 74076743 0x46a5247 0 2 73202877 0x45cfcbd 0 3 72732559 0x455cf8f 0 1 71221505 0x43ec101 0 4 73314037 0x45eaef5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000002 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202760 0x45cfc48 0 3 72732442 0x455cf1a 0 1 71221389 0x43ec08d 0 4 73313921 0x45eae81 0 5 72496237 0x452346d 0 0 74076626 0x46a51d2 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000046 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221623 0x43ec177 0 4 73314155 0x45eaf6b 0 5 72496471 0x4523557 0 0 74076861 0x46a52bd 0 2 73202995 0x45cfd33 0 3 72732677 0x455d005 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000009 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221585 0x43ec151 0 4 73314117 0x45eaf45 0 5 72496433 0x4523531 0 0 74076822 0x46a5296 0 2 73202956 0x45cfd0c 0 3 72732638 0x455cfde 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000019 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076835 0x46a52a3 0 2 73202969 0x45cfd19 0 3 72732652 0x455cfec 0 1 71221599 0x43ec15f 0 4 73314131 0x45eaf53 0 5 72496447 0x452353f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000058 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314170 0x45eaf7a 0 5 72496486 0x4523566 0 0 74076877 0x46a52cd 0 2 73203012 0x45cfd44 0 3 72732695 0x455d017 0 1 71221641 0x43ec189 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000050 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496354 0x45234e2 0 0 74076744 0x46a5248 0 2 73202878 0x45cfcbe 0 3 72732560 0x455cf90 0 1 71221508 0x43ec104 0 4 73314039 0x45eaef7 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000017 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221586 0x43ec152 0 4 73314118 0x45eaf46 0 5 72496434 0x4523532 0 0 74076823 0x46a5297 0 2 73202957 0x45cfd0d 0 3 72732639 0x455cfdf 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000038 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496340 0x45234d4 0 0 74076729 0x46a5239 0 2 73202864 0x45cfcb0 0 3 72732546 0x455cf82 0 1 71221493 0x43ec0f5 0 4 73314024 0x45eaee8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000036 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076730 0x46a523a 0 2 73202865 0x45cfcb1 0 3 72732547 0x455cf83 0 1 71221494 0x43ec0f6 0 4 73314026 0x45eaeea 0 5 72496342 0x45234d6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000060 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202784 0x45cfc60 0 3 72732466 0x455cf32 0 1 71221413 0x43ec0a5 0 4 73313945 0x45eae99 0 5 72496262 0x4523486 0 0 74076651 0x46a51eb 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000015 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313995 0x45eaecb 0 5 72496312 0x45234b8 0 0 74076703 0x46a521f 0 2 73202840 0x45cfc98 0 3 72732522 0x455cf6a 0 1 71221470 0x43ec0de 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000044 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076810 0x46a528a 0 2 73202944 0x45cfd00 0 3 72732628 0x455cfd4 0 1 71221575 0x43ec147 0 4 73314108 0x45eaf3c 0 5 72496424 0x4523528 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000045 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202974 0x45cfd1e 0 3 72732657 0x455cff1 0 1 71221606 0x43ec166 0 4 73314138 0x45eaf5a 0 5 72496455 0x4523547 0 0 74076844 0x46a52ac 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000057 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732626 0x455cfd2 0 1 71221574 0x43ec146 0 4 73314106 0x45eaf3a 0 5 72496422 0x4523526 0 0 74076812 0x46a528c 0 2 73202947 0x45cfd03 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000049 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314166 0x45eaf76 0 5 72496482 0x4523562 0 0 74076871 0x46a52c7 0 2 73203006 0x45cfd3e 0 3 72732689 0x455d011 0 1 71221637 0x43ec185 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000040 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732449 0x455cf21 0 1 71221397 0x43ec095 0 4 73313930 0x45eae8a 0 5 72496247 0x4523477 0 0 74076637 0x46a51dd 0 2 73202772 0x45cfc54 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000031 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732651 0x455cfeb 0 1 71221598 0x43ec15e 0 4 73314130 0x45eaf52 0 5 72496446 0x452353e 0 0 74076836 0x46a52a4 0 2 73202970 0x45cfd1a 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000046 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076659 0x46a51f3 0 2 73202795 0x45cfc6b 0 3 72732478 0x455cf3e 0 1 71221425 0x43ec0b1 0 4 73313957 0x45eaea5 0 5 72496273 0x4523491 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000018 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202868 0x45cfcb4 0 3 72732550 0x455cf86 0 1 71221498 0x43ec0fa 0 4 73314031 0x45eaeef 0 5 72496347 0x45234db 0 0 74076737 0x46a5241 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000047 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732598 0x455cfb6 0 1 71221545 0x43ec129 0 4 73314077 0x45eaf1d 0 5 72496394 0x452350a 0 0 74076783 0x46a526f 0 2 73202917 0x45cfce5 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000031 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314046 0x45eaefe 0 5 72496362 0x45234ea 0 0 74076751 0x46a524f 0 2 73202886 0x45cfcc6 0 3 72732568 0x455cf98 0 1 71221516 0x43ec10c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000040 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732688 0x455d010 0 1 71221636 0x43ec184 0 4 73314169 0x45eaf79 0 5 72496485 0x4523565 0 0 74076874 0x46a52ca 0 2 73203008 0x45cfd40 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000061 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221479 0x43ec0e7 0 4 73314011 0x45eaedb 0 5 72496327 0x45234c7 0 0 74076718 0x46a522e 0 2 73202852 0x45cfca4 0 3 72732535 0x455cf77 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000009 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313958 0x45eaea6 0 5 72496274 0x4523492 0 0 74076663 0x46a51f7 0 2 73202797 0x45cfc6d 0 3 72732479 0x455cf3f 0 1 71221426 0x43ec0b2 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000029 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076856 0x46a52b8 0 2 73202990 0x45cfd2e 0 3 72732672 0x455d000 0 1 71221619 0x43ec173 0 4 73314152 0x45eaf68 0 5 72496468 0x4523554 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000053 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314059 0x45eaf0b 0 5 72496375 0x45234f7 0 0 74076764 0x46a525c 0 2 73202898 0x45cfcd2 0 3 72732580 0x455cfa4 0 1 71221528 0x43ec118 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000034 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202885 0x45cfcc5 0 3 72732567 0x455cf97 0 1 71221515 0x43ec10b 0 4 73314048 0x45eaf00 0 5 72496364 0x45234ec 0 0 74076753 0x46a5251 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000048 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221482 0x43ec0ea 0 4 73314014 0x45eaede 0 5 72496331 0x45234cb 0 0 74076720 0x46a5230 0 2 73202854 0x45cfca6 0 3 72732537 0x455cf79 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000024 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202949 0x45cfd05 0 3 72732632 0x455cfd8 0 1 71221579 0x43ec14b 0 4 73314111 0x45eaf3f 0 5 72496426 0x452352a 0 0 74076815 0x46a528f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000034 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202778 0x45cfc5a 0 3 72732462 0x455cf2e 0 1 71221410 0x43ec0a2 0 4 73313942 0x45eae96 0 5 72496259 0x4523483 0 0 74076648 0x46a51e8 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000010 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202993 0x45cfd31 0 3 72732675 0x455d003 0 1 71221622 0x43ec176 0 4 73314154 0x45eaf6a 0 5 72496470 0x4523556 0 0 74076859 0x46a52bb 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000019 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221542 0x43ec126 0 4 73314074 0x45eaf1a 0 5 72496390 0x4523506 0 0 74076779 0x46a526b 0 2 73202913 0x45cfce1 0 3 72732595 0x455cfb3 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000058 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496392 0x4523508 0 0 74076781 0x46a526d 0 2 73202915 0x45cfce3 0 3 72732597 0x455cfb5 0 1 71221543 0x43ec127 0 4 73314075 0x45eaf1b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000034 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202946 0x45cfd02 0 3 72732629 0x455cfd5 0 1 71221576 0x43ec148 0 4 73314109 0x45eaf3d 0 5 72496425 0x4523529 0 0 74076814 0x46a528e 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000003 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202888 0x45cfcc8 0 3 72732570 0x455cf9a 0 1 71221517 0x43ec10d 0 4 73314049 0x45eaf01 0 5 72496365 0x45234ed 0 0 74076755 0x46a5253 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000035 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732599 0x455cfb7 0 1 71221547 0x43ec12b 0 4 73314079 0x45eaf1f 0 5 72496395 0x452350b 0 0 74076784 0x46a5270 0 2 73202918 0x45cfce6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000058 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076685 0x46a520d 0 2 73202819 0x45cfc83 0 3 72732502 0x455cf56 0 1 71221451 0x43ec0cb 0 4 73313984 0x45eaec0 0 5 72496300 0x45234ac 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000014 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732549 0x455cf85 0 1 71221496 0x43ec0f8 0 4 73314028 0x45eaeec 0 5 72496344 0x45234d8 0 0 74076733 0x46a523d 0 2 73202867 0x45cfcb3 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000000 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076665 0x46a51f9 0 2 73202799 0x45cfc6f 0 3 72732481 0x455cf41 0 1 71221428 0x43ec0b4 0 4 73313960 0x45eaea8 0 5 72496276 0x4523494 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000060 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496359 0x45234e7 0 0 74076748 0x46a524c 0 2 73202882 0x45cfcc2 0 3 72732564 0x455cf94 0 1 71221511 0x43ec107 0 4 73314043 0x45eaefb 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000020 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202933 0x45cfcf5 0 3 72732615 0x455cfc7 0 1 71221562 0x43ec13a 0 4 73314094 0x45eaf2e 0 5 72496410 0x452351a 0 0 74076799 0x46a527f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000035 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221437 0x43ec0bd 0 4 73313969 0x45eaeb1 0 5 72496285 0x452349d 0 0 74076674 0x46a5202 0 2 73202808 0x45cfc78 0 3 72732490 0x455cf4a 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000029 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202896 0x45cfcd0 0 3 72732578 0x455cfa2 0 1 71221525 0x43ec115 0 4 73314057 0x45eaf09 0 5 72496373 0x45234f5 0 0 74076762 0x46a525a 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000017 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221417 0x43ec0a9 0 4 73313952 0x45eaea0 0 5 72496268 0x452348c 0 0 74076657 0x46a51f1 0 2 73202792 0x45cfc68 0 3 72732474 0x455cf3a 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000008 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496328 0x45234c8 0 0 74076719 0x46a522f 0 2 73202853 0x45cfca5 0 3 72732536 0x455cf78 0 1 71221483 0x43ec0eb 0 4 73314016 0x45eaee0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000030 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496406 0x4523516 0 0 74076795 0x46a527b 0 2 73202929 0x45cfcf1 0 3 72732611 0x455cfc3 0 1 71221558 0x43ec136 0 4 73314090 0x45eaf2a 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000063 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314068 0x45eaf14 0 5 72496384 0x4523500 0 0 74076773 0x46a5265 0 2 73202908 0x45cfcdc 0 3 72732590 0x455cfae 0 1 71221537 0x43ec121 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000028 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202829 0x45cfc8d 0 3 72732512 0x455cf60 0 1 71221461 0x43ec0d5 0 4 73313996 0x45eaecc 0 5 72496314 0x45234ba 0 0 74076704 0x46a5220 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000041 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496383 0x45234ff 0 0 74076775 0x46a5267 0 2 73202909 0x45cfcdd 0 3 72732593 0x455cfb1 0 1 71221541 0x43ec125 0 4 73314073 0x45eaf19 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000035 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076715 0x46a522b 0 2 73202850 0x45cfca2 0 3 72732533 0x455cf75 0 1 71221481 0x43ec0e9 0 4 73314013 0x45eaedd 0 5 72496330 0x45234ca 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000037 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221402 0x43ec09a 0 4 73313935 0x45eae8f 0 5 72496252 0x452347c 0 0 74076642 0x46a51e2 0 2 73202776 0x45cfc58 0 3 72732458 0x455cf2a 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000004 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496275 0x4523493 0 0 74076664 0x46a51f8 0 2 73202798 0x45cfc6e 0 3 72732480 0x455cf40 0 1 71221427 0x43ec0b3 0 4 73313959 0x45eaea7 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000018 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496479 0x452355f 0 0 74076868 0x46a52c4 0 2 73203002 0x45cfd3a 0 3 72732684 0x455d00c 0 1 71221631 0x43ec17f 0 4 73314163 0x45eaf73 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000063 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314133 0x45eaf55 0 5 72496450 0x4523542 0 0 74076839 0x46a52a7 0 2 73202975 0x45cfd1f 0 3 72732658 0x455cff2 0 1 71221605 0x43ec165 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000006 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076808 0x46a5288 0 2 73202942 0x45cfcfe 0 3 72732624 0x455cfd0 0 1 71221571 0x43ec143 0 4 73314103 0x45eaf37 0 5 72496419 0x4523523 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000027 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221520 0x43ec110 0 4 73314052 0x45eaf04 0 5 72496368 0x45234f0 0 0 74076757 0x46a5255 0 2 73202892 0x45cfccc 0 3 72732574 0x455cf9e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000021 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496474 0x452355a 0 0 74076863 0x46a52bf 0 2 73202999 0x45cfd37 0 3 72732681 0x455d009 0 1 71221628 0x43ec17c 0 4 73314161 0x45eaf71 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000018 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313970 0x45eaeb2 0 5 72496286 0x452349e 0 0 74076676 0x46a5204 0 2 73202810 0x45cfc7a 0 3 72732492 0x455cf4c 0 1 71221439 0x43ec0bf 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000055 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732534 0x455cf76 0 1 71221484 0x43ec0ec 0 4 73314017 0x45eaee1 0 5 72496333 0x45234cd 0 0 74076722 0x46a5232 0 2 73202856 0x45cfca8 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000033 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076875 0x46a52cb 0 2 73203009 0x45cfd41 0 3 72732692 0x455d014 0 1 71221638 0x43ec186 0 4 73314171 0x45eaf7b 0 5 72496488 0x4523568 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000061 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221445 0x43ec0c5 0 4 73313977 0x45eaeb9 0 5 72496293 0x45234a5 0 0 74076684 0x46a520c 0 2 73202817 0x45cfc81 0 3 72732499 0x455cf53 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000050 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496415 0x452351f 0 0 74076804 0x46a5284 0 2 73202938 0x45cfcfa 0 3 72732621 0x455cfcd 0 1 71221568 0x43ec140 0 4 73314100 0x45eaf34 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000047 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732648 0x455cfe8 0 1 71221595 0x43ec15b 0 4 73314127 0x45eaf4f 0 5 72496443 0x452353b 0 0 74076832 0x46a52a0 0 2 73202966 0x45cfd16 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000046 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732631 0x455cfd7 0 1 71221578 0x43ec14a 0 4 73314110 0x45eaf3e 0 5 72496427 0x452352b 0 0 74076816 0x46a5290 0 2 73202950 0x45cfd06 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000020 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202768 0x45cfc50 0 3 72732452 0x455cf24 0 1 71221401 0x43ec099 0 4 73313933 0x45eae8d 0 5 72496250 0x452347a 0 0 74076639 0x46a51df 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000006 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202992 0x45cfd30 0 3 72732674 0x455d002 0 1 71221621 0x43ec175 0 4 73314153 0x45eaf69 0 5 72496469 0x4523555 0 0 74076858 0x46a52ba 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000007 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202905 0x45cfcd9 0 3 72732589 0x455cfad 0 1 71221535 0x43ec11f 0 4 73314070 0x45eaf16 0 5 72496387 0x4523503 0 0 74076777 0x46a5269 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000015 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202890 0x45cfcca 0 3 72732572 0x455cf9c 0 1 71221521 0x43ec111 0 4 73314053 0x45eaf05 0 5 72496369 0x45234f1 0 0 74076758 0x46a5256 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000052 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314157 0x45eaf6d 0 5 72496473 0x4523559 0 0 74076862 0x46a52be 0 2 73202998 0x45cfd36 0 3 72732682 0x455d00a 0 1 71221629 0x43ec17d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000023 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076759 0x46a5257 0 2 73202894 0x45cfcce 0 3 72732576 0x455cfa0 0 1 71221523 0x43ec113 0 4 73314056 0x45eaf08 0 5 72496372 0x45234f4 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000029 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076749 0x46a524d 0 2 73202883 0x45cfcc3 0 3 72732566 0x455cf96 0 1 71221513 0x43ec109 0 4 73314045 0x45eaefd 0 5 72496361 0x45234e9 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000027 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496244 0x4523474 0 0 74076633 0x46a51d9 0 2 73202769 0x45cfc51 0 3 72732451 0x455cf23 0 1 71221399 0x43ec097 0 4 73313932 0x45eae8c 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000018 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221573 0x43ec145 0 4 73314105 0x45eaf39 0 5 72496421 0x4523525 0 0 74076811 0x46a528b 0 2 73202945 0x45cfd01 0 3 72732627 0x455cfd3 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000008 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221556 0x43ec134 0 4 73314088 0x45eaf28 0 5 72496404 0x4523514 0 0 74076793 0x46a5279 0 2 73202927 0x45cfcef 0 3 72732609 0x455cfc1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000025 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314150 0x45eaf66 0 5 72496467 0x4523553 0 0 74076857 0x46a52b9 0 2 73202991 0x45cfd2f 0 3 72732673 0x455d001 0 1 71221620 0x43ec174 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000005 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202766 0x45cfc4e 0 3 72732448 0x455cf20 0 1 71221395 0x43ec093 0 4 73313927 0x45eae87 0 5 72496243 0x4523473 0 0 74076632 0x46a51d8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000039 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732585 0x455cfa9 0 1 71221532 0x43ec11c 0 4 73314064 0x45eaf10 0 5 72496380 0x45234fc 0 0 74076769 0x46a5261 0 2 73202903 0x45cfcd7 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000015 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314147 0x45eaf63 0 5 72496463 0x452354f 0 0 74076852 0x46a52b4 0 2 73202987 0x45cfd2b 0 3 72732669 0x455cffd 0 1 71221616 0x43ec170 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000039 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732655 0x455cfef 0 1 71221604 0x43ec164 0 4 73314136 0x45eaf58 0 5 72496452 0x4523544 0 0 74076841 0x46a52a9 0 2 73202977 0x45cfd21 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000049 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202824 0x45cfc88 0 3 72732506 0x455cf5a 0 1 71221453 0x43ec0cd 0 4 73313985 0x45eaec1 0 5 72496301 0x45234ad 0 0 74076690 0x46a5212 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000051 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076675 0x46a5203 0 2 73202809 0x45cfc79 0 3 72732491 0x455cf4b 0 1 71221438 0x43ec0be 0 4 73313971 0x45eaeb3 0 5 72496288 0x45234a0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000033 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496318 0x45234be 0 0 74076708 0x46a5224 0 2 73202843 0x45cfc9b 0 3 72732526 0x455cf6e 0 1 71221472 0x43ec0e0 0 4 73314004 0x45eaed4 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000009 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314055 0x45eaf07 0 5 72496371 0x45234f3 0 0 74076761 0x46a5259 0 2 73202895 0x45cfccf 0 3 72732577 0x455cfa1 0 1 71221524 0x43ec114 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000047 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221507 0x43ec103 0 4 73314040 0x45eaef8 0 5 72496356 0x45234e4 0 0 74076746 0x46a524a 0 2 73202880 0x45cfcc0 0 3 72732562 0x455cf92 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000001 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202943 0x45cfcff 0 3 72732625 0x455cfd1 0 1 71221572 0x43ec144 0 4 73314104 0x45eaf38 0 5 72496420 0x4523524 0 0 74076809 0x46a5289 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000053 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221414 0x43ec0a6 0 4 73313948 0x45eae9c 0 5 72496265 0x4523489 0 0 74076655 0x46a51ef 0 2 73202791 0x45cfc67 0 3 72732475 0x455cf3b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000002 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314162 0x45eaf72 0 5 72496478 0x452355e 0 0 74076867 0x46a52c3 0 2 73203001 0x45cfd39 0 3 72732683 0x455d00b 0 1 71221630 0x43ec17e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000022 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202996 0x45cfd34 0 3 72732678 0x455d006 0 1 71221625 0x43ec179 0 4 73314158 0x45eaf6e 0 5 72496476 0x452355c 0 0 74076865 0x46a52c1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000022 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202781 0x45cfc5d 0 3 72732464 0x455cf30 0 1 71221412 0x43ec0a4 0 4 73313944 0x45eae98 0 5 72496260 0x4523484 0 0 74076649 0x46a51e9 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000026 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076790 0x46a5276 0 2 73202925 0x45cfced 0 3 72732607 0x455cfbf 0 1 71221554 0x43ec132 0 4 73314086 0x45eaf26 0 5 72496402 0x4523512 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000004 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496461 0x452354d 0 0 74076851 0x46a52b3 0 2 73202986 0x45cfd2a 0 3 72732667 0x455cffb 0 1 71221614 0x43ec16e 0 4 73314146 0x45eaf62 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000043 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732679 0x455d007 0 1 71221626 0x43ec17a 0 4 73314159 0x45eaf6f 0 5 72496475 0x452355b 0 0 74076864 0x46a52c0 0 2 73203000 0x45cfd38 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000056 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314067 0x45eaf13 0 5 72496386 0x4523502 0 0 74076776 0x46a5268 0 2 73202911 0x45cfcdf 0 3 72732594 0x455cfb2 0 1 71221540 0x43ec124 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000031 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732579 0x455cfa3 0 1 71221526 0x43ec116 0 4 73314058 0x45eaf0a 0 5 72496374 0x45234f6 0 0 74076763 0x46a525b 0 2 73202897 0x45cfcd1 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000021 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732510 0x455cf5e 0 1 71221459 0x43ec0d3 0 4 73313992 0x45eaec8 0 5 72496308 0x45234b4 0 0 74076697 0x46a5219 0 2 73202834 0x45cfc92 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000014 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221449 0x43ec0c9 0 4 73313982 0x45eaebe 0 5 72496298 0x45234aa 0 0 74076688 0x46a5210 0 2 73202821 0x45cfc85 0 3 72732503 0x455cf57 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000017 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732606 0x455cfbe 0 1 71221553 0x43ec131 0 4 73314085 0x45eaf25 0 5 72496401 0x4523511 0 0 74076791 0x46a5277 0 2 73202924 0x45cfcec 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000024 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202807 0x45cfc77 0 3 72732489 0x455cf49 0 1 71221436 0x43ec0bc 0 4 73313968 0x45eaeb0 0 5 72496284 0x452349c 0 0 74076673 0x46a5201 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000055 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313946 0x45eae9a 0 5 72496263 0x4523487 0 0 74076652 0x46a51ec 0 2 73202786 0x45cfc62 0 3 72732469 0x455cf35 0 1 71221418 0x43ec0aa 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000008 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076631 0x46a51d7 0 2 73202765 0x45cfc4d 0 3 72732447 0x455cf1f 0 1 71221394 0x43ec092 0 4 73313926 0x45eae86 0 5 72496242 0x4523472 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000007 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496302 0x45234ae 0 0 74076691 0x46a5213 0 2 73202825 0x45cfc89 0 3 72732508 0x455cf5c 0 1 71221457 0x43ec0d1 0 4 73313989 0x45eaec5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000032 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076833 0x46a52a1 0 2 73202967 0x45cfd17 0 3 72732649 0x455cfe9 0 1 71221596 0x43ec15c 0 4 73314128 0x45eaf50 0 5 72496445 0x452353d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000055 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076766 0x46a525e 0 2 73202900 0x45cfcd4 0 3 72732582 0x455cfa6 0 1 71221529 0x43ec119 0 4 73314061 0x45eaf0d 0 5 72496377 0x45234f9 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000021 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313964 0x45eaeac 0 5 72496280 0x4523498 0 0 74076669 0x46a51fd 0 2 73202803 0x45cfc73 0 3 72732485 0x455cf45 0 1 71221432 0x43ec0b8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000004 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313990 0x45eaec6 0 5 72496311 0x45234b7 0 0 74076701 0x46a521d 0 2 73202835 0x45cfc93 0 3 72732516 0x455cf64 0 1 71221463 0x43ec0d7 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000024 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313987 0x45eaec3 0 5 72496304 0x45234b0 0 0 74076695 0x46a5217 0 2 73202833 0x45cfc91 0 3 72732518 0x455cf66 0 1 71221465 0x43ec0d9 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000007 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202762 0x45cfc4a 0 3 72732444 0x455cf1c 0 1 71221391 0x43ec08f 0 4 73313924 0x45eae84 0 5 72496240 0x4523470 0 0 74076629 0x46a51d5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000042 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313981 0x45eaebd 0 5 72496297 0x45234a9 0 0 74076687 0x46a520f 0 2 73202822 0x45cfc86 0 3 72732504 0x455cf58 0 1 71221452 0x43ec0cc 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000010 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202839 0x45cfc97 0 3 72732521 0x455cf69 0 1 71221468 0x43ec0dc 0 4 73314001 0x45eaed1 0 5 72496317 0x45234bd 0 0 74076706 0x46a5222 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000038 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314093 0x45eaf2d 0 5 72496409 0x4523519 0 0 74076798 0x46a527e 0 2 73202932 0x45cfcf4 0 3 72732614 0x455cfc6 0 1 71221561 0x43ec139 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000052 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732500 0x455cf54 0 1 71221448 0x43ec0c8 0 4 73313980 0x45eaebc 0 5 72496296 0x45234a8 0 0 74076686 0x46a520e 0 2 73202820 0x45cfc84 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000032 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202934 0x45cfcf6 0 3 72732616 0x455cfc8 0 1 71221564 0x43ec13c 0 4 73314097 0x45eaf31 0 5 72496414 0x452351e 0 0 74076803 0x46a5283 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000023 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221460 0x43ec0d4 0 4 73313994 0x45eaeca 0 5 72496310 0x45234b6 0 0 74076698 0x46a521a 0 2 73202831 0x45cfc8f 0 3 72732514 0x455cf62 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000045 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496393 0x4523509 0 0 74076782 0x46a526e 0 2 73202916 0x45cfce4 0 3 72732600 0x455cfb8 0 1 71221546 0x43ec12a 0 4 73314078 0x45eaf1e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000016 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496465 0x4523551 0 0 74076855 0x46a52b7 0 2 73202989 0x45cfd2d 0 3 72732671 0x455cfff 0 1 71221618 0x43ec172 0 4 73314151 0x45eaf67 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000009 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732565 0x455cf95 0 1 71221512 0x43ec108 0 4 73314044 0x45eaefc 0 5 72496360 0x45234e8 0 0 74076750 0x46a524e 0 2 73202884 0x45cfcc4 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000052 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221499 0x43ec0fb 0 4 73314032 0x45eaef0 0 5 72496349 0x45234dd 0 0 74076738 0x46a5242 0 2 73202872 0x45cfcb8 0 3 72732554 0x455cf8a 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000022 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732543 0x455cf7f 0 1 71221490 0x43ec0f2 0 4 73314022 0x45eaee6 0 5 72496338 0x45234d2 0 0 74076727 0x46a5237 0 2 73202861 0x45cfcad 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000050 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076876 0x46a52cc 0 2 73203011 0x45cfd43 0 3 72732694 0x455d016 0 1 71221642 0x43ec18a 0 4 73314174 0x45eaf7e 0 5 72496491 0x452356b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000043 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732529 0x455cf71 0 1 71221476 0x43ec0e4 0 4 73314008 0x45eaed8 0 5 72496324 0x45234c4 0 0 74076713 0x46a5229 0 2 73202847 0x45cfc9f 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000004 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221527 0x43ec117 0 4 73314060 0x45eaf0c 0 5 72496376 0x45234f8 0 0 74076765 0x46a525d 0 2 73202899 0x45cfcd3 0 3 72732581 0x455cfa5 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000003 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221455 0x43ec0cf 0 4 73313988 0x45eaec4 0 5 72496305 0x45234b1 0 0 74076694 0x46a5216 0 2 73202830 0x45cfc8e 0 3 72732513 0x455cf61 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000039 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496313 0x45234b9 0 0 74076702 0x46a521e 0 2 73202838 0x45cfc96 0 3 72732519 0x455cf67 0 1 71221466 0x43ec0da 0 4 73313999 0x45eaecf 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000060 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496400 0x4523510 0 0 74076789 0x46a5275 0 2 73202923 0x45cfceb 0 3 72732605 0x455cfbd 0 1 71221552 0x43ec130 0 4 73314084 0x45eaf24 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000041 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202973 0x45cfd1d 0 3 72732656 0x455cff0 0 1 71221603 0x43ec163 0 4 73314135 0x45eaf57 0 5 72496454 0x4523546 0 0 74076843 0x46a52ab 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000012 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313928 0x45eae88 0 5 72496245 0x4523475 0 0 74076634 0x46a51da 0 2 73202770 0x45cfc52 0 3 72732453 0x455cf25 0 1 71221403 0x43ec09b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000049 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732584 0x455cfa8 0 1 71221531 0x43ec11b 0 4 73314063 0x45eaf0f 0 5 72496379 0x45234fb 0 0 74076768 0x46a5260 0 2 73202902 0x45cfcd6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000054 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221443 0x43ec0c3 0 4 73313975 0x45eaeb7 0 5 72496291 0x45234a3 0 0 74076681 0x46a5209 0 2 73202816 0x45cfc80 0 3 72732498 0x455cf52 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000050 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221400 0x43ec098 0 4 73313934 0x45eae8e 0 5 72496251 0x452347b 0 0 74076641 0x46a51e1 0 2 73202774 0x45cfc56 0 3 72732456 0x455cf28 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000035 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202997 0x45cfd35 0 3 72732680 0x455d008 0 1 71221627 0x43ec17b 0 4 73314160 0x45eaf70 0 5 72496477 0x452355d 0 0 74076866 0x46a52c2 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000010 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496398 0x452350e 0 0 74076787 0x46a5273 0 2 73202921 0x45cfce9 0 3 72732603 0x455cfbb 0 1 71221550 0x43ec12e 0 4 73314083 0x45eaf23 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000044 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221633 0x43ec181 0 4 73314167 0x45eaf77 0 5 72496483 0x4523563 0 0 74076872 0x46a52c8 0 2 73203007 0x45cfd3f 0 3 72732690 0x455d012 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000019 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202841 0x45cfc99 0 3 72732523 0x455cf6b 0 1 71221471 0x43ec0df 0 4 73314003 0x45eaed3 0 5 72496320 0x45234c0 0 0 74076709 0x46a5225 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000020 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221497 0x43ec0f9 0 4 73314029 0x45eaeed 0 5 72496345 0x45234d9 0 0 74076735 0x46a523f 0 2 73202870 0x45cfcb6 0 3 72732552 0x455cf88 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000006 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732473 0x455cf39 0 1 71221421 0x43ec0ad 0 4 73313954 0x45eaea2 0 5 72496271 0x452348f 0 0 74076661 0x46a51f5 0 2 73202796 0x45cfc6c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000015 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313949 0x45eae9d 0 5 72496266 0x452348a 0 0 74076654 0x46a51ee 0 2 73202788 0x45cfc64 0 3 72732470 0x455cf36 0 1 71221419 0x43ec0ab 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000016 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076682 0x46a520a 0 2 73202818 0x45cfc82 0 3 72732501 0x455cf55 0 1 71221447 0x43ec0c7 0 4 73313979 0x45eaebb 0 5 72496295 0x45234a7 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000011 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076828 0x46a529c 0 2 73202962 0x45cfd12 0 3 72732644 0x455cfe4 0 1 71221593 0x43ec159 0 4 73314125 0x45eaf4d 0 5 72496441 0x4523539 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000037 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221580 0x43ec14c 0 4 73314112 0x45eaf40 0 5 72496428 0x452352c 0 0 74076817 0x46a5291 0 2 73202951 0x45cfd07 0 3 72732633 0x455cfd9 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000027 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732532 0x455cf74 0 1 71221480 0x43ec0e8 0 4 73314012 0x45eaedc 0 5 72496329 0x45234c9 0 0 74076717 0x46a522d 0 2 73202851 0x45cfca3 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000014 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496436 0x4523534 0 0 74076825 0x46a5299 0 2 73202959 0x45cfd0f 0 3 72732641 0x455cfe1 0 1 71221588 0x43ec154 0 4 73314120 0x45eaf48 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000048 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496456 0x4523548 0 0 74076845 0x46a52ad 0 2 73202979 0x45cfd23 0 3 72732661 0x455cff5 0 1 71221608 0x43ec168 0 4 73314140 0x45eaf5c 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000016 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314054 0x45eaf06 0 5 72496370 0x45234f2 0 0 74076760 0x46a5258 0 2 73202893 0x45cfccd 0 3 72732575 0x455cf9f 0 1 71221522 0x43ec112 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000048 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732467 0x455cf33 0 1 71221415 0x43ec0a7 0 4 73313951 0x45eae9f 0 5 72496267 0x452348b 0 0 74076656 0x46a51f0 0 2 73202790 0x45cfc66 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000059 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732459 0x455cf2b 0 1 71221407 0x43ec09f 0 4 73313941 0x45eae95 0 5 72496258 0x4523482 0 0 74076647 0x46a51e7 0 2 73202783 0x45cfc5f 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000042 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314042 0x45eaefa 0 5 72496358 0x45234e6 0 0 74076747 0x46a524b 0 2 73202881 0x45cfcc1 0 3 72732563 0x455cf93 0 1 71221510 0x43ec106 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000025 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314035 0x45eaef3 0 5 72496351 0x45234df 0 0 74076740 0x46a5244 0 2 73202875 0x45cfcbb 0 3 72732556 0x455cf8c 0 1 71221504 0x43ec100 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000036 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203010 0x45cfd42 0 3 72732693 0x455d015 0 1 71221643 0x43ec18b 0 4 73314175 0x45eaf7f 0 5 72496492 0x452356c 0 0 74076881 0x46a52d1 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000022 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732608 0x455cfc0 0 1 71221555 0x43ec133 0 4 73314087 0x45eaf27 0 5 72496403 0x4523513 0 0 74076792 0x46a5278 0 2 73202926 0x45cfcee 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000028 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221396 0x43ec094 0 4 73313929 0x45eae89 0 5 72496246 0x4523476 0 0 74076635 0x46a51db 0 2 73202771 0x45cfc53 0 3 72732454 0x455cf26 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000023 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221450 0x43ec0ca 0 4 73313983 0x45eaebf 0 5 72496299 0x45234ab 0 0 74076689 0x46a5211 0 2 73202823 0x45cfc87 0 3 72732505 0x455cf59 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000037 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202837 0x45cfc95 0 3 72732520 0x455cf68 0 1 71221467 0x43ec0db 0 4 73314000 0x45eaed0 0 5 72496316 0x45234bc 0 0 74076705 0x46a5221 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000056 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496283 0x452349b 0 0 74076672 0x46a5200 0 2 73202806 0x45cfc76 0 3 72732488 0x455cf48 0 1 71221435 0x43ec0bb 0 4 73313967 0x45eaeaf 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000019 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221406 0x43ec09e 0 4 73313939 0x45eae93 0 5 72496257 0x4523481 0 0 74076646 0x46a51e6 0 2 73202782 0x45cfc5e 0 3 72732465 0x455cf31 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000045 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202793 0x45cfc69 0 3 72732476 0x455cf3c 0 1 71221423 0x43ec0af 0 4 73313955 0x45eaea3 0 5 72496272 0x4523490 0 0 74076662 0x46a51f6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000038 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203004 0x45cfd3c 0 3 72732687 0x455d00f 0 1 71221635 0x43ec183 0 4 73314168 0x45eaf78 0 5 72496484 0x4523564 0 0 74076873 0x46a52c9 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000021 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496378 0x45234fa 0 0 74076767 0x46a525f 0 2 73202901 0x45cfcd5 0 3 72732583 0x455cfa7 0 1 71221530 0x43ec11a 0 4 73314062 0x45eaf0e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000010 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496253 0x452347d 0 0 74076643 0x46a51e3 0 2 73202777 0x45cfc59 0 3 72732460 0x455cf2c 0 1 71221408 0x43ec0a0 0 4 73313940 0x45eae94 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000042 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732653 0x455cfed 0 1 71221600 0x43ec160 0 4 73314132 0x45eaf54 0 5 72496448 0x4523540 0 0 74076837 0x46a52a5 0 2 73202971 0x45cfd1b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000063 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496249 0x4523479 0 0 74076638 0x46a51de 0 2 73202773 0x45cfc55 0 3 72732455 0x455cf27 0 1 71221404 0x43ec09c 0 4 73313937 0x45eae91 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000030 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076627 0x46a51d3 0 2 73202761 0x45cfc49 0 3 72732443 0x455cf1b 0 1 71221390 0x43ec08e 0 4 73313922 0x45eae82 0 5 72496238 0x452346e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000055 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496451 0x4523543 0 0 74076840 0x46a52a8 0 2 73202976 0x45cfd20 0 3 72732659 0x455cff3 0 1 71221607 0x43ec167 0 4 73314139 0x45eaf5b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000028 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314089 0x45eaf29 0 5 72496405 0x4523515 0 0 74076794 0x46a527a 0 2 73202928 0x45cfcf0 0 3 72732610 0x455cfc2 0 1 71221557 0x43ec135 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000026 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732670 0x455cffe 0 1 71221617 0x43ec171 0 4 73314149 0x45eaf65 0 5 72496466 0x4523552 0 0 74076854 0x46a52b6 0 2 73202988 0x45cfd2c 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000046 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221474 0x43ec0e2 0 4 73314006 0x45eaed6 0 5 72496322 0x45234c2 0 0 74076710 0x46a5226 0 2 73202844 0x45cfc9c 0 3 72732528 0x455cf70 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000011 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314030 0x45eaeee 0 5 72496346 0x45234da 0 0 74076736 0x46a5240 0 2 73202871 0x45cfcb7 0 3 72732553 0x455cf89 0 1 71221501 0x43ec0fd 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000061 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221519 0x43ec10f 0 4 73314051 0x45eaf03 0 5 72496367 0x45234ef 0 0 74076756 0x46a5254 0 2 73202891 0x45cfccb 0 3 72732573 0x455cf9d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000058 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496306 0x45234b2 0 0 74076696 0x46a5218 0 2 73202836 0x45cfc94 0 3 72732517 0x455cf65 0 1 71221464 0x43ec0d8 0 4 73313998 0x45eaece 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000051 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076772 0x46a5264 0 2 73202910 0x45cfcde 0 3 72732591 0x455cfaf 0 1 71221539 0x43ec123 0 4 73314072 0x45eaf18 0 5 72496389 0x4523505 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000008 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076850 0x46a52b2 0 2 73202984 0x45cfd28 0 3 72732666 0x455cffa 0 1 71221613 0x43ec16d 0 4 73314145 0x45eaf61 0 5 72496462 0x452354e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000013 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732483 0x455cf43 0 1 71221430 0x43ec0b6 0 4 73313962 0x45eaeaa 0 5 72496278 0x4523496 0 0 74076667 0x46a51fb 0 2 73202801 0x45cfc71 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000023 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314121 0x45eaf49 0 5 72496437 0x4523535 0 0 74076826 0x46a529a 0 2 73202960 0x45cfd10 0 3 72732642 0x455cfe2 0 1 71221591 0x43ec157 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000040 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076806 0x46a5286 0 2 73202940 0x45cfcfc 0 3 72732622 0x455cfce 0 1 71221570 0x43ec142 0 4 73314102 0x45eaf36 0 5 72496418 0x4523522 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000056 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076742 0x46a5246 0 2 73202876 0x45cfcbc 0 3 72732558 0x455cf8e 0 1 71221506 0x43ec102 0 4 73314038 0x45eaef6 0 5 72496355 0x45234e3 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000013 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202826 0x45cfc8a 0 3 72732509 0x455cf5d 0 1 71221458 0x43ec0d2 0 4 73313991 0x45eaec7 0 5 72496307 0x45234b3 0 0 74076699 0x46a521b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000053 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221590 0x43ec156 0 4 73314123 0x45eaf4b 0 5 72496439 0x4523537 0 0 74076830 0x46a529e 0 2 73202964 0x45cfd14 0 3 72732646 0x455cfe6 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000032 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202873 0x45cfcb9 0 3 72732555 0x455cf8b 0 1 71221503 0x43ec0ff 0 4 73314036 0x45eaef4 0 5 72496352 0x45234e0 0 0 74076741 0x46a5245 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000033 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496261 0x4523485 0 0 74076650 0x46a51ea 0 2 73202785 0x45cfc61 0 3 72732468 0x455cf34 0 1 71221416 0x43ec0a8 0 4 73313950 0x45eae9e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000062 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496487 0x4523567 0 0 74076879 0x46a52cf 0 2 73203014 0x45cfd46 0 3 72732696 0x455d018 0 1 71221644 0x43ec18c 0 4 73314176 0x45eaf80 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000030 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076860 0x46a52bc 0 2 73202994 0x45cfd32 0 3 72732676 0x455d004 0 1 71221624 0x43ec178 0 4 73314156 0x45eaf6c 0 5 72496472 0x4523558 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000063 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076714 0x46a522a 0 2 73202848 0x45cfca0 0 3 72732530 0x455cf72 0 1 71221477 0x43ec0e5 0 4 73314009 0x45eaed9 0 5 72496325 0x45234c5 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000000 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732588 0x455cfac 0 1 71221536 0x43ec120 0 4 73314071 0x45eaf17 0 5 72496388 0x4523504 0 0 74076778 0x46a526a 0 2 73202912 0x45cfce0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000043 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314069 0x45eaf15 0 5 72496385 0x4523501 0 0 74076774 0x46a5266 0 2 73202907 0x45cfcdb 0 3 72732592 0x455cfb0 0 1 71221538 0x43ec122 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000012 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076734 0x46a523e 0 2 73202869 0x45cfcb5 0 3 72732551 0x455cf87 0 1 71221500 0x43ec0fc 0 4 73314033 0x45eaef1 0 5 72496348 0x45234dc 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000002 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732602 0x455cfba 0 1 71221549 0x43ec12d 0 4 73314081 0x45eaf21 0 5 72496397 0x452350d 0 0 74076786 0x46a5272 0 2 73202920 0x45cfce8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000000 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221456 0x43ec0d0 0 4 73313993 0x45eaec9 0 5 72496309 0x45234b5 0 0 74076700 0x46a521c 0 2 73202832 0x45cfc90 0 3 72732515 0x455cf63 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000045 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076693 0x46a5215 0 2 73202828 0x45cfc8c 0 3 72732511 0x455cf5f 0 1 71221462 0x43ec0d6 0 4 73313997 0x45eaecd 0 5 72496315 0x45234bb 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000062 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221569 0x43ec141 0 4 73314101 0x45eaf35 0 5 72496417 0x4523521 0 0 74076807 0x46a5287 0 2 73202941 0x45cfcfd 0 3 72732623 0x455cfcf 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000026 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076745 0x46a5249 0 2 73202879 0x45cfcbf 0 3 72732561 0x455cf91 0 1 71221509 0x43ec105 0 4 73314041 0x45eaef9 0 5 72496357 0x45234e5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000061 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202981 0x45cfd25 0 3 72732663 0x455cff7 0 1 71221610 0x43ec16a 0 4 73314142 0x45eaf5e 0 5 72496458 0x452354a 0 0 74076847 0x46a52af 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000044 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313923 0x45eae83 0 5 72496239 0x452346f 0 0 74076628 0x46a51d4 0 2 73202763 0x45cfc4b 0 3 72732446 0x455cf1e 0 1 71221393 0x43ec091 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000038 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202814 0x45cfc7e 0 3 72732496 0x455cf50 0 1 71221444 0x43ec0c4 0 4 73313976 0x45eaeb8 0 5 72496292 0x45234a4 0 0 74076683 0x46a520b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000048 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221563 0x43ec13b 0 4 73314095 0x45eaf2f 0 5 72496412 0x452351c 0 0 74076801 0x46a5281 0 2 73202936 0x45cfcf8 0 3 72732618 0x455cfca 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000041 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221469 0x43ec0dd 0 4 73314002 0x45eaed2 0 5 72496319 0x45234bf 0 0 74076707 0x46a5223 0 2 73202842 0x45cfc9a 0 3 72732525 0x455cf6d 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000027 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314119 0x45eaf47 0 5 72496435 0x4523533 0 0 74076824 0x46a5298 0 2 73202958 0x45cfd0e 0 3 72732640 0x455cfe0 0 1 71221587 0x43ec153 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000059 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221502 0x43ec0fe 0 4 73314034 0x45eaef2 0 5 72496350 0x45234de 0 0 74076739 0x46a5243 0 2 73202874 0x45cfcba 0 3 72732557 0x455cf8d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000053 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732527 0x455cf6f 0 1 71221475 0x43ec0e3 0 4 73314007 0x45eaed7 0 5 72496323 0x45234c3 0 0 74076712 0x46a5228 0 2 73202846 0x45cfc9e 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000030 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314015 0x45eaedf 0 5 72496332 0x45234cc 0 0 74076721 0x46a5231 0 2 73202855 0x45cfca7 0 3 72732538 0x455cf7a 0 1 71221485 0x43ec0ed 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000003 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496269 0x452348d 0 0 74076658 0x46a51f2 0 2 73202794 0x45cfc6a 0 3 72732477 0x455cf3d 0 1 71221424 0x43ec0b0 0 4 73313956 0x45eaea4 0 Start IOR test posix_directio_32m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐B ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_directio_32m [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 22:33:38 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐B ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_directio_32mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.15099/posix_directio_32m access = file‐per‐process ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 1737.83 1737.83 1737.83 0.00 54.31 54.31 54.31 0.00 301.69082 64 2 1 1 1 1 0 0 1 0 33554432 549755813888 ‐1 POSIX EXCEL Max Write: 1737.83 MiB/sec (1822.25 MB/sec) Perf_Data IOR 33554432 write 1737.832 MiB/sec Run finished: Wed Apr 20 22:38:40 2016 Application 183680 resources: utime ~21s, stime ~50s, Rss ~4744, inblocks ~0, outblocks ~0 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐B ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_directio_32m [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 22:38:48 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐B ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_directio_32mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.15099/posix_directio_32m access = file‐per‐process ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 1815.80 1815.80 1815.80 0.00 56.74 56.74 56.74 0.00 288.73608 64 2 1 1 1 1 0 0 1 0 33554432 549755813888 ‐1 POSIX EXCEL Max Read: 1815.80 MiB/sec (1904.01 MB/sec) Perf_Data IOR 33554432 read 1815.804 MiB/sec Run finished: Wed Apr 20 22:43:37 2016 Application 183681 resources: utime ~697s, stime ~52s, Rss ~4744, inblocks ~0, outblocks ~0 Done IOR posix_directio_32m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes posix_directio_32m, down= 3 total 2.6T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000063 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000063 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000063 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000063 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000063 /lus/husk/dmoen/testdir1.15099 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000047 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732445 0x455cf1d 0 1 71221392 0x43ec090 0 4 73313925 0x45eae85 0 5 72496241 0x4523471 0 0 74076630 0x46a51d6 0 2 73202764 0x45cfc4c 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000012 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496411 0x452351b 0 0 74076800 0x46a5280 0 2 73202935 0x45cfcf7 0 3 72732617 0x455cfc9 0 1 71221565 0x43ec13d 0 4 73314098 0x45eaf32 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000042 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732620 0x455cfcc 0 1 71221567 0x43ec13f 0 4 73314099 0x45eaf33 0 5 72496416 0x4523520 0 0 74076805 0x46a5285 0 2 73202939 0x45cfcfb 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000001 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202767 0x45cfc4f 0 3 72732450 0x455cf22 0 1 71221398 0x43ec096 0 4 73313931 0x45eae8b 0 5 72496248 0x4523478 0 0 74076636 0x46a51dc 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000054 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076796 0x46a527c 0 2 73202930 0x45cfcf2 0 3 72732612 0x455cfc4 0 1 71221559 0x43ec137 0 4 73314091 0x45eaf2b 0 5 72496407 0x4523517 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000060 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221640 0x43ec188 0 4 73314173 0x45eaf7d 0 5 72496490 0x452356a 0 0 74076880 0x46a52d0 0 2 73203015 0x45cfd47 0 3 72732697 0x455d019 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000016 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314018 0x45eaee2 0 5 72496334 0x45234ce 0 0 74076723 0x46a5233 0 2 73202857 0x45cfca9 0 3 72732539 0x455cf7b 0 1 71221486 0x43ec0ee 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000018 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314191 0x45eaf8f 0 5 72496507 0x452357b 0 0 74076896 0x46a52e0 0 2 73203034 0x45cfd5a 0 3 72732716 0x455d02c 0 1 71221664 0x43ec1a0 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000032 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496287 0x452349f 0 0 74076677 0x46a5205 0 2 73202811 0x45cfc7b 0 3 72732494 0x455cf4e 0 1 71221441 0x43ec0c1 0 4 73313973 0x45eaeb5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000037 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496457 0x4523549 0 0 74076846 0x46a52ae 0 2 73202980 0x45cfd24 0 3 72732662 0x455cff6 0 1 71221609 0x43ec169 0 4 73314141 0x45eaf5d 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000013 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076869 0x46a52c5 0 2 73203003 0x45cfd3b 0 3 72732685 0x455d00d 0 1 71221632 0x43ec180 0 4 73314164 0x45eaf74 0 5 72496480 0x4523560 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000025 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076640 0x46a51e0 0 2 73202775 0x45cfc57 0 3 72732457 0x455cf29 0 1 71221405 0x43ec09d 0 4 73313938 0x45eae92 0 5 72496255 0x452347f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000057 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313936 0x45eae90 0 5 72496254 0x452347e 0 0 74076644 0x46a51e4 0 2 73202779 0x45cfc5b 0 3 72732461 0x455cf2d 0 1 71221409 0x43ec0a1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000053 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221660 0x43ec19c 0 4 73314195 0x45eaf93 0 5 72496511 0x452357f 0 0 74076900 0x46a52e4 0 2 73203035 0x45cfd5b 0 3 72732717 0x455d02d 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000026 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202800 0x45cfc70 0 3 72732482 0x455cf42 0 1 71221429 0x43ec0b5 0 4 73313961 0x45eaea9 0 5 72496277 0x4523495 0 0 74076666 0x46a51fa 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000013 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202904 0x45cfcd8 0 3 72732586 0x455cfaa 0 1 71221533 0x43ec11d 0 4 73314065 0x45eaf11 0 5 72496381 0x45234fd 0 0 74076770 0x46a5262 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000011 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076754 0x46a5252 0 2 73202889 0x45cfcc9 0 3 72732571 0x455cf9b 0 1 71221518 0x43ec10e 0 4 73314050 0x45eaf02 0 5 72496366 0x45234ee 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000028 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076819 0x46a5293 0 2 73202954 0x45cfd0a 0 3 72732637 0x455cfdd 0 1 71221583 0x43ec14f 0 4 73314115 0x45eaf43 0 5 72496431 0x452352f 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000052 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314096 0x45eaf30 0 5 72496413 0x452351d 0 0 74076802 0x46a5282 0 2 73202937 0x45cfcf9 0 3 72732619 0x455cfcb 0 1 71221566 0x43ec13e 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000040 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202866 0x45cfcb2 0 3 72732548 0x455cf84 0 1 71221495 0x43ec0f7 0 4 73314027 0x45eaeeb 0 5 72496343 0x45234d7 0 0 74076732 0x46a523c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000041 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221431 0x43ec0b7 0 4 73313963 0x45eaeab 0 5 72496279 0x4523497 0 0 74076668 0x46a51fc 0 2 73202802 0x45cfc72 0 3 72732484 0x455cf44 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000012 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496449 0x4523541 0 0 74076838 0x46a52a6 0 2 73202972 0x45cfd1c 0 3 72732654 0x455cfee 0 1 71221601 0x43ec161 0 4 73314134 0x45eaf56 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000005 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221491 0x43ec0f3 0 4 73314023 0x45eaee7 0 5 72496339 0x45234d3 0 0 74076728 0x46a5238 0 2 73202862 0x45cfcae 0 3 72732544 0x455cf80 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000001 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202985 0x45cfd29 0 3 72732668 0x455cffc 0 1 71221615 0x43ec16f 0 4 73314148 0x45eaf64 0 5 72496464 0x4523550 0 0 74076853 0x46a52b5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000000 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076820 0x46a5294 0 2 73202953 0x45cfd09 0 3 72732636 0x455cfdc 0 1 71221584 0x43ec150 0 4 73314116 0x45eaf44 0 5 72496432 0x4523530 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000051 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221611 0x43ec16b 0 4 73314143 0x45eaf5f 0 5 72496459 0x452354b 0 0 74076848 0x46a52b0 0 2 73202982 0x45cfd26 0 3 72732664 0x455cff8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000017 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313986 0x45eaec2 0 5 72496303 0x45234af 0 0 74076692 0x46a5214 0 2 73202827 0x45cfc8b 0 3 72732507 0x455cf5b 0 1 71221454 0x43ec0ce 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000033 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732587 0x455cfab 0 1 71221534 0x43ec11e 0 4 73314066 0x45eaf12 0 5 72496382 0x45234fe 0 0 74076771 0x46a5263 0 2 73202906 0x45cfcda 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000057 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314165 0x45eaf75 0 5 72496481 0x4523561 0 0 74076870 0x46a52c6 0 2 73203005 0x45cfd3d 0 3 72732686 0x455d00e 0 1 71221634 0x43ec182 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000059 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314107 0x45eaf3b 0 5 72496423 0x4523527 0 0 74076813 0x46a528d 0 2 73202948 0x45cfd04 0 3 72732630 0x455cfd6 0 1 71221577 0x43ec149 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000031 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202789 0x45cfc65 0 3 72732472 0x455cf38 0 1 71221422 0x43ec0ae 0 4 73313953 0x45eaea1 0 5 72496270 0x452348e 0 0 74076660 0x46a51f4 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000043 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313947 0x45eae9b 0 5 72496264 0x4523488 0 0 74076653 0x46a51ed 0 2 73202787 0x45cfc63 0 3 72732471 0x455cf37 0 1 71221420 0x43ec0ac 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000060 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732730 0x455d03a 0 1 71221678 0x43ec1ae 0 4 73314210 0x45eafa2 0 5 72496527 0x452358f 0 0 74076916 0x46a52f4 0 2 73203051 0x45cfd6b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000005 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496442 0x452353a 0 0 74076831 0x46a529f 0 2 73202965 0x45cfd15 0 3 72732647 0x455cfe7 0 1 71221594 0x43ec15a 0 4 73314126 0x45eaf4e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000003 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496444 0x452353c 0 0 74076834 0x46a52a2 0 2 73202968 0x45cfd18 0 3 72732650 0x455cfea 0 1 71221597 0x43ec15d 0 4 73314129 0x45eaf51 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000057 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732524 0x455cf6c 0 1 71221473 0x43ec0e1 0 4 73314005 0x45eaed5 0 5 72496321 0x45234c1 0 0 74076711 0x46a5227 0 2 73202845 0x45cfc9d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000054 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221514 0x43ec10a 0 4 73314047 0x45eaeff 0 5 72496363 0x45234eb 0 0 74076752 0x46a5250 0 2 73202887 0x45cfcc7 0 3 72732569 0x455cf99 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000034 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221589 0x43ec155 0 4 73314122 0x45eaf4a 0 5 72496438 0x4523536 0 0 74076829 0x46a529d 0 2 73202963 0x45cfd13 0 3 72732645 0x455cfe5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000039 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496256 0x4523480 0 0 74076645 0x46a51e5 0 2 73202780 0x45cfc5c 0 3 72732463 0x455cf2f 0 1 71221411 0x43ec0a3 0 4 73313943 0x45eae97 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000007 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203038 0x45cfd5e 0 3 72732721 0x455d031 0 1 71221668 0x43ec1a4 0 4 73314201 0x45eaf99 0 5 72496517 0x4523585 0 0 74076906 0x46a52ea 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000039 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732724 0x455d034 0 1 71221672 0x43ec1a8 0 4 73314205 0x45eaf9d 0 5 72496520 0x4523588 0 0 74076910 0x46a52ee 0 2 73203046 0x45cfd66 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000004 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076902 0x46a52e6 0 2 73203036 0x45cfd5c 0 3 72732718 0x455d02e 0 1 71221665 0x43ec1a1 0 4 73314198 0x45eaf96 0 5 72496513 0x4523581 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000029 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076670 0x46a51fe 0 2 73202804 0x45cfc74 0 3 72732486 0x455cf46 0 1 71221433 0x43ec0b9 0 4 73313965 0x45eaead 0 5 72496281 0x4523499 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000011 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076678 0x46a5206 0 2 73202812 0x45cfc7c 0 3 72732495 0x455cf4f 0 1 71221442 0x43ec0c2 0 4 73313974 0x45eaeb6 0 5 72496290 0x45234a2 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000036 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202919 0x45cfce7 0 3 72732601 0x455cfb9 0 1 71221548 0x43ec12c 0 4 73314080 0x45eaf20 0 5 72496396 0x452350c 0 0 74076785 0x46a5271 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000014 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314082 0x45eaf22 0 5 72496399 0x452350f 0 0 74076788 0x46a5274 0 2 73202922 0x45cfcea 0 3 72732604 0x455cfbc 0 1 71221551 0x43ec12f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000024 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314113 0x45eaf41 0 5 72496429 0x452352d 0 0 74076818 0x46a5292 0 2 73202952 0x45cfd08 0 3 72732634 0x455cfda 0 1 71221581 0x43ec14d 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000036 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496282 0x452349a 0 0 74076671 0x46a51ff 0 2 73202805 0x45cfc75 0 3 72732487 0x455cf47 0 1 71221434 0x43ec0ba 0 4 73313966 0x45eaeae 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000051 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732531 0x455cf73 0 1 71221478 0x43ec0e6 0 4 73314010 0x45eaeda 0 5 72496326 0x45234c6 0 0 74076716 0x46a522c 0 2 73202849 0x45cfca1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000059 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076849 0x46a52b1 0 2 73202983 0x45cfd27 0 3 72732665 0x455cff9 0 1 71221612 0x43ec16c 0 4 73314144 0x45eaf60 0 5 72496460 0x452354c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000007 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496430 0x452352e 0 0 74076821 0x46a5295 0 2 73202955 0x45cfd0b 0 3 72732635 0x455cfdb 0 1 71221582 0x43ec14e 0 4 73314114 0x45eaf42 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000005 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314092 0x45eaf2c 0 5 72496408 0x4523518 0 0 74076797 0x46a527d 0 2 73202931 0x45cfcf3 0 3 72732613 0x455cfc5 0 1 71221560 0x43ec138 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000044 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202863 0x45cfcaf 0 3 72732545 0x455cf81 0 1 71221492 0x43ec0f4 0 4 73314025 0x45eaee9 0 5 72496341 0x45234d5 0 0 74076731 0x46a523b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000020 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076827 0x46a529b 0 2 73202961 0x45cfd11 0 3 72732643 0x455cfe3 0 1 71221592 0x43ec158 0 4 73314124 0x45eaf4c 0 5 72496440 0x4523538 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000014 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076923 0x46a52fb 0 2 73203058 0x45cfd72 0 3 72732740 0x455d044 0 1 71221688 0x43ec1b8 0 4 73314220 0x45eafac 0 5 72496536 0x4523598 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000049 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076680 0x46a5208 0 2 73202815 0x45cfc7f 0 3 72732497 0x455cf51 0 1 71221446 0x43ec0c6 0 4 73313978 0x45eaeba 0 5 72496294 0x45234a6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000054 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732691 0x455d013 0 1 71221639 0x43ec187 0 4 73314172 0x45eaf7c 0 5 72496489 0x4523569 0 0 74076878 0x46a52ce 0 2 73203013 0x45cfd45 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000025 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496391 0x4523507 0 0 74076780 0x46a526c 0 2 73202914 0x45cfce2 0 3 72732596 0x455cfb4 0 1 71221544 0x43ec128 0 4 73314076 0x45eaf1c 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000006 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496335 0x45234cf 0 0 74076724 0x46a5234 0 2 73202858 0x45cfcaa 0 3 72732540 0x455cf7c 0 1 71221487 0x43ec0ef 0 4 73314019 0x45eaee3 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000056 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221602 0x43ec162 0 4 73314137 0x45eaf59 0 5 72496453 0x4523545 0 0 74076842 0x46a52aa 0 2 73202978 0x45cfd22 0 3 72732660 0x455cff4 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000020 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496540 0x452359c 0 0 74076929 0x46a5301 0 2 73203063 0x45cfd77 0 3 72732747 0x455d04b 0 1 71221694 0x43ec1be 0 4 73314225 0x45eafb1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000026 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221683 0x43ec1b3 0 4 73314215 0x45eafa7 0 5 72496531 0x4523593 0 0 74076920 0x46a52f8 0 2 73203054 0x45cfd6e 0 3 72732736 0x455d040 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000062 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732493 0x455cf4d 0 1 71221440 0x43ec0c0 0 4 73313972 0x45eaeb4 0 5 72496289 0x45234a1 0 0 74076679 0x46a5207 0 2 73202813 0x45cfc7d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000001 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202860 0x45cfcac 0 3 72732542 0x455cf7e 0 1 71221489 0x43ec0f1 0 4 73314021 0x45eaee5 0 5 72496337 0x45234d1 0 0 74076726 0x46a5236 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000002 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076725 0x46a5235 0 2 73202859 0x45cfcab 0 3 72732541 0x455cf7d 0 1 71221488 0x43ec0f0 0 4 73314020 0x45eaee4 0 5 72496336 0x45234d0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000062 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496353 0x45234e1 0 0 74076743 0x46a5247 0 2 73202877 0x45cfcbd 0 3 72732559 0x455cf8f 0 1 71221505 0x43ec101 0 4 73314037 0x45eaef5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000006 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314228 0x45eafb4 0 5 72496544 0x45235a0 0 0 74076934 0x46a5306 0 2 73203068 0x45cfd7c 0 3 72732750 0x455d04e 0 1 71221697 0x43ec1c1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000002 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202760 0x45cfc48 0 3 72732442 0x455cf1a 0 1 71221389 0x43ec08d 0 4 73313921 0x45eae81 0 5 72496237 0x452346d 0 0 74076626 0x46a51d2 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000046 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221623 0x43ec177 0 4 73314155 0x45eaf6b 0 5 72496471 0x4523557 0 0 74076861 0x46a52bd 0 2 73202995 0x45cfd33 0 3 72732677 0x455d005 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000009 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221585 0x43ec151 0 4 73314117 0x45eaf45 0 5 72496433 0x4523531 0 0 74076822 0x46a5296 0 2 73202956 0x45cfd0c 0 3 72732638 0x455cfde 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000019 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076835 0x46a52a3 0 2 73202969 0x45cfd19 0 3 72732652 0x455cfec 0 1 71221599 0x43ec15f 0 4 73314131 0x45eaf53 0 5 72496447 0x452353f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000036 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076935 0x46a5307 0 2 73203069 0x45cfd7d 0 3 72732751 0x455d04f 0 1 71221698 0x43ec1c2 0 4 73314230 0x45eafb6 0 5 72496546 0x45235a2 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000058 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314170 0x45eaf7a 0 5 72496486 0x4523566 0 0 74076877 0x46a52cd 0 2 73203012 0x45cfd44 0 3 72732695 0x455d017 0 1 71221641 0x43ec189 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000050 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496354 0x45234e2 0 0 74076744 0x46a5248 0 2 73202878 0x45cfcbe 0 3 72732560 0x455cf90 0 1 71221508 0x43ec104 0 4 73314039 0x45eaef7 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000017 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221586 0x43ec152 0 4 73314118 0x45eaf46 0 5 72496434 0x4523532 0 0 74076823 0x46a5297 0 2 73202957 0x45cfd0d 0 3 72732639 0x455cfdf 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000024 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203029 0x45cfd55 0 3 72732711 0x455d027 0 1 71221659 0x43ec19b 0 4 73314193 0x45eaf91 0 5 72496509 0x452357d 0 0 74076899 0x46a52e3 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000038 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496340 0x45234d4 0 0 74076729 0x46a5239 0 2 73202864 0x45cfcb0 0 3 72732546 0x455cf82 0 1 71221493 0x43ec0f5 0 4 73314024 0x45eaee8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000036 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076730 0x46a523a 0 2 73202865 0x45cfcb1 0 3 72732547 0x455cf83 0 1 71221494 0x43ec0f6 0 4 73314026 0x45eaeea 0 5 72496342 0x45234d6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000060 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202784 0x45cfc60 0 3 72732466 0x455cf32 0 1 71221413 0x43ec0a5 0 4 73313945 0x45eae99 0 5 72496262 0x4523486 0 0 74076651 0x46a51eb 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000015 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313995 0x45eaecb 0 5 72496312 0x45234b8 0 0 74076703 0x46a521f 0 2 73202840 0x45cfc98 0 3 72732522 0x455cf6a 0 1 71221470 0x43ec0de 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000044 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076810 0x46a528a 0 2 73202944 0x45cfd00 0 3 72732628 0x455cfd4 0 1 71221575 0x43ec147 0 4 73314108 0x45eaf3c 0 5 72496424 0x4523528 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000045 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202974 0x45cfd1e 0 3 72732657 0x455cff1 0 1 71221606 0x43ec166 0 4 73314138 0x45eaf5a 0 5 72496455 0x4523547 0 0 74076844 0x46a52ac 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000057 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732626 0x455cfd2 0 1 71221574 0x43ec146 0 4 73314106 0x45eaf3a 0 5 72496422 0x4523526 0 0 74076812 0x46a528c 0 2 73202947 0x45cfd03 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000049 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314166 0x45eaf76 0 5 72496482 0x4523562 0 0 74076871 0x46a52c7 0 2 73203006 0x45cfd3e 0 3 72732689 0x455d011 0 1 71221637 0x43ec185 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000040 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732449 0x455cf21 0 1 71221397 0x43ec095 0 4 73313930 0x45eae8a 0 5 72496247 0x4523477 0 0 74076637 0x46a51dd 0 2 73202772 0x45cfc54 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000031 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732651 0x455cfeb 0 1 71221598 0x43ec15e 0 4 73314130 0x45eaf52 0 5 72496446 0x452353e 0 0 74076836 0x46a52a4 0 2 73202970 0x45cfd1a 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000046 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076659 0x46a51f3 0 2 73202795 0x45cfc6b 0 3 72732478 0x455cf3e 0 1 71221425 0x43ec0b1 0 4 73313957 0x45eaea5 0 5 72496273 0x4523491 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000018 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202868 0x45cfcb4 0 3 72732550 0x455cf86 0 1 71221498 0x43ec0fa 0 4 73314031 0x45eaeef 0 5 72496347 0x45234db 0 0 74076737 0x46a5241 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000047 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732598 0x455cfb6 0 1 71221545 0x43ec129 0 4 73314077 0x45eaf1d 0 5 72496394 0x452350a 0 0 74076783 0x46a526f 0 2 73202917 0x45cfce5 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000031 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314046 0x45eaefe 0 5 72496362 0x45234ea 0 0 74076751 0x46a524f 0 2 73202886 0x45cfcc6 0 3 72732568 0x455cf98 0 1 71221516 0x43ec10c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000040 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732688 0x455d010 0 1 71221636 0x43ec184 0 4 73314169 0x45eaf79 0 5 72496485 0x4523565 0 0 74076874 0x46a52ca 0 2 73203008 0x45cfd40 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000035 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314197 0x45eaf95 0 5 72496514 0x4523582 0 0 74076904 0x46a52e8 0 2 73203039 0x45cfd5f 0 3 72732722 0x455d032 0 1 71221669 0x43ec1a5 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000061 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221479 0x43ec0e7 0 4 73314011 0x45eaedb 0 5 72496327 0x45234c7 0 0 74076718 0x46a522e 0 2 73202852 0x45cfca4 0 3 72732535 0x455cf77 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000009 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313958 0x45eaea6 0 5 72496274 0x4523492 0 0 74076663 0x46a51f7 0 2 73202797 0x45cfc6d 0 3 72732479 0x455cf3f 0 1 71221426 0x43ec0b2 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000029 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076856 0x46a52b8 0 2 73202990 0x45cfd2e 0 3 72732672 0x455d000 0 1 71221619 0x43ec173 0 4 73314152 0x45eaf68 0 5 72496468 0x4523554 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000053 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314059 0x45eaf0b 0 5 72496375 0x45234f7 0 0 74076764 0x46a525c 0 2 73202898 0x45cfcd2 0 3 72732580 0x455cfa4 0 1 71221528 0x43ec118 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000034 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202885 0x45cfcc5 0 3 72732567 0x455cf97 0 1 71221515 0x43ec10b 0 4 73314048 0x45eaf00 0 5 72496364 0x45234ec 0 0 74076753 0x46a5251 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000048 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221482 0x43ec0ea 0 4 73314014 0x45eaede 0 5 72496331 0x45234cb 0 0 74076720 0x46a5230 0 2 73202854 0x45cfca6 0 3 72732537 0x455cf79 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000024 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202949 0x45cfd05 0 3 72732632 0x455cfd8 0 1 71221579 0x43ec14b 0 4 73314111 0x45eaf3f 0 5 72496426 0x452352a 0 0 74076815 0x46a528f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000034 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202778 0x45cfc5a 0 3 72732462 0x455cf2e 0 1 71221410 0x43ec0a2 0 4 73313942 0x45eae96 0 5 72496259 0x4523483 0 0 74076648 0x46a51e8 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000010 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202993 0x45cfd31 0 3 72732675 0x455d003 0 1 71221622 0x43ec176 0 4 73314154 0x45eaf6a 0 5 72496470 0x4523556 0 0 74076859 0x46a52bb 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000019 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221542 0x43ec126 0 4 73314074 0x45eaf1a 0 5 72496390 0x4523506 0 0 74076779 0x46a526b 0 2 73202913 0x45cfce1 0 3 72732595 0x455cfb3 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000058 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496392 0x4523508 0 0 74076781 0x46a526d 0 2 73202915 0x45cfce3 0 3 72732597 0x455cfb5 0 1 71221543 0x43ec127 0 4 73314075 0x45eaf1b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000034 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202946 0x45cfd02 0 3 72732629 0x455cfd5 0 1 71221576 0x43ec148 0 4 73314109 0x45eaf3d 0 5 72496425 0x4523529 0 0 74076814 0x46a528e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000051 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203053 0x45cfd6d 0 3 72732735 0x455d03f 0 1 71221682 0x43ec1b2 0 4 73314214 0x45eafa6 0 5 72496530 0x4523592 0 0 74076919 0x46a52f7 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000003 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202888 0x45cfcc8 0 3 72732570 0x455cf9a 0 1 71221517 0x43ec10d 0 4 73314049 0x45eaf01 0 5 72496365 0x45234ed 0 0 74076755 0x46a5253 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000035 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732599 0x455cfb7 0 1 71221547 0x43ec12b 0 4 73314079 0x45eaf1f 0 5 72496395 0x452350b 0 0 74076784 0x46a5270 0 2 73202918 0x45cfce6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000047 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203027 0x45cfd53 0 3 72732710 0x455d026 0 1 71221658 0x43ec19a 0 4 73314192 0x45eaf90 0 5 72496508 0x452357c 0 0 74076897 0x46a52e1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000058 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076685 0x46a520d 0 2 73202819 0x45cfc83 0 3 72732502 0x455cf56 0 1 71221451 0x43ec0cb 0 4 73313984 0x45eaec0 0 5 72496300 0x45234ac 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000014 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732549 0x455cf85 0 1 71221496 0x43ec0f8 0 4 73314028 0x45eaeec 0 5 72496344 0x45234d8 0 0 74076733 0x46a523d 0 2 73202867 0x45cfcb3 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000000 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076665 0x46a51f9 0 2 73202799 0x45cfc6f 0 3 72732481 0x455cf41 0 1 71221428 0x43ec0b4 0 4 73313960 0x45eaea8 0 5 72496276 0x4523494 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000060 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496359 0x45234e7 0 0 74076748 0x46a524c 0 2 73202882 0x45cfcc2 0 3 72732564 0x455cf94 0 1 71221511 0x43ec107 0 4 73314043 0x45eaefb 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000003 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314216 0x45eafa8 0 5 72496532 0x4523594 0 0 74076921 0x46a52f9 0 2 73203055 0x45cfd6f 0 3 72732737 0x455d041 0 1 71221684 0x43ec1b4 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000019 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732745 0x455d049 0 1 71221692 0x43ec1bc 0 4 73314224 0x45eafb0 0 5 72496541 0x452359d 0 0 74076930 0x46a5302 0 2 73203064 0x45cfd78 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000020 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202933 0x45cfcf5 0 3 72732615 0x455cfc7 0 1 71221562 0x43ec13a 0 4 73314094 0x45eaf2e 0 5 72496410 0x452351a 0 0 74076799 0x46a527f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000035 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221437 0x43ec0bd 0 4 73313969 0x45eaeb1 0 5 72496285 0x452349d 0 0 74076674 0x46a5202 0 2 73202808 0x45cfc78 0 3 72732490 0x455cf4a 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000029 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202896 0x45cfcd0 0 3 72732578 0x455cfa2 0 1 71221525 0x43ec115 0 4 73314057 0x45eaf09 0 5 72496373 0x45234f5 0 0 74076762 0x46a525a 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000017 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221417 0x43ec0a9 0 4 73313952 0x45eaea0 0 5 72496268 0x452348c 0 0 74076657 0x46a51f1 0 2 73202792 0x45cfc68 0 3 72732474 0x455cf3a 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000008 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496328 0x45234c8 0 0 74076719 0x46a522f 0 2 73202853 0x45cfca5 0 3 72732536 0x455cf78 0 1 71221483 0x43ec0eb 0 4 73314016 0x45eaee0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000030 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496406 0x4523516 0 0 74076795 0x46a527b 0 2 73202929 0x45cfcf1 0 3 72732611 0x455cfc3 0 1 71221558 0x43ec136 0 4 73314090 0x45eaf2a 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000063 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314068 0x45eaf14 0 5 72496384 0x4523500 0 0 74076773 0x46a5265 0 2 73202908 0x45cfcdc 0 3 72732590 0x455cfae 0 1 71221537 0x43ec121 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000028 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202829 0x45cfc8d 0 3 72732512 0x455cf60 0 1 71221461 0x43ec0d5 0 4 73313996 0x45eaecc 0 5 72496314 0x45234ba 0 0 74076704 0x46a5220 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000041 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496383 0x45234ff 0 0 74076775 0x46a5267 0 2 73202909 0x45cfcdd 0 3 72732593 0x455cfb1 0 1 71221541 0x43ec125 0 4 73314073 0x45eaf19 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000062 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203042 0x45cfd62 0 3 72732725 0x455d035 0 1 71221673 0x43ec1a9 0 4 73314207 0x45eaf9f 0 5 72496523 0x452358b 0 0 74076912 0x46a52f0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000035 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076715 0x46a522b 0 2 73202850 0x45cfca2 0 3 72732533 0x455cf75 0 1 71221481 0x43ec0e9 0 4 73314013 0x45eaedd 0 5 72496330 0x45234ca 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000037 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221402 0x43ec09a 0 4 73313935 0x45eae8f 0 5 72496252 0x452347c 0 0 74076642 0x46a51e2 0 2 73202776 0x45cfc58 0 3 72732458 0x455cf2a 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000004 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496275 0x4523493 0 0 74076664 0x46a51f8 0 2 73202798 0x45cfc6e 0 3 72732480 0x455cf40 0 1 71221427 0x43ec0b3 0 4 73313959 0x45eaea7 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000018 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496479 0x452355f 0 0 74076868 0x46a52c4 0 2 73203002 0x45cfd3a 0 3 72732684 0x455d00c 0 1 71221631 0x43ec17f 0 4 73314163 0x45eaf73 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000063 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314133 0x45eaf55 0 5 72496450 0x4523542 0 0 74076839 0x46a52a7 0 2 73202975 0x45cfd1f 0 3 72732658 0x455cff2 0 1 71221605 0x43ec165 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000006 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076808 0x46a5288 0 2 73202942 0x45cfcfe 0 3 72732624 0x455cfd0 0 1 71221571 0x43ec143 0 4 73314103 0x45eaf37 0 5 72496419 0x4523523 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000027 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221520 0x43ec110 0 4 73314052 0x45eaf04 0 5 72496368 0x45234f0 0 0 74076757 0x46a5255 0 2 73202892 0x45cfccc 0 3 72732574 0x455cf9e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000021 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496474 0x452355a 0 0 74076863 0x46a52bf 0 2 73202999 0x45cfd37 0 3 72732681 0x455d009 0 1 71221628 0x43ec17c 0 4 73314161 0x45eaf71 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000018 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313970 0x45eaeb2 0 5 72496286 0x452349e 0 0 74076676 0x46a5204 0 2 73202810 0x45cfc7a 0 3 72732492 0x455cf4c 0 1 71221439 0x43ec0bf 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000055 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732534 0x455cf76 0 1 71221484 0x43ec0ec 0 4 73314017 0x45eaee1 0 5 72496333 0x45234cd 0 0 74076722 0x46a5232 0 2 73202856 0x45cfca8 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000033 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076875 0x46a52cb 0 2 73203009 0x45cfd41 0 3 72732692 0x455d014 0 1 71221638 0x43ec186 0 4 73314171 0x45eaf7b 0 5 72496488 0x4523568 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000061 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221445 0x43ec0c5 0 4 73313977 0x45eaeb9 0 5 72496293 0x45234a5 0 0 74076684 0x46a520c 0 2 73202817 0x45cfc81 0 3 72732499 0x455cf53 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000050 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496415 0x452351f 0 0 74076804 0x46a5284 0 2 73202938 0x45cfcfa 0 3 72732621 0x455cfcd 0 1 71221568 0x43ec140 0 4 73314100 0x45eaf34 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000047 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732648 0x455cfe8 0 1 71221595 0x43ec15b 0 4 73314127 0x45eaf4f 0 5 72496443 0x452353b 0 0 74076832 0x46a52a0 0 2 73202966 0x45cfd16 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000046 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732631 0x455cfd7 0 1 71221578 0x43ec14a 0 4 73314110 0x45eaf3e 0 5 72496427 0x452352b 0 0 74076816 0x46a5290 0 2 73202950 0x45cfd06 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000020 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202768 0x45cfc50 0 3 72732452 0x455cf24 0 1 71221401 0x43ec099 0 4 73313933 0x45eae8d 0 5 72496250 0x452347a 0 0 74076639 0x46a51df 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000034 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221703 0x43ec1c7 0 4 73314235 0x45eafbb 0 5 72496552 0x45235a8 0 0 74076942 0x46a530e 0 2 73203075 0x45cfd83 0 3 72732759 0x455d057 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000006 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202992 0x45cfd30 0 3 72732674 0x455d002 0 1 71221621 0x43ec175 0 4 73314153 0x45eaf69 0 5 72496469 0x4523555 0 0 74076858 0x46a52ba 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000007 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202905 0x45cfcd9 0 3 72732589 0x455cfad 0 1 71221535 0x43ec11f 0 4 73314070 0x45eaf16 0 5 72496387 0x4523503 0 0 74076777 0x46a5269 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000015 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202890 0x45cfcca 0 3 72732572 0x455cf9c 0 1 71221521 0x43ec111 0 4 73314053 0x45eaf05 0 5 72496369 0x45234f1 0 0 74076758 0x46a5256 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000052 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314157 0x45eaf6d 0 5 72496473 0x4523559 0 0 74076862 0x46a52be 0 2 73202998 0x45cfd36 0 3 72732682 0x455d00a 0 1 71221629 0x43ec17d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000023 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076759 0x46a5257 0 2 73202894 0x45cfcce 0 3 72732576 0x455cfa0 0 1 71221523 0x43ec113 0 4 73314056 0x45eaf08 0 5 72496372 0x45234f4 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000029 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076749 0x46a524d 0 2 73202883 0x45cfcc3 0 3 72732566 0x455cf96 0 1 71221513 0x43ec109 0 4 73314045 0x45eaefd 0 5 72496361 0x45234e9 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000041 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732754 0x455d052 0 1 71221701 0x43ec1c5 0 4 73314233 0x45eafb9 0 5 72496549 0x45235a5 0 0 74076938 0x46a530a 0 2 73203073 0x45cfd81 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000013 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076914 0x46a52f2 0 2 73203049 0x45cfd69 0 3 72732731 0x455d03b 0 1 71221677 0x43ec1ad 0 4 73314209 0x45eafa1 0 5 72496526 0x452358e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000027 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496244 0x4523474 0 0 74076633 0x46a51d9 0 2 73202769 0x45cfc51 0 3 72732451 0x455cf23 0 1 71221399 0x43ec097 0 4 73313932 0x45eae8c 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000018 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221573 0x43ec145 0 4 73314105 0x45eaf39 0 5 72496421 0x4523525 0 0 74076811 0x46a528b 0 2 73202945 0x45cfd01 0 3 72732627 0x455cfd3 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000030 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076941 0x46a530d 0 2 73203076 0x45cfd84 0 3 72732758 0x455d056 0 1 71221707 0x43ec1cb 0 4 73314240 0x45eafc0 0 5 72496556 0x45235ac 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000044 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496525 0x452358d 0 0 74076915 0x46a52f3 0 2 73203050 0x45cfd6a 0 3 72732732 0x455d03c 0 1 71221679 0x43ec1af 0 4 73314211 0x45eafa3 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000008 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221556 0x43ec134 0 4 73314088 0x45eaf28 0 5 72496404 0x4523514 0 0 74076793 0x46a5279 0 2 73202927 0x45cfcef 0 3 72732609 0x455cfc1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000025 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314150 0x45eaf66 0 5 72496467 0x4523553 0 0 74076857 0x46a52b9 0 2 73202991 0x45cfd2f 0 3 72732673 0x455d001 0 1 71221620 0x43ec174 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000032 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496537 0x4523599 0 0 74076926 0x46a52fe 0 2 73203061 0x45cfd75 0 3 72732743 0x455d047 0 1 71221690 0x43ec1ba 0 4 73314222 0x45eafae 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000052 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496543 0x452359f 0 0 74076932 0x46a5304 0 2 73203066 0x45cfd7a 0 3 72732748 0x455d04c 0 1 71221695 0x43ec1bf 0 4 73314227 0x45eafb3 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000005 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202766 0x45cfc4e 0 3 72732448 0x455cf20 0 1 71221395 0x43ec093 0 4 73313927 0x45eae87 0 5 72496243 0x4523473 0 0 74076632 0x46a51d8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000039 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732585 0x455cfa9 0 1 71221532 0x43ec11c 0 4 73314064 0x45eaf10 0 5 72496380 0x45234fc 0 0 74076769 0x46a5261 0 2 73202903 0x45cfcd7 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000031 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314204 0x45eaf9c 0 5 72496521 0x4523589 0 0 74076909 0x46a52ed 0 2 73203045 0x45cfd65 0 3 72732728 0x455d038 0 1 71221676 0x43ec1ac 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000015 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314147 0x45eaf63 0 5 72496463 0x452354f 0 0 74076852 0x46a52b4 0 2 73202987 0x45cfd2b 0 3 72732669 0x455cffd 0 1 71221616 0x43ec170 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000039 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732655 0x455cfef 0 1 71221604 0x43ec164 0 4 73314136 0x45eaf58 0 5 72496452 0x4523544 0 0 74076841 0x46a52a9 0 2 73202977 0x45cfd21 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000049 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202824 0x45cfc88 0 3 72732506 0x455cf5a 0 1 71221453 0x43ec0cd 0 4 73313985 0x45eaec1 0 5 72496301 0x45234ad 0 0 74076690 0x46a5212 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000051 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076675 0x46a5203 0 2 73202809 0x45cfc79 0 3 72732491 0x455cf4b 0 1 71221438 0x43ec0be 0 4 73313971 0x45eaeb3 0 5 72496288 0x45234a0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000033 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496318 0x45234be 0 0 74076708 0x46a5224 0 2 73202843 0x45cfc9b 0 3 72732526 0x455cf6e 0 1 71221472 0x43ec0e0 0 4 73314004 0x45eaed4 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000009 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314055 0x45eaf07 0 5 72496371 0x45234f3 0 0 74076761 0x46a5259 0 2 73202895 0x45cfccf 0 3 72732577 0x455cfa1 0 1 71221524 0x43ec114 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000047 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221507 0x43ec103 0 4 73314040 0x45eaef8 0 5 72496356 0x45234e4 0 0 74076746 0x46a524a 0 2 73202880 0x45cfcc0 0 3 72732562 0x455cf92 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000001 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202943 0x45cfcff 0 3 72732625 0x455cfd1 0 1 71221572 0x43ec144 0 4 73314104 0x45eaf38 0 5 72496420 0x4523524 0 0 74076809 0x46a5289 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000053 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221414 0x43ec0a6 0 4 73313948 0x45eae9c 0 5 72496265 0x4523489 0 0 74076655 0x46a51ef 0 2 73202791 0x45cfc67 0 3 72732475 0x455cf3b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000002 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314162 0x45eaf72 0 5 72496478 0x452355e 0 0 74076867 0x46a52c3 0 2 73203001 0x45cfd39 0 3 72732683 0x455d00b 0 1 71221630 0x43ec17e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000015 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076883 0x46a52d3 0 2 73203017 0x45cfd49 0 3 72732701 0x455d01d 0 1 71221648 0x43ec190 0 4 73314180 0x45eaf84 0 5 72496496 0x4523570 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000005 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076933 0x46a5305 0 2 73203067 0x45cfd7b 0 3 72732749 0x455d04d 0 1 71221696 0x43ec1c0 0 4 73314229 0x45eafb5 0 5 72496545 0x45235a1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000022 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202996 0x45cfd34 0 3 72732678 0x455d006 0 1 71221625 0x43ec179 0 4 73314158 0x45eaf6e 0 5 72496476 0x452355c 0 0 74076865 0x46a52c1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000022 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202781 0x45cfc5d 0 3 72732464 0x455cf30 0 1 71221412 0x43ec0a4 0 4 73313944 0x45eae98 0 5 72496260 0x4523484 0 0 74076649 0x46a51e9 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000026 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076790 0x46a5276 0 2 73202925 0x45cfced 0 3 72732607 0x455cfbf 0 1 71221554 0x43ec132 0 4 73314086 0x45eaf26 0 5 72496402 0x4523512 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000054 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203043 0x45cfd63 0 3 72732726 0x455d036 0 1 71221674 0x43ec1aa 0 4 73314208 0x45eafa0 0 5 72496524 0x452358c 0 0 74076913 0x46a52f1 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000042 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203062 0x45cfd76 0 3 72732744 0x455d048 0 1 71221691 0x43ec1bb 0 4 73314223 0x45eafaf 0 5 72496539 0x452359b 0 0 74076928 0x46a5300 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000004 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496461 0x452354d 0 0 74076851 0x46a52b3 0 2 73202986 0x45cfd2a 0 3 72732667 0x455cffb 0 1 71221614 0x43ec16e 0 4 73314146 0x45eaf62 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000063 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732755 0x455d053 0 1 71221702 0x43ec1c6 0 4 73314234 0x45eafba 0 5 72496550 0x45235a6 0 0 74076939 0x46a530b 0 2 73203072 0x45cfd80 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000043 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732679 0x455d007 0 1 71221626 0x43ec17a 0 4 73314159 0x45eaf6f 0 5 72496475 0x452355b 0 0 74076864 0x46a52c0 0 2 73203000 0x45cfd38 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000012 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496497 0x4523571 0 0 74076886 0x46a52d6 0 2 73203020 0x45cfd4c 0 3 72732702 0x455d01e 0 1 71221650 0x43ec192 0 4 73314182 0x45eaf86 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000056 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314067 0x45eaf13 0 5 72496386 0x4523502 0 0 74076776 0x46a5268 0 2 73202911 0x45cfcdf 0 3 72732594 0x455cfb2 0 1 71221540 0x43ec124 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000031 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732579 0x455cfa3 0 1 71221526 0x43ec116 0 4 73314058 0x45eaf0a 0 5 72496374 0x45234f6 0 0 74076763 0x46a525b 0 2 73202897 0x45cfcd1 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000021 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732510 0x455cf5e 0 1 71221459 0x43ec0d3 0 4 73313992 0x45eaec8 0 5 72496308 0x45234b4 0 0 74076697 0x46a5219 0 2 73202834 0x45cfc92 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000014 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221449 0x43ec0c9 0 4 73313982 0x45eaebe 0 5 72496298 0x45234aa 0 0 74076688 0x46a5210 0 2 73202821 0x45cfc85 0 3 72732503 0x455cf57 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000011 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496493 0x452356d 0 0 74076882 0x46a52d2 0 2 73203016 0x45cfd48 0 3 72732699 0x455d01b 0 1 71221646 0x43ec18e 0 4 73314178 0x45eaf82 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000017 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732606 0x455cfbe 0 1 71221553 0x43ec131 0 4 73314085 0x45eaf25 0 5 72496401 0x4523511 0 0 74076791 0x46a5277 0 2 73202924 0x45cfcec 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000024 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202807 0x45cfc77 0 3 72732489 0x455cf49 0 1 71221436 0x43ec0bc 0 4 73313968 0x45eaeb0 0 5 72496284 0x452349c 0 0 74076673 0x46a5201 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000055 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313946 0x45eae9a 0 5 72496263 0x4523487 0 0 74076652 0x46a51ec 0 2 73202786 0x45cfc62 0 3 72732469 0x455cf35 0 1 71221418 0x43ec0aa 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000057 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221670 0x43ec1a6 0 4 73314202 0x45eaf9a 0 5 72496518 0x4523586 0 0 74076907 0x46a52eb 0 2 73203041 0x45cfd61 0 3 72732723 0x455d033 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000008 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076631 0x46a51d7 0 2 73202765 0x45cfc4d 0 3 72732447 0x455cf1f 0 1 71221394 0x43ec092 0 4 73313926 0x45eae86 0 5 72496242 0x4523472 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000007 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496302 0x45234ae 0 0 74076691 0x46a5213 0 2 73202825 0x45cfc89 0 3 72732508 0x455cf5c 0 1 71221457 0x43ec0d1 0 4 73313989 0x45eaec5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000059 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314203 0x45eaf9b 0 5 72496519 0x4523587 0 0 74076908 0x46a52ec 0 2 73203044 0x45cfd64 0 3 72732727 0x455d037 0 1 71221675 0x43ec1ab 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000032 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076833 0x46a52a1 0 2 73202967 0x45cfd17 0 3 72732649 0x455cfe9 0 1 71221596 0x43ec15c 0 4 73314128 0x45eaf50 0 5 72496445 0x452353d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000055 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076766 0x46a525e 0 2 73202900 0x45cfcd4 0 3 72732582 0x455cfa6 0 1 71221529 0x43ec119 0 4 73314061 0x45eaf0d 0 5 72496377 0x45234f9 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000000 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221649 0x43ec191 0 4 73314181 0x45eaf85 0 5 72496498 0x4523572 0 0 74076887 0x46a52d7 0 2 73203021 0x45cfd4d 0 3 72732704 0x455d020 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000049 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732756 0x455d054 0 1 71221706 0x43ec1ca 0 4 73314239 0x45eafbf 0 5 72496555 0x45235ab 0 0 74076945 0x46a5311 0 2 73203079 0x45cfd87 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000021 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313964 0x45eaeac 0 5 72496280 0x4523498 0 0 74076669 0x46a51fd 0 2 73202803 0x45cfc73 0 3 72732485 0x455cf45 0 1 71221432 0x43ec0b8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000004 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313990 0x45eaec6 0 5 72496311 0x45234b7 0 0 74076701 0x46a521d 0 2 73202835 0x45cfc93 0 3 72732516 0x455cf64 0 1 71221463 0x43ec0d7 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000024 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313987 0x45eaec3 0 5 72496304 0x45234b0 0 0 74076695 0x46a5217 0 2 73202833 0x45cfc91 0 3 72732518 0x455cf66 0 1 71221465 0x43ec0d9 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000007 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202762 0x45cfc4a 0 3 72732444 0x455cf1c 0 1 71221391 0x43ec08f 0 4 73313924 0x45eae84 0 5 72496240 0x4523470 0 0 74076629 0x46a51d5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000042 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313981 0x45eaebd 0 5 72496297 0x45234a9 0 0 74076687 0x46a520f 0 2 73202822 0x45cfc86 0 3 72732504 0x455cf58 0 1 71221452 0x43ec0cc 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000010 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202839 0x45cfc97 0 3 72732521 0x455cf69 0 1 71221468 0x43ec0dc 0 4 73314001 0x45eaed1 0 5 72496317 0x45234bd 0 0 74076706 0x46a5222 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000038 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314093 0x45eaf2d 0 5 72496409 0x4523519 0 0 74076798 0x46a527e 0 2 73202932 0x45cfcf4 0 3 72732614 0x455cfc6 0 1 71221561 0x43ec139 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000017 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076922 0x46a52fa 0 2 73203057 0x45cfd71 0 3 72732739 0x455d043 0 1 71221687 0x43ec1b7 0 4 73314219 0x45eafab 0 5 72496535 0x4523597 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000052 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732500 0x455cf54 0 1 71221448 0x43ec0c8 0 4 73313980 0x45eaebc 0 5 72496296 0x45234a8 0 0 74076686 0x46a520e 0 2 73202820 0x45cfc84 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000032 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202934 0x45cfcf6 0 3 72732616 0x455cfc8 0 1 71221564 0x43ec13c 0 4 73314097 0x45eaf31 0 5 72496414 0x452351e 0 0 74076803 0x46a5283 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000023 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221460 0x43ec0d4 0 4 73313994 0x45eaeca 0 5 72496310 0x45234b6 0 0 74076698 0x46a521a 0 2 73202831 0x45cfc8f 0 3 72732514 0x455cf62 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000021 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203032 0x45cfd58 0 3 72732714 0x455d02a 0 1 71221662 0x43ec19e 0 4 73314196 0x45eaf94 0 5 72496512 0x4523580 0 0 74076901 0x46a52e5 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000045 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496393 0x4523509 0 0 74076782 0x46a526e 0 2 73202916 0x45cfce4 0 3 72732600 0x455cfb8 0 1 71221546 0x43ec12a 0 4 73314078 0x45eaf1e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000016 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496465 0x4523551 0 0 74076855 0x46a52b7 0 2 73202989 0x45cfd2d 0 3 72732671 0x455cfff 0 1 71221618 0x43ec172 0 4 73314151 0x45eaf67 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000008 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221656 0x43ec198 0 4 73314188 0x45eaf8c 0 5 72496505 0x4523579 0 0 74076894 0x46a52de 0 2 73203030 0x45cfd56 0 3 72732712 0x455d028 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000009 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732565 0x455cf95 0 1 71221512 0x43ec108 0 4 73314044 0x45eaefc 0 5 72496360 0x45234e8 0 0 74076750 0x46a524e 0 2 73202884 0x45cfcc4 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000052 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221499 0x43ec0fb 0 4 73314032 0x45eaef0 0 5 72496349 0x45234dd 0 0 74076738 0x46a5242 0 2 73202872 0x45cfcb8 0 3 72732554 0x455cf8a 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000022 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732543 0x455cf7f 0 1 71221490 0x43ec0f2 0 4 73314022 0x45eaee6 0 5 72496338 0x45234d2 0 0 74076727 0x46a5237 0 2 73202861 0x45cfcad 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000016 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076890 0x46a52da 0 2 73203024 0x45cfd50 0 3 72732706 0x455d022 0 1 71221653 0x43ec195 0 4 73314185 0x45eaf89 0 5 72496501 0x4523575 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000050 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076876 0x46a52cc 0 2 73203011 0x45cfd43 0 3 72732694 0x455d016 0 1 71221642 0x43ec18a 0 4 73314174 0x45eaf7e 0 5 72496491 0x452356b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000043 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732529 0x455cf71 0 1 71221476 0x43ec0e4 0 4 73314008 0x45eaed8 0 5 72496324 0x45234c4 0 0 74076713 0x46a5229 0 2 73202847 0x45cfc9f 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000004 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221527 0x43ec117 0 4 73314060 0x45eaf0c 0 5 72496376 0x45234f8 0 0 74076765 0x46a525d 0 2 73202899 0x45cfcd3 0 3 72732581 0x455cfa5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000038 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496551 0x45235a7 0 0 74076940 0x46a530c 0 2 73203074 0x45cfd82 0 3 72732757 0x455d055 0 1 71221705 0x43ec1c9 0 4 73314238 0x45eafbe 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000037 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076903 0x46a52e7 0 2 73203037 0x45cfd5d 0 3 72732720 0x455d030 0 1 71221667 0x43ec1a3 0 4 73314200 0x45eaf98 0 5 72496516 0x4523584 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000003 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221455 0x43ec0cf 0 4 73313988 0x45eaec4 0 5 72496305 0x45234b1 0 0 74076694 0x46a5216 0 2 73202830 0x45cfc8e 0 3 72732513 0x455cf61 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000002 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496502 0x4523576 0 0 74076891 0x46a52db 0 2 73203026 0x45cfd52 0 3 72732709 0x455d025 0 1 71221657 0x43ec199 0 4 73314189 0x45eaf8d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000039 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496313 0x45234b9 0 0 74076702 0x46a521e 0 2 73202838 0x45cfc96 0 3 72732519 0x455cf67 0 1 71221466 0x43ec0da 0 4 73313999 0x45eaecf 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000009 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203056 0x45cfd70 0 3 72732738 0x455d042 0 1 71221686 0x43ec1b6 0 4 73314218 0x45eafaa 0 5 72496534 0x4523596 0 0 74076925 0x46a52fd 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000060 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496400 0x4523510 0 0 74076789 0x46a5275 0 2 73202923 0x45cfceb 0 3 72732605 0x455cfbd 0 1 71221552 0x43ec130 0 4 73314084 0x45eaf24 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000022 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496538 0x452359a 0 0 74076927 0x46a52ff 0 2 73203060 0x45cfd74 0 3 72732742 0x455d046 0 1 71221689 0x43ec1b9 0 4 73314221 0x45eafad 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000041 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202973 0x45cfd1d 0 3 72732656 0x455cff0 0 1 71221603 0x43ec163 0 4 73314135 0x45eaf57 0 5 72496454 0x4523546 0 0 74076843 0x46a52ab 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000012 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313928 0x45eae88 0 5 72496245 0x4523475 0 0 74076634 0x46a51da 0 2 73202770 0x45cfc52 0 3 72732453 0x455cf25 0 1 71221403 0x43ec09b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000049 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732584 0x455cfa8 0 1 71221531 0x43ec11b 0 4 73314063 0x45eaf0f 0 5 72496379 0x45234fb 0 0 74076768 0x46a5260 0 2 73202902 0x45cfcd6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000061 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732703 0x455d01f 0 1 71221651 0x43ec193 0 4 73314183 0x45eaf87 0 5 72496499 0x4523573 0 0 74076889 0x46a52d9 0 2 73203023 0x45cfd4f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000043 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203031 0x45cfd57 0 3 72732713 0x455d029 0 1 71221661 0x43ec19d 0 4 73314194 0x45eaf92 0 5 72496510 0x452357e 0 0 74076898 0x46a52e2 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000029 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203025 0x45cfd51 0 3 72732707 0x455d023 0 1 71221654 0x43ec196 0 4 73314186 0x45eaf8a 0 5 72496503 0x4523577 0 0 74076892 0x46a52dc 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000048 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221671 0x43ec1a7 0 4 73314206 0x45eaf9e 0 5 72496522 0x452358a 0 0 74076911 0x46a52ef 0 2 73203047 0x45cfd67 0 3 72732729 0x455d039 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000058 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496547 0x45235a3 0 0 74076937 0x46a5309 0 2 73203071 0x45cfd7f 0 3 72732753 0x455d051 0 1 71221700 0x43ec1c4 0 4 73314232 0x45eafb8 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000054 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221443 0x43ec0c3 0 4 73313975 0x45eaeb7 0 5 72496291 0x45234a3 0 0 74076681 0x46a5209 0 2 73202816 0x45cfc80 0 3 72732498 0x455cf52 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000050 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221400 0x43ec098 0 4 73313934 0x45eae8e 0 5 72496251 0x452347b 0 0 74076641 0x46a51e1 0 2 73202774 0x45cfc56 0 3 72732456 0x455cf28 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000035 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202997 0x45cfd35 0 3 72732680 0x455d008 0 1 71221627 0x43ec17b 0 4 73314160 0x45eaf70 0 5 72496477 0x452355d 0 0 74076866 0x46a52c2 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000010 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496398 0x452350e 0 0 74076787 0x46a5273 0 2 73202921 0x45cfce9 0 3 72732603 0x455cfbb 0 1 71221550 0x43ec12e 0 4 73314083 0x45eaf23 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000044 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221633 0x43ec181 0 4 73314167 0x45eaf77 0 5 72496483 0x4523563 0 0 74076872 0x46a52c8 0 2 73203007 0x45cfd3f 0 3 72732690 0x455d012 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000019 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202841 0x45cfc99 0 3 72732523 0x455cf6b 0 1 71221471 0x43ec0df 0 4 73314003 0x45eaed3 0 5 72496320 0x45234c0 0 0 74076709 0x46a5225 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000020 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221497 0x43ec0f9 0 4 73314029 0x45eaeed 0 5 72496345 0x45234d9 0 0 74076735 0x46a523f 0 2 73202870 0x45cfcb6 0 3 72732552 0x455cf88 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000006 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732473 0x455cf39 0 1 71221421 0x43ec0ad 0 4 73313954 0x45eaea2 0 5 72496271 0x452348f 0 0 74076661 0x46a51f5 0 2 73202796 0x45cfc6c 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000015 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313949 0x45eae9d 0 5 72496266 0x452348a 0 0 74076654 0x46a51ee 0 2 73202788 0x45cfc64 0 3 72732470 0x455cf36 0 1 71221419 0x43ec0ab 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000016 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076682 0x46a520a 0 2 73202818 0x45cfc82 0 3 72732501 0x455cf55 0 1 71221447 0x43ec0c7 0 4 73313979 0x45eaebb 0 5 72496295 0x45234a7 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000011 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076828 0x46a529c 0 2 73202962 0x45cfd12 0 3 72732644 0x455cfe4 0 1 71221593 0x43ec159 0 4 73314125 0x45eaf4d 0 5 72496441 0x4523539 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000037 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221580 0x43ec14c 0 4 73314112 0x45eaf40 0 5 72496428 0x452352c 0 0 74076817 0x46a5291 0 2 73202951 0x45cfd07 0 3 72732633 0x455cfd9 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000027 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732532 0x455cf74 0 1 71221480 0x43ec0e8 0 4 73314012 0x45eaedc 0 5 72496329 0x45234c9 0 0 74076717 0x46a522d 0 2 73202851 0x45cfca3 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000014 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496436 0x4523534 0 0 74076825 0x46a5299 0 2 73202959 0x45cfd0f 0 3 72732641 0x455cfe1 0 1 71221588 0x43ec154 0 4 73314120 0x45eaf48 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000048 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496456 0x4523548 0 0 74076845 0x46a52ad 0 2 73202979 0x45cfd23 0 3 72732661 0x455cff5 0 1 71221608 0x43ec168 0 4 73314140 0x45eaf5c 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000016 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314054 0x45eaf06 0 5 72496370 0x45234f2 0 0 74076760 0x46a5258 0 2 73202893 0x45cfccd 0 3 72732575 0x455cf9f 0 1 71221522 0x43ec112 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000048 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732467 0x455cf33 0 1 71221415 0x43ec0a7 0 4 73313951 0x45eae9f 0 5 72496267 0x452348b 0 0 74076656 0x46a51f0 0 2 73202790 0x45cfc66 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000059 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732459 0x455cf2b 0 1 71221407 0x43ec09f 0 4 73313941 0x45eae95 0 5 72496258 0x4523482 0 0 74076647 0x46a51e7 0 2 73202783 0x45cfc5f 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000042 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314042 0x45eaefa 0 5 72496358 0x45234e6 0 0 74076747 0x46a524b 0 2 73202881 0x45cfcc1 0 3 72732563 0x455cf93 0 1 71221510 0x43ec106 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000025 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314035 0x45eaef3 0 5 72496351 0x45234df 0 0 74076740 0x46a5244 0 2 73202875 0x45cfcbb 0 3 72732556 0x455cf8c 0 1 71221504 0x43ec100 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000040 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732719 0x455d02f 0 1 71221666 0x43ec1a2 0 4 73314199 0x45eaf97 0 5 72496515 0x4523583 0 0 74076905 0x46a52e9 0 2 73203040 0x45cfd60 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000036 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203010 0x45cfd42 0 3 72732693 0x455d015 0 1 71221643 0x43ec18b 0 4 73314175 0x45eaf7f 0 5 72496492 0x452356c 0 0 74076881 0x46a52d1 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000022 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732608 0x455cfc0 0 1 71221555 0x43ec133 0 4 73314087 0x45eaf27 0 5 72496403 0x4523513 0 0 74076792 0x46a5278 0 2 73202926 0x45cfcee 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000028 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221396 0x43ec094 0 4 73313929 0x45eae89 0 5 72496246 0x4523476 0 0 74076635 0x46a51db 0 2 73202771 0x45cfc53 0 3 72732454 0x455cf26 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000023 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221450 0x43ec0ca 0 4 73313983 0x45eaebf 0 5 72496299 0x45234ab 0 0 74076689 0x46a5211 0 2 73202823 0x45cfc87 0 3 72732505 0x455cf59 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000037 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202837 0x45cfc95 0 3 72732520 0x455cf68 0 1 71221467 0x43ec0db 0 4 73314000 0x45eaed0 0 5 72496316 0x45234bc 0 0 74076705 0x46a5221 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000056 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496283 0x452349b 0 0 74076672 0x46a5200 0 2 73202806 0x45cfc76 0 3 72732488 0x455cf48 0 1 71221435 0x43ec0bb 0 4 73313967 0x45eaeaf 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000019 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221406 0x43ec09e 0 4 73313939 0x45eae93 0 5 72496257 0x4523481 0 0 74076646 0x46a51e6 0 2 73202782 0x45cfc5e 0 3 72732465 0x455cf31 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000045 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202793 0x45cfc69 0 3 72732476 0x455cf3c 0 1 71221423 0x43ec0af 0 4 73313955 0x45eaea3 0 5 72496272 0x4523490 0 0 74076662 0x46a51f6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000038 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203004 0x45cfd3c 0 3 72732687 0x455d00f 0 1 71221635 0x43ec183 0 4 73314168 0x45eaf78 0 5 72496484 0x4523564 0 0 74076873 0x46a52c9 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000021 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496378 0x45234fa 0 0 74076767 0x46a525f 0 2 73202901 0x45cfcd5 0 3 72732583 0x455cfa7 0 1 71221530 0x43ec11a 0 4 73314062 0x45eaf0e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000010 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496253 0x452347d 0 0 74076643 0x46a51e3 0 2 73202777 0x45cfc59 0 3 72732460 0x455cf2c 0 1 71221408 0x43ec0a0 0 4 73313940 0x45eae94 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000042 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732653 0x455cfed 0 1 71221600 0x43ec160 0 4 73314132 0x45eaf54 0 5 72496448 0x4523540 0 0 74076837 0x46a52a5 0 2 73202971 0x45cfd1b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000010 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221685 0x43ec1b5 0 4 73314217 0x45eafa9 0 5 72496533 0x4523595 0 0 74076924 0x46a52fc 0 2 73203059 0x45cfd73 0 3 72732741 0x455d045 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000063 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496249 0x4523479 0 0 74076638 0x46a51de 0 2 73202773 0x45cfc55 0 3 72732455 0x455cf27 0 1 71221404 0x43ec09c 0 4 73313937 0x45eae91 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000030 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076627 0x46a51d3 0 2 73202761 0x45cfc49 0 3 72732443 0x455cf1b 0 1 71221390 0x43ec08e 0 4 73313922 0x45eae82 0 5 72496238 0x452346e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000055 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496451 0x4523543 0 0 74076840 0x46a52a8 0 2 73202976 0x45cfd20 0 3 72732659 0x455cff3 0 1 71221607 0x43ec167 0 4 73314139 0x45eaf5b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000028 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314089 0x45eaf29 0 5 72496405 0x4523515 0 0 74076794 0x46a527a 0 2 73202928 0x45cfcf0 0 3 72732610 0x455cfc2 0 1 71221557 0x43ec135 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000026 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732670 0x455cffe 0 1 71221617 0x43ec171 0 4 73314149 0x45eaf65 0 5 72496466 0x4523552 0 0 74076854 0x46a52b6 0 2 73202988 0x45cfd2c 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000046 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221474 0x43ec0e2 0 4 73314006 0x45eaed6 0 5 72496322 0x45234c2 0 0 74076710 0x46a5226 0 2 73202844 0x45cfc9c 0 3 72732528 0x455cf70 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000011 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314030 0x45eaeee 0 5 72496346 0x45234da 0 0 74076736 0x46a5240 0 2 73202871 0x45cfcb7 0 3 72732553 0x455cf89 0 1 71221501 0x43ec0fd 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000061 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221519 0x43ec10f 0 4 73314051 0x45eaf03 0 5 72496367 0x45234ef 0 0 74076756 0x46a5254 0 2 73202891 0x45cfccb 0 3 72732573 0x455cf9d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000058 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496306 0x45234b2 0 0 74076696 0x46a5218 0 2 73202836 0x45cfc94 0 3 72732517 0x455cf65 0 1 71221464 0x43ec0d8 0 4 73313998 0x45eaece 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000051 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076772 0x46a5264 0 2 73202910 0x45cfcde 0 3 72732591 0x455cfaf 0 1 71221539 0x43ec123 0 4 73314072 0x45eaf18 0 5 72496389 0x4523505 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000008 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076850 0x46a52b2 0 2 73202984 0x45cfd28 0 3 72732666 0x455cffa 0 1 71221613 0x43ec16d 0 4 73314145 0x45eaf61 0 5 72496462 0x452354e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000013 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732483 0x455cf43 0 1 71221430 0x43ec0b6 0 4 73313962 0x45eaeaa 0 5 72496278 0x4523496 0 0 74076667 0x46a51fb 0 2 73202801 0x45cfc71 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000056 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314237 0x45eafbd 0 5 72496554 0x45235aa 0 0 74076944 0x46a5310 0 2 73203078 0x45cfd86 0 3 72732761 0x455d059 0 1 71221708 0x43ec1cc 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000023 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314121 0x45eaf49 0 5 72496437 0x4523535 0 0 74076826 0x46a529a 0 2 73202960 0x45cfd10 0 3 72732642 0x455cfe2 0 1 71221591 0x43ec157 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000033 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314190 0x45eaf8e 0 5 72496506 0x452357a 0 0 74076895 0x46a52df 0 2 73203033 0x45cfd59 0 3 72732715 0x455d02b 0 1 71221663 0x43ec19f 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000045 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732700 0x455d01c 0 1 71221647 0x43ec18f 0 4 73314179 0x45eaf83 0 5 72496495 0x452356f 0 0 74076885 0x46a52d5 0 2 73203019 0x45cfd4b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000040 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076806 0x46a5286 0 2 73202940 0x45cfcfc 0 3 72732622 0x455cfce 0 1 71221570 0x43ec142 0 4 73314102 0x45eaf36 0 5 72496418 0x4523522 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000050 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732698 0x455d01a 0 1 71221645 0x43ec18d 0 4 73314177 0x45eaf81 0 5 72496494 0x452356e 0 0 74076884 0x46a52d4 0 2 73203018 0x45cfd4a 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000056 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076742 0x46a5246 0 2 73202876 0x45cfcbc 0 3 72732558 0x455cf8e 0 1 71221506 0x43ec102 0 4 73314038 0x45eaef6 0 5 72496355 0x45234e3 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000013 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202826 0x45cfc8a 0 3 72732509 0x455cf5d 0 1 71221458 0x43ec0d2 0 4 73313991 0x45eaec7 0 5 72496307 0x45234b3 0 0 74076699 0x46a521b 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000053 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221590 0x43ec156 0 4 73314123 0x45eaf4b 0 5 72496439 0x4523537 0 0 74076830 0x46a529e 0 2 73202964 0x45cfd14 0 3 72732646 0x455cfe6 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000025 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732746 0x455d04a 0 1 71221693 0x43ec1bd 0 4 73314226 0x45eafb2 0 5 72496542 0x452359e 0 0 74076931 0x46a5303 0 2 73203065 0x45cfd79 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000001 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203048 0x45cfd68 0 3 72732733 0x455d03d 0 1 71221680 0x43ec1b0 0 4 73314212 0x45eafa4 0 5 72496528 0x4523590 0 0 74076917 0x46a52f5 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000032 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202873 0x45cfcb9 0 3 72732555 0x455cf8b 0 1 71221503 0x43ec0ff 0 4 73314036 0x45eaef4 0 5 72496352 0x45234e0 0 0 74076741 0x46a5245 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000023 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732708 0x455d024 0 1 71221655 0x43ec197 0 4 73314187 0x45eaf8b 0 5 72496504 0x4523578 0 0 74076893 0x46a52dd 0 2 73203028 0x45cfd54 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000033 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496261 0x4523485 0 0 74076650 0x46a51ea 0 2 73202785 0x45cfc61 0 3 72732468 0x455cf34 0 1 71221416 0x43ec0a8 0 4 73313950 0x45eae9e 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000062 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496487 0x4523567 0 0 74076879 0x46a52cf 0 2 73203014 0x45cfd46 0 3 72732696 0x455d018 0 1 71221644 0x43ec18c 0 4 73314176 0x45eaf80 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000030 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076860 0x46a52bc 0 2 73202994 0x45cfd32 0 3 72732676 0x455d004 0 1 71221624 0x43ec178 0 4 73314156 0x45eaf6c 0 5 72496472 0x4523558 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000063 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076714 0x46a522a 0 2 73202848 0x45cfca0 0 3 72732530 0x455cf72 0 1 71221477 0x43ec0e5 0 4 73314009 0x45eaed9 0 5 72496325 0x45234c5 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000000 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732588 0x455cfac 0 1 71221536 0x43ec120 0 4 73314071 0x45eaf17 0 5 72496388 0x4523504 0 0 74076778 0x46a526a 0 2 73202912 0x45cfce0 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000043 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314069 0x45eaf15 0 5 72496385 0x4523501 0 0 74076774 0x46a5266 0 2 73202907 0x45cfcdb 0 3 72732592 0x455cfb0 0 1 71221538 0x43ec122 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000055 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221704 0x43ec1c8 0 4 73314236 0x45eafbc 0 5 72496553 0x45235a9 0 0 74076943 0x46a530f 0 2 73203077 0x45cfd85 0 3 72732760 0x455d058 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000012 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076734 0x46a523e 0 2 73202869 0x45cfcb5 0 3 72732551 0x455cf87 0 1 71221500 0x43ec0fc 0 4 73314033 0x45eaef1 0 5 72496348 0x45234dc 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000002 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732602 0x455cfba 0 1 71221549 0x43ec12d 0 4 73314081 0x45eaf21 0 5 72496397 0x452350d 0 0 74076786 0x46a5272 0 2 73202920 0x45cfce8 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000000 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221456 0x43ec0d0 0 4 73313993 0x45eaec9 0 5 72496309 0x45234b5 0 0 74076700 0x46a521c 0 2 73202832 0x45cfc90 0 3 72732515 0x455cf63 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000045 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076693 0x46a5215 0 2 73202828 0x45cfc8c 0 3 72732511 0x455cf5f 0 1 71221462 0x43ec0d6 0 4 73313997 0x45eaecd 0 5 72496315 0x45234bb 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000062 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221569 0x43ec141 0 4 73314101 0x45eaf35 0 5 72496417 0x4523521 0 0 74076807 0x46a5287 0 2 73202941 0x45cfcfd 0 3 72732623 0x455cfcf 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000046 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73203070 0x45cfd7e 0 3 72732752 0x455d050 0 1 71221699 0x43ec1c3 0 4 73314231 0x45eafb7 0 5 72496548 0x45235a4 0 0 74076936 0x46a5308 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000026 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076745 0x46a5249 0 2 73202879 0x45cfcbf 0 3 72732561 0x455cf91 0 1 71221509 0x43ec105 0 4 73314041 0x45eaef9 0 5 72496357 0x45234e5 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000061 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202981 0x45cfd25 0 3 72732663 0x455cff7 0 1 71221610 0x43ec16a 0 4 73314142 0x45eaf5e 0 5 72496458 0x452354a 0 0 74076847 0x46a52af 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000044 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73313923 0x45eae83 0 5 72496239 0x452346f 0 0 74076628 0x46a51d4 0 2 73202763 0x45cfc4b 0 3 72732446 0x455cf1e 0 1 71221393 0x43ec091 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000038 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 73202814 0x45cfc7e 0 3 72732496 0x455cf50 0 1 71221444 0x43ec0c4 0 4 73313976 0x45eaeb8 0 5 72496292 0x45234a4 0 0 74076683 0x46a520b 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_1m.00000048 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221563 0x43ec13b 0 4 73314095 0x45eaf2f 0 5 72496412 0x452351c 0 0 74076801 0x46a5281 0 2 73202936 0x45cfcf8 0 3 72732618 0x455cfca 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000041 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221469 0x43ec0dd 0 4 73314002 0x45eaed2 0 5 72496319 0x45234bf 0 0 74076707 0x46a5223 0 2 73202842 0x45cfc9a 0 3 72732525 0x455cf6d 0 /lus/husk/dmoen/testdir1.15099/posix_directio_1m.00000027 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314119 0x45eaf47 0 5 72496435 0x4523533 0 0 74076824 0x46a5298 0 2 73202958 0x45cfd0e 0 3 72732640 0x455cfe0 0 1 71221587 0x43ec153 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000059 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221502 0x43ec0fe 0 4 73314034 0x45eaef2 0 5 72496350 0x45234de 0 0 74076739 0x46a5243 0 2 73202874 0x45cfcba 0 3 72732557 0x455cf8d 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000053 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732527 0x455cf6f 0 1 71221475 0x43ec0e3 0 4 73314007 0x45eaed7 0 5 72496323 0x45234c3 0 0 74076712 0x46a5228 0 2 73202846 0x45cfc9e 0 /lus/husk/dmoen/testdir1.15099/posix_buffered_32m.00000030 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314015 0x45eaedf 0 5 72496332 0x45234cc 0 0 74076721 0x46a5231 0 2 73202855 0x45cfca7 0 3 72732538 0x455cf7a 0 1 71221485 0x43ec0ed 0 /lus/husk/dmoen/testdir1.15099/posix_directio_64m.00000003 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 72496269 0x452348d 0 0 74076658 0x46a51f2 0 2 73202794 0x45cfc6a 0 3 72732477 0x455cf3d 0 1 71221424 0x43ec0b0 0 4 73313956 0x45eaea4 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000027 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 74076888 0x46a52d8 0 2 73203022 0x45cfd4e 0 3 72732705 0x455d021 0 1 71221652 0x43ec194 0 4 73314184 0x45eaf88 0 5 72496500 0x4523574 0 /lus/husk/dmoen/testdir1.15099/posix_directio_32m.00000028 lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 72732734 0x455d03e 0 1 71221681 0x43ec1b1 0 4 73314213 0x45eafa5 0 5 72496529 0x4523591 0 0 74076918 0x46a52f6 0 2 73203052 0x45cfd6c 0 Start IOR test posix_buffered_64m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_buffered_64m [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 22:43:45 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_buffered_64mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.15099/posix_buffered_64m access = file‐per‐process ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 2008.13 2008.13 2008.13 0.00 31.38 31.38 31.38 0.00 261.08210 64 2 1 1 1 1 0 0 1 0 67108864 549755813888 ‐1 POSIX EXCEL Max Write: 2008.13 MiB/sec (2105.68 MB/sec) Perf_Data IOR 67108864 write 2008.135 MiB/sec Run finished: Wed Apr 20 22:48:06 2016 Application 183682 resources: utime ~539s, stime ~163s, Rss ~4744, inblocks ~0, outblocks ~402653184 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_buffered_64m [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 22:48:17 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐F ‐e ‐g ‐C ‐k ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.15099/posix_buffered_64mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.15099/posix_buffered_64m access = file‐per‐process ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 2338.99 2338.99 2338.99 0.00 36.55 36.55 36.55 0.00 224.15139 64 2 1 1 1 1 0 0 1 0 67108864 549755813888 ‐1 POSIX EXCEL Max Read: 2338.99 MiB/sec (2452.61 MB/sec) Perf_Data IOR 67108864 read 2338.991 MiB/sec Run finished: Wed Apr 20 22:52:02 2016 Application 183683 resources: utime ~369s, stime ~133s, Rss ~4744, inblocks ~0, outblocks ~0 Done IOR posix_buffered_64m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes posix_buffered_64m, down= 3 + ls ‐lh /lus/husk/dmoen/testdir1.15099 total 3.1T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:18 posix_buffered_1m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:19 posix_buffered_1m.00000063 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:10 posix_buffered_32m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:11 posix_buffered_32m.00000063 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:46 posix_buffered_64m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:46 posix_buffered_64m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:46 posix_buffered_64m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:46 posix_buffered_64m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:47 posix_buffered_64m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:48 posix_buffered_64m.00000063 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:28 posix_directio_1m.00000063 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:38 posix_directio_32m.00000063 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000000 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000001 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000002 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000003 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000004 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000005 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000006 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000007 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000008 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000009 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000010 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000011 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000012 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000013 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000014 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000015 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000016 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000017 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000018 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000019 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000020 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000021 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000022 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000023 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000024 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000025 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000026 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000027 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000028 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000029 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000030 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000031 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000032 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000033 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000034 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000035 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000036 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000037 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000038 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000039 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000040 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000041 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000042 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000043 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000044 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000045 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000046 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000047 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000048 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000049 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000050 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000051 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000052 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000053 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000054 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000055 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000056 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000057 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000058 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000059 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000060 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000061 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000062 ‐rw‐r‐‐r‐‐ 1 dmoen criemp 8.0G Apr 20 22:02 posix_directio_64m.00000063 + lfs df UUID 1K‐blocks Used Available Use% Mounted on husk‐MDT0000_UUID 392737536 871652 364937196 0% /lus/husk[MDT:0] husk‐MDT0001_UUID 392737536 79816 365727484 0% /lus/husk[MDT:1] husk‐MDT0002_UUID 392737536 80020 365727280 0% /lus/husk[MDT:2] husk‐MDT0003_UUID 392737536 79996 365727304 0% /lus/husk[MDT:3] husk‐OST0000_UUID 7805162980 638445068 6762966600 9% /lus/husk[OST:0] husk‐OST0001_UUID 7805162980 638538856 6762882172 9% /lus/husk[OST:1] husk‐OST0002_UUID 7805148880 643564824 6757797648 9% /lus/husk[OST:2] husk‐OST0003_UUID 7805148880 638354212 6763053640 9% /lus/husk[OST:3] husk‐OST0004_UUID 7805148880 638331440 6763072764 9% /lus/husk[OST:4] husk‐OST0005_UUID 7805148880 638304136 6763093208 9% /lus/husk[OST:5] filesystem summary: 46830921480 3835538536 40572866032 9% /lus/husk + rm ‐rf /lus/husk/dmoen/testdir1.15099 + RANKS=64 + RANKS_PER_NODE=2 + FS=/lus/husk/dmoen + CFGFILE=./IOR_posix_shr.cfg + BIN=../bin + RUN_IOR=../bin/run_IOR.pl + JOBID=16566 + TESTDIR1=/lus/husk/dmoen/testdir1.16566 + mkdir /lus/husk/dmoen/testdir1.16566 + lfs setstripe ‐S 1m ‐c ‐1 /lus/husk/dmoen/testdir1.16566 + module list ++ /opt/cray/pe/modules/3.2.10.4/bin/modulecmd bash list Currently Loaded Modulefiles: 1) modules/3.2.10.4 2) alps/6.1.3‐52.7 3) nodestat/2.2‐1.0000.58434.41.84.ari 4) sdb/2.2.0‐2.1256 5) udreg/2.3.2‐6.1 6) ugni/6.0.12‐2.3 7) gni‐headers/5.0.7‐2.3 8) dmapp/7.1.0‐28.15 9) xpmem/0.1‐2.0000.66820.89.2.ari 10) llm/20.2.1‐2.12 11) nodehealth/5.2.0‐14.17 12) sysadm/2.2.1‐2.118 13) Base‐opts/2.1.1‐2.4 14) cce/8.5.0.4664 15) craype‐network‐aries 16) craype/2.5.5.2 17) cray‐libsci/16.03.1 18) pmi/5.0.10‐1.0000.11050.0.0.ari 19) rca/1.0.0‐2.0000.66809.157.31.ari 20) atp/2.0.1 21) PrgEnv‐cray/6.0.2 22) cray‐mpich/7.3.2 23) craype‐sandybridge 24) craype‐hugepages512M 25) cray‐hdf5‐parallel/1.8.14 + eval + cat /proc/fs/lustre/version lustre: 2.7.1 kernel: patchless_client build: 2.7.0‐trunk‐1.0000.f114a17.0.0‐jmiller‐lustre‐filesystem.git@f114a17‐2016‐04‐19‐14:45 + lfs df UUID 1K‐blocks Used Available Use% Mounted on husk‐MDT0000_UUID 392737536 871652 364937196 0% /lus/husk[MDT:0] husk‐MDT0001_UUID 392737536 79816 365727484 0% /lus/husk[MDT:1] husk‐MDT0002_UUID 392737536 80020 365727280 0% /lus/husk[MDT:2] husk‐MDT0003_UUID 392737536 79996 365727304 0% /lus/husk[MDT:3] husk‐OST0000_UUID 7805162980 638445068 6762966600 9% /lus/husk[OST:0] husk‐OST0001_UUID 7805162980 638538856 6762882172 9% /lus/husk[OST:1] husk‐OST0002_UUID 7805148880 643564824 6757797648 9% /lus/husk[OST:2] husk‐OST0003_UUID 7805148880 638354212 6763053640 9% /lus/husk[OST:3] husk‐OST0004_UUID 7805148880 638331440 6763072764 9% /lus/husk[OST:4] husk‐OST0005_UUID 7805148880 638304136 6763093208 9% /lus/husk[OST:5] filesystem summary: 46830921480 3835538536 40572866032 9% /lus/husk + export MPICH_VERSION_DISPLAY=1 + MPICH_VERSION_DISPLAY=1 + export MPICH_MPIIO_HINTS_DISPLAY=1 + MPICH_MPIIO_HINTS_DISPLAY=1 + export MPICH_MPIIO_STATS=1 + MPICH_MPIIO_STATS=1 + export MPICH_MPIIO_ABORT_ON_RW_ERROR=enable + MPICH_MPIIO_ABORT_ON_RW_ERROR=enable + export MPICH_CPUMASK_DISPLAY=1 + MPICH_CPUMASK_DISPLAY=1 + export IOR_HINT__MPI__romio_no_indep_rw=true + IOR_HINT__MPI__romio_no_indep_rw=true + export IOR_HINT__MPI__romio_cb_write=enable + IOR_HINT__MPI__romio_cb_write=enable + ../bin/run_IOR.pl ‐n 64 ‐N 2 /lus/husk/dmoen/testdir1.16566 ./IOR_posix_shr.cfg ALPS Node Information total 0 /lus/husk/dmoen/testdir1.16566 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 Start IOR test posix_buffered_32m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐e ‐g ‐C ‐k ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.16566/posix_buffered_32m [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 22:52:12 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐e ‐g ‐C ‐k ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.16566/posix_buffered_32mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.16566/posix_buffered_32m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 2424.98 2424.98 2424.98 0.00 75.78 75.78 75.78 0.00 216.20323 64 2 1 0 1 1 0 0 1 0 33554432 549755813888 ‐1 POSIX EXCEL Max Write: 2424.98 MiB/sec (2542.77 MB/sec) Perf_Data IOR 33554432 write 2424.978 MiB/sec Run finished: Wed Apr 20 22:55:48 2016 Application 183684 resources: utime ~289s, stime ~177s, Rss ~4744, inblocks ~0, outblocks ~402653184 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐e ‐g ‐C ‐k ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.16566/posix_buffered_32m [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 22:56:00 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐e ‐g ‐C ‐k ‐t 32m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.16566/posix_buffered_32mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.16566/posix_buffered_32m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 32 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 2388.95 2388.95 2388.95 0.00 74.65 74.65 74.65 0.00 219.46398 64 2 1 0 1 1 0 0 1 0 33554432 549755813888 ‐1 POSIX EXCEL Max Read: 2388.95 MiB/sec (2504.99 MB/sec) Perf_Data IOR 33554432 read 2388.948 MiB/sec Run finished: Wed Apr 20 22:59:39 2016 Application 183685 resources: utime ~368s, stime ~134s, Rss ~4744, inblocks ~0, outblocks ~0 Done IOR posix_buffered_32m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes posix_buffered_32m, down= 3 total 513G ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 22:55 posix_buffered_32m /lus/husk/dmoen/testdir1.16566 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir1.16566/posix_buffered_32m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221773 0x43ec20d 0 4 73314305 0x45eb001 0 5 72496621 0x45235ed 0 0 74077010 0x46a5352 0 2 73203144 0x45cfdc8 0 3 72732826 0x455d09a 0 Start IOR test posix_buffered_1m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐e ‐g ‐C ‐k ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.16566/posix_buffered_1m [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 22:59:50 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐e ‐g ‐C ‐k ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.16566/posix_buffered_1mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.16566/posix_buffered_1m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 2396.07 2396.07 2396.07 0.00 2396.07 2396.07 2396.07 0.00 218.81155 64 2 1 0 1 1 0 0 1 0 1048576 549755813888 ‐1 POSIX EXCEL Max Write: 2396.07 MiB/sec (2512.46 MB/sec) Perf_Data IOR 1048576 write 2396.071 MiB/sec Run finished: Wed Apr 20 23:03:29 2016 Application 183686 resources: utime ~263s, stime ~170s, Rss ~4744, inblocks ~0, outblocks ~402653184 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐e ‐g ‐C ‐k ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.16566/posix_buffered_1m [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 23:03:41 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐e ‐g ‐C ‐k ‐t 1m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.16566/posix_buffered_1mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.16566/posix_buffered_1m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 1 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 2422.12 2422.12 2422.12 0.00 2422.12 2422.12 2422.12 0.00 216.45811 64 2 1 0 1 1 0 0 1 0 1048576 549755813888 ‐1 POSIX EXCEL Max Read: 2422.12 MiB/sec (2539.78 MB/sec) Perf_Data IOR 1048576 read 2422.122 MiB/sec Run finished: Wed Apr 20 23:07:17 2016 Application 183687 resources: utime ~360s, stime ~118s, Rss ~4744, inblocks ~0, outblocks ~0 Done IOR posix_buffered_1m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes posix_buffered_1m, down= 3 total 1.1T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 23:03 posix_buffered_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 22:55 posix_buffered_32m /lus/husk/dmoen/testdir1.16566 stripe_count: ‐1 stripe_size: 1048576 stripe_offset: ‐1 /lus/husk/dmoen/testdir1.16566/posix_buffered_1m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 73314306 0x45eb002 0 5 72496622 0x45235ee 0 0 74077011 0x46a5353 0 2 73203145 0x45cfdc9 0 3 72732827 0x455d09b 0 1 71221774 0x43ec20e 0 /lus/husk/dmoen/testdir1.16566/posix_buffered_32m lmm_stripe_count: 6 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 71221773 0x43ec20d 0 4 73314305 0x45eb001 0 5 72496621 0x45235ed 0 0 74077010 0x46a5352 0 2 73203144 0x45cfdc8 0 3 72732826 0x455d09a 0 Start IOR test posix_buffered_64m aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐e ‐g ‐C ‐k ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.16566/posix_buffered_64m [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 23:07:29 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐w ‐a POSIX ‐e ‐g ‐C ‐k ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.16566/posix_buffered_64mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.16566/posix_buffered_64m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ write 2105.16 2105.16 2105.16 0.00 32.89 32.89 32.89 0.00 249.04906 64 2 1 0 1 1 0 0 1 0 67108864 549755813888 ‐1 POSIX EXCEL Max Write: 2105.16 MiB/sec (2207.42 MB/sec) Perf_Data IOR 67108864 write 2105.160 MiB/sec Run finished: Wed Apr 20 23:11:38 2016 Application 183688 resources: utime ~322s, stime ~181s, Rss ~4744, inblocks ~0, outblocks ~402653184 aprun ‐n 64 ‐N 2 ‐j 1 ‐T /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐e ‐g ‐C ‐k ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.16566/posix_buffered_64m [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = NULL PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 IOR‐2.10.2: MPI Coordinated Test of Parallel I/O Run began: Wed Apr 20 23:11:50 2016 Command line used: /home/users/dmoen/lockahead/bin/IOR/src/C/IOR ‐r ‐a POSIX ‐e ‐g ‐C ‐k ‐t 64m ‐s 1 ‐b 8g ‐o /lus/husk/dmoen/testdir1.16566/posix_buffered_64mMachine: Linux nid00032 Summary: api = POSIX test filename = /lus/husk/dmoen/testdir1.16566/posix_buffered_64m access = single‐shared‐file ordering in a file = sequential offsets ordering inter file=constant task offsets = 1 clients = 64 (2 per node) repetitions = 1 xfersize = 64 MiB blocksize = 8 GiB aggregate filesize = 512 GiB Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ ‐‐‐‐‐‐‐ read 2329.44 2329.44 2329.44 0.00 36.40 36.40 36.40 0.00 225.07018 64 2 1 0 1 1 0 0 1 0 67108864 549755813888 ‐1 POSIX EXCEL Max Read: 2329.44 MiB/sec (2442.60 MB/sec) Perf_Data IOR 67108864 read 2329.442 MiB/sec Run finished: Wed Apr 20 23:15:35 2016 Application 183689 resources: utime ~411s, stime ~138s, Rss ~4744, inblocks ~0, outblocks ~0 Done IOR posix_buffered_64m 141 0x8d c0‐0c2s3n1 compute down interactive 169 0xa9 c0‐0c2s10n1 compute down interactive 170 0xaa c0‐0c2s10n2 compute admindown interactive Down compute nodes posix_buffered_64m, down= 3 + ls ‐lh /lus/husk/dmoen/testdir1.16566 total 1.6T ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 23:03 posix_buffered_1m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 22:55 posix_buffered_32m ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 20 23:11 posix_buffered_64m + lfs df UUID 1K‐blocks Used Available Use% Mounted on husk‐MDT0000_UUID 392737536 867712 364941128 0% /lus/husk[MDT:0] husk‐MDT0001_UUID 392737536 79816 365727484 0% /lus/husk[MDT:1] husk‐MDT0002_UUID 392737536 80020 365727280 0% /lus/husk[MDT:2] husk‐MDT0003_UUID 392737536 79996 365727304 0% /lus/husk[MDT:3] husk‐OST0000_UUID 7805162980 370008796 7030878584 5% /lus/husk[OST:0] husk‐OST0001_UUID 7805162980 370101556 7030795180 5% /lus/husk[OST:1] husk‐OST0002_UUID 7805148880 375123432 7025714740 5% /lus/husk[OST:2] husk‐OST0003_UUID 7805148880 369907664 7030975880 5% /lus/husk[OST:3] husk‐OST0004_UUID 7805148880 369905436 7030974500 5% /lus/husk[OST:4] husk‐OST0005_UUID 7805148880 369874016 7030999048 5% /lus/husk[OST:5] filesystem summary: 46830921480 2224920900 42180337932 5% /lus/husk + rm ‐rf /lus/husk/dmoen/testdir1.16566 + RANKS=64 + RANKS_PER_NODE=2 + FS=/lus/husk/dmoen + BIN=../bin + JOBID=17318 + TESTDIR1=/lus/husk/dmoen/read_after_write/testdir1.17318 + TESTDIR2=/lus/husk/dmoen/read_after_write/testdir2.17318 + TESTDIR3=/lus/husk/dmoen/read_after_write/testdir3.17318 + mkdir ‐p /lus/husk/dmoen/read_after_write/testdir1.17318 + mkdir ‐p /lus/husk/dmoen/read_after_write/testdir2.17318 + mkdir ‐p /lus/husk/dmoen/read_after_write/testdir3.17318 + lfs setstripe ‐S 1m ‐c ‐1 /lus/husk/dmoen/read_after_write ++ lfs df /lus/husk/dmoen/read_after_write ++ grep OST ++ wc ‐l + NUM_OSTS=6 + module list ++ /opt/cray/pe/modules/3.2.10.4/bin/modulecmd bash list Currently Loaded Modulefiles: 1) modules/3.2.10.4 2) alps/6.1.3‐52.7 3) nodestat/2.2‐1.0000.58434.41.84.ari 4) sdb/2.2.0‐2.1256 5) udreg/2.3.2‐6.1 6) ugni/6.0.12‐2.3 7) gni‐headers/5.0.7‐2.3 8) dmapp/7.1.0‐28.15 9) xpmem/0.1‐2.0000.66820.89.2.ari 10) llm/20.2.1‐2.12 11) nodehealth/5.2.0‐14.17 12) sysadm/2.2.1‐2.118 13) Base‐opts/2.1.1‐2.4 14) cce/8.5.0.4664 15) craype‐network‐aries 16) craype/2.5.5.2 17) cray‐libsci/16.03.1 18) pmi/5.0.10‐1.0000.11050.0.0.ari 19) rca/1.0.0‐2.0000.66809.157.31.ari 20) atp/2.0.1 21) PrgEnv‐cray/6.0.2 22) cray‐mpich/7.3.2 23) craype‐sandybridge 24) craype‐hugepages512M 25) cray‐hdf5‐parallel/1.8.14 + eval + cat /proc/fs/lustre/version lustre: 2.7.1 kernel: patchless_client build: 2.7.0‐trunk‐1.0000.f114a17.0.0‐jmiller‐lustre‐filesystem.git@f114a17‐2016‐04‐19‐14:45 + lfs df UUID 1K‐blocks Used Available Use% Mounted on husk‐MDT0000_UUID 392737536 867712 364941128 0% /lus/husk[MDT:0] husk‐MDT0001_UUID 392737536 79816 365727484 0% /lus/husk[MDT:1] husk‐MDT0002_UUID 392737536 80020 365727280 0% /lus/husk[MDT:2] husk‐MDT0003_UUID 392737536 79996 365727304 0% /lus/husk[MDT:3] husk‐OST0000_UUID 7805162980 370008796 7030878584 5% /lus/husk[OST:0] husk‐OST0001_UUID 7805162980 370101556 7030795180 5% /lus/husk[OST:1] husk‐OST0002_UUID 7805148880 375123432 7025714740 5% /lus/husk[OST:2] husk‐OST0003_UUID 7805148880 369907664 7030975880 5% /lus/husk[OST:3] husk‐OST0004_UUID 7805148880 369905436 7030974500 5% /lus/husk[OST:4] husk‐OST0005_UUID 7805148880 369874016 7030999048 5% /lus/husk[OST:5] filesystem summary: 46830921480 2224920900 42180337932 5% /lus/husk + export MPICH_VERSION_DISPLAY=1 + MPICH_VERSION_DISPLAY=1 + export MPICH_MPIIO_HINTS_DISPLAY=1 + MPICH_MPIIO_HINTS_DISPLAY=1 + export MPICH_MPIIO_STATS=1 + MPICH_MPIIO_STATS=1 + export MPICH_MPIIO_ABORT_ON_RW_ERROR=enable + MPICH_MPIIO_ABORT_ON_RW_ERROR=enable + export MPICH_CPUMASK_DISPLAY=1 + MPICH_CPUMASK_DISPLAY=1 + export ’MPICH_MPIIO_HINTS=*:romio_no_indep_rw=true:romio_cb_write=enable:cray_cb_write_lock_mode=0:cray_cb_nodes_multiplier=6’ + MPICH_MPIIO_HINTS=’*:romio_no_indep_rw=true:romio_cb_write=enable:cray_cb_write_lock_mode=0:cray_cb_nodes_multiplier=6’ + aprun ‐n 64 ‐N 2 ../bin/mpi‐io‐test ‐i 128 ‐b 67198864 ‐f /lus/husk/dmoen/read_after_write/testdir3.17318/outfile [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = *:romio_no_indep_rw=true:romio_cb_write=enable:cray_cb_write_lock_mode=0:cray_cb_nodes_multiplier=6 PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 Rank 13 of 64 is on nid00042 Rank 12 of 64 is on nid00042 Rank 45 of 64 is on nid00110 Rank 44 of 64 is on nid00110 Rank 51 of 64 is on nid00113 Rank 50 of 64 is on nid00113 Rank 61 of 64 is on nid00118 Rank 60 of 64 is on nid00118 Rank 59 of 64 is on nid00117 Rank 58 of 64 is on nid00117 Rank 55 of 64 is on nid00115 Rank 54 of 64 is on nid00115 Rank 53 of 64 is on nid00114 Rank 52 of 64 is on nid00114 Rank 49 of 64 is on nid00112 Rank 48 of 64 is on nid00112 Rank 46 of 64 is on nid00111 Rank 47 of 64 is on nid00111 Rank 42 of 64 is on nid00109 Rank 43 of 64 is on nid00109 Rank 40 of 64 is on nid00108 Rank 41 of 64 is on nid00108 Rank 39 of 64 is on nid00107 Rank 38 of 64 is on nid00107 Rank 36 of 64 is on nid00106 Rank 37 of 64 is on nid00106 Rank 35 of 64 is on nid00105 Rank 34 of 64 is on nid00105 Rank 32 of 64 is on nid00104 Rank 33 of 64 is on nid00104 Rank 31 of 64 is on nid00099 Rank 30 of 64 is on nid00099 Rank 29 of 64 is on nid00098 Rank 28 of 64 is on nid00098 Rank 27 of 64 is on nid00097 Rank 26 of 64 is on nid00097 Rank 25 of 64 is on nid00096 Rank 24 of 64 is on nid00096 Rank 23 of 64 is on nid00067 Rank 22 of 64 is on nid00067 Rank 21 of 64 is on nid00066 Rank 20 of 64 is on nid00066 Rank 19 of 64 is on nid00065 Rank 18 of 64 is on nid00065 Rank 17 of 64 is on nid00064 Rank 16 of 64 is on nid00064 Rank 15 of 64 is on nid00043 Rank 14 of 64 is on nid00043 Rank 10 of 64 is on nid00041 Rank 11 of 64 is on nid00041 Rank 9 of 64 is on nid00040 Rank 8 of 64 is on nid00040 Rank 7 of 64 is on nid00035 Rank 6 of 64 is on nid00035 Rank 4 of 64 is on nid00034 Rank 5 of 64 is on nid00034 Rank 2 of 64 is on nid00033 Rank 3 of 64 is on nid00033 Rank 1 of 64 is on nid00032 Rank 0 of 64 is on nid00032 Rank 57 of 64 is on nid00116 Rank 56 of 64 is on nid00116 Rank 63 of 64 is on nid00119 Rank 62 of 64 is on nid00119 PE 0: MPIIO hints for /lus/husk/dmoen/read_after_write/testdir3.17318/outfile: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 6 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 1 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/read_after_write/testdir3.17318/outfile | independent writes = 0 | collective writes = 8192 | independent writers = 0 | aggregators = 6 | stripe count = 1 | stripe size = 1048576 | system writes = 525119 | stripe sized writes = 524864 | total bytes for writes = 550493093888 = 524991 MiB = 512 GiB | ave system write size = 1048320 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 392 | ave write gap size = 2291960460 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/read_after_write/testdir3.17318/outfile | independent reads = 4858 | collective reads = 0 | independent readers = 64 | aggregators = 6 | stripe count = 1 | stripe size = 1048576 | system reads = 4858 | stripe sized reads = 0 | total bytes for reads = 326452081312 = 311328 MiB = 304 GiB | ave system read size = 67198864 | number of read gaps = 4800 | ave read gap size = 89387240 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/read_after_write/testdir3.17318/outfile: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 6 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 1 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/read_after_write/testdir3.17318/outfile | independent reads = 0 | collective reads = 8192 | independent readers = 0 | aggregators = 6 | stripe count = 1 | stripe size = 1048576 | system reads = 525119 | stripe sized reads = 524864 | total bytes for reads = 550493093888 = 524991 MiB = 512 GiB | ave system read size = 1048320 | number of read gaps = 0 | ave read gap size = NA | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ nr_procs = 64, nr_iter = 128, blk_sz = 67198864, coll = 1 # total_size = 550493093888 # Write: min_t = 1164.063372, max_t = 1164.068238, mean_t = 1164.065036, var_t = 0.000001 # Read: min_t = 45.989815, max_t = 1367.534605, mean_t = 771.796060, var_t = 180363.512328 Write bandwidth = 472.904488 Mbytes/sec Read bandwidth = 402.544178 Mbytes/sec Correctness test passed. Application 183690 resources: utime ~67113s, stime ~565s, Rss ~4744, inblocks ~0, outblocks ~537592360 + aprun ‐n 64 ‐N 2 ../bin/mpi‐io‐test ‐i 256 ‐b 33554432 ‐f /lus/husk/dmoen/read_after_write/testdir2.17318/outfile [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = *:romio_no_indep_rw=true:romio_cb_write=enable:cray_cb_write_lock_mode=0:cray_cb_nodes_multiplier=6 PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 Rank 10 of 64 is on nid00041 Rank 11 of 64 is on nid00041 Rank 13 of 64 is on nid00042 Rank 12 of 64 is on nid00042 Rank 18 of 64 is on nid00065 Rank 19 of 64 is on nid00065 Rank 35 of 64 is on nid00105 Rank 34 of 64 is on nid00105 Rank 43 of 64 is on nid00109 Rank 42 of 64 is on nid00109 Rank 45 of 64 is on nid00110 Rank 44 of 64 is on nid00110 Rank 51 of 64 is on nid00113 Rank 50 of 64 is on nid00113 Rank 62 of 64 is on nid00119 Rank 63 of 64 is on nid00119 Rank 61 of 64 is on nid00118 Rank 60 of 64 is on nid00118 Rank 59 of 64 is on nid00117 Rank 58 of 64 is on nid00117 Rank 57 of 64 is on nid00116 Rank 56 of 64 is on nid00116 Rank 55 of 64 is on nid00115 Rank 54 of 64 is on nid00115 Rank 52 of 64 is on nid00114 Rank 53 of 64 is on nid00114 Rank 49 of 64 is on nid00112 Rank 48 of 64 is on nid00112 Rank 47 of 64 is on nid00111 Rank 46 of 64 is on nid00111 Rank 41 of 64 is on nid00108 Rank 40 of 64 is on nid00108 Rank 38 of 64 is on nid00107 Rank 39 of 64 is on nid00107 Rank 36 of 64 is on nid00106 Rank 37 of 64 is on nid00106 Rank 33 of 64 is on nid00104 Rank 32 of 64 is on nid00104 Rank 30 of 64 is on nid00099 Rank 31 of 64 is on nid00099 Rank 29 of 64 is on nid00098 Rank 28 of 64 is on nid00098 Rank 26 of 64 is on nid00097 Rank 27 of 64 is on nid00097 Rank 24 of 64 is on nid00096 Rank 25 of 64 is on nid00096 Rank 23 of 64 is on nid00067 Rank 22 of 64 is on nid00067 Rank 21 of 64 is on nid00066 Rank 20 of 64 is on nid00066 Rank 17 of 64 is on nid00064 Rank 16 of 64 is on nid00064 Rank 15 of 64 is on nid00043 Rank 14 of 64 is on nid00043 Rank 8 of 64 is on nid00040 Rank 9 of 64 is on nid00040 Rank 7 of 64 is on nid00035 Rank 6 of 64 is on nid00035 Rank 4 of 64 is on nid00034 Rank 5 of 64 is on nid00034 Rank 2 of 64 is on nid00033 Rank 3 of 64 is on nid00033 Rank 0 of 64 is on nid00032 Rank 1 of 64 is on nid00032 PE 0: MPIIO hints for /lus/husk/dmoen/read_after_write/testdir2.17318/outfile: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 6 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 1 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/read_after_write/testdir2.17318/outfile | independent writes = 0 | collective writes = 16384 | independent writers = 0 | aggregators = 6 | stripe count = 1 | stripe size = 1048576 | system writes = 524288 | stripe sized writes = 524288 | total bytes for writes = 549755813888 = 524288 MiB = 512 GiB | ave system write size = 1048576 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 790 | ave write gap size = 1151523598 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/read_after_write/testdir2.17318/outfile | independent reads = 9817 | collective reads = 0 | independent readers = 64 | aggregators = 6 | stripe count = 1 | stripe size = 1048576 | system reads = 9817 | stripe sized reads = 0 | total bytes for reads = 329403858944 = 314144 MiB = 306 GiB | ave system read size = 33554432 | number of read gaps = 9759 | ave read gap size = 22154404 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/read_after_write/testdir2.17318/outfile: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 6 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 1 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/read_after_write/testdir2.17318/outfile | independent reads = 0 | collective reads = 16384 | independent readers = 0 | aggregators = 6 | stripe count = 1 | stripe size = 1048576 | system reads = 524288 | stripe sized reads = 524288 | total bytes for reads = 549755813888 = 524288 MiB = 512 GiB | ave system read size = 1048576 | number of read gaps = 0 | ave read gap size = NA | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ nr_procs = 64, nr_iter = 256, blk_sz = 33554432, coll = 1 # total_size = 549755813888 # Write: min_t = 949.175712, max_t = 949.179771, mean_t = 949.177418, var_t = 0.000001 # Read: min_t = 45.579205, max_t = 1330.646948, mean_t = 753.981982, var_t = 170353.835523 Write bandwidth = 579.190403 Mbytes/sec Read bandwidth = 413.149269 Mbytes/sec Correctness test passed. Application 183691 resources: utime ~61806s, stime ~570s, Rss ~4744, inblocks ~0, outblocks ~536872960 + aprun ‐n 64 ‐N 2 ../bin/mpi‐io‐test ‐i 8192 ‐b 1048576 ‐f /lus/husk/dmoen/read_after_write/testdir1.17318/outfile [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = *:romio_no_indep_rw=true:romio_cb_write=enable:cray_cb_write_lock_mode=0:cray_cb_nodes_multiplier=6 PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 Rank 11 of 64 is on nid00041 Rank 10 of 64 is on nid00041 Rank 12 of 64 is on nid00042 Rank 13 of 64 is on nid00042 Rank 21 of 64 is on nid00066 Rank 20 of 64 is on nid00066 Rank 35 of 64 is on nid00105 Rank 34 of 64 is on nid00105 Rank 43 of 64 is on nid00109 Rank 42 of 64 is on nid00109 Rank 45 of 64 is on nid00110 Rank 44 of 64 is on nid00110 Rank 51 of 64 is on nid00113 Rank 50 of 64 is on nid00113 Rank 63 of 64 is on nid00119 Rank 62 of 64 is on nid00119 Rank 61 of 64 is on nid00118 Rank 60 of 64 is on nid00118 Rank 59 of 64 is on nid00117 Rank 58 of 64 is on nid00117 Rank 57 of 64 is on nid00116 Rank 56 of 64 is on nid00116 Rank 55 of 64 is on nid00115 Rank 54 of 64 is on nid00115 Rank 53 of 64 is on nid00114 Rank 52 of 64 is on nid00114 Rank 49 of 64 is on nid00112 Rank 48 of 64 is on nid00112 Rank 47 of 64 is on nid00111 Rank 46 of 64 is on nid00111 Rank 41 of 64 is on nid00108 Rank 40 of 64 is on nid00108 Rank 39 of 64 is on nid00107 Rank 38 of 64 is on nid00107 Rank 37 of 64 is on nid00106 Rank 36 of 64 is on nid00106 Rank 32 of 64 is on nid00104 Rank 33 of 64 is on nid00104 Rank 31 of 64 is on nid00099 Rank 30 of 64 is on nid00099 Rank 28 of 64 is on nid00098 Rank 29 of 64 is on nid00098 Rank 26 of 64 is on nid00097 Rank 27 of 64 is on nid00097 Rank 24 of 64 is on nid00096 Rank 25 of 64 is on nid00096 Rank 23 of 64 is on nid00067 Rank 22 of 64 is on nid00067 Rank 19 of 64 is on nid00065 Rank 18 of 64 is on nid00065 Rank 17 of 64 is on nid00064 Rank 16 of 64 is on nid00064 Rank 15 of 64 is on nid00043 Rank 14 of 64 is on nid00043 Rank 9 of 64 is on nid00040 Rank 8 of 64 is on nid00040 Rank 7 of 64 is on nid00035 Rank 6 of 64 is on nid00035 Rank 4 of 64 is on nid00034 Rank 5 of 64 is on nid00034 Rank 3 of 64 is on nid00033 Rank 2 of 64 is on nid00033 Rank 0 of 64 is on nid00032 Rank 1 of 64 is on nid00032 PE 0: MPIIO hints for /lus/husk/dmoen/read_after_write/testdir1.17318/outfile: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 6 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 1 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/read_after_write/testdir1.17318/outfile | independent writes = 0 | collective writes = 524288 | independent writers = 0 | aggregators = 6 | stripe count = 1 | stripe size = 1048576 | system writes = 524288 | stripe sized writes = 524288 | total bytes for writes = 549755813888 = 524288 MiB = 512 GiB | ave system write size = 1048576 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 24510 | ave write gap size = 33104370 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/read_after_write/testdir1.17318/outfile | independent reads = 314826 | collective reads = 0 | independent readers = 64 | aggregators = 6 | stripe count = 1 | stripe size = 1048576 | system reads = 314826 | stripe sized reads = 314826 | total bytes for reads = 330118987776 = 314826 MiB = 307 GiB | ave system read size = 1048576 | number of read gaps = 309757 | ave read gap size = 19437 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/read_after_write/testdir1.17318/outfile: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 6 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 1 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/read_after_write/testdir1.17318/outfile | independent reads = 0 | collective reads = 524288 | independent readers = 0 | aggregators = 6 | stripe count = 1 | stripe size = 1048576 | system reads = 524288 | stripe sized reads = 524288 | total bytes for reads = 549755813888 = 524288 MiB = 512 GiB | ave system read size = 1048576 | number of read gaps = 0 | ave read gap size = NA | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ nr_procs = 64, nr_iter = 8192, blk_sz = 1048576, coll = 1 # total_size = 549755813888 # Write: min_t = 590.931070, max_t = 590.977739, mean_t = 590.946272, var_t = 0.000095 # Read: min_t = 96.220369, max_t = 1299.179201, mean_t = 780.364960, var_t = 147034.984734 Write bandwidth = 930.247922 Mbytes/sec Read bandwidth = 423.156262 Mbytes/sec Correctness test passed. Application 183692 resources: utime ~63723s, stime ~538s, Rss ~4744, inblocks ~0, outblocks ~536872960 + ls ‐lh /lus/husk/dmoen/read_after_write/testdir1.17318 total 513G ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 21 01:19 outfile + ls ‐lh /lus/husk/dmoen/read_after_write/testdir2.17318 total 513G ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 21 00:31 outfile + ls ‐lh /lus/husk/dmoen/read_after_write/testdir3.17318 total 513G ‐rw‐r‐‐r‐‐ 1 dmoen criemp 513G Apr 20 23:43 outfile + lfs df ‐h UUID bytes Used Available Use% Mounted on husk‐MDT0000_UUID 374.5G 847.4M 348.0G 0% /lus/husk[MDT:0] husk‐MDT0001_UUID 374.5G 77.9M 348.8G 0% /lus/husk[MDT:1] husk‐MDT0002_UUID 374.5G 78.1M 348.8G 0% /lus/husk[MDT:2] husk‐MDT0003_UUID 374.5G 78.1M 348.8G 0% /lus/husk[MDT:3] husk‐OST0000_UUID 7.3T 608.9G 6.3T 9% /lus/husk[OST:0] husk‐OST0001_UUID 7.3T 97.0G 6.8T 1% /lus/husk[OST:1] husk‐OST0002_UUID 7.3T 613.7G 6.3T 9% /lus/husk[OST:2] husk‐OST0003_UUID 7.3T 96.8G 6.8T 1% /lus/husk[OST:3] husk‐OST0004_UUID 7.3T 96.8G 6.8T 1% /lus/husk[OST:4] husk‐OST0005_UUID 7.3T 609.4G 6.3T 9% /lus/husk[OST:5] filesystem summary: 43.6T 2.1T 39.3T 5% /lus/husk + rm ‐rf /lus/husk/dmoen/read_after_write/testdir1.17318 + rm ‐rf /lus/husk/dmoen/read_after_write/testdir2.17318 + rm ‐rf /lus/husk/dmoen/read_after_write/testdir3.17318 + mkdir ‐p /lus/husk/dmoen/read_after_write/testdir1.17318 + mkdir ‐p /lus/husk/dmoen/read_after_write/testdir2.17318 + mkdir ‐p /lus/husk/dmoen/read_after_write/testdir3.17318 + export ’MPICH_MPIIO_HINTS=*:romio_no_indep_rw=true:romio_cb_write=enable:cray_cb_write_lock_mode=4:cray_cb_nodes_multiplier=6’ + MPICH_MPIIO_HINTS=’*:romio_no_indep_rw=true:romio_cb_write=enable:cray_cb_write_lock_mode=4:cray_cb_nodes_multiplier=6’ + aprun ‐n 64 ‐N 2 ../bin/mpi‐io‐test ‐i 128 ‐b 67198864 ‐f /lus/husk/dmoen/read_after_write/testdir3.17318/outfile [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = *:romio_no_indep_rw=true:romio_cb_write=enable:cray_cb_write_lock_mode=4:cray_cb_nodes_multiplier=6 PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 Rank 11 of 64 is on nid00041 Rank 10 of 64 is on nid00041 Rank 13 of 64 is on nid00042 Rank 12 of 64 is on nid00042 Rank 61 of 64 is on nid00118 Rank 60 of 64 is on nid00118 Rank 59 of 64 is on nid00117 Rank 58 of 64 is on nid00117 Rank 55 of 64 is on nid00115 Rank 54 of 64 is on nid00115 Rank 53 of 64 is on nid00114 Rank 52 of 64 is on nid00114 Rank 51 of 64 is on nid00113 Rank 50 of 64 is on nid00113 Rank 49 of 64 is on nid00112 Rank 48 of 64 is on nid00112 Rank 47 of 64 is on nid00111 Rank 46 of 64 is on nid00111 Rank 45 of 64 is on nid00110 Rank 44 of 64 is on nid00110 Rank 42 of 64 is on nid00109 Rank 43 of 64 is on nid00109 Rank 41 of 64 is on nid00108 Rank 40 of 64 is on nid00108 Rank 36 of 64 is on nid00106 Rank 37 of 64 is on nid00106 Rank 34 of 64 is on nid00105 Rank 35 of 64 is on nid00105 Rank 29 of 64 is on nid00098 Rank 28 of 64 is on nid00098 Rank 27 of 64 is on nid00097 Rank 26 of 64 is on nid00097 Rank 23 of 64 is on nid00067 Rank 22 of 64 is on nid00067 Rank 21 of 64 is on nid00066 Rank 20 of 64 is on nid00066 Rank 19 of 64 is on nid00065 Rank 18 of 64 is on nid00065 Rank 17 of 64 is on nid00064 Rank 16 of 64 is on nid00064 Rank 15 of 64 is on nid00043 Rank 14 of 64 is on nid00043 Rank 9 of 64 is on nid00040 Rank 8 of 64 is on nid00040 Rank 5 of 64 is on nid00034 Rank 4 of 64 is on nid00034 Rank 3 of 64 is on nid00033 Rank 2 of 64 is on nid00033 Rank 0 of 64 is on nid00032 Rank 1 of 64 is on nid00032 Rank 6 of 64 is on nid00035 Rank 7 of 64 is on nid00035 Rank 24 of 64 is on nid00096 Rank 25 of 64 is on nid00096 Rank 31 of 64 is on nid00099 Rank 30 of 64 is on nid00099 Rank 33 of 64 is on nid00104 Rank 32 of 64 is on nid00104 Rank 39 of 64 is on nid00107 Rank 38 of 64 is on nid00107 Rank 57 of 64 is on nid00116 Rank 56 of 64 is on nid00116 Rank 63 of 64 is on nid00119 Rank 62 of 64 is on nid00119 PE 0: MPIIO hints for /lus/husk/dmoen/read_after_write/testdir3.17318/outfile: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/read_after_write/testdir3.17318/outfile | independent writes = 0 | collective writes = 8192 | independent writers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system writes = 525119 | stripe sized writes = 524864 | total bytes for writes = 550493093888 = 524991 MiB = 512 GiB | ave system write size = 1048320 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 2624 | ave write gap size = 2338842649 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/read_after_write/testdir3.17318/outfile | independent reads = 4855 | collective reads = 0 | independent readers = 64 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system reads = 4855 | stripe sized reads = 0 | total bytes for reads = 326250484720 = 311136 MiB = 303 GiB | ave system read size = 67198864 | number of read gaps = 4820 | ave read gap size = 55942316 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/read_after_write/testdir3.17318/outfile: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/read_after_write/testdir3.17318/outfile | independent reads = 0 | collective reads = 8192 | independent readers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system reads = 525119 | stripe sized reads = 524864 | total bytes for reads = 550493093888 = 524991 MiB = 512 GiB | ave system read size = 1048320 | number of read gaps = 0 | ave read gap size = NA | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ nr_procs = 64, nr_iter = 128, blk_sz = 67198864, coll = 1 # total_size = 550493093888 # Write: min_t = 168.647661, max_t = 168.651179, mean_t = 168.649415, var_t = 0.000001 # Read: min_t = 38.044702, max_t = 232.195891, mean_t = 189.559952, var_t = 3754.136785 Write bandwidth = 3264.092772 Mbytes/sec Read bandwidth = 2370.813241 Mbytes/sec Correctness test passed. Application 183693 resources: utime ~37997s, stime ~480s, Rss ~4744, inblocks ~0, outblocks ~477862424 + aprun ‐n 64 ‐N 2 ../bin/mpi‐io‐test ‐i 256 ‐b 33554432 ‐f /lus/husk/dmoen/read_after_write/testdir2.17318/outfile [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = *:romio_no_indep_rw=true:romio_cb_write=enable:cray_cb_write_lock_mode=4:cray_cb_nodes_multiplier=6 PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 Rank 9 of 64 is on nid00040 Rank 8 of 64 is on nid00040 Rank 10 of 64 is on nid00041 Rank 11 of 64 is on nid00041 Rank 13 of 64 is on nid00042 Rank 12 of 64 is on nid00042 Rank 35 of 64 is on nid00105 Rank 34 of 64 is on nid00105 Rank 42 of 64 is on nid00109 Rank 43 of 64 is on nid00109 Rank 45 of 64 is on nid00110 Rank 44 of 64 is on nid00110 Rank 51 of 64 is on nid00113 Rank 50 of 64 is on nid00113 Rank 63 of 64 is on nid00119 Rank 62 of 64 is on nid00119 Rank 61 of 64 is on nid00118 Rank 60 of 64 is on nid00118 Rank 59 of 64 is on nid00117 Rank 58 of 64 is on nid00117 Rank 57 of 64 is on nid00116 Rank 56 of 64 is on nid00116 Rank 55 of 64 is on nid00115 Rank 54 of 64 is on nid00115 Rank 52 of 64 is on nid00114 Rank 53 of 64 is on nid00114 Rank 49 of 64 is on nid00112 Rank 48 of 64 is on nid00112 Rank 46 of 64 is on nid00111 Rank 47 of 64 is on nid00111 Rank 41 of 64 is on nid00108 Rank 40 of 64 is on nid00108 Rank 39 of 64 is on nid00107 Rank 38 of 64 is on nid00107 Rank 37 of 64 is on nid00106 Rank 36 of 64 is on nid00106 Rank 33 of 64 is on nid00104 Rank 32 of 64 is on nid00104 Rank 30 of 64 is on nid00099 Rank 31 of 64 is on nid00099 Rank 29 of 64 is on nid00098 Rank 28 of 64 is on nid00098 Rank 27 of 64 is on nid00097 Rank 26 of 64 is on nid00097 Rank 24 of 64 is on nid00096 Rank 25 of 64 is on nid00096 Rank 23 of 64 is on nid00067 Rank 22 of 64 is on nid00067 Rank 20 of 64 is on nid00066 Rank 21 of 64 is on nid00066 Rank 19 of 64 is on nid00065 Rank 18 of 64 is on nid00065 Rank 16 of 64 is on nid00064 Rank 17 of 64 is on nid00064 Rank 15 of 64 is on nid00043 Rank 14 of 64 is on nid00043 Rank 7 of 64 is on nid00035 Rank 6 of 64 is on nid00035 Rank 5 of 64 is on nid00034 Rank 4 of 64 is on nid00034 Rank 3 of 64 is on nid00033 Rank 2 of 64 is on nid00033 Rank 0 of 64 is on nid00032 Rank 1 of 64 is on nid00032 PE 0: MPIIO hints for /lus/husk/dmoen/read_after_write/testdir2.17318/outfile: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/read_after_write/testdir2.17318/outfile | independent writes = 0 | collective writes = 16384 | independent writers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system writes = 524288 | stripe sized writes = 524288 | total bytes for writes = 549755813888 = 524288 MiB = 512 GiB | ave system write size = 1048576 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 5337 | ave write gap size = 1162728930 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/read_after_write/testdir2.17318/outfile | independent reads = 9797 | collective reads = 0 | independent readers = 64 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system reads = 9797 | stripe sized reads = 0 | total bytes for reads = 328732770304 = 313504 MiB = 306 GiB | ave system read size = 33554432 | number of read gaps = 9625 | ave read gap size = 15167474 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/read_after_write/testdir2.17318/outfile: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/read_after_write/testdir2.17318/outfile | independent reads = 0 | collective reads = 16384 | independent readers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system reads = 524288 | stripe sized reads = 524288 | total bytes for reads = 549755813888 = 524288 MiB = 512 GiB | ave system read size = 1048576 | number of read gaps = 0 | ave read gap size = NA | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ nr_procs = 64, nr_iter = 256, blk_sz = 33554432, coll = 1 # total_size = 549755813888 # Write: min_t = 129.889682, max_t = 129.893814, mean_t = 129.891554, var_t = 0.000001 # Read: min_t = 40.687884, max_t = 235.437707, mean_t = 190.515805, var_t = 3570.503398 Write bandwidth = 4232.347920 Mbytes/sec Read bandwidth = 2335.037244 Mbytes/sec Correctness test passed. Application 183694 resources: utime ~38640s, stime ~484s, Rss ~4744, inblocks ~0, outblocks ~477224960 + aprun ‐n 64 ‐N 2 ../bin/mpi‐io‐test ‐i 8192 ‐b 1048576 ‐f /lus/husk/dmoen/read_after_write/testdir1.17318/outfile [PE_16]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000001 [PE_17]: cpumask set to 1 cpu on nid00064, cpumask = 0000000000000000000000000000000000000010 [PE_18]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000001 [PE_19]: cpumask set to 1 cpu on nid00065, cpumask = 0000000000000000000000000000000000000010 [PE_48]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000001 [PE_49]: cpumask set to 1 cpu on nid00112, cpumask = 0000000000000000000000000000000000000010 [PE_46]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_47]: cpumask set to 1 cpu on nid00111, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_42]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_43]: cpumask set to 1 cpu on nid00109, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_38]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_39]: cpumask set to 1 cpu on nid00107, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_36]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_37]: cpumask set to 1 cpu on nid00106, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_34]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_35]: cpumask set to 1 cpu on nid00105, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_32]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_33]: cpumask set to 1 cpu on nid00104, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_24]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000001 [PE_25]: cpumask set to 1 cpu on nid00096, cpumask = 00000000000000000000000000000010 [PE_22]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000001 [PE_23]: cpumask set to 1 cpu on nid00067, cpumask = 0000000000000000000000000000000000000010 [PE_21]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000010 [PE_20]: cpumask set to 1 cpu on nid00066, cpumask = 0000000000000000000000000000000000000001 [PE_2]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000001 [PE_3]: cpumask set to 1 cpu on nid00033, cpumask = 00000000000000000000000000000010 [PE_0]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000001 [PE_1]: cpumask set to 1 cpu on nid00032, cpumask = 00000000000000000000000000000010 [PE_4]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000001 [PE_5]: cpumask set to 1 cpu on nid00034, cpumask = 00000000000000000000000000000010 [PE_6]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000001 [PE_7]: cpumask set to 1 cpu on nid00035, cpumask = 00000000000000000000000000000010 [PE_8]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000001 [PE_9]: cpumask set to 1 cpu on nid00040, cpumask = 00000000000000000000000000000010 [PE_10]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000001 [PE_11]: cpumask set to 1 cpu on nid00041, cpumask = 00000000000000000000000000000010 [PE_12]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000001 [PE_13]: cpumask set to 1 cpu on nid00042, cpumask = 00000000000000000000000000000010 [PE_15]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000010 [PE_14]: cpumask set to 1 cpu on nid00043, cpumask = 00000000000000000000000000000001 [PE_27]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000010 [PE_26]: cpumask set to 1 cpu on nid00097, cpumask = 00000000000000000000000000000001 [PE_28]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000001 [PE_29]: cpumask set to 1 cpu on nid00098, cpumask = 00000000000000000000000000000010 [PE_31]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000010 [PE_30]: cpumask set to 1 cpu on nid00099, cpumask = 00000000000000000000000000000001 [PE_40]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_41]: cpumask set to 1 cpu on nid00108, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_44]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000001 [PE_45]: cpumask set to 1 cpu on nid00110, cpumask = 0000000000000000000000000000000000000000000000000000000000000010 [PE_50]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000001 [PE_51]: cpumask set to 1 cpu on nid00113, cpumask = 0000000000000000000000000000000000000010 [PE_52]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000001 [PE_53]: cpumask set to 1 cpu on nid00114, cpumask = 0000000000000000000000000000000000000010 [PE_54]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000001 [PE_55]: cpumask set to 1 cpu on nid00115, cpumask = 0000000000000000000000000000000000000010 [PE_56]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000001 [PE_57]: cpumask set to 1 cpu on nid00116, cpumask = 0000000000000000000000000000000000000010 [PE_58]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000001 [PE_59]: cpumask set to 1 cpu on nid00117, cpumask = 0000000000000000000000000000000000000010 [PE_60]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000001 [PE_61]: cpumask set to 1 cpu on nid00118, cpumask = 0000000000000000000000000000000000000010 [PE_62]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000001 [PE_63]: cpumask set to 1 cpu on nid00119, cpumask = 0000000000000000000000000000000000000010 MPI VERSION : CRAY MPICH version 7.3.2 (ANL base 3.2rc1) MPI BUILD INFO : Built Tue Feb 23 17:49:31 2016 (git hash 5ef867d) MT‐G PE 0: MPICH MPIIO environment settings: PE 0: MPICH_MPIIO_HINTS_DISPLAY = 1 PE 0: MPICH_MPIIO_HINTS = *:romio_no_indep_rw=true:romio_cb_write=enable:cray_cb_write_lock_mode=4:cray_cb_nodes_multiplier=6 PE 0: MPICH_MPIIO_ABORT_ON_RW_ERROR = enable PE 0: MPICH_MPIIO_CB_ALIGN = 2 PE 0: MPICH MPIIO statistics environment settings: PE 0: MPICH_MPIIO_STATS = 1 Rank 11 of 64 is on nid00041 Rank 10 of 64 is on nid00041 Rank 53 of 64 is on nid00114 Rank 52 of 64 is on nid00114 Rank 51 of 64 is on nid00113 Rank 50 of 64 is on nid00113 Rank 49 of 64 is on nid00112 Rank 48 of 64 is on nid00112 Rank 47 of 64 is on nid00111 Rank 46 of 64 is on nid00111 Rank 45 of 64 is on nid00110 Rank 44 of 64 is on nid00110 Rank 43 of 64 is on nid00109 Rank 42 of 64 is on nid00109 Rank 41 of 64 is on nid00108 Rank 40 of 64 is on nid00108 Rank 37 of 64 is on nid00106 Rank 36 of 64 is on nid00106 Rank 34 of 64 is on nid00105 Rank 35 of 64 is on nid00105 Rank 29 of 64 is on nid00098 Rank 28 of 64 is on nid00098 Rank 26 of 64 is on nid00097 Rank 27 of 64 is on nid00097 Rank 21 of 64 is on nid00066 Rank 20 of 64 is on nid00066 Rank 19 of 64 is on nid00065 Rank 18 of 64 is on nid00065 Rank 15 of 64 is on nid00043 Rank 14 of 64 is on nid00043 Rank 12 of 64 is on nid00042 Rank 13 of 64 is on nid00042 Rank 8 of 64 is on nid00040 Rank 9 of 64 is on nid00040 Rank 4 of 64 is on nid00034 Rank 5 of 64 is on nid00034 Rank 3 of 64 is on nid00033 Rank 2 of 64 is on nid00033 Rank 0 of 64 is on nid00032 Rank 1 of 64 is on nid00032 Rank 7 of 64 is on nid00035 Rank 6 of 64 is on nid00035 Rank 17 of 64 is on nid00064 Rank 16 of 64 is on nid00064 Rank 23 of 64 is on nid00067 Rank 22 of 64 is on nid00067 Rank 25 of 64 is on nid00096 Rank 24 of 64 is on nid00096 Rank 31 of 64 is on nid00099 Rank 30 of 64 is on nid00099 Rank 32 of 64 is on nid00104 Rank 33 of 64 is on nid00104 Rank 39 of 64 is on nid00107 Rank 38 of 64 is on nid00107 Rank 55 of 64 is on nid00115 Rank 54 of 64 is on nid00115 Rank 56 of 64 is on nid00116 Rank 57 of 64 is on nid00116 Rank 59 of 64 is on nid00117 Rank 58 of 64 is on nid00117 Rank 61 of 64 is on nid00118 Rank 60 of 64 is on nid00118 Rank 63 of 64 is on nid00119 Rank 62 of 64 is on nid00119 PE 0: MPIIO hints for /lus/husk/dmoen/read_after_write/testdir1.17318/outfile: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO write access patterns for /lus/husk/dmoen/read_after_write/testdir1.17318/outfile | independent writes = 0 | collective writes = 524288 | independent writers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system writes = 524288 | stripe sized writes = 524288 | total bytes for writes = 549755813888 = 524288 MiB = 512 GiB | ave system write size = 1048576 | read‐modify‐write count = 0 | read‐modify‐write bytes = 0 | number of write gaps = 130708 | ave write gap size = 28275235 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/read_after_write/testdir1.17318/outfile | independent reads = 314878 | collective reads = 0 | independent readers = 64 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system reads = 314878 | stripe sized reads = 314878 | total bytes for reads = 330173513728 = 314878 MiB = 307 GiB | ave system read size = 1048576 | number of read gaps = 180932 | ave read gap size = 13856 | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ PE 0: MPIIO hints for /lus/husk/dmoen/read_after_write/testdir1.17318/outfile: cb_buffer_size = 16777216 romio_cb_read = enable romio_cb_write = enable cb_nodes = 36 cb_align = 2 romio_no_indep_rw = true romio_cb_pfr = disable romio_cb_fr_types = aar romio_cb_fr_alignment = 1 romio_cb_ds_threshold = 0 romio_cb_alltoall = automatic ind_rd_buffer_size = 4194304 ind_wr_buffer_size = 524288 romio_ds_read = disable romio_ds_write = disable striping_factor = 6 striping_unit = 1048576 romio_lustre_start_iodevice = 0 direct_io = false aggregator_placement_stride = ‐1 abort_on_rw_error = enable cray_cb_nodes_multiplier = 6 cray_cb_write_lock_mode = 4 cray_cb_lock_ahead_num_extents = 500 cray_cb_lock_ahead_flags = 0 cb_config_list = *:* romio_filesystem_type = CRAY ADIO: +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | MPIIO read access patterns for /lus/husk/dmoen/read_after_write/testdir1.17318/outfile | independent reads = 0 | collective reads = 524288 | independent readers = 0 | aggregators = 36 | stripe count = 6 | stripe size = 1048576 | system reads = 524288 | stripe sized reads = 524288 | total bytes for reads = 549755813888 = 524288 MiB = 512 GiB | ave system read size = 1048576 | number of read gaps = 0 | ave read gap size = NA | See "Optimizing MPI I/O on Cray XE Systems" S‐0013‐20 for explanations. +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ nr_procs = 64, nr_iter = 8192, blk_sz = 1048576, coll = 1 # total_size = 549755813888 # Write: min_t = 30.154866, max_t = 30.219611, mean_t = 30.177388, var_t = 0.000204 # Read: min_t = 88.496904, max_t = 236.377858, mean_t = 175.751888, var_t = 2253.048226 Write bandwidth = 18192.021284 Mbytes/sec Read bandwidth = 2325.750041 Mbytes/sec Correctness test passed. Application 183695 resources: utime ~40089s, stime ~451s, Rss ~4744, inblocks ~0, outblocks ~477224960 + ls ‐lh /lus/husk/dmoen/read_after_write/testdir1.17318 total 513G ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 21 03:06 outfile + ls ‐lh /lus/husk/dmoen/read_after_write/testdir2.17318 total 513G ‐rw‐r‐‐r‐‐ 1 dmoen criemp 512G Apr 21 02:36 outfile + ls ‐lh /lus/husk/dmoen/read_after_write/testdir3.17318 total 513G ‐rw‐r‐‐r‐‐ 1 dmoen criemp 513G Apr 21 02:07 outfile + lfs df ‐h UUID bytes Used Available Use% Mounted on husk‐MDT0000_UUID 374.5G 847.4M 348.0G 0% /lus/husk[MDT:0] husk‐MDT0001_UUID 374.5G 77.9M 348.8G 0% /lus/husk[MDT:1] husk‐MDT0002_UUID 374.5G 78.1M 348.8G 0% /lus/husk[MDT:2] husk‐MDT0003_UUID 374.5G 78.1M 348.8G 0% /lus/husk[MDT:3] husk‐OST0000_UUID 7.3T 353.0G 6.5T 5% /lus/husk[OST:0] husk‐OST0001_UUID 7.3T 353.1G 6.5T 5% /lus/husk[OST:1] husk‐OST0002_UUID 7.3T 357.9G 6.5T 5% /lus/husk[OST:2] husk‐OST0003_UUID 7.3T 352.9G 6.5T 5% /lus/husk[OST:3] husk‐OST0004_UUID 7.3T 352.9G 6.5T 5% /lus/husk[OST:4] husk‐OST0005_UUID 7.3T 352.9G 6.5T 5% /lus/husk[OST:5] filesystem summary: 43.6T 2.1T 39.3T 5% /lus/husk + rm ‐rf /lus/husk/dmoen/read_after_write/testdir1.17318 + rm ‐rf /lus/husk/dmoen/read_after_write/testdir2.17318 + rm ‐rf /lus/husk/dmoen/read_after_write/testdir3.17318 + rmdir /lus/husk/dmoen/read_after_write Test results from saturn‐CLFS‐rts‐Apr21 Attached are the results from the IOR lock ahead test runs on saturn‐p2 last night. This includes David's request for posix shared file I/O testing. All tests passed and no downed nodes. saturn‐p2 has 1 lustre file system. It is: /lus/husk saturn‐p2’s lustre client is 2.7.1 Lustre client build: 2.7.0‐trunk‐1.0000.e442c1f.67.7‐abuild‐lustre‐filesystem.git@e442c1f‐2016‐04‐16‐22:28 /lus/husk is a CLFS file system /lus/husk’s lustre server is running 2.7.1 The luster server build is: 2.7.0‐trunk‐1.0000.f114a17.0.0‐jmiller‐lustre‐filesystem.git@f114a17‐2016‐04‐19‐15:04 /lus/husk has 6 OSTs. The ESMS is husk‐esms1 The MDS is node husk‐esf1‐mds1. The OSS node(s): husk‐esf1‐oss1 husk‐esf1‐oss2 husk‐esf1‐oss3 husk‐esf1‐oss4 husk‐esf1‐oss5 husk‐esf1‐oss6 clients = 64 (2 per node) blocksize = 8 GiB filesize = 512 GiB One cb aggregator per OST (cray_cb_nodes_multiplier = 6) sl = Standard Lustre Locking gl = Group Lock la = Lock Ahead fpp = Posix file per process shr = Posix shared file dir = Posix direct buf = Posix buffered Test Op Xfer Size Rate (MiB/sec) CL_IOR_all_mpiio_cb_gl_w_64m write 67108864 2528.005 CL_IOR_all_mpiio_cb_la_w_64m write 67108864 2530.037 CL_IOR_all_hdf5_cb_sl_w_64m write 67108864 1666.582 CL_IOR_all_hdf5_cb_la_w_64m write 67108864 2541.630 CL_IOR_all_posix_dir_fpp_w_64m write 67108864 1329.662 CL_IOR_all_posix_buf_fpp_w_64m write 67108864 2008.135 CL_IOR_all_posix_buf_shr_w_64m write 67108864 2105.160 CL_IOR_all_mpiio_cb_gl_w_32m write 33554432 2446.070 CL_IOR_all_mpiio_cb_la_w_32m write 33554432 2447.365 CL_IOR_all_hdf5_cb_sl_w_32m write 33554432 1730.573 CL_IOR_all_hdf5_cb_la_w_32m write 33554432 2650.620 CL_IOR_all_posix_dir_fpp_w_32m write 33554432 1737.832 CL_IOR_all_posix_buf_fpp_w_32m write 33554432 2683.552 CL_IOR_all_posix_buf_shr_w_32m write 33554432 2424.978 CL_IOR_all_mpiio_cb_gl_w_1m write 1048576 2312.579 CL_IOR_all_mpiio_cb_la_w_1m write 1048576 2286.717 CL_IOR_all_hdf5_cb_sl_w_1m write 1048576 689.288 CL_IOR_all_hdf5_cb_la_w_1m write 1048576 1781.704 CL_IOR_all_posix_dir_fpp_w_1m write 1048576 1551.678 CL_IOR_all_posix_buf_fpp_w_1m write 1048576 2362.193 CL_IOR_all_posix_buf_shr_w_1m write 1048576 2396.071 Test results from saturn‐RAM‐rts‐May03 Attached are my results from the automated IOR lock ahead test runs on saturn‐p2 using the ram‐backed fs. One attachment is a summary of the test runs and second is the entire test output. The posix, buffered, file per process tests failed; out of disk space. From David Knaak on posix fpp test failures: I think the reason that the POSIX FPP tests failed is that the files were not evenly spread across OSTs and so some filled up. It is a little tricky to set it up to spread them evenly ‐ Doug Petesch does it for his FPP benchmarks. I have the script somewhere for this. System Configuration: OS CNL ARCH XC: DATE Tue May 3 14:24:53 CDT 2016 UNAME UNICOSlc saturn‐p2 6.1.104DV USER 22569 LUSTRE /lus/largeram ARCHITECTURE XC RELEASE 312 CONFIG "" ENV TESTDIR Lustre MPI RUNCMD EXPECT TSTTOOL APPS CPUSET ALPS date 05/03/16 tag tcid testcase status contact ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ CL_lustre_info ‐ ‐ pass overby CL_IOR_all_mpiio_cb_gl_w_1m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_gl_r_1m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_la_w_1m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_la_r_1m ‐ ‐ pass dmoen CL_IOR_all_hdf5_cb_sl_w_1m ‐ ‐ pass dmoen CL_IOR_all_hdf5_cb_sl_r_1m ‐ ‐ pass dmoen CL_IOR_all_hdf5_cb_la_w_1m ‐ ‐ pass dmoen CL_IOR_all_hdf5_cb_la_r_1m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_fpp_w_1m ‐ ‐ fail dmoen CL_IOR_all_posix_buf_fpp_r_1m ‐ ‐ fail dmoen CL_IOR_all_posix_dir_fpp_w_1m ‐ ‐ pass dmoen CL_IOR_all_posix_dir_fpp_r_1m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_shr_w_1m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_shr_r_1m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_gl_w_32m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_gl_r_32m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_la_w_32m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_la_r_32m ‐ ‐ pass dmoen CL_IOR_all_hdf5_cb_sl_w_32m ‐ ‐ pass dmoen CL_IOR_all_hdf5_cb_sl_r_32m ‐ ‐ pass dmoen CL_IOR_all_hdf5_cb_la_w_32m ‐ ‐ pass dmoen CL_IOR_all_hdf5_cb_la_r_32m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_fpp_w_32m ‐ ‐ fail dmoen CL_IOR_all_posix_buf_fpp_r_32m ‐ ‐ fail dmoen CL_IOR_all_posix_dir_fpp_w_32m ‐ ‐ pass dmoen CL_IOR_all_posix_dir_fpp_r_32m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_shr_w_32m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_shr_r_32m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_gl_w_64m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_gl_r_64m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_la_w_64m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_la_r_64m ‐ ‐ pass dmoen CL_IOR_all_hdf5_cb_sl_w_64m ‐ ‐ pass dmoen CL_IOR_all_hdf5_cb_sl_r_64m ‐ ‐ pass dmoen CL_IOR_all_hdf5_cb_la_w_64m ‐ ‐ pass dmoen CL_IOR_all_hdf5_cb_la_r_64m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_fpp_w_64m ‐ ‐ fail dmoen CL_IOR_all_posix_buf_fpp_r_64m ‐ ‐ fail dmoen CL_IOR_all_posix_dir_fpp_w_64m ‐ ‐ pass dmoen CL_IOR_all_posix_dir_fpp_r_64m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_shr_w_64m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_shr_r_64m ‐ ‐ pass dmoen UUID 1K‐blocks Used Available Use% Mounted on largeram‐MDT0000_UUID 22802556 464800 20776856 2% /lus/largeram[MDT:0] largeram‐MDT0001_UUID 22802556 464716 20776940 2% /lus/largeram[MDT:1] largeram‐OST0000_UUID 99240380 49169960 44995100 52% /lus/largeram[OST:0] largeram‐OST0001_UUID 99240380 49099888 45000772 52% /lus/largeram[OST:1] largeram‐OST0002_UUID 99240380 49137748 44999848 52% /lus/largeram[OST:2] largeram‐OST0003_UUID 99240380 49099860 45000800 52% /lus/largeram[OST:3] largeram‐OST0004_UUID 99240380 49099868 44999768 52% /lus/largeram[OST:4] largeram‐OST0005_UUID 99240380 49099856 44997732 52% /lus/largeram[OST:5] largeram‐OST0006_UUID 99240380 49099860 45000800 52% /lus/largeram[OST:6] largeram‐OST0007_UUID 99240380 49165392 44946608 52% /lus/largeram[OST:7] filesystem summary: 793923040 392972432 359941428 52% /lus/largeram saturn‐p2 has 1 lustre file system. It is: /lus/largeram saturn‐p2’s lustre client is 2.7.1 Lustre client build: 2.7.0‐trunk‐1.0000.1f777d0.0.0‐jmiller‐lustre‐filesystem.git@1f777d0‐2016‐05‐02‐21:33 /lus/largeram is a CLFS file system /lus/largeram’s lustre server is running The luster server build is: /lus/largeram has 8 OSTs. The ESMS is The MDS is node . The OSS node(s): saturn‐p2 /lus/largeram CLFS 2.7.1 2.7.0‐trunk‐1.0000.1f777d0.0.0‐jmiller‐lustre‐filesystem.git@1f777d0‐2016‐05‐02‐21:33 clients = 64 (2 per node) aggregators = 16 (cray_cb_nodes_multiplier = 2) sl: standard lock gl: group lock la: lock ahead Test Op Xfer Size Rate (MiB/sec) CL_IOR_all_mpiio_cb_gl_w_1m write 1048576 794.334 CL_IOR_all_mpiio_cb_la_w_1m write 1048576 755.131 CL_IOR_all_hdf5_cb_sl_w_1m write 1048576 544.863 CL_IOR_all_hdf5_cb_la_w_1m write 1048576 682.028 CL_IOR_all_posix_dir_fpp_w_1m write 1048576 15270.327 CL_IOR_all_posix_buf_shr_w_1m write 1048576 4405.359 CL_IOR_all_mpiio_cb_gl_w_32m write 33554432 7030.617 CL_IOR_all_mpiio_cb_la_w_32m write 33554432 6677.504 CL_IOR_all_hdf5_cb_sl_w_32m write 33554432 2808.108 CL_IOR_all_hdf5_cb_la_w_32m write 33554432 5599.880 CL_IOR_all_posix_dir_fpp_w_32m write 33554432 13398.811 CL_IOR_all_posix_buf_shr_w_32m write 33554432 4414.457 CL_IOR_all_mpiio_cb_gl_w_64m write 67108864 7175.958 CL_IOR_all_mpiio_cb_la_w_64m write 67108864 6810.539 CL_IOR_all_hdf5_cb_sl_w_64m write 67108864 3023.737 CL_IOR_all_hdf5_cb_la_w_64m write 67108864 5153.182 CL_IOR_all_posix_dir_fpp_w_64m write 67108864 14611.597 CL_IOR_all_posix_buf_shr_w_64m write 67108864 4189.254 Test results from pollux‐SU16‐rts‐May20 Attached are my results from the automated IOR lock ahead test runs on pollux‐p6 using snx11011 with su 16 (lock ahead) installed. In addition to the lock ahead tests, I ran the RTS regression write_verify and relrun io test suites. IOR hdf5 test failures were due to missing cray pe libraries. error while loading shared libraries: libmpich_cray.so.3: cannot open shared object file: No such file or directory One (1) downed compute node while running RTS suite (relrun io). LustreError: 3609:0:(osc_cache.c:2954:osc_cache_writeback_range()) ASSERTION( !ext‐>oe_hp ) failed: LustreError: 3609:0:(osc_cache.c:2954:osc_cache_writeback_range()) LBUG Two (2) read tests failed due to timeouts; slow reads. These tests pass, written data is good, if given more time to complete. pollux‐p6’s lustre client is 2.7.1 Lustre client build: 2.7.0‐trunk‐1.0502.081230a.0.0‐jmiller‐lustre‐filesystem.git@081230a‐2016‐05‐16‐15:28 /lus/snx11011 lustre server is running 2.5.1 The lustre server build is: jenkins‐Changeling_Lustre‐279‐279‐gc49bc98‐CHANGED‐2.6.32‐431.17.1.x2.0.76.x86_64 /lus/snx11011 has 16 OSTs The MDS is node snx11011n003. The OSS node(s): snx11011n004 snx11011n005 snx11011n006 snx11011n007 clients = 64 (2 per node) blocksize = 8 GiB filesize = 512 GiB sl = Standard Lustre Locking (cray_cb_write_lock_mode = 0) gl = Group Lock (cray_cb_write_lock_mode = 1) la = Lock Ahead (cray_cb_write_lock_mode = 4) fpp = Posix file per process shr = Posix shared file dir = Posix direct buf = Posix buffered System Configuration: OS CNL ARCH XC: DATE Thu May 19 15:09:41 CDT 2016 UNAME UNICOSlc pollux‐p6 5.2.82 USER 22569 LUSTRE /lus/snx11011 ARCHITECTURE XC RELEASE 5.2.82 CONFIG "" ENV TESTDIR Lustre MPI EXPECT TSTTOOL APPS CPUSET ALPS date 05/19/16 tag tcid testcase status contact ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ CL_lustre_info ‐ ‐ pass overby CL_IOR_all_mpiio_cb_gl_w_1m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_gl_r_1m ‐ ‐ timeout dmoen CL_IOR_all_mpiio_cb_la_w_1m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_la_r_1m ‐ ‐ timeout dmoen CL_IOR_all_posix_buf_fpp_w_1m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_fpp_r_1m ‐ ‐ pass dmoen CL_IOR_all_posix_dir_fpp_w_1m ‐ ‐ pass dmoen CL_IOR_all_posix_dir_fpp_r_1m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_shr_w_1m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_shr_r_1m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_gl_w_32m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_gl_r_32m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_la_w_32m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_la_r_32m ‐ ‐ pass dmoen CL_IOR_all_hdf5_cb_sl_w_32m ‐ ‐ fail dmoen CL_IOR_all_hdf5_cb_sl_r_32m ‐ ‐ fail dmoen CL_IOR_all_hdf5_cb_la_w_32m ‐ ‐ fail dmoen CL_IOR_all_hdf5_cb_la_r_32m ‐ ‐ fail dmoen CL_IOR_all_posix_buf_fpp_w_32m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_fpp_r_32m ‐ ‐ pass dmoen CL_IOR_all_posix_dir_fpp_w_32m ‐ ‐ pass dmoen CL_IOR_all_posix_dir_fpp_r_32m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_shr_w_32m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_shr_r_32m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_gl_w_64m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_gl_r_64m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_la_w_64m ‐ ‐ pass dmoen CL_IOR_all_mpiio_cb_la_r_64m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_fpp_w_64m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_fpp_r_64m ‐ ‐ pass dmoen CL_IOR_all_posix_dir_fpp_w_64m ‐ ‐ pass dmoen CL_IOR_all_posix_dir_fpp_r_64m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_shr_w_64m ‐ ‐ pass dmoen CL_IOR_all_posix_buf_shr_r_64m ‐ ‐ pass dmoen ***** Results (out.all) ***** Test Op Xfer Size Cb Nodes Multi Rate (MiB/sec) CL_IOR_all_mpiio_cb_gl_w_1m write 1048576 4 339.906 CL_IOR_all_mpiio_cb_la_w_1m write 1048576 4 283.526 CL_IOR_all_posix_buf_fpp_w_1m write 1048576 0 7815.367 CL_IOR_all_posix_dir_fpp_w_1m write 1048576 0 1075.029 CL_IOR_all_posix_buf_shr_w_1m write 1048576 0 5950.421 CL_IOR_all_mpiio_cb_gl_w_32m write 33554432 4 2373.630 CL_IOR_all_mpiio_cb_la_w_32m write 33554432 4 2413.987 CL_IOR_all_posix_buf_fpp_w_32m write 33554432 0 6542.209 CL_IOR_all_posix_dir_fpp_w_32m write 33554432 0 1285.084 CL_IOR_all_posix_buf_shr_w_32m write 33554432 0 5663.417 CL_IOR_all_mpiio_cb_gl_w_64m write 67108864 4 1490.737 CL_IOR_all_mpiio_cb_la_w_64m write 67108864 4 3331.462 CL_IOR_all_posix_buf_fpp_w_64m write 67108864 0 7559.196 CL_IOR_all_posix_dir_fpp_w_64m write 67108864 0 520.442 CL_IOR_all_posix_buf_shr_w_64m write 67108864 0 1355.851 CL_IOR_all_posix_buf_fpp_r_1m read 1048576 0 4476.193 CL_IOR_all_posix_dir_fpp_r_1m read 1048576 0 1409.612 CL_IOR_all_posix_buf_shr_r_1m read 1048576 0 3840.387 CL_IOR_all_mpiio_cb_gl_r_32m read 33554432 4 736.051 CL_IOR_all_mpiio_cb_la_r_32m read 33554432 4 737.732 CL_IOR_all_posix_buf_fpp_r_32m read 33554432 0 2460.537 CL_IOR_all_posix_dir_fpp_r_32m read 33554432 0 697.613 CL_IOR_all_posix_buf_shr_r_32m read 33554432 0 3521.591 CL_IOR_all_mpiio_cb_gl_r_64m read 67108864 4 721.895 CL_IOR_all_mpiio_cb_la_r_64m read 67108864 4 2320.702 CL_IOR_all_posix_buf_fpp_r_64m read 67108864 0 4115.056 CL_IOR_all_posix_dir_fpp_r_64m read 67108864 0 337.546 CL_IOR_all_posix_buf_shr_r_64m read 67108864 0 2984.182 CL_IOR_all_mpiio_cb_gl_r_1m FAIL CL_IOR_all_mpiio_cb_la_r_1m FAIL CL_IOR_all_hdf5_cb_sl_w_32m FAIL CL_IOR_all_hdf5_cb_sl_r_32m FAIL CL_IOR_all_hdf5_cb_la_w_32m FAIL CL_IOR_all_hdf5_cb_la_r_32m FAIL ***** Command lines (out.all) ***** CL_IOR_all_mpiio_cb_gl_w_1m write /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_mpiio_cb_gl_w_1m/IORfile_1m ‐a MPIIO ‐w ‐H ‐C ‐k ‐c ‐t 1m ‐b 8g ‐s 1 CL_IOR_all_mpiio_cb_la_w_1m write /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_mpiio_cb_la_w_1m/IORfile_1m ‐a MPIIO ‐w ‐H ‐C ‐k ‐c ‐t 1m ‐b 8g ‐s 1 CL_IOR_all_posix_buf_fpp_w_1m write /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_buf_fpp_w_1m/IORfile_1m ‐a POSIX ‐w ‐F ‐e ‐g ‐C ‐k ‐t 1m ‐b 8g ‐s 1 CL_IOR_all_posix_dir_fpp_w_1m write /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_dir_fpp_w_1m/IORfile_1m ‐a POSIX ‐w ‐F ‐e ‐g ‐C ‐k ‐B ‐t 1m ‐b 8g ‐s 1 CL_IOR_all_posix_buf_shr_w_1m write /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_buf_shr_w_1m/IORfile_1m ‐a POSIX ‐w ‐e ‐g ‐C ‐k ‐t 1m ‐b 8g ‐s 1 CL_IOR_all_mpiio_cb_gl_w_32m write /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_mpiio_cb_gl_w_32m/IORfile_32m ‐a MPIIO ‐w ‐H ‐C ‐k ‐c ‐t 32m ‐b 8g ‐s 1 CL_IOR_all_mpiio_cb_la_w_32m write /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_mpiio_cb_la_w_32m/IORfile_32m ‐a MPIIO ‐w ‐H ‐C ‐k ‐c ‐t 32m ‐b 8g ‐s 1 CL_IOR_all_posix_buf_fpp_w_32m write /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_buf_fpp_w_32m/IORfile_32m ‐a POSIX ‐w ‐F ‐e ‐g ‐C ‐k ‐t 32m ‐b 8g ‐s 1 CL_IOR_all_posix_dir_fpp_w_32m write /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_dir_fpp_w_32m/IORfile_32m ‐a POSIX ‐w ‐F ‐e ‐g ‐C ‐k ‐B ‐t 32m ‐b 8g ‐s 1 CL_IOR_all_posix_buf_shr_w_32m write /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_buf_shr_w_32m/IORfile_32m ‐a POSIX ‐w ‐e ‐g ‐C ‐k ‐t 32m ‐b 8g ‐s 1 CL_IOR_all_mpiio_cb_gl_w_64m write /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_mpiio_cb_gl_w_64m/IORfile_64m ‐a MPIIO ‐w ‐H ‐C ‐k ‐c ‐t 64m ‐b 8g ‐s 1 CL_IOR_all_mpiio_cb_la_w_64m write /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_mpiio_cb_la_w_64m/IORfile_64m ‐a MPIIO ‐w ‐H ‐C ‐k ‐c ‐t 64m ‐b 8g ‐s 1 CL_IOR_all_posix_buf_fpp_w_64m write /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_buf_fpp_w_64m/IORfile_64m ‐a POSIX ‐w ‐F ‐e ‐g ‐C ‐k ‐t 64m ‐b 8g ‐s 1 CL_IOR_all_posix_dir_fpp_w_64m write /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_dir_fpp_w_64m/IORfile_64m ‐a POSIX ‐w ‐F ‐e ‐g ‐C ‐k ‐B ‐t 64m ‐b 8g ‐s 1 CL_IOR_all_posix_buf_shr_w_64m write /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_buf_shr_w_64m/IORfile_64m ‐a POSIX ‐w ‐e ‐g ‐C ‐k ‐t 64m ‐b 8g ‐s 1 CL_IOR_all_posix_buf_fpp_r_1m read /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_buf_fpp_w_1m/IORfile_1m ‐a POSIX ‐r ‐F ‐e ‐g ‐C ‐k ‐t 1m ‐b 8g ‐s 1 CL_IOR_all_posix_dir_fpp_r_1m read /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_dir_fpp_w_1m/IORfile_1m ‐a POSIX ‐r ‐F ‐e ‐g ‐C ‐k ‐B ‐t 1m ‐b 8g ‐s 1 CL_IOR_all_posix_buf_shr_r_1m read /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_buf_shr_w_1m/IORfile_1m ‐a POSIX ‐r ‐e ‐g ‐C ‐k ‐t 1m ‐b 8g ‐s 1 CL_IOR_all_mpiio_cb_gl_r_32m read /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_mpiio_cb_gl_w_32m/IORfile_32m ‐a MPIIO ‐r ‐H ‐C ‐k ‐c ‐t 32m ‐b 8g ‐s 1 CL_IOR_all_mpiio_cb_la_r_32m read /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_mpiio_cb_la_w_32m/IORfile_32m ‐a MPIIO ‐r ‐H ‐C ‐k ‐c ‐t 32m ‐b 8g ‐s 1 CL_IOR_all_posix_buf_fpp_r_32m read /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_buf_fpp_w_32m/IORfile_32m ‐a POSIX ‐r ‐F ‐e ‐g ‐C ‐k ‐t 32m ‐b 8g ‐s 1 CL_IOR_all_posix_dir_fpp_r_32m read /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_dir_fpp_w_32m/IORfile_32m ‐a POSIX ‐r ‐F ‐e ‐g ‐C ‐k ‐B ‐t 32m ‐b 8g ‐s 1 CL_IOR_all_posix_buf_shr_r_32m read /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_buf_shr_w_32m/IORfile_32m ‐a POSIX ‐r ‐e ‐g ‐C ‐k ‐t 32m ‐b 8g ‐s 1 CL_IOR_all_mpiio_cb_gl_r_64m read /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_mpiio_cb_gl_w_64m/IORfile_64m ‐a MPIIO ‐r ‐H ‐C ‐k ‐c ‐t 64m ‐b 8g ‐s 1 CL_IOR_all_mpiio_cb_la_r_64m read /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_mpiio_cb_la_w_64m/IORfile_64m ‐a MPIIO ‐r ‐H ‐C ‐k ‐c ‐t 64m ‐b 8g ‐s 1 CL_IOR_all_posix_buf_fpp_r_64m read /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_buf_fpp_w_64m/IORfile_64m ‐a POSIX ‐r ‐F ‐e ‐g ‐C ‐k ‐t 64m ‐b 8g ‐s 1 CL_IOR_all_posix_dir_fpp_r_64m read /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_dir_fpp_w_64m/IORfile_64m ‐a POSIX ‐r ‐F ‐e ‐g ‐C ‐k ‐B ‐t 64m ‐b 8g ‐s 1 CL_IOR_all_posix_buf_shr_r_64m read /cray/css/ostest/binaries/xt/rel.52.gemini.cray/xtcnl/apps/ROOT.latest/perf/IOR‐2.10.3/RUN/IOR ‐o /lus/snx11011/dmoen/CL_IOR_all_posix_buf_shr_w_64m/IORfile_64m ‐a POSIX ‐r ‐e ‐g ‐C ‐k ‐t 64m ‐b 8g ‐s 1