================ test started at: Thu May 12 23:43:13 PDT 2011 ================

=============================== disks to be used ===============================
/dev/md0

================================ create the PVs ================================
  Physical volume "/dev/md0" successfully created

================================ create the VG ================================
  Volume group "large_vg" successfully created
  --- Volume group ---
  VG Name               large_vg
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               199.39 TB
  PE Size               4.00 MB
  Total PE              52269049
  Alloc PE / Size       0 / 0   
  Free  PE / Size       52269049 / 199.39 TB
  VG UUID               w1uEH1-EwuA-736R-THRY-gvBW-MS6x-0gSWkd
   

================================ create the LVs ================================
  Logical volume "mdt_lv" created
  Logical volume "ost_lv" created
  --- Logical volume ---
  LV Name                /dev/large_vg/mdt_lv
  VG Name                large_vg
  LV UUID                lh1J8T-1SKy-eSeC-x6a2-prHx-Pilv-An7AY3
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                2.00 TB
  Current LE             524288
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     57344
  Block device           253:2
   
  --- Logical volume ---
  LV Name                /dev/large_vg/ost_lv
  VG Name                large_vg
  LV UUID                EsaifZ-7Nk0-CnQC-zt8J-yAEb-DDi3-AYX2E2
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                128.00 TB
  Current LE             33554432
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     57344
  Block device           253:3
   

================= run llverdev on the OST /dev/large_vg/ost_lv =================
# time llverdev -vpf /dev/large_vg/ost_lv
llverdev: /dev/large_vg/ost_lv is 140737488355328 bytes (131072 GB) in size
Timestamp: 1305268994
write offset:     1755316224kB 669.8 MB/s            
write offset:     3619684352kB 711.2 MB/s            
write offset:     5427429376kB 689.6 MB/s            
write offset:     7103053824kB 639.2 MB/s            
write offset:     8691646464kB   606 MB/s            
write offset:    10267656192kB 601.2 MB/s            
write offset:    11737759744kB 560.8 MB/s            
write offset:    13160677376kB 542.8 MB/s            
write offset:    14519631872kB 518.4 MB/s            
write offset:    16419651584kB 724.8 MB/s            
write offset:    19901972480kB 1328.4 MB/s            
write offset:    24106762240kB  1604 MB/s            
write offset:    27987542016kB 1480.4 MB/s            
write offset:    31901876224kB 1493.2 MB/s            
write offset:    34664873984kB  1054 MB/s            
write offset:    37505466368kB 1083.6 MB/s            
write offset:    40507539456kB 1145.2 MB/s            
write offset:    43355471872kB 1086.4 MB/s            
write offset:    46105886720kB 1049.2 MB/s            
write offset:    48974790656kB 1094.4 MB/s            
write offset:    51747225600kB 1057.6 MB/s            
write offset:    54614032384kB 1093.6 MB/s            
write offset:    57339281408kB 1039.6 MB/s            
write offset:    60026781696kB 1025.2 MB/s            
write offset:    62642978816kB   998 MB/s            
write offset:    65240301568kB 990.8 MB/s            
write offset:    67834478592kB 989.6 MB/s            
write offset:    70348963840kB 959.2 MB/s            
write offset:    73016541184kB 1017.6 MB/s            
write offset:    75877056512kB 1091.2 MB/s            
write offset:    78589722624kB 1034.8 MB/s            
write offset:    81351671808kB 1053.6 MB/s            
write offset:    83830505472kB 945.6 MB/s            
write offset:    86470819840kB 1007.2 MB/s            
write offset:    88962236416kB 950.4 MB/s            
write offset:    91489304576kB   964 MB/s            
write offset:    94011129856kB   962 MB/s            
write offset:    96852770816kB  1084 MB/s            
write offset:    99424927744kB 981.2 MB/s            
write offset:   101927878656kB 954.8 MB/s            
write offset:   104526249984kB 991.2 MB/s            
write offset:   107136155648kB 995.6 MB/s            
write offset:   109551026176kB 921.2 MB/s            
write offset:   112145203200kB 989.6 MB/s            
write offset:   114800197632kB 1012.8 MB/s            
write offset:   117444706304kB 1008.8 MB/s            
write offset:   120330387456kB 1100.8 MB/s            
write offset:   122948681728kB 998.8 MB/s            
write offset:   125665542144kB 1036.4 MB/s            
write offset:   128131792896kB 940.8 MB/s            
write offset:   130622160896kB   950 MB/s            
write offset:   133256183808kB 1004.8 MB/s            
write offset:   135814709248kB   976 MB/s            

write complete
write offset:   137438953472kB 0.0909091 MB/s            

write complete
read offset:       23068672kB     9 MB/s            
read offset:      189792256kB  63.6 MB/s            
read offset:      409993216kB    84 MB/s            
read offset:      630194176kB    84 MB/s            
read offset:      847249408kB  82.8 MB/s            
read offset:     1064304640kB  82.8 MB/s            
read offset:     1284505600kB    84 MB/s            
read offset:     1501560832kB  82.8 MB/s            
read offset:     1719664640kB  83.2 MB/s            
read offset:     1936719872kB  82.8 MB/s            
read offset:     2152726528kB  82.4 MB/s            
read offset:     2371878912kB  83.6 MB/s            
read offset:     2588934144kB  82.8 MB/s            
read offset:     2809135104kB    84 MB/s            
read offset:     3030384640kB  84.4 MB/s            
read offset:     3249537024kB  83.6 MB/s            
read offset:     3469737984kB    84 MB/s            
read offset:     3690987520kB  84.4 MB/s            
read offset:     3909091328kB  83.2 MB/s            
read offset:     4129292288kB    84 MB/s            
read offset:     4348444672kB  83.6 MB/s            
read offset:     4568645632kB    84 MB/s            
read offset:     4787798016kB  83.6 MB/s            
read offset:     5006950400kB  83.6 MB/s            
read offset:     5227151360kB    84 MB/s            
read offset:     5445255168kB  83.2 MB/s            
read offset:     5664407552kB  83.6 MB/s            
read offset:     5883559936kB  83.6 MB/s            
read offset:     6101663744kB  83.2 MB/s            
read offset:     6320816128kB  83.6 MB/s            
read offset:     6539968512kB  83.6 MB/s            
read offset:     6760169472kB    84 MB/s            
read offset:     6980370432kB    84 MB/s            
read offset:     7199522816kB  83.6 MB/s            
read offset:     7419723776kB    84 MB/s            
read offset:     7638876160kB  83.6 MB/s            
read offset:     7859077120kB    84 MB/s            
read offset:     8079278080kB    84 MB/s            
read offset:     8298430464kB  83.6 MB/s            
read offset:     8516534272kB  83.2 MB/s            
read offset:     8738832384kB  84.8 MB/s            
read offset:     8956936192kB  83.2 MB/s            
read offset:     9177137152kB    84 MB/s            
read offset:     9396289536kB  83.6 MB/s            
read offset:     9614393344kB  83.2 MB/s            
read offset:     9833545728kB  83.6 MB/s            
read offset:    10053746688kB    84 MB/s            
read offset:    10272899072kB  83.6 MB/s            
read offset:    10488905728kB  82.4 MB/s            
read offset:    10705960960kB  82.8 MB/s            
read offset:    10920919040kB    82 MB/s            
read offset:    11139022848kB  83.2 MB/s            
read offset:    11356078080kB  82.8 MB/s            
read offset:    11574181888kB  83.2 MB/s            
read offset:    11793334272kB  83.6 MB/s            
read offset:    12011438080kB  83.2 MB/s            
read offset:    12229541888kB  83.2 MB/s            
read offset:    12447645696kB  83.2 MB/s            
read offset:    12665749504kB  83.2 MB/s            
read offset:    12881756160kB  82.4 MB/s            
read offset:    13099859968kB  83.2 MB/s            
read offset:    13319012352kB  83.6 MB/s            
read offset:    13538164736kB  83.6 MB/s            
read offset:    13754171392kB  82.4 MB/s            
read offset:    13970178048kB  82.4 MB/s            
read offset:    14185136128kB    82 MB/s            
read offset:    14405337088kB    84 MB/s            
read offset:    14620295168kB    82 MB/s            
read offset:    14839447552kB  83.6 MB/s            
read offset:    15054405632kB    82 MB/s            
read offset:    15272509440kB  83.2 MB/s            
read offset:    15489564672kB  82.8 MB/s            
read offset:    15705571328kB  82.4 MB/s            
read offset:    15923675136kB  83.2 MB/s            
read offset:    16139681792kB  82.4 MB/s            
read offset:    16356737024kB  82.8 MB/s            
read offset:    16572743680kB  82.4 MB/s            
read offset:    16790847488kB  83.2 MB/s            
read offset:    17006854144kB  82.4 MB/s            
read offset:    17227055104kB    84 MB/s            
read offset:    17444110336kB  82.8 MB/s            
read offset:    17664311296kB    84 MB/s            
read offset:    17882415104kB  83.2 MB/s            
read offset:    18098421760kB  82.4 MB/s            
read offset:    18315476992kB  82.8 MB/s            
read offset:    18534629376kB  83.6 MB/s            
read offset:    18750636032kB  82.4 MB/s            
read offset:    18969788416kB  83.6 MB/s            
read offset:    19186843648kB  82.8 MB/s            
read offset:    19405996032kB  83.6 MB/s            
read offset:    19625148416kB  83.6 MB/s            
read offset:    19843252224kB  83.2 MB/s            
read offset:    20063453184kB    84 MB/s            
read offset:    20281556992kB  83.2 MB/s            
read offset:    20499660800kB  83.2 MB/s            
read offset:    20717764608kB  83.2 MB/s            
read offset:    20934819840kB  82.8 MB/s            
read offset:    21151875072kB  82.8 MB/s            
read offset:    21369978880kB  83.2 MB/s            
read offset:    21589131264kB  83.6 MB/s            
read offset:    21807235072kB  83.2 MB/s            
read offset:    22025338880kB  83.2 MB/s            
read offset:    22242394112kB  82.8 MB/s            
read offset:    22458400768kB  82.4 MB/s            
read offset:    22675456000kB  82.8 MB/s            
read offset:    22893559808kB  83.2 MB/s            
read offset:    23111663616kB  83.2 MB/s            
read offset:    23327670272kB  82.4 MB/s            
read offset:    23546822656kB  83.6 MB/s            
read offset:    23765975040kB  83.6 MB/s            
read offset:    23984078848kB  83.2 MB/s            
read offset:    24200085504kB  82.4 MB/s            
read offset:    24417140736kB  82.8 MB/s            
read offset:    24634195968kB  82.8 MB/s            
read offset:    24851251200kB  82.8 MB/s            
read offset:    25069355008kB  83.2 MB/s            
read offset:    25288507392kB  83.6 MB/s            
read offset:    25505562624kB  82.8 MB/s            
read offset:    25721569280kB  82.4 MB/s            
read offset:    25939673088kB  83.2 MB/s            
read offset:    26156728320kB  82.8 MB/s            
read offset:    26373783552kB  82.8 MB/s            
read offset:    26591887360kB  83.2 MB/s            
read offset:    26811039744kB  83.6 MB/s            
read offset:    27029143552kB  83.2 MB/s            
read offset:    27246198784kB  82.8 MB/s            
read offset:    27465351168kB  83.6 MB/s            
read offset:    27682406400kB  82.8 MB/s            
read offset:    27900510208kB  83.2 MB/s            
read offset:    28118614016kB  83.2 MB/s            
read offset:    28336717824kB  83.2 MB/s            
read offset:    28551675904kB    82 MB/s            
read offset:    28770828288kB  83.6 MB/s            
read offset:    28991029248kB    84 MB/s            
read offset:    29208084480kB  82.8 MB/s            
read offset:    29428285440kB    84 MB/s            
read offset:    29647437824kB  83.6 MB/s            
read offset:    29866590208kB  83.6 MB/s            
read offset:    30082596864kB  82.4 MB/s            
read offset:    30302797824kB    84 MB/s            
read offset:    30520901632kB  83.2 MB/s            
read offset:    30740054016kB  83.6 MB/s            
read offset:    30959206400kB  83.6 MB/s            
read offset:    31177310208kB  83.2 MB/s            
read offset:    31396462592kB  83.6 MB/s            
read offset:    31614566400kB  83.2 MB/s            
read offset:    31832670208kB  83.2 MB/s            
read offset:    32050774016kB  83.2 MB/s            
read offset:    32268877824kB  83.2 MB/s            
read offset:    32485933056kB  82.8 MB/s            
read offset:    32702988288kB  82.8 MB/s            
read offset:    32920043520kB  82.8 MB/s            
read offset:    33139195904kB  83.6 MB/s            
read offset:    33356251136kB  82.8 MB/s            
read offset:    33574354944kB  83.2 MB/s            
read offset:    33792458752kB  83.2 MB/s            
read offset:    34011611136kB  83.6 MB/s            
read offset:    34229714944kB  83.2 MB/s            
read offset:    34442575872kB  81.2 MB/s            
read offset:    34657533952kB    82 MB/s            
read offset:    34873540608kB  82.4 MB/s            
read offset:    35091644416kB  83.2 MB/s            
read offset:    35307651072kB  82.4 MB/s            
read offset:    35524706304kB  82.8 MB/s            
read offset:    35740712960kB  82.4 MB/s            
read offset:    35957768192kB  82.8 MB/s            
read offset:    36175872000kB  83.2 MB/s            
read offset:    36395024384kB  83.6 MB/s            
read offset:    36613128192kB  83.2 MB/s            
read offset:    36831232000kB  83.2 MB/s            
read offset:    37048287232kB  82.8 MB/s            
read offset:    37266391040kB  83.2 MB/s            
read offset:    37484494848kB  83.2 MB/s            
read offset:    37702598656kB  83.2 MB/s            
read offset:    37921751040kB  83.6 MB/s            
read offset:    38138806272kB  82.8 MB/s            
read offset:    38355861504kB  82.8 MB/s            
read offset:    38573965312kB  83.2 MB/s            
read offset:    38794166272kB    84 MB/s            
read offset:    39012270080kB  83.2 MB/s            
read offset:    39230373888kB  83.2 MB/s            
read offset:    39449526272kB  83.6 MB/s            
read offset:    39667630080kB  83.2 MB/s            
read offset:    39886782464kB  83.6 MB/s            
read offset:    40105934848kB  83.6 MB/s            
read offset:    40324038656kB  83.2 MB/s            
read offset:    40543191040kB  83.6 MB/s            
read offset:    40760246272kB  82.8 MB/s            
read offset:    40979398656kB  83.6 MB/s            
read offset:    41195405312kB  82.4 MB/s            
read offset:    41413509120kB  83.2 MB/s            
read offset:    41630564352kB  82.8 MB/s            
read offset:    41844473856kB  81.6 MB/s            
read offset:    42060480512kB  82.4 MB/s            
read offset:    42276487168kB  82.4 MB/s            
read offset:    42493542400kB  82.8 MB/s            
read offset:    42709549056kB  82.4 MB/s            
read offset:    42927652864kB  83.2 MB/s            
read offset:    43143659520kB  82.4 MB/s            
read offset:    43358617600kB    82 MB/s            
read offset:    43575672832kB  82.8 MB/s            
read offset:    43790630912kB    82 MB/s            
read offset:    44006637568kB  82.4 MB/s            
read offset:    44223692800kB  82.8 MB/s            
read offset:    44440748032kB  82.8 MB/s            
read offset:    44658851840kB  83.2 MB/s            
read offset:    44875907072kB  82.8 MB/s            
read offset:    45092962304kB  82.8 MB/s            
read offset:    45308968960kB  82.4 MB/s            
read offset:    45526024192kB  82.8 MB/s            
read offset:    45743079424kB  82.8 MB/s            
read offset:    45961183232kB  83.2 MB/s            
read offset:    46176141312kB    82 MB/s            
read offset:    46393196544kB  82.8 MB/s            
read offset:    46607106048kB  81.6 MB/s            
read offset:    46823112704kB  82.4 MB/s            
read offset:    47039119360kB  82.4 MB/s            
read offset:    47253028864kB  81.6 MB/s            
read offset:    47467986944kB    82 MB/s            
read offset:    47683993600kB  82.4 MB/s            
read offset:    47898951680kB    82 MB/s            
read offset:    48116006912kB  82.8 MB/s            
read offset:    48332013568kB  82.4 MB/s            
read offset:    48546971648kB    82 MB/s            
read offset:    48762978304kB  82.4 MB/s            
read offset:    48980033536kB  82.8 MB/s            
read offset:    49193943040kB  81.6 MB/s            
read offset:    49410998272kB  82.8 MB/s            
read offset:    49628053504kB  82.8 MB/s            
read offset:    49840914432kB  81.2 MB/s            
read offset:    50056921088kB  82.4 MB/s            
read offset:    50272927744kB  82.4 MB/s            
read offset:    50489982976kB  82.8 MB/s            
read offset:    50704941056kB    82 MB/s            
read offset:    50919899136kB    82 MB/s            
read offset:    51134857216kB    82 MB/s            
read offset:    51351912448kB  82.8 MB/s            
read offset:    51567919104kB  82.4 MB/s            
read offset:    51783925760kB  82.4 MB/s            
read offset:    51999932416kB  82.4 MB/s            
read offset:    52216987648kB  82.8 MB/s            
read offset:    52432994304kB  82.4 MB/s            
read offset:    52644806656kB  80.8 MB/s            
read offset:    52858716160kB  81.6 MB/s            
read offset:    53073674240kB    82 MB/s            
read offset:    53289680896kB  82.4 MB/s            
read offset:    53504638976kB    82 MB/s            
read offset:    53717499904kB  81.2 MB/s            
read offset:    53931409408kB  81.6 MB/s            
read offset:    54148464640kB  82.8 MB/s            
read offset:    54362374144kB  81.6 MB/s            
read offset:    54577332224kB    82 MB/s            
read offset:    54793338880kB  82.4 MB/s            
read offset:    55007248384kB  81.6 MB/s            
read offset:    55221157888kB  81.6 MB/s            
read offset:    55436115968kB    82 MB/s            
read offset:    55650025472kB  81.6 MB/s            
read offset:    55864983552kB    82 MB/s            
read offset:    56079941632kB    82 MB/s            
read offset:    56294899712kB    82 MB/s            
read offset:    56509857792kB    82 MB/s            
read offset:    56723767296kB  81.6 MB/s            
read offset:    56937676800kB  81.6 MB/s            
read offset:    57151586304kB  81.6 MB/s            
read offset:    57365495808kB  81.6 MB/s            
read offset:    57581502464kB  82.4 MB/s            
read offset:    57793314816kB  80.8 MB/s            
read offset:    58007224320kB  81.6 MB/s            
read offset:    58220085248kB  81.2 MB/s            
read offset:    58436091904kB  82.4 MB/s            
read offset:    58651049984kB    82 MB/s            
read offset:    58864959488kB  81.6 MB/s            
read offset:    59080966144kB  82.4 MB/s            
read offset:    59296972800kB  82.4 MB/s            
read offset:    59514028032kB  82.8 MB/s            
read offset:    59731083264kB  82.8 MB/s            
read offset:    59948138496kB  82.8 MB/s            
read offset:    60164145152kB  82.4 MB/s            
read offset:    60379103232kB    82 MB/s            
read offset:    60595109888kB  82.4 MB/s            
read offset:    60811116544kB  82.4 MB/s            
read offset:    61020831744kB    80 MB/s            
read offset:    61235789824kB    82 MB/s            
read offset:    61450747904kB    82 MB/s            
read offset:    61668851712kB  83.2 MB/s            
read offset:    61886955520kB  83.2 MB/s            
read offset:    62102962176kB  82.4 MB/s            
read offset:    62313725952kB  80.4 MB/s            
read offset:    62526586880kB  81.2 MB/s            
read offset:    62740496384kB  81.6 MB/s            
read offset:    62952308736kB  80.8 MB/s            
read offset:    63168315392kB  82.4 MB/s            
read offset:    63382224896kB  81.6 MB/s            
read offset:    63597182976kB    82 MB/s            
read offset:    63812141056kB    82 MB/s            
read offset:    64026050560kB  81.6 MB/s            
read offset:    64239960064kB  81.6 MB/s            
read offset:    64455966720kB  82.4 MB/s            
read offset:    64671973376kB  82.4 MB/s            
read offset:    64887980032kB  82.4 MB/s            
read offset:    65102938112kB    82 MB/s            
read offset:    65315799040kB  81.2 MB/s            
read offset:    65532854272kB  82.8 MB/s            
read offset:    65748860928kB  82.4 MB/s            
read offset:    65964867584kB  82.4 MB/s            
read offset:    66179825664kB    82 MB/s            
read offset:    66394783744kB    82 MB/s            
read offset:    66611838976kB  82.8 MB/s            
read offset:    66827845632kB  82.4 MB/s            
read offset:    67042803712kB    82 MB/s            
read offset:    67256713216kB  81.6 MB/s            
read offset:    67471671296kB    82 MB/s            
read offset:    67687677952kB  82.4 MB/s            
read offset:    67901587456kB  81.6 MB/s            
read offset:    68115496960kB  81.6 MB/s            
read offset:    68331503616kB  82.4 MB/s            
read offset:    68547510272kB  82.4 MB/s            
read offset:    68762468352kB    82 MB/s            
read offset:    68977426432kB    82 MB/s            
read offset:    69192384512kB    82 MB/s            
read offset:    69407342592kB    82 MB/s            
read offset:    69622300672kB    82 MB/s            
read offset:    69839355904kB  82.8 MB/s            
read offset:    70052216832kB  81.2 MB/s            
read offset:    70267174912kB    82 MB/s            
read offset:    70480035840kB  81.2 MB/s            
read offset:    70692896768kB  81.2 MB/s            
read offset:    70905757696kB  81.2 MB/s            
read offset:    71120715776kB    82 MB/s            
read offset:    71334625280kB  81.6 MB/s            
read offset:    71550631936kB  82.4 MB/s            
read offset:    71764541440kB  81.6 MB/s            
read offset:    71979499520kB    82 MB/s            
read offset:    72195506176kB  82.4 MB/s            
read offset:    72409415680kB  81.6 MB/s            
read offset:    72623325184kB  81.6 MB/s            
read offset:    72837234688kB  81.6 MB/s            
read offset:    73053241344kB  82.4 MB/s            
read offset:    73269248000kB  82.4 MB/s            
read offset:    73483157504kB  81.6 MB/s            
read offset:    73698115584kB    82 MB/s            
read offset:    73912025088kB  81.6 MB/s            
read offset:    74125934592kB  81.6 MB/s            
read offset:    74336698368kB  80.4 MB/s            
read offset:    74548510720kB  80.8 MB/s            
read offset:    74759274496kB  80.4 MB/s            
read offset:    74974232576kB    82 MB/s            
read offset:    75187093504kB  81.2 MB/s            
read offset:    75398905856kB  80.8 MB/s            
read offset:    75610718208kB  80.8 MB/s            
read offset:    75827773440kB  82.8 MB/s            
read offset:    76043780096kB  82.4 MB/s            
read offset:    76259786752kB  82.4 MB/s            
read offset:    76475793408kB  82.4 MB/s            
read offset:    76689702912kB  81.6 MB/s            
read offset:    76905709568kB  82.4 MB/s            
read offset:    77118570496kB  81.2 MB/s            
read offset:    77332480000kB  81.6 MB/s            
read offset:    77545340928kB  81.2 MB/s            
read offset:    77757153280kB  80.8 MB/s            
read offset:    77971062784kB  81.6 MB/s            
read offset:    78184972288kB  81.6 MB/s            
read offset:    78393638912kB  79.6 MB/s            
read offset:    78608596992kB    82 MB/s            
read offset:    78822506496kB  81.6 MB/s            
read offset:    79035367424kB  81.2 MB/s            
read offset:    79249276928kB  81.6 MB/s            
read offset:    79462137856kB  81.2 MB/s            
read offset:    79678144512kB  82.4 MB/s            
read offset:    79893102592kB    82 MB/s            
read offset:    80105963520kB  81.2 MB/s            
read offset:    80318824448kB  81.2 MB/s            
read offset:    80532733952kB  81.6 MB/s            
read offset:    80744546304kB  80.8 MB/s            
read offset:    80958455808kB  81.6 MB/s            
read offset:    81172365312kB  81.6 MB/s            
read offset:    81386274816kB  81.6 MB/s            
read offset:    81599135744kB  81.2 MB/s            
read offset:    81813045248kB  81.6 MB/s            
read offset:    82025906176kB  81.2 MB/s            
read offset:    82240864256kB    82 MB/s            
read offset:    82453725184kB  81.2 MB/s            
read offset:    82665537536kB  80.8 MB/s            
read offset:    82875252736kB    80 MB/s            
read offset:    83089162240kB  81.6 MB/s            
read offset:    83303071744kB  81.6 MB/s            
read offset:    83518029824kB    82 MB/s            
read offset:    83735085056kB  82.8 MB/s            
read offset:    83950043136kB    82 MB/s            
read offset:    84165001216kB    82 MB/s            
read offset:    84378910720kB  81.6 MB/s            
read offset:    84591771648kB  81.2 MB/s            
read offset:    84802535424kB  80.4 MB/s            
read offset:    85015396352kB  81.2 MB/s            
read offset:    85229305856kB  81.6 MB/s            
read offset:    85443215360kB  81.6 MB/s            
read offset:    85657124864kB  81.6 MB/s            
read offset:    85872082944kB    82 MB/s            
read offset:    86085992448kB  81.6 MB/s            
read offset:    86300950528kB    82 MB/s            
read offset:    86513811456kB  81.2 MB/s            
read offset:    86725623808kB  80.8 MB/s            
read offset:    86939533312kB  81.6 MB/s            
read offset:    87153442816kB  81.6 MB/s            
read offset:    87367352320kB  81.6 MB/s            
read offset:    87580213248kB  81.2 MB/s            
read offset:    87795171328kB    82 MB/s            
read offset:    88008032256kB  81.2 MB/s            
read offset:    88219844608kB  80.8 MB/s            
read offset:    88434802688kB    82 MB/s            
read offset:    88648712192kB  81.6 MB/s            
read offset:    88862621696kB  81.6 MB/s            
read offset:    89075482624kB  81.2 MB/s            
read offset:    89287294976kB  80.8 MB/s            
read offset:    89501204480kB  81.6 MB/s            
read offset:    89714065408kB  81.2 MB/s            
read offset:    89927974912kB  81.6 MB/s            
read offset:    90141884416kB  81.6 MB/s            
read offset:    90355793920kB  81.6 MB/s            
read offset:    90568654848kB  81.2 MB/s            
read offset:    90783612928kB    82 MB/s            
read offset:    90998571008kB    82 MB/s            
read offset:    91211431936kB  81.2 MB/s            
read offset:    91427438592kB  82.4 MB/s            
read offset:    91641348096kB  81.6 MB/s            
read offset:    91852111872kB  80.4 MB/s            
read offset:    92063924224kB  80.8 MB/s            
read offset:    92275736576kB  80.8 MB/s            
read offset:    92488597504kB  81.2 MB/s            
read offset:    92701458432kB  81.2 MB/s            
read offset:    92916416512kB    82 MB/s            
read offset:    93131374592kB    82 MB/s            
read offset:    93346332672kB    82 MB/s            
read offset:    93559193600kB  81.2 MB/s            
read offset:    93773103104kB  81.6 MB/s            
read offset:    93988061184kB    82 MB/s            
read offset:    94199873536kB  80.8 MB/s            
read offset:    94411685888kB  80.8 MB/s            
read offset:    94626643968kB    82 MB/s            
read offset:    94839504896kB  81.2 MB/s            
read offset:    95051317248kB  80.8 MB/s            
read offset:    95264178176kB  81.2 MB/s            
read offset:    95478087680kB  81.6 MB/s            
read offset:    95690948608kB  81.2 MB/s            
read offset:    95906955264kB  82.4 MB/s            
read offset:    96119816192kB  81.2 MB/s            
read offset:    96332677120kB  81.2 MB/s            
read offset:    96544489472kB  80.8 MB/s            
read offset:    96759447552kB    82 MB/s            
read offset:    96974405632kB    82 MB/s            
read offset:    97186217984kB  80.8 MB/s            
read offset:    97400127488kB  81.6 MB/s            
read offset:    97612988416kB  81.2 MB/s            
read offset:    97824800768kB  80.8 MB/s            
read offset:    98038710272kB  81.6 MB/s            
read offset:    98253668352kB    82 MB/s            
read offset:    98468626432kB    82 MB/s            
read offset:    98681487360kB  81.2 MB/s            
read offset:    98893299712kB  80.8 MB/s            
read offset:    99106160640kB  81.2 MB/s            
read offset:    99317972992kB  80.8 MB/s            
read offset:    99531882496kB  81.6 MB/s            
read offset:    99746840576kB    82 MB/s            
read offset:    99960750080kB  81.6 MB/s            
read offset:   100174659584kB  81.6 MB/s            
read offset:   100387520512kB  81.2 MB/s            
read offset:   100599332864kB  80.8 MB/s            
read offset:   100812193792kB  81.2 MB/s            
read offset:   101027151872kB    82 MB/s            
read offset:   101242109952kB    82 MB/s            
read offset:   101457068032kB    82 MB/s            
read offset:   101668880384kB  80.8 MB/s            
read offset:   101881741312kB  81.2 MB/s            
read offset:   102097747968kB  82.4 MB/s            
read offset:   102311657472kB  81.6 MB/s            
read offset:   102525566976kB  81.6 MB/s            
read offset:   102741573632kB  82.4 MB/s            
read offset:   102955483136kB  81.6 MB/s            
read offset:   103171489792kB  82.4 MB/s            
read offset:   103384350720kB  81.2 MB/s            
read offset:   103596163072kB  80.8 MB/s            
read offset:   103810072576kB  81.6 MB/s            
read offset:   104023982080kB  81.6 MB/s            
read offset:   104237891584kB  81.6 MB/s            
read offset:   104451801088kB  81.6 MB/s            
read offset:   104665710592kB  81.6 MB/s            
read offset:   104878571520kB  81.2 MB/s            
read offset:   105094578176kB  82.4 MB/s            
read offset:   105307439104kB  81.2 MB/s            
read offset:   105521348608kB  81.6 MB/s            
read offset:   105736306688kB    82 MB/s            
read offset:   105944973312kB  79.6 MB/s            
read offset:   106150494208kB  78.4 MB/s            
read offset:   106360209408kB    80 MB/s            
read offset:   106573070336kB  81.2 MB/s            
read offset:   106788028416kB    82 MB/s            
read offset:   107000889344kB  81.2 MB/s            
read offset:   107212701696kB  80.8 MB/s            
read offset:   107424514048kB  80.8 MB/s            
read offset:   107639472128kB    82 MB/s            
read offset:   107854430208kB    82 MB/s            
read offset:   108067291136kB  81.2 MB/s            
read offset:   108282249216kB    82 MB/s            
read offset:   108495110144kB  81.2 MB/s            
read offset:   108707971072kB  81.2 MB/s            
read offset:   108920832000kB  81.2 MB/s            
read offset:   109134741504kB  81.6 MB/s            
read offset:   109348651008kB  81.6 MB/s            
read offset:   109560463360kB  80.8 MB/s            
read offset:   109775421440kB    82 MB/s            
read offset:   109988282368kB  81.2 MB/s            
read offset:   110201143296kB  81.2 MB/s            
read offset:   110414004224kB  81.2 MB/s            
read offset:   110626865152kB  81.2 MB/s            
read offset:   110840774656kB  81.6 MB/s            
read offset:   111052587008kB  80.8 MB/s            
read offset:   111264399360kB  80.8 MB/s            
read offset:   111476211712kB  80.8 MB/s            
read offset:   111689072640kB  81.2 MB/s            
read offset:   111902982144kB  81.6 MB/s            
read offset:   112114794496kB  80.8 MB/s            
read offset:   112327655424kB  81.2 MB/s            
read offset:   112537370624kB    80 MB/s            
read offset:   112752328704kB    82 MB/s            
read offset:   112965189632kB  81.2 MB/s            
read offset:   113175953408kB  80.4 MB/s            
read offset:   113385668608kB    80 MB/s            
read offset:   113596432384kB  80.4 MB/s            
read offset:   113809293312kB  81.2 MB/s            
read offset:   114022154240kB  81.2 MB/s            
read offset:   114233966592kB  80.8 MB/s            
read offset:   114447876096kB  81.6 MB/s            
read offset:   114660737024kB  81.2 MB/s            
read offset:   114873597952kB  81.2 MB/s            
read offset:   115085410304kB  80.8 MB/s            
read offset:   115298271232kB  81.2 MB/s            
read offset:   115512180736kB  81.6 MB/s            
read offset:   115723993088kB  80.8 MB/s            
read offset:   115937902592kB  81.6 MB/s            
read offset:   116151812096kB  81.6 MB/s            
read offset:   116364673024kB  81.2 MB/s            
read offset:   116576485376kB  80.8 MB/s            
read offset:   116786200576kB    80 MB/s            
read offset:   116998012928kB  80.8 MB/s            
read offset:   117210873856kB  81.2 MB/s            
read offset:   117422686208kB  80.8 MB/s            
read offset:   117635547136kB  81.2 MB/s            
read offset:   117849456640kB  81.6 MB/s            
read offset:   118060220416kB  80.4 MB/s            
read offset:   118272032768kB  80.8 MB/s            
read offset:   118484893696kB  81.2 MB/s            
read offset:   118697754624kB  81.2 MB/s            
read offset:   118910615552kB  81.2 MB/s            
read offset:   119124525056kB  81.6 MB/s            
read offset:   119336337408kB  80.8 MB/s            
read offset:   119547101184kB  80.4 MB/s            
read offset:   119758913536kB  80.8 MB/s            
read offset:   119969677312kB  80.4 MB/s            
read offset:   120181489664kB  80.8 MB/s            
read offset:   120394350592kB  81.2 MB/s            
read offset:   120606162944kB  80.8 MB/s            
read offset:   120817975296kB  80.8 MB/s            
read offset:   121029787648kB  80.8 MB/s            
read offset:   121243697152kB  81.6 MB/s            
read offset:   121454460928kB  80.4 MB/s            
read offset:   121667321856kB  81.2 MB/s            
read offset:   121879134208kB  80.8 MB/s            
read offset:   122090946560kB  80.8 MB/s            
read offset:   122305904640kB    82 MB/s            
read offset:   122518765568kB  81.2 MB/s            
read offset:   122730577920kB  80.8 MB/s            
read offset:   122943438848kB  81.2 MB/s            
read offset:   123153154048kB    80 MB/s            
read offset:   123363917824kB  80.4 MB/s            
read offset:   123577827328kB  81.6 MB/s            
read offset:   123789639680kB  80.8 MB/s            
read offset:   124002500608kB  81.2 MB/s            
read offset:   124214312960kB  80.8 MB/s            
read offset:   124424028160kB    80 MB/s            
read offset:   124636889088kB  81.2 MB/s            
read offset:   124851847168kB    82 MB/s            
read offset:   125064708096kB  81.2 MB/s            
read offset:   125277569024kB  81.2 MB/s            
read offset:   125490429952kB  81.2 MB/s            
read offset:   125702242304kB  80.8 MB/s            
read offset:   125915103232kB  81.2 MB/s            
read offset:   126125867008kB  80.4 MB/s            
read offset:   126337679360kB  80.8 MB/s            
read offset:   126550540288kB  81.2 MB/s            
read offset:   126761304064kB  80.4 MB/s            
read offset:   126972067840kB  80.4 MB/s            
read offset:   127183880192kB  80.8 MB/s            
read offset:   127396741120kB  81.2 MB/s            
read offset:   127607504896kB  80.4 MB/s            
read offset:   127819317248kB  80.8 MB/s            
read offset:   128031129600kB  80.8 MB/s            
read offset:   128241893376kB  80.4 MB/s            
read offset:   128452657152kB  80.4 MB/s            
read offset:   128664469504kB  80.8 MB/s            
read offset:   128875233280kB  80.4 MB/s            
read offset:   129085997056kB  80.4 MB/s            
read offset:   129296760832kB  80.4 MB/s            
read offset:   129506476032kB    80 MB/s            
read offset:   129719336960kB  81.2 MB/s            
read offset:   129930100736kB  80.4 MB/s            
read offset:   130142961664kB  81.2 MB/s            
read offset:   130352676864kB    80 MB/s            
read offset:   130563440640kB  80.4 MB/s            
read offset:   130773155840kB    80 MB/s            
read offset:   130983919616kB  80.4 MB/s            
read offset:   131195731968kB  80.8 MB/s            
read offset:   131407544320kB  80.8 MB/s            
read offset:   131618308096kB  80.4 MB/s            
read offset:   131828023296kB    80 MB/s            
read offset:   132037738496kB    80 MB/s            
read offset:   132247453696kB    80 MB/s            
read offset:   132459266048kB  80.8 MB/s            
read offset:   132670029824kB  80.4 MB/s            
read offset:   132881842176kB  80.8 MB/s            
read offset:   133093654528kB  80.8 MB/s            
read offset:   133304418304kB  80.4 MB/s            
read offset:   133515182080kB  80.4 MB/s            
read offset:   133726994432kB  80.8 MB/s            
read offset:   133935661056kB  79.6 MB/s            
read offset:   134148521984kB  81.2 MB/s            
read offset:   134359285760kB  80.4 MB/s            
read offset:   134569000960kB    80 MB/s            
read offset:   134778716160kB    80 MB/s            
read offset:   134989479936kB  80.4 MB/s            
read offset:   135198146560kB  79.6 MB/s            
read offset:   135405764608kB  79.2 MB/s            
read offset:   135615479808kB    80 MB/s            
read offset:   135826243584kB  80.4 MB/s            
read offset:   136038055936kB  80.8 MB/s            
read offset:   136248819712kB  80.4 MB/s            
read offset:   136459583488kB  80.4 MB/s            
read offset:   136670347264kB  80.4 MB/s            
read offset:   136881111040kB  80.4 MB/s            
read offset:   137090826240kB    80 MB/s            
read offset:   137302638592kB  80.8 MB/s            

read complete
read offset:   137438953472kB 0.142857 MB/s            

read complete

llverdev: data verified successfully

real	58m2.904s
user	0m2.725s
sys	9m47.551s

===================== format the MDT /dev/large_vg/mdt_lv =====================
# time mkfs.lustre --reformat --fsname=largefs --mdt --mgs --mkfsoptions='-t ext4' /dev/large_vg/mdt_lv

   Permanent disk data:
Target:     largefs-MDTffff
Index:      unassigned
Lustre FS:  largefs
Mount type: ldiskfs
Flags:      0x75
              (MDT MGS needs_index first_time update )
Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro
Parameters: mdt.group_upcall=/usr/sbin/l_getgroups

device size = 2097152MB
2 6 18
formatting backing filesystem ldiskfs on /dev/large_vg/mdt_lv
	target name  largefs-MDTffff
	4k blocks     536870912
	options       -t ext4 -J size=400 -i 4096 -I 512 -q -O dir_index,uninit_groups -F
mkfs_cmd = mke2fs -j -b 4096 -L largefs-MDTffff -t ext4 -J size=400 -i 4096 -I 512 -q -O dir_index,uninit_groups -F /dev/large_vg/mdt_lv 536870912
Writing CONFIGS/mountdata

real	4m20.365s
user	0m19.411s
sys	3m49.944s

===================== format the OST /dev/large_vg/ost_lv =====================
# time mkfs.lustre --reformat --fsname=largefs --ost --mgsnode=192.168.77.1@o2ib --mkfsoptions='-t ext4 -i 1058576' --mountfsoptions='errors=remount-ro,extents,mballoc,force_over_16tb' /dev/large_vg/ost_lv

   Permanent disk data:
Target:     largefs-OSTffff
Index:      unassigned
Lustre FS:  largefs
Mount type: ldiskfs
Flags:      0x72
              (OST needs_index first_time update )
Persistent mount opts: errors=remount-ro,extents,mballoc,force_over_16tb
Parameters: mgsnode=192.168.77.1@o2ib

device size = 134217728MB
2 6 18
formatting backing filesystem ldiskfs on /dev/large_vg/ost_lv
	target name  largefs-OSTffff
	4k blocks     34359738368
	options       -t ext4 -i 1058576 -J size=400 -I 256 -q -O dir_index,extents,uninit_groups -F
mkfs_cmd = mke2fs -j -b 4096 -L largefs-OSTffff -t ext4 -i 1058576 -J size=400 -I 256 -q -O dir_index,extents,uninit_groups -F /dev/large_vg/ost_lv 34359738368
Writing CONFIGS/mountdata

real	1m22.322s
user	0m6.786s
sys	0m33.467s

=========== dump the super block information on /dev/large_vg/ost_lv ===========
# dumpe2fs -h /dev/large_vg/ost_lv
dumpe2fs 1.41.90.wc1 (18-Mar-2011)
Filesystem volume name:   largefs-OSTffff
Last mounted on:          /tmp/mnt6d98sv/tmp/mnt6d98sv
Filesystem UUID:          5e2a898b-da07-48a4-93cf-1c3a3159b517
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr dir_index filetype extent 64bit flex_bg sparse_super huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              134217728
Block count:              34359738368
Reserved block count:     1717986918
Free blocks:              34348675033
Free inodes:              134217715
First block:              0
Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         128
Inode blocks per group:   8
Flex block group size:    16
Filesystem created:       Fri May 13 00:45:38 2011
Last mount time:          Fri May 13 00:47:00 2011
Last write time:          Fri May 13 00:47:01 2011
Mount count:              1
Maximum mount count:      20
Last checked:             Fri May 13 00:45:38 2011
Check interval:           0 (<none>)
Lifetime writes:          34 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      32b48ffa-1ae0-4da0-9cfd-6d0edbeece5a
Journal backup:           inode blocks
Journal features:         FEATURE_I1
Journal size:             400M
Journal length:           102400
Journal sequence:         0x00000004
Journal start:            0


========= mount the OST /dev/large_vg/ost_lv as an ldiskfs filesystem =========

# mount -t ldiskfs -o extents,mballoc,force_over_16tb /dev/large_vg/ost_lv /mnt/ost1
/dev/mapper/large_vg-ost_lv on /mnt/ost1 type ldiskfs (rw,extents,mballoc,force_over_16tb)

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/large_vg-ost_lv
                      128T  401M  122T   1% /mnt/ost1


===== run llverfs in partial mode on the OST ldiskfs filesystem /mnt/ost1 =====
# time llverfs -vpl /mnt/ost1
Timestamp: 1305272823
write File name: /mnt/ost1/dir00000/file000          
write File name: /mnt/ost1/dir00010/file000          
write File name: /mnt/ost1/dir00164/file000          
write File name: /mnt/ost1/dir00391/file000          
write File name: /mnt/ost1/dir00626/file000          
write File name: /mnt/ost1/dir00863/file000          
write File name: /mnt/ost1/dir01120/file000          
write File name: /mnt/ost1/dir01397/file000          
write File name: /mnt/ost1/dir01687/file000          
write File name: /mnt/ost1/dir01959/file000          
write File name: /mnt/ost1/dir02244/file000          
write File name: /mnt/ost1/dir02508/file000          
write File name: /mnt/ost1/dir02701/file000          
write File name: /mnt/ost1/dir02914/file000          
write File name: /mnt/ost1/dir03168/file000          
write File name: /mnt/ost1/dir03473/file000          
write File name: /mnt/ost1/dir03766/file000          
write File name: /mnt/ost1/dir04067/file000          
write File name: /mnt/ost1/dir04358/file000          
write File name: /mnt/ost1/dir04650/file000          
write File name: /mnt/ost1/dir04815/file000          
write File name: /mnt/ost1/dir05048/file000          
write File name: /mnt/ost1/dir05349/file000          
write File name: /mnt/ost1/dir05627/file000          
write File name: /mnt/ost1/dir05924/file000          
write File name: /mnt/ost1/dir06245/file000          
write File name: /mnt/ost1/dir06546/file000          
write File name: /mnt/ost1/dir06815/file000          
write File name: /mnt/ost1/dir07121/file000          
write File name: /mnt/ost1/dir07439/file000          
write File name: /mnt/ost1/dir07750/file000          
write File name: /mnt/ost1/dir08057/file000          
write File name: /mnt/ost1/dir08224/file000          
write File name: /mnt/ost1/dir08317/file000          
write File name: /mnt/ost1/dir08636/file000          
write File name: /mnt/ost1/dir08948/file000          
write File name: /mnt/ost1/dir09268/file000          
write File name: /mnt/ost1/dir09592/file000          
write File name: /mnt/ost1/dir09907/file000          
write File name: /mnt/ost1/dir10221/file000          
write File name: /mnt/ost1/dir10537/file000          
write File name: /mnt/ost1/dir10865/file000          
write File name: /mnt/ost1/dir11171/file000          
write File name: /mnt/ost1/dir11483/file000          
write File name: /mnt/ost1/dir11643/file000          
write File name: /mnt/ost1/dir11749/file000          
write File name: /mnt/ost1/dir12031/file000          
write File name: /mnt/ost1/dir12345/file000          
write File name: /mnt/ost1/dir12692/file000          
write File name: /mnt/ost1/dir13028/file000          
write File name: /mnt/ost1/dir13333/file000          
write File name: /mnt/ost1/dir13630/file000          
write File name: /mnt/ost1/dir13929/file000          
write File name: /mnt/ost1/dir14256/file000          
write File name: /mnt/ost1/dir14591/file000          
write File name: /mnt/ost1/dir14921/file000          
write File name: /mnt/ost1/dir15096/file000          
write File name: /mnt/ost1/dir15197/file000          
write File name: /mnt/ost1/dir15538/file000          
write File name: /mnt/ost1/dir15880/file000          
write File name: /mnt/ost1/dir16222/file000          
write File name: /mnt/ost1/dir16534/file000          
write File name: /mnt/ost1/dir16859/file000          
write File name: /mnt/ost1/dir17184/file000          
write File name: /mnt/ost1/dir17509/file000          
write File name: /mnt/ost1/dir17829/file000          
write File name: /mnt/ost1/dir18157/file000          
write File name: /mnt/ost1/dir18499/file000          
write File name: /mnt/ost1/dir18671/file000          
write File name: /mnt/ost1/dir18758/file000          
write File name: /mnt/ost1/dir19067/file000          
write File name: /mnt/ost1/dir19383/file000          
write File name: /mnt/ost1/dir19708/file000          
write File name: /mnt/ost1/dir20044/file000          
write File name: /mnt/ost1/dir20380/file000          
write File name: /mnt/ost1/dir20704/file000          
write File name: /mnt/ost1/dir21030/file000          
write File name: /mnt/ost1/dir21355/file000          
write File name: /mnt/ost1/dir21666/file000          
write File name: /mnt/ost1/dir22003/file000          
write File name: /mnt/ost1/dir22156/file000          
write File name: /mnt/ost1/dir22266/file000          
write File name: /mnt/ost1/dir22597/file000          
write File name: /mnt/ost1/dir22931/file000          
write File name: /mnt/ost1/dir23255/file000          
write File name: /mnt/ost1/dir23579/file000          
write File name: /mnt/ost1/dir23914/file000          
write File name: /mnt/ost1/dir24232/file000          
write File name: /mnt/ost1/dir24546/file000          
write File name: /mnt/ost1/dir24871/file000          
write File name: /mnt/ost1/dir25191/file000          
write File name: /mnt/ost1/dir25517/file000          
write File name: /mnt/ost1/dir25674/file000          
write File name: /mnt/ost1/dir25800/file000          
write File name: /mnt/ost1/dir26122/file000          
write File name: /mnt/ost1/dir26456/file000          
write File name: /mnt/ost1/dir26810/file000          
write File name: /mnt/ost1/dir27164/file000          
write File name: /mnt/ost1/dir27509/file000          
write File name: /mnt/ost1/dir27806/file000          
write File name: /mnt/ost1/dir28098/file000          
write File name: /mnt/ost1/dir28440/file000          
write File name: /mnt/ost1/dir28780/file000          
write File name: /mnt/ost1/dir29106/file000          
write File name: /mnt/ost1/dir29266/file000          
write File name: /mnt/ost1/dir29397/file000          
write File name: /mnt/ost1/dir29712/file000          
write File name: /mnt/ost1/dir29998/file000          

write complete
read File name: /mnt/ost1/dir00194/file000          
read File name: /mnt/ost1/dir00562/file000          
read File name: /mnt/ost1/dir00859/file000          
read File name: /mnt/ost1/dir01248/file000          
read File name: /mnt/ost1/dir01623/file000          
read File name: /mnt/ost1/dir02070/file000          
read File name: /mnt/ost1/dir02536/file000          
read File name: /mnt/ost1/dir02940/file000          
read File name: /mnt/ost1/dir03383/file000          
read File name: /mnt/ost1/dir03701/file000          
read File name: /mnt/ost1/dir03967/file000          
read File name: /mnt/ost1/dir04355/file000          
read File name: /mnt/ost1/dir04710/file000          
read File name: /mnt/ost1/dir04981/file000          
read File name: /mnt/ost1/dir05300/file000          
read File name: /mnt/ost1/dir05677/file000          
read File name: /mnt/ost1/dir06034/file000          
read File name: /mnt/ost1/dir06292/file000          
read File name: /mnt/ost1/dir06628/file000          
read File name: /mnt/ost1/dir12455/file000          
read File name: /mnt/ost1/dir25781/file000          
read File name: /mnt/ost1/dir26116/file000          
read File name: /mnt/ost1/dir26442/file000          
read File name: /mnt/ost1/dir26810/file000          
read File name: /mnt/ost1/dir26862/file000          
read File name: /mnt/ost1/dir29006/file000          
read File name: /mnt/ost1/dir29359/file000          
read File name: /mnt/ost1/dir29731/file000          
read File name: /mnt/ost1/dir29998/file000          

read complete

real	11m23.279s
user	0m1.217s
sys	1m33.825s

===================== unmount the OST /dev/large_vg/ost_lv =====================
# umount /mnt/ost1

================== run e2fsck on device /dev/large_vg/ost_lv ==================
# time e2fsck -fy -tt /dev/large_vg/ost_lv
e2fsck 1.41.90.wc1 (18-Mar-2011)
Pass 1: Checking inodes, blocks, and sizes
Pass 1: Memory used: 276k/166364k (71k/206k), time: 208.19/ 3.52/ 3.93
Pass 1: I/O read: 118MB, write: 1MB, rate: 0.57MB/s
Pass 2: Checking directory structure
Pass 2: Memory used: 276k/202552k (64k/213k), time: 318.25/ 0.91/ 1.27
Pass 2: I/O read: 118MB, write: 0MB, rate: 0.37MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 16624k/202552k (16449k/176k), time: 533.78/ 6.95/ 8.32
Pass 3A: Memory used: 16624k/202552k (16448k/177k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 16624k/202200k (64k/16561k), time:  0.01/ 0.01/ 0.01
Pass 3: I/O read: 1MB, write: 0MB, rate: 74.33MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 16624k/110620k (64k/16561k), time:  2.63/ 2.62/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 24816k/77844k (16448k/8369k), time: 492.52/54.49/ 4.83
Pass 5: I/O read: 259MB, write: 0MB, rate: 0.53MB/s
largefs-OSTffff: 60012/134217728 files (0.2% non-contiguous), 18773235/34359738368 blocks
Memory used: 24816k/77844k (16448k/8369k), time: 1029.31/64.43/13.15
I/O read: 558MB, write: 1MB, rate: 0.54MB/s

real	17m9.881s
user	1m4.495s
sys	0m13.575s

====== mount the OST /dev/large_vg/ost_lv as an ldiskfs filesystem again ======

# mount -t ldiskfs -o extents,mballoc,force_over_16tb /dev/large_vg/ost_lv /mnt/ost1
/dev/mapper/large_vg-ost_lv on /mnt/ost1 type ldiskfs (rw,extents,mballoc,force_over_16tb)

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/large_vg-ost_lv
                      128T   30G  122T   1% /mnt/ost1


# time rm -rf /mnt/ost1/{*.filecount,dir*}

real	18m36.392s
user	0m0.743s
sys	0m9.191s

===================== unmount the OST /dev/large_vg/ost_lv =====================
# umount /mnt/ost1

========================= setup the Lustre filesystem =========================

# mount -t lustre -o user_xattr,acl /dev/large_vg/mdt_lv /mnt/mds1
/dev/large_vg/mdt_lv on /mnt/mds1 type lustre (rw,user_xattr,acl)

Filesystem            Size  Used Avail Use% Mounted on
/dev/large_vg/mdt_lv  1.8T  471M  1.7T   1% /mnt/mds1


# mount -t lustre -o extents,mballoc,force_over_16tb /dev/large_vg/ost_lv /mnt/ost1
/dev/large_vg/ost_lv on /mnt/ost1 type lustre (rw,extents,mballoc,force_over_16tb)

Filesystem            Size  Used Avail Use% Mounted on
/dev/large_vg/ost_lv  128T  401M  122T   1% /mnt/ost1


# mount -t lustre -o user_xattr,acl,flock 192.168.77.1@o2ib:/largefs /mnt/lustre
192.168.77.1@o2ib:/largefs on /mnt/lustre type lustre (rw,user_xattr,acl,flock)

Filesystem            Size  Used Avail Use% Mounted on
192.168.77.1@o2ib:/largefs
                      128T  401M  122T   1% /mnt/lustre


# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      9.1G  3.3G  5.4G  38% /
/dev/hda1              99M   20M   75M  21% /boot
tmpfs                  16G     0   16G   0% /dev/shm
/dev/large_vg/mdt_lv  1.8T  471M  1.7T   1% /mnt/mds1
/dev/large_vg/ost_lv  128T  401M  122T   1% /mnt/ost1
192.168.77.1@o2ib:/largefs
                      128T  401M  122T   1% /mnt/lustre

# lfs df -h /mnt/lustre
UUID                       bytes        Used   Available Use% Mounted on
largefs-MDT0000_UUID        1.7T      470.1M        1.6T   0% /mnt/lustre[MDT:0]
largefs-OST0000_UUID      128.0T      400.8M      121.6T   0% /mnt/lustre[OST:0]

filesystem summary:       128.0T      400.8M      121.6T   0% /mnt/lustre


# lfs df -i /mnt/lustre
UUID                      Inodes       IUsed       IFree IUse% Mounted on
largefs-MDT0000_UUID   469606378          25   469606353   0% /mnt/lustre[MDT:0]
largefs-OST0000_UUID   134217728          56   134217672   0% /mnt/lustre[OST:0]

filesystem summary:    469606378          25   469606353   0% /mnt/lustre

lnet.debug=0x33f0404
lnet.subsystem_debug=0xffb7e3ff
lnet.debug_mb=10

=========== dump the super block information on /dev/large_vg/ost_lv ===========
# dumpe2fs -h /dev/large_vg/ost_lv
dumpe2fs 1.41.90.wc1 (18-Mar-2011)
Filesystem volume name:   largefs-OST0000
Last mounted on:          /
Filesystem UUID:          5e2a898b-da07-48a4-93cf-1c3a3159b517
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr dir_index filetype needs_recovery extent 64bit flex_bg sparse_super huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              134217728
Block count:              34359738368
Reserved block count:     1717986918
Free blocks:              34348674877
Free inodes:              134217715
First block:              0
Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         128
Inode blocks per group:   8
Flex block group size:    16
Filesystem created:       Fri May 13 00:45:38 2011
Last mount time:          Fri May 13 01:34:34 2011
Last write time:          Fri May 13 01:34:34 2011
Mount count:              3
Maximum mount count:      20
Last checked:             Fri May 13 00:58:32 2011
Check interval:           0 (<none>)
Lifetime writes:          34 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      32b48ffa-1ae0-4da0-9cfd-6d0edbeece5a
Journal backup:           inode blocks
Journal features:         journal_incompat_revoke FEATURE_I1
Journal size:             400M
Journal length:           102400
Journal sequence:         0x00000173
Journal start:            1


======= run llverfs in partial mode on the Lustre filesystem /mnt/lustre =======
# time llverfs -vpl /mnt/lustre
Timestamp: 1305275704
write File name: /mnt/lustre/dir00000/file000          
write File name: /mnt/lustre/dir00509/file000          
write File name: /mnt/lustre/dir01090/file000          
write File name: /mnt/lustre/dir01663/file000          
write File name: /mnt/lustre/dir02225/file000          
write File name: /mnt/lustre/dir02787/file000          
write File name: /mnt/lustre/dir03341/file000          
write File name: /mnt/lustre/dir03900/file000          
write File name: /mnt/lustre/dir04450/file000          
write File name: /mnt/lustre/dir05003/file000          
write File name: /mnt/lustre/dir05553/file000          
write File name: /mnt/lustre/dir06103/file000          
write File name: /mnt/lustre/dir06642/file000          
write File name: /mnt/lustre/dir07180/file000          
write File name: /mnt/lustre/dir07718/file000          
write File name: /mnt/lustre/dir08243/file000          
write File name: /mnt/lustre/dir08766/file000          
write File name: /mnt/lustre/dir09296/file000          
write File name: /mnt/lustre/dir09819/file000          
write File name: /mnt/lustre/dir10339/file000          
write File name: /mnt/lustre/dir10856/file000          
write File name: /mnt/lustre/dir11365/file000          
write File name: /mnt/lustre/dir11880/file000          
write File name: /mnt/lustre/dir12387/file000          
write File name: /mnt/lustre/dir12894/file000          
write File name: /mnt/lustre/dir13412/file000          
write File name: /mnt/lustre/dir13924/file000          
write File name: /mnt/lustre/dir14440/file000          
write File name: /mnt/lustre/dir14942/file000          
write File name: /mnt/lustre/dir15448/file000          
write File name: /mnt/lustre/dir15955/file000          
write File name: /mnt/lustre/dir16459/file000          
write File name: /mnt/lustre/dir16960/file000          
write File name: /mnt/lustre/dir17467/file000          
write File name: /mnt/lustre/dir17962/file000          
write File name: /mnt/lustre/dir18462/file000          
write File name: /mnt/lustre/dir18956/file000          
write File name: /mnt/lustre/dir19459/file000          
write File name: /mnt/lustre/dir19962/file000          
write File name: /mnt/lustre/dir20462/file000          
write File name: /mnt/lustre/dir20964/file000          
write File name: /mnt/lustre/dir21464/file000          
write File name: /mnt/lustre/dir21966/file000          
write File name: /mnt/lustre/dir22465/file000          
write File name: /mnt/lustre/dir22964/file000          
write File name: /mnt/lustre/dir23464/file000          
write File name: /mnt/lustre/dir23968/file000          
write File name: /mnt/lustre/dir24462/file000          
write File name: /mnt/lustre/dir24960/file000          
write File name: /mnt/lustre/dir25458/file000          
write File name: /mnt/lustre/dir25959/file000          
write File name: /mnt/lustre/dir26448/file000          
write File name: /mnt/lustre/dir26947/file000          
write File name: /mnt/lustre/dir27438/file000          
write File name: /mnt/lustre/dir27929/file000          
write File name: /mnt/lustre/dir28423/file000          
write File name: /mnt/lustre/dir28920/file000          
write File name: /mnt/lustre/dir29410/file000          
write File name: /mnt/lustre/dir29896/file000          
write File name: /mnt/lustre/dir29998/file000          

write complete
read File name: /mnt/lustre/dir00217/file000          
read File name: /mnt/lustre/dir00395/file000          
read File name: /mnt/lustre/dir00510/file000          
read File name: /mnt/lustre/dir00685/file000          
read File name: /mnt/lustre/dir00939/file000          
read File name: /mnt/lustre/dir01183/file000          
read File name: /mnt/lustre/dir01407/file000          
read File name: /mnt/lustre/dir01649/file000          
read File name: /mnt/lustre/dir01908/file000          
read File name: /mnt/lustre/dir02153/file000          
read File name: /mnt/lustre/dir02394/file000          
read File name: /mnt/lustre/dir02654/file000          
read File name: /mnt/lustre/dir02879/file000          
read File name: /mnt/lustre/dir03092/file000          
read File name: /mnt/lustre/dir03330/file000          
read File name: /mnt/lustre/dir03568/file000          
read File name: /mnt/lustre/dir03823/file000          
read File name: /mnt/lustre/dir04090/file000          
read File name: /mnt/lustre/dir04330/file000          
read File name: /mnt/lustre/dir04563/file000          
read File name: /mnt/lustre/dir04814/file000          
read File name: /mnt/lustre/dir05077/file000          
read File name: /mnt/lustre/dir05332/file000          
read File name: /mnt/lustre/dir05582/file000          
read File name: /mnt/lustre/dir05845/file000          
read File name: /mnt/lustre/dir06096/file000          
read File name: /mnt/lustre/dir06366/file000          
read File name: /mnt/lustre/dir06635/file000          
read File name: /mnt/lustre/dir06893/file000          
read File name: /mnt/lustre/dir07157/file000          
read File name: /mnt/lustre/dir07423/file000          
read File name: /mnt/lustre/dir07685/file000          
read File name: /mnt/lustre/dir07943/file000          
read File name: /mnt/lustre/dir08200/file000          
read File name: /mnt/lustre/dir08457/file000          
read File name: /mnt/lustre/dir08739/file000          
read File name: /mnt/lustre/dir09006/file000          
read File name: /mnt/lustre/dir09277/file000          
read File name: /mnt/lustre/dir09562/file000          
read File name: /mnt/lustre/dir09839/file000          
read File name: /mnt/lustre/dir10102/file000          
read File name: /mnt/lustre/dir10373/file000          
read File name: /mnt/lustre/dir10646/file000          
read File name: /mnt/lustre/dir10895/file000          
read File name: /mnt/lustre/dir11174/file000          
read File name: /mnt/lustre/dir11421/file000          
read File name: /mnt/lustre/dir11685/file000          
read File name: /mnt/lustre/dir11957/file000          
read File name: /mnt/lustre/dir12213/file000          
read File name: /mnt/lustre/dir12479/file000          
read File name: /mnt/lustre/dir12744/file000          
read File name: /mnt/lustre/dir13000/file000          
read File name: /mnt/lustre/dir13255/file000          
read File name: /mnt/lustre/dir13525/file000          
read File name: /mnt/lustre/dir13782/file000          
read File name: /mnt/lustre/dir14051/file000          
read File name: /mnt/lustre/dir14319/file000          
read File name: /mnt/lustre/dir14573/file000          
read File name: /mnt/lustre/dir14823/file000          
read File name: /mnt/lustre/dir15086/file000          
read File name: /mnt/lustre/dir15330/file000          
read File name: /mnt/lustre/dir15561/file000          
read File name: /mnt/lustre/dir15825/file000          
read File name: /mnt/lustre/dir16086/file000          
read File name: /mnt/lustre/dir16344/file000          
read File name: /mnt/lustre/dir16612/file000          
read File name: /mnt/lustre/dir16892/file000          
read File name: /mnt/lustre/dir17160/file000          
read File name: /mnt/lustre/dir17425/file000          
read File name: /mnt/lustre/dir17690/file000          
read File name: /mnt/lustre/dir17958/file000          
read File name: /mnt/lustre/dir18237/file000          
read File name: /mnt/lustre/dir18500/file000          
read File name: /mnt/lustre/dir18772/file000          
read File name: /mnt/lustre/dir19053/file000          
read File name: /mnt/lustre/dir19315/file000          
read File name: /mnt/lustre/dir19596/file000          
read File name: /mnt/lustre/dir19875/file000          
read File name: /mnt/lustre/dir20142/file000          
read File name: /mnt/lustre/dir20414/file000          
read File name: /mnt/lustre/dir20681/file000          
read File name: /mnt/lustre/dir20964/file000          
read File name: /mnt/lustre/dir21233/file000          
read File name: /mnt/lustre/dir21488/file000          
read File name: /mnt/lustre/dir21755/file000          
read File name: /mnt/lustre/dir22021/file000          
read File name: /mnt/lustre/dir22293/file000          
read File name: /mnt/lustre/dir22555/file000          
read File name: /mnt/lustre/dir22815/file000          
read File name: /mnt/lustre/dir23112/file000          
read File name: /mnt/lustre/dir23401/file000          
read File name: /mnt/lustre/dir23687/file000          
read File name: /mnt/lustre/dir23972/file000          
read File name: /mnt/lustre/dir24236/file000          
read File name: /mnt/lustre/dir24507/file000          
read File name: /mnt/lustre/dir24773/file000          
read File name: /mnt/lustre/dir25048/file000          
read File name: /mnt/lustre/dir25321/file000          
read File name: /mnt/lustre/dir25587/file000          
read File name: /mnt/lustre/dir25867/file000          
read File name: /mnt/lustre/dir26135/file000          
read File name: /mnt/lustre/dir26411/file000          
read File name: /mnt/lustre/dir26684/file000          
read File name: /mnt/lustre/dir26957/file000          
read File name: /mnt/lustre/dir27245/file000          
read File name: /mnt/lustre/dir27517/file000          
read File name: /mnt/lustre/dir27789/file000          
read File name: /mnt/lustre/dir28050/file000          
read File name: /mnt/lustre/dir28320/file000          
read File name: /mnt/lustre/dir28583/file000          
read File name: /mnt/lustre/dir28849/file000          
read File name: /mnt/lustre/dir29125/file000          
read File name: /mnt/lustre/dir29391/file000          
read File name: /mnt/lustre/dir29665/file000          
read File name: /mnt/lustre/dir29938/file000          
read File name: /mnt/lustre/dir29998/file000          

read complete

real	14m26.419s
user	0m1.484s
sys	7m27.234s

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      9.1G  3.3G  5.4G  38% /
/dev/hda1              99M   20M   75M  21% /boot
tmpfs                  16G     0   16G   0% /dev/shm
/dev/large_vg/mdt_lv  1.8T  588M  1.7T   1% /mnt/mds1
/dev/large_vg/ost_lv  128T   30G  122T   1% /mnt/ost1
192.168.77.1@o2ib:/largefs
                      128T   30G  122T   1% /mnt/lustre

# lfs df -h /mnt/lustre
UUID                       bytes        Used   Available Use% Mounted on
largefs-MDT0000_UUID        1.7T      587.9M        1.6T   0% /mnt/lustre[MDT:0]
largefs-OST0000_UUID      128.0T       29.7G      121.5T   0% /mnt/lustre[OST:0]

filesystem summary:       128.0T       29.7G      121.5T   0% /mnt/lustre


# lfs df -i /mnt/lustre
UUID                      Inodes       IUsed       IFree IUse% Mounted on
largefs-MDT0000_UUID   469636214       60024   469576190   0% /mnt/lustre[MDT:0]
largefs-OST0000_UUID   134217728       30072   134187656   0% /mnt/lustre[OST:0]

filesystem summary:    469636214       60024   469576190   0% /mnt/lustre


======================== cleanup the Lustre filesystem ========================
# umount /mnt/lustre
# umount /mnt/mds1
# umount /mnt/ost1

================== run e2fsck on device /dev/large_vg/ost_lv ==================
# time e2fsck -fy -tt /dev/large_vg/ost_lv
e2fsck 1.41.90.wc1 (18-Mar-2011)
Pass 1: Checking inodes, blocks, and sizes
Pass 1: Memory used: 276k/164368k (88k/189k), time: 209.41/ 2.93/ 3.82
Pass 1: I/O read: 125MB, write: 1MB, rate: 0.60MB/s
Pass 2: Checking directory structure
Pass 2: Memory used: 276k/201728k (65k/212k), time:  0.07/ 0.04/ 0.02
Pass 2: I/O read: 2MB, write: 0MB, rate: 28.45MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 16660k/201728k (16449k/212k), time: 215.09/ 5.62/ 6.69
Pass 3A: Memory used: 16660k/201728k (16449k/212k), time:  0.00/ 0.00/ 0.00
Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 3: Memory used: 16660k/201728k (64k/16597k), time:  0.01/ 0.00/ 0.01
Pass 3: I/O read: 1MB, write: 0MB, rate: 113.26MB/s
Pass 4: Checking reference counts
Pass 4: Memory used: 16660k/110620k (64k/16597k), time:  2.41/ 2.41/ 0.00
Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 24816k/77844k (16448k/8369k), time: 492.69/55.35/ 4.35
Pass 5: I/O read: 261MB, write: 0MB, rate: 0.53MB/s
largefs-OST0000: 30072/134217728 files (18.7% non-contiguous), 18743454/34359738368 blocks
Memory used: 24816k/77844k (16448k/8369k), time: 710.55/63.70/11.04
I/O read: 452MB, write: 1MB, rate: 0.64MB/s

real	11m51.076s
user	1m3.763s
sys	0m11.424s

======================== cleanup the Lustre filesystem ========================

============================= test duration: 8304s =============================

================ test stopped at: Fri May 13 02:01:38 PDT 2011 ================