================ test started at: Thu May 12 23:43:13 PDT 2011 ================ =============================== disks to be used =============================== /dev/md0 ================================ create the PVs ================================ Physical volume "/dev/md0" successfully created ================================ create the VG ================================ Volume group "large_vg" successfully created --- Volume group --- VG Name large_vg System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 1 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 199.39 TB PE Size 4.00 MB Total PE 52269049 Alloc PE / Size 0 / 0 Free PE / Size 52269049 / 199.39 TB VG UUID w1uEH1-EwuA-736R-THRY-gvBW-MS6x-0gSWkd ================================ create the LVs ================================ Logical volume "mdt_lv" created Logical volume "ost_lv" created --- Logical volume --- LV Name /dev/large_vg/mdt_lv VG Name large_vg LV UUID lh1J8T-1SKy-eSeC-x6a2-prHx-Pilv-An7AY3 LV Write Access read/write LV Status available # open 0 LV Size 2.00 TB Current LE 524288 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 57344 Block device 253:2 --- Logical volume --- LV Name /dev/large_vg/ost_lv VG Name large_vg LV UUID EsaifZ-7Nk0-CnQC-zt8J-yAEb-DDi3-AYX2E2 LV Write Access read/write LV Status available # open 0 LV Size 128.00 TB Current LE 33554432 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 57344 Block device 253:3 ================= run llverdev on the OST /dev/large_vg/ost_lv ================= # time llverdev -vpf /dev/large_vg/ost_lv llverdev: /dev/large_vg/ost_lv is 140737488355328 bytes (131072 GB) in size Timestamp: 1305268994 write offset: 1755316224kB 669.8 MB/s write offset: 3619684352kB 711.2 MB/s write offset: 5427429376kB 689.6 MB/s write offset: 7103053824kB 639.2 MB/s write offset: 8691646464kB 606 MB/s write offset: 10267656192kB 601.2 MB/s write offset: 11737759744kB 560.8 MB/s write offset: 13160677376kB 542.8 MB/s write offset: 14519631872kB 518.4 MB/s write offset: 16419651584kB 724.8 MB/s write offset: 19901972480kB 1328.4 MB/s write offset: 24106762240kB 1604 MB/s write offset: 27987542016kB 1480.4 MB/s write offset: 31901876224kB 1493.2 MB/s write offset: 34664873984kB 1054 MB/s write offset: 37505466368kB 1083.6 MB/s write offset: 40507539456kB 1145.2 MB/s write offset: 43355471872kB 1086.4 MB/s write offset: 46105886720kB 1049.2 MB/s write offset: 48974790656kB 1094.4 MB/s write offset: 51747225600kB 1057.6 MB/s write offset: 54614032384kB 1093.6 MB/s write offset: 57339281408kB 1039.6 MB/s write offset: 60026781696kB 1025.2 MB/s write offset: 62642978816kB 998 MB/s write offset: 65240301568kB 990.8 MB/s write offset: 67834478592kB 989.6 MB/s write offset: 70348963840kB 959.2 MB/s write offset: 73016541184kB 1017.6 MB/s write offset: 75877056512kB 1091.2 MB/s write offset: 78589722624kB 1034.8 MB/s write offset: 81351671808kB 1053.6 MB/s write offset: 83830505472kB 945.6 MB/s write offset: 86470819840kB 1007.2 MB/s write offset: 88962236416kB 950.4 MB/s write offset: 91489304576kB 964 MB/s write offset: 94011129856kB 962 MB/s write offset: 96852770816kB 1084 MB/s write offset: 99424927744kB 981.2 MB/s write offset: 101927878656kB 954.8 MB/s write offset: 104526249984kB 991.2 MB/s write offset: 107136155648kB 995.6 MB/s write offset: 109551026176kB 921.2 MB/s write offset: 112145203200kB 989.6 MB/s write offset: 114800197632kB 1012.8 MB/s write offset: 117444706304kB 1008.8 MB/s write offset: 120330387456kB 1100.8 MB/s write offset: 122948681728kB 998.8 MB/s write offset: 125665542144kB 1036.4 MB/s write offset: 128131792896kB 940.8 MB/s write offset: 130622160896kB 950 MB/s write offset: 133256183808kB 1004.8 MB/s write offset: 135814709248kB 976 MB/s write complete write offset: 137438953472kB 0.0909091 MB/s write complete read offset: 23068672kB 9 MB/s read offset: 189792256kB 63.6 MB/s read offset: 409993216kB 84 MB/s read offset: 630194176kB 84 MB/s read offset: 847249408kB 82.8 MB/s read offset: 1064304640kB 82.8 MB/s read offset: 1284505600kB 84 MB/s read offset: 1501560832kB 82.8 MB/s read offset: 1719664640kB 83.2 MB/s read offset: 1936719872kB 82.8 MB/s read offset: 2152726528kB 82.4 MB/s read offset: 2371878912kB 83.6 MB/s read offset: 2588934144kB 82.8 MB/s read offset: 2809135104kB 84 MB/s read offset: 3030384640kB 84.4 MB/s read offset: 3249537024kB 83.6 MB/s read offset: 3469737984kB 84 MB/s read offset: 3690987520kB 84.4 MB/s read offset: 3909091328kB 83.2 MB/s read offset: 4129292288kB 84 MB/s read offset: 4348444672kB 83.6 MB/s read offset: 4568645632kB 84 MB/s read offset: 4787798016kB 83.6 MB/s read offset: 5006950400kB 83.6 MB/s read offset: 5227151360kB 84 MB/s read offset: 5445255168kB 83.2 MB/s read offset: 5664407552kB 83.6 MB/s read offset: 5883559936kB 83.6 MB/s read offset: 6101663744kB 83.2 MB/s read offset: 6320816128kB 83.6 MB/s read offset: 6539968512kB 83.6 MB/s read offset: 6760169472kB 84 MB/s read offset: 6980370432kB 84 MB/s read offset: 7199522816kB 83.6 MB/s read offset: 7419723776kB 84 MB/s read offset: 7638876160kB 83.6 MB/s read offset: 7859077120kB 84 MB/s read offset: 8079278080kB 84 MB/s read offset: 8298430464kB 83.6 MB/s read offset: 8516534272kB 83.2 MB/s read offset: 8738832384kB 84.8 MB/s read offset: 8956936192kB 83.2 MB/s read offset: 9177137152kB 84 MB/s read offset: 9396289536kB 83.6 MB/s read offset: 9614393344kB 83.2 MB/s read offset: 9833545728kB 83.6 MB/s read offset: 10053746688kB 84 MB/s read offset: 10272899072kB 83.6 MB/s read offset: 10488905728kB 82.4 MB/s read offset: 10705960960kB 82.8 MB/s read offset: 10920919040kB 82 MB/s read offset: 11139022848kB 83.2 MB/s read offset: 11356078080kB 82.8 MB/s read offset: 11574181888kB 83.2 MB/s read offset: 11793334272kB 83.6 MB/s read offset: 12011438080kB 83.2 MB/s read offset: 12229541888kB 83.2 MB/s read offset: 12447645696kB 83.2 MB/s read offset: 12665749504kB 83.2 MB/s read offset: 12881756160kB 82.4 MB/s read offset: 13099859968kB 83.2 MB/s read offset: 13319012352kB 83.6 MB/s read offset: 13538164736kB 83.6 MB/s read offset: 13754171392kB 82.4 MB/s read offset: 13970178048kB 82.4 MB/s read offset: 14185136128kB 82 MB/s read offset: 14405337088kB 84 MB/s read offset: 14620295168kB 82 MB/s read offset: 14839447552kB 83.6 MB/s read offset: 15054405632kB 82 MB/s read offset: 15272509440kB 83.2 MB/s read offset: 15489564672kB 82.8 MB/s read offset: 15705571328kB 82.4 MB/s read offset: 15923675136kB 83.2 MB/s read offset: 16139681792kB 82.4 MB/s read offset: 16356737024kB 82.8 MB/s read offset: 16572743680kB 82.4 MB/s read offset: 16790847488kB 83.2 MB/s read offset: 17006854144kB 82.4 MB/s read offset: 17227055104kB 84 MB/s read offset: 17444110336kB 82.8 MB/s read offset: 17664311296kB 84 MB/s read offset: 17882415104kB 83.2 MB/s read offset: 18098421760kB 82.4 MB/s read offset: 18315476992kB 82.8 MB/s read offset: 18534629376kB 83.6 MB/s read offset: 18750636032kB 82.4 MB/s read offset: 18969788416kB 83.6 MB/s read offset: 19186843648kB 82.8 MB/s read offset: 19405996032kB 83.6 MB/s read offset: 19625148416kB 83.6 MB/s read offset: 19843252224kB 83.2 MB/s read offset: 20063453184kB 84 MB/s read offset: 20281556992kB 83.2 MB/s read offset: 20499660800kB 83.2 MB/s read offset: 20717764608kB 83.2 MB/s read offset: 20934819840kB 82.8 MB/s read offset: 21151875072kB 82.8 MB/s read offset: 21369978880kB 83.2 MB/s read offset: 21589131264kB 83.6 MB/s read offset: 21807235072kB 83.2 MB/s read offset: 22025338880kB 83.2 MB/s read offset: 22242394112kB 82.8 MB/s read offset: 22458400768kB 82.4 MB/s read offset: 22675456000kB 82.8 MB/s read offset: 22893559808kB 83.2 MB/s read offset: 23111663616kB 83.2 MB/s read offset: 23327670272kB 82.4 MB/s read offset: 23546822656kB 83.6 MB/s read offset: 23765975040kB 83.6 MB/s read offset: 23984078848kB 83.2 MB/s read offset: 24200085504kB 82.4 MB/s read offset: 24417140736kB 82.8 MB/s read offset: 24634195968kB 82.8 MB/s read offset: 24851251200kB 82.8 MB/s read offset: 25069355008kB 83.2 MB/s read offset: 25288507392kB 83.6 MB/s read offset: 25505562624kB 82.8 MB/s read offset: 25721569280kB 82.4 MB/s read offset: 25939673088kB 83.2 MB/s read offset: 26156728320kB 82.8 MB/s read offset: 26373783552kB 82.8 MB/s read offset: 26591887360kB 83.2 MB/s read offset: 26811039744kB 83.6 MB/s read offset: 27029143552kB 83.2 MB/s read offset: 27246198784kB 82.8 MB/s read offset: 27465351168kB 83.6 MB/s read offset: 27682406400kB 82.8 MB/s read offset: 27900510208kB 83.2 MB/s read offset: 28118614016kB 83.2 MB/s read offset: 28336717824kB 83.2 MB/s read offset: 28551675904kB 82 MB/s read offset: 28770828288kB 83.6 MB/s read offset: 28991029248kB 84 MB/s read offset: 29208084480kB 82.8 MB/s read offset: 29428285440kB 84 MB/s read offset: 29647437824kB 83.6 MB/s read offset: 29866590208kB 83.6 MB/s read offset: 30082596864kB 82.4 MB/s read offset: 30302797824kB 84 MB/s read offset: 30520901632kB 83.2 MB/s read offset: 30740054016kB 83.6 MB/s read offset: 30959206400kB 83.6 MB/s read offset: 31177310208kB 83.2 MB/s read offset: 31396462592kB 83.6 MB/s read offset: 31614566400kB 83.2 MB/s read offset: 31832670208kB 83.2 MB/s read offset: 32050774016kB 83.2 MB/s read offset: 32268877824kB 83.2 MB/s read offset: 32485933056kB 82.8 MB/s read offset: 32702988288kB 82.8 MB/s read offset: 32920043520kB 82.8 MB/s read offset: 33139195904kB 83.6 MB/s read offset: 33356251136kB 82.8 MB/s read offset: 33574354944kB 83.2 MB/s read offset: 33792458752kB 83.2 MB/s read offset: 34011611136kB 83.6 MB/s read offset: 34229714944kB 83.2 MB/s read offset: 34442575872kB 81.2 MB/s read offset: 34657533952kB 82 MB/s read offset: 34873540608kB 82.4 MB/s read offset: 35091644416kB 83.2 MB/s read offset: 35307651072kB 82.4 MB/s read offset: 35524706304kB 82.8 MB/s read offset: 35740712960kB 82.4 MB/s read offset: 35957768192kB 82.8 MB/s read offset: 36175872000kB 83.2 MB/s read offset: 36395024384kB 83.6 MB/s read offset: 36613128192kB 83.2 MB/s read offset: 36831232000kB 83.2 MB/s read offset: 37048287232kB 82.8 MB/s read offset: 37266391040kB 83.2 MB/s read offset: 37484494848kB 83.2 MB/s read offset: 37702598656kB 83.2 MB/s read offset: 37921751040kB 83.6 MB/s read offset: 38138806272kB 82.8 MB/s read offset: 38355861504kB 82.8 MB/s read offset: 38573965312kB 83.2 MB/s read offset: 38794166272kB 84 MB/s read offset: 39012270080kB 83.2 MB/s read offset: 39230373888kB 83.2 MB/s read offset: 39449526272kB 83.6 MB/s read offset: 39667630080kB 83.2 MB/s read offset: 39886782464kB 83.6 MB/s read offset: 40105934848kB 83.6 MB/s read offset: 40324038656kB 83.2 MB/s read offset: 40543191040kB 83.6 MB/s read offset: 40760246272kB 82.8 MB/s read offset: 40979398656kB 83.6 MB/s read offset: 41195405312kB 82.4 MB/s read offset: 41413509120kB 83.2 MB/s read offset: 41630564352kB 82.8 MB/s read offset: 41844473856kB 81.6 MB/s read offset: 42060480512kB 82.4 MB/s read offset: 42276487168kB 82.4 MB/s read offset: 42493542400kB 82.8 MB/s read offset: 42709549056kB 82.4 MB/s read offset: 42927652864kB 83.2 MB/s read offset: 43143659520kB 82.4 MB/s read offset: 43358617600kB 82 MB/s read offset: 43575672832kB 82.8 MB/s read offset: 43790630912kB 82 MB/s read offset: 44006637568kB 82.4 MB/s read offset: 44223692800kB 82.8 MB/s read offset: 44440748032kB 82.8 MB/s read offset: 44658851840kB 83.2 MB/s read offset: 44875907072kB 82.8 MB/s read offset: 45092962304kB 82.8 MB/s read offset: 45308968960kB 82.4 MB/s read offset: 45526024192kB 82.8 MB/s read offset: 45743079424kB 82.8 MB/s read offset: 45961183232kB 83.2 MB/s read offset: 46176141312kB 82 MB/s read offset: 46393196544kB 82.8 MB/s read offset: 46607106048kB 81.6 MB/s read offset: 46823112704kB 82.4 MB/s read offset: 47039119360kB 82.4 MB/s read offset: 47253028864kB 81.6 MB/s read offset: 47467986944kB 82 MB/s read offset: 47683993600kB 82.4 MB/s read offset: 47898951680kB 82 MB/s read offset: 48116006912kB 82.8 MB/s read offset: 48332013568kB 82.4 MB/s read offset: 48546971648kB 82 MB/s read offset: 48762978304kB 82.4 MB/s read offset: 48980033536kB 82.8 MB/s read offset: 49193943040kB 81.6 MB/s read offset: 49410998272kB 82.8 MB/s read offset: 49628053504kB 82.8 MB/s read offset: 49840914432kB 81.2 MB/s read offset: 50056921088kB 82.4 MB/s read offset: 50272927744kB 82.4 MB/s read offset: 50489982976kB 82.8 MB/s read offset: 50704941056kB 82 MB/s read offset: 50919899136kB 82 MB/s read offset: 51134857216kB 82 MB/s read offset: 51351912448kB 82.8 MB/s read offset: 51567919104kB 82.4 MB/s read offset: 51783925760kB 82.4 MB/s read offset: 51999932416kB 82.4 MB/s read offset: 52216987648kB 82.8 MB/s read offset: 52432994304kB 82.4 MB/s read offset: 52644806656kB 80.8 MB/s read offset: 52858716160kB 81.6 MB/s read offset: 53073674240kB 82 MB/s read offset: 53289680896kB 82.4 MB/s read offset: 53504638976kB 82 MB/s read offset: 53717499904kB 81.2 MB/s read offset: 53931409408kB 81.6 MB/s read offset: 54148464640kB 82.8 MB/s read offset: 54362374144kB 81.6 MB/s read offset: 54577332224kB 82 MB/s read offset: 54793338880kB 82.4 MB/s read offset: 55007248384kB 81.6 MB/s read offset: 55221157888kB 81.6 MB/s read offset: 55436115968kB 82 MB/s read offset: 55650025472kB 81.6 MB/s read offset: 55864983552kB 82 MB/s read offset: 56079941632kB 82 MB/s read offset: 56294899712kB 82 MB/s read offset: 56509857792kB 82 MB/s read offset: 56723767296kB 81.6 MB/s read offset: 56937676800kB 81.6 MB/s read offset: 57151586304kB 81.6 MB/s read offset: 57365495808kB 81.6 MB/s read offset: 57581502464kB 82.4 MB/s read offset: 57793314816kB 80.8 MB/s read offset: 58007224320kB 81.6 MB/s read offset: 58220085248kB 81.2 MB/s read offset: 58436091904kB 82.4 MB/s read offset: 58651049984kB 82 MB/s read offset: 58864959488kB 81.6 MB/s read offset: 59080966144kB 82.4 MB/s read offset: 59296972800kB 82.4 MB/s read offset: 59514028032kB 82.8 MB/s read offset: 59731083264kB 82.8 MB/s read offset: 59948138496kB 82.8 MB/s read offset: 60164145152kB 82.4 MB/s read offset: 60379103232kB 82 MB/s read offset: 60595109888kB 82.4 MB/s read offset: 60811116544kB 82.4 MB/s read offset: 61020831744kB 80 MB/s read offset: 61235789824kB 82 MB/s read offset: 61450747904kB 82 MB/s read offset: 61668851712kB 83.2 MB/s read offset: 61886955520kB 83.2 MB/s read offset: 62102962176kB 82.4 MB/s read offset: 62313725952kB 80.4 MB/s read offset: 62526586880kB 81.2 MB/s read offset: 62740496384kB 81.6 MB/s read offset: 62952308736kB 80.8 MB/s read offset: 63168315392kB 82.4 MB/s read offset: 63382224896kB 81.6 MB/s read offset: 63597182976kB 82 MB/s read offset: 63812141056kB 82 MB/s read offset: 64026050560kB 81.6 MB/s read offset: 64239960064kB 81.6 MB/s read offset: 64455966720kB 82.4 MB/s read offset: 64671973376kB 82.4 MB/s read offset: 64887980032kB 82.4 MB/s read offset: 65102938112kB 82 MB/s read offset: 65315799040kB 81.2 MB/s read offset: 65532854272kB 82.8 MB/s read offset: 65748860928kB 82.4 MB/s read offset: 65964867584kB 82.4 MB/s read offset: 66179825664kB 82 MB/s read offset: 66394783744kB 82 MB/s read offset: 66611838976kB 82.8 MB/s read offset: 66827845632kB 82.4 MB/s read offset: 67042803712kB 82 MB/s read offset: 67256713216kB 81.6 MB/s read offset: 67471671296kB 82 MB/s read offset: 67687677952kB 82.4 MB/s read offset: 67901587456kB 81.6 MB/s read offset: 68115496960kB 81.6 MB/s read offset: 68331503616kB 82.4 MB/s read offset: 68547510272kB 82.4 MB/s read offset: 68762468352kB 82 MB/s read offset: 68977426432kB 82 MB/s read offset: 69192384512kB 82 MB/s read offset: 69407342592kB 82 MB/s read offset: 69622300672kB 82 MB/s read offset: 69839355904kB 82.8 MB/s read offset: 70052216832kB 81.2 MB/s read offset: 70267174912kB 82 MB/s read offset: 70480035840kB 81.2 MB/s read offset: 70692896768kB 81.2 MB/s read offset: 70905757696kB 81.2 MB/s read offset: 71120715776kB 82 MB/s read offset: 71334625280kB 81.6 MB/s read offset: 71550631936kB 82.4 MB/s read offset: 71764541440kB 81.6 MB/s read offset: 71979499520kB 82 MB/s read offset: 72195506176kB 82.4 MB/s read offset: 72409415680kB 81.6 MB/s read offset: 72623325184kB 81.6 MB/s read offset: 72837234688kB 81.6 MB/s read offset: 73053241344kB 82.4 MB/s read offset: 73269248000kB 82.4 MB/s read offset: 73483157504kB 81.6 MB/s read offset: 73698115584kB 82 MB/s read offset: 73912025088kB 81.6 MB/s read offset: 74125934592kB 81.6 MB/s read offset: 74336698368kB 80.4 MB/s read offset: 74548510720kB 80.8 MB/s read offset: 74759274496kB 80.4 MB/s read offset: 74974232576kB 82 MB/s read offset: 75187093504kB 81.2 MB/s read offset: 75398905856kB 80.8 MB/s read offset: 75610718208kB 80.8 MB/s read offset: 75827773440kB 82.8 MB/s read offset: 76043780096kB 82.4 MB/s read offset: 76259786752kB 82.4 MB/s read offset: 76475793408kB 82.4 MB/s read offset: 76689702912kB 81.6 MB/s read offset: 76905709568kB 82.4 MB/s read offset: 77118570496kB 81.2 MB/s read offset: 77332480000kB 81.6 MB/s read offset: 77545340928kB 81.2 MB/s read offset: 77757153280kB 80.8 MB/s read offset: 77971062784kB 81.6 MB/s read offset: 78184972288kB 81.6 MB/s read offset: 78393638912kB 79.6 MB/s read offset: 78608596992kB 82 MB/s read offset: 78822506496kB 81.6 MB/s read offset: 79035367424kB 81.2 MB/s read offset: 79249276928kB 81.6 MB/s read offset: 79462137856kB 81.2 MB/s read offset: 79678144512kB 82.4 MB/s read offset: 79893102592kB 82 MB/s read offset: 80105963520kB 81.2 MB/s read offset: 80318824448kB 81.2 MB/s read offset: 80532733952kB 81.6 MB/s read offset: 80744546304kB 80.8 MB/s read offset: 80958455808kB 81.6 MB/s read offset: 81172365312kB 81.6 MB/s read offset: 81386274816kB 81.6 MB/s read offset: 81599135744kB 81.2 MB/s read offset: 81813045248kB 81.6 MB/s read offset: 82025906176kB 81.2 MB/s read offset: 82240864256kB 82 MB/s read offset: 82453725184kB 81.2 MB/s read offset: 82665537536kB 80.8 MB/s read offset: 82875252736kB 80 MB/s read offset: 83089162240kB 81.6 MB/s read offset: 83303071744kB 81.6 MB/s read offset: 83518029824kB 82 MB/s read offset: 83735085056kB 82.8 MB/s read offset: 83950043136kB 82 MB/s read offset: 84165001216kB 82 MB/s read offset: 84378910720kB 81.6 MB/s read offset: 84591771648kB 81.2 MB/s read offset: 84802535424kB 80.4 MB/s read offset: 85015396352kB 81.2 MB/s read offset: 85229305856kB 81.6 MB/s read offset: 85443215360kB 81.6 MB/s read offset: 85657124864kB 81.6 MB/s read offset: 85872082944kB 82 MB/s read offset: 86085992448kB 81.6 MB/s read offset: 86300950528kB 82 MB/s read offset: 86513811456kB 81.2 MB/s read offset: 86725623808kB 80.8 MB/s read offset: 86939533312kB 81.6 MB/s read offset: 87153442816kB 81.6 MB/s read offset: 87367352320kB 81.6 MB/s read offset: 87580213248kB 81.2 MB/s read offset: 87795171328kB 82 MB/s read offset: 88008032256kB 81.2 MB/s read offset: 88219844608kB 80.8 MB/s read offset: 88434802688kB 82 MB/s read offset: 88648712192kB 81.6 MB/s read offset: 88862621696kB 81.6 MB/s read offset: 89075482624kB 81.2 MB/s read offset: 89287294976kB 80.8 MB/s read offset: 89501204480kB 81.6 MB/s read offset: 89714065408kB 81.2 MB/s read offset: 89927974912kB 81.6 MB/s read offset: 90141884416kB 81.6 MB/s read offset: 90355793920kB 81.6 MB/s read offset: 90568654848kB 81.2 MB/s read offset: 90783612928kB 82 MB/s read offset: 90998571008kB 82 MB/s read offset: 91211431936kB 81.2 MB/s read offset: 91427438592kB 82.4 MB/s read offset: 91641348096kB 81.6 MB/s read offset: 91852111872kB 80.4 MB/s read offset: 92063924224kB 80.8 MB/s read offset: 92275736576kB 80.8 MB/s read offset: 92488597504kB 81.2 MB/s read offset: 92701458432kB 81.2 MB/s read offset: 92916416512kB 82 MB/s read offset: 93131374592kB 82 MB/s read offset: 93346332672kB 82 MB/s read offset: 93559193600kB 81.2 MB/s read offset: 93773103104kB 81.6 MB/s read offset: 93988061184kB 82 MB/s read offset: 94199873536kB 80.8 MB/s read offset: 94411685888kB 80.8 MB/s read offset: 94626643968kB 82 MB/s read offset: 94839504896kB 81.2 MB/s read offset: 95051317248kB 80.8 MB/s read offset: 95264178176kB 81.2 MB/s read offset: 95478087680kB 81.6 MB/s read offset: 95690948608kB 81.2 MB/s read offset: 95906955264kB 82.4 MB/s read offset: 96119816192kB 81.2 MB/s read offset: 96332677120kB 81.2 MB/s read offset: 96544489472kB 80.8 MB/s read offset: 96759447552kB 82 MB/s read offset: 96974405632kB 82 MB/s read offset: 97186217984kB 80.8 MB/s read offset: 97400127488kB 81.6 MB/s read offset: 97612988416kB 81.2 MB/s read offset: 97824800768kB 80.8 MB/s read offset: 98038710272kB 81.6 MB/s read offset: 98253668352kB 82 MB/s read offset: 98468626432kB 82 MB/s read offset: 98681487360kB 81.2 MB/s read offset: 98893299712kB 80.8 MB/s read offset: 99106160640kB 81.2 MB/s read offset: 99317972992kB 80.8 MB/s read offset: 99531882496kB 81.6 MB/s read offset: 99746840576kB 82 MB/s read offset: 99960750080kB 81.6 MB/s read offset: 100174659584kB 81.6 MB/s read offset: 100387520512kB 81.2 MB/s read offset: 100599332864kB 80.8 MB/s read offset: 100812193792kB 81.2 MB/s read offset: 101027151872kB 82 MB/s read offset: 101242109952kB 82 MB/s read offset: 101457068032kB 82 MB/s read offset: 101668880384kB 80.8 MB/s read offset: 101881741312kB 81.2 MB/s read offset: 102097747968kB 82.4 MB/s read offset: 102311657472kB 81.6 MB/s read offset: 102525566976kB 81.6 MB/s read offset: 102741573632kB 82.4 MB/s read offset: 102955483136kB 81.6 MB/s read offset: 103171489792kB 82.4 MB/s read offset: 103384350720kB 81.2 MB/s read offset: 103596163072kB 80.8 MB/s read offset: 103810072576kB 81.6 MB/s read offset: 104023982080kB 81.6 MB/s read offset: 104237891584kB 81.6 MB/s read offset: 104451801088kB 81.6 MB/s read offset: 104665710592kB 81.6 MB/s read offset: 104878571520kB 81.2 MB/s read offset: 105094578176kB 82.4 MB/s read offset: 105307439104kB 81.2 MB/s read offset: 105521348608kB 81.6 MB/s read offset: 105736306688kB 82 MB/s read offset: 105944973312kB 79.6 MB/s read offset: 106150494208kB 78.4 MB/s read offset: 106360209408kB 80 MB/s read offset: 106573070336kB 81.2 MB/s read offset: 106788028416kB 82 MB/s read offset: 107000889344kB 81.2 MB/s read offset: 107212701696kB 80.8 MB/s read offset: 107424514048kB 80.8 MB/s read offset: 107639472128kB 82 MB/s read offset: 107854430208kB 82 MB/s read offset: 108067291136kB 81.2 MB/s read offset: 108282249216kB 82 MB/s read offset: 108495110144kB 81.2 MB/s read offset: 108707971072kB 81.2 MB/s read offset: 108920832000kB 81.2 MB/s read offset: 109134741504kB 81.6 MB/s read offset: 109348651008kB 81.6 MB/s read offset: 109560463360kB 80.8 MB/s read offset: 109775421440kB 82 MB/s read offset: 109988282368kB 81.2 MB/s read offset: 110201143296kB 81.2 MB/s read offset: 110414004224kB 81.2 MB/s read offset: 110626865152kB 81.2 MB/s read offset: 110840774656kB 81.6 MB/s read offset: 111052587008kB 80.8 MB/s read offset: 111264399360kB 80.8 MB/s read offset: 111476211712kB 80.8 MB/s read offset: 111689072640kB 81.2 MB/s read offset: 111902982144kB 81.6 MB/s read offset: 112114794496kB 80.8 MB/s read offset: 112327655424kB 81.2 MB/s read offset: 112537370624kB 80 MB/s read offset: 112752328704kB 82 MB/s read offset: 112965189632kB 81.2 MB/s read offset: 113175953408kB 80.4 MB/s read offset: 113385668608kB 80 MB/s read offset: 113596432384kB 80.4 MB/s read offset: 113809293312kB 81.2 MB/s read offset: 114022154240kB 81.2 MB/s read offset: 114233966592kB 80.8 MB/s read offset: 114447876096kB 81.6 MB/s read offset: 114660737024kB 81.2 MB/s read offset: 114873597952kB 81.2 MB/s read offset: 115085410304kB 80.8 MB/s read offset: 115298271232kB 81.2 MB/s read offset: 115512180736kB 81.6 MB/s read offset: 115723993088kB 80.8 MB/s read offset: 115937902592kB 81.6 MB/s read offset: 116151812096kB 81.6 MB/s read offset: 116364673024kB 81.2 MB/s read offset: 116576485376kB 80.8 MB/s read offset: 116786200576kB 80 MB/s read offset: 116998012928kB 80.8 MB/s read offset: 117210873856kB 81.2 MB/s read offset: 117422686208kB 80.8 MB/s read offset: 117635547136kB 81.2 MB/s read offset: 117849456640kB 81.6 MB/s read offset: 118060220416kB 80.4 MB/s read offset: 118272032768kB 80.8 MB/s read offset: 118484893696kB 81.2 MB/s read offset: 118697754624kB 81.2 MB/s read offset: 118910615552kB 81.2 MB/s read offset: 119124525056kB 81.6 MB/s read offset: 119336337408kB 80.8 MB/s read offset: 119547101184kB 80.4 MB/s read offset: 119758913536kB 80.8 MB/s read offset: 119969677312kB 80.4 MB/s read offset: 120181489664kB 80.8 MB/s read offset: 120394350592kB 81.2 MB/s read offset: 120606162944kB 80.8 MB/s read offset: 120817975296kB 80.8 MB/s read offset: 121029787648kB 80.8 MB/s read offset: 121243697152kB 81.6 MB/s read offset: 121454460928kB 80.4 MB/s read offset: 121667321856kB 81.2 MB/s read offset: 121879134208kB 80.8 MB/s read offset: 122090946560kB 80.8 MB/s read offset: 122305904640kB 82 MB/s read offset: 122518765568kB 81.2 MB/s read offset: 122730577920kB 80.8 MB/s read offset: 122943438848kB 81.2 MB/s read offset: 123153154048kB 80 MB/s read offset: 123363917824kB 80.4 MB/s read offset: 123577827328kB 81.6 MB/s read offset: 123789639680kB 80.8 MB/s read offset: 124002500608kB 81.2 MB/s read offset: 124214312960kB 80.8 MB/s read offset: 124424028160kB 80 MB/s read offset: 124636889088kB 81.2 MB/s read offset: 124851847168kB 82 MB/s read offset: 125064708096kB 81.2 MB/s read offset: 125277569024kB 81.2 MB/s read offset: 125490429952kB 81.2 MB/s read offset: 125702242304kB 80.8 MB/s read offset: 125915103232kB 81.2 MB/s read offset: 126125867008kB 80.4 MB/s read offset: 126337679360kB 80.8 MB/s read offset: 126550540288kB 81.2 MB/s read offset: 126761304064kB 80.4 MB/s read offset: 126972067840kB 80.4 MB/s read offset: 127183880192kB 80.8 MB/s read offset: 127396741120kB 81.2 MB/s read offset: 127607504896kB 80.4 MB/s read offset: 127819317248kB 80.8 MB/s read offset: 128031129600kB 80.8 MB/s read offset: 128241893376kB 80.4 MB/s read offset: 128452657152kB 80.4 MB/s read offset: 128664469504kB 80.8 MB/s read offset: 128875233280kB 80.4 MB/s read offset: 129085997056kB 80.4 MB/s read offset: 129296760832kB 80.4 MB/s read offset: 129506476032kB 80 MB/s read offset: 129719336960kB 81.2 MB/s read offset: 129930100736kB 80.4 MB/s read offset: 130142961664kB 81.2 MB/s read offset: 130352676864kB 80 MB/s read offset: 130563440640kB 80.4 MB/s read offset: 130773155840kB 80 MB/s read offset: 130983919616kB 80.4 MB/s read offset: 131195731968kB 80.8 MB/s read offset: 131407544320kB 80.8 MB/s read offset: 131618308096kB 80.4 MB/s read offset: 131828023296kB 80 MB/s read offset: 132037738496kB 80 MB/s read offset: 132247453696kB 80 MB/s read offset: 132459266048kB 80.8 MB/s read offset: 132670029824kB 80.4 MB/s read offset: 132881842176kB 80.8 MB/s read offset: 133093654528kB 80.8 MB/s read offset: 133304418304kB 80.4 MB/s read offset: 133515182080kB 80.4 MB/s read offset: 133726994432kB 80.8 MB/s read offset: 133935661056kB 79.6 MB/s read offset: 134148521984kB 81.2 MB/s read offset: 134359285760kB 80.4 MB/s read offset: 134569000960kB 80 MB/s read offset: 134778716160kB 80 MB/s read offset: 134989479936kB 80.4 MB/s read offset: 135198146560kB 79.6 MB/s read offset: 135405764608kB 79.2 MB/s read offset: 135615479808kB 80 MB/s read offset: 135826243584kB 80.4 MB/s read offset: 136038055936kB 80.8 MB/s read offset: 136248819712kB 80.4 MB/s read offset: 136459583488kB 80.4 MB/s read offset: 136670347264kB 80.4 MB/s read offset: 136881111040kB 80.4 MB/s read offset: 137090826240kB 80 MB/s read offset: 137302638592kB 80.8 MB/s read complete read offset: 137438953472kB 0.142857 MB/s read complete llverdev: data verified successfully real 58m2.904s user 0m2.725s sys 9m47.551s ===================== format the MDT /dev/large_vg/mdt_lv ===================== # time mkfs.lustre --reformat --fsname=largefs --mdt --mgs --mkfsoptions='-t ext4' /dev/large_vg/mdt_lv Permanent disk data: Target: largefs-MDTffff Index: unassigned Lustre FS: largefs Mount type: ldiskfs Flags: 0x75 (MDT MGS needs_index first_time update ) Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro Parameters: mdt.group_upcall=/usr/sbin/l_getgroups device size = 2097152MB 2 6 18 formatting backing filesystem ldiskfs on /dev/large_vg/mdt_lv target name largefs-MDTffff 4k blocks 536870912 options -t ext4 -J size=400 -i 4096 -I 512 -q -O dir_index,uninit_groups -F mkfs_cmd = mke2fs -j -b 4096 -L largefs-MDTffff -t ext4 -J size=400 -i 4096 -I 512 -q -O dir_index,uninit_groups -F /dev/large_vg/mdt_lv 536870912 Writing CONFIGS/mountdata real 4m20.365s user 0m19.411s sys 3m49.944s ===================== format the OST /dev/large_vg/ost_lv ===================== # time mkfs.lustre --reformat --fsname=largefs --ost --mgsnode=192.168.77.1@o2ib --mkfsoptions='-t ext4 -i 1058576' --mountfsoptions='errors=remount-ro,extents,mballoc,force_over_16tb' /dev/large_vg/ost_lv Permanent disk data: Target: largefs-OSTffff Index: unassigned Lustre FS: largefs Mount type: ldiskfs Flags: 0x72 (OST needs_index first_time update ) Persistent mount opts: errors=remount-ro,extents,mballoc,force_over_16tb Parameters: mgsnode=192.168.77.1@o2ib device size = 134217728MB 2 6 18 formatting backing filesystem ldiskfs on /dev/large_vg/ost_lv target name largefs-OSTffff 4k blocks 34359738368 options -t ext4 -i 1058576 -J size=400 -I 256 -q -O dir_index,extents,uninit_groups -F mkfs_cmd = mke2fs -j -b 4096 -L largefs-OSTffff -t ext4 -i 1058576 -J size=400 -I 256 -q -O dir_index,extents,uninit_groups -F /dev/large_vg/ost_lv 34359738368 Writing CONFIGS/mountdata real 1m22.322s user 0m6.786s sys 0m33.467s =========== dump the super block information on /dev/large_vg/ost_lv =========== # dumpe2fs -h /dev/large_vg/ost_lv dumpe2fs 1.41.90.wc1 (18-Mar-2011) Filesystem volume name: largefs-OSTffff Last mounted on: /tmp/mnt6d98sv/tmp/mnt6d98sv Filesystem UUID: 5e2a898b-da07-48a4-93cf-1c3a3159b517 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr dir_index filetype extent 64bit flex_bg sparse_super huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 134217728 Block count: 34359738368 Reserved block count: 1717986918 Free blocks: 34348675033 Free inodes: 134217715 First block: 0 Block size: 4096 Fragment size: 4096 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 128 Inode blocks per group: 8 Flex block group size: 16 Filesystem created: Fri May 13 00:45:38 2011 Last mount time: Fri May 13 00:47:00 2011 Last write time: Fri May 13 00:47:01 2011 Mount count: 1 Maximum mount count: 20 Last checked: Fri May 13 00:45:38 2011 Check interval: 0 () Lifetime writes: 34 GB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 32b48ffa-1ae0-4da0-9cfd-6d0edbeece5a Journal backup: inode blocks Journal features: FEATURE_I1 Journal size: 400M Journal length: 102400 Journal sequence: 0x00000004 Journal start: 0 ========= mount the OST /dev/large_vg/ost_lv as an ldiskfs filesystem ========= # mount -t ldiskfs -o extents,mballoc,force_over_16tb /dev/large_vg/ost_lv /mnt/ost1 /dev/mapper/large_vg-ost_lv on /mnt/ost1 type ldiskfs (rw,extents,mballoc,force_over_16tb) Filesystem Size Used Avail Use% Mounted on /dev/mapper/large_vg-ost_lv 128T 401M 122T 1% /mnt/ost1 ===== run llverfs in partial mode on the OST ldiskfs filesystem /mnt/ost1 ===== # time llverfs -vpl /mnt/ost1 Timestamp: 1305272823 write File name: /mnt/ost1/dir00000/file000 write File name: /mnt/ost1/dir00010/file000 write File name: /mnt/ost1/dir00164/file000 write File name: /mnt/ost1/dir00391/file000 write File name: /mnt/ost1/dir00626/file000 write File name: /mnt/ost1/dir00863/file000 write File name: /mnt/ost1/dir01120/file000 write File name: /mnt/ost1/dir01397/file000 write File name: /mnt/ost1/dir01687/file000 write File name: /mnt/ost1/dir01959/file000 write File name: /mnt/ost1/dir02244/file000 write File name: /mnt/ost1/dir02508/file000 write File name: /mnt/ost1/dir02701/file000 write File name: /mnt/ost1/dir02914/file000 write File name: /mnt/ost1/dir03168/file000 write File name: /mnt/ost1/dir03473/file000 write File name: /mnt/ost1/dir03766/file000 write File name: /mnt/ost1/dir04067/file000 write File name: /mnt/ost1/dir04358/file000 write File name: /mnt/ost1/dir04650/file000 write File name: /mnt/ost1/dir04815/file000 write File name: /mnt/ost1/dir05048/file000 write File name: /mnt/ost1/dir05349/file000 write File name: /mnt/ost1/dir05627/file000 write File name: /mnt/ost1/dir05924/file000 write File name: /mnt/ost1/dir06245/file000 write File name: /mnt/ost1/dir06546/file000 write File name: /mnt/ost1/dir06815/file000 write File name: /mnt/ost1/dir07121/file000 write File name: /mnt/ost1/dir07439/file000 write File name: /mnt/ost1/dir07750/file000 write File name: /mnt/ost1/dir08057/file000 write File name: /mnt/ost1/dir08224/file000 write File name: /mnt/ost1/dir08317/file000 write File name: /mnt/ost1/dir08636/file000 write File name: /mnt/ost1/dir08948/file000 write File name: /mnt/ost1/dir09268/file000 write File name: /mnt/ost1/dir09592/file000 write File name: /mnt/ost1/dir09907/file000 write File name: /mnt/ost1/dir10221/file000 write File name: /mnt/ost1/dir10537/file000 write File name: /mnt/ost1/dir10865/file000 write File name: /mnt/ost1/dir11171/file000 write File name: /mnt/ost1/dir11483/file000 write File name: /mnt/ost1/dir11643/file000 write File name: /mnt/ost1/dir11749/file000 write File name: /mnt/ost1/dir12031/file000 write File name: /mnt/ost1/dir12345/file000 write File name: /mnt/ost1/dir12692/file000 write File name: /mnt/ost1/dir13028/file000 write File name: /mnt/ost1/dir13333/file000 write File name: /mnt/ost1/dir13630/file000 write File name: /mnt/ost1/dir13929/file000 write File name: /mnt/ost1/dir14256/file000 write File name: /mnt/ost1/dir14591/file000 write File name: /mnt/ost1/dir14921/file000 write File name: /mnt/ost1/dir15096/file000 write File name: /mnt/ost1/dir15197/file000 write File name: /mnt/ost1/dir15538/file000 write File name: /mnt/ost1/dir15880/file000 write File name: /mnt/ost1/dir16222/file000 write File name: /mnt/ost1/dir16534/file000 write File name: /mnt/ost1/dir16859/file000 write File name: /mnt/ost1/dir17184/file000 write File name: /mnt/ost1/dir17509/file000 write File name: /mnt/ost1/dir17829/file000 write File name: /mnt/ost1/dir18157/file000 write File name: /mnt/ost1/dir18499/file000 write File name: /mnt/ost1/dir18671/file000 write File name: /mnt/ost1/dir18758/file000 write File name: /mnt/ost1/dir19067/file000 write File name: /mnt/ost1/dir19383/file000 write File name: /mnt/ost1/dir19708/file000 write File name: /mnt/ost1/dir20044/file000 write File name: /mnt/ost1/dir20380/file000 write File name: /mnt/ost1/dir20704/file000 write File name: /mnt/ost1/dir21030/file000 write File name: /mnt/ost1/dir21355/file000 write File name: /mnt/ost1/dir21666/file000 write File name: /mnt/ost1/dir22003/file000 write File name: /mnt/ost1/dir22156/file000 write File name: /mnt/ost1/dir22266/file000 write File name: /mnt/ost1/dir22597/file000 write File name: /mnt/ost1/dir22931/file000 write File name: /mnt/ost1/dir23255/file000 write File name: /mnt/ost1/dir23579/file000 write File name: /mnt/ost1/dir23914/file000 write File name: /mnt/ost1/dir24232/file000 write File name: /mnt/ost1/dir24546/file000 write File name: /mnt/ost1/dir24871/file000 write File name: /mnt/ost1/dir25191/file000 write File name: /mnt/ost1/dir25517/file000 write File name: /mnt/ost1/dir25674/file000 write File name: /mnt/ost1/dir25800/file000 write File name: /mnt/ost1/dir26122/file000 write File name: /mnt/ost1/dir26456/file000 write File name: /mnt/ost1/dir26810/file000 write File name: /mnt/ost1/dir27164/file000 write File name: /mnt/ost1/dir27509/file000 write File name: /mnt/ost1/dir27806/file000 write File name: /mnt/ost1/dir28098/file000 write File name: /mnt/ost1/dir28440/file000 write File name: /mnt/ost1/dir28780/file000 write File name: /mnt/ost1/dir29106/file000 write File name: /mnt/ost1/dir29266/file000 write File name: /mnt/ost1/dir29397/file000 write File name: /mnt/ost1/dir29712/file000 write File name: /mnt/ost1/dir29998/file000 write complete read File name: /mnt/ost1/dir00194/file000 read File name: /mnt/ost1/dir00562/file000 read File name: /mnt/ost1/dir00859/file000 read File name: /mnt/ost1/dir01248/file000 read File name: /mnt/ost1/dir01623/file000 read File name: /mnt/ost1/dir02070/file000 read File name: /mnt/ost1/dir02536/file000 read File name: /mnt/ost1/dir02940/file000 read File name: /mnt/ost1/dir03383/file000 read File name: /mnt/ost1/dir03701/file000 read File name: /mnt/ost1/dir03967/file000 read File name: /mnt/ost1/dir04355/file000 read File name: /mnt/ost1/dir04710/file000 read File name: /mnt/ost1/dir04981/file000 read File name: /mnt/ost1/dir05300/file000 read File name: /mnt/ost1/dir05677/file000 read File name: /mnt/ost1/dir06034/file000 read File name: /mnt/ost1/dir06292/file000 read File name: /mnt/ost1/dir06628/file000 read File name: /mnt/ost1/dir12455/file000 read File name: /mnt/ost1/dir25781/file000 read File name: /mnt/ost1/dir26116/file000 read File name: /mnt/ost1/dir26442/file000 read File name: /mnt/ost1/dir26810/file000 read File name: /mnt/ost1/dir26862/file000 read File name: /mnt/ost1/dir29006/file000 read File name: /mnt/ost1/dir29359/file000 read File name: /mnt/ost1/dir29731/file000 read File name: /mnt/ost1/dir29998/file000 read complete real 11m23.279s user 0m1.217s sys 1m33.825s ===================== unmount the OST /dev/large_vg/ost_lv ===================== # umount /mnt/ost1 ================== run e2fsck on device /dev/large_vg/ost_lv ================== # time e2fsck -fy -tt /dev/large_vg/ost_lv e2fsck 1.41.90.wc1 (18-Mar-2011) Pass 1: Checking inodes, blocks, and sizes Pass 1: Memory used: 276k/166364k (71k/206k), time: 208.19/ 3.52/ 3.93 Pass 1: I/O read: 118MB, write: 1MB, rate: 0.57MB/s Pass 2: Checking directory structure Pass 2: Memory used: 276k/202552k (64k/213k), time: 318.25/ 0.91/ 1.27 Pass 2: I/O read: 118MB, write: 0MB, rate: 0.37MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 16624k/202552k (16449k/176k), time: 533.78/ 6.95/ 8.32 Pass 3A: Memory used: 16624k/202552k (16448k/177k), time: 0.00/ 0.00/ 0.00 Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Memory used: 16624k/202200k (64k/16561k), time: 0.01/ 0.01/ 0.01 Pass 3: I/O read: 1MB, write: 0MB, rate: 74.33MB/s Pass 4: Checking reference counts Pass 4: Memory used: 16624k/110620k (64k/16561k), time: 2.63/ 2.62/ 0.00 Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 5: Checking group summary information Pass 5: Memory used: 24816k/77844k (16448k/8369k), time: 492.52/54.49/ 4.83 Pass 5: I/O read: 259MB, write: 0MB, rate: 0.53MB/s largefs-OSTffff: 60012/134217728 files (0.2% non-contiguous), 18773235/34359738368 blocks Memory used: 24816k/77844k (16448k/8369k), time: 1029.31/64.43/13.15 I/O read: 558MB, write: 1MB, rate: 0.54MB/s real 17m9.881s user 1m4.495s sys 0m13.575s ====== mount the OST /dev/large_vg/ost_lv as an ldiskfs filesystem again ====== # mount -t ldiskfs -o extents,mballoc,force_over_16tb /dev/large_vg/ost_lv /mnt/ost1 /dev/mapper/large_vg-ost_lv on /mnt/ost1 type ldiskfs (rw,extents,mballoc,force_over_16tb) Filesystem Size Used Avail Use% Mounted on /dev/mapper/large_vg-ost_lv 128T 30G 122T 1% /mnt/ost1 # time rm -rf /mnt/ost1/{*.filecount,dir*} real 18m36.392s user 0m0.743s sys 0m9.191s ===================== unmount the OST /dev/large_vg/ost_lv ===================== # umount /mnt/ost1 ========================= setup the Lustre filesystem ========================= # mount -t lustre -o user_xattr,acl /dev/large_vg/mdt_lv /mnt/mds1 /dev/large_vg/mdt_lv on /mnt/mds1 type lustre (rw,user_xattr,acl) Filesystem Size Used Avail Use% Mounted on /dev/large_vg/mdt_lv 1.8T 471M 1.7T 1% /mnt/mds1 # mount -t lustre -o extents,mballoc,force_over_16tb /dev/large_vg/ost_lv /mnt/ost1 /dev/large_vg/ost_lv on /mnt/ost1 type lustre (rw,extents,mballoc,force_over_16tb) Filesystem Size Used Avail Use% Mounted on /dev/large_vg/ost_lv 128T 401M 122T 1% /mnt/ost1 # mount -t lustre -o user_xattr,acl,flock 192.168.77.1@o2ib:/largefs /mnt/lustre 192.168.77.1@o2ib:/largefs on /mnt/lustre type lustre (rw,user_xattr,acl,flock) Filesystem Size Used Avail Use% Mounted on 192.168.77.1@o2ib:/largefs 128T 401M 122T 1% /mnt/lustre # df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 9.1G 3.3G 5.4G 38% / /dev/hda1 99M 20M 75M 21% /boot tmpfs 16G 0 16G 0% /dev/shm /dev/large_vg/mdt_lv 1.8T 471M 1.7T 1% /mnt/mds1 /dev/large_vg/ost_lv 128T 401M 122T 1% /mnt/ost1 192.168.77.1@o2ib:/largefs 128T 401M 122T 1% /mnt/lustre # lfs df -h /mnt/lustre UUID bytes Used Available Use% Mounted on largefs-MDT0000_UUID 1.7T 470.1M 1.6T 0% /mnt/lustre[MDT:0] largefs-OST0000_UUID 128.0T 400.8M 121.6T 0% /mnt/lustre[OST:0] filesystem summary: 128.0T 400.8M 121.6T 0% /mnt/lustre # lfs df -i /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on largefs-MDT0000_UUID 469606378 25 469606353 0% /mnt/lustre[MDT:0] largefs-OST0000_UUID 134217728 56 134217672 0% /mnt/lustre[OST:0] filesystem summary: 469606378 25 469606353 0% /mnt/lustre lnet.debug=0x33f0404 lnet.subsystem_debug=0xffb7e3ff lnet.debug_mb=10 =========== dump the super block information on /dev/large_vg/ost_lv =========== # dumpe2fs -h /dev/large_vg/ost_lv dumpe2fs 1.41.90.wc1 (18-Mar-2011) Filesystem volume name: largefs-OST0000 Last mounted on: / Filesystem UUID: 5e2a898b-da07-48a4-93cf-1c3a3159b517 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr dir_index filetype needs_recovery extent 64bit flex_bg sparse_super huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 134217728 Block count: 34359738368 Reserved block count: 1717986918 Free blocks: 34348674877 Free inodes: 134217715 First block: 0 Block size: 4096 Fragment size: 4096 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 128 Inode blocks per group: 8 Flex block group size: 16 Filesystem created: Fri May 13 00:45:38 2011 Last mount time: Fri May 13 01:34:34 2011 Last write time: Fri May 13 01:34:34 2011 Mount count: 3 Maximum mount count: 20 Last checked: Fri May 13 00:58:32 2011 Check interval: 0 () Lifetime writes: 34 GB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 32b48ffa-1ae0-4da0-9cfd-6d0edbeece5a Journal backup: inode blocks Journal features: journal_incompat_revoke FEATURE_I1 Journal size: 400M Journal length: 102400 Journal sequence: 0x00000173 Journal start: 1 ======= run llverfs in partial mode on the Lustre filesystem /mnt/lustre ======= # time llverfs -vpl /mnt/lustre Timestamp: 1305275704 write File name: /mnt/lustre/dir00000/file000 write File name: /mnt/lustre/dir00509/file000 write File name: /mnt/lustre/dir01090/file000 write File name: /mnt/lustre/dir01663/file000 write File name: /mnt/lustre/dir02225/file000 write File name: /mnt/lustre/dir02787/file000 write File name: /mnt/lustre/dir03341/file000 write File name: /mnt/lustre/dir03900/file000 write File name: /mnt/lustre/dir04450/file000 write File name: /mnt/lustre/dir05003/file000 write File name: /mnt/lustre/dir05553/file000 write File name: /mnt/lustre/dir06103/file000 write File name: /mnt/lustre/dir06642/file000 write File name: /mnt/lustre/dir07180/file000 write File name: /mnt/lustre/dir07718/file000 write File name: /mnt/lustre/dir08243/file000 write File name: /mnt/lustre/dir08766/file000 write File name: /mnt/lustre/dir09296/file000 write File name: /mnt/lustre/dir09819/file000 write File name: /mnt/lustre/dir10339/file000 write File name: /mnt/lustre/dir10856/file000 write File name: /mnt/lustre/dir11365/file000 write File name: /mnt/lustre/dir11880/file000 write File name: /mnt/lustre/dir12387/file000 write File name: /mnt/lustre/dir12894/file000 write File name: /mnt/lustre/dir13412/file000 write File name: /mnt/lustre/dir13924/file000 write File name: /mnt/lustre/dir14440/file000 write File name: /mnt/lustre/dir14942/file000 write File name: /mnt/lustre/dir15448/file000 write File name: /mnt/lustre/dir15955/file000 write File name: /mnt/lustre/dir16459/file000 write File name: /mnt/lustre/dir16960/file000 write File name: /mnt/lustre/dir17467/file000 write File name: /mnt/lustre/dir17962/file000 write File name: /mnt/lustre/dir18462/file000 write File name: /mnt/lustre/dir18956/file000 write File name: /mnt/lustre/dir19459/file000 write File name: /mnt/lustre/dir19962/file000 write File name: /mnt/lustre/dir20462/file000 write File name: /mnt/lustre/dir20964/file000 write File name: /mnt/lustre/dir21464/file000 write File name: /mnt/lustre/dir21966/file000 write File name: /mnt/lustre/dir22465/file000 write File name: /mnt/lustre/dir22964/file000 write File name: /mnt/lustre/dir23464/file000 write File name: /mnt/lustre/dir23968/file000 write File name: /mnt/lustre/dir24462/file000 write File name: /mnt/lustre/dir24960/file000 write File name: /mnt/lustre/dir25458/file000 write File name: /mnt/lustre/dir25959/file000 write File name: /mnt/lustre/dir26448/file000 write File name: /mnt/lustre/dir26947/file000 write File name: /mnt/lustre/dir27438/file000 write File name: /mnt/lustre/dir27929/file000 write File name: /mnt/lustre/dir28423/file000 write File name: /mnt/lustre/dir28920/file000 write File name: /mnt/lustre/dir29410/file000 write File name: /mnt/lustre/dir29896/file000 write File name: /mnt/lustre/dir29998/file000 write complete read File name: /mnt/lustre/dir00217/file000 read File name: /mnt/lustre/dir00395/file000 read File name: /mnt/lustre/dir00510/file000 read File name: /mnt/lustre/dir00685/file000 read File name: /mnt/lustre/dir00939/file000 read File name: /mnt/lustre/dir01183/file000 read File name: /mnt/lustre/dir01407/file000 read File name: /mnt/lustre/dir01649/file000 read File name: /mnt/lustre/dir01908/file000 read File name: /mnt/lustre/dir02153/file000 read File name: /mnt/lustre/dir02394/file000 read File name: /mnt/lustre/dir02654/file000 read File name: /mnt/lustre/dir02879/file000 read File name: /mnt/lustre/dir03092/file000 read File name: /mnt/lustre/dir03330/file000 read File name: /mnt/lustre/dir03568/file000 read File name: /mnt/lustre/dir03823/file000 read File name: /mnt/lustre/dir04090/file000 read File name: /mnt/lustre/dir04330/file000 read File name: /mnt/lustre/dir04563/file000 read File name: /mnt/lustre/dir04814/file000 read File name: /mnt/lustre/dir05077/file000 read File name: /mnt/lustre/dir05332/file000 read File name: /mnt/lustre/dir05582/file000 read File name: /mnt/lustre/dir05845/file000 read File name: /mnt/lustre/dir06096/file000 read File name: /mnt/lustre/dir06366/file000 read File name: /mnt/lustre/dir06635/file000 read File name: /mnt/lustre/dir06893/file000 read File name: /mnt/lustre/dir07157/file000 read File name: /mnt/lustre/dir07423/file000 read File name: /mnt/lustre/dir07685/file000 read File name: /mnt/lustre/dir07943/file000 read File name: /mnt/lustre/dir08200/file000 read File name: /mnt/lustre/dir08457/file000 read File name: /mnt/lustre/dir08739/file000 read File name: /mnt/lustre/dir09006/file000 read File name: /mnt/lustre/dir09277/file000 read File name: /mnt/lustre/dir09562/file000 read File name: /mnt/lustre/dir09839/file000 read File name: /mnt/lustre/dir10102/file000 read File name: /mnt/lustre/dir10373/file000 read File name: /mnt/lustre/dir10646/file000 read File name: /mnt/lustre/dir10895/file000 read File name: /mnt/lustre/dir11174/file000 read File name: /mnt/lustre/dir11421/file000 read File name: /mnt/lustre/dir11685/file000 read File name: /mnt/lustre/dir11957/file000 read File name: /mnt/lustre/dir12213/file000 read File name: /mnt/lustre/dir12479/file000 read File name: /mnt/lustre/dir12744/file000 read File name: /mnt/lustre/dir13000/file000 read File name: /mnt/lustre/dir13255/file000 read File name: /mnt/lustre/dir13525/file000 read File name: /mnt/lustre/dir13782/file000 read File name: /mnt/lustre/dir14051/file000 read File name: /mnt/lustre/dir14319/file000 read File name: /mnt/lustre/dir14573/file000 read File name: /mnt/lustre/dir14823/file000 read File name: /mnt/lustre/dir15086/file000 read File name: /mnt/lustre/dir15330/file000 read File name: /mnt/lustre/dir15561/file000 read File name: /mnt/lustre/dir15825/file000 read File name: /mnt/lustre/dir16086/file000 read File name: /mnt/lustre/dir16344/file000 read File name: /mnt/lustre/dir16612/file000 read File name: /mnt/lustre/dir16892/file000 read File name: /mnt/lustre/dir17160/file000 read File name: /mnt/lustre/dir17425/file000 read File name: /mnt/lustre/dir17690/file000 read File name: /mnt/lustre/dir17958/file000 read File name: /mnt/lustre/dir18237/file000 read File name: /mnt/lustre/dir18500/file000 read File name: /mnt/lustre/dir18772/file000 read File name: /mnt/lustre/dir19053/file000 read File name: /mnt/lustre/dir19315/file000 read File name: /mnt/lustre/dir19596/file000 read File name: /mnt/lustre/dir19875/file000 read File name: /mnt/lustre/dir20142/file000 read File name: /mnt/lustre/dir20414/file000 read File name: /mnt/lustre/dir20681/file000 read File name: /mnt/lustre/dir20964/file000 read File name: /mnt/lustre/dir21233/file000 read File name: /mnt/lustre/dir21488/file000 read File name: /mnt/lustre/dir21755/file000 read File name: /mnt/lustre/dir22021/file000 read File name: /mnt/lustre/dir22293/file000 read File name: /mnt/lustre/dir22555/file000 read File name: /mnt/lustre/dir22815/file000 read File name: /mnt/lustre/dir23112/file000 read File name: /mnt/lustre/dir23401/file000 read File name: /mnt/lustre/dir23687/file000 read File name: /mnt/lustre/dir23972/file000 read File name: /mnt/lustre/dir24236/file000 read File name: /mnt/lustre/dir24507/file000 read File name: /mnt/lustre/dir24773/file000 read File name: /mnt/lustre/dir25048/file000 read File name: /mnt/lustre/dir25321/file000 read File name: /mnt/lustre/dir25587/file000 read File name: /mnt/lustre/dir25867/file000 read File name: /mnt/lustre/dir26135/file000 read File name: /mnt/lustre/dir26411/file000 read File name: /mnt/lustre/dir26684/file000 read File name: /mnt/lustre/dir26957/file000 read File name: /mnt/lustre/dir27245/file000 read File name: /mnt/lustre/dir27517/file000 read File name: /mnt/lustre/dir27789/file000 read File name: /mnt/lustre/dir28050/file000 read File name: /mnt/lustre/dir28320/file000 read File name: /mnt/lustre/dir28583/file000 read File name: /mnt/lustre/dir28849/file000 read File name: /mnt/lustre/dir29125/file000 read File name: /mnt/lustre/dir29391/file000 read File name: /mnt/lustre/dir29665/file000 read File name: /mnt/lustre/dir29938/file000 read File name: /mnt/lustre/dir29998/file000 read complete real 14m26.419s user 0m1.484s sys 7m27.234s # df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00 9.1G 3.3G 5.4G 38% / /dev/hda1 99M 20M 75M 21% /boot tmpfs 16G 0 16G 0% /dev/shm /dev/large_vg/mdt_lv 1.8T 588M 1.7T 1% /mnt/mds1 /dev/large_vg/ost_lv 128T 30G 122T 1% /mnt/ost1 192.168.77.1@o2ib:/largefs 128T 30G 122T 1% /mnt/lustre # lfs df -h /mnt/lustre UUID bytes Used Available Use% Mounted on largefs-MDT0000_UUID 1.7T 587.9M 1.6T 0% /mnt/lustre[MDT:0] largefs-OST0000_UUID 128.0T 29.7G 121.5T 0% /mnt/lustre[OST:0] filesystem summary: 128.0T 29.7G 121.5T 0% /mnt/lustre # lfs df -i /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on largefs-MDT0000_UUID 469636214 60024 469576190 0% /mnt/lustre[MDT:0] largefs-OST0000_UUID 134217728 30072 134187656 0% /mnt/lustre[OST:0] filesystem summary: 469636214 60024 469576190 0% /mnt/lustre ======================== cleanup the Lustre filesystem ======================== # umount /mnt/lustre # umount /mnt/mds1 # umount /mnt/ost1 ================== run e2fsck on device /dev/large_vg/ost_lv ================== # time e2fsck -fy -tt /dev/large_vg/ost_lv e2fsck 1.41.90.wc1 (18-Mar-2011) Pass 1: Checking inodes, blocks, and sizes Pass 1: Memory used: 276k/164368k (88k/189k), time: 209.41/ 2.93/ 3.82 Pass 1: I/O read: 125MB, write: 1MB, rate: 0.60MB/s Pass 2: Checking directory structure Pass 2: Memory used: 276k/201728k (65k/212k), time: 0.07/ 0.04/ 0.02 Pass 2: I/O read: 2MB, write: 0MB, rate: 28.45MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 16660k/201728k (16449k/212k), time: 215.09/ 5.62/ 6.69 Pass 3A: Memory used: 16660k/201728k (16449k/212k), time: 0.00/ 0.00/ 0.00 Pass 3A: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 3: Memory used: 16660k/201728k (64k/16597k), time: 0.01/ 0.00/ 0.01 Pass 3: I/O read: 1MB, write: 0MB, rate: 113.26MB/s Pass 4: Checking reference counts Pass 4: Memory used: 16660k/110620k (64k/16597k), time: 2.41/ 2.41/ 0.00 Pass 4: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 5: Checking group summary information Pass 5: Memory used: 24816k/77844k (16448k/8369k), time: 492.69/55.35/ 4.35 Pass 5: I/O read: 261MB, write: 0MB, rate: 0.53MB/s largefs-OST0000: 30072/134217728 files (18.7% non-contiguous), 18743454/34359738368 blocks Memory used: 24816k/77844k (16448k/8369k), time: 710.55/63.70/11.04 I/O read: 452MB, write: 1MB, rate: 0.64MB/s real 11m51.076s user 1m3.763s sys 0m11.424s ======================== cleanup the Lustre filesystem ======================== ============================= test duration: 8304s ============================= ================ test stopped at: Fri May 13 02:01:38 PDT 2011 ================