[LU-1805] Striping question

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.1.3
• Type: Task
• Priority: Minor
• Reporter: George Jackson (Inactive)
• Assignee: Cliff White (Inactive)
• Resolution: Fixed
• Labels: striping

Environment

RHEL6 x86_64

• Epic: striping

Description

After setting up our Lustre fs and began writing data to it, we noticed data was not being striped across the OSTs, only going to one OST. The question is: should data be getting striped across all OSTs? If striping is optional, how do we determine if we should stripe or not? If we find we should be striping any data that written to the filesystem mount, how do we set that up? We've tried the lfs setstripe commands from the manual and what we learned at training but still no striping. Thanks!

Comments

[Peter Jones]

Cliff

Could you please help with this one

Thanks

Peter

[Cliff White (Inactive)]

The default for a new Lustre filesystem is stripe_count = 1. Each new file open will use a different OST, in a round robin fashion.
Example:

# mkdir default 
# touch default/foo default/bar default/baz
# lfs getstripe default
default
stripe_count:   1 stripe_size:    1048576 stripe_offset:  -1 
default/bar
lmm_stripe_count:   1
lmm_stripe_size:    1048576
lmm_layout_gen:     0
lmm_stripe_offset:  24
        obdidx           objid          objid            group
            24           95261        0x1741d                0

default/baz
lmm_stripe_count:   1
lmm_stripe_size:    1048576
lmm_layout_gen:     0
lmm_stripe_offset:  27
        obdidx           objid          objid            group
            27           95259        0x1741b                0

default/foo
lmm_stripe_count:   1
lmm_stripe_size:    1048576
lmm_layout_gen:     0
lmm_stripe_offset:  22
        obdidx           objid          objid            group
            22           95246        0x1740e                0

Notice the different obdidx values.
When you stripe a file you obtain two things:
1. A single file can be larger than one OST. A single striped file can be as big as your storage.
2. Each stripe adds bandwidth to IO operations.
For example, when a client reads a file that is striped across 4 servers (assuming the read is large)
4 parallel IO requests are created, and the 4 servers will respond in parallel. When to stripe and how big is stripe is entirely application-specific.
If you need extremely large files, or large IO bandwidth, striping will help provide this.
Striping policy is per-directory, create a new directory and use lfs setstripe.
Example:

# mkdir big small
# lfs getstripe big
big
stripe_count:   1 stripe_size:    1048576 stripe_offset:  -1
#  lfs setstripe -c 10 big
#  lfs getstripe big
big
stripe_count:   10 stripe_size:    1048576 stripe_offset:  -1
#  lfs setstripe -c 2 small
#  touch big/fat
#  touch small/skinny
#  lfs getstripe big/fat
big/fat
lmm_stripe_count:   10
lmm_stripe_size:    1048576
lmm_layout_gen:     0
lmm_stripe_offset:  11
        obdidx           objid          objid            group
            11           97021        0x17afd                0
            10           95330        0x17462                0
            14           95168        0x173c0                0
            15           95256        0x17418                0
            12           95253        0x17415                0
             6           95672        0x175b8                0
            16           95285        0x17435                0
            17           95277        0x1742d                0
            18           95246        0x1740e                0
            19           95258        0x1741a                0
#  lfs getstripe small/skinny
small/skinny
lmm_stripe_count:   2
lmm_stripe_size:    1048576
lmm_layout_gen:     0
lmm_stripe_offset:  21
        obdidx           objid          objid            group
            21           95233        0x17401                0
            20           95217        0x173f1                0

Hope this helps

[George Jackson (Inactive)]

Cliff

Thanks for this information as it is what we needed to know to move forward. We will review our data requirements and make striping decisions accordingly. If we run into any specific issues while running the commands to stripe we will report that in a separate ticket. You may close this ticket as resolved.

Thanks again for your help!
George

[Cliff White (Inactive)]

Great, glad we could help. Since striping can be set per-directory, it's usually pretty easy to test variations in striping.