https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1810 Lustre <p>After running &lt;lfs setstripe -d /data&gt; we ran the below command to start striping /data</p> <p>lfs setstripe -s 0 -c 7 --offset=4</p> <p>When we execute &lt;lfs getstripe -v /data&gt; it shows:</p> <p>/data stripe count: 7 stripe_size: 1048576 stripe_offset: -1</p> <p>Why is it that when we execute &lt;watch lfs df&gt; it still shows only ost0 being written to? We have stopped all services that write to this directory and umount /data but after remounting /data and restarting the services to continue writing it still shows only writing to ost0. To us it seems that the default striping is still going on even though we removed it before running the &lt;lfs getstripe&gt;.</p> rhel6 LU-1810

Striping of mount point

Task Minor Resolved Fixed Cliff White Douglas Allen Cain striping Fri, 31 Aug 2012 13:19:33 +0000 Fri, 21 Sep 2012 09:17:04 +0000 Fri, 21 Sep 2012 09:16:40 +0000 Lustre 2.1.3 0 1 <p>Cliff</p> <p>Could you please help with this one?</p> <p>Thanks</p> <p>Peter</p> <p>How large are the files you are creating? <br/> If you run lfs getstripe &lt;filename&gt; what do you get? </p> <p>Remember, a change in striping policy only affects files created after the change. <br/> Existing files are not re-striped.</p> <p>Cliff,</p> <p>I remember reading that but what I don't understand is when we upgraded to<br/> v2.2.0, I mounted the file system and then ran setstripe. Once I started<br/> the service all osts started striping. I did the samething with v2.1.3 but<br/> only one ost is striping. So in order to get striping to work, will I need<br/> to remove all files under /data run lfs setstripe -d /data and then run the<br/> new setstripe?</p> <p>To answer your question, "How large are the files you are creating?" After<br/> running lfs getstripe, we receive:<br/> lmm_stripe_count: 1<br/> lmm_stripe_size: 1048576<br/> lmm_stripe_offset: 0<br/> Obdidx objid objid group<br/> 0 497369 0x796d9 0</p> <p>V/r,<br/> Douglas Cain<br/> Cyber Systems Administrator, CENTAUR Operations<br/> Northrop Grumman Information Systems<br/> DISA PEO-MA<br/> COM 850-452-3560<br/> DSN 312-922-3560</p> <p>Cliff,</p> <p>Actually, in reviewing the data being sent to the filesystem, the sizes written are anywhere from 250K to 10M in size.</p> <p>Thanks,<br/> George</p> <p>As you can see from the lfs getstripe,(lmm_stripe_count: 1) the file you examined was created with a stripe count of 1. <br/> Was this file created after the change in striping policy or before? <br/> Please create a new file in the directory with the new striping policy and run lfs getstripe on that file </p> <p>Striping policy changes only apply to files created after the striping policy was set, as file striping is set at file creation time.<br/> If you wish to re-stripe an existing file, simply cp it to a new filename - it will be restriped<br/> so, # cp myfile tmpname; cp tmpname myfile<br/> will re-stripe any file. </p> <p>If you run lfs getstripe on the directory, what is the result?</p> <p>Also, lfs df is not especially useful for instantaneous measurement of performance.<br/> If you want detailed results, you can look at rpc_stats on the client or brw_stats on a server.<br/> on a client, for example for OST0001:</p> <ol> <li>lctl get_param osc.lustre-OST0001*.rpc_stats<br/> If you wish to check IO on an ost, from a command prompt on the OSS, run</li> <li>lctl get_param obdfilter.*.brw_stats</li> </ol> <p>That provides a much more accurate picture. However, in this case, i think the issue is with the stripe settings for the directory,<br/> lfs getstripe of the directory should show you the issue.</p> <p>Cliff,</p> <p>Even with creating a new file after we run lfs setstripe on a newly created dir, /data/big, we're still unable to stripe. The lfs getstripe reports</p> <p>/data/big<br/> stripe_count: 7 stripe_size: 1048576 stripe_offset: 4<br/> .<br/> .<br/> .<br/> lmm_stripe_count: 1<br/> lmm_stripe_size: 1048576<br/> lmm_stripe_pattern: 1<br/> lmm_stripe_offset 0<br/> .<br/> .</p> <p>The newly created file shows an lmm_stripe_count of 1 as well. Could it be because /data is the mount point of the filesystem and needs to be reformatted to include a stripe count of 7? Also, at what size does the file start striping? In other words, we ran a 'dd if=/dev/zero of=/data/big/test1.file bs=5000000' but even after 2 GB we still didn't see striping occurring.</p> <p>If there is any other info you need please let us know. Thanks, George</p> <p>It looks like you have set a fixed stripe_offset on /data/big. That's is likely causing your problem<br/> You should not have stripe_offset = 4. Setting stripe_offset forces you to use<br/> a specific OST. Make sure stripe_offset = -1 (lfs setstripe -i -1 ) <br/> Please do this:<br/> lfs setstripe -i -1 /data/big<br/> cd /data/big<br/> touch foo<br/> lfs getstripe foo</p> <p>Attache the <em>complete</em> output of lfs getstripe to the bug.<br/> You do not have to reformat. Stripe policy is per-directory, regardless of the policy at the mount point.</p> <p>You should really leave rest of the defaults alone, and only set stripe_count (-c) Also, there is no need to erase striping (-d option)<br/> Just set the new policy.</p> <p>Cliff,</p> <p>Ran the below commands:</p> <p>rm -rf /data/big<br/> mkdir /data/big<br/> lfs getstripe /data/big - saw that stripe_count: 1 stripe_size:<br/> 1048576 stripe_offset: -1 (that's fine because we did not set anything yet)</p> <p>Then ran:<br/> lfs setstripe -c 7 /data/big - we have 7 osts<br/> touch test<br/> lfs getstripe /data/big <br/> /data/big<br/> stripe_count: 7 stripe_size: 1048576 stripe_offset: -1<br/> /data/big/test<br/> lmm_stripe_count: 1<br/> lmm_stripe_size: 1048576<br/> lmm_stripe_offset:0<br/> Obdidx objid objid group<br/> 0 671056 0xa3d50 0</p> <p>V/r,<br/> Douglas Cain<br/> Cyber Systems Administrator, CENTAUR Operations<br/> Northrop Grumman Information Systems<br/> DISA PEO-MA<br/> COM 850-452-3560<br/> DSN 312-922-3560</p> <p>That's quite bizarre. <br/> If i do the same thing here on a sample system, i see:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># mkdir foo # lfs getstripe foo foo stripe_count: 1 stripe_size: 1048576 stripe_offset: -1 # lfs setstripe -c 7 foo # lfs getstripe foo foo stripe_count: 7 stripe_size: 1048576 stripe_offset: -1 # cd foo # touch bar # lfs getstripe bar bar lmm_stripe_count: 7 lmm_stripe_size: 1048576 lmm_layout_gen: 0 lmm_stripe_offset: 14 obdidx objid objid group 14 238461 0x3a37d 0 12 238461 0x3a37d 0 15 238429 0x3a35d 0 5 238366 0x3a31e 0 18 238494 0x3a39e 0 19 238366 0x3a31e 0 17 238367 0x3a31f 0 </pre> </div></div> <p>Do you have some mount options set? Did you set any mkfsoptions when creating the filesystem? <br/> First, run the command 'script lu-1829' - this will create a script file<br/> Then, repeat the striping/file creation as above.<br/> After you are done, exit script with ctrl-D. Attach the output file to this bug. <br/> I need to see exactly what you are doing. </p> <p>Did you actually create the 'test' file in the data/big directory, or did you 'mv' it there? <br/> I can replicate your results if i do this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # touch bob # lfs getstripe bob bob lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_layout_gen: 0 lmm_stripe_offset: 18 obdidx objid objid group 18 238434 0x3a362 0 # mv bob foo # lfs getstripe foo/bob foo/bob lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_layout_gen: 0 lmm_stripe_offset: 18 obdidx objid objid group 18 238434 0x3a362 0 </pre> </div></div> <p>a 'mv' does not restripe the file. If I wish to restripe an existing file, this works:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># touch baz # lfs getstripe baz baz lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_layout_gen: 0 lmm_stripe_offset: 33 obdidx objid objid group 33 239920 0x3a930 0 # cp baz foo/baz # lfs getstripe foo/baz foo/baz lmm_stripe_count: 7 lmm_stripe_size: 1048576 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 240095 0x3a9df 0 7 240031 0x3a99f 0 10 240096 0x3a9e0 0 11 239743 0x3a87f 0 9 239746 0x3a882 0 8 239488 0x3a780 0 14 240160 0x3aa20 0 </pre> </div></div> <p>Cliff,</p> <p>We are not getting the same results and to answer your question "Did you<br/> actually create the 'test' file in the data/big directory, or did you 'mv'<br/> it there?" I created it in the /data/big directory with the following<br/> commands:</p> <p>rm -rf /data/big<br/> mkdir /data/big<br/> lfs getstripe /data/big - saw that stripe_count: 1 stripe_size:<br/> 1048576 stripe_offset: -1 (that's fine because we did not set anything yet)</p> <p>Then ran:<br/> lfs setstripe -c 7 /data/big - we have 7 osts <br/> touch /data/big/test <br/> lfs getstripe /data/big <br/> /data/big<br/> stripe_count: 7 stripe_size: 1048576 stripe_offset: -1<br/> /data/big/test<br/> lmm_stripe_count: 1<br/> lmm_stripe_size: 1048576<br/> lmm_stripe_offset:0<br/> Obdidx objid objid group<br/> 0 671056 0xa3d50 0</p> <p>We are running version 2.1.3 on rhel 6. We formatted the ost file system<br/> with:<br/> Mkfs.lustre --ost --fsname=lustre01<br/> --failnode=ip_address@o2ib:ip_address@tcp0<br/> --mgsnode=ip_address@o2ib:ip_address@tcp0 /dev/mapper/mpathXX</p> <p>Thanks,<br/> Douglas</p> <p>another possibility from our testing - are you certain all the OSTs are on line and up? <br/> If your OSTs are off-line, the files will only put stripes on functioning OSTs.</p> <p>I would take a step back at this point. Is the filesystem otherwise healthy? Any other client errors?</p> <p>1) Verify that all your OSTs are online and accessible to the Lustre MGS</p> <ul class="alternate" type="square"> <li>check mounts on the OSS nodes.</li> <li>check syslogs on all server nodes. Report any LustreError messages.</li> </ul> <p>2) Please attach the contents of 'tune2fs -print &lt;your device&gt;' for both the MGS and MDS (if MGS is separate) where &lt;your device&gt; is replaced by the actual /dev/ path. </p> <p>3) If stripe_count is == 1, each new file create show go to a different OST. to verify this, try something like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">mkdir st1 lfs setstipe -c 1 st1 <span class="code-keyword">for</span> i in a b c d e f g;<span class="code-keyword">do</span> touch st1/$i; done </pre> </div></div> <p>If you then run 'lfs getstripe st1' each file should have a different obdidx value. (and different lmm_stripe_offset) Please verify this on your system. </p> <p>Yes all osts are on line.</p> <p>Cliff,</p> <p>I have verified that all mount points are mounted on our stand alone mgs and<br/> two mds'.<br/> I have verified that all osts nine in total are up and have all mount points<br/> mounted. Seven ost for one mount point on mdt01 and two for one mount point<br/> mdt02.</p> <p>After logging into the mds and executing lctl dl, I confirmed that the mds<br/> is talking to the mgs by seeing UP for mgc and UP for all listed osc, lov,<br/> mdt, and mds.</p> <p>After running your for I in I received the same obdidx value = 0 same<br/> lmm_stripe_offset = 0</p> <p>After running tune2fs on both mgs and mds it returned:<br/> Setting multiple mount protection update interval to 5 seconds</p> <p>Thank you,<br/> Douglas</p> <p>Douglas, I asked you specifically for the <em>output</em> of 'tunefs.lustre --print &lt;device&gt;' run on all your MGS,MDS and OST devices. Please attach that to the bug.</p> <p>Cliff,</p> <p>I am waiting on my boss so I can get permission due to the area that we work<br/> in.</p> <p>V/r,<br/> Douglas</p> <p>Cliff,</p> <p>Still awaiting permission to send you the output file but I can tell you<br/> that on all osts, mds, mgs we receive:<br/> tune2fs 1.41.90.wc4 (01-Sep-2011)<br/> Setting multiple mount protection update internal to 5 seconds</p> <p>I have to unmount all mount points if I do not unmount than I receive:<br/> tune2fs 1.41.90.wc4 (01-Sep-2011)<br/> ^[tune2fs: MMP: device currently active while trying to open<br/> /dev/mapper/mpathX - where X is the path.</p> <p>V/r,<br/> Douglas </p> <p>You seem to be misreading my instructions. The command i asked you to run is "tunefs.lustre --print &lt;device&gt;" - the --print option is crucial. Please attach the full, complete output to this bug.</p> <p>And, do not umount the devices. This can happen on a live system.</p> <p>Cliff,</p> <p>In this email you asked me in step 2 to run tune2fs. I will run tunefs now.</p> <p>V/r,<br/> Douglas</p> <p>Cliff,</p> <p>I updated the ticket through the website but I have been advised not to send<br/> the output of the server itself due to it being in a classified area. I<br/> have provided an attachment where I typed the results from tunefs.</p> <p>V/r,<br/> Douglas</p> <p>Thank you for explaining this is a classified site. That explains a few things. What you are attempting is a very, very basic part of Lustre. It's worked for a long time for a lot of people. <br/> The code in that area is mature, and hasn't changed in quite awhile. It's routinely tested. </p> <p>I see absolutely nothing unique or unusual in your setup, based on the data you have provided me. In the absence of error messages from a Lustre client or server, or a full script capture of <br/> your actions, i can see no reason why you are having this problem. <br/> Things I can suggest:</p> <ul class="alternate" type="square"> <li>You appear to have two filesystems, lustre01 with 7 OSTs and lustre02 with 2 OSTs. Do you have clients mounting one or both filesystems? try your striping tests on both, if possible. Compare results.</li> <li>Check everywhere for errors. Check every system console, every system error log (/var/log/messages, normally). Monitor the logs when you do tests.</li> <li>As I've already mentioned, try creating many files with stripe_count = 1 and verify they are allocated on different OSTs</li> <li>Try different values for stripe_count - try stripe_count = -1 (all)</li> <li>Try using stripe_index to force the objects onto specific OSTs.</li> </ul> <p>The important thing is that 'lfs getstripe' should return a list of objects, when you create a striped file. Focus on getting that to work. </p> <p>Cliff,</p> <p>You asked if we were receiving any errors. On one of our clients we are<br/> receiving this error:</p> <p>(file.c:2196:ll_inode_revalidate_fini()) failure -13</p> <p>Could you please shed some light on this?</p> <p>V/r,<br/> Douglas</p> <p>The error is EACCES 13 /* Permission denied */ I would need more context from the system log to be able to say more. It's unlikely to have anything to do with the striping issue.</p> <p>Cliff,</p> <p>Also, on the mds side it shows, an error while communicating with<br/> ip_address@tcp. The ost_connect operation failed with -19. We know that -19<br/> means ENODEV no such device is available. The server stopped or failed over.<br/> I checked and the server is up and running and hasn't failed over.</p> <p>Thanks,<br/> Douglas </p> <p>Again, i would need to see the actual error, and some context.</p> <p>Cliff, our striping issue is now resolved. Your comments on this were very helpful but we found some underlying issues with our initial configuration related to the indexes of the OSTs. We ended up reformatting all mgt, mdt, and ost using the correct indexing and are now able to stripe. Thanks again for your help, you may close this issue as resolved.</p> <p>Thanks for letting us know George!</p> Development Rank 1|hzw0yn: Rank (Obsolete) 10236