https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2871 Lustre <p>I found this problem during the investigation on <a href="https://jira.whamcloud.com/browse/LU-2809" title="fix ll_setxattr() to always ignore ll_stripe_offset" class="issue-link" data-issue-key="LU-2809"><del>LU-2809</del></a>. While running "lfs setstripe -c -1 -i n testfile", if ost index n doesn't start from 0, it shows that data can't be striped across all the OSTs and OST0 is always ignored. </p> LU-2871

Data can't be striped across all the OSTs correctly by running "lfs setstripe -c -1 -i n" (n>0)

Bug Blocker Closed Fixed Emoly Liu Emoly Liu MB Tue, 26 Feb 2013 20:38:03 +0000 Fri, 15 Mar 2013 01:53:12 +0000 Fri, 15 Mar 2013 01:52:55 +0000 Lustre 2.4.0 Lustre 2.4.0 0 10 <p>OSTCOUNT=4,</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[root@centos6-1 ~]# cd /mnt/lustre [root@centos6-1 lustre]# mkdir test;cd test [root@centos6-1 test]# <span class="code-keyword">for</span> i in 0 1 2 3; <span class="code-keyword">do</span> lfs setstripe -i $i -c -1 testfile$i;dd <span class="code-keyword">if</span>=/dev/zero of=testfile$i bs=2M count=5;done 5+0 records in 5+0 records out 10485760 bytes (10 MB) copied, 0.0220632 s, 475 MB/s 5+0 records in 5+0 records out 10485760 bytes (10 MB) copied, 0.0193019 s, 543 MB/s 5+0 records in 5+0 records out 10485760 bytes (10 MB) copied, 0.0200823 s, 522 MB/s 5+0 records in 5+0 records out 10485760 bytes (10 MB) copied, 0.0190184 s, 551 MB/s [root@centos6-1 test]# lfs getstripe * testfile0 lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1 0x1 0 1 1 0x1 0 2 1 0x1 0 3 1 0x1 0 testfile1 lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0 2 2 0x2 0 3 2 0x2 0 1 3 0x3 0 testfile2 lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 3 0x3 0 3 3 0x3 0 1 4 0x4 0 2 4 0x4 0 testfile3 lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_layout_gen: 0 lmm_stripe_offset: 3 obdidx objid objid group 3 4 0x4 0 1 5 0x5 0 2 5 0x5 0 3 5 0x5 0 </pre> </div></div> <p>I Added some debug messages, </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/lustre/lod/lod_dev.c b/lustre/lod/lod_dev.c index 17dca0c..851caee 100644 --- a/lustre/lod/lod_dev.c +++ b/lustre/lod/lod_dev.c @@ -68,7 +68,10 @@ <span class="code-object">int</span> lod_fld_lookup(<span class="code-keyword">const</span> struct lu_env *env, struct lod_device *lod, LASSERTF(fid_is_sane(fid), <span class="code-quote">"Invalid FID "</span>DFID<span class="code-quote">"\n"</span>, PFID(fid)); <span class="code-keyword">if</span> (fid_is_idif(fid)) { + printk(<span class="code-quote">"before ostindex:%d, FID "</span>DFID<span class="code-quote">"\n"</span>, + cpu_to_le32(*tgt), PFID(fid)); *tgt = fid_idif_ost_idx(fid); + printk(<span class="code-quote">"after ostindex:%d\n"</span>, cpu_to_le32(*tgt)); RETURN(rc); } diff --git a/lustre/osd-ldiskfs/osd_handler.c b/lustre/osd-ldiskfs/osd_handler.c index f6dad39..39decd5 100644 --- a/lustre/osd-ldiskfs/osd_handler.c +++ b/lustre/osd-ldiskfs/osd_handler.c @@ -2206,6 +2206,20 @@ <span class="code-keyword">static</span> inline <span class="code-object">int</span> __osd_xattr_set(struct osd_thread_info *info, ll_vfs_dq_init(inode); dentry-&gt;d_inode = inode; + <span class="code-keyword">if</span> (strcmp(name, XATTR_NAME_LOV) == 0) { + struct lov_mds_md_v1 *lmm = (struct lov_mds_md_v1 *)buf; + <span class="code-object">int</span> stripe_count = lmm-&gt;lmm_stripe_count; + struct lov_ost_data *objects = lmm-&gt;lmm_objects; + <span class="code-object">int</span> i; + + printk(<span class="code-quote">"stripecount=%d, stripesize=%d\n"</span>, + stripe_count, lmm-&gt;lmm_stripe_size); + <span class="code-keyword">for</span> (i = 0; i &lt; stripe_count; i++) { + <span class="code-object">int</span> idx = objects[i].l_ost_idx; + printk(<span class="code-quote">"here ostindex:%d\n"</span>, idx); + } + } <span class="code-keyword">return</span> inode-&gt;i_op-&gt;setxattr(dentry, name, buf, buflen, fl); } </pre> </div></div> <p>and dmesg showed</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400):0:mdt before ostindex:-30720, FID [0x100000000:0x1:0x0] after ostindex:0 before ostindex:0, FID [0x100010000:0x1:0x0] after ostindex:1 before ostindex:1, FID [0x100020000:0x1:0x0] after ostindex:2 before ostindex:2, FID [0x100030000:0x1:0x0] after ostindex:3 stripecount=4, stripesize=1048576 here ostindex:0 here ostindex:1 here ostindex:2 here ostindex:3 before ostindex:-30720, FID [0x100010000:0x2:0x0] after ostindex:1 before ostindex:1, FID [0x100020000:0x2:0x0] after ostindex:2 before ostindex:2, FID [0x100030000:0x2:0x0] after ostindex:3 before ostindex:3, FID [0x100010000:0x3:0x0] after ostindex:1 stripecount=4, stripesize=1048576 here ostindex:1 here ostindex:2 here ostindex:3 here ostindex:1 before ostindex:-30720, FID [0x100020000:0x3:0x0] after ostindex:2 before ostindex:2, FID [0x100030000:0x3:0x0] after ostindex:3 before ostindex:3, FID [0x100010000:0x4:0x0] after ostindex:1 before ostindex:1, FID [0x100020000:0x4:0x0] after ostindex:2 stripecount=4, stripesize=1048576 here ostindex:2 here ostindex:3 here ostindex:1 here ostindex:2 before ostindex:-30720, FID [0x100030000:0x4:0x0] after ostindex:3 before ostindex:3, FID [0x100010000:0x5:0x0] after ostindex:1 before ostindex:1, FID [0x100020000:0x5:0x0] after ostindex:2 before ostindex:2, FID [0x100030000:0x5:0x0] after ostindex:3 stripecount=4, stripesize=1048576 here ostindex:3 here ostindex:1 here ostindex:2 here ostindex:3 </pre> </div></div> <p>There is something wrong with fid sequence.</p> <p>I found the root cause.</p> <p>in lod_qos_ost_in_use_clear(), the ost_in_use array is initialised to 0, and in lod_qos_prep_create()-&gt;old_alloc_specific(), the ost_idx is</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">for</span> (i = 0; i &lt; ost_count; i++, array_idx = (array_idx + 1) % ost_count) { ost_idx = osts-&gt;op_array[array_idx]; </pre> </div></div> <p>and the ost_idx will be checked upon ost_in_use array</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (lod_qos_is_ost_used(env, ost_idx, stripe_num)) <span class="code-keyword">continue</span>; </pre> </div></div> <p>If the stripe_offset starts from 0, and in the 1st iteration, stripe_num is also 0, and lod_qos_is_ost_used() will return false, then object will be allocated on the first OST device.</p> <p>While if file stripe starting from a number other than 0, when the loop comes to which ost_idx is 0, the lod_qos_is_ost_used(env, 0, stripe_num) will return true, and the 1st OST device will be skipped.</p> <p>The fix should be in lod_qos_ost_in_use_clear(). With following patch, the object stripe allocation will be correct.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/lustre/lod/lod_qos.c b/lustre/lod/lod_qos.c index 2b81ad8..2f46e7c 100644 --- a/lustre/lod/lod_qos.c +++ b/lustre/lod/lod_qos.c @@ -629,7 +629,7 @@ <span class="code-keyword">static</span> inline <span class="code-object">int</span> lod_qos_ost_in_use_clear(<span class="code-keyword">const</span> struct lu_env *env, <span class="code-object">int</span> stripes CERROR(<span class="code-quote">"can't allocate memory <span class="code-keyword">for</span> ost-in-use array\n"</span>); <span class="code-keyword">return</span> -ENOMEM; } - memset(info-&gt;lti_ea_store, 0, sizeof(<span class="code-object">int</span>) * stripes); + memset(info-&gt;lti_ea_store, -1, sizeof(<span class="code-object">int</span>) * stripes); <span class="code-keyword">return</span> 0; } </pre> </div></div> <p>pretty much correct. please put a patch into gerrit, thanks.</p> <p>Another way, run lod_qos_ost_in_use() after lod_qos_is_ost_used() check, right?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/lustre/lod/lod_qos.c b/lustre/lod/lod_qos.c index 2b81ad8..92b3b36 100644 --- a/lustre/lod/lod_qos.c +++ b/lustre/lod/lod_qos.c @@ -887,6 +887,7 @@ repeat_find: */ <span class="code-keyword">if</span> (lod_qos_is_ost_used(env, ost_idx, stripe_num)) <span class="code-keyword">continue</span>; + lod_qos_ost_in_use(env, stripe_num, ost_idx); /* Drop slow OSCs <span class="code-keyword">if</span> we can, but not <span class="code-keyword">for</span> requested start idx. * </pre> </div></div> <p>I will add a sanity test for this case.</p> <p>Patch is at <a href="http://review.whamcloud.com/5554" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5554</a></p> <p>liuying, it's better to mark index used after successful lod_qos_declare_object_on(). and I don't think this is an option to the change by Zhenyu Xu, I think the both changes should be applied.</p> <p>Sure, I made the both changes in the patch and will update it per Ned Bass' advice later. Thanks!</p> <p>Landed for 2.4</p> Duplicate LU-2893 Related LU-2809 Development Rank 1|hzvjv3: Rank (Obsolete) 6940 Severity