Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-3036] Interop 1.8.9<->2.4 Failure on test suite sanityn test_23: atime doesn't update among nodes https://jira.whamcloud.com/browse/LU-3036 Lustre <p>This issue was created by maloo for sarah <sarah@whamcloud.com></p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/25418d26-948b-11e2-93c6-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/25418d26-948b-11e2-93c6-52540035b04c</a>.</p> <p>The sub-test test_23 failed with the following error:</p> <blockquote> <p>test_23 failed with 1</p></blockquote> <p>Info required for matching: sanityn 23</p> client: 1.8.9<br/> server: lustre-master build #1338 LU-3036

Interop 1.8.9<->2.4 Failure on test suite sanityn test_23: atime doesn't update among nodes

Bug Minor Resolved Fixed Emoly Liu Maloo Tue, 26 Mar 2013 21:06:04 +0000 Mon, 17 Jun 2013 02:14:43 +0000 Mon, 17 Jun 2013 02:14:43 +0000 Lustre 2.4.0 Lustre 2.4.0 Lustre 1.8.9 0 8 <p>I noticed that in the b2_1 the test is different, it sends the signal to multiop before reading the time with a comment that says:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> multiop_bg_pause $DIR1/f23 or20_c || <span class="code-keyword">return</span> 1 # with SOM and opencache enabled, we need to close a file and cancel # open lock to get atime propogated to MDS kill -USR1 $! </pre> </div></div> <p>I imagine it could be that b1_8 interop needs the same thing?</p> <p>Also somebody needs to check correctness of the updated test in 2.x code, it looks pretty strange now to me, what's the point of a pause if we stop it the very next thing?</p> <p>Emoly</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>I will try per Oleg's advice.</p> <p>This failure is easy to reproduce on my local VMs. I tried the method that Oleg mentioned but failed.</p> <p>I will investigate more.</p> <p>sanity test_203 on b1_8 is another atime test, whose script is similar to sanityN test_23. But I found there was some lustre version judgement there, introduced by bz=23766 for 1.8.6<->2.1.0 interop (<a href="https://projectlava.xyratex.com/show_bug.cgi?id=23766#c56" class="external-link" target="_blank" rel="nofollow noopener">https://projectlava.xyratex.com/show_bug.cgi?id=23766#c56</a>).</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> test_203() { + local lustre_version=$(get_lustre_version mds) + <span class="code-keyword">if</span> [[ $lustre_version != 1.8* ]]; then + skip bug23766 mds running $lustre_version + <span class="code-keyword">return</span> + fi + local ATIME=`do_facet mds lctl get_param -n mds.*.atime_diff` echo <span class="code-quote">"atime should be updated on the MDS when closing file"</span> > $DIR/$tfile sync </pre> </div></div> <p>Why we added this ? If I remove this judgement, apparently test_203 will fail for b1_8<->master interop test, as same as test_23 does. Is there anything blocking interop atime update since then?</p> <p>BTW, I know the path of parameter "atime_diff" is a little different between b1_8 and others, but I don't think that will be a reason to skip this test, we can use md*.*.atime_diff instead.</p> <p>I forgot to say in my last two comments that the problem of "no such process" really can be fixed by Oleg's suggestion, but atime update problem still exists in b18<->master interop. The maloo test report is at <a href="https://maloo.whamcloud.com/test_sessions/a1e209cc-b872-11e2-891d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/a1e209cc-b872-11e2-891d-52540035b04c</a></p> <p>On master, we update atime on close if (la_valid == LA_ATIME),</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">int</span> mdt_mfd_close(struct mdt_thread_info *info, struct mdt_file_data *mfd) { ... <span class="code-comment">/* Update atime on close only. */</span> <span class="code-keyword">if</span> ((mode & MDS_FMODE_EXEC || mode & FMODE_READ || mode & FMODE_WRITE) && (ma->ma_valid & MA_INODE) && (ma->ma_attr.la_valid & LA_ATIME)) { <span class="code-comment">/* Set the atime only. */</span> ma->ma_valid = MA_INODE; ma->ma_attr.la_valid = LA_ATIME; rc = mo_attr_set(info->mti_env, next, ma); } ... } <span class="code-keyword">static</span> <span class="code-object">int</span> mdd_fix_attr(<span class="code-keyword">const</span> struct lu_env *env, struct mdd_object *obj, struct lu_attr *la, <span class="code-keyword">const</span> unsigned <span class="code-object">long</span> flags) { ... <span class="code-keyword">if</span> (la->la_valid == LA_ATIME) { <span class="code-comment">/* This is atime only set <span class="code-keyword">for</span> read atime update on close. */</span> <span class="code-keyword">if</span> (la->la_atime >= tmp_la->la_atime && la->la_atime < (tmp_la->la_atime + mdd_obj2mdd_dev(obj)->mdd_atime_diff)) la->la_valid &= ~LA_ATIME; RETURN(0); } ... } </pre> </div></div> <p>but my debug message showed that "valid" from b18 client wasn't mapped to LA_ATIME finally. I will investigate more.</p> <p>After looking at the code between master and b1_8, I found they used different attribute flags to trigger atime update.</p> <p>MASTER: ll_close_inode_openhandle()->ll_prepare_close(): ATTR_MODE | <font color="red">ATTR_ATIME_SET</font> | ATTR_MTIME_SET | ATTR_CTIME_SET;</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>---------------------------------------------------------------- client | server ---------------------------------------------------------------- ATTR_* -----> MDS_ATTR* ------> ATTR_* ------> LA_* attr_pack() attr_unpack() mdt_attr_valid_xlate() ---> if (in & ATTR_ATIME_SET) ---------------------------------------------------------------- out |= LA_ATIME; </pre> </div></div> <p>B1_8: ll_close_inode_openhandle(): OBD_MD_FLTYPE | OBD_MD_FLMODE | <font color="red">OBD_MD_FLATIME</font> | OBD_MD_FLMTIME | OBD_MD_FLCTIME;</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>-------------------------------------------------- client | server -------------------------------------------------- OBD_MD_FL* -----> MDS_ATTR* ------> ATTR_* mdc_close_pack_20() OBD_MD_FL* mdc_close_pack_18() mds_mfd_close() -----> if ((request_body->valid & OBD_MD_FLATIME) && -------------------------------------------------- ((request_body->atime > LTIME_S(inode->i_atime) + mds->mds_atime_diff) )) { LTIME_S(iattr.ia_atime) = request_body->atime; iattr.ia_valid |= ATTR_ATIME; } </pre> </div></div> <p>I think we don't need to force 2.x to understand the flags from b18 client, so I decide to add lustre version check to skip this test for interop test.</p> <p>I will push a patch later.</p> <p>patch for b18 is at <a href="http://review.whamcloud.com/#change,6289" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,6289</a></p> <p>Sorry, I don't understand your comment. The 1.8 client has support for the 2.x protocol in <tt>mdc_close_pack_20()</tt>, so it should be sending the same MDS_ATTR_* flags as the 2.x client. The <tt>mdc_close_pack_18()</tt> codepath is only used when the client is communicating with a 1.8 MDS.</p> <p>So, I don't think we should be so quick to just skip this test. If the 1.8 client sends the same MDS_ATTR_* flags as the 2.x client, then the 2.x server should treat the atime update in the same way.</p> <p>Probably the code path in my last comment misled you.</p> <p>The flags translation between 1.8 and 2.x has no problem. The problem is that they use different flags to trigger atime update.</p> <ul class="alternate" type="square"> <li>2.x uses ATTR_ATIME_SET. From the code we can see, 2.x updates atime on close only if (la->la_valid == LA_ATIME), and only ATTR_ATIME_SET can be translated into LA_ATIME.</li> <li>but 1.8 uses ATTR_ATIME.</li> </ul> <p>The protocol is <em>supposed</em> to be that <tt>ATTR_ATIME</tt> means the atime should be updated, but the server generates the timestamp if <tt>ATTR_ATIME_SET</tt> is not also set. See e.g. <tt>mds_fix_attr()</tt> in 1.8 updating xtime locally if <tt>ATTR_xTIME_SET</tt> is missing, and <tt>ll_setattr_raw()</tt> on the client setting <tt>ATTR_xTIME_SET</tt> if it is actually sending the atime value along in the RPC.</p> <p>Looking at the code more closely, I see that there some more improvements that should be made. The conversion in <tt>mdt_setatt_unpack_rec()</tt> from <tt>MDS_ATTR_<b></tt> to <tt>ATTR_</b></tt> to <tt>LA_<b></tt> doesn't make sense. <tt>mdt_attr_valid_xlate()</tt> should be changed to convert from <tt>MDS_ATTR_</b></tt> directly to <tt>LA_*</tt> and <tt>attr_unpack()</tt> should just be removed (it is not used anywhere else).</p> <p>There are several patches that are needed to fix this issue properly (in order of decreasing priority):</p> <ul class="alternate" type="square"> <li>2.1/2.4 MDS should check in <tt>mdt_setattr_unpack_rec()</tt> if <tt>MDS_ATTR_xTIME</tt> is set without <tt>MDS_ATTR_xTIME_SET</tt>, and set <tt>MDS_ATTR_xTIME_SET</tt> if the client does not have <tt>OBD_CONNECT_FULL20</tt> (i.e. 1.8 client), before calling <tt>mdt_attr_valid_xlate()</tt>. This will fix old 1.8 clients with new 2.x servers.</li> <li>2.1/2.4 clients should set both <tt>MDS_ATTR_xTIME | MDS_ATTR_xTIME_SET</tt> for timestamps in <tt>ll_prepare_close()</tt>. This is for protocol correctness and allows us to fix the server-side timestamp setting in the future.</li> <li>2.1/2.4 clients should add sanity.sh test_203() if this is checking something different than test_23(). This improves test coverage.</li> <li>2.4 MDS should remove <tt>attr_unpack()</tt></li> <li>1.8 clients should set both <tt>MDS_ATTR_xTIME | MDS_ATTR_xTIME_SET</tt> when converting from <tt>OBD_MD_FLATIME</tt> in <tt>mdc_close_pack_20()</tt>. This will fix new 1.8 clients with old 2.x servers.</li> <li>1.8 sanity.sh test_203() should remove the "skip" check, since it <em>should</em> be working properly with 2.x MDS. This ensures interop is working again.</li> </ul> <p>Thanks, Andreas! I will prepare for these patches.</p> <p>BTW, do we keep supporting 1.8 client for 2.5 and later?</p> <p>No, interoperability with 1.8 clients will end with Lustre 2.4, and 2.5 will not need to interoperate with these older clients anymore. </p> <p>patch for 2.4 is at <a href="http://review.whamcloud.com/6327" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6327</a>. It includes the following fixes</p> <blockquote> <ul class="alternate" type="square"> <li>2.1/2.4 MDS should check in mdt_setattr_unpack_rec() if MDS_ATTR_xTIME is set without MDS_ATTR_xTIME_SET, and set MDS_ATTR_xTIME_SET if the client does not have OBD_CONNECT_FULL20 (i.e. 1.8 client), before calling mdt_attr_valid_xlate(). This will fix old 1.8 clients with new 2.x servers.</li> <li>2.1/2.4 clients should set both MDS_ATTR_xTIME | MDS_ATTR_xTIME_SET for timestamps in ll_prepare_close(). This is for protocol correctness and allows us to fix the server-side timestamp setting in the future.</li> <li>2.1/2.4 clients should add sanity.sh test_203() if this is checking something different than test_23(). This improves test coverage.</li> <li>2.4 MDS should remove attr_unpack()</li> </ul> </blockquote> <p>BTW, sanityn.sh test_23 does the same check to sanity.sh test_203, so I didn't add test_203 but improve test_23 instead.</p> <p>Patch landed for 2.4 and b18</p> Development Rank 1|hzvmc7: Rank (Obsolete) 7407 Severity