https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9490 Lustre <p>Looking at the Lustre ADIO driver, it appears that it is retrieving the layout parameters (stripe count, stripe size) incorrectly from the kernel. It is calling an ioctl() directly to the kernel, but then not checking that the returned layout magic is something that it understands before dereferencing the fields of <tt>struct lov_mds_md</tt>. For composite files that for not have <tt>struct lov_mds_md</tt> directly at the start of the layout, this will deteference random fields in the layout and return garbage to the library. </p> LU-9490

MPI-IO Lustre ADIO driver gets Lustre layout parameters incorrectly

Bug Critical Resolved Fixed Emoly Liu Andreas Dilger Thu, 11 May 2017 14:54:17 +0000 Sun, 12 Nov 2017 21:08:57 +0000 Wed, 7 Jun 2017 20:47:11 +0000 Lustre 2.10.0 Lustre 2.10.0 0 12 <p>In particular: <a href="https://trac.mpich.org/projects/mpich/browser/src/mpi/romio/adio/ad_lustre/ad_lustre_open.c?rev=f818af4a15b0a168ecab7f73baf0491c2baed26b" class="external-link" target="_blank" rel="nofollow noopener">https://trac.mpich.org/projects/mpich/browser/src/mpi/romio/adio/ad_lustre/ad_lustre_open.c?rev=f818af4a15b0a168ecab7f73baf0491c2baed26b</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">void ADIOI_LUSTRE_Open(ADIO_File fd, <span class="code-object">int</span> *error_code) { : : <span class="code-comment">/* get file striping information and set it in info */</span> lum.lmm_magic = LOV_USER_MAGIC; err = ioctl(fd-&gt;fd_sys, LL_IOC_LOV_GETSTRIPE, (void *) &amp;lum); <span class="code-keyword">if</span> (!err) { sprintf(value, <span class="code-quote">"%d"</span>, lum.lmm_stripe_size); MPI_Info_set(fd-&gt;info, <span class="code-quote">"striping_unit"</span>, value); sprintf(value, <span class="code-quote">"%d"</span>, lum.lmm_stripe_count); MPI_Info_set(fd-&gt;info, <span class="code-quote">"striping_factor"</span>, value); sprintf(value, <span class="code-quote">"%d"</span>, lum.lmm_stripe_offset); MPI_Info_set(fd-&gt;info, <span class="code-quote">"start_iodevice"</span>, value); } </pre> </div></div> <p>Some options that I see to handle this:</p> <ul class="alternate" type="square"> <li>set LOV_USER_MAGIC = LOV_USER_MAGIC_COMP_V1 in lustreapi and then detect this in the kernel <tt>LL_IOC_LOV_GETSTRIPE</tt> handling, and return a fake LOV_USER_MAGIC_V1 xattr to the caller. I don't think this will be very robust.</li> <li>fix ADIO_Lustre to check the returned magic value and then interpret the returned composite xattr based on the magic</li> <li>fix ADIO_Lustre to use <tt>llapi_layout_get_by_path()</tt>, which creates a dependency on liblustreapi (not sure if this is an issue, is it always installed on Lustre clients?)</li> </ul> <p>It appears that we have similar problems in our own code, for example any program that calls <tt>llapi_file_get_stripe()</tt> will always return <tt>struct lov_user_md</tt> to the caller. For better or worse, this will result in <tt>lum-&gt;lmm_stripe_count</tt>, <tt>lum-&gt;lmm_stripe_size</tt>, and <tt>lum-&gt;lmm_stripe_offset</tt> being zero, since these fields overlap with <tt>lcm_padding2</tt>.</p> <p>Niu</p> <p>Could you please assist with this issue?</p> <p>Thanks</p> <p>Peter</p> <p>It looks like <tt>lfs.c::lfs_migrate()</tt> is calling <tt>llapi_file_get_stripe()</tt> to get the stripe size, and this should be changed to use <tt>llapi_layout_get_by_path()</tt> and the stripe count extracted from the returned layout. One option is to set the component pointer to be the last initialized component, so that attributes returned are what the user would expect if they don't understand composite files.</p> <p>Yep we hit this problem during our PFL test shot. Prevented us from running one of our apps.</p> <p>James, could you please include the output from the Cray MPI-IO test run, since it was referencing a function that does not exist in the upstream MPICH code.</p> <blockquote> <p>set LOV_USER_MAGIC = LOV_USER_MAGIC_COMP_V1 in lustreapi and then detect this in the kernel LL_IOC_LOV_GETSTRIPE handling, and return a fake LOV_USER_MAGIC_V1 xattr to the caller. I don't think this will be very robust.</p></blockquote> <p>Yes, I also don't think it will work well. The fake v1 xattr can't provide accurate stripe information for ADIO to leverage on, imaging FLR or other more complex layout introduced later, it'll be more messy with the fake v1. If some application cares about Lustre file layout, it has to understand the layout anyway.</p> <blockquote> <p>fix ADIO_Lustre to check the returned magic value and then interpret the returned composite xattr based on the magic<br/> fix ADIO_Lustre to use llapi_layout_get_by_path(), which creates a dependency on liblustreapi (not sure if this is an issue, is it always installed on Lustre clients?)</p></blockquote> <p>I vote for using llapi_layout_xxx interfaces, but I don't know how it should be fixed, should we inform the maintainer to update the ADIO driver?</p> <blockquote> <p>It appears that we have similar problems in our own code, for example any program that calls llapi_file_get_stripe() will always return struct lov_user_md to the caller. For better or worse, this will result in lum-&gt;lmm_stripe_count, lum-&gt;lmm_stripe_size, and lum-&gt;lmm_stripe_offset being zero, since these fields overlap with lcm_padding2.</p></blockquote> <p>I see only some places used llapi_file_get_stripe() (or call the ioctl) directly:</p> <ul class="alternate" type="square"> <li>'lfs find', 'lfs getstripe' calls the ioctl directly, but we've handled it properly.</li> <li>'lfs migrate' calls llapi_file_get_stripe() directly and uses result for v1/v3 only, not an error but can be improved.</li> <li>ll_dirstripe_verify.c calls llapi_file_get_stripe() directly, and it's used in sanity.sh, that'll cause test failures when root default layout is set as composite, I think it should be handled by <a href="https://jira.whamcloud.com/browse/LU-9324" title="sanity-pfl test 10 needs to reset the file system default layout" class="issue-link" data-issue-key="LU-9324"><del>LU-9324</del></a>. (or already being handled?)</li> </ul> <p>This is the error we saw during the test shot: </p> <p>Assertion failed in file /notbackedup/users/ulib/mpt_base/mpich2/src/mpi/romio/adio/ad_cray/ad_cray_adio_open.c at line 474: _mpiio_o_lov_delay_create != 0</p> <p>Niu, it looks like ll_dirstripe_verify.c is not handling non-v1/v3 layouts properly. It is probably just comparing "0 == 0" in this case. We should probably deprecate the use of llapi_file_get_stripe() via _<em>attribute</em>_((deprecated)) to start (compile time), then via a message printed out in 2.11.</p> <p>In the meantime, it probably makes sense to have it return something reasonable for PFL files when used, such as the plain layout of the last component (if file has no size) and the last init component if it has a size.</p> <p>Separately, Emoly, can you please work with Niu to create and push a patch for MPICH, as I believe you worked on this code originally:<br/> <a href="http://git.mpich.org/mpich.git/blob/HEAD:/src/mpi/romio/adio/ad_lustre/ad_lustre_open.c" class="external-link" target="_blank" rel="nofollow noopener">http://git.mpich.org/mpich.git/blob/HEAD:/src/mpi/romio/adio/ad_lustre/ad_lustre_open.c</a> It should really be using the llapi_layout_* interface. I don't know if applications pass in a hint for the file size, but if they do it would make it easy to decide which component to use for the returned layout parameters. </p> <p>However, for existing applications that call the ioctl directly, we may still need to return something reasonable from the kernel, since otherwise every MPI-IO application running on a new version of Lustre will fail if it hasn't been patched. One option is to detect the magic value passed to the kernel <tt>LL_IOC_LOV_GETSTRIPE</tt> handling in <tt>lov_getstripe()</tt>, and if LOV_MAGIC_V1/V3 return the last init component, and only return the full composite layout if the caller passes in LOV_MAGIC_COMP_V1? This would add complexity to the kernel, but would avoid compatibility issues with older applications, at the expense of returning <em>something</em> useful to userspace rather than causing outright application failures.</p> <blockquote> <p>Niu, it looks like ll_dirstripe_verify.c is not handling non-v1/v3 layouts properly. It is probably just comparing "0 == 0" in this case. We should probably deprecate the use of llapi_file_get_stripe() via <em>attribute</em>((deprecated)) to start (compile time), then via a message printed out in 2.11.<br/> In the meantime, it probably makes sense to have it return something reasonable for PFL files when used, such as the plain layout of the last component (if file has no size) and the last init component if it has a size.</p></blockquote> <p>Ok, sure. What do you mean "via a message printed out in 2.11."?</p> <blockquote> <p>However, for existing applications that call the ioctl directly, we may still need to return something reasonable from the kernel, since otherwise every MPI-IO application running on a new version of Lustre will fail if it hasn't been patched. One option is to detect the magic value passed to the kernel LL_IOC_LOV_GETSTRIPE handling in lov_getstripe(), and if LOV_MAGIC_V1/V3 return the last init component, and only return the full composite layout if the caller passes in LOV_MAGIC_COMP_V1? This would add complexity to the kernel, but would avoid compatibility issues with older applications, at the expense of returning something useful to userspace rather than causing outright application failures.</p></blockquote> <p>I'm wondering if it's better to return error instead of the last instantiated component?</p> <p>I suppose application usually calls getstripe on open/create file, so the first component will be returned in most cases, then optimizing based on first component's attributes will highly likely get opposite result. If we return an error instead of last instantiated component, app may either fallback to non-optimizing default setting (which is better than optimizing on first component), or error out (so that user will know it's time to upgrade).</p> <p>BTW, except ADIO, I didn't see other callers passing lmm magic to LL_IOC_LOV_GETSTRIPE ioctl, so this workaround only works for the apps which providing lmm magic.</p> <blockquote> <p>What do you mean "via a message printed out in 2.11."?</p></blockquote> <p>I mean that <tt>llapi_file_get_stripe()</tt> should start to print out a warning about calling this function in 2.11, and that they should change to using <tt>llapi_layout_get_by_path()</tt> instead.</p> <blockquote> <p>I'm wondering if it's better to return error instead of the last instantiated component? I suppose application usually calls getstripe on open/create file, so the first component will be returned in most cases, then optimizing based on first component's attributes will highly likely get opposite result.</p></blockquote> <p>That's why I suggested to "have it return something reasonable for PFL files when used, such as the plain layout of the last component (if file has no size) and the last init component if it has a size." That would return the last component if the file was just created.</p> <p>Another alternative is to return the fields of interest in the current padding fields:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">struct lov_user_md_v1 { <span class="code-comment">/* LOV EA user data (host-endian) */</span> : __u32 lmm_stripe_size; <span class="code-comment">/* size of stripe in bytes */</span> __u16 lmm_stripe_count; <span class="code-comment">/* num stripes in use <span class="code-keyword">for</span> <span class="code-keyword">this</span> object */</span> __u16 lmm_layout_gen; /* layout generation number * used when reading */ </pre> </div></div> <p>These map to the <tt>__u64 lcm_padding2</tt> in <tt>struct lov_comp_md_v1</tt>, which is currently unused. If this ioctl() is called, it would be possible to extract those values from the proper component, temporarily store them into <tt>lov_user_md_v1</tt>, and return it to userspace. This will likely allow 99% of apps to continue using Lustre without any change.</p> <p>If course, Ideally we will get a proper fix into the hands of the users, but I think there also needs to be a workaround in place so that sites which don't apply the fix.</p> <p>Niu Yawei (yawei.niu@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/27180" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27180</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9490" title="MPI-IO Lustre ADIO driver gets Lustre layout parameters incorrectly " class="issue-link" data-issue-key="LU-9490"><del>LU-9490</del></a> tests: ll_dirstripe_verify handles PFL layout<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ecf66cbfaf2e5734a18098db77882a344c944f9f</p> <p>Niu Yawei (yawei.niu@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/27183" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27183</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9490" title="MPI-IO Lustre ADIO driver gets Lustre layout parameters incorrectly " class="issue-link" data-issue-key="LU-9490"><del>LU-9490</del></a> llite: return v1/v3 layout for legacy app<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 87da9d4b51f6874412c46ff130b36e938fa85fe4</p> <p>We still need a fix for MPICH ADIO, otherwise it will never work properly with composite layouts. The current solution allows it to <em>function</em>, but it won't generate any optimal IO patterns.</p> <p>Emoly to work on ADIO driver ww21</p> <p>I will look into this one.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a>, I'm cloning MPICH git repo. Do you mean I need to create a patch there? <br/> BTW, the lustre ADIO driver patch in our repo (lustre/contrib/adio_driver_mpich2-1.0.7.patch) is too old, I remembered it was released by MPICH2 about 8 years ago. Should it be removed?</p> <p>Emoly, yes the fix should be submitted to the MPICH Git repo. The ADIO patches in <tt>lustre/contrib</tt> should be deleted.</p> <p>Discussed with Niu and worked on how to apply each component stripe information to the I/O pattern redistribution.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/27310" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27310</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9490" title="MPI-IO Lustre ADIO driver gets Lustre layout parameters incorrectly " class="issue-link" data-issue-key="LU-9490"><del>LU-9490</del></a> llite: return v1/v3 layout for legacy app<br/> Project: fs/lustre-release<br/> Branch: pfl<br/> Current Patch Set: 1<br/> Commit: f3e00e27e8f86be9400df19f074b23da6d2b17fb</p> <p>Andreas Dilger (andreas.dilger@intel.com) merged in patch <a href="https://review.whamcloud.com/27310/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27310/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9490" title="MPI-IO Lustre ADIO driver gets Lustre layout parameters incorrectly " class="issue-link" data-issue-key="LU-9490"><del>LU-9490</del></a> llite: return v1/v3 layout for legacy app<br/> Project: fs/lustre-release<br/> Branch: pfl<br/> Current Patch Set: <br/> Commit: fac27756d5f2a283a5ec7a908e6434d30748a6bd</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/27183/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27183/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9490" title="MPI-IO Lustre ADIO driver gets Lustre layout parameters incorrectly " class="issue-link" data-issue-key="LU-9490"><del>LU-9490</del></a> llite: return v1/v3 layout for legacy app<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 545f464b68ee21ccc425113db9b57c63ac2840b1</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/27180/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27180/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9490" title="MPI-IO Lustre ADIO driver gets Lustre layout parameters incorrectly " class="issue-link" data-issue-key="LU-9490"><del>LU-9490</del></a> tests: ll_dirstripe_verify handles PFL layout<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c1359f54314084e0613a5ae70168595f296b7dfc</p> <p>Lustre changes all landed for 2.10. ADIO changes being tracked separately</p> <p>ADIO changes being tracked in <a href="https://jira.whamcloud.com/browse/LU-9657" title="Make Lustre ADIO driver work with PFL correctly" class="issue-link" data-issue-key="LU-9657"><del>LU-9657</del></a></p> Related LU-9349 LU-10232 LU-7964 LU-9550 Development Rank 1|hzzci7: Rank (Obsolete) 9223372036854775807 Severity