https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11985 Lustre <p> </p> <p>We are getting errors when accessing a Lustre 2.12.0 filesystem with client version 2.10.5 or 2.11.0.</p> <ul> <li>we can make directory, no problem</li> <li>for files, we get errors like</li> <li>ls: cannot access file55: Invalid argument<br/> ls: cannot access file22: Invalid argument<br/> ls: cannot access file74: Invalid argument<br/> ls: cannot access file90: Invalid argument<br/> ls: cannot access file46: Invalid argument<br/> ls: cannot access file31: Invalid argument</li> <li>-bash-4.1$ ls -las<br/> ls: cannot access one: Invalid argument<br/> total 50<br/> 25 drwxr-xr-x 2 manu1729 csd102 25600 Feb 20 09:30 .<br/> 25 drwxr-xr-x 6 manu1729 csd102 25600 Feb 20 09:30 ..<br/> ? -????????? ? ? ? ? ? one</li> <li>using client 2.12, all looks fine.</li> <li>2.12.0 ChangeLog says: Clients &amp; Servers: Latest 2.10.X and Latest 2.11.X</li> </ul> <p>Am I missing something here?</p> server side: <br/> Linux aeon-eval-nvme-xeon 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux <br/> lustre-2.12.0-1.el7.x86_64 <br/> ZFS 0.7.9 <br/> <br/> client side: <br/> 2.10.5 <br/> 2.11.0 <br/> 2.12.0 LU-11985

Lustre 2.12.0 client compatibility question

Bug Major Resolved Not a Bug Patrick Farrell Haisong Cai interop Wed, 20 Feb 2019 20:36:05 +0000 Wed, 29 Jun 2022 19:33:46 +0000 Thu, 21 Feb 2019 17:14:47 +0000 Lustre 2.10.5 0 4 <p>Cai,</p> <p>Can you do lfs getstripe (from a 2.12 client) on one of these files?  You may be using a layout feature not supported in the earlier version.  (Though you should get -EOPNOTSUP, not -EINVAL, generally.)</p> <p> </p> <p><span class="error">&#91;root@comet-26-02 dir1&#93;</span># ls -las</p> <p>...</p> <p>13 <del>rw-r</del><del>r</del>- 1 manu1729 csd102 1024 Feb 20 09:24 file97<br/> 13 <del>rw-r</del><del>r</del>- 1 manu1729 csd102 1024 Feb 20 09:24 file98<br/> 13 <del>rw-r</del><del>r</del>- 1 manu1729 csd102 1024 Feb 20 09:24 file99<br/> ? -????????? ? ? ? ? ? one</p> <p><span class="error">&#91;root@comet-26-02 dir1&#93;</span># lfs getstripe one<br/> lfs getstripe: error opening one: Invalid argument (22)<br/> one<br/> lcm_layout_gen: 2<br/> lcm_mirror_count: 1<br/> lcm_entry_count: 2<br/> lcme_id: 1<br/> lcme_mirror_id: 0<br/> lcme_flags: init<br/> lcme_extent.e_start: 0<br/> lcme_extent.e_end: 131072<br/> lmm_stripe_count: 0<br/> lmm_stripe_size: 131072<br/> lmm_pattern: mdt<br/> lmm_layout_gen: 0<br/> lmm_stripe_offset: 0</p> <p>lcme_id: 2<br/> lcme_mirror_id: 0<br/> lcme_flags: 0<br/> lcme_extent.e_start: 131072<br/> lcme_extent.e_end: EOF<br/> lmm_stripe_count: -1<br/> lmm_stripe_size: 4194304<br/> lmm_pattern: raid0<br/> lmm_layout_gen: 0<br/> lmm_stripe_offset: -1</p> <p> </p> <p>dmesg:</p> <p>LustreError: 2054:0:(lcommon_cl.c:181:cl_file_inode_init()) Skipped 1 previous similar message<br/> LustreError: 2054:0:(llite_lib.c:2328:ll_prep_inode()) new_inode -fatal: rc -22<br/> LustreError: 2054:0:(llite_lib.c:2328:ll_prep_inode()) Skipped 11 previous similar messages<br/> LustreError: 2078:0:(llite_lib.c:2328:ll_prep_inode()) new_inode -fatal: rc -22<br/> LustreError: 2078:0:(llite_lib.c:2328:ll_prep_inode()) Skipped 4 previous similar messages</p> <p>Cai,</p> <p>Is this from a 2.12 client?  I thought you said the 2.12 client didn't get these errors?  If this is not from a 2.12 client, can you try again from a 2.12 client?</p> <p>But, since you gave a portion of it - Can you share all of dmesg from comet-26-02 ?  (Which I assume is not running 2.12?)</p> <p>Also, it looks like you've got a data-on-MDT component in this file.  That is not going to work with a 2.10 client, because it lacks the feature entirely.</p> <p> </p> <p>All above messages are coming from comet-26-02 which is running Lustre 2.11.0</p> <p> </p> <p>dmesg from the same client is coming</p> <p> </p> <p>Interesting, OK.</p> <p>Let's get some debug, from this client and from the MDS.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>DEBUGMB=`lctl get_param -n debug_mb` lctl set_param *debug=-1 debug_mb=10000 lctl clear lctl mark "before" # do the ls -la command on one file lctl mark "after" #Write out the log lctl dk &gt; /tmp/log #Set debug back to defaults lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" lctl set_param debug_mb=$DEBUGMB </pre> </div></div> <p>Please gather the debug log from the client &amp; the MDS and post those here.</p> <p>Logs uploaded.</p> <p> </p> <p>Haisong</p> <p>The client this dklog is from is running 2.10.X (I believe 2.10.5?), and it is rejecting a DoM component for having no stripes.  This is expected behavior.  You cannot use DoM files with a 2.10 client.</p> <p>Can you please double check your versions and interop issues with this in mind?</p> <p> </p> <p>Note also in the dmesg you attached:<br/> Lustre: Lustre: Build Version: 2.10.5</p> <p>Not 2.11.</p> <p>Also, FWIW the MDS is running spl/zfs 0.7.12. </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=aeonjeffj" class="user-hover" rel="aeonjeffj">aeonjeffj</a> good to know, but please check the client versions, etc - The logs I have been shown don't show any bugs.</p> <p>A 2.10 client can't use DOM.  This is what it looks like when you try (I understand it's not the best representation of that incompatibility, sorry.).</p> <p>If you're having issues with a 2.11 client and DOM files - or issues with non-DOM or FLR files and a 2.10 client - let us know.</p> <p>Hi Patrick,</p> <p>After seeing your message yesterday afternoon, I tried unloading Lustre (lustre_rmmod) and reloading it again, this time it appeared permission errors went away. I am attaching a command history output to reference I did see the errors on 2.11 clients after the first time loading the Lustre, with mounting the Lustre f/s. </p> <p> </p> <p>The history file was taken on one of the 2 clients (not the one I took debug_kernel on). line 1-30, was the first attempt mounting the f/s with 2.10.5. from 31-109 was when I uninstall 2.10.5 and install 2.11. from 110 to the end was when I made the unloading-reloading yesterday afternoon.</p> <p>So far we have ran a small sets of tests and haven't seen compatibility issues. Please keep this ticket open for a couple of days as we are about to ramp up the tests.</p> <p> </p> <p>Thanks for the help,</p> <p>Haisong</p> <p>Haisong,</p> <p>Sure, glad to help.</p> <p>Given the description of this ticket is specific to an issue that is now resolved, I'm actually going to ask you to open a new ticket for the next issue.  This is just so we can keep the ticket closely aligned to the problem being discussed.  We'll still help you out.</p> <p> </p> <p>Thanks!</p> <p>Some confusion over exact versions live on the nodes, now resolved.</p> Development Rank 1|i00by7: Rank (Obsolete) 9223372036854775807 Severity Site Affected: SDSC