[LU-11985] Lustre 2.12.0 client compatibility question

• Type: Bug
• Resolution: Not a Bug
• Priority: Major
• Affects Version/s: Lustre 2.10.5
• Labels: interop
• Environment: server side: ; Linux aeon-eval-nvme-xeon 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux ; lustre-2.12.0-1.el7.x86_64 ; ZFS 0.7.9 ; ; client side: ; 2.10.5 ; 2.11.0 ; 2.12.0
• Severity: 3
• Assignee: Patrick Farrell (Inactive)
• Reporter: Haisong Cai (Inactive)

Description

 

We are getting errors when accessing a Lustre 2.12.0 filesystem with client version 2.10.5 or 2.11.0.

• we can make directory, no problem
• for files, we get errors like
• ls: cannot access file55: Invalid argument
  ls: cannot access file22: Invalid argument
  ls: cannot access file74: Invalid argument
  ls: cannot access file90: Invalid argument
  ls: cannot access file46: Invalid argument
  ls: cannot access file31: Invalid argument
• -bash-4.1$ ls -las
  ls: cannot access one: Invalid argument
  total 50
  25 drwxr-xr-x 2 manu1729 csd102 25600 Feb 20 09:30 .
  25 drwxr-xr-x 6 manu1729 csd102 25600 Feb 20 09:30 ..
  ? -????????? ? ? ? ? ? one
• using client 2.12, all looks fine.
• 2.12.0 ChangeLog says: Clients & Servers: Latest 2.10.X and Latest 2.11.X

Am I missing something here?

Comments

[Patrick Farrell (Inactive)]

Some confusion over exact versions live on the nodes, now resolved.

[Patrick Farrell (Inactive)]

Haisong,

Sure, glad to help.

Given the description of this ticket is specific to an issue that is now resolved, I'm actually going to ask you to open a new ticket for the next issue.  This is just so we can keep the ticket closely aligned to the problem being discussed.  We'll still help you out.

 

Thanks!

[Haisong Cai (Inactive)]

Hi Patrick,

After seeing your message yesterday afternoon, I tried unloading Lustre (lustre_rmmod) and reloading it again, this time it appeared permission errors went away. I am attaching a command history output to reference I did see the errors on 2.11 clients after the first time loading the Lustre, with mounting the Lustre f/s.

 

The history file was taken on one of the 2 clients (not the one I took debug_kernel on). line 1-30, was the first attempt mounting the f/s with 2.10.5. from 31-109 was when I uninstall 2.10.5 and install 2.11. from 110 to the end was when I made the unloading-reloading yesterday afternoon.

So far we have ran a small sets of tests and haven't seen compatibility issues. Please keep this ticket open for a couple of days as we are about to ramp up the tests.

 

Thanks for the help,

Haisong

[Patrick Farrell (Inactive)]

aeonjeffj good to know, but please check the client versions, etc - The logs I have been shown don't show any bugs.

A 2.10 client can't use DOM.  This is what it looks like when you try (I understand it's not the best representation of that incompatibility, sorry.).

If you're having issues with a 2.11 client and DOM files - or issues with non-DOM or FLR files and a 2.10 client - let us know.

[Jeff Johnson (Inactive)]

Also, FWIW the MDS is running spl/zfs 0.7.12. 

[Patrick Farrell (Inactive)]

Note also in the dmesg you attached:
Lustre: Lustre: Build Version: 2.10.5

Not 2.11.

[Patrick Farrell (Inactive)]

The client this dklog is from is running 2.10.X (I believe 2.10.5?), and it is rejecting a DoM component for having no stripes.  This is expected behavior.  You cannot use DoM files with a 2.10 client.

Can you please double check your versions and interop issues with this in mind?

 

[Haisong Cai (Inactive)]

Logs uploaded.

 

Haisong

[Patrick Farrell (Inactive)]

Interesting, OK.

Let's get some debug, from this client and from the MDS.

DEBUGMB=`lctl get_param -n debug_mb`
lctl set_param *debug=-1 debug_mb=10000
lctl clear
lctl mark "before"

# do the ls -la command on one file

lctl mark "after"
#Write out the log
lctl dk > /tmp/log

#Set debug back to defaults
lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck"
lctl set_param debug_mb=$DEBUGMB 

Please gather the debug log from the client & the MDS and post those here.

[Haisong Cai (Inactive)]

 

All above messages are coming from comet-26-02 which is running Lustre 2.11.0

 

dmesg from the same client is coming