[#LU-1308] 2.2 clients unable to mount upgraded MDT

[LU-1308] 2.2 clients unable to mount upgraded MDT Created: 11/Apr/12 Updated: 07/Jun/12 Resolved: 03/May/12
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 2.2.0, Lustre 2.3.0
Fix Version/s:	Lustre 2.3.0, Lustre 2.1.2

Type:

Bug

Priority:

Blocker

Reporter:

Marek Magrys

Assignee:

Oleg Drokin

Resolution:

Fixed

Votes:

Labels:

None

Environment:

Scientific Linux 5.5, Lustre 2.2.0 on servers, patchless 2.1.1 and 2.2.0 on clients

Attachments:

debug_logs.tar.gz

dumps.tar.gz

mount-debug-patch2.log.gz

mount.debug.gz

Severity:

Epic:

client, interop

Rank (Obsolete):

4641

Description

We are hitting a strange bug while upgrading to 2.2.0. We moved all the servers and some clients to 2.2 already, however our TCP clients are unable to mount the filesystem, because they are unable to find a suitable NID to connect to the MDT. 2.1.1 clients work fine.
In our case o2ib

{0,1}

are the first networks listed in all configs (MGS/MDT/OST config), and the tcp one is occuring as the third one. All the clients which use o2ib work fine, as the first MDT NID they get from MGS works for them, however TCP ones fail (at least thats what we supose).
Our mds params are:
Parameters: mgsnode=172.16.193.1@o2ib,172.16.126.1@tcp mgsnode=172.16.193.3@o2ib,172.16.126.2@tcp failover.node=172.16.193.3@o2ib,172.16.126.2@tcp mdd.quota_type=ug

Servers have:
options lnet networks="o2ib0(ib0),o2ib1(ib1),tcp0(eth0)"
TCP clients:
options lnet networks="tcp0(eth0)"

And the client gets this:
[root@n1-4-1 ~]# lctl which_nid 172.16.126.1@tcp
172.16.126.1@tcp

[root@n1-4-1 ~]# lctl ping 172.16.126.1@tcp
12345-0@lo
12345-172.16.193.1@o2ib
12345-192.168.193.2@o2ib1
12345-172.16.126.1@tcp

[root@n1-4-1 ~]# mount -t lustre 172.16.126.1@tcp:/scratch /mnt/lustre/scratch/
mount.lustre: mount 172.16.126.1@tcp:/scratch at /mnt/lustre/scratch failed: No such file or directory
Is the MGS specification correct?
Is the filesystem name correct?
If upgrading, is the copied client log valid? (see upgrade docs)

Dmesg says:
Apr 11 16:55:53 n1-4-1 kernel: Lustre: MGC172.16.126.1@tcp: Reactivating import
Apr 11 16:55:53 n1-4-1 kernel: LustreError: 2469:0:(ldlm_lib.c:381:client_obd_setup()) can't add initial connection
Apr 11 16:55:53 n1-4-1 kernel: LustreError: 2469:0:(obd_config.c:521:class_setup()) setup scratch-MDT0000-mdc-ffff81018d9d6400 failed (-2)
Apr 11 16:55:53 n1-4-1 kernel: LustreError: 2469:0:(obd_config.c:1362:class_config_llog_handler()) Err -2 on cfg command:
Apr 11 16:55:53 n1-4-1 kernel: Lustre: cmd=cf003 0:scratch-MDT0000-mdc 1:scratch-MDT0000_UUID 2:172.16.193.1@o2ib
Apr 11 16:55:53 n1-4-1 kernel: LustreError: 15c-8: MGC172.16.126.1@tcp: The configuration from log 'scratch-client' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
Apr 11 16:55:53 n1-4-1 kernel: LustreError: 2457:0:(llite_lib.c:978:ll_fill_super()) Unable to process log: -2
Apr 11 16:55:53 n1-4-1 kernel: LustreError: 2457:0:(obd_config.c:566:class_cleanup()) Device 3 not setup
Apr 11 16:55:53 n1-4-1 kernel: LustreError: 2457:0:(ldlm_request.c:1170:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
Apr 11 16:55:53 n1-4-1 kernel: LustreError: 2457:0:(ldlm_request.c:1796:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
Apr 11 16:55:53 n1-4-1 kernel: Lustre: client ffff81018d9d6400 umount complete
Apr 11 16:55:53 n1-4-1 kernel: LustreError: 2457:0:(obd_mount.c:2349:lustre_fill_super()) Unable to mount (-2)

I'm also attaching two debug dumps (lctl dk) for 2.1.1 client (works fine) and 2.2.0 client (fails).

Comments

Comment by Marek Magrys [ 11/Apr/12 ]

I should've mention it in the description: previous version on the servers was 2.1.0, clients were running mostly 2.1.1.

Comment by Jinshan Xiong (Inactive) [ 13/Apr/12 ]

Hi Marek, can you please dump the log on both client and MGS and post it here? Please set debug and subsystem_debug to -1 before you mount the client. thanks.

Comment by Marek Magrys [ 13/Apr/12 ]

Here are the logs you asked for, the MGS/MDS one is rather long, as the system is in production.

Comment by Peter Jones [ 16/Apr/12 ]

Oleg will be working on this one

Comment by Oleg Drokin [ 16/Apr/12 ]

The problem was introduced by this patch http://review.whamcloud.com/1189

Copy and paste error in class_add_uuid().

Patch in testing at cyfronet

Comment by Lukasz Flis [ 16/Apr/12 ]

client debug log with test patch by Oleg.

Comment by Lukasz Flis [ 16/Apr/12 ]

Client with the test patch is still unable to mount via tcp.
dk attached: mount-debug.log.gz

Comment by Supporto Lustre Jnet2000 (Inactive) [ 17/Apr/12 ]

Oleg, Which Lustre version are affected by this bug? Is It affected Lustre server version 2.1.x ?

Thanks in advance.

Comment by Peter Jones [ 17/Apr/12 ]

This is a 2.2 issue only so it will not be present in 2.1.x.

Comment by Oleg Drokin [ 17/Apr/12 ]

Please try this patch, I think it fixes all issues: http://review.whamcloud.com/2561

Comment by Lukasz Flis [ 17/Apr/12 ]

Oleg, thanks for the patch.
We did some testing and it appears that in our case only 5th or 6th "mount -a" attempt completes successfuly.

I am attaching client logs: mount-debug-patch2.log.gz

Comment by Lukasz Flis [ 17/Apr/12 ]

Debug log from clients with patch#2 applied.
Mounting works after several attempts (5-6) of "mount -a" invocation