[LU-4475] mount command errors: "Communicating with 0@lo, operation mds_connect failed with -11" AND "Transport endpoint is not connected"

• Type: Story
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.4.2
• Labels: llnl

Environment

# uname -a
Linux es0 2.6.32-358.23.2.el6_lustre.x86_64 #1 SMP Thu Dec 19 19:57:45 PST 2013 x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa | egrep "lustre|e2fs" | sort
e2fsprogs-1.42.7.wc2-7.el6.x86_64
e2fsprogs-libs-1.42.7.wc2-7.el6.x86_64
kernel-2.6.32-358.23.2.el6_lustre.x86_64
kernel-firmware-2.6.32-358.23.2.el6_lustre.x86_64
lustre-2.4.2-2.6.32_358.23.2.el6_lustre.x86_64.x86_64
lustre-ldiskfs-4.1.0-2.6.32_358.23.2.el6_lustre.x86_64.x86_64
lustre-modules-2.4.2-2.6.32_358.23.2.el6_lustre.x86_64.x86_64
lustre-osd-ldiskfs-2.4.2-2.6.32_358.23.2.el6_lustre.x86_64.x86_64

• Assignee: WC Triage
• Reporter: Mark Duffield

Description

I created the mgs/mdt:

mkfs.lustre --fsname=lfs1 --mgs --mdt --index=0 /dev/vg_root/es0-00

and the ost, on another node:

mkfs.lustre --fsname=lfs1 --mgsnode=172.18.54.21@tcp0 --ost --index=0 /dev/vg_root/es2-00

When mounting either I receive a comm error.

When mounting the ost I see "Transport endpoint is not connected":

# mount -vvv -t lustre /dev/dm-3 /mnt/ost0
mount: fstab path: "/etc/fstab"
mount: mtab path:  "/etc/mtab"
mount: lock path:  "/etc/mtab~"
mount: temp path:  "/etc/mtab.tmp"
mount: UID:        0
mount: eUID:       0
mount: spec:  "/dev/mapper/vg_root-es2--00"
mount: node:  "/mnt/ost0"
mount: types: "lustre"
mount: opts:  "(null)"
final mount options: '(null)'
mount: external mount: argv[0] = "/sbin/mount.lustre"
mount: external mount: argv[1] = "/dev/mapper/vg_root-es2--00"
mount: external mount: argv[2] = "/mnt/ost0"
mount: external mount: argv[3] = "-v"
mount: external mount: argv[4] = "-o"
mount: external mount: argv[5] = "rw"
arg[0] = /sbin/mount.lustre
arg[1] = -v
arg[2] = -o
arg[3] = rw
arg[4] = /dev/mapper/vg_root-es2--00
arg[5] = /mnt/ost0
source = /dev/mapper/vg_root-es2--00 (/dev/mapper/vg_root-es2--00), target = /mnt/ost0
options = rw
checking for existing Lustre data: found
Reading CONFIGS/mountdata
mounting device /dev/mapper/vg_root-es2--00 at /mnt/ost0, flags=0x1000000 options=osd=osd-ldiskfs,errors=remount-ro,mgsnode=172.18.54.21@tcp,virgin,param=mgsnode=172.18.54.21@tcp,svname=lfs1-OST0000,device=/dev/mapper/vg_root-es2--00
mount.lustre: mount /dev/mapper/vg_root-es2--00 at /mnt/ost0 failed: Transport endpoint is not connected retries left: 0
mount.lustre: mount /dev/mapper/vg_root-es2--00 at /mnt/ost0 failed: Transport endpoint is not connected

And when mounting the mgs/mdt I see "Communicating with 0@lo, operation mds_connect failed with -11":

Jan 11 11:14:30 es0 kernel: LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. quota=on. Opts: 
Jan 11 11:14:30 es0 kernel: Lustre: lfs1-MDT0000: used disk, loading
Jan 11 11:14:30 es0 kernel: LustreError: 11-0: lfs1-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11.

The communication looks fine between nodes:

From es0:

[root@es0 log]# lctl
lctl > ping es2
12345-0@lo
12345-172.18.54.23@tcp

From es2:

[root@es2 log]# lctl
lctl > ping es0
12345-0@lo
12345-172.18.54.21@tcp