[LU-11119] A 'mv' of a file from a local file system to a lustre file system hangs

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Affects Version/s: Lustre 2.10.3

Environment

uname -a
Linux monarch-login2 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

 cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

lctl lustre_build_version
Lustre version: 2.10.3

• Severity: 3
• Assignee: John Hammond
• Reporter: Monash HPC

Description

I have found a weird problem on our Lustre system when we try to move a file from a different file system (here /tmp) onto the lustre file server. This problem only affects a mv. A cp works ok. The problem is that the 'mv' hangs forever, and the process can not be a killed WHen I did a strace on the mv, the program hangs on fchown.

strace mv /tmp/simon.small.txt  /mnt/lustre/projects/pMOSP/simon
<stuff>
write(4, "1\n", 2)                      = 2
read(3, "", 4194304)                    = 0
utimensat(4, NULL, [{1530777797, 478293939}, {1530777797, 478293939}], 0) = 0
fchown(4, 10001, 10025 

If you look at demsg, you see these multiple errors start appearing at the same time:
The errors don't stop as we can't kill the 'mv' process

Thu Jul  5 18:08:43 2018] Lustre: lustre-MDT0000-mdc-ffff88351771f000: Connection restored to 172.16.231.50@o2ib (at 172.16.231.50@o2ib)
[Thu Jul  5 18:08:43 2018] Lustre: Skipped 140105 previous similar messages
[Thu Jul  5 18:09:47 2018] Lustre: lustre-MDT0000-mdc-ffff88351771f000: Connection to lustre-MDT0000 (at 172.16.231.50@o2ib) was lost; in progress operations using this service will wait for recovery to complete
[Thu Jul  5 18:09:47 2018] Lustre: Skipped 285517 previous similar messages
[Thu Jul  5 18:09:47 2018] Lustre: lustre-MDT0000-mdc-ffff88351771f000: Connection restored to 172.16.231.50@o2ib (at 172.16.231.50@o2ib)
[Thu Jul  5 18:09:47 2018] Lustre: Skipped 285516 previous similar messages

We have the following ofed drivers, which I believe have a known problem with connecting to Lustre servers

ofed_info | head -1
MLNX_OFED_LINUX-4.2-1.2.0.0 (OFED-4.2-1.2.0):

Issue Links

• is related to: LU-11236 client MDT OST ENOTCONN loops
• is related to: LU-10310 Change l_getidentity error level from NOTICE to WARNING
• is related to: LU-11227 client process hangs when lod_sync accesses deactivated OSTs

Comments

[Ian Costello]

Hi Peter,

Problem resolved, we can close this one.

Thanks,
Ian

[Peter Jones]

Not any more - thx

[Malcolm Haak - NCI (Inactive)]

Also Peter,

LU-11227 is on the list on the change log page for 2.10.5

http://wiki.lustre.org/Lustre_2.10.5_Changelog

Thanks

[Malcolm Haak - NCI (Inactive)]

Hi Peter,

All good. Just though I might have been going crazy was all

[Peter Jones]

Ah. I apologize - it is not actually in 2.10.5. We had discussed including it but decided against it in the end because at the time it had not had very long in testing. It should be in 2.10.6

[Malcolm Haak - NCI (Inactive)]

Hi Peter,

Which commit ID? I looked at the git log for the 2.10.5 tag and could not see a commit claiming to pull this fix.

Was it merged into a different commit?

[Peter Jones]

Simon

JFYI- the above-mentioned fix for LU-11227 was included in the recently release Lustre 2.10.5.

Peter

[Julien Wallior]

Simon, John,
We had a similar (same?) issue on 2.10.4 servers/2.10.1 clients. We applied the patch from LU-11227 and it seems good now.

Regarding the assertion(bd_md_count==0), how does your trace look like? Do you have the {client,server}_bulk_callback() LustreErrors like in LU-8573? We were having this assertion too (before patching with LU-11227), but the trace looks a bit different than LU-8573 and no bulk_callback() LustreError.

[ 142.947358] [<ffffffff8100471d>] dump_trace+0x7d/0x2d0
[ 142.947369] [<ffffffffa0b4176a>] libcfs_call_trace+0x4a/0x60 [libcfs]
[ 142.947388] [<ffffffffa0b417e8>] lbug_with_loc+0x48/0xa0 [libcfs]
[ 142.947427] [<ffffffffa0ece3c4>] ptlrpc_register_bulk+0x7c4/0x990 [ptlrpc]
[ 142.947462] [<ffffffffa0ecef26>] ptl_send_rpc+0x236/0xe20 [ptlrpc]
[ 142.947497] [<ffffffffa0ec984c>] ptlrpc_check_set.part.23+0x18bc/0x1dd0 [ptlrpc]
[ 142.947531] [<ffffffffa0eca2b1>] ptlrpc_set_wait+0x481/0x8a0 [ptlrpc]
[ 142.947563] [<ffffffffa0eca74c>] ptlrpc_queue_wait+0x7c/0x220 [ptlrpc]
[ 142.947586] [<ffffffffa0e630fb>] mdc_getpage+0x18b/0x620 [mdc]
[ 142.947592] [<ffffffffa0e636ab>] mdc_read_page_remote+0x11b/0x650 [mdc]
[ 142.947598] [<ffffffff8113ea0e>] do_read_cache_page+0x7e/0x1c0
[ 142.947602] [<ffffffffa0e5eb07>] mdc_read_page+0x167/0x960 [mdc]
[ 142.947610] [<ffffffffa10010ce>] lmv_read_page+0x1ae/0x520 [lmv]
[ 142.947628] [<ffffffffa10964d0>] ll_get_dir_page+0xb0/0x350 [lustre]
[ 142.947637] [<ffffffffa10968c9>] ll_dir_read+0x99/0x310 [lustre]
[ 142.947648] [<ffffffffa10c9cf0>] ll_get_name+0x110/0x2d0 [lustre]
[ 142.947660] [<ffffffff8121b3b2>] exportfs_get_name+0x32/0x50
[ 142.947663] [<ffffffff8121b526>] reconnect_path+0x156/0x2e0
[ 142.947666] [<ffffffff8121b9df>] exportfs_decode_fh+0xef/0x2c0
[ 142.947684] [<ffffffffa04cd425>] fh_verify+0x2f5/0x5e0 [nfsd]
[ 142.947694] [<ffffffffa04d142c>] nfsd_access+0x2c/0x140 [nfsd]
[ 142.947705] [<ffffffffa04d7378>] nfsd3_proc_access+0x68/0xb0 [nfsd]
[ 142.947719] [<ffffffffa04c9cc2>] nfsd_dispatch+0xb2/0x200 [nfsd]
[ 142.947734] [<ffffffffa0320cab>] svc_process_common+0x43b/0x680 [sunrpc]
[ 142.947754] [<ffffffffa0320ffc>] svc_process+0x10c/0x160 [sunrpc]
[ 142.947770] [<ffffffffa04c96cf>] nfsd+0xaf/0x120 [nfsd]
[ 142.947775] [<ffffffff8107ae74>] kthread+0xb4/0xc0
[ 142.947780] [<ffffffff8152ae18>] ret_from_fork+0x58/0x90

 

[John Hammond]

Hi Simon,

Is OST000c offline or just deactivated to prevent new files from being created on it?

If it's only to prevent new files from being created then you should use the max_create_count parameter. See http://doc.lustre.org/lustre_manual.xhtml#section_remove_ost.

[Monash HPC]

Dear John,
we have been updating our mofed drivers and lustre client versions to try and resolve both bugs. So some clients are
lctl lustre_build_version
Lustre version: 2.10.3

and some
lctl lustre_build_version
Lustre version: 2.10.4

Both versions show this problem.

As for the OST000c, this is an inactive OST
lfs osts
OBDS:
0: lustre-OST0000_UUID ACTIVE
1: lustre-OST0001_UUID ACTIVE
2: lustre-OST0002_UUID ACTIVE
3: lustre-OST0003_UUID ACTIVE
4: lustre-OST0004_UUID ACTIVE
5: lustre-OST0005_UUID ACTIVE
6: lustre-OST0006_UUID ACTIVE
7: lustre-OST0007_UUID ACTIVE
8: lustre-OST0008_UUID ACTIVE
9: lustre-OST0009_UUID ACTIVE
10: lustre-OST000a_UUID ACTIVE
11: lustre-OST000b_UUID ACTIVE
12: lustre-OST000c_UUID INACTIVE
13: lustre-OST000d_UUID ACTIVE
14: lustre-OST000e_UUID ACTIVE

But the file belongs on a different OST ( I ran some code I found on the internet to find this)
/mnt/lustre/projects/pMOSP/simon/simon.small.txt.3: ['lustre-OST0009']

The MDT Lustre version is
[root@rclmddc1r14-02-e1 ~]# lctl lustre_build_version
Lustre version: 2.10.58_46_ge528677

regards
Simon