Details
-
Bug
-
Resolution: Fixed
-
Major
-
None
-
Lustre 2.10.3
-
None
-
3
-
9223372036854775807
Description
I have found a weird problem on our Lustre system when we try to move a file from a different file system (here /tmp) onto the lustre file server. This problem only affects a mv. A cp works ok. The problem is that the 'mv' hangs forever, and the process can not be a killed WHen I did a strace on the mv, the program hangs on fchown.
strace mv /tmp/simon.small.txt /mnt/lustre/projects/pMOSP/simon
<stuff>
write(4, "1\n", 2) = 2
read(3, "", 4194304) = 0
utimensat(4, NULL, [{1530777797, 478293939}, {1530777797, 478293939}], 0) = 0
fchown(4, 10001, 10025
If you look at demsg, you see these multiple errors start appearing at the same time:
The errors don't stop as we can't kill the 'mv' process
Thu Jul 5 18:08:43 2018] Lustre: lustre-MDT0000-mdc-ffff88351771f000: Connection restored to 172.16.231.50@o2ib (at 172.16.231.50@o2ib)
[Thu Jul 5 18:08:43 2018] Lustre: Skipped 140105 previous similar messages
[Thu Jul 5 18:09:47 2018] Lustre: lustre-MDT0000-mdc-ffff88351771f000: Connection to lustre-MDT0000 (at 172.16.231.50@o2ib) was lost; in progress operations using this service will wait for recovery to complete
[Thu Jul 5 18:09:47 2018] Lustre: Skipped 285517 previous similar messages
[Thu Jul 5 18:09:47 2018] Lustre: lustre-MDT0000-mdc-ffff88351771f000: Connection restored to 172.16.231.50@o2ib (at 172.16.231.50@o2ib)
[Thu Jul 5 18:09:47 2018] Lustre: Skipped 285516 previous similar messages
We have the following ofed drivers, which I believe have a known problem with connecting to Lustre servers
ofed_info | head -1 MLNX_OFED_LINUX-4.2-1.2.0.0 (OFED-4.2-1.2.0):
Attachments
Issue Links
Activity
John,
Firstly I'd like to note that when I tried running the chgrp command, the OS became unstable and crashed on me. This has happened twice on me:
*Message from syslogd@monarch-login2 at Aug 3 17:06:27 ...
kernel:LustreError: 12556:0:(niobuf.c:330:ptlrpc_register_bulk()) ASSERTION( desc->bd_md_count == 0 ) failed:
Message from syslogd@monarch-login2 at Aug 3 17:06:27 ...
kernel:LustreError: 12556:0:(niobuf.c:330:ptlrpc_register_bulk()) LBUG*
Would you be interested in any of the contents of the /var/crash directories, i.e either the large vmcore or the vmcore-dmesg files? This may/may not be related to another bug we have in lustre which crashes our machines regularly. I am led to believe that bug is caused by an issue with the current version of mofed drivers.
I ran the jobs an have uploaded the files for you, with the Aug4 postfix on them.
regards Simon
Dear John
I uploaded 2 of the files. I am afraid I could not find the program strack1. Is this part of the lustre distribution or a tool I need to install?
regards
Simon
Hi Simon,
The attachments from the 17th show that the client is waiting for the MDT to respond. But the previously attached stack1 output from the MDT show that the server is idle. Can you do the following:
# On the client and MDT: lctl set_param debug='+dlmtrace rpctrace vfstrace inode trace' lctl set_param debug_mb=64 lctl clear # On the client: chgrp pMOSP /mnt/lustre/projects/pMOSP/simon/simon.small.txt.3 & # Wait for chgrp to hang in fchown() lctl dk > /tmp/client-chgrp-dk.out strack1 > /tmp/client-chgrp-stack1.out # On the MDT: lctl dk > /tmp/mdt-chgrp-dk.out strack1 > /tmp/mdt-chgrp-stack1.out
Dear John
I have uploaded the files to you. These were run on the client monarch-dtn
regards
Simon
Hi Simon,
OK this is interesting. The file you attached shows that the MDT was completely idle when stack1 was run. So I'm sorry to say we may have been going in the wrong direction? Could you run it on the client after chown/chgrp hangs? Could you also enable some more debugging and run the following on the client:
lctl set_param debug='+dlmtrace rpctrace vfstrace inode trace' lctl set_param debug_mb=64 lctl clear chgrp pMOSP /mnt/lustre/projects/pMOSP/simon/simon.small.txt.3 & ### Wait for chgrp to hang in fchown() lctl dk > /tmp/chgrp-dk.out strack1 > /tmp/chgrp-stack1.out
And then attach /tmp/chgrp-dk.out and /tmp/chgrp-stack1.out?
Dear John,
I ran the command and attached the file.
The client executed the command
strace chgrp pMOSP /mnt/lustre/projects/pMOSP/simon/simon.small.txt.3
which goes up to..
read(4, "\1\0\0\0\0\0\0\0L'\0\0\20\0\0\0pMOSP\0*\0smichnow"..., 136) = 136
newfstatat(AT_FDCWD, "/mnt/lustre/projects/pMOSP/simon/simon.small.txt.3",
, AT_SYMLINK_NOFOLLOW) = 0
fchownat(AT_FDCWD, "/mnt/lustre/projects/pMOSP/simon/simon.small.txt.3", -1, 10060, 0
and hangs
regards
Simon
Could you run your reproducer and wait until the client blocks in fchown(), then run the attached script stack1 on the MDT (to collect and consolidate the kernel stack traces) and attach the output here?
John
the dmesg errors from the node that hangs upon a chgrp are
[Mon Jul 16 17:14:54 2018] Lustre: lustre-MDT0000-mdc-ffff88351a566800: Connection to lustre-MDT0000 (at 172.16.231.50@o2ib) was lost; in progress operations using this service will wait for recovery to complete
[Mon Jul 16 17:14:55 2018] Lustre: Skipped 1175070 previous similar messages
[Mon Jul 16 17:14:55 2018] Lustre: lustre-MDT0000-mdc-ffff88351a566800: Connection restored to 172.16.231.50@o2ib (at 172.16.231.50@o2ib)
[Mon Jul 16 17:14:55 2018] Lustre: Skipped 1175070 previous similar messages
regards
Simon
OK this is better. The chgrp is failing because the MDT is not connected to OST000c. What is the status of that OST? It appears that the client and server are not handling this condition correctly.
The MDT logs you provided are not from Lustre 2.10.3. What version of Lustre is the MDT running?
The failed assertions are due to
LU-8573and possibly OFED issues.