Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
Lustre 2.4.0
-
PPC client
-
3
-
8173
Description
We updated a client to 2.3.64-4chaos and tried to mount a 2.3.63-6chaos server. The mount fails with
LustreError: 15c-8: MGC172.20.20.201@o2ib500: The configuration from log 'fsv-client' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. LustreError: 14351:0:(llite_lib.c:1043:ll_fill_super()) Unable to process log: -2 Lustre: Unmounted fsv-client LustreError: 14351:0:(obd_mount.c:1265:lustre_fill_super()) Unable to mount (-2)
Using git bisect I found the mount failure was introduced with this patch:
http://review.whamcloud.com/#change,5820
LU-2684 fid: unify ostid and FID
The critical questions at this point are:
- Can we solve this problem by updating both server and client to 2.3.64-4chaos?
- Can we safely upgrade the server, or does the above patch introduce on-disk format incompatibilities?
- Will we be able to safely revert the server to 2.3.63 in case we find problems, or will it write new objects in an incompatible format?
LLNL-bug-id: TOSS-2060
Attachments
Issue Links
- is related to
-
-
- Resolved
-
Activity
Haven't tried 2.3.65 yet, but initial testing suggests updating the server to 2.3.64 lets the mount succeed. Here's what I did:
1. Tried to mount a 2.3.62 server from a 2.3.64 PPC client. Fails with "ll_fill_super() Unable to process log: -2"
2. Updated the server to 2.3.64. Mount from 2.3.64 PPC client succeeds.
John, okay, we're getting a test environment set up where I should be able to do that test.
Possibly. Please see LU-3294.
Ned, it would be interesting to know what happens when you create a new 2.3.65 FS on ppc, unmount, and then remount it.
John, it was mentioned to me that you have already found some endian issues with the FID-on-OST code? Could you please point out where they are, it might be that this is the source of the problem being seen here, since we didn't see any problems with our x86_64 clients for interoperability.
Ned, is this a PPC client? It would be useful to include this information in the "Environment" section when filing a bug.
The 6044/LU-2888 patch fixed the handling on the server, but the original problem patch from LU-2684 wasn't in 2.3.63, so it shouldn't be relevant.
I did notice the mgs got ENOENT handling opcodes LLOG_ORIGIN_HANDLE_CREATE and LLOG_ORIGIN_HANDLE_READ_HEADER:
20000000:01000000:6.0:1368040430.786604:0:18265:0:(mgs_handler.c:757:mgs_handle()) @@@ MGS fail to handle opc = 501: rc = -2 req@ffff881019ecb050 x1434494006460492/t0(0) o501->2e89e428-68d9-71a1-75f0-147bc1963566@172.20.16.10@o2ib500:0/0 lens 296/0 e 0 to 0 dl 1368040491 ref 1 fl Interpret:/0/ffffffff rc 0/-1 ... 20000000:01000000:6.0:1368040430.788063:0:18265:0:(mgs_handler.c:757:mgs_handle()) @@@ MGS fail to handle opc = 503: rc = -2 req@ffff881019f14850 x1434494006460504/t0(0) o503->2e89e428-68d9-71a1-75f0-147bc1963566@172.20.16.10@o2ib500:0/0 lens 272/0 e 0 to 0 dl 1368040491 ref 1 fl Interpret:/0/ffffffff rc 0/-1
Attaching -1 debug logs for client and MDS. Note these were not captured from the same mount attempt.
The NID of the client is 172.20.16.10@o2ib500.
Ned, I just checked the debug log, it seems client get correct log ID after swab,
Here are the client log
But somehow server can not find the log object by this ID. Unfortunately, I can not find correspondent mgs handling information in the MDS debug log. Could you please redo the test update the debug log.
In the mean time, I do see there are some problem during the logid swab(John also point out one in
LU-3294) I will cook the patch now.