Details
-
Bug
-
Resolution: Cannot Reproduce
-
Minor
-
None
-
Lustre 2.4.0, Lustre 2.1.1
-
3
-
9748
Description
We occasionally see the message in the summary show up in the MDS console log during server recovery. What might cause this?
Attachments
Issue Links
Activity
The syslog messages doesn't provide much information. IMHO a possible cause may be that a file was created, and then opened, but MDS failed and unfortunately this inode was not synced to disk yet, then during MDS recovery, client tried to replay open, but MDS couldn't find this file, and printed this error message. If it's all that has happened, current handling looks reasonable.
Will you do such recovery test again? If so, I can provide a debug patch to print more information.
I wasn't able to get the debug log files but the attachment has syslog messages from the MDS. The OPEN & CREAT messages appear starting at May 3 12:16:33. We don't see any problems obviously connected with this error, although we are running into other recovery-related bugs in 2.1, namely LU-1352 and LU-1368.
In MDS recovery, all opened files (on client) needs to be opened again, this is called open replay. But in your case, the replayed open failed with -ENOENT, and open is not called with O_CREAT (it means this open should not create a new file), this is abnormal and this error message is printed. Did you see anything wrong after this?
If you met this again, could you get /var/log/messages and dump debug log of both client and server?
We see this on production servers, not during recovery testing. We can include a debug patch in our tree and wait for it to happen again. If we learn that this is a valid case under Lustre's consistency protocol then perhaps the message should not go to the console.