|
Crash file has been stored at lhn:/scratch/crashdumps/lu-7430/127.0.0.1-2015-11-13-16:28:09 and can be uploaded on demand to desired storage location.
|
|
Just investigate this problem a bit. So there are two problems here.
1. It seems MDT assign duplicate generations in last rcvd file.
This causes the mount failure. Hmm, this looks like multiple slot RPC patch problem.
2. The error handler for this mount failure seems not be handled correctly, which caused the panic here.
|
|
the console output contains a lot of I/O errors. can this is a root cause?
|
|
Yes, but this happened several times (> 5 times) specially at this place in soak-tests, so I doubt this because of the I/O errors.
|
|
How can I have a look at the lustre logs of the failing MDS ?
|
|
Here you are.
|
|
Alex: No it isn't. The multipathd queries for some reason the device of the Emc^2 (or is it Dell now  )
controller devices although they're blacklisted in mulitpath configuration file. If you refer to this:
the messages can be ignored.
|
|
Grégoire, Alex, is there a reason that the generation needs to be unique between different clients? If yes, then we need to examine the code that assigns the generation, as well as the recovery code to ensure that there aren't old slots on disk that are not cleared after recovery is completed.
|
|
well, it should be unique as it's used to bind the replies to the clients. essentially it's unique client id growing monotonically. it's calculated from last_rcvd on every boot. given 1 was duplicated, it looks like the whole process of last_rcvd scanning was skipped for a reason.
|
|
I have reproduced the error handler problem in MDT recovery that leads to the GPF.
In tgt_clients_data_init(), a new export is created for each valid client area detected in the last_rcvd file. Valid client area is based on a non-zero lcd_uuid field. Creation operations include calls to class_new_export() that initializes the obd_export structure and add it to the hash tables, and tgt_client_add() routines that update the lu_target related fields (lut_client_bitmap).
When an error is encountered, the exports deletions are postponed to be handled later by obd_zombid thread. Export deletion operation includes call to tgt_client_free() routine that updates the lu_target related fields (especially the lut_client_bitmap).
But, the error in tgt_clients_data_init() is reported back up to tgt_server_data_init() and tgt_init() routines, which frees the lu_target data including the lut_client_bitmap.
When obd_zombid thread calls tgt_client_free() the lut_client_bitmap has been poison'ed which make the MDS crash.
I am going to see how it is possible to manage correctly that error path, and push a patch.
Note that the issue was already present before multi slot RPC feature, but the error path was probably never called.
About the duplicate generation in the last_rcvd file, I have no idea of what could lead to that situation for now.
|
|
Grégoire Pichon (gregoire.pichon@bull.net) uploaded a new patch: http://review.whamcloud.com/17424
Subject: LU-7430 mdt: better handle MDT recovery error path
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b3341ed8f0b2568eebdec1677bcb074e3da9d410
|
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/17424/
Subject: LU-7430 mdt: better handle MDT recovery error path
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0d3a07a8aa46bd190813b6e6e3da0e12c61a9d09
|
|
Landed for 2.8
|
Generated at Sat Feb 10 02:08:50 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
console-lola-9.log.gz