|
Unfortunately, there is no OI scrub functionality for ZFS today, so it isn't possible to just delete the corrupted OI file and have OI Scrub rebuild it. ZFS is not supposed to be corrupted during usage, and none of the APIs that Lustre is using to modify the filesystem should allow the pool to be corrupt.
However, in LU-7798 Frank wrote:
I forgot to mention that the OSS nodes have been extended to operate in active-active failover configuration
for disk resources by Feb, 17th 2016. So the failover partner node lola-4 can see of node lola-5 all disks and has its ZFS pools imported also.
There's no start-up (boot) wrapper script that prevents the (primary) zpools of the other node from being imported.
ZFS is definitely vulnerable to corruption if it is mounted on multiple nodes at the same time, and there is not currently an MMP feature like ldiskfs that actively prevents it from being accessed by two nodes. This kind of corruption would show up first for files that are being modified frequently (e.g. the OI file seen here) because each node will be assigning different blocks and updating the tree differently. There needs to be strict Linux HA control of the zpools so that they are not imported on the backup node unless they are failed over, and when failover happens there needs to be STONITH to turn off the primary node before the pool is imported on the backup to avoid concurrent access. It is worthwhile to contact Gabriele or Zhiqi to see if there is a best-practices guide to installing ZFS in HA failover configuration.
Also, see patch http://review.whamcloud.com/16611 "LU-7134 utils: Ensure hostid set for ZFS during mkfs" which requires /etc/hostid is set and would at least provide basic protection from pool import if it is in use on another node.
You may be able to recover the corrupted OST zpool if it hasn't been mounted for a long time by reverting to an older uberblock or snapshot that does not have the corruption in it. See for example http://www.solarisinternals.com/wiki/index.php/ZFS_forensics_scrollback_script or https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSMetadataRecovery for details. If that doesn't work then it would be necessary to reformat the filesystem. However, at a minimum the /etc/hostid should be set (and unique!) to prevent gratuitous imports, and failover between OSSes should be disabled until proper HA configuration is done.
| 
console-lola-5.log.bz2