[LU-7938] Recovery on secondary OSS node stalled

• Type: Bug
• Resolution: Not a Bug
• Priority: Blocker
• Fix Version/s: Lustre 2.9.0
• Affects Version/s: Lustre 2.8.0, Lustre 2.9.0
• Labels: soak
• Environment: lola; build: 2.8 GA + patches
• Severity: 3
• Assignee: nasf (Inactive)
• Reporter: Frank Heckes (Inactive)

Description

Description

Error happens during soak testing of build '20160324' (see https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160324). DNE is enabled. MDTs had been formatted with ldiskfs, OSTs using zfs. MDS and OSS nodes are configured in HA active-active failover configuration. MDS nodes operated wiht 1 MDT per MDS, while OSSes running 4 OST per node.
Nodes lola-4 and lola-5 form a HA cluster.

Event history

• 2016-03-29 07:48:04,307:fsmgmt.fsmgmt:INFO triggering fault oss_failover of node lola-4
  powercyle node
• 2016-03-29 07:52:27,876:fsmgmt.fsmgmt:INFO lola-4 is up
• 2016-03-29 07:53:09,557:fsmgmt.fsmgmt:INFO zpool import and mount of lola-4's OSTs complete
• Recovery don't complete after recovery_time is zero
• 2016-03-29 08:03 (approximately) Aborted recovery manually (lctl --device ... abort_recovery)

Attached files:
messages, console and debug log before (lustre-log-20160329-0759-recovery-stalled) and after recovery was aborted (lustre-log-20160329-0803-recovery-aborted)