[LU-3353] import_sec_validate_get() import ffff88061d4a7000 (FULL) with no sec

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.7.0, Lustre 2.5.5
• Affects Version/s: Lustre 2.1.4
• Labels: llnl
• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: Ned Bass (Inactive)

Description

We started getting many of these messages on our 2.1.4 MDS. The timing corresponded with client reconnection problems, i.e. LU-1934.

2013-05-15 15:54:44 Lustre: lsd-MDT0000: Client a22a8a8a-754a-c1ab-859d-eda3c476a27d (at 192.168.112.1@o2ib6) reconnecting
2013-05-15 15:54:44 Lustre: Skipped 143 previous similar messages
2013-05-15 15:54:44 Lustre: lsd-MDT0000: Client a22a8a8a-754a-c1ab-859d-eda3c476a27d (at 192.168.112.1@o2ib6) refused reconnection, still busy with 1 active RPCs
2013-05-15 15:54:44 Lustre: Skipped 143 previous similar messages
2013-05-15 15:56:31 LustreError: 5204:0:(sec.c:385:import_sec_validate_get()) import ffff88061d4a7000 (FULL) with no sec
2013-05-15 15:56:31 LustreError: 5204:0:(sec.c:385:import_sec_validate_get()) Skipped 2399 previous similar messages

The MDS was just rebooted without a crash dump so, unless this reproduces, we won't be able to tell much about the imports in question.

Issue Links

• is related to: LU-7030 import_sec_validate_get()) import ffff8819c6497800 (FULL) with no sec (again)

Comments

[Jodi Levi (Inactive)]

Patch landed to Master. Patch for other branches will be tracked outside of this ticket.

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13254
Subject: LU-3353 ptlrpc: Suppress error message when imp_sec is freed
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: 2d6469ec1a1ffc820cb7d5a6e18006a4e41bad19

[D. Marc Stearman (Inactive)]

Can a patch be landed for 2.5? We still see this on our production clusters, and it would be nice to reduce unneeded log messages.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/10200/
Subject: LU-3353 ptlrpc: Suppress error message when imp_sec is freed
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: edf6116663724467207422dcc0c6120320055cac

[Amir Shehata (Inactive)]

the core reason for the race condition, where a reverse import is destroyed while there are client bound requests in flight, is when a client reconnects. The area in the code where I could identify is in:

target_handle_connect()
{
...
 spin_lock(&export->exp_lock);
 if (export->exp_imp_reverse != NULL)
  /* destroyed import can be still referenced in ctxt */
  tmp_imp = export->exp_imp_reverse;
 export->exp_imp_reverse = revimp;
 spin_unlock(&export->exp_lock);
...
}

later on the tmp_imp is destroyed.

While it's being destroyed import_sec_validate_get() could be getting called, and the security could've been cleared already.

the suggested solution is to suppress the error message in import_sec_validate_get() if the import is being destroyed

[Alexey Lyashkov]

it's bug in reconnect logic, we are disconnect an rev import but lack an invalidate list of requests on import.
in that case lock cancel callback may still in sending queue with waiting sec context state and don't killed at all but it's need some network flap also.
i have crash dump with such situation.

[Niu Yawei (Inactive)]

This looks like a race: when the export is being destroyed, it'll kill the imp_sec on it's reverse import, and in the meantime, there is still some inflight requests on it's reverse import, which triggered the error message of "import_sec_validate_get() import ... with no sec". I don't think it could cause any serious damange so far.

[Ned Bass (Inactive)]

I don't think we specify any security flavor, so it should be null.

[Niu Yawei (Inactive)]

I'm not familiar with lustre security. Ned, do you know what kind of security flavor was specified for the cluster? Thanks.

[Peter Jones]

Niu

Could you please comment on this one?

Thanks

Peter