Details
-
Bug
-
Resolution: Fixed
-
Major
-
Lustre 2.6.0
-
None
-
3
-
12491
Description
This issue was created by maloo for Di Wang <di.wang@whamcloud.com>
This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/9794753a-8b8b-11e3-b099-52540035b04c.
The sub-test test_17n failed with the following error:
create remote dir error 0
Info required for matching: sanity 17n
Attachments
Issue Links
Activity
So according to the discussion, we have two options.
1. Resend seq lookup RPC if the LWP is evicted.
2. Or not evict LWP during reconnect after reboot, but then it needs quota to set special flag not to replay the quota lock during recovery.
right now I will go with option 1. Niu, could you please comment on 2.
Here is the update patch http://review.whamcloud.com/9106
further discussion with Johann
[2/5/14, 1:17:05 PM] Johann Lombardi: which are hold as long unused space is hold by each quota slave [2/5/14, 1:17:32 PM] Johann Lombardi: if the MDT reboots, those locks should be cancelled and new one with correct attributes (in the lvb) should be fetched [2/5/14, 1:17:54 PM] Johann Lombardi: the quota code relies on eviction to clean up all those locks [2/5/14, 1:18:07 PM] Johann Lombardi: on lock revocation, a resync process is triggered [2/5/14, 1:18:21 PM] Johann Lombardi: to resync quota space hold by each slave with the master [2/5/14, 1:18:34 PM] Johann Lombardi: and new locks are acquired if needed in this case [2/5/14, 1:18:52 PM] Johann Lombardi: the issue if you allow reconnect to make it when the target restarts [2/5/14, 1:19:08 PM] Johann Lombardi: is that those locks will remain there and won't be cancelled [2/5/14, 1:19:22 PM] Johann Lombardi: so no quota resync process will be triggered on the slave [2/5/14, 1:19:45 PM] Johann Lombardi: the resync process is called reconciliation or reintegration [2/5/14, 1:20:08 PM] Johann Lombardi: is that clear? [2/5/14, 1:20:18 PM] wangdi: yes [2/5/14, 1:20:31 PM] wangdi: so eviction is necessary for quota [2/5/14, 1:20:54 PM] Johann Lombardi: well, we at least need locks to be cancelled [2/5/14, 1:21:25 PM] Johann Lombardi: do you know what will happen to lock if you allow the reconnection now? [2/5/14, 1:21:34 PM] wangdi: are there any other event can trigger this, instead of eviction? [2/5/14, 1:21:38 PM] Johann Lombardi: client won't replay them, right? [2/5/14, 1:21:47 PM] wangdi: no [2/5/14, 1:21:53 PM] Johann Lombardi: so we will have troubles anyway [2/5/14, 1:22:05 PM] Johann Lombardi: with locks that exists on the client and not on the server [2/5/14, 1:23:00 PM] Johann Lombardi: i think the lwp has its own lock ns on the client so we could cancel all those locks manually [2/5/14, 1:23:17 PM] Johann Lombardi: but this only works when we can clearly make the difference between regular reconnect [2/5/14, 1:23:23 PM] Johann Lombardi: and reconnect after restart [2/5/14, 1:23:30 PM] Johann Lombardi: this is becoming complicated .... [2/5/14, 1:24:40 PM] wangdi: there is an initial connect flag in the connect request [2/5/14, 1:24:53 PM] wangdi: I think this is easy to tell on the server side [2/5/14, 1:25:02 PM] Johann Lombardi: yes, but this is useful on the server side only [2/5/14, 1:25:11 PM] Johann Lombardi: right, we need to find this out on the client now too [2/5/14, 1:25:49 PM] wangdi: server will return recovering flag in this case “reconnect after restart” [2/5/14, 1:27:13 PM] Johann Lombardi: ok, this might work then. we could simulate a local eviction w/o killing the delayed list, i think [2/5/14, 1:28:13 PM] wangdi: where is this code? could you please point it to me? [2/5/14, 1:28:26 PM] Johann Lombardi: which one? [2/5/14, 1:29:19 PM] wangdi: ah, forget it [2/5/14, 1:30:15 PM] wangdi: so quota needs to simulate a local eviction [2/5/14, 1:30:22 PM] wangdi: instead of LWP [2/5/14, 1:30:27 PM] wangdi: that is what you mean right? [2/5/14, 1:30:58 PM] Johann Lombardi: well, the trick is that all lock replay is done in ptlrpc itself [2/5/14, 1:31:11 PM] Johann Lombardi: i was thinking that we might have to patch ptlrpc actually [2/5/14, 1:31:15 PM] Johann Lombardi: which isn't nice [2/5/14, 1:32:15 PM] Johann Lombardi: if the ptlrpc client gets the recovery flag on regular reconnect, it should still kill all locks [2/5/14, 1:32:16 PM] wangdi: hmm, the point is that I do not want evict LWP using by fld, but only for quota [2/5/14, 1:32:17 PM] Johann Lombardi: like a regular eviction [2/5/14, 1:32:26 PM] Johann Lombardi: understood [2/5/14, 1:33:28 PM] wangdi: sounds like a hack to me [2/5/14, 1:36:19 PM] Johann Lombardi: well, if we want to allow lwp to reconnect, we have to clean up locks [2/5/14, 1:36:26 PM] Johann Lombardi: since those latters won't be replayed [2/5/14, 1:36:39 PM] Johann Lombardi: and we can't afford to be out of sync between client & server [2/5/14, 1:37:07 PM] Johann Lombardi: it is a hack to fix holes of another hack :) [2/5/14, 1:37:47 PM] wangdi: probably LWP is being over-used [2/5/14, 1:38:28 PM] Johann Lombardi: indeed [2/5/14, 1:39:19 PM] wangdi: probably just resend seq lookup RPC, not touch this thing [2/5/14, 1:39:38 PM] wangdi: but we will see other problem here later [2/5/14, 1:39:46 PM] Johann Lombardi: like? [2/5/14, 1:41:09 PM] wangdi: I mean if LWP can not recovery itself through failover, it will cause application error during recovery [2/5/14, 1:41:40 PM] Johann Lombardi: well, the initial purpose of LWP was to do no recovery [2/5/14, 1:42:03 PM] Johann Lombardi: just to provide very basic connection [2/5/14, 1:42:25 PM] wangdi: not for seq allocation and lookup [2/5/14, 1:42:46 PM] Johann Lombardi: right [2/5/14, 1:42:55 PM] wangdi: I guess the user need to maintain the recovery thing itself [2/5/14, 1:43:10 PM] Johann Lombardi: we just used it with different idea of how it would work .... [2/5/14, 1:43:23 PM] wangdi: but then it is no point for sequence to use this LWP at all [2/5/14, 1:43:43 PM] Johann Lombardi: yup, it is mostly to share the conn and avoid adding more [2/5/14, 1:44:08 PM] wangdi: it is not good for sequence. hmm [2/5/14, 1:44:36 PM] wangdi: sigh, we did not think through when implement this. [2/5/14, 1:45:26 PM] Johann Lombardi: initially, we were using a different approach for our quota locks [2/5/14, 1:45:32 PM] Johann Lombardi: maybe we could reconsider it [2/5/14, 1:45:43 PM] Johann Lombardi: we actually allowed lock replay on lwp [2/5/14, 1:45:56 PM] Johann Lombardi: and added a new lock flag on all the quota locks [2/5/14, 1:46:04 PM] Johann Lombardi: to tell that they should not be replayed [2/5/14, 1:46:11 PM] Johann Lombardi: it was working well [2/5/14, 1:46:19 PM] Johann Lombardi: we removed this when we switched to lwp [2/5/14, 1:46:24 PM] wangdi: it will save my trouble. hehe [2/5/14, 1:46:42 PM] wangdi: easy to redo this? [2/5/14, 1:46:47 PM] Johann Lombardi: well, we can talk to Niu on Friday, but i think it is not very difficult to redo this [2/5/14, 1:46:57 PM] Johann Lombardi: it is just a local dlm flag [2/5/14, 1:47:15 PM] Johann Lombardi: and a fix in the lock replay loop [2/5/14, 1:47:23 PM] Johann Lombardi: to cancel the ones with this flag set [2/5/14, 1:48:53 PM] wangdi: ok, sounds good. [2/5/14, 1:50:13 PM] wangdi: so we actually do not need this eviction at all [2/5/14, 1:50:28 PM] wangdi: if it is not for lock cancellation [2/5/14, 1:50:38 PM] Johann Lombardi: well, this eviction was a way for us to: [2/5/14, 1:50:50 PM] Johann Lombardi: - have the locks cancelled [2/5/14, 1:51:14 PM] Johann Lombardi: - have all requests associated with previous connection instance aborted [2/5/14, 1:51:23 PM] Johann Lombardi: but we could do this with different means [2/5/14, 1:51:39 PM] wangdi: but all requests are synchronized with LWP right? [2/5/14, 1:52:02 PM] Johann Lombardi: what do you mean? [2/5/14, 1:52:07 PM] wangdi: no except lock, we do not need care other inflight RPC [2/5/14, 1:52:34 PM] Johann Lombardi: well, we could make sure they are dropped with rq_no_delay [2/5/14, 1:52:48 PM] wangdi: oh, yes [2/5/14, 1:53:04 PM] Johann Lombardi: that's what i meant by "other means" [2/5/14, 1:53:20 PM] wangdi: which way is cleaner? [2/5/14, 1:54:23 PM] Johann Lombardi: well, 1.5 years ago, i thought that doing this through eviction of lwp is cleaner :) [2/5/14, 1:54:36 PM] Johann Lombardi: but i am fine to change this if this is causing you troubles [2/5/14, 1:54:39 PM] wangdi: I think LWP is light weight, but it is allowed to be replay, so eviction might not be good [2/5/14, 1:55:20 PM] wangdi: I can fix this in other way, but I thought the other way is dirty on my side. [2/5/14, 1:55:44 PM] Johann Lombardi: well, let's chat with Niu on Friday if this can wait till then. [2/5/14, 1:56:01 PM] Johann Lombardi: i think it is fixable on our side w/o too much work/hack
With the normal MDT->OST connections, if the UUID is the same (which it should be, since it is using the MDT target name) it will allow it to reconnect to the same export. Should that also be the case for LWP connections?
Since LWP does not write anything into last rcvd, it does not remember any connection, during restart. The MDT can not recognize the old connection(if the reconnection is for the same export) at all. The patch(9106) tries to treat such reconnection as the old export, then not to evict the import. But unfortunately, the quota relies on the eviction to cancel the local lock. So it seems we can not remove the eviction easily. hmm.
discussion on Skype
[2/5/14, 9:19:26 AM] wangdi: if we just evict the LWP, it might cause some in-flight RPC failed [2/5/14, 9:19:40 AM] Johann Lombardi: yup [2/5/14, 9:19:41 AM] wangdi: which then cause application error during recovery [2/5/14, 9:19:44 AM] Johann Lombardi: yes [2/5/14, 9:19:54 AM] Johann Lombardi: that's what the quota code expects [2/5/14, 9:20:03 AM] wangdi: which then cause some test failure, [2/5/14, 9:20:13 AM] Johann Lombardi: all the states associated with the previous "connection" should go away [2/5/14, 9:20:47 AM] Johann Lombardi: could you please remind me how you use LWP? [2/5/14, 9:20:59 AM] Johann Lombardi: for seq allocation mostly, right? [2/5/14, 9:21:15 AM] wangdi: no, mostly for fld(seq) lookup [2/5/14, 9:21:35 AM] wangdi: seq allocation is rare actually [2/5/14, 9:21:38 AM] Johann Lombardi: ah, then the lack of recovery should not be a problem [2/5/14, 9:21:45 AM] Johann Lombardi: you can just resend the request yourself [2/5/14, 9:22:06 AM] wangdi: that would complicate thing [2/5/14, 9:22:24 AM] wangdi: I am thinking some easy way. :) [2/5/14, 9:22:38 AM] Johann Lombardi: well, the purpose of lwp is to be lightweight [2/5/14, 9:22:52 AM] Johann Lombardi: as such, it has no entry in last_rcvd [2/5/14, 9:22:58 AM] Johann Lombardi: and gets evicted upon restart [2/5/14, 9:23:07 AM] wangdi: I understand this. [2/5/14, 9:23:14 AM] Johann Lombardi: it is like this by design [2/5/14, 9:23:54 AM] wangdi: quota need this evict thing? or just design [2/5/14, 9:24:01 AM] wangdi: if there are no replay [2/5/14, 9:25:06 AM] wangdi: why do we need this eviction. [2/5/14, 9:25:24 AM] Johann Lombardi: quota needs all outstanding requests associated with previous connection to fail, yes [2/5/14, 9:25:31 AM] wangdi: I see [2/5/14, 9:25:36 AM] wangdi: thanks [2/5/14, 9:25:46 AM] Johann Lombardi: well, the eviction is a side effect of not being in the last_rcvd file [2/5/14, 9:26:15 AM] Johann Lombardi: quota updates are not synchronous to disk and not replayed [2/5/14, 9:26:28 AM] wangdi: if the eviction can chose not to fail those read-only in-flight RPC [2/5/14, 9:26:32 AM] wangdi: that would be best [2/5/14, 9:26:35 AM] Johann Lombardi: when the mdt restarts, it might have a different view of allocation [2/5/14, 9:26:49 AM] Johann Lombardi: so we should restart from scratch each time [2/5/14, 9:27:00 AM] Johann Lombardi: and forget about previous requests that did not complete [2/5/14, 9:27:09 AM] Johann Lombardi: because there are not relevant any more [2/5/14, 9:27:22 AM] Johann Lombardi: i guess we could change the behavior a bit [2/5/14, 9:27:42 AM] Johann Lombardi: and not fail requests which are in the delayed list [2/5/14, 9:28:16 AM] Johann Lombardi: quota could then set the no_delay flag (if not done already) to bypass this behavior [2/5/14, 9:28:47 AM] Johann Lombardi: however, i guess it will require to patch ptlrpc no to fail those requests on eviction which will happen anyway since we have not entry in the last_rcvd file [2/5/14, 9:28:50 AM] Johann Lombardi: what do you think? [2/5/14, 9:29:31 AM] wangdi: the purpose of eviction is for the "replay" req, not for delay req in this case. [2/5/14, 9:29:55 AM] wangdi: so probably change ptlrpc makes sense, but that would complicate the protocol [2/5/14, 9:29:57 AM] wangdi: I hate that [2/5/14, 9:30:00 AM] Johann Lombardi: well, requests from the delayed list will be failed too [2/5/14, 9:30:08 AM] Johann Lombardi: in the case of eviction [2/5/14, 9:30:24 AM] wangdi: yes, but it should not for lightweight connection [2/5/14, 9:30:54 AM] Johann Lombardi: yes, so you could change this behavior and set rq_no_delay on quota request (against, if not done already) [2/5/14, 9:31:12 AM] Johann Lombardi: yours will be handled [2/5/14, 9:32:15 AM] wangdi: yes, probably [2/5/14, 9:32:27 AM] wangdi: need think a bit. thanks for the discussion. [2/5/14, 9:32:27 AM] Johann Lombardi: i don't see any other way to do that [2/5/14, 9:32:44 AM] Johann Lombardi: except adding lwp to last_rcvd [2/5/14, 9:32:52 AM] Johann Lombardi: which would defeat the original purpose [2/5/14, 9:33:03 AM] Johann Lombardi: np [2/5/14, 9:33:40 AM] wangdi: the other way is to resend the fld rpc as you said, but that would complicate thing [2/5/14, 9:34:17 AM] Johann Lombardi: it might, indeed [2/5/14, 10:32:30 AM] wangdi: set no_delay will cause quota RPC failed un any unhealthy connection. not just restart. [2/5/14, 10:32:35 AM] wangdi: is it ok for quota? [2/5/14, 10:48:13 AM] wangdi: hmm, we already set no_delay flag for quota request [2/5/14, 10:48:27 AM] wangdi: so I guess this eviction is unnecessary anymore? [2/5/14, 10:48:49 AM] wangdi: I will disable part of recovery-small test then
could you please look at
LU-4571, I do not understand why LWP needs to be evicted after restart. (recovery-small.sh 106), which basically make the fix onLU-4571fails on this test. please post here, if you remember sth. Thanks
The purpose of LWP was to have a stateless connection (no entry in last_rcvd file, no lock replay, ...).
What's exactly the problem?
patch landed to master