nasa-mtd_open-v1.patch
|
Related |
|||
| is related to |
|
Interop 2.3.0<->2.5 failure on test s... | Resolved |
[LU-4179] LBUG ASSERTION( !lustre_handle_is_used(&lhc->mlh_reg_lh) ) failed: Created: 29/Oct/13 Updated: 31/Dec/13 Resolved: 03/Dec/13 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.4.0, Lustre 2.4.1 |
| Fix Version/s: | Lustre 2.6.0, Lustre 2.4.2, Lustre 2.5.1 |
| Type: | Bug | Priority: | Critical |
| Reporter: | Mahmoud Hanafi | Assignee: | Di Wang |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | server | ||
| Environment: |
our source is at git://github.com/jlan/lustre-nas.git |
||
| Attachments: |
nasa-mtd_open-v1.patch
|
||||||||
| Issue Links: |
|
||||||||
| Severity: | 3 | ||||||||
| Epic: | server | ||||||||
| Rank (Obsolete): | 11313 | ||||||||
| Description |
|
We have the crash dumps. But requires analysis by US personal only. LustreError: 6425:0:(mdt_open.c:1690:mdt_reint_open()) ASSERTION( !lustre_handle_is_used(&lhc->mlh_reg_lh) ) failed: ^M |
| Comments |
| Comment by Mahmoud Hanafi [ 29/Oct/13 ] |
|
Correction Effected version should be 2.4 not 2.1.5 |
| Comment by Peter Jones [ 29/Oct/13 ] |
|
Keith is a US citizen and so should be ok to look over the crash dump. |
| Comment by Keith Mannthey (Inactive) [ 29/Oct/13 ] |
|
I have sent email asking for the steps to access the crashdump. |
| Comment by Keith Mannthey (Inactive) [ 29/Oct/13 ] |
|
Can you confirm both Lustre Server and Client Version? |
| Comment by Mahmoud Hanafi [ 29/Oct/13 ] |
|
clients are 2.1.5 and a few 2.3 See git://github.com/jlan/lustre-nas.git for our source tree. I'll will try to get you the crash dumps. |
| Comment by Jay Lan (Inactive) [ 30/Oct/13 ] |
|
If you are looking for source at github, |
| Comment by Keith Mannthey (Inactive) [ 31/Oct/13 ] |
|
Is DNE being used? How often are you hitting this issue? Can you say when it started? There is no easy answer from the crashdump and it is in a complicated area of the code. I have unwound the stack and have the struct mdt_thread_info *info being passed into mdt_reint_open. I am going to see if I can find the mdt_lock_handle tomorrow and continue to dig into this issue. |
| Comment by Keith Mannthey (Inactive) [ 31/Oct/13 ] |
|
I was able to extract a good Lustre Debug log from the crash. The system was busy with service.c:1079:ptlrpc_update_export_timer() during the time of the error. I don't know if this is important or not yet. |
| Comment by Mahmoud Hanafi [ 31/Oct/13 ] |
|
We not using DNE. We are hitting this bug at least once a day. So this is number one on our priority list to be fixed. |
| Comment by Keith Mannthey (Inactive) [ 01/Nov/13 ] |
|
I have attached a possible fix from Di Wang. Please test it and report back. If you get an LASSERT with this patch applied please send a fresh crashdump. |
| Comment by Keith Mannthey (Inactive) [ 01/Nov/13 ] |
|
A patch against master can be tracked here: http://review.whamcloud.com/8142 |
| Comment by Jay Lan (Inactive) [ 01/Nov/13 ] |
|
I applied the patch against 2.4.0 and built. Hit build errors: /usr/src/redhat/BUILD/lustre-2.4.0/lustre/mdt/mdt_open.c: In function 'mdt_reint_open': |
| Comment by Keith Mannthey (Inactive) [ 01/Nov/13 ] |
|
Sorry Jay, Let me work with this patch some more. We see this with the Master build as well. |
| Comment by Di Wang [ 01/Nov/13 ] |
|
Hmm, the patch post on http://review.whamcloud.com/8142 missing a "}" diff --git a/lustre/mdt/mdt_open.c b/lustre/mdt/mdt_open.c
index b4057a0..570d07c 100644
--- a/lustre/mdt/mdt_open.c
+++ b/lustre/mdt/mdt_open.c
@@ -1841,17 +1841,18 @@ int mdt_reint_open(struct mdt_thread_info *info, struct mdt_lock_handle *lhc)
}
}
- LASSERT(!lustre_handle_is_used(&lhc->mlh_reg_lh));
-
- /* get openlock if this is not replay and if a client requested it */
- if (!req_is_replay(req)) {
- rc = mdt_object_open_lock(info, child, lhc, &ibits);
- if (rc != 0)
- GOTO(out_child_unlock, result = rc);
- else if (create_flags & MDS_OPEN_LOCK)
- mdt_set_disposition(info, ldlm_rep, DISP_OPEN_LOCK);
+ if (lustre_handle_is_used(&lhc->mlh_reg_lh)) {
+ LASSERT((lustre_msg_get_flags(req->rq_reqmsg) & MSG_RESENT));
+ } else {
+ /* get openlock if this is not replay and if a client requested it */
+ if (!req_is_replay(req)) {
+ rc = mdt_object_open_lock(info, child, lhc, &ibits);
+ if (rc != 0)
+ GOTO(out_child_unlock, result = rc);
+ else if (create_flags & MDS_OPEN_LOCK)
+ mdt_set_disposition(info, ldlm_rep, DISP_OPEN_LOCK);
+ }
}
-
/* Try to open it now. */
rc = mdt_finish_open(info, parent, child, create_flags,
created, ldlm_rep);
But please hold off a bit to try, since I need revisit these code a bit, Thanks! |
| Comment by Di Wang [ 02/Nov/13 ] |
|
I just updated the patch http://review.whamcloud.com/#/c/8142/ , please try again. Thanks. |
| Comment by Jay Lan (Inactive) [ 02/Nov/13 ] |
|
Hi Di, The new patch can not be applied to 2.4.0. Your new patch contains this line: Could you write a patch that is applicable on top of 2.4.0? Thanks! |
| Comment by Di Wang [ 02/Nov/13 ] |
|
http://review.whamcloud.com/8145 Here is the one for b2_4, and the patch should be able to applied to 2.4.0 as well, please try. |
| Comment by Peter Jones [ 02/Nov/13 ] |
|
Jay Just to be clear - please only use the patch in a test environment until we have completed our validation Thanks Peter |
| Comment by Mahmoud Hanafi [ 04/Nov/13 ] |
|
What is the risk with this patch. We can't fully test the patch using test env because we didn't isolate trigger. We would need to deploy it on production to fully test it. |
| Comment by Di Wang [ 04/Nov/13 ] |
|
IMHO, the patch itself is fine, but we can not say it is "safe', until it went through all of our validation processes, i.e. reviewing, passing internal tests and landing, as peter mentioned. |
| Comment by Di Wang [ 05/Nov/13 ] |
|
http://review.whamcloud.com/8145 has the wrong tag "based on master), actually that patch is based on b2_4, so I resubmit a new one http://review.whamcloud.com/8173 please track this one. |
| Comment by Jay Lan (Inactive) [ 05/Nov/13 ] |
|
Hmm, I had no problem patching and building based on I compared one in 8145 with one in 8173, the patches seemed to be identical to me? |
| Comment by Di Wang [ 05/Nov/13 ] |
|
Yes, they are same, but just some tag issue. If you just pull out the patch and build yourself, you can ignore it. |
| Comment by Jian Yu [ 26/Nov/13 ] |
|
Lustre client: http://build.whamcloud.com/job/lustre-b2_4/57/ racer test also hit this LBUG: |
| Comment by Mahmoud Hanafi [ 03/Dec/13 ] |
|
Patched applied and haven't see this bug. Please close |
| Comment by Peter Jones [ 03/Dec/13 ] |
|
Great - thanks Mahmoud! |