Lai
Could you please look into this one as your top priority?
Thanks
Peter
I made a quick look and think waitq using in seq_client_alloc_fid() is suspicious:
rc = seq_fid_alloc_prep(seq, &link);
if (rc)
continue;
rc = seq_client_alloc_seq(seq, &seqnr);
if (rc) {
CERROR("%s: Can't allocate new sequence, "
"rc %d\n", seq->lcs_name, rc);
cfs_up(&seq->lcs_sem);
RETURN(rc);
}
if we got error returned by seq_client_alloc_seq(), we will leave @link on waitq (added by seq_fid_alloc_seq) and return, but @link is in stack of this function which can be overwritten anytime.
FYI
Liang
Yes, this looks to be a bug, but maloo is not accessible at the moment, and I can't make certain whether this is the cause.
I didn't find any error message for seq_client_alloc_seq() failure, so it's not the cause for this panic. After checking seq_fid_alloc_prep():
static int seq_fid_alloc_prep(struct lu_client_seq *seq,
cfs_waitlink_t *link)
{
cfs_waitq_add(&seq->lcs_waitq, link);
if (seq->lcs_update) {
cfs_up(&seq->lcs_sem);
cfs_set_current_state(CFS_TASK_UNINT);
cfs_waitq_wait(link, CFS_TASK_UNINT);
cfs_set_current_state(CFS_TASK_RUNNING);
cfs_down(&seq->lcs_sem);
return -EAGAIN;
}
++seq->lcs_update;
cfs_up(&seq->lcs_sem);
return 0;
}
Once seq is updating, current thread will schedule and try again later, however it doesn't remove itself from waitq, therefore it may add itself more than once, and cause the panic.
review is on http://review.whamcloud.com/#change,1216
After the upgrading, running metabench test also hit the similar issue:
https://maloo.whamcloud.com/test_sets/955e4d74-c4bc-11e0-9a20-52540025f9af
I also got such list_add corruption error while running the interop tests locally, looks it's quite easy to trigger this bug, and it's not an interop bug actually, so I'm confused that why it was not hit in 2.1 daily tests, any idea?
I also feel a bit strange of it. One possible cause might be that upgrading is tested with physical machine, while autotest using VM. And this panic is triggered by parallel metadata operation test like metabench (multiple threads race on fid seq allocation), which is easier to occur on physical machines.
Integrated in 
lustre-master » x86_64,client,el5,ofa #259
LU-586 1.8 <-> 2.1 interop: list_add corruption
Oleg Drokin : 2807ae2826aa10aca69beaa924f69093326f8351
Files :
- lustre/fid/fid_request.c
Integrated in lustre-master » i686,client,el6,inkernel #259
LU-586 1.8 <-> 2.1 interop: list_add corruption
Oleg Drokin : 2807ae2826aa10aca69beaa924f69093326f8351
Files :
- lustre/fid/fid_request.c
Integrated in lustre-master » x86_64,server,el6,inkernel #259
LU-586 1.8 <-> 2.1 interop: list_add corruption
Oleg Drokin : 2807ae2826aa10aca69beaa924f69093326f8351
Files :
- lustre/fid/fid_request.c
Integrated in lustre-master » x86_64,client,el5,inkernel #259
LU-586 1.8 <-> 2.1 interop: list_add corruption
Oleg Drokin : 2807ae2826aa10aca69beaa924f69093326f8351
Files :
- lustre/fid/fid_request.c
Integrated in lustre-master » x86_64,client,el6,inkernel #259
LU-586 1.8 <-> 2.1 interop: list_add corruption
Oleg Drokin : 2807ae2826aa10aca69beaa924f69093326f8351
Files :
- lustre/fid/fid_request.c
Integrated in lustre-master » x86_64,server,el5,ofa #259
LU-586 1.8 <-> 2.1 interop: list_add corruption
Oleg Drokin : 2807ae2826aa10aca69beaa924f69093326f8351
Files :
- lustre/fid/fid_request.c
Integrated in lustre-master » i686,client,el5,inkernel #259
LU-586 1.8 <-> 2.1 interop: list_add corruption
Oleg Drokin : 2807ae2826aa10aca69beaa924f69093326f8351
Files :
- lustre/fid/fid_request.c
Integrated in lustre-master » i686,server,el5,ofa #259
LU-586 1.8 <-> 2.1 interop: list_add corruption
Oleg Drokin : 2807ae2826aa10aca69beaa924f69093326f8351
Files :
- lustre/fid/fid_request.c
Integrated in lustre-master » i686,server,el6,inkernel #259
LU-586 1.8 <-> 2.1 interop: list_add corruption
Oleg Drokin : 2807ae2826aa10aca69beaa924f69093326f8351
Files :
- lustre/fid/fid_request.c
Integrated in lustre-master » x86_64,client,sles11,inkernel #259
LU-586 1.8 <-> 2.1 interop: list_add corruption
Oleg Drokin : 2807ae2826aa10aca69beaa924f69093326f8351
Files :
- lustre/fid/fid_request.c
Integrated in lustre-master » x86_64,client,ubuntu1004,inkernel #259
LU-586 1.8 <-> 2.1 interop: list_add corruption
Oleg Drokin : 2807ae2826aa10aca69beaa924f69093326f8351
Files :
- lustre/fid/fid_request.c
Integrated in lustre-master » i686,client,el5,ofa #259
LU-586 1.8 <-> 2.1 interop: list_add corruption
Oleg Drokin : 2807ae2826aa10aca69beaa924f69093326f8351
Files :
- lustre/fid/fid_request.c
Integrated in lustre-master » i686,server,el5,inkernel #259
LU-586 1.8 <-> 2.1 interop: list_add corruption
Oleg Drokin : 2807ae2826aa10aca69beaa924f69093326f8351
Files :
- lustre/fid/fid_request.c
Integrated in lustre-master » x86_64,server,el5,inkernel #259
LU-586 1.8 <-> 2.1 interop: list_add corruption
Oleg Drokin : 2807ae2826aa10aca69beaa924f69093326f8351
Files :
- lustre/fid/fid_request.c