[#LU-352] obdecho don't work with verify mode

[LU-352] obdecho don't work with verify mode Created: 22/May/11 Updated: 20/Nov/12 Resolved: 23/Aug/12
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 2.0.0, Lustre 2.1.0
Fix Version/s:	Lustre 2.3.0

Type:

Bug

Priority:

Major

Reporter:

Alexey Lyashkov

Assignee:

Hongchao Zhang

Resolution:

Fixed

Votes:

Labels:

None

Environment:

RHEL6, or RHEL5 with any lustre 2.x

Issue Links:

Related
is related to	~~LU-1400~~	obdfilter-survey test_1c failed but s...	Resolved

Severity:

Rank (Obsolete):

4658

Description

obdecho doesn't work on verify mode.

[root@rh5-64 utils]# [root@rh5-64 obdfilter-survey]# ./obdfilter-survey
Mon May 23 07:38:32 EEST 2011 Obdfilter-survey for case=disk from rh5-64.shadowland
ost 2 sz 33554432K rsz 1024K obj 2 thr 2 write Terminated
[root@rh5-64 obdfilter-survey]# verify=1 ./obdfilter-survey
Mon May 23 07:39:01 EEST 2011 Obdfilter-survey for case=disk from rh5-64.shadowland
ost 2 sz 33554432K rsz 1024K obj 2 thr 2 write 1336928.79 ERROR rewrite 1075489.49 ERROR read

LustreError: 19113:0:(debug.c:119:block_debug_check()) test_brw: id 0x2 offset 1040384 end id: 0xdeadbeef00c0ffee != 0x2
LustreError: 19113:0:(debug.c:102:block_debug_check()) test_brw: id 0x2 offset 1044480 off: 0xdeadbeef00c0ffee != 0xff000
LustreError: 19113:0:(debug.c:107:block_debug_check()) test_brw: id 0x2 offset 1044480 id: 0xdeadbeef00c0ffee != 0x2
LustreError: 19113:0:(debug.c:114:block_debug_check()) test_brw: id 0x2 offset 1044480 end off: 0xdeadbeef00c0ffee != 0xff000

Comments

Comment by Jinshan Xiong (Inactive) [ 22/May/11 ]

Hi Shadow - long time no see.
After checking the code, I saw the verification only happens in read. However, I notice that there seemed erroneous on write phase from your log, let's make sure the write works firstly.

Comment by Alexey Lyashkov [ 23/May/11 ]

Hi Jay

Thanks for review a obdecho code. I will look to obdecho sources and try to fix it, because it's blocking bug for us.

Comment by Alexey Lyashkov [ 24/May/11 ]

I spent some time to make investigation.
That badnes a result of r/o cache landing, so now obd_preprw allocate a new pages for read from mapping and that is overwrite a page pointers submited from obdecho.
Also it's broke a 0-copy testing.

Comment by Alexey Lyashkov [ 24/May/11 ]

In fact fix is easy
lctl / obdecho has a two cases to work with lower fs. one is prep + commit, second is brw.

BRW methods is broked for obdecho and for 'disk' case we need to choose a prep + commit method.

Comment by Peter Jones [ 25/May/11 ]

Niu

Could you please look at this one when you are able to

Thanks

Peter

Comment by Alexey Lyashkov [ 25/May/11 ]

Peter, i have submitted fix to Gerit.
http://review.whamcloud.com/#change,598

Comment by Peter Jones [ 25/May/11 ]

Thanks Shadow - I saw. However, even if we have a contributed fix, someone still needs to be responsible for checking it gets tested, inspected and landed. I just noticed that HongChao is already working on this area of code for another ticket, so reassigning to him.

Comment by Jinshan Xiong (Inactive) [ 25/May/11 ]

Hi Shadow,

Can you please tell me a bit more about where is the problem? Is the problem happening at read or write phase?

Comment by Alexey Lyashkov [ 25/May/11 ]

i think on both.

without extra changes in obdfilter-survey,
obdecho uses a obd_brw() call to read or to write a data to the obdfilter device.
but after landing R/O cache obd_preprw don't want to have allocated pages from TLS, and allocated pages himself. So pointers to pages submitted by obdecho is replaced by allocated in obd_preprw and none data copped to obdecho module.
in that case obdecho see a poisoned buffer
LustreError: 19113:0:(debug.c:119:block_debug_check()) test_brw: id 0x2 offset 1040384 end id: 0xdeadbeef00c0ffee != 0x2
instead of data from write phase.

that can be fixed by two ways.
one way is adding a special parameter in lctl test_brw command to tell obdecho uses obd_preprw + obd_commitrw instead of obd_brw().
second way to add a extra check in obdecho module "is lower device is obdfilter" and force to use a obd_preprw + obd_commitrw in that case.

My submitted fix implement a both of them.

Jay it's enough ?

Comment by Jinshan Xiong (Inactive) [ 25/May/11 ]

Indeed.

It seems filter_brw() won't work at all, because the page pointer will be cleared by filter_preprw anyway. It seems more reasonable to me to fix this problem by revising filter_preprw(). What do you think?

Comment by Alexey Lyashkov [ 25/May/11 ]

if you will be fix filter_preprw to use a pages from obdecho you will stop to use a zero-copy feature in mdraid because it is need to have mapping with special flag.

so - i think better to force obdecho to use existent code (obd_preprw + obd_commitrw) for obdfilter, it 3 lines fix in obdecho.

/** obdfilter don't supported obd_brw now, simulate via prep_commit */
test_mode = (long)data->ioc_pbuf1;
if (ed->ed_next == NULL)

{ test_mode = 3; data->ioc_plen1 = data->ioc_count; }

switch(test_mode) {

or you need to force obdfilter-survey to use a 'p' flag in case 'disk'.

+# obdfilter need to use prep+commit mode
+# osc need to use kbrw mode
+testcase2mode()

{ + case $case in + disk) echo "p$1";; + *) echo "g";; + esac +}

test_brw $count `testname2type $test` q $pages ${thr_per_obj}t${first_obj} &"
+ test_brw $count `testname2type $test` q $pages ${thr_per_obj}t${first_obj} `testcase2mode $pages` &"

so you will have same behavior without touch any lines in code.

Comment by Jinshan Xiong (Inactive) [ 25/May/11 ]

In that case, let's return an error code in filter_brw() so that we won't be confused by it again.

Comment by Alexey Lyashkov [ 25/May/11 ]

just killl that method and obd_brw will return ENOSUPP.

Comment by Peter Jones [ 30/Aug/11 ]

http://review.whamcloud.com/#change,598

Comment by Build Master (Inactive) [ 02/Apr/12 ]

Integrated in lustre-reviews » x86_64,client,sles11,inkernel #4630
~~LU-352~~ obdfilter: fix obdfilter-survey to work on verify mode. (Revision f9b01623ff16b17eae22ee5eebb1838a21ef846a)

Result = SUCCESS
alexander_boyko : f9b01623ff16b17eae22ee5eebb1838a21ef846a
Files :

lustre/obdfilter/filter_io.c
lustre/obdecho/echo_client.c
lustre/obdfilter/filter.c
lustre/obdfilter/filter_internal.h
lustre/tests/obdfilter-survey.sh
lustre-iokit/obdfilter-survey/obdfilter-survey

Comment by Build Master (Inactive) [ 02/Apr/12 ]

Integrated in lustre-reviews » x86_64,client,el5,inkernel #4630
~~LU-352~~ obdfilter: fix obdfilter-survey to work on verify mode. (Revision f9b01623ff16b17eae22ee5eebb1838a21ef846a)

Result = SUCCESS
alexander_boyko : f9b01623ff16b17eae22ee5eebb1838a21ef846a
Files :

lustre/obdecho/echo_client.c
lustre/obdfilter/filter.c
lustre/tests/obdfilter-survey.sh
lustre/obdfilter/filter_io.c
lustre/obdfilter/filter_internal.h
lustre-iokit/obdfilter-survey/obdfilter-survey

Comment by Build Master (Inactive) [ 02/Apr/12 ]

Integrated in lustre-reviews » x86_64,client,el6,inkernel #4630
~~LU-352~~ obdfilter: fix obdfilter-survey to work on verify mode. (Revision f9b01623ff16b17eae22ee5eebb1838a21ef846a)

Result = SUCCESS
alexander_boyko : f9b01623ff16b17eae22ee5eebb1838a21ef846a
Files :

lustre/obdfilter/filter_io.c
lustre-iokit/obdfilter-survey/obdfilter-survey
lustre/obdfilter/filter_internal.h
lustre/obdfilter/filter.c
lustre/tests/obdfilter-survey.sh
lustre/obdecho/echo_client.c

Comment by Build Master (Inactive) [ 02/Apr/12 ]

Integrated in lustre-reviews » i686,server,el5,inkernel #4630
~~LU-352~~ obdfilter: fix obdfilter-survey to work on verify mode. (Revision f9b01623ff16b17eae22ee5eebb1838a21ef846a)

Result = SUCCESS
alexander_boyko : f9b01623ff16b17eae22ee5eebb1838a21ef846a
Files :

lustre/obdfilter/filter_io.c
lustre/obdfilter/filter_internal.h
lustre/tests/obdfilter-survey.sh
lustre-iokit/obdfilter-survey/obdfilter-survey
lustre/obdfilter/filter.c
lustre/obdecho/echo_client.c

Comment by Build Master (Inactive) [ 02/Apr/12 ]

Integrated in lustre-reviews » i686,client,el6,inkernel #4630
~~LU-352~~ obdfilter: fix obdfilter-survey to work on verify mode. (Revision f9b01623ff16b17eae22ee5eebb1838a21ef846a)

Result = SUCCESS
alexander_boyko : f9b01623ff16b17eae22ee5eebb1838a21ef846a
Files :

lustre-iokit/obdfilter-survey/obdfilter-survey
lustre/obdfilter/filter_io.c
lustre/obdfilter/filter.c
lustre/tests/obdfilter-survey.sh
lustre/obdfilter/filter_internal.h
lustre/obdecho/echo_client.c

Comment by Build Master (Inactive) [ 02/Apr/12 ]

Integrated in lustre-reviews » x86_64,server,el5,inkernel #4630
~~LU-352~~ obdfilter: fix obdfilter-survey to work on verify mode. (Revision f9b01623ff16b17eae22ee5eebb1838a21ef846a)

Result = SUCCESS
alexander_boyko : f9b01623ff16b17eae22ee5eebb1838a21ef846a
Files :

lustre/obdfilter/filter_io.c
lustre/tests/obdfilter-survey.sh
lustre/obdfilter/filter_internal.h
lustre-iokit/obdfilter-survey/obdfilter-survey
lustre/obdfilter/filter.c
lustre/obdecho/echo_client.c

Comment by Build Master (Inactive) [ 02/Apr/12 ]

Integrated in lustre-reviews » i686,client,el5,inkernel #4630
~~LU-352~~ obdfilter: fix obdfilter-survey to work on verify mode. (Revision f9b01623ff16b17eae22ee5eebb1838a21ef846a)

Result = SUCCESS
alexander_boyko : f9b01623ff16b17eae22ee5eebb1838a21ef846a
Files :

lustre/obdfilter/filter.c
lustre/obdfilter/filter_io.c
lustre/obdfilter/filter_internal.h
lustre/tests/obdfilter-survey.sh
lustre/obdecho/echo_client.c
lustre-iokit/obdfilter-survey/obdfilter-survey

Comment by Build Master (Inactive) [ 02/Apr/12 ]

Integrated in lustre-reviews » x86_64,server,el6,inkernel #4630
~~LU-352~~ obdfilter: fix obdfilter-survey to work on verify mode. (Revision f9b01623ff16b17eae22ee5eebb1838a21ef846a)

Result = SUCCESS
alexander_boyko : f9b01623ff16b17eae22ee5eebb1838a21ef846a
Files :

lustre/obdfilter/filter_io.c
lustre/obdfilter/filter_internal.h
lustre-iokit/obdfilter-survey/obdfilter-survey
lustre/tests/obdfilter-survey.sh
lustre/obdecho/echo_client.c
lustre/obdfilter/filter.c

Comment by Build Master (Inactive) [ 02/Apr/12 ]

Integrated in lustre-reviews » i686,server,el6,inkernel #4630
~~LU-352~~ obdfilter: fix obdfilter-survey to work on verify mode. (Revision f9b01623ff16b17eae22ee5eebb1838a21ef846a)

Result = SUCCESS
alexander_boyko : f9b01623ff16b17eae22ee5eebb1838a21ef846a
Files :

lustre-iokit/obdfilter-survey/obdfilter-survey
lustre/obdfilter/filter_internal.h
lustre/tests/obdfilter-survey.sh
lustre/obdfilter/filter.c
lustre/obdecho/echo_client.c
lustre/obdfilter/filter_io.c