[LU-746] obdfilter-survey FAIL: test_1b ost4: hndls expected > 8, have 2 Created: 10/Oct/11  Updated: 16/Aug/16  Resolved: 16/Aug/16

Status: Closed
Project: Lustre
Component/s: None
Affects Version/s: Lustre 1.8.8, Lustre 1.8.7
Fix Version/s: Lustre 2.2.0, Lustre 1.8.8

Type: Bug Priority: Minor
Reporter: Jian Yu Assignee: Lai Siyao
Resolution: Won't Fix Votes: 0
Labels: None
Environment:

Lustre Branch: b1_8
Lustre Build: http://newbuild.whamcloud.com/job/lustre-b1_8/134/
Distro/Arch: RHEL5/x86_64 (kernel version: 2.6.18_274.3.1.el5)
Network: TCP (1GigE)


Issue Links:
Duplicate
is duplicated by LU-756 Test failure on test suite obdfilter-... Resolved
Severity: 3
Rank (Obsolete): 4749

 Description   

obdfilter-survey test 1b failed as follows:

Sat Oct  8 23:12:57 PDT 2011 Obdfilter-survey for case=disk from client-26vm1.lab.whamcloud.com
ost  7 sz  7340032K rsz 1024K obj    7 thr   28 write   10.21 [   0.00,   6.99] rewrite   16.39 [   0.00,  22.96] read   28.63 [   0.00,  25.96] 
starting run for test: write rsz: 1024 threads: 4 objects: 1
starting run for test: rewrite rsz: 1024 threads: 4 objects: 1
starting run for test: read rsz: 1024 threads: 4 objects: 1
R    1024  0     5000  0     0     137   38     2     3    
R    1025  0     5000  0     0     123   2      16    17   
R    1008  0     5000  0     0     198   11     2     3    
R    1009  0     5000  0     0     172   2      16    17   
R    1009  0     5000  0     0     2117  15     2     3    
R    1010  0     5000  0     0     259   2      16    17   
R    1010  0     5000  0     0     399   2      2     3    
R    1011  0     5000  0     0     175   2      16    17   
 obdfilter-survey test_1b: @@@@@@ FAIL: ost4: hndls expected > 8, have 2 
Dumping lctl log to /logdir/test_logs/2011-10-08/lustre-b1_8-el5-x86_64__134__-7f5c39000658/obdfilter-survey.test_1b.*.1318142207.log
R    1008  0     5000  0     0     1845  3      2     3    
R    1009  0     5000  0     0     233   2      16    17   
 obdfilter-survey test_1b: @@@@@@ FAIL: ost5: hndls expected > 8, have 3 
Dumping lctl log to /logdir/test_logs/2011-10-08/lustre-b1_8-el5-x86_64__134__-7f5c39000658/obdfilter-survey.test_1b.*.1318142217.log
R    1004  0     5000  0     0     1031  1      2     3    
R    1005  0     5000  0     0     180   2      16    17   
 obdfilter-survey test_1b: @@@@@@ FAIL: ost6: hndls expected > 8, have 1 
Dumping lctl log to /logdir/test_logs/2011-10-08/lustre-b1_8-el5-x86_64__134__-7f5c39000658/obdfilter-survey.test_1b.*.1318142222.log
R    998   0     5000  0     0     2262  5      2     3    
R    999   0     5000  0     0     101   2      16    17   
 obdfilter-survey test_1b: @@@@@@ FAIL: ost7: hndls expected > 8, have 5 
Dumping lctl log to /logdir/test_logs/2011-10-08/lustre-b1_8-el5-x86_64__134__-7f5c39000658/obdfilter-survey.test_1b.*.1318142227.log
 obdfilter-survey test_1b: @@@@@@ FAIL: test_1b failed with 8 
Dumping lctl log to /logdir/test_logs/2011-10-08/lustre-b1_8-el5-x86_64__134__-7f5c39000658/obdfilter-survey.test_1b.*.1318142232.log
Resetting fail_loc on all nodes...done.
FAIL   (1499s)

Maloo report: https://maloo.whamcloud.com/test_sets/d09bdfde-f24b-11e0-908b-52540025f9af

 Comments   
Comment by Peter Jones [ 13/Oct/11 ]

Lai

Could you please look into this possible 1.8.7 blocker as your to priority?

Thanks

Peter

Comment by Sarah Liu [ 14/Oct/11 ]

passed on manual test:https://maloo.whamcloud.com/test_sets/b8cb27c6-f6a0-11e0-a451-52540025f9af

Comment by Lai Siyao [ 17/Oct/11 ]

The last brw write ends at time 1318141924:

00002000:00100000:0:1318141924.703756:0:24425:0:(filter.c:198:filter_finish_transno()) wrote trans 21474842626 for client ECHO_UUID at #3: err = 0

The last object destroy ends at time 1318142186:

00002000:00020000:0:1318142186.737229:0:26178:0:(filter.c:1557:filter_destroy_internal()) destroying objid 3 ino 56 nlink 1 count 1
00002000:00100000:0:1318142186.740251:0:26178:0:(filter.c:198:filter_finish_transno()) wrote trans 12884908035 for client ECHO_UUID at #3: err = 0

jbd hndls checked at time 1318142207:

00000001:02000400:0:1318142207.957982:0:27184:0:(debug.c:535:libcfs_debug_mark_buffer()) DEBUG MARKER: obdfilter-survey test_1b: @@@@@@ FAIL: ost4: hndls expected > 8, have 2

obdfilter-survey.sh '1b' test checked jbd history data of the past 15 seconds, but unfortunately in this round of test, system is idle at that period [1318142192, 1318142207], so the stats collected is not the one wanted. I guess this is caused by slow VMs (and this can't be reproduced on physical machines manually by Sarah, see above).

To fix this, we may need to make `check_jbd_values_facets` execute right after brw.

Comment by Lai Siyao [ 18/Oct/11 ]

review is on http://review.whamcloud.com/#change,1534

Comment by Peter Jones [ 03/Nov/11 ]

Lai

Would it be useful to port this patch to master?

Peter

Comment by Lai Siyao [ 03/Nov/11 ]

Peter, yes, I'll port it to master later.

Comment by Minh Diep [ 06/Feb/12 ]

Can we get this patch to land? this bug is preventing lustre-review to have a good run

Comment by Build Master (Inactive) [ 07/Feb/12 ]

Integrated in lustre-b1_8 » x86_64,client,el5,ofa #170
LU-746 obdfilter-survey FAIL: test_1b ost4: hndls expected > 8, have 2 (Revision e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44)

Result = SUCCESS
Johann Lombardi : e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 07/Feb/12 ]

Integrated in lustre-b1_8 » x86_64,server,el5,inkernel #170
LU-746 obdfilter-survey FAIL: test_1b ost4: hndls expected > 8, have 2 (Revision e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44)

Result = SUCCESS
Johann Lombardi : e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 07/Feb/12 ]

Integrated in lustre-b1_8 » x86_64,client,el5,inkernel #170
LU-746 obdfilter-survey FAIL: test_1b ost4: hndls expected > 8, have 2 (Revision e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44)

Result = SUCCESS
Johann Lombardi : e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 07/Feb/12 ]

Integrated in lustre-b1_8 » x86_64,client,el6,inkernel #170
LU-746 obdfilter-survey FAIL: test_1b ost4: hndls expected > 8, have 2 (Revision e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44)

Result = SUCCESS
Johann Lombardi : e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 07/Feb/12 ]

Integrated in lustre-b1_8 » i686,client,el5,inkernel #170
LU-746 obdfilter-survey FAIL: test_1b ost4: hndls expected > 8, have 2 (Revision e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44)

Result = SUCCESS
Johann Lombardi : e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 07/Feb/12 ]

Integrated in lustre-b1_8 » x86_64,client,ubuntu1004,inkernel #170
LU-746 obdfilter-survey FAIL: test_1b ost4: hndls expected > 8, have 2 (Revision e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44)

Result = SUCCESS
Johann Lombardi : e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 07/Feb/12 ]

Integrated in lustre-b1_8 » x86_64,server,el5,ofa #170
LU-746 obdfilter-survey FAIL: test_1b ost4: hndls expected > 8, have 2 (Revision e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44)

Result = SUCCESS
Johann Lombardi : e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 07/Feb/12 ]

Integrated in lustre-b1_8 » i686,client,el5,ofa #170
LU-746 obdfilter-survey FAIL: test_1b ost4: hndls expected > 8, have 2 (Revision e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44)

Result = SUCCESS
Johann Lombardi : e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 07/Feb/12 ]

Integrated in lustre-b1_8 » i686,client,el6,inkernel #170
LU-746 obdfilter-survey FAIL: test_1b ost4: hndls expected > 8, have 2 (Revision e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44)

Result = SUCCESS
Johann Lombardi : e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 07/Feb/12 ]

Integrated in lustre-b1_8 » i686,server,el5,inkernel #170
LU-746 obdfilter-survey FAIL: test_1b ost4: hndls expected > 8, have 2 (Revision e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44)

Result = SUCCESS
Johann Lombardi : e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 07/Feb/12 ]

Integrated in lustre-b1_8 » i686,server,el5,ofa #170
LU-746 obdfilter-survey FAIL: test_1b ost4: hndls expected > 8, have 2 (Revision e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44)

Result = SUCCESS
Johann Lombardi : e93c4dae50fba04b8083a6afc5a7a79b8e4f0a44
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 07/Feb/12 ]

Integrated in lustre-master » x86_64,client,el5,ofa #456
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = SUCCESS
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 07/Feb/12 ]

Integrated in lustre-master » x86_64,client,el5,inkernel #456
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = SUCCESS
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 08/Feb/12 ]

Integrated in lustre-master » x86_64,server,el6,inkernel #456
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = SUCCESS
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 08/Feb/12 ]

Integrated in lustre-master » i686,server,el6,inkernel #456
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = SUCCESS
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 08/Feb/12 ]

Integrated in lustre-master » x86_64,client,el6,inkernel #456
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = SUCCESS
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 08/Feb/12 ]

Integrated in lustre-master » x86_64,server,el5,inkernel #456
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = SUCCESS
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 08/Feb/12 ]

Integrated in lustre-master » x86_64,server,el5,ofa #456
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = SUCCESS
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 08/Feb/12 ]

Integrated in lustre-master » i686,server,el5,ofa #456
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = SUCCESS
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 08/Feb/12 ]

Integrated in lustre-master » i686,client,el6,inkernel #456
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = SUCCESS
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 08/Feb/12 ]

Integrated in lustre-master » x86_64,client,ubuntu1004,inkernel #456
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = SUCCESS
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 08/Feb/12 ]

Integrated in lustre-master » i686,server,el5,inkernel #456
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = SUCCESS
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 08/Feb/12 ]

Integrated in lustre-master » x86_64,client,sles11,inkernel #456
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = SUCCESS
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 08/Feb/12 ]

Integrated in lustre-master » i686,client,el5,inkernel #456
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = SUCCESS
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 08/Feb/12 ]

Integrated in lustre-master » i686,client,el5,ofa #456
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = SUCCESS
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Peter Jones [ 08/Feb/12 ]

Landed for 2.2

Comment by Build Master (Inactive) [ 17/Feb/12 ]

Integrated in lustre-master » x86_64,server,el6,ofa #480
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = FAILURE
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 17/Feb/12 ]

Integrated in lustre-master » x86_64,client,el6,ofa #480
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = FAILURE
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Build Master (Inactive) [ 17/Feb/12 ]

Integrated in lustre-master » i686,client,el6,ofa #480
LU-746 test: obdfilter-survey FAIL hndls expected >8, have 2 (Revision 65ec69f84b6452fd626bc3da980d31a4ef05a00e)

Result = ABORTED
Oleg Drokin : 65ec69f84b6452fd626bc3da980d31a4ef05a00e
Files :

  • lustre/tests/obdfilter-survey.sh
Comment by Jian Yu [ 16/May/12 ]

Lustre Tag: v1_8_8_WC1_RC1
Lustre Build: http://build.whamcloud.com/job/lustre-b1_8/195/
Distro/Arch: RHEL5.8/x86_64(server), RHEL6.2/x86_64(client)
Network: TCP (1GigE)
ENABLE_QUOTA=yes

The issue still occurred: https://maloo.whamcloud.com/test_sets/ee85f612-9d5c-11e1-8587-52540035b04c

Comment by Jian Yu [ 14/Jan/13 ]

Lustre Branch: b1_8
Lustre Build: http://build.whamcloud.com/job/lustre-b1_8/241

The obdfilter-survey test 2b failed as follows:

obd survey finished in 957 seconds
CMD: client-28vm4 dv=\$(lctl get_param -n *.lustre-OST0000.mntdev);
if foo=\$(lvdisplay -c \$dv 2>/dev/null); then
    echo dm-\${foo##*:};
else
    echo \$(basename \$dv);
fi;
CMD: client-28vm4 cat /proc/fs/jbd*/dm-0*/history
R    2298  0     5001  0     0     1709  12     2     3    
CMD: client-28vm4 dv=\$(lctl get_param -n *.lustre-OST0001.mntdev);
if foo=\$(lvdisplay -c \$dv 2>/dev/null); then
    echo dm-\${foo##*:};
else
    echo \$(basename \$dv);
fi;
CMD: client-28vm4 cat /proc/fs/jbd*/dm-1*/history
R    2327  0     5000  0     0     2071  19     2     3    
CMD: client-28vm4 dv=\$(lctl get_param -n *.lustre-OST0002.mntdev);
if foo=\$(lvdisplay -c \$dv 2>/dev/null); then
    echo dm-\${foo##*:};
else
    echo \$(basename \$dv);
fi;
CMD: client-28vm4 cat /proc/fs/jbd*/dm-2*/history
R    2333  0     5004  0     0     2013  20     2     3    
CMD: client-28vm4 dv=\$(lctl get_param -n *.lustre-OST0003.mntdev);
if foo=\$(lvdisplay -c \$dv 2>/dev/null); then
    echo dm-\${foo##*:};
else
    echo \$(basename \$dv);
fi;
CMD: client-28vm4 cat /proc/fs/jbd*/dm-3*/history
R    2331  0     5000  0     0     1119  15     2     3    
CMD: client-28vm4 dv=\$(lctl get_param -n *.lustre-OST0004.mntdev);
if foo=\$(lvdisplay -c \$dv 2>/dev/null); then
    echo dm-\${foo##*:};
else
    echo \$(basename \$dv);
fi;
CMD: client-28vm4 cat /proc/fs/jbd*/dm-4*/history
R    2326  1     4999  0     0     2035  20     2     3    
 obdfilter-survey test_2b: @@@@@@ FAIL: ost5: run expected 5000, have 4999

Maloo report: https://maloo.whamcloud.com/test_sets/83456338-5daa-11e2-8199-52540035b04c

Comment by James A Simmons [ 16/Aug/16 ]

Old ticket for unsupported version

Generated at Sat Feb 10 01:10:01 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.