[LU-479] Test failure on test suite sanity, subtest test_124a

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.1.0
• Type: Bug
• Priority: Minor
• Reporter: Maloo
• Assignee: Lai Siyao
• Resolution: Cannot Reproduce
• Severity: 3

Description

This issue was created by maloo for yujian <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/282aae9a-a451-11e0-b786-52540025f9af.

The sub-test test_124a failed with the following error:

No locks dropped in 50s. LRU size: 2003

Comments

[Jian Yu]

More failure instances:
https://maloo.whamcloud.com/test_sets/4b97a594-a448-11e0-b786-52540025f9af
https://maloo.whamcloud.com/test_sets/c597e884-a44f-11e0-b786-52540025f9af

[Lai Siyao]

Maloo test logs shows:
LRU=2003
LVF=0

Herein LVF should be a big number (around 10000), but it's 0 here, I will print more tunables in the script to find why.

[Andreas Dilger]

This problem is causing quite a lot of sanity test failures lately. Has any investigation been done to determine when this test started failing, and what changes were made around that time?

[Lai Siyao]

This doesn't always fail, and a possible caused may be autotest environment change (moving from physical machines to VM, and using lvm as backend instead of /dev/sdx).

[Peter Jones]

Landed for Lustre 2.1

[Peter Jones]

Oops. I think that the patch landed is actually a diagnostic patch rather than a fix so reopening

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el5,inkernel #199
LU-479 sanity 124a failed

Oleg Drokin : c4cb5e5f0e316f986eae3b11be4f5d81756b85b6
Files :

• lustre/tests/sanity.sh

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el6,inkernel #199
LU-479 sanity 124a failed

Oleg Drokin : c4cb5e5f0e316f986eae3b11be4f5d81756b85b6
Files :

• lustre/tests/sanity.sh

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el6,inkernel #199
LU-479 sanity 124a failed

Oleg Drokin : c4cb5e5f0e316f986eae3b11be4f5d81756b85b6
Files :

• lustre/tests/sanity.sh

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el5,ofa #199
LU-479 sanity 124a failed

Oleg Drokin : c4cb5e5f0e316f986eae3b11be4f5d81756b85b6
Files :

• lustre/tests/sanity.sh

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el5,inkernel #199
LU-479 sanity 124a failed

Oleg Drokin : c4cb5e5f0e316f986eae3b11be4f5d81756b85b6
Files :

• lustre/tests/sanity.sh

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el5,ofa #199
LU-479 sanity 124a failed

Oleg Drokin : c4cb5e5f0e316f986eae3b11be4f5d81756b85b6
Files :

• lustre/tests/sanity.sh

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el5,ofa #199
LU-479 sanity 124a failed

Oleg Drokin : c4cb5e5f0e316f986eae3b11be4f5d81756b85b6
Files :

• lustre/tests/sanity.sh

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,sles11,inkernel #199
LU-479 sanity 124a failed

Oleg Drokin : c4cb5e5f0e316f986eae3b11be4f5d81756b85b6
Files :

• lustre/tests/sanity.sh

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el5,inkernel #199
LU-479 sanity 124a failed

Oleg Drokin : c4cb5e5f0e316f986eae3b11be4f5d81756b85b6
Files :

• lustre/tests/sanity.sh

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,ubuntu1004,inkernel #199
LU-479 sanity 124a failed

Oleg Drokin : c4cb5e5f0e316f986eae3b11be4f5d81756b85b6
Files :

• lustre/tests/sanity.sh

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el5,inkernel #199
LU-479 sanity 124a failed

Oleg Drokin : c4cb5e5f0e316f986eae3b11be4f5d81756b85b6
Files :

• lustre/tests/sanity.sh

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el5,ofa #199
LU-479 sanity 124a failed

Oleg Drokin : c4cb5e5f0e316f986eae3b11be4f5d81756b85b6
Files :

• lustre/tests/sanity.sh

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el6,inkernel #199
LU-479 sanity 124a failed

Oleg Drokin : c4cb5e5f0e316f986eae3b11be4f5d81756b85b6
Files :

• lustre/tests/sanity.sh

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el6,inkernel #199
LU-479 sanity 124a failed

Oleg Drokin : c4cb5e5f0e316f986eae3b11be4f5d81756b85b6
Files :

• lustre/tests/sanity.sh

[Peter Jones]

Does it make sense to try and reproduce this manually in a vm environment as with LU482?

[Lai Siyao]

Okay, I will test it manually.

[Lai Siyao]

Hi Chris,

I ran sanity and test_124a for 2 days on client16-vm[1-3], but couldn't reproduce this failure.

Last weekend Toro nodes were reserved, and today these VMs can't be accessed yet, Chris, could you help me?

thanks,

• Lai

[Chris Gearing (Inactive)]

I've rebuilt the VM's after Sam's work. They should be usable now.

Chris

[Lai Siyao]

In maloo test result, one log message are repeated from the beginning of test_124a:

(ldlm_request.c:1218:ldlm_cli_update_pool()) @@@ Zero SLV or Limit found (SLV: 0, Limit: 102336)

Because zero SLV is invalid, client pool limit and slv won't be updated.

It's strange that MDS replies zero SLV to client, because the only chance obd->obd_pool_slv equals 0 is that ldlm_pool_recalc() is never called.

[Lai Siyao]

Hi Peter, I couldn't reproduce this failure (for 5 days), and to make debug easier, I'd suggest to enable this test on regression to help collect debug logs.

[Peter Jones]

Chris

Can you take care of this?

Peter

[Peter Jones]

Dropping as blocker for now. May re-add if it starts reoccurring regularly.

[Zhenyu Xu]

got another hit on https://maloo.whamcloud.com/test_sets/5ecde210-d96b-11e0-8d02-52540025f9af

[Kit Westneat (Inactive)]

It looks like these hit it as well:
https://maloo.whamcloud.com/test_sessions/7c897254-fdde-11e1-a1b4-52540035b04c https://maloo.whamcloud.com/test_sessions/cb9c8b6c-f6e3-11e1-b320-52540035b04c

[Andreas Dilger]

Close old issue.