[LU-2194] Test failure on test suite recovery-small, subtest test_19b

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.0
• Fix Version/s: Lustre 2.4.0, Lustre 2.8.0
• Type: Bug
• Priority: Blocker
• Reporter: Maloo
• Assignee: Hongchao Zhang
• Resolution: Fixed
• Labels: LB
• Severity: 3

Description

This issue was created by maloo for Li Wei <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/a7b63102-1737-11e2-afe1-52540035b04c.

The sub-test test_19b failed with the following error:

no eviction

Info required for matching: recovery-small 19b

Comments

[Nathaniel Clark]

http://review.whamcloud.com/4652

[Nathaniel Clark]

https://maloo.whamcloud.com/test_sets/d9395c3c-3d83-11e2-9127-52540035b04c

[Hongchao Zhang]

same error as above link https://maloo.whamcloud.com/test_sets/d9395c3c-3d83-11e2-9127-52540035b04c

https://maloo.whamcloud.com/test_sets/3e897880-432c-11e2-b57a-52540035b04c

[Hongchao Zhang]

the reason is the client is evicted by OST, which is the result of previous subtest "test_19a"
in "test_19a", fail_loc OBD_FAIL_LDLM_CANCEL_NET is set at MDT & OSTs, and it detect an eviction (was from MDT),
then it assume the test passed and continue to run test_19b, which encountered the eviction from OST.

[Peter Jones]

Landed for 2.4

[Peng Tao]

The same error as comment 2/3 above, is seen again with patch in comment 1 (commit cc980df563ef86847aae1e0a3f0a5b17589e6297) applied.

https://maloo.whamcloud.com/test_sets/070528a6-4cea-11e2-bf7d-52540035b04c

https://maloo.whamcloud.com/test_sets/befc32f8-4c25-11e2-875d-52540035b04c

[Keith Mannthey (Inactive)]

I don't know if this is exactly the same error originally reported but it is the same test and the same errors Peng Tao is seeing.

Error: 'test_19b failed with 1'

https://maloo.whamcloud.com/test_sessions/4d2113e8-471a-11e2-9537-52540035b04c

In my example the on the MDS the lnet service thead gets tied up:
"LNet: Service thread pid 7757 completed after 150.32s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources)."
There is a 40 second thread hung stack dump and plenty of other errors in the dmesg log about this Lnet service thread 7757.

[Keith Mannthey (Inactive)]

It seems test_19b is still having issues.

[Hongchao Zhang]

same error:
https://maloo.whamcloud.com/test_sets/befc32f8-4c25-11e2-875d-52540035b04c

[nasf (Inactive)]

Another failure:

https://maloo.whamcloud.com/test_sets/ac8aa71c-6034-11e2-84d4-52540035b04c

[Keith Mannthey (Inactive)]

Is the environment not set before the tests are starting in some way?

[Bob Glossman (Inactive)]

another instance:
https://maloo.whamcloud.com/test_sets/119c11ae-6233-11e2-b20c-52540035b04c

[Nathaniel Clark]

http://review.whamcloud.com/5141

[nasf (Inactive)]

Another failure instance:

https://maloo.whamcloud.com/test_sets/da6627c0-64b1-11e2-a1aa-52540035b04c

[Sebastien Buisson (Inactive)]

Yet another failure instance:
https://maloo.whamcloud.com/test_sets/1120911e-69a9-11e2-954a-52540035b04c

[Henri Doreau (Inactive)]

Another one:
https://maloo.whamcloud.com/test_sets/145ca582-6ab4-11e2-bea0-52540035b04c

[Bob Glossman (Inactive)]

Another instance:
https://maloo.whamcloud.com/test_sets/1fb3c870-6ae1-11e2-bea0-52540035b04c

[Peter Jones]

Has this reoccurred since the extra debugging has been landed?

[Nathaniel Clark]

Found one post debugging:
https://maloo.whamcloud.com/test_sets/e3c10bd4-7878-11e2-9928-52540035b04c

[Nathaniel Clark]

Client error for multiop:

Lustre: DEBUG MARKER: multiop /mnt/lustre/f.recovery-small.19b Ow
LustreError: 11-0: lustre-OST0000-osc-ffff88007c0b8000: Communicating with 10.10.17.16@tcp, operation ldlm_enqueue failed with -107.
LustreError: 23456:0:(ldlm_request.c:1267:ldlm_cli_cancel_req()) Got rc -5 from cancel RPC: canceling anyway
LustreError: 23456:0:(ldlm_request.c:1904:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -5
LustreError: 23456:0:(ldlm_request.c:1904:ldlm_cli_cancel_list()) Skipped 1 previous similar message
Lustre: DEBUG MARKER: /usr/sbin/lctl mark recovery-small test_19b: @@@@@@ FAIL: failed to run multiop: 5

[Bob Glossman (Inactive)]

another instance:
https://maloo.whamcloud.com/test_sets/48d621c0-7271-11e2-9b41-52540035b04c

[Zhenyu Xu]

another hit at https://maloo.whamcloud.com/test_sets/87356fc4-7f6f-11e2-afad-52540035b04c

from OST console message, it says

07:12:30:Lustre: DEBUG MARKER: == recovery-small test 19b: test expired_lock_main on ost (2867) == 07:12:23 (1361805143)
07:12:30:LustreError: 0:0:(ldlm_lockd.c:391:waiting_locks_callback()) ### lock callback timer expired after 153s: evicting client at 10.10.17.14@tcp ns: filter-ffff88006b003000 lock: ffff8800679ac000/0xf574bdda33b2fbc1 lrc: 3/0,0 mode: PR/PR res: 36/0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.10.17.14@tcp remote: 0x2e77606da0e4cfde expref: 4 pid: 29406 timeout: 4307850023 lvb_type: 1
07:12:30:LNet: Service thread pid 29405 completed after 150.98s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
07:12:30:Lustre: DEBUG MARKER: /usr/sbin/lctl mark recovery-small test_19b: @@@@@@ FAIL: failed to run multiop: 5

and OST evicts client

[Bob Glossman (Inactive)]

another instance:
https://maloo.whamcloud.com/test_sets/eeae8204-85d6-11e2-9f8d-52540035b04c

[Peter Jones]

Hongchao

Could you please look into this one?

Thanks

Peter

[Hongchao Zhang]

the patch is tracked at http://review.whamcloud.com/#change,5679

[Peter Jones]

Landed for 2.4

[Andreas Dilger]

recovery-small test_19b is still being skipped due to this bug.

[Gerrit Updater]

Hongchao Zhang ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13671
Subject: LU-2194 test: remove test_19b from except list
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 7531f2ac0495f021d79427bf9dfab41a2d269db6

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13671/
Subject: LU-2194 test: remove test_19b from except list
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 87e12095044c905078008efe11b3476374088c3e