[LU-4421] Failure on test suite sanity test_120e: 1 blocking RPC occured

• Type: Bug
• Resolution: Duplicate
• Priority: Minor
• Labels: dne
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/8cbadcc4-706c-11e3-a3b4-52540035b04c.

The sub-test test_120e failed with the following error:

1 blocking RPC occured.

test log shows:

== sanity test 120e: Early Lock Cancel: unlink test == 10:25:53 (1388255153)
ldlm.namespaces.lustre-MDT0000-mdc-ffff8800697aa400.lru_size=200
ldlm.namespaces.lustre-MDT0001-mdc-ffff8800697aa400.lru_size=200
ldlm.namespaces.lustre-MDT0002-mdc-ffff8800697aa400.lru_size=200
ldlm.namespaces.lustre-MDT0003-mdc-ffff8800697aa400.lru_size=200
ldlm.namespaces.lustre-OST0000-osc-ffff8800697aa400.lru_size=200
ldlm.namespaces.lustre-OST0001-osc-ffff8800697aa400.lru_size=200
ldlm.namespaces.lustre-OST0002-osc-ffff8800697aa400.lru_size=200
ldlm.namespaces.lustre-OST0003-osc-ffff8800697aa400.lru_size=200
ldlm.namespaces.lustre-OST0004-osc-ffff8800697aa400.lru_size=200
ldlm.namespaces.lustre-OST0005-osc-ffff8800697aa400.lru_size=200
ldlm.namespaces.lustre-OST0006-osc-ffff8800697aa400.lru_size=200
ldlm.namespaces.lustre-OST0007-osc-ffff8800697aa400.lru_size=200
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.00208266 s, 246 kB/s
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.00540223 s, 94.8 kB/s
error: get_param: /proc/{fs,sys}/{lnet,lustre}/ldlm/services/ldlm_canceld/stats: Found no match
error: get_param: /proc/{fs,sys}/{lnet,lustre}/ldlm/services/ldlm_canceld/stats: Found no match
 sanity test_120e: @@@@@@ FAIL: 1 blocking RPC occured. 

Hit this issue in patch http://review.whamcloud.com/#/c/7087/, not sure if this is master problem or just the patch

Issue Links

• duplicates: LU-4206 Sanity test_120e fails with 1 blocking RPC occured.
• is related to: LU-4747 sanity tests 120* call to "lctl get_param ldlm.services.ldlm_canceld.stats" on client fail

Comments

[Andreas Dilger]

Duplicate of LU-4206.

[James Nunez (Inactive)]

I've hit the same error at https://maloo.whamcloud.com/test_sets/805a8136-c27f-11e3-a886-52540035b04c in review-dne-part-1. The error message is the same, but, as you can see below, I don't get the get_param error

...
ldlm.namespaces.lustre-OST0005-osc-ffff88007e7b4c00.lru_size=200
ldlm.namespaces.lustre-OST0006-osc-ffff88007e7b4c00.lru_size=200
ldlm.namespaces.lustre-OST0007-osc-ffff88007e7b4c00.lru_size=200
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.00499895 s, 102 kB/s
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.00506382 s, 101 kB/s
CMD: client-32vm3 /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats
CMD: client-32vm3 /usr/sbin/lctl get_param -n ldlm.services.ldlm_canceld.stats
 sanity test_120e: @@@@@@ FAIL: 1 blocking RPC occured. 

Looking at the two cases above https://maloo.whamcloud.com/test_sets/a9708b10-b799-11e3-97ab-52540035b04c and https://maloo.whamcloud.com/test_sets/42cc4338-b92c-11e3-a578-52540035b04c, there is no get_param error in those logs. So, I think the get_param error was corrected in LU-4747, but there still an issue with the blocking RPC.

Should we close this ticket and open a new one with the slightly modified client test log?

[Bob Glossman (Inactive)]

another:
https://maloo.whamcloud.com/test_sets/42cc4338-b92c-11e3-a578-52540035b04c

[Swapnil Pimpale (Inactive)]

Another instance where this bug was hit: https://maloo.whamcloud.com/test_sets/a9708b10-b799-11e3-97ab-52540035b04c

[Jodi Levi (Inactive)]

Fan Yong,
Can you look into this one and comment?