[LU-703] liblustre timeout

• Type: Bug
• Resolution: Won't Fix
• Priority: Major
• Affects Version/s: Lustre 2.2.0, Lustre 2.1.1

Environment

Lustre Clients:
Tag: v2_1_0_0_RC2
Distro/Arch: RHEL6/i686 (kernel version: 2.6.32-131.6.1.el6)
Build:https://build.whamcloud.com/job/lustre-master/283/arch=i686,build_type=client,distro=el6,ib_stack=inkernel/
Network: TCP

Lustre Servers:
Tag: v2_1_0_0_RC2
Distro/Arch: RHEL6/x86_64 (kernel version: 2.6.32-131.6.1.el6)
Build: https://build.whamcloud.com/job/lustre-master/283/arch=x86_64,build_type=cserver,distro=el6,ib_stack=inkernel/
Network: TCP

• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Minh Diep

Description

v2_1_0_0-RC2 testing

liblustre seems failing to start

Report: https://maloo.whamcloud.com/test_sets/4fb0d978-e419-11e0-9909-52540025f9af

== liblustre test 1: liblustre sanity == 04:55:11 (1316519711)
/usr/lib64/lustre/liblustre/tests/sanity --target=client-20vm3@tcp:/lustre

1316519711.649109:10453:client-20vm1.lab.whamcloud.com:(class_obd.c:513:init_obdclass()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(class_obd.c:513:init_obdclass()): OBD class driver, http://wiki.whamcloud.com/
1316519711.649122:10453:client-20vm1.lab.whamcloud.com:(class_obd.c:514:init_obdclass()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(class_obd.c:514:init_obdclass()): Lustre Version: 2.1.0
1316519711.649127:10453:client-20vm1.lab.whamcloud.com:(class_obd.c:515:init_obdclass()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(class_obd.c:515:init_obdclass()): Build Version: RC2-g9d71fe8-PRISTINE-2.6.32-131.6.1.el6.i686
1316519711.663673:10453:client-20vm1.lab.whamcloud.com:(lu_object.c:1745:lu_global_init()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(lu_object.c:1745:lu_global_init()): Lustre LU module (0x82b3a80).
1316519712.143066:10453:client-20vm1.lab.whamcloud.com:(lov_obd.c:2892:lov_init()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(lov_obd.c:2892:lov_init()): Lustre LOV module (0x82a2920).
1316519712.144675:10453:client-20vm1.lab.whamcloud.com:(osc_request.c:4628:osc_init()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(osc_request.c:4628:osc_init()): Lustre OSC module (0x82a2d60).
LibLustre: obd timeout=100 seconds
1316519712.151016:10453:client-20vm1.lab.whamcloud.com:(sec.c:1474:sptlrpc_import_sec_adapt()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(sec.c:1474:sptlrpc_import_sec_adapt()): import mgc_dev->10.10.4.102@tcp netid 20000: select flavor null
1316519717.000598:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481537 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316519712] [real_sent 1316519712] [current 1316519717] [deadline 5s] [delay 0s] req@0x9a71e68 x1380470972481537/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316519717 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316519722.000555:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481539 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316519717] [real_sent 1316519717] [current 1316519722] [deadline 5s] [delay 0s] req@0x9a74aa0 x1380470972481539/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316519722 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316519722.000577:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 5s
1316519732.000568:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481540 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316519722] [real_sent 1316519722] [current 1316519732] [deadline 10s] [delay 0s] req@0x9a71e68 x1380470972481540/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316519732 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316519742.000558:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481541 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316519732] [real_sent 1316519732] [current 1316519742] [deadline 10s] [delay 0s] req@0x9a74aa0 x1380470972481541/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316519742 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316519742.000581:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 10s
1316519757.000556:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481542 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316519742] [real_sent 1316519742] [current 1316519757] [deadline 15s] [delay 0s] req@0x9a71e68 x1380470972481542/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316519757 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316519772.000558:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481543 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316519757] [real_sent 1316519757] [current 1316519772] [deadline 15s] [delay 0s] req@0x9a74aa0 x1380470972481543/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316519772 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316519772.000577:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 15s
1316519792.000553:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481544 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316519772] [real_sent 1316519772] [current 1316519792] [deadline 20s] [delay 0s] req@0x9a71e68 x1380470972481544/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316519792 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316519812.000565:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481545 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316519792] [real_sent 1316519792] [current 1316519812] [deadline 20s] [delay 0s] req@0x9a74aa0 x1380470972481545/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316519812 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316519812.000583:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 20s
1316519837.000553:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481546 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316519812] [real_sent 1316519812] [current 1316519837] [deadline 25s] [delay 0s] req@0x9a71e68 x1380470972481546/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316519837 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316519862.000555:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481547 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316519837] [real_sent 1316519837] [current 1316519862] [deadline 25s] [delay 0s] req@0x9a74aa0 x1380470972481547/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316519862 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316519862.000579:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 25s
1316519892.000552:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481548 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316519862] [real_sent 1316519862] [current 1316519892] [deadline 30s] [delay 0s] req@0x9a71e68 x1380470972481548/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316519892 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316519922.000558:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481549 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316519892] [real_sent 1316519892] [current 1316519922] [deadline 30s] [delay 0s] req@0x9a74aa0 x1380470972481549/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316519922 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316519922.000576:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 30s
1316519957.000562:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481550 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316519922] [real_sent 1316519922] [current 1316519957] [deadline 35s] [delay 0s] req@0x9a71e68 x1380470972481550/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316519957 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316519992.000560:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481551 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316519957] [real_sent 1316519957] [current 1316519992] [deadline 35s] [delay 0s] req@0x9a74aa0 x1380470972481551/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316519992 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316519992.000581:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 35s
1316520032.000557:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481552 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316519992] [real_sent 1316519992] [current 1316520032] [deadline 40s] [delay 0s] req@0x9a71e68 x1380470972481552/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520032 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520072.000568:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481553 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520032] [real_sent 1316520032] [current 1316520072] [deadline 40s] [delay 0s] req@0x9a74aa0 x1380470972481553/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520072 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520072.000592:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 40s
1316520117.000562:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481554 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520072] [real_sent 1316520072] [current 1316520117] [deadline 45s] [delay 0s] req@0x9a71e68 x1380470972481554/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520117 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520162.000563:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481555 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520117] [real_sent 1316520117] [current 1316520162] [deadline 45s] [delay 0s] req@0x9a74aa0 x1380470972481555/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520162 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520162.000581:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 45s
1316520212.000552:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481556 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520162] [real_sent 1316520162] [current 1316520212] [deadline 50s] [delay 0s] req@0x9a71e68 x1380470972481556/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520212 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520262.000560:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481557 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520212] [real_sent 1316520212] [current 1316520262] [deadline 50s] [delay 0s] req@0x9a74aa0 x1380470972481557/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520262 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520262.000578:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316520317.000555:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481558 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520262] [real_sent 1316520262] [current 1316520317] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481558/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520317 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520372.000555:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481559 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520317] [real_sent 1316520317] [current 1316520372] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481559/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520372 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520372.000574:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316520427.000584:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481560 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520372] [real_sent 1316520372] [current 1316520427] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481560/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520427 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520482.000576:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481561 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520427] [real_sent 1316520427] [current 1316520482] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481561/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520482 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520482.001192:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316520537.000552:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481562 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520482] [real_sent 1316520482] [current 1316520537] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481562/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520537 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520592.000564:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481563 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520537] [real_sent 1316520537] [current 1316520592] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481563/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520592 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520592.000582:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316520647.000552:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481564 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520592] [real_sent 1316520592] [current 1316520647] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481564/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520647 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520702.000554:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481565 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520647] [real_sent 1316520647] [current 1316520702] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481565/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520702 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520702.000574:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316520757.000565:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481566 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520702] [real_sent 1316520702] [current 1316520757] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481566/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520757 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520812.000563:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481567 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520757] [real_sent 1316520757] [current 1316520812] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481567/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520812 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520812.001242:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316520867.000579:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481568 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520812] [real_sent 1316520812] [current 1316520867] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481568/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520867 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520922.000567:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481569 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520867] [real_sent 1316520867] [current 1316520922] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481569/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520922 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316520922.000590:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316520977.000550:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481570 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520922] [real_sent 1316520922] [current 1316520977] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481570/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316520977 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521032.000558:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481571 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316520977] [real_sent 1316520977] [current 1316521032] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481571/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521032 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521032.000607:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316521087.000550:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481572 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521032] [real_sent 1316521032] [current 1316521087] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481572/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521087 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521142.000562:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481573 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521087] [real_sent 1316521087] [current 1316521142] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481573/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521142 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521142.001108:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316521197.000551:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481574 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521142] [real_sent 1316521142] [current 1316521197] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481574/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521197 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521252.000552:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481575 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521197] [real_sent 1316521197] [current 1316521252] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481575/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521252 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521252.000571:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316521307.000551:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481576 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521252] [real_sent 1316521252] [current 1316521307] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481576/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521307 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521362.000552:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481577 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521307] [real_sent 1316521307] [current 1316521362] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481577/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521362 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521362.000575:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316521417.000562:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481578 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521362] [real_sent 1316521362] [current 1316521417] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481578/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521417 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521472.000560:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481579 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521417] [real_sent 1316521417] [current 1316521472] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481579/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521472 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521472.000577:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316521527.000552:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481580 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521472] [real_sent 1316521472] [current 1316521527] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481580/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521527 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521582.000557:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481581 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521527] [real_sent 1316521527] [current 1316521582] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481581/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521582 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521582.000577:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316521637.000554:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481582 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521582] [real_sent 1316521582] [current 1316521637] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481582/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521637 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521692.000560:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481583 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521637] [real_sent 1316521637] [current 1316521692] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481583/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521692 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521692.000577:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316521747.000553:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481584 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521692] [real_sent 1316521692] [current 1316521747] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481584/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521747 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521802.000578:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481585 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521747] [real_sent 1316521747] [current 1316521802] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481585/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521802 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521802.000603:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316521857.000556:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481586 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521802] [real_sent 1316521802] [current 1316521857] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481586/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521857 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521912.000562:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481587 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521857] [real_sent 1316521857] [current 1316521912] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481587/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521912 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316521912.000584:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316521967.000553:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481588 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521912] [real_sent 1316521912] [current 1316521967] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481588/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316521967 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522022.000555:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481589 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316521967] [real_sent 1316521967] [current 1316522022] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481589/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522022 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522022.000575:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316522077.000582:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481590 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522022] [real_sent 1316522022] [current 1316522077] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481590/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522077 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522132.000557:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481591 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522077] [real_sent 1316522077] [current 1316522132] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481591/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522132 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522132.000582:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316522187.000574:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481592 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522132] [real_sent 1316522132] [current 1316522187] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481592/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522187 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522242.000573:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481593 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522187] [real_sent 1316522187] [current 1316522242] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481593/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522242 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522242.000591:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316522297.000571:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481594 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522242] [real_sent 1316522242] [current 1316522297] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481594/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522297 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522352.000556:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481595 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522297] [real_sent 1316522297] [current 1316522352] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481595/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522352 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522352.000576:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316522407.000559:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481596 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522352] [real_sent 1316522352] [current 1316522407] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481596/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522407 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522462.001372:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481597 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522407] [real_sent 1316522407] [current 1316522462] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481597/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522462 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522462.001396:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316522517.000570:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481598 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522462] [real_sent 1316522462] [current 1316522517] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481598/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522517 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522572.000561:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481599 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522517] [real_sent 1316522517] [current 1316522572] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481599/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522572 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522572.000582:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316522627.000562:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481600 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522572] [real_sent 1316522572] [current 1316522627] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481600/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522627 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522682.000558:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481601 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522627] [real_sent 1316522627] [current 1316522682] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481601/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522682 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522682.000579:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316522737.000620:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481602 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522682] [real_sent 1316522682] [current 1316522737] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481602/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522737 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522792.000569:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481603 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522737] [real_sent 1316522737] [current 1316522792] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481603/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522792 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522792.000587:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316522847.000561:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481604 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522792] [real_sent 1316522792] [current 1316522847] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481604/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522847 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522902.000561:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481605 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522847] [real_sent 1316522847] [current 1316522902] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481605/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522902 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316522902.001233:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316522957.000559:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481606 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522902] [real_sent 1316522902] [current 1316522957] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481606/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316522957 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316523012.000567:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481607 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316522957] [real_sent 1316522957] [current 1316523012] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481607/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316523012 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316523012.000592:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316523067.000568:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481608 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316523012] [real_sent 1316523012] [current 1316523067] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481608/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316523067 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316523122.000553:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481609 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316523067] [real_sent 1316523067] [current 1316523122] [deadline 55s] [delay 0s] req@0x9a74aa0 x1380470972481609/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316523122 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1
1316523122.000583:10453:client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(import.c:526:import_select_connection()): mgc_dev: tried all connections, increasing latency to 50s
1316523177.000548:10453:client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): Lustre: 10453-client-20vm1.lab.whamcloud.com:(client.c:1778:ptlrpc_expire_one_request()): @@@ Request x1380470972481610 sent from mgc_dev to NID 10.10.4.102@tcp has timed out for slow reply: [sent 1316523122] [real_sent 1316523122] [current 1316523177] [deadline 55s] [delay 0s] req@0x9a71e68 x1380470972481610/t0(0) o-1->MGS@10.10.4.102@tcp:26/25 lens 368/512 e 0 to 1 dl 1316523177 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1

Issue Links

• is duplicated by: LU-801 Test failure on test suite liblustre, subtest test_1
• Trackbacks: Lustre 2.1.0 release testing tracker Lustre 2.1.0 RC2 Tag: v2100RC2 Build:
• Trackbacks: Lustre 2.1.1 release testing tracker Lustre 2.1.1 RC2 Tag: v2110RC2 Build: