2012-01-16 21:09:10 Lustre: DEBUG MARKER: -----============= acceptance-small: parallel-scale ============----- Mon Jan 16 21:09:09 PST 2012
2012-01-16 21:09:11 Lustre: DEBUG MARKER: only running test mdtestssf mdtestfpp mib iorssf iorfpp mdsrate
2012-01-16 21:09:12 Lustre: DEBUG MARKER: excepting tests: parallel_grouplock
2012-01-16 21:09:48 Lustre: DEBUG MARKER: Using TIMEOUT=100
2012-01-16 21:11:36 Lustre: DEBUG MARKER: 7 OST are inactive after 100 seconds, give up
2012-01-16 21:11:37 Lustre: 6705:0:(quota_master.c:793:close_quota_files()) quota[0] is off already
2012-01-16 21:11:37 Lustre: 6705:0:(quota_master.c:793:close_quota_files()) Skipped 1 previous similar message
2012-01-16 21:11:38 Lustre: 13732:0:(debug.c:326:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
2012-01-16 21:11:38 Lustre: 13732:0:(debug.c:326:libcfs_debug_str2mask()) Skipped 1 previous similar message
2012-01-16 21:11:53 LustreError: 5991:0:(osc_create.c:175:osc_interpret_create()) @@@ Unknown rc -107 from async create: failing oscc  req@ffff88022d736c00 x1391205468613621/t0(0) o5->lustre-OST001c-osc-MDT0000@192.168.127.66@o2ib1:7/4 lens 400/400 e 0 to 1 dl 1326777081 ref 1 fl Interpret:RXN/0/ffffffff rc -107/-1
2012-01-16 21:11:53 LustreError: 5991:0:(osc_create.c:175:osc_interpret_create()) Skipped 6 previous similar messages
2012-01-16 21:12:10 Lustre: DEBUG MARKER: == parallel-scale test mdtestssf: mdtestssf == 21:12:09 (1326777129)
2012-01-16 21:12:11 LustreError: 13392:0:(lov_obd.c:1069:lov_clear_orphans()) error in orphan recovery on OST idx 28/60: rc = -5
2012-01-16 21:12:11 LustreError: 13392:0:(lov_obd.c:1069:lov_clear_orphans()) Skipped 7 previous similar messages
2012-01-16 21:12:11 LustreError: 13392:0:(mds_lov.c:882:__mds_lov_synchronize()) lustre-OST001c_UUID failed at mds_lov_clear_orphans: -5
2012-01-16 21:12:11 LustreError: 13392:0:(mds_lov.c:882:__mds_lov_synchronize()) Skipped 7 previous similar messages
2012-01-16 21:12:11 LustreError: 13392:0:(mds_lov.c:903:__mds_lov_synchronize()) lustre-OST001c_UUID sync failed -5, deactivating
2012-01-16 21:12:11 LustreError: 13392:0:(mds_lov.c:903:__mds_lov_synchronize()) Skipped 7 previous similar messages
2012-01-16 21:12:50 LustreError: 13981:0:(lov_obd.c:1069:lov_clear_orphans()) error in orphan recovery on OST idx 31/60: rc = -5
2012-01-16 21:12:50 LustreError: 13981:0:(lov_obd.c:1069:lov_clear_orphans()) Skipped 1 previous similar message
2012-01-16 21:12:50 LustreError: 13981:0:(mds_lov.c:882:__mds_lov_synchronize()) lustre-OST001f_UUID failed at mds_lov_clear_orphans: -5
2012-01-16 21:12:50 LustreError: 13981:0:(mds_lov.c:882:__mds_lov_synchronize()) Skipped 1 previous similar message
2012-01-16 21:12:50 LustreError: 13981:0:(mds_lov.c:903:__mds_lov_synchronize()) lustre-OST001f_UUID sync failed -5, deactivating
2012-01-16 21:12:50 LustreError: 13981:0:(mds_lov.c:903:__mds_lov_synchronize()) Skipped 1 previous similar message
2012-01-16 21:13:46 LustreError: 6711:0:(osd_handler.c:1573:osd_attr_set()) ASSERTION((oh)->ot_declare_attr_set > 0) failed
2012-01-16 21:13:46 LustreError: 6711:0:(osd_handler.c:1573:osd_attr_set()) LBUG
2012-01-16 21:13:46 Pid: 6711, comm: mdt_15
2012-01-16 21:13:46 
2012-01-16 21:13:46 Jan 16 21:13:46 Call Trace:
2012-01-16 21:13:46 hyperion-rst6 ke [<ffffffffa044d855>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
2012-01-16 21:13:46 rnel: LustreErro [<ffffffffa044de95>] lbug_with_loc+0x75/0xe0 [libcfs]
2012-01-16 21:13:46 r: 6711:0:(osd_h [<ffffffffa0458d86>] libcfs_assertion_failed+0x66/0x70 [libcfs]
2012-01-16 21:13:46 andler.c:1573:os [<ffffffffa0a922d7>] osd_attr_set+0x4a7/0x510 [osd_ldiskfs]
2012-01-16 21:13:46 d_attr_set()) AS [<ffffffffa0a9111a>] ? osd_xattr_get+0x1ea/0x270 [osd_ldiskfs]
2012-01-16 21:13:46 SERTION((oh)->ot [<ffffffff814dd936>] ? rwsem_down_read_failed+0x26/0x30
2012-01-16 21:13:46 _declare_attr_se [<ffffffffa09c75a4>] mdd_attr_set_internal+0xb4/0x310 [mdd]
2012-01-16 21:13:46 t > 0) failed
2012-01-16 21:13:46 Ja [<ffffffffa09c7b55>] mdd_attr_check_set_internal+0x355/0x390 [mdd]
2012-01-16 21:13:46 n 16 21:13:46 hy [<ffffffffa09d5274>] ? mdd_lov_set_md+0x484/0x610 [mdd]
2012-01-16 21:13:46 perion-rst6 kern [<ffffffffa09c7bf9>] mdd_attr_check_set_internal_locked+0x69/0x180 [mdd]
2012-01-16 21:13:46 el: LustreError: [<ffffffffa09e99e8>] mdd_create+0x1ec8/0x2470 [mdd]
2012-01-16 21:13:46  6711:0:(osd_han [<ffffffffa05528ad>] ? htable_lookup+0xed/0x190 [obdclass]
2012-01-16 21:13:46 dler.c:1573:osd_ [<ffffffffa045c599>] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs]
2012-01-16 21:13:46 attr_set()) LBUG [<ffffffffa0648565>] ? lustre_msg_buf+0x85/0x90 [ptlrpc]
2012-01-16 21:13:46 
2012-01-16 21:13:46  [<ffffffffa0ab611f>] ? cml_lookup+0x8f/0x1f0 [cmm]
2012-01-16 21:13:46  [<ffffffffa06764fb>] ? __req_capsule_get+0x14b/0x6b0 [ptlrpc]
2012-01-16 21:13:46  [<ffffffffa0ab752c>] cml_create+0xbc/0x280 [cmm]
2012-01-16 21:13:46  [<ffffffffa0a386a6>] ? mdt_version_save+0x96/0x170 [mdt]
2012-01-16 21:13:46  [<ffffffffa0a396e1>] mdt_md_create+0x481/0x6a0 [mdt]
2012-01-16 21:13:46  [<ffffffffa0623991>] ? ldlm_request_cancel+0x351/0x420 [ptlrpc]
2012-01-16 21:13:46  [<ffffffffa023298c>] ? lprocfs_counter_add+0x12c/0x196 [lvfs]
2012-01-16 21:13:46  [<ffffffffa0a3a448>] mdt_reint_create+0x118/0x740 [mdt]
2012-01-16 21:13:46  [<ffffffffa09ee346>] ? md_ucred+0x26/0x60 [mdd]
2012-01-16 21:13:46  [<ffffffffa09ee346>] ? md_ucred+0x26/0x60 [mdd]
2012-01-16 21:13:46  [<ffffffffa0a3776f>] mdt_reint_rec+0x3f/0x100 [mdt]
2012-01-16 21:13:46  [<ffffffffa064aa44>] ? lustre_msg_get_flags+0x34/0xa0 [ptlrpc]
2012-01-16 21:13:46  [<ffffffffa0a2fbf4>] mdt_reint_internal+0x6d4/0x9f0 [mdt]
2012-01-16 21:13:46  [<ffffffffa0a257e6>] ? mdt_reint_opcode+0x96/0x160 [mdt]
2012-01-16 21:13:46  [<ffffffffa0a2ff5c>] mdt_reint+0x4c/0x120 [mdt]
2012-01-16 21:13:46  [<ffffffffa064a518>] ? lustre_msg_check_version+0xc8/0xe0 [ptlrpc]
2012-01-16 21:13:46  [<ffffffffa0a22ec5>] mdt_handle_common+0x8d5/0x1810 [mdt]
2012-01-16 21:13:46  [<ffffffffa06481a4>] ? lustre_msg_get_opc+0x94/0x100 [ptlrpc]
2012-01-16 21:13:46  [<ffffffffa0a23ed5>] mdt_regular_handle+0x15/0x20 [mdt]
2012-01-16 21:13:46  [<ffffffffa0658f29>] ptlrpc_main+0xbc9/0x19a0 [ptlrpc]
2012-01-16 21:13:46  [<ffffffffa0658360>] ? ptlrpc_main+0x0/0x19a0 [ptlrpc]
2012-01-16 21:13:46  [<ffffffff8100c1ca>] child_rip+0xa/0x20
2012-01-16 21:13:46  [<ffffffffa0658360>] ? ptlrpc_main+0x0/0x19a0 [ptlrpc]
2012-01-16 21:13:46  [<ffffffffa0658360>] ? ptlrpc_main+0x0/0x19a0 [ptlrpc]
2012-01-16 21:13:46  [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
2012-01-16 21:13:46 
2012-01-16 21:13:46 Kernel panic - not syncing: LBUG
2012-01-16 21:13:46 Pid: 6711, comm: mdt_15 Not tainted 2.6.32-131.17.1.el6_lustre.x86_64 #1
2012-01-16 21:13:46 Call Trace:
2012-01-16 21:13:46 Jan 16 21:13:46  [<ffffffff814dac78>] ? panic+0x78/0x143
2012-01-16 21:13:46 hyperion-rst6 ke [<ffffffffa044deeb>] ? lbug_with_loc+0xcb/0xe0 [libcfs]
2012-01-16 21:13:46 rnel: Kernel pan [<ffffffffa0458d86>] ? libcfs_assertion_failed+0x66/0x70 [libcfs]
2012-01-16 21:13:46 ic - not syncing [<ffffffffa0a922d7>] ? osd_attr_set+0x4a7/0x510 [osd_ldiskfs]
2012-01-16 21:13:46 : LBUG
2012-01-16 21:13:46  [<ffffffffa0a9111a>] ? osd_xattr_get+0x1ea/0x270 [osd_ldiskfs]
2012-01-16 21:13:46  [<ffffffff814dd936>] ? rwsem_down_read_failed+0x26/0x30
2012-01-16 21:13:46  [<ffffffffa09c75a4>] ? mdd_attr_set_internal+0xb4/0x310 [mdd]
2012-01-16 21:13:46  [<ffffffffa09c7b55>] ? mdd_attr_check_set_internal+0x355/0x390 [mdd]
2012-01-16 21:13:46  [<ffffffffa09d5274>] ? mdd_lov_set_md+0x484/0x610 [mdd]
2012-01-16 21:13:46  [<ffffffffa09c7bf9>] ? mdd_attr_check_set_internal_locked+0x69/0x180 [mdd]
2012-01-16 21:13:46  [<ffffffffa09e99e8>] ? mdd_create+0x1ec8/0x2470 [mdd]
2012-01-16 21:13:46  [<ffffffffa05528ad>] ? htable_lookup+0xed/0x190 [obdclass]
2012-01-16 21:13:46  [<ffffffffa045c599>] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs]
2012-01-16 21:13:46  [<ffffffffa0648565>] ? lustre_msg_buf+0x85/0x90 [ptlrpc]
2012-01-16 21:13:46  [<ffffffffa0ab611f>] ? cml_lookup+0x8f/0x1f0 [cmm]
2012-01-16 21:13:46  [<ffffffffa06764fb>] ? __req_capsule_get+0x14b/0x6b0 [ptlrpc]
2012-01-16 21:13:46  [<ffffffffa0ab752c>] ? cml_create+0xbc/0x280 [cmm]
2012-01-16 21:13:46  [<ffffffffa0a386a6>] ? mdt_version_save+0x96/0x170 [mdt]
2012-01-16 21:13:46  [<ffffffffa0a396e1>] ? mdt_md_create+0x481/0x6a0 [mdt]
2012-01-16 21:13:46  [<ffffffffa0623991>] ? ldlm_request_cancel+0x351/0x420 [ptlrpc]
2012-01-16 21:13:46  [<ffffffffa023298c>] ? lprocfs_counter_add+0x12c/0x196 [lvfs]
2012-01-16 21:13:46  [<ffffffffa0a3a448>] ? mdt_reint_create+0x118/0x740 [mdt]
2012-01-16 21:13:46  [<ffffffffa09ee346>] ? md_ucred+0x26/0x60 [mdd]
2012-01-16 21:13:46  [<ffffffffa09ee346>] ? md_ucred+0x26/0x60 [mdd]
2012-01-16 21:13:46  [<ffffffffa0a3776f>] ? mdt_reint_rec+0x3f/0x100 [mdt]
2012-01-16 21:13:46  [<ffffffffa064aa44>] ? lustre_msg_get_flags+0x34/0xa0 [ptlrpc]
2012-01-16 21:13:46  [<ffffffffa0a2fbf4>] ? mdt_reint_internal+0x6d4/0x9f0 [mdt]
2012-01-16 21:13:46  [<ffffffffa0a257e6>] ? mdt_reint_opcode+0x96/0x160 [mdt]
2012-01-16 21:13:46  [<ffffffffa0a2ff5c>] ? mdt_reint+0x4c/0x120 [mdt]
2012-01-16 21:13:46  [<ffffffffa064a518>] ? lustre_msg_check_version+0xc8/0xe0 [ptlrpc]
2012-01-16 21:13:47  [<ffffffffa0a22ec5>] ? mdt_handle_common+0x8d5/0x1810 [mdt]
2012-01-16 21:13:47  [<ffffffffa06481a4>] ? lustre_msg_get_opc+0x94/0x100 [ptlrpc]
2012-01-16 21:13:47  [<ffffffffa0a23ed5>] ? mdt_regular_handle+0x15/0x20 [mdt]
2012-01-16 21:13:47  [<ffffffffa0658f29>] ? ptlrpc_main+0xbc9/0x19a0 [ptlrpc]
2012-01-16 21:13:47  [<ffffffffa0658360>] ? ptlrpc_main+0x0/0x19a0 [ptlrpc]
2012-01-16 21:13:47  [<ffffffff8100c1ca>] ? child_rip+0xa/0x20
2012-01-16 21:13:47  [<ffffffffa0658360>] ? ptlrpc_main+0x0/0x19a0 [ptlrpc]
2012-01-16 21:13:47  [<ffffffffa0658360>] ? ptlrpc_main+0x0/0x19a0 [ptlrpc]
2012-01-16 21:13:47  [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
2012-01-16 21:13:47 Initializing cgroup subsys cpuset
2012-01-16 21:13:47 Initializing cgroup subsys cpu