Jul 14 03:35:01 ucbn001 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="3912" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jul 14 03:35:02 ucbn001 rhsmd: This system is registered to RHN Classic.
Jul 15 03:13:01 ucbn001 logrotate: ALERT exited abnormally with [1]
Jul 15 03:13:02 ucbn001 rhsmd: This system is registered to RHN Classic.
Jul 15 14:30:44 ucbn001 kernel: LustreError: 11-0: pfscdat2-OST0000-osc-ffff88087250a800: Communicating with 172.26.17.4@o2ib, operation ldlm_enqueue failed with -19.
Jul 15 14:30:44 ucbn001 kernel: Lustre: pfscdat2-OST0000-osc-ffff88087250a800: Connection to pfscdat2-OST0000 (at 172.26.17.4@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jul 15 14:31:36 ucbn001 kernel: Lustre: pfscdat2-OST0000-osc-ffff88087250a800: Connection restored to pfscdat2-OST0000 (at 172.26.17.3@o2ib)
Jul 15 14:36:46 ucbn001 kernel: LustreError: 11-0: pfscdat2-OST0000-osc-ffff88087250a800: Communicating with 172.26.17.3@o2ib, operation ost_write failed with -107.
Jul 15 14:36:46 ucbn001 kernel: Lustre: pfscdat2-OST0000-osc-ffff88087250a800: Connection to pfscdat2-OST0000 (at 172.26.17.3@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jul 15 14:36:46 ucbn001 kernel: LustreError: Skipped 1 previous similar message
Jul 15 14:37:40 ucbn001 kernel: Lustre: pfscdat2-OST0000-osc-ffff88087250a800: Connection restored to pfscdat2-OST0000 (at 172.26.17.4@o2ib)
Jul 15 15:01:02 ucbn001 nscd_restart.cron: NSCD is dead. Starting service again
Jul 15 15:01:02 ucbn001 nscd_restart.cron: Starting nscd: [  OK  ]#015
Jul 15 15:37:13 ucbn001 kernel: INFO: task ll_sa_71331:90417 blocked for more than 120 seconds.
Jul 15 15:37:13 ucbn001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 15 15:37:13 ucbn001 kernel: ll_sa_71331   D 000000000000001e     0 90417      2 0x00000080
Jul 15 15:37:13 ucbn001 kernel: ffff881030ce7b10 0000000000000046 ffff881030ce7a80 ffffffff00000050
Jul 15 15:37:13 ucbn001 kernel: ffff881030ce7ab0 0000000000000246 0000000000000010 ffff88086aac23c0
Jul 15 15:37:13 ucbn001 kernel: ffff881042c59af8 ffff881030ce7fd8 000000000000fb88 ffff881042c59af8
Jul 15 15:37:13 ucbn001 kernel: Call Trace:
Jul 15 15:37:13 ucbn001 kernel: [<ffffffff81096f8e>] ? prepare_to_wait+0x4e/0x80
Jul 15 15:37:13 ucbn001 kernel: [<ffffffff8119c928>] __wait_on_freeing_inode+0x98/0xc0
Jul 15 15:37:13 ucbn001 kernel: [<ffffffff81096ce0>] ? wake_bit_function+0x0/0x50
Jul 15 15:37:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 15:37:13 ucbn001 kernel: [<ffffffff8119cad4>] find_inode+0x64/0x90
Jul 15 15:37:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 15:37:13 ucbn001 kernel: [<ffffffff8119dc1d>] ifind+0x4d/0xd0
Jul 15 15:37:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 15:37:13 ucbn001 kernel: [<ffffffff8119e099>] iget5_locked+0x59/0x1b0
Jul 15 15:37:13 ucbn001 kernel: [<ffffffffa0b1fe00>] ? ll_set_inode+0x0/0x1a0 [lustre]
Jul 15 15:37:13 ucbn001 kernel: [<ffffffffa0b1fb05>] ll_iget+0x65/0x360 [lustre]
Jul 15 15:37:13 ucbn001 kernel: [<ffffffffa0bf88d3>] ? lmv_get_lustre_md+0x153/0x3d0 [lmv]
Jul 15 15:37:13 ucbn001 kernel: [<ffffffffa076d9a1>] ? ldlm_revalidate_lock_handle+0x81/0x250 [ptlrpc]
Jul 15 15:37:13 ucbn001 kernel: [<ffffffffa0b13080>] ll_prep_inode+0x6e0/0xf60 [lustre]
Jul 15 15:37:13 ucbn001 kernel: [<ffffffffa0c0d8c4>] ? lmv_revalidate_lock+0x2b4/0x550 [lmv]
Jul 15 15:37:13 ucbn001 kernel: [<ffffffffa0b36a47>] ll_post_statahead+0x2f7/0xa80 [lustre]
Jul 15 15:37:13 ucbn001 kernel: [<ffffffffa0b3b4b8>] ll_statahead_thread+0xd38/0xfa0 [lustre]
Jul 15 15:37:13 ucbn001 kernel: [<ffffffff81063310>] ? default_wake_function+0x0/0x20
Jul 15 15:37:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 15:37:13 ucbn001 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jul 15 15:37:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 15:37:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 15:37:13 ucbn001 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jul 15 15:39:13 ucbn001 kernel: INFO: task ll_sa_71331:90417 blocked for more than 120 seconds.
Jul 15 15:39:13 ucbn001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 15 15:39:13 ucbn001 kernel: ll_sa_71331   D 000000000000001e     0 90417      2 0x00000080
Jul 15 15:39:13 ucbn001 kernel: ffff881030ce7b10 0000000000000046 ffff881030ce7a80 ffffffff00000050
Jul 15 15:39:13 ucbn001 kernel: ffff881030ce7ab0 0000000000000246 0000000000000010 ffff88086aac23c0
Jul 15 15:39:13 ucbn001 kernel: ffff881042c59af8 ffff881030ce7fd8 000000000000fb88 ffff881042c59af8
Jul 15 15:39:13 ucbn001 kernel: Call Trace:
Jul 15 15:39:13 ucbn001 kernel: [<ffffffff81096f8e>] ? prepare_to_wait+0x4e/0x80
Jul 15 15:39:13 ucbn001 kernel: [<ffffffff8119c928>] __wait_on_freeing_inode+0x98/0xc0
Jul 15 15:39:13 ucbn001 kernel: [<ffffffff81096ce0>] ? wake_bit_function+0x0/0x50
Jul 15 15:39:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 15:39:13 ucbn001 kernel: [<ffffffff8119cad4>] find_inode+0x64/0x90
Jul 15 15:39:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 15:39:13 ucbn001 kernel: [<ffffffff8119dc1d>] ifind+0x4d/0xd0
Jul 15 15:39:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 15:39:13 ucbn001 kernel: [<ffffffff8119e099>] iget5_locked+0x59/0x1b0
Jul 15 15:39:13 ucbn001 kernel: [<ffffffffa0b1fe00>] ? ll_set_inode+0x0/0x1a0 [lustre]
Jul 15 15:39:13 ucbn001 kernel: [<ffffffffa0b1fb05>] ll_iget+0x65/0x360 [lustre]
Jul 15 15:39:13 ucbn001 kernel: [<ffffffffa0bf88d3>] ? lmv_get_lustre_md+0x153/0x3d0 [lmv]
Jul 15 15:39:13 ucbn001 kernel: [<ffffffffa076d9a1>] ? ldlm_revalidate_lock_handle+0x81/0x250 [ptlrpc]
Jul 15 15:39:13 ucbn001 kernel: [<ffffffffa0b13080>] ll_prep_inode+0x6e0/0xf60 [lustre]
Jul 15 15:39:13 ucbn001 kernel: [<ffffffffa0c0d8c4>] ? lmv_revalidate_lock+0x2b4/0x550 [lmv]
Jul 15 15:39:13 ucbn001 kernel: [<ffffffffa0b36a47>] ll_post_statahead+0x2f7/0xa80 [lustre]
Jul 15 15:39:13 ucbn001 kernel: [<ffffffffa0b3b4b8>] ll_statahead_thread+0xd38/0xfa0 [lustre]
Jul 15 15:39:13 ucbn001 kernel: [<ffffffff81063310>] ? default_wake_function+0x0/0x20
Jul 15 15:39:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 15:39:13 ucbn001 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jul 15 15:39:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 15:39:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 15:39:13 ucbn001 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jul 15 15:41:13 ucbn001 kernel: INFO: task ll_sa_71331:90417 blocked for more than 120 seconds.
Jul 15 15:41:13 ucbn001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 15 15:41:13 ucbn001 kernel: ll_sa_71331   D 000000000000001e     0 90417      2 0x00000080
Jul 15 15:41:13 ucbn001 kernel: ffff881030ce7b10 0000000000000046 ffff881030ce7a80 ffffffff00000050
Jul 15 15:41:13 ucbn001 kernel: ffff881030ce7ab0 0000000000000246 0000000000000010 ffff88086aac23c0
Jul 15 15:41:13 ucbn001 kernel: ffff881042c59af8 ffff881030ce7fd8 000000000000fb88 ffff881042c59af8
Jul 15 15:41:13 ucbn001 kernel: Call Trace:
Jul 15 15:41:13 ucbn001 kernel: [<ffffffff81096f8e>] ? prepare_to_wait+0x4e/0x80
Jul 15 15:41:13 ucbn001 kernel: [<ffffffff8119c928>] __wait_on_freeing_inode+0x98/0xc0
Jul 15 15:41:13 ucbn001 kernel: [<ffffffff81096ce0>] ? wake_bit_function+0x0/0x50
Jul 15 15:41:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 15:41:13 ucbn001 kernel: [<ffffffff8119cad4>] find_inode+0x64/0x90
Jul 15 15:41:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 15:41:13 ucbn001 kernel: [<ffffffff8119dc1d>] ifind+0x4d/0xd0
Jul 15 15:41:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 15:41:13 ucbn001 kernel: [<ffffffff8119e099>] iget5_locked+0x59/0x1b0
Jul 15 15:41:13 ucbn001 kernel: [<ffffffffa0b1fe00>] ? ll_set_inode+0x0/0x1a0 [lustre]
Jul 15 15:41:13 ucbn001 kernel: [<ffffffffa0b1fb05>] ll_iget+0x65/0x360 [lustre]
Jul 15 15:41:13 ucbn001 kernel: [<ffffffffa0bf88d3>] ? lmv_get_lustre_md+0x153/0x3d0 [lmv]
Jul 15 15:41:13 ucbn001 kernel: [<ffffffffa076d9a1>] ? ldlm_revalidate_lock_handle+0x81/0x250 [ptlrpc]
Jul 15 15:41:13 ucbn001 kernel: [<ffffffffa0b13080>] ll_prep_inode+0x6e0/0xf60 [lustre]
Jul 15 15:41:13 ucbn001 kernel: [<ffffffffa0c0d8c4>] ? lmv_revalidate_lock+0x2b4/0x550 [lmv]
Jul 15 15:41:13 ucbn001 kernel: [<ffffffffa0b36a47>] ll_post_statahead+0x2f7/0xa80 [lustre]
Jul 15 15:41:13 ucbn001 kernel: [<ffffffffa0b3b4b8>] ll_statahead_thread+0xd38/0xfa0 [lustre]
Jul 15 15:41:13 ucbn001 kernel: [<ffffffff81063310>] ? default_wake_function+0x0/0x20
Jul 15 15:41:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 15:41:13 ucbn001 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jul 15 15:41:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 15:41:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 15:41:13 ucbn001 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jul 15 15:43:13 ucbn001 kernel: INFO: task ll_sa_71331:90417 blocked for more than 120 seconds.
Jul 15 15:43:13 ucbn001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 15 15:43:13 ucbn001 kernel: ll_sa_71331   D 000000000000001e     0 90417      2 0x00000080
Jul 15 15:43:13 ucbn001 kernel: ffff881030ce7b10 0000000000000046 ffff881030ce7a80 ffffffff00000050
Jul 15 15:43:13 ucbn001 kernel: ffff881030ce7ab0 0000000000000246 0000000000000010 ffff88086aac23c0
Jul 15 15:43:13 ucbn001 kernel: ffff881042c59af8 ffff881030ce7fd8 000000000000fb88 ffff881042c59af8
Jul 15 15:43:13 ucbn001 kernel: Call Trace:
Jul 15 15:43:13 ucbn001 kernel: [<ffffffff81096f8e>] ? prepare_to_wait+0x4e/0x80
Jul 15 15:43:13 ucbn001 kernel: [<ffffffff8119c928>] __wait_on_freeing_inode+0x98/0xc0
Jul 15 15:43:13 ucbn001 kernel: [<ffffffff81096ce0>] ? wake_bit_function+0x0/0x50
Jul 15 15:43:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 15:43:13 ucbn001 kernel: [<ffffffff8119cad4>] find_inode+0x64/0x90
Jul 15 15:43:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 15:43:13 ucbn001 kernel: [<ffffffff8119dc1d>] ifind+0x4d/0xd0
Jul 15 15:43:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 15:43:13 ucbn001 kernel: [<ffffffff8119e099>] iget5_locked+0x59/0x1b0
Jul 15 15:43:13 ucbn001 kernel: [<ffffffffa0b1fe00>] ? ll_set_inode+0x0/0x1a0 [lustre]
Jul 15 15:43:13 ucbn001 kernel: [<ffffffffa0b1fb05>] ll_iget+0x65/0x360 [lustre]
Jul 15 15:43:13 ucbn001 kernel: [<ffffffffa0bf88d3>] ? lmv_get_lustre_md+0x153/0x3d0 [lmv]
Jul 15 15:43:13 ucbn001 kernel: [<ffffffffa076d9a1>] ? ldlm_revalidate_lock_handle+0x81/0x250 [ptlrpc]
Jul 15 15:43:13 ucbn001 kernel: [<ffffffffa0b13080>] ll_prep_inode+0x6e0/0xf60 [lustre]
Jul 15 15:43:13 ucbn001 kernel: [<ffffffffa0c0d8c4>] ? lmv_revalidate_lock+0x2b4/0x550 [lmv]
Jul 15 15:43:13 ucbn001 kernel: [<ffffffffa0b36a47>] ll_post_statahead+0x2f7/0xa80 [lustre]
Jul 15 15:43:13 ucbn001 kernel: [<ffffffffa0b3b4b8>] ll_statahead_thread+0xd38/0xfa0 [lustre]
Jul 15 15:43:13 ucbn001 kernel: [<ffffffff81063310>] ? default_wake_function+0x0/0x20
Jul 15 15:43:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 15:43:13 ucbn001 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jul 15 15:43:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 15:43:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 15:43:13 ucbn001 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jul 15 15:55:49 ucbn001 kernel: LustreError: 11-0: pfscdat2-MDT0000-mdc-ffff88087250a800: Communicating with 172.26.17.2@o2ib, operation mds_reint failed with -19.
Jul 15 15:55:49 ucbn001 kernel: LustreError: Skipped 2 previous similar messages
Jul 15 15:55:49 ucbn001 kernel: Lustre: pfscdat2-MDT0000-mdc-ffff88087250a800: Connection to pfscdat2-MDT0000 (at 172.26.17.2@o2ib) was lost; in progress operations using this service will wait for recovery to complete
Jul 15 15:56:43 ucbn001 kernel: Lustre: pfscdat2-MDT0000-mdc-ffff88087250a800: Connection restored to pfscdat2-MDT0000 (at 172.26.17.1@o2ib)
Jul 15 16:37:13 ucbn001 kernel: INFO: task rm:91092 blocked for more than 120 seconds.
Jul 15 16:37:13 ucbn001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 15 16:37:13 ucbn001 kernel: rm            D 0000000000000002     0 91092  70318 0x00000080
Jul 15 16:37:13 ucbn001 kernel: ffff881058b7b9b8 0000000000000086 ffff881058b7b968 0000000000000050
Jul 15 16:37:13 ucbn001 kernel: ffff881058b7b958 0000000000000246 0000000000000010 ffff88086aac23c0
Jul 15 16:37:13 ucbn001 kernel: ffff8810554205f8 ffff881058b7bfd8 000000000000fb88 ffff8810554205f8
Jul 15 16:37:13 ucbn001 kernel: Call Trace:
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff81096f8e>] ? prepare_to_wait+0x4e/0x80
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119c928>] __wait_on_freeing_inode+0x98/0xc0
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff81096ce0>] ? wake_bit_function+0x0/0x50
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119cad4>] find_inode+0x64/0x90
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119dc1d>] ifind+0x4d/0xd0
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119e099>] iget5_locked+0x59/0x1b0
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1fe00>] ? ll_set_inode+0x0/0x1a0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1fb05>] ll_iget+0x65/0x360 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0bf88d3>] ? lmv_get_lustre_md+0x153/0x3d0 [lmv]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b13080>] ll_prep_inode+0x6e0/0xf60 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa07cd2c8>] ? req_capsule_server_get+0x18/0x20 [ptlrpc]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0c11dea>] ? lmv_intent_lookup+0x25a/0x770 [lmv]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b20930>] ? ll_md_blocking_ast+0x0/0x750 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0aea04e>] ? ll_check_acl+0x3e/0x160 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b23f3a>] ll_lookup_it_finish+0x1da/0xe80 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0c12eda>] ? lmv_intent_lock+0x31a/0x370 [lmv]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b20930>] ? ll_md_blocking_ast+0x0/0x750 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b24fd7>] ll_lookup_it+0x3f7/0xbf0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b20930>] ? ll_md_blocking_ast+0x0/0x750 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0ae9b94>] ? ll_get_acl+0x34/0xe0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b2585c>] ll_lookup_nd+0x8c/0x430 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119ae07>] ? d_alloc+0x137/0x1b0
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8118e202>] __lookup_hash+0x102/0x160
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8118e8aa>] lookup_hash+0x3a/0x50
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff81191ea1>] do_unlinkat+0xb1/0x1b0
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff811c326e>] ? dnotify_flush+0x7e/0x140
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff810dc847>] ? audit_syscall_entry+0x1d7/0x200
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff81192102>] sys_unlinkat+0x22/0x40
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Jul 15 16:37:13 ucbn001 kernel: INFO: task ll_sa_91092:96049 blocked for more than 120 seconds.
Jul 15 16:37:13 ucbn001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 15 16:37:13 ucbn001 kernel: ll_sa_91092   D 0000000000000008     0 96049      2 0x00000080
Jul 15 16:37:13 ucbn001 kernel: ffff880812de5b10 0000000000000046 0000000000000000 ffffffff810522fd
Jul 15 16:37:13 ucbn001 kernel: ffff880812de5a80 0000000000000086 ffff880812de5af0 ffffffff8106317e
Jul 15 16:37:13 ucbn001 kernel: ffff88081d0a5ab8 ffff880812de5fd8 000000000000fb88 ffff88081d0a5ab8
Jul 15 16:37:13 ucbn001 kernel: Call Trace:
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff810522fd>] ? check_preempt_curr+0x6d/0x90
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8106317e>] ? try_to_wake_up+0x24e/0x3e0
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119c928>] __wait_on_freeing_inode+0x98/0xc0
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff81096ce0>] ? wake_bit_function+0x0/0x50
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119cad4>] find_inode+0x64/0x90
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119dc1d>] ifind+0x4d/0xd0
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119e099>] iget5_locked+0x59/0x1b0
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1fe00>] ? ll_set_inode+0x0/0x1a0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1fb05>] ll_iget+0x65/0x360 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0bf88d3>] ? lmv_get_lustre_md+0x153/0x3d0 [lmv]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa076d9a1>] ? ldlm_revalidate_lock_handle+0x81/0x250 [ptlrpc]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b13080>] ll_prep_inode+0x6e0/0xf60 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0c0d8c4>] ? lmv_revalidate_lock+0x2b4/0x550 [lmv]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b36a47>] ll_post_statahead+0x2f7/0xa80 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b3a9e8>] ll_statahead_thread+0x268/0xfa0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff81063310>] ? default_wake_function+0x0/0x20
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jul 15 16:37:13 ucbn001 kernel: INFO: task ll_sa_91092:96113 blocked for more than 120 seconds.
Jul 15 16:37:13 ucbn001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 15 16:37:13 ucbn001 kernel: ll_sa_91092   D 0000000000000019     0 96113      2 0x00000080
Jul 15 16:37:13 ucbn001 kernel: ffff88081e84fb10 0000000000000046 ffff88081e84fa80 ffffffff00000050
Jul 15 16:37:13 ucbn001 kernel: ffff88081e84fab0 0000000000000246 0000000000000010 ffff88086aac23c0
Jul 15 16:37:13 ucbn001 kernel: ffff88083028e5f8 ffff88081e84ffd8 000000000000fb88 ffff88083028e5f8
Jul 15 16:37:13 ucbn001 kernel: Call Trace:
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff81096f8e>] ? prepare_to_wait+0x4e/0x80
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119c928>] __wait_on_freeing_inode+0x98/0xc0
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff81096ce0>] ? wake_bit_function+0x0/0x50
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119cad4>] find_inode+0x64/0x90
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119dc1d>] ifind+0x4d/0xd0
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119e099>] iget5_locked+0x59/0x1b0
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1fe00>] ? ll_set_inode+0x0/0x1a0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1fb05>] ll_iget+0x65/0x360 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0bf88d3>] ? lmv_get_lustre_md+0x153/0x3d0 [lmv]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa076d9a1>] ? ldlm_revalidate_lock_handle+0x81/0x250 [ptlrpc]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b13080>] ll_prep_inode+0x6e0/0xf60 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0c0d8c4>] ? lmv_revalidate_lock+0x2b4/0x550 [lmv]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b36a47>] ll_post_statahead+0x2f7/0xa80 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b3b4b8>] ll_statahead_thread+0xd38/0xfa0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff81063310>] ? default_wake_function+0x0/0x20
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jul 15 16:37:13 ucbn001 kernel: INFO: task ll_sa_91092:96115 blocked for more than 120 seconds.
Jul 15 16:37:13 ucbn001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 15 16:37:13 ucbn001 kernel: ll_sa_91092   D 0000000000000019     0 96115      2 0x00000080
Jul 15 16:37:13 ucbn001 kernel: ffff880812143b10 0000000000000046 0000000000000000 ffffffff00000050
Jul 15 16:37:13 ucbn001 kernel: ffff880812143ab0 0000000000000246 0000000000000010 ffff88086aac23c0
Jul 15 16:37:13 ucbn001 kernel: ffff8808214d0638 ffff880812143fd8 000000000000fb88 ffff8808214d0638
Jul 15 16:37:13 ucbn001 kernel: Call Trace:
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119c928>] __wait_on_freeing_inode+0x98/0xc0
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff81096ce0>] ? wake_bit_function+0x0/0x50
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119cad4>] find_inode+0x64/0x90
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119dc1d>] ifind+0x4d/0xd0
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8119e099>] iget5_locked+0x59/0x1b0
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1fe00>] ? ll_set_inode+0x0/0x1a0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b1fb05>] ll_iget+0x65/0x360 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0bf88d3>] ? lmv_get_lustre_md+0x153/0x3d0 [lmv]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa076d9a1>] ? ldlm_revalidate_lock_handle+0x81/0x250 [ptlrpc]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b13080>] ll_prep_inode+0x6e0/0xf60 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0c0d8c4>] ? lmv_revalidate_lock+0x2b4/0x550 [lmv]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b36a47>] ll_post_statahead+0x2f7/0xa80 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b3b4b8>] ll_statahead_thread+0xd38/0xfa0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff81063310>] ? default_wake_function+0x0/0x20
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 16:37:13 ucbn001 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jul 15 16:39:13 ucbn001 kernel: INFO: task rm:91092 blocked for more than 120 seconds.
Jul 15 16:39:13 ucbn001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 15 16:39:13 ucbn001 kernel: rm            D 0000000000000002     0 91092  70318 0x00000080
Jul 15 16:39:13 ucbn001 kernel: ffff881058b7b9b8 0000000000000086 ffff881058b7b968 0000000000000050
Jul 15 16:39:13 ucbn001 kernel: ffff881058b7b958 0000000000000246 0000000000000010 ffff88086aac23c0
Jul 15 16:39:13 ucbn001 kernel: ffff8810554205f8 ffff881058b7bfd8 000000000000fb88 ffff8810554205f8
Jul 15 16:39:13 ucbn001 kernel: Call Trace:
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff81096f8e>] ? prepare_to_wait+0x4e/0x80
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff8119c928>] __wait_on_freeing_inode+0x98/0xc0
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff81096ce0>] ? wake_bit_function+0x0/0x50
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff8119cad4>] find_inode+0x64/0x90
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff8119dc1d>] ifind+0x4d/0xd0
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff8119e099>] iget5_locked+0x59/0x1b0
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b1fe00>] ? ll_set_inode+0x0/0x1a0 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b1fb05>] ll_iget+0x65/0x360 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0bf88d3>] ? lmv_get_lustre_md+0x153/0x3d0 [lmv]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b13080>] ll_prep_inode+0x6e0/0xf60 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa07cd2c8>] ? req_capsule_server_get+0x18/0x20 [ptlrpc]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0c11dea>] ? lmv_intent_lookup+0x25a/0x770 [lmv]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b20930>] ? ll_md_blocking_ast+0x0/0x750 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0aea04e>] ? ll_check_acl+0x3e/0x160 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b23f3a>] ll_lookup_it_finish+0x1da/0xe80 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0c12eda>] ? lmv_intent_lock+0x31a/0x370 [lmv]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b20930>] ? ll_md_blocking_ast+0x0/0x750 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b24fd7>] ll_lookup_it+0x3f7/0xbf0 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b20930>] ? ll_md_blocking_ast+0x0/0x750 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0ae9b94>] ? ll_get_acl+0x34/0xe0 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b2585c>] ll_lookup_nd+0x8c/0x430 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff8119ae07>] ? d_alloc+0x137/0x1b0
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff8118e202>] __lookup_hash+0x102/0x160
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff8118e8aa>] lookup_hash+0x3a/0x50
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff81191ea1>] do_unlinkat+0xb1/0x1b0
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff811c326e>] ? dnotify_flush+0x7e/0x140
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff810dc847>] ? audit_syscall_entry+0x1d7/0x200
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff81192102>] sys_unlinkat+0x22/0x40
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Jul 15 16:39:13 ucbn001 kernel: INFO: task ll_sa_91092:96049 blocked for more than 120 seconds.
Jul 15 16:39:13 ucbn001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 15 16:39:13 ucbn001 kernel: ll_sa_91092   D 0000000000000008     0 96049      2 0x00000080
Jul 15 16:39:13 ucbn001 kernel: ffff880812de5b10 0000000000000046 0000000000000000 ffffffff810522fd
Jul 15 16:39:13 ucbn001 kernel: ffff880812de5a80 0000000000000086 ffff880812de5af0 ffffffff8106317e
Jul 15 16:39:13 ucbn001 kernel: ffff88081d0a5ab8 ffff880812de5fd8 000000000000fb88 ffff88081d0a5ab8
Jul 15 16:39:13 ucbn001 kernel: Call Trace:
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff810522fd>] ? check_preempt_curr+0x6d/0x90
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff8106317e>] ? try_to_wake_up+0x24e/0x3e0
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff8119c928>] __wait_on_freeing_inode+0x98/0xc0
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff81096ce0>] ? wake_bit_function+0x0/0x50
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff8119cad4>] find_inode+0x64/0x90
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff8119dc1d>] ifind+0x4d/0xd0
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b1f630>] ? ll_test_inode+0x0/0x90 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff8119e099>] iget5_locked+0x59/0x1b0
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b1fe00>] ? ll_set_inode+0x0/0x1a0 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b1fb05>] ll_iget+0x65/0x360 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0bf88d3>] ? lmv_get_lustre_md+0x153/0x3d0 [lmv]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa076d9a1>] ? ldlm_revalidate_lock_handle+0x81/0x250 [ptlrpc]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b13080>] ll_prep_inode+0x6e0/0xf60 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0c0d8c4>] ? lmv_revalidate_lock+0x2b4/0x550 [lmv]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b36a47>] ll_post_statahead+0x2f7/0xa80 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b3a9e8>] ll_statahead_thread+0x268/0xfa0 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff81063310>] ? default_wake_function+0x0/0x20
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffffa0b3a780>] ? ll_statahead_thread+0x0/0xfa0 [lustre]
Jul 15 16:39:13 ucbn001 kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jul 15 18:39:15 ucbn001 kernel: igb: eth0 NIC Link is Down
Jul 15 18:39:41 ucbn001 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jul 15 18:43:06 ucbn001 kernel: igb: eth0 NIC Link is Down
Jul 15 18:43:24 ucbn001 kernel: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jul 16 03:49:02 ucbn001 logrotate: ALERT exited abnormally with [1]
Jul 16 03:49:12 ucbn001 rhsmd: This system is registered to RHN Classic.