Apr 28 14:38:20 meerkat-oss-11-1 kernel: LustreError: 5128:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0013 qtype:usr id:504888 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 14:39:45 meerkat-oss-6-2 kernel: LustreError: 8174:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:509912 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 14:41:19 meerkat-oss-11-2 kernel: LustreError: 2137:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 14:41:19 meerkat-oss-11-2 kernel: LustreError: 2137:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 17718 previous similar messages
Apr 28 14:41:19 meerkat-oss-11-2 kernel: Lustre: 2137:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 14:41:19 meerkat-oss-11-2 kernel: Lustre: 2137:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 17713 previous similar messages
Apr 28 14:44:13 meerkat-oss-5-3 kernel: LustreError: 8020:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 14:44:13 meerkat-oss-5-3 kernel: LustreError: 8020:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 247261 previous similar messages
Apr 28 14:44:14 meerkat-oss-6-1 kernel: LustreError: 3971:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0009 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 14:49:58 meerkat-oss-11-3 kernel: LustreError: 4880:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0025 qtype:usr id:456090 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 14:49:58 meerkat-oss-11-3 kernel: LustreError: 4880:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 14:50:34 meerkat-oss-11-4 kernel: LustreError: 4835:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:456090 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 14:50:34 meerkat-oss-11-4 kernel: LustreError: 4835:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 11 previous similar messages
Apr 28 14:51:19 meerkat-oss-11-2 kernel: LustreError: 20656:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 14:51:19 meerkat-oss-11-2 kernel: LustreError: 20656:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 18036 previous similar messages
Apr 28 14:51:19 meerkat-oss-11-2 kernel: Lustre: 20656:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 14:51:19 meerkat-oss-11-2 kernel: Lustre: 20656:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 18036 previous similar messages
Apr 28 14:54:13 meerkat-oss-5-3 kernel: LustreError: 8020:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 14:54:13 meerkat-oss-5-3 kernel: LustreError: 8020:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 248759 previous similar messages
Apr 28 14:57:08 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 150s: evicting client at 192.168.230.52@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff880249652d80/0x492ec3326a42579b lrc: 3/0,0 mode: PW/PW res: [0xe2a644:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000080010020 nid: 192.168.230.52@tcp remote: 0xc36633803192807e expref: 19 pid: 8354 timeout: 4381104507 lvb_type: 0
Apr 28 14:57:08 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) Skipped 2 previous similar messages
Apr 28 15:00:01 oasis-meerkat CROND[4619]: (root) CMD (/opt/rocks/bin/rocks run host lustre-oss lustre-mds command="uptime" collate=yes | grep -v X11 | /bin/sort -n -r --key=11 | /bin/mail -s "Oasis Meerkat servers uptime" cai@postal.sdsc.edu)
Apr 28 15:00:02 oasis-meerkat rockscommand[4621]: user root called "run host lustre-oss lustre-mds command=uptime collate=yes"
Apr 28 15:01:19 meerkat-oss-11-2 kernel: LustreError: 8374:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:01:19 meerkat-oss-11-2 kernel: LustreError: 8374:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 19766 previous similar messages
Apr 28 15:01:19 meerkat-oss-11-2 kernel: Lustre: 8374:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 15:01:19 meerkat-oss-11-2 kernel: Lustre: 8374:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 19765 previous similar messages
Apr 28 15:01:20 meerkat-oss-11-5 kernel: LustreError: 4835:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0017 qtype:usr id:510226 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:01:20 meerkat-oss-11-5 kernel: LustreError: 4835:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 15:01:21 meerkat-oss-11-3 kernel: LustreError: 5242:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000d qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:01:21 meerkat-oss-11-3 kernel: LustreError: 5242:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 15:03:51 meerkat-oss-6-1 kernel: LustreError: 4989:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0009 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:03:51 meerkat-oss-6-1 kernel: LustreError: 4989:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 15:03:51 meerkat-oss-11-1 kernel: LustreError: 5031:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0013 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:03:51 meerkat-oss-11-1 kernel: LustreError: 5031:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 15:03:51 meerkat-oss-11-5 kernel: LustreError: 3753:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002f qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:03:51 meerkat-oss-11-5 kernel: LustreError: 3753:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 15:03:51 meerkat-oss-11-1 kernel: LustreError: 5031:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0013: fail to locate lqe for id:502676, type:0
Apr 28 15:03:51 meerkat-oss-6-2 kernel: LustreError: 7577:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000a qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:03:51 meerkat-oss-11-4 kernel: LustreError: 5033:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:03:51 meerkat-oss-11-4 kernel: LustreError: 5033:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 13 previous similar messages
Apr 28 15:03:51 meerkat-oss-11-3 kernel: LustreError: 5033:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST000d: fail to locate lqe for id:502676, type:0
Apr 28 15:03:51 meerkat-oss-11-3 kernel: LustreError: 5033:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 9 previous similar messages
Apr 28 15:03:51 meerkat-oss-5-3 kernel: LustreError: 4846:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0030: fail to locate lqe for id:502676, type:0
Apr 28 15:04:13 meerkat-oss-5-3 kernel: LustreError: 3742:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:04:13 meerkat-oss-5-3 kernel: LustreError: 3742:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 248608 previous similar messages
Apr 28 15:09:02 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 192.168.230.54@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff8804ae579880/0x492ec3326a42675b lrc: 3/0,0 mode: PW/PW res: [0xf743a0:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000080010020 nid: 192.168.230.54@tcp remote: 0x3d16df50363636a0 expref: 17 pid: 8370 timeout: 4381818488 lvb_type: 0
Apr 28 15:11:19 meerkat-oss-11-2 kernel: LustreError: 28222:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:11:19 meerkat-oss-11-2 kernel: LustreError: 28222:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 19654 previous similar messages
Apr 28 15:11:19 meerkat-oss-11-2 kernel: Lustre: 28222:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 15:11:19 meerkat-oss-11-2 kernel: Lustre: 28222:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 19652 previous similar messages
Apr 28 15:12:54 meerkat-oss-11-1 kernel: LustreError: 5128:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:12:54 meerkat-oss-11-1 kernel: LustreError: 5128:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 15:12:56 meerkat-oss-11-5 kernel: LustreError: 7763:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002f qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:12:56 meerkat-oss-11-5 kernel: LustreError: 7763:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 15:13:15 meerkat-oss-6-1 kernel: LustreError: 4980:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0031 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:13:15 meerkat-oss-6-1 kernel: LustreError: 4980:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 15:13:25 meerkat-oss-11-3 kernel: LustreError: 4637:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000d qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:13:25 meerkat-oss-11-3 kernel: LustreError: 4637:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 15:13:30 meerkat-oss-6-2 kernel: LustreError: 3742:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:13:35 meerkat-oss-6-2 kernel: Lustre: meerkat-OST0002: haven't heard from client 6ed5a247-3136-4f0a-bfdc-16b3cf4d318c (at 132.249.107.88@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88034d1d8000, cur 1430259215 expire 1430259065 last 1430258988
Apr 28 15:13:35 meerkat-oss-6-2 kernel: Lustre: Skipped 7 previous similar messages
Apr 28 15:13:35 meerkat-oss-5-3 kernel: Lustre: meerkat-OST0000: haven't heard from client 6ed5a247-3136-4f0a-bfdc-16b3cf4d318c (at 132.249.107.88@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bc86ec00, cur 1430259215 expire 1430259065 last 1430258988
Apr 28 15:13:35 meerkat-oss-5-3 kernel: Lustre: Skipped 7 previous similar messages
Apr 28 15:13:35 meerkat-oss-11-1 kernel: Lustre: meerkat-OST002b: haven't heard from client 6ed5a247-3136-4f0a-bfdc-16b3cf4d318c (at 132.249.107.88@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805d082c400, cur 1430259215 expire 1430259065 last 1430258988
Apr 28 15:13:35 meerkat-oss-11-1 kernel: Lustre: Skipped 7 previous similar messages
Apr 28 15:13:35 meerkat-mds-10-1 kernel: Lustre: MGS: haven't heard from client 9d233a59-1c97-4570-dfba-61f0c3fd9f05 (at 132.249.107.88@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802fda55c00, cur 1430259215 expire 1430259065 last 1430258988
Apr 28 15:13:35 meerkat-oss-11-3 kernel: Lustre: meerkat-OST000d: haven't heard from client 6ed5a247-3136-4f0a-bfdc-16b3cf4d318c (at 132.249.107.88@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804f55eb800, cur 1430259215 expire 1430259065 last 1430258988
Apr 28 15:13:35 meerkat-oss-11-3 kernel: Lustre: Skipped 7 previous similar messages
Apr 28 15:13:35 meerkat-oss-11-5 kernel: Lustre: meerkat-OST000f: haven't heard from client 6ed5a247-3136-4f0a-bfdc-16b3cf4d318c (at 132.249.107.88@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805cde1ec00, cur 1430259215 expire 1430259065 last 1430258988
Apr 28 15:13:35 meerkat-oss-11-5 kernel: Lustre: Skipped 7 previous similar messages
Apr 28 15:13:35 meerkat-oss-6-1 kernel: Lustre: meerkat-OST0001: haven't heard from client 6ed5a247-3136-4f0a-bfdc-16b3cf4d318c (at 132.249.107.88@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880474b86400, cur 1430259215 expire 1430259065 last 1430258988
Apr 28 15:13:35 meerkat-oss-11-2 kernel: Lustre: meerkat-OST0004: haven't heard from client 6ed5a247-3136-4f0a-bfdc-16b3cf4d318c (at 132.249.107.88@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88050109e000, cur 1430259215 expire 1430259065 last 1430258988
Apr 28 15:13:35 meerkat-oss-11-4 kernel: Lustre: meerkat-OST0006: haven't heard from client 6ed5a247-3136-4f0a-bfdc-16b3cf4d318c (at 132.249.107.88@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806275a4800, cur 1430259215 expire 1430259065 last 1430258988
Apr 28 15:13:35 meerkat-oss-11-2 kernel: Lustre: Skipped 6 previous similar messages
Apr 28 15:13:35 meerkat-oss-11-4 kernel: Lustre: Skipped 7 previous similar messages
Apr 28 15:14:13 meerkat-oss-5-3 kernel: LustreError: 8031:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:14:13 meerkat-oss-5-3 kernel: LustreError: 8031:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 246085 previous similar messages
Apr 28 15:14:54 meerkat-oss-11-1 kernel: LustreError: 5399:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:14:54 meerkat-oss-11-1 kernel: LustreError: 5399:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 25 previous similar messages
Apr 28 15:14:54 meerkat-oss-11-4 kernel: LustreError: 4876:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:15:01 oasis-meerkat CROND[4781]: (root) CMD (/opt/rocks/bin/rocks run host lustre-mds command="/usr/bin/vmstat -s | grep buffer" collate=yes | grep -v X11 | /bin/mail -s "meerkat MDS memory usage" cai@postal.sdsc.edu)
Apr 28 15:15:01 oasis-meerkat rockscommand[4782]: user root called "run host lustre-mds command=/usr/bin/vmstat -s | grep buffer collate=yes"
Apr 28 15:15:13 meerkat-oss-6-2 kernel: LustreError: 4863:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000a qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:15:25 meerkat-oss-6-2 kernel: LustreError: 5204:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000a qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:18:15 meerkat-oss-11-5 kernel: LustreError: 3800:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002f qtype:usr id:510115 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:18:15 meerkat-oss-6-2 kernel: LustreError: 4868:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0002 qtype:usr id:510115 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:18:15 meerkat-oss-11-1 kernel: LustreError: 3871:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:510115 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:18:15 meerkat-oss-11-1 kernel: LustreError: 3871:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 15:18:15 meerkat-oss-11-5 kernel: LustreError: 3800:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 18 previous similar messages
Apr 28 15:18:15 meerkat-oss-6-1 kernel: LustreError: 4971:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0019 qtype:usr id:510115 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:18:15 meerkat-oss-6-1 kernel: LustreError: 4971:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 15:18:15 meerkat-oss-11-2 kernel: LustreError: 15122:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0014: fail to locate lqe for id:510115, type:0
Apr 28 15:18:15 meerkat-oss-11-4 kernel: LustreError: 4747:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0036: fail to locate lqe for id:510115, type:0
Apr 28 15:18:15 meerkat-oss-11-3 kernel: LustreError: 5050:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST002d: fail to locate lqe for id:510115, type:0
Apr 28 15:18:15 meerkat-oss-11-3 kernel: LustreError: 5050:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 2 previous similar messages
Apr 28 15:18:15 meerkat-oss-6-2 kernel: LustreError: 4863:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0032: fail to locate lqe for id:510115, type:0
Apr 28 15:18:15 meerkat-oss-6-2 kernel: LustreError: 4863:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 3 previous similar messages
Apr 28 15:18:15 meerkat-oss-11-1 kernel: LustreError: 7723:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0013: fail to locate lqe for id:510115, type:0
Apr 28 15:18:15 meerkat-oss-11-1 kernel: LustreError: 7723:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 1 previous similar message
Apr 28 15:18:15 meerkat-oss-5-3 kernel: LustreError: 4854:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0008: fail to locate lqe for id:510115, type:0
Apr 28 15:18:15 meerkat-oss-11-5 kernel: LustreError: 7522:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0037: fail to locate lqe for id:510115, type:0
Apr 28 15:18:15 meerkat-oss-6-1 kernel: LustreError: 7624:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0011: fail to locate lqe for id:510115, type:0
Apr 28 15:20:54 meerkat-oss-6-2 kernel: LustreError: 4676:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000a qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:20:54 meerkat-oss-6-2 kernel: LustreError: 4676:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 11 previous similar messages
Apr 28 15:21:19 meerkat-oss-11-2 kernel: Lustre: 16119:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 15:21:19 meerkat-oss-11-2 kernel: Lustre: 16119:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 19859 previous similar messages
Apr 28 15:21:19 meerkat-oss-11-2 kernel: LustreError: 16120:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:21:19 meerkat-oss-11-2 kernel: LustreError: 16120:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 19881 previous similar messages
Apr 28 15:22:35 meerkat-oss-6-1 kernel: LustreError: 8223:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0031 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:22:35 meerkat-oss-6-1 kernel: LustreError: 8223:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 4 previous similar messages
Apr 28 15:23:14 meerkat-oss-6-2 kernel: LustreError: 4723:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0002 qtype:usr id:510115 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:24:13 meerkat-oss-5-3 kernel: LustreError: 8020:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:24:13 meerkat-oss-5-3 kernel: LustreError: 8020:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 244390 previous similar messages
Apr 28 15:24:24 meerkat-oss-11-3 kernel: LustreError: 4837:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0015 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:24:24 meerkat-oss-11-3 kernel: LustreError: 4837:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 20 previous similar messages
Apr 28 15:25:05 meerkat-oss-11-4 kernel: LustreError: 4688:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:510115 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:25:05 meerkat-oss-11-4 kernel: LustreError: 4688:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 34 previous similar messages
Apr 28 15:26:44 meerkat-oss-6-2 kernel: LustreError: 5187:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:504215 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:26:44 meerkat-oss-6-2 kernel: LustreError: 5187:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 15:28:05 meerkat-oss-6-1 kernel: LustreError: 4691:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0011 qtype:usr id:510115 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:28:05 meerkat-oss-6-1 kernel: LustreError: 4691:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 4 previous similar messages
Apr 28 15:28:05 meerkat-oss-11-1 kernel: LustreError: 5234:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:510115 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:28:05 meerkat-oss-11-1 kernel: LustreError: 5234:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 13 previous similar messages
Apr 28 15:28:28 meerkat-oss-11-5 kernel: LustreError: 4828:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0037 qtype:usr id:504215 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:28:28 meerkat-oss-11-5 kernel: LustreError: 4828:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 21 previous similar messages
Apr 28 15:30:52 meerkat-oss-11-2 kernel: LustreError: 23435:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST001c: fail to locate lqe for id:510115, type:0
Apr 28 15:30:52 meerkat-oss-11-2 kernel: LustreError: 23435:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 8 previous similar messages
Apr 28 15:31:19 meerkat-oss-11-2 kernel: Lustre: 3684:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 15:31:19 meerkat-oss-11-2 kernel: Lustre: 3684:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 19906 previous similar messages
Apr 28 15:31:19 meerkat-oss-11-2 kernel: LustreError: 3685:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 15:31:19 meerkat-oss-11-2 kernel: LustreError: 3685:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 19916 previous similar messages
Apr 28 17:11:19 meerkat-oss-11-2 kernel: Lustre: 28002:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 17:11:19 meerkat-oss-11-2 kernel: Lustre: 28002:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 22724 previous similar messages
Apr 28 17:11:19 meerkat-oss-11-2 kernel: LustreError: 28006:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:11:19 meerkat-oss-11-2 kernel: LustreError: 28006:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 22726 previous similar messages
Apr 28 17:12:55 meerkat-oss-11-5 kernel: LustreError: 4613:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0037 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:12:55 meerkat-oss-11-5 kernel: LustreError: 4613:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 17:13:44 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:13:44 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 26567 previous similar messages
Apr 28 17:14:13 meerkat-oss-5-3 kernel: LustreError: 3743:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:14:13 meerkat-oss-5-3 kernel: LustreError: 3743:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 229797 previous similar messages
Apr 28 17:15:01 oasis-meerkat CROND[5422]: (root) CMD (/opt/rocks/bin/rocks run host lustre-mds command="/usr/bin/vmstat -s | grep buffer" collate=yes | grep -v X11 | /bin/mail -s "meerkat MDS memory usage" cai@postal.sdsc.edu)
Apr 28 17:15:02 oasis-meerkat rockscommand[5423]: user root called "run host lustre-mds command=/usr/bin/vmstat -s | grep buffer collate=yes"
Apr 28 17:15:27 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:15:27 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 13220 previous similar messages
Apr 28 17:17:37 meerkat-oss-6-2 kernel: LustreError: 5187:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002a qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:17:37 meerkat-oss-6-2 kernel: LustreError: 5187:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 17:21:19 meerkat-oss-11-2 kernel: Lustre: 18839:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 17:21:19 meerkat-oss-11-2 kernel: Lustre: 18839:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 22795 previous similar messages
Apr 28 17:21:19 meerkat-oss-11-2 kernel: LustreError: 18842:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:21:19 meerkat-oss-11-2 kernel: LustreError: 18842:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 22797 previous similar messages
Apr 28 17:21:42 meerkat-oss-6-1 kernel: LustreError: 4691:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0019 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:21:42 meerkat-oss-6-1 kernel: LustreError: 4691:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 4 previous similar messages
Apr 28 17:22:48 meerkat-oss-6-2 kernel: LustreError: 4892:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0002 qtype:usr id:502699 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:22:48 meerkat-oss-6-2 kernel: LustreError: 4892:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 17:22:55 meerkat-oss-11-3 kernel: LustreError: 3883:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000d qtype:usr id:510900 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:22:55 meerkat-oss-11-3 kernel: LustreError: 3883:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 17:23:15 meerkat-oss-11-1 kernel: LustreError: 5391:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:23:45 meerkat-oss-6-2 kernel: LustreError: 5054:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000a qtype:usr id:510900 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:23:46 meerkat-oss-11-4 kernel: LustreError: 4802:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:23:46 meerkat-oss-11-4 kernel: LustreError: 4802:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 30332 previous similar messages
Apr 28 17:24:13 meerkat-oss-5-3 kernel: LustreError: 4800:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:24:13 meerkat-oss-5-3 kernel: LustreError: 4800:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 229018 previous similar messages
Apr 28 17:25:29 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:25:29 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 13220 previous similar messages
Apr 28 17:29:01 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 150s: evicting client at 192.168.230.54@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff8805c3383280/0x492ec3326a4325c0 lrc: 3/0,0 mode: PW/PW res: [0xf7449b:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->262143) flags: 0x60000000010020 nid: 192.168.230.54@tcp remote: 0x3d16df503c5057c2 expref: 17 pid: 5479 timeout: 4390217894 lvb_type: 0
Apr 28 17:29:32 meerkat-oss-11-5 kernel: LustreError: 8191:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0017 qtype:usr id:510900 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:29:32 meerkat-oss-11-5 kernel: LustreError: 8191:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 17:30:42 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.51@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff8805f4afc4c0/0x492ec3326a4323d6 lrc: 3/0,0 mode: PW/PW res: [0xf74486:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000080010020 nid: 192.168.230.51@tcp remote: 0x9bbc69787fb37142 expref: 24 pid: 15124 timeout: 4390318562 lvb_type: 0
Apr 28 17:31:19 meerkat-oss-11-2 kernel: Lustre: 9553:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 17:31:19 meerkat-oss-11-2 kernel: Lustre: 9553:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 22781 previous similar messages
Apr 28 17:31:19 meerkat-oss-11-2 kernel: LustreError: 9556:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:31:19 meerkat-oss-11-2 kernel: LustreError: 9556:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 22784 previous similar messages
Apr 28 17:32:24 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff880105685380/0x492ec3326a42e07d lrc: 3/0,0 mode: PW/PW res: [0xf74032:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000080010020 nid: 192.168.230.53@tcp remote: 0xab5cfbed7133fd15 expref: 25 pid: 8391 timeout: 4390420592 lvb_type: 0
Apr 28 17:33:00 meerkat-oss-11-3 kernel: LustreError: 4827:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002d qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:33:00 meerkat-oss-11-3 kernel: LustreError: 4827:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 7 previous similar messages
Apr 28 17:33:48 meerkat-oss-11-4 kernel: LustreError: 4844:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:33:48 meerkat-oss-11-4 kernel: LustreError: 4844:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 29314 previous similar messages
Apr 28 17:34:13 meerkat-oss-5-3 kernel: LustreError: 8033:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:34:13 meerkat-oss-5-3 kernel: LustreError: 8033:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 226574 previous similar messages
Apr 28 17:34:41 meerkat-oss-6-2 kernel: LustreError: 5197:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:35:03 meerkat-oss-6-1 kernel: LustreError: 4842:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0019 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:35:03 meerkat-oss-6-1 kernel: LustreError: 4842:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 11 previous similar messages
Apr 28 17:35:29 meerkat-oss-11-4 kernel: LustreError: 4817:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:35:29 meerkat-oss-11-4 kernel: LustreError: 4817:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 18894 previous similar messages
Apr 28 17:38:37 meerkat-oss-11-1 kernel: LustreError: 5005:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0013 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:40:16 meerkat-oss-11-5 kernel: LustreError: 5120:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0017 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:40:16 meerkat-oss-11-5 kernel: LustreError: 5120:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 6 previous similar messages
Apr 28 17:41:19 meerkat-oss-11-2 kernel: Lustre: 32265:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 17:41:19 meerkat-oss-11-2 kernel: Lustre: 32265:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 22543 previous similar messages
Apr 28 17:41:19 meerkat-oss-11-2 kernel: LustreError: 32269:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:41:19 meerkat-oss-11-2 kernel: LustreError: 32269:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 22546 previous similar messages
Apr 28 17:43:23 meerkat-oss-11-1 kernel: LustreError: 4778:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:508145 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:43:24 meerkat-oss-11-3 kernel: LustreError: 4825:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000d qtype:usr id:508145 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:43:24 meerkat-oss-11-3 kernel: LustreError: 4825:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 4 previous similar messages
Apr 28 17:43:26 meerkat-oss-11-1 kernel: LustreError: 4778:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST001b qtype:usr id:508145 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:43:26 meerkat-oss-11-1 kernel: LustreError: 4778:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 17:43:26 meerkat-oss-6-2 kernel: LustreError: 5054:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:508145 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:43:48 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:460 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:43:48 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 181551 previous similar messages
Apr 28 17:43:50 meerkat-oss-6-2 kernel: LustreError: 5190:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000a qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:43:50 meerkat-oss-6-2 kernel: LustreError: 5190:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 17:44:13 meerkat-oss-5-3 kernel: LustreError: 8031:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:44:13 meerkat-oss-5-3 kernel: LustreError: 8031:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 226882 previous similar messages
Apr 28 17:44:42 meerkat-oss-6-2 kernel: LustreError: 5054:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000a qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:45:29 meerkat-oss-11-4 kernel: LustreError: 4604:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:45:29 meerkat-oss-11-4 kernel: LustreError: 4604:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 67746 previous similar messages
Apr 28 17:47:17 meerkat-oss-6-1 kernel: LustreError: 5437:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0031 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:47:17 meerkat-oss-6-1 kernel: LustreError: 5437:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 13 previous similar messages
Apr 28 17:48:00 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 192.168.230.52@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff8801d8e09480/0x492ec3326a432d29 lrc: 3/0,0 mode: PW/PW res: [0xf7404a:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000080010020 nid: 192.168.230.52@tcp remote: 0xc3663380420f21b1 expref: 19 pid: 8439 timeout: 4391356187 lvb_type: 0
Apr 28 17:48:29 meerkat-oss-11-1 kernel: LustreError: 5221:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:48:29 meerkat-oss-11-1 kernel: LustreError: 5221:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 17:49:00 meerkat-oss-6-2 kernel: LustreError: 5269:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0002 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:51:19 meerkat-oss-11-2 kernel: Lustre: 22896:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 17:51:19 meerkat-oss-11-2 kernel: Lustre: 22896:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 22596 previous similar messages
Apr 28 17:51:19 meerkat-oss-11-2 kernel: LustreError: 22901:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:51:19 meerkat-oss-11-2 kernel: LustreError: 22901:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 22607 previous similar messages
Apr 28 17:53:50 meerkat-oss-11-4 kernel: LustreError: 4835:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:53:50 meerkat-oss-11-4 kernel: LustreError: 4835:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 145986 previous similar messages
Apr 28 17:54:13 meerkat-oss-5-3 kernel: LustreError: 3743:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:54:13 meerkat-oss-5-3 kernel: LustreError: 3743:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 227676 previous similar messages
Apr 28 17:55:33 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:55:33 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 44037 previous similar messages
Apr 28 17:57:00 meerkat-oss-11-1 kernel: LustreError: 3882:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST001b qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:57:00 meerkat-oss-11-1 kernel: LustreError: 3882:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 17:59:48 meerkat-oss-6-1 kernel: LustreError: 5432:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0019 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 17:59:48 meerkat-oss-6-1 kernel: LustreError: 5432:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 18:00:01 oasis-meerkat CROND[5604]: (root) CMD (/opt/rocks/bin/rocks run host lustre-oss lustre-mds command="uptime" collate=yes | grep -v X11 | /bin/sort -n -r --key=11 | /bin/mail -s "Oasis Meerkat servers uptime" cai@postal.sdsc.edu)
Apr 28 18:00:02 oasis-meerkat rockscommand[5606]: user root called "run host lustre-oss lustre-mds command=uptime collate=yes"
Apr 28 18:00:15 meerkat-oss-11-1 kernel: LustreError: 5012:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0013 qtype:usr id:510900 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:00:28 meerkat-oss-6-2 kernel: LustreError: 5054:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:01:19 meerkat-oss-11-2 kernel: Lustre: 13736:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 18:01:19 meerkat-oss-11-2 kernel: Lustre: 13736:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 22744 previous similar messages
Apr 28 18:01:19 meerkat-oss-11-2 kernel: LustreError: 13742:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:01:19 meerkat-oss-11-2 kernel: LustreError: 13742:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 22748 previous similar messages
Apr 28 18:03:54 meerkat-oss-11-4 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:03:54 meerkat-oss-11-4 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 26562 previous similar messages
Apr 28 18:04:13 meerkat-oss-5-3 kernel: LustreError: 8033:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:04:13 meerkat-oss-5-3 kernel: LustreError: 8033:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 226946 previous similar messages
Apr 28 18:04:27 meerkat-oss-6-2 kernel: LustreError: 4972:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002a qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:05:37 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:05:37 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 13220 previous similar messages
Apr 28 18:08:17 meerkat-oss-11-5 kernel: LustreError: 4822:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0017 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:08:17 meerkat-oss-11-5 kernel: LustreError: 4822:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 19 previous similar messages
Apr 28 18:09:36 meerkat-oss-11-5 kernel: LustreError: 8181:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0037 qtype:usr id:510900 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:09:36 meerkat-oss-11-5 kernel: LustreError: 8181:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 18:11:19 meerkat-oss-11-2 kernel: Lustre: 4128:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 18:11:19 meerkat-oss-11-2 kernel: Lustre: 4128:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 22693 previous similar messages
Apr 28 18:11:19 meerkat-oss-11-2 kernel: LustreError: 4131:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:11:19 meerkat-oss-11-2 kernel: LustreError: 4131:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 22691 previous similar messages
Apr 28 18:13:57 meerkat-oss-11-4 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:13:57 meerkat-oss-11-4 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 26559 previous similar messages
Apr 28 18:14:13 meerkat-oss-5-3 kernel: LustreError: 3743:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:14:13 meerkat-oss-5-3 kernel: LustreError: 3743:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 227057 previous similar messages
Apr 28 18:14:15 meerkat-oss-6-1 kernel: LustreError: 3851:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0019 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:14:15 meerkat-oss-6-1 kernel: LustreError: 3851:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 18:14:19 meerkat-oss-11-5 kernel: LustreError: 3763:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000f qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:14:33 meerkat-oss-11-3 kernel: LustreError: 4825:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0025 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:14:33 meerkat-oss-11-3 kernel: LustreError: 4825:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 7 previous similar messages
Apr 28 18:15:01 oasis-meerkat CROND[5751]: (root) CMD (/opt/rocks/bin/rocks run host lustre-mds command="/usr/bin/vmstat -s | grep buffer" collate=yes | grep -v X11 | /bin/mail -s "meerkat MDS memory usage" cai@postal.sdsc.edu)
Apr 28 18:15:01 oasis-meerkat rockscommand[5752]: user root called "run host lustre-mds command=/usr/bin/vmstat -s | grep buffer collate=yes"
Apr 28 18:15:40 meerkat-oss-11-4 kernel: LustreError: 4839:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:15:40 meerkat-oss-11-4 kernel: LustreError: 4839:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 13220 previous similar messages
Apr 28 18:21:19 meerkat-oss-11-2 kernel: Lustre: 27413:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 18:21:19 meerkat-oss-11-2 kernel: Lustre: 27413:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 22850 previous similar messages
Apr 28 18:21:19 meerkat-oss-11-2 kernel: LustreError: 27417:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:21:19 meerkat-oss-11-2 kernel: LustreError: 27417:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 22855 previous similar messages
Apr 28 18:24:00 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:24:00 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 26559 previous similar messages
Apr 28 18:24:13 meerkat-oss-5-3 kernel: LustreError: 3741:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:24:13 meerkat-oss-5-3 kernel: LustreError: 3741:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 228890 previous similar messages
Apr 28 18:24:33 meerkat-oss-6-1 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0009 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:24:33 meerkat-oss-6-1 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 18:25:05 meerkat-oss-11-5 kernel: LustreError: 8181:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000f qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:25:05 meerkat-oss-11-5 kernel: LustreError: 8181:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 18:25:43 meerkat-oss-11-4 kernel: LustreError: 4839:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:25:43 meerkat-oss-11-4 kernel: LustreError: 4839:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 13220 previous similar messages
Apr 28 18:26:46 meerkat-oss-11-1 kernel: LustreError: 5374:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST001b qtype:usr id:504796 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:26:46 meerkat-oss-6-2 kernel: LustreError: 3784:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:504796 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:26:56 meerkat-oss-6-2 kernel: LustreError: 5208:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:504796 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:26:56 meerkat-oss-11-1 kernel: LustreError: 5374:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST001b qtype:usr id:504796 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:26:56 meerkat-oss-11-1 kernel: LustreError: 5374:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 18:27:47 meerkat-oss-11-1 kernel: LustreError: 5012:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0013 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:27:47 meerkat-oss-11-1 kernel: LustreError: 5012:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 18:29:50 meerkat-oss-6-2 kernel: LustreError: 5271:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000a qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:30:04 meerkat-oss-11-1 kernel: LustreError: 5012:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0013 qtype:usr id:504888 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:31:00 meerkat-oss-6-2 kernel: LustreError: 5190:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0002 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:31:13 meerkat-oss-6-2 kernel: LustreError: 3730:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002a qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:31:19 meerkat-oss-11-2 kernel: Lustre: 18494:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 18:31:19 meerkat-oss-11-2 kernel: Lustre: 18494:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23036 previous similar messages
Apr 28 18:31:19 meerkat-oss-11-2 kernel: LustreError: 18498:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:31:19 meerkat-oss-11-2 kernel: LustreError: 18498:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23037 previous similar messages
Apr 28 18:32:45 meerkat-oss-11-1 kernel: LustreError: 5388:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:32:45 meerkat-oss-11-1 kernel: LustreError: 5388:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 18:34:03 meerkat-oss-11-4 kernel: LustreError: 4807:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:34:03 meerkat-oss-11-4 kernel: LustreError: 4807:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 26565 previous similar messages
Apr 28 18:34:57 meerkat-oss-11-3 kernel: LustreError: 4907:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000d qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:34:57 meerkat-oss-11-3 kernel: LustreError: 4907:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 18:35:29 meerkat-oss-11-3 kernel: LustreError: 4914:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000d qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:35:29 meerkat-oss-11-3 kernel: LustreError: 4914:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 18:35:29 meerkat-oss-5-3 kernel: LustreError: 5308:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0008 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:35:29 meerkat-oss-5-3 kernel: LustreError: 5308:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 175786 previous similar messages
Apr 28 18:35:46 meerkat-oss-11-4 kernel: LustreError: 4807:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:35:46 meerkat-oss-11-4 kernel: LustreError: 4807:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 13220 previous similar messages
Apr 28 18:36:02 meerkat-oss-11-3 kernel: LustreError: 4850:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0025 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:36:02 meerkat-oss-11-3 kernel: LustreError: 4850:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 18:37:40 meerkat-oss-6-1 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0009 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:37:40 meerkat-oss-6-1 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 18:39:30 meerkat-oss-11-3 kernel: LustreError: 4907:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002d qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:39:30 meerkat-oss-11-3 kernel: LustreError: 4907:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 18:40:00 meerkat-oss-11-5 kernel: LustreError: 5239:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000f qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:40:00 meerkat-oss-11-5 kernel: LustreError: 5239:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 4 previous similar messages
Apr 28 18:40:15 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 150s: evicting client at 192.168.230.54@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff880525d71c80/0x492ec3326a433df3 lrc: 3/0,0 mode: PW/PW res: [0xe2a2ed:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000080010020 nid: 192.168.230.54@tcp remote: 0x3d16df503e603c56 expref: 22 pid: 8205 timeout: 4394491637 lvb_type: 0
Apr 28 18:41:19 meerkat-oss-11-2 kernel: Lustre: 9567:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 18:41:19 meerkat-oss-11-2 kernel: Lustre: 9567:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23138 previous similar messages
Apr 28 18:41:19 meerkat-oss-11-2 kernel: LustreError: 9571:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:41:19 meerkat-oss-11-2 kernel: LustreError: 9571:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23139 previous similar messages
Apr 28 18:42:08 meerkat-oss-11-3 kernel: LustreError: 4863:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0015 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:42:08 meerkat-oss-11-3 kernel: LustreError: 4863:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 18:44:05 meerkat-oss-11-4 kernel: LustreError: 4807:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:44:05 meerkat-oss-11-4 kernel: LustreError: 4807:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 26561 previous similar messages
Apr 28 18:45:48 meerkat-oss-11-4 kernel: LustreError: 4844:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:45:48 meerkat-oss-11-4 kernel: LustreError: 4844:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 13220 previous similar messages
Apr 28 18:45:50 meerkat-oss-5-3 kernel: LustreError: 3741:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:502676 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:45:50 meerkat-oss-5-3 kernel: LustreError: 3741:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 18:45:51 meerkat-oss-11-3 kernel: LustreError: 4901:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000d qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:49:39 meerkat-oss-6-2 kernel: LustreError: 8174:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:51:19 meerkat-oss-11-2 kernel: Lustre: 460:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 18:51:19 meerkat-oss-11-2 kernel: Lustre: 460:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23193 previous similar messages
Apr 28 18:51:20 meerkat-oss-11-2 kernel: LustreError: 465:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:51:20 meerkat-oss-11-2 kernel: LustreError: 465:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23194 previous similar messages
Apr 28 18:53:48 meerkat-oss-11-1 kernel: LustreError: 5392:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:54:07 meerkat-oss-11-4 kernel: LustreError: 4802:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:54:07 meerkat-oss-11-4 kernel: LustreError: 4802:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 26560 previous similar messages
Apr 28 18:54:38 meerkat-oss-11-5 kernel: Lustre: meerkat-OST000f: Client f5b33f01-51f3-96f6-f709-bb48716dddd6 (at 192.168.5.142@tcp) reconnecting
Apr 28 18:54:38 meerkat-oss-11-5 kernel: Lustre: meerkat-OST0017: Client f5b33f01-51f3-96f6-f709-bb48716dddd6 (at 192.168.5.142@tcp) reconnecting
Apr 28 18:54:38 meerkat-oss-11-5 kernel: Lustre: Skipped 2 previous similar messages
Apr 28 18:54:38 meerkat-oss-11-5 kernel: Lustre: Skipped 2 previous similar messages
Apr 28 18:54:38 meerkat-oss-11-1 kernel: Lustre: meerkat-OST0003: Client f5b33f01-51f3-96f6-f709-bb48716dddd6 (at 192.168.5.142@tcp) reconnecting
Apr 28 18:54:38 meerkat-oss-11-1 kernel: Lustre: Skipped 2 previous similar messages
Apr 28 18:54:38 meerkat-mds-10-1 kernel: Lustre: MGS: Client db47b52a-3887-9515-9513-e71530c0845d (at 192.168.5.142@tcp) reconnecting
Apr 28 18:54:38 meerkat-mds-10-1 kernel: Lustre: Skipped 5 previous similar messages
Apr 28 18:54:38 meerkat-oss-5-3 kernel: Lustre: meerkat-OST0010: Client f5b33f01-51f3-96f6-f709-bb48716dddd6 (at 192.168.5.142@tcp) reconnecting
Apr 28 18:54:38 meerkat-oss-5-3 kernel: Lustre: Skipped 1 previous similar message
Apr 28 18:54:38 meerkat-oss-6-2 kernel: Lustre: meerkat-OST0012: Client f5b33f01-51f3-96f6-f709-bb48716dddd6 (at 192.168.5.142@tcp) reconnecting
Apr 28 18:54:38 meerkat-oss-11-4 kernel: Lustre: meerkat-OST001e: Client f5b33f01-51f3-96f6-f709-bb48716dddd6 (at 192.168.5.142@tcp) reconnecting
Apr 28 18:54:38 meerkat-oss-11-2 kernel: Lustre: meerkat-OST000c: Client f5b33f01-51f3-96f6-f709-bb48716dddd6 (at 192.168.5.142@tcp) reconnecting
Apr 28 18:54:38 meerkat-oss-11-4 kernel: Lustre: Skipped 6 previous similar messages
Apr 28 18:54:38 meerkat-oss-11-3 kernel: Lustre: meerkat-OST000d: Client f5b33f01-51f3-96f6-f709-bb48716dddd6 (at 192.168.5.142@tcp) reconnecting
Apr 28 18:54:39 meerkat-oss-6-1 kernel: Lustre: meerkat-OST0011: Client f5b33f01-51f3-96f6-f709-bb48716dddd6 (at 192.168.5.142@tcp) reconnecting
Apr 28 18:54:39 meerkat-oss-6-1 kernel: Lustre: Skipped 1 previous similar message
Apr 28 18:54:42 meerkat-oss-11-3 kernel: LustreError: 4833:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0015 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:54:42 meerkat-oss-11-3 kernel: LustreError: 4833:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 18:54:47 meerkat-oss-11-5 kernel: LustreError: 8182:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002f qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:54:47 meerkat-oss-11-5 kernel: LustreError: 8182:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 18:55:35 meerkat-oss-6-2 kernel: LustreError: 5054:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0002 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:55:50 meerkat-oss-11-4 kernel: LustreError: 4802:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:55:50 meerkat-oss-11-4 kernel: LustreError: 4802:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 13220 previous similar messages
Apr 28 18:56:07 meerkat-oss-5-3 kernel: LustreError: 5299:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0008 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 18:56:07 meerkat-oss-5-3 kernel: LustreError: 5299:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 19:00:01 oasis-meerkat CROND[5816]: (root) CMD (/opt/rocks/bin/rocks run host lustre-oss lustre-mds command="uptime" collate=yes | grep -v X11 | /bin/sort -n -r --key=11 | /bin/mail -s "Oasis Meerkat servers uptime" cai@postal.sdsc.edu)
Apr 28 19:00:01 oasis-meerkat rockscommand[5818]: user root called "run host lustre-oss lustre-mds command=uptime collate=yes"
Apr 28 19:01:19 meerkat-oss-11-2 kernel: Lustre: 24266:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 19:01:19 meerkat-oss-11-2 kernel: Lustre: 24266:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23233 previous similar messages
Apr 28 19:01:20 meerkat-oss-11-2 kernel: LustreError: 24271:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:01:20 meerkat-oss-11-2 kernel: LustreError: 24271:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23237 previous similar messages
Apr 28 19:01:45 meerkat-oss-6-2 kernel: LustreError: 3743:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002a qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:01:45 meerkat-oss-11-1 kernel: LustreError: 5384:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:01:45 meerkat-oss-11-1 kernel: LustreError: 5384:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 19:04:08 meerkat-oss-11-4 kernel: LustreError: 4802:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:04:08 meerkat-oss-11-4 kernel: LustreError: 4802:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 26569 previous similar messages
Apr 28 19:04:51 meerkat-oss-11-3 kernel: LustreError: 3881:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0015 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:04:51 meerkat-oss-11-3 kernel: LustreError: 3881:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 7 previous similar messages
Apr 28 19:05:49 meerkat-oss-6-1 kernel: LustreError: 4807:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0009 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:05:51 meerkat-oss-11-4 kernel: LustreError: 4817:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:05:51 meerkat-oss-11-4 kernel: LustreError: 4817:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 13220 previous similar messages
Apr 28 19:06:15 meerkat-oss-5-3 kernel: LustreError: 3743:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:06:15 meerkat-oss-5-3 kernel: LustreError: 3743:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 4 previous similar messages
Apr 28 19:06:48 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 223s: evicting client at 192.168.230.52@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff8801d20c4980/0x492ec3326a434a25 lrc: 3/0,0 mode: PW/PW res: [0xe2a303:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000080010020 nid: 192.168.230.52@tcp remote: 0xc3663380464232dc expref: 24 pid: 8330 timeout: 4396084164 lvb_type: 0
Apr 28 19:07:35 meerkat-oss-6-1 kernel: LustreError: 4691:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0019 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:10:35 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 226s: evicting client at 192.168.230.51@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff8806149cdc80/0x492ec3326a4341f1 lrc: 3/0,0 mode: PW/PW res: [0xf743cd:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000080010020 nid: 192.168.230.51@tcp remote: 0x9bbc69787fb46ee6 expref: 23 pid: 6033 timeout: 4396311549 lvb_type: 0
Apr 28 19:11:19 meerkat-oss-11-2 kernel: Lustre: 15632:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 19:11:19 meerkat-oss-11-2 kernel: Lustre: 15632:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23318 previous similar messages
Apr 28 19:11:20 meerkat-oss-11-2 kernel: LustreError: 15638:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:11:20 meerkat-oss-11-2 kernel: LustreError: 15638:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23321 previous similar messages
Apr 28 19:11:21 meerkat-oss-11-1 kernel: LustreError: 3887:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:11:21 meerkat-oss-11-5 kernel: LustreError: 4873:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0037 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:11:21 meerkat-oss-11-5 kernel: LustreError: 4873:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 4 previous similar messages
Apr 28 19:11:22 meerkat-oss-6-2 kernel: LustreError: 5208:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002a qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:12:31 meerkat-oss-11-1 kernel: LustreError: 5381:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST001b qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:12:31 meerkat-oss-11-1 kernel: LustreError: 5381:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 19:14:10 meerkat-oss-11-4 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:14:10 meerkat-oss-11-4 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 26563 previous similar messages
Apr 28 19:15:01 oasis-meerkat CROND[5936]: (root) CMD (/opt/rocks/bin/rocks run host lustre-mds command="/usr/bin/vmstat -s | grep buffer" collate=yes | grep -v X11 | /bin/mail -s "meerkat MDS memory usage" cai@postal.sdsc.edu)
Apr 28 19:15:01 oasis-meerkat rockscommand[5937]: user root called "run host lustre-mds command=/usr/bin/vmstat -s | grep buffer collate=yes"
Apr 28 19:15:10 meerkat-oss-11-3 kernel: LustreError: 4863:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0015 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:15:10 meerkat-oss-11-3 kernel: LustreError: 4863:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 19:15:52 meerkat-oss-11-4 kernel: LustreError: 4817:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:15:52 meerkat-oss-11-4 kernel: LustreError: 4817:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 13220 previous similar messages
Apr 28 19:16:12 meerkat-oss-6-2 kernel: LustreError: 5207:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000a qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:17:00 meerkat-oss-6-2 kernel: LustreError: 5137:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002a qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:17:00 meerkat-oss-11-1 kernel: LustreError: 3889:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST001b qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:17:00 meerkat-oss-11-1 kernel: LustreError: 3889:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 19:19:12 meerkat-oss-11-1 kernel: LustreError: 5005:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0013 qtype:usr id:504888 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:19:12 meerkat-oss-11-1 kernel: LustreError: 5005:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 19:21:19 meerkat-oss-11-2 kernel: Lustre: 6439:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 19:21:19 meerkat-oss-11-2 kernel: Lustre: 6439:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 22946 previous similar messages
Apr 28 19:21:20 meerkat-oss-11-2 kernel: LustreError: 6445:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:21:20 meerkat-oss-11-2 kernel: LustreError: 6445:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 22948 previous similar messages
Apr 28 19:21:34 meerkat-oss-11-1 kernel: LustreError: 3888:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:26:41 meerkat-oss-11-5 kernel: LustreError: 5239:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0017 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:26:41 meerkat-oss-11-5 kernel: LustreError: 5239:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 4 previous similar messages
Apr 28 19:31:19 meerkat-oss-11-2 kernel: Lustre: 29999:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 19:31:19 meerkat-oss-11-2 kernel: Lustre: 29999:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23321 previous similar messages
Apr 28 19:31:20 meerkat-oss-11-2 kernel: LustreError: 30006:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:31:20 meerkat-oss-11-2 kernel: LustreError: 30006:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23323 previous similar messages
Apr 28 19:32:21 meerkat-oss-11-4 kernel: LustreError: 4604:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 19:32:21 meerkat-oss-11-4 kernel: LustreError: 4604:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 11578 previous similar messages
Apr 28 19:32:24 meerkat-oss-6-1 kernel: LustreError: 5437:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0009 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:32:24 meerkat-oss-6-1 kernel: LustreError: 5437:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 19:32:25 meerkat-oss-11-4 kernel: LustreError: 4604:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 19:32:25 meerkat-oss-11-4 kernel: LustreError: 4604:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 3502 previous similar messages
Apr 28 19:35:49 meerkat-oss-5-3 kernel: LustreError: 3745:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0010 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:37:02 meerkat-oss-11-3 kernel: LustreError: 4908:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000d qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:37:02 meerkat-oss-11-3 kernel: LustreError: 4908:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 7 previous similar messages
Apr 28 19:37:36 meerkat-oss-11-5 kernel: LustreError: 3763:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002f qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:37:36 meerkat-oss-11-5 kernel: LustreError: 3763:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 19:37:36 meerkat-oss-11-1 kernel: LustreError: 5381:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST001b qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:37:36 meerkat-oss-11-1 kernel: LustreError: 5381:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 19:37:53 meerkat-oss-5-3 kernel: LustreError: 3900:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:37:53 meerkat-oss-6-2 kernel: LustreError: 4877:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000a qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:37:53 meerkat-oss-6-1 kernel: LustreError: 4775:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0009 qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:37:53 meerkat-oss-6-1 kernel: LustreError: 4775:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0009: fail to locate lqe for id:507750, type:0
Apr 28 19:37:53 meerkat-oss-6-1 kernel: LustreError: 4775:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 2 previous similar messages
Apr 28 19:37:53 meerkat-oss-6-2 kernel: LustreError: 4675:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST000a: fail to locate lqe for id:507750, type:0
Apr 28 19:37:53 meerkat-oss-6-2 kernel: LustreError: 4675:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 5 previous similar messages
Apr 28 19:37:53 meerkat-oss-11-1 kernel: LustreError: 5020:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST001b: fail to locate lqe for id:507750, type:0
Apr 28 19:37:53 meerkat-oss-11-1 kernel: LustreError: 5020:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 3 previous similar messages
Apr 28 19:37:53 meerkat-oss-11-3 kernel: LustreError: 5059:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0025: fail to locate lqe for id:507750, type:0
Apr 28 19:37:53 meerkat-oss-11-3 kernel: LustreError: 5059:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 6 previous similar messages
Apr 28 19:37:53 meerkat-oss-11-4 kernel: LustreError: 7713:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0036: fail to locate lqe for id:507750, type:0
Apr 28 19:37:54 meerkat-oss-5-3 kernel: LustreError: 4843:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0010: fail to locate lqe for id:507750, type:0
Apr 28 19:37:54 meerkat-oss-5-3 kernel: LustreError: 4843:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 3 previous similar messages
Apr 28 19:37:54 meerkat-oss-11-2 kernel: LustreError: 6033:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST000c: fail to locate lqe for id:507750, type:0
Apr 28 19:37:54 meerkat-oss-11-5 kernel: LustreError: 7515:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST002f: fail to locate lqe for id:507750, type:0
Apr 28 19:37:54 meerkat-oss-11-5 kernel: LustreError: 7515:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 5 previous similar messages
Apr 28 19:38:56 meerkat-oss-11-1 kernel: LustreError: 5390:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST001b qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:38:56 meerkat-oss-11-1 kernel: LustreError: 5390:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 4 previous similar messages
Apr 28 19:39:34 meerkat-oss-11-3 kernel: LustreError: 5042:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002d qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:39:34 meerkat-oss-11-3 kernel: LustreError: 5042:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 15 previous similar messages
Apr 28 19:39:34 meerkat-oss-6-2 kernel: LustreError: 4871:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000a qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:39:34 meerkat-oss-6-2 kernel: LustreError: 4871:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 11 previous similar messages
Apr 28 19:39:34 meerkat-oss-6-1 kernel: LustreError: 3971:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0009 qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:39:34 meerkat-oss-6-1 kernel: LustreError: 3971:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 19:39:34 meerkat-oss-11-1 kernel: LustreError: 4681:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST001b: fail to locate lqe for id:507750, type:0
Apr 28 19:39:34 meerkat-oss-11-1 kernel: LustreError: 4681:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 1 previous similar message
Apr 28 19:39:34 meerkat-oss-5-3 kernel: LustreError: 4493:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0008: fail to locate lqe for id:507750, type:0
Apr 28 19:39:34 meerkat-oss-5-3 kernel: LustreError: 4493:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 1 previous similar message
Apr 28 19:39:34 meerkat-oss-11-2 kernel: LustreError: 23116:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0014: fail to locate lqe for id:507750, type:0
Apr 28 19:39:34 meerkat-oss-11-3 kernel: LustreError: 3919:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST002d: fail to locate lqe for id:507750, type:0
Apr 28 19:39:34 meerkat-oss-11-3 kernel: LustreError: 3919:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 5 previous similar messages
Apr 28 19:39:34 meerkat-oss-6-2 kernel: LustreError: 4871:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST000a: fail to locate lqe for id:507750, type:0
Apr 28 19:39:34 meerkat-oss-6-2 kernel: LustreError: 4871:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 4 previous similar messages
Apr 28 19:39:34 meerkat-oss-6-1 kernel: LustreError: 4987:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0009: fail to locate lqe for id:507750, type:0
Apr 28 19:39:34 meerkat-oss-11-4 kernel: LustreError: 5000:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0036: fail to locate lqe for id:507750, type:0
Apr 28 19:39:34 meerkat-oss-11-4 kernel: LustreError: 5000:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 3 previous similar messages
Apr 28 19:39:34 meerkat-oss-11-5 kernel: LustreError: 7518:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST002f: fail to locate lqe for id:507750, type:0
Apr 28 19:39:34 meerkat-oss-11-5 kernel: LustreError: 7518:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 2 previous similar messages
Apr 28 19:39:39 meerkat-oss-5-3 kernel: LustreError: 4493:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0008: fail to locate lqe for id:507750, type:0
Apr 28 19:39:39 meerkat-oss-5-3 kernel: LustreError: 4493:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 2 previous similar messages
Apr 28 19:39:39 meerkat-oss-11-2 kernel: LustreError: 23116:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0024: fail to locate lqe for id:507750, type:0
Apr 28 19:39:39 meerkat-oss-11-2 kernel: LustreError: 23116:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 3 previous similar messages
Apr 28 19:40:39 meerkat-oss-6-2 kernel: LustreError: 8171:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002a qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:40:39 meerkat-oss-6-2 kernel: LustreError: 8171:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 19:40:48 meerkat-oss-5-3 kernel: LustreError: 8027:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0010 qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:40:48 meerkat-oss-5-3 kernel: LustreError: 8027:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 12 previous similar messages
Apr 28 19:40:48 meerkat-oss-11-1 kernel: LustreError: 5374:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST001b qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:40:48 meerkat-oss-11-1 kernel: LustreError: 5374:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 12 previous similar messages
Apr 28 19:41:19 meerkat-oss-11-2 kernel: Lustre: 21398:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 19:41:19 meerkat-oss-11-2 kernel: Lustre: 21398:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23357 previous similar messages
Apr 28 19:41:20 meerkat-oss-11-2 kernel: LustreError: 21404:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:41:20 meerkat-oss-11-2 kernel: LustreError: 21404:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23377 previous similar messages
Apr 28 19:42:25 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 19:42:25 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 174 previous similar messages
Apr 28 19:42:32 meerkat-oss-11-3 kernel: LustreError: 4863:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0015 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:42:32 meerkat-oss-11-3 kernel: LustreError: 4863:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 6 previous similar messages
Apr 28 19:42:55 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 19:42:55 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 18 previous similar messages
Apr 28 19:47:57 meerkat-oss-5-3 kernel: LustreError: 3748:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0000 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:47:57 meerkat-oss-5-3 kernel: LustreError: 3748:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 19:48:57 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.54@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff8803e656f2c0/0x492ec3326a435117 lrc: 3/0,0 mode: PW/PW res: [0xf744e0:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->12287) flags: 0x60000000010020 nid: 192.168.230.54@tcp remote: 0x3d16df50427a20b1 expref: 17 pid: 5519 timeout: 4398613566 lvb_type: 0
Apr 28 19:48:57 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) Skipped 1 previous similar message
Apr 28 19:49:06 meerkat-oss-11-3 kernel: LustreError: 4908:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000d qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:49:06 meerkat-oss-11-3 kernel: LustreError: 4908:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 19:49:27 meerkat-oss-11-5 kernel: LustreError: 8181:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0017 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:49:27 meerkat-oss-11-5 kernel: LustreError: 8181:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 15 previous similar messages
Apr 28 19:50:27 meerkat-oss-11-1 kernel: LustreError: 3888:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:50:27 meerkat-oss-11-1 kernel: LustreError: 3888:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 19:51:19 meerkat-oss-11-2 kernel: Lustre: 12962:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 19:51:19 meerkat-oss-11-2 kernel: Lustre: 12962:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23533 previous similar messages
Apr 28 19:51:20 meerkat-oss-11-2 kernel: LustreError: 12967:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:51:20 meerkat-oss-11-2 kernel: LustreError: 12967:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23532 previous similar messages
Apr 28 19:52:29 meerkat-oss-11-4 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 19:52:29 meerkat-oss-11-4 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 156 previous similar messages
Apr 28 19:52:50 meerkat-oss-6-2 kernel: LustreError: 4888:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0002 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:52:50 meerkat-oss-6-2 kernel: LustreError: 4888:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 19:53:25 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 19:53:25 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 17 previous similar messages
Apr 28 19:53:56 meerkat-oss-6-2 kernel: LustreError: 4888:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:453407 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:54:20 meerkat-oss-6-1 kernel: LustreError: 5450:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0009 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:54:20 meerkat-oss-6-1 kernel: LustreError: 5450:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 19:55:32 meerkat-oss-6-2 kernel: LustreError: 5197:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:55:32 meerkat-oss-6-2 kernel: LustreError: 5197:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 19:58:14 meerkat-oss-6-1 kernel: LustreError: 5450:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0009 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:59:03 meerkat-oss-11-1 kernel: LustreError: 3884:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0013 qtype:usr id:504888 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 19:59:03 meerkat-oss-11-1 kernel: LustreError: 3884:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 4 previous similar messages
Apr 28 19:59:22 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 192.168.230.51@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff8805f5686dc0/0x492ec3326a43522f lrc: 3/0,0 mode: PW/PW res: [0xf744e0:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000080010020 nid: 192.168.230.51@tcp remote: 0x9bbc69787fb59127 expref: 17 pid: 23116 timeout: 4399238398 lvb_type: 0
Apr 28 20:00:01 oasis-meerkat CROND[6008]: (root) CMD (/opt/rocks/bin/rocks run host lustre-oss lustre-mds command="uptime" collate=yes | grep -v X11 | /bin/sort -n -r --key=11 | /bin/mail -s "Oasis Meerkat servers uptime" cai@postal.sdsc.edu)
Apr 28 20:00:01 oasis-meerkat rockscommand[6009]: user root called "run host lustre-oss lustre-mds command=uptime collate=yes"
Apr 28 20:00:50 meerkat-oss-6-1 kernel: LustreError: 5450:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0009 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:01:20 meerkat-oss-11-2 kernel: Lustre: 4160:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 20:01:20 meerkat-oss-11-2 kernel: Lustre: 4160:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23451 previous similar messages
Apr 28 20:01:20 meerkat-oss-11-2 kernel: LustreError: 4168:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:01:20 meerkat-oss-11-2 kernel: LustreError: 4168:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23465 previous similar messages
Apr 28 20:02:31 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 20:02:31 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 154 previous similar messages
Apr 28 20:02:46 meerkat-oss-6-2 kernel: LustreError: 5204:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:02:46 meerkat-oss-6-2 kernel: LustreError: 5204:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 20:03:55 meerkat-oss-11-4 kernel: LustreError: 4604:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 20:03:55 meerkat-oss-11-4 kernel: LustreError: 4604:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 17 previous similar messages
Apr 28 20:04:00 meerkat-oss-5-3 kernel: LustreError: 8033:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:04:00 meerkat-oss-5-3 kernel: LustreError: 8033:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 20:04:28 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 192.168.230.52@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff880115bec080/0x492ec3326a44f2e7 lrc: 3/0,0 mode: PW/PW res: [0xf744e0:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000080010020 nid: 192.168.230.52@tcp remote: 0xc36633804d71d323 expref: 16 pid: 8439 timeout: 4399544274 lvb_type: 0
Apr 28 20:05:01 meerkat-oss-11-5 kernel: LustreError: 3763:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0017 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:05:01 meerkat-oss-11-5 kernel: LustreError: 3763:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 5 previous similar messages
Apr 28 20:05:41 meerkat-oss-6-2 kernel: LustreError: 4713:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:05:41 meerkat-oss-6-2 kernel: LustreError: 4713:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 20:06:31 meerkat-oss-6-1 kernel: LustreError: 5450:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0031 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:06:31 meerkat-oss-6-1 kernel: LustreError: 5450:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 20:07:56 meerkat-oss-6-2 kernel: LustreError: 4974:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000a qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:07:56 meerkat-oss-6-2 kernel: LustreError: 4974:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 20:08:54 meerkat-oss-11-1 kernel: LustreError: 7713:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:08:54 meerkat-oss-11-1 kernel: LustreError: 7713:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 6 previous similar messages
Apr 28 20:08:54 meerkat-oss-6-2 kernel: LustreError: 4868:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0032: fail to locate lqe for id:507750, type:0
Apr 28 20:08:54 meerkat-oss-6-2 kernel: LustreError: 4868:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 2 previous similar messages
Apr 28 20:08:54 meerkat-oss-11-2 kernel: LustreError: 15123:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST000c: fail to locate lqe for id:507750, type:0
Apr 28 20:08:54 meerkat-oss-11-1 kernel: LustreError: 7713:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST002b: fail to locate lqe for id:507750, type:0
Apr 28 20:08:54 meerkat-oss-11-1 kernel: LustreError: 7713:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 4 previous similar messages
Apr 28 20:08:54 meerkat-oss-6-1 kernel: LustreError: 4767:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0019 qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:08:59 meerkat-oss-11-3 kernel: LustreError: 4645:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0015 qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:08:59 meerkat-oss-11-3 kernel: LustreError: 4645:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 5 previous similar messages
Apr 28 20:08:59 meerkat-oss-11-5 kernel: LustreError: 7515:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0017: fail to locate lqe for id:507750, type:0
Apr 28 20:08:59 meerkat-oss-11-5 kernel: LustreError: 7515:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 1 previous similar message
Apr 28 20:08:59 meerkat-oss-11-2 kernel: LustreError: 15114:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0014: fail to locate lqe for id:507750, type:0
Apr 28 20:08:59 meerkat-oss-5-3 kernel: LustreError: 4491:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0030: fail to locate lqe for id:507750, type:0
Apr 28 20:08:59 meerkat-oss-11-4 kernel: LustreError: 4747:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0026: fail to locate lqe for id:507750, type:0
Apr 28 20:08:59 meerkat-oss-11-4 kernel: LustreError: 4747:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 2 previous similar messages
Apr 28 20:08:59 meerkat-oss-11-1 kernel: LustreError: 7725:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST001b: fail to locate lqe for id:507750, type:0
Apr 28 20:08:59 meerkat-oss-6-1 kernel: LustreError: 4591:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0031: fail to locate lqe for id:507750, type:0
Apr 28 20:08:59 meerkat-oss-6-1 kernel: LustreError: 4591:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 1 previous similar message
Apr 28 20:08:59 meerkat-oss-11-3 kernel: LustreError: 5056:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0025: fail to locate lqe for id:507750, type:0
Apr 28 20:08:59 meerkat-oss-11-3 kernel: LustreError: 5056:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 1 previous similar message
Apr 28 20:08:59 meerkat-oss-6-2 kernel: LustreError: 3731:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0032: fail to locate lqe for id:507750, type:0
Apr 28 20:09:04 meerkat-oss-6-2 kernel: LustreError: 4877:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0002 qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:09:04 meerkat-oss-6-2 kernel: LustreError: 4877:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 11 previous similar messages
Apr 28 20:09:04 meerkat-oss-11-4 kernel: LustreError: 5015:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0026: fail to locate lqe for id:507750, type:0
Apr 28 20:09:04 meerkat-oss-11-4 kernel: LustreError: 5015:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 7 previous similar messages
Apr 28 20:09:04 meerkat-oss-11-3 kernel: LustreError: 5046:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0025: fail to locate lqe for id:507750, type:0
Apr 28 20:09:04 meerkat-oss-11-3 kernel: LustreError: 5046:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 1 previous similar message
Apr 28 20:09:04 meerkat-oss-6-2 kernel: LustreError: 4877:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0002: fail to locate lqe for id:507750, type:0
Apr 28 20:09:04 meerkat-oss-6-2 kernel: LustreError: 4877:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 3 previous similar messages
Apr 28 20:09:04 meerkat-oss-11-2 kernel: LustreError: 8401:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST000c: fail to locate lqe for id:507750, type:0
Apr 28 20:09:04 meerkat-oss-11-2 kernel: LustreError: 8401:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 8 previous similar messages
Apr 28 20:09:04 meerkat-oss-5-3 kernel: LustreError: 7469:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0010: fail to locate lqe for id:507750, type:0
Apr 28 20:09:04 meerkat-oss-5-3 kernel: LustreError: 7469:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 3 previous similar messages
Apr 28 20:09:04 meerkat-oss-6-1 kernel: LustreError: 7624:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0009: fail to locate lqe for id:507750, type:0
Apr 28 20:09:04 meerkat-oss-6-1 kernel: LustreError: 7624:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 2 previous similar messages
Apr 28 20:09:04 meerkat-oss-11-1 kernel: LustreError: 5032:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST002b: fail to locate lqe for id:507750, type:0
Apr 28 20:09:04 meerkat-oss-11-1 kernel: LustreError: 5032:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 1 previous similar message
Apr 28 20:09:04 meerkat-oss-11-5 kernel: LustreError: 4515:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0017: fail to locate lqe for id:507750, type:0
Apr 28 20:09:04 meerkat-oss-11-5 kernel: LustreError: 4515:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 4 previous similar messages
Apr 28 20:09:08 meerkat-oss-11-3 kernel: LustreError: 7678:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0015: fail to locate lqe for id:507750, type:0
Apr 28 20:09:08 meerkat-oss-11-3 kernel: LustreError: 7678:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 6 previous similar messages
Apr 28 20:09:09 meerkat-oss-5-3 kernel: LustreError: 7473:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0030: fail to locate lqe for id:507750, type:0
Apr 28 20:09:09 meerkat-oss-5-3 kernel: LustreError: 7473:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 8 previous similar messages
Apr 28 20:09:09 meerkat-oss-6-1 kernel: LustreError: 7634:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0031: fail to locate lqe for id:507750, type:0
Apr 28 20:09:09 meerkat-oss-6-2 kernel: LustreError: 4496:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0032: fail to locate lqe for id:507750, type:0
Apr 28 20:09:09 meerkat-oss-6-2 kernel: LustreError: 4496:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 3 previous similar messages
Apr 28 20:09:09 meerkat-oss-11-1 kernel: LustreError: 7718:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0013: fail to locate lqe for id:507750, type:0
Apr 28 20:09:09 meerkat-oss-11-1 kernel: LustreError: 7718:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 1 previous similar message
Apr 28 20:09:09 meerkat-oss-11-4 kernel: LustreError: 5001:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0026: fail to locate lqe for id:507750, type:0
Apr 28 20:09:09 meerkat-oss-11-4 kernel: LustreError: 5001:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 2 previous similar messages
Apr 28 20:09:09 meerkat-oss-11-2 kernel: LustreError: 15114:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0024: fail to locate lqe for id:507750, type:0
Apr 28 20:09:09 meerkat-oss-11-2 kernel: LustreError: 15114:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 5 previous similar messages
Apr 28 20:09:09 meerkat-oss-11-5 kernel: LustreError: 4688:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST000f: fail to locate lqe for id:507750, type:0
Apr 28 20:09:09 meerkat-oss-11-5 kernel: LustreError: 4688:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 1 previous similar message
Apr 28 20:09:14 meerkat-oss-11-1 kernel: LustreError: 7723:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0013: fail to locate lqe for id:507750, type:0
Apr 28 20:09:14 meerkat-oss-11-1 kernel: LustreError: 7723:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 5 previous similar messages
Apr 28 20:09:14 meerkat-oss-11-4 kernel: LustreError: 4586:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0026: fail to locate lqe for id:507750, type:0
Apr 28 20:09:14 meerkat-oss-11-4 kernel: LustreError: 4586:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 3 previous similar messages
Apr 28 20:09:14 meerkat-oss-6-2 kernel: LustreError: 4494:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST000a: fail to locate lqe for id:507750, type:0
Apr 28 20:09:14 meerkat-oss-6-2 kernel: LustreError: 4494:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 3 previous similar messages
Apr 28 20:09:14 meerkat-oss-11-3 kernel: LustreError: 5058:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST000d: fail to locate lqe for id:507750, type:0
Apr 28 20:09:14 meerkat-oss-11-3 kernel: LustreError: 5058:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 1 previous similar message
Apr 28 20:09:14 meerkat-oss-11-2 kernel: LustreError: 14332:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0014: fail to locate lqe for id:507750, type:0
Apr 28 20:09:14 meerkat-oss-6-1 kernel: LustreError: 4767:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0009: fail to locate lqe for id:507750, type:0
Apr 28 20:09:14 meerkat-oss-6-1 kernel: LustreError: 4767:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 6 previous similar messages
Apr 28 20:09:14 meerkat-oss-11-5 kernel: LustreError: 3752:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0017: fail to locate lqe for id:507750, type:0
Apr 28 20:09:14 meerkat-oss-5-3 kernel: LustreError: 4505:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0030: fail to locate lqe for id:507750, type:0
Apr 28 20:09:14 meerkat-oss-5-3 kernel: LustreError: 4505:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 6 previous similar messages
Apr 28 20:09:19 meerkat-oss-11-4 kernel: LustreError: 5001:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST002e: fail to locate lqe for id:507750, type:0
Apr 28 20:09:19 meerkat-oss-11-4 kernel: LustreError: 5001:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 4 previous similar messages
Apr 28 20:09:19 meerkat-oss-11-3 kernel: LustreError: 5042:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0025: fail to locate lqe for id:507750, type:0
Apr 28 20:09:19 meerkat-oss-11-3 kernel: LustreError: 5042:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 1 previous similar message
Apr 28 20:09:19 meerkat-oss-5-3 kernel: LustreError: 3902:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0030: fail to locate lqe for id:507750, type:0
Apr 28 20:09:19 meerkat-oss-5-3 kernel: LustreError: 3902:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 4 previous similar messages
Apr 28 20:09:19 meerkat-oss-6-1 kernel: LustreError: 4987:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0009: fail to locate lqe for id:507750, type:0
Apr 28 20:09:19 meerkat-oss-11-5 kernel: LustreError: 4688:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST000f: fail to locate lqe for id:507750, type:0
Apr 28 20:09:19 meerkat-oss-11-5 kernel: LustreError: 4688:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 1 previous similar message
Apr 28 20:09:24 meerkat-oss-11-2 kernel: LustreError: 6027:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST000c: fail to locate lqe for id:507750, type:0
Apr 28 20:09:24 meerkat-oss-11-2 kernel: LustreError: 6027:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 10 previous similar messages
Apr 28 20:09:24 meerkat-oss-11-1 kernel: LustreError: 7713:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST002b: fail to locate lqe for id:507750, type:0
Apr 28 20:09:24 meerkat-oss-11-1 kernel: LustreError: 7713:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 8 previous similar messages
Apr 28 20:09:24 meerkat-oss-6-2 kernel: LustreError: 7577:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0002: fail to locate lqe for id:507750, type:0
Apr 28 20:09:24 meerkat-oss-6-2 kernel: LustreError: 7577:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 9 previous similar messages
Apr 28 20:09:29 meerkat-oss-6-1 kernel: LustreError: 4979:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0031: fail to locate lqe for id:507750, type:0
Apr 28 20:09:29 meerkat-oss-6-1 kernel: LustreError: 4979:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 6 previous similar messages
Apr 28 20:09:29 meerkat-oss-11-3 kernel: LustreError: 7679:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST002d: fail to locate lqe for id:507750, type:0
Apr 28 20:09:29 meerkat-oss-11-3 kernel: LustreError: 7679:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 9 previous similar messages
Apr 28 20:09:29 meerkat-oss-11-4 kernel: LustreError: 5015:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0026: fail to locate lqe for id:507750, type:0
Apr 28 20:09:29 meerkat-oss-11-4 kernel: LustreError: 5015:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 14 previous similar messages
Apr 28 20:09:29 meerkat-oss-5-3 kernel: LustreError: 4492:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0030: fail to locate lqe for id:507750, type:0
Apr 28 20:09:29 meerkat-oss-5-3 kernel: LustreError: 4492:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 7 previous similar messages
Apr 28 20:09:29 meerkat-oss-11-5 kernel: LustreError: 7515:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST000f: fail to locate lqe for id:507750, type:0
Apr 28 20:09:29 meerkat-oss-11-5 kernel: LustreError: 7515:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 8 previous similar messages
Apr 28 20:09:30 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff88017c2f1ac0/0x492ec3326a45fc54 lrc: 3/0,0 mode: PW/PW res: [0xf74486:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000080010020 nid: 192.168.230.53@tcp remote: 0xab5cfbed71360331 expref: 20 pid: 8379 timeout: 4399846664 lvb_type: 0
Apr 28 20:09:44 meerkat-oss-6-2 kernel: LustreError: 4865:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0002: fail to locate lqe for id:507750, type:0
Apr 28 20:09:44 meerkat-oss-6-2 kernel: LustreError: 4865:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 14 previous similar messages
Apr 28 20:09:44 meerkat-oss-11-1 kernel: LustreError: 5021:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST001b: fail to locate lqe for id:507750, type:0
Apr 28 20:09:44 meerkat-oss-11-1 kernel: LustreError: 5021:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 14 previous similar messages
Apr 28 20:09:44 meerkat-oss-11-2 kernel: LustreError: 6031:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0014: fail to locate lqe for id:507750, type:0
Apr 28 20:09:44 meerkat-oss-11-2 kernel: LustreError: 6031:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 19 previous similar messages
Apr 28 20:09:49 meerkat-oss-11-4 kernel: LustreError: 3997:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0036: fail to locate lqe for id:507750, type:0
Apr 28 20:09:49 meerkat-oss-5-3 kernel: LustreError: 3902:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0030: fail to locate lqe for id:507750, type:0
Apr 28 20:09:49 meerkat-oss-5-3 kernel: LustreError: 3902:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 18 previous similar messages
Apr 28 20:09:49 meerkat-oss-11-4 kernel: LustreError: 3997:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 18 previous similar messages
Apr 28 20:09:49 meerkat-oss-11-3 kernel: LustreError: 5055:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0025: fail to locate lqe for id:507750, type:0
Apr 28 20:09:49 meerkat-oss-11-3 kernel: LustreError: 5055:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 24 previous similar messages
Apr 28 20:09:49 meerkat-oss-6-1 kernel: LustreError: 4981:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0009: fail to locate lqe for id:507750, type:0
Apr 28 20:09:49 meerkat-oss-6-1 kernel: LustreError: 4981:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 8 previous similar messages
Apr 28 20:09:49 meerkat-oss-11-5 kernel: LustreError: 4847:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST000f: fail to locate lqe for id:507750, type:0
Apr 28 20:09:49 meerkat-oss-11-5 kernel: LustreError: 4847:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 12 previous similar messages
Apr 28 20:11:20 meerkat-oss-11-2 kernel: Lustre: 27696:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 20:11:20 meerkat-oss-11-2 kernel: Lustre: 27696:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23100 previous similar messages
Apr 28 20:11:20 meerkat-oss-11-2 kernel: LustreError: 27702:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:11:20 meerkat-oss-11-2 kernel: LustreError: 27702:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23236 previous similar messages
Apr 28 20:12:35 meerkat-oss-11-4 kernel: LustreError: 4802:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 20:12:35 meerkat-oss-11-4 kernel: LustreError: 4802:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 281 previous similar messages
Apr 28 20:13:44 meerkat-oss-6-2 kernel: LustreError: 5207:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:13:44 meerkat-oss-6-2 kernel: LustreError: 5207:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 93 previous similar messages
Apr 28 20:13:55 meerkat-oss-11-4 kernel: LustreError: 4802:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 20:13:55 meerkat-oss-11-4 kernel: LustreError: 4802:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 16 previous similar messages
Apr 28 20:15:01 oasis-meerkat CROND[6127]: (root) CMD (/opt/rocks/bin/rocks run host lustre-mds command="/usr/bin/vmstat -s | grep buffer" collate=yes | grep -v X11 | /bin/mail -s "meerkat MDS memory usage" cai@postal.sdsc.edu)
Apr 28 20:15:01 oasis-meerkat rockscommand[6128]: user root called "run host lustre-mds command=/usr/bin/vmstat -s | grep buffer collate=yes"
Apr 28 20:15:56 meerkat-oss-5-3 kernel: LustreError: 5302:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0010 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:15:56 meerkat-oss-5-3 kernel: LustreError: 5302:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 141 previous similar messages
Apr 28 20:18:17 meerkat-oss-6-1 kernel: LustreError: 4816:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0009 qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:18:17 meerkat-oss-6-1 kernel: LustreError: 4816:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 83 previous similar messages
Apr 28 20:18:17 meerkat-oss-6-2 kernel: LustreError: 5207:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002a qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:18:45 meerkat-oss-11-5 kernel: LustreError: 8181:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0017 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:18:45 meerkat-oss-11-5 kernel: LustreError: 8181:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 76 previous similar messages
Apr 28 20:20:06 meerkat-oss-11-3 kernel: LustreError: 4914:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000d qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:20:06 meerkat-oss-11-3 kernel: LustreError: 4914:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 114 previous similar messages
Apr 28 20:20:13 meerkat-oss-11-1 kernel: LustreError: 5225:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0013 qtype:usr id:507750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:20:13 meerkat-oss-11-1 kernel: LustreError: 5225:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 85 previous similar messages
Apr 28 20:21:20 meerkat-oss-11-2 kernel: Lustre: 18684:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 20:21:20 meerkat-oss-11-2 kernel: Lustre: 18684:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 22942 previous similar messages
Apr 28 20:21:20 meerkat-oss-11-2 kernel: LustreError: 18695:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:21:20 meerkat-oss-11-2 kernel: LustreError: 18695:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 22952 previous similar messages
Apr 28 20:22:39 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 20:22:39 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1763 previous similar messages
Apr 28 20:24:25 meerkat-oss-11-4 kernel: LustreError: 4802:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 20:24:25 meerkat-oss-11-4 kernel: LustreError: 4802:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 17 previous similar messages
Apr 28 20:24:45 meerkat-oss-11-2 kernel: LustreError: 23435:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0024: fail to locate lqe for id:505418, type:0
Apr 28 20:24:45 meerkat-oss-11-2 kernel: LustreError: 23435:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 18 previous similar messages
Apr 28 20:24:45 meerkat-oss-6-2 kernel: LustreError: 4868:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST0032: fail to locate lqe for id:505418, type:0
Apr 28 20:24:45 meerkat-oss-6-2 kernel: LustreError: 4868:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 12 previous similar messages
Apr 28 20:24:45 meerkat-oss-11-3 kernel: LustreError: 5046:0:(qsd_handler.c:1155:qsd_op_adjust()) meerkat-OST002d: fail to locate lqe for id:505418, type:0
Apr 28 20:24:45 meerkat-oss-11-3 kernel: LustreError: 5046:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 8 previous similar messages
Apr 28 20:31:05 meerkat-oss-5-3 kernel: LustreError: 4800:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0000 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:31:05 meerkat-oss-5-3 kernel: LustreError: 4800:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 17 previous similar messages
Apr 28 20:31:07 meerkat-oss-11-1 kernel: LustreError: 3887:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0013 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:31:07 meerkat-oss-11-1 kernel: LustreError: 3887:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 9 previous similar messages
Apr 28 20:31:20 meerkat-oss-11-2 kernel: Lustre: 9784:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 20:31:20 meerkat-oss-11-2 kernel: Lustre: 9784:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23164 previous similar messages
Apr 28 20:31:20 meerkat-oss-11-2 kernel: LustreError: 9794:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:31:20 meerkat-oss-11-2 kernel: LustreError: 9794:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23175 previous similar messages
Apr 28 20:31:53 meerkat-oss-11-3 kernel: LustreError: 4912:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002d qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:31:53 meerkat-oss-11-3 kernel: LustreError: 4912:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 14 previous similar messages
Apr 28 20:32:42 meerkat-oss-11-4 kernel: LustreError: 4604:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 20:32:42 meerkat-oss-11-4 kernel: LustreError: 4604:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 158 previous similar messages
Apr 28 20:34:55 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 20:34:55 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 17 previous similar messages
Apr 28 20:36:17 meerkat-oss-11-5 kernel: LustreError: 3773:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0017 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:36:17 meerkat-oss-11-5 kernel: LustreError: 3773:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 4 previous similar messages
Apr 28 20:39:57 meerkat-oss-6-2 kernel: LustreError: 5204:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:39:57 meerkat-oss-6-2 kernel: LustreError: 5204:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 18 previous similar messages
Apr 28 20:41:20 meerkat-oss-11-2 kernel: Lustre: 769:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 20:41:20 meerkat-oss-11-2 kernel: Lustre: 769:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23277 previous similar messages
Apr 28 20:41:20 meerkat-oss-11-2 kernel: LustreError: 778:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:41:20 meerkat-oss-11-2 kernel: LustreError: 778:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23280 previous similar messages
Apr 28 20:42:47 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 20:42:47 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 153 previous similar messages
Apr 28 20:43:07 meerkat-oss-11-3 kernel: LustreError: 4637:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002d qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:43:07 meerkat-oss-11-3 kernel: LustreError: 4637:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 6 previous similar messages
Apr 28 20:43:12 meerkat-oss-11-1 kernel: LustreError: 3882:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:43:12 meerkat-oss-11-1 kernel: LustreError: 3882:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 20:43:52 meerkat-oss-6-1 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0009 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:43:52 meerkat-oss-6-1 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 20:44:45 meerkat-oss-5-3 kernel: LustreError: 5308:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0008 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:44:45 meerkat-oss-5-3 kernel: LustreError: 5308:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 20:45:25 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 20:45:25 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 17 previous similar messages
Apr 28 20:46:53 meerkat-oss-11-5 kernel: LustreError: 3771:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0017 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:46:53 meerkat-oss-11-5 kernel: LustreError: 3771:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 4 previous similar messages
Apr 28 20:49:10 meerkat-oss-6-1 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0009 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:49:10 meerkat-oss-6-1 kernel: LustreError: 4817:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 20:50:57 meerkat-oss-6-2 kernel: Lustre: meerkat-OST001a: Client 6c95e621-a5b4-26a4-5697-a926a6224271 (at 198.202.118.93@tcp) reconnecting
Apr 28 20:50:57 meerkat-oss-6-2 kernel: Lustre: meerkat-OST0012: Client 6c95e621-a5b4-26a4-5697-a926a6224271 (at 198.202.118.93@tcp) reconnecting
Apr 28 20:50:57 meerkat-oss-6-2 kernel: Lustre: Skipped 7 previous similar messages
Apr 28 20:50:57 meerkat-oss-6-2 kernel: Lustre: Skipped 7 previous similar messages
Apr 28 20:51:20 meerkat-oss-11-2 kernel: Lustre: 24655:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 20:51:20 meerkat-oss-11-2 kernel: Lustre: 24655:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23379 previous similar messages
Apr 28 20:51:20 meerkat-oss-11-2 kernel: LustreError: 24665:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:51:20 meerkat-oss-11-2 kernel: LustreError: 24665:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23390 previous similar messages
Apr 28 20:51:22 meerkat-oss-6-2 kernel: Lustre: meerkat-OST0012: Client 6c95e621-a5b4-26a4-5697-a926a6224271 (at 198.202.118.93@tcp) reconnecting
Apr 28 20:51:22 meerkat-oss-6-2 kernel: Lustre: Skipped 4 previous similar messages
Apr 28 20:52:50 meerkat-oss-11-4 kernel: LustreError: 4835:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 20:52:50 meerkat-oss-11-4 kernel: LustreError: 4835:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 155 previous similar messages
Apr 28 20:53:44 meerkat-oss-11-3 kernel: LustreError: 5411:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000d qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:53:44 meerkat-oss-11-3 kernel: LustreError: 5411:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 20:54:11 meerkat-oss-6-2 kernel: LustreError: 4972:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0002 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:54:11 meerkat-oss-6-2 kernel: LustreError: 4972:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 20:55:55 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 20:55:55 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 17 previous similar messages
Apr 28 20:56:15 meerkat-oss-6-2 kernel: LustreError: 4893:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:56:38 meerkat-oss-5-3 kernel: LustreError: 4800:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0000 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:56:38 meerkat-oss-5-3 kernel: LustreError: 4800:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 20:56:49 meerkat-oss-11-1 kernel: LustreError: 5006:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0013 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:56:49 meerkat-oss-11-1 kernel: LustreError: 5006:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 20:57:04 meerkat-oss-6-2 kernel: LustreError: 4893:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0002 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:57:04 meerkat-oss-6-2 kernel: LustreError: 4893:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 20:58:48 meerkat-oss-6-1 kernel: LustreError: 5437:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0011 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 20:58:48 meerkat-oss-6-1 kernel: LustreError: 5437:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 21:00:01 oasis-meerkat CROND[6446]: (root) CMD (/opt/rocks/bin/rocks run host lustre-oss lustre-mds command="uptime" collate=yes | grep -v X11 | /bin/sort -n -r --key=11 | /bin/mail -s "Oasis Meerkat servers uptime" cai@postal.sdsc.edu)
Apr 28 21:00:02 oasis-meerkat rockscommand[6448]: user root called "run host lustre-oss lustre-mds command=uptime collate=yes"
Apr 28 21:01:20 meerkat-oss-11-2 kernel: Lustre: 16305:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 21:01:20 meerkat-oss-11-2 kernel: Lustre: 16305:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23541 previous similar messages
Apr 28 21:01:20 meerkat-oss-11-2 kernel: LustreError: 16316:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:01:20 meerkat-oss-11-2 kernel: LustreError: 16316:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23546 previous similar messages
Apr 28 21:02:23 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.54@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff88051c7f2b40/0x492ec3326a517425 lrc: 3/0,0 mode: PW/PW res: [0xf74382:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000080010020 nid: 192.168.230.54@tcp remote: 0x3d16df50459a10d8 expref: 22 pid: 8403 timeout: 4403019671 lvb_type: 0
Apr 28 21:02:54 meerkat-oss-11-4 kernel: LustreError: 4835:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 21:02:54 meerkat-oss-11-4 kernel: LustreError: 4835:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 154 previous similar messages
Apr 28 21:03:48 meerkat-oss-11-5 kernel: LustreError: 5236:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0017 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:03:48 meerkat-oss-11-5 kernel: LustreError: 5236:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 21:04:07 meerkat-oss-11-3 kernel: LustreError: 5042:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002d qtype:usr id:509731 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:04:07 meerkat-oss-11-3 kernel: LustreError: 5042:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 21:05:46 meerkat-oss-6-1 kernel: LustreError: 4601:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0019 qtype:usr id:510900 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:05:46 meerkat-oss-6-1 kernel: LustreError: 4601:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 21:05:55 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 21:05:55 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 16 previous similar messages
Apr 28 21:08:55 meerkat-oss-5-3 kernel: LustreError: 5309:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:08:55 meerkat-oss-5-3 kernel: LustreError: 5309:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 21:09:10 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 192.168.230.52@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff8802104b4280/0x492ec3326a52b506 lrc: 3/0,0 mode: PW/PW res: [0xf74382:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000080010020 nid: 192.168.230.52@tcp remote: 0xc3663380539f07cf expref: 23 pid: 8329 timeout: 4403426104 lvb_type: 0
Apr 28 21:09:26 meerkat-oss-11-1 kernel: LustreError: 5367:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0013 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:09:26 meerkat-oss-11-1 kernel: LustreError: 5367:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 21:11:20 meerkat-oss-11-2 kernel: Lustre: 7420:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 21:11:20 meerkat-oss-11-2 kernel: Lustre: 7420:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23241 previous similar messages
Apr 28 21:11:20 meerkat-oss-11-2 kernel: LustreError: 7428:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:11:20 meerkat-oss-11-2 kernel: LustreError: 7428:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23249 previous similar messages
Apr 28 21:12:50 meerkat-oss-6-1 kernel: LustreError: 5168:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0019 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:12:55 meerkat-oss-11-4 kernel: LustreError: 3838:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:460 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 21:12:55 meerkat-oss-11-4 kernel: LustreError: 3838:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 153 previous similar messages
Apr 28 21:12:56 meerkat-oss-6-2 kernel: LustreError: 4975:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0002 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:14:18 meerkat-oss-11-3 kernel: LustreError: 5411:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0015 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:14:18 meerkat-oss-11-3 kernel: LustreError: 5411:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 6 previous similar messages
Apr 28 21:15:01 oasis-meerkat CROND[6565]: (root) CMD (/opt/rocks/bin/rocks run host lustre-mds command="/usr/bin/vmstat -s | grep buffer" collate=yes | grep -v X11 | /bin/mail -s "meerkat MDS memory usage" cai@postal.sdsc.edu)
Apr 28 21:15:02 oasis-meerkat rockscommand[6566]: user root called "run host lustre-mds command=/usr/bin/vmstat -s | grep buffer collate=yes"
Apr 28 21:15:12 meerkat-oss-6-2 kernel: LustreError: 4713:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:15:12 meerkat-oss-6-2 kernel: LustreError: 4713:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 21:15:47 meerkat-oss-11-5 kernel: LustreError: 4613:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0017 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:15:47 meerkat-oss-11-5 kernel: LustreError: 4613:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 21:16:25 meerkat-oss-11-4 kernel: LustreError: 4604:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 21:16:25 meerkat-oss-11-4 kernel: LustreError: 4604:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 17 previous similar messages
Apr 28 21:19:29 meerkat-oss-5-3 kernel: LustreError: 5309:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0030 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:19:29 meerkat-oss-5-3 kernel: LustreError: 5309:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 4 previous similar messages
Apr 28 21:19:56 meerkat-oss-11-1 kernel: LustreError: 5221:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002b qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:19:56 meerkat-oss-11-1 kernel: LustreError: 5221:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 21:20:01 meerkat-oss-6-2 kernel: LustreError: 3746:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0002 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:20:01 meerkat-oss-6-2 kernel: LustreError: 3746:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 21:21:20 meerkat-oss-11-2 kernel: Lustre: 30855:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 21:21:20 meerkat-oss-11-2 kernel: Lustre: 30855:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23209 previous similar messages
Apr 28 21:21:20 meerkat-oss-11-2 kernel: LustreError: 30863:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:21:20 meerkat-oss-11-2 kernel: LustreError: 30863:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23214 previous similar messages
Apr 28 21:22:56 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 21:22:56 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 154 previous similar messages
Apr 28 21:26:55 meerkat-oss-11-4 kernel: LustreError: 3838:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 21:26:55 meerkat-oss-11-4 kernel: LustreError: 3838:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 17 previous similar messages
Apr 28 21:29:16 meerkat-oss-6-1 kernel: LustreError: 5437:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0011 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:29:16 meerkat-oss-6-1 kernel: LustreError: 5437:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 21:30:08 meerkat-oss-11-3 kernel: LustreError: 3883:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002d qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:30:08 meerkat-oss-11-3 kernel: LustreError: 3883:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 4 previous similar messages
Apr 28 21:30:30 meerkat-oss-5-3 kernel: LustreError: 5299:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0000 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:30:30 meerkat-oss-5-3 kernel: LustreError: 5299:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 21:31:20 meerkat-oss-11-2 kernel: Lustre: 22380:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 21:31:20 meerkat-oss-11-2 kernel: Lustre: 22380:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23484 previous similar messages
Apr 28 21:31:20 meerkat-oss-11-2 kernel: LustreError: 22387:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:31:20 meerkat-oss-11-2 kernel: LustreError: 22387:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23488 previous similar messages
Apr 28 21:32:59 meerkat-oss-11-4 kernel: LustreError: 3838:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 21:32:59 meerkat-oss-11-4 kernel: LustreError: 3838:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 157 previous similar messages
Apr 28 21:33:32 meerkat-oss-11-2 kernel: LustreError: 57:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.230.53@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff8802a8dfbd40/0x492ec3326a52e002 lrc: 3/0,0 mode: PW/PW res: [0xf744c2:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000080010020 nid: 192.168.230.53@tcp remote: 0xab5cfbed71388d51 expref: 20 pid: 8338 timeout: 4404888907 lvb_type: 0
Apr 28 21:33:55 meerkat-oss-11-1 kernel: LustreError: 5384:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST001b qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:33:55 meerkat-oss-11-1 kernel: LustreError: 5384:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 21:34:22 meerkat-oss-11-2 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 150s: evicting client at 192.168.230.51@tcp  ns: filter-meerkat-OST0034_UUID lock: ffff880469f2e2c0/0x492ec3326a4f0e47 lrc: 3/0,0 mode: PW/PW res: [0xf744ed:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000080010020 nid: 192.168.230.51@tcp remote: 0x9bbc69788003be74 expref: 10203 pid: 15127 timeout: 4404938916 lvb_type: 0
Apr 28 21:36:14 meerkat-oss-11-4 kernel: Lustre: meerkat-OST0016: Client 4f316e03-1113-0db9-3f00-d1d53950f781 (at 198.202.118.201@tcp) reconnecting
Apr 28 21:36:14 meerkat-oss-11-4 kernel: Lustre: meerkat-OST001e: Client 4f316e03-1113-0db9-3f00-d1d53950f781 (at 198.202.118.201@tcp) reconnecting
Apr 28 21:36:14 meerkat-oss-11-4 kernel: Lustre: Skipped 7 previous similar messages
Apr 28 21:36:14 meerkat-oss-11-4 kernel: Lustre: Skipped 7 previous similar messages
Apr 28 21:36:46 meerkat-oss-6-2 kernel: LustreError: 4713:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0032 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:36:46 meerkat-oss-6-2 kernel: LustreError: 4713:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 21:37:25 meerkat-oss-11-4 kernel: LustreError: 3838:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 21:37:25 meerkat-oss-11-4 kernel: LustreError: 3838:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 18 previous similar messages
Apr 28 21:39:25 meerkat-oss-11-5 kernel: LustreError: 4525:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0017 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:39:25 meerkat-oss-6-1 kernel: LustreError: 5437:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0011 qtype:usr id:508963 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:39:25 meerkat-oss-6-1 kernel: LustreError: 5437:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 3 previous similar messages
Apr 28 21:41:20 meerkat-oss-11-2 kernel: Lustre: 13859:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 21:41:20 meerkat-oss-11-2 kernel: Lustre: 13859:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23447 previous similar messages
Apr 28 21:41:20 meerkat-oss-11-2 kernel: LustreError: 13866:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:41:20 meerkat-oss-11-2 kernel: LustreError: 13866:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23451 previous similar messages
Apr 28 21:43:03 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 21:43:03 meerkat-oss-11-4 kernel: LustreError: 4845:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 158 previous similar messages
Apr 28 21:43:25 meerkat-oss-6-2 kernel: LustreError: 3746:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0002 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:43:35 meerkat-oss-11-3 kernel: LustreError: 5411:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0015 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:43:35 meerkat-oss-11-3 kernel: LustreError: 5411:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 2 previous similar messages
Apr 28 21:46:14 meerkat-oss-6-2 kernel: LustreError: 5193:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST002a qtype:usr id:511076 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:46:29 meerkat-oss-6-2 kernel: LustreError: 3746:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0002 qtype:usr id:511076 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:46:29 meerkat-oss-6-2 kernel: LustreError: 3746:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 21:46:43 meerkat-oss-5-3 kernel: LustreError: 4700:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0010 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:46:43 meerkat-oss-5-3 kernel: LustreError: 4700:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 6 previous similar messages
Apr 28 21:47:46 meerkat-oss-11-1 kernel: LustreError: 3889:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST001b qtype:usr id:511076 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:47:46 meerkat-oss-11-1 kernel: LustreError: 3889:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 4 previous similar messages
Apr 28 21:47:55 meerkat-oss-11-4 kernel: LustreError: 4604:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master & slave, and quota reintegration needs be re-triggered. qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 21:47:55 meerkat-oss-11-4 kernel: LustreError: 4604:0:(qsd_handler.c:767:qsd_op_begin0()) Skipped 17 previous similar messages
Apr 28 21:48:14 meerkat-oss-11-5 kernel: LustreError: 4832:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0037 qtype:usr id:511076 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:48:14 meerkat-oss-11-5 kernel: LustreError: 4832:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 21:49:20 meerkat-oss-6-2 kernel: LustreError: 4713:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000a qtype:usr id:511076 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:49:20 meerkat-oss-6-2 kernel: LustreError: 4713:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 1 previous similar message
Apr 28 21:51:20 meerkat-oss-11-2 kernel: Lustre: 5044:0:(qsd_reint.c:486:qsd_reint_main()) meerkat-OST0034: reint global for [0x200000006:0x1020000:0x0] failed. -3
Apr 28 21:51:20 meerkat-oss-11-2 kernel: Lustre: 5044:0:(qsd_reint.c:486:qsd_reint_main()) Skipped 23497 previous similar messages
Apr 28 21:51:20 meerkat-oss-11-2 kernel: LustreError: 5052:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:grp id:46987 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:51:20 meerkat-oss-11-2 kernel: LustreError: 5052:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 23503 previous similar messages
Apr 28 21:52:17 meerkat-oss-6-2 kernel: LustreError: 4888:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0002 qtype:usr id:508990 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:52:36 meerkat-oss-11-5 kernel: LustreError: 5236:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0017 qtype:usr id:511076 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:52:36 meerkat-oss-11-5 kernel: LustreError: 5236:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 6 previous similar messages
Apr 28 21:53:06 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0036 qtype:usr id:15835 enforced:1 granted:65536 pending:0 waiting:0 req:0 usage:0 qunit:65536 qtune:16384 edquot:0
Apr 28 21:53:06 meerkat-oss-11-4 kernel: LustreError: 4813:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 158 previous similar messages
Apr 28 21:55:36 meerkat-oss-11-3 kernel: LustreError: 4824:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST000d qtype:usr id:511076 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Apr 28 21:55:36 meerkat-oss-11-3 kernel: LustreError: 4824:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 5 previous similar messages