[LU-6308] OST failed to read disk usage

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.3
• Type: Bug
• Priority: Critical
• Reporter: Haisong Cai (Inactive)
• Assignee: Niu Yawei (Inactive)
• Resolution: Cannot Reproduce

Environment

Linux meerkat-mds-10-2.local 2.6.32-358.23.2.el6_lustre.x86_64 #1 SMP Thu Dec 19 19:57:45 PST 2013 x86_64 x86_64 x86_64 GNU/Linux

• Severity: 3

Description

Server logging errors:

Mar 1 05:46:30 meerkat-oss-11-2 kernel: LustreError: 4551:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:506231 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 06:06:06 meerkat-oss-11-2 kernel: LustreError: 5111:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:505750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 06:13:04 meerkat-oss-11-2 kernel: LustreError: 10239:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:505750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 06:13:17 meerkat-oss-11-2 kernel: LustreError: 10239:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:505750 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 06:46:31 meerkat-oss-11-2 kernel: LustreError: 4643:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:506231 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 07:58:53 meerkat-oss-11-2 kernel: LustreError: 6660:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:510329 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 08:11:40 meerkat-oss-11-2 kernel: LustreError: 6660:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:510329 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 08:11:57 meerkat-oss-11-2 kernel: LustreError: 6660:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:510329 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 08:15:23 meerkat-oss-11-2 kernel: LustreError: 8700:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:510329 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 08:16:04 meerkat-oss-11-2 kernel: LustreError: 7677:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:510329 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 08:16:14 meerkat-oss-11-2 kernel: LustreError: 5086:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:510329 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 08:16:32 meerkat-oss-11-2 kernel: LustreError: 5086:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:510329 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 09:18:20 meerkat-oss-11-2 kernel: LustreError: 8699:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:507280 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 09:18:22 meerkat-oss-11-2 kernel: LustreError: 29143:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:507280 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 09:18:26 meerkat-oss-11-2 kernel: LustreError: 7657:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:507280 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 09:21:51 meerkat-oss-11-2 kernel: LustreError: 5076:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:507280 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 09:22:00 meerkat-oss-11-2 kernel: LustreError: 5114:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:507280 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 09:23:27 meerkat-oss-11-2 kernel: LustreError: 10244:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:510297 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 09:24:24 meerkat-oss-11-2 kernel: LustreError: 5047:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:507280 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 09:26:35 meerkat-oss-11-2 kernel: LustreError: 8566:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:510297 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 09:30:17 meerkat-oss-11-2 kernel: LustreError: 8566:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:510297 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 09:34:33 meerkat-oss-11-2 kernel: LustreError: 6664:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:510297 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 09:43:06 meerkat-oss-11-2 kernel: LustreError: 7677:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:510297 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 09:53:06 meerkat-oss-11-2 kernel: LustreError: 6664:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:510297 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0
Mar 1 10:03:06 meerkat-oss-11-2 kernel: LustreError: 5036:0:(qsd_entry.c:215:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:meerkat-OST0034 qtype:usr id:510297 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0 qunit:0 qtune:0 edquot:0

Client having write fail:

LustreError: 11-0: meerkat-OST0034-osc-ffff881030667400: Communicating with 172.25.32.115@tcp, operation ost_write failed with -3.
LustreError: 11-0: meerkat-OST0034-osc-ffff881030667400: Communicating with 172.25.32.115@tcp, operation ost_write failed with -3.
LustreError: 11-0: meerkat-OST0034-osc-ffff881030667400: Communicating with 172.25.32.115@tcp, operation ost_write failed with -3.
LustreError: 11-0: meerkat-OST0034-osc-ffff881030667400: Communicating with 172.25.32.115@tcp, operation ost_write failed with -3.

Comments

[Peter Jones]

Niu

Could you please advise?

Thanks

Peter

[Niu Yawei (Inactive)]

Did you enable quota feature on the OST0034? You can check it by "cat /proc/fs/lustre/osd-ldiskfs/$OSTNAME/quota_slave/info" on OST0034, the output will look like:

target name:    lustre-OST0001
pool ID:        0
type:           dt
quota enabled:  none
conn to master: setup
space acct:     ug
user uptodate:  glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]

"space acct: ug" means the quota feature (user & group quota accounting) are enabled.

If the quota feature isn't enabled, you can enable it by 'tune2fs -O quota $OSTDEV" (the OST must be offline).

[Niu Yawei (Inactive)]

Any feedback? I think it the reason could be quota not enabled on OST-00034. (-3 ESRCH).

[Haisong Cai (Inactive)]

Hi Niu,

This is a old ticket. We've moved on to different version since.
You may go ahead and close the case.

thanks,
Haisong

[Niu Yawei (Inactive)]

Thanks, Haisong.