Details
-
Bug
-
Resolution: Won't Fix
-
Major
-
None
-
None
-
None
-
[root@storage06 ~]# uname -r
3.10.0-327.3.1.el7_lustre.x86_64
[root@storage06 ~]# cat /etc/redhat-release
Scientific Linux release 7.2 (Nitrogen)
-
3
-
9223372036854775807
Description
Dear Support,
One of our customer contacted us because of the following errors:
[root@mds01 ~]# dmesg -T | grep -E "opd_pre_status -116|opd_pre_status -115"
[Thu Mar 9 14:50:45 2017] LustreError: 25696:0:(osp_precreate.c:1243:osp_precreate_ready_condition()) wurfs-OST001b-osc-MDT0000: precreate failed opd_pre_status -115
[Thu Mar 9 15:01:44 2017] LustreError: 25753:0:(osp_precreate.c:1243:osp_precreate_ready_condition()) wurfs-OST001b-osc-MDT0000: precreate failed opd_pre_status -115
[Thu Mar 9 15:11:46 2017] LustreError: 12529:0:(osp_precreate.c:1243:osp_precreate_ready_condition()) wurfs-OST001b-osc-MDT0000: precreate failed opd_pre_status -115
[Thu Mar 9 15:22:04 2017] LustreError: 25542:0:(osp_precreate.c:1243:osp_precreate_ready_condition()) wurfs-OST001b-osc-MDT0000: precreate failed opd_pre_status -115
[Thu Mar 9 15:33:14 2017] LustreError: 25248:0:(osp_precreate.c:1243:osp_precreate_ready_condition()) wurfs-OST0021-osc-MDT0000: precreate failed opd_pre_status -116
[Thu Mar 9 15:43:56 2017] LustreError: 25656:0:(osp_precreate.c:1243:osp_precreate_ready_condition()) wurfs-OST001b-osc-MDT0000: precreate failed opd_pre_status -115
[Thu Mar 9 15:54:21 2017] LustreError: 25645:0:(osp_precreate.c:1243:osp_precreate_ready_condition()) wurfs-OST001c-osc-MDT0000: precreate failed opd_pre_status -115
I have started focusing on one of the osts that are reporting the error: ost001b.
It is currently on storage06:
[9752632.984562] Lustre: wurfs-OST001c-os: trigger OI scrub by RPC for [0x1001c0000:0x1ad9d9d:0x0], rc = 0 [1]
[9752632.984569] Lustre: Skipped 127 previous similar messages
[9752877.938323] LustreError: 23938:0:(ofd_dev.c:1781:ofd_create_hdl()) wurfs-OST0021: unable to precreate: rc = -116
[9752877.938334] LustreError: 23938:0:(ofd_dev.c:1781:ofd_create_hdl()) Skipped 266 previous similar messages
[9752884.305673] mpt2sas1: log_info(0x31120302): originator(PL), code(0x12), sub_code(0x0302)
[9753043.238296] LustreError: 42247:0:(osd_compat.c:593:osd_obj_update_entry()) wurfs-OST001c-osd: the FID [0x580000400:0x2d2c:0x0] is used by two objects: 524529/2876391915 1582/813719656
[9753043.238307] LustreError: 42247:0:(osd_compat.c:593:osd_obj_update_entry()) Skipped 3455 previous similar messages
[9753235.723470] Lustre: wurfs-OST001b-os: trigger OI scrub by RPC for [0x1001b0000:0x19a5c22:0x0], rc = 0 [1]
[9753235.723478] Lustre: Skipped 130 previous similar messages
[9753429.709498] mpt2sas1: log_info(0x31120302): originator(PL), code(0x12), sub_code(0x0302)
[9753478.487827] LustreError: 42848:0:(ofd_dev.c:1781:ofd_create_hdl()) wurfs-OST001c: unable to precreate: rc = -115
[9753478.487835] LustreError: 42848:0:(ofd_dev.c:1781:ofd_create_hdl()) Skipped 263 previous similar messages
[9753648.598453] LustreError: 4652:0:(osd_compat.c:593:osd_obj_update_entry()) wurfs-OST001c-osd: the FID [0x580000400:0x2d2c:0x0] is used by two objects: 524529/2876391915 1582/813719656
[9753648.598462] LustreError: 4652:0:(osd_compat.c:593:osd_obj_update_entry()) Skipped 3509 previous similar messages
I can see these messages repeated a lot of times.
[root@storage06 ~]# mount | grep mapper
/dev/mapper/ost001b on /lustre/ost1b type lustre (ro)
/dev/mapper/ost001c on /lustre/ost1c type lustre (ro)
/dev/mapper/ost001d on /lustre/ost1d type lustre (ro)
/dev/mapper/ost0021 on /lustre/ost21 type lustre (ro)
/dev/mapper/ost0022 on /lustre/ost22 type lustre (ro)
/dev/mapper/ost0023 on /lustre/ost23 type lustre (ro)
The customer has run: lctl lfsck_start -M wurfs-OST001b but it didnt help.
I suppose that the next step is:
debugfs -c -R "ncheck 524529 1582" /dev//dev/mapper/ost001b
(I wonder if it can be done while the fs is mounted.)