Blocker Description
Our sysadmins updated one of out Lustre 2.1 filesystem to lustre 2.4.0-19chaos. Note that this filesystem was likely originally formatted under 1.8. It looks like oi_scrub ran automatically this time, but failed to make any updates:
> cat osd-ldiskfs/lsd-MDT0000/oi_scrub name: OI_scrub magic: 0x4c5fd252 oi_files: 1 status: completed flags: param: time_since_last_completed: 505891 seconds time_since_latest_start: 521998 seconds time_since_last_checkpoint: 505891 seconds latest_start_position: 12 last_checkpoint_position: 991133697 first_failure_position: N/A checked: 200636112 updated: 0 failed: 0 prior_updated: 0 noscrub: 3090 igif: 15492100 success_count: 2 run_time: 16107 seconds average_speed: 12456 objects/sec real-time_speed: N/A current_position: N/A
You'll recall that we have oi scrub problems when we tried to upgrade the first ldiskfs filesystem to 2.4 in LU-3934. This time we are using a version of lustre with the suggested patches included.
We are seeing similar symptoms as last time. For example, directory listings show ????????? for permissions flags for some of the subdirectories, and we are seeing errors on the MDS console like this:
Nov 7 08:06:19 momus-mds1 kernel: LustreError: 7326:0:(fld_handler.c:169:fld_server_lookup()) srv-lsd-MDT0000: Cannot find sequence 0x607000002000000: rc = -5 Nov 7 08:06:19 momus-mds1 kernel: LustreError: 7326:0:(fld_handler.c:169:fld_server_lookup()) Skipped 20 previous similar messages Nov 7 08:06:19 momus-mds1 kernel: LustreError: 7326:0:(osd_handler.c:2125:osd_fld_lookup()) lsd-MDT0000-osd: cannot find FLD range for [0x607000002000000:0x8a0:0x0]: rc = -5 Nov 7 08:06:19 momus-mds1 kernel: LustreError: 7326:0:(osd_handler.c:2125:osd_fld_lookup()) Skipped 14 previous similar messages Nov 7 08:06:19 momus-mds1 kernel: LustreError: 7326:0:(osd_handler.c:3317:osd_remote_fid()) lsd-MDT0000-osd: Can not lookup fld for [0x607000002000000:0x8a0:0x0] Nov 7 08:06:19 momus-mds1 kernel: LustreError: 7326:0:(osd_handler.c:3317:osd_remote_fid()) Skipped 14 previous similar messages
The filesystem is unusable many of our users.
