Details
-
Bug
-
Resolution: Fixed
-
Major
-
Lustre 2.4.1
-
3
-
12442
Description
We are running Lustre 2.4.0-21chaos (see http://github/com/chaos/lustre), and most likely of particular interest are these two patches that we are carrying:
We now find that, at least on the ldiskfs MDS nodes, OI scrub runs on every start up of the MDS. The console message looks something like this:
2014-01-28 09:27:28 sumom-mds1 login: LustreError: 0-0: lsc-MDT0000: trigger OI scrub by RPC for [0x7e4d2310f09:0x2ddf:0x0], rc = 0 [1]
Given the frequency of MDS reboots (i.e. often) required lately for other bugs, OI scrub is running far too much.
Attachments
Activity
This patch hasn't landed to master yet, but I created a b2_5 and b2_4 patch at:
b2_4 - http://review.whamcloud.com/#/c/9140/
b2_5 - http://review.whamcloud.com/#/c/9139/
James, Lai is on holiday this week. Could you please cherry-pick this patch to b2_4 and b2_5 once it has landed to master. This can now be done directly in Gerrit. Please also add the "Lustre-change:" and "Lustre-commit:" tags to the commit messages as described on https://wiki.hpdd.intel.com/display/PUB/Commit+Comments
I peeked at the /OI_scrub file while an auto-scrub was running. It showed that bit 0 was set in sf->sf_oi_bitmap. This is wrong, because the OI already exists and OI_scrub has already run to completion several times.
I think I see the problem in osd_oi_table_open(). Note the format string assumes the OI containers have names like oi.16.0, oi.16.1, and so on. However, for our upgraded filesystems we have only one OI container named oi.16. So osd_oi_open() returns ENOENT and we proceed to set the "recreated" bit in the bitmap.
I also notice osd_fid_lookup() starts the scrub using osd_scrub_start(dev), which only enables the flag SS_AUTO. So even though (ldiskfs_test_bit(osd_oi_fid2idx(dev,fid), sf->sf_oi_bitmap)) is true, (unless I misunderstand something) we would not see the "recreated" flag in oi_scrub.
Fan Yong, the -1 debug log is from a classified system, so I can't send it, but if you have specific questions about it I can look for you. While the scrub was in progress, the flags field only had 'auto'. An example FID from that system that followed the "trigger" path was [0x1a89082ad98:0x4d:0x0].
Then it is strange, the code should not jump from such place. Is there any (-1 level) debug log when the OI scrub triggered?
Here is the oi_scrub output from one of the MDS nodes that was rebooted on our open network this morning:
> cat osd-ldiskfs/lsc-MDT0000/oi_scrub name: OI_scrub magic: 0x4c5fd252 oi_files: 1 status: completed flags: param: time_since_last_completed: 28001 seconds time_since_latest_start: 36680 seconds time_since_last_checkpoint: 28001 seconds latest_start_position: 12 last_checkpoint_position: 991133697 first_failure_position: N/A checked: 170974399 updated: 0 failed: 0 prior_updated: 0 noscrub: 2284 igif: 12381456 success_count: 12 run_time: 8678 seconds average_speed: 19702 objects/sec real-time_speed: N/A current_position: N/A
The FID [0x7e4d2310f09:0x2ddf:0x0] is an normal FID, according to above logic, the condition:
"(ldiskfs_test_bit(osd_oi_fid2idx(dev,fid), sf->sf_oi_bitmap))" should true.
That means the OSD thinks that it is in OI files re-creating, so it is not sure whether it is because the object does not non-exist or because related OI mapping is not re-inserted yet.
So would you please to show the OI_scrub file under /proc to check whether the system is really in re-creating the OI files or not? Thanks!
Landed for 2.5.1 and 2.6