[LU-9952] soft lockup in osd_inode_iteration() for lustre 2.8.1

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Affects Version/s: Lustre 2.8.0
• Environment: Lustre ldiskfs server back end running version 2.8.1 with a few additional patches. The OS is RHEL6.9
• Severity: 2
• Assignee: nasf (Inactive)
• Reporter: James A Simmons

Description

One of production file systems running lustre 2.8.1 experienced a soft lock up very similar to LU-9488. I attempted to back port the patch but way to many changes have happened between 2.8.1 and lustre 2.10.0. Unsure if I would get the port right. I have attached the back trace.

Issue Links

• is related to: LU-9040 Soft lockup on CPU during lfsck
• is related to: LU-9488 soft lockup in osd_inode_iteration()

Comments

[James A Simmons]

We wouldn't be running the test framework on our production system. It looks like I just need to create a bunch of files on the file system.

lctl set_param -n osd*.MDT.force_sync=1
lctl set_param fail_val=1 fail_loc=0x190
lctl lfsck_start -M lustre-MDT0000
lctl set_param fail_val=0 fail_loc=0x198

While you check status:
lctl get_param -n osd-ldiskfs.lustre-MDT000.oi_scrub | grep status

Does this look right? What values do I use to reset it back to normal working conditions.

[nasf (Inactive)]

Let check whether this one https://review.whamcloud.com/#/c/29133/ works or not.

[James A Simmons]

Could you create a test condition before the 30th of September?

[nasf (Inactive)]

I think that we need some new fail_loc to simulate osd_inode_iteration() trouble. For example, inject the new failure stub in the osd_iit_next() to simulate kinds of bitmap layout cases.

[James A Simmons]

We are in the process of testing these patches. I attempted to recreate the problem with "lctl set_param fail_loc=0x1504" but that didn't work. What would you recommend to recreate this problem on a 2.8 system? Note we removed the offending files to make our production file system usable again.

[nasf (Inactive)]

The known patches on master that are related with the OI scrub soft lockup are back ported as following:

https://review.whamcloud.com/28903
https://review.whamcloud.com/28904
https://review.whamcloud.com/28905
https://review.whamcloud.com/28906

[Peter Jones]

Fan Yong

Can you please advise on this one?

Thanks

Peter