[LU-11089] Performance improvements for lu_object locking

• Type: Improvement
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.13.0, Lustre 2.12.5
• Affects Version/s: Lustre 2.12.0
• Assignee: James A Simmons
• Reporter: James A Simmons

Description

While porting the LU-6800 work upstream the reaction to the approach was disliked since it wasn't a real improvement. Neil has created a patch series to break up the global lock to increase its performance.

Issue Links

• is related to: LU-8346 conf-sanity test_93: test failed to respond and timed out
• is related to: LU-12565 Use bit locking in obd_echo
• is related to: LU-6800 Significant performance regression with patch LU-5264
• is related to: LU-9679 Prepare lustre for adoption into the linux kernel

Comments

[Shuichi Ihara]

Patrick, sorry! forgot to update this ticket. Here is new ticket. LU-11624.

[Patrick Farrell (Inactive)]

Ihara,

Could you link that ticket here?  I'm interested in tracking it.  Our MDSses running 2.12 are crashing when we fail them over under load.  Pretty reliably.

[James A Simmons]

Thanks. I have a patch based on LU-8130 work that should fix this.

[Shuichi Ihara]

James, crashing servers were not related to your patches (LU-11089), but looks like more general problem in master. me open new jira ticket for this.

[Peter Jones]

Could we please have a separate ticket for any instances seen on 2.12 or earlier releases without James's unlanded patches being applied? Is there any suggestion that this is happening more frequently on 2.12 compared to 2.11 and earlier releases?

[Patrick Farrell (Inactive)]

Ah, I see now that none of the patches have landed.  So it is definitely a pre-existing bug.  Interesting.

[Patrick Farrell (Inactive)]

Seen here during recovery as well.  Interesting.  I imagine even if the bug was already there, the changes made it easier to hit.  (Doesn't mean the changes are wrong, just that there's probably a reason we're suddenly seeing it.)

[James A Simmons]

Since the NID hash seems to be broken in general I did a port to rhashtables. Still need to work on the /proc entries to display hash stats. Please try it out to see if no longer crashes your nodes. Patch is at:

https://review.whamcloud.com/#/c/33518

The build breakage is only on SLES12SP3.

[James A Simmons]

Thanks for the info. Ruth has pointed out that this is a general bug. I have started the port of the nid hash to rhashtable and I'm seeing hidden issues with the original code.

[Shuichi Ihara]

This happens quite offten. I saw crash even at intial mount. e.g. create filesystem and mount Lustre on 32 clients, then got crash of one of OSS.

[Ruth Klundt (Inactive)]

One off with no patches applied.

After I brought the node back all the clients mounted and tests ran.