[LU-14580] Lustre 2.12.6 performance regression

• Type: Bug
• Resolution: Unresolved
• Priority: Major
• Affects Version/s: Lustre 2.12.6
• Labels: performance
• Environment: 16 Socket Superdome Flex w/8 EDR IB interfaces
• Severity: 2
• Assignee: Neil Brown
• Reporter: Stephen Champion

Description

Between Lustre 2.12.5 and 2.12.6 releases, we have a major performance regression on large systems with multiple interfaces.

On a 16 socket Superdome Flex with 8 EDR interfaces functioning as a client of a filesystem capable of > 100 GB/s, using Lustre 2.12.5, we are measuring up to 50 GB/s. With Lustre 2.12.6, the same test is down to 13 GB/s.

Reverting commit ID f92c7a161242c478658af09159a127bc21cba611 restores performance.

Issue Links

• is related to: LU-14055 Write performance regression caused by an commit from LU-13344
• is related to: LU-12820 obd_dirty_transit_pages is always zero and can be removed

Comments

[Patrick Farrell]

I'm pretty confident this and LU-14055 are the same issue, so I'm snagging both of these.

I'm going to leave this one open for now, but I think LU-14055 is the better place to continue the discussion.

[Peter Jones]

Did you get any updated results on this Steve?

[Stephen Champion]

@John Hammond Good idea! I tried re-introducing the padding to no effect on x86_64. 128-bit cache line architectures may have different results.

[Mikhail Pershin]

I don't see problems with patch itself. Increment in osc_consume_write_grant() was removed because it is done by atomic_long_add_return() now outside that call and it is done in both places where it is called. But maybe the patch "LU-12687 osc: consume grants for direct I/O" itself causes slowdown? Now grants are taken for Direct IO as well, so maybe that is related to not enough grants problem or similar. Are there any complains about grants on client during IOR run?

[Mikhail Pershin]

Sure, I am checking that . These changes are combination of two patches actually: first ID is Ia047affc33fb9277e6c28a8f6d7d088c385b51a8 and next one is already referred f92c7a161242c478658af09159a127bc21cba611

Their ports to 2.12 base were a bit different from master branch so I am checking if some functionality was broken.