[LU-3932] HSM can only handle ~4G requests at most

• Type: Technical task
• Resolution: Won't Fix
• Priority: Major
• Affects Version/s: Lustre 2.5.0
• Labels: hsm
• Assignee: John Hammond
• Reporter: Jinshan Xiong (Inactive)

Description

In current implementation, a catalog llog can only have ~4G entries in its lifetime which means HSM can only handle ~4G requests at most.

We have to improve catalog. It seems reasonable for me to roll catalog index back when it reaches maximum, assuming that it's impossible to have ~4G entries on the same time.

Comments

[Andreas Dilger]

It is worth noting that this only affects the case of the MDS crashes or is shut down while the first and last ChangeLog catalog llogs are straddling the start of the catalog. That would return ChangeLog entries in a slightly wrong order. That is not a problem for HSM, but may be for lustre-rsync and similar.

[Jinshan Xiong (Inactive)]

Andreas says that the catalog should store the catalog index for the first (or last) index that is used. When processing the ChangeLog, it should start at the first (or one after the last) catalog index so that the individual llog files are processed in order.

[Jodi Levi (Inactive)]

After speaking to Jinshan, this will not be fixed.

[Jinshan Xiong (Inactive)]

After taking a further look at the implementation of catalog llog implementation, unlike plain llog, it reuses its index number. However, after the indices are reused, reading catalog entries will not be in the order when they were added.

This doesn't seem to be a severe problem because we have ~4G slot anyway.

I will drop the priority.