[#LU-499] grant_rate and cancel_rate are static when an OST is idle

[LU-499] grant_rate and cancel_rate are static when an OST is idle Created: 11/Jul/11 Updated: 08/Apr/12 Resolved: 13/Oct/11
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 2.1.0
Fix Version/s:	Lustre 2.2.0, Lustre 2.1.2

Type:

Bug

Priority:

Minor

Reporter:

Prakash Surya (Inactive)

Assignee:

Lai Siyao

Resolution:

Fixed

Votes:

Labels:

None

Environment:

Lustre 2.0.63-1chaos

Severity:

Rank (Obsolete):

4887

Description

When an OST is idle and has no outstanding locks (i.e. lock_count reports 0) it seems like the pool/grant_rate and pool/cancel_rate values become static. Also, the "samples" value for these fields in pool/stats remains static under the same circumstance (lock_count being 0). If lock_count is non zero, these values get updated at a given interval (which appears to be every second on our system).

This comes up when running IORs. For example, when an IOR finishes on our test system, and no other activity is occurring, these values remain static until the next IOR is started. Once the second IOR is started, the values drop back to zero and then fluctuate through the duration of the test.

Is this expected behavior? Talking with Chris, it seems that the grant_rate, cancel_rate, etc. should continue to be updated when there is no activity for a given OST (or at least zeroed out). Although, this is not the behavior we are seeing.

Comments

Comment by Peter Jones [ 11/Jul/11 ]

Yang Sheng

Could you please look into this one when you have finished with your current priority task

Thanks

Peter

Comment by Prakash Surya (Inactive) [ 26/Jul/11 ]

Is this correct behaviour? Can anybody comment?

Comment by Peter Jones [ 01/Aug/11 ]

Lai

Could you help with this ticket?

Thanks

Peter

Comment by Lai Siyao [ 04/Aug/11 ]

Hi Ericm,

This looks to be introduced by bz21519: ns->ns_bref equals zero doesn't necessarily mean the namespace is to be destroyed, but no lock exists, so it's better to recalc namespace pool stats anyway.

Any comment is welcome!

Thanks,

Comment by Eric Mei (Inactive) [ 04/Aug/11 ]

Hi Lai, my impression is that namespace must be referenced at least once by its user, a target device or client, even if there's no cached lock. reference dropped to 0 only possible when it is being destructed. Can you show me the code where else reference can drop to 0? (currently I don't even have Lustre source to check that)

Comment by Lai Siyao [ 04/Aug/11 ]

Hi Ericm,

The comment of namespace->ns_bref is /* big refcount (by bucket) */, that means only when a new hash bucket is created, this refcount will be increased. The initial value of ns_bref is 0 for a newly created namespace, and ldlm_namespace_put() doesn't free namespace directly.

void ldlm_namespace_put(struct ldlm_namespace *ns)
{
        if (cfs_atomic_dec_and_lock(&ns->ns_bref, &ns->ns_lock)) {
                cfs_waitq_signal(&ns->ns_waitq);
                cfs_spin_unlock(&ns->ns_lock);
        }
}

Instead, each device calls ldlm_namespace_free() explicitly in cleanup time.

Comment by Lai Siyao [ 05/Aug/11 ]

Hi Prakash, review is on http://review.whamcloud.com/#change,1185, could you verify it works?

Comment by Prakash Surya (Inactive) [ 05/Aug/11 ]

Hi Lai, Yes I will try this out today and make sure it works. I'll comment back here and on the gerrit patch once I get it installed and tested. Thank You!

Comment by Prakash Surya (Inactive) [ 08/Aug/11 ]

Sorry for taking so long to test this patch out. I've installed http://review.whamcloud.com/#change,1185 on one of out OSS nodes and everything looks good. The grant_rate and cancel_rate are now being updated even when lock_count is zero.