[#LU-170] OSCC_FLAG_LOW is always set and oscc_grow

[LU-170] OSCC_FLAG_LOW is always set and oscc_grow_count will never grow Created: 28/Mar/11 Updated: 28/Jun/11 Resolved: 24/May/11
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 1.8.6
Fix Version/s:	Lustre 2.1.0, Lustre 1.8.6

Type:

Bug

Priority:

Blocker

Reporter:

Liang Zhen (Inactive)

Assignee:

Liang Zhen (Inactive)

Resolution:

Fixed

Votes:

Labels:

None

Severity:

Rank (Obsolete):

5080

Description

I found osc_precreate is not working in the expected way, we always precreate 32 objects for some reason.

After digging into code I found there is a bug:

we are using req::rq_async_args.space[0] to store original value of oscc_grow_count, and using req::rq_async_args.pointer_arg[0] to store oscc, however, ptlrpc_async_args is a union, which means req::rq_async_args.space[0] will always be overwritten by a ossc (a pointer), and osc_interpret_create will always get true on this condition "if (diff < (int) req->rq_async_args.space[0])" and reset oscc_grow_count to OST_MIN_PRECREATE and set OSCC_FLAG_LOW.

The fix will be very simple but it has big impact on performance of file creation so I set priority as "major", I will post patch for it very soon.

Comments

Comment by Build Master (Inactive) [ 28/Mar/11 ]

Integrated in reviews-centos5 #579
~~LU-170~~ oscc_grow_count will never grow

Liang Zhen : 72cd44851509863fc36597d75e3d1c7917b4d62c
Files :

lustre/osc/osc_create.c

Comment by Liang Zhen (Inactive) [ 28/Mar/11 ]

I've posted the patch on http://review.whamcloud.com/#change,371
one concern is, do we really want to have so big OST_MAX_PRECREATE which is 20000
we have this bug on both 1.8.* and 2.*, and everybody is living with precreate 32 objects. I think 20000 is too big and it's easy to take seconds to create 20000 objects, this is dangerous for MDS because all threads could be waiting for these 20000 creating...

Comment by Oleg Drokin [ 28/Mar/11 ]

Not to worry about the big create count, there is a time-based limit on the OST side that would prevent OST spending too much time in precreate.
Good catch otherwise!

Comment by Oleg Drokin [ 28/Mar/11 ]

I did some research and I suspect I know how it was not initially caught.
On 32bit arch the cast of the oscc address to int would make a negative integer that would lead to always allowing the growth.
(this was introduced in bug 18577.)
I verified that on x86_64 the oscc address is split in such a way that it results in positive integer once casted to int.
I contacted Cray and they don't remember anymore if they were able to demonstrate performance improvements of the patch that introduced this issue (fighting the same cause too).
There is also hope that ORNL might be able to give a try to this patch during a dedicated testrun. They are interested because currently there is unexplained effect of too narrow of a gap between shared and uniq dir creates being observed.

Comment by Build Master (Inactive) [ 28/Mar/11 ]

Integrated in reviews-centos5 #594
~~LU-170~~ oscc_grow_count will never grow

Liang Zhen : eec7388d80a437fdf20ebf46ea42355cf45f8b77
Files :

lustre/osc/osc_create.c

Comment by Build Master (Inactive) [ 28/Mar/11 ]

Integrated in lustre-reviews » server,el5 #18
~~LU-170~~ oscc_grow_count will never grow

Liang Zhen : eec7388d80a437fdf20ebf46ea42355cf45f8b77
Files :

lustre/osc/osc_create.c

Comment by Build Master (Inactive) [ 28/Mar/11 ]

Integrated in lustre-reviews » server,el6 #18
~~LU-170~~ oscc_grow_count will never grow

Liang Zhen : eec7388d80a437fdf20ebf46ea42355cf45f8b77
Files :

lustre/osc/osc_create.c

Comment by Build Master (Inactive) [ 28/Mar/11 ]

Integrated in lustre-reviews » client,el5 #18
~~LU-170~~ oscc_grow_count will never grow

Liang Zhen : eec7388d80a437fdf20ebf46ea42355cf45f8b77
Files :

lustre/osc/osc_create.c

Comment by Build Master (Inactive) [ 28/Mar/11 ]

Integrated in lustre-reviews » client,el6 #18
~~LU-170~~ oscc_grow_count will never grow

Liang Zhen : eec7388d80a437fdf20ebf46ea42355cf45f8b77
Files :

lustre/osc/osc_create.c

Comment by Build Master (Inactive) [ 28/Mar/11 ]

Integrated in lustre-reviews » client,ubuntu #18
~~LU-170~~ oscc_grow_count will never grow

Liang Zhen : eec7388d80a437fdf20ebf46ea42355cf45f8b77
Files :

lustre/osc/osc_create.c

Comment by Eric Barton (Inactive) [ 29/Mar/11 ]

This use of rq_async_args (i.e. dereferencing 'space[]' and
the declaration of 'pointer_arg[]' and use in ldlm_cb_interpret(),
mdc_intent_getattr_sync_interpret() etc) invites defects and is
UNACCEPTABLE.

Each new or different use of this scratchpad must have its
own structure definition - see how it's done with osc_brw_async_args,
osc_create_args etc.

Ideally, rq_async_args should be declared as the union of all these
structs, but since that would require subsystem-specific structs
to be declared too globally, it's acceptable to simply reserve space
(that's the ONLY permissable use of 'space' here) AND CLASSERT that
enough space has been reserved.

These issues must either be fixed in this bug, or a new bug created
with this as its first comment.

Comment by Build Master (Inactive) [ 29/Mar/11 ]

Integrated in reviews-centos5 #608
~~LU-170~~ oscc_grow_count will never grow

Liang Zhen : eec7388d80a437fdf20ebf46ea42355cf45f8b77
Files :

lustre/osc/osc_create.c

Comment by Build Master (Inactive) [ 30/Mar/11 ]

Integrated in reviews-centos5 #610
~~LU-170~~ oscc_grow_count will never grow

Liang Zhen : 515135a06055dd5047877786998ff400a5f12ba1
Files :

lustre/mdc/mdc_locks.c
lustre/ldlm/ldlm_request.c
lustre/ldlm/ldlm_internal.h
lustre/mdc/mdc_request.c
lustre/include/lustre_net.h
lustre/osc/osc_create.c
lustre/mdc/mdc_internal.h
lustre/ldlm/ldlm_lockd.c
lustre/ptlrpc/recov_thread.c

Comment by Build Master (Inactive) [ 31/Mar/11 ]

Integrated in reviews-centos5 #630
~~LU-170~~ oscc_grow_count will never grow

Liang Zhen : 3e44a66438ece2da0e2467a8dd1f00c0933e87f4
Files :

lustre/include/lustre_net.h
lustre/mdc/mdc_request.c
lustre/ldlm/ldlm_lockd.c
lustre/ptlrpc/recov_thread.c
lustre/osc/osc_create.c
lustre/mdc/mdc_locks.c

Comment by Build Master (Inactive) [ 06/Apr/11 ]

Integrated in lustre-master » client,el5-x86_64 #17
~~LU-170~~ oscc_grow_count will never grow

Oleg Drokin : b9ab6348153fd8d236871f75e5bd3793a49c9a45
Files :

lustre/mdc/mdc_locks.c
lustre/ptlrpc/recov_thread.c
lustre/mdc/mdc_request.c
lustre/osc/osc_create.c
lustre/ldlm/ldlm_lockd.c
lustre/include/lustre_net.h

Comment by Build Master (Inactive) [ 06/Apr/11 ]

Integrated in lustre-master » client,el6-x86_64 #17
~~LU-170~~ oscc_grow_count will never grow

Oleg Drokin : b9ab6348153fd8d236871f75e5bd3793a49c9a45
Files :

lustre/mdc/mdc_locks.c
lustre/ldlm/ldlm_lockd.c
lustre/ptlrpc/recov_thread.c
lustre/osc/osc_create.c
lustre/mdc/mdc_request.c
lustre/include/lustre_net.h

Comment by Build Master (Inactive) [ 06/Apr/11 ]

Integrated in lustre-master-centos5 #183
~~LU-170~~ oscc_grow_count will never grow

Oleg Drokin : b9ab6348153fd8d236871f75e5bd3793a49c9a45
Files :

lustre/mdc/mdc_request.c
lustre/mdc/mdc_locks.c
lustre/ldlm/ldlm_lockd.c
lustre/include/lustre_net.h
lustre/osc/osc_create.c
lustre/ptlrpc/recov_thread.c

Comment by Build Master (Inactive) [ 06/Apr/11 ]

Integrated in lustre-master » client,ubuntu-x86_64 #17
~~LU-170~~ oscc_grow_count will never grow

Oleg Drokin : b9ab6348153fd8d236871f75e5bd3793a49c9a45
Files :

lustre/include/lustre_net.h
lustre/osc/osc_create.c
lustre/ldlm/ldlm_lockd.c
lustre/mdc/mdc_request.c
lustre/mdc/mdc_locks.c
lustre/ptlrpc/recov_thread.c

Comment by Build Master (Inactive) [ 06/Apr/11 ]

Integrated in lustre-master » client,el6-i686 #17
~~LU-170~~ oscc_grow_count will never grow

Oleg Drokin : b9ab6348153fd8d236871f75e5bd3793a49c9a45
Files :

lustre/ptlrpc/recov_thread.c
lustre/ldlm/ldlm_lockd.c
lustre/osc/osc_create.c
lustre/mdc/mdc_request.c
lustre/include/lustre_net.h
lustre/mdc/mdc_locks.c

Comment by Build Master (Inactive) [ 06/Apr/11 ]

Integrated in lustre-master » server,el5-i686 #17
~~LU-170~~ oscc_grow_count will never grow

Oleg Drokin : b9ab6348153fd8d236871f75e5bd3793a49c9a45
Files :

lustre/mdc/mdc_locks.c
lustre/mdc/mdc_request.c
lustre/ldlm/ldlm_lockd.c
lustre/osc/osc_create.c
lustre/include/lustre_net.h
lustre/ptlrpc/recov_thread.c

Comment by Build Master (Inactive) [ 06/Apr/11 ]

Integrated in lustre-master » server,el5-x86_64 #17
~~LU-170~~ oscc_grow_count will never grow

Oleg Drokin : b9ab6348153fd8d236871f75e5bd3793a49c9a45
Files :

lustre/ldlm/ldlm_lockd.c
lustre/include/lustre_net.h
lustre/osc/osc_create.c
lustre/mdc/mdc_locks.c
lustre/ptlrpc/recov_thread.c
lustre/mdc/mdc_request.c

Comment by Build Master (Inactive) [ 06/Apr/11 ]

Integrated in lustre-master » client,el5-i686 #17
~~LU-170~~ oscc_grow_count will never grow

Oleg Drokin : b9ab6348153fd8d236871f75e5bd3793a49c9a45
Files :

lustre/osc/osc_create.c
lustre/mdc/mdc_locks.c
lustre/include/lustre_net.h
lustre/ldlm/ldlm_lockd.c
lustre/ptlrpc/recov_thread.c
lustre/mdc/mdc_request.c

Comment by Build Master (Inactive) [ 06/Apr/11 ]

Integrated in lustre-master » server,el6-x86_64 #17
~~LU-170~~ oscc_grow_count will never grow

Oleg Drokin : b9ab6348153fd8d236871f75e5bd3793a49c9a45
Files :

lustre/mdc/mdc_locks.c
lustre/ptlrpc/recov_thread.c
lustre/osc/osc_create.c
lustre/mdc/mdc_request.c
lustre/include/lustre_net.h
lustre/ldlm/ldlm_lockd.c

Comment by Peter Jones [ 06/Apr/11 ]

Fix landed to master. Please reopen if more work required.

Comment by Build Master (Inactive) [ 07/Apr/11 ]

Integrated in lustre-master » server,el6-i686 #20
~~LU-170~~ oscc_grow_count will never grow

Oleg Drokin : b9ab6348153fd8d236871f75e5bd3793a49c9a45
Files :

lustre/mdc/mdc_request.c
lustre/ptlrpc/recov_thread.c
lustre/osc/osc_create.c
lustre/ldlm/ldlm_lockd.c
lustre/mdc/mdc_locks.c
lustre/include/lustre_net.h

Comment by Liang Zhen (Inactive) [ 14/Apr/11 ]

it's still on b1_8, I will workout a patch for b1_8

Comment by Liang Zhen (Inactive) [ 14/Apr/11 ]

reopen it for b1_8