[#LU-1137] flock owner incorrectly handled on the server side

[LU-1137] flock owner incorrectly handled on the server side Created: 24/Feb/12 Updated: 20/Nov/12 Resolved: 06/Apr/12
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 2.1.0, Lustre 2.3.0
Fix Version/s:	Lustre 2.3.0, Lustre 2.1.2

Type:

Bug

Priority:

Blocker

Reporter:

Iurii Golovach (Inactive)

Assignee:

WC Triage

Resolution:

Fixed

Votes:

Labels:

None

Environment:

Client 1.8.x and less. Server 2.1.x

Attachments:

Log1.zip

Log2.log

Issue Links:

Related
is related to	~~LU-1575~~	Add flock policy support	Resolved

Severity:

Rank (Obsolete):

4662

Description

During flock transmission on the server side the owner field incorrectly handled. This is a reason of errors on 1.8 & 1.6 clients.

whole history is traced at the CLSTR-23
http://jira-nss.xy01.xyratex.com:8080/browse/CLSTR-23

Comments

Comment by Iurii Golovach (Inactive) [ 24/Feb/12 ]

The patch for the issue is placed in review here:
http://review.whamcloud.com/#change,2193

Comment by Andreas Dilger [ 24/Feb/12 ]

Sorry, I'm unable to access http://jira-nss.xy01.xyratex.com:8080/browse/CLSTR-23. Is that an internal website? If yes, please attach logs/description here.

Comment by Iurii Golovach (Inactive) [ 27/Feb/12 ]

Hi Andreas. My apologize. I'm attaching two logs and the brief from the ticket:

---------------------------------------------------
the fcntl() call to unlock a file seems to fail. We have a test which needs file locking to serialize the updates of the file position. Our standard test of this function passes on other Lustre file systems, even at more than 20k ranks. The same test fails on s4 even with as few as 2 ranks. One rank will lock, read, write, unlock the file; other ranks hang, never getting the lock.
---------------------------------------------------
first of all thanks a lot for these logs - they will save us some time.

There is a patch http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=commitdiff;h=3bb8d1b9656994b0313cdba6ad8eeb7b84f5ee9f which wasn't fully ported and looks like a fix for this issue. Tomorrow I'll create a patch for your code and attach it here for you to run the test and let us know if it fix the issue.
----------------------------------------------------
Issue observed on the function ll_file_flock(). In this function there is a variable ldlm_policy_data_t flock; which has to have all fields initialized properly.

The issue is that only next 3 fields are initialized:
flock.l_flock.pid = file_lock->fl_pid;
flock.l_flock.start = file_lock->fl_start;
flock.l_flock.end = file_lock->fl_end;

While the l_flock.owner field used uninitialized in this function and is the reason of errors in log.

Looks like there was an issue when you integrated the fix for ~~LU-104~~ into your code.
You integrated the http://review.whamcloud.com/279 only while this is a small part of the fix which has the hash 3bb8d1b9656994b0313cdba6ad8eeb7b84f5ee9f (in WC repo) and was attached at the https://bugzilla.lustre.org/attachment.cgi?id=30902 like a fix for the https://bugzilla.lustre.org/show_bug.cgi?id=22040 wasn't integrated.

This fix contains next lines which resolve the current issue:

+ lpolicy->l_flock.owner = wpolicy->l_flock.lfw_owner;
+ /* Compat code, old clients had no idea about owner field and
+ * relied solely on pid for ownership. Introduced in 2.1, April 2011 */
+ if (!lpolicy->l_flock.owner)
+ lpolicy->l_flock.owner = wpolicy->l_flock.lfw_pid;

....I'll attach the modified patch which can be applied on your code in close hour.
----------------------------------------------------
Bad news. flock test still fails with patch applied. Attaching dk logs from today's test. Again, I have ldumps if you want them (takes some effort to package up in a readable format; otherwise I'd just send them to you).

FWIW, we picked up the partial flock fix from Lustre bug 22660 att 29853 as part of the official 1.8.6 release.
----------------------------------------------------
From the logs you attached I see next:

p1) you applied the patch and the flock owner value was transferred correctly first two times, but then there is an owner corruption on the client side:

00010000:00000001:0.0F:1329519637.982734:0:3717:0:(ldlm_flock.c:516:ldlm_flock_completion_ast()) Process entered
00010000:00010000:0.0F:1329519637.982734:0:3717:0:(ldlm_flock.c:519:ldlm_flock_completion_ast()) flags: 0x0 data: (null) getlk: ffff88021f782e48
00010000:00010000:0.0:1329519637.982735:0:3717:0:(ldlm_flock.c:593:ldlm_flock_completion_ast()) ### client-side enqueue granted ns: castor-MDT0000-mdc-ffff880414355400 lock: ffff880216374900/0x472ca55e16a1d2d7 lrc: 4/0,1 mode: PW/PW res: 28738515467/7 rrc: 2 type: FLK pid: 3717 [0->7] flags: 0x0 remote: 0x8f11d572f8beb3c2 expref: -99 pid: 3717 timeout: 0
00010000:00000001:0.0F:1329519637.982740:0:3717:0:(ldlm_flock.c:166:ldlm_process_flock_lock()) Process entered
00010000:00010000:0.0F:1329519637.982741:0:3717:0:(ldlm_flock.c:172:ldlm_process_flock_lock()) flags 0x40 owner 18446612141434674600 pid 3717 mode 2 start 0 end 7
00010000:00000001:0.0:1329519637.982771:0:3717:0:(ldlm_flock.c:469:ldlm_process_flock_lock()) Process leaving (rc=1 : 1 : 1)
00010000:00000001:0.0:1329519637.982772:0:3717:0:(ldlm_flock.c:634:ldlm_flock_completion_ast()) Process leaving (rc=0 : 0 : 0)

....

00010000:00000001:0.0F:1329519637.986301:0:3717:0:(ldlm_flock.c:516:ldlm_flock_completion_ast()) Process entered
00010000:00010000:0.0F:1329519637.986302:0:3717:0:(ldlm_flock.c:519:ldlm_flock_completion_ast()) flags: 0x40000 data: (null) getlk: ffff88021f782e48
00010000:00010000:0.0:1329519637.986303:0:3717:0:(ldlm_flock.c:593:ldlm_flock_completion_ast()) ### client-side enqueue granted ns: castor-MDT0000-mdc-ffff880414355400 lock: ffff880216374240/0x472ca55e16a1d308 lrc: 4/1,0 mode: NL/NL res: 28738515467/7 rrc: 3 type: FLK pid: 3717 [0->7] flags: 0x0 remote: 0x8f11d572f8beb3d0 expref: -99 pid: 3717 timeout: 0
00010000:00000001:0.0F:1329519637.986307:0:3717:0:(ldlm_flock.c:166:ldlm_process_flock_lock()) Process entered
00010000:00010000:0.0F:1329519637.986308:0:3717:0:(ldlm_flock.c:172:ldlm_process_flock_lock()) flags 0x40 owner 1329519637 pid 3717 mode 32 start 0 end 7

As you may see the owner value was changed from 18446612141434674600 to 1329519637 and was corrupted.

This cause the situation when process 3717 wants to release the lock but can't do this since owner value is incorrect.
This is the reason why the process with pid 3718 can't finish the unlink.
----------------------------------------------------
The latest patch works. The flock test no longer hangs.

Comment by Iurii Golovach (Inactive) [ 07/Mar/12 ]

Looks like an issue with build system (Hudson failed the build with the next error):

+++ mv '/var/lib/jenkins/lbuild-data/kernelrpm/2.6.18/rhel5/i686/yumW12939/*' /var/lib/jenkins/lbuild-data/kernelrpm/2.6.18/rhel5/i686
mv: cannot stat `/var/lib/jenkins/lbuild-data/kernelrpm/2.6.18/rhel5/i686/yumW12939/*': No such file or directory
++++ set +x
+++ rmdir /var/lib/jenkins/lbuild-data/kernelrpm/2.6.18/rhel5/i686/yumW12939
+++ echo '*'
+++ return 0
++ found_rpm='An unexpected error has occurred at lbuild-rhel:190.

http://build.whamcloud.com/job/lustre-reviews/5093/arch=i686,build_type=client,distro=el5,ib_stack=inkernel/console