[LU-4398] mdt_object_open_lock() may not flush conflicting handles

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.13.0, Lustre 2.12.4
• Affects Version/s: Lustre 2.6.0, Lustre 2.5.1
• Labels: mdt
• Severity: 3
• Assignee: Gu Zheng (Inactive)
• Reporter: John Hammond

Description

Calls to mdt_object_open_lock() which do not have MDS_OPEN_LOCK in open flags may fail to flush conflicting handles.

t:~# rm /mnt/lustre/*
t:~# ls /mnt/lustre/
t:~# cp /bin/echo /mnt/lustre/echo
t:~# lfs path2fid /mnt/lustre/echo
[0x280000401:0x9:0x0]
t:~# /mnt/lustre/echo Hi
Hi
t:~# lctl clear
t:~# echo Bye > /mnt/lustre2/echo
-bash: /mnt/lustre2/echo: Text file busy
t:~# lctl dk > 2.dk

00000004:00000002:0.0:1387473999.026142:0:15064:0:(mdt_open.c:1610:mdt_reint_open()) I am going to open [0x200000007:0x1:0x0]/(echo->[0x280000402:0x4:0x0]) cr_flag=01102 mode=0100666 msg_flag=0x0

** There is a typo in the CDEBUG for the next message, the open count
** is really printed.

        CDEBUG(D_INODE, "normal open:"DFID" lease count: %d, lm: %d\n",
               PFID(mdt_object_fid(obj)),
               atomic_read(&obj->mot_open_count), lm);

00000004:00000002:0.0:1387473999.026365:0:15064:0:(mdt_open.c:1242:mdt_object_open_lock()) normal open:[0x280000401:0x9:0x0] lease count: 1, lm: 16

A CR LOOKUP LAYOUT is granted:

00000004:00000002:0.0:1387473999.026430:0:15064:0:(mdt_open.c:1269:mdt_object_open_lock()) Requested bits lock:[0x280000401:0x9:0x0], ibits = 0x9, open_flags = 01102, try_layout = 1, rc = 0
00000004:00000001:0.0:1387473999.026433:0:15064:0:(mdt_open.c:1332:mdt_object_open_lock()) Process leaving via out (rc=0 : 0 : 0x0)

00000004:00000001:0.0:1387473999.026456:0:15064:0:(mdt_open.c:536:mdt_write_get()) Process leaving (rc=18446744073709551590 : -26 : ffffffffffffffe6)
00000004:00000001:0.0:1387473999.026458:0:15064:0:(mdt_open.c:723:mdt_mfd_open()) Process leaving (rc=18446744073709551590 : -26 : ffffffffffffffe6)
00000004:00000001:0.0:1387473999.026459:0:15064:0:(mdt_open.c:994:mdt_finish_open()) Process leaving (rc=18446744073709551590 : -26 : ffffffffffffffe6)

Issue Links

• duplicates: LU-4773 "Text file busy" error when creating executables on NFS share and trying to run them on lustre clients
• is duplicated by: LU-4996 Interop 2.5.1<->2.6 failure on test suite sanityn test_14e: Text file busy
• is related to: LU-12661 sanity test_817: FAIL: failed to execute 'true' command
• is related to: LU-5197 A performance regression of "FileRead" metadata operation
• is related to: LU-4520 Text file busy error -- mainline 3.12 client
• is related to: LU-6232 Text file busy error -- lustre 2.6.0 client
• is related to: LU-10457 open_by_handle_at() in write mode triggers ETXTBSY
• is related to: LU-4671 sanity test_43: multiop still running
• is related to: LU-10948 client cache open lock after N opens
• is related to: LU-4152 layout locks can cause deadlock
• is related to: LU-10300 Can the Lustre 2.10.x clients support 64K kernel page?

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/36680/
Subject: LU-4398 llite: do not cache write open lock for exec file
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 07f782635c7b5de59a13371c6d14aa2c4910d257

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36680
Subject: LU-4398 llite: do not cache write open lock for exec file
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 639b51789ebcae59ba8ed1ba383bdc9e3126d67d

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36142
Subject: LU-4398 tests: re-enable sanity test 817
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e800ae379b506fb6f5f4917fe4e896ff5873dfb3

[Gu Zheng (Inactive)]

Fix patch has been landed, it's fine to close this ticket.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32265/
Subject: LU-4398 llite: do not cache write open lock for exec file
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6dd9d57bc006a37731d34409ce43de13c192e0cc

[Gu Zheng (Inactive)]

Seems nfs is problem on aarch64, the script failed even ran without lustre.

[root@trevis-80vm4 ~]# bash run_test.sh
+ systemctl restart nfs-server.service
+ rm -rf /tmp/nfs_test /tmp/nfs-60b4UP/
+ mkdir /tmp/nfs_test /tmp/nfs-60b4UP/
+ exportfs -o rw,no_root_squash localhost:/tmp/nfs_test
+ mount -t nfs localhost:/tmp/nfs_test /tmp/nfs-60b4UP/
+ cp /bin/true /tmp/nfs-60b4UP/
+ /tmp/nfs_test/true
run_test.sh: line 14: /tmp/nfs_test/true: Text file busy
+ umount /tmp/nfs-60b4UP/
+ systemctl stop nfs-server.service

Even use nfsV3:

[root@trevis-80vm4 ~]# bash run_test.sh
+ systemctl restart nfs-server.service
+ rm -rf /tmp/nfs_test /tmp/nfs-60b4UP/
+ mkdir /tmp/nfs_test /tmp/nfs-60b4UP/
+ exportfs -o rw,no_root_squash localhost:/tmp/nfs_test
+ mount -t nfs -o vers=3 localhost:/tmp/nfs_test /tmp/nfs-60b4UP/
+ cp /bin/true /tmp/nfs-60b4UP/
+ /tmp/nfs_test/true
run_test.sh: line 13: /tmp/nfs_test/true: Text file busy
+ umount /tmp/nfs-60b4UP/
+ systemctl stop nfs-server.service

Will skip the test case for aarch64, temporarily.

[Gu Zheng (Inactive)]

I back ported Jinshan's new fix patch (https://review.whamcloud.com/#/c/32265) to master,  which always sends an real mds close from the client, if the executable file was opened in write mode. But it's strange that, the nfs test always failed on aarch64 client (other arches clients are fine), and I can easy reproduce it on trevis-80vm3.

+ systemctl restart nfs-server.service
+ mount -t lustre 10.9.6.161@tcp:/lustre /mnt/lustre
+ mkdir /mnt/lustre/nfs_test
+ exportfs -orw,no_root_squash localhost:/mnt/lustre/nfs_test
+ mount -t nfs localhost:/mnt/lustre/nfs_test /tmp/nfs-60b4UP/
+ cp /bin/true /tmp/nfs-60b4UP/
+ /mnt/lustre/nfs_test/true
LU-4398.sh: line 10: /mnt/lustre/nfs_test/true: Text file busy
+ umount /tmp/nfs-60b4UP/
+ systemctl stop nfs-server.service
+ rm -rf /mnt/lustre/nfs_test
+ umount /mnt/lustre

kernel: 4.14.0-115.2.2.el7a.aarch64

[Andreas Dilger]

Colin, I agree it seems the patches on this ticket have not made any progress lately. It looks like the most recent version of https://review.whamcloud.com/32020 indicate that the patch is causing a regression in sanity-hsm, and needs to be refreshed in any case. It looks like there is also a test case in https://pastebin.com/GHj0rqxT that would be sueful to include.

[Colin Faber [X] (Inactive)]

Has this been abandoned?

[Gerrit Updater]

Jinshan Xiong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32265
Subject: LU-4398 llite: do not cache write open lock for exec file
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ba79fe03a7fcdbc1c7e8ef186c987905c2cb6e85