[LU-4840] Deadlock when truncating file during lfs migrate

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.4.2
• Labels: cea
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Patrick Valentin (Inactive)

Description

While migrating a file with "lfs migrate", if a process tries to truncate the file, both lfs migrate and truncating processes will deadlock.

This will result in both processes never finishing (unless it is killed) and watchdog messages saying that the processes did not progress for the last XXX seconds.

Here is a reproducer:

[root@lustre24cli ~]# cat reproducer.sh
#!/bin/sh

FS=/test
FILE=${FS}/file

rm -f ${FILE}
# Create a file on OST 1 of size 512M
lfs setstripe -o 1 -c 1 ${FILE}
dd if=/dev/zero of=${FILE} bs=1M count=512

echo 3 > /proc/sys/vm/drop_caches

# Launch a migrate to OST 0 and a bit later open it for write
lfs migrate -i 0 --block ${FILE} &
sleep 2
dd if=/dev/zero of=${FILE} bs=1M count=512 

Once the last dd tries to open the file, both lfs and dd processes stay forever with this stack:

lfs stack:

[<ffffffff8128e864>] call_rwsem_down_read_failed+0x14/0x30
[<ffffffffa08d98dd>] ll_file_io_generic+0x29d/0x600 [lustre]
[<ffffffffa08d9d7f>] ll_file_aio_read+0x13f/0x2c0 [lustre]
[<ffffffffa08da61c>] ll_file_read+0x16c/0x2a0 [lustre]
[<ffffffff811896b5>] vfs_read+0xb5/0x1a0
[<ffffffff811897f1>] sys_read+0x51/0x90
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

dd stack:

[<ffffffffa03436fe>] cfs_waitq_wait+0xe/0x10 [libcfs]
[<ffffffffa04779fa>] cl_lock_state_wait+0x1aa/0x320 [obdclass]
[<ffffffffa04781eb>] cl_enqueue_locked+0x15b/0x1f0 [obdclass]
[<ffffffffa0478d6e>] cl_lock_request+0x7e/0x270 [obdclass]
[<ffffffffa047e00c>] cl_io_lock+0x3cc/0x560 [obdclass]
[<ffffffffa047e242>] cl_io_loop+0xa2/0x1b0 [obdclass]
[<ffffffffa092a8c8>] cl_setattr_ost+0x208/0x2c0 [lustre]
[<ffffffffa08f8a0e>] ll_setattr_raw+0x9ce/0x1000 [lustre]
[<ffffffffa08f909b>] ll_setattr+0x5b/0xf0 [lustre]
[<ffffffff811a7348>] notify_change+0x168/0x340
[<ffffffff81187074>] do_truncate+0x64/0xa0
[<ffffffff8119bcc1>] do_filp_open+0x861/0xd20
[<ffffffff81185d39>] do_sys_open+0x69/0x140
[<ffffffff81185e50>] sys_open+0x20/0x30
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

Issue Links

• is related to: LU-6785 Interop 2.7.0<->master sanity test_56w: cannot swap layouts: Device or resource busy
• is related to: LU-5915 racer test_1: FAIL: test_1 failed with 4
• is related to: LU-7073 racer with OST object migration hangs on cleanup
• is related to: LU-6903 racer file migration crash ASSERTION( lov->lo_type == LLT_RAID0 )

Comments

[Jean-Baptiste Riaux (Inactive)]

Backport to b2_7_fe http://review.whamcloud.com/#/c/21513/

[Andreas Dilger]

Closing LU-4840 since the patches here are landed to fix the problems described here, except http://review.whamcloud.com/13669 to enable racer with object migrate. LU-7073 has been opened to track the new racer + object migrate failure.

[Sebastien Buisson (Inactive)]

Hi,

Now that patch at http://review.whamcloud.com/10013 has been merged into master, is it possible to have a backport to b2_5?
Indeed, the original issue was met on one of our customer clusters running Lustre 2.4, that was updated afterwards in Lustre 2.5.

Thanks,
Sebastien.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/10013/
Subject: LU-4840 lfs: Use file lease to implement migration
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 85bd36cc69563d7a79e3ed34f8fadb4ed1a72b7c

[Henri Doreau (Inactive)]

Racer patch: http://review.whamcloud.com/#/c/13669

[Gerrit Updater]

Henri Doreau ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13669
Subject: LU-4840 tests: Add file migration to racer
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 739d73d58b2877711893e2df0b5f487e6b141060

[Oleg Drokin]

So in order to move things forward, and based on my understanding that this patch actually helps some user-ereported problems (and also on the assumption thatthe racer problems are now possibly a separate bug)- let's split the patch into two parts: the actual fix and the racer test patch.

We can then land the code fix (if it otherwise does not introduce any more failures) and we can wait with the racer test patch until we better understand why it fails and fix those. I am a bit nervous about landing tests that tend to fail as that invalidates our testing strategy quite a bit though.

[Jinshan Xiong (Inactive)]

Henri, that's great. Please let me know if you need any help from me.

[Henri Doreau (Inactive)]

Hello,

same here, I have not been able to reproduce the issues aforementioned... I can try investigating from crash dumps if Oleg has any that he can share, though it's indeed harder than with a reproducer.

This patch introduces no regression I'm aware of, and it fixes the original problem. If its size/complexity makes it "unlandable" I can try to split it into two subpatches (one fixing the blocking mode, another one adding the non-blocking mode). I was actually not expecting it to grow that much

Edit: I have just triggered deadlocks with racer, as descibed by Oleg, with all rpms installed, not simply running from the source tree. There was concurrent activity going on, but I'll try to reproduce in a non-disturbed environment.

[Jinshan Xiong (Inactive)]

1. I can confirm that the original problem is fixed by this patch;
2. I tried to reproduce the problem mentioned by Oleg, i.e., running racer, for over 10 hours but I couldn't reproduce the problem. I'm not saying that this issue doesn't exist but is hard to reproduce.