[LU-4840] Deadlock when truncating file during lfs migrate

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.4.2
• Labels: cea
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Patrick Valentin (Inactive)

Description

While migrating a file with "lfs migrate", if a process tries to truncate the file, both lfs migrate and truncating processes will deadlock.

This will result in both processes never finishing (unless it is killed) and watchdog messages saying that the processes did not progress for the last XXX seconds.

Here is a reproducer:

[root@lustre24cli ~]# cat reproducer.sh
#!/bin/sh

FS=/test
FILE=${FS}/file

rm -f ${FILE}
# Create a file on OST 1 of size 512M
lfs setstripe -o 1 -c 1 ${FILE}
dd if=/dev/zero of=${FILE} bs=1M count=512

echo 3 > /proc/sys/vm/drop_caches

# Launch a migrate to OST 0 and a bit later open it for write
lfs migrate -i 0 --block ${FILE} &
sleep 2
dd if=/dev/zero of=${FILE} bs=1M count=512 

Once the last dd tries to open the file, both lfs and dd processes stay forever with this stack:

lfs stack:

[<ffffffff8128e864>] call_rwsem_down_read_failed+0x14/0x30
[<ffffffffa08d98dd>] ll_file_io_generic+0x29d/0x600 [lustre]
[<ffffffffa08d9d7f>] ll_file_aio_read+0x13f/0x2c0 [lustre]
[<ffffffffa08da61c>] ll_file_read+0x16c/0x2a0 [lustre]
[<ffffffff811896b5>] vfs_read+0xb5/0x1a0
[<ffffffff811897f1>] sys_read+0x51/0x90
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

dd stack:

[<ffffffffa03436fe>] cfs_waitq_wait+0xe/0x10 [libcfs]
[<ffffffffa04779fa>] cl_lock_state_wait+0x1aa/0x320 [obdclass]
[<ffffffffa04781eb>] cl_enqueue_locked+0x15b/0x1f0 [obdclass]
[<ffffffffa0478d6e>] cl_lock_request+0x7e/0x270 [obdclass]
[<ffffffffa047e00c>] cl_io_lock+0x3cc/0x560 [obdclass]
[<ffffffffa047e242>] cl_io_loop+0xa2/0x1b0 [obdclass]
[<ffffffffa092a8c8>] cl_setattr_ost+0x208/0x2c0 [lustre]
[<ffffffffa08f8a0e>] ll_setattr_raw+0x9ce/0x1000 [lustre]
[<ffffffffa08f909b>] ll_setattr+0x5b/0xf0 [lustre]
[<ffffffff811a7348>] notify_change+0x168/0x340
[<ffffffff81187074>] do_truncate+0x64/0xa0
[<ffffffff8119bcc1>] do_filp_open+0x861/0xd20
[<ffffffff81185d39>] do_sys_open+0x69/0x140
[<ffffffff81185e50>] sys_open+0x20/0x30
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

Issue Links

• is related to: LU-6785 Interop 2.7.0<->master sanity test_56w: cannot swap layouts: Device or resource busy
• is related to: LU-5915 racer test_1: FAIL: test_1 failed with 4
• is related to: LU-7073 racer with OST object migration hangs on cleanup
• is related to: LU-6903 racer file migration crash ASSERTION( lov->lo_type == LLT_RAID0 )

Comments

[Henri Doreau (Inactive)]

Racer patch: http://review.whamcloud.com/#/c/13669

[Gerrit Updater]

Henri Doreau ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13669
Subject: LU-4840 tests: Add file migration to racer
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 739d73d58b2877711893e2df0b5f487e6b141060

[Oleg Drokin]

So in order to move things forward, and based on my understanding that this patch actually helps some user-ereported problems (and also on the assumption thatthe racer problems are now possibly a separate bug)- let's split the patch into two parts: the actual fix and the racer test patch.

We can then land the code fix (if it otherwise does not introduce any more failures) and we can wait with the racer test patch until we better understand why it fails and fix those. I am a bit nervous about landing tests that tend to fail as that invalidates our testing strategy quite a bit though.

[Jinshan Xiong (Inactive)]

Henri, that's great. Please let me know if you need any help from me.

[Henri Doreau (Inactive)]

Hello,

same here, I have not been able to reproduce the issues aforementioned... I can try investigating from crash dumps if Oleg has any that he can share, though it's indeed harder than with a reproducer.

This patch introduces no regression I'm aware of, and it fixes the original problem. If its size/complexity makes it "unlandable" I can try to split it into two subpatches (one fixing the blocking mode, another one adding the non-blocking mode). I was actually not expecting it to grow that much

Edit: I have just triggered deadlocks with racer, as descibed by Oleg, with all rpms installed, not simply running from the source tree. There was concurrent activity going on, but I'll try to reproduce in a non-disturbed environment.

[Jinshan Xiong (Inactive)]

1. I can confirm that the original problem is fixed by this patch;
2. I tried to reproduce the problem mentioned by Oleg, i.e., running racer, for over 10 hours but I couldn't reproduce the problem. I'm not saying that this issue doesn't exist but is hard to reproduce.

[Andreas Dilger]

Frank, Henri, Jinshan,
according to Oleg's last comments, he was still able to hit this deadlock even when the patch was applied, which raises a concern whether the risk of landing this complex patch is worth the risk at this late stage.

Could you please confirm that the current patch has resolved the deadlock in your testing? It may be that Oleg is hitting a second issue that is not directly related.

The second question is whether you are currently running with this patch in your other testing and can confirm that it doesn't introduce other problems?

[Jinshan Xiong (Inactive)]

I couldn't reproduce the deadlock problem on MDT. Please collect a core dump when you see the deadlock issue again.

[Oleg Drokin]

for the record: using Jinshan's patch did not help all that much and I was still seeing deadlocks on mdt

[Frank Zago (Inactive)]

Patch that adds some tests for the new API: http://review.whamcloud.com/13441/
It has a couple questions left (see BUG?? in source) but otherwise is complete.
It has to be applied on top on Henri's patch.

[Jinshan Xiong (Inactive)]

please apply http://review.whamcloud.com/13344 to your tree. It worked well after that patch was applied in my test.