[LU-7759] umount hanging in modern distros when OST is unavailable

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.9.0
• Affects Version/s: Lustre 2.8.0, Lustre 2.9.0
• Labels: easy sles12
• Severity: 3
• Assignee: Yang Sheng
• Reporter: Oleg Drokin

Description

It looks like modern distros: SLES12, Fedora 21+ (possibly earlier) always do statfs on the mountpoint before issuing the umount call.

In case one of the OSTs is not available at that time - the statfs will hang.

In our tests there's a bunch of tests in conf-sanity that have this problem and it was "dealt with" in LU-5472 by just adding the -f umount option.
But this still leaves palces like recovery-single test 89 and something in recovery-small at the very least.

And most of all regular users would be affected too. So I wonder if we should deal with this issue somewhat more intelligently so that the unmount does not really hang in such a case?

Issue Links

• is related to: LU-4039 Failure on test suite replay-single test_90: wrong stripe: f0, uuid: lustre-OST0000_UUID
• is related to: LU-8544 recovery-double-scale test_pairwise_fail: start client on trevis-54vm5 failed
• is related to: LU-8731 lfs df exits with status 0 on failures
• is related to: LU-6233 recovery-small test_10d failed with 'file contents differ'
• is related to: LU-8069 Allow remount to include "flock" and "localflock", as well as "lazystatfs" and "nolazystatfs"
• is related to: LU-5472 conf-sanity test_32a: failed with 1

Comments

[Andreas Dilger]

Rather than open up an old ticket that hasn't been seen in 2+ years, it is better to open up a new ticket for the new problem, and if they seem related they can be linked together. Otherwise, tracking the old re-opened issue fix version is more difficult than necessary.

[Jinshan Xiong (Inactive)]

There is a new occurrence on recent test: https://testing.hpdd.intel.com/test_sets/0d7d92de-cad2-11e7-9840-52540065bddc

The test failure looks the same but not sure if they are the same.

[Yang Sheng]

Patch landed. Close ticket.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/19195/
Subject: LU-7759 llite: handle inactive OSTs better in statfs
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 23fde1f89bec0adf4f7181ccce5a236eac371a38

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/18820/
Subject: LU-7759 utils: build mount.lustre with libmount
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: f1de339d881958de8fc47065fb31a5c8e0c14b60

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: http://review.whamcloud.com/19195
Subject: LU-7759 llite: handle inactive OSTs better in statfs
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 169a33aaf665fdaef6fc5734665a04d758a443e9

[Gerrit Updater]

Yang Sheng ([e-mail]) uploaded a new patch: http://review.whamcloud.com/18820
Subject: LU-7759 utils: build mount.lustre with libmount
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ffebbb3e30c247e453480f9219985effdf035b05

[Yang Sheng]

I found a way to handle it. We can add a entry to utab(default located at /run/mount/utab) when mount a lustre filesystem. Then umount will use this entry to find out fstype to avoid invoke statfs. What we need to do just change mount_lustre.c is enough. I'll produce a patch.

[Andreas Dilger]

I guess it is possible to check strstr("unmount", current->comm) and set sbi->ll_statfs |= LL_SBI_LAZYSTATFS on the filesystem, but this would add overhead to every statfs() call.

It might be possible to set LL_SBI_LAZYSTATFS by default, but that may also cause problems with the recovery tests that wait on "df" to return to indicate recovery is complete.

[Yang Sheng]

Looks like umount.lustre is called after statfs. Since umount need figure out fstype before invoke umount.{fstype}. This is why it call statfs.The hard part is no way to tell kernel the statfs called from umount. So avoid invoke statfs is only thing we can do. Of course, skip unavailable OST maybe a reasonable solution?