[LU-13296] statfs isn't work properly with MDT statfs proxy - Whamcloud Community JIRA

Details

Type: Bug
Resolution: Fixed
Priority: Blocker
Fix Version/s: Lustre 2.14.0, Lustre 2.12.5
Affects Version/s: Lustre 2.12.0, Lustre 2.12.5
Labels:
None

Severity:
3
Rank (Obsolete):
9223372036854775807

Description

statfs don't work fine sometimes. It's easy replicate with start a several processes on same time which ask about block size.

all instances run on same node.

% aprun -n 8 ./a.out /lus/snx11242
statfs (/lus/snx11242) f_bsize 0X9003B000
statfs (/lus/snx11242) f_bsize 0X8C19F000
statfs (/lus/snx11242) f_bsize 0X82BFD000
statfs (/lus/snx11242) f_bsize 0X85C31000
statfs (/lus/snx11242) f_bsize 0X8BED2000
statfs (/lus/snx11242) f_bsize 0X835B1000
statfs (/lus/snx11242) f_bsize 0X828E3000
statfs (/lus/snx11242) f_bsize 0X1000

log for failed application.

00000080:00200000:0.0:1582645192.532388:0:16422:0:(llite_lib.c:1890:ll_statfs()) VFS Op: at 4296762105 jiffies
00000080:00000001:0.0:1582645192.532391:0:16422:0:(llite_lib.c:1838:ll_statfs_internal()) Process entered
00000080:00000001:0.0:1582645192.532392:0:16422:0:(obd_class.h:1051:obd_statfs()) Process entered
00000080:00000004:0.0:1582645192.532394:0:16422:0:(obd_class.h:1063:obd_statfs()) snx11242-clilmv-ffff881f8b680000: age 7262, max_age 7478
00000080:00000004:0.0:1582645192.535406:0:16422:0:(obd_class.h:1083:obd_statfs()) snx11242-clilmv-ffff881f8b680000: new ffff881f9cad8228 cache blocks 3366320026/3419988114 objects 630715697
0/6372092768
00000080:00000001:0.0:1582645192.535408:0:16422:0:(obd_class.h:1103:obd_statfs()) Process leaving (rc=0 : 0 : 0)
00000080:00000004:0.0:1582645192.535409:0:16422:0:(llite_lib.c:1848:ll_statfs_internal()) MDC blocks 18446744071580744875/18446744071582034486 objects 0/1
00000080:00000001:0.0:1582645192.535410:0:16422:0:(llite_lib.c:1851:ll_statfs_internal()) Process leaving via out (rc=0 : 0 : 0x0)
00000080:00000001:0.0:1582645192.535411:0:16422:0:(llite_lib.c:1881:ll_statfs_internal()) Process leaving (rc=0 : 0 : 0)

regression introduced by

commit b500d5193360711a6c6b07497f34e61cc590cf19
Author: Alex Zhuravlev <bzzz@whamcloud.com>
Date:   Thu Sep 21 18:24:18 2017 +0300

    LU-10018 protocol: MDT as a statfs proxy

Attachments

Issue Links

is duplicated by

LU-15361 lfs df output is incorrect

Resolved

is related to

LU-13285 multiple DF returns bad info

Resolved

LU-12368 concurrent statfs() calls on the client should be blocked

Resolved

Activity

[LU-13296] statfs isn't work properly with MDT statfs proxy

Gerrit Updater added a comment - 25/Mar/20 6:31 AM

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37819/
Subject: ~~LU-13296~~ obd: make statfs cache working again
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 6db1fd1cfc67c639c4817ae00cfe4198882d8e9c

Gerrit Updater added a comment - 25/Mar/20 6:31 AM Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37819/ Subject: LU-13296 obd: make statfs cache working again Project: fs/lustre-release Branch: b2_12 Current Patch Set: Commit: 6db1fd1cfc67c639c4817ae00cfe4198882d8e9c

Gerrit Updater added a comment - 06/Mar/20 1:05 AM

Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/37819
Subject: ~~LU-13296~~ obd: make statfs cache working again
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 785965b33c9993103a703034d5ee2a32067eeb8a

Gerrit Updater added a comment - 06/Mar/20 1:05 AM Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/37819 Subject: LU-13296 obd: make statfs cache working again Project: fs/lustre-release Branch: b2_12 Current Patch Set: 1 Commit: 785965b33c9993103a703034d5ee2a32067eeb8a

Peter Jones added a comment - 05/Mar/20 10:44 PM

Landed for 2.14

Peter Jones added a comment - 05/Mar/20 10:44 PM Landed for 2.14

Gerrit Updater added a comment - 05/Mar/20 10:35 PM

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37753/
Subject: ~~LU-13296~~ obd: make statfs cache working again
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 7281635521a823548d497bce2f19acfa3318dfe9

Gerrit Updater added a comment - 05/Mar/20 10:35 PM Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37753/ Subject: LU-13296 obd: make statfs cache working again Project: fs/lustre-release Branch: master Current Patch Set: Commit: 7281635521a823548d497bce2f19acfa3318dfe9

Gerrit Updater added a comment - 27/Feb/20 2:54 PM

Alexey Lyashkov (alexey.lyashkov@hpe.com) uploaded a new patch: https://review.whamcloud.com/37753
Subject: ~~LU-13296~~ obd: make statfs cache working again
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 3773a5c542a9d2f5c1c2ba868e5ce8832700e8a5

Gerrit Updater added a comment - 27/Feb/20 2:54 PM Alexey Lyashkov (alexey.lyashkov@hpe.com) uploaded a new patch: https://review.whamcloud.com/37753 Subject: LU-13296 obd: make statfs cache working again Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 3773a5c542a9d2f5c1c2ba868e5ce8832700e8a5

Alexey Lyashkov added a comment - 26/Feb/20 6:14 PM

Root cause of this bug - cached copy ins't used in case of race between obtain statfs info and refreshing it.
I will submit fix in near time

Alexey Lyashkov added a comment - 26/Feb/20 6:14 PM Root cause of this bug - cached copy ins't used in case of race between obtain statfs info and refreshing it. I will submit fix in near time

Alexey Lyashkov added a comment - 26/Feb/20 6:13 PM

issue replicated easy with several statfs in parallel.

# cat test
./a.out &
./a.out &
./a.out &
./a.out &
./a.out &
./a.out &
./a.out &
./a.out &

sleep 1

# cat ttt.c
#include <stdio.h>
#include  <sys/statfs.h>

int main() {
    struct statfs buf;

    statfs("/mnt/lustre", &buf);
    printf("b_size %llx\n", buf.f_bsize);
    return 0;
}

# bash test
b_size 1000
b_size 1000
b_size 0
b_size 0
b_size 0
b_size 0
b_size 0
b_size 0

Alexey Lyashkov added a comment - 26/Feb/20 6:13 PM issue replicated easy with several statfs in parallel. # cat test ./a.out & ./a.out & ./a.out & ./a.out & ./a.out & ./a.out & ./a.out & ./a.out & sleep 1 # cat ttt.c #include <stdio.h> #include <sys/statfs.h> int main() { struct statfs buf; statfs( "/mnt/lustre" , &buf); printf( "b_size %llx\n" , buf.f_bsize); return 0; } # bash test b_size 1000 b_size 1000 b_size 0 b_size 0 b_size 0 b_size 0 b_size 0 b_size 0

People

Assignee:: Alexey Lyashkov

Reporter:: Alexey Lyashkov

Votes:: 0 Vote for this issue

Watchers:: 9 Start watching this issue

Dates

Created:: 26/Feb/20 12:40 PM

Updated:: 11/Dec/21 4:54 AM

Resolved:: 05/Mar/20 10:44 PM