[LU-9663] sanity test_103a: LU-2561 test failed

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.10.0
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.2
• Type: Bug
• Priority: Minor
• Reporter: Maloo
• Assignee: Jian Yu
• Resolution: Fixed

Environment

Servers - 2.9.58, 3591, master
Client - 2.9.0, build 22, b2_9

• Severity: 3

Description

This issue was created by maloo for Saurabh Tandan <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/f3de6c42-4af5-11e7-b400-5254006e85c2.

The sub-test test_103a failed with the following error:

LU-2561 test failed

test_logs:

11 commands (11 passed, 0 failed)
LU-2561 newly created file is same size as directory...
[3] $ mkdir -p 2561 -- ok
[4] $ cd 2561 -- ok
[5] $ getfacl --access . | setfacl -d -M- . -- ok
[6] $ touch f1 -- ok
[7] $ ls -s f1 -- failed
0 f1                                  ? 4 f1                                   
[9] $ cd .. -- ok
[10] $ rm -rf 2561 -- ok
7 commands (6 passed, 1 failed)
 sanity test_103a: @@@@@@ FAIL: LU-2561 test failed 

This issue is seen only during rolling upgrade/downgrade testing.

Issue Links

• is related to: LU-2561 Newly created file is same size as di...

Comments

[Peter Jones]

Jian

Can you please investigate?

Thanks

Peter

[Saurabh Tandan (Inactive)]

Another instance for tag 2.9.59 with 1 MDT, 2 OSTs and 2 Clients for rolling upgrade testing
https://testing.hpdd.intel.com/test_sets/05e12a22-56d3-11e7-8a1b-5254006e85c2

[Jian Yu]

As Saurabh reported, sanity test 103a failed under the configuration of Lustre 2.9.0 client with rolling upgraded 2.9.59 servers but passed under the configuration of Lustre 2.9.0 client with freshly installed 2.9.59 servers.
I did some experiments under the configuration of Lustre 2.9.0 client with freshly installed 2.9.59 servers and found the failure occurred in another way:

# pwd
/mnt/lustre
# ls -s
total 0
# mkdir -p 2561
# cd 2561
# ls -s
total 0
# touch f1
# ls -s f1
0 f1
# echo a > f1
# ls -s f1
1 f1
# ls -s f1
4 f1

As we can see, the results of the latest two runs of "ls -s f1" were different. So, without upgrading, the issue also exists under interop configuration. Continue investigating.

[Jian Yu]

It turns out the above "ls -s" issue also exists on the latest master branch. I filed a new ticket LU-9708 to track that.

For the upgrading issue in this ticket, I did more experiments under the configuration of Lustre 2.9.0 client with rolling upgraded 2.9.59 servers and found:
On 2.9.0 client:

# pwd
/mnt/lustre
# ls -s
total 0
# touch f
# ls -s f
4 f
# stat f
  File: ‘f’
  Size: 0               Blocks: 8          IO Block: 4194304 regular empty file
Device: 2c54f966h/743766374d    Inode: 144115205272502275  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-06-23 19:39:41.000000000 +0000
Modify: 2017-06-23 19:39:41.000000000 +0000
Change: 2017-06-23 19:39:41.000000000 +0000
 Birth: -

The allocated size of the touched file "f" is 4 blocks instead of 0. And the "Blocks" showed by "stat f" is 8 instead of 0.

On 2.9.59 MDS server:

# umount /dev/vda3
# mount -t ldiskfs /dev/vda3 /mnt/lustre-mds1
# cd /mnt/lustre-mds1
# stat ROOT/f 
  File: ‘ROOT/f’
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: fd03h/64771d    Inode: 134         Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-06-23 19:39:41.000000000 +0000
Modify: 2017-06-23 19:39:41.000000000 +0000
Change: 2017-06-23 19:39:41.000000000 +0000
 Birth: -

Both "Size" and "Blocks" are 0.

After unmouting and remounting all of the 2.9.0 clients, "ls -s f" is still 4 instead of 0. Touching a new file also hit the same issue.

[Jian Yu]

The same issue also can be reproduced with clean upgrading (even only upgrading OSS from Lustre 2.9.0 to 2.9.59). The result of "ls -s" on a newly touched file is 4 instead of 0.

[Andreas Dilger]

I think the test here should be improved. Something like changing the "ls -s <file>" check of the allocated block count to "stat -c '%s' <file>" to check the file size would also detect this problem, and would not suffer from the change in the allocated block count.

[Gerrit Updater]

Jian Yu ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27961
Subject: LU-9663 tests: replace “ls -s” with “stat -c '%s'” to get file size
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 23131216b6407114445456b0b46a4b5e2a88557d

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/27961/
Subject: LU-9663 tests: replace “ls -s” with “stat -c '%s'” to get file size
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 45b8c6cf6916cf012ad0512f9cf249c152b45236

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/29816
Subject: LU-9663 tests: replace “ls -s” with “stat -c '%s'” to get file size
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 96cc82bbd18b0f5daa847a5a4027db6ea9d6855b

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/29816/
Subject: LU-9663 tests: replace “ls -s” with “stat -c '%s'” to get file size
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: f80d37cea5f5c4ad971afa1eac47dda6859c6a2a