Details
-
Bug
-
Resolution: Fixed
-
Minor
-
Lustre 2.10.0
-
None
-
Servers - 2.9.58, 3591, master
Client - 2.9.0, build 22, b2_9
-
3
-
9223372036854775807
Description
This issue was created by maloo for Saurabh Tandan <saurabh.tandan@intel.com>
This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/f3de6c42-4af5-11e7-b400-5254006e85c2.
The sub-test test_103a failed with the following error:
LU-2561 test failed
test_logs:
11 commands (11 passed, 0 failed) LU-2561 newly created file is same size as directory... [3] $ mkdir -p 2561 -- ok [4] $ cd 2561 -- ok [5] $ getfacl --access . | setfacl -d -M- . -- ok [6] $ touch f1 -- ok [7] $ ls -s f1 -- failed 0 f1 ? 4 f1 [9] $ cd .. -- ok [10] $ rm -rf 2561 -- ok 7 commands (6 passed, 1 failed) sanity test_103a: @@@@@@ FAIL: LU-2561 test failed
This issue is seen only during rolling upgrade/downgrade testing.
Attachments
Issue Links
![[Whamcloud Community Wiki] Page](/images/icons/generic_link_16.png "[Whamcloud Community Wiki] Page"){kind=icon}
Activity
Minh Diep (minh.diep@intel.com) uploaded a new patch: https://review.whamcloud.com/29816
Subject: LU-9663 tests: replace “ls -s” with “stat -c '%s'” to get file size
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 96cc82bbd18b0f5daa847a5a4027db6ea9d6855b
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/27961/
Subject: LU-9663 tests: replace “ls -s” with “stat -c '%s'” to get file size
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 45b8c6cf6916cf012ad0512f9cf249c152b45236
Jian Yu (jian.yu@intel.com) uploaded a new patch: https://review.whamcloud.com/27961
Subject: LU-9663 tests: replace “ls -s” with “stat -c '%s'” to get file size
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 23131216b6407114445456b0b46a4b5e2a88557d
I think the test here should be improved. Something like changing the "ls -s <file>" check of the allocated block count to "stat -c '%s' <file>" to check the file size would also detect this problem, and would not suffer from the change in the allocated block count.
The same issue also can be reproduced with clean upgrading (even only upgrading OSS from Lustre 2.9.0 to 2.9.59). The result of "ls -s" on a newly touched file is 4 instead of 0.
It turns out the above "ls -s" issue also exists on the latest master branch. I filed a new ticket LU-9708 to track that.
For the upgrading issue in this ticket, I did more experiments under the configuration of Lustre 2.9.0 client with rolling upgraded 2.9.59 servers and found:
On 2.9.0 client:
# pwd /mnt/lustre # ls -s total 0 # touch f # ls -s f 4 f # stat f File: ‘f’ Size: 0 Blocks: 8 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502275 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2017-06-23 19:39:41.000000000 +0000 Modify: 2017-06-23 19:39:41.000000000 +0000 Change: 2017-06-23 19:39:41.000000000 +0000 Birth: -
The allocated size of the touched file "f" is 4 blocks instead of 0. And the "Blocks" showed by "stat f" is 8 instead of 0.
On 2.9.59 MDS server:
# umount /dev/vda3 # mount -t ldiskfs /dev/vda3 /mnt/lustre-mds1 # cd /mnt/lustre-mds1 # stat ROOT/f File: ‘ROOT/f’ Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: fd03h/64771d Inode: 134 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2017-06-23 19:39:41.000000000 +0000 Modify: 2017-06-23 19:39:41.000000000 +0000 Change: 2017-06-23 19:39:41.000000000 +0000 Birth: -
Both "Size" and "Blocks" are 0.
After unmouting and remounting all of the 2.9.0 clients, "ls -s f" is still 4 instead of 0. Touching a new file also hit the same issue.
As Saurabh reported, sanity test 103a failed under the configuration of Lustre 2.9.0 client with rolling upgraded 2.9.59 servers but passed under the configuration of Lustre 2.9.0 client with freshly installed 2.9.59 servers.
I did some experiments under the configuration of Lustre 2.9.0 client with freshly installed 2.9.59 servers and found the failure occurred in another way:
# pwd
/mnt/lustre
# ls -s
total 0
# mkdir -p 2561
# cd 2561
# ls -s
total 0
# touch f1
# ls -s f1
0 f1
# echo a > f1
# ls -s f1
1 f1
# ls -s f1
4 f1
As we can see, the results of the latest two runs of "ls -s f1" were different. So, without upgrading, the issue also exists under interop configuration. Continue investigating.
Another instance for tag 2.9.59 with 1 MDT, 2 OSTs and 2 Clients for rolling upgrade testing
https://testing.hpdd.intel.com/test_sets/05e12a22-56d3-11e7-8a1b-5254006e85c2
John L. Hammond (john.hammond@intel.com) merged in patch https://review.whamcloud.com/29816/
Subject:
LU-9663tests: replace “ls -s” with “stat -c '%s'” to get file sizeProject: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: f80d37cea5f5c4ad971afa1eac47dda6859c6a2a