[LU-5490] sanity test_133d: FAIL: samedir_rename_size error

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.4
• Affects Version/s: Lustre 2.7.0, Lustre 2.5.3, Lustre 2.9.0
• Labels: zfs
• Environment: Lustre build: https://build.hpdd.intel.com/job/lustre-master/2615/; Distro/Arch: RHEL6.5/x86_64; FSTYPE=zfs
• Severity: 3
• Assignee: Nathaniel Clark
• Reporter: Jian Yu

Description

sanity test 133d failed as follows:

== sanity test 133d: Verifying rename_stats ========================================== 01:32:10 (1408005130)
CMD: onyx-47vm7 /usr/sbin/lctl list_param mdt.*.rename_stats
mdt.lustre-MDT0000.rename_stats
CMD: onyx-47vm7 /usr/sbin/lctl set_param mdt.*.rename_stats=clear
mdt.lustre-MDT0000.rename_stats=clear
total: 512 creates in 1.52 seconds: 336.70 creates/second
source rename dir size: 64K
target rename dir size: 8K
CMD: onyx-47vm7 /usr/sbin/lctl get_param mdt.*.rename_stats
mdt.lustre-MDT0000.rename_stats=
rename_stats:
- snapshot_time:  1408005133.29235
- same_dir       
      512bytes: { sample:   1, pct: 100, cum_pct: 100 }
CMD: onyx-47vm7 /usr/sbin/lctl get_param mdt.*.rename_stats
CMD: onyx-47vm7 /usr/sbin/lctl get_param mdt.*.rename_stats
 sanity test_133d: @@@@@@ FAIL: samedir_rename_size error

Maloo report: https://testing.hpdd.intel.com/test_sets/4b98ba1c-23f5-11e4-b2ba-5254006e85c2

Issue Links

• is related to: LU-4203 Test failure sanity test_133d: crossdir_rename_size error
• is related to: LU-8066 Move lustre procfs handling to sysfs and debugfs.

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/31585/
Subject: LU-5490 tests: Sanity/133d ensure stats read is on correct MDT
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1d80fd72bf42f653a5d6a4a31fc2c5df571f1afc

[Gerrit Updater]

Nathaniel Clark ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31585
Subject: LU-5490 tests: Sanity/133d ensure stats read is on correct MDT
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: fa38f33d74bcd91ecf8768dde3dece69d629c099

[Nathaniel Clark]

This test can fail for DNE as test_133d only checks 1 MDS.

[Gerrit Updater]

Nathaniel Clark ([e-mail]) uploaded a new patch: https://review.whamcloud.com/29053
Subject: LU-5490 tests: FORTESTONLY
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 821baea4626beb70dbca822f3109d58180db3fd2

[Patrick Farrell (Inactive)]

+1 on master:
https://testing.hpdd.intel.com/test_sets/1d65eb6a-975b-11e7-b761-5254006e85c2

[Bob Glossman (Inactive)]

another on master:
https://testing.hpdd.intel.com/test_sets/233fe796-0749-11e6-b5f1-5254006e85c2

[Nathaniel Clark]

This is still happening on master:
https://testing.hpdd.intel.com/test_sets/242be33c-e0d2-11e5-97b5-5254006e85c2
https://testing.hpdd.intel.com/test_sets/75019ae8-dfee-11e5-9400-5254006e85c2
https://testing.hpdd.intel.com/test_sets/29b44c56-df1d-11e5-8471-5254006e85c2

[Nathaniel Clark]

"cause" of error seems to be size of directory reported by ls/stat is one thing (i.e. "64K") and size in mdt.lustre-MDT0000.rename_stats is another (i.e. "512bytes"). Not sure why the discrepancy yet, but this is why the test is failing.

[Andreas Dilger]

Nathaniel, could you take a crack at this if you have time. Not critical, but a source of ongoing annoyance for patch landings.

[James Nunez (Inactive)]

More failures on master at:
2015-07-10 03:43:38 - https://testing.hpdd.intel.com/test_sets/840b7844-26ec-11e5-b3d7-5254006e85c2
2015-07-22 20:01:38 - https://testing.hpdd.intel.com/test_sets/c4fdb934-30f1-11e5-a788-5254006e85c2
2015-08-24 15:36:45 - https://testing.hpdd.intel.com/test_sets/fb01ec64-4aad-11e5-88e8-5254006e85c2
2016-01-04 01:56:27 - https://testing.hpdd.intel.com/test_sets/fafafd4a-b2b4-11e5-8114-5254006e85c2
2016-02-02 00:40:04 - https://testing.hpdd.intel.com/test_sets/95eca8d4-c968-11e5-9e6a-5254006e85c2
2016-02-04 07:09:54 - https://testing.hpdd.intel.com/test_sets/57eaf714-cb3b-11e5-b49e-5254006e85c2
2016-02-04 09:47:15 - https://testing.hpdd.intel.com/test_sets/4b7d3746-cb4d-11e5-be8d-5254006e85c2

[Andreas Dilger]

Between this failure and LU-4203 there are 11 failures in the past month. I suspect there is some inconsistency with how osd-zfs is recording the directory size in the rename stats (which I think is the number of entries) vs. how the test or /proc is checking them (which is the number of bytes). I don't think we need to change osd-zfs to record the stats differently, but rather fix the test so that it handles this consistently. It isn't clear why the test only fails intermittently.