[LU-10509] sanity-dom test_fsx: FAIL: test_fsx failed with 110

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.11.0, Lustre 2.12.0
• Fix Version/s: Lustre 2.12.0
• Type: Bug
• Priority: Critical
• Reporter: Jian Yu
• Assignee: Mikhail Pershin
• Resolution: Fixed
• Labels: zfs

Environment

FSTYPE=zfs

• Severity: 3

Description

sanity-dom test fsx failed as follows in ZFS test sessions:

000069[0] 1515877521.496798 read       0x098d90 thru 0x0a0330	(0x75a1 bytes)
       1515877521.499774 read done
READ BAD DATA: offset = 0x98d90, size = 0x75a1
OFFSET	GOOD	BAD	RANGE
0x9902f	000000	0x9b0c	 0x72c3
operation# (mod 256) for the bad datamay be 12
LOG DUMP (71 total operations):
1[0]: 1515877520.983988 TRUNCATE UP	from 0x0 to 0x67f16
2[1]: 1515877520.986934 MAPWRITE 0x1c3ee0 thru 0x1c857f (0x46a0 bytes)
3[1]: 1515877520.999046 WRITE    0x1d29a7 thru 0x1d30c7 (0x721 bytes) HOLE
4[1]: 1515877521.001075 READ     0x7aec0 thru 0x88bd3 (0xdd14 bytes)
5[0]: 1515877521.007460 WRITE    0x9ae5 thru 0x19ad8 (0xfff4 bytes)
6[0]: 1515877521.012003 WRITE    0x1128a4 thru 0x117b0e (0x526b bytes)
7[0]: 1515877521.018228 SKIPPED (no operation)
8[0]: 1515877521.018334 MAPREAD  0x189684 thru 0x18ab5b (0x14d8 bytes)
9[0]: 1515877521.019975 WRITE    0x142af6 thru 0x14c614 (0x9b1f bytes)
10[1]: 1515877521.021589 READ     0x83f12 thru 0x8b809 (0x78f8 bytes)
11[1]: 1515877521.046885 MAPWRITE 0xb2a3f thru 0xbd3c7 (0xa989 bytes)
12[1]: 1515877521.075449 MAPWRITE 0x9902f thru 0xa189c (0x886e bytes)	******WWWW
13[1]: 1515877521.084757 MAPWRITE 0x1b6bb7 thru 0x1b8413 (0x185d bytes)
14[1]: 1515877521.092561 MAPWRITE 0x1413ee thru 0x149b9e (0x87b1 bytes)
15[0]: 1515877521.119883 WRITE    0x1c882d thru 0x1cbf20 (0x36f4 bytes)
16[1]: 1515877521.126775 MAPWRITE 0x4de7e thru 0x4fe37 (0x1fba bytes)
17[0]: 1515877521.131847 MAPREAD  0xc420d thru 0xca339 (0x612d bytes)
18[0]: 1515877521.151499 TRUNCATE DOWN	from 0x1d30c8 to 0x1f2c3	******WWWW
19[1]: 1515877521.156735 MAPREAD  0x8391 thru 0x17b73 (0xf7e3 bytes)
20[0]: 1515877521.165539 MAPREAD  0x1e5f0 thru 0x1e78d (0x19e bytes)
21[1]: 1515877521.167973 MAPREAD  0x13dcf thru 0x15138 (0x136a bytes)
22[0]: 1515877521.168883 WRITE    0x109324 thru 0x10c5ca (0x32a7 bytes) HOLE	***WWWW
23[0]: 1515877521.170715 READ     0x48590 thru 0x51c91 (0x9702 bytes)
24[0]: 1515877521.171655 WRITE    0x594f2 thru 0x68780 (0xf28f bytes)
25[1]: 1515877521.180898 MAPREAD  0x10a656 thru 0x10c5ca (0x1f75 bytes)
26[0]: 1515877521.187353 TRUNCATE DOWN	from 0x10c5cb to 0x5e5ea	******WWWW
27[0]: 1515877521.192553 MAPREAD  0x2d83b thru 0x3b32c (0xdaf2 bytes)
28[1]: 1515877521.202051 WRITE    0x11e3b3 thru 0x1205c2 (0x2210 bytes) HOLE	***WWWW
29[1]: 1515877521.204904 WRITE    0x2ccce thru 0x33cbf (0x6ff2 bytes)
30[1]: 1515877521.210134 READ     0x6c52c thru 0x727f3 (0x62c8 bytes)
31[1]: 1515877521.211592 READ     0x6f8aa thru 0x730ba (0x3811 bytes)
32[0]: 1515877521.213285 MAPREAD  0x92bf thru 0x16bff (0xd941 bytes)
33[1]: 1515877521.233063 WRITE    0x6cdb2 thru 0x70fb7 (0x4206 bytes)
34[1]: 1515877521.237368 TRUNCATE DOWN	from 0x1205c3 to 0x65303	******WWWW
35[0]: 1515877521.239654 MAPREAD  0x55060 thru 0x6074f (0xb6f0 bytes)
36[0]: 1515877521.247115 MAPREAD  0x20b5f thru 0x255fa (0x4a9c bytes)
37[0]: 1515877521.260646 READ     0x255c0 thru 0x2a93d (0x537e bytes)
38[1]: 1515877521.262403 WRITE    0x1a94e1 thru 0x1b6f62 (0xda82 bytes) HOLE	***WWWW
39[1]: 1515877521.263005 TRUNCATE DOWN	from 0x1b6f63 to 0x72c13	******WWWW
40[1]: 1515877521.268324 MAPREAD  0x1e6e3 thru 0x1ff6c (0x188a bytes)
41[1]: 1515877521.270353 READ     0x12f89 thru 0x1b28e (0x8306 bytes)
42[0]: 1515877521.285472 MAPREAD  0x6cae5 thru 0x72c12 (0x612e bytes)
43[0]: 1515877521.292353 READ     0x3f0d4 thru 0x44716 (0x5643 bytes)
44[1]: 1515877521.300557 MAPWRITE 0x16f3fe thru 0x17cc34 (0xd837 bytes)
45[0]: 1515877521.311396 MAPWRITE 0x1e0a6 thru 0x1e34b (0x2a6 bytes)
46[1]: 1515877521.316861 CLOSE/OPEN
47[0]: 1515877521.318235 WRITE    0x9c826 thru 0x9d9d0 (0x11ab bytes)
48[0]: 1515877521.321503 TRUNCATE DOWN	from 0x17cc35 to 0x1355d3
49[0]: 1515877521.331467 WRITE    0x1ac4b5 thru 0x1adf12 (0x1a5e bytes) HOLE
50[0]: 1515877521.332478 READ     0x19c59e thru 0x1a2889 (0x62ec bytes)
51[1]: 1515877521.333859 MAPREAD  0x185357 thru 0x194012 (0xecbc bytes)
52[0]: 1515877521.347262 TRUNCATE DOWN	from 0x1adf13 to 0x1274fb
53[1]: 1515877521.351347 WRITE    0x1a8fd6 thru 0x1aea03 (0x5a2e bytes) HOLE
54[0]: 1515877521.354876 MAPWRITE 0x1b0381 thru 0x1b932c (0x8fac bytes)
55[1]: 1515877521.364888 MAPREAD  0x13c15f thru 0x147d5f (0xbc01 bytes)
56[0]: 1515877521.392780 WRITE    0x1b9901 thru 0x1bd759 (0x3e59 bytes) HOLE
57[0]: 1515877521.395476 TRUNCATE DOWN	from 0x1bd75a to 0x7215a	******WWWW
58[0]: 1515877521.399666 CLOSE/OPEN
59[0]: 1515877521.400981 MAPWRITE 0x1f436e thru 0x1fe841 (0xa4d4 bytes)
60[0]: 1515877521.410748 READ     0x1f67dd thru 0x1fa3bb (0x3bdf bytes)
61[1]: 1515877521.412885 MAPREAD  0x3cb thru 0x9184 (0x8dba bytes)
62[1]: 1515877521.453479 MAPREAD  0x64823 thru 0x6a34c (0x5b2a bytes)
63[1]: 1515877521.456049 TRUNCATE DOWN	from 0x1fe842 to 0x67852	******WWWW
64[1]: 1515877521.464637 TRUNCATE UP	from 0x67852 to 0x1353b7	******WWWW
65[1]: 1515877521.467812 TRUNCATE DOWN	from 0x1353b7 to 0x12410c
66[0]: 1515877521.471279 WRITE    0x86ae0 thru 0x8bad5 (0x4ff6 bytes)
67[1]: 1515877521.475306 MAPWRITE 0x11e8ba thru 0x11ed54 (0x49b bytes)
68[0]: 1515877521.478767 WRITE    0xf5575 thru 0xf85ee (0x307a bytes)
69[0]: 1515877521.481865 MAPREAD  0x113d5c thru 0x119413 (0x56b8 bytes)
70[0]: 1515877521.489381 MAPWRITE 0xe0bc1 thru 0xeeb1b (0xdf5b bytes)
71[0]: 1515877521.496798 READ     0x98d90 thru 0xa0330 (0x75a1 bytes)	***RRRR***
Correct content saved for comparison
(maybe hexdump "/mnt/lustre/ffsx.sanity-dom" vs "/mnt/lustre/ffsx.sanity-dom.fsxgood")
 sanity-dom test_fsx: @@@@@@ FAIL: test_fsx failed with 110 

Maloo reports:
https://testing.hpdd.intel.com/test_sets/dfd214d2-f8a7-11e7-a6ad-52540065bddc
https://testing.hpdd.intel.com/test_sets/6d5895e2-f8a8-11e7-a7cd-52540065bddc

Issue Links

• is related to: LU-11033 truncate to DoM-file results in two RPCs

Comments

[Jian Yu]

Hi Mike,
I found this failure occurred consistently in ZFS test sessions, which blocks sanity-dom being added into autotest review-zfs-part-1 test group for regression testing. Could you please look at this failure?

https://testing.hpdd.intel.com/test_sets/f5a8be24-1034-11e8-bd00-52540065bddc
https://testing.hpdd.intel.com/test_sets/3e52e906-1035-11e8-a6ad-52540065bddc

[Wang Shilong (Inactive)]

https://testing.whamcloud.com/test_sets/a8b1b3ca-9953-11e8-b0aa-52540065bddc

[James Nunez (Inactive)]

This test is failing 100% of the time in review-dne-zfs-part-4 which is currently run as an optional test session and not required for patches to land. We do want to make this a required test session in the future.

Some recent failures are at
https://testing.whamcloud.com/test_sets/22a096d8-9be3-11e8-8ee3-52540065bddc
https://testing.whamcloud.com/test_sets/e323eb32-9baf-11e8-87f3-52540065bddc

[Peter Jones]

Mike

Please can you investigate

Peter

[Mikhail Pershin]

I have found the reason of this, working on patch

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33008
Subject: LU-10509 mdd: don't set size attr for DOM file
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 7995b14a2c862bc192dcce3047553d7467be34a2

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33008/
Subject: LU-10509 mdd: don't set size attr for DOM file
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: f49579786d41d9d8933e50948a6cac342a6f87a6

[Mikhail Pershin]

Patch was landed