[LU-9096] sanity test_253: File creation failed after rm

• Type: Bug
• Resolution: Unresolved
• Priority: Critical
• Affects Version/s: Lustre 2.9.0, Lustre 2.10.0, Lustre 2.14.0
• Labels: always_except zfs
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for Andreas Dilger <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/b06b5a90-e8f0-11e6-935d-5254006e85c2.

The sub-test test_253 failed with the following error:

CMD: trevis-43vm3 /usr/sbin/lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_status
prealloc_status -28
dd: failed to open '/mnt/lustre/d253.sanity/2': No space left on device
5+0 records in
5+0 records out
5242880 bytes (5.2 MB) copied, 0.0534031 s, 98.2 MB/s
CMD: trevis-43vm3 /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1
CMD: trevis-43vm3 /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
:
:
CMD: trevis-43vm3 /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Delete is not completed in 28 seconds
Waiting for local destroys to complete
CMD: trevis-43vm3 lctl get_param -n lov.*.qos_maxage
 sanity test_253: @@@@@@ FAIL: File creation failed after rm

The MDS console log shows that there are no precreated objects for this OST:

20:44:10:[11571.982460] LustreError: 22503:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x20000234c:0x13b8:0x0]: have 0 want 1

Info required for matching: sanity 253

Issue Links

• is duplicated by: LU-9450 precreate logic badness between lod_statfs_and_check() and lod_check_and_reserve_ost()
• is duplicated by: LU-9454 sanity test_255a: f255a.sanity failed
• is related to: LU-9457 sanity test_253() must restore osp.*.reserved_mb_{high,low} on error
• is related to: LU-7754 DNE3: osd-zfs gets into a livelock if transaction is too big
• is related to: LU-15553 replay-vbr test 12a fails with 'test_12a failed with 4'
• is related to: LU-10070 PFL self-extending file layout

Comments

[Andreas Dilger]

Patch https://review.whamcloud.com/51861 "LU-15553 test: mkdir_on_mdt0 in sanity" is adding this subtest to the always_except list again.

[Sergey Cheremencev]

+1 on master https://testing.whamcloud.com/test_sessions/c590176e-852a-4080-abb9-e601015fbed9

[Andreas Dilger]

It looks like this test was re-enabled by patch https://review.whamcloud.com/33778 "LU-10070 tests: New test-framework functionality". Since it currently is not failing very offer, I'm not in a hurry to turn it off again, especially if it is adding coverage for SEL.

[Andreas Dilger]

For some reason this test is now running again and failing intermittently (about 4x per month).

[Peter Jones]

The test is no longer being run so I will remove the fixversion from this ticket and it can be reintroduced when the test has been given a clean bill of health

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/27013/
Subject: LU-9096 test: add sanity 253 to ALWAYS_EXCEPT
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: cdc7b3bb16537fc513a64b4f20824b9832efcdf9

[James Nunez (Inactive)]

Since sanity 253 is failing intermittently and the correctness of the test is being questioned, I've uploaded a patch to add the test to the ALWAYS_EXCEPT list and, thus, not be run.

Patch https://review.whamcloud.com/27013

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27013
Subject: LU-9096 test: add sanity 253 to ALWAYS_EXCEPT
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d62f40a8af0fa7d4041517e81a0183bb051c1785

[Nathaniel Clark]

bzzz,

Instead of hard coding a constant, would it make sense to use test_framework.sh::fs_log_size() to ensure there's enough room for the fs log?

[John Hammond]

In the interim it would be good to dump all the osp params when this fails (rather than just prealloc_status).