[LU-9096] sanity test_253: File creation failed after rm

• Type: Bug
• Resolution: Unresolved
• Priority: Critical
• Affects Version/s: Lustre 2.9.0, Lustre 2.10.0, Lustre 2.14.0
• Labels: always_except zfs
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for Andreas Dilger <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/b06b5a90-e8f0-11e6-935d-5254006e85c2.

The sub-test test_253 failed with the following error:

CMD: trevis-43vm3 /usr/sbin/lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_status
prealloc_status -28
dd: failed to open '/mnt/lustre/d253.sanity/2': No space left on device
5+0 records in
5+0 records out
5242880 bytes (5.2 MB) copied, 0.0534031 s, 98.2 MB/s
CMD: trevis-43vm3 /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1
CMD: trevis-43vm3 /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
:
:
CMD: trevis-43vm3 /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Delete is not completed in 28 seconds
Waiting for local destroys to complete
CMD: trevis-43vm3 lctl get_param -n lov.*.qos_maxage
 sanity test_253: @@@@@@ FAIL: File creation failed after rm

The MDS console log shows that there are no precreated objects for this OST:

20:44:10:[11571.982460] LustreError: 22503:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x20000234c:0x13b8:0x0]: have 0 want 1

Info required for matching: sanity 253

Issue Links

• is duplicated by: LU-9450 precreate logic badness between lod_statfs_and_check() and lod_check_and_reserve_ost()
• is duplicated by: LU-9454 sanity test_255a: f255a.sanity failed
• is related to: LU-9457 sanity test_253() must restore osp.*.reserved_mb_{high,low} on error
• is related to: LU-7754 DNE3: osd-zfs gets into a livelock if transaction is too big
• is related to: LU-15553 replay-vbr test 12a fails with 'test_12a failed with 4'
• is related to: LU-10070 PFL self-extending file layout

Comments

[Andreas Dilger]

It looks like this test was re-enabled by patch https://review.whamcloud.com/33778 "LU-10070 tests: New test-framework functionality". Since it currently is not failing very offer, I'm not in a hurry to turn it off again, especially if it is adding coverage for SEL.

[Andreas Dilger]

For some reason this test is now running again and failing intermittently (about 4x per month).

[Peter Jones]

The test is no longer being run so I will remove the fixversion from this ticket and it can be reintroduced when the test has been given a clean bill of health

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/27013/
Subject: LU-9096 test: add sanity 253 to ALWAYS_EXCEPT
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: cdc7b3bb16537fc513a64b4f20824b9832efcdf9

[James Nunez (Inactive)]

Since sanity 253 is failing intermittently and the correctness of the test is being questioned, I've uploaded a patch to add the test to the ALWAYS_EXCEPT list and, thus, not be run.

Patch https://review.whamcloud.com/27013

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27013
Subject: LU-9096 test: add sanity 253 to ALWAYS_EXCEPT
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d62f40a8af0fa7d4041517e81a0183bb051c1785

[Nathaniel Clark]

bzzz,

Instead of hard coding a constant, would it make sense to use test_framework.sh::fs_log_size() to ensure there's enough room for the fs log?

[John Hammond]

In the interim it would be good to dump all the osp params when this fails (rather than just prealloc_status).

[John Hammond]

> solves the problem.. I'm wondering where 10 comes from.

I think it matches the 10 used a few lines before:
dd if=/dev/zero of=$DIR/$tdir/0 bs=1M count=10

[Alex Zhuravlev]

@@ -14318,7 +14318,7 @@ test_253() {
local blocks=$($LFS df $MOUNT | grep $ost_name | awk '

{ print $4 }

')
echo "OST still has $((blocks/1024)) mbytes free"

• local new_lwm=$((blocks/1024-10))
  + local new_lwm=$((blocks/1024-20))
  do_facet $SINGLEMDS $LCTL set_param \
  osp.$mdtosc_proc1.reserved_mb_high=$((new_lwm+5))
  do_facet $SINGLEMDS $LCTL set_param \

solves the problem.. I'm wondering where 10 comes from.

[Alex Zhuravlev]

updated statfs: rc = 0, state, ffree 57923, bavail 52864, avail 206, low 1
updated statfs: rc = 0, state, ffree 57923, bavail 52864, avail 206, low 1
updated statfs: rc = 0, state enospc, ffree 57923, bavail 39552, avail 154, low 209
updated statfs: rc = 0, state enospc, ffree 18345, bavail 44512, avail 173, low 209
updated statfs: rc = 0, state enospc, ffree 18345, bavail 44512, avail 173, low 209
updated statfs: rc = 0, state enospc, ffree 18345, bavail 44512, avail 173, low 209
updated statfs: rc = 0, state enospc, ffree 18345, bavail 44512, avail 173, low 209
updated statfs: rc = 0, state enospc, ffree 18345, bavail 42848, avail 167, low 209
updated statfs: rc = 0, state enospc, ffree 18345, bavail 42848, avail 167, low 209
updated statfs: rc = 0, state enospc, ffree 57908, bavail 52864, avail 206, low 209
updated statfs: rc = 0, state enospc, ffree 57908, bavail 52864, avail 206, low 209

sanity/253 sets low to 209:
local blocks=$($LFS df $MOUNT | grep $ost_name | awk '

{ print $4 }

')
local new_lwm=$((blocks/1024-10))

osp.lustre-OST0000-osc-MDT0000.reserved_mb_low=209