[LU-11987] sanity test 59 fails with “test_59 failed with 1” Created: 21/Feb/19  Updated: 25/Oct/20

Status: Open
Project: Lustre
Component/s: None
Affects Version/s: Lustre 2.13.0, Lustre 2.12.1, Lustre 2.12.2, Lustre 2.14.0
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: James Nunez (Inactive) Assignee: WC Triage
Resolution: Unresolved Votes: 0
Labels: None
Environment:

DNE/ZFS


Issue Links:
Related
is related to LU-12317 sanity test 239a fails with 'test_59 ... Closed
is related to LU-10934 integrate statx() API with Lustre Resolved
is related to LU-12572 sanity-pfl test_20b: Delete is not co... Open
Severity: 3
Rank (Obsolete): 9223372036854775807

 Description   

sanity test_59 fails in wait_delete_completed_mds() with “test_59 failed with 1” for review-dne-zfs-part-1 only.

sanity test 59 is a simple test that creates 130 files, unlinks the files and waits for 20 seconds, by default, until all osc.*MDT*.sync_* parameters are zero.

Looking at the suite_log for https://testing.whamcloud.com/test_sets/299f3fba-358b-11e9-ae87-52540065bddc , we see that the sync parameters are not all zero for all MDSs within 28 seconds; some parameters that are zero are removed from the following output and replaced with “…”

Delete is not completed in 28 seconds
CMD: trevis-33vm4,trevis-33vm5 /usr/sbin/lctl get_param osc.*MDT*.sync_*
…
osc.lustre-OST0001-osc-MDT0000.sync_changes=0
osc.lustre-OST0001-osc-MDT0000.sync_in_flight=0
osc.lustre-OST0001-osc-MDT0000.sync_in_progress=16
…
osc.lustre-OST0002-osc-MDT0000.sync_changes=0
osc.lustre-OST0002-osc-MDT0000.sync_in_flight=0
osc.lustre-OST0002-osc-MDT0000.sync_in_progress=16
…
osc.lustre-OST0003-osc-MDT0000.sync_changes=0
osc.lustre-OST0003-osc-MDT0000.sync_in_flight=0
osc.lustre-OST0003-osc-MDT0000.sync_in_progress=17
…
osc.lustre-OST0004-osc-MDT0000.sync_changes=0
osc.lustre-OST0004-osc-MDT0000.sync_in_flight=0
osc.lustre-OST0004-osc-MDT0000.sync_in_progress=16
…
osc.lustre-OST0005-osc-MDT0000.sync_changes=0
osc.lustre-OST0005-osc-MDT0000.sync_in_flight=0
osc.lustre-OST0005-osc-MDT0000.sync_in_progress=16
…
osc.lustre-OST0007-osc-MDT0000.sync_changes=0
osc.lustre-OST0007-osc-MDT0000.sync_in_flight=0
osc.lustre-OST0007-osc-MDT0000.sync_in_progress=16
…
 sanity test_59: @@@@@@ FAIL: test_59 failed with 1 

There are no errors or other indication of a problem in the console logs.

Note: We should check the return code of wait_delete_completed() and call error() when the return code is anything other than 0 to make the error message more descriptive/helpful.

Logs for more failures are at:
https://testing.whamcloud.com/test_sets/57398b58-2d41-11e9-90fb-52540065bddc
https://testing.whamcloud.com/test_sets/42fcf44c-25a6-11e9-b901-52540065bddc
https://testing.whamcloud.com/test_sets/455c7bc2-2574-11e9-b54c-52540065bddc



 Comments   
Comment by Jian Yu [ 28/Aug/19 ]

+1 on master branch: https://testing.whamcloud.com/test_sets/42267b06-c9a7-11e9-9fc9-52540065bddc

Comment by Chris Horn [ 23/Oct/19 ]

+1 on master https://testing.whamcloud.com/test_sessions/ac6b159b-d75f-4549-897c-5623fa253a3f

Comment by Emoly Liu [ 20/Nov/19 ]

+1 on master: https://testing.whamcloud.com/test_sets/98cb5c76-0af0-11ea-b934-52540065bddc

Comment by Andreas Dilger [ 12/Dec/19 ]

+1 on master https://testing.whamcloud.com/test_sets/01759ca2-1bcf-11ea-b0f4-52540065bddc

Comment by Jian Yu [ 26/Jan/20 ]

Still failed on master branch: https://testing.whamcloud.com/test_sets/ee464ed2-406e-11ea-9543-52540065bddc

Comment by Chris Horn [ 04/May/20 ]

+1 on master https://testing.whamcloud.com/test_sessions/f85132f8-4407-4088-a0af-c734aa297cf7

Comment by Andreas Dilger [ 30/Jul/20 ]

+1 on master https://testing.whamcloud.com/test_sets/0a4bb37a-3f5d-42c7-ade9-c39efcdce19d

Comment by John Hammond [ 07/Oct/20 ]

+1 on master https://testing.whamcloud.com/test_sets/35669252-36f6-4240-8c31-6df8ed5cd902

Comment by Bruno Faccini (Inactive) [ 25/Oct/20 ]

+1 with latest master at https://testing.whamcloud.com/test_sets/73fc9f7a-40ca-4366-a0cc-4e4104e39f1c

Generated at Sat Feb 10 02:48:41 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.