[LU-8622] recovery-small test_24b: no discarded dirty page found!

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.9.0, Lustre 2.10.0, Lustre 2.12.0, Lustre 2.13.0, Lustre 2.12.3, Lustre 2.12.5, Lustre 2.12.6
• Type: Bug
• Priority: Minor
• Reporter: Maloo
• Assignee: WC Triage
• Resolution: Unresolved

Environment

master
Lustre Version: 2.8.57.60.g556f5cd

• Severity: 3

Description

This issue was created by maloo for Saurabh Tandan <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/532f66fc-7a85-11e6-8afd-5254006e85c2.

The sub-test test_24b failed with the following error:

no discarded dirty page found!

test logs:

== recovery-small test 24b: test dirty page discard due to client eviction =========================== 05:29:01 (1473856141)
warning: '--size|-s' deprecated, use '--stripe-size|-S' instead
multiop /mnt/lustre/d24b.recovery-small/f24b.recovery-small-1 vOw8192_yc
TMPPIPE=/tmp/multiop_open_wait_pipe.18656
multiop /mnt/lustre/d24b.recovery-small/f24b.recovery-small-2 vOw8192_c
TMPPIPE=/tmp/multiop_open_wait_pipe.18656
CMD: onyx-38vm8 /usr/sbin/lctl set_param -n obdfilter.lustre-OST0000.evict_client abc79743-c199-597e-592e-fb5c9d3e1a6b
fsync: Input/output error
fail_loc=0x0
CMD: onyx-38vm1.onyx.hpdd.intel.com,onyx-38vm2 /usr/bin/lfs df /mnt/lustre; uname -n >> /mnt/lustre/recon
Connected clients: onyx-38vm1.onyx.hpdd.intel.com
onyx-38vm1.onyx.hpdd.intel.com
onyx-38vm2.onyx.hpdd.intel.com
 recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 5 or close: 0 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:4835:error_ignore()
  = /usr/lib64/lustre/tests/recovery-small.sh:932:test_24b()
  = /usr/lib64/lustre/tests/test-framework.sh:5095:run_one()
  = /usr/lib64/lustre/tests/test-framework.sh:5134:run_one_logged()
  = /usr/lib64/lustre/tests/test-framework.sh:4981:run_test()
  = /usr/lib64/lustre/tests/recovery-small.sh:938:main()
Dumping lctl log to /logdir/test_logs/2016-09-14/lustre-reviews-el7-x86_64--review-zfs-part-2--1_9_1__41448__-70017634587780-002935/recovery-small.test_24b.*.1473856143.log
CMD: onyx-38vm1.onyx.hpdd.intel.com,onyx-38vm2,onyx-38vm7,onyx-38vm8 /usr/sbin/lctl dk > /logdir/test_logs/2016-09-14/lustre-reviews-el7-x86_64--review-zfs-part-2--1_9_1__41448__-70017634587780-002935/recovery-small.test_24b.debug_log.\$(hostname -s).1473856143.log;
         dmesg > /logdir/test_logs/2016-09-14/lustre-reviews-el7-x86_64--review-zfs-part-2--1_9_1__41448__-70017634587780-002935/recovery-small.test_24b.dmesg.\$(hostname -s).1473856143.log
CMD: onyx-38vm7,onyx-38vm8 /usr/sbin/lctl set_param debug=\"\"
onyx-38vm7: error: set_param: setting debug: no value
onyx-38vm8: error: set_param: setting debug: no value
Resetting fail_loc on all nodes...CMD: onyx-38vm1.onyx.hpdd.intel.com,onyx-38vm2,onyx-38vm7,onyx-38vm8 lctl set_param -n fail_loc=0 	    fail_val=0 2>/dev/null
done.
 recovery-small test_24b: @@@@@@ FAIL: no discarded dirty page found! 

Comments

[nasf (Inactive)]

+1 on master:
https://testing.hpdd.intel.com/test_sets/41e226bc-f3f4-11e6-8862-5254006e85c2

[Emoly Liu]

+1 on master:
https://testing.hpdd.intel.com/test_sets/ef1e78de-f3cd-11e6-8862-5254006e85c2

[Alexander Boyko]

+3 on master
https://testing.hpdd.intel.com/test_sets/6167708e-f2fd-11e6-a0f1-5254006e85c2
https://testing.hpdd.intel.com/test_sets/c4855122-f2d0-11e6-88d4-5254006e85c2
https://testing.hpdd.intel.com/test_sets/0be25dc8-f306-11e6-a0f1-5254006e85c2

[Bob Glossman (Inactive)]

more on master:
https://testing.hpdd.intel.com/test_sets/f99fd206-f442-11e6-8291-5254006e85c2
https://testing.hpdd.intel.com/test_sets/4e61fc5e-f3f5-11e6-ba2a-5254006e85c2
https://testing.hpdd.intel.com/test_sets/b79692ae-f3e9-11e6-8862-5254006e85c2
https://testing.hpdd.intel.com/test_sets/da0b41d4-f3f5-11e6-ba2a-5254006e85c2

[nasf (Inactive)]

The patch for this issue:
https://review.whamcloud.com/#/c/25481/

[Bob Glossman (Inactive)]

another on b2_10:
https://testing.hpdd.intel.com/test_sets/d452a7fe-0d0e-11e8-a6ad-52540065bddc

[James Nunez (Inactive)]

I think we are still seeing this for 2.12:
https://testing.whamcloud.com/test_sets/ae22c4b6-d710-11e8-9238-52540065bddc

[Sebastien Buisson]

Another occurence on b2_12:
https://testing.whamcloud.com/test_sets/f5a38d28-615a-11e9-9720-52540065bddc

[Minh Diep]

+1 on b2_12 https://testing.whamcloud.com/test_sets/3f44f314-834c-11e9-8c65-52540065bddc

[Chris Horn]

+1 on master https://testing.whamcloud.com/test_sets/2486dafc-ab55-11e9-a0be-52540065bddc

[Jian Yu]

+1 on master: https://testing.whamcloud.com/test_sets/72d600ae-c8cc-11e9-a25b-52540065bddc

[James Nunez (Inactive)]

It looks like we are still seeing this on the b2_12 (2.12.5 RC1) branch at https://testing.whamcloud.com/test_sets/f9847425-112b-4de4-9cd1-53c3e4ef8a75, but the error message on the client does not contain the "error: set_param: setting debug: no value" errors:

== recovery-small test 24b: test dirty page discard due to client eviction =========================== 19:27:08 (1590866828)
multiop /mnt/lustre/d24b.recovery-small/f24b.recovery-small-1 vOw8192_yc
TMPPIPE=/tmp/multiop_open_wait_pipe.13430
multiop /mnt/lustre/d24b.recovery-small/f24b.recovery-small-2 vOw8192_c
TMPPIPE=/tmp/multiop_open_wait_pipe.13430
CMD: trevis-51vm3 /usr/sbin/lctl set_param -n obdfilter.lustre-OST0000.evict_client 0a9e3c9c-5287-6aad-04b4-e8daecb3761f
fsync: Input/output error
fail_loc=0x0
CMD: trevis-51vm1.trevis.whamcloud.com,trevis-51vm2 /usr/bin/lfs df /mnt/lustre; uname -n >> /mnt/lustre/recon
Connected clients: trevis-51vm1.trevis.whamcloud.com
trevis-51vm1.trevis.whamcloud.com
trevis-51vm2.trevis.whamcloud.com
 recovery-small test_24b: @@@@@@ IGNORE (bz5494): multiop didn't fail fsync: 5 or close: 0 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:5923:error_ignore()
  = /usr/lib64/lustre/tests/recovery-small.sh:1058:test_24b()
  = /usr/lib64/lustre/tests/test-framework.sh:6210:run_one()
  = /usr/lib64/lustre/tests/test-framework.sh:6259:run_one_logged()
  = /usr/lib64/lustre/tests/test-framework.sh:6099:run_test()
  = /usr/lib64/lustre/tests/recovery-small.sh:1064:main()
Dumping lctl log to /autotest/autotest2/2020-05-29/lustre-b2_12_full_86_1_3_590417cc-5242-465e-99ac-474e865329c6/recovery-small.test_24b.*.1590866831.log
CMD: trevis-51vm1.trevis.whamcloud.com,trevis-51vm2,trevis-51vm3,trevis-51vm4,trevis-51vm5 /usr/sbin/lctl dk > /autotest/autotest2/2020-05-29/lustre-b2_12_full_86_1_3_590417cc-5242-465e-99ac-474e865329c6/recovery-small.test_24b.debug_log.\$(hostname -s).1590866831.log;
         dmesg > /autotest/autotest2/2020-05-29/lustre-b2_12_full_86_1_3_590417cc-5242-465e-99ac-474e865329c6/recovery-small.test_24b.dmesg.\$(hostname -s).1590866831.log
Resetting fail_loc on all nodes...CMD: trevis-51vm1.trevis.whamcloud.com,trevis-51vm2,trevis-51vm3,trevis-51vm4,trevis-51vm5 lctl set_param -n fail_loc=0 	    fail_val=0 2>/dev/null
done.
 recovery-small test_24b: @@@@@@ FAIL: no discarded dirty page found! 

[Chris Horn]

+1 on master https://testing.whamcloud.com/test_sessions/cd730a9a-30af-4794-840b-8d1593e2a931

[Sergey Cheremencev]

+1 on master https://testing.whamcloud.com/test_sets/5a494dc2-32e4-4b2c-921a-76c4e2ead1dc

[Artem Blagodarenko (Inactive)]

https://testing.whamcloud.com/test_sets/860709dd-da42-484d-aeac-08ece2ce2ba0

[Nikitas Angelinas]

+1 on master: https://testing.whamcloud.com/test_sets/0eebbd35-dcc7-412b-8a64-22ca18755ad5

[Serguei Smirnov]

+1 on master: https://testing.whamcloud.com/test_sets/49a83f29-6961-4a8e-b6bc-5cf6130578fa

[Arshad Hussain]

+1 on master (https://testing.whamcloud.com/test_sets/95476a3a-2973-4c01-82a5-4783b7bb434d)