Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-2461] recovery-small test_26a: @@@@@@ FAIL: client not evicted from OST https://jira.whamcloud.com/browse/LU-2461 Lustre <p>Commit: bb1f6e0ee5aaa7b71a2b7584fdb7fc502fd6eb37<br/> Maloo: <a href="https://maloo.whamcloud.com/test_sets/cf3fb7ac-c369-11e0-9a20-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/cf3fb7ac-c369-11e0-9a20-52540025f9af</a></p> <p>I happened to notice this in an Autotest report:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== recovery-small test 26a: evict dead exports ======================================================= 01:42:25 (1312965745) starting with 4 OST exports fail_loc=0x505 Waiting 55 secs for update Waiting 50 secs for update Waiting 45 secs for update Waiting 40 secs for update Waiting 35 secs for update Waiting 30 secs for update Waiting 25 secs for update Waiting 20 secs for update Waiting 15 secs for update Waiting 10 secs for update Waiting 5 secs for update Update not seen after 55 sec: wanted '3' got '4' fail_loc=0x0 UUID 1K-blocks Used Available Use% Mounted on error: llapi_obd_statfs: opening '/mnt/lustre': Interrupted system call (4) MDT0000 : Interrupted system call error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0001 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0002 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0003 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0004 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0005 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0006 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0007 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0008 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0009 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT000a : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT000b : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT000c : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT000d : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT000e : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT000f : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0010 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0011 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0012 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0013 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0014 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0015 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0016 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0017 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0018 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0019 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT001a : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT001b : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT001c : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT001d : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT001e : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT001f : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0020 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0021 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0022 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0023 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0024 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0025 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0026 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0027 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0028 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0029 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT002a : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT002b : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT002c : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT002d : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT002e : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT002f : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0030 : Cannot send after transport endpoint shutdown error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108) MDT0031 : Cannot send after transport endpoint shutdown lustre-OST0000_UUID 7123992 327812 6421560 5% /mnt/lustre[OST:0] lustre-OST0001_UUID 7123992 323712 6424632 5% /mnt/lustre[OST:1] lustre-OST0002_UUID 7123992 323712 6425656 5% /mnt/lustre[OST:2] lustre-OST0003_UUID 7123992 323712 6425656 5% /mnt/lustre[OST:3] lustre-OST0004_UUID 7123992 323712 6425656 5% /mnt/lustre[OST:4] lustre-OST0005_UUID 7123992 323712 6425656 5% /mnt/lustre[OST:5] lustre-OST0006_UUID 7123992 327812 6421564 5% /mnt/lustre[OST:6] filesystem summary: 49867944 2274184 44970380 5% /mnt/lustre osp.lustre-OST0000-osp-MDT0000.sync_changes=0 osp.lustre-OST0000-osp-MDT0000.sync_in_flight=0 osp.lustre-OST0000-osp-MDT0000.sync_in_progress=0 osp.lustre-OST0001-osp-MDT0000.sync_changes=0 osp.lustre-OST0001-osp-MDT0000.sync_in_flight=0 osp.lustre-OST0001-osp-MDT0000.sync_in_progress=0 osp.lustre-OST0002-osp-MDT0000.sync_changes=0 osp.lustre-OST0002-osp-MDT0000.sync_in_flight=0 osp.lustre-OST0002-osp-MDT0000.sync_in_progress=0 osp.lustre-OST0003-osp-MDT0000.sync_changes=0 osp.lustre-OST0003-osp-MDT0000.sync_in_flight=0 osp.lustre-OST0003-osp-MDT0000.sync_in_progress=0 osp.lustre-OST0004-osp-MDT0000.sync_changes=0 osp.lustre-OST0004-osp-MDT0000.sync_in_flight=0 osp.lustre-OST0004-osp-MDT0000.sync_in_progress=0 osp.lustre-OST0005-osp-MDT0000.sync_changes=0 osp.lustre-OST0005-osp-MDT0000.sync_in_flight=0 osp.lustre-OST0005-osp-MDT0000.sync_in_progress=0 osp.lustre-OST0006-osp-MDT0000.sync_changes=0 osp.lustre-OST0006-osp-MDT0000.sync_in_flight=0 osp.lustre-OST0006-osp-MDT0000.sync_in_progress=0 osp.lustre-OST0000-osp-MDT0000.prealloc_status=0 osp.lustre-OST0001-osp-MDT0000.prealloc_status=0 osp.lustre-OST0002-osp-MDT0000.prealloc_status=0 osp.lustre-OST0003-osp-MDT0000.prealloc_status=0 osp.lustre-OST0004-osp-MDT0000.prealloc_status=0 osp.lustre-OST0005-osp-MDT0000.prealloc_status=0 osp.lustre-OST0006-osp-MDT0000.prealloc_status=0 recovery-small test_26a: @@@@@@ FAIL: client not evicted from OST Dumping lctl log to /logdir/test_logs/2011-08-09/lustre-dev-el5-x86_64__108__-7fedefc99de0/recovery-small.test_26a.*.1312965806.log </pre> </div></div> LU-2461 recovery-small test_26a: @@@@@@ FAIL: client not evicted from OST Bug Minor Resolved Duplicate Alex Zhuravlev Li Wei Mon, 15 Aug 2011 22:06:49 +0000 Mon, 11 Feb 2019 16:03:33 +0000 Mon, 11 Feb 2019 16:03:33 +0000 Lustre 2.4.0 Lustre 2.11.0 0 3 <p>another failure: <a href="https://maloo.whamcloud.com/test_sets/f52af154-4296-11e2-8dba-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f52af154-4296-11e2-8dba-52540035b04c</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/fe065d76-0baa-11e8-a7cd-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fe065d76-0baa-11e8-a7cd-52540065bddc</a><br/> Another failure of this test but with bash syntax error:<br/> Lustre 2.10.57.89</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== recovery-small test 26a: evict dead exports ======================================================= 00:55:25 (1517964925) CMD: trevis-6vm9 lctl get_param -n timeout CMD: trevis-6vm6.trevis.hpdd.intel.com lctl set_param fail_loc=0x505 fail_loc=0x505 CMD: trevis-6vm6.trevis.hpdd.intel.com lctl set_param fail_loc=0x0 fail_loc=0x0 CMD: trevis-6vm6.trevis.hpdd.intel.com df df: '/mnt/lustre': Cannot send after transport endpoint shutdown Check state for lustre-OST0000-osc-ffff880039c89800 CMD: trevis-6vm6.trevis.hpdd.intel.com /usr/sbin/lctl get_param osc.lustre-OST0000-osc-ffff880039c89800.state /usr/lib64/lustre/tests/test-framework.sh: line 8496: ((: &gt; 1517964926: syntax error: operand expected (error token is "&gt; 1517964926") Check state for lustre-OST0001-osc-ffff880039c89800 </pre> </div></div> <p>Looks like exact duplicate of newer <a href="https://jira.whamcloud.com/browse/LU-10632" title="recovery-small test 26a fails with ‘client not evicted from OST’" class="issue-link" data-issue-key="LU-10632"><del>LU-10632</del></a>.</p> Duplicate LU-10632 LU-2082 Related LU-10632 Development Rank 1|hzuusv: Rank (Obsolete) 2652 Severity Story Points 1.0