[LU-2461] recovery-small test_26a: @@@@@@ FAIL: client not evicted from OST

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.0, Lustre 2.11.0
• Type: Bug
• Priority: Minor
• Reporter: Li Wei (Inactive)
• Assignee: Alex Zhuravlev
• Resolution: Duplicate
• Story Points: 1
• Severity: 3

Description

Commit: bb1f6e0ee5aaa7b71a2b7584fdb7fc502fd6eb37
Maloo: https://maloo.whamcloud.com/test_sets/cf3fb7ac-c369-11e0-9a20-52540025f9af

I happened to notice this in an Autotest report:

== recovery-small test 26a: evict dead exports ======================================================= 01:42:25 (1312965745)
starting with 4 OST exports
fail_loc=0x505
Waiting 55 secs for update
Waiting 50 secs for update
Waiting 45 secs for update
Waiting 40 secs for update
Waiting 35 secs for update
Waiting 30 secs for update
Waiting 25 secs for update
Waiting 20 secs for update
Waiting 15 secs for update
Waiting 10 secs for update
Waiting 5 secs for update
Update not seen after 55 sec: wanted '3' got '4'
fail_loc=0x0
UUID                   1K-blocks        Used   Available Use% Mounted on
error: llapi_obd_statfs: opening '/mnt/lustre': Interrupted system call (4)
MDT0000             : Interrupted system call
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0001             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0002             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0003             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0004             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0005             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0006             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0007             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0008             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0009             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT000a             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT000b             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT000c             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT000d             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT000e             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT000f             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0010             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0011             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0012             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0013             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0014             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0015             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0016             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0017             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0018             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0019             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT001a             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT001b             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT001c             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT001d             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT001e             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT001f             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0020             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0021             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0022             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0023             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0024             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0025             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0026             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0027             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0028             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0029             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT002a             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT002b             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT002c             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT002d             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT002e             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT002f             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0030             : Cannot send after transport endpoint shutdown
error: llapi_obd_statfs: opening '/mnt/lustre': Cannot send after transport endpoint shutdown (108)
MDT0031             : Cannot send after transport endpoint shutdown
lustre-OST0000_UUID      7123992      327812     6421560   5% /mnt/lustre[OST:0]
lustre-OST0001_UUID      7123992      323712     6424632   5% /mnt/lustre[OST:1]
lustre-OST0002_UUID      7123992      323712     6425656   5% /mnt/lustre[OST:2]
lustre-OST0003_UUID      7123992      323712     6425656   5% /mnt/lustre[OST:3]
lustre-OST0004_UUID      7123992      323712     6425656   5% /mnt/lustre[OST:4]
lustre-OST0005_UUID      7123992      323712     6425656   5% /mnt/lustre[OST:5]
lustre-OST0006_UUID      7123992      327812     6421564   5% /mnt/lustre[OST:6]

filesystem summary:     49867944     2274184    44970380   5% /mnt/lustre

osp.lustre-OST0000-osp-MDT0000.sync_changes=0
osp.lustre-OST0000-osp-MDT0000.sync_in_flight=0
osp.lustre-OST0000-osp-MDT0000.sync_in_progress=0
osp.lustre-OST0001-osp-MDT0000.sync_changes=0
osp.lustre-OST0001-osp-MDT0000.sync_in_flight=0
osp.lustre-OST0001-osp-MDT0000.sync_in_progress=0
osp.lustre-OST0002-osp-MDT0000.sync_changes=0
osp.lustre-OST0002-osp-MDT0000.sync_in_flight=0
osp.lustre-OST0002-osp-MDT0000.sync_in_progress=0
osp.lustre-OST0003-osp-MDT0000.sync_changes=0
osp.lustre-OST0003-osp-MDT0000.sync_in_flight=0
osp.lustre-OST0003-osp-MDT0000.sync_in_progress=0
osp.lustre-OST0004-osp-MDT0000.sync_changes=0
osp.lustre-OST0004-osp-MDT0000.sync_in_flight=0
osp.lustre-OST0004-osp-MDT0000.sync_in_progress=0
osp.lustre-OST0005-osp-MDT0000.sync_changes=0
osp.lustre-OST0005-osp-MDT0000.sync_in_flight=0
osp.lustre-OST0005-osp-MDT0000.sync_in_progress=0
osp.lustre-OST0006-osp-MDT0000.sync_changes=0
osp.lustre-OST0006-osp-MDT0000.sync_in_flight=0
osp.lustre-OST0006-osp-MDT0000.sync_in_progress=0
osp.lustre-OST0000-osp-MDT0000.prealloc_status=0
osp.lustre-OST0001-osp-MDT0000.prealloc_status=0
osp.lustre-OST0002-osp-MDT0000.prealloc_status=0
osp.lustre-OST0003-osp-MDT0000.prealloc_status=0
osp.lustre-OST0004-osp-MDT0000.prealloc_status=0
osp.lustre-OST0005-osp-MDT0000.prealloc_status=0
osp.lustre-OST0006-osp-MDT0000.prealloc_status=0
 recovery-small test_26a: @@@@@@ FAIL: client not evicted from OST 
Dumping lctl log to /logdir/test_logs/2011-08-09/lustre-dev-el5-x86_64__108__-7fedefc99de0/recovery-small.test_26a.*.1312965806.log

Issue Links

• duplicates: LU-10632 recovery-small test 26a fails with ‘c...
• is duplicated by: LU-2082 Test failure on test suite recovery-s...
• is related to: LU-10632 recovery-small test 26a fails with ‘c...

Comments

[Sarah Liu]

another failure: https://maloo.whamcloud.com/test_sets/f52af154-4296-11e2-8dba-52540035b04c

[Nathaniel Clark]

https://testing.hpdd.intel.com/test_sets/fe065d76-0baa-11e8-a7cd-52540065bddc
Another failure of this test but with bash syntax error:
Lustre 2.10.57.89

== recovery-small test 26a: evict dead exports ======================================================= 00:55:25 (1517964925)
CMD: trevis-6vm9 lctl get_param -n timeout
CMD: trevis-6vm6.trevis.hpdd.intel.com lctl set_param fail_loc=0x505
fail_loc=0x505
CMD: trevis-6vm6.trevis.hpdd.intel.com lctl set_param fail_loc=0x0
fail_loc=0x0
CMD: trevis-6vm6.trevis.hpdd.intel.com df
df: '/mnt/lustre': Cannot send after transport endpoint shutdown
Check state for lustre-OST0000-osc-ffff880039c89800
CMD: trevis-6vm6.trevis.hpdd.intel.com /usr/sbin/lctl get_param osc.lustre-OST0000-osc-ffff880039c89800.state
/usr/lib64/lustre/tests/test-framework.sh: line 8496: ((: > 1517964926: syntax error: operand expected (error token is "> 1517964926")
Check state for lustre-OST0001-osc-ffff880039c89800

[Patrick Farrell (Inactive)]

Looks like exact duplicate of newer LU-10632.