[LU-11933] sanity-hsm: test_406 sanity: test_230b osd_declare_dir_delete() ASSERTION( dt_object_exists(dt) )

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.13.0, Lustre 2.12.4
• Affects Version/s: Lustre 2.13.0, Lustre 2.12.3
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Alexander Boyko

Description

This issue was created by maloo for Alexander Boyko <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/3f081d66-2971-11e9-a318-52540065bddc

[ 6056.684483] Lustre: DEBUG MARKER: == sanity-hsm test 406: attempting to migrate HSM archived files is safe ============================= 14:13:40 (1549376020)
[ 6060.313122] LustreError: 4422:0:(osd_index.c:1117:osd_declare_dir_delete()) ASSERTION( dt_object_exists(dt) ) failed: 
[ 6060.315139] LustreError: 4422:0:(osd_index.c:1117:osd_declare_dir_delete()) LBUG
[ 6060.316456] Pid: 4422, comm: mdt00_000 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Jan 11 05:58:47 UTC 2019
[ 6060.318137] Call Trace:
[ 6060.318641]  [<ffffffffc0a127cc>] libcfs_call_trace+0x8c/0xc0 [libcfs]
[ 6060.319891]  [<ffffffffc0a1287c>] lbug_with_loc+0x4c/0xa0 [libcfs]
[ 6060.321802]  [<ffffffffc1012acd>] osd_declare_dir_delete+0x27d/0x2b0 [osd_zfs]
[ 6060.323583]  [<ffffffffc129870e>] lod_sub_declare_delete+0xbe/0x1f0 [lod]
[ 6060.324964]  [<ffffffffc1273be4>] lod_declare_delete+0x24/0x30 [lod]
[ 6060.326378]  [<ffffffffc130964f>] mdd_orphan_declare_insert+0x33f/0x690 [mdd]
[ 6060.327797]  [<ffffffffc12f0fe8>] mdd_declare_create+0xa48/0xcb0 [mdd]
[ 6060.329315]  [<ffffffffc1310d8d>] mdd_declare_migrate_create+0x159/0x3ad [mdd]
[ 6060.330679]  [<ffffffffc12f61bb>] mdd_migrate+0xe9b/0x1810 [mdd]
[ 6060.332092]  [<ffffffffc1187786>] mdt_reint_migrate_internal+0xf56/0x14e0 [mdt]
[ 6060.333573]  [<ffffffffc1187f75>] mdt_reint_rename_or_migrate.isra.51+0x265/0x860 [mdt]
[ 6060.335338]  [<ffffffffc1188580>] mdt_reint_migrate+0x10/0x20 [mdt]
[ 6060.336542]  [<ffffffffc1188633>] mdt_reint_rec+0x83/0x210 [mdt]
[ 6060.337928]  [<ffffffffc1165183>] mdt_reint_internal+0x6e3/0xaf0 [mdt]
[ 6060.339487]  [<ffffffffc11704e7>] mdt_reint+0x67/0x140 [mdt]
[ 6060.340551]  [<ffffffffc0e1740a>] tgt_request_handle+0xafa/0x1590 [ptlrpc]
[ 6060.342292]  [<ffffffffc0dba99e>] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc]
[ 6060.343705]  [<ffffffffc0dbe47c>] ptlrpc_main+0xbbc/0x2090 [ptlrpc]
[ 6060.344944]  [<ffffffffa52c1c31>] kthread+0xd1/0xe0
[ 6060.345964]  [<ffffffffa5974c37>] ret_from_fork_nospec_end+0x0/0x39
[ 6060.347201]  [<ffffffffffffffff>] 0xffffffffffffffff

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity test_230b - crashed during sanity test_230b
sanity-hsm test_406 - crashed during sanity-hsm test_406

Issue Links

• is related to: LU-12617 sanity test_230b: migrate failed: No such file or directory (-2)

Comments

[Peter Jones]

Landed for 2.13

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/36154/
Subject: LU-11933 mdt: clear sp_cr_flags in migrate unpack
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d4da3b55a8303d937828e74341b3ab5c4dfd52b2

[Gerrit Updater]

Lai Siyao ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36154
Subject: LU-11933 mdt: clear sp_cr_flags in migrate unpack
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9f8a7b36fd9aeca8e8a752d92091da4de988db0f

[Andreas Dilger]

It looks like this is only crashing during directory migration, mostly in sanity test_230b and much less often in sanity-hsm test_406:
https://testing.whamcloud.com/sub_tests/query?utf8=%E2%9C%93&warn%5Bnotice%5D=&test_set_script_id=&status%5B%5D=CRASH&query_bugs=LU-11933&horizon=2332800&commit=Update+results&buggable_class=SubTest&num_results=250

It's hit about once a day - not terrible, but probably deserves some attention.

[Andreas Dilger]

+1 on master https://testing.whamcloud.com/test_sets/3c2314e6-ca9a-11e9-90ad-52540065bddc sanity test_230b

[Minh Diep]

+1 on b2_12 https://testing.whamcloud.com/test_sessions/26c1e599-2cb2-4191-9492-37ca65e906d9

[Andreas Dilger]

+1 on master https://testing.whamcloud.com/test_sets/149b75f8-c5a4-11e9-a2b6-52540065bddc

[Lai Siyao]

No, it's related with volatile file migration, I'll check details later.

[Patrick Farrell (Inactive)]

laisiyao,

Do you think this crash is related to https://review.whamcloud.com/#/c/35360/ / LU-12485?  It's ... similar looking, at least.

[Wang Shilong (Inactive)]

https://testing.whamcloud.com/test_sets/81a7f204-b9a9-11e9-a2b6-52540065bddc

[Jian Yu]

+1 on master branch: https://testing.whamcloud.com/test_sets/e02cf1b6-b603-11e9-b753-52540065bddc