[LU-14033] panic after "ldiskfs_free_blocks:5437: IO failure"

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.15.0
• Severity: 3
• Assignee: Shaun Tancheff
• Reporter: Sergey Cheremencev

Description

We faced this issue while failover/failback testing of Pool Quotas.
Cluster configuration
Test description

Server: lustre-2.13.56_3.10.0_957.1.3957.1.3.x4.4.35.x86_64.rpm
Client: lustre-client-2.12.4.1_cray_180_gee19431_3.10.0_957.5.1.el7.x86_64.rpm

[161423.697898] LDISKFS-fs error (device md0) in ldiskfs_free_blocks:5437: IO failure
[161423.697901] LDISKFS-fs error (device md0) in ldiskfs_free_blocks:5437: IO failure
[161423.698349] LDISKFS-fs error (device md0) in ldiskfs_free_blocks:5437: IO failure
[161423.698621] LDISKFS-fs error (device md0) in ldiskfs_free_blocks:5437: IO failure
[161423.744231] Aborting journal on device md129.
[161423.744560] Aborting journal on device md129.
[161423.744565] Quota error (device md0): qtree_write_dquot: dquota write failed
[161423.744567] LDISKFS-fs error (device md0) in ldiskfs_write_dquot:5495: Journal has aborted
[161423.744575] JBD2: Spotted dirty metadata buffer (dev = md0, blocknr = 0). There's a risk of filesystem corruption in case of system crash.
[161423.744580] LDISKFS-fs error (device md0) in ldiskfs_orphan_add:3367: Journal has aborted
...
[161423.748560] LustreError: 23845:0:(ofd_dev.c:1818:ofd_destroy_hdl()) cslmo17-OST0002: error destroying object [0x100020000:0x2fb5f4a:0x0]: -30
[161423.899356] Kernel panic - not syncing: LDISKFS-fs (device md0): panic forced after error


[161423.912479] CPU: 1 PID: 21924 Comm: ll_ost00_001 Kdump: loaded Tainted: P           OE  ------------   3.10.0-957.1.3957.1.3.x4.4.35.x86_64 #1
[161423.928609] Hardware name: Seagate SATI-TL/Type2 - Board Product Sati2, BIOS SATI-TL.v0046.0002 01/13/2015
[161423.939944] Call Trace:
[161423.944047]  [<ffffffff93d64e41>] dump_stack+0x19/0x1b
[161423.950817]  [<ffffffff93d5e550>] panic+0xe8/0x21f
[161423.957195]  [<ffffffffc1d95416>] ldiskfs_handle_error.part.190+0xa6/0xb0 [ldiskfs]
[161423.966451]  [<ffffffffc1d95a9b>] __ldiskfs_std_error+0x7b/0x100 [ldiskfs]
[161423.974911]  [<ffffffffc1dad13a>] ldiskfs_free_blocks+0xa1a/0xbb0 [ldiskfs]
[161423.983432]  [<ffffffff9387946c>] ? __find_get_block+0xbc/0x120
[161423.990868]  [<ffffffffc0ba1f72>] ? jbd2_journal_get_write_access+0x32/0x40 [jbd2]
[161423.999911]  [<ffffffffc1db3039>] ldiskfs_ext_remove_space+0x8a9/0x1150 [ldiskfs]
[161424.008840]  [<ffffffffc1d7ae75>] ? ldiskfs_do_update_inode+0x525/0x810 [ldiskfs]
[161424.017794]  [<ffffffffc1db57b0>] ldiskfs_ext_truncate+0xb0/0xe0 [ldiskfs]
[161424.026051]  [<ffffffffc1d7e0ba>] ldiskfs_truncate+0x3da/0x430 [ldiskfs]
[161424.034126]  [<ffffffffc1d7ee3a>] ldiskfs_evict_inode+0x58a/0x630 [ldiskfs]
[161424.042500]  [<ffffffff9385ee14>] evict+0xb4/0x180
[161424.048627]  [<ffffffff9385f71c>] iput+0xfc/0x190
[161424.054651]  [<ffffffffc1dfd017>] osd_object_delete+0x1e7/0x360 [osd_ldiskfs]
[161424.063102]  [<ffffffffc0e67078>] lu_object_free.isra.27+0xb8/0x1c0 [obdclass]
[161424.071629]  [<ffffffffc0e6b315>] lu_object_put+0xa5/0x430 [obdclass]
[161424.079386]  [<ffffffffc172a47e>] ofd_destroy_by_fid+0x20e/0x500 [ofd]
[161424.087228]  [<ffffffffc1121810>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
[161424.096059]  [<ffffffffc111db00>] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc]
[161424.105099]  [<ffffffffc171fee7>] ofd_destroy_hdl+0x267/0x9f0 [ofd]
[161424.112623]  [<ffffffffc11bb38a>] tgt_request_handle+0x96a/0x1700 [ptlrpc]
[161424.120762]  [<ffffffffc1195981>] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc]
[161424.129592]  [<ffffffffc0c4b02e>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
[161424.137938]  [<ffffffffc115ea76>] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc]
[161424.146855]  [<ffffffff936c3050>] ? wake_up_atomic_t+0x30/0x30
[161424.153998]  [<ffffffffc11635cc>] ptlrpc_main+0xb3c/0x14d0 [ptlrpc]
[161424.161607]  [<ffffffffc1162a90>] ? ptlrpc_register_service+0xf90/0xf90 [ptlrpc]
[161424.170278]  [<ffffffff936c1f81>] kthread+0xd1/0xe0
[161424.176422]  [<ffffffff936c1eb0>] ? insert_kthread_work+0x40/0x40
[161424.183719]  [<ffffffff93d77c1d>] ret_from_fork_nospec_begin+0x7/0x21
[161424.191365]  [<ffffffff936c1eb0>] ? insert_kthread_work+0x40/0x40