Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-14033

panic after "ldiskfs_free_blocks:5437: IO failure"

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • Lustre 2.15.0
    • None
    • 3
    • 9223372036854775807

    Description

      We faced this issue while failover/failback testing of Pool Quotas.
      Cluster configuration
      Test description

      Server: lustre-2.13.56_3.10.0_957.1.3957.1.3.x4.4.35.x86_64.rpm
      Client: 
      lustre-client-2.12.4.1_cray_180_gee19431_3.10.0_957.5.1.el7.x86_64.rpm

      [161423.697898] LDISKFS-fs error (device md0) in ldiskfs_free_blocks:5437: IO failure
      [161423.697901] LDISKFS-fs error (device md0) in ldiskfs_free_blocks:5437: IO failure
      [161423.698349] LDISKFS-fs error (device md0) in ldiskfs_free_blocks:5437: IO failure
      [161423.698621] LDISKFS-fs error (device md0) in ldiskfs_free_blocks:5437: IO failure
      [161423.744231] Aborting journal on device md129.
      [161423.744560] Aborting journal on device md129.
      [161423.744565] Quota error (device md0): qtree_write_dquot: dquota write failed
      [161423.744567] LDISKFS-fs error (device md0) in ldiskfs_write_dquot:5495: Journal has aborted
      [161423.744575] JBD2: Spotted dirty metadata buffer (dev = md0, blocknr = 0). There's a risk of filesystem corruption in case of system crash.
      [161423.744580] LDISKFS-fs error (device md0) in ldiskfs_orphan_add:3367: Journal has aborted
      ...
      [161423.748560] LustreError: 23845:0:(ofd_dev.c:1818:ofd_destroy_hdl()) cslmo17-OST0002: error destroying object [0x100020000:0x2fb5f4a:0x0]: -30
      [161423.899356] Kernel panic - not syncing: LDISKFS-fs (device md0): panic forced after error
      
      
      [161423.912479] CPU: 1 PID: 21924 Comm: ll_ost00_001 Kdump: loaded Tainted: P           OE  ------------   3.10.0-957.1.3957.1.3.x4.4.35.x86_64 #1
      [161423.928609] Hardware name: Seagate SATI-TL/Type2 - Board Product Sati2, BIOS SATI-TL.v0046.0002 01/13/2015
      [161423.939944] Call Trace:
      [161423.944047]  [<ffffffff93d64e41>] dump_stack+0x19/0x1b
      [161423.950817]  [<ffffffff93d5e550>] panic+0xe8/0x21f
      [161423.957195]  [<ffffffffc1d95416>] ldiskfs_handle_error.part.190+0xa6/0xb0 [ldiskfs]
      [161423.966451]  [<ffffffffc1d95a9b>] __ldiskfs_std_error+0x7b/0x100 [ldiskfs]
      [161423.974911]  [<ffffffffc1dad13a>] ldiskfs_free_blocks+0xa1a/0xbb0 [ldiskfs]
      [161423.983432]  [<ffffffff9387946c>] ? __find_get_block+0xbc/0x120
      [161423.990868]  [<ffffffffc0ba1f72>] ? jbd2_journal_get_write_access+0x32/0x40 [jbd2]
      [161423.999911]  [<ffffffffc1db3039>] ldiskfs_ext_remove_space+0x8a9/0x1150 [ldiskfs]
      [161424.008840]  [<ffffffffc1d7ae75>] ? ldiskfs_do_update_inode+0x525/0x810 [ldiskfs]
      [161424.017794]  [<ffffffffc1db57b0>] ldiskfs_ext_truncate+0xb0/0xe0 [ldiskfs]
      [161424.026051]  [<ffffffffc1d7e0ba>] ldiskfs_truncate+0x3da/0x430 [ldiskfs]
      [161424.034126]  [<ffffffffc1d7ee3a>] ldiskfs_evict_inode+0x58a/0x630 [ldiskfs]
      [161424.042500]  [<ffffffff9385ee14>] evict+0xb4/0x180
      [161424.048627]  [<ffffffff9385f71c>] iput+0xfc/0x190
      [161424.054651]  [<ffffffffc1dfd017>] osd_object_delete+0x1e7/0x360 [osd_ldiskfs]
      [161424.063102]  [<ffffffffc0e67078>] lu_object_free.isra.27+0xb8/0x1c0 [obdclass]
      [161424.071629]  [<ffffffffc0e6b315>] lu_object_put+0xa5/0x430 [obdclass]
      [161424.079386]  [<ffffffffc172a47e>] ofd_destroy_by_fid+0x20e/0x500 [ofd]
      [161424.087228]  [<ffffffffc1121810>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
      [161424.096059]  [<ffffffffc111db00>] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc]
      [161424.105099]  [<ffffffffc171fee7>] ofd_destroy_hdl+0x267/0x9f0 [ofd]
      [161424.112623]  [<ffffffffc11bb38a>] tgt_request_handle+0x96a/0x1700 [ptlrpc]
      [161424.120762]  [<ffffffffc1195981>] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc]
      [161424.129592]  [<ffffffffc0c4b02e>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
      [161424.137938]  [<ffffffffc115ea76>] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc]
      [161424.146855]  [<ffffffff936c3050>] ? wake_up_atomic_t+0x30/0x30
      [161424.153998]  [<ffffffffc11635cc>] ptlrpc_main+0xb3c/0x14d0 [ptlrpc]
      [161424.161607]  [<ffffffffc1162a90>] ? ptlrpc_register_service+0xf90/0xf90 [ptlrpc]
      [161424.170278]  [<ffffffff936c1f81>] kthread+0xd1/0xe0
      [161424.176422]  [<ffffffff936c1eb0>] ? insert_kthread_work+0x40/0x40
      [161424.183719]  [<ffffffff93d77c1d>] ret_from_fork_nospec_begin+0x7/0x21
      [161424.191365]  [<ffffffff936c1eb0>] ? insert_kthread_work+0x40/0x40 

      Attachments

        1. dmesg
          415 kB
        2. logs.txt.gz
          1.42 MB

        Activity

          People

            stancheff Shaun Tancheff
            sergey Sergey Cheremencev
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: