[LU-2983] ASSERTION in osd_bufs_get_read()

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.0
• Fix Version/s: Lustre 2.4.0
• Type: Bug
• Priority: Critical
• Reporter: Brian Behlendorf
• Assignee: Alex Zhuravlev
• Resolution: Fixed
• Labels: sequoia

Environment

ZFS OSDs

• Epic/Theme: OSD_API
• Story Points: 1
• Severity: 3
• Epic: server

Description

Despite our parity checking hardware RAID on Grove we appear to have run in to a case where ZFS is getting bad block data from disk. The root cause for this still isn't clear and we're looking in to it.

However, it clearly exposed that right now the ZFS OSD doesn't even try to handle IO errors on read from the DMU. Lustre hit the following assertion when ZFS returned the IO error. We need to update osd_bufs_get_read() to handle the error and return it up the stack.

<ConMan> Console [grove250] log at 2013-03-17 23:00:00 PDT.
2013-03-17 23:50:10 LustreError: 7462:0:(osd_io.c:276:osd_bufs_get_read()) ASSERTION( rc == 0 ) failed: 
2013-03-17 23:50:10 LustreError: 7462:0:(osd_io.c:276:osd_bufs_get_read()) LBUG
2013-03-17 23:50:10 Pid: 7462, comm: ll_ost_io00_060
2013-03-17 23:50:10
2013-03-17 23:50:10 Call Trace:
2013-03-17 23:50:10  [<ffffffffa0346965>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
2013-03-17 23:50:10  [<ffffffffa0346f77>] lbug_with_loc+0x47/0xb0 [libcfs]
2013-03-17 23:50:10  [<ffffffffa0d36796>] osd_bufs_get+0x996/0xa10 [osd_zfs]
2013-03-17 23:50:10  [<ffffffffa06cc386>] ? lu_object_find+0x16/0x20 [obdclass]
2013-03-17 23:50:10  [<ffffffffa0dd540f>] ofd_preprw_read+0x13f/0x850 [ofd]
2013-03-17 23:50:10  [<ffffffffa0dd6073>] ofd_preprw+0x553/0x12b0 [ofd]
2013-03-17 23:50:10  [<ffffffffa0d9030c>] obd_preprw+0x12c/0x3d0 [ost]
2013-03-17 23:50:10  [<ffffffffa0d95af4>] ost_brw_read+0xd14/0x12f0 [ost]
2013-03-17 23:50:10  [<ffffffff8126c489>] ? cpumask_next_and+0x29/0x50
2013-03-17 23:50:10  [<ffffffff810551d4>] ? find_busiest_group+0x244/0x9f0
2013-03-17 23:50:10  [<ffffffffa085d52c>] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc]
2013-03-17 23:50:10  [<ffffffffa085d688>] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc]
2013-03-17 23:50:10  [<ffffffffa0d9c658>] ost_handle+0x2a68/0x46a0 [ost]
2013-03-17 23:50:10  [<ffffffffa0864c2b>] ? ptlrpc_update_export_timer+0x4b/0x470 [ptlrpc]
2013-03-17 23:50:10  [<ffffffffa086d08c>] ptlrpc_server_handle_request+0x41c/0xe00 [ptlrpc]
2013-03-17 23:50:10  [<ffffffffa03476be>] ? cfs_timer_arm+0xe/0x10 [libcfs]
2013-03-17 23:50:10  [<ffffffffa035914f>] ? lc_watchdog_touch+0x6f/0x180 [libcfs]
2013-03-17 23:50:10  [<ffffffffa0864459>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
2013-03-17 23:50:10  [<ffffffff81051ba3>] ? __wake_up+0x53/0x70
2013-03-17 23:50:10  [<ffffffffa086e625>] ptlrpc_main+0xbb5/0x1970 [ptlrpc]
2013-03-17 23:50:10  [<ffffffffa086da70>] ? ptlrpc_main+0x0/0x1970 [ptlrpc]
2013-03-17 23:50:10  [<ffffffff8100c14a>] child_rip+0xa/0x20
2013-03-17 23:50:10  [<ffffffffa086da70>] ? ptlrpc_main+0x0/0x1970 [ptlrpc]
2013-03-17 23:50:10  [<ffffffffa086da70>] ? ptlrpc_main+0x0/0x1970 [ptlrpc]
2013-03-17 23:50:10  [<ffffffff8100c140>] ? child_rip+0x0/0x20

Comments

[Peter Jones]

Alex will look into this one

[Brian Behlendorf]

Related to this we're trying to map the ZFS object number from the OST (which has a bad checksum) back to the full Lustre path for the file. What's the right way to go about this these days?

[Alex Zhuravlev]

http://review.whamcloud.com/5784

[Alex Zhuravlev]

landed