https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1232 Lustre <p>Running large lun with 24T OST on Juelich cluster, got this error when running llverfs in full mode on the OST ldiskfs filesystem </p> <p>write filename: /mnt/ost1/dir00157/file025, current 787.039 MB/s, overall 100.624 MB/s, est 4294967248:4294967257:4294967237 left<br/> write filename: /mnt/ost1/dir00157/file026, current 794.458 MB/s, overall 100.642 MB/s, est 4294967248:4294967252:429496724<br/> llverfs: Open '/mnt/ost1/dir00172/file002' failed:Input/output error</p> <p>Please see the attached for console log and dmesg.</p> 2.2-RC1-RHEL6 server and client LU-1232

Input/Output error during large lun test

Bug Blocker Resolved Won't Fix Yang Sheng Sarah Liu Mon, 19 Mar 2012 02:26:55 +0000 Wed, 21 Mar 2012 12:53:58 +0000 Wed, 21 Mar 2012 10:17:36 +0000 Lustre 2.2.0 0 2 <p>Yangsheng</p> <p>Could you please advise on tihs one?</p> <p>Thanks</p> <p>Peter</p> <p>This issue looks like cause by a hardware problem.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sd 6:0:27:0: rejecting I/O to offline device LDISKFS-fs error (device dm-1): ldiskfs_find_entry: reading directory #22708225 offset 0 sd 6:0:27:0: rejecting I/O to offline device LDISKFS-fs error (device dm-1): ldiskfs_read_inode_bitmap: Cannot read inode bitmap - block_group = 177408, inode_bitmap = 5813305600 LDISKFS-fs error (device dm-1) in ldiskfs_new_inode: IO failure sd 6:0:27:0: rejecting I/O to offline device LDISKFS-fs (dm-1): delayed block allocation failed for inode 22544416 at logical offset 995328 with max blocks 2048 with error -5 This should not happen!! Data will be lost JBD2: Detected IO errors while flushing file data on dm-1-8 </pre> </div></div> <p>I reran this test in partial mode, failed again. Please see the attached for console log and demsg.</p> <p>From demsg_partial, it is very obvious a storage error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Buffer I/O error on device dm-0, logical block 65598925 lost page write due to I/O error on dm-0 sd 6:0:22:0: [sdu] Unhandled error code sd 6:0:22:0: [sdu] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK sd 6:0:22:0: [sdu] CDB: Write(10): 2a 00 1f 61 89 08 00 00 08 00 end_request: I/O error, dev sdu, sector 526485768 </pre> </div></div> <p>I'll look into RHEL6 bugzilla trying to found if has this kind issue with this driver.</p> <p>mpt2sas0: LSISAS2008: FWVersion(11.00.00.00), ChipRevision(0x03), BiosVersion(07.21.00.00)</p> <p>Thanks for the whole dmesg log.</p> <p>I try to install tag-2.1.56 build again and got following error:<br/> -------------------<br/> BUG: unable to handle kernel NULL pointer dereference at 0000000000000006<br/> IP: <span class="error">&#91;&lt;ffffffffa00b9f01&gt;&#93;</span> ses_intf_add+0x2f1/0x5e0 <span class="error">&#91;ses&#93;</span><br/> PGD 630ec3067 PUD 631ed5067 PMD 0 <br/> Oops: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP <br/> last sysfs file: /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.1/device<br/> CPU 0 <br/> Modules linked in: ses<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/add.png" height="16" width="16" align="absmiddle" alt="" border="0"/> enclosure mlx4_ib ib_mad ib_core mlx4_en mlx4_core scsi_wait_scan<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/add.png" height="16" width="16" align="absmiddle" alt="" border="0"/> igb iTCO_wdt i2c_i801<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/add.png" height="16" width="16" align="absmiddle" alt="" border="0"/> i2c_core i7core_edac<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/add.png" height="16" width="16" align="absmiddle" alt="" border="0"/> ioatdma<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/add.png" height="16" width="16" align="absmiddle" alt="" border="0"/> iTCO_vendor_support dca edac_core microcode serio_raw shpchp<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/add.png" height="16" width="16" align="absmiddle" alt="" border="0"/> ext3 jbd mbcache sd_mod crc_t10dif mpt2sas<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/add.png" height="16" width="16" align="absmiddle" alt="" border="0"/> scsi_transport_sas raid_class ahci dm_mirror dm_region_hash dm_log dm_mod</p> <p>Pid: 2080, comm: modprobe Not tainted 2.6.32-220.4.2.el6_lustre.gddd1a7c.x86_64 #1 SGI.COM C1104-2TY9/X8DTT-IBQF<br/> RIP: 0010:<span class="error">&#91;&lt;ffffffffa00b9f01&gt;&#93;</span> <span class="error">&#91;&lt;ffffffffa00b9f01&gt;&#93;</span> ses_intf_add+0x2f1/0x5e0 <span class="error">&#91;ses&#93;</span><br/> RSP: 0018:ffff8803340bfe38 EFLAGS: 00010246<br/> RAX: ffff88032db03800 RBX: ffff88032f5ae800 RCX: 0000000000000017<br/> RDX: 0000000000000000 RSI: ffffffff8126cdd0 RDI: 0000000000000000<br/> RBP: ffff8803340bfe98 R08: ffffffff81c00280 R09: 0000000000000000<br/> R10: 0000000000000000 R11: 0000000000000000 R12: ffff880637d78920<br/> R13: 0000000000000000 R14: ffff880330ec0400 R15: ffff88063006cdc0<br/> FS: 00007f9665162700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000<br/> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> CR2: 0000000000000006 CR3: 0000000637094000 CR4: 00000000000006f0<br/> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Process modprobe (pid: 2080, threadinfo ffff8803340be000, task ffff880336dd74c0)<br/> Stack:<br/> ffff8803340bfe78 ffffffff814d3aef ffff880330048800 ffff88032f5aeb58<br/> &lt;0&gt; ffff88032f5ae938 0000000000000010 0000000000000000 ffffffffa00ba460<br/> &lt;0&gt; ffffffff81b01ee0 ffff8803340bfea8 0000000000000000 0000000000000000<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff814d3aef&gt;&#93;</span> ? klist_next+0x7f/0xf0<br/> <span class="error">&#91;&lt;ffffffff81347599&gt;&#93;</span> class_interface_register+0xa9/0xe0<br/> <span class="error">&#91;&lt;ffffffffa00fe000&gt;&#93;</span> ? ses_init+0x0/0x3c <span class="error">&#91;ses&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81367366&gt;&#93;</span> scsi_register_interface+0x16/0x20<br/> <span class="error">&#91;&lt;ffffffffa00fe014&gt;&#93;</span> ses_init+0x14/0x3c <span class="error">&#91;ses&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100204c&gt;&#93;</span> do_one_initcall+0x3c/0x1d0<br/> <span class="error">&#91;&lt;ffffffff810af4e1&gt;&#93;</span> sys_init_module+0xe1/0x250<br/> <span class="error">&#91;&lt;ffffffff8100b0f2&gt;&#93;</span> system_call_fastpath+0x16/0x1b<br/> Code: 29 e1 48 85 c0 75 13 eb 51 90 48 8b 3b 48 89 c6 e8 d5 f0 29 e1 48 85 c0 74 40 8b b8 84 00 00 00 85 ff 75 e6 48 8b 90 a8 00 00 00 &lt;f6&gt; 42 06 40 75 d9 48 89 c6 4c 89 f7 48 89 45 b0 e8 aa fa ff ff <br/> RIP <span class="error">&#91;&lt;ffffffffa00b9f01&gt;&#93;</span> ses_intf_add+0x2f1/0x5e0 <span class="error">&#91;ses&#93;</span><br/> RSP &lt;ffff8803340bfe38&gt;<br/> CR2: 0000000000000006<br/> --<del>[ end trace c653e9e779d07a3e ]</del>--<br/> Kernel panic - not syncing: Fatal exception<br/> Pid: 2080, comm: modprobe Tainted: G D ---------------- 2.6.32-220.4.2.el6_lustre.gddd1a7c.x86_64 #1<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff814ec61a&gt;&#93;</span> ? panic+0x78/0x143<br/> <span class="error">&#91;&lt;ffffffff814f07a4&gt;&#93;</span> ? oops_end+0xe4/0x100<br/> <span class="error">&#91;&lt;ffffffff8104234b&gt;&#93;</span> ? no_context+0xfb/0x260<br/> <span class="error">&#91;&lt;ffffffffffff81250984&gt;&#93;</span> ?3&gt;] ? __do_page_ft+0x49/0x60<br/> [&lt;f0x190<br/> [&lt;ffffffge_fault+0x25/0x30<br/> <span class="error">&#91;&lt;ffffffff8126cdd0&gt;&#93;</span> ? kobject_release+0x0/0x240<br/> <span class="error">&#91;&lt;ffffffffa00b9f01&gt;&#93;</span> ? ses_intf_add+0x2f1/0x5e0 <span class="error">&#91;ses&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa00b9f25&gt;&#93;</span> ? ses_intf_add+0x315/0x5e0 <span class="error">&#91;ses&#93;</span><br/> <span class="error">&#91;&lt;ffffffff814d3aef&gt;&#93;</span> ? klist_next+0x7f/0xf0<br/> <span class="error">&#91;&lt;ffffffff81347599&gt;&#93;</span> ? class_interface_register+0xa9/0xe0<br/> <span class="error">&#91;&lt;ffffffffa00fe000&gt;&#93;</span> ? ses_init+0x0/0x3c <span class="error">&#91;ses&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81367366&gt;&#93;</span> ? scsi_register_interface+0x16/0x20<br/> <span class="error">&#91;&lt;ffffffffa00fe014&gt;&#93;</span> ? ses_init+0x14/0x3c <span class="error">&#91;ses&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100204c&gt;&#93;</span> ? do_one_initcall+0x3c/0x1d0<br/> <span class="error">&#91;&lt;ffffffff810af4e1&gt;&#93;</span> ? sys_init_module+0xe1/0x250<br/> <span class="error">&#91;&lt;ffffffff8100b0f2&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b</p> <p>Hi Sarah,</p> <p>you're right, two disks of the pool assigned the OSS nodes are broken:</p> <p>ID 5000c50040cf7d9d /dev/sdu ST2000NM0001 (2TB disk)<br/> ID 5000c50034003265 /dev/sdz ST33000650SS (3TB disk)</p> <p>I removed them from the JBOD. Could you remove them from the autotest resource file, till we receive the spare parts?</p> <p>These are too many HW failures in 3 month for such a little environment. I'll try to get in touch with our supplier whether there's a quality issue with the disks or maybe some problem with MPT driver, disk firmware or ...<br/> I'm very sorry for the delay caused by these failures. </p> <p>Thanks Frank. I am closing this ticket because it is now clear that it is not related to a Lustre software issue.</p> <p>Thanks Frank, I will remove them from the script.</p> Development Rank 1|hzvh5z: Rank (Obsolete) 6430 Severity