Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-6584] OSS hit LBUG and crash https://jira.whamcloud.com/browse/LU-6584 Lustre <p>No sign or indication, ie lustre-log or error messages, OSS unexpectantly crash (please see console image).</p> <p>/var/log/messages is attached</p> [<a href='mailto:root@panda-oss-25-4'>root@panda-oss-25-4</a> ~]# uname -a<br/> Linux panda-oss-25-4.sdsc.edu 3.10.73-1.el6.elrepo.x86_64 #1 SMP Thu Mar 26 16:28:30 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux<br/> [<a href='mailto:root@panda-oss-25-4'>root@panda-oss-25-4</a> ~]# rpm -aq | grep lustre<br/> lustre-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64<br/> lustre-osd-zfs-mount-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64<br/> lustre-iokit-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64<br/> lustre-source-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64<br/> lustre-osd-zfs-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64<br/> lustre-modules-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64<br/> lustre-tests-2.7.51-3.10.73_1.el6.elrepo.x86_64_gb019b03.x86_64<br/> <br/> LU-6584

OSS hit LBUG and crash

Bug Critical Resolved Fixed Mikhail Pershin Haisong Cai sdsc Fri, 8 May 2015 00:05:20 +0000 Mon, 21 Dec 2015 18:49:00 +0000 Wed, 7 Oct 2015 18:11:35 +0000 Lustre 2.7.0 Lustre 2.8.0 0 15 <p>like to add an observation:</p> <p>In 2 cases happened to 2 separated OSS so far, an OSS crashed and was brought back, OSTs mounted, then almost immediately it crashed again. </p> <p>The console image was taken from second crash.</p> <p>Haisong</p> <p>Hello Haisong,<br/> According to your messages/syslog file attached, seems that there is no crash-dump tool (kdump?) enabled on the OSS, right? If this is the case, is it possible to have this setup in order to take a vmcore image for post-mortem analysis upon a new occurrence?</p> <p>Only interesting infos found in the messages file is :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>May 7 15:18:13 panda-oss-23-6 kernel: LustreError: 26222:0:(client.c:173:__ptlrpc_prep_bulk_page()) ASSERTION( pageoffset + len <= ((1UL) << 12) ) failed: May 7 15:18:13 panda-oss-23-6 kernel: LustreError: 26222:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG May 7 15:18:13 panda-oss-23-6 kernel: Pid: 26222, comm: ll_ost_io00_013 May 7 15:18:13 panda-oss-23-6 kernel: May 7 15:18:13 panda-oss-23-6 kernel: Call Trace: May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffffa094b857>] libcfs_debug_dumpstack+0x57/0x80 [libcfs] May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffffa094bdd7>] lbug_with_loc+0x47/0xc0 [libcfs] May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffffa0fe921b>] __ptlrpc_prep_bulk_page+0xcb/0x190 [ptlrpc] May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffffa105f590>] tgt_brw_read+0xab0/0x11d0 [ptlrpc] May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffffa0ffa486>] ? lustre_pack_reply_flags+0xa6/0x1e0 [ptlrpc] May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffff8109518d>] ? sched_clock_cpu+0xcd/0x110 May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffffa105c7ae>] tgt_handle_request0+0x9e/0x3f0 [ptlrpc] May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffffa10576c0>] ? tgt_handle_recovery+0x30/0x360 [ptlrpc] May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffffa105d371>] tgt_request_handle+0x1c1/0x770 [ptlrpc] May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffffa100a5e3>] ptlrpc_server_handle_request+0x2e3/0xbc0 [ptlrpc] May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffffa094c3de>] ? cfs_timer_arm+0xe/0x10 [libcfs] May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffffa095ba0a>] ? lc_watchdog_touch+0x7a/0x190 [libcfs] May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffffa1003209>] ? ptlrpc_wait_event+0xa9/0x2f0 [ptlrpc] May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffff8108d273>] ? __wake_up+0x53/0x70 May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffffa100cd3c>] ptlrpc_main+0x9dc/0xd90 [ptlrpc] May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffffa100c360>] ? ptlrpc_main+0x0/0xd90 [ptlrpc] May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffff810821be>] kthread+0xce/0xe0 May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffff810820f0>] ? kthread+0x0/0xe0 May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffff815f93c8>] ret_from_fork+0x58/0x90 May 7 15:18:13 panda-oss-23-6 kernel: [<ffffffff810820f0>] ? kthread+0x0/0xe0 May 7 15:18:13 panda-oss-23-6 kernel: </pre> </div></div> <p>so, did something go wrong during remote <-> local buffers mapping?</p> <p>One of the OSS crashed 2 days ago crashed again this morning. I am attaching dmesg here.<br/> This dmesg was taken between the first and second crash the scenario appears at every time now. </p> <p>Just to list all LBUG we have encounter on this filesystem so far (this morning was panda-oss-25-4)</p> <p>May 6 09:06:46 panda-oss-25-4 kernel: LustreError: 30076:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> May 6 10:18:54 panda-oss-25-4 kernel: LustreError: 11585:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> May 7 15:18:13 panda-oss-23-6 kernel: LustreError: 26222:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> May 7 16:47:23 panda-oss-23-6 kernel: LustreError: 10154:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> May 8 07:45:59 panda-oss-25-4 kernel: LustreError: 12448:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> May 8 09:12:59 panda-oss-25-4 kernel: LustreError: 10032:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG</p> <p>To answer your question about kdump, no we don't have kdump enable on our OSS/MDS. <br/> We will have to exam our environment before we can enable it. At the meantime, please suggest alternative way to collect debug information, if there is any.</p> <p>thanks,<br/> Haisong</p> <p>Adding 2 lustre-logs for this morning's crash, on panda-oss-25-4</p> <p>Another OSS crashed, third OSS. Same pattern: the first crash, brought back OSS/OST, in less then a minite it comes with the second crash. It generally stay on and functional for hours until next crash. </p> <p>Mike</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Haisong,</p> <p>This is hitting wombat (our test partition) when I'm just running IOR. Both partitions have the same stack, now, so we may able reproduce it quickly. This also eliminates it being cause by pathological client IO on the production side.</p> <p>--Rick</p> <p>Mike,</p> <p>If it's any help, we were running v2.6.92 with a handful of patches related to large block support with ZFS (<a href="https://jira.whamcloud.com/browse/LU-4820" title="extra memcpy in read path" class="issue-link" data-issue-key="LU-4820"><del>LU-4820</del></a>, <a href="https://jira.whamcloud.com/browse/LU-5278" title="ZFS - many OST watchdogs with IOR" class="issue-link" data-issue-key="LU-5278"><del>LU-5278</del></a>, <a href="https://jira.whamcloud.com/browse/LU-6038" title="ZFS 0.6.4 Compatibility" class="issue-link" data-issue-key="LU-6038"><del>LU-6038</del></a>, <a href="https://jira.whamcloud.com/browse/LU-6038" title="ZFS 0.6.4 Compatibility" class="issue-link" data-issue-key="LU-6038"><del>LU-6038</del></a>, <a href="https://jira.whamcloud.com/browse/LU-6152" title="zfs large block support" class="issue-link" data-issue-key="LU-6152"><del>LU-6152</del></a>, <a href="https://jira.whamcloud.com/browse/LU-6155" title="osd_count_not_mapped() calls dbuf_hold_impl() without the lock" class="issue-link" data-issue-key="LU-6155"><del>LU-6155</del></a>) and did not see this. After 2.7 was released, we moved to v2.7.52 with the patches that hadn't landed yet. So, whatever's going on crept in between v2.6.92 and v2.7.52.</p> <p>--Rick</p> <p>Thanks for info, I am investigating this.</p> <p>Mike Pershin (mike.pershin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/14926" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14926</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6584" title="OSS hit LBUG and crash" class="issue-link" data-issue-key="LU-6584"><del>LU-6584</del></a> osd: add more debug to check buffer size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9a7be00ed34a2c7f44a42f2603200977b47118d4</p> <p>I wasn't able to find exact place of problem by inspecting related code. There is patch to get a little bit more info when this bug happens.</p> <p>Mike, could you please take a look though the patches between 2.6.92 and 2.7.52 to see if there are any likely candidates?</p> <p>We hit another LBUG and it looks like the same kind.<br/> ====</p> <p>Jun 12 01:01:45 panda-oss-25-2 kernel: LustreError: 29442:0:(client.c:173:__ptlrpc_prep_bulk_page()) ASSERTION( pageoffset + len <= ((1UL) << 12) ) failed: <br/> Jun 12 01:01:45 panda-oss-25-2 kernel: LustreError: 29442:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> Jun 12 01:01:45 panda-oss-25-2 kernel: Pid: 29442, comm: ll_ost_io01_105<br/> Jun 12 01:01:45 panda-oss-25-2 kernel: <br/> Jun 12 01:01:45 panda-oss-25-2 kernel: Call Trace:<br/> Jun 12 01:01:45 panda-oss-25-2 kernel: <span class="error">[<ffffffffa0940857>]</span> libcfs_debug_dumpstack+0x57/0x80 <span class="error">[libcfs]</span><br/> Jun 12 01:01:45 panda-oss-25-2 kernel: <span class="error">[<ffffffffa0940dd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span><br/> Jun 12 01:01:45 panda-oss-25-2 kernel: <span class="error">[<ffffffffa0fde21b>]</span> __ptlrpc_prep_bulk_page+0xcb/0x190 <span class="error">[ptlrpc]</span><br/> Jun 12 01:01:45 panda-oss-25-2 kernel: <span class="error">[<ffffffffa1054590>]</span> tgt_brw_read+0xab0/0x11d0 <span class="error">[ptlrpc]</span><br/> Jun 12 01:01:45 panda-oss-25-2 kernel: <span class="error">[<ffffffffa0fef486>]</span> ? lustre_pack_reply_flags+0xa6/0x1e0 <span class="error">[ptlrpc]</span><br/> Jun 12 01:01:45 panda-oss-25-2 kernel: <span class="error">[<ffffffff8109518d>]</span> ? sched_clock_cpu+0xcd/0x110<br/> Jun 12 01:01:45 panda-oss-25-2 kernel: <span class="error">[<ffffffffa10517ae>]</span> tgt_handle_request0+0x9e/0x3f0 <span class="error">[ptlrpc]</span><br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffffa104c6c0>]</span> ? tgt_handle_recovery+0x30/0x360 <span class="error">[ptlrpc]</span><br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffffa1052371>]</span> tgt_request_handle+0x1c1/0x770 <span class="error">[ptlrpc]</span><br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffffa0fff5e3>]</span> ptlrpc_server_handle_request+0x2e3/0xbc0 <span class="error">[ptlrpc]</span><br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffffa09413de>]</span> ? cfs_timer_arm+0xe/0x10 <span class="error">[libcfs]</span><br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffffa0950a0a>]</span> ? lc_watchdog_touch+0x7a/0x190 <span class="error">[libcfs]</span><br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffffa0ff8209>]</span> ? ptlrpc_wait_event+0xa9/0x2f0 <span class="error">[ptlrpc]</span><br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffff8108d273>]</span> ? __wake_up+0x53/0x70<br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffffa1001d3c>]</span> ptlrpc_main+0x9dc/0xd90 <span class="error">[ptlrpc]</span><br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffffa1001360>]</span> ? ptlrpc_main+0x0/0xd90 <span class="error">[ptlrpc]</span><br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffff810821be>]</span> kthread+0xce/0xe0<br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffff810820f0>]</span> ? kthread+0x0/0xe0<br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffff815f93c8>]</span> ret_from_fork+0x58/0x90<br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffff810820f0>]</span> ? kthread+0x0/0xe0<br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <br/> Jun 12 01:01:46 panda-oss-25-2 kernel: Kernel panic - not syncing: LBUG<br/> Jun 12 01:01:46 panda-oss-25-2 kernel: CPU: 11 PID: 29442 Comm: ll_ost_io01_105 Tainted: P O 3.10.73-1.el6.elrepo.x86_64 #1<br/> Jun 12 01:01:46 panda-oss-25-2 kernel: Hardware name: Intel Corporation S2600CP/S2600CP, BIOS SE5C600.86B.02.04.0003.102320141138 10/23/2014<br/> Jun 12 01:01:46 panda-oss-25-2 kernel: 000000006fe000c0 ffff881e2f76fa88 ffffffff815ece4a ffff881e2f76fb08<br/> Jun 12 01:01:46 panda-oss-25-2 kernel: ffffffff815ecbeb ffffffff00000008 ffff881e2f76fb18 ffff881e2f76fab8<br/> Jun 12 01:01:46 panda-oss-25-2 kernel: ffffffffa106cb20 ffff881e2f76fac8 0000000000000000 ffffffffa095f5b1<br/> Jun 12 01:01:46 panda-oss-25-2 kernel: Call Trace:<br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffff815ece4a>]</span> dump_stack+0x19/0x1f<br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffff815ecbeb>]</span> panic+0xc4/0x1e4<br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffffa0940e3b>]</span> lbug_with_loc+0xab/0xc0 <span class="error">[libcfs]</span><br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffffa0fde21b>]</span> __ptlrpc_prep_bulk_page+0xcb/0x190 <span class="error">[ptlrpc]</span><br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffffa1054590>]</span> tgt_brw_read+0xab0/0x11d0 <span class="error">[ptlrpc]</span><br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffffa0fef486>]</span> ? lustre_pack_reply_flags+0xa6/0x1e0 <span class="error">[ptlrpc]</span><br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffff8109518d>]</span> ? sched_clock_cpu+0xcd/0x110<br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffffa10517ae>]</span> tgt_handle_request0+0x9e/0x3f0 <span class="error">[ptlrpc]</span><br/> Jun 12 01:01:46 panda-oss-25-2 kernel: <span class="error">[<ffffffffa104c6c0>]</span> ? tgt_handle_recovery+0x30/0x360 <span class="error">[ptlrpc]</span></p> <p>Haisong, it looks like you are not yet running the debug patch from Mike (<a href="http://review.whamcloud.com/14926" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14926</a>) on your systems. It would be useful if you applied that patch (only needed on the servers) so that we can capture more information about this failure.</p> <p>Rick, what version of ZFS are you using, 0.6.4?</p> <p>Hi Mikhail,</p> <p>Yes, 0.6.4, but with the first large block support pull request, 2865.</p> <p>Here's the SPL and ZFS build process I used.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>git clone https://github.com/zfsonlinux/zpl.git cd spl git checkout spl-0.6.4 ./autogen.sh ./configure --disable-debug make pkg rpm -ivh *x86_64.rpm git clone https://github.com/zfsonlinux/zfs.git cd zfs git fetch -t https://github.com/zfsonlinux/zfs.git refs/pull/2865/head:lgblock git checkout zfs-0.6.4 git merge lgblock ./autogen.sh ./configure --disable-debug make pkg rpm -ivh *x86_64.rpm </pre> </div></div> <p>Rick, thank you for that info, I am trying to reproduce that situation. Meanwhile, were there any other specific tuning in ZFS, e.g. maximum block size or anything related to the block size?</p> <p>Also what Lustre patches are you using over the stock Lustre 2.7.52? I am asking because 2.7.52 can't be built with zfs version you are using, because of SPA_MAXBLOCKSHIFT is bigger now</p> <p>Mikhail,</p> <p>For ZFS with large block support, our record size is set to 1024k. On the Lustre side we've got a few patches applied, one or more of which has not landed:</p> <ul> <li><a href="https://jira.whamcloud.com/browse/LU-4820" title="extra memcpy in read path" class="issue-link" data-issue-key="LU-4820"><del>LU-4820</del></a> osd: drop memcpy in zfs osd</li> <li><a href="https://jira.whamcloud.com/browse/LU-6038" title="ZFS 0.6.4 Compatibility" class="issue-link" data-issue-key="LU-6038"><del>LU-6038</del></a> osd-zfs: Avoid redefining KM_SLEEP</li> <li><a href="https://jira.whamcloud.com/browse/LU-6038" title="ZFS 0.6.4 Compatibility" class="issue-link" data-issue-key="LU-6038"><del>LU-6038</del></a> osd-zfs: sa_spill_alloc()/sa_spill_free() compat</li> <li><a href="https://jira.whamcloud.com/browse/LU-6152" title="zfs large block support" class="issue-link" data-issue-key="LU-6152"><del>LU-6152</del></a> osd-zfs: ZFS large block compat</li> <li><a href="https://jira.whamcloud.com/browse/LU-6155" title="osd_count_not_mapped() calls dbuf_hold_impl() without the lock" class="issue-link" data-issue-key="LU-6155"><del>LU-6155</del></a> osd-zfs: dbuf_hold_impl() called without the lock</li> </ul> <p><a href="https://jira.whamcloud.com/browse/LU-6152" title="zfs large block support" class="issue-link" data-issue-key="LU-6152"><del>LU-6152</del></a> in particular is the one that deals with SPA_MAXBLOCKSIZE and SPA_MAXBLOCKSHIFT.</p> <p>At our site, I've handed over the build process to Dima Mishin for maintenance. Dima, can you pass on your current build (base commit, cherry picks, and any manual patches) to Mikhail?</p> <p>I have a build from commit 8a11cb62 in master, with patch from a44175d (<a href="https://jira.whamcloud.com/browse/LU-6155" title="osd_count_not_mapped() calls dbuf_hold_impl() without the lock" class="issue-link" data-issue-key="LU-6155"><del>LU-6155</del></a>), and cherry-picked c39e02e8 (<a href="https://jira.whamcloud.com/browse/LU-6584" title="OSS hit LBUG and crash" class="issue-link" data-issue-key="LU-6584"><del>LU-6584</del></a> - debug), and latest master commits of zfs and spl (a7b10a931, 77ab5dd33a)</p> <p>Haisong,</p> <p>How often do you see this in production? could you apply the debug patch (<a href="http://review.whamcloud.com/14926)?" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14926)?</a></p> <p>Thanks<br/> -Minh</p> <p>Hi Minh,</p> <p>Depending on how file-system is being used, we hit the bug between every day (as I described when I created this ticket) to several weeks apart.</p> <p>We are in the process of applying the patch on to our production file-system.</p> <p>Haisong</p> <p>We are very close to get the debug patch deployed onto our production systems.<br/> But a FYI, the following is a lit of all the hits of the LBUG and each hit is an OSS downtime:</p> <p>This is the number of times we have hit the lbug <em>today</em> Each hit is an OSS downtime.</p> <p><span class="error">[root@oasis-panda cai]</span># grep -i lbug /var/log/messages<br/> Jul 13 12:27:36 panda-oss-24-4 kernel: LustreError: 29775:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> Jul 13 12:27:37 panda-oss-24-4 kernel: <span class="error">[<ffffffffa0951dd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span><br/> Jul 13 12:27:37 panda-oss-24-4 kernel: Kernel panic - not syncing: LBUG<br/> Jul 13 12:48:52 panda-oss-24-4 kernel: LustreError: 10351:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> Jul 13 13:12:49 panda-oss-22-4 kernel: LustreError: 30300:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> Jul 13 13:12:49 panda-oss-22-4 kernel: <span class="error">[<ffffffffa094edd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span><br/> Jul 13 13:46:07 panda-oss-22-4 kernel: LustreError: 10383:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> Jul 13 13:46:07 panda-oss-22-4 kernel: <span class="error">[<ffffffffa091bdd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span><br/> Jul 13 17:00:40 panda-oss-25-4 kernel: LustreError: 10468:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> Jul 13 17:00:40 panda-oss-25-4 kernel: <span class="error">[<ffffffffa090fdd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span><br/> Jul 13 17:35:02 panda-oss-25-4 kernel: LustreError: 10310:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> Jul 13 17:47:21 panda-oss-24-1 kernel: LustreError: 29652:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> Jul 13 17:47:21 panda-oss-24-1 kernel: <span class="error">[<ffffffffa0944dd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span><br/> Jul 13 19:14:13 panda-oss-24-1 kernel: LustreError: 10101:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> Jul 13 19:14:13 panda-oss-24-1 kernel: <span class="error">[<ffffffffa091edd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span><br/> Jul 13 19:14:14 panda-oss-24-1 kernel: Kernel panic - not syncing: LBUG<br/> Jul 13 19:14:14 panda-oss-24-1 kernel: <span class="error">[<ffffffffa091ee3b>]</span> lbug_with_loc+0xab/0xc0 <span class="error">[libcfs]</span><br/> Jul 13 22:48:29 panda-oss-25-4 kernel: LustreError: 10161:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> Jul 13 22:48:29 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0914dd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span><br/> Jul 13 23:15:00 panda-oss-25-4 kernel: LustreError: 10185:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> Jul 13 23:15:01 panda-oss-25-4 kernel: <span class="error">[<ffffffffa090edd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span><br/> Jul 13 23:15:01 panda-oss-25-4 kernel: Kernel panic - not syncing: LBUG<br/> Jul 13 23:15:01 panda-oss-25-4 kernel: <span class="error">[<ffffffffa090ee3b>]</span> lbug_with_loc+0xab/0xc0 <span class="error">[libcfs]</span><br/> Jul 13 23:21:26 panda-oss-25-4 kernel: LustreError: 10265:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG<br/> Jul 13 23:21:26 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0916dd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span></p> <p>In all crash cases these 2 lines always coupled:</p> <p>Jul 13 23:21:26 panda-oss-25-4 kernel: LustreError: 10265:0:(client.c:173:__ptlrpc_prep_bulk_page()) ASSERTION( pageoffset + len <= ((1UL) << 12) ) failed: <br/> Jul 13 23:21:26 panda-oss-25-4 kernel: LustreError: 10265:0:(client.c:173:__ptlrpc_prep_bulk_page()) LBUG</p> <p>and most of the time follow by this line:<br/> Jul 13 23:21:26 panda-oss-25-4 kernel: Pid: 10265, comm: ll_ost_io00_031</p> <p>I am attaching /var/log/messages here that include all incidences from yesterday.</p> <p>collected with command line:</p> <p>cat /var/log/messages | egrep -iv "sshd|cron|run-parts|postfix|rsyslog|audispd|named|rockscommand|channel|alert-handler|411-alert" > /tmp/log.$$</p> <p>thanks,<br/> Haisong</p> <p>Haisong, what do you mean by 'OSS downtime'? </p> <p>Hi Mikhail,</p> <p>As I described at the very beginning of this ticket, when LBUG is hit and OSS crashes. <br/> Machine will have to be power reset. During that time OSS will be unavailable.</p> <p>Haisong</p> <p>Hi, what the progress of applying the debug patch and collect the log again? The latest log is still without the debug patch.</p> <p>Zhenyu, we're rolling out an update to our production file system today. After discussion with Minh we've decided to rebase our code on later releases of ZFS and Lustre that have our necessary patches. The only additional patch we've added is the debugging one for this ticket. After we're done, I'll ask Dima Mishin to post the exact releases that we're working from.</p> <p>We used:<br/> kernel-lt-3.10.73-1.el6.elrepo from kernel-lt<br/> spl 77ab5dd33a99<br/> zfs f1512ee61e2f22<br/> lustre 1ef0185e8c12 with applied dba1c12e debug patch</p> <p>We hit another LBUG today. OSS is running debug patch. <br/> Here is the /var/log/message and let me know what else you want me to collect.</p> <p>Jul 30 15:08:25 panda-oss-25-4 kernel: LustreError: 11719:0:(client.c:211:__ptlrpc_prep_bulk_page()) ASSERTION( pageoffset + len <= PAGE_CACHE_SIZE ) failed: offset 0, len 1913970688<br/> Jul 30 15:08:25 panda-oss-25-4 kernel: LustreError: 11719:0:(client.c:211:__ptlrpc_prep_bulk_page()) LBUG<br/> Jul 30 15:08:26 panda-oss-25-4 kernel: Pid: 11719, comm: ll_ost_io00_045<br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <br/> Jul 30 15:08:26 panda-oss-25-4 kernel: Call Trace:<br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0919857>]</span> libcfs_debug_dumpstack+0x57/0x80 <span class="error">[libcfs]</span><br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0919dd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span><br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0ca02fc>]</span> __ptlrpc_prep_bulk_page+0x12c/0x1f0 <span class="error">[ptlrpc]</span><br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0ca03cc>]</span> ptlrpc_prep_bulk_page_nopin+0xc/0x10 <span class="error">[ptlrpc]</span><br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0d1d432>]</span> tgt_brw_read+0xab2/0x1210 <span class="error">[ptlrpc]</span><br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0cb06b6>]</span> ? lustre_pack_reply_flags+0xa6/0x1e0 <span class="error">[ptlrpc]</span><br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffff8109518d>]</span> ? sched_clock_cpu+0xcd/0x110<br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0d1aca6>]</span> tgt_handle_request0+0x2c6/0x430 <span class="error">[ptlrpc]</span><br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0d1567a>]</span> ? tgt_handle_recovery+0x14a/0x310 <span class="error">[ptlrpc]</span><br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0d1b025>]</span> tgt_request_handle+0x215/0x990 <span class="error">[ptlrpc]</span><br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0cc07c3>]</span> ptlrpc_server_handle_request+0x2e3/0xbc0 <span class="error">[ptlrpc]</span><br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffffa092760a>]</span> ? lc_watchdog_touch+0x7a/0x190 <span class="error">[libcfs]</span><br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0cb9ba9>]</span> ? ptlrpc_wait_event+0xa9/0x300 <span class="error">[ptlrpc]</span><br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffff8108d273>]</span> ? __wake_up+0x53/0x70<br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0cc22cc>]</span> ptlrpc_main+0x9ec/0xd30 <span class="error">[ptlrpc]</span><br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0cc18e0>]</span> ? ptlrpc_main+0x0/0xd30 <span class="error">[ptlrpc]</span><br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffff810821be>]</span> kthread+0xce/0xe0<br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffff810820f0>]</span> ? kthread+0x0/0xe0<br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffff815f93c8>]</span> ret_from_fork+0x58/0x90<br/> Jul 30 15:08:26 panda-oss-25-4 kernel: <span class="error">[<ffffffff810820f0>]</span> ? kthread+0x0/0xe0<br/> Jul 30 15:08:26 panda-oss-25-4 kernel: </p> <p>In little more than an hour:</p> <p>Jul 30 15:08:25 panda-oss-25-4 kernel: LustreError: 11719:0:(client.c:211:__ptlrpc_prep_bulk_page()) LBUG<br/> Jul 30 15:25:13 panda-oss-25-4 kernel: LustreError: 9907:0:(client.c:211:__ptlrpc_prep_bulk_page()) LBUG<br/> Jul 30 15:25:13 panda-oss-25-4 kernel: Kernel panic - not syncing: LBUG<br/> Jul 30 16:16:39 panda-oss-25-4 kernel: LustreError: 10230:0:(client.c:211:__ptlrpc_prep_bulk_page()) LBUG</p> <p>Is it worthwhile to print out the whole lnb struct (maybe with neighboring values also) to see if there is memory corruption that can be identified? Is the oops always in the same place? Then it seems likely that there is some systematic memory corruption (stack overflow, out of bounds array access, etc) rather than random corruption from another thread, which would cause crashes in other parts of the code. </p> <p>Hi Haisong,</p> <p><a href="http://review.whamcloud.com/#/c/14926/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14926/</a> has been updated to include more debug information to catch the LBUG, would you please apply it and re-hit the issue and collect core dump? Please add "ha" debug before re-run the test, as </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lctl set_param debug=<span class="code-quote">"+ha"</span> </pre> </div></div> <p>Hi Zhenyu,</p> <p>We will try to apply the new debug patch. But since this is our production file-system, we will have to plan it.<br/> We hit this LBUG randomly depend on how the file-system is used. For example we just had 7 hits during last 24hrs, <br/> after about 12 days quiet period. </p> <p>So I have 2 questions:</p> <p>1) by any chance you know a way to reproduce the LBUG? The reason I ask is that we have a non-production file-system that<br/> we may use for this debugging.<br/> 2) about collecting core dump, do you mean the core dump of specific lustre processes or the entire system? When LBUG is hit, <br/> the OSS servers panic. I want to make sure we setup so that core dump can be saved properly. </p> <p>I like to remind everyone that this LBUG event usually occurs 2 or 3 consecutively, meaning OSS crash, standing OSS up, <br/> crash again, and standing OSS up again, and the third time, before it stabilized. Here are examples we had for the last 24hr. <br/> Please notice the server names and time stamps:</p> <p><span class="error">[root@oasis-panda log]</span># grep -i lbug messages<br/> Aug 11 12:32:47 panda-oss-25-4 kernel: LustreError: 10267:0:(client.c:211:__ptlrpc_prep_bulk_page()) LBUG<br/> Aug 11 12:32:47 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0920dd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span><br/> Aug 11 12:32:47 panda-oss-25-4 kernel: Kernel panic - not syncing: LBUG<br/> Aug 11 12:32:47 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0920e3b>]</span> lbug_with_loc+0xab/0xc0 <span class="error">[libcfs]</span><br/> Aug 11 13:29:54 panda-oss-25-4 kernel: LustreError: 10206:0:(client.c:211:__ptlrpc_prep_bulk_page()) LBUG<br/> Aug 11 13:29:54 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0927dd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span><br/> Aug 11 13:29:55 panda-oss-25-4 kernel: Kernel panic - not syncing: LBUG<br/> Aug 11 13:29:55 panda-oss-25-4 kernel: <span class="error">[<ffffffffa0927e3b>]</span> lbug_with_loc+0xab/0xc0 <span class="error">[libcfs]</span><br/> Aug 11 13:36:50 panda-oss-24-4 kernel: LustreError: 11848:0:(client.c:211:__ptlrpc_prep_bulk_page()) LBUG<br/> Aug 11 13:45:55 panda-oss-24-4 kernel: LustreError: 10229:0:(client.c:211:__ptlrpc_prep_bulk_page()) LBUG<br/> Aug 11 13:45:55 panda-oss-24-4 kernel: <span class="error">[<ffffffffa0923dd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span><br/> Aug 12 08:09:43 panda-oss-25-3 kernel: LustreError: 11728:0:(client.c:211:__ptlrpc_prep_bulk_page()) LBUG<br/> Aug 12 09:48:28 panda-oss-25-3 kernel: LustreError: 11232:0:(client.c:211:__ptlrpc_prep_bulk_page()) LBUG<br/> Aug 12 09:48:29 panda-oss-25-3 kernel: <span class="error">[<ffffffffa0919dd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span><br/> Aug 12 09:48:29 panda-oss-25-3 kernel: Kernel panic - not syncing: LBUG<br/> Aug 12 09:48:29 panda-oss-25-3 kernel: <span class="error">[<ffffffffa0919e3b>]</span> lbug_with_loc+0xab/0xc0 <span class="error">[libcfs]</span><br/> Aug 12 09:52:56 panda-oss-25-3 kernel: LustreError: 10548:0:(client.c:211:__ptlrpc_prep_bulk_page()) LBUG<br/> Aug 12 09:52:56 panda-oss-25-3 kernel: <span class="error">[<ffffffffa0916dd7>]</span> lbug_with_loc+0x47/0xc0 <span class="error">[libcfs]</span><br/> Aug 12 09:52:58 panda-oss-25-3 kernel: Kernel panic - not syncing: LBUG<br/> Aug 12 09:52:58 panda-oss-25-3 kernel: <span class="error">[<ffffffffa0916e3b>]</span> lbug_with_loc+0xab/0xc0 <span class="error">[libcfs]</span><br/> <span class="error">[root@oasis-panda log]</span># </p> <p>thank,<br/> Haisong</p> <p>1. Unfortunately we don't know the reason of the LBUG, we just saw that some IO has invalid page length, and it seems there is some specific memory corruption there, so the debug patch is try to print out IO pages information if abnormal happens.</p> <p>2. You can setup kdump to capture the kernel crash dump as <a href="http://fedoraproject.org/wiki/How_to_use_kdump_to_debug_kernel_crashes" class="external-link" target="_blank" rel="nofollow noopener">http://fedoraproject.org/wiki/How_to_use_kdump_to_debug_kernel_crashes</a> guides.</p> <p>If the OSS is crashing repeatedly after startup, it may mean that the bad data is arriving from the client during replay and is not being verified properly? Are there checks up in the ost layer to verify the niobuf_remote contains valid data before it is used in the OSD? It may be that the corruption is happening on the network or on the client.</p> <p>In the boot log I see you are using ZFS 0.6.4. It looks like there may be fixes in 0.6.4.1 and 0.6.4.2 that may be helpful in this case and <a href="https://jira.whamcloud.com/browse/LU-6607" title="MDS ( 2 node DNE) running out of memory and crash" class="issue-link" data-issue-key="LU-6607"><del>LU-6607</del></a> (reducing CPU usage on metadata intensive workloads):<br/> <a href="https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.6.4.1" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.6.4.1</a></p> <blockquote> <p> Fixed io-spare.sh script for ZED.<br/> Fixed multiple deadlocks which might occur when reclaiming memory.<br/> Fixed excessive CPU usage for meta data heavy workloads when reclaiming the ARC.</p></blockquote> <p><a href="https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.6.4.2" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.6.4.2</a></p> <blockquote> <p> Fix panic due to corrupt nvlist when running utilities zfsonlinux/zfs#3335<br/> Fix hard lockup due to infinite loop in zfs_zget() zfsonlinux/zfs#3349<br/> Fix panic on unmount due to iput taskq zfsonlinux/zfs#3281<br/> Improve metadata shrinker performance on pre-3.1 kernels zfsonlinux/zfs#3501<br/> Linux 4.1 compat: use read_iter() / write_iter()<br/> Linux 3.12 compat: NUMA-aware per-superblock shrinker<br/> Fix spurious hung task watchdog stack traces zfsonlinux/zfs#3402<br/> Fix module loading in zfs import systemd service zfsonlinux/zfs#3440<br/> Fix intermittent libzfs_init() failure to open /dev/zfs zfsonlinux/zfs#2556</p></blockquote> <p>Also, just to confirm - are these OSS nodes running with 1MB ZFS blocksize?</p> <p>We're using the f1512ee61e commit from master ZFS branch (large block support). It's later than 0.6.4.1, and I had problems running with latest master version.</p> <p>Andreas, yes, we're using 1MB block sizes on the ZFS datasets that handle the OSTs.</p> <p>Bobijam,<br/> can you please make a new patch that checks the contents of niobuf_remote when it is first accessed by the OST (<tt>tgt_brw_read()</tt> and <tt>tgt_brw_write()</tt>) to verify that the contents are sane, and print out all the values under D_BUFFS debugging. If the values are incorrect a CERROR() should be printed and an <tt>-EPROTO</tt> error returned to the client, and we can debug this problem as a network corruption.</p> <p>This niobuf verification should be in a helper function that can also be called before the currently-failing LASSERT() checks are being handled (and elsewhere in the code if you think it is helpful), and those functions can return an <tt><del>EIO</tt> error to the caller rather than triggering the LASSERT. At that point the client should resend the BRW RPC due to <tt>brw_interpret()</del>>osc_recoverable_error()</tt> and hopefully it will succeed on the second try.</p> <p>While I don't think this is a proper solution, it will at least tell us if the corruption is happening on the client and/or on the network, or in memory on the OSS, and it will potentially allow debugging to continue without the high frequency of OSS failures.</p> <p>Rick, the other possible avenue for debugging is to disable the 1MB blocksize tunable on one or more of your OST datasets, and see if this correlates to a reduction or elimination of the occurrence of this failure. This is one of the main deltas between your ZFS environment and other ZFS users, so this would allow us to isolate the memory corruption to the code handling 1MB blocksize.</p> <p><a href="http://review.whamcloud.com/#/c/14926/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14926/</a> has been updated to add more remote/local buffer check.</p> <p>We've scheduled a maintenance window for Sep. 8 to roll out this latest patch after testing.</p> <p>Andreas, I'll consider changing the recordsize on some of the OSTs. The most likely scenario where we get solid information from this is if the LBUG is still hit on one of the OSSes with the changed setting. I am being a little cautious considering this since it will mean having a ZFS dataset with varying recordsizes. I don't believe the ZFS layer will care, but it's not something I've dealt with before.</p> <p>Hi Rick, any news on this front? Have you looked into upgrading to ZFS 0.6.5 to get the native large block support? The patch <a href="http://review.whamcloud.com/15127" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15127</a> "<a href="https://jira.whamcloud.com/browse/LU-4865" title="osd-zfs: increase object block size dynamically as object grows" class="issue-link" data-issue-key="LU-4865"><del>LU-4865</del></a> zfs: grow block size by write pattern" should also help performance when dealing with files under 1MB in size.</p> <p>Hi Andreas, since our last update to the code tree based on <a href="http://review.whamcloud.com/#/c/14926/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14926/</a> we've been stable. It's possible that we've pulled in a bugfix along with the debugging patch although I couldn't point to a specific one.</p> <p>We are looking at ZFS 0.6.5 to get away from the unreleased version for ZFS we've had to run. I would probably do that along with another rebase to a later unpatched tag of Lustre, maybe once <a href="https://jira.whamcloud.com/browse/LU-4865" title="osd-zfs: increase object block size dynamically as object grows" class="issue-link" data-issue-key="LU-4865"><del>LU-4865</del></a> is included.</p> <p>On a related note, I think this issue could be removed from the 2.8 blocker list, since we started with patched versions of Lustre and ZFS.</p> <p>Mike Pershin (mike.pershin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16685" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16685</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6584" title="OSS hit LBUG and crash" class="issue-link" data-issue-key="LU-6584"><del>LU-6584</del></a> osd: prevent int type overflow in osd_read_prep()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 687338302147dad5b09b964b8615a3b3adb78a7d</p> <p>It seems the reason of this issue is the int type overflow in lnb_rc. Instead of writing the (eof - file_offset) right into lnb_rc we have to check first it is not negative.</p> <p>Will SDSC be able to try this patch out to confirm whether it fixes the issues that they have been experiencing?</p> <p>Yes, we're scheduling a PM and push this out. Could this patch be related to <a href="https://jira.whamcloud.com/browse/LU-7106" title="Lustre client fail with error vvp_io.c:1081:vvp_io_commit_write()) even went there are space in OST and MDT" class="issue-link" data-issue-key="LU-7106"><del>LU-7106</del></a>? In other words could the current code create an error that propagates back to the client as ENOSPC even when there's capacity on the OST?</p> <p>Rick, this particular issue existed in IO READ code path and doesn't related to <a href="https://jira.whamcloud.com/browse/LU-7106" title="Lustre client fail with error vvp_io.c:1081:vvp_io_commit_write()) even went there are space in OST and MDT" class="issue-link" data-issue-key="LU-7106"><del>LU-7106</del></a>. I check OSD code quickly and didn't notice other similar issues at first glance.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16685/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16685/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6584" title="OSS hit LBUG and crash" class="issue-link" data-issue-key="LU-6584"><del>LU-6584</del></a> osd: prevent int type overflow in osd_read_prep()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: efe3842c76b8041a048457779554ffa5ba76567d</p> <p>Fix landed for 2.8. We'll reopen if this issue still is hit on Hyperion. If there is still an issue at SDSC and it is not, as hoped, a duplicate of this issue then please open a new ticket to track that issue.</p> Duplicate Related Development Rank 1|hzxctb: Rank (Obsolete) 9223372036854775807 Severity