https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1121 Lustre <p>While running recovery-mds-scale with FLAVOR=OSS, it failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>==== Checking the clients loads AFTER failover -- failure NOT OK ost3 has failed over 1 times, and counting... sleeping 582 seconds ... tar: etc/selinux/targeted/modules/active/modules/sandbox.pp: Wrote only 4096 of 7168 bytes tar: Exiting with failure status due to previous errors Found the END_RUN_FILE file: /home/yujian/test_logs/end_run_file client-1-ib Client load failed on node client-1-ib client client-1-ib load stdout and debug files : /tmp/recovery-mds-scale.log_run_tar.sh-client-1-ib /tmp/recovery-mds-scale.log_run_tar.sh-client-1-ib.debug </pre> </div></div> <p>/tmp/recovery-mds-scale.log_run_tar.sh-client-1-ib:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>tar: etc/selinux/targeted/modules/active/modules/sandbox.pp: Wrote only 4096 of 7168 bytes tar: Exiting with failure status due to previous errors </pre> </div></div> <p>/tmp/recovery-mds-scale.log_run_tar.sh-client-1-ib.debug:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;~snip~&gt; 2012-02-18 22:30:41: tar run starting + mkdir -p /mnt/lustre/d0.tar-client-1-ib + cd /mnt/lustre/d0.tar-client-1-ib + wait 7567 + do_tar + tar cf - /etc + tar xf - + tee /tmp/recovery-mds-scale.log_run_tar.sh-client-1-ib tar: Removing leading `/' from member names + return 2 + RC=2 ++ grep 'exit delayed from previous errors' /tmp/recovery-mds-scale.log_run_tar.sh-client-1-ib + PREV_ERRORS= + true + '[' 2 -ne 0 -a '' -a '' ']' + '[' 2 -eq 0 ']' ++ date '+%F %H:%M:%S' + echoerr '2012-02-18 22:37:10: tar failed' + echo '2012-02-18 22:37:10: tar failed' 2012-02-18 22:37:10: tar failed &lt;~snip~&gt; </pre> </div></div> <p>Syslog on client node client-1-ib showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Feb 18 22:34:54 client-1 kernel: INFO: task flush-lustre-1:3510 blocked for more than 120 seconds. Feb 18 22:34:54 client-1 kernel: "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 18 22:34:54 client-1 kernel: flush-lustre- D 0000000000000000 0 3510 2 0x00000080 Feb 18 22:34:54 client-1 kernel: ffff8801f70e99a0 0000000000000046 ffff8801f70e9920 ffffffffa0942434 Feb 18 22:34:54 client-1 kernel: 0000000000000000 ffff880331d24980 ffff8801f70e9930 0000000000000000 Feb 18 22:34:54 client-1 kernel: ffff88027d12b0b8 ffff8801f70e9fd8 000000000000f4e8 ffff88027d12b0b8 Feb 18 22:34:54 client-1 kernel: Call Trace: Feb 18 22:34:54 client-1 kernel: [&lt;ffffffffa0942434&gt;] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs] Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff8109b809&gt;] ? ktime_get_ts+0xa9/0xe0 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff81110b10&gt;] ? sync_page+0x0/0x50 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff814ed1c3&gt;] io_schedule+0x73/0xc0 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff81110b4d&gt;] sync_page+0x3d/0x50 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff814eda2a&gt;] __wait_on_bit_lock+0x5a/0xc0 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff81110ae7&gt;] __lock_page+0x67/0x70 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff81090c30&gt;] ? wake_bit_function+0x0/0x50 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff81124c97&gt;] ? __writepage+0x17/0x40 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff811261f2&gt;] write_cache_pages+0x392/0x4a0 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff81052600&gt;] ? __dequeue_entity+0x30/0x50 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff81124c80&gt;] ? __writepage+0x0/0x40 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff8126a5c9&gt;] ? cpumask_next_and+0x29/0x50 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff81054754&gt;] ? find_busiest_group+0x244/0xb20 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff81126324&gt;] generic_writepages+0x24/0x30 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff81126351&gt;] do_writepages+0x21/0x40 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff811a046d&gt;] writeback_single_inode+0xdd/0x2c0 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff811a08ae&gt;] writeback_sb_inodes+0xce/0x180 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff811a0a0b&gt;] writeback_inodes_wb+0xab/0x1b0 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff811a0dab&gt;] wb_writeback+0x29b/0x3f0 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff814eca20&gt;] ? thread_return+0x4e/0x77e Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff8107cc02&gt;] ? del_timer_sync+0x22/0x30 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff811a1099&gt;] wb_do_writeback+0x199/0x240 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff811a11a3&gt;] bdi_writeback_task+0x63/0x1b0 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff81090ab7&gt;] ? bit_waitqueue+0x17/0xd0 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff81134d40&gt;] ? bdi_start_fn+0x0/0x100 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff81134dc6&gt;] bdi_start_fn+0x86/0x100 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff81134d40&gt;] ? bdi_start_fn+0x0/0x100 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff81090886&gt;] kthread+0x96/0xa0 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff810907f0&gt;] ? kthread+0x0/0xa0 Feb 18 22:34:54 client-1 kernel: [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/f3b4fe94-5af9-11e1-8801-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f3b4fe94-5af9-11e1-8801-5254004bbbd3</a></p> <p>Please refer to the attached recovery-oss-scale.1329633991.log.tar.bz2 for more logs.</p> <p>It seems this is issue <a href="https://jira.whamcloud.com/browse/LU-874" title="Client eviction on lock callback timeout " class="issue-link" data-issue-key="LU-874"><del>LU-874</del></a>.</p> <br/> Lustre Tag: v2_1_1_0_RC4<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/44/">http://build.whamcloud.com/job/lustre-b2_1/44/</a><br/> e2fsprogs Build: <a href="http://build.whamcloud.com/job/e2fsprogs-master/217/">http://build.whamcloud.com/job/e2fsprogs-master/217/</a><br/> Distro/Arch: RHEL6/x86_64 (kernel version: 2.6.32-220.el6)<br/> Network: IB (in-kernel OFED)<br/> ENABLE_QUOTA=yes<br/> FAILURE_MODE=HARD<br/> FLAVOR=OSS<br/> <br/> MGS/MDS Nodes: client-8-ib<br/> <br/> OSS Nodes: client-18-ib(active), client-19-ib(active)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\ /<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST1 (active in client-18-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST2 (active in client-19-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST3 (active in client-18-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST4 (active in client-19-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST5 (active in client-18-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST6 (active in client-19-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;client-9-ib(OST7)<br/> <br/> Client Nodes: client-[1,4,17],fat-amd-2,fat-intel-2<br/> <br/> Network Addresses:<br/> client-1-ib: 192.168.4.1<br/> client-4-ib: 192.168.4.4<br/> client-8-ib: 192.168.4.8<br/> client-9-ib: 192.168.4.9<br/> client-17-ib: 192.168.4.17<br/> client-18-ib: 192.168.4.18<br/> client-19-ib: 192.168.4.19<br/> fat-amd-2-ib: 192.168.4.133<br/> fat-intel-2-ib: 192.168.4.129<br/> LU-1121

recovery-mds-scale (FLAVOR=OSS): tar: Wrote only 4096 of 7168 bytes

Bug Blocker Closed Won't Fix Oleg Drokin Jian Yu Sun, 19 Feb 2012 08:39:03 +0000 Sun, 14 Aug 2016 17:18:01 +0000 Sun, 14 Aug 2016 17:18:01 +0000 Lustre 2.1.1 Lustre 2.1.3 0 5 <p>Hi Jay,<br/> Could you please take a look to see whether this is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-874" title="Client eviction on lock callback timeout " class="issue-link" data-issue-key="LU-874"><del>LU-874</del></a> or not? Thanks.</p> <p>Looking at the logs, I am not all that sure it's related to lu-874 after all.</p> <p>I don't see the affected filename ever being written too, so possibly the logs are too short to see the problem, or we just don't have the necessary debug level enabled to see where the write failed to look around that time and the tar output itself is not timed.</p> <p>Additionally the last part of client-1 log is pretty strange. It appear that after 22;36, where there is last cd into d0-tar... dir (after the failure was already registered), a period of very frequent pings to OSTs begin for some reason, no other activity happens.<br/> It continues for ~561 seconds, I am not really sure what is that part about.</p> <p>In my humble opinion, the stack trace is not related with the issue here, neither is it related to <a href="https://jira.whamcloud.com/browse/LU-874" title="Client eviction on lock callback timeout " class="issue-link" data-issue-key="LU-874"><del>LU-874</del></a>. The real issue is due to short write where 7168 bytes was supposed to be written but actually 4096 bytes was written. I didn't find any clue from logs. I suspect that there must exist issues with kms handling.</p> <p>The reason why there was no activity on client-1 after 22:36 is because the test loop exited due to failure on tar.</p> <p>Thanks Oleg and Jinshan for the analysis. I'm going to reproduce this issue and gather more logs. The current debug level is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@client-18-ib ~]# lctl get_param debug debug= super warning dlmtrace error emerg ha rpctrace vfstrace config console [root@client-18-ib ~]# lctl get_param subsystem_debug subsystem_debug= undefined mdc mds osc ost class log llite rpc filter echo ldlm lov lquota lmv sec gss mgc mgs fid fld [root@client-18-ib ~]# lctl get_param debug_mb debug_mb=48 </pre> </div></div> <p>Could you please suggest what extra debug code I should add to gather more logs? Thanks.</p> <p>Jinshan, please note the abnormal high rate of pngs on all osts after 22:36, it's unrelated to the bug at hand, but sill looks strange.</p> <p>YuJian, I imagine having vfstrace enabled would have been helpful, also somewhat increased debug buffer size.<br/> The kms tracking is done under D_INODE flag, though we usually don't run with it and I don't know how much it will impact testing.<br/> If you are loking to do extending testing, +vfstrace +inode +ha look like sensible options to me. Also a 1024 in debug_mb if you have the RAM. </p> <blockquote> <p>Could you please suggest what extra debug code I should add to gather more logs? Thanks.</p></blockquote> <p>On the client side, I always do the following:</p> <p>lctl set_param debug=-1<br/> lctl set_param debug=-trace</p> <p>In this way, I can get reasonable log and not need a super size debug memory. Just a hint <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>After I set the debug level and size as follows on all of the nodes:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>debug=-1 subsystem_debug=0xffb7e3ff debug_mb=200 debug=-trace </pre> </div></div> <p>Running recovery-mds-scale with FLAVOR=OSS hit the following LBUG on OSS client-18-ib:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 5227:0:(filter.c:4141:filter_destroy()) lustre-OST0003: can not find olg of group 0^M LustreError: 5227:0:(filter.c:3683:filter_handle_precreate()) ASSERTION(diff &gt;= 0) failed: lustre-OST0003: 163 - 225 = -62^M LustreError: 5227:0:(filter.c:3683:filter_handle_precreate()) LBUG^M Pid: 5227, comm: tgt_recov^M ^M Call Trace:^M [&lt;ffffffffa0636855&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs]^M [&lt;ffffffffa0636e95&gt;] lbug_with_loc+0x75/0xe0 [libcfs]^M [&lt;ffffffffa090ec4d&gt;] filter_create+0x160d/0x1640 [obdfilter]^M [&lt;ffffffffa0919b11&gt;] filter_preprw_write+0x6c1/0x1f10 [obdfilter]^M [&lt;ffffffffa083e87b&gt;] ? _debug_req+0x51b/0x660 [ptlrpc]^M [&lt;ffffffffa091c258&gt;] filter_preprw+0x68/0x90 [obdfilter]^M [&lt;ffffffffa07c110e&gt;] obd_preprw+0x11e/0x420 [ost]^M [&lt;ffffffffa07cb18c&gt;] ost_brw_write+0x98c/0x1870 [ost]^M [&lt;ffffffffa06416a1&gt;] ? libcfs_debug_vmsg2+0x4d1/0xb50 [libcfs]^M [&lt;ffffffffa083b264&gt;] ? lustre_msg_get_opc+0x94/0x100 [ptlrpc]^M [&lt;ffffffffa07cfce5&gt;] ost_handle+0x3325/0x4b90 [ost]^M [&lt;ffffffffa0b16b16&gt;] ? vvp_session_key_init+0xd6/0x1d0 [lustre]^M [&lt;ffffffffa07cc9c0&gt;] ? ost_handle+0x0/0x4b90 [ost]^M [&lt;ffffffffa0800cf6&gt;] handle_recovery_req+0x1f6/0x330 [ptlrpc]^M [&lt;ffffffffa0801257&gt;] target_recovery_thread+0x3a7/0xf50 [ptlrpc]^M [&lt;ffffffffa0800eb0&gt;] ? target_recovery_thread+0x0/0xf50 [ptlrpc]^M [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20^M [&lt;ffffffffa0800eb0&gt;] ? target_recovery_thread+0x0/0xf50 [ptlrpc]^M [&lt;ffffffffa0800eb0&gt;] ? target_recovery_thread+0x0/0xf50 [ptlrpc]^M [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20^M ^M Kernel panic - not syncing: LBUG^M Pid: 5227, comm: tgt_recov Not tainted 2.6.32-220.el6_lustre.g4554b65.x86_64 #1^M Call Trace:^M [&lt;ffffffff814ec661&gt;] ? panic+0x78/0x143^M [&lt;ffffffffa0636eeb&gt;] ? lbug_with_loc+0xcb/0xe0 [libcfs]^M [&lt;ffffffffa090ec4d&gt;] ? filter_create+0x160d/0x1640 [obdfilter]^M [&lt;ffffffffa0919b11&gt;] ? filter_preprw_write+0x6c1/0x1f10 [obdfilter]^M [&lt;ffffffffa083e87b&gt;] ? _debug_req+0x51b/0x660 [ptlrpc]^M [&lt;ffffffffa091c258&gt;] ? filter_preprw+0x68/0x90 [obdfilter]^M [&lt;ffffffffa07c110e&gt;] ? obd_preprw+0x11e/0x420 [ost]^M [&lt;ffffffffa07cb18c&gt;] ? ost_brw_write+0x98c/0x1870 [ost]^M [&lt;ffffffffa06416a1&gt;] ? libcfs_debug_vmsg2+0x4d1/0xb50 [libcfs]^M [&lt;ffffffffa083b264&gt;] ? lustre_msg_get_opc+0x94/0x100 [ptlrpc]^M [&lt;ffffffffa07cfce5&gt;] ? ost_handle+0x3325/0x4b90 [ost]^M [&lt;ffffffffa0b16b16&gt;] ? vvp_session_key_init+0xd6/0x1d0 [lustre]^M [&lt;ffffffffa07cc9c0&gt;] ? ost_handle+0x0/0x4b90 [ost]^M [&lt;ffffffffa0800cf6&gt;] ? handle_recovery_req+0x1f6/0x330 [ptlrpc]^M [&lt;ffffffffa0801257&gt;] ? target_recovery_thread+0x3a7/0xf50 [ptlrpc]^M [&lt;ffffffffa0800eb0&gt;] ? target_recovery_thread+0x0/0xf50 [ptlrpc]^M [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20^M [&lt;ffffffffa0800eb0&gt;] ? target_recovery_thread+0x0/0xf50 [ptlrpc]^M [&lt;ffffffffa0800eb0&gt;] ? target_recovery_thread+0x0/0xf50 [ptlrpc]^M [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20^M </pre> </div></div> <p>Please refer to /scratch/logs/2.1.1/recovery-oss-scale.1329896304.log.tar.bz2 on brent node for more logs.</p> <p>I'm disabling panic_on_lbug to get debug log...</p> <blockquote><p>I'm disabling panic_on_lbug to get debug log...</p></blockquote> <p>After disabling panic_on_lbug, I could not reproduce the above LBUG and the original issue of this ticket, but kept hitting the known issue: <a href="https://jira.whamcloud.com/browse/LU-463" title="orphan recovery happens too late, causing writes to fail with ENOENT after recovery" class="issue-link" data-issue-key="LU-463"><del>LU-463</del></a>.</p> <p>The second crash is now tracked under <a href="https://jira.whamcloud.com/browse/LU-1129" title="filter_handle_precreate()) ASSERTION(diff &gt;= 0) failed" class="issue-link" data-issue-key="LU-1129"><del>LU-1129</del></a></p> <p>Lustre Tag: v2_1_3_RC1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/113/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/113/</a><br/> Distro/Arch: RHEL6.3/x86_64 (kernel version: 2.6.32-279.2.1.el6)<br/> Network: IB (in-kernel OFED)<br/> ENABLE_QUOTA=yes<br/> FAILURE_MODE=HARD</p> <p>The original issue occurred again while running recovery-mds-scale failover_ost test:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>tar: etc/selinux/targeted/modules/active/modules/rhgb.pp: Wrote only 4096 of 7680 bytes tar: Exiting with failure status due to previous errors </pre> </div></div> <p><a href="https://maloo.whamcloud.com/test_sets/dc54205c-e534-11e1-ae4e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/dc54205c-e534-11e1-ae4e-52540035b04c</a></p> <p>After setting PTLDEBUG=-1 and DEBUG_SIZE=200 to reproduce the issue and gather more logs, I hit <a href="https://jira.whamcloud.com/browse/LU-463" title="orphan recovery happens too late, causing writes to fail with ENOENT after recovery" class="issue-link" data-issue-key="LU-463"><del>LU-463</del></a> again:<br/> <a href="https://maloo.whamcloud.com/test_sets/b18a1330-e5ad-11e1-ae4e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b18a1330-e5ad-11e1-ae4e-52540035b04c</a></p> <p>Really old blocker for unsupported version</p> Related LU-1129 Development Rank 1|hzv31r: Rank (Obsolete) 3993 Severity