https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1080 Lustre <p>Running on a real machine:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">$ mkfs.lustre --fsname=survey --mdt --index=0 /dev/sda3 $ mount -t lustre /dev/sda3 /mnt $ thrhi=64 file_count=200000 sh mds-survey </pre> </div></div> <p>then crash:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Build Version: jenkins-arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel-4610-g614 Lustre: Added LNI 10.45.1.8@tcp [8/256/0/180] Lustre: Accept all, port 988 LDISKFS-fs (sda3): recovery complete LDISKFS-fs (sda3): mounted filesystem with ordered data mode. Opts: LDISKFS-fs (sda3): mounted filesystem with ordered data mode. Opts: Lustre: MGC10.45.1.8@tcp: Reactivating <span class="code-keyword">import</span> Lustre: survey-MDT0000: used disk, loading Lustre: Echo OBD driver; http:<span class="code-comment">//www.lustre.org/ </span>LustreError: 1821:0:(echo_client.c:1810:echo_md_destroy_internal()) Can not unlink child tests: rc = -39 LustreError: 1823:0:(echo_client.c:1810:echo_md_destroy_internal()) Can not unlink child tests1: rc = -39 LustreError: 1831:0:(osd_handler.c:2294:osd_object_ref_del()) ASSERTION((oh)-&gt;ot_declare_ref_del &gt; 0) failed LustreError: 1831:0:(osd_handler.c:2294:osd_object_ref_del()) LBUG Pid: 1831, comm: lctl Call Trace: [&lt;ffffffffa038e855&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa038ee95&gt;] lbug_with_loc+0x75/0xe0 [libcfs] [&lt;ffffffffa0399d96&gt;] libcfs_assertion_failed+0x66/0x70 [libcfs] [&lt;ffffffffa0a1781a&gt;] osd_object_ref_del+0x14a/0x180 [osd_ldiskfs] [&lt;ffffffffa096ecbb&gt;] __mdd_ref_del+0x5b/0xa0 [mdd] [&lt;ffffffffa09777a2&gt;] mdd_create+0x1ae2/0x2470 [mdd] [&lt;ffffffffa051190d&gt;] ? htable_lookup+0xed/0x190 [obdclass] [&lt;ffffffffa041b5a9&gt;] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] [&lt;ffffffff81275894&gt;] ? vsnprintf+0x484/0x5f0 [&lt;ffffffffa0a6822b&gt;] echo_md_create_internal+0xab/0x4b0 [obdecho] [&lt;ffffffff81275a40&gt;] ? sprintf+0x40/0x50 [&lt;ffffffffa0a6ff40&gt;] echo_md_handler+0x1380/0x1dd0 [obdecho] [&lt;ffffffffa040d87e&gt;] ? cfs_mem_cache_free+0xe/0x10 [libcfs] [&lt;ffffffffa0a75ae6&gt;] echo_client_iocontrol+0x1c86/0x2a30 [obdecho] [&lt;ffffffff81127e77&gt;] ? ____pagevec_lru_add+0x167/0x180 [&lt;ffffffffa040da13&gt;] ? cfs_alloc+0x63/0x90 [libcfs] [&lt;ffffffffa04c0f52&gt;] ? obd_ioctl_getdata+0x172/0x1060 [obdclass] [&lt;ffffffffa04d6264&gt;] class_handle_ioctl+0x14d4/0x2340 [obdclass] [&lt;ffffffff8120d5df&gt;] ? security_inode_permission+0x1f/0x30 [&lt;ffffffffa04c0313&gt;] obd_class_ioctl+0x53/0x240 [obdclass] [&lt;ffffffff8100bc0e&gt;] ? apic_timer_interrupt+0xe/0x20 [&lt;ffffffff81189342&gt;] vfs_ioctl+0x22/0xa0 [&lt;ffffffff811894c9&gt;] ? do_vfs_ioctl+0x69/0x580 [&lt;ffffffff811894e4&gt;] do_vfs_ioctl+0x84/0x580 [&lt;ffffffff8100bc0e&gt;] ? apic_timer_interrupt+0xe/0x20 [&lt;ffffffff81189a61&gt;] sys_ioctl+0x81/0xa0 [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b Kernel panic - not syncing: LBUG Pid: 7014, comm: lctl Not tainted 2.6.32-220.el6_lustre.x86_64 #1 Call Trace: [&lt;ffffffff814ec701&gt;] ? panic+0x78/0x143 [&lt;ffffffffa040ceeb&gt;] ? lbug_with_loc+0xcb/0xe0 [libcfs] [&lt;ffffffffa0417d96&gt;] ? libcfs_assertion_failed+0x66/0x70 [libcfs] [&lt;ffffffffa0a1781a&gt;] ? osd_object_ref_del+0x14a/0x180 [osd_ldiskfs] [&lt;ffffffffa096ecbb&gt;] ? __mdd_ref_del+0x5b/0xa0 [mdd] [&lt;ffffffffa09777a2&gt;] ? mdd_create+0x1ae2/0x2470 [mdd] [&lt;ffffffffa051190d&gt;] ? htable_lookup+0xed/0x190 [obdclass] [&lt;ffffffffa041b5a9&gt;] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] [&lt;ffffffff81275894&gt;] ? vsnprintf+0x484/0x5f0 [&lt;ffffffffa0a6822b&gt;] ? echo_md_create_internal+0xab/0x4b0 [obdecho] [&lt;ffffffff81275a40&gt;] ? sprintf+0x40/0x50 [&lt;ffffffffa0a6ff40&gt;] ? echo_md_handler+0x1380/0x1dd0 [obdecho] [&lt;ffffffffa040d87e&gt;] ? cfs_mem_cache_free+0xe/0x10 [libcfs] [&lt;ffffffffa0a75ae6&gt;] ? echo_client_iocontrol+0x1c86/0x2a30 [obdecho] [&lt;ffffffff81127e77&gt;] ? ____pagevec_lru_add+0x167/0x180 [&lt;ffffffffa040da13&gt;] ? cfs_alloc+0x63/0x90 [libcfs] [&lt;ffffffffa04c0f52&gt;] ? obd_ioctl_getdata+0x172/0x1060 [obdclass] [&lt;ffffffffa04d6264&gt;] ? class_handle_ioctl+0x14d4/0x2340 [obdclass] [&lt;ffffffff8120d5df&gt;] ? security_inode_permission+0x1f/0x30 [&lt;ffffffffa04c0313&gt;] ? obd_class_ioctl+0x53/0x240 [obdclass] [&lt;ffffffff8100bc0e&gt;] ? apic_timer_interrupt+0xe/0x20 [&lt;ffffffff81189342&gt;] ? vfs_ioctl+0x22/0xa0 [&lt;ffffffff811894c9&gt;] ? do_vfs_ioctl+0x69/0x580 [&lt;ffffffff811894e4&gt;] ? do_vfs_ioctl+0x84/0x580 [&lt;ffffffff8100bc0e&gt;] ? apic_timer_interrupt+0xe/0x20 [&lt;ffffffff81189a61&gt;] ? sys_ioctl+0x81/0xa0 [&lt;ffffffff8100b0f2&gt;] ? system_call_fastpath+0x16/0x1b </pre> </div></div> Lustre Master LU-1080

mds-survey crash

Bug Blocker Resolved Cannot Reproduce Alex Zhuravlev Richard Henwood Wed, 8 Feb 2012 12:29:37 +0000 Tue, 21 Feb 2012 13:00:54 +0000 Tue, 21 Feb 2012 13:00:54 +0000 Lustre 2.2.0 Lustre 2.2.0 0 6 <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># rpm -qa | grep lustre lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g61f62a1.x86_64 kernel-2.6.32-131.17.1.el6_lustre.g60f4e35.x86_64 kernel-2.6.32-220.el6_lustre.x86_64 lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g61f62a1.x86_64 lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g61f62a1.x86_64 kernel-firmware-2.6.32-220.el6_lustre.x86_64 lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g61f62a1.x86_64 </pre> </div></div> <p>mds-survey and libecho are from:<br/> <a href="http://review.whamcloud.com/#change,1969,patchset=7" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1969,patchset=7</a></p> <p>It seems this is related with recent osd API change, instead of mds-survey bug. Richard, it seems mdd_create is failed somewhere? could you be able to find the debug log of LBUG.</p> <p>Assign this to Alex.</p> <p>Alex, do you have any ideas on how this might be fixed?</p> <p>sorry, still thinking how to solve this easily ...</p> <p>Since we can't declare undo operations, I've removed this LASSERT in <a href="https://jira.whamcloud.com/browse/LU-993" title="1.8&lt;-&gt;2.1.54 Test failure on test suite sanity :osd_attr_set()) ASSERTION((oh)-&gt;ot_declare_attr_set &gt; 0) failed" class="issue-link" data-issue-key="LU-993"><del>LU-993</del></a>(see ec20be97b9f977d3f4944523baaffb1bf95cf76c <a href="https://jira.whamcloud.com/browse/LU-993" title="1.8&lt;-&gt;2.1.54 Test failure on test suite sanity :osd_attr_set()) ASSERTION((oh)-&gt;ot_declare_attr_set &gt; 0) failed" class="issue-link" data-issue-key="LU-993"><del>LU-993</del></a> osd: code cleanup for directory nlink count), but I'm not sure why the echo create failed, is it a normal failure during the test?</p> <p>well, we can declare undo ops - that could be the easiest solution, but that results in more credits.</p> <p>btw, any idea why <span class="error">&#91;DTO_INDEX_DELETE&#93;</span> = 16 ? ldiskfs never shrinks dir nor it updates neighbor blocks during entry removal<br/> nor it changes quota usage. I'd think 1 should be enough?</p> <blockquote> <p>btw, any idea why <span class="error">&#91;DTO_INDEX_DELETE&#93;</span> = 16 ? ldiskfs never shrinks dir nor it updates neighbor blocks during entry removal<br/> nor it changes quota usage. I'd think 1 should be enough?</p></blockquote> <p>I have no idea why it was 16. You are the ext3/4 expert, I believe you are right <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>, 1 block could be enough.</p> <p>I'm beginning to think that if deletion consts 1 credit, then 1..2 more credits to set nlink.<br/> so 2..3 additional credits for undo path won't hurt us, probably? at least in mdd_create() case.<br/> another (enormous) case is mdd_rename() - it'll take mode credits for undo, but probably still<br/> acceptable. and at some point we're going to change the approach to be more object-based than<br/> just summing ops.</p> <p>Andreas/Johann</p> <p>Could one of you please comment on this 2.2 blocker?</p> <p>Thanks</p> <p>Peter</p> <p>FYI: I have been running a more recent Lustre and I have not been able to reproduce this issue.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># rpm -qa | grep lustre lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.gfd1c51d.x86_64_g0204171.x86_64 kernel-2.6.32-220.el6_lustre.gfd1c51d.x86_64 lustre-modules-2.1.55-2.6.32_220.el6_lustre.gfd1c51d.x86_64_g0204171.x86_64 kernel-firmware-2.6.32-220.el6_lustre.gfd1c51d.x86_64 lustre-2.1.55-2.6.32_220.el6_lustre.gfd1c51d.x86_64_g0204171.x86_64 </pre> </div></div> <p>These rpms are from build 480:<br/> <a href="http://build.whamcloud.com/job/lustre-master/480/arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/480/arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/</a></p> <p><a href="https://jira.whamcloud.com/browse/LU-1082" title="add mds-survey to lustre-tests" class="issue-link" data-issue-key="LU-1082"><del>LU-1082</del></a> is tracking the test which will run mds-survey during normal testing ensure it keeps working.</p> Related LU-1082 Development Rank 1|hzvhef: Rank (Obsolete) 6468 Severity