https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1993 Lustre <p>Hit this LBUG when running racer on the OFED build</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: -----============= acceptance-small: racer ============----- Wed Sep 19 20:26:36 PDT 2012 Lustre: DEBUG MARKER: excepting tests: LustreError: 152-6: Ignoring deprecated mount option 'acl'. Lustre: Increasing default stripe size to min 1048576 Lustre: Mounted lustre-client LNet: 27947:0:(debug.c:324:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. LNet: 27947:0:(debug.c:324:libcfs_debug_str2mask()) Skipped 1 previous similar message Lustre: DEBUG MARKER: Using TIMEOUT=20 LNet: 29272:0:(debug.c:324:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. LNet: 29272:0:(debug.c:324:libcfs_debug_str2mask()) Skipped 1 previous similar message Lustre: DEBUG MARKER: == racer test 1: racer on clients: client-5-ib,client-17-ib DURATION=900 == 20:27:22 (1348111642) LustreError: 18092:0:(file.c:2328:ll_inode_revalidate_fini()) failure -116 inode 144115406632653105 LustreError: 8396:0:(file.c:626:ll_file_open()) ASSERTION( it_disposition(it, 0x00800000) ) failed: LustreError: 8396:0:(file.c:626:ll_file_open()) LBUG Pid: 8396, comm: ls Call Trace: [&lt;ffffffffa0c19905&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Message from [&lt;ffffffffa0c19f17&gt;] lbug_with_loc+0x47/0xb0 [libcfs] syslogd@client- [&lt;ffffffffa0af04e0&gt;] ll_lov_recreate.clone.0+0x0/0xdd0 [lustre] 5 at Sep 19 20:3 [&lt;ffffffffa0c25048&gt;] ? libcfs_log_return+0x28/0x40 [libcfs] 0:46 ... kern [&lt;ffffffffa0ad75d0&gt;] ? ll_dir_open+0x0/0xf0 [lustre] el:LustreError: [&lt;ffffffffa0ad7681&gt;] ll_dir_open+0xb1/0xf0 [lustre] 8396:0:(file.c:6 [&lt;ffffffff811789ba&gt;] __dentry_open+0x10a/0x360 26:ll_file_open( [&lt;ffffffff8121439f&gt;] ? security_inode_permission+0x1f/0x30 )) ASSERTION( it [&lt;ffffffff81178d24&gt;] nameidata_to_filp+0x54/0x70 _disposition(it, [&lt;ffffffff8118c230&gt;] do_filp_open+0x6c0/0xd60 0x00800000) ) f [&lt;ffffffff8119a460&gt;] ? mntput_no_expire+0x30/0x110 ailed: Me [&lt;ffffffff811982b2&gt;] ? alloc_fd+0x92/0x160 ssage from syslo [&lt;ffffffff81178769&gt;] do_sys_open+0x69/0x140 gd@client-5 at S [&lt;ffffffff81178880&gt;] sys_open+0x20/0x30 ep 19 20:30:46 . [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b .. kernel:Lus treError: 8396:0:(file.c:626:ll_file_open()) LBUG Kernel panic - not syncing: LBUG Pid: 8396, comm: ls Not tainted 2.6.32-279.5.1.el6.x86_64 #1 Call Trace: [&lt;ffffffff814fd24a&gt;] ? panic+0xa0/0x168 [&lt;ffffffffa0c19f6b&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [&lt;ffffffffa0af04e0&gt;] ? ll_lov_recreate.clone.0+0x0/0xdd0 [lustre] [&lt;ffffffffa0c25048&gt;] ? libcfs_log_return+0x28/0x40 [libcfs] [&lt;ffffffffa0ad75d0&gt;] ? ll_dir_open+0x0/0xf0 [lustre] [&lt;ffffffffa0ad7681&gt;] ? ll_dir_open+0xb1/0xf0 [lustre] [&lt;ffffffff811789ba&gt;] ? __dentry_open+0x10a/0x360 [&lt;ffffffff8121439f&gt;] ? security_inode_permission+0x1f/0x30 [&lt;ffffffff81178d24&gt;] ? nameidata_to_filp+0x54/0x70 [&lt;ffffffff8118c230&gt;] ? do_filp_open+0x6c0/0xd60 [&lt;ffffffff8119a460&gt;] ? mntput_no_expire+0x30/0x110 [&lt;ffffffff811982b2&gt;] ? alloc_fd+0x92/0x160 [&lt;ffffffff81178769&gt;] ? do_sys_open+0x69/0x140 [&lt;ffffffff81178880&gt;] ? sys_open+0x20/0x30 [&lt;ffffffff8100b0f2&gt;] ? system_call_fastpath+0x16/0x1b Initializing cgroup subsys cpuset Initializing cgroup subsys cpu </pre> </div></div> server/client b2_3-tag-2.2.96 RHEL6 external OFED build LU-1993

racer test: ASSERTION( it_disposition(it, 0x00800000) ) failed

Bug Minor Resolved Fixed Bob Glossman Sarah Liu Thu, 20 Sep 2012 00:20:45 +0000 Mon, 3 Mar 2014 21:58:26 +0000 Mon, 3 Mar 2014 21:58:26 +0000 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.6.0 0 6 <p>Bob will look into this one</p> <p>I tried running racer alone and the whole sequence, cannot reprode it. I think it's ok to drop the priority since it's not reproducible</p> <p>Lustre: DEBUG MARKER: == racer test 1: racer on clients: client-5-ib,client-17-ib DURATION=900 == 20:30:06 (1348284606)<br/> LustreError: 15339:0:(file.c:2328:ll_inode_revalidate_fini()) failure -2 inode 144115406615943713<br/> LustreError: 2684:0:(file.c:2328:ll_inode_revalidate_fini()) failure -2 inode 144115406615946176<br/> LustreError: 19085:0:(file.c:2328:ll_inode_revalidate_fini()) failure -2 inode 144115406666237683<br/> Lustre: DEBUG MARKER: == racer racer.sh test complete, duration 1038 sec == 20:45:44 (1348285544)<br/> LustreError: 6300:0:(ldlm_request.c:1166:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway<br/> LustreError: 6300:0:(ldlm_request.c:1166:ldlm_cli_cancel_req()) Skipped 1 previous similar message<br/> LustreError: 6300:0:(ldlm_request.c:1792:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108<br/> LustreError: 6300:0:(ldlm_request.c:1792:ldlm_cli_cancel_list()) Skipped 1 previous similar message<br/> Lustre: Unmounted lustre-client<br/> racer returned 0</p> <p>Agreed</p> <p>Lustre Tag: v2_3_0_RC6<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_3/41" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/41</a><br/> Distro/Arch: RHEL6.3/x86_64(server), SLES11SP1/x86_64(client)<br/> Network: TCP</p> <p>The same issue occurred again:<br/> <a href="https://maloo.whamcloud.com/test_sets/ed87458a-1bc5-11e2-9d79-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ed87458a-1bc5-11e2-9d79-52540035b04c</a></p> <p>Syslog on Client 2 (client-28vm2) showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 20 19:00:43 client-28vm2 kernel: [20814.731833] Lustre: DEBUG MARKER: DURATION=900 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre2/racer Oct 20 19:00:43 client-28vm2 kernel: [20814.738668] Lustre: DEBUG MARKER: DURATION=900 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre/racer Oct 20 19:00:43 client-28vm2 in.rshd[7495]: connect from 10.10.4.173 (10.10.4.173) Oct 20 19:00:48 client-28vm2 rshd[7495]: pam_rhosts(rsh:auth): allowed access to root@client-28vm2.lab.whamcloud.com as root Oct 20 19:00:48 client-28vm2 in.rshd[7500]: connect from 10.10.4.173 (10.10.4.173) Oct 20 19:00:48 client-28vm2 rshd[7501]: root@client-28vm2.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= MGSFSTYPE=ldiskfs MDSFSTYPE=ldiskfs OSTFSTYPE=ldiskfs FSTYPE=ldiskfs sh -c "DURATION=900 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre2/racer ");echo XXRETCODE:$?' Oct 20 19:00:48 client-28vm2 rshd[7500]: pam_rhosts(rsh:auth): allowed access to root@client-28vm2.lab.whamcloud.com as root Oct 20 19:00:48 client-28vm2 rshd[7508]: root@client-28vm2.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= MGSFSTYPE=ldiskfs MDSFSTYPE=ldiskfs OSTFSTYPE=ldiskfs FSTYPE=ldiskfs sh -c "DURATION=900 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre/racer ");echo XXRETCODE:$?' Oct 20 19:06:53 client-28vm2 kernel: [21184.731271] LustreError: 19933:0:(file.c:626:ll_file_open()) ASSERTION( it_disposition(it, 0x00800000) ) failed: </pre> </div></div> <p>The back trace was not gathered somehow.</p> <p>This is likely related to <a href="https://jira.whamcloud.com/browse/LU-2275" title="Open request leak" class="issue-link" data-issue-key="LU-2275"><del>LU-2275</del></a> (mdt side changes) (note the patch referenced there does not fully fix this particular assert yet).</p> <p>another failure seen in 2.4 testing<br/> <a href="https://maloo.whamcloud.com/test_sets/badce3ee-3d6a-11e2-b377-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/badce3ee-3d6a-11e2-b377-52540035b04c</a></p> <p>Hit this on my local test system after about 12 minutes of racer testing on v2_3_64_0-93-g441ae91:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> LustreError: 22208:0:(file.c:640:ll_file_open()) ASSERTION( it_disposition(it, 0x00800000) ) failed: Pid: 22208, comm: ls Call Trace: [&lt;ffffffffa0ea8905&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa0ea8f07&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0f946e3&gt;] ll_file_open+0xc23/0xc30 [lustre] [&lt;ffffffffa0f75d40&gt;] ? ll_dir_open+0x0/0xf0 [lustre] [&lt;ffffffffa0f75e1b&gt;] ll_dir_open+0xdb/0xf0 [lustre] [&lt;ffffffff811789ba&gt;] __dentry_open+0x10a/0x360 [&lt;ffffffff8121443f&gt;] ? security_inode_permission+0x1f/0x30 [&lt;ffffffff81178d24&gt;] nameidata_to_filp+0x54/0x70 [&lt;ffffffff8118c230&gt;] do_filp_open+0x6c0/0xd60 [&lt;ffffffff81178769&gt;] do_sys_open+0x69/0x140 [&lt;ffffffff81178880&gt;] sys_open+0x20/0x30 </pre> </div></div> <p>I pushed <a href="http://review.whamcloud.com/6250" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6250</a> with an improvement to the LASSERT() message, and enabling full debug during racer test runs.</p> <p>Patch landed to Master. Please reopen ticket or create a new ticket if more work is needed.</p> Development Rank 1|hzvq3j: Rank (Obsolete) 8101 Severity