https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4252 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/03808e92-49af-11e3-9f02-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/03808e92-49af-11e3-9f02-52540035b04c</a>.</p> <p>The sub-test test_1 failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>test log shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== racer test 1: racer on clients: client-16vm1,client-16vm2 DURATION=900 ============================ 04:13:57 (1383999237) racers pids: 28586 28587 CMD: client-16vm1,client-16vm2 DURATION=900 MDSCOUNT=1 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre2/racer CMD: client-16vm1,client-16vm2 DURATION=900 MDSCOUNT=1 /usr/lib64/lustre/tests/racer/racer.sh /mnt/lustre/racer client-16vm2: file_create.sh: no process found client-16vm2: dir_create.sh: no process found client-16vm2: file_rm.sh: no process found client-16vm2: file_rename.sh: no process found client-16vm2: file_link.sh: no process found client-16vm2: file_symlink.sh: no process found client-16vm2: file_list.sh: no process found client-16vm2: file_create.sh: no process found client-16vm2: file_create.sh: no process found client-16vm2: dir_create.sh: no process found client-16vm2: file_rm.sh: no process found client-16vm2: dir_create.sh: no process found client-16vm2: file_rename.sh: no process found client-16vm2: file_rm.sh: no process found client-16vm2: file_link.sh: no process found client-16vm2: file_rename.sh: no process found client-16vm2: file_symlink.sh: no process found client-16vm2: file_link.sh: no process found client-16vm2: file_list.sh: no process found client-16vm2: file_symlink.sh: no process found client-16vm2: file_concat.sh: no process found client-16vm2: file_list.sh: no process found client-16vm2: file_exec.sh: no process found client-16vm2: file_concat.sh: no process found client-16vm2: file_exec.sh: no process found </pre> </div></div> lustre-master build # 1753 ldiskfs <br/> client is running SLES11 SP3 LU-4252

Failure on test suite racer test_1

Bug Blocker Resolved Fixed Yang Sheng Maloo Thu, 14 Nov 2013 18:43:41 +0000 Fri, 15 Nov 2019 20:51:15 +0000 Thu, 20 Mar 2014 12:42:57 +0000 Lustre 2.6.0 Lustre 2.5.1 Lustre 2.6.0 Lustre 2.5.2 0 8 <p>John,<br/> Could you please take a look if this is different that the many other racer bugs we have? There are quite a few and I don't know the subtleties of each one. </p> <p>Sorry, I can't tell from what's on maloo. It looks like a client rebooted but there are no console files for either client.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/5/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/5/</a><br/> Distro/Arch: RHEL6.4/x86_64(server), SLES11SP3/x86_64(client)</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/2dbe5d7a-7640-11e3-b3c0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2dbe5d7a-7640-11e3-b3c0-52540035b04c</a></p> <p>Two threads that appear to be deadlocked:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[34058.700432] ldlm_elt        D ffff880062ffe010     0 32149      2 0x00000000 [34058.700432]  ffff880062fff9f8 0000000000000046 ffff880062ffe010 0000000000011800 [34058.700432]  0000000000011800 0000000000011800 0000000000011800 ffff880062ffffd8 [34058.700432]  ffff880062ffffd8 0000000000011800 ffff88006b7103c0 ffff8800772e8140 [34058.700432] Call Trace: [34058.700432]  [&lt;ffffffff8145fda8&gt;] __mutex_lock_slowpath+0xf8/0x150 [34058.700432]  [&lt;ffffffff8145f83a&gt;] mutex_lock+0x1a/0x40 [34058.700432]  [&lt;ffffffffa062cfc5&gt;] ldiskfs_sync_file+0xc5/0x360 [ldiskfs] [34058.700432]  [&lt;ffffffffa0e66b9c&gt;] osd_object_sync+0x11c/0x150 [osd_ldiskfs] [34058.700432]  [&lt;ffffffffa0ee8590&gt;] ofd_sync+0x540/0x930 [ofd] [34058.700432]  [&lt;ffffffffa0dd802f&gt;] obd_sync+0xdf/0x240 [ost] [34058.700432]  [&lt;ffffffffa0de72aa&gt;] ost_blocking_ast+0x48a/0xee0 [ost] [34058.700432]  [&lt;ffffffffa09d910f&gt;] ldlm_cancel_callback+0x5f/0x180 [ptlrpc] [34058.700432]  [&lt;ffffffffa09d9284&gt;] ldlm_lock_cancel+0x54/0x1e0 [ptlrpc] [34058.700432]  [&lt;ffffffffa09da1ef&gt;] ldlm_cancel_locks_for_export_cb+0x8f/0x200 [ptlrpc] [34058.700432]  [&lt;ffffffffa06b3ba8&gt;] cfs_hash_for_each_relax+0x178/0x340 [libcfs] [34058.700432]  [&lt;ffffffffa06b5cae&gt;] cfs_hash_for_each_empty+0x10e/0x1f0 [libcfs] [34058.700432]  [&lt;ffffffffa09d59f6&gt;] ldlm_cancel_locks_for_export+0x26/0x30 [ptlrpc] [34058.700432]  [&lt;ffffffffa09e6bad&gt;] server_disconnect_export+0x5d/0x1a0 [ptlrpc] [34058.700432]  [&lt;ffffffffa0ede6c0&gt;] ofd_obd_disconnect+0x60/0x1f0 [ofd] [34058.700432]  [&lt;ffffffffa07b2528&gt;] class_fail_export+0x118/0x580 [obdclass] [34058.700432]  [&lt;ffffffffa09ff52f&gt;] expired_lock_main+0x41f/0x880 [ptlrpc] [34058.700432]  [&lt;ffffffff81082966&gt;] kthread+0x96/0xa0 [34058.700432]  [&lt;ffffffff81469ee4&gt;] kernel_thread_helper+0x4/0x10 </pre> </div></div> <p>vs</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[34058.700432] ldlm_cn00_003   D ffff8800691ffa88     0  1252      2 0x00000000 [34058.700432]  ffff8800691ffa50 0000000000000046 ffff8800691fe010 0000000000011800 [34058.700432]  0000000000011800 0000000000011800 0000000000011800 ffff8800691fffd8 [34058.700432]  ffff8800691fffd8 0000000000011800 ffff88006e08a5c0 ffff88007bcae300 [34058.700432] Call Trace: [34058.700432]  [&lt;ffffffffa0612205&gt;] jbd2_log_wait_commit+0xb5/0x130 [jbd2] [34058.700432]  [&lt;ffffffffa062d0dd&gt;] ldiskfs_sync_file+0x1dd/0x360 [ldiskfs] [34058.700432]  [&lt;ffffffffa0e66b9c&gt;] osd_object_sync+0x11c/0x150 [osd_ldiskfs] [34058.700432]  [&lt;ffffffffa0ee8590&gt;] ofd_sync+0x540/0x930 [ofd] [34058.700432]  [&lt;ffffffffa0dd802f&gt;] obd_sync+0xdf/0x240 [ost] [34058.700432]  [&lt;ffffffffa0de72aa&gt;] ost_blocking_ast+0x48a/0xee0 [ost] [34058.700432]  [&lt;ffffffffa09d910f&gt;] ldlm_cancel_callback+0x5f/0x180 [ptlrpc] [34058.700432]  [&lt;ffffffffa09d9284&gt;] ldlm_lock_cancel+0x54/0x1e0 [ptlrpc] [34058.700432]  [&lt;ffffffffa09fcdcb&gt;] ldlm_request_cancel+0x25b/0x420 [ptlrpc] [34058.700432]  [&lt;ffffffffa09fd0d5&gt;] ldlm_handle_cancel+0x145/0x250 [ptlrpc] [34058.700432]  [&lt;ffffffffa0a02329&gt;] ldlm_cancel_handler+0x1e9/0x510 [ptlrpc] [34058.700432]  [&lt;ffffffffa0a32fcb&gt;] ptlrpc_server_handle_request+0x37b/0xcd0 [ptlrpc] [34058.700432]  [&lt;ffffffffa0a35627&gt;] ptlrpc_main+0x917/0xd70 [ptlrpc] [34058.700432]  [&lt;ffffffff81082966&gt;] kthread+0x96/0xa0 [34058.700432]  [&lt;ffffffff81469ee4&gt;] kernel_thread_helper+0x4/0x10 </pre> </div></div> <p>Hongchao</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>thread "ll_ost_io00_037" has started journal but can't get the "i_mutex" lock.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ll_ost_io00_037 D ffff880048206010 0 1448 2 0x00000000 [34058.700432] ffff880048207a38 0000000000000046 ffff880048206010 0000000000011800 [34058.700432] 0000000000011800 0000000000011800 0000000000011800 ffff880048207fd8 [34058.700432] ffff880048207fd8 0000000000011800 ffff880048204080 ffff880068ea6200 [34058.700432] Call Trace: [34058.700432] [&lt;ffffffff8145fda8&gt;] __mutex_lock_slowpath+0xf8/0x150 [34058.700432] [&lt;ffffffff8145f83a&gt;] mutex_lock+0x1a/0x40 [34058.700432] [&lt;ffffffffa0e83649&gt;] osd_punch+0x479/0x610 [osd_ldiskfs] [34058.700432] [&lt;ffffffffa0ef1bd1&gt;] ofd_object_punch+0x741/0x960 [ofd] [34058.700432] [&lt;ffffffffa0ee71ed&gt;] ofd_punch+0x5dd/0xb20 [ofd] [34058.700432] [&lt;ffffffffa0de493f&gt;] ost_punch+0x3af/0xa30 [ost] [34058.700432] [&lt;ffffffffa0de6b1e&gt;] ost_handle+0x1b5e/0x1e60 [ost] [34058.700432] [&lt;ffffffffa0a32fcb&gt;] ptlrpc_server_handle_request+0x37b/0xcd0 [ptlrpc] [34058.700432] [&lt;ffffffffa0a35627&gt;] ptlrpc_main+0x917/0xd70 [ptlrpc] [34058.700432] [&lt;ffffffff81082966&gt;] kthread+0x96/0xa0 [34058.700432] [&lt;ffffffff81469ee4&gt;] kernel_thread_helper+0x4/0x10 </pre> </div></div> <p>thread ldlm_cn00_003 got the "i_mutex" lock and start to commit the journal, but the above thread holding one journal handle,<br/> then the journal can't be committed. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[34058.700432] ldlm_cn00_003 D ffff8800691ffa88 0 1252 2 0x00000000 [34058.700432] ffff8800691ffa50 0000000000000046 ffff8800691fe010 0000000000011800 [34058.700432] 0000000000011800 0000000000011800 0000000000011800 ffff8800691fffd8 [34058.700432] ffff8800691fffd8 0000000000011800 ffff88006e08a5c0 ffff88007bcae300 [34058.700432] Call Trace: [34058.700432] [&lt;ffffffffa0612205&gt;] jbd2_log_wait_commit+0xb5/0x130 [jbd2] [34058.700432] [&lt;ffffffffa062d0dd&gt;] ldiskfs_sync_file+0x1dd/0x360 [ldiskfs] [34058.700432] [&lt;ffffffffa0e66b9c&gt;] osd_object_sync+0x11c/0x150 [osd_ldiskfs] [34058.700432] [&lt;ffffffffa0ee8590&gt;] ofd_sync+0x540/0x930 [ofd] [34058.700432] [&lt;ffffffffa0dd802f&gt;] obd_sync+0xdf/0x240 [ost] [34058.700432] [&lt;ffffffffa0de72aa&gt;] ost_blocking_ast+0x48a/0xee0 [ost] [34058.700432] [&lt;ffffffffa09d910f&gt;] ldlm_cancel_callback+0x5f/0x180 [ptlrpc] [34058.700432] [&lt;ffffffffa09d9284&gt;] ldlm_lock_cancel+0x54/0x1e0 [ptlrpc] [34058.700432] [&lt;ffffffffa09fcdcb&gt;] ldlm_request_cancel+0x25b/0x420 [ptlrpc] [34058.700432] [&lt;ffffffffa09fd0d5&gt;] ldlm_handle_cancel+0x145/0x250 [ptlrpc] [34058.700432] [&lt;ffffffffa0a02329&gt;] ldlm_cancel_handler+0x1e9/0x510 [ptlrpc] [34058.700432] [&lt;ffffffffa0a32fcb&gt;] ptlrpc_server_handle_request+0x37b/0xcd0 [ptlrpc] [34058.700432] [&lt;ffffffffa0a35627&gt;] ptlrpc_main+0x917/0xd70 [ptlrpc] [34058.700432] [&lt;ffffffff81082966&gt;] kthread+0x96/0xa0 [34058.700432] [&lt;ffffffff81469ee4&gt;] kernel_thread_helper+0x4/0x10 </pre> </div></div> <p>the patch is tracked at <a href="http://review.whamcloud.com/#/c/9644/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9644/</a></p> <p>after discussing with Yangsheng and Niu, this issue should be introduced by the patch <a href="http://review.whamcloud.com/#/c/7794/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7794/</a> in <a href="https://jira.whamcloud.com/browse/LU-3373" title="ldiskfs patches for FC19" class="issue-link" data-issue-key="LU-3373"><del>LU-3373</del></a></p> <p>Reassign to yangsheng to fix it.</p> <p>Landed for 2.6</p> <p>Port to b2_5: <a href="http://review.whamcloud.com/9742" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9742</a></p> Related LU-6446 LU-12977 Development Rank 1|hzw8uf: Rank (Obsolete) 11590 Severity