https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9156 Lustre <p><a href="https://testing.hpdd.intel.com/test_sessions/e072c616-f8f8-11e6-aac4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/e072c616-f8f8-11e6-aac4-5254006e85c2</a></p> <p>From Client 2 console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>05:47:50:[15560.679336] kworker/0:1 D 0000000000000000 0 6749 2 0x00000080 05:47:50:[15560.680977] Workqueue: events key_garbage_collector 05:47:50:[15560.682439] ffff88003604bc40 0000000000000046 ffff88007aafce70 ffff88003604bfd8 05:47:50:[15560.684148] ffff88003604bfd8 ffff88003604bfd8 ffff88007aafce70 ffff88003604bd80 05:47:50:[15560.685840] ffff88003604bd88 7fffffffffffffff ffff88007aafce70 0000000000000000 05:47:50:[15560.687529] Call Trace: 05:47:50:[15560.688780] [&lt;ffffffff8168b979&gt;] schedule+0x29/0x70 05:47:50:[15560.690227] [&lt;ffffffff816893c9&gt;] schedule_timeout+0x239/0x2d0 05:47:50:[15560.691732] [&lt;ffffffff811dc341&gt;] ? __slab_free+0x81/0x2f0 05:47:50:[15560.693210] [&lt;ffffffff810a5cdc&gt;] ? insert_work+0x4c/0xa0 05:47:50:[15560.694666] [&lt;ffffffff811dc341&gt;] ? __slab_free+0x81/0x2f0 05:47:50:[15560.696107] [&lt;ffffffff8168bd56&gt;] wait_for_completion+0x116/0x170 05:47:50:[15560.697604] [&lt;ffffffff810c4fd0&gt;] ? wake_up_state+0x20/0x20 05:47:50:[15560.699063] [&lt;ffffffff81138a30&gt;] ? __call_rcu+0x2c0/0x2c0 05:47:50:[15560.700518] [&lt;ffffffff810ad1bb&gt;] wait_rcu_gp+0x5b/0x80 05:47:50:[15560.701979] [&lt;ffffffff810ad140&gt;] ? ftrace_raw_output_rcu_utilization+0x50/0x50 05:47:50:[15560.703560] [&lt;ffffffff8113862b&gt;] synchronize_sched+0x3b/0x50 05:47:50:[15560.705020] [&lt;ffffffff8129b47d&gt;] key_garbage_collector+0x1cd/0x390 05:47:50:[15560.706513] [&lt;ffffffff810a805b&gt;] process_one_work+0x17b/0x470 05:47:50:[15560.707970] [&lt;ffffffff810a8e96&gt;] worker_thread+0x126/0x410 05:47:50:[15560.709415] [&lt;ffffffff810a8d70&gt;] ? rescuer_thread+0x460/0x460 05:47:50:[15560.710882] [&lt;ffffffff810b064f&gt;] kthread+0xcf/0xe0 05:47:50:[15560.712267] [&lt;ffffffff810b0580&gt;] ? kthread_create_on_node+0x140/0x140 05:47:50:[15560.713767] [&lt;ffffffff81696818&gt;] ret_from_fork+0x58/0x90 05:47:50:[15560.715157] [&lt;ffffffff810b0580&gt;] ? kthread_create_on_node+0x140/0x140 </pre> </div></div> trevis-36vm1/2/7/8, Full Group test, <br/> RHEL7.3/zfs, branch master, v2.9.53.19.gdad9ce4, b3534<br/> LU-9156

parallel-scale test_metabench: test failed to respond and timed out

Bug Minor Open Unresolved WC Triage James Casper Fri, 24 Feb 2017 18:23:22 +0000 Mon, 9 Oct 2017 22:09:59 +0000 Lustre 2.10.0 Lustre 2.11.0 0 4 <p>Jim, could you please elaborate on why this particular client stack is the important one?</p> <p>Have also seen "kworker D" with mmp and sanity-sec. Latest timeout with mmp:</p> <p>test_7: stops at mds umount:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Stopping /mnt/lustre-mds1failover (opts:) on trevis-44vm3 CMD: trevis-44vm3 umount -d /mnt/lustre-mds1failover </pre> </div></div> <p>A passing test_7 continues after the mds umount:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Stopping /mnt/lustre-mds1failover (opts:) on onyx-37vm3 CMD: onyx-37vm3 umount -d /mnt/lustre-mds1failover CMD: onyx-37vm3 lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' CMD: onyx-37vm4 grep -c /mnt/lustre-ost1failover' ' /proc/mounts Stopping /mnt/lustre-ost1failover (opts:) on onyx-37vm4 CMD: onyx-37vm4 umount -d /mnt/lustre-ost1failover CMD: onyx-37vm4 lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' CMD: onyx-37vm7 grep -c /mnt/lustre-mds1' ' /proc/mounts CMD: onyx-37vm7 lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' CMD: onyx-37vm8 grep -c /mnt/lustre-ost1' ' /proc/mounts CMD: onyx-37vm8 lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' Resetting fail_loc on all nodes...CMD: onyx-37vm1.onyx.hpdd.intel.com,onyx-37vm5,onyx-37vm6,onyx-37vm7,onyx-37vm8 lctl set_param -n fail_loc=0 fail_val=0 2&gt;/dev/null done. 22:35:01 (1485498901) waiting for onyx-37vm1.onyx.hpdd.intel.com network 5 secs ... 22:35:01 (1485498901) network interface is UP </pre> </div></div> <p>An Instance on EE 3.1.1.0 , b_ieel3_0, build# 185<br/> <a href="https://testing.hpdd.intel.com/test_sets/8d28da46-0798-11e7-8406-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8d28da46-0798-11e7-8406-5254006e85c2</a></p> <p>While testing patch <a href="https://review.whamcloud.com/25336" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25336</a> on master branch, many tests failed with the similar issue to this one.<br/> <a href="https://testing.hpdd.intel.com/test_sessions/c145c4a3-b193-4734-b4c0-1aec2522f09a" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/c145c4a3-b193-4734-b4c0-1aec2522f09a</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/6fdca020-ca43-42c8-9fc3-3ffac60c1ba1" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/6fdca020-ca43-42c8-9fc3-3ffac60c1ba1</a></p> <p>I did not find any abnormal messages on consoles but only this stack:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>kworker/0:0 D 0000000000000000 0 7535 2 0x00000080 Workqueue: events key_garbage_collector ffff88004e1f3c40 0000000000000046 ffff880058ec2f10 ffff88004e1f3fd8 ffff88004e1f3fd8 ffff88004e1f3fd8 ffff880058ec2f10 ffff88004e1f3d80 ffff88004e1f3d88 7fffffffffffffff ffff880058ec2f10 0000000000000000 Call Trace: [&lt;ffffffff8168c169&gt;] schedule+0x29/0x70 [&lt;ffffffff81689bc9&gt;] schedule_timeout+0x239/0x2c0 [&lt;ffffffff811dc861&gt;] ? __slab_free+0x81/0x2f0 [&lt;ffffffff810b1c17&gt;] ? wake_bit_function_rh+0x37/0x40 [&lt;ffffffff8168c546&gt;] wait_for_completion+0x116/0x170 [&lt;ffffffff810c54c0&gt;] ? wake_up_state+0x20/0x20 [&lt;ffffffff81138f20&gt;] ? __call_rcu+0x2c0/0x2c0 [&lt;ffffffff810ad5bb&gt;] wait_rcu_gp+0x5b/0x80 [&lt;ffffffff810ad540&gt;] ? ftrace_raw_output_rcu_utilization+0x50/0x50 [&lt;ffffffff81138b1b&gt;] synchronize_sched+0x3b/0x50 [&lt;ffffffff8129b9bd&gt;] key_garbage_collector+0x1cd/0x390 [&lt;ffffffff810a845b&gt;] process_one_work+0x17b/0x470 [&lt;ffffffff810a9296&gt;] worker_thread+0x126/0x410 [&lt;ffffffff810a9170&gt;] ? rescuer_thread+0x460/0x460 [&lt;ffffffff810b0a4f&gt;] kthread+0xcf/0xe0 [&lt;ffffffff810b0980&gt;] ? kthread_create_on_node+0x140/0x140 [&lt;ffffffff816970d8&gt;] ret_from_fork+0x58/0x90 [&lt;ffffffff810b0980&gt;] ? kthread_create_on_node+0x140/0x140 </pre> </div></div> <p>Still looking for guidance with this ticket. If the kworker hang is not the root cause, please point us in the right direction. This is the only indication we can find of something that might be wrong.</p> <p>2.9.57, b3575:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/0800ff00-8d4c-4627-878e-566e8a697c01" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/0800ff00-8d4c-4627-878e-566e8a697c01</a><br/> (Interop w/2.10 on server)</p> <p>+1 on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/401fc94c-9ef8-11e7-b778-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/401fc94c-9ef8-11e7-b778-5254006e85c2</a></p> Development Rank 1|hzz4w7: Rank (Obsolete) 9223372036854775807 Severity