Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-2829] Timeout on sanityn test_33a: zfs slow when commit_on_sharing enabled https://jira.whamcloud.com/browse/LU-2829 Lustre <p>This issue was created by maloo for sarah <sarah@whamcloud.com></p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/2b779918-7787-11e2-987d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2b779918-7787-11e2-987d-52540035b04c</a>.</p> <p>The sub-test test_33a failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> LU-2829

Timeout on sanityn test_33a: zfs slow when commit_on_sharing enabled

Bug Blocker Resolved Fixed Nathaniel Clark Maloo performance zfs Mon, 18 Feb 2013 20:08:15 +0000 Tue, 31 Dec 2013 15:54:00 +0000 Mon, 2 Dec 2013 18:38:20 +0000 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.4.2 Lustre 2.5.1 0 7 <p>Also the test reports<br/> test_33a <br/> Error: 'test failed to respond and timed out'<br/> Failure Rate: 100.00% of last 100 executions <span class="error">[all branches]</span> </p> <p>The systems look like they just waited around for an hours. </p> <p>The test log as a good bit of "/proc/fs/jbd2/mdt1:*/info: No such file or directory" in it. I am guessing the test got lost. I don't expect zfs to have jbd2 business. </p> <p><a href="http://review.whamcloud.com/5530" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5530</a> - ABANDONED</p> <p>Fix for 33a and 33b need to fix assumptions about device names for these tests to be useful even on autotests since they use dm devices.</p> <p>This particular failure is not actually related to zfs proc entries.</p> <p>One of the createmany processes (run on client1) seems to go out to lunch</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 17:43:26 client-19vm1 xinetd[5727]: START: mshell pid=4698 from=::ffff:10.10.4.220 Feb 14 17:43:26 client-19vm1 mrshd[4699]: root@client-19vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" MGSFSTYPE=zfs MDSFSTYPE=zfs OSTFSTYPE=zfs FSTYPE=zfs sh -c "createmany -o /mnt/lustre/d0.sanityn/d33-\$(hostname)-1/f- -r /mnt/lustre2/d0.sanityn/d33-\$(hostname)-1/f- 10000 > /dev/null 2>&1");echo XXRETCODE:$?' Feb 14 18:40:17 client-19vm1 xinetd[5727]: START: shell pid=28648 from=::ffff:10.10.4.220 Feb 14 18:40:17 client-19vm1 rshd[28650]: autotest@client-19vm1.lab.whamcloud.com as root: cmd='/home/autotest/.autotest/dynamic_bash/70180499028200+1360896014.64875' Feb 14 18:40:18 client-19vm1 kernel: SysRq : Show State ... </pre> </div></div> <p>OST has one process blocked on:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>18:41:13:txg_sync D 0000000000000000 0 2707 2 0x00000080 18:41:13: ffff88007c881bc0 0000000000000046 ffff88007cecf680 0000000000000086 18:41:13: ffff88007c881b70 ffff88007cfeda48 0000000000000001 ffff88007cfeda60 18:41:13: ffff8800796a9af8 ffff88007c881fd8 000000000000fb88 ffff8800796a9af8 18:41:13:Call Trace: 18:41:13: [<ffffffff81090b9e>] ? prepare_to_wait_exclusive+0x4e/0x80 18:41:13: [<ffffffffa016b5ac>] cv_wait_common+0x9c/0x1a0 [spl] 18:41:13: [<ffffffffa02d5160>] ? zio_execute+0x0/0xf0 [zfs] 18:41:13: [<ffffffff81090990>] ? autoremove_wake_function+0x0/0x40 18:41:13: [<ffffffffa016b6e3>] __cv_wait+0x13/0x20 [spl] 18:41:13: [<ffffffffa02d533b>] zio_wait+0xeb/0x160 [zfs] 18:41:13: [<ffffffffa026b807>] dsl_pool_sync+0x2a7/0x480 [zfs] 18:41:13: [<ffffffffa027e147>] spa_sync+0x397/0x9a0 [zfs] 18:41:13: [<ffffffffa028fd41>] txg_sync_thread+0x2c1/0x490 [zfs] 18:41:13: [<ffffffff810527f9>] ? set_user_nice+0xc9/0x130 18:41:13: [<ffffffffa028fa80>] ? txg_sync_thread+0x0/0x490 [zfs] 18:41:13: [<ffffffffa0164668>] thread_generic_wrapper+0x68/0x80 [spl] 18:41:13: [<ffffffffa0164600>] ? thread_generic_wrapper+0x0/0x80 [spl] 18:41:14: [<ffffffff81090626>] kthread+0x96/0xa0 18:41:14: [<ffffffff8100c0ca>] child_rip+0xa/0x20 18:41:14: [<ffffffff81090590>] ? kthread+0x0/0xa0 18:41:14: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20 </pre> </div></div> <p>This txg_sync stack trace looks very similar to the one posted in <a href="https://jira.whamcloud.com/browse/LU-2887" title="sanity-quota test_12a: slow due to ZFS VMs sharing single disk" class="issue-link" data-issue-key="LU-2887"><del>LU-2887</del></a></p> <p>This test will complete on ZFS but it takes an inordinate amount of time (usually slightly over 1 hour).</p> <p>zfs times:<br/> commit_on_sharing=0 ~58s<br/> commit_on_sharing=1 ~1300s</p> <p>ldiskfs times:<br/> commit_on_sharing=0 ~58s<br/> commit_on_sharing=1 ~240s</p> <p>Patch to make logging print fewer errors:<br/> <a href="http://review.whamcloud.com/5693" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5693</a></p> <p>Landed for 2.4</p> <p>The core issue isn't resolved. This could be de-prioritized because the test will EVENTUALLY pass, just not within a reasonable time.</p> <p>The speed is probably related to bug <a href="https://jira.whamcloud.com/browse/LU-2887" title="sanity-quota test_12a: slow due to ZFS VMs sharing single disk" class="issue-link" data-issue-key="LU-2887"><del>LU-2887</del></a> and ilk.</p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/29/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/29/</a></p> <p>With FSTYPE=zfs, sanityn test 33a cannot pass within 3600s:<br/> <a href="https://maloo.whamcloud.com/test_sets/3fbb7eca-0257-11e3-b384-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/3fbb7eca-0257-11e3-b384-52540035b04c</a></p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/44/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/44/</a> (2.4.1 RC1)<br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p>sanityn test 33a timed out with 18000s:<br/> <a href="https://maloo.whamcloud.com/test_sets/fbf13be0-1556-11e3-8938-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fbf13be0-1556-11e3-8938-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/46/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/46/</a><br/> FSTYPE=zfs</p> <p>sanityn test 33a timed out with 3600s:<br/> <a href="https://maloo.whamcloud.com/test_sets/cff9f9da-305b-11e3-b28a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/cff9f9da-305b-11e3-b28a-52540035b04c</a></p> <p>We'll see whether the timeout failure can disappear after TEI-790 is resolved.</p> <p>Even with OSTCOUNT=2, sanityn test 33a still timed out in 18000s:<br/> <a href="https://maloo.whamcloud.com/test_sets/70536e32-444f-11e3-8472-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/70536e32-444f-11e3-8472-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/fb7ca724-43ed-11e3-8599-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fb7ca724-43ed-11e3-8599-52540035b04c</a></p> <p><a href="http://review.whamcloud.com/8161" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8161</a></p> <p>The above patch is also needed on Lustre b2_4 and b2_5 branches.</p> <p>backport to b2_4<br/> <a href="http://review.whamcloud.com/8273" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8273</a></p> <p>8161 cherry-picks over to b2_5 just fine</p> <p>Landed post 2.5.51</p> <p>Patch landed on Lustre b2_4 branch.</p> <blockquote><p>backport to b2_4 <a href="http://review.whamcloud.com/8273" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8273</a></p></blockquote> <p>The failure still occurred:</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/58/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/58/</a><br/> Distro/Arch: RHEL6.4/x86_64</p> <p>FSTYPE=zfs<br/> MDSCOUNT=1<br/> MDSSIZE=2097152<br/> OSTCOUNT=2<br/> OSTSIZE=2097152</p> <p><a href="https://maloo.whamcloud.com/test_sets/0f0a6172-5605-11e3-8e94-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0f0a6172-5605-11e3-8e94-52540035b04c</a></p> <p>Patches to decrease the number of files again for sanityn test 33a and 33b:<br/> For master branch: <a href="http://review.whamcloud.com/8440" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8440</a><br/> For b2_4 branch: <a href="http://review.whamcloud.com/8439" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8439</a></p> <p>The above patches landed on master and b2_4 branches.</p> Duplicate LU-4251 Related LU-2887 Development Rank 1|hzvjd3: Rank (Obsolete) 6852 Severity