https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4177 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/861fabce-40a6-11e3-a5c2-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/861fabce-40a6-11e3-a5c2-52540035b04c</a>.</p> <p>The sub-test test_iorssf failed with the following error:</p> <blockquote> <p>test_iorssf returned 1</p></blockquote> <p>client dmesg shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == parallel-scale-nfsv3 test iorssf: iorssf == 01:40:35 (1383036035) INFO: task IOR:13356 blocked for more than 120 seconds. "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. IOR D 0000000000000000 0 13356 13352 0x00000080 ffff880324b4bc88 0000000000000082 0000000000000000 ffff880324b4bc4c 0000000000000000 ffff88033fc24300 ffff880032e76700 0000000000000400 ffff880323f4c638 ffff880324b4bfd8 000000000000fb88 ffff880323f4c638 Call Trace: [&lt;ffffffff8150f78e&gt;] __mutex_lock_slowpath+0x13e/0x180 [&lt;ffffffff8150f62b&gt;] mutex_lock+0x2b/0x50 [&lt;ffffffff8111c4b1&gt;] generic_file_aio_write+0x71/0x100 [&lt;ffffffff8100b9ce&gt;] ? common_interrupt+0xe/0x13 [&lt;ffffffffa0386f8e&gt;] nfs_file_write+0xde/0x1f0 [nfs] [&lt;ffffffff8118106a&gt;] do_sync_write+0xfa/0x140 [&lt;ffffffff81096da0&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffff8100bb8e&gt;] ? apic_timer_interrupt+0xe/0x20 [&lt;ffffffff8121bed6&gt;] ? security_file_permission+0x16/0x20 [&lt;ffffffff81181368&gt;] vfs_write+0xb8/0x1a0 [&lt;ffffffff81181c61&gt;] sys_write+0x51/0x90 [&lt;ffffffff810dc685&gt;] ? __audit_syscall_exit+0x265/0x290 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b Lustre: DEBUG MARKER: parallel-scale-nfsv3 test_iorssf: @@@@@@ FAIL: test_iorssf failed with 1 </pre> </div></div> client and server: lustre-b2_5 build #2 RHEL6 ldiskfs LU-4177

Failure on test suite parallel-scale-nfsv3 test_iorssf

Bug Minor Resolved Duplicate Di Wang Maloo dne Tue, 29 Oct 2013 14:39:25 +0000 Wed, 8 Jan 2014 18:56:06 +0000 Wed, 30 Oct 2013 05:22:31 +0000 Lustre 2.5.0 0 4 <p>same error seen in parallel-scale-nfsv4 test_iorssf:</p> <p><a href="https://maloo.whamcloud.com/test_sets/01509f4c-40a7-11e3-a5c2-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/01509f4c-40a7-11e3-a5c2-52540035b04c</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == parallel-scale-nfsv4 test iorssf: iorssf == 02:27:02 (1383038822) INFO: task IOR:17809 blocked for more than 120 seconds. "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. IOR D 0000000000000002 0 17809 17806 0x00000080 ffff8803293e9c88 0000000000000086 0000000000000000 ffff880032e36700 0000000000000002 0000000000000001 ffff8803293e9c28 ffffffff81065d54 ffff88032cfb1058 ffff8803293e9fd8 000000000000fb88 ffff88032cfb1058 Call Trace: [&lt;ffffffff81065d54&gt;] ? enqueue_task_fair+0x64/0x100 [&lt;ffffffff8150f78e&gt;] __mutex_lock_slowpath+0x13e/0x180 [&lt;ffffffff8150f62b&gt;] mutex_lock+0x2b/0x50 [&lt;ffffffff8111c4b1&gt;] generic_file_aio_write+0x71/0x100 [&lt;ffffffffa0386f8e&gt;] nfs_file_write+0xde/0x1f0 [nfs] [&lt;ffffffff8118106a&gt;] do_sync_write+0xfa/0x140 [&lt;ffffffff81096da0&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffff8121bed6&gt;] ? security_file_permission+0x16/0x20 [&lt;ffffffff81181368&gt;] vfs_write+0xb8/0x1a0 [&lt;ffffffff81181c61&gt;] sys_write+0x51/0x90 [&lt;ffffffff810dc685&gt;] ? __audit_syscall_exit+0x265/0x290 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b INFO: task IOR:17810 blocked for more than 120 seconds. "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. IOR D 0000000000000002 0 17810 17806 0x00000080 ffff88032bb81c88 0000000000000086 0000000000000000 ffff88032bb81c88 0000000000000000 0000000000000000 00000000ffffffff ffff88032bb80000 ffff880334f01ab8 ffff88032bb81fd8 000000000000fb88 ffff880334f01ab8 Call Trace: [&lt;ffffffff8150e18a&gt;] ? thread_return+0xa8/0x76e [&lt;ffffffff8150f78e&gt;] __mutex_lock_slowpath+0x13e/0x180 [&lt;ffffffff8150f62b&gt;] mutex_lock+0x2b/0x50 [&lt;ffffffff8111c4b1&gt;] generic_file_aio_write+0x71/0x100 [&lt;ffffffffa0386f8e&gt;] nfs_file_write+0xde/0x1f0 [nfs] [&lt;ffffffff8118106a&gt;] do_sync_write+0xfa/0x140 [&lt;ffffffff81096da0&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffff811c419b&gt;] ? inotify_inode_queue_event+0xeb/0x120 [&lt;ffffffff8100bb8e&gt;] ? apic_timer_interrupt+0xe/0x20 [&lt;ffffffff8121bed6&gt;] ? security_file_permission+0x16/0x20 [&lt;ffffffff81181368&gt;] vfs_write+0xb8/0x1a0 [&lt;ffffffff8117f8fa&gt;] ? vfs_llseek+0x2a/0x40 [&lt;ffffffff81181c61&gt;] sys_write+0x51/0x90 [&lt;ffffffff810dc685&gt;] ? __audit_syscall_exit+0x265/0x290 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Di,<br/> Could you please have a look and comment on this one?</p> <p>Path: /mnt/lustre/d0.ior.ssf<br/> FS: 28.2 GiB Used FS: 4.4% Inodes: 1.8 Mi Used Inodes: 0.0%<br/> Participating tasks: 4</p> <p>Summary:<br/> api = POSIX<br/> test filename = /mnt/lustre/d0.ior.ssf/iorData<br/> access = single-shared-file<br/> pattern = segmented (1 segment)<br/> ordering in a file = sequential offsets<br/> ordering inter file= no tasks offsets<br/> clients = 4 (2 per node)<br/> repetitions = 1<br/> xfersize = 2 MiB<br/> blocksize = 3 GiB<br/> aggregate filesize = 12 GiB</p> <p>Commencing write performance test.<br/> Tue Oct 29 01:40:36 2013</p> <p>Commencing read performance test.<br/> Tue Oct 29 01:52:55 2013</p> <p>Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s) Op grep #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize</p> <p>--------- --------- --------- ---------- ------- --------- --------- ---------- ------- --------<br/> write 16.64 16.64 16.64 0.00 8.32 8.32 8.32 0.00 738.47005 4 2 1 0 0 1 0 0 1 3221225472 2097152 12884901888 -1 POSIX EXCEL<br/> read 177.79 177.79 177.79 0.00 88.89 88.89 88.89 0.00 69.11676 4 2 1 0 0 1 0 0 1 3221225472 2097152 12884901888 -1 POSIX EXCEL</p> <p>Max Write: 16.64 MiB/sec (17.45 MB/sec)<br/> Max Read: 177.79 MiB/sec (186.42 MB/sec)</p> <p>Run finished: Tue Oct 29 01:54:04 2013<br/> rm: cannot remove `/mnt/lustre/d0.ior.ssf': Directory not empty<br/> parallel-scale-nfsv3 test_iorssf: @@@@@@ FAIL: test_iorssf failed with 1 <br/> Trace dump:<br/> = /usr/lib64/lustre/tests/test-framework.sh:4264:error_noexit()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4291:error()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4530:run_one()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4563:run_one_logged()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4433:run_test()<br/> = /usr/lib64/lustre/tests/parallel-scale-nfs.sh:93:main()<br/> Dumping lctl log to /tmp/test_logs/2013-10-29/012032/parallel-scale-nfsv3.test_iorssf.*.1383036845.log</p> <p>It seems duplicate with <a href="https://jira.whamcloud.com/browse/LU-1639" title="Test failure parallel-scale-nfsv3, test_iorssf" class="issue-link" data-issue-key="LU-1639">LU-1639</a>.</p> <p>I checked debug log on MDS, it seems show the directory is indeed not empty</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00000001:3.0:1383036844.842826:0:3111:0:(osd_handler.c:4729:osd_it_ea_next()) Process entered 00000004:00000001:3.0:1383036844.842827:0:3111:0:(osd_handler.c:4737:osd_it_ea_next()) Process leaving (rc=0 : 0 : 0) 00000004:00000001:3.0:1383036844.842828:0:3111:0:(osd_handler.c:4565:osd_it_ea_fini()) Process entered 00000020:00000001:3.0:1383036844.842830:0:3111:0:(lustre_fid.h:719:fid_flatten32()) Process leaving (rc=29361006 : 29361006 : 1c0036e) 00000004:00000001:3.0:1383036844.842831:0:3111:0:(osd_handler.c:4568:osd_it_ea_fini()) Process leaving 00000004:00000001:3.0:1383036844.842832:0:3111:0:(mdd_dir.c:271:mdd_dir_is_empty()) Process leaving (rc=18446744073709551577 : -39 : ffffffffffffffd9) 00000004:00000001:3.0:1383036844.842833:0:3111:0:(mdd_dir.c:462:mdd_may_delete()) Process leaving (rc=18446744073709551577 : -39 : ffffffffffffffd9) 00000004:00000001:3.0:1383036844.842835:0:3111:0:(mdd_dir.c:1351:mdd_unlink_sanity_check()) Process leaving (rc=18446744073709551577 : -39 : ffffffffffffffd9) 00000004:00000001:3.0:1383036844.842836:0:3111:0:(mdd_dir.c:1502:mdd_unlink()) Process leaving via cleanup (rc=18446744073709551577 : -39 : 0xffffffffffffffd9) </pre> </div></div> <p>Though no idea about why iorData is not being deleted before rmdir d0.ior.ssf. Unfortunately there are no debug log here to know the reason. Anyway, it is nothing about DNE, should be duplicate with <a href="https://jira.whamcloud.com/browse/LU-1639" title="Test failure parallel-scale-nfsv3, test_iorssf" class="issue-link" data-issue-key="LU-1639">LU-1639</a>.</p> Bugzilla ID 1639.0 Development Rank 1|hzw76n: Rank (Obsolete) 11308 Severity