Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-451] Test failure on test suite parallel-scale https://jira.whamcloud.com/browse/LU-451 Lustre <p>This issue was created by maloo for sarah <sarah@whamcloud.com></p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/6fc062aa-9d34-11e0-9a27-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6fc062aa-9d34-11e0-9a27-52540025f9af</a>.<br/> IOR failed on the latest master build NFS3</p> LU-451

Test failure on test suite parallel-scale

Bug Blocker Resolved Fixed Oleg Drokin Maloo Wed, 22 Jun 2011 21:08:30 +0000 Fri, 5 Aug 2011 13:28:38 +0000 Fri, 5 Aug 2011 13:28:38 +0000 Lustre 2.1.0 0 5 <p>Unfortunately the maloo still does not report and logs from the nodes other than nfs client.<br/> These runs must include logs from luster nfs exporting client, also MDS and OST to be fully selfcontained.</p> <p>As things stand now we see from the nfs client that the NFS server went missing somewhere during run, but I have no way of knowing why.<br/> Need logs from 10.10.4.15 to find out.</p> <p>Thanks.</p> <p>connectathon on rhel5/x85_64 has the similar issue, during the lock test, nfs server does not respond.</p> <p>console log from nfs client:<br/> -----------------<br/> Lustre: DEBUG MARKER: == parallel-scale test connectathon: connectathon ==================================================== 22:38:38 (1308893918)<br/> Lustre: DEBUG MARKER: ./runtests -N 2 -b -f /mnt/lustre/d0.connectathon<br/> Lustre: DEBUG MARKER: ./runtests -N 2 -g -f /mnt/lustre/d0.connectathon<br/> Lustre: DEBUG MARKER: ./runtests -N 2 -s -f /mnt/lustre/d0.connectathon<br/> Lustre: DEBUG MARKER: ./runtests -N 2 -l -f /mnt/lustre/d0.connectathon<br/> lockd: server 10.10.4.15 not responding, timed out<br/> lockd: server 10.10.4.15 not responding, timed out</p> <p>lockd: server 10.10.4.15 not responding, timed out<br/> lockd: server 10.10.4.15 not responding, timed out</p> <p>lockd: server 10.10.4.15 not responding, timed out</p> <p>Attached is the logs from lustre client, mds and ost</p> <p>hm, nothing interesting in dmesg logs.<br/> Is it possible to get a backtrace list from lustre client?</p> <p>Interesting that I have not seen any problems like this, I guess I need to try with a remote node.</p> <p>I will update the back trace tomorrow</p> <p>Here is the back trace of lustre client(nfs server)</p> <p>got the same error on 2.0.65 rhel6-x86_64/i686 </p> <p>Lustre: DEBUG MARKER: == parallel-scale test ior: ior == 13:52:39 (1311195159)<br/> nfs: server 10.10.4.15 not responding, still trying<br/> nfs: server 10.10.4.15 not responding, still trying<br/> nfs: server 10.10.4.15 not responding, still trying<br/> nfs: server 10.10.4.15 not responding, still trying<br/> INFO: task IOR:9286 blocked for more than 120 seconds.<br/> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> IOR D c306be18 0 9286 9284 0x00000080<br/> c31db030 00000082 00000002 c306be18 c3e03b54 00000000 c04b63d5 c0459a2e<br/> 11768000 f09ac740 0000300f 5952697c 0000300f c0ae0120 c0ae0120 c31db2d8<br/> c0ae0120 c0adbb54 c0ae0120 c31db2d8 c3e03b54 e47e41ec 03230e0c c31db030<br/> Call Trace:<br/> <span class="error">[<c04b63d5>]</span> ? rcu_process_callbacks+0x35/0x40<br/> <span class="error">[<c0459a2e>]</span> ? __do_softirq+0xae/0x1a0<br/> <span class="error">[<c0459c85>]</span> ? irq_exit+0x35/0x70<br/> <span class="error">[<c04279f3>]</span> ? smp_apic_timer_interrupt+0x53/0x90<br/> <span class="error">[<c08239c5>]</span> ? apic_timer_interrupt+0x31/0x38<br/> <span class="error">[<c044007b>]</span> ? print_cfs_rq+0x56b/0xc90<br/> <span class="error">[<c08222d8>]</span> ? __mutex_lock_slowpath+0xd8/0x140<br/> <span class="error">[<c08221dd>]</span> ? mutex_lock+0x1d/0x40<br/> <span class="error">[<c04de7a9>]</span> ? generic_file_aio_write+0x49/0xc0<br/> <span class="error">[<f86dfc9c>]</span> ? nfs_file_write+0xcc/0x1d0 <span class="error">[nfs]</span><br/> <span class="error">[<c04631e5>]</span> ? run_timer_softirq+0x35/0x2c0<br/> <span class="error">[<c05277f5>]</span> ? do_sync_write+0xd5/0x120<br/> <span class="error">[<c0473cb0>]</span> ? autoremove_wake_function+0x0/0x40<br/> <span class="error">[<c0459c85>]</span> ? irq_exit+0x35/0x70<br/> <span class="error">[<c04279f3>]</span> ? smp_apic_timer_interrupt+0x53/0x90<br/> <span class="error">[<c0459a2e>]</span> ? __do_softirq+0xae/0x1a0<br/> <span class="error">[<c08239c5>]</span> ? apic_timer_interrupt+0x31/0x38<br/> <span class="error">[<c059cf4c>]</span> ? security_file_permission+0xc/0x10<br/> <span class="error">[<c05279c6>]</span> ? rw_verify_area+0x66/0xe0<br/> <span class="error">[<c0527720>]</span> ? do_sync_write+0x0/0x120<br/> <span class="error">[<c0527ae0>]</span> ? vfs_write+0xa0/0x190<br/> <span class="error">[<c04adb3c>]</span> ? audit_syscall_entry+0x21c/0x240<br/> <span class="error">[<c0528561>]</span> ? sys_write+0x41/0x70<br/> <span class="error">[<c0409adf>]</span> ? sysenter_do_call+0x12/0x28<br/> INFO: task IOR:9286 blocked for more than 120 seconds.<br/> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> IOR D c306be18 0 9286 9284 0x00000080<br/> c31db030 00000082 00000002 c306be18 c3e03b54 00000000 c04b63d5 c0459a2e<br/> 11768000 f09ac740 0000300f 5952697c 0000300f c0ae0120 c0ae0120 c31db2d8<br/> c0ae0120 c0adbb54 c0ae0120 c31db2d8 c3e03b54 e47e41ec 03230e0c c31db030<br/> Call Trace:<br/> <span class="error">[<c04b63d5>]</span> ? rcu_process_callbacks+0x35/0x40<br/> <span class="error">[<c0459a2e>]</span> ? __do_softirq+0xae/0x1a0<br/> <span class="error">[<c0459c85>]</span> ? irq_exit+0x35/0x70<br/> <span class="error">[<c04279f3>]</span> ? smp_apic_timer_interrupt+0x53/0x90<br/> <span class="error">[<c08239c5>]</span> ? apic_timer_interrupt+0x31/0x38<br/> <span class="error">[<c044007b>]</span> ? print_cfs_rq+0x56b/0xc90<br/> <span class="error">[<c08222d8>]</span> ? __mutex_lock_slowpath+0xd8/0x140<br/> <span class="error">[<c08221dd>]</span> ? mutex_lock+0x1d/0x40<br/> <span class="error">[<c04de7a9>]</span> ? generic_file_aio_write+0x49/0xc0<br/> <span class="error">[<f86dfc9c>]</span> ? nfs_file_write+0xcc/0x1d0 <span class="error">[nfs]</span><br/> <span class="error">[<c04631e5>]</span> ? run_timer_softirq+0x35/0x2c0<br/> <span class="error">[<c05277f5>]</span> ? do_sync_write+0xd5/0x120<br/> <span class="error">[<c0473cb0>]</span> ? autoremove_wake_function+0x0/0x40<br/> <span class="error">[<c0459c85>]</span> ? irq_exit+0x35/0x70<br/> <span class="error">[<c04279f3>]</span> ? smp_apic_timer_interrupt+0x53/0x90<br/> <span class="error">[<c0459a2e>]</span> ? __do_softirq+0xae/0x1a0<br/> <span class="error">[<c08239c5>]</span> ? apic_timer_interrupt+0x31/0x38<br/> <span class="error">[<c059cf4c>]</span> ? security_file_permission+0xc/0x10<br/> <span class="error">[<c05279c6>]</span> ? rw_verify_area+0x66/0xe0<br/> <span class="error">[<c0527720>]</span> ? do_sync_write+0x0/0x120<br/> <span class="error">[<c0527ae0>]</span> ? vfs_write+0xa0/0x190<br/> <span class="error">[<c04adb3c>]</span> ? audit_syscall_entry+0x21c/0x240<br/> <span class="error">[<c0528561>]</span> ? sys_write+0x41/0x70<br/> <span class="error">[<c0409adf>]</span> ? sysenter_do_call+0x12/0x28<br/> INFO: task IOR:9286 blocked for more than 120 seconds.<br/> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> IOR D c306be18 0 9286 9284 0x00000080<br/> c31db030 00000082 00000002 c306be18 c3e03b54 00000000 c04b63d5 c0459a2e<br/> 11768000 f09ac740 0000300f 5952697c 0000300f c0ae0120 c0ae0120 c31db2d8<br/> c0ae0120 c0adbb54 c0ae0120 c31db2d8 c3e03b54 e47e41ec 03230e0c c31db030<br/> Call Trace:<br/> <span class="error">[<c04b63d5>]</span> ? rcu_process_callbacks+0x35/0x40<br/> <span class="error">[<c0459a2e>]</span> ? __do_softirq+0xae/0x1a0<br/> <span class="error">[<c0459c85>]</span> ? irq_exit+0x35/0x70<br/> <span class="error">[<c04279f3>]</span> ? smp_apic_timer_interrupt+0x53/0x90<br/> <span class="error">[<c08239c5>]</span> ? apic_timer_interrupt+0x31/0x38<br/> <span class="error">[<c044007b>]</span> ? print_cfs_rq+0x56b/0xc90<br/> <span class="error">[<c08222d8>]</span> ? __mutex_lock_slowpath+0xd8/0x140<br/> <span class="error">[<c08221dd>]</span> ? mutex_lock+0x1d/0x40<br/> <span class="error">[<c04de7a9>]</span> ? generic_file_aio_write+0x49/0xc0<br/> <span class="error">[<f86dfc9c>]</span> ? nfs_file_write+0xcc/0x1d0 <span class="error">[nfs]</span><br/> <span class="error">[<c04631e5>]</span> ? run_timer_softirq+0x35/0x2c0<br/> <span class="error">[<c05277f5>]</span> ? do_sync_write+0xd5/0x120<br/> <span class="error">[<c0473cb0>]</span> ? autoremove_wake_function+0x0/0x40<br/> <span class="error">[<c0459c85>]</span> ? irq_exit+0x35/0x70<br/> <span class="error">[<c04279f3>]</span> ? smp_apic_timer_interrupt+0x53/0x90<br/> <span class="error">[<c0459a2e>]</span> ? __do_softirq+0xae/0x1a0<br/> <span class="error">[<c08239c5>]</span> ? apic_timer_interrupt+0x31/0x38<br/> <span class="error">[<c059cf4c>]</span> ? security_file_permission+0xc/0x10<br/> <span class="error">[<c05279c6>]</span> ? rw_verify_area+0x66/0xe0<br/> <span class="error">[<c0527720>]</span> ? do_sync_write+0x0/0x120<br/> <span class="error">[<c0527ae0>]</span> ? vfs_write+0xa0/0x190<br/> <span class="error">[<c04adb3c>]</span> ? audit_syscall_entry+0x21c/0x240<br/> <span class="error">[<c0528561>]</span> ? sys_write+0x41/0x70<br/> <span class="error">[<c0409adf>]</span> ? sysenter_do_call+0x12/0x28</p> <p>Marking as blocker because this seems to regularly fail in NFS testing</p> <p>Hm, can you try the latest build from <a href="http://review.whamcloud.com/#change,911" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,911</a> and see if IOR still hangs?<br/> Also nfs server (lustre client) backtrace would be useful, when there is a report of nfs server not responding, the problem is somewhere on that end.</p> <p>With this patch, it passed on RHEL6 nfs3: <a href="https://maloo.whamcloud.com/test_sets/b0149d4a-b6f6-11e0-8bdf-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b0149d4a-b6f6-11e0-8bdf-52540025f9af</a></p> <p>On RHEL6-x86_64 server and RHEL6-i686 lustre client and nfs client, got this error.<br/> NFS client console log:<br/> -------------------------<del>Lustre: DEBUG MARKER: -----============= acceptance-small: parallel-scale ============</del>---- Mon Jul 25 15:43:16 PDT 2011<br/> Lustre: DEBUG MARKER: only running test ior<br/> Lustre: DEBUG MARKER: excepting tests: parallel_grouplock<br/> Lustre: DEBUG MARKER: == parallel-scale test ior: ior == 15:43:18 (1311633798)<br/> nfs: server 10.10.4.15 not responding, still trying<br/> nfs: server 10.10.4.15 not responding, still trying<br/> INFO: task IOR:3628 blocked for more than 120 seconds.<br/> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> IOR D e8c07e18 0 3628 3627 0x00000080<br/> c2ca6ab0 00000082 00000002 e8c07e18 c3e03b54 00000000 c04b63d5 00000a3e<br/> 00000000 c2c18740 000001e7 a04f6bc3 000001e7 c0ae0120 c0ae0120 c2ca6d58<br/> c0ae0120 c0adbb54 c0ae0120 c2ca6d58 c3e03b54 e958147c 001b6c91 c2ca6ab0<br/> Call Trace:<br/> <span class="error">[<c04b63d5>]</span> ? rcu_process_callbacks+0x35/0x40<br/> <span class="error">[<c0459c85>]</span> ? irq_exit+0x35/0x70<br/> <span class="error">[<c04279f3>]</span> ? smp_apic_timer_interrupt+0x53/0x90<br/> <span class="error">[<c08239c5>]</span> ? apic_timer_interrupt+0x31/0x38<br/> <span class="error">[<c08222d8>]</span> ? __mutex_lock_slowpath+0xd8/0x140<br/> <span class="error">[<c08221dd>]</span> ? mutex_lock+0x1d/0x40<br/> <span class="error">[<c04de7a9>]</span> ? generic_file_aio_write+0x49/0xc0<br/> <span class="error">[<f86dfc9c>]</span> ? nfs_file_write+0xcc/0x1d0 <span class="error">[nfs]</span><br/> <span class="error">[<c05277f5>]</span> ? do_sync_write+0xd5/0x120<br/> <span class="error">[<c0473cb0>]</span> ? autoremove_wake_function+0x0/0x40<br/> <span class="error">[<c0459a2e>]</span> ? __do_softirq+0xae/0x1a0<br/> <span class="error">[<c059cf4c>]</span> ? security_file_permission+0xc/0x10<br/> <span class="error">[<c05279c6>]</span> ? rw_verify_area+0x66/0xe0<br/> <span class="error">[<c0527720>]</span> ? do_sync_write+0x0/0x120<br/> <span class="error">[<c0527ae0>]</span> ? vfs_write+0xa0/0x190<br/> <span class="error">[<c04adb3c>]</span> ? audit_syscall_entry+0x21c/0x240<br/> <span class="error">[<c0528561>]</span> ? sys_write+0x41/0x70<br/> <span class="error">[<c0409adf>]</span> ? sysenter_do_call+0x12/0x28<br/> nfs: server 10.10.4.15 not responding, still trying</p> <p>NFS server console log, sorry cannot get the trace since the following messages keep showing up<br/> ---------------------------------------<br/> Call Trace:<br/> <span class="error">[<c04ed81e>]</span> ? __alloc_pages_nodemask+0x67e/0x800<br/> <span class="error">[<c0442233>]</span> ? find_busiest_group+0x1d3/0x930<br/> <span class="error">[<c051981c>]</span> ? cache_alloc_refill+0x2bc/0x510<br/> <span class="error">[<c0519dd2>]</span> ? __kmalloc+0x142/0x180<br/> <span class="error">[<c0775dff>]</span> ? __alloc_skb+0x4f/0x140<br/> <span class="error">[<c0775dff>]</span> ? __alloc_skb+0x4f/0x140<br/> <span class="error">[<c07760eb>]</span> ? __netdev_alloc_skb+0x1b/0x40<br/> <span class="error">[<f8115942>]</span> ? igb_alloc_rx_buffers_adv+0x2c2/0x320 <span class="error">[igb]</span><br/> <span class="error">[<c04b5171>]</span> ? move_native_irq+0x11/0x50<br/> <span class="error">[<f8115f16>]</span> ? igb_poll+0x576/0xd20 <span class="error">[igb]</span><br/> <span class="error">[<f8117dbe>]</span> ? igb_msix_ring+0x4e/0x70 <span class="error">[igb]</span><br/> <span class="error">[<c04b1d75>]</span> ? handle_IRQ_event+0x45/0x140<br/> <span class="error">[<c04b5171>]</span> ? move_native_irq+0x11/0x50<br/> <span class="error">[<c040a030>]</span> ? common_interrupt+0x30/0x38<br/> <span class="error">[<c0781b5e>]</span> ? net_rx_action+0xde/0x280<br/> <span class="error">[<c0459a0a>]</span> ? __do_softirq+0x8a/0x1a0<br/> <span class="error">[<c05f846c>]</span> ? __const_udelay+0x2c/0x30<br/> <span class="error">[<c04b5171>]</span> ? move_native_irq+0x11/0x50<br/> <span class="error">[<f811db2f>]</span> ? igb_release_swfw_sync_82575+0xf/0x40 <span class="error">[igb]</span><br/> <span class="error">[<c0459b5d>]</span> ? do_softirq+0x3d/0x50<br/> <span class="error">[<c0459cb5>]</span> ? irq_exit+0x65/0x70<br/> <span class="error">[<c040b1c0>]</span> ? do_IRQ+0x50/0xc0<br/> <span class="error">[<c04f9aa0>]</span> ? vmstat_update+0x0/0x30<br/> <span class="error">[<c040a030>]</span> ? common_interrupt+0x30/0x38<br/> <span class="error">[<c04f9aa0>]</span> ? vmstat_update+0x0/0x30<br/> <span class="error">[<c04f9ab0>]</span> ? vmstat_update+0x10/0x30<br/> <span class="error">[<c046f58b>]</span> ? worker_thread+0x11b/0x230<br/> <span class="error">[<c0473cb0>]</span> ? autoremove_wake_function+0x0/0x40<br/> <span class="error">[<c046f470>]</span> ? worker_thread+0x0/0x230<br/> <span class="error">[<c0473a74>]</span> ? kthread+0x74/0x80<br/> <span class="error">[<c0473a00>]</span> ? kthread+0x0/0x80<br/> <span class="error">[<c040a03f>]</span> ? kernel_thread_helper+0x7/0x10<br/> Mem-Info:<br/> DMA per-cpu:<br/> CPU 0: hi: 0, btch: 1 usd: 0<br/> CPU 1: hi: 0, btch: 1 usd: 0<br/> CPU 2: hi: 0, btch: 1 usd: 0<br/> CPU 3: hi: 0, btch: 1 usd: 0<br/> Normal per-cpu:<br/> CPU 0: hi: 186, btch: 31 usd: 0<br/> CPU 1: hi: 186, btch: 31 usd: 0<br/> CPU 2: hi: 186, btch: 31 usd: 0<br/> CPU 3: hi: 186, btch: 31 usd: 0<br/> HighMem per-cpu:<br/> CPU 0: hi: 186, btch: 31 usd: 0<br/> CPU 1: hi: 186, btch: 31 usd: 0<br/> CPU 2: hi: 186, btch: 31 usd: 0<br/> CPU 3: hi: 186, btch: 31 usd: 0<br/> active_anon:2077 inactive_anon:354 isolated_anon:0<br/> active_<a href="file:2444" class="external-link" target="_blank" rel="nofollow noopener">file:2444</a> inactive_<a href="file:756607" class="external-link" target="_blank" rel="nofollow noopener">file:756607</a> isolated_<a href="file:0" class="external-link" target="_blank" rel="nofollow noopener">file:0</a><br/> unevictable:0 dirty:0 writeback:0 unstable:0<br/> free:2146797 slab_reclaimable:3746 slab_unreclaimable:127322<br/> mapped:2502 shmem:46 pagetables:316 bounce:0<br/> DMA free:3472kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_<a href="file:0kB" class="external-link" target="_blank" rel="nofollow noopener">file:0kB</a> inactive_<a href="file:0kB" class="external-link" target="_blank" rel="nofollow noopener">file:0kB</a> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15792kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:64kB slab_unreclaimable:3772kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes<br/> lowmem_reserve[]: 0 863 12159 12159<br/> Normal free:1400kB min:3724kB low:4652kB high:5584kB active_anon:0kB inactive_anon:0kB active_<a href="file:184kB" class="external-link" target="_blank" rel="nofollow noopener">file:184kB</a> inactive_<a href="file:0kB" class="external-link" target="_blank" rel="nofollow noopener">file:0kB</a> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:883912kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:14920kB slab_unreclaimable:505516kB kernel_stack:1488kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:334 all_unreclaimable? yes<br/> lowmem_reserve[]: 0 0 90370 90370<br/> HighMem free:8582316kB min:512kB low:12700kB high:24888kB active_anon:8308kB inactive_anon:1416kB active_<a href="file:9592kB" class="external-link" target="_blank" rel="nofollow noopener">file:9592kB</a> inactive_<a href="file:3026428kB" class="external-link" target="_blank" rel="nofollow noopener">file:3026428kB</a> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:11567412kB mlocked:0kB dirty:0kB writeback:0kB mapped:10004kB shmem:184kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:1264kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no<br/> lowmem_reserve[]: 0 0 0 0<br/> DMA: 6*4kB 3*8kB 2*16kB 4*32kB 3*64kB 6*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 3472kB<br/> Normal: 60*4kB 7*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1400kB<br/> HighMem: 1139*4kB 342*8kB 189*16kB 215*32kB 180*64kB 77*128kB 46*256kB 30*512kB 15*1024kB 5*2048kB 2073*4096kB = 8582316kB<br/> 759078 total pagecache pages<br/> 0 pages in swap cache<br/> Swap cache stats: add 0, delete 0, find 0/0<br/> Free swap = 14565368kB<br/> Total swap = 14565368kB<br/> 3145712 pages RAM<br/> 2918914 pages HighMem<br/> 62162 pages reserved<br/> 804322 pages shared<br/> 151475 pages non-shared<br/> events/2: page allocation failure. order:0, mode:0x20<br/> Pid: 21, comm: events/2 Tainted: G ---------------- T 2.6.32-131.2.1.el6.i686 #1</p> <p>connectathon on RHEL5 still has this problem, please find attached for trace.</p> <p>Well, this is out of memory on i686<br/> Can you please test on x86_64 instead and see if that works?</p> <p>It works on RHEL6-x86_64 and failed on RHEL5-x86_64(connectathon test, not ior)</p> <p>How does the failure looked like on rhel5 x86_64? Is there a maloo run link?<br/> Can you run rhel5 and rhel6 each several times and see how consistently they fail, please?</p> <p>Thanks.</p> <p>No, there is no maloo link. It looks like this, trace information please see the attached.</p> <p>Lustre: DEBUG MARKER: == parallel-scale test connectathon: connectathon ==================================================== 15:16:44 (1311718604)<br/> Lustre: DEBUG MARKER: ./runtests -N 2 -b -f /mnt/lustre/d0.connectathon<br/> Lustre: DEBUG MARKER: ./runtests -N 2 -g -f /mnt/lustre/d0.connectathon<br/> Lustre: DEBUG MARKER: ./runtests -N 2 -s -f /mnt/lustre/d0.connectathon<br/> Lustre: DEBUG MARKER: ./runtests -N 2 -l -f /mnt/lustre/d0.connectathon<br/> lockd: server 10.10.4.15 not responding, timed out<br/> lockd: server 10.10.4.15 not responding, timed out<br/> lockd: server 10.10.4.15 not responding, timed out<br/> lockd: server 10.10.4.15 not responding, timed out</p> <p>I ran connectathon on RHEL5 4 times and only one pass: <a href="https://maloo.whamcloud.com/test_sets/4d895790-b7e3-11e0-8bdf-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4d895790-b7e3-11e0-8bdf-52540025f9af</a></p> <p>I also ran ior on RHEL6 2 times today against this patch, they all failed.</p> <p>Hm, the stacktrace points at the locking problems which is quite strange, I think those all are fixed and I have not seen them in quite a while.</p> <p>Are such connectathon failures only happening with nfs3 and never with nfs4?</p> <blockquote> <p>Are such connectathon failures only happening with nfs3 and never with nfs4?</p></blockquote> <p>From the record, only nfs3 has this problem, never with nfs4</p> <p>Can you please try with <a href="http://review.whamcloud.com/1174" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1174</a> this seems to be working well for me now.</p> <p>run 3 times connectathon on RHEL5, all pass; while ior on RHEL6 still has this problem.Please see the attached for trace from nfs server</p> <p>Interesting, I did my testing on rhel6 and met no problem, though I concentrated on locking only.</p> <p>Anyway the problem seen in teh stack trace is a different one.<br/> Can you please file a different bug for it please? Thus bug is already overloaded with various failures and it's about time we bring some order to it.</p> <p>I've created a new issue <a href="https://jira.whamcloud.com/browse/LU-567" title="ior failed on RHEL6-x86_64 NFS3 testing" class="issue-link" data-issue-key="LU-567"><del>LU-567</del></a>, please check.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/242/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #242</a><br/> <a href="https://jira.whamcloud.com/browse/LU-451" title="Test failure on test suite parallel-scale" class="issue-link" data-issue-key="LU-451"><del>LU-451</del></a> Workaround for lockd (nfsv3) locking</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=71535dbbb89774eecb2f83eff80b19515e14da70" class="external-link" target="_blank" rel="nofollow noopener">71535dbbb89774eecb2f83eff80b19515e14da70</a><br/> Files : </p> <ul> <li>lustre/llite/file.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/242/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #242</a><br/> <a href="https://jira.whamcloud.com/browse/LU-451" title="Test failure on test suite parallel-scale" class="issue-link" data-issue-key="LU-451"><del>LU-451</del></a> Workaround for lockd (nfsv3) locking</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=71535dbbb89774eecb2f83eff80b19515e14da70" class="external-link" target="_blank" rel="nofollow noopener">71535dbbb89774eecb2f83eff80b19515e14da70</a><br/> Files : </p> <ul> <li>lustre/llite/file.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/242/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #242</a><br/> <a href="https://jira.whamcloud.com/browse/LU-451" title="Test failure on test suite parallel-scale" class="issue-link" data-issue-key="LU-451"><del>LU-451</del></a> Workaround for lockd (nfsv3) locking</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=71535dbbb89774eecb2f83eff80b19515e14da70" class="external-link" target="_blank" rel="nofollow noopener">71535dbbb89774eecb2f83eff80b19515e14da70</a><br/> Files : </p> <ul> <li>lustre/llite/file.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/242/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #242</a><br/> <a href="https://jira.whamcloud.com/browse/LU-451" title="Test failure on test suite parallel-scale" class="issue-link" data-issue-key="LU-451"><del>LU-451</del></a> Workaround for lockd (nfsv3) locking</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=71535dbbb89774eecb2f83eff80b19515e14da70" class="external-link" target="_blank" rel="nofollow noopener">71535dbbb89774eecb2f83eff80b19515e14da70</a><br/> Files : </p> <ul> <li>lustre/llite/file.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/242/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #242</a><br/> <a href="https://jira.whamcloud.com/browse/LU-451" title="Test failure on test suite parallel-scale" class="issue-link" data-issue-key="LU-451"><del>LU-451</del></a> Workaround for lockd (nfsv3) locking</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=71535dbbb89774eecb2f83eff80b19515e14da70" class="external-link" target="_blank" rel="nofollow noopener">71535dbbb89774eecb2f83eff80b19515e14da70</a><br/> Files : </p> <ul> <li>lustre/llite/file.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/242/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #242</a><br/> <a href="https://jira.whamcloud.com/browse/LU-451" title="Test failure on test suite parallel-scale" class="issue-link" data-issue-key="LU-451"><del>LU-451</del></a> Workaround for lockd (nfsv3) locking</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=71535dbbb89774eecb2f83eff80b19515e14da70" class="external-link" target="_blank" rel="nofollow noopener">71535dbbb89774eecb2f83eff80b19515e14da70</a><br/> Files : </p> <ul> <li>lustre/llite/file.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/242/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #242</a><br/> <a href="https://jira.whamcloud.com/browse/LU-451" title="Test failure on test suite parallel-scale" class="issue-link" data-issue-key="LU-451"><del>LU-451</del></a> Workaround for lockd (nfsv3) locking</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=71535dbbb89774eecb2f83eff80b19515e14da70" class="external-link" target="_blank" rel="nofollow noopener">71535dbbb89774eecb2f83eff80b19515e14da70</a><br/> Files : </p> <ul> <li>lustre/llite/file.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/242/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #242</a><br/> <a href="https://jira.whamcloud.com/browse/LU-451" title="Test failure on test suite parallel-scale" class="issue-link" data-issue-key="LU-451"><del>LU-451</del></a> Workaround for lockd (nfsv3) locking</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=71535dbbb89774eecb2f83eff80b19515e14da70" class="external-link" target="_blank" rel="nofollow noopener">71535dbbb89774eecb2f83eff80b19515e14da70</a><br/> Files : </p> <ul> <li>lustre/llite/file.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/242/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #242</a><br/> <a href="https://jira.whamcloud.com/browse/LU-451" title="Test failure on test suite parallel-scale" class="issue-link" data-issue-key="LU-451"><del>LU-451</del></a> Workaround for lockd (nfsv3) locking</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=71535dbbb89774eecb2f83eff80b19515e14da70" class="external-link" target="_blank" rel="nofollow noopener">71535dbbb89774eecb2f83eff80b19515e14da70</a><br/> Files : </p> <ul> <li>lustre/llite/file.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/242/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #242</a><br/> <a href="https://jira.whamcloud.com/browse/LU-451" title="Test failure on test suite parallel-scale" class="issue-link" data-issue-key="LU-451"><del>LU-451</del></a> Workaround for lockd (nfsv3) locking</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=71535dbbb89774eecb2f83eff80b19515e14da70" class="external-link" target="_blank" rel="nofollow noopener">71535dbbb89774eecb2f83eff80b19515e14da70</a><br/> Files : </p> <ul> <li>lustre/llite/file.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/242/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #242</a><br/> <a href="https://jira.whamcloud.com/browse/LU-451" title="Test failure on test suite parallel-scale" class="issue-link" data-issue-key="LU-451"><del>LU-451</del></a> Workaround for lockd (nfsv3) locking</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=71535dbbb89774eecb2f83eff80b19515e14da70" class="external-link" target="_blank" rel="nofollow noopener">71535dbbb89774eecb2f83eff80b19515e14da70</a><br/> Files : </p> <ul> <li>lustre/llite/file.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/242/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #242</a><br/> <a href="https://jira.whamcloud.com/browse/LU-451" title="Test failure on test suite parallel-scale" class="issue-link" data-issue-key="LU-451"><del>LU-451</del></a> Workaround for lockd (nfsv3) locking</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=71535dbbb89774eecb2f83eff80b19515e14da70" class="external-link" target="_blank" rel="nofollow noopener">71535dbbb89774eecb2f83eff80b19515e14da70</a><br/> Files : </p> <ul> <li>lustre/llite/file.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/242/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #242</a><br/> <a href="https://jira.whamcloud.com/browse/LU-451" title="Test failure on test suite parallel-scale" class="issue-link" data-issue-key="LU-451"><del>LU-451</del></a> Workaround for lockd (nfsv3) locking</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=71535dbbb89774eecb2f83eff80b19515e14da70" class="external-link" target="_blank" rel="nofollow noopener">71535dbbb89774eecb2f83eff80b19515e14da70</a><br/> Files : </p> <ul> <li>lustre/llite/file.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/242/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #242</a><br/> <a href="https://jira.whamcloud.com/browse/LU-451" title="Test failure on test suite parallel-scale" class="issue-link" data-issue-key="LU-451"><del>LU-451</del></a> Workaround for lockd (nfsv3) locking</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=71535dbbb89774eecb2f83eff80b19515e14da70" class="external-link" target="_blank" rel="nofollow noopener">71535dbbb89774eecb2f83eff80b19515e14da70</a><br/> Files : </p> <ul> <li>lustre/llite/file.c</li> </ul> <p>Landed for 2.1</p> Development Rank 1|hzv8mn: Rank (Obsolete) 4932 Severity