https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-683 Lustre <p>While running recovery-mds-scale with FLAVOR=OSS, it failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;~snip~&gt; ==== Checking the clients loads AFTER failover -- failure NOT OK Client load failed on node client-2-ib, rc=1 Client load failed during failover. Exiting 2011-09-13 22:12:19 Terminating clients loads ... Duration: 7200 Server failover period: 600 seconds Exited after: 4991 seconds Number of failovers before exit: mds1: 0 times ost1: 1 times ost2: 1 times ost3: 0 times ost4: 0 times ost5: 1 times ost6: 0 times ost7: 1 times Status: FAIL: rc=5 </pre> </div></div> <p>Client node client-2-ib hit the following LBUG:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 32289:0:(osc_io.c:336:osc_io_commit_write()) ASSERTION(to &gt; 0) failed^M LustreError: 32289:0:(osc_io.c:336:osc_io_commit_write()) LBUG^M Pid: 32289, comm: dd^M ^M Call Trace:^M [&lt;ffffffffa03b9855&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs]^M [&lt;ffffffffa03b9e95&gt;] lbug_with_loc+0x75/0xe0 [libcfs]^M [&lt;ffffffffa03c4d76&gt;] libcfs_assertion_failed+0x66/0x70 [libcfs]^M [&lt;ffffffffa079dcc0&gt;] osc_io_commit_write+0x1d0/0x1e0 [osc]^M [&lt;ffffffff8119b6cf&gt;] ? __mark_inode_dirty+0x13f/0x160^M [&lt;ffffffffa04e241f&gt;] cl_io_commit_write+0xaf/0x1f0 [obdclass]^M [&lt;ffffffffa07fe6dd&gt;] ? lov_page_subio+0xad/0x260 [lov]^M [&lt;ffffffffa04d37d8&gt;] ? cl_page_export+0x58/0x80 [obdclass]^M [&lt;ffffffffa07fe937&gt;] lov_io_commit_write+0xa7/0x1d0 [lov]^M^@ [&lt;ffffffffa04e241f&gt;] cl_io_commit_write+0xaf/0x1f0 [obdclass]^M [&lt;ffffffffa04d26b9&gt;] ? cl_env_get+0x29/0x350 [obdclass]^M [&lt;ffffffffa0871e0d&gt;] ll_commit_write+0xdd/0x2b0 [lustre]^M [&lt;ffffffff814e4d3c&gt;] ? bad_to_user+0x66/0x612^M [&lt;ffffffffa08894a0&gt;] ll_write_end+0x30/0x60 [lustre]^M [&lt;ffffffff8110dc84&gt;] generic_file_buffered_write+0x174/0x2a0^M [&lt;ffffffff8106dd17&gt;] ? current_fs_time+0x27/0x30^M [&lt;ffffffff8110f570&gt;] __generic_file_aio_write+0x250/0x480^M [&lt;ffffffff8110f80f&gt;] generic_file_aio_write+0x6f/0xe0^M [&lt;ffffffffa089b001&gt;] vvp_io_write_start+0xa1/0x270 [lustre]^M [&lt;ffffffffa04df318&gt;] cl_io_start+0x68/0x170 [obdclass]^M [&lt;ffffffffa04e37d0&gt;] cl_io_loop+0x110/0x1c0 [obdclass]^M [&lt;ffffffffa084291b&gt;] ll_file_io_generic+0x44b/0x580 [lustre]^M [&lt;ffffffffa03c8424&gt;] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs]^M [&lt;ffffffffa04d26b9&gt;] ? cl_env_get+0x29/0x350 [obdclass]^M [&lt;ffffffffa0842b8f&gt;] ll_file_aio_write+0x13f/0x310 [lustre]^M [&lt;ffffffffa04d282e&gt;] ? cl_env_get+0x19e/0x350 [obdclass]^M [&lt;ffffffffa04d0d8f&gt;] ? cl_env_put+0x1af/0x2e0 [obdclass]^M [&lt;ffffffffa0849301&gt;] ll_file_write+0x171/0x310 [lustre]^M [&lt;ffffffff8126e781&gt;] ? __clear_user+0x21/0x70^M [&lt;ffffffff81172748&gt;] vfs_write+0xb8/0x1a0^M [&lt;ffffffff810d1ad2&gt;] ? audit_syscall_entry+0x272/0x2a0^M [&lt;ffffffff81173181&gt;] sys_write+0x51/0x90^M [&lt;ffffffff8100b172&gt;] system_call_fastpath+0x16/0x1b^M ^M Kernel panic - not syncing: LBUG^M Pid: 32289, comm: dd Tainted: G ---------------- T 2.6.32-131.6.1.el6.x86_64 #1^M Call Trace:^M [&lt;ffffffff814da518&gt;] ? panic+0x78/0x143^M [&lt;ffffffffa03b9eeb&gt;] ? lbug_with_loc+0xcb/0xe0 [libcfs]^M [&lt;ffffffffa03c4d76&gt;] ? libcfs_assertion_failed+0x66/0x70 [libcfs]^M [&lt;ffffffffa079dcc0&gt;] ? osc_io_commit_write+0x1d0/0x1e0 [osc]^M [&lt;ffffffff8119b6cf&gt;] ? __mark_inode_dirty+0x13f/0x160^M [&lt;ffffffffa04e241f&gt;] ? cl_io_commit_write+0xaf/0x1f0 [obdclass]^M [&lt;ffffffffa07fe6dd&gt;] ? lov_page_subio+0xad/0x260 [lov]^M [&lt;ffffffffa04d37d8&gt;] ? cl_page_export+0x58/0x80 [obdclass]^M [&lt;ffffffffa07fe937&gt;] ? lov_io_commit_write+0xa7/0x1d0 [lov]^M [&lt;ffffffffa04e241f&gt;] ? cl_io_commit_write+0xaf/0x1f0 [obdclass]^M [&lt;ffffffffa04d26b9&gt;] ? cl_env_get+0x29/0x350 [obdclass]^M [&lt;ffffffffa0871e0d&gt;] ? ll_commit_write+0xdd/0x2b0 [lustre]^M [&lt;ffffffff814e4d3c&gt;] ? bad_to_user+0x66/0x612^M [&lt;ffffffffa08894a0&gt;] ? ll_write_end+0x30/0x60 [lustre]^M [&lt;ffffffff8110dc84&gt;] ? generic_file_buffered_write+0x174/0x2a0^M [&lt;ffffffff8106dd17&gt;] ? current_fs_time+0x27/0x30^M [&lt;ffffffff8110f570&gt;] ? __generic_file_aio_write+0x250/0x480^M [&lt;ffffffff8110f80f&gt;] ? generic_file_aio_write+0x6f/0xe0^M [&lt;ffffffffa089b001&gt;] ? vvp_io_write_start+0xa1/0x270 [lustre]^M [&lt;ffffffffa04df318&gt;] ? cl_io_start+0x68/0x170 [obdclass]^M [&lt;ffffffffa04e37d0&gt;] ? cl_io_loop+0x110/0x1c0 [obdclass]^M [&lt;ffffffffa084291b&gt;] ? ll_file_io_generic+0x44b/0x580 [lustre]^M [&lt;ffffffffa03c8424&gt;] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs]^M [&lt;ffffffffa04d26b9&gt;] ? cl_env_get+0x29/0x350 [obdclass]^M [&lt;ffffffffa0842b8f&gt;] ? ll_file_aio_write+0x13f/0x310 [lustre]^M [&lt;ffffffffa04d282e&gt;] ? cl_env_get+0x19e/0x350 [obdclass]^M [&lt;ffffffffa04d0d8f&gt;] ? cl_env_put+0x1af/0x2e0 [obdclass]^M [&lt;ffffffffa0849301&gt;] ? ll_file_write+0x171/0x310 [lustre]^M [&lt;ffffffff8126e781&gt;] ? __clear_user+0x21/0x70^M [&lt;ffffffff81172748&gt;] ? vfs_write+0xb8/0x1a0^M [&lt;ffffffff810d1ad2&gt;] ? audit_syscall_entry+0x272/0x2a0^M [&lt;ffffffff81173181&gt;] ? sys_write+0x51/0x90^M [&lt;ffffffff8100b172&gt;] ? system_call_fastpath+0x16/0x1b^M </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/943d6cda-de9e-11e0-9909-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/943d6cda-de9e-11e0-9909-52540025f9af</a></p> <p>Please refer to the attached recovery-oss-scale.1315977145.log.tar.bz2 for more logs.</p> <br/> Lustre Branch: master<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-reviews/2177/">http://build.whamcloud.com/job/lustre-reviews/2177/</a><br/> e2fsprogs Build: <a href="http://newbuild.whamcloud.com/job/e2fsprogs-master/54/">http://newbuild.whamcloud.com/job/e2fsprogs-master/54/</a><br/> Distro/Arch: RHEL6/x86_64(in-kernel OFED, kernel version: 2.6.32-131.6.1.el6.x86_64)<br/> ENABLE_QUOTA=yes<br/> FAILURE_MODE=HARD<br/> FLAVOR=OSS<br/> <br/> MGS/MDS Node: fat-amd-1-ib<br/> <br/> OSS Nodes: fat-amd-3-ib(active), fat-amd-4-ib(active)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\ /<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST1 (active in fat-amd-3-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST2 (active in fat-amd-4-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST3 (active in fat-amd-3-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST4 (active in fat-amd-4-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST5 (active in fat-amd-3-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST6 (active in fat-amd-4-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;fat-amd-2-ib(OST7)<br/> <br/> Client Nodes: client-[1,2,4,5,12,13,15],fat-intel-4<br/> <br/> Network Addresses:<br/> fat-amd-1-ib: 192.168.4.132<br/> fat-amd-2-ib: 192.168.4.133<br/> fat-amd-3-ib: 192.168.4.134<br/> fat-amd-4-ib: 192.168.4.135<br/> client-1-ib: 192.168.4.1<br/> client-2-ib: 192.168.4.2<br/> client-4-ib: 192.168.4.4<br/> client-5-ib: 192.168.4.5<br/> client-12-ib: 192.168.4.12<br/> client-13-ib: 192.168.4.13<br/> client-15-ib: 192.168.4.15<br/> fat-intel-4-ib: 192.168.4.131<br/> LU-683

LBUG: (osc_io.c:336:osc_io_commit_write()) ASSERTION(to > 0) failed

Bug Critical Closed Fixed WC Triage Jian Yu Wed, 14 Sep 2011 03:25:42 +0000 Tue, 5 Jun 2012 01:46:26 +0000 Tue, 5 Jun 2012 01:46:14 +0000 Lustre 2.1.0 Lustre 2.1.0 0 4 <p>From the kernel implementation, variable `to' in osc_io_commit_write can be zero. I'm cooking a patch.</p> <p>patch is at: <a href="http://review.whamcloud.com/1383" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1383</a></p> <p>We really need to fix our testing system because no lustre log was caught on the client side.</p> <blockquote><p>patch is at: <a href="http://review.whamcloud.com/1383" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1383</a></p></blockquote> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-reviews/2204/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/2204/</a><br/> Distro/Arch: RHEL6/x86_64(in-kernel OFED, kernel version: 2.6.32-131.6.1.el6.x86_64)<br/> ENABLE_QUOTA=yes<br/> FAILURE_MODE=HARD<br/> FLAVOR=OSS</p> <p>Running recovery-mds-scale with FLAVOR=OSS failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;~snip~&gt; ==== Checking the clients loads AFTER failover -- failure NOT OK Client load failed on node client-12-ib, rc=1 Client load failed during failover. Exiting Found the END_RUN_FILE file: /home/yujian/test_logs/end_run_file client-2-ib Client load failed on node client-2-ib client client-2-ib load stdout and debug files : /tmp/recovery-mds-scale.log_run_dd.sh-client-2-ib /tmp/recovery-mds-scale.log_run_dd.sh-client-2-ib.debug 2011-09-15 04:08:07 Terminating clients loads ... Duration: 7200 Server failover period: 900 seconds Exited after: 3509 seconds Number of failovers before exit: mds1: 0 times ost1: 0 times ost2: 0 times ost3: 0 times ost4: 0 times ost5: 1 times ost6: 1 times ost7: 0 times Status: FAIL: rc=1 </pre> </div></div> <p>/tmp/recovery-mds-scale.log_run_dd.sh-client-2-ib.debug:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;~snip~&gt; 2011-09-15 04:04:43: dd succeeded + cd /tmp + rm -rf /mnt/lustre/d0.dd-client-2-ib ++ date '+%F %H:%M:%S' + echoerr '2011-09-15 04:04:45: dd run finished' + echo '2011-09-15 04:04:45: dd run finished' 2011-09-15 04:04:45: dd run finished + '[' '!' -e /home/yujian/test_logs/end_run_file ']' + true ++ date '+%F %H:%M:%S' + echoerr '2011-09-15 04:04:45: dd run starting' + echo '2011-09-15 04:04:45: dd run starting' 2011-09-15 04:04:45: dd run starting + mkdir -p /mnt/lustre/d0.dd-client-2-ib + cd /mnt/lustre/d0.dd-client-2-ib + load_pid=31837 + wait 31837 + dd bs=4k count=1000000 status=noxfer if=/dev/zero of=/mnt/lustre/d0.dd-client-2-ib/dd-file dd: writing `/mnt/lustre/d0.dd-client-2-ib/dd-file': No space left on device 617072+0 records in 617071+0 records out + '[' 1 -eq 0 ']' ++ date '+%F %H:%M:%S' + echoerr '2011-09-15 04:05:44: dd failed' + echo '2011-09-15 04:05:44: dd failed' 2011-09-15 04:05:44: dd failed </pre> </div></div> <p>Syslog on client node client-2-ib showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Sep 15 04:05:44 client-2 kernel: LustreError: 31837:0:(vvp_io.c:1001:vvp_io_commit_write()) Write page 617071 of inode ffff880127479cb8 failed -28 </pre> </div></div> <p>The dd error and syslog on client-2-ib showed that "No space left on device", however:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@client-2-ib ~]# lfs df UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 7498624 443864 6554760 6% /mnt/lustre[MDT:0] lustre-OST0000_UUID 15939896 2922084 12217684 19% /mnt/lustre[OST:0] lustre-OST0001_UUID 15939896 478888 14653776 3% /mnt/lustre[OST:1] lustre-OST0002_UUID 15939896 455896 14682912 3% /mnt/lustre[OST:2] lustre-OST0003_UUID 15939896 451792 14680872 3% /mnt/lustre[OST:3] lustre-OST0004_UUID 15939896 453048 14686784 3% /mnt/lustre[OST:4] lustre-OST0005_UUID 15939896 452036 14687796 3% /mnt/lustre[OST:5] lustre-OST0006_UUID 15939896 480232 14645024 3% /mnt/lustre[OST:6] filesystem summary: 111579272 5693976 100254848 5% /mnt/lustre [root@client-2-ib ~]# lfs df -i UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 5000040 148 4999892 0% /mnt/lustre[MDT:0] lustre-OST0000_UUID 236160 107 236053 0% /mnt/lustre[OST:0] lustre-OST0001_UUID 236160 93 236067 0% /mnt/lustre[OST:1] lustre-OST0002_UUID 236160 109 236051 0% /mnt/lustre[OST:2] lustre-OST0003_UUID 236160 90 236070 0% /mnt/lustre[OST:3] lustre-OST0004_UUID 236160 97 236063 0% /mnt/lustre[OST:4] lustre-OST0005_UUID 236160 95 236065 0% /mnt/lustre[OST:5] lustre-OST0006_UUID 236160 90 236070 0% /mnt/lustre[OST:6] filesystem summary: 5000040 148 4999892 0% /mnt/lustre </pre> </div></div> <p>An oom issue also occurred on client node client-12-ib:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>dd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0 dd cpuset=/ mems_allowed=0 Pid: 31050, comm: dd Tainted: G ---------------- T 2.6.32-131.6.1.el6.x86_64 #1 Call Trace: [&lt;ffffffff810c0071&gt;] ? cpuset_print_task_mems_allowed+0x91/0xb0 [&lt;ffffffff8111020b&gt;] ? oom_kill_process+0xcb/0x2e0 [&lt;ffffffff811107d0&gt;] ? select_bad_process+0xd0/0x110 [&lt;ffffffff81110868&gt;] ? __out_of_memory+0x58/0xc0 [&lt;ffffffff81110a69&gt;] ? out_of_memory+0x199/0x210 [&lt;ffffffff811201b2&gt;] ? __alloc_pages_nodemask+0x812/0x8b0 [&lt;ffffffff8115465a&gt;] ? alloc_pages_current+0xaa/0x110 [&lt;ffffffff8110d667&gt;] ? __page_cache_alloc+0x87/0x90 [&lt;ffffffff8110e37c&gt;] ? grab_cache_page_write_begin+0x7c/0xc0 [&lt;ffffffffa088957c&gt;] ? ll_write_begin+0x5c/0x200 [lustre] [&lt;ffffffffa0882182&gt;] ? ll_getxattr+0xf2/0x580 [lustre] [&lt;ffffffff8110dc1e&gt;] ? generic_file_buffered_write+0x10e/0x2a0 [&lt;ffffffff8106dd17&gt;] ? current_fs_time+0x27/0x30 [&lt;ffffffff8110f570&gt;] ? __generic_file_aio_write+0x250/0x480 [&lt;ffffffff8110f80f&gt;] ? generic_file_aio_write+0x6f/0xe0 [&lt;ffffffffa089b051&gt;] ? vvp_io_write_start+0xa1/0x270 [lustre] [&lt;ffffffffa04df318&gt;] ? cl_io_start+0x68/0x170 [obdclass] [&lt;ffffffffa04e37d0&gt;] ? cl_io_loop+0x110/0x1c0 [obdclass] [&lt;ffffffffa084291b&gt;] ? ll_file_io_generic+0x44b/0x580 [lustre] [&lt;ffffffffa03c8424&gt;] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs] [&lt;ffffffffa04d26b9&gt;] ? cl_env_get+0x29/0x350 [obdclass] [&lt;ffffffffa0842b8f&gt;] ? ll_file_aio_write+0x13f/0x310 [lustre] [&lt;ffffffffa04d282e&gt;] ? cl_env_get+0x19e/0x350 [obdclass] [&lt;ffffffffa04d0d8f&gt;] ? cl_env_put+0x1af/0x2e0 [obdclass] [&lt;ffffffffa0849301&gt;] ? ll_file_write+0x171/0x310 [lustre] [&lt;ffffffff8126e781&gt;] ? __clear_user+0x21/0x70 [&lt;ffffffff81172748&gt;] ? vfs_write+0xb8/0x1a0 [&lt;ffffffff810d1ad2&gt;] ? audit_syscall_entry+0x272/0x2a0 [&lt;ffffffff81173181&gt;] ? sys_write+0x51/0x90 [&lt;ffffffff8100b172&gt;] ? system_call_fastpath+0x16/0x1b Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 CPU 2: hi: 0, btch: 1 usd: 0 CPU 3: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 186, btch: 31 usd: 16 CPU 1: hi: 186, btch: 31 usd: 0 CPU 2: hi: 186, btch: 31 usd: 132 CPU 3: hi: 186, btch: 31 usd: 54 Node 0 Normal per-cpu: CPU 0: hi: 186, btch: 31 usd: 53 CPU 1: hi: 186, btch: 31 usd: 30 CPU 2: hi: 186, btch: 31 usd: 178 CPU 3: hi: 186, btch: 31 usd: 59 active_anon:2493 inactive_anon:419 isolated_anon:0 active_file:6765 inactive_file:2834208 isolated_file:64 unevictable:0 dirty:0 writeback:62 unstable:0 free:30040 slab_reclaimable:4392 slab_unreclaimable:97285 mapped:2752 shmem:1 pagetables:936 bounce:0 Node 0 DMA free:15564kB min:80kB low:100kB high:120kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15156kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 2991 12081 12081 Node 0 DMA32 free:53164kB min:16712kB low:20888kB high:25068kB active_anon:0kB inactive_anon:4kB active_file:88kB inactive_file:2643896kB unevictable:0kB isolated(anon):0kB isolated(file):256kB present:3063392kB mlocked:0kB dirty:0kB writeback:248kB mapped:0kB shmem:0kB slab_reclaimable:5104kB slab_unreclaimable:32240kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:189512 all_unreclaimable? no lowmem_reserve[]: 0 0 9090 9090 Node 0 Normal free:51432kB min:50784kB low:63480kB high:76176kB active_anon:9972kB inactive_anon:1672kB active_file:26972kB inactive_file:8692936kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:9308160kB mlocked:0kB dirty:0kB writeback:0kB mapped:11008kB shmem:4kB slab_reclaimable:12464kB slab_unreclaimable:356900kB kernel_stack:1592kB pagetables:3744kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1090588 all_unreclaimable? no lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 1*4kB 1*8kB 0*16kB 2*32kB 0*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15564kB Node 0 DMA32: 583*4kB 630*8kB 548*16kB 373*32kB 228*64kB 34*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 53164kB Node 0 Normal: 1169*4kB 1168*8kB 658*16kB 324*32kB 178*64kB 7*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 51300kB 389500 total pagecache pages 322 pages in swap cache Swap cache stats: add 3069, delete 2747, find 535/672 Free swap = 14410400kB Total swap = 14417912kB 3145712 pages RAM 98606 pages reserved 397745 pages shared 2558959 pages non-shared Out of memory: kill process 6405 (qmgr) score 16072 or a child Killed process 6405 (qmgr) vsz:64288kB, anon-rss:668kB, file-rss:1884kB </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/aeb95442-df8f-11e0-9909-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/aeb95442-df8f-11e0-9909-52540025f9af</a></p> <p>Please refer to the attached recovery-oss-scale.1316084894.log.tar.bz2 for more logs.</p> <p>The "ENOSPC" issue is similar to <a href="https://jira.whamcloud.com/browse/LU-657" title="recovery-mds-scale (FLAVOR=MDS): client load failed" class="issue-link" data-issue-key="LU-657"><del>LU-657</del></a>.</p> <p>The second run hit <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a>, so I pushed a combined patch in <a href="http://review.whamcloud.com/1383" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1383</a> to trigger a Jenkins build which contains both the fixes. I'm verifying the new build.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/282/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #282</a><br/> <a href="https://jira.whamcloud.com/browse/LU-683" title="LBUG: (osc_io.c:336:osc_io_commit_write()) ASSERTION(to &gt; 0) failed" class="issue-link" data-issue-key="LU-683"><del>LU-683</del></a>: handle short write in cl_io_commit_write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4" class="external-link" target="_blank" rel="nofollow noopener">ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4</a><br/> Files : </p> <ul> <li>lustre/llite/rw.c</li> <li>lustre/llite/rw26.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/282/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #282</a><br/> <a href="https://jira.whamcloud.com/browse/LU-683" title="LBUG: (osc_io.c:336:osc_io_commit_write()) ASSERTION(to &gt; 0) failed" class="issue-link" data-issue-key="LU-683"><del>LU-683</del></a>: handle short write in cl_io_commit_write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4" class="external-link" target="_blank" rel="nofollow noopener">ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4</a><br/> Files : </p> <ul> <li>lustre/llite/rw.c</li> <li>lustre/llite/rw26.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/282/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #282</a><br/> <a href="https://jira.whamcloud.com/browse/LU-683" title="LBUG: (osc_io.c:336:osc_io_commit_write()) ASSERTION(to &gt; 0) failed" class="issue-link" data-issue-key="LU-683"><del>LU-683</del></a>: handle short write in cl_io_commit_write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4" class="external-link" target="_blank" rel="nofollow noopener">ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4</a><br/> Files : </p> <ul> <li>lustre/llite/rw26.c</li> <li>lustre/llite/rw.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/282/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #282</a><br/> <a href="https://jira.whamcloud.com/browse/LU-683" title="LBUG: (osc_io.c:336:osc_io_commit_write()) ASSERTION(to &gt; 0) failed" class="issue-link" data-issue-key="LU-683"><del>LU-683</del></a>: handle short write in cl_io_commit_write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4" class="external-link" target="_blank" rel="nofollow noopener">ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4</a><br/> Files : </p> <ul> <li>lustre/llite/rw.c</li> <li>lustre/llite/rw26.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/282/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #282</a><br/> <a href="https://jira.whamcloud.com/browse/LU-683" title="LBUG: (osc_io.c:336:osc_io_commit_write()) ASSERTION(to &gt; 0) failed" class="issue-link" data-issue-key="LU-683"><del>LU-683</del></a>: handle short write in cl_io_commit_write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4" class="external-link" target="_blank" rel="nofollow noopener">ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4</a><br/> Files : </p> <ul> <li>lustre/llite/rw.c</li> <li>lustre/llite/rw26.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/282/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #282</a><br/> <a href="https://jira.whamcloud.com/browse/LU-683" title="LBUG: (osc_io.c:336:osc_io_commit_write()) ASSERTION(to &gt; 0) failed" class="issue-link" data-issue-key="LU-683"><del>LU-683</del></a>: handle short write in cl_io_commit_write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4" class="external-link" target="_blank" rel="nofollow noopener">ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4</a><br/> Files : </p> <ul> <li>lustre/llite/rw.c</li> <li>lustre/llite/rw26.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/282/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #282</a><br/> <a href="https://jira.whamcloud.com/browse/LU-683" title="LBUG: (osc_io.c:336:osc_io_commit_write()) ASSERTION(to &gt; 0) failed" class="issue-link" data-issue-key="LU-683"><del>LU-683</del></a>: handle short write in cl_io_commit_write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4" class="external-link" target="_blank" rel="nofollow noopener">ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4</a><br/> Files : </p> <ul> <li>lustre/llite/rw.c</li> <li>lustre/llite/rw26.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/282/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #282</a><br/> <a href="https://jira.whamcloud.com/browse/LU-683" title="LBUG: (osc_io.c:336:osc_io_commit_write()) ASSERTION(to &gt; 0) failed" class="issue-link" data-issue-key="LU-683"><del>LU-683</del></a>: handle short write in cl_io_commit_write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4" class="external-link" target="_blank" rel="nofollow noopener">ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4</a><br/> Files : </p> <ul> <li>lustre/llite/rw26.c</li> <li>lustre/llite/rw.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/282/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #282</a><br/> <a href="https://jira.whamcloud.com/browse/LU-683" title="LBUG: (osc_io.c:336:osc_io_commit_write()) ASSERTION(to &gt; 0) failed" class="issue-link" data-issue-key="LU-683"><del>LU-683</del></a>: handle short write in cl_io_commit_write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4" class="external-link" target="_blank" rel="nofollow noopener">ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4</a><br/> Files : </p> <ul> <li>lustre/llite/rw.c</li> <li>lustre/llite/rw26.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/282/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #282</a><br/> <a href="https://jira.whamcloud.com/browse/LU-683" title="LBUG: (osc_io.c:336:osc_io_commit_write()) ASSERTION(to &gt; 0) failed" class="issue-link" data-issue-key="LU-683"><del>LU-683</del></a>: handle short write in cl_io_commit_write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4" class="external-link" target="_blank" rel="nofollow noopener">ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4</a><br/> Files : </p> <ul> <li>lustre/llite/rw26.c</li> <li>lustre/llite/rw.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/282/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #282</a><br/> <a href="https://jira.whamcloud.com/browse/LU-683" title="LBUG: (osc_io.c:336:osc_io_commit_write()) ASSERTION(to &gt; 0) failed" class="issue-link" data-issue-key="LU-683"><del>LU-683</del></a>: handle short write in cl_io_commit_write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4" class="external-link" target="_blank" rel="nofollow noopener">ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4</a><br/> Files : </p> <ul> <li>lustre/llite/rw26.c</li> <li>lustre/llite/rw.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/282/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #282</a><br/> <a href="https://jira.whamcloud.com/browse/LU-683" title="LBUG: (osc_io.c:336:osc_io_commit_write()) ASSERTION(to &gt; 0) failed" class="issue-link" data-issue-key="LU-683"><del>LU-683</del></a>: handle short write in cl_io_commit_write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4" class="external-link" target="_blank" rel="nofollow noopener">ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4</a><br/> Files : </p> <ul> <li>lustre/llite/rw26.c</li> <li>lustre/llite/rw.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/282/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #282</a><br/> <a href="https://jira.whamcloud.com/browse/LU-683" title="LBUG: (osc_io.c:336:osc_io_commit_write()) ASSERTION(to &gt; 0) failed" class="issue-link" data-issue-key="LU-683"><del>LU-683</del></a>: handle short write in cl_io_commit_write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4" class="external-link" target="_blank" rel="nofollow noopener">ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4</a><br/> Files : </p> <ul> <li>lustre/llite/rw.c</li> <li>lustre/llite/rw26.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/282/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #282</a><br/> <a href="https://jira.whamcloud.com/browse/LU-683" title="LBUG: (osc_io.c:336:osc_io_commit_write()) ASSERTION(to &gt; 0) failed" class="issue-link" data-issue-key="LU-683"><del>LU-683</del></a>: handle short write in cl_io_commit_write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4" class="external-link" target="_blank" rel="nofollow noopener">ff38d3f6ba13e8c251a2fdc070bae3cb2098a4a4</a><br/> Files : </p> <ul> <li>lustre/llite/rw.c</li> <li>lustre/llite/rw26.c</li> </ul> Development Rank 1|hzv8fz: Rank (Obsolete) 4902 Severity