https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-653 Lustre <p>While running recovery-mds-scale with FLAVOR=OSS, it failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;~snip~&gt; Client load failed during failover. Exiting Found the END_RUN_FILE file: /home/yujian/test_logs/end_run_file client-15-ib Client load failed on node client-15-ib client client-15-ib load stdout and debug files : /tmp/recovery-mds-scale.log_run_dbench.sh-client-15-ib /tmp/recovery-mds-scale.log_run_dbench.sh-client-15-ib.debug 2011-08-31 01:45:08 Terminating clients loads ... Duration: 43200 Server failover period: 600 seconds Exited after: 0 seconds Number of failovers before exit: mds1: 0 times ost1: 0 times ost2: 0 times ost3: 0 times ost4: 0 times ost5: 0 times ost6: 0 times ost7: 0 times Status: FAIL: rc=1 </pre> </div></div> <p>/tmp/recovery-mds-scale.log_run_dbench.sh-client-15-ib:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;~snip~&gt; 2 5122 0.00 MB/sec execute 40 sec latency 144211.791 ms 2 5122 0.00 MB/sec execute 41 sec latency 145211.965 ms 2 5122 0.00 MB/sec execute 42 sec latency 146212.143 ms [5748] write failed on handle 11108 (Cannot send after transport endpoint shutdown) Child failed with status 1 </pre> </div></div> <p>/tmp/recovery-mds-scale.log_run_dbench.sh-client-15-ib.debug:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2011-08-31 01:40:59: dbench run starting + mkdir -p /mnt/lustre/d0.dbench-client-15-ib + load_pid=3602 + wait 3602 + rundbench -D /mnt/lustre/d0.dbench-client-15-ib 2 touch: missing file operand Try `touch --help' for more information. + '[' 1 -eq 0 ']' ++ date '+%F %H:%M:%S' + echoerr '2011-08-31 01:43:43: dbench failed' + echo '2011-08-31 01:43:43: dbench failed' 2011-08-31 01:43:43: dbench failed </pre> </div></div> <p>Syslog on client node client-15-ib showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Aug 31 01:42:51 client-15 kernel: Lustre: 2534:0:(client.c:2530:ptlrpc_replay_interpret()) @@@ Version mismatch during replay Aug 31 01:42:51 client-15 kernel: req@ffff880308397400 x1378646783695240/t908(908) o-1-&gt;lustre-OST0000_UUID@192.168.4.135@o2ib:6/4 lens 512/400 e 1 to 0 dl 1314780217 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1Aug 31 01:42:55 client-15 kernel: Lustre: 2534:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1378646783702455 sent from lustre-OST0002-osc-ffff880326e33c00 to NID 192.168.4.134@o2ib has failed due to network error: [sent 1314780175] [real_sent 1314780175] [current 1314780175] [deadline 26s] [delay -26s] req@ffff8803118e7000 x13786467837 02455/t0(0) o-1-&gt;lustre-OST0002_UUID@192.168.4.134@o2ib:28/4 lens 368/512 e 0 to 1 dl 1314780201 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Aug 31 01:42:55 client-15 kernel: Lustre: 2534:0:(client.c:1778:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Aug 31 01:43:10 client-15 kernel: LustreError: 11-0: an error occurred while communicating with 192.168.4.135@o2ib. The ost_connect operation failed with -19Aug 31 01:43:10 client-15 kernel: LustreError: Skipped 6 previous similar messagesAug 31 01:43:15 client-15 kernel: Lustre: 2535:0:(import.c:526:import_select_connection()) lustre-OST0002-osc-ffff880326e33c00: tried all connections, increasing latency to 21sAug 31 01:43:15 client-15 kernel: Lustre: 2535:0:(import.c:526:import_select_connection()) Skipped 6 previous similar messages Aug 31 01:43:32 client-15 kernel: Lustre: lustre-OST0002-osc-ffff880326e33c00: Connection restored to service lustre-OST0002 using nid 192.168.4.135@o2ib. Aug 31 01:43:36 client-15 kernel: INFO: task flush-lustre-1:2690 blocked for more than 120 seconds. Aug 31 01:43:36 client-15 kernel: "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 31 01:43:36 client-15 kernel: flush-lustre- D 0000000000000002 0 2690 2 0x00000080Aug 31 01:43:36 client-15 kernel: ffff88030e35d9a0 0000000000000046 0000000000000000 ffffffffa0303424 Aug 31 01:43:36 client-15 kernel: 0000000000000000 ffff88031f3fdc00 ffff88030e35d930 00000001002932fc Aug 31 01:43:36 client-15 kernel: ffff880325e47078 ffff88030e35dfd8 000000000000f598 ffff880325e47078 Aug 31 01:43:36 client-15 kernel: Call Trace:Aug 31 01:43:36 client-15 kernel: [&lt;ffffffffa0303424&gt;] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs]Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8110d320&gt;] ? sync_page+0x0/0x50Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff814db3c3&gt;] io_schedule+0x73/0xc0 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8110d35d&gt;] sync_page+0x3d/0x50 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff814dbada&gt;] __wait_on_bit_lock+0x5a/0xc0 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8110d2f7&gt;] __lock_page+0x67/0x70 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8108e140&gt;] ? wake_bit_function+0x0/0x50 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff81120d17&gt;] ? __writepage+0x17/0x40 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff81122272&gt;] write_cache_pages+0x392/0x4a0 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff81120d00&gt;] ? __writepage+0x0/0x40 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff811223a4&gt;] generic_writepages+0x24/0x30 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff811223d1&gt;] do_writepages+0x21/0x40 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8119bbdd&gt;] writeback_single_inode+0xdd/0x2c0 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8119bfde&gt;] writeback_sb_inodes+0xce/0x180 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8119c13b&gt;] writeback_inodes_wb+0xab/0x1b0 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8119c4db&gt;] wb_writeback+0x29b/0x3f0 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff814dac27&gt;] ? thread_return+0x4e/0x777 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8107a1a2&gt;] ? del_timer_sync+0x22/0x30 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8119c7c9&gt;] wb_do_writeback+0x199/0x240 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8119c8d3&gt;] bdi_writeback_task+0x63/0x1b0 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8108dfc7&gt;] ? bit_waitqueue+0x17/0xd0 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff81130bd0&gt;] ? bdi_start_fn+0x0/0x100 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff81130c56&gt;] bdi_start_fn+0x86/0x100 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff81130bd0&gt;] ? bdi_start_fn+0x0/0x100 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8108dd96&gt;] kthread+0x96/0xa0 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8100c1ca&gt;] child_rip+0xa/0x20 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8108dd00&gt;] ? kthread+0x0/0xa0 Aug 31 01:43:36 client-15 kernel: [&lt;ffffffff8100c1c0&gt;] ? child_rip+0x0/0x20 Aug 31 01:43:43 client-15 kernel: Lustre: 2534:0:(import.c:1160:completed_replay_interpret()) lustre-OST0000-osc-ffff880326e33c00: version recovery fails, reconnecting Aug 31 01:43:43 client-15 kernel: LustreError: 167-0: This client was evicted by lustre-OST0000; in progress operations using this service will fail. </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/e68e0d04-d3b4-11e0-8d02-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e68e0d04-d3b4-11e0-8d02-52540025f9af</a></p> <p>Please refer to the attached recovery-mds-scale.1314780314.log.tar.bz2 for more logs.</p> <br/> Lustre Tag: v2_1_0_0_RC1<br/> Lustre Build: <a href="http://newbuild.whamcloud.com/job/lustre-master/274/">http://newbuild.whamcloud.com/job/lustre-master/274/</a><br/> e2fsprogs Build: <a href="http://newbuild.whamcloud.com/job/e2fsprogs-master/42/">http://newbuild.whamcloud.com/job/e2fsprogs-master/42/</a><br/> Distro/Arch: RHEL6/x86_64(in-kernel OFED, kernel version: 2.6.32-131.6.1.el6.x86_64)<br/> ENABLE_QUOTA=yes<br/> FAILURE_MODE=HARD<br/> FLAVOR=OSS<br/> <br/> MGS/MDS Nodes: fat-amd-1-ib<br/> <br/> OSS Nodes: fat-amd-3-ib(active), fat-amd-4-ib(active)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\ /<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST1 (active in fat-amd-3-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST2 (active in fat-amd-4-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST3 (active in fat-amd-3-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST4 (active in fat-amd-4-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST5 (active in fat-amd-3-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST6 (active in fat-amd-4-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;fat-amd-2-ib(OST7)<br/> <br/> Client Nodes: client-[1,2,4,5,12,13,15],fat-intel-4<br/> <br/> Network Addresses:<br/> fat-amd-1-ib: 192.168.4.132<br/> fat-amd-2-ib: 192.168.4.133<br/> fat-amd-3-ib: 192.168.4.134<br/> fat-amd-4-ib: 192.168.4.135<br/> client-1-ib: 192.168.4.1<br/> client-2-ib: 192.168.4.2<br/> client-4-ib: 192.168.4.4<br/> client-5-ib: 192.168.4.5<br/> client-12-ib: 192.168.4.12<br/> client-13-ib: 192.168.4.13<br/> client-15-ib: 192.168.4.15<br/> fat-intel-4-ib: 192.168.4.131<br/> LU-653

recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)

Bug Blocker Resolved Fixed Oleg Drokin Jian Yu Wed, 31 Aug 2011 06:15:55 +0000 Fri, 16 Sep 2011 11:12:06 +0000 Fri, 16 Sep 2011 11:12:06 +0000 Lustre 2.1.0 Lustre 2.1.0 0 2 <p>Oleg is going to look into this one</p> <p>After reviewing the logs I had a discussion with Mike about this about some strange stuff in there.</p> <p>First of all there seems to be a problem where we wait for clients to replay long-committed transactions. This should not introduce a problem like visible here.</p> <p>It seems we need more logs, in particular "+inode +rpctrace +ha"</p> <p>YuJian can you please run this test several times with the additional logging specified above (make sure it propagates to OSTs)<br/> and with patch for the incorrect gap in sequence that is located at <a href="http://review.whamcloud.com/1318" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1318</a></p> <blockquote><p>YuJian can you please run this test several times with the additional logging specified above (make sure it propagates to OSTs) and with patch for the incorrect gap in sequence that is located at <a href="http://review.whamcloud.com/1318" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1318</a></p></blockquote> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-reviews/1967/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/1967/</a><br/> Distro/Arch: RHEL6/x86_64 (in-kernel OFED)<br/> ENABLE_QUOTA=yes<br/> FAILURE_MODE=HARD<br/> FLAVOR=OSS</p> <p>recovery-mds-scale(FLAVOR=OSS) failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>==== Checking the clients loads AFTER failover -- failure NOT OK Client load failed on node client-12-ib, rc=1 Client load failed during failover. Exiting Found the END_RUN_FILE file: /home/yujian/test_logs/end_run_file client-15-ib Client load failed on node client-15-ib client client-15-ib load stdout and debug files : /tmp/recovery-mds-scale.log_run_dbench.sh-client-15-ib /tmp/recovery-mds-scale.log_run_dbench.sh-client-15-ib.debug </pre> </div></div> <p>/tmp/recovery-mds-scale.log_run_dbench.sh-client-15-ib:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;~snip~&gt; 2 3122 0.00 MB/sec execute 211 sec latency 283720.897 ms 2 3122 0.00 MB/sec execute 212 sec latency 284720.990 ms [3811] unlink ./clients/client0/~dmtmp/PM/PMC184.TMP failed (Input/output error) - expected NT_STATUS_OK ERROR: child 0 failed at line 3811 Child failed with status 1 </pre> </div></div> <p>/tmp/recovery-mds-scale.log_run_dbench.sh-client-15-ib.debug:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;~snip~&gt; 2011-09-02 04:10:10: dbench run starting + mkdir -p /mnt/lustre/d0.dbench-client-15-ib + load_pid=15556 + wait 15556 + rundbench -D /mnt/lustre/d0.dbench-client-15-ib 2 touch: missing file operand Try `touch --help' for more information. + '[' 1 -eq 0 ']' ++ date '+%F %H:%M:%S' + echoerr '2011-09-02 04:15:44: dbench failed' + echo '2011-09-02 04:15:44: dbench failed' 2011-09-02 04:15:44: dbench failed </pre> </div></div> <p>Syslog on client-15-ib showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Sep 2 04:06:11 client-15 kernel: Lustre: 13220:0:(client.c:2530:ptlrpc_replay_interpret()) @@@ Version mismatch during replay Sep 2 04:06:11 client-15 kernel: req@ffff880311f5e000 x1378836125083746/t16967(16967) o-1-&gt;lustre-OST0005_UUID@192.168.4.135@o2ib:28/4 lens 408/400 e 0 to 0 dl 1314961626 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 Sep 2 04:06:11 client-15 kernel: Lustre: 13220:0:(client.c:2530:ptlrpc_replay_interpret()) @@@ Version mismatch during replay Sep 2 04:06:11 client-15 kernel: req@ffff8803127f4000 x1378836125084042/t16981(16981) o-1-&gt;lustre-OST0005_UUID@192.168.4.135@o2ib:28/4 lens 408/400 e 0 to 0 dl 1314961634 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 Sep 2 04:07:19 client-15 kernel: INFO: task flush-lustre-1:13377 blocked for more than 120 seconds. Sep 2 04:07:19 client-15 kernel: "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 2 04:07:19 client-15 kernel: flush-lustre- D 0000000000000001 0 13377 2 0x00000080 Sep 2 04:07:19 client-15 kernel: ffff8803139c39a0 0000000000000046 0000000000000000 ffffffffa08d7424 Sep 2 04:07:19 client-15 kernel: 0000000000000000 ffff88030a086300 ffff8803139c3930 0000000100cf4d29 Sep 2 04:07:19 client-15 kernel: ffff8803139c1b38 ffff8803139c3fd8 000000000000f598 ffff8803139c1b38 Sep 2 04:07:19 client-15 kernel: Call Trace: Sep 2 04:07:19 client-15 kernel: [&lt;ffffffffa08d7424&gt;] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs] Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8110d320&gt;] ? sync_page+0x0/0x50 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff814db3c3&gt;] io_schedule+0x73/0xc0 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8110d35d&gt;] sync_page+0x3d/0x50 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff814dbada&gt;] __wait_on_bit_lock+0x5a/0xc0 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8110d2f7&gt;] __lock_page+0x67/0x70 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8108e140&gt;] ? wake_bit_function+0x0/0x50 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff81120d17&gt;] ? __writepage+0x17/0x40 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff81122272&gt;] write_cache_pages+0x392/0x4a0 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff81120d00&gt;] ? __writepage+0x0/0x40 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff811223a4&gt;] generic_writepages+0x24/0x30 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff811223d1&gt;] do_writepages+0x21/0x40 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8119bbdd&gt;] writeback_single_inode+0xdd/0x2c0 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8119bfde&gt;] writeback_sb_inodes+0xce/0x180 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8119c13b&gt;] writeback_inodes_wb+0xab/0x1b0 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8119c4db&gt;] wb_writeback+0x29b/0x3f0 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff814dac27&gt;] ? thread_return+0x4e/0x777 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8107a1a2&gt;] ? del_timer_sync+0x22/0x30 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8119c7c9&gt;] wb_do_writeback+0x199/0x240 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8119c8d3&gt;] bdi_writeback_task+0x63/0x1b0 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8108dfc7&gt;] ? bit_waitqueue+0x17/0xd0 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff81130bd0&gt;] ? bdi_start_fn+0x0/0x100 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff81130c56&gt;] bdi_start_fn+0x86/0x100 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff81130bd0&gt;] ? bdi_start_fn+0x0/0x100 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8108dd96&gt;] kthread+0x96/0xa0 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8100c1ca&gt;] child_rip+0xa/0x20 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8108dd00&gt;] ? kthread+0x0/0xa0 Sep 2 04:07:19 client-15 kernel: [&lt;ffffffff8100c1c0&gt;] ? child_rip+0x0/0x20 Sep 2 04:08:16 client-15 kernel: Lustre: 13220:0:(import.c:1160:completed_replay_interpret()) lustre-OST0005-osc-ffff88030d16ec00: version recovery fails, reconnecting Sep 2 04:08:16 client-15 kernel: LustreError: 167-0: This client was evicted by lustre-OST0005; in progress operations using this service will fail. &lt;~snip~&gt; Sep 2 04:14:22 client-15 kernel: Lustre: 13220:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1378836125097599 sent from lustre-OST0003-osc-ffff88030d16ec00 to NID 192.168.4.135@o2ib has failed due to network error: [sent 1314962062] [real_sent 1314962062] [current 1314962062] [deadline 26s] [delay -26s] req@ffff8803145b7000 x1378836125097599/t0(0) o-1-&gt;lustre-OST0003_UUID@192.168.4.135@o2ib:28/4 lens 368/512 e 0 to 1 dl 1314962088 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1 Sep 2 04:14:22 client-15 kernel: Lustre: 13220:0:(client.c:1778:ptlrpc_expire_one_request()) Skipped 92 previous similar messages Sep 2 04:14:40 client-15 kernel: Lustre: 13220:0:(client.c:2530:ptlrpc_replay_interpret()) @@@ Version mismatch during replay Sep 2 04:14:40 client-15 kernel: req@ffff88031387c800 x1378836125088753/t24049(24049) o-1-&gt;lustre-OST0003_UUID@192.168.4.134@o2ib:6/4 lens 512/400 e 0 to 0 dl 1314962107 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 Sep 2 04:15:44 client-15 kernel: Lustre: 13220:0:(import.c:1160:completed_replay_interpret()) lustre-OST0003-osc-ffff88030d16ec00: version recovery fails, reconnecting Sep 2 04:15:44 client-15 kernel: LustreError: 167-0: This client was evicted by lustre-OST0003; in progress operations using this service will fail. Sep 2 04:15:44 client-15 kernel: LustreError: 13219:0:(client.c:1060:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff880312372400 x1378836125097748/t0(0) o-1-&gt;lustre-OST0003_UUID@192.168.4.134@o2ib:28/4 lens 296/352 e 0 to 0 dl 0 ref 1 fl Rpc:/ffffffff/ffffffff rc 0/-1 Sep 2 04:15:44 client-15 kernel: LustreError: 13219:0:(client.c:1060:ptlrpc_import_delay_req()) Skipped 4 previous similar messages Sep 2 04:15:44 client-15 kernel: LustreError: 15565:0:(namei.c:1111:ll_objects_destroy()) obd destroy objid 0x466 error -5 Sep 2 04:15:44 client-15 kernel: Lustre: lustre-OST0003-osc-ffff88030d16ec00: Connection restored to service lustre-OST0003 using nid 192.168.4.134@o2ib. </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/d53a0c1a-d55d-11e0-8d02-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d53a0c1a-d55d-11e0-8d02-52540025f9af</a></p> <p>Please refer to the attached recovery-oss-scale.1314962198.log.tar.bz2 for more logs.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-reviews/2042/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/2042/</a><br/> Distro/Arch: RHEL6/x86_64 (in-kernel OFED)<br/> ENABLE_QUOTA=yes<br/> FAILURE_MODE=HARD<br/> FLAVOR=OSS</p> <p>recovery-mds-scale(FLAVOR=OSS) failed: <a href="https://maloo.whamcloud.com/test_sets/bade20c4-d93e-11e0-8d02-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/bade20c4-d93e-11e0-8d02-52540025f9af</a></p> <p>Please refer to the attached recovery-oss-scale.1315388587.log.tar.bz2 for more logs.</p> <p>Oleg,</p> <p>Can you take a look at the updated logs from Yu Jian so that we can make a decision on RC2 tomorrow?</p> <p>Thanks,</p> <p>Jessica</p> <p>Jessica, Oleg, I did look already, there is another patch was pushed to gerrit to get more info. It will be tested today</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-reviews/2087/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/2087/</a><br/> Distro/Arch: RHEL6/x86_64 (in-kernel OFED)<br/> ENABLE_QUOTA=yes<br/> FAILURE_MODE=HARD<br/> FLAVOR=OSS</p> <p>recovery-mds-scale(FLAVOR=OSS) failed:<br/> <a href="https://maloo.whamcloud.com/test_sets/a7591bee-dac9-11e0-8d02-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a7591bee-dac9-11e0-8d02-52540025f9af</a><br/> <a href="https://maloo.whamcloud.com/test_sets/57731e6e-dac8-11e0-8d02-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/57731e6e-dac8-11e0-8d02-52540025f9af</a></p> <p>Please refer to the attached recovery-oss-scale.1315558735.log.tar.bz2 and recovery-oss-scale.1315560362.log.tar.bz2 for more logs.</p> <p>I found the issue causing that. Let's look at fsfilt version get code:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">static</span> __u64 get_i_version(struct inode *inode) { #<span class="code-keyword">if</span> (LINUX_VERSION_CODE &gt;= KERNEL_VERSION(2,6,27)) &amp;&amp; defined(HAVE_EXT4_LDISKFS) <span class="code-keyword">return</span> inode-&gt;i_version; #<span class="code-keyword">else</span> <span class="code-keyword">return</span> EXT3_I(inode)-&gt;i_fs_version; #endif } </pre> </div></div> <p>With RHEL6+ext4 it uses inode-&gt;i_version field to store versions. Meanwhile this field is used internally by ext4 and can be changed. This breaks recovery and cause client eviction.</p> <p>We need to use the same i_fs_version in RHEL6 too, moreover the series contains already patch to support it. I've updated review tracker with the fix and new build showed no signs of issue.</p> <p>The latest version of this patch is confirmed as resolving this issue - <a href="http://review.whamcloud.com/#change,1342" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1342</a> . It should land subject to inspections.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/280/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #280</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> Ignore last transno from clients with no outstanding transactions</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=280d8b6a1538f4ad9d2acdd045b970811e895c43" class="external-link" target="_blank" rel="nofollow noopener">280d8b6a1538f4ad9d2acdd045b970811e895c43</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/280/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #280</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> Ignore last transno from clients with no outstanding transactions</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=280d8b6a1538f4ad9d2acdd045b970811e895c43" class="external-link" target="_blank" rel="nofollow noopener">280d8b6a1538f4ad9d2acdd045b970811e895c43</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/280/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #280</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> Ignore last transno from clients with no outstanding transactions</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=280d8b6a1538f4ad9d2acdd045b970811e895c43" class="external-link" target="_blank" rel="nofollow noopener">280d8b6a1538f4ad9d2acdd045b970811e895c43</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/280/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #280</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> Ignore last transno from clients with no outstanding transactions</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=280d8b6a1538f4ad9d2acdd045b970811e895c43" class="external-link" target="_blank" rel="nofollow noopener">280d8b6a1538f4ad9d2acdd045b970811e895c43</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/280/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #280</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> Ignore last transno from clients with no outstanding transactions</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=280d8b6a1538f4ad9d2acdd045b970811e895c43" class="external-link" target="_blank" rel="nofollow noopener">280d8b6a1538f4ad9d2acdd045b970811e895c43</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/280/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #280</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> Ignore last transno from clients with no outstanding transactions</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=280d8b6a1538f4ad9d2acdd045b970811e895c43" class="external-link" target="_blank" rel="nofollow noopener">280d8b6a1538f4ad9d2acdd045b970811e895c43</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/280/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #280</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> Ignore last transno from clients with no outstanding transactions</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=280d8b6a1538f4ad9d2acdd045b970811e895c43" class="external-link" target="_blank" rel="nofollow noopener">280d8b6a1538f4ad9d2acdd045b970811e895c43</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/280/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #280</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> Ignore last transno from clients with no outstanding transactions</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=280d8b6a1538f4ad9d2acdd045b970811e895c43" class="external-link" target="_blank" rel="nofollow noopener">280d8b6a1538f4ad9d2acdd045b970811e895c43</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/280/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #280</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> Ignore last transno from clients with no outstanding transactions</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=280d8b6a1538f4ad9d2acdd045b970811e895c43" class="external-link" target="_blank" rel="nofollow noopener">280d8b6a1538f4ad9d2acdd045b970811e895c43</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/280/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #280</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> Ignore last transno from clients with no outstanding transactions</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=280d8b6a1538f4ad9d2acdd045b970811e895c43" class="external-link" target="_blank" rel="nofollow noopener">280d8b6a1538f4ad9d2acdd045b970811e895c43</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/280/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #280</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> Ignore last transno from clients with no outstanding transactions</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=280d8b6a1538f4ad9d2acdd045b970811e895c43" class="external-link" target="_blank" rel="nofollow noopener">280d8b6a1538f4ad9d2acdd045b970811e895c43</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/280/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #280</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> Ignore last transno from clients with no outstanding transactions</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=280d8b6a1538f4ad9d2acdd045b970811e895c43" class="external-link" target="_blank" rel="nofollow noopener">280d8b6a1538f4ad9d2acdd045b970811e895c43</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/280/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #280</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> Ignore last transno from clients with no outstanding transactions</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=280d8b6a1538f4ad9d2acdd045b970811e895c43" class="external-link" target="_blank" rel="nofollow noopener">280d8b6a1538f4ad9d2acdd045b970811e895c43</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/280/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #280</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> Ignore last transno from clients with no outstanding transactions</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=280d8b6a1538f4ad9d2acdd045b970811e895c43" class="external-link" target="_blank" rel="nofollow noopener">280d8b6a1538f4ad9d2acdd045b970811e895c43</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_lib.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/281/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #281</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> i_version shouldn't be used for VBR</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=1e9326917af52f3d01920411465476154b2807d0" class="external-link" target="_blank" rel="nofollow noopener">1e9326917af52f3d01920411465476154b2807d0</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/281/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #281</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> i_version shouldn't be used for VBR</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=1e9326917af52f3d01920411465476154b2807d0" class="external-link" target="_blank" rel="nofollow noopener">1e9326917af52f3d01920411465476154b2807d0</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/281/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #281</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> i_version shouldn't be used for VBR</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=1e9326917af52f3d01920411465476154b2807d0" class="external-link" target="_blank" rel="nofollow noopener">1e9326917af52f3d01920411465476154b2807d0</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/281/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #281</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> i_version shouldn't be used for VBR</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=1e9326917af52f3d01920411465476154b2807d0" class="external-link" target="_blank" rel="nofollow noopener">1e9326917af52f3d01920411465476154b2807d0</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/281/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #281</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> i_version shouldn't be used for VBR</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=1e9326917af52f3d01920411465476154b2807d0" class="external-link" target="_blank" rel="nofollow noopener">1e9326917af52f3d01920411465476154b2807d0</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/281/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #281</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> i_version shouldn't be used for VBR</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=1e9326917af52f3d01920411465476154b2807d0" class="external-link" target="_blank" rel="nofollow noopener">1e9326917af52f3d01920411465476154b2807d0</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/281/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #281</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> i_version shouldn't be used for VBR</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=1e9326917af52f3d01920411465476154b2807d0" class="external-link" target="_blank" rel="nofollow noopener">1e9326917af52f3d01920411465476154b2807d0</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/281/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #281</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> i_version shouldn't be used for VBR</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=1e9326917af52f3d01920411465476154b2807d0" class="external-link" target="_blank" rel="nofollow noopener">1e9326917af52f3d01920411465476154b2807d0</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/281/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #281</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> i_version shouldn't be used for VBR</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=1e9326917af52f3d01920411465476154b2807d0" class="external-link" target="_blank" rel="nofollow noopener">1e9326917af52f3d01920411465476154b2807d0</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/281/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #281</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> i_version shouldn't be used for VBR</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=1e9326917af52f3d01920411465476154b2807d0" class="external-link" target="_blank" rel="nofollow noopener">1e9326917af52f3d01920411465476154b2807d0</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/281/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #281</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> i_version shouldn't be used for VBR</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=1e9326917af52f3d01920411465476154b2807d0" class="external-link" target="_blank" rel="nofollow noopener">1e9326917af52f3d01920411465476154b2807d0</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/281/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #281</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> i_version shouldn't be used for VBR</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=1e9326917af52f3d01920411465476154b2807d0" class="external-link" target="_blank" rel="nofollow noopener">1e9326917af52f3d01920411465476154b2807d0</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/281/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #281</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> i_version shouldn't be used for VBR</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=1e9326917af52f3d01920411465476154b2807d0" class="external-link" target="_blank" rel="nofollow noopener">1e9326917af52f3d01920411465476154b2807d0</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/281/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #281</a><br/> <a href="https://jira.whamcloud.com/browse/LU-653" title="recovery-mds-scale (FLAVOR=OSS): dbench write failed on handle 11108 (Cannot send after transport endpoint shutdown)" class="issue-link" data-issue-key="LU-653"><del>LU-653</del></a> i_version shouldn't be used for VBR</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=1e9326917af52f3d01920411465476154b2807d0" class="external-link" target="_blank" rel="nofollow noopener">1e9326917af52f3d01920411465476154b2807d0</a><br/> Files : </p> <ul> <li>lustre/lvfs/fsfilt_ext3.c</li> </ul> <p>Landed for 2.1</p> Development Rank 1|hzv8g7: Rank (Obsolete) 4903 Severity