Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-4292] Lustre data loss when client hang up https://jira.whamcloud.com/browse/LU-4292 Lustre <p>Error watched on Lustre 2.1.6 and later. Rhel 6.x data loss. If the file was open or modified approximately in 30 seconds before the server failed, all data in this file will be erased. Somebody watched a similar error?</p> OS RHEL 6.4, 6.3, Lustre versions lustre 2.1.6, 2.3, 2.4.1, 2.5. The error was watched on the different hardwares. LU-4292

Lustre data loss when client hang up

Bug Critical Closed Cannot Reproduce James Nunez Alexander server Fri, 22 Nov 2013 06:55:06 +0000 Wed, 26 Apr 2017 20:02:08 +0000 Wed, 26 Apr 2017 20:02:08 +0000 Lustre 2.4.1 0 6 <p>Could you please explain your testing process further? Does this affect existing files that are opened, or only newly created files? If the server fails, but the client is still running, and then the server is restarted, is there data loss?</p> <p>There is a prototype consisting of 2 servers. One server of meta data and object data. The second server is used as the client. Let's allow in file system already there was a certain file_1.txt file in which "a, b, c" was written. If from the client to execute record in this file "echo d >> /mnt/file_1.txt" that in file /proc/fs/osc/lustre-OST-*/cur_dirty_bytes will appear instead of 0 4096 (byte). Data are stored in "dirty_data" about 30 seconds then probably it is carried out from disk writing. And so, if the client machine fails before these will expire 30 seconds that in the /mnt/file_1.txt file will be gone not only those data which were just written (by means of the "echo d >> /mnt/file_1.txt" command), but also those that were written to it. I.e. this file will be empty. <br/> In case of study file Lustre-x.x/lustre/ChangeLog found a similar error which can be found bugzilla 12181, 12203, but it is dated 2007-04-01.</p> <p>James</p> <p>Could you please see if you can reproduce this reported behavior?</p> <p>Thanks</p> <p>Peter</p> <p>It would be desirable to learn, whether it managed to you to repeat this situation?</p> <p>Alexander, </p> <p>I'm having trouble recreating this problem. </p> <p>I have a single server with MDS/MGS and OSS and a single client. I write to a file using "echo" and I can see the cur_dirty_bytes stay at 4096 for a while and then change to 0, but I can't get any data loss or corruption. To fail the client I've used reboot and shutdown immediately after the echo command was issued. </p> <p>So, I may be doing things differently from you. How did you fail the client?</p> <p>James,</p> <p>In case of reboot or poweroff loss of data isn't watched. Loss of data is watched in case of hangup of an operating system or for example forced switch-off of power supply from the client. Respectively, if the client hangs up or will be compulsorily disconnected from power supply at that moment while in the cur_dirty_bytes file there will be a value 4096 that there will be a loss of data.</p> <p>Alexander, </p> <p>I've moved my testing to using VMs so that I can shut off machines to mimic a node going down. Here is what I can reproduce, using the b2_3 branch of Lustre:</p> <p>I have a single node MDS and OSS and two clients. One client writes data to a file called test_file and the write completes normally. That same client writes data to the existing file, I print cur_dirty_bytes which is 4096 and then I power off the VM, which does not flush the cache. I go to the second client and try and list the contents of the Lustre file system and it hangs. If I control C the listing, I get something like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@client2 ~]# ls -l /lustre/scratch/ ^Cls: cannot access /lustre/scratch/test_file: Interrupted system call total 0 -????????? ? ? ? ? ? test_file </pre> </div></div> <p>If I wait for a couple of minutes, the original file/data is restored, but the data in flight never makes it to the file:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@client2 ~]# ls /lustre/scratch/ client_2_touch test_file [root@client2 ~]# more /lustre/scratch/test_file a, b, c d, e, f g, h, i </pre> </div></div> <p>Does this match your experience? If you wait a few minutes, does the file that your client was writing to, /mnt/file_1 in your example, come back and is the original data "a, b, c" there?</p> <p>James,</p> <p> The experiment made by you matches that I saw off except for one point. I made experiment not on the VM and on real servers, respectively the cache on real servers was cleared (dropped).<br/> Probably on that the test_file file was recovered from a cache only after booted switched off by you VMs machine. Whether so it?<br/> Try to repeat your experiment, but so that in case of forced VM reset/switching off its cache would be dropped (is cleared).</p> <p>Hi Alexander, when you saw the empty, do you notice what's the file size?</p> <p>You can try to write something into the file, and then make sure the content has been written to OST, and then reproduce this problem. For example:</p> <p> echo "hello world" > test_file;<br/> sync; sync; sync<br/> run your test to append the file</p> <p>After you've reproduced the problem, please check what the file size is.</p> <p>Thanks</p> <p>Hi Jinshan,</p> <p>I wrote similar procedure:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#!/bin/bash path=/proc/fs/lustre/osc ost_num=test-OST000 name=cur_dirty_bytes <span class="code-keyword">while</span> [ 1 ]; <span class="code-keyword">do</span> <span class="code-keyword">for</span> j in 0 1; <span class="code-keyword">do</span> c=`cat $(path)/$(ost_num)$(j)*/$(name)`; <span class="code-keyword">if</span> [ $(c) -ne 0 ]; then sync; fi done sleep 0.3; done </pre> </div></div> <p>This script forces to drop the memories this from a cache (if they there are) on a disk with an interval approximately to equal 1 second.</p> <p>But it only minimizes possibility of a similar situation. On default about 30 seconds are stored in file system of cur_dirty_bytes then they automatically register from a memory cache in a disk. Thus the probability of loss of data remains if the client hangs up in that timepoint when the memories this in a cache still will be and the sync command yet didn't pass.</p> <p>Alexander, </p> <p>Can you reproduce the data loss on your cluster/nodes? In other words, is this a persistent problem, does it happen occasionally or did it stop happening?</p> <p>Thanks</p> <p>James,</p> <p>Yes at me this problem arises stablly, in case of described by me in the previous messages conditions. I can reproduce this error on the equipment also.</p> <p>James,</p> <p>Hello, whether there were at you any news on this subject?</p> <p>Alexander, </p> <p>Nothing new to report; I am not able to reproduce this error. Would you please let me know what your mount options are when you experience the data loss? Also, do you have logs from these nodes when the data loss happens and would you make those available to me?</p> <p>Thanks,</p> <p>James,</p> <p> I reproduced this situation on the following prototype:<br/> Two virtual machines, the first (VM1) is used as MDS/OSS, the second (VM2) as the client. X86_64 RHEL 6.4 operating system, lustre 2.4.1.</p> <p> The sequence of actions was following:</p> <p> On VM2 write data to /mnt/file:<br/> <span class="error">[root@n2 ~]</span># date && dmesg > /mnt/file && ll /mnt/file && date<br/> Thu Apr 10 02:58:16 EDT 2014<br/> <del>rw-r</del><del>r</del>- 1 root root 79650 Apr 10 02:58 /mnt/file<br/> Thu Apr 10 02:58:16 EDT 2014<br/> <span class="error">[root@n2 ~]</span>#</p> <p> On VM1 do:<br/> <span class="error">[root@n1 common]</span># date && ll /mnt/file && date<br/> Thu Apr 10 02:58:22 EDT 2014<br/> <del>rw-r</del><del>r</del>- 1 root root 79650 Apr 10 02:58 /mnt/file<br/> Thu Apr 10 02:58:22 EDT 2014</p> <p> The file at /mnt have size 79650.</p> <p> Force poweroff VM2.</p> <p> On VM1 do:<br/> <span class="error">[root@n1 common]</span># date && ll /mnt/file && date<br/> Thu Apr 10 02:58:33 EDT 2014<br/> <del>rw-r</del><del>r</del>- 1 root root 79650 Apr 10 02:58 /mnt/file<br/> Thu Apr 10 03:02:07 EDT 2014</p> <p> After about four minutes of waiting the command was executed also the file it seems not empty, but in case of repeated execution of the same command we receive:</p> <p><span class="error">[root@n1 common]</span># date && ll /mnt/file && date<br/> Thu Apr 10 03:02:13 EDT 2014<br/> <del>rw-r</del><del>r</del>- 1 root root 0 Apr 10 02:58 /mnt/file<br/> Thu Apr 10 03:02:13 EDT 2014<br/> <span class="error">[root@n1 common]</span># </p> <p> The file is empty. The VM2 server thus remained switched off.</p> <p> The output of file /var/log/messages of the VM1 server from the moment of removal of a supply from VM2 is given below:</p> <p>Apr 10 02:58:40 n1 kernel: Lustre: 3118:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">[sent 1397113113/real 1397113113]</span> req@ffff88005dc90400 x1464979110756852/t0(0) o106->lustre-OST0000@11.0.0.32@tcp:15/16 lens 296/280 e 0 to 1 dl 1397113120 ref 1 fl Rpc:X/0/ffffffff rc 0/-1<br/> Apr 10 02:58:48 n1 kernel: Lustre: 3118:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">[sent 1397113120/real 1397113120]</span> req@ffff88005dc90400 x1464979110756852/t0(0) o106->lustre-OST0000@11.0.0.32@tcp:15/16 lens 296/280 e 0 to 1 dl 1397113127 ref 1 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 10 02:58:55 n1 kernel: Lustre: 3118:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">[sent 1397113127/real 1397113127]</span> req@ffff88005dc90400 x1464979110756852/t0(0) o106->lustre-OST0000@11.0.0.32@tcp:15/16 lens 296/280 e 0 to 1 dl 1397113134 ref 1 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 10 02:59:02 n1 kernel: Lustre: 3118:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">[sent 1397113134/real 1397113134]</span> req@ffff88005dc90400 x1464979110756852/t0(0) o106->lustre-OST0000@11.0.0.32@tcp:15/16 lens 296/280 e 0 to 1 dl 1397113141 ref 1 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 10 02:59:09 n1 kernel: Lustre: 3118:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">[sent 1397113141/real 1397113142]</span> req@ffff88005dc90400 x1464979110756852/t0(0) o106->lustre-OST0000@11.0.0.32@tcp:15/16 lens 296/280 e 0 to 1 dl 1397113148 ref 1 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 10 02:59:23 n1 kernel: Lustre: 3118:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">[sent 1397113155/real 1397113157]</span> req@ffff88005dc90400 x1464979110756852/t0(0) o106->lustre-OST0000@11.0.0.32@tcp:15/16 lens 296/280 e 0 to 1 dl 1397113162 ref 1 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 10 02:59:23 n1 kernel: Lustre: 3118:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message<br/> Apr 10 02:59:38 n1 kernel: Lustre: 3118:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: <span class="error">[sent 1397113178/real 1397113178]</span> req@ffff88005dc90400 x1464979110756852/t0(0) o106->lustre-OST0000@11.0.0.32@tcp:15/16 lens 296/280 e 0 to 1 dl 1397113185 ref 1 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 10 02:59:38 n1 kernel: Lustre: 3118:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 495793 previous similar messages<br/> Apr 10 03:00:11 n1 kernel: Lustre: 3118:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: <span class="error">[sent 1397113210/real 1397113210]</span> req@ffff88005dc90400 x1464979110756852/t0(0) o106->lustre-OST0000@11.0.0.32@tcp:15/16 lens 296/280 e 0 to 1 dl 1397113217 ref 1 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 10 03:00:11 n1 kernel: Lustre: 3118:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4198137 previous similar messages<br/> Apr 10 03:01:14 n1 kernel: Lustre: 3118:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: <span class="error">[sent 1397113274/real 1397113274]</span> req@ffff88005dc90400 x1464979110756852/t0(0) o106->lustre-OST0000@11.0.0.32@tcp:15/16 lens 296/280 e 0 to 1 dl 1397113281 ref 1 fl Rpc:X/2/ffffffff rc 0/-1<br/> Apr 10 03:01:14 n1 kernel: Lustre: 3118:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9459123 previous similar messages<br/> Apr 10 03:01:53 n1 kernel: LNet: Service thread pid 3118 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:<br/> Apr 10 03:01:53 n1 kernel: Pid: 3118, comm: ll_ost00_002<br/> Apr 10 03:01:53 n1 kernel: <br/> Apr 10 03:01:53 n1 kernel: Call Trace:<br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa037c075>]</span> ? libcfs_nid2str+0x155/0x160 <span class="error">[libcfs]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffff81055ad3>]</span> ? __wake_up+0x53/0x70<br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa03ffc86>]</span> ? lnet_send+0x6d6/0xb60 <span class="error">[lnet]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa0400c33>]</span> ? LNetPut+0x313/0x860 <span class="error">[lnet]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa06319ca>]</span> ? ptl_send_buf+0x12a/0x550 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa03f8007>]</span> ? LNetMDAttach+0x427/0x5a0 <span class="error">[lnet]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa0634f05>]</span> ? ptl_send_rpc+0x565/0xc40 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa062c702>]</span> ? ptlrpc_expire_one_request+0xa2/0x440 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa062d956>]</span> ? ptlrpc_check_set+0xd26/0x1b20 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffff8109715c>]</span> ? remove_wait_queue+0x3c/0x50<br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa062e8b5>]</span> ? ptlrpc_set_wait+0x165/0x8c0 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffff81063410>]</span> ? default_wake_function+0x0/0x20<br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa05ee0a0>]</span> ? ldlm_work_gl_ast_lock+0x0/0x290 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa05f0f7b>]</span> ? ldlm_run_ast_work+0x1db/0x490 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa061298b>]</span> ? ldlm_glimpse_locks+0x3b/0x100 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa0ea2096>]</span> ? ofd_intent_policy+0x516/0x7d0 <span class="error">[ofd]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa05f0831>]</span> ? ldlm_lock_enqueue+0x361/0x8d0 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa06171ef>]</span> ? ldlm_handle_enqueue0+0x4ef/0x10b0 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa0617e16>]</span> ? ldlm_handle_enqueue+0x66/0x70 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa0617e20>]</span> ? ldlm_server_completion_ast+0x0/0x6c0 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa0e54300>]</span> ? ost_blocking_ast+0x0/0x10f0 <span class="error">[ost]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa0614830>]</span> ? ldlm_server_glimpse_ast+0x0/0x3b0 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa0e5d318>]</span> ? ost_handle+0x1db8/0x48e0 <span class="error">[ost]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa0640beb>]</span> ? ptlrpc_update_export_timer+0x4b/0x560 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa06493c8>]</span> ? ptlrpc_server_handle_request+0x398/0xc60 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa03705de>]</span> ? cfs_timer_arm+0xe/0x10 <span class="error">[libcfs]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa0381d9f>]</span> ? lc_watchdog_touch+0x6f/0x170 <span class="error">[libcfs]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa0640729>]</span> ? ptlrpc_wait_event+0xa9/0x290 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffff81055ad3>]</span> ? __wake_up+0x53/0x70<br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa064a75e>]</span> ? ptlrpc_main+0xace/0x1700 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa0649c90>]</span> ? ptlrpc_main+0x0/0x1700 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffff8100c0ca>]</span> ? child_rip+0xa/0x20<br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa0649c90>]</span> ? ptlrpc_main+0x0/0x1700 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffffa0649c90>]</span> ? ptlrpc_main+0x0/0x1700 <span class="error">[ptlrpc]</span><br/> Apr 10 03:01:53 n1 kernel: <span class="error">[<ffffffff8100c0c0>]</span> ? child_rip+0x0/0x20<br/> Apr 10 03:01:53 n1 kernel: <br/> Apr 10 03:01:53 n1 kernel: LustreError: dumping log to /tmp/lustre-log.1397113313.3118<br/> Apr 10 03:02:07 n1 kernel: Lustre: lustre-OST0000: haven't heard from client 7511411c-dd05-5423-aa8a-098f78d9739a (at 11.0.0.32@tcp) in 231 seconds. I think it's dead, and I am evicting it. exp ffff88007c62b400, cur 1397113327 expire 1397113177 last 1397113096<br/> Apr 10 03:02:07 n1 kernel: LustreError: 3118:0:(client.c:1048:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88005dc90400 x1464979110756852/t0(0) o106->lustre-OST0000@11.0.0.32@tcp:15/16 lens 296/280 e 0 to 1 dl 1397113334 ref 1 fl Rpc:X/2/ffffffff rc -5/-1<br/> Apr 10 03:02:07 n1 kernel: LustreError: 3118:0:(ldlm_lockd.c:709:ldlm_handle_ast_error()) ### client (nid 11.0.0.32@tcp) returned 0 from glimpse AST ns: filter-lustre-OST0000_UUID lock: ffff88005dd47000/0x1151412baa60ebe9 lrc: 2/0,0 mode: -<del>/PW res: <span class="error">[0x8ba2:0x0:0x0]</span>.0 rrc: 2 type: EXT <span class="error">[0->18446744073709551615]</span> (req 0</del>>81919) flags: 0x801000000000 nid: 11.0.0.32@tcp remote: 0xdcdb6436fc134adb expref: 2 pid: 3118 timeout: 0 lvb_type: 0<br/> Apr 10 03:02:07 n1 kernel: LNet: Service thread pid 3118 completed after 213.86s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).<br/> Apr 10 03:02:43 n1 kernel: Lustre: MGS: haven't heard from client 6abc5e98-73d8-60a9-2c7a-c9503f5979d6 (at 11.0.0.32@tcp) in 272 seconds. I think it's dead, and I am evicting it. exp ffff88005dc6e800, cur 1397113363 expire 1397113213 last 1397113091</p> <p>Alexander,<br/> Are you still experiencing this data corruption? If you are, what version of Lustre are you using?</p> <p>Thanks</p> <p>Please open a new ticket if you are still experiencing this problem.</p> Development Rank 1|hzw9wn: Rank (Obsolete) 11775 Severity