https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-316 Lustre <p>system hang when running lustre-rsync-test test_5b</p> <p>Lustre: DEBUG MARKER: == lustre-rsync-test test 5b: Kill / restart lustre_rsync == 19:47:48 (1305254868)<br/> INFO: task lustre_rsync:8898 blocked for more than 120 seconds.<br/> "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> lustre_rsync D ffff88033fc24700 0 8898 1 0x00000080<br/> ffff8802fd019c40 0000000000000082 0000000000000000 ffff880032e769f0<br/> ffff8802fd019be0 ffffffff8105fe65 ffff8802fd019bc0 0000000100115300<br/> ffff88030a7bc678 ffff8802fd019fd8 0000000000010518 ffff88030a7bc678<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff8105fe65&gt;&#93;</span> ? task_new_fair+0xb5/0x100<br/> <span class="error">&#91;&lt;ffffffff814c9cf5&gt;&#93;</span> schedule_timeout+0x225/0x2f0<br/> <span class="error">&#91;&lt;ffffffffa02ec6de&gt;&#93;</span> ? cfs_waitq_del+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffff814c9963&gt;&#93;</span> wait_for_common+0x123/0x180<br/> <span class="error">&#91;&lt;ffffffff8105c540&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> <span class="error">&#91;&lt;ffffffff814c9a7d&gt;&#93;</span> wait_for_completion+0x1d/0x20<br/> <span class="error">&#91;&lt;ffffffffa03f9ad5&gt;&#93;</span> llog_process_flags+0x115/0x680 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0552897&gt;&#93;</span> ? llog_client_read_header+0x187/0x640 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa03fce98&gt;&#93;</span> llog_cat_process_flags+0x188/0x2d0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa03fbf8f&gt;&#93;</span> ? llog_init_handle+0x17f/0xa70 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0709170&gt;&#93;</span> ? changelog_show_cb+0x0/0x310 <span class="error">&#91;mdc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0712aae&gt;&#93;</span> mdc_changelog_send_thread+0x4ce/0xb90 <span class="error">&#91;mdc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81068bc4&gt;&#93;</span> ? __mmdrop+0x44/0x60<br/> <span class="error">&#91;&lt;ffffffff81059e2c&gt;&#93;</span> ? finish_task_switch+0xac/0xd0<br/> <span class="error">&#91;&lt;ffffffff810141ca&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa07125e0&gt;&#93;</span> ? mdc_changelog_send_thread+0x0/0xb90 <span class="error">&#91;mdc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810141c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> INFO: task lustre_rsync:8899 blocked for more than 120 seconds.<br/> "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> lustre_rsync D ffff88033fc24900 0 8899 8898 0x00000080<br/> ffff88030e1c9c10 0000000000000082 0000000000000000 00000000000022c4<br/> ffff88030e1c9b80 ffffffff810348fe ffff88030e1c9b90 000000010011535c<br/> ffff8802fcd9f068 ffff88030e1c9fd8 0000000000010518 ffff8802fcd9f068<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff810348fe&gt;&#93;</span> ? physflat_send_IPI_mask+0xe/0x10<br/> <span class="error">&#91;&lt;ffffffff814c9cf5&gt;&#93;</span> schedule_timeout+0x225/0x2f0<br/> <span class="error">&#91;&lt;ffffffffa0552897&gt;&#93;</span> ? llog_client_read_header+0x187/0x640 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff814c9963&gt;&#93;</span> wait_for_common+0x123/0x180<br/> <span class="error">&#91;&lt;ffffffff8105c540&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> <span class="error">&#91;&lt;ffffffff814c9a7d&gt;&#93;</span> wait_for_completion+0x1d/0x20<br/> <span class="error">&#91;&lt;ffffffffa03f9ad5&gt;&#93;</span> llog_process_flags+0x115/0x680 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81096d4f&gt;&#93;</span> ? up+0x2f/0x50<br/> <span class="error">&#91;&lt;ffffffffa03fdd59&gt;&#93;</span> llog_cat_process_cb+0x329/0x400 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa03fb653&gt;&#93;</span> llog_process_thread+0x9a3/0xe70 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8111f059&gt;&#93;</span> ? free_pages+0x49/0x50<br/> <span class="error">&#91;&lt;ffffffff810141ca&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa0709170&gt;&#93;</span> ? changelog_show_cb+0x0/0x310 <span class="error">&#91;mdc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa03facb0&gt;&#93;</span> ? llog_process_thread+0x0/0xe70 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810141c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107<br/> LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.<br/> Lustre: lustre-OST0005-osc-ffff880318f86000: Connection to service lustre-OST0005 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete.<br/> LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.<br/> Lustre: lustre-MDT0000-mdc-ffff880318f86000: Connection restored to service lustre-MDT0000 using nid 192.168.4.128@o2ib.<br/> Lustre: 2501:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x2af0a31218ffe297 to 0x2af0a31219beaacb<br/> Lustre: MGC192.168.4.128@o2ib: Reactivating import<br/> Lustre: MGC192.168.4.128@o2ib: Connection restored to service MGS using nid 192.168.4.128@o2ib.<br/> Lustre: Skipped 1 previous similar message<br/> INFO: task lustre_rsync:8898 blocked for more than 120 seconds.<br/> LustreError: 11-0: an error occurred while communicating with 192.168.4.128@o2ib. The obd_ping operation failed with -107<br/> LustreError: Skipped 2 previous similar messages<br/> LustreError: 166-1: MGC192.168.4.128@o2ib: Connection to service MGS via nid 192.168.4.128@o2ib was lost; in progress operations using this service will fail.<br/> Lustre: lustre-OST0000-osc-ffff880318f86000: Connection to service lustre-OST0000 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete.<br/> Lustre: Skipped 1 previous similar message<br/> Lustre: 2501:0:(import.c:885:ptlrpc_connect_interpret()) MGS@192.168.4.128@o2ib changed server handle from 0x2af0a31219beaacb to 0x2af0a31219beaad9<br/> LustreError: 167-0: This client was evicted by lustre-OST0000; in progress operations using this service will fail.<br/> LustreError: Skipped 1 previous similar message<br/> Lustre: lustre-OST0002-osc-ffff880318f86000: Connection restored to service lustre-OST0002 using nid 192.168.4.129@o2ib.<br/> LustreError: 9082:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8802ff556400 x1368658213493291/t0(0) o-1-&gt;MGS@192.168.4.128@o2ib:26/25 lens 296/352 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1<br/> Lustre: MGC192.168.4.128@o2ib: Reactivating import</p> lustre-master/rhel6-x86_64/#114 LU-316

system hang when running lustre-rsync-test test_5b

Bug Minor Resolved Fixed Oleg Drokin Sarah Liu Thu, 12 May 2011 20:52:05 +0000 Wed, 25 May 2011 14:50:09 +0000 Wed, 25 May 2011 14:50:09 +0000 Lustre 2.1.0 Lustre 2.1.0 0 0 <p>syslog from ost:<br/> -----------------------------</p> <p>Lustre: DEBUG MARKER: == lustre-rsync-test test 5b: Kill / restart lustre_rsync == 19:47:48 (1305254868)<br/> Lustre: lustre-OST0005: haven't heard from client 6a761607-aedc-55b0-0bdc-d007d810ea33 (at 192.168.4.5@o2ib) in 55 seconds. I think it's dead, and I am evicting it. exp ffff88061fdfb400, cur 1305255094 expire 1305255064 last 1305255039<br/> Lustre: lustre-OST0004: haven't heard from client 6a761607-aedc-55b0-0bdc-d007d810ea33 (at 192.168.4.5@o2ib) in 55 seconds. I think it's dead, and I am evicting it. exp ffff88061f62e400, cur 1305255094 expire 1305255064 last 1305255039<br/> LustreError: 4153:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (<del>107) req@ffff88061e86a400 x1368658213493272/t0(0) o-1</del>&gt;&lt;?&gt;@&lt;?&gt;:0/0 lens 192/0 e 0 to 0 dl 1305255100 ref 1 fl Interpret:H/ffffffff/ffffffff rc -107/-1<br/> Lustre: 4155:0:(ldlm_lib.c:871:target_handle_connect()) lustre-OST0005: connection from 6a761607-aedc-55b0-0bdc-d007d810ea33@192.168.4.5@o2ib t0 exp (null) cur 1305255094 last 0<br/> Lustre: 4155:0:(filter.c:2806:filter_connect()) lustre-OST0005: Received MDS connection (0x19c578d6a8d29552); group 0<br/> Lustre: 4155:0:(filter.c:2806:filter_connect()) Skipped 1 previous similar message<br/> Lustre: 4155:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0005-&gt;NET_0x50000c0a80405_UUID netid 50000: select flavor null<br/> Lustre: 4155:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages<br/> Lustre: lustre-OST0005: haven't heard from client 6a761607-aedc-55b0-0bdc-d007d810ea33 (at 192.168.4.5@o2ib) in 55 seconds. I think it's dead, and I am evicting it. exp ffff88032fa68000, cur 1305255149 expire 1305255119 last 1305255094<br/> LustreError: 4151:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (<del>107) req@ffff88061a52e800 x1368658213493279/t0(0) o-1</del>&gt;&lt;?&gt;@&lt;?&gt;:0/0 lens 192/0 e 0 to 0 dl 1305255183 ref 1 fl Interpret:H/ffffffff/ffffffff rc -107/-1<br/> Lustre: 4157:0:(ldlm_lib.c:871:target_handle_connect()) lustre-OST0003: connection from 6a761607-aedc-55b0-0bdc-d007d810ea33@192.168.4.5@o2ib t0 exp (null) cur 1305255177 last 0<br/> Lustre: 4157:0:(filter.c:2806:filter_connect()) lustre-OST0003: Received MDS connection (0x19c578d6a8d29560); group 0<br/> Lustre: 4157:0:(filter.c:2806:filter_connect()) Skipped 1 previous similar message<br/> Lustre: 4163:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000-&gt;NET_0x50000c0a80405_UUID netid 50000: select flavor null<br/> Lustre: 4163:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 2 previous similar messages<br/> LustreError: 4151:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 5 previous similar messages</p> <p>syslog from mds:<br/> --------------------------<br/> Lustre: DEBUG MARKER: ----<del>============= acceptance-small: lustre-rsync-test ============</del>---- Fri May 13 12:04:49 PDT 2011<br/> Lustre: DEBUG MARKER: Using TIMEOUT=20<br/> Lustre: Disabling parameter lustre-MDT0000.mdd.quota_type in log lustre-MDT0000<br/> Lustre: Skipped 1 previous similar message<br/> LustreError: 10125:0:(mgs_llog.c:2652:mgs_write_log_param()) err -19 on param 'mdd.quota_type='<br/> LustreError: 10125:0:(mgs_handler.c:969:mgs_iocontrol()) setparam err -19<br/> Lustre: 10302:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.<br/> Lustre: DEBUG MARKER: only running test 5b<br/> Lustre: DEBUG MARKER: excepting tests:<br/> Lustre: DEBUG MARKER: == lustre-rsync-test test 5b: Kill / restart lustre_rsync == 12:04:56 (1305313496)<br/> Lustre: mdd_obd-lustre-MDT0000-0: changelog on<br/> Lustre: mdd_obd-lustre-MDT0000-0: changelog off<br/> Lustre: MGS: haven't heard from client 2bae2e67-a05e-33c2-0d9a-6856e66881f6 (at 192.168.4.5@o2ib) in 48 seconds. I think it's dead, and I am evicting it. exp ffff88062b134000, cur 1305313632 expire 1305313602 last 1305313584<br/> Lustre: lustre-MDT0000: haven't heard from client 9cb3a3d1-f9b3-8d76-cf39-fa485e920451 (at 192.168.4.5@o2ib) in 52 seconds. I think it's dead, and I am evicting it. exp ffff8805da0c3000, cur 1305313637 expire 1305313607 last 1305313585<br/> LustreError: 9234:0:(mgs_handler.c:677:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS<br/> LustreError: 10576:0:(mdt_handler.c:2813:mdt_recovery()) operation 400 on unconnected MDS from 12345-192.168.4.5@o2ib<br/> LustreError: 10576:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (<del>107) req@ffff88062b09c000 x1368720312157349/t0(0) o-1</del>&gt;&lt;?&gt;@&lt;?&gt;:0/0 lens 192/0 e 0 to 0 dl 1305313695 ref 1 fl Interpret:H/ffffffff/ffffffff rc -107/-1<br/> Lustre: 10576:0:(ldlm_lib.c:871:target_handle_connect()) lustre-MDT0000: connection from 9cb3a3d1-f9b3-8d76-cf39-fa485e920451@192.168.4.5@o2ib t4294983304 exp (null) cur 1305313689 last 0<br/> Lustre: 10576:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 2 previous similar messages<br/> Lustre: 10576:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000-&gt;NET_0x50000c0a80405_UUID netid 50000: select flavor null<br/> Lustre: 10576:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 2 previous similar messages<br/> Lustre: MGS: haven't heard from client 2bae2e67-a05e-33c2-0d9a-6856e66881f6 (at 192.168.4.5@o2ib) in 48 seconds. I think it's dead, and I am evicting it. exp ffff8805d7cd9400, cur 1305313737 expire 1305313707 last 1305313689<br/> Lustre: lustre-MDT0000: haven't heard from client 9cb3a3d1-f9b3-8d76-cf39-fa485e920451 (at 192.168.4.5@o2ib) in 53 seconds. I think it's dead, and I am evicting it. exp ffff8802cc611c00, cur 1305313742 expire 1305313712 last 1305313689<br/> LustreError: 9251:0:(mdt_handler.c:2813:mdt_recovery()) operation 400 on unconnected MDS from 12345-192.168.4.5@o2ib<br/> LustreError: 9234:0:(mgs_handler.c:677:mgs_handle()) lustre_mgs: operation 400 on unconnected MGS<br/> LustreError: 9234:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (<del>107) req@ffff8805d9cfb050 x1368720312157364/t0(0) o-1</del>&gt;&lt;?&gt;@&lt;?&gt;:0/0 lens 192/0 e 0 to 0 dl 1305313796 ref 1 fl Interpret:H/ffffffff/ffffffff rc -107/-1<br/> LustreError: 9234:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 1 previous similar message<br/> Lustre: 9234:0:(ldlm_lib.c:871:target_handle_connect()) MGS: connection from 2bae2e67-a05e-33c2-0d9a-6856e66881f6@192.168.4.5@o2ib t0 exp (null) cur 1305313790 last 0<br/> Lustre: 9234:0:(ldlm_lib.c:871:target_handle_connect()) Skipped 1 previous similar message<br/> Lustre: 9234:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS-&gt;NET_0x50000c0a80405_UUID netid 50000: select flavor null<br/> Lustre: 9234:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 2 previous similar messages</p> <p>Oleg</p> <p>Could you look into this one please?</p> <p>Thanks</p> <p>Peter</p> <p>Something pretty weird is going on as evidenced by the stack traces below.<br/> The rsync thread calls mdc_ioc_changelog_send that creates a new "Sending to userspace thread" that calls llog_process_flags that schedules a new thread that in turn calls llog_process_flags again that schedules another thread that tries to write to some userspace pipe where nobody reads on the other end and hangs.</p> <p>I did some more digging and I see that changelog communications is a total mess.<br/> The userspace is responsible to create a pipe for communications with kernel.</p> <p>Then our implementation (in liblustreapi.c::llapi_changelog_start) just does the ioctl causing the action from above to start.<br/> Then userspace closes the write descriptor so that if kernel closes its side, it's notified.<br/> One problem is - the read descriptor is not closed prior to ioctl, so even when the userspace side dies (due to signal), there are both write and read parts of the pipe opened by the kernel threads and so EPIPE never arises.<br/> Even if we fix this in userspace, malicious people can construct a similar case and cause needless kernel resource usage, so we need to also address this from the kernel side. (is this a real concern or is this only accessible to root?)</p> <p>In fact I just did the closing in the kernel and it seems to help, the commends in the code are all invalidly presume that the files must remain open where in fact we already hold the pipe reference in the kernel.</p> <p>I am running the full test now to make sure that nothing break.</p> <p>[ 284.928469] lustre_rsync D ffff88003f824288 4856 3398 1 0x00000000<br/> [ 284.928469] ffff880039731c40 0000000000000082 0000000000000000 ffff880037460400<br/> [ 284.928469] ffff880039731bb0 0000000000000282 ffff880039731bd0 00000000ffff17b4<br/> [ 284.928469] ffff880030666738 ffff880039731fd8 0000000000010488 ffff880030666738<br/> [ 284.928469] Call Trace:<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff814d4e15&gt;&#93;</span> schedule_timeout+0x225/0x2f0<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff81158d36&gt;&#93;</span> ? kfree_debugcheck+0x16/0x40<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff814d4a7b&gt;&#93;</span> wait_for_common+0x12b/0x180<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff8105c700&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff814d4b8d&gt;&#93;</span> wait_for_completion+0x1d/0x20<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa0305975&gt;&#93;</span> llog_process_flags+0x115/0x580 <span class="error">&#91;obdclass&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa045a1c7&gt;&#93;</span> ? llog_client_read_header+0x187/0x640 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa01f89b3&gt;&#93;</span> ? cfs_alloc+0x63/0x90 <span class="error">&#91;libcfs&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa0308a78&gt;&#93;</span> llog_cat_process_flags+0x188/0x2d0 <span class="error">&#91;obdclass&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa0307c0f&gt;&#93;</span> ? llog_init_handle+0x17f/0x9d0 <span class="error">&#91;obdclass&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff8115ba03&gt;&#93;</span> ? __kmalloc+0x1c3/0x2e0<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa01f89b3&gt;&#93;</span> ? cfs_alloc+0x63/0x90 <span class="error">&#91;libcfs&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa05a1050&gt;&#93;</span> ? changelog_show_cb+0x0/0x310 <span class="error">&#91;mdc&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa05aa63e&gt;&#93;</span> mdc_changelog_send_thread+0x41e/0xad0 <span class="error">&#91;mdc&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff81069044&gt;&#93;</span> ? __mmdrop+0x44/0x60<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff8105a17e&gt;&#93;</span> ? finish_task_switch+0xae/0xd0<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff8101420a&gt;&#93;</span> child_rip+0xa/0x20<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa05aa220&gt;&#93;</span> ? mdc_changelog_send_thread+0x0/0xad0 <span class="error">&#91;mdc&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff81014200&gt;&#93;</span> ? child_rip+0x0/0x20<br/> [ 284.928469] lustre_rsync D ffff880025cafcf0 4488 3399 3398 0x00000000<br/> [ 284.928469] ffff880025cafc10 0000000000000082 ffff880001f16a80 ffff88003d0ce200<br/> [ 284.928469] ffff880025cafb80 0000000000000282 ffff880025cafba0 0000000000000282<br/> [ 284.928469] ffff8800374609b8 ffff880025caffd8 0000000000010488 ffff8800374609b8<br/> [ 284.928469] Call Trace:<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff814d4e15&gt;&#93;</span> schedule_timeout+0x225/0x2f0<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa045a1c7&gt;&#93;</span> ? llog_client_read_header+0x187/0x640 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa01f89b3&gt;&#93;</span> ? cfs_alloc+0x63/0x90 <span class="error">&#91;libcfs&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff814d4a7b&gt;&#93;</span> wait_for_common+0x12b/0x180<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff8105c700&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff814d4b8d&gt;&#93;</span> wait_for_completion+0x1d/0x20<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa0305975&gt;&#93;</span> llog_process_flags+0x115/0x580 <span class="error">&#91;obdclass&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff810976bf&gt;&#93;</span> ? up+0x2f/0x50<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa0309939&gt;&#93;</span> llog_cat_process_cb+0x329/0x400 <span class="error">&#91;obdclass&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa0307303&gt;&#93;</span> llog_process_thread+0x943/0xde0 <span class="error">&#91;obdclass&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff8101420a&gt;&#93;</span> child_rip+0xa/0x20<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa05a1050&gt;&#93;</span> ? changelog_show_cb+0x0/0x310 <span class="error">&#91;mdc&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa03069c0&gt;&#93;</span> ? llog_process_thread+0x0/0xde0 <span class="error">&#91;obdclass&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff81014200&gt;&#93;</span> ? child_rip+0x0/0x20<br/> [ 284.928469] lustre_rsync S 000000000000000e 4320 3400 3399 0x00000000<br/> [ 284.928469] ffff880025cc9b20 0000000000000082 0000000000000246 ffff88003c80f120<br/> [ 284.928469] ffff880025cc9af0 ffffffff814d59cc ffff88003c80f138 0000000000000282<br/> [ 284.928469] ffff88003d0ce7b8 ffff880025cc9fd8 0000000000010488 ffff88003d0ce7b8<br/> [ 284.928469] Call Trace:<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff814d59cc&gt;&#93;</span> ? __mutex_lock_slowpath+0x21c/0x2c0<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff8117c67b&gt;&#93;</span> pipe_wait+0x5b/0x80<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff81092780&gt;&#93;</span> ? autoremove_wake_function+0x0/0x40<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff814d5aa1&gt;&#93;</span> ? mutex_lock+0x31/0x50<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff8117d392&gt;&#93;</span> pipe_write+0x322/0x660<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff811728ea&gt;&#93;</span> do_sync_write+0xfa/0x140<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff81092780&gt;&#93;</span> ? autoremove_wake_function+0x0/0x40<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa046fee9&gt;&#93;</span> ? sptlrpc_req_put_ctx+0xa9/0x1b0 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa01f8a46&gt;&#93;</span> cfs_user_write+0x66/0xa0 <span class="error">&#91;libcfs&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa020a94b&gt;&#93;</span> libcfs_kkuc_msg_put+0x5b/0x180 <span class="error">&#91;libcfs&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa045b087&gt;&#93;</span> ? llog_client_next_block+0x1c7/0x550 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa05a1112&gt;&#93;</span> changelog_show_cb+0xc2/0x310 <span class="error">&#91;mdc&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa0307303&gt;&#93;</span> llog_process_thread+0x943/0xde0 <span class="error">&#91;obdclass&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff8101420a&gt;&#93;</span> child_rip+0xa/0x20<br/> [ 284.928469] <span class="error">&#91;&lt;ffffffffa03069c0&gt;&#93;</span> ? llog_process_thread+0x0/0xde0 <span class="error">&#91;obdclass&#93;</span><br/> [ 284.928469] <span class="error">&#91;&lt;ffffffff81014200&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>Sarah, please try patch in <a href="http://review.whamcloud.com/555" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/555</a> to see if it works for the entire rsync and changelog related testsuites.<br/> Seems to be working fine for me.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/128/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #128</a><br/> <a href="https://jira.whamcloud.com/browse/LU-316" title="system hang when running lustre-rsync-test test_5b" class="issue-link" data-issue-key="LU-316"><del>LU-316</del></a> Fix changelog deadlock when changelog client is killed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=9e6a0fcbf02942caff9d769d644913964d34c028" class="external-link" target="_blank" rel="nofollow noopener">9e6a0fcbf02942caff9d769d644913964d34c028</a><br/> Files : </p> <ul> <li>lustre/mdc/mdc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/128/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #128</a><br/> <a href="https://jira.whamcloud.com/browse/LU-316" title="system hang when running lustre-rsync-test test_5b" class="issue-link" data-issue-key="LU-316"><del>LU-316</del></a> Fix changelog deadlock when changelog client is killed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=9e6a0fcbf02942caff9d769d644913964d34c028" class="external-link" target="_blank" rel="nofollow noopener">9e6a0fcbf02942caff9d769d644913964d34c028</a><br/> Files : </p> <ul> <li>lustre/mdc/mdc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/128/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #128</a><br/> <a href="https://jira.whamcloud.com/browse/LU-316" title="system hang when running lustre-rsync-test test_5b" class="issue-link" data-issue-key="LU-316"><del>LU-316</del></a> Fix changelog deadlock when changelog client is killed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=9e6a0fcbf02942caff9d769d644913964d34c028" class="external-link" target="_blank" rel="nofollow noopener">9e6a0fcbf02942caff9d769d644913964d34c028</a><br/> Files : </p> <ul> <li>lustre/mdc/mdc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/128/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #128</a><br/> <a href="https://jira.whamcloud.com/browse/LU-316" title="system hang when running lustre-rsync-test test_5b" class="issue-link" data-issue-key="LU-316"><del>LU-316</del></a> Fix changelog deadlock when changelog client is killed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=9e6a0fcbf02942caff9d769d644913964d34c028" class="external-link" target="_blank" rel="nofollow noopener">9e6a0fcbf02942caff9d769d644913964d34c028</a><br/> Files : </p> <ul> <li>lustre/mdc/mdc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/128/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #128</a><br/> <a href="https://jira.whamcloud.com/browse/LU-316" title="system hang when running lustre-rsync-test test_5b" class="issue-link" data-issue-key="LU-316"><del>LU-316</del></a> Fix changelog deadlock when changelog client is killed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=9e6a0fcbf02942caff9d769d644913964d34c028" class="external-link" target="_blank" rel="nofollow noopener">9e6a0fcbf02942caff9d769d644913964d34c028</a><br/> Files : </p> <ul> <li>lustre/mdc/mdc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/128/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #128</a><br/> <a href="https://jira.whamcloud.com/browse/LU-316" title="system hang when running lustre-rsync-test test_5b" class="issue-link" data-issue-key="LU-316"><del>LU-316</del></a> Fix changelog deadlock when changelog client is killed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=9e6a0fcbf02942caff9d769d644913964d34c028" class="external-link" target="_blank" rel="nofollow noopener">9e6a0fcbf02942caff9d769d644913964d34c028</a><br/> Files : </p> <ul> <li>lustre/mdc/mdc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/128/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #128</a><br/> <a href="https://jira.whamcloud.com/browse/LU-316" title="system hang when running lustre-rsync-test test_5b" class="issue-link" data-issue-key="LU-316"><del>LU-316</del></a> Fix changelog deadlock when changelog client is killed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=9e6a0fcbf02942caff9d769d644913964d34c028" class="external-link" target="_blank" rel="nofollow noopener">9e6a0fcbf02942caff9d769d644913964d34c028</a><br/> Files : </p> <ul> <li>lustre/mdc/mdc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/128/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #128</a><br/> <a href="https://jira.whamcloud.com/browse/LU-316" title="system hang when running lustre-rsync-test test_5b" class="issue-link" data-issue-key="LU-316"><del>LU-316</del></a> Fix changelog deadlock when changelog client is killed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=9e6a0fcbf02942caff9d769d644913964d34c028" class="external-link" target="_blank" rel="nofollow noopener">9e6a0fcbf02942caff9d769d644913964d34c028</a><br/> Files : </p> <ul> <li>lustre/mdc/mdc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/128/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #128</a><br/> <a href="https://jira.whamcloud.com/browse/LU-316" title="system hang when running lustre-rsync-test test_5b" class="issue-link" data-issue-key="LU-316"><del>LU-316</del></a> Fix changelog deadlock when changelog client is killed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=9e6a0fcbf02942caff9d769d644913964d34c028" class="external-link" target="_blank" rel="nofollow noopener">9e6a0fcbf02942caff9d769d644913964d34c028</a><br/> Files : </p> <ul> <li>lustre/mdc/mdc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/128/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #128</a><br/> <a href="https://jira.whamcloud.com/browse/LU-316" title="system hang when running lustre-rsync-test test_5b" class="issue-link" data-issue-key="LU-316"><del>LU-316</del></a> Fix changelog deadlock when changelog client is killed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=9e6a0fcbf02942caff9d769d644913964d34c028" class="external-link" target="_blank" rel="nofollow noopener">9e6a0fcbf02942caff9d769d644913964d34c028</a><br/> Files : </p> <ul> <li>lustre/mdc/mdc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/128/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #128</a><br/> <a href="https://jira.whamcloud.com/browse/LU-316" title="system hang when running lustre-rsync-test test_5b" class="issue-link" data-issue-key="LU-316"><del>LU-316</del></a> Fix changelog deadlock when changelog client is killed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=9e6a0fcbf02942caff9d769d644913964d34c028" class="external-link" target="_blank" rel="nofollow noopener">9e6a0fcbf02942caff9d769d644913964d34c028</a><br/> Files : </p> <ul> <li>lustre/mdc/mdc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/128/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #128</a><br/> <a href="https://jira.whamcloud.com/browse/LU-316" title="system hang when running lustre-rsync-test test_5b" class="issue-link" data-issue-key="LU-316"><del>LU-316</del></a> Fix changelog deadlock when changelog client is killed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=9e6a0fcbf02942caff9d769d644913964d34c028" class="external-link" target="_blank" rel="nofollow noopener">9e6a0fcbf02942caff9d769d644913964d34c028</a><br/> Files : </p> <ul> <li>lustre/mdc/mdc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=ofa/128/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,ofa #128</a><br/> <a href="https://jira.whamcloud.com/browse/LU-316" title="system hang when running lustre-rsync-test test_5b" class="issue-link" data-issue-key="LU-316"><del>LU-316</del></a> Fix changelog deadlock when changelog client is killed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=9e6a0fcbf02942caff9d769d644913964d34c028" class="external-link" target="_blank" rel="nofollow noopener">9e6a0fcbf02942caff9d769d644913964d34c028</a><br/> Files : </p> <ul> <li>lustre/mdc/mdc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/128/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #128</a><br/> <a href="https://jira.whamcloud.com/browse/LU-316" title="system hang when running lustre-rsync-test test_5b" class="issue-link" data-issue-key="LU-316"><del>LU-316</del></a> Fix changelog deadlock when changelog client is killed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=9e6a0fcbf02942caff9d769d644913964d34c028" class="external-link" target="_blank" rel="nofollow noopener">9e6a0fcbf02942caff9d769d644913964d34c028</a><br/> Files : </p> <ul> <li>lustre/mdc/mdc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/128/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #128</a><br/> <a href="https://jira.whamcloud.com/browse/LU-316" title="system hang when running lustre-rsync-test test_5b" class="issue-link" data-issue-key="LU-316"><del>LU-316</del></a> Fix changelog deadlock when changelog client is killed</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=9e6a0fcbf02942caff9d769d644913964d34c028" class="external-link" target="_blank" rel="nofollow noopener">9e6a0fcbf02942caff9d769d644913964d34c028</a><br/> Files : </p> <ul> <li>lustre/mdc/mdc_request.c</li> </ul> <p>verified on the latest master build RHEL6/x86_64/#134</p> <p>Fixed</p> Development Rank 1|hzv97z: Rank (Obsolete) 5028 Severity