https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-187 Lustre <p>umount mds hang when running conf-sanity test 29</p> <p>MDS:</p> <p>INFO: task umount:24781 blocked for more than 120 seconds.<br/> "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> umount D ffffffff80150d6a 0 24781 24780 (NOTLB)<br/> ffff8102f2283978 0000000000000086 0000003000000030 ffff8102f22839e0<br/> ffff8102f22838d8 0000000000000007 ffff8103389b67a0 ffff81034aae77a0<br/> 000017607cb40f62 0000000002787517 ffff8103389b6988 0000000700005486<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff80064613&gt;&#93;</span> __down_write_nested+0x7a/0x92<br/> <span class="error">&#91;&lt;ffffffff88bcf960&gt;&#93;</span> :mdd:mdd_init_capa_ctxt+0xe0/0x1a0<br/> <span class="error">&#91;&lt;ffffffff889213e0&gt;&#93;</span> :fld:fld_cache_fini+0x350/0x360<br/> <span class="error">&#91;&lt;ffffffff88c84066&gt;&#93;</span> :cmm:cmm_init_capa_ctxt+0xd6/0x170<br/> <span class="error">&#91;&lt;ffffffff88bf32d2&gt;&#93;</span> :mdt:mdt_fld_fini+0x1b2/0x1c0<br/> <span class="error">&#91;&lt;ffffffff88c0087e&gt;&#93;</span> :mdt:mdt_device_fini+0x36e/0x650<br/> <span class="error">&#91;&lt;ffffffff88738aea&gt;&#93;</span> :obdclass:class_cleanup+0xbca/0xe40<br/> <span class="error">&#91;&lt;ffffffff8873cc4c&gt;&#93;</span> :obdclass:class_process_config+0x1e5c/0x3200<br/> <span class="error">&#91;&lt;ffffffff8873f647&gt;&#93;</span> :obdclass:class_manual_cleanup+0xad7/0xe80<br/> <span class="error">&#91;&lt;ffffffff8874d10c&gt;&#93;</span> :obdclass:server_put_super+0x5ac/0xe70<br/> <span class="error">&#91;&lt;ffffffff80051848&gt;&#93;</span> vfs_quota_sync+0x14b/0x15a<br/> <span class="error">&#91;&lt;ffffffff80034ee9&gt;&#93;</span> dispose_list+0xc7/0xe0<br/> <span class="error">&#91;&lt;ffffffff800ec87a&gt;&#93;</span> invalidate_inodes+0xce/0xe0<br/> <span class="error">&#91;&lt;ffffffff800e3f71&gt;&#93;</span> generic_shutdown_super+0x79/0xfb<br/> <span class="error">&#91;&lt;ffffffff800e4041&gt;&#93;</span> kill_anon_super+0x9/0x35<br/> <span class="error">&#91;&lt;ffffffff800e40f2&gt;&#93;</span> deactivate_super+0x6a/0x82<br/> <span class="error">&#91;&lt;ffffffff800edf47&gt;&#93;</span> sys_umount+0x245/0x27b<br/> <span class="error">&#91;&lt;ffffffff800236ba&gt;&#93;</span> sys_newstat+0x19/0x31<br/> <span class="error">&#91;&lt;ffffffff8005d116&gt;&#93;</span> system_call+0x7e/0x83</p> LU-187

umount hang when running conf-sanity test 29

Bug Minor Closed Cannot Reproduce Jian Yu Sarah Liu Sat, 2 Apr 2011 00:04:02 +0000 Thu, 30 May 2013 08:08:21 +0000 Wed, 1 Jun 2011 20:27:42 +0000 Lustre 2.1.0 0 1 <p>Yu Jian</p> <p>Can you please try and understand the cause of this hang?</p> <p>Thanks</p> <p>Peter</p> <p>From the syslog of the MDS node, we could see:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: Permanently reactivating lustre-OST0001 Lustre: lustre-OST0001-osc-MDT0000: Connection to service lustre-OST0001 via nid 192.168.4.129@o2ib was lost; in progress operations using this service will wait for recovery to complete. LustreError: 167-0: This client was evicted by lustre-OST0001; in progress operations using this service will fail. Lustre: lustre-OST0001-osc-MDT0000: Connection restored to service lustre-OST0001 using nid 192.168.4.129@o2ib. LustreError: 11-0: an error occurred while communicating with 192.168.4.129@o2ib. The ost_set_info operation failed with -107 </pre> </div></div> <p>After reactivating the lustre-OST0001, the communication between MDS and OSS hit -107 (-ENOTCONN, Transport endpoint is not connected) error.</p> <p>Sarah, do you have the dmesg/syslog of the OSS node? I could not reproduce this issue on Toro cluster, and then could not figure out what happened on the OSS node.</p> <p>sorry, I don't have those logs. I will try to reproduce this bug and gather the logs.</p> <p>The issue has not been reproduced for two months. Let's close the ticket. If you hit the issue again, please feel free to reopen it.</p> <p>Hi,</p> <p>I'm able to reproduce this pretty reliably:</p> <p>INFO: task touch:22268 blocked for more than 120 seconds.<br/> "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> touch D ffff880038923a18 0 22268 22237 0x00000080<br/> ffff880038923938 0000000000000082 ffff8800389238e8 ffffffff810117bc<br/> 0000000000000000 0000000000000000 0000000000000000 ffff88002ff9c100<br/> ffff880039cbbb18 ffff880038923fd8 0000000000010518 ffff880039cbbb18<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff810117bc&gt;&#93;</span> ? __switch_to+0x1ac/0x320<br/> <span class="error">&#91;&lt;ffffffff814c95f6&gt;&#93;</span> ? thread_return+0x4e/0x778<br/> <span class="error">&#91;&lt;ffffffff814ca335&gt;&#93;</span> schedule_timeout+0x225/0x2f0<br/> <span class="error">&#91;&lt;ffffffff814c9fa3&gt;&#93;</span> wait_for_common+0x123/0x180<br/> <span class="error">&#91;&lt;ffffffff8105c580&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> <span class="error">&#91;&lt;ffffffff814ca0bd&gt;&#93;</span> wait_for_completion+0x1d/0x20<br/> <span class="error">&#91;&lt;ffffffffa08b3558&gt;&#93;</span> osc_io_setattr_end+0x28/0x90 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0691260&gt;&#93;</span> cl_io_end+0x60/0x130 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0905c86&gt;&#93;</span> lov_io_end_wrapper+0x116/0x120 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0905b70&gt;&#93;</span> ? lov_io_end_wrapper+0x0/0x120 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0905b70&gt;&#93;</span> ? lov_io_end_wrapper+0x0/0x120 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0905a88&gt;&#93;</span> lov_io_call+0x58/0x140 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa09065a6&gt;&#93;</span> ? lov_io_start+0xc6/0x170 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0906457&gt;&#93;</span> lov_io_end+0x47/0xd0 <span class="error">&#91;lov&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0691260&gt;&#93;</span> cl_io_end+0x60/0x130 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06951de&gt;&#93;</span> cl_io_loop+0x11e/0x1d0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0693a14&gt;&#93;</span> ? cl_io_init+0x64/0xc0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0b24f3b&gt;&#93;</span> cl_setattr_ost+0x1fb/0x290 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0afc4ea&gt;&#93;</span> ll_setattr_raw+0xa4a/0x1020 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0afcb1f&gt;&#93;</span> ll_setattr+0x5f/0x100 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81188f98&gt;&#93;</span> notify_change+0x168/0x340<br/> <span class="error">&#91;&lt;ffffffff8119c2cc&gt;&#93;</span> utimes_common+0xdc/0x1b0<br/> <span class="error">&#91;&lt;ffffffff8119c439&gt;&#93;</span> do_utimes+0x99/0xf0<br/> <span class="error">&#91;&lt;ffffffff810d4372&gt;&#93;</span> ? audit_syscall_entry+0x272/0x2a0<br/> <span class="error">&#91;&lt;ffffffff8119c592&gt;&#93;</span> sys_utimensat+0x32/0x90<br/> <span class="error">&#91;&lt;ffffffff81013172&gt;&#93;</span> system_call_fastpath+0x16/0x1b<br/> LustreError: 9781:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (<del>19) req@ffff88000baf3400 x1372341458271667/t0(0) o-1</del>&gt;&lt;?&gt;@&lt;?&gt;:0/0 lens 368/0 e 0 to 0 dl 1308766557 ref 1 fl Interpret:/ffffffff/ffffffff rc -19/-1<br/> LustreError: 9781:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 51 previous similar messages</p> <p>I tripped over this running acceptance on a SLES11SP1 / Lustre 2.1.5 client :</p> <p>crash&gt; bt 24884<br/> PID: 24884 TASK: ffff8802f9688aa0 CPU: 5 COMMAND: "umount"<br/> #0 <span class="error">&#91;ffff8802f23438d8&#93;</span> schedule at ffffffff8150ba32<br/> #1 <span class="error">&#91;ffff8802f23439a0&#93;</span> rwsem_down_failed_common at ffffffff8150dfe5<br/> #2 <span class="error">&#91;ffff8802f2343a00&#93;</span> rwsem_down_write_failed at ffffffff8150e143<br/> #3 <span class="error">&#91;ffff8802f2343a40&#93;</span> call_rwsem_down_write_failed at ffffffff81281463<br/> #4 <span class="error">&#91;ffff8802f2343aa0&#93;</span> mdd_init_capa_ctxt at ffffffffa07da6c8 <span class="error">&#91;mdd&#93;</span><br/> #5 <span class="error">&#91;ffff8802f2343b00&#93;</span> cmm_init_capa_ctxt at ffffffffa091b873 <span class="error">&#91;cmm&#93;</span><br/> #6 <span class="error">&#91;ffff8802f2343b40&#93;</span> mdt_device_fini at ffffffffa0834abf <span class="error">&#91;mdt&#93;</span><br/> #7 <span class="error">&#91;ffff8802f2343b80&#93;</span> class_cleanup at ffffffffa0b730f7 <span class="error">&#91;obdclass&#93;</span><br/> #8 <span class="error">&#91;ffff8802f2343c00&#93;</span> class_process_config at ffffffffa0b75103 <span class="error">&#91;obdclass&#93;</span><br/> #9 <span class="error">&#91;ffff8802f2343ca0&#93;</span> class_manual_cleanup at ffffffffa0b75e49 <span class="error">&#91;obdclass&#93;</span><br/> #10 <span class="error">&#91;ffff8802f2343d60&#93;</span> server_put_super at ffffffffa0b81cec <span class="error">&#91;obdclass&#93;</span><br/> #11 <span class="error">&#91;ffff8802f2343e30&#93;</span> generic_shutdown_super at ffffffff811828fb<br/> #12 <span class="error">&#91;ffff8802f2343e50&#93;</span> kill_anon_super at ffffffff811829e6<br/> #13 <span class="error">&#91;ffff8802f2343e70&#93;</span> lustre_kill_super at ffffffffa0b77a86 <span class="error">&#91;obdclass&#93;</span><br/> #14 <span class="error">&#91;ffff8802f2343e90&#93;</span> deactivate_super at ffffffff81183187<br/> #15 <span class="error">&#91;ffff8802f2343eb0&#93;</span> mntput_no_expire at ffffffff811a11af<br/> #16 <span class="error">&#91;ffff8802f2343ee0&#93;</span> sys_umount at ffffffff811a1c1b<br/> #17 <span class="error">&#91;ffff8802f2343f80&#93;</span> system_call_fastpath at ffffffff8100b072</p> <p>I doubt I'll have time to spare chasing it down, but I can make the core and logs available to anyone who wants to take a look.</p> Development Rank 1|hzvs9r: Rank (Obsolete) 8477 Severity