https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3946 Lustre <p>Dear support,</p> <p>our customer is experiencing an eviction problem on the login/client nodes in his lustre cluster.<br/> The eviction seems too frequent and, since seems to be no other way to recover then reboot the node, this interrupts the users work heavily.<br/> The cause of eviction seems to be that the login nodes may be temporarily stuck by different applications for fractions of seconds and consequently the oss does not see the client for minutes (according to logs).<br/> The customer attempted to solve the issue by booting the login nodes with kernel options "notsc" and "clocksource=hpet" to no avail.<br/> Also, they mounted lustre over tcp instead of IB, which also did not help. </p> <p>Infrastructure info:<br/> 2 node for MDS/MGS with pacemaker/corosync cluster<br/> 8 node for OSS in a 2 node cluster configuration with pacemaker/corosync cluster with a dothill storage controller per pair.<br/> ~1000 client</p> <p>We attach the messages log file for several days.<br/> For example we found about 280 clients evicted in 5 days, it is normal?</p> <p>For clarity login nodes have names brutus<span class="error">&#91;1-4&#93;</span> and IB IP 10.201.32.31-34 and Ethernet IP 10.201.0.31-34.</p> <p>Is it possible that this issue arise from the different version of lustre software btw clients and servers?</p> <p>Many thanks in advance for your help.</p> Lustre servers:<br/> CentOS 6.2 with Linux version 2.6.32-220.4.2.el6_lustre.x86_64 (<a href='mailto:jenkins@client'>jenkins@client</a> 31.lab.whamcloud.com) (gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC) ) #1 SMP Wed Mar 14 13:03:47 PDT 2012<br/> <br/> Build Version: 2.2.0-RC2--PRISTINE-2.6.32-220.4.2.el6_lustre.x86_64<br/> <br/> HW: 2xIntel(R) Xeon(R) CPU E5645 @ 2.40GHz, RAM 48GB<br/> <br/> Lustre clients:<br/> CentOS 6.4 with Linux version 2.6.32-358.6.2.el6.x86_64 (<a href='mailto:mockbuild@c6b8.bsys.dev.centos.org'>mockbuild@c6b8.bsys.dev.centos.org</a>) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Thu May 16 20:59:36 UTC 2013<br/> <br/> Build Version: 2.4.0-RC2--CHANGED-2.6.32-358.6.2.el6.x86_64<br/> <br/> HW: Login nodes are of type 12-core AMD Opteron 6174<br/> Compute nodes are a mix of: <br/> &nbsp;&nbsp;&nbsp;&nbsp;12-core AMD Opteron 6174<br/> &nbsp;&nbsp;&nbsp;&nbsp;quad-core AMD Opteron 8380 <br/> &nbsp;&nbsp;&nbsp;&nbsp;quad-core AMD Opteron 8384 <br/> &nbsp;&nbsp;&nbsp;&nbsp;8-core Intel Xeon E7-8837 LU-3946

Frequent client eviction

Bug Minor Resolved Duplicate Bruno Faccini Matteo Piccinini Fri, 13 Sep 2013 16:48:11 +0000 Mon, 28 Apr 2014 13:41:53 +0000 Mon, 28 Apr 2014 13:41:21 +0000 Lustre 2.4.0 0 8 <p>Matteo</p> <p>It is certainly true that 2.2 servers with 2.4 clients is not a combination that we officially test or support. However, rather than jump to conclusions, we should have an engineer review the evidence and see if this is indeed the reason for the problems.</p> <p>Hongchao</p> <p>Could you please review the information provided and make an assessment</p> <p>Thanks</p> <p>Peter</p> <p>Hi Matteo</p> <p>what is the format of the second log file "lustre_and_login_nodes_logs.tar.bz2ab", which isn't a bzip2 file just like the first one.<br/> the logs in the first file "lustre_and_login_nodes_logs.tar.bz2aa" only contains log of the clients, and the server logs are needed to<br/> get the cause of eviction.</p> <p>BTW, does the "stuck" of login nodes affect the whole system or only the applications? how long is the maximum duration of the stuck?</p> <p>Thanks</p> <p>Hi Peter and Hongchao,</p> <p>thanks for your further investigations.<br/> I'm sorry, I forgot to mention that I attached an archive splitted in two pieces (GNU coreutils split) because of the 10MB limits, if necessary I can upload a new one.</p> <p>The maximun duration of the "stuck" is variable, from seconds to several minutes and it affects only the login node itself requiring a manual system reboot.</p> <p>Thanks.</p> <p>Hi Matteo,</p> <p>the eviction is related to the "stuck" as you mentioned above, for there is even no "ping" request for more than 227 (the ping request is sent<br/> by an individual thread started alongside of the Lustre). the time interval before eviction can be extended as followings,</p> <p>echo "newtimeout" &gt; /proc/sys/lustre/timeout</p> <p>the interval will be "newtimeout * 9 / 4"</p> <p>if the stuck can't be avoided, how about increasing the timeout value?</p> <p>Thanks</p> <p>Hi Hongchao,</p> <p>we considered to change the timeout values.<br/> We will do tests and let you know as soon as possible if it solves the problem.</p> <p>Thanks.</p> <p>P.S. I don't know how to change the ticket status</p> <p>Hi Matteo</p> <p>It is ok - you do not need to change the ticket status. Please just let us know how you get on with your tests.</p> <p>Thanks</p> <p>Peter</p> <p>Log of eviction with timeout value of 200 seconds</p> <p>Hi Hongchao,</p> <p>we configure the timeout from the default 100 seconds to 200 seconds with the following command but the nodes was evicted around 457 seconds.</p> <p>lctl conf_param nero.sys.timeout=200</p> <p>We try to set a timeout value of 300 seconds and again found eviction around 667 seconds.<br/> Where i can find some best practises for setting the static timeout in proportion to the number of node?</p> <p>Do you have any other useful advice to try to solve this problem?</p> <p>Thanks,<br/> Matteo</p> <p>P.S. We attached the log file with timeout value of 200s <span class="error">&#91;logs_timeout_200s.tar.bz2&#93;</span></p> <p>Hello Matteo,</p> <p>Sorry for the late update.<br/> Could it be possible that you boot at least one of the impacted login nodes with "nohz=off" boot-parameter and see how it runs with it ??</p> <p>Comment from the customer:</p> <p>&lt;&lt;Dear Gabriele</p> <p>I have set nohz=off on two out of the four problematic servers at 5.12.<br/> Yesterday evening, one of them has been evicted again (nodename brutus2).</p> <p>So, this did not help. </p> <p>BTW: the other node has nohz=off as well as notsc and clocksource=hpet set.&gt;&gt;</p> <p>Thanks to attach the update Gabriele. I also attach both files/logs (brutus2_eviction.txt, messages_brutus2.txt) that were provided with this update.</p> <p>Matteo, does this mean that nohz=off helped to avoid the issue for some time ?</p> <p>Also, I am afraid I don't fully understand what you mean by "stuck", is it a scheduling issue caused by the heavy load on the login nodes ?</p> <p>Don't you run with some special ptlrpcd module multi-thread/numa-policy parameter ?</p> <p>I see in the logs that the evictions almost always start from OSS side with the "LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after …." msg, so can you provide the lustre debug-log (with full debug enabled and biggest debug buffer) at the time of the evict and from Client and OSSs sides ?</p> <p>Adding latest infos/exchanges from/with customer :</p> <p> _ Since it was never provided, I requested to get Client/OSS Lustre debug log during an evict. Since debug mask current settings was only "ioctl neterror warning error emerg ha config console" on all Clients/Servers, I requested "+dlmtrace +rpctrace". This is now setup on one login/Client node on-site.</p> <p> _ I requested to get current ptlrpcd Numa settings. On all Clients/Servers nodes nothing specific is actually configured, max_ptlrpcds=0 and ptlrpcd_bind_policy=3.</p> <p> _ I also requested to get current vm.zone_reclaim_mode setting, and since it was 1/on, I requested to have it changed to 0/off at least on one login/Client node. This has been done by customer.</p> <p>I also attach the latest Lustre+Numa configurations infos provided by customer (requested_outputs.tar.bz2).</p> <p>Customer provided more infos (Client lustre debug-log, Client/OSS syslog, ...) taken during a new occurence. It is "ETHZ_client_eviction_brutus3_n-oss07_20131220.tar.bz2" attachment I just uploaded.</p> <p>Here are my 1st analysis comments about these new infos.</p> <p>The following similar stack dumps for multiple ldlm_bl_xx threads before the evict :<br/> ================================================================<br/> Dec 20 10:00:54 brutus3 kernel: INFO: task ldlm_bl_43:12517 blocked for more than 120 seconds.<br/> Dec 20 10:00:54 brutus3 kernel: "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> Dec 20 10:00:54 brutus3 kernel: ldlm_bl_43 D 000000000000000c 0 12517 2 0x00000080<br/> Dec 20 10:00:54 brutus3 kernel: ffff880ae35a3d50 0000000000000046 0000000000000000 ffffffffa0582967<br/> Dec 20 10:00:54 brutus3 kernel: 0000000100000000 0000000000000000 ffff880ae35a3cf0 ffffffffa0581f22<br/> Dec 20 10:00:54 brutus3 kernel: ffff880ad9e7c5f8 ffff880ae35a3fd8 000000000000fb88 ffff880ad9e7c5f8<br/> Dec 20 10:00:54 brutus3 kernel: Call Trace:<br/> Dec 20 10:00:54 brutus3 kernel: <span class="error">&#91;&lt;ffffffffa0582967&gt;&#93;</span> ? cfs_hash_bd_lookup_intent+0x37/0x130 <span class="error">&#91;libcfs&#93;</span><br/> Dec 20 10:00:54 brutus3 kernel: <span class="error">&#91;&lt;ffffffffa0581f22&gt;&#93;</span> ? cfs_hash_bd_add_locked+0x62/0x90 <span class="error">&#91;libcfs&#93;</span><br/> Dec 20 10:00:54 brutus3 kernel: <span class="error">&#91;&lt;ffffffff8150f1ee&gt;&#93;</span> __mutex_lock_slowpath+0x13e/0x180<br/> Dec 20 10:00:54 brutus3 kernel: <span class="error">&#91;&lt;ffffffff8150f08b&gt;&#93;</span> mutex_lock+0x2b/0x50<br/> Dec 20 10:00:54 brutus3 kernel: <span class="error">&#91;&lt;ffffffffa06d5ebf&gt;&#93;</span> cl_lock_mutex_get+0x6f/0xd0 <span class="error">&#91;obdclass&#93;</span><br/> Dec 20 10:00:54 brutus3 kernel: <span class="error">&#91;&lt;ffffffffa0a4495a&gt;&#93;</span> osc_ldlm_blocking_ast+0x7a/0x350 <span class="error">&#91;osc&#93;</span><br/> Dec 20 10:00:54 brutus3 kernel: <span class="error">&#91;&lt;ffffffffa057d2c1&gt;&#93;</span> ? libcfs_debug_msg+0x41/0x50 <span class="error">&#91;libcfs&#93;</span><br/> Dec 20 10:00:54 brutus3 kernel: <span class="error">&#91;&lt;ffffffffa07f1f00&gt;&#93;</span> ldlm_handle_bl_callback+0x130/0x400 <span class="error">&#91;ptlrpc&#93;</span><br/> Dec 20 10:00:54 brutus3 kernel: <span class="error">&#91;&lt;ffffffffa07f2451&gt;&#93;</span> ldlm_bl_thread_main+0x281/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> Dec 20 10:00:54 brutus3 kernel: <span class="error">&#91;&lt;ffffffff81063310&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> Dec 20 10:00:54 brutus3 kernel: <span class="error">&#91;&lt;ffffffffa07f21d0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> Dec 20 10:00:54 brutus3 kernel: <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> Dec 20 10:00:54 brutus3 kernel: <span class="error">&#91;&lt;ffffffffa07f21d0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> Dec 20 10:00:54 brutus3 kernel: <span class="error">&#91;&lt;ffffffffa07f21d0&gt;&#93;</span> ? ldlm_bl_thread_main+0x0/0x3d0 <span class="error">&#91;ptlrpc&#93;</span><br/> Dec 20 10:00:54 brutus3 kernel: <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> ================================================================<br/> looks like an indication of a problem/dead-lock on the Client-side.</p> <p>And the associated Lustre debug-traces sequence is also always the same and like :<br/> ==================================================================<br/> 00000100:00100000:33.0:1387529928.782898:0:3546:0:(events.c:352:request_in_callback()) peer: 12345-10.201.62.37@o2ib<br/> 00000100:00100000:25.0:1387529928.782936:0:3640:0:(service.c:1867:ptlrpc_server_handle_req_in()) got req x1444519122204514<br/> 00000100:00080000:25.0:1387529928.782961:0:3640:0:(service.c:1079:ptlrpc_update_export_timer()) updating export LOV_OSC_UUID at 1387529928 exp ffff8820303d4000<br/> 00000100:00100000:25.0:1387529928.782985:0:3640:0:(nrs_fifo.c:182:nrs_fifo_req_get()) NRS start fifo request from 12345-10.201.62.37@o2ib, seq: 15852<br/> 00000100:00100000:25.0:1387529928.783003:0:3640:0:(service.c:2011:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ldlm_cb03_002:LOV_OSC_UUID+4:4310:x1444519122204514:12345-10.201.62.37@o2ib:106<br/> 00010000:00010000:25.0:1387529928.783029:0:3640:0:(ldlm_lockd.c:1882:ldlm_handle_gl_callback()) ### client glimpse AST callback handler ns: nero-OST000e-osc-ffff88203a24a800 lock: ffff88091b327a00/0x30c32bca9aefeafc lrc: 8/0,0 mode: PW/PW res: 119352871/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0-&gt;18446744073709551615) flags: 0x29400000000 nid: local remote: 0x87d5d77ddb4e8fb0 expref: -99 pid: 45673 timeout: 0 lvb_type: 1<br/> 00000020:00010000:25.0:1387529928.783063:0:3640:0:(cl_object.c:305:cl_object_glimpse()) header@ffff88035bd6a6a0[0x0, 5, <span class="error">&#91;0x20001c158:0xdd90:0x0&#93;</span> hash]<br/> 00000020:00010000:25.0:1387529928.783065:0:3640:0:(cl_object.c:305:cl_object_glimpse()) size: 348 mtime: 1387455174 atime: 1387456028 ctime: 1387455174 blocks: 8<br/> 00000020:00010000:25.0:1387529928.783068:0:3640:0:(cl_object.c:305:cl_object_glimpse()) header@ffff880334d7e7f0[0x0, 3, <span class="error">&#91;0x1000e0000:0x71d2e27:0x0&#93;</span> hash]<br/> 00000020:00010000:25.0:1387529928.783069:0:3640:0:(cl_object.c:305:cl_object_glimpse()) size: 348 mtime: 1387455174 atime: 1387456028 ctime: 1387455174 blocks: 8<br/> 00000100:00100000:25.0:1387529928.783125:0:3640:0:(service.c:2055:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ldlm_cb03_002:LOV_OSC_UUID+4:4310:x1444519122204514:12345-10.201.62.37@o2ib:106 Request procesed in 135us (231us total) trans 0 rc 0/0<br/> 00000100:00100000:25.0:1387529928.783131:0:3640:0:(nrs_fifo.c:244:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.201.62.37@o2ib, seq: 15852<br/> 00010000:00010000:12.0:1387529928.783139:0:12517:0:(ldlm_lockd.c:1696:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: nero-OST000e-osc-ffff88203a24a800 lock: ffff88091b327a00/0x30c32bca9aefeafc lrc: 8/0,0 mode: PW/PW res: 119352871/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0-&gt;18446744073709551615) flags: 0x29400000000 nid: local remote: 0x87d5d77ddb4e8fb0 expref: -99 pid: 45673 timeout: 0 lvb_type: 1<br/> 00010000:00010000:12.0:1387529928.783150:0:12517:0:(ldlm_lockd.c:1709:ldlm_handle_bl_callback()) Lock ffff88091b327a00 already unused, calling callback (ffffffffa0a448e0)<br/> ==================================================================</p> <p>At a first look, problem seems related to <a href="https://jira.whamcloud.com/browse/LU-874" title="Client eviction on lock callback timeout " class="issue-link" data-issue-key="LU-874"><del>LU-874</del></a>/<a href="https://jira.whamcloud.com/browse/LU-2683" title="Client deadlock in cl_lock_mutex_get" class="issue-link" data-issue-key="LU-2683"><del>LU-2683</del></a>, but all the concerned patches have been landed to b2_4 so it is unlikely to be the case finally.<br/> A good thing would be to get a crash-dump at the time of the evict or at least a full threads stack-traces, to understand who owns the cl_lock mutex preventing ldlm_bl_xx threads to complete/answer the blocking-AST.</p> <p>I will try to come back with a procedure to get more infos during other occurrences or may be a with specific debug patch.</p> <p>Humm, in fact I wonder if this problem could not be a new facet of the same problem than <a href="https://jira.whamcloud.com/browse/LU-874" title="Client eviction on lock callback timeout " class="issue-link" data-issue-key="LU-874"><del>LU-874</del></a>/<a href="https://jira.whamcloud.com/browse/LU-2683" title="Client deadlock in cl_lock_mutex_get" class="issue-link" data-issue-key="LU-2683"><del>LU-2683</del></a> but under 2.2/2.4 hybrid un-tested/supported configuration ??…</p> <p>Could it be possible to install+run with an instrumented version of Lustre-Client modules, at least on one of the impacted login nodes, that will allow to get a crash-dump (instead of lustre-debug log dump!) upon eviction ??</p> <p>Just created a set of Client RPMs for the same Kernel 2.6.32-358.6.2.el6.x86_64 and Lustre 2.4.0-RC2 versions used on-site, allowing for a LBUG() (in ptlrpc_invalidate_import_thread()) to occur instead of a debug-log dump (if obd_dump_on_eviction exactly = -1) upon eviction. This should allow to get a crash-dump (if panic_on_lbug is also != 0) upon eviction.<br/> I am currently testing its functionality in-house, will provide an update soon.</p> <p>On the other hand, I am also trying to setup a similar Lustre-Server/Lustre-Client Version (2.2.0-RC2-<del>PRISTINE/2.4.0-RC2</del>-CHANGED) platform to try reproduce the issue with the prog from <a href="https://jira.whamcloud.com/browse/LU-874" title="Client eviction on lock callback timeout " class="issue-link" data-issue-key="LU-874"><del>LU-874</del></a>.</p> <p>My RPMs, including the LBUG() upon eviction patch/instrumentation, testing has been successful. The instrumented RPMs have been pushed on customer upload for installation and in order to allow for a crash-dump to be taken during an eviction.</p> <p>Customer will also try to run the <a href="https://jira.whamcloud.com/browse/LU-874" title="Client eviction on lock callback timeout " class="issue-link" data-issue-key="LU-874"><del>LU-874</del></a> reproducer on its platform.</p> <p>We have the RPMs on half of the login nodes in place, checked our kdump config with sysrq-trigger and waiting for the first crash-dump during an eviction.</p> <p>We run reproducer <a href="https://jira.whamcloud.com/browse/LU-874" title="Client eviction on lock callback timeout " class="issue-link" data-issue-key="LU-874"><del>LU-874</del></a> and also <a href="https://jira.whamcloud.com/browse/LU-4112" title="Random eviction of clients on lock callback timeout" class="issue-link" data-issue-key="LU-4112"><del>LU-4112</del></a> aggressively but no eviction nor anything else happened.</p> <p>Cheers<br/> Allen</p> <p>Allen, thanks for helping so much.</p> <p>crash dump text file uploaded to jira and vmcore dump provided to Gabriele Paciucci.</p> <p>Another dump file for reference. <br/> Full coredump uploaded to Gabriele.</p> <p>Sorry for the delay to complete crash-dumps from instrumented RPMs analysis.<br/> Now, after spending some time doing so, I think the problem you face is the same than the one already tracked as part of <a href="https://jira.whamcloud.com/browse/LU-4300" title="ptlrpcd threads deadlocked in cl_lock_mutex_get" class="issue-link" data-issue-key="LU-4300"><del>LU-4300</del></a>.<br/> To confirm this, could it be possible that you disable ELC (with "echo 0 &gt; /proc/fs/lustre/ldlm/namespaces/*/early_lock_cancel") on one (all?) of the login nodes that trigger the evictions ?</p> <p>Thanks for the analysis of the crash dump, Bruno.<br/> I have disabled ELC on all the involved login nodes now. Lets see, what happens.</p> <p>Hello Eric, can you give some feedback if running with ELC disabled helped or not?</p> <p>I cannot tell if it is successful as we had no eviction prior to set ELC=0 for at least a fortnight.<br/> I have tried to stress the login nodes more than usual, but I could not force an eviction. But this is not unusual,<br/> as we never managed to trigger the eviction with a certain usage pattern…</p> <p>I will of course update asap we have a new eviction, although I hope that it never ever occurs again (best case).</p> <p>Eric, can you give some update ?? Thanks in advance.</p> <p>We had no eviction on the 4 nodes until 02.04.2014. At this time, we rebooted one of these nodes with the longest uptime of all (53 days) and let the default early_lock_cancel (1) configured. This node run for 6 days until 08.04.2014 and then has been evicted by first a single OSS and rapidly by others. I have seen users doing many rsync operations on Lustre at that time (apart from other usual load). <br/> Lustre logs showed the following:</p> <p>00010000:00010000:45.0:1396944724.065662:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff881853511200/0x5d1861be72b7c5eb lrc: 0/0,0 mode: -<del>/PR res: 138708343/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5df055 expref: -99 pid: 43945 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065672:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff881d59ae9a00/0x5d1861be72b7b7f2 lrc: 0/0,0 mode: -<del>/PR res: 138708334/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5def52 expref: -99 pid: 43945 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065678:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff8818726e8600/0x5d1861be72b7a728 lrc: 0/0,0 mode: -<del>/PR res: 138708324/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5dee41 expref: -99 pid: 43945 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065684:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff881a88e77400/0x5d1861be72cf6d93 lrc: 0/0,0 mode: -<del>/PR res: 138711793/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5fbaff expref: -99 pid: 44920 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065703:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff881e9e464c00/0x5d1861be72cfd56d lrc: 0/0,0 mode: -<del>/PR res: 138711859/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5fc1b2 expref: -99 pid: 44920 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065709:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff881c15923e00/0x5d1861be72cf1691 lrc: 0/0,0 mode: -<del>/PR res: 138711753/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5fb5cd expref: -99 pid: 44920 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065716:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff881cab0b1400/0x5d1861be72b78afe lrc: 0/0,0 mode: -<del>/PR res: 138708312/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5debd2 expref: -99 pid: 43945 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065722:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff880b360b0c00/0x5d1861be72cfc8b6 lrc: 0/0,0 mode: -<del>/PR res: 138711851/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5fc0cb expref: -99 pid: 44920 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065728:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff8815907c4200/0x5d1861be72cfac3f lrc: 0/0,0 mode: -<del>/PR res: 138711835/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5fbee8 expref: -99 pid: 44920 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065734:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff8815399ed400/0x5d1861be72cf7232 lrc: 0/0,0 mode: -<del>/PR res: 138711798/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5fbb53 expref: -99 pid: 44920 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065749:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff881915410400/0x5d1861be72cf90cb lrc: 0/0,0 mode: -<del>/PR res: 138711819/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5fbd52 expref: -99 pid: 44920 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065759:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff88146b2d0c00/0x5d1861be72cf1ef6 lrc: 0/0,0 mode: -<del>/PR res: 138711760/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5fb63d expref: -99 pid: 44920 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065766:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff8813f3483600/0x5d1861be72b78333 lrc: 0/0,0 mode: -<del>/PR res: 138708303/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5deac8 expref: -99 pid: 43945 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065772:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff880725999600/0x5d1861be72cfb752 lrc: 0/0,0 mode: -<del>/PR res: 138711843/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5fbfb3 expref: -99 pid: 44920 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065779:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff8819f2a30200/0x5d1861be72cf3084 lrc: 0/0,0 mode: -<del>/PR res: 138711769/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5fb708 expref: -99 pid: 44920 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065785:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff8814e99b5600/0x5d1861be72cf69bf lrc: 0/0,0 mode: -<del>/PR res: 138711791/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5fba9d expref: -99 pid: 44920 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065791:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff8804d6b30200/0x5d1861be72d291cf lrc: 0/0,0 mode: -<del>/PW res: 138711739/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869480000000 nid: local remote: 0x35306a32ed604737 expref: -99 pid: 5386 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065801:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff881902ad2200/0x5d1861be72cf58f5 lrc: 0/0,0 mode: -<del>/PR res: 138711785/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5fb99a expref: -99 pid: 44920 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065808:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff88201e820600/0x5d1861be72cf4449 lrc: 0/0,0 mode: -<del>/PR res: 138711776/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;18446744073709551615) flags: 0x869400000000 nid: local remote: 0x35306a32ed5fb85f expref: -99 pid: 44920 timeout: 0 lvb_type: 1<br/> 00010000:00010000:45.0:1396944724.065818:0:12697:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: nero-OST0019-osc-ffff880c39f1ec00 lock: ffff881108e46000/0x5d1861be72cf6e34 lrc: 0/0,0 mode: -<del>/PW res: 138711795/0 rrc: 1 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0</del>&gt;4095) flags: 0x869400000000 nid: local remote: 0x35306a32ed5fbb06 expref: -99 pid: 44920 timeout: 0 lvb_type: 1</p> <p>We have now set ELC on all the nodes again. <br/> So, there might be a reasonable chance that ELC will work out (hopefully!).</p> <p>Hello Eric,<br/> Can you give us some feedback on how "worked" ELC ?? I hope we will definitely confirm that this ticket is a dup of <a href="https://jira.whamcloud.com/browse/LU-4300" title="ptlrpcd threads deadlocked in cl_lock_mutex_get" class="issue-link" data-issue-key="LU-4300"><del>LU-4300</del></a> and we can close it accordingly ...</p> <p>Hello Bruno<br/> All nodes are up for at least 45 days now without any new evictions. I guess we can say that ELC works and may close this ticket.<br/> Thanks for collaboration!</p> <p>Thanks Matteo!</p> Development Rank 1|hzw2db: Rank (Obsolete) 10466 Severity