https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4797 Lustre <p>Hi,</p> <p>After 3 days in production with Lustre 2.4.2, CEA is suffering from the following "assertion failed" issue about 5 times a day:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 4089:0:(lovsub_lock.c:103:lovsub_lock_state()) ASSERTION( cl_lock_is_mutexed(slice-&gt;cls_lock) ) failed: LustreError: 4089:0:(lovsub_lock.c:103:lovsub_lock_state()) LBUG Pid: 4089, comm: %%AQC.P.I.O Call Trace: [&lt;ffffffffa0af4895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa0af4e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa1065d51&gt;] lovsub_lock_state+0x1a1/0x1b0 [lov] [&lt;ffffffffa0bd7a88&gt;] cl_lock_state_signal+0x68/0x160 [obdclass] [&lt;ffffffffa0bd7bd5&gt;] cl_lock_state_set+0x55/0x190 [obdclass] [&lt;ffffffffa0bdb8d9&gt;] cl_enqueue_try+0x149/0x300 [obdclass] [&lt;ffffffffa105e0da&gt;] lov_lock_enqueue+0x22a/0x850 [lov] [&lt;ffffffffa0bdb88c&gt;] cl_enqueue_try+0xfc/0x300 [obdclass] [&lt;ffffffffa0bdcc7f&gt;] cl_enqueue_locked+0x6f/0x1f0 [obdclass] [&lt;ffffffffa0bdd8ee&gt;] cl_lock_request+0x7e/0x270 [obdclass] [&lt;ffffffffa0be2b8c&gt;] cl_io_lock+0x3cc/0x560 [obdclass] [&lt;ffffffffa0be2dc2&gt;] cl_io_loop+0xa2/0x1b0 [obdclass] [&lt;ffffffffa10dba90&gt;] ll_file_io_generic+0x450/0x600 [lustre] [&lt;ffffffffa10dc9d2&gt;] ll_file_aio_write+0x142/0x2c0 [lustre] [&lt;ffffffffa10dccbc&gt;] ll_file_write+0x16c/0x2a0 [lustre] [&lt;ffffffff811895d8&gt;] vfs_write+0xb8/0x1a0 [&lt;ffffffff81189ed1&gt;] sys_write+0x51/0x90 [&lt;ffffffff81091039&gt;] ? sys_times+0x29/0x70 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>This issue is very similar to <a href="https://jira.whamcloud.com/browse/LU-4693" title="(lovsub_lock.c:103:lovsub_lock_state()) ASSERTION( cl_lock_is_mutexed(slice-&gt;cls_lock) ) failed:" class="issue-link" data-issue-key="LU-4693"><del>LU-4693</del></a>, which is itself a duplicate of <a href="https://jira.whamcloud.com/browse/LU-4692" title="(osc_lock.c:1204:osc_lock_enqueue()) ASSERTION( ols-&gt;ols_state == OLS_NEW ) failed: Impossible state: 6" class="issue-link" data-issue-key="LU-4692"><del>LU-4692</del></a>, for which there is unfortunately no fix yet.</p> <p>Please ask if you need additional information that could help the diagnostic and resolution of the problem.</p> <p>Sebastien.</p> LU-4797

ASSERTION( cl_lock_is_mutexed(slice->cls_lock) ) failed

Bug Major Resolved Duplicate Jinshan Xiong Sebastien Buisson Fri, 21 Mar 2014 14:00:33 +0000 Mon, 28 Apr 2014 14:21:49 +0000 Fri, 4 Apr 2014 16:24:42 +0000 Lustre 2.4.2 0 9 <p>Hello Sebastien, are there any crash-dump available ?? If yes, could it be possible to extract the debug-log content with the crash-tool expansion described in CFS BZ #13155 (source to be re-compiled are available, and I know you may need them to install+use on-site) ?? BTW, waht is the default debug mask you run with ??</p> <p>Hi Bruno,</p> <p>I have forwarded your request to on-site Support team. Do you want us to attach the requested debug-log content to this ticket? Or could we have a look by ourselves and search for something specific?</p> <p>Cheers,<br/> Sebastien.</p> <p>Bobijam</p> <p>Does this appear to be a duplicate of <a href="https://jira.whamcloud.com/browse/LU-4692" title="(osc_lock.c:1204:osc_lock_enqueue()) ASSERTION( ols-&gt;ols_state == OLS_NEW ) failed: Impossible state: 6" class="issue-link" data-issue-key="LU-4692"><del>LU-4692</del></a>? Is there anything additional that would assist with debugging this issue?</p> <p>Peter</p> <p>Beside crash-dump, is it possible to find a rehit procedure?</p> <p>Hi Bobijam,</p> <p>All I know is that the impacted file is a log file in which several processes write.</p> <p>I have forwarded your request to our on-site Support team.</p> <p>Cheers,<br/> Sebastien.</p> <p>The workload should be reproduced by launching the script run_reproducer_2.sh with 4 processes on 2 nodes.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>:::::::::::::: run_reproducer_2.sh :::::::::::::: #!/bin/bash sleeptime=$(( ( ${SLURM_PROCID} * 10000 ) + 1000000 )) reproducer2.sh 10 /&lt;path&gt;/mylog ${sleeptime} ${SLURM_JOBID}_${SLURM_PROCID} :::::::::::::: reproducer2.sh :::::::::::::: #!/bin/bash # for i in $(seq 1 $1) do usleep $3 echo $(date) $(date '+%N') $4 $3 testing write in append mode &gt;&gt; $2 done </pre> </div></div> <p>LLNL also hit this in testing Lustre version 2.4.2-6chaos on a Lustre client.</p> <p>I've been being trying the reproduce script since yesterday, haven't gotten a hit yet. I did a little change for my VM test environment.</p> <p>$ cat ~/tmp/reproducer.sh<br/> #!/bin/bash</p> <p>ls &gt; /dev/null &amp;<br/> PID=$!</p> <p>for i in $(seq 1 10000)<br/> do<br/> echo $PID - $i<br/> usleep $((($PID * 100) + 1500000))<br/> echo "$(date) $(date '+%N') $PID-$i ***** testing write in append mode" &gt;&gt; /mnt/lustre/file<br/> done</p> <p>and on 2 nodes, run "$~/tmp/reproducer.sh &amp;" five times, I think the basic idea is the same. </p> <p>Sebastien, How long would it rehit the issue in your case?</p> <p>Is it possible to collect a crash dump for this issue?</p> <p>The only difference between 2.4.1 and 2.4.2 is <a href="https://jira.whamcloud.com/browse/LU-3027" title="Failure on test suite parallel-scale test_write_disjoint: invalid file size 140329 instead of 160376 = 20047 * 8" class="issue-link" data-issue-key="LU-3027"><del>LU-3027</del></a>, there are two patches landed for it. Can anyone please revert them and try again? Probably this way we can get some clues for this issue.</p> <p>Please note that Sebastien's script is not a reproducer of this crash, but something similar to the workload that leads to this crash. This code only easily triggers a lot of evictions.</p> <p>Seb, sorry to only answer to your own comment/reply on "21/Mar/14 3:53 PM", so yes it could be useful as a 1st debugging info to get the debug-log content extracted from the crash-dump. BTW, I hope that you run with a default debug-levels mask that will be enough to gather accurate traces for this problem ??...</p> <p>Aurelien, concerning the evictions likely to be reproduced on site with this script, is it also possible to get a Lustre debug-log, at least from evicted Client side and with the full debug mask/traces enabled ?</p> <p>Bobijam,</p> <p>To make it clear, 2.4.2 is the first b2_4 release used at CEA, so we are not comparing with 2.4.1, but with 2.1.6.</p> <p>Sebastien.</p> <p>About the testcase :</p> <ul class="alternate" type="square"> <li>the waiting time is too large than the initial reproducer by x15</li> <li>fix the run a processor/core with taskset for example when<br/> you run /tmp/reproducer.sh on the 2 nodes <br/> for cpu in $(seq 0 5) # nb cpus nodes<br/> taskset -c $cpu /tmp/reproducer.sh &amp; ; done</li> <li>My first testcase was without usleep and 32 task/process by nodes<br/> and 1000 write/echo by task</li> <li>no lustre trace enable</li> </ul> <p>my wait time is 1.5x of the initial reproducer, I'll try the taskset.</p> <p>update: tried taskset, usleep 1000000, each cpu took 250 echo taskes, 2 VM nodes, each node has 2 cpus, haven't got the hit yet.</p> <p>Bobi, I attach the Client-side debug logs that have been taken during an on-site run of the reproducer on 2x nodes (lascaux<span class="error">&#91;2890-2891&#93;</span>) with evictions :</p> <p> _ file "eviction" is a log showing the reproducer run+errors and the actions taken.<br/> _ files lascaux289<span class="error">&#91;0,1&#93;</span><em>dump_lustre_pb_eviction</em><span class="error">&#91;1,2&#93;</span>.gz are Lustre debug logs taken on both nodes and as shown in "eviction" file/trace.</p> <p>Hope this will allow you to get more infos to qualify.</p> <p>still checking the eviction logs, there are lock enqueue and blocking ast call trace intertwined, but that's normal, haven't found where the race happens causing this mutex assertion violation.</p> <p>Please try patch: <a href="http://review.whamcloud.com/9881" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9881</a></p> <p>I believe this is the same issue in <a href="https://jira.whamcloud.com/browse/LU-4591" title="Related cl_lock failures on master/2.5" class="issue-link" data-issue-key="LU-4591"><del>LU-4591</del></a>.</p> <p>duplicate of <a href="https://jira.whamcloud.com/browse/LU-4558" title="Crash in cl_lock_put on racer" class="issue-link" data-issue-key="LU-4558"><del>LU-4558</del></a></p> <p>Hi Zhenyu Xu,<br/> Could you explain me why you think "there are lock enqueue and blocking ast call trace intertwined" <br/> and eviction. You said "that's normal". I don't understand why there a lot contention on this because<br/> we just add some bytes at the end of one file with just 4 process on 2 nodes . for me and on a 2.1.x <br/> lustre distribution we haven't this contention.<br/> and that could explain why sometime we meet the race fix by <a href="https://jira.whamcloud.com/browse/LU-4558" title="Crash in cl_lock_put on racer" class="issue-link" data-issue-key="LU-4558"><del>LU-4558</del></a><br/> <a href="https://jira.whamcloud.com/browse/LU-3027" title="Failure on test suite parallel-scale test_write_disjoint: invalid file size 140329 instead of 160376 = 20047 * 8" class="issue-link" data-issue-key="LU-3027"><del>LU-3027</del></a> and <a href="https://jira.whamcloud.com/browse/LU-4495" title="client evicted on parallel append write to the shared file." class="issue-link" data-issue-key="LU-4495"><del>LU-4495</del></a> could explain this contention ?<br/> thanks</p> <p>every write needs a exclusive lock, write from other node will cause the lock holder to relinquish the lock, and multiple write upon the same file from different node will cause lock enqueue and lock blocking ast intertwined, by that I meant normal.</p> Duplicate LU-4693 Related LU-4558 LU-4591 Development Rank 1|hzwi3b: Rank (Obsolete) 13202 Severity