https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4851 Lustre <p>How to trigger the kernel panic:</p> <p>1. start-up amplxe-gui and create a new project, e.g. "test".</p> <p>2. Then set the executable to use to say /bin/ls.</p> <p>3. Now create a new analysis and select any of the ones available to your process architecture and run it.</p> <p>4 .Once the simulation has completed, exit amplxe-gui.</p> <p>5. Load amplxe-gui again and re-run the simulation in step 3.</p> <p>6. Exit amplxe-gui once the simulation completes.</p> <p>7. Repeat steps 5 to 6 until the node kernel panics (normally takes three or four attempts)</p> <p>It should be noted that users' home directories are stored on Lustre and also the environment variable $TMPDIR is set to a directory within users' homespaces on Lustre. The Vtune "project files" are therefore stored on Lustre. I suspect that the reading or writing of these files by Vtune could be the cause of the kernel panic.</p> LU-4851

Lustre kernel panic when using Intel Vtune Amplier XE 2013

Bug Major Resolved Fixed Bruno Faccini Rajeshwaran Ganesan Wed, 2 Apr 2014 09:54:39 +0000 Fri, 16 May 2014 14:06:10 +0000 Fri, 16 May 2014 14:06:09 +0000 Lustre 2.4.3 0 7 <p>This issue occurs only on the client, </p> <p>here is the lustre versions</p> <p>The kernel panic only occurs on the client where Vtune was being executed.</p> <p>The MDS and OSS nodes do not suffer any issues.</p> <p>The clients are running SLES 11 SP3:</p> <p>lustre-client-2.4.3-3.0.93_0.8_default_gfc544a1<br/> lustre-client-modules-2.4.3-3.0.93_0.8_default_gfc544a1<br/> lustre-client-tests-2.4.3-3.0.93_0.8_default_gfc544a1<br/> lustre-iokit-1.4.0-1</p> <p>Kernel: 3.0.93-0.8-default</p> <p>The MDS and OSS nodes are running CentOS:</p> <p>kernel-2.6.32-358.18.1.el6_lustre.es50.x86_64<br/> kernel-devel-2.6.32-358.18.1.el6_lustre.es50.x86_64<br/> kernel-firmware-2.6.32-358.18.1.el6_lustre.es50.x86_64<br/> kernel-headers-2.6.32-358.18.1.el6_lustre.es50.x86_64<br/> kernel-ib-1.5.3-2.6.32_358.18.1.el6_lustre.es50.x86_64_2.6.32_358.18.1.el6_lustre.es50.x86_64<br/> kernel-ib-devel-1.5.3-2.6.32_358.18.1.el6_lustre.es50.x86_64_2.6.32_358.18.1.el6_lustre.es50.x86_64<br/> lustre-2.4.1-ddn1.0_2.6.32_358.18.1.el6_lustre.es50.x86_64_ES.x86_64<br/> lustre-ldiskfs-4.1.0-2.6.32_358.18.1.el6_lustre.es50.x86_64.x86_64<br/> lustre-modules-2.4.1-ddn1.0_2.6.32_358.18.1.el6_lustre.es50.x86_64_ES.x86_64<br/> lustre-osd-ldisfs-2.4.1-ddn1.0_2.6.32_358.18.1.el6_lustre.es50.x86_64_ES.x86_64<br/> lustre-source-2.4.1-ddn1.0_2.6.32_358.18.1.el6_lustre.es50.x86_64_ES.x86_64</p> <p>Hello Rajeshwaran,<br/> Is there a crash-dump available for one of the occurrences, and if yes can you provide it ??<br/> Also was there some Lustre debug (like rpctrace, dlmtrace,...) enabled at the time of the crashes ?</p> <p>This actually might be a dup of <a href="https://jira.whamcloud.com/browse/LU-4403" title="ASSERTION( lock-&gt;l_readers &gt; 0 )" class="issue-link" data-issue-key="LU-4403"><del>LU-4403</del></a></p> <p>Oleg, even if <a href="https://jira.whamcloud.com/browse/LU-4403" title="ASSERTION( lock-&gt;l_readers &gt; 0 )" class="issue-link" data-issue-key="LU-4403"><del>LU-4403</del></a> occurs on Server/MDS side ??</p> <p>actually, with a client crash, it might be something else. We definitely need a full backtrace from the crash here at the very least.</p> <p>Hello Rajeshwaran,<br/> As we agreed during the conf-call, I am trying to reproduce the issue in-house. I am currently unable to reproduce the LBUG ("(ldlm_lock.c:851:ldlm_lock_decref_internal_nolock()) ASSERTION( lock-&gt;l_readers &gt; 0 ) failed:") you encountered.<br/> BTW, you seem to use VTune's GUI, but did you try to run with command-line only ("amplxe-cl -<span class="error">&#91;collect,report&#93;</span> ..."), just to see if we can simplify the reproducer ??</p> <p>Hello Rajeshwaran, <br/> Could you please have a shot at running VTune using just the command line interface? (As requested above).<br/> If you are able to provide us with a reproducer from a command line run, it will improve our chances that we will be successful in reproducing this issue, on our in-house platform.</p> <p>Many thanks,<br/> ~ jfc.</p> <p>Yes, it would be helpful and easier to reproduce in-house if you could confirm that you also reproduce by running VTune via its command-line interface. So, if I try to mimic your GUI's actions to reproduce wit command-line, it should be something like :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>for i in `seq 1 100` ; do /opt/intel/vtune_amplifier_xe_2013/bin64/amplxe-cl -collect hotspots -result-dir=/mnt/lustre/vtune/intel/amplxe/projects/ls_lustre2/r${i}hs -app-working-dir=/mnt/lustre/vtune/ ls -laR /mnt/lustre /opt/intel/vtune_amplifier_xe_2013/bin64/amplxe-cl -report hotspots -r /mnt/lustre/vtune/intel/amplxe/projects/ls_lustre2/r${i}hs done </pre> </div></div> <p>On the other hand, I am still trying and unable to reproduce even using the GUI interface as you reported ... So, I begin to think that it is more configuration dependent (OSTs number, default/used striping, ...) and not only a pure VTune behavior consequence ...</p> <p>BTW, I forgot to ask which mount options are used on your Client ? And particularly, do you mount Lustre with flock/localflock/noflock ?</p> <p>Hello, </p> <p>Lustre is mounted on our login nodes with the following options: “rw”, “_netdev” and “flock”</p> <p>We mount it under /scratch</p> <p>The mount definitions for Lustre are stored in /etc/fstab and we manually mount the file system once the node has been booted, e.g.:</p> <p> mount /scratch</p> <p>No output from the mount command is produced. In dmesg we see:</p> <p>Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-22597-gfc544a1-PRISTINE-../lustre/scripts</p> <p>LNet: Added LNI XXX.XXX.XXX.XXX@o2ib <span class="error">&#91;YYY/YYY/YYY/YYY&#93;</span></p> <p>Lustre: Layout lock feature supported.</p> <p>Lustre: Mounted scratch-client</p> <p>Note: IP info redacted from the output above.</p> <p>The default stripe count is 12 (all OSTs).</p> <p>Please find attached a dmesg trace that is produced when the LBUG kernel panic occurs.</p> <p>I've not be able to recreate the issue with the command line utility.</p> <p>However, I have found a quicker way to trigger the bug by just merely closing and opening the Vtune GUI (i.e. no need to create any projects or analyses)</p> <p>E.g.</p> <p>cd ~<br/> rm -rf .intel intel tmp/*<br/> amplx-gui</p> <ol> <li>now close the GUI (if it loads) and rerun the "amplx-gui" command</li> <li>repeat until crash occurs (normally by 4th attempt)</li> </ol> <p>The crash is triggered whilst the splash screen for Vtune is visible but before the actual main Vtune window is displayed. Therefore, I assume the bug is being triggered by some of the Vtune GUI's start-up code?</p> <p>As suspected, I cannot recreate the issue if I move my homespace to a non-Lustre file system.</p> <p>Hope this helps,</p> <p>Hello Bruno, </p> <p>Please let me know, if you need any other logs or any other commands to try. </p> <p>I can send them and get the results.</p> <p>Thanks,<br/> Rajesh</p> <p>Hello Rajeshwaran,<br/> Thanks for all these additional infos!<br/> BTW, with the new dmesg you provided, and the panic/LBUG stack now available, it is clear that the problem occurs during FLock operations.<br/> This allows me to suspect that you may trigger the problem/race I already worked on as part of <a href="https://jira.whamcloud.com/browse/LU-3684" title="LBUG/&quot;ldlm_lock_decref_internal_nolock()) ASSERTION(lock-&gt;l_readers &gt; 0) failed&quot; running Bull&#39;s NFS locktests" class="issue-link" data-issue-key="LU-3684"><del>LU-3684</del></a>, where I pushed a patch to fix but does not seem to be in b2_4 ...<br/> Will also try again to reproduce using your new+simplified instructions.</p> <p>Hello Bruno,</p> <p>Cu. tried running the CLI as shown below, but after 100 iterations the login node refuses to crash.</p> <p>Yet if they switch back to the GUI, they can get the login node to crash after a few attempted launches as described previously.<br/> Thanks,<br/> Rajesh</p> <p>Hello Bruno,</p> <p>some good news, the localflock is not causing the crash. </p> <p>if they mount using flock, it is crashing as usual. </p> <p>Is there any harm in mounting our Lustre file system with “localflock” on our clients rather then “flock”?</p> <p>Whats your suggestion on localflock and flock</p> <p>Thanks,<br/> Rajesh</p> <p>After I found that the issue is FLock related, I had also in mind to ask you to try with either localflock and/or noflock mount options, but decided not to do so right now since I assumed that your customer is likely to use applications that require cluster-wide/multi-nodes FLock support (guaranteed with flock option), and not only local/single-node FLock support (localflock scope). But since you asked/tried, may be you can check with your customer if this can feet with their production/applications ??...</p> <p>Are we getting any fix for flock option?</p> <p>1st of all I would like to add an update about my reproduction efforts in-house ... Unfortunatelly I am still unable to reproduce until now, and this even after I used your latest instructions and configuration details.</p> <p>Concerning a possible fix to cover the flock option usage, I made a b2_4 back-port of my previous patch for <a href="https://jira.whamcloud.com/browse/LU-1126" title="Client file locking issue. Assertion triggered when decrementing a read lock on an item that has no existing read locks." class="issue-link" data-issue-key="LU-1126"><del>LU-1126</del></a>/<a href="https://jira.whamcloud.com/browse/LU-3684" title="LBUG/&quot;ldlm_lock_decref_internal_nolock()) ASSERTION(lock-&gt;l_readers &gt; 0) failed&quot; running Bull&#39;s NFS locktests" class="issue-link" data-issue-key="LU-3684"><del>LU-3684</del></a>, it is available at <a href="http://review.whamcloud.com/9968" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9968</a>, and running our tests suites.</p> <p>Last, and since I am unable to reproduce, did you pursue (and succeed) with the process of having informations out from the site and as we discussed during conf-call with customer ?? As we already insisted, the full debug log from Client side taken during reproducer run wouls be more than helpful to understand the issue, and also to confirm that my patch is the fix ...</p> <p>Hello Bruno,</p> <p>Could you please provide the source RPM with the patch, I can ask them rebuild and install it. we can verify it in the client. </p> <p>Thanks,<br/> Rajesh</p> <p>Builds with my patch are available under <a href="http://build.whamcloud.com/job/lustre-reviews/23081/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/23081/</a>. Can you access it ? If yes, you can check the target OS from the build matrix, and then follow the "Build Artifacts" link where you can find the corresponding source rpm.</p> <p>But I suggest you wait for the success of our test suites run before to apply it on-site.</p> <p>Thanks for your help, Sure I can wait, please let me know once it passes the test. </p> <p>#9968 has successfully passed almost all Maloo tests, only one unrelated failure in lustre-rsync-test/test_8 for a known+unrelated failure already tracked in <a href="https://jira.whamcloud.com/browse/LU-3573" title="lustre-rsync-test test_8: @@@@@@ FAIL: Failure in replication; differences found. " class="issue-link" data-issue-key="LU-3573"><del>LU-3573</del></a>. So, it is safe for on-site exposure !</p> <p>Concerning how-to get the lustre debug-log upon LBUG, here my instructions to be applied on the Client node where the reproducer will run :</p> <p> _ as already requested, ensure /proc/sys/lnet/debug<span class="error">&#91;_mb&#93;</span> are respectively set to at least rpctrace+dlmtrace in addition to the default value (-1 would be the best!) for the trace-mask and to a reasonable value (2048 or even 4096) for the debug-buffer size.</p> <p> _ unset/0 /proc/sys/lnet/panic_on_lbug</p> <p> _ run reproducer/VTune to get to the LBUG</p> <p> _ the Lustre debug-log should be dumped automatically upon LBUG in a file with name/path /tmp/lustre-log.&lt;seconds-since-the-Epoch.milliseconds&gt;, but if not you can force this with "lctl dk &lt;path/file&gt;" command.</p> <p> _ if for any reason panic_on_lbug can not be unset ... I can also provide you with the information+stuff necessary to extract Lustre debug-log from a crash-dump!</p> <p> _ when you will have ensured the Lustre debug-log has been collected/saved, you will need to reboot the Client node to get Lustre functional again.</p> <p>Hello Bruno,</p> <p>What could be best option for the mounting for vtunes application. flock or localflock. what is the best practice for the vtunes application. Could you please check with Vtunes team</p> <p>VTune application use flock to protect database changes across multiple users. If you don't share results between different computers the options "localflock" will be enough. If you have concurrent access to the same result directory from different computers you definitely need "flock" option.</p> <p>Hello Bruno - Cu. is applying the patch this week, once I have an update, I will let you know. </p> <p>Thanks,<br/> Rajesh</p> <p>Hello Rajesh,<br/> Do you have any news/update from the site ?</p> <p>Hello Bruno,</p> <p>Cu have updated the patch and not seeing the issue, they are in the process of updating the remaining clients. </p> <p>Thanks,<br/> Rajesh</p> <p>Rajesh, how are we doing?<br/> Can we mark this as resolved?<br/> Thanks!<br/> ~ jfc.</p> <p>Hello John -</p> <p>Please go ahead and close this LU. </p> <p>Thanks for your help,<br/> Rajesh</p> <p>Thanks Rajesh</p> Development Rank 1|hzwj3r: Rank (Obsolete) 13382 Severity