Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-3684] LBUG/"ldlm_lock_decref_internal_nolock()) ASSERTION(lock->l_readers > 0) failed" running Bull's NFS locktests https://jira.whamcloud.com/browse/LU-3684 Lustre <p>We need to fix the bug reported here: <a href="https://jira.hpdd.intel.com/browse/LU-1126" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-1126</a> before installing Sas Grid Manager.</p> <p>The Lustre filesystem is mounted on client using the -o flock option</p> Client Lustre 1.8.7 jenkins-wc1--PRISTINE-2.6.18-274.3.1.el5 RHEL 5.7<br/> Server Lustre 1.8.7 jenkins-wc1--PRISTINE-2.6.18-274.3.1.el5_lustre.g9500ebf RHEL 5.7 LU-3684

LBUG/"ldlm_lock_decref_internal_nolock()) ASSERTION(lock->l_readers > 0) failed" running Bull's NFS locktests

Task Major Resolved Fixed Bruno Faccini Supporto Lustre Jnet2000 Thu, 1 Aug 2013 14:53:31 +0000 Wed, 16 Apr 2014 09:42:40 +0000 Mon, 3 Mar 2014 12:01:59 +0000 Lustre 1.8.7 0 4 <p>Thanks for the report. We are looking into the best option</p> <p>Hello, <br/> I wonder if you can better detail the issue you experience by running SAS Grid Manager on top of Lustre 1.8 ?? This could help me to better qualify this ticket and also to change its title accordingly!</p> <p>Since your feeling is that your problem is still the one originally addressed by <a href="https://jira.whamcloud.com/browse/LU-1126" title="Client file locking issue. Assertion triggered when decrementing a read lock on an item that has no existing read locks." class="issue-link" data-issue-key="LU-1126"><del>LU-1126</del></a> and that you may already be aware of my last work/update in it, I would like you to help me determine if it is still the original one described in <a href="https://jira.whamcloud.com/browse/LU-1126" title="Client file locking issue. Assertion triggered when decrementing a read lock on an item that has no existing read locks." class="issue-link" data-issue-key="LU-1126"><del>LU-1126</del></a> (wrong lock mode used versus readers/writers counters) and likely to be reproduced by flock.c provided program, or the one I reported to be still present in master (race during lock destroy when overlap detection) and easily reproducible with "BULL's NFS Locktests".</p> <p>Do you think you can again provide the Lustre debug-log (The way Oleg described in <a href="https://jira.whamcloud.com/browse/LU-1126" title="Client file locking issue. Assertion triggered when decrementing a read lock on an item that has no existing read locks." class="issue-link" data-issue-key="LU-1126"><del>LU-1126</del></a> would be the best since it is for a full-trace, but at least with "dlmtrace" enabled) of a new occurence ??</p> <p>Thanks in advance for your help.</p> <p>Dear Bruno, currently we don't have SAS Grid Manager installed on our system. Before installing it, sas support team require that the bug reported here <a href="https://jira.hpdd.intel.com/browse/LU-1126" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-1126</a> must to be fixed.<br/> Instead, we have surely the bug that can be reproduced with "BULL's NFS Locktests".<br/> If you wish, we can provide the lustre log of the BULL test run.<br/> Regards</p> <p>That would be nice if I can get the Lustre debug-log taken during "BULL's NFS Locktests" run at your site! Thanks in advance.</p> <p>Dear Bruno, we have attached the log of lustre client crashed during the execution of the BULL test. <br/> Regards</p> <p>I checked the lustre-log you provided and it is definitelly the same problem triggered by "BULL's NFS Locktests" (and not the original one in <a href="https://jira.whamcloud.com/browse/LU-1126" title="Client file locking issue. Assertion triggered when decrementing a read lock on an item that has no existing read locks." class="issue-link" data-issue-key="LU-1126"><del>LU-1126</del></a> with custom reproducer) that I fixed with change <a href="http://review.whamcloud.com/7134" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7134</a> in master, like already described in <a href="https://jira.whamcloud.com/browse/LU-1126" title="Client file locking issue. Assertion triggered when decrementing a read lock on an item that has no existing read locks." class="issue-link" data-issue-key="LU-1126"><del>LU-1126</del></a>.</p> <p>Since this ticket definitely addresses a different scenario (even if LBUG/"ldlm_lock_decref_internal_nolock()) ASSERTION(lock->l_readers > 0) failed" is the same!!) than the original reported as part of <a href="https://jira.whamcloud.com/browse/LU-1126" title="Client file locking issue. Assertion triggered when decrementing a read lock on an item that has no existing read locks." class="issue-link" data-issue-key="LU-1126"><del>LU-1126</del></a> but where it was also/already described, I would like to set this ticket as the main/tracking one for this particular case.</p> <p>Just to be complete about the differences between the 2 problems for <a href="https://jira.whamcloud.com/browse/LU-1126" title="Client file locking issue. Assertion triggered when decrementing a read lock on an item that has no existing read locks." class="issue-link" data-issue-key="LU-1126"><del>LU-1126</del></a> and this ticket :</p> <p> _ <a href="https://jira.whamcloud.com/browse/LU-1126" title="Client file locking issue. Assertion triggered when decrementing a read lock on an item that has no existing read locks." class="issue-link" data-issue-key="LU-1126"><del>LU-1126</del></a>, has been open for a particular race/scenario where a transient lock, result/return for a F_GETLK request, has to be destroyed due to overlap, but this occurs during the very short window where its changed mode (PR/PW) has become incompatible with its counter (l_readers/l_writers). Thus the LBUG occurs because the wrong counter will be decremented. This particular problem only show up with the custom reproducer (flock.c) provided for <a href="https://jira.whamcloud.com/browse/LU-1126" title="Client file locking issue. Assertion triggered when decrementing a read lock on an item that has no existing read locks." class="issue-link" data-issue-key="LU-1126"><del>LU-1126</del></a>.</p> <p> _ this is different to the scenario for this ticket's problem, where a race can occur between 2 threads who want to destroy (one to finish corresponding request processing, the other due to overlap rules) the same lock, mainly during F_UNLCK multiple/concurent requests handling. Thus the LBUG occurs because 2nd thread found counter already set to 0. This particular problem show up very easily when running, as you experienced, "Bull's NFS Locktests". This test is available at <a href="http://nfsv4.bullopensource.org/tools/tests/locktest.php" class="external-link" target="_blank" rel="nofollow noopener">http://nfsv4.bullopensource.org/tools/tests/locktest.php</a>, and provided as "locktests.tar.gz" distro I attached here. Easy way to reproduce is to run in pthread mode like "locktests -n 10 -T -f <Lustre-File>" on a single+full Lustre node (ie, like after intalling Lustre and running "llmount.sh").</p> <p>As I said, problem has been fixed in master with <a href="http://review.whamcloud.com/7134" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7134</a>, b1_8 patch is at <a href="http://review.whamcloud.com/7420" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7420</a> now.</p> <p>Also, I would like to change this ticket's title as it is definitely not the same problem/race than the one addressed in <a href="https://jira.whamcloud.com/browse/LU-1126" title="Client file locking issue. Assertion triggered when decrementing a read lock on an item that has no existing read locks." class="issue-link" data-issue-key="LU-1126"><del>LU-1126</del></a>, something like "<span class="error">[ldlm_lock_decref_internal_nolock()) ASSERTION(lock->l_readers > 0) failed]</span> running Bull's NFS locktests".</p> <p>Will also add reference to this ticket in <a href="https://jira.whamcloud.com/browse/LU-1126" title="Client file locking issue. Assertion triggered when decrementing a read lock on an item that has no existing read locks." class="issue-link" data-issue-key="LU-1126"><del>LU-1126</del></a> to complete split-up between the 2 different problems.</p> <p>The customer is currently in the process to upgrade to 1.8.9+patch. So please close this ticket.</p> <p>b2_4 patch version is at <a href="http://review.whamcloud.com/9968" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9968</a>.</p> Development Rank 1|hzvx0f: Rank (Obsolete) 9511