https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-39 Lustre <p>A sysadmin was shutting down an MDS node cleanly in preparation for scheduled upgrades. During the umount of the MGS device, we hit the following assertion:</p> <p>LustreError ... (lprocfs_status.c:1060:lprocfs_free_client_stats())<br/> ASSERTION(atomic_read(&amp;client_stat-&gt;nid_exp_ref_count) == 0) failed: count 1</p> <p>And the stack trace was:</p> <p>:obdclass:lprocfs_free_client_stats<br/> :obdclass:lprocfs_free_per_client_stats<br/> :mgs:lproc_mgs_cleanup<br/> :mgs:mgs_cleanup<br/> :obdclass:class_decref<br/> :obdclass:class_export_destroy<br/> :obdclass:obd_zombie_impexp_cull<br/> :obdclass:obd_zombie_impexp_thread</p> <p>We have seen this same assertion from OSTs as well. Some investigation was done in <a href="https://bugzilla.lustre.org/show_bug.cgi?id=23499" class="external-link" target="_blank" rel="nofollow noopener">bug 23499</a>, but there is not yet a solution.</p> lustre 1.8.3.0-6chaos LU-39

ASSERTION(atomic_read(&client_stat->nid_exp_ref_count) == 0) failed: count 1

Bug Minor Resolved Fixed Zhenyu Xu Christopher Morrone Tue, 11 Jan 2011 11:24:42 +0000 Fri, 25 May 2012 02:42:37 +0000 Wed, 23 Mar 2011 00:42:48 +0000 Lustre 1.8.6 Lustre 2.1.0 Lustre 1.8.6 0 4 <p>Bobijam is taking this bug. </p> <p>There is a race window between server obd cleanup and its handling<br/> of client connection.</p> <p>In class_cleanup() before the server obd's stopping flag is set, a new<br/> client connection request can be handled in target_handle_connect();<br/> after the target obd (server obd) is found, and before its refcount is<br/> increased, class_cleanup() can be processed and run through<br/> class_decref(); it find that the server obd's refcount becomes to 0;<br/> at this point, target_handle_connect() can go on and add target obd<br/> (server obd) refcount and create the client export.</p> <p>So the race makes a supposed 0 referenced server obd has export on it,<br/> thus the server obd's obd_nid_stats contains non 0 referenced client<br/> nid stat.</p> <p>patch is tracked at <a href="http://review.whamcloud.com/211" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/211</a></p> <p>As Niu mentioned in Gerrit review "The nid_stats_hash will be destroyed before the obd refcount drops to zero (see class_cleanup()), so the target_handle_reconnect() will not able to add any elements into the nid_stats_hash. I'm afraid this crash is caused by other defects. Maybe in the lprocfs_exp_setup(), it handles the (old_stat != new_stat) case incorrectly?"</p> <p>Yes, the patch I posted will solve another bug issue, which will not go this further to this assertion failure.</p> <p>I checked lprocfs_exp_setup() code, it handles the (old_stat != new_stat) case correctly, there should be other race place bring this extra nid_exp_ref_count at cleanup phase (in obd cleanup phase, obd_nid_stats_hash was destroyed in class_cleanup(), and mgs_cleanup() will not called until the obd's refcount becomes to 0 which means all client exports are dereferenced - class_export_destroy() decreases obd's refcount).</p> <p>Christopher,</p> <p>Do some of the clients use multip NIDs to connect servers?</p> <p>If a client is configed with multiple NIDs, when the connected NID encounters problem, it will reconnect server with another new NID, the lprocfs_exp_setup() misses releasing the old NID's stats refcount.</p> <p>The OSS nodes and the MDS have multiple NIDs, and the clients only have a single NID that they use to talk to the servers. Could this be caused by the MDS losing connection with the OSS nodes, and reconnecting using a different NID? The MDS would be similar to a client in this case.</p> <p>yes, that could be the case.</p> <p>bobi, I think we might want this patch for 2.x as well, I just made a quick look and found it's also in 2.x.</p> <p>Thanks</p> <p>HEAD version patch.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/reviews-centos5/529/" class="external-link" target="_blank" rel="nofollow noopener">reviews-centos5 #529</a><br/> <a href="https://jira.whamcloud.com/browse/LU-39" title="ASSERTION(atomic_read(&amp;client_stat-&gt;nid_exp_ref_count) == 0) failed: count 1" class="issue-link" data-issue-key="LU-39"><del>LU-39</del></a> ASSERTION(atomic_read(&amp;client_stat-&gt;nid_exp_ref_count) == 0)</p> <p>Bobi Jam : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-release.git&amp;a=commit&amp;h=8efdff9aeb1933f9b1e7320ff48ad84983e4daa3" class="external-link" target="_blank" rel="nofollow noopener">8efdff9aeb1933f9b1e7320ff48ad84983e4daa3</a><br/> Files : </p> <ul> <li>lustre/mgs/mgs_fs.c</li> <li>lustre/mdt/mdt_fs.c</li> <li>lustre/mdt/mdt_internal.h</li> <li>lustre/include/lprocfs_status.h</li> <li>lustre/mgs/mgs_handler.c</li> <li>lustre/obdfilter/filter.c</li> <li>lustre/obdclass/lprocfs_status.c</li> <li>lustre/mdt/mdt_handler.c</li> <li>lustre/mgs/mgs_internal.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master-centos5/159/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master-centos5 #159</a><br/> <a href="https://jira.whamcloud.com/browse/LU-39" title="ASSERTION(atomic_read(&amp;client_stat-&gt;nid_exp_ref_count) == 0) failed: count 1" class="issue-link" data-issue-key="LU-39"><del>LU-39</del></a> ASSERTION(atomic_read(&amp;client_stat-&gt;nid_exp_ref_count) == 0)</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-release.git&amp;a=commit&amp;h=2a6045403fbd46bb6501df907f0321f5401924ba" class="external-link" target="_blank" rel="nofollow noopener">2a6045403fbd46bb6501df907f0321f5401924ba</a><br/> Files : </p> <ul> <li>lustre/mdt/mdt_fs.c</li> <li>lustre/mgs/mgs_internal.h</li> <li>lustre/mdt/mdt_internal.h</li> <li>lustre/include/lprocfs_status.h</li> <li>lustre/mgs/mgs_fs.c</li> <li>lustre/mgs/mgs_handler.c</li> <li>lustre/obdfilter/filter.c</li> <li>lustre/obdclass/lprocfs_status.c</li> <li>lustre/mdt/mdt_handler.c</li> </ul> Bugzilla ID 23499.0 Development Rank 1|hzv9mf: Rank (Obsolete) 5093 Severity