https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1248 Lustre <p>The load average on the MDS for a classified production 2.1 filesystem jumped to over 400. Top showed mdt_rdpg_* threads all using 4-7% CPU time. This may have been due to a pathological workload, but we were wondering if there's something like an overly contended lock in ldiskfs going on here. </p> <p>Most of the stacks looked like this:</p> <p>__cond_resched <br/> _cond_resched <br/> ifind_fast <br/> iget_locked <br/> ldiskfs_iget <br/> ? generic_detach_inode <br/> osd_iget <br/> osd_ea_fid_get <br/> osd_it_ea_rec <br/> mdd_readpage <br/> cml_readpage <br/> mdt_readpage <br/> ? mdt_unpack_req_pack_rep <br/> mdt_handle_common <br/> ? lustre_msg_get_transno <br/> mdt_readpage_handle <br/> ptlrpc_main <br/> child_rip</p> <a href="https://github.com/chaos/lustre/tree/2.1.0-llnl">https://github.com/chaos/lustre/tree/2.1.0-llnl</a> LU-1248

all mdt_rdpg_* threads busy in osd_ea_fid_get()

Bug Minor Resolved Fixed Lai Siyao Ned Bass Wed, 21 Mar 2012 17:16:55 +0000 Sat, 1 Feb 2014 08:35:15 +0000 Tue, 26 Jun 2012 10:03:13 +0000 Lustre 2.1.0 0 8 <p>Lsi</p> <p>Could you please comment on this one?</p> <p>Thanks</p> <p>Peter</p> <p>This looks normal from the code. Basically it's an inode scalability problem, these busy threads are contending on inode_lock, and NIck Piggin's inode lock scalability patches are getting merged into kernel 3.x.</p> <p>Currently MDT still uses directory+ea to store metadata, while IAM looks to have better performance and scalability, but I'm not clear why it's not enabled yet.</p> <p>Ned, is the system upgraded from 1.8? Normally readdir can get fid from dir data, and then it doesn't need read fid from ea, but for a upgraded system, it needs query each inode. Besides, did you see any error messages related with this dir?</p> <p>Yes the system was upgraded from 1.8. Will files created after the upgrade store the fid in the dir data?</p> <p>There are a few "osd_object_delete() Failed to cleanup: -2" console messages on the MDS from around that time. I don't find any other errors worth mentioning.</p> <p>Yes, on upgraded system even newly created dir won't store fid in dir data; I'll try whether it's easy to implement this.</p> <p>I don't find a easy way to implement: without change for disk format, there's no way to distinguish 1.8 directory and newly created directory. The original design for 1.8 &lt;-&gt; 2.x interoperatability server is on <a href="https://bugzilla.lustre.org/show_bug.cgi?id=11826" class="external-link" target="_blank" rel="nofollow noopener">bz11826</a>.</p> <p>There are incompatible format between b1_8 and b2_1: in b1_8, lvfs_dentry_params is appended after the name entry in parent directory; but in b2_1, it is ldiskfs_dentry_param. They are different and incompatible. So when system upgraded from b1_8 to b2_1, the new created fie cannot append ldiskfs_dentry_param (which contains the FID) after its name entry in parent directory, otherwise, the system cannot downgrade back to b1_8.</p> <p>But if without FID appended after name entry in parent directory, it will cause performance (for dir readpage) regression. I do not think it is good solution, because upgrade is more often used than downgrade.</p> <p>We should make some patch in b2_x to support appending FID after the name entry in parent directory for upgrading case and skip lvfs_dentry_params after the name entry for old files. On the other hand, need another patch against b1_8_x (x &gt;= 8) to skip FID after the name entry in parent directory to support downgrade back to b1_8_x (x &gt;= 8).</p> <p>Andreas, it looks like we need change both 2.x and 1.8 ldiskfs code to keep both backward and forward compatibility for this, any suggestion for this?</p> <p>The need to be able to downgrade from 2.x to 1.8 is only in the case of "simple" upgrade to 2.x that has hit problems and needs to be able to downgrade. If the upgrade has been successful, and then the admin (separately) enables the "dir_data" feature using tune2fs on the filesystem, this should be enough to allow storing FIDs in the directory entries. After that point, the filesystem should not be downgraded to 1.8 anymore.</p> <p>What definitely <em>should</em> be avoided is any automatic enabling of the "dir_data" feature on the filesystem when it is first mounted, since this will cause problems if there are FIDs stored in the directory entries, then the filesystem is downgraded to 1.8, the FID-in-LMA is deleted upon access (reverting to IGIF for that inode), and then the filesystem is upgraded again. That would cause the FID-in-dirent to contain invalid data that OI scrub and e2fsck will not fix yet.</p> <p>So, my understanding is that if you are sure there is no need to downgrade to 1.8, it should be possible with 2.1+ to use:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>tune2fs -O dirdata /dev/{mdtdev} </pre> </div></div> <p>to enable this feature, and then newly-created files/links will store the FID in the directory. I don't know if we have tested this process or not.</p> <p>Assuming this is OK, it would then be possible in that case to "refresh" the directory with a script to re-link filenames that are expected to live for a long time, assuming they are not in use, something like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs find /mnt/lustre/some/dir -type f | while read F; do FTMP="$F.XXXXXX.$$.$RANDOM" mv "$F" "$FTMP" &amp;&amp; mv "$FTMP" "$F" done </pre> </div></div> <p>In a later phase of LFSCK, the FID-in-dirent data will be verified and refreshed if missing, but this is not part of the Phase I deliverable.</p> <p>This existing behaviour is not a net performance loss in many use cases, since it is prefetching the inode into MDS memory for use as soon as e.g. "ls" does a stat() on the file. There would only be a visible slowdown in the case of e.g. "find" that is not accessing any of the file attributes, and only generating pathnames.</p> <p>Andreas, thanks for your detailed explanation! I'll verify `tune2fs` to enable 'dir_data' feature later.</p> <p>Ned, are you fine with the result?</p> <p>The command should be `tune2fs -O dirdata /dev/</p> {mdtdev} <p>`, and I've verified that newly created dir will store FID in it.</p> <p>Lai, enabling dir_data seems like a reasonable course of action. We'll start some local testing and propose it to our sysadmin team. Thanks</p> <p>If 1.8 system is upgraded to 2.x successfully, tunefs can be used to enable dirdata feature, then new directory will contain inode FID in its data.</p> <p>The Lustre Manual should be updated to inform users about how to enable "dirdata" on an upgraded 1.8-&gt;2.x MDT, once they are sure that they will not be downgrading the MDS to 1.8 again. This will minimize performance impact on newly created files.</p> <p>Cliff could you please create an LUDOC ticket to track Andreas's request?</p> <p><a href="http://jira.whamcloud.com/browse/LUDOC-68" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LUDOC-68</a> has been created to track the manual changes</p> Development Rank 1|hzvh5j: Rank (Obsolete) 6428 Severity