https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1632 Lustre <p>On a 2.1 filesystem - server and client both running 2.1.2, and a filesystem created with Lustre 2.1.x, many files are created on the same client using the same FID sequence number. That's how I expect it to work.</p> <p>With servers and clients running 2.1.2, but with a filesystem that was originally created with Lustre 1.8, only one file is created per sequence number before the client requests another one from the MDS. For example, for several consecutive files created on the same client, I get FIDs like this:</p> <p><span class="error">&#91;0x33c0e251df8:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x33c0e254b37:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x33c0e257876:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x33c0e25a5b5:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x33c0e25d2f4:0x1:0x0&#93;</span><br/> <span class="error">&#91;0x33c0e260033:0x1:0x0&#93;</span></p> <p>lctl get_param 'seq.cli-srv-nbp*.* shows a space of <span class="error">&#91;0x0 - 0x0&#93;</span>:0:0 for the filesystems that were formatted under Lustre 1.8.</p> <p>Is this the way it's supposed to work?</p> <p>Thanks,<br/> Jason</p> Lustre 2.1.2 LU-1632

FID sequence numbers not working properly with filesystems formatted using 1.8?

Bug Blocker Resolved Fixed Di Wang jason.rappleye@nasa.gov fid Fri, 13 Jul 2012 19:22:13 +0000 Thu, 7 Nov 2013 22:22:24 +0000 Fri, 21 Dec 2012 13:38:00 +0000 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.1.2 Lustre 2.4.0 Lustre 2.1.4 0 6 <p>Definitely seems unusual, and not what should be happening.</p> <p>Hmm, definitely not right, once you are using 2.x client. Both client and server are little-endian nodes? Could you please collect some -1 debug log on the client side, when you create these file?</p> <p>files.txt: ls -l including fid for each file<br/> dk.seq.debug.gz: logs collected while creating each file, with all debug flags enabled</p> <p>Interesting - this only happens when touching a file, e.g.</p> <p>$ for i in </p> {1..100}; do touch foo-${i}; done<br/> <br/> If I write data to the file, it behaves as expected:<br/> <br/> $ for i in {1..100} <p>; do echo foo &gt; foo-${i}; done<br/> $ for i in $(ls foo*); do echo -n "$(ls -l $i) "; lfs path2fid $i; done</p> <p><del>rw-r</del><del>r</del>- 1 jrappley cstaff 4 Jul 13 18:04 foo-1 <span class="error">&#91;0x33c5f3bb04a:0x1:0x0&#93;</span><br/> <del>rw-r</del><del>r</del>- 1 jrappley cstaff 4 Jul 13 18:04 foo-10 <span class="error">&#91;0x33c5f3bb04a:0xa:0x0&#93;</span><br/> <del>rw-r</del><del>r</del>- 1 jrappley cstaff 4 Jul 13 18:04 foo-100 <span class="error">&#91;0x33c5f3bb04a:0x64:0x0&#93;</span><br/> <del>rw-r</del><del>r</del>- 1 jrappley cstaff 4 Jul 13 18:04 foo-11 <span class="error">&#91;0x33c5f3bb04a:0xb:0x0&#93;</span><br/> <del>rw-r</del><del>r</del>- 1 jrappley cstaff 4 Jul 13 18:04 foo-12 <span class="error">&#91;0x33c5f3bb04a:0xc:0x0&#93;</span><br/> <del>rw-r</del><del>r</del>- 1 jrappley cstaff 4 Jul 13 18:04 foo-13 <span class="error">&#91;0x33c5f3bb04a:0xd:0x0&#93;</span><br/> ...</p> <p>Still smells like a bug, though.</p> <p>Definitely shouldn't be happening this way. This causes many orders of magnitude (10^5) too many sequences to be allocated by the MDS. While it isn't fatal, it isn't what we expect and would cause some long-term overhead on the clients to have to fetch so many new FLDB entries.</p> <p>Hmm, it seem very strange. Did you do umount/mount between touch and echo test. Did you do that in the same client. It seems seq width of this client, which is exposed under /proc, was reset to 0 somehow. Could you please try this</p> <p>lctl get_param seq.*.width</p> <p>And post the result here. Thanks.</p> <p>No umount/mount, and the client didn't disconnect/reconnect to the MDS, either.</p> <p>Width is fine, but space isn't what I'd expect:</p> <p>$ lctl get_param 'seq.cli-srv-nbp*.*'seq.cli-srv-nbp6-MDT0000-mdc-ffff88040990cc00.fid=<span class="error">&#91;0x0:0x0:0x0&#93;</span><br/> seq.cli-srv-nbp6-MDT0000-mdc-ffff88040990cc00.server=nbp6-MDT0000_UUID<br/> seq.cli-srv-nbp6-MDT0000-mdc-ffff88040990cc00.space=<span class="error">&#91;0x0 - 0x0&#93;</span>:0:0<br/> seq.cli-srv-nbp6-MDT0000-mdc-ffff88040990cc00.width=131072</p> <p>Note that I can reproduce this on multiple filesystems and from different clients.</p> <p>We have six production filesystem; one was created with 2.1.x, the rest were 1.8 before upgrading to 2.1. We did not use the Xyratex migration patch.</p> <p>Since upgrading from 1.8 to 2.1.1, and now 2.1.2, we've had several incidents of high load average on our MDSes that are apparently due to a large number of SEQ_QUERY RPCs. They might be related to this issue. We see many mdss threads with this stack trace:</p> <p> <span class="error">&#91;&lt;ffffffffa0577a13&gt;&#93;</span> ? cfs_alloc+0x63/0x90 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffff815221f5&gt;&#93;</span> schedule_timeout+0x215/0x2e0<br/> <span class="error">&#91;&lt;ffffffffa079b6c4&gt;&#93;</span> ? sptlrpc_svc_alloc_rs+0x74/0x2d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa076d2b4&gt;&#93;</span> ? lustre_msg_add_version+0x94/0x110 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81523112&gt;&#93;</span> __down+0x72/0xb0<br/> <span class="error">&#91;&lt;ffffffff81095e11&gt;&#93;</span> down+0x41/0x50<br/> <span class="error">&#91;&lt;ffffffffa08a2531&gt;&#93;</span> seq_server_alloc_meta+0x41/0x720 <span class="error">&#91;fid&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa063c830&gt;&#93;</span> ? lustre_swab_lu_seq_range+0x0/0x30 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08a2fc8&gt;&#93;</span> seq_query+0x3b8/0x680 <span class="error">&#91;fid&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa076c004&gt;&#93;</span> ? lustre_msg_get_opc+0x94/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0bdfc65&gt;&#93;</span> mdt_handle_common+0x8d5/0x1810 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa076c004&gt;&#93;</span> ? lustre_msg_get_opc+0x94/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0be0c15&gt;&#93;</span> mdt_mdss_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span></p> <p>and one like this:</p> <p> <span class="error">&#91;&lt;ffffffff810902de&gt;&#93;</span> ? prepare_to_wait+0x4e/0x80<br/> <span class="error">&#91;&lt;ffffffffa0a79785&gt;&#93;</span> jbd2_log_wait_commit+0xc5/0x140 <span class="error">&#91;jbd2&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8108fff0&gt;&#93;</span> ? autoremove_wake_function+0x0/0x40<br/> <span class="error">&#91;&lt;ffffffffa0a79836&gt;&#93;</span> ? __jbd2_log_start_commit+0x36/0x40 <span class="error">&#91;jbd2&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0a71b4b&gt;&#93;</span> jbd2_journal_stop+0x2cb/0x320 <span class="error">&#91;jbd2&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0aca048&gt;&#93;</span> __ldiskfs_journal_stop+0x68/0xa0 <span class="error">&#91;ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c448f8&gt;&#93;</span> osd_trans_stop+0xb8/0x290 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08a3b06&gt;&#93;</span> ? seq_store_write+0xc6/0x2b0 <span class="error">&#91;fid&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08a3867&gt;&#93;</span> seq_store_trans_stop+0x57/0xe0 <span class="error">&#91;fid&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08a3d8c&gt;&#93;</span> seq_store_update+0x9c/0x1e0 <span class="error">&#91;fid&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08a299a&gt;&#93;</span> seq_server_alloc_meta+0x4aa/0x720 <span class="error">&#91;fid&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa063c830&gt;&#93;</span> ? lustre_swab_lu_seq_range+0x0/0x30 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08a2fc8&gt;&#93;</span> seq_query+0x3b8/0x680 <span class="error">&#91;fid&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa076c004&gt;&#93;</span> ? lustre_msg_get_opc+0x94/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0bdfc65&gt;&#93;</span> mdt_handle_common+0x8d5/0x1810 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa076c004&gt;&#93;</span> ? lustre_msg_get_opc+0x94/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0be0c15&gt;&#93;</span> mdt_mdss_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span></p> <p>I haven't looked into how the sequence allocation works on the server side, but my first guess is that we're bound by the time it takes to commit the latest (sequence number, width) to disk (in the OI)? Of course, if we had fewer SEQ_QUERY RPCs being issued by clients, there might not be a problem!</p> <p>I'm going to use SystemTap to see if I can understand what's going on. I'll report back what I find.</p> <p>Ah, I know where is the problem. You did not erase the config log when you upgrade 1.8 to 2.1 right? So the problem is </p> <p>void ll_delete_inode(struct inode *inode)<br/> {<br/> struct ll_sb_info *sbi = ll_i2sbi(inode);<br/> int rc;<br/> ENTRY;</p> <p> rc = obd_fid_delete(sbi-&gt;ll_md_exp, ll_inode2fid(inode));<br/> if (rc)<br/> CERROR("fid_delete() failed, rc %d\n", rc);</p> <p>it will call obd_fid_delete to delete the lmv object, but since you did not erase the config log, so it did not create the lmv layer at all. then it goes to mdc layer directly, and it will do sth obsolete there, which is completely wrong in 2.1 structure, I will cook a fix right now.</p> <p>My investigation with SystemTap so far has shown that every time seq_client_alloc_fid is called, the seq parameter is zeroed out. Some time after the call to seq_client_alloc_fid, mdc_fid_delete is called. It calls seq_client_flush, which zeros out the FID.</p> <p>On the filesystem that was formatted with Lustre 2.1, this call sequence does not happen.</p> <p>Btw: it seems pretty serious problem, since it will cause 1 extra RPC for every create.</p> <p>Yup, I see that in the `lctl dl` output - on the clients, the filesystem that's OK has an lmv device, while the rest don't.</p> <p>Will unmounting the MGS, removing the client config log, and remounting the MGS cause the client config log to be regenerated, presumably allowing new clients mounts to pick up the correct config log? Or do we need to do the usual `tunefs.lustre --writeconf` procedure? Though will only be slightly less intrusive for us than a client upgrade.</p> <p>We've definitely seen operational issues due to the extra RPCs - see my previous comment with stack traces. We've seen load averages of ~500 while all of the mdss threads are processing requests.</p> <p>Also, I don't see anything in the manual regarding deleting the config logs as part of the 1.8 -&gt; 2.x upgrade procedure.</p> <p>No, it should not need to erase the config log indeed. But this problem will only happen if you do not erase the log, so it is a bug. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/>. Probably only delete the client config log would not work right now. And you probably need tunefs.lustre --writeconf procedure here.</p> <p><a href="http://review.whamcloud.com/#change,3422" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3422</a> Here is the fix based on b2_1.</p> <p>Would it be sufficient to use this patch (after it's been reviewed, of course) to fix this problem, at least until we can take the filesystem down to regenerate the client config logs? Might there be other unintended consequences of not having the lmv layer in place?</p> <p>I ask because it's relatively easy to update the Lustre clients, versus taking each filesystem down.</p> <p>Jason, Di can answer authoritatively, but I believe the fix on the client should be enough to resolve the problem. The LMV layer is only needed on the client when DNE is enabled on the server. This means you have at least until 2.4 to regenerate the config.</p> <p>Yes, this patch, which is only on client side, should be enough to fix the problem. As Andreas said, LMV layer is only needed when you enable DNE on your system, which will not happen until 2.4.</p> <p>What is DNE?<br/> Also, on filesystems we did not erase the config log when we upgraded and thus no lmv device. However, we will still do not see the lmv device but it would work correctly with this client side patch. Is my understanding correct? Thanks!</p> <p>DNE means Distributed NamespacE, with which you can have multiple Metadata targets, <a href="http://wiki.whamcloud.com/display/PUB/Remote+Directories+Solution+Architecture" class="external-link" target="_blank" rel="nofollow noopener">http://wiki.whamcloud.com/display/PUB/Remote+Directories+Solution+Architecture</a>.</p> <p>Yes, if you apply this patch on client side, you do not need erase the config log. it will work correctly even there are no lmv device.</p> <p>I tried to forward port the 2.1 patch to 2.3, but it seems some of the routines involved got changed.</p> <p>Is 2.3 client still need the patch?</p> <p>yes, 2.3 client needs this patch. hmm, the patch seems only land on 2.1 right now.</p> <p>The patch has been landed to 2.1, and I will make a patch for 2.4 soon.</p> <p><a href="http://review.whamcloud.com/#change,4606" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4606</a> patch for current master.</p> <p>Landed to Master</p> Related LU-3318 Development Rank 1|hzv347: Rank (Obsolete) 4004 Severity