https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7659 Lustre <p>The Kernel Userland Communication (KUC) subsystem is a lustre-specific API for something relatively common (deliver stream of records from kernel to userland, transmit feedback from userland to kernel). We propose to replace it by character devices.</p> <p>Besides being more standard, it can also increase performance significantly. A process can read large chunks from the character device. Our proof of concept shows a 5~10x speedup for reading changelogs by blocks of 4k.</p> <p>I would like feedback and suggestions. The proposed implementation works as follows:</p> <ul class="alternate" type="square"> <li>register a misc char device at mdc_setup (eg. /dev/changelog-lustre0-MDT0000). The minor number is associated to the corresponding OBD.</li> <li>The .open handler starts a thread in the background, that iterates over the llog and enqueues up to X records into a ring buffer</li> <li>The .read dequeues records from the ring buffer. We can make it blocking or not.</li> <li>.release stops the background thread and releases resources</li> <li>changelog clear is not yet implemented. It can either be a .write or a .unlocked_ioctl handler. Which would be preferable?</li> </ul> <p>The implementation for the copytool has not been done yet but would work in a similar way.</p> LU-7659

Replace KUC by more standard mechanisms

Improvement Minor Reopened Unresolved Yohan Pipereau Henri Doreau cea patch Wed, 13 Jan 2016 12:00:05 +0000 Wed, 15 Dec 2021 08:51:22 +0000 Lustre 2.10.0 Upstream 0 21 <p>Hello Henri,<br/> This looks really promising, and also much more fitting with the growing volume and required need of bandwidth for ChangeLogs, than KUC can offer now.<br/> Can you detail the need for the back-ground thread being started during open?</p> <p>Thanks Bruno,</p> <p>this thread makes records retrieval operations asynchronous and speeds up the whole thing. While the userland processes a batch of records, a new one gets retrieved in the background. Do you think it's overkill? <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I should add that kuc can deliver ~100k records per second in my benchmarks, though we have a robinhood setup able to consume 70k/s. This motived my work. It would also make it easier to read records from any language. And would look nicer.</p> <p>Some comments and questions:</p> <ul class="alternate" type="square"> <li>would this new mechanism be able to handle multiple ChangeLog consumers?</li> <li>my preference would be to use read and write for the interface, instead of ioctl, since this can be used even from scripts</li> <li>I would have suggested a /proc file instead of a char device, but new /proc files are frowned upon, and /sys files are only one value per file. The (minor) issue with a char device is the registration of the char major/minor, but it could use a misc char device?</li> <li>the .llseek() operation should allow seeking to a specific record, so that if there are multiple consumers and old records are not yet cancelled the new records can be found easily</li> <li>the char device should also have a .poll() method so that userspace can wait for new records efficiently instead of busy looping</li> </ul> <p>One issue that had come up with ChangeLogs in the past was that they are single-threaded in the kernel, which limits performance during metadata operations. If we are changing the API in userspace, it might also be good to change the on-disk format to allow multiple ChangeLog files to be written in parallel. Probably not one per core (that may become too many on large MDS nodes), but maybe 4-8 or so. The records could be merge sorted in the kernel by the helper thread at read time.</p> <p>Also, in theory the user-kernel interface could be changed without changing the userspace API, though this may be less desirable because of the licensing.</p> <p>Also note that it may be possible to just change the existing pipe interface to allow reading multiple records at once, instead of the current implementation that does two read() calls per record (one for the header and one for the body). It would be possible to read up to 64KB chunks from the pipe I think, and it could get as many full records as fit into the buffer and return a short read.</p> <p>The inotify API could be a good model for this, particularly providing a file descriptor that can be used with select or poll. </p> <p>Henri Doreau (henri.doreau@cea.fr) uploaded a new patch: <a href="http://review.whamcloud.com/18900" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18900</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7659" title="Replace KUC by more standard mechanisms" class="issue-link" data-issue-key="LU-7659">LU-7659</a> mdc: expose changelog through char devices<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d34ff747f04f30e37a6f13b970bbcaa6ffe9e813</p> <p>I was thinking that this might also be more useful to expose via <tt>.lustre/changelog/MDTxxxx</tt> rather than /dev/XXX so that it is easily accessed by applications when multiple filesystems are mounted on the same node.</p> <p>Also, very similar to this would be exposing (probably only on the server?) the virtual "all objects" iterator for each target under similar <tt>.lustre/iterator/MDTxxxx</tt> and <tt>.lustre/iterator/OSTxxxx</tt> virtual files (or similar). The MDTxxxx iterators are useful for listing all inodes in order so that they can efficiently be processed for initial RBH scans of all files. The OSTxxxx iterators might be useful for e.g. migrating objects off OSTs, replication of file data, and other operations that touch every object on an <em>online</em> OST, but could be implemented separately as needed. The caveat is that this would only be easily accessed if the OST is online, unless it was handled virtually by traversing the MDT layouts when the OST is offline which would not be nearly as efficient.</p> <p>As I explore netlink I wonder if the API could be used for this? In in my research I discovered it being used by the SCSI layer which surprised me.</p> <p>Quentin Bouget (quentin.bouget.ocre@cea.fr) uploaded a new patch: <a href="http://review.whamcloud.com/20327" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20327</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7659" title="Replace KUC by more standard mechanisms" class="issue-link" data-issue-key="LU-7659">LU-7659</a> mdc: expose hsm requests through char devices<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 52935cd7190a1b4d4b5def5a9244ce1e5ca60c3a</p> <p>Quentin Bouget (quentin.bouget.ocre@cea.fr) uploaded a new patch: <a href="http://review.whamcloud.com/20501" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20501</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7659" title="Replace KUC by more standard mechanisms" class="issue-link" data-issue-key="LU-7659">LU-7659</a> mdc: add an ioctl call to the copytool char device<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3d4f22eab430874560db611f9bd95fb31f63350f</p> <p>Quentin Bouget (quentin.bouget.ocre@cea.fr) uploaded a new patch: <a href="http://review.whamcloud.com/20502" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20502</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7659" title="Replace KUC by more standard mechanisms" class="issue-link" data-issue-key="LU-7659">LU-7659</a> mdc: revise copytool char device locking<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ebae9fe2b1fdf458638cabbde8a02fc1522ebb75</p> <p>Andreas, I realize that have not answered your questions, sorry for that, see below.</p> <blockquote> <p>would this new mechanism be able to handle multiple ChangeLog consumers?</p></blockquote> <p>Yes, multiple processes can open the char device. By default they start reading from the beginning of the llog and they can lseek to wherever they want in the log to start at a given record. Very similar to the existing implementation in this sense.</p> <blockquote> <p>my preference would be to use read and write for the interface, instead of ioctl, since this can be used even from scripts</p></blockquote> <p>Done.</p> <blockquote> <p>I would have suggested a /proc file instead of a char device, but new /proc files are frowned upon, and /sys files are only one value per file. The (minor) issue with a char device is the registration of the char major/minor, but it could use a misc char device?</p></blockquote> <p>It is a misc char device.</p> <blockquote> <p>the .llseek() operation should allow seeking to a specific record, so that if there are multiple consumers and old records are not yet cancelled the new records can be found easily</p></blockquote> <p>Done, using the record number as the offset to jump to.</p> <blockquote> <p>the char device should also have a .poll() method so that userspace can wait for new records efficiently instead of busy looping</p></blockquote> <p>Done.</p> <blockquote> <p>One issue that had come up with ChangeLogs in the past was that they are single-threaded in the kernel, which limits performance during metadata operations. If we are changing the API in userspace, it might also be good to change the on-disk format to allow multiple ChangeLog files to be written in parallel. Probably not one per core (that may become too many on large MDS nodes), but maybe 4-8 or so. The records could be merge sorted in the kernel by the helper thread at read time.</p></blockquote> <p>I'd love that. It is beyond the scope of this patch I'd' say, but I keep it in mind. Maybe indexes instead of llog catalogs?</p> <p>Yes, we've discussed changing llogs over to use an index instead of a flat file. The benefit of the llog file is that it can be written mostly sequentially, and record cancellation only needs to update the bitmap in the header. The drawback is that updating the header is serialized, reserving space in the llog file is difficult if the record size is unknown, and there is added complexity the order of the log records does not match the order that transactions are completed.</p> <p>On a related note, did you look into connecting the LFSCK iterator to the new char interface to speed up the initial scanning for RBH?</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/18900/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/18900/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7659" title="Replace KUC by more standard mechanisms" class="issue-link" data-issue-key="LU-7659">LU-7659</a> mdc: expose changelog through char devices<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1d40214d96dd6e36bd39a35f8419f753bae8d305</p> <p>Landed for 2.10</p> <p>Yohan Pipereau (yohan.pipereau.ocre@cea.fr) uploaded a new patch: <a href="https://review.whamcloud.com/32941" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32941</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7659" title="Replace KUC by more standard mechanisms" class="issue-link" data-issue-key="LU-7659">LU-7659</a> libcfs: Use netlink for KUC communication<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 31a839a7e846b7e53c0e846452435fffe83a0585</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/34258" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34258</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7659" title="Replace KUC by more standard mechanisms" class="issue-link" data-issue-key="LU-7659">LU-7659</a> hsm: Use netlink for KUC communication<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e3e1c2f7fa81dd149308b41c74ad190e32c858ae</p> Related LU-15373 LU-10968 LU-12506 LU-11626 LU-9680 LU-10141 Development End date Mon, 30 May 2016 12:00:05 +0000 Rank 1|hzxy27: Rank (Obsolete) 9223372036854775807 Start date Wed, 13 Jan 2016 12:00:05 +0000