https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3974 Lustre <p>Tracker for 3.11 kernel support</p> <p>As fc19 just recently updated their kernel from 3.10 to 3.11 we will need to support the 3.11 kernel soon.</p> LU-3974

Support for linux 3.11 kernel

Improvement Minor Resolved Fixed Bob Glossman Bob Glossman Thu, 19 Sep 2013 20:43:17 +0000 Wed, 27 Jan 2016 23:02:44 +0000 Sun, 2 Mar 2014 14:18:14 +0000 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.6.0 0 12 <p>Already found at least one incompatibility with current lustre. num_physpages, a global kernel export used by lustre code, no longer exists. I'm thinking we will need some autoconf change(s) to adapt to this fact.</p> <p>num_physpages can be replaced by inline function get_num_physpages()</p> <p>address_space_operations.invalidatepage now take 3 args instead of 2. This will force mods in ll_invalidatepage, truncate_complete_page, maybe ldiskfs, maybe elsewhere I haven't identified yet.</p> <p>Last time I tried this also d_compare api changed as well which affects some of the llite code. </p> <p>I see a couple of recent patches that do part of the job of getting client builds working again:</p> <p><a href="http://review.whamcloud.com/#/c/7726" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7726</a><br/> <a href="http://review.whamcloud.com/#/c/7729" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7729</a></p> <p>These are in addition to the set of patches listed in <a href="https://jira.whamcloud.com/browse/LU-3319" title="Adapt to 3.10 upstream kernel proc_dir_entry change" class="issue-link" data-issue-key="LU-3319"><del>LU-3319</del></a>.</p> <p>Yep. More coming. I just need to do some more testing to make sure they work every where.</p> <p>For inital 3.11 client support you need the following:</p> <p>All the patches from <a href="https://jira.whamcloud.com/browse/LU-3319" title="Adapt to 3.10 upstream kernel proc_dir_entry change" class="issue-link" data-issue-key="LU-3319"><del>LU-3319</del></a> - <a href="http://review.whamcloud.com/#/q/status:open+message:LU-3319,n,z" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/q/status:open+message:LU-3319,n,z</a><br/> (One of the patches will need to fixed up)</p> <p><a href="https://jira.whamcloud.com/browse/LU-2800" title="build: clean out old autoconf options" class="issue-link" data-issue-key="LU-2800"><del>LU-2800</del></a> - <a href="http://review.whamcloud.com/#/c/5380" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/5380</a></p> <p>That will give you basic 3.10 support. The rest needed to be applied in the below order</p> <p><a href="http://review.whamcloud.com/#/c/7741" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7741</a><br/> <a href="http://review.whamcloud.com/#/c/7746" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7746</a><br/> <a href="http://review.whamcloud.com/#/c/7747" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7747</a><br/> <a href="http://review.whamcloud.com/#/c/7729" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7729</a></p> <p>Changes affecting NFS happened so that needs to be tested for possible regressions.</p> <p>I think you left out <a href="http://review.whamcloud.com/#/c/7726" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7726</a></p> <p>I've verified that with the new additional patches client builds work again for fc19 with the 3.11 kernel.</p> <p>Any ETA for server builds?</p> <p>Hm. I seeing problems with the infiniband stack. Are you?</p> <p>Lustre: Lustre: Build Version: 2.4.92-gb1bc221-CHANGED-3.11.1<br/> ko2iblnd: no symbol version for ib_fmr_pool_unmap<br/> ko2iblnd: Unknown symbol ib_fmr_pool_unmap (err -22)<br/> ko2iblnd: no symbol version for ib_create_cq<br/> ko2iblnd: Unknown symbol ib_create_cq (err -22)<br/> ko2iblnd: no symbol version for rdma_resolve_addr<br/> ko2iblnd: Unknown symbol rdma_resolve_addr (err -22)<br/> ko2iblnd: no symbol version for ib_reg_phys_mr<br/> ko2iblnd: Unknown symbol ib_reg_phys_mr (err -22)<br/> ko2iblnd: no symbol version for ib_create_fmr_pool<br/> ko2iblnd: Unknown symbol ib_create_fmr_pool (err -22)<br/> ko2iblnd: no symbol version for ib_flush_fmr_pool<br/> ko2iblnd: Unknown symbol ib_flush_fmr_pool (err -22)</p> <p>As for the server support. That will be awhile still. I'm half way done.</p> <p>So far have only done building, haven't actually tried an install so I wouldn't have seen missing symbols. Will try that in a while.</p> <p>I think all those symbols should be in infiniband .ko loadable modules. You didn't maybe config your kernel without infiniband or try to force load ko2iblnd lnet module without having inifiniband modules loaded, did you?</p> <p><span class="error">&#91;root@ninja05 infiniband&#93;</span># lsmod | grep mlx<br/> mlx4_ib 104462 0 <br/> ib_sa 26820 6 ib_srp,ib_ipoib,rdma_ucm,rdma_cm,ib_cm,mlx4_ib<br/> ib_mad 36559 4 ib_umad,ib_cm,mlx4_ib,ib_sa<br/> ib_core 52348 12 ib_srp,ib_ipoib,rdma_ucm,ib_ucm,ib_uverbs,ib_umad,rdma_cm,ib_cm,iw_cm,mlx4_ib,ib_sa,ib_mad<br/> mlx4_en 72499 0 <br/> mlx4_core 165690 2 mlx4_ib,mlx4_en</p> <p>I haven't been able to reproduce your failure with IB symbols. I'm running in VMs without any IB, but if I force load of ko2iblnd with 'modprobe ko2iblnd' I don't seen any Unknown symbol errors and I do see the following set of previously not loaded kernel modules getting loaded:</p> <p>rdma_cm 42363 1 ko2iblnd<br/> ib_addr 13883 1 rdma_cm<br/> iw_cm 18153 1 rdma_cm<br/> ib_cm 41722 1 rdma_cm<br/> ib_sa 29038 2 rdma_cm,ib_cm<br/> ib_mad 42262 2 ib_cm,ib_sa<br/> ib_core 74075 6 rdma_cm,ib_cm,ib_sa,iw_cm,ko2iblnd,ib_mad</p> <p>That's the good news. On the bad news side I can't get the fc19 client to actually function. It builds fine, appears to install fine. I can do a mount of a lustre file system on Centos servers that works fine in Centos and SuSE clients, but any access fails. example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fedora19 x86_64]# mount -t lustre centos2:/lustre /mnt/lustre [root@fedora19 x86_64]# ls -l /mnt/lustre ls: reading directory /mnt/lustre: Input/output error total 0 </pre> </div></div> <p>I'm still trying to debug what it's unhappy about.<br/> At least it doesn't crash.</p> <p>It seems the failures I'm seeing only happen during directory access. I can actually create some files, put data in them, read the data back all from the fc19 client. Only see errors during ls (so far).</p> <p>Pretty sure the directories and inodes on disk are really OK. All look fine viewed from other clients.</p> <p>Have you tried the lustre client that comes with 3.11. I'm curious to see if that works.</p> <p>I did make some attempt to build it. Needs some non-obvious manual edits to Kconfig files just to get it to try to build. Doesn't build in its current form, at least for me. Strongly suspect it need one or more of the patches in this ticket or the 3.10 kernel ticket or both.</p> <p>Bob,<br/> If you haven't already, please file a bug with your steps and problems for compiling the in-kernel client, and CC Tao Peng and me on it, and link it to this one. It makes no sense that we spend time to get code into the upstream kernel that doesn't even build. I thought Tao has been building that code?</p> <p>We can also use that ticket (or this one) to track the landing of patches into master that are getting applied to the upstream kernel (where that makes sense). I know there are a number of good cleanups going into the Lustre client in the kernel, but there are also some code deletions that we either need to rework in order to get something acceptable to upstream, or keep them in the out-of-tree client for our own needs (e.g. watchdogs and stack tracing on servers, etc). </p> <p>About the problems with directory access. Each attempt to do ls /mnt/lustre generates some errors that look like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 1 09:33:45 fedora19 kernel: [174400.937657] LustreError: 24568:0:(dir.c:398:ll_get_dir_page()) dir page locate: [0x200000007:0x1:0x0] at 18446744073709551614: rc -5 Oct 1 09:33:45 fedora19 kernel: [174400.937661] LustreError: 24568:0:(dir.c:595:ll_dir_read()) error reading dir [0x200000007:0x1:0x0] at 18446744073709551614: rc -5 </pre> </div></div> <p>Attempting to mkdir in /mnt/lustre just hangs.<br/> Not getting closer to the underlying cause(s) very quickly.</p> <p>Current build recipe for clients is:</p> <p><a href="http://review.whamcloud.com/#/c/5512" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/5512</a> # <a href="https://jira.whamcloud.com/browse/LU-2158" title="remove lvfs and fsfilt code" class="issue-link" data-issue-key="LU-2158"><del>LU-2158</del></a></p> <p><a href="http://review.whamcloud.com/#/c/7135" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7135</a> # <a href="https://jira.whamcloud.com/browse/LU-3319" title="Adapt to 3.10 upstream kernel proc_dir_entry change" class="issue-link" data-issue-key="LU-3319"><del>LU-3319</del></a><br/> <a href="http://review.whamcloud.com/#/c/7139" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7139</a><br/> <a href="http://review.whamcloud.com/#/c/7140" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7140</a><br/> <a href="http://review.whamcloud.com/#/c/7290" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7290</a><br/> <a href="http://review.whamcloud.com/#/c/7291" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7291</a><br/> <a href="http://review.whamcloud.com/#/c/7292" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7292</a><br/> <a href="http://review.whamcloud.com/#/c/7293" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7293</a><br/> <a href="http://review.whamcloud.com/#/c/7294" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7294</a><br/> <a href="http://review.whamcloud.com/#/c/7295" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7295</a><br/> <a href="http://review.whamcloud.com/#/c/7296" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7296</a></p> <p><a href="http://review.whamcloud.com/#/c/5380" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/5380</a> # <a href="https://jira.whamcloud.com/browse/LU-2800" title="build: clean out old autoconf options" class="issue-link" data-issue-key="LU-2800"><del>LU-2800</del></a></p> <p><a href="http://review.whamcloud.com/#/c/7741" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7741</a> # <a href="https://jira.whamcloud.com/browse/LU-3974" title="Support for linux 3.11 kernel" class="issue-link" data-issue-key="LU-3974"><del>LU-3974</del></a><br/> <a href="http://review.whamcloud.com/#/c/7746" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7746</a><br/> <a href="http://review.whamcloud.com/#/c/7747" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7747</a><br/> <a href="http://review.whamcloud.com/#/c/7729" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7729</a></p> <p>A few of these need refresh. In particular <a href="http://review.whamcloud.com/#/c/5512" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/5512</a> adds a couple more instances of d_refcount() that aren't yet fixed in <a href="http://review.whamcloud.com/#/c/7741" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7741</a>.</p> <p>No solution yet about functional problems.</p> <p>In an attempt to narrow down the source of the runtime problems with the 3.11 client code I applied and built the same recipe (- the patches from <a href="https://jira.whamcloud.com/browse/LU-3974" title="Support for linux 3.11 kernel" class="issue-link" data-issue-key="LU-3974"><del>LU-3974</del></a>) onto the 3.10 kernel in fc18. That seems to work fine. Suggests to me that the problem(s) are in the additional set of 3.11 specific patches, the 3.11 upstream kernel, or both.</p> <p>Retried the same recipe using the recent refresh of the <a href="https://jira.whamcloud.com/browse/LU-3319" title="Adapt to 3.10 upstream kernel proc_dir_entry change" class="issue-link" data-issue-key="LU-3319"><del>LU-3319</del></a> patches on both fc18 (3.10.13) and fc19 (3.11.2). Still builds in both, but works nicely on 3.10 and not so nicely on 3.11.</p> <p>Bob, I didn't see the ll_read_dir() error message in upstream kernel client but I did see mkdir hung. It is because Lustre always saves the first page of a dir inode mapping at index ~0UL. And after 3.11 commit 5a72039 (mm: teach truncate_inode_pages_range() to handle non page aligned ranges), truncate_inode_pages_range() <em>NO LONGER</em> truncates the page that is sitting at index ~0UL.</p> <p>My patch to fix it is at:<br/> <a href="https://github.com/bergwolf/linux/commit/28fde12d26d7d4b54cb4689dd6117c2ed791f985" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/bergwolf/linux/commit/28fde12d26d7d4b54cb4689dd6117c2ed791f985</a></p> <p>It is not merge in upstream kernel yet because the problem was not there when I tested the code in staging tree before 3.11.</p> <p>Is that mod to lustre_lite.h safe in any lustre version or only good in upstream lustre on 3.11 that has the commit 5a72039 (mm: teach truncate_inode_pages_range() to handle non page aligned ranges) you mention above?</p> <p>It is safe to any Lustre version.</p> <p>Tried out the patch in <a href="https://github.com/bergwolf/linux/commit/28fde12d26d7d4b54cb4689dd6117c2ed791f985" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/bergwolf/linux/commit/28fde12d26d7d4b54cb4689dd6117c2ed791f985</a>. Does solve the problem of mkdir hang. No impact on the other directory access issues. Still see errors like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> # ll -aR /mnt/lustre /mnt/lustre: ls: reading directory /mnt/lustre: Input/output error total 20 drwxr-xr-x 4 root root 4096 Oct 15 09:37 . drwxr-xr-x. 5 root root 4096 Sep 24 12:20 .. drwxr-xr-x 2 root root 4096 Oct 15 09:37 dtest drwxr-xr-x 3 root root 4096 Aug 29 14:05 .lustre -rw-r--r-- 1 root root 4 Sep 25 09:10 xxx /mnt/lustre/dtest: ls: reading directory /mnt/lustre/dtest: Input/output error total 8 drwxr-xr-x 2 root root 4096 Oct 15 09:37 . drwxr-xr-x 4 root root 4096 Oct 15 09:37 .. /mnt/lustre/.lustre: ls: reading directory /mnt/lustre/.lustre: Input/output error total 12 drwxr-xr-x 3 root root 4096 Aug 29 14:05 . drwxr-xr-x 4 root root 4096 Oct 15 09:37 .. d--x------ 2 root root 4096 Aug 29 14:05 fid /mnt/lustre/.lustre/fid: ls: reading directory /mnt/lustre/.lustre/fid: Input/output error total 8 d--x------ 2 root root 4096 Aug 29 14:05 . drwxr-xr-x 3 root root 4096 Aug 29 14:05 .. </pre> </div></div> <p>coupled with log errors like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> Oct 15 09:37:04 fedora19 kernel: [ 204.735226] LustreError: 2647:0:(dir.c:422:ll_get_dir_page()) read cache page: [0x200000007:0x1:0x0] at 18446744073709551614: rc -5 Oct 15 09:37:04 fedora19 kernel: [ 204.735231] LustreError: 2647:0:(dir.c:422:ll_get_dir_page()) Skipped 2 previous similar messages Oct 15 09:37:04 fedora19 kernel: [ 204.735235] LustreError: 2647:0:(dir.c:595:ll_dir_read()) error reading dir [0x200000007:0x1:0x0] at 18446744073709551614: rc -5 Oct 15 09:37:04 fedora19 kernel: [ 204.735236] LustreError: 2647:0:(dir.c:595:ll_dir_read()) Skipped 2 previous similar messages Oct 15 09:37:04 fedora19 kernel: [ 204.735555] LustreError: 2647:0:(dir.c:398:ll_get_dir_page()) dir page locate: [0x200000002:0x1:0x0] at 18446744073709551614: rc -5 </pre> </div></div> <p>I think I know what the bug might be. Patch coming soon.</p> <p>Bob I just updated <a href="http://review.whamcloud.com/#/c/7747" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7747</a>. Can you try it now with Peng's fix.</p> <p>The refresh of #7747 had no effect that I can see. Still get similar errors on directory access. Peng's fix still appears to work.</p> <p>I think the refresh did have some effect. It looks like there is now exactly one log error associated with each directory failure. 1 per directory and no more. Think it was a bit more variable before the refresh.</p> <p>Can you post your last error.</p> <p>most recent errors. stdout/stderr:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ll -aR /mnt/lustre /mnt/lustre: ls: reading directory /mnt/lustre: Input/output error total 20 drwxr-xr-x 4 root root 4096 Oct 15 11:18 . drwxr-xr-x. 5 root root 4096 Sep 24 12:20 .. drwxr-xr-x 2 root root 4096 Oct 15 10:48 ddd drwxr-xr-x 3 root root 4096 Aug 29 14:05 .lustre -rw-r--r-- 1 root root 4 Sep 25 09:10 xxx /mnt/lustre/ddd: ls: reading directory /mnt/lustre/ddd: Input/output error total 8 drwxr-xr-x 2 root root 4096 Oct 15 10:48 . drwxr-xr-x 4 root root 4096 Oct 15 11:18 .. /mnt/lustre/.lustre: ls: reading directory /mnt/lustre/.lustre: Input/output error total 12 drwxr-xr-x 3 root root 4096 Aug 29 14:05 . drwxr-xr-x 4 root root 4096 Oct 15 11:18 .. d--x------ 2 root root 4096 Aug 29 14:05 fid /mnt/lustre/.lustre/fid: ls: reading directory /mnt/lustre/.lustre/fid: Input/output error total 8 d--x------ 2 root root 4096 Aug 29 14:05 . drwxr-xr-x 3 root root 4096 Aug 29 14:05 .. </pre> </div></div> <p>/var/log/messages:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 15 13:41:21 fedora19 kernel: [ 8177.767876] LustreError: 25405:0:(dir.c:398:ll_get_dir_page()) dir page locate: [0x200000007:0x1:0x0] at 18446744073709551614: rc -5 Oct 15 13:41:21 fedora19 kernel: [ 8177.767884] LustreError: 25405:0:(dir.c:596:ll_dir_read()) error reading dir [0x200000007:0x1:0x0] at 18446744073709551614: rc -5 Oct 15 13:41:21 fedora19 kernel: [ 8177.767887] LustreError: 25405:0:(dir.c:596:ll_dir_read()) Skipped 2 previous similar messages Oct 15 13:41:21 fedora19 kernel: [ 8177.769821] LustreError: 25405:0:(dir.c:398:ll_get_dir_page()) dir page locate: [0x200000002:0x1:0x0] at 18446744073709551614: rc -5 Oct 15 13:41:21 fedora19 kernel: [ 8177.769824] LustreError: 25405:0:(dir.c:398:ll_get_dir_page()) Skipped 1 previous similar message </pre> </div></div> <p>The offset reported in all those logged errors is 18446744073709551614 == 0xfffffffffffffffe == MDS_DIR_END_OFF. I wonder if that's a clue.</p> <p>I think I may have trial &amp; errored my way to a fix. Don't know if it's a good fix. If I add the following the errors go away.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> diff --git a/lustre/llite/dir.c b/lustre/llite/dir.c index 5f9e46b..9836c02 100644 --- a/lustre/llite/dir.c +++ b/lustre/llite/dir.c @@ -640,7 +640,9 @@ static int ll_readdir(struct file *filp, void *cookie, filld GOTO(out, rc = 0); #ifdef HAVE_DIR_CONTEXT + ctx-&gt;pos = pos; rc = ll_dir_read(inode, ctx); + pos = ctx-&gt;pos; #else rc = ll_dir_read(inode, &amp;pos, cookie, filldir); #endif </pre> </div></div> <p>Seems like a piece missing from #7747.</p> <p>So the patch in question seems to be <a href="http://review.whamcloud.com/#/c/7747" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7747</a>.</p> <p>James, I do not have Fedora 19 kernel but in upstream kernel, 3.11 release converts .readdir to .iterate in struct file_operations. And it comes along with the struct dir_context change. Does Fedora 19 kernel include them all or just the dir_context part?</p> <p>In upstream kernel client, my patch for dealing with .iterate change is <a href="https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=0b09d381bb2b4ccab15711cf98858a7146b24749" class="external-link" target="_blank" rel="nofollow noopener">https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=0b09d381bb2b4ccab15711cf98858a7146b24749</a>. You may want to take a look and consider merging it into your patch.</p> <p>Peng I also only test with the upstream kernels. Thanks for pointing out the patch. Bob got part of the fix but I see one other thing missing. Bob your clue points also to the ctx-&gt;pos not being updated when the pos is set to <br/> MDS_DIR_END_OFF. Next patch will have this update.</p> <p>Patch has been updated.</p> <p>Today's refresh of #7747 seems to do the job. No errors on dir access seen.</p> <p>On a different topic, how close are working server patches? With the ldiskfs patch set in <a href="https://jira.whamcloud.com/browse/LU-3373" title="ldiskfs patches for FC19" class="issue-link" data-issue-key="LU-3373"><del>LU-3373</del></a> now building OK it would be nice if we could complete and test a server build.</p> <p>Currently I have pushed what has been working for me to gerrit. I also have patches for the lod, mdd, mdt, and osp layer as well but currently their are show stopper bugs that need to be addressed. I said a one to two weeks of work to sort it get it going.</p> <p>James, thanks for the update. I have been paying close attention to the new patches as they show up in <a href="https://jira.whamcloud.com/browse/LU-3319" title="Adapt to 3.10 upstream kernel proc_dir_entry change" class="issue-link" data-issue-key="LU-3319"><del>LU-3319</del></a>. I look forward to having a more complete set in a few weeks.</p> <p>Peng, do you plan to push a version of <a href="https://github.com/bergwolf/linux/commit/28fde12d26d7d4b54cb4689dd6117c2ed791f985" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/bergwolf/linux/commit/28fde12d26d7d4b54cb4689dd6117c2ed791f985</a> to gerrit? Seems like we will need it in master.</p> <p>James, I took a look at v3.11 upstream kernel, it already requires iterate replacing readdir in struct file_operations, by kernel commit 2233f31aade393641f0eaed43a71110e629bb900</p> <p>commit 2233f31aade393641f0eaed43a71110e629bb900<br/> Author: Al Viro &lt;viro@zeniv.linux.org.uk&gt;<br/> Date: Wed May 22 21:44:23 2013 -0400</p> <p> <span class="error">&#91;readdir&#93;</span> -&gt;readdir() is gone</p> <p> everything's converted to -&gt;iterate()</p> <p> Signed-off-by: Al Viro &lt;viro@zeniv.linux.org.uk&gt;</p> <p>diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking<br/> index 0706d32..bdd82b2 100644<br/> &#8212; a/Documentation/filesystems/Locking<br/> +++ b/Documentation/filesystems/Locking<br/> @@ -414,7 +414,7 @@ prototypes:<br/> ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *);<br/> ssize_t (*aio_read) (struct kiocb *, const struct iovec *, unsigned long, loff_t);<br/> ssize_t (*aio_write) (struct kiocb *, const struct iovec *, unsigned long, loff_t);</p> <ul class="alternate" type="square"> <li>int (*readdir) (struct file *, void *, filldir_t);<br/> + int (*iterate) (struct file *, struct dir_context *);<br/> unsigned int (*poll) (struct file *, struct poll_table_struct *);<br/> long (*unlocked_ioctl) (struct file *, unsigned int, unsigned long);<br/> long (*compat_ioctl) (struct file *, unsigned int, unsigned long);<br/> &lt;snip&gt;</li> </ul> <p><span class="error">&#91;mpfssvr@linux-lustre&#93;</span>$git tag --contains 2233f31aade393641f0eaed43a71110e629bb900<br/> v3.11<br/> v3.11-rc1<br/> v3.11-rc2<br/> v3.11-rc3<br/> v3.11-rc4<br/> v3.11-rc5<br/> v3.11-rc6<br/> v3.11-rc7<br/> v3.12-rc1<br/> v3.12-rc2</p> <p>So I think we need a similar patch like <a href="https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=0b09d381bb2b4ccab15711cf98858a7146b24749" class="external-link" target="_blank" rel="nofollow noopener">https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=0b09d381bb2b4ccab15711cf98858a7146b24749</a> rather than just coping with struct dir_context. In fact, struct dir_context is introduced in the same kernel commit as .iterate. So whenever dir_context is there, .iterate exists. Please check your local tree to see if it somehow missed the commit.</p> <p>Sorry I mis-read 7747. Jemes' patch looks good. I will submit <a href="https://github.com/bergwolf/linux/commit/28fde12d26d7d4b54cb4689dd6117c2ed791f985" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/bergwolf/linux/commit/28fde12d26d7d4b54cb4689dd6117c2ed791f985</a> once it is merged in upstream kernel.</p> <p>Patches needed to support 3.11 kernel are:</p> <p><a href="http://review.whamcloud.com/#/c/7741" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7741</a><br/> <a href="http://review.whamcloud.com/#/c/7746" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7746</a><br/> <a href="http://review.whamcloud.com/#/c/7747" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7747</a><br/> <a href="http://review.whamcloud.com/#/c/7729" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7729</a></p> <p>Now that we have support for server side my testing for ldiskfs also shows we need to update osd_handler.c due the readdir -&gt; iterate changes. We have</p> <p>lustre-2.5.50/lustre/osd-ldiskfs/osd_handler.c: In function 'osd_ldiskfs_it_fill':<br/> lustre-2.5.50/lustre/osd-ldiskfs/osd_handler.c:4695: error: 'const struct file_operations' has no member named 'readdir'</p> <p>to be address as well.</p> <p>With these patches plus the patches listed in <a href="https://jira.whamcloud.com/browse/LU-3675" title="Support for linux 3.10 kernel" class="issue-link" data-issue-key="LU-3675"><del>LU-3675</del></a> I have managed to get ZFS working on a 3.11.2 kernel. Now onto ldiskfs support.</p> <p><span class="error">&#91;root@spoon46 tests&#93;</span># df<br/> Filesystem 1K-blocks Used Available Use% Mounted on<br/> unionfs 88763392 63498240 20757504 76% /<br/> none 100 0 100 0% /dev/shm<br/> none 2957124 19668 2937456 1% /tmp<br/> 172.30.248.91:/chexport<br/> 46445568 32107520 11978752 73% /chexport<br/> lustre-mdt1/mdt1 157440 3072 152320 2% /tmp/lustre/mds1<br/> lustre-ost1/ost1 157440 1152 154240 1% /tmp/lustre/ost1<br/> lustre-ost2/ost2 157440 1152 154240 1% /tmp/lustre/ost2<br/> 10.37.248.48@o2ib1:/lustre<br/> 314880 2304 308480 1% /lustre/barry<br/> <span class="error">&#91;root@spoon46 tests&#93;</span># uname -r<br/> 3.11.1</p> <p>Note that Yang Sheng gas ldiskfs patches for 3.11 in <a href="http://review.whamcloud.com/7263" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7263</a></p> <p>Excellent news. I have built a real file system this time using a 3.11.1 kernel on all the servers. Now to mount it on our cray test bed and run a bunch of jobs on it.</p> <p>Patches needed to support 3.11 kernel are:</p> <p><a href="http://review.whamcloud.com/#/c/7741" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7741</a><br/> <a href="http://review.whamcloud.com/#/c/7746" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7746</a><br/> <a href="http://review.whamcloud.com/#/c/7747" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7747</a><br/> <a href="http://review.whamcloud.com/#/c/7729" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7729</a></p> <p>These patches are ready for review and merger.</p> <p><a href="http://review.whamcloud.com/#/c/7746" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7746</a> is ready for review and possible merger.</p> <p>Finally got patch 7746 to pass maloo. Patch <a href="http://review.whamcloud.com/#/c/7746" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7746</a> is ready for inspection and possible landing.</p> <p>Two patches left to inspect and possibly merge.</p> <p><a href="http://review.whamcloud.com/#/c/7746" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7746</a><br/> <a href="http://review.whamcloud.com/#/c/7729" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7729</a></p> <p>All patches have landed. This ticket can be closed.</p> <p>Thanks James</p> Related LU-3373 LU-7711 LU-4011 Development Rank 1|hzw38f: Rank (Obsolete) 10608