https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4351 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/0ac2cba4-5de4-11e3-8f3c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/0ac2cba4-5de4-11e3-8f3c-52540035b04c</a>.</p> <p>The sub-test test_54c failed with the following error:</p> <blockquote> <p>error mounting /mnt/lustre/loop54c on /mnt/lustre/d54c</p></blockquote> <p>client dmesg shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 2564.211169] LustreError: 9888:0:(rw.c:128:ll_cl_init()) lustre: [0x200001b70:0xeff1:0x0] no active IO, please file a ticket. [ 2564.211172] Pid: 9888, comm: loop3 [ 2564.211173] [ 2564.211174] Call Trace: [ 2564.211186] [&lt;ffffffff81004935&gt;] dump_trace+0x75/0x310 [ 2564.211210] [&lt;ffffffffa05c282a&gt;] libcfs_debug_dumpstack+0x4a/0x70 [libcfs] [ 2564.211250] [&lt;ffffffffa0becb8e&gt;] ll_cl_init+0x22e/0x330 [lustre] [ 2564.211280] [&lt;ffffffffa0c0901a&gt;] ll_write_begin+0x8a/0x5d0 [lustre] [ 2564.211305] [&lt;ffffffffa02cb50d&gt;] do_lo_send_aops+0xad/0x1b0 [loop] [ 2564.211310] [&lt;ffffffffa02cb7a0&gt;] do_bio_filebacked+0x190/0x280 [loop] [ 2564.211314] [&lt;ffffffffa02cb952&gt;] loop_thread+0xc2/0x250 [loop] [ 2564.211319] [&lt;ffffffff81082306&gt;] kthread+0x96/0xa0 [ 2564.211325] [&lt;ffffffff81467864&gt;] kernel_thread_helper+0x4/0x10 [ 2564.211328] [ 2564.211335] Buffer I/O error on device loop3, logical block 0 [ 2564.211336] lost page write due to I/O error on loop3 [ 2564.211345] Pid: 9888, comm: loop3 [ 2564.211346] [ 2564.211346] Call Trace: [ 2564.211350] [&lt;ffffffff81004935&gt;] dump_trace+0x75/0x310 [ 2564.211360] [&lt;ffffffffa05c282a&gt;] libcfs_debug_dumpstack+0x4a/0x70 [libcfs] [ 2564.211377] [&lt;ffffffffa0becb8e&gt;] ll_cl_init+0x22e/0x330 [lustre] [ 2564.211403] [&lt;ffffffffa0c0901a&gt;] ll_write_begin+0x8a/0x5d0 [lustre] [ 2564.211421] [&lt;ffffffffa02cb50d&gt;] do_lo_send_aops+0xad/0x1b0 [loop] [ 2564.211426] [&lt;ffffffffa02cb7a0&gt;] do_bio_filebacked+0x190/0x280 [loop] [ 2564.211430] [&lt;ffffffffa02cb952&gt;] loop_thread+0xc2/0x250 [loop] [ 2564.211434] [&lt;ffffffff81082306&gt;] kthread+0x96/0xa0 [ 2564.211445] [&lt;ffffffff81467864&gt;] kernel_thread_helper+0x4/0x10 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 2564.237136] EXT2-fs (loop3): error: can't find an ext2 filesystem on dev loop3. [ 2568.722796] Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity test_54c: @@@@@@ FAIL: error mounting \/mnt\/lustre\/loop54c on \/mnt\/lustre\/d54c </pre> </div></div> lustre-master build #1791 ldiskfs <br/> client is running SLES11 SP3 LU-4351

sanity test_54c: can't find an ext2 filesystem on dev loop3

Bug Minor Resolved Fixed Bob Glossman Maloo Thu, 5 Dec 2013 22:02:41 +0000 Mon, 21 Jul 2014 15:43:27 +0000 Tue, 3 Jun 2014 14:48:32 +0000 Lustre 2.6.0 Lustre 2.6.0 Lustre 2.5.2 0 11 <p>too bad, this kernel is too old which still calls write_being() and write_end() to write pages.</p> <p>what do you mean the kernel is OLD? It's 3.4 or something like that, definitely very new compared to rhel6.x we run.</p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-reviews/20841/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/20841/</a><br/> Distro/Arch: SLES11SP3/x86_64 (both server and client, kernel version: 3.0.101-0.8)</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/7e296c20-7986-11e3-a27b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7e296c20-7986-11e3-a27b-52540035b04c</a></p> <p>This is actually a distinct SLES11 specific bug and not a dup. The underlying cause is the quite variant implementation of kernel loop devices in SLES11. It differs from the implementation of similar devices in both earlier and later kernel versions in all other distros yet examined, including much later kernel.org versions.</p> <p>The basic problem is that the SLES implementation uses address space ops while all the other linux'es use only regular file I/O ops in their write code for kernel loop devices. This doesn't play nicely with embedded assumptions in lustre code.</p> <p>Since the problem is expected to go away in later kernel versions, even later SLES kernels, the plan is to just skip the test in current SLES clients as it will always fail there.</p> <p><a href="http://review.whamcloud.com/9955" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9955</a></p> <p>in b2_5<br/> <a href="http://review.whamcloud.com/10085" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10085</a></p> <p>I'm a bit puzzled:<br/> Cray has been running sanity on 2.x Lustre+SLES11SP3 for some time now. We only see this test failure when testing master, not 2.5 or 2.4.<br/> So it seem strange to say the fault lies completely with SLES.</p> <p>I'm not saying SLES doesn't have a bug in it, but wasn't there a change made on the Lustre side that caused this failure to start happening? Have you actually seen this test failure with b2_5?</p> <p>yes, this problem has been reproduced in various branches. not just seen in master. It's possible that the SLES11 flaw in their kernel device implementation only cut in with some specific kernel version over the long life of SLES11. I only looked at recent 3.0.x versions, didn't check back through all of SLES11 history.</p> <p>It's also possible that the lustre sensitivity to the use of address space ops called directly came about due to recent changes in lustre, like clio or unified target. Even if that's true we aren't likely to back out such new features due to a very SLES11 specific incompatibility.</p> <p>Just as an aside have already confirmed that the current SLES12 beta doesn't have this flaw in its loop devices. Nor does any other linux kernel we've been able to examine.</p> <p>Thanks, Bob. That clarifies. </p> <p>Well, if this use case turns out to be something we/our customers truly care about - I doubt it will, but... - Cray will continue looking in to it to try to figure out a fix and pass on whatever we find.</p> <p>I still have the error on Xeon Phi:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[12215.497838] LustreError: 33755:0:(rw.c:128:ll_cl_init()) lustre25: [0x200000bd1:0xbfda:0x0] no active IO, please file a ticket. [12215.497865] Pid: 33755, comm: loop3 [12215.497871] [12215.497874] Call Trace: [12215.497934] [&lt;ffffffffa01615f9&gt;] libcfs_debug_dumpstack+0x5b/0x6d [libcfs] [12215.497991] [&lt;ffffffffa066e177&gt;] ll_cl_init+0x1e1/0x30a [lustre] [12215.498043] [&lt;ffffffffa0684745&gt;] ll_write_begin+0x120/0x609 [lustre] [12215.498068] [&lt;ffffffff8109f2de&gt;] pagecache_write_begin+0x1c/0x1e [12215.498091] [&lt;ffffffff81204c8f&gt;] do_lo_send_aops+0x9c/0x17a [12215.498110] [&lt;ffffffff81022c8c&gt;] ? set_next_entity+0x45/0x99 [12215.498127] [&lt;ffffffff81028ae7&gt;] ? pick_next_task_fair+0xaf/0xbf [12215.498145] [&lt;ffffffff81204600&gt;] do_bio_filebacked+0xfd/0x27f [12215.498161] [&lt;ffffffff81204bf3&gt;] ? do_lo_send_aops+0x0/0x17a [12215.498178] [&lt;ffffffff81204bbd&gt;] loop_thread+0x1db/0x211 [12215.498194] [&lt;ffffffff8104dba0&gt;] ? autoremove_wake_function+0x0/0x38 [12215.498211] [&lt;ffffffff812049e2&gt;] ? loop_thread+0x0/0x211 [12215.498229] [&lt;ffffffff8104d611&gt;] kthread+0x84/0x8c [12215.498247] [&lt;ffffffff81003814&gt;] kernel_thread_helper+0x4/0x10 [12215.498264] [&lt;ffffffff8104d58d&gt;] ? kthread+0x0/0x8c [12215.498278] [&lt;ffffffff81003810&gt;] ? kernel_thread_helper+0x0/0x10 [12215.498286] [12215.498321] Buffer I/O error on device loop3, logical block 0 [12215.498333] lost page write due to I/O error on loop3 [12215.498416] Pid: 33755, comm: loop3 [12215.498423] [12215.498426] Call Trace: [12215.498459] [&lt;ffffffffa01615f9&gt;] libcfs_debug_dumpstack+0x5b/0x6d [libcfs] [12215.498508] [&lt;ffffffffa066e177&gt;] ll_cl_init+0x1e1/0x30a [lustre] [12215.498560] [&lt;ffffffffa0684745&gt;] ll_write_begin+0x120/0x609 [lustre] [12215.498577] [&lt;ffffffff8109f2de&gt;] pagecache_write_begin+0x1c/0x1e [12215.498594] [&lt;ffffffff81204c8f&gt;] do_lo_send_aops+0x9c/0x17a [12215.498615] [&lt;ffffffff81351ed8&gt;] ? __slab_free+0x78/0xd8 [12215.498630] [&lt;ffffffff810a323b&gt;] ? mempool_free_slab+0x17/0x19 [12215.498647] [&lt;ffffffff81204600&gt;] do_bio_filebacked+0xfd/0x27f [12215.498663] [&lt;ffffffff81204bf3&gt;] ? do_lo_send_aops+0x0/0x17a [12215.498685] [&lt;ffffffff8110605f&gt;] ? bio_free+0x3d/0x54 [12215.498700] [&lt;ffffffff8110608b&gt;] ? bio_fs_destructor+0x15/0x17 [12215.498716] [&lt;ffffffff81105cc3&gt;] ? bio_put+0x2b/0x2d [12215.498731] [&lt;ffffffff81204bbd&gt;] loop_thread+0x1db/0x211 [12215.498745] [&lt;ffffffff8104dba0&gt;] ? autoremove_wake_function+0x0/0x38 [12215.498762] [&lt;ffffffff812049e2&gt;] ? loop_thread+0x0/0x211 [12215.498778] [&lt;ffffffff8104d611&gt;] kthread+0x84/0x8c [12215.498794] [&lt;ffffffff81003814&gt;] kernel_thread_helper+0x4/0x10 [12215.498811] [&lt;ffffffff8104d58d&gt;] ? kthread+0x0/0x8c [12215.498825] [&lt;ffffffff81003810&gt;] ? kernel_thread_helper+0x0/0x10 [12215.498833] [12215.498862] Buffer I/O error on device loop3, logical block 1 [12215.498874] lost page write due to I/O error on loop3 [12215.498950] Pid: 33755, comm: loop3 ... </pre> </div></div> <p>Dmitry,<br/> Are you using a SLES kernel on the Phi? if so then this is expected behavior. If not then it may be an entirely different problem.</p> <p>No, it's upstream kernel 2.6.38 with patches for Xeon Phi.</p> <p>I see your call stack has do_lo_send_aops in it. That says this is likely the same bug that's in the SLES11 kernel. Does any of your Phi specific patches apply in drivers/block/loop.c? If not then this bug probably exists in 2.6.38 upstream kernel source you are using. Previously had only seen this internal routine in SLES 11 (3.0.x) kernels. Not seen in el6 (2.6.32-xxx) kernels or fc20, rhel7, sles12.</p> <p>As far as I know there is no workaround.</p> <p>I have been researching the use of address space ops in the kernel loop device code a bit. As far as I can tell it's been part of the mainline upstream linux kernel for a very long time until the following commit in 2011:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>author Christoph Hellwig &lt;hch@infradead.org&gt; 2011-10-17 10:57:20 (GMT) committer Jens Axboe &lt;axboe@kernel.dk&gt; 2011-10-17 10:57:20 (GMT) commit 456be1484ffc72a24bdb4200b5847c4fa90139d9 (patch) tree 570f0818bd6cfa245ab23d0121853b7b1e5a649b /drivers/block/loop.c parent 8bc03e8f3a334e09e89a7dffb486ee97a5ce84ae (diff) loop: remove the incorrect write_begin/write_end shortcut Currently the loop device tries to call directly into write_begin/write_end instead of going through -&gt;write if it can. This is a fairly nasty shortcut as write_begin and write_end are only callbacks for the generic write code and expect to be called with filesystem specific locks held. This code currently causes various issues for clustered filesystems as it doesn't take the required cluster locks, and it also causes issues for XFS as it doesn't properly lock against the swapext ioctl as called by the defragmentation tools. This in case causes data corruption if defragmentation hits a busy loop device in the wrong time window, as reported by RH QA. The reason why we have this shortcut is that it saves a data copy when doing a transformation on the loop device, which is the technical term for using cryptoloop (or an XOR transformation). Given that cryptoloop has been deprecated in favour of dm-crypt my opinion is that we should simply drop this shortcut instead of finding complicated ways to to introduce a formal interface for this shortcut. Signed-off-by: Christoph Hellwig &lt;hch@lst.de&gt; Signed-off-by: Jens Axboe &lt;axboe@kernel.dk&gt; </pre> </div></div> <p>I suspect the reason we don't see it in el6 is that yanking it out was one of the RH customizations of their distro kernel.<br/> I suspect the reason it's in SLES11 is the branch their kernel is derived from is too old to include the upstream kernel commit.</p> <p>Recent bug report submitted upstream to SuSE and marked fixed suggests this bug will very likely disappear after the next sles11sp3 kernel update.</p> <p>Bob: Are you able to provide a link or other info about that bug report?</p> <p>Thanks</p> <p>Patrick,<br/> It's <a href="https://bugzilla.novell.com/show_bug.cgi?id=878123" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.novell.com/show_bug.cgi?id=878123</a>. Don't know if that's generally accessible.</p> <p>btw, the fix isn't in the latest update to 3.0.101-0.29. I expect it in the next one.</p> <p>Thanks, Bob - It doesn't seem to be, but Cray has support contracts with SUSE, so someone around here should be able to get me in. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Much appreciated.</p> <p>Patches have landed.</p> <p>fixed in sles11sp3 as well in the most recent upstream kernel update from SuSE to version 3.0.101-0.35</p> Duplicate LU-5032 Development Rank 1|hzwarj: Rank (Obsolete) 11919 Severity