https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-65 Lustre <p>These are the results of running the b1_8 acc_sm test. I also listed the results at <a href="https://bugzilla.lustre.org/show_bug.cgi?id=21367" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=21367</a>.</p> <p>sanity 64b - bug 22703<br/> sanity 72b - bug 24226<br/> replay-single 65a - bug 19960<br/> sanity-quota - totally broken. Does not work at all. Locks up clients</p> <p>obdfilter-survey - locks the client up. No debug output <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/><br/> config-sanity 55,56,57 - no bug report yet. Seeing the following error</p> <p>LustreError: 13996:0:(mdt_handler.c:4521:mdt_init0()) CMD Operation not allowed in IOP mode<br/> LustreError: 13996:0:(obd_config.c:495:class_setup()) setup lustre-MDT0001 failed (-22)<br/> LustreError: 13996:0:(obd_config.c:1338:class_config_llog_handler()) Err -22 on cfg command:<br/> Lustre: cmd=cf003 0:lustre-MDT0001 1:lustre-MDT0001_UUID 2:1 3:lustre-MDT0001-mdtlov 4:f <br/> LustreError: 15b-f: MGC10.36.230.2@o2ib: The configuration from log 'lustre-MDT0001'failed from the<br/> MGS (-22). Make sure this client and the MGS are running compatible versions of Lustre.<br/> LustreError: 15c-8: MGC10.36.230.2@o2ib: The configuration from log 'lustre-MDT0001' failed (-22).<br/> This may be the result of communication errors between this node and the MGS, a bad configuration,<br/> or other errors. See the syslog for more information.</p> <p>I will provide more info and logs as very soon.</p> 4 OSS, each with 7 OSTS with one MDS with a disk devoted to a MDT and another disk devoted to the MGS; all running lustre 2.0.59. I have 4 clients running lustre 1.8.5.54 LU-65

Interop testing results for 1.8.5.54 clients with Lustre 2.0.59 servers

Bug Blocker Resolved Fixed nasf James A Simmons Tue, 8 Feb 2011 12:16:29 +0000 Tue, 28 Jun 2011 15:01:40 +0000 Tue, 29 Mar 2011 11:13:16 +0000 Lustre 1.8.6 Lustre 2.1.0 0 2 <p>Here are some lustre logs produced for the failed runs.</p> <p>Nasf</p> <p>Can you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Debug log from OBDFilter test 1b from 1.8.X test suite on 1.8 client</p> <p>No debug logs for Obdfilter-survey but I do have a dmesg that could be of some interest. For some reason it appears the client can't communicate with the OSS runing 2.X</p> <p>Also for Obdfilter_survery test 2b on the OSS running lustre 2.X I'm seeing in dmesg</p> <p>Lustre: DEBUG MARKER: == test 2b: Stripe F/S over the Network, async journal == 11:45:14 (1297269914)<br/> LustreError: 137-5: UUID 'lustre-OST0008_UUIlustre-OST0008_UUID' is not available for connect (no target)<br/> LustreError: Skipped 37 previous similar messages<br/> LustreError: 14469:0:(ldlm_lib.c:2118:target_send_reply_msg()) @@@ processing error (<del>19) req@ffff8104233e6400 x1355126983196980/t0(0) o-1</del>&gt;&lt;?&gt;@&lt;?&gt;:0/0 lens 368/0 e 0 to 0 dl 1297270075 ref 1 fl Interpret:/ffffffff/ffffffff rc -19/-1<br/> LustreError: 14469:0:(ldlm_lib.c:2118:target_send_reply_msg()) Skipped 42 previous similar messages</p> <p>Their is no lustre-OST008_UUID for the OSS. Where is it getting this info?</p> <p>Patch for "sanity 72b - bug 24226" is in inspection:</p> <p><a href="http://review.whamcloud.com/#change,238" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,238</a></p> <p>That patch falls short of behaving correctly. Try this.</p> <p>touch $DIR/$tfile<br/> chmod 777 $DIR/$tfile<br/> chmod ug+s $DIR/$tfile<br/> $RUNAS dd if=/dev/urandom of=$DIR/$tfile</p> <p>While it runs look at the file's permission. You will notice the suid bits are still there. If you try this on a ext<span class="error">&#91;3,4&#93;</span> file system you will notice the suid bits are gone soon as you start writing to the file. The current lustre code handles the fixup of attr when the file is closed. It needs to be managed when the file is opened. I have a patch but it needs to be worked on since their exist the case of a suid file copied from a non lustre file system to a lustre file system to preserve those suid bits. The case is very specific for the removal of the suid. As soon as I have it working I will post a patch.</p> <p>The SUID/SGID should be removed just when you start to write the file, which is the expected behavior. With the patch applied, Lustre-2.x behavior is the same as ext3/4 does, but not you mentioned, at least I can not reproduce it (means the SUID/SGID bits still exist when writes until file closed). Have you verified my patch? or your description is based on the test result against old Lustre code?</p> <p>I will give your patches a try monday. Right now I'm running some test for oleg.</p> <p>Okay I did test your patch and it appears to work.</p> <p>I have a bunch of patches to fix various parts of the 2.X test suite. Shoudl I opena different bug for those patches?</p> <p>I thing you can create some sub-tasks under this one, then it is more easy to be tracked.</p> <p>Jsmes</p> <p>Personally I think that it is easier to track issues when there is a 1:1 relationship between tickets and issues\fixes</p> <p>Peter</p> <p>&gt;replay-single 65a - bug 19960</p> <p>Sorry, I can not reproduce this failure. Can you show me an easy way to reproduce it? I think it is a duplicate of bug 22560, which has been fixed on master and lustre-1.8.5. Would you like to verify it again?</p> <p>Just tried it. Also the test fails with 2.X clients with 2.X servers.</p> <p>I believe I found the problem for replay-single 65a. Please look at patch <a href="http://review.whamcloud.com/#change,284" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,284</a></p> <p>&gt; sanity 64b - bug 22703</p> <p>I have made patch for it:<br/> <a href="http://review.whamcloud.com/#change,286" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,286</a> (for master)<br/> <a href="http://review.whamcloud.com/#change,287" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,287</a> (for b1_8)</p> <p>&gt; sanity-quota - totally broken. Does not work at all. Locks up clients</p> <p>Are there any logs related with sanity_quota interoperability test which caused client locked up? Because sanity-quota interoperability test passed in my local environment. I have checked bugzilla also, and found recent test result:</p> <p><a href="https://bugzilla.lustre.org/show_bug.cgi?id=24207#c4" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=24207#c4</a></p> <p>That means sanity-quota interoperability works under TCP environment, but failed under IB case for bug 24055, and related patch for bug 24055 has been landed.</p> <p>So would you like to check whether such patch applied in your test. On the other hand, I think bug 24055's patch is not enough, you need above patch for bug 22703 also.</p> <p>Thanks!</p> <p>&gt; config-sanity 55,56,57 - no bug report yet.</p> <p>According to the MDS side log (lustre_conf-sanity_test_55.1297196706.gz), the system is not ready to accept client(10.36.230.36@o2ib) connection yet.</p> <p>==========<br/> 00000004:00000001:2.0:1297196688.391043:0:19245:0:(mdt_handler.c:2528:mdt_req_handle()) Process entered<br/> 00000004:00000001:2.0:1297196688.391045:0:19245:0:(mdt_handler.c:2482:mdt_unpack_req_pack_rep()) Process entered<br/> 00000004:00000001:2.0:1297196688.391046:0:19245:0:(mdt_handler.c:2503:mdt_unpack_req_pack_rep()) Process leaving (rc=0 : 0 : 0)<br/> 00010000:00000001:2.0:1297196688.391049:0:19245:0:(ldlm_lib.c:667:target_handle_connect()) Process entered<br/> 00010000:02000400:2.0:1297196688.391055:0:19245:0:(ldlm_lib.c:694:target_handle_connect()) lustre-MDT0000: temporarily refusing client connection from 10.36.230.36@o2ib<br/> 00010000:00000001:2.0:1297196688.406588:0:19245:0:(ldlm_lib.c:695:target_handle_connect()) Process leaving via out (rc=18446744073709551605 : -11 : 0xfffffffffffffff5)<br/> 00010000:00000001:2.0:1297196688.406591:0:19245:0:(ldlm_lib.c:1082:target_handle_connect()) Process leaving (rc=18446744073709551605 : -11 : fffffffffffffff5)<br/> ==========</p> <p>That means MDS returned "EAGAIN" to client to tell it retry later, which is normal case. But from the log, I can not find any other communication between client and MDS after that, until MDS reported test_55 failure.</p> <p>00000001:00000001:5.0:1297196705.949788:0:19367:0:(debug.c:439:libcfs_debug_mark_buffer()) ***************************************************<br/> 00000001:02000400:5.0:1297196705.949789:0:19367:0:(debug.c:440:libcfs_debug_mark_buffer()) DEBUG MARKER: conf-sanity test_55: @@@@@@ FAIL: client start failed<br/> 00000001:00000001:5.0:1297196705.963380:0:19367:0:(debug.c:441:libcfs_debug_mark_buffer()) ***************************************************</p> <p>I need client side log to investigate what happened on client after MDS returned "EAGAIN".<br/> James, are there any logs for that? It seems not easy to reproduce conf_sanity test_55 failure.</p> <p>Thanks!</p> <p>For <a href="http://review.whamcloud.com/#change,286" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,286</a> the two sets of patches conflict. Is the second patch the only valid one?</p> <p>&gt; For <a href="http://review.whamcloud.com/#change,286" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,286</a> the two sets of patches conflict. Is the second patch the only valid one?</p> <p>Yes, set 2 is the right one.</p> <p>Sorry I haven't been able to test. The build system is broken.</p> <p>/data/buildsystem/jsimmons-head/rpmbuild/BUILD/lustre-2.0.59/lustre/lvfs/fsfilt-ldiskfs.c: In function 'fsfilt_ldiskfs_fid2dentry':<br/> /data/buildsystem/jsimmons-head/rpmbuild/BUILD/lustre-2.0.59/lustre/lvfs/fsfilt-ldiskfs.c:2352: error: implicit declaration of function 'exportfs_decode_fh'<br/> /data/buildsystem/jsimmons-head/rpmbuild/BUILD/lustre-2.0.59/lustre/lvfs/fsfilt-ldiskfs.c:2352: error: 'FILEID_INO32_GEN' undeclared (first use in this function)<br/> /data/buildsystem/jsimmons-head/rpmbuild/BUILD/lustre-2.0.59/lustre/lvfs/fsfilt-ldiskfs.c:2352: error: (Each undeclared identifier is reported only once<br/> /data/buildsystem/jsimmons-head/rpmbuild/BUILD/lustre-2.0.59/lustre/lvfs/fsfilt-ldiskfs.c:2352: error: for each function it appears in.)<br/> cc1: warnings being treated as errors</p> <p>The patch of "http://review.whamcloud.com/#change,286" has been landed, I think you can test with the latest code.</p> <p>On the other hand, would you like to update your patch of "http://review.whamcloud.com/#change,284" to make it more compatible?</p> <p>Thanks</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/reviews-centos5/541/" class="external-link" target="_blank" rel="nofollow noopener">reviews-centos5 #541</a><br/> <a href="https://jira.whamcloud.com/browse/LU-65" title="Interop testing results for 1.8.5.54 clients with Lustre 2.0.59 servers" class="issue-link" data-issue-key="LU-65"><del>LU-65</del></a> ORNL Lustre 2.X testing</p> <p>James Simmons : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-release.git&amp;a=commit&amp;h=43d727e089f1a1cf237da4251dc2aa661de05a0b" class="external-link" target="_blank" rel="nofollow noopener">43d727e089f1a1cf237da4251dc2aa661de05a0b</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> <li>libcfs/libcfs/darwin/darwin-proc.c</li> <li>lustre/obdclass/class_obd.c</li> <li>lustre/include/obd_support.h</li> <li>libcfs/include/libcfs/Makefile.am</li> <li>lustre/obdclass/darwin/darwin-sysctl.c</li> <li>lustre/lvfs/lvfs_lib.c</li> <li>lustre/obdfilter/filter.c</li> <li>lustre/obdclass/linux/linux-sysctl.c</li> <li>libcfs/libcfs/module.c</li> <li>lustre/mdt/mdt_internal.h</li> <li>libcfs/libcfs/Makefile.in</li> <li>lustre/liblustre/tests/recovery_small.c</li> <li>libcfs/libcfs/linux/linux-proc.c</li> <li>libcfs/include/libcfs/libcfs.h</li> <li>lustre/include/darwin/obd_support.h</li> <li>lustre/include/linux/obd_support.h</li> <li>lustre/tests/sanity-gss.sh</li> <li>libcfs/libcfs/autoMakefile.am</li> <li>libcfs/libcfs/fail.c</li> <li>libcfs/include/libcfs/libcfs_fail.h</li> </ul> <p>FWIW, your ubuntu reviews builds are failing due to <a href="https://jira.whamcloud.com/browse/LU-92" title="Missing files in dist tarball" class="issue-link" data-issue-key="LU-92"><del>LU-92</del></a>. I've just submitted <a href="http://review.whamcloud.com/#change,356" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,356</a> to test a patch to fix this. If it passes it's review testing, you could try to import that patch into your branch, putting it patch before your patch to see if it resolves your ubuntu build issue.</p> <p>Believed resolved. ORNL will reopen or open a new ticket if their reproducer still has issues</p> Bugzilla ID 21367.0 Development Epic interop results test Rank 1|hzvabz: Rank (Obsolete) 5209 Severity