https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2201 Lustre <p><a href="https://maloo.whamcloud.com/test_sets/90a83b74-872a-11e1-aa0e-525400d2bfa6" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/90a83b74-872a-11e1-aa0e-525400d2bfa6</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>18:40:51:Lustre: DEBUG MARKER: == conf-sanity test 32a: Upgrade (not live) == 15:40:49 (1334443249) 18:40:52:LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=off. Opts: 18:40:52:Lustre: 20270:0:(mdt_lproc.c:410:lprocfs_wr_identity_upcall()) t32fs-MDT0000: identity upcall set to /usr/sbin/l_getidentity 18:40:53:LDISKFS-fs (loop1): mounted filesystem with ordered data mode. quota=off. Opts: 18:40:53:Lustre: 20424:0:(ofd_fs.c:257:ofd_groups_init()) t32fs-OST0000: 1 groups initialized 18:40:53:Lustre: Setting parameter t32fs-OST0000-osc.osc.max_dirty_mb in log t32fs-client 18:40:58:LustreError: 20580:0:(osp_dev.c:242:osp_process_config()) t32fs-OST0000-osc-MDT0000: unknown param osc.max_dirty_mb=15 18:40:58:LustreError: 20580:0:(obd_config.c:730:class_add_conn()) can't add connection on non-client dev 18:40:59:Lustre: Failing over t32fs-MDT0000 18:41:05:LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=off. Opts: 18:41:06:Lustre: 20744:0:(mdt_lproc.c:410:lprocfs_wr_identity_upcall()) t32fs-MDT0000: identity upcall set to /usr/sbin/l_getidentity 18:41:06:LustreError: 20744:0:(genops.c:317:class_newdev()) Device t32fs-OST0000-osc-MDT0000 already exists at 6, won't add 18:41:06:LustreError: 20744:0:(obd_config.c:334:class_attach()) Cannot create device t32fs-OST0000-osc-MDT0000 of type osp : -17 18:41:06:LustreError: 20744:0:(obd_config.c:1407:class_config_llog_handler()) Err -17 on cfg command: 18:41:06:Lustre: cmd=cf001 0:t32fs-OST0000-osc-MDT0000 1:osp 2:t32fs-MDT0000-mdtlov_UUID 18:41:06:LustreError: 15c-8: MGC172.29.3.95@tcp: The configuration from log 't32fs-MDT0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. 18:41:06:LustreError: 20714:0:(obd_mount.c:984:server_start_targets()) failed to start server t32fs-MDT0000: -17 18:41:06:LustreError: 20714:0:(obd_mount.c:1280:lustre_server_mount()) Unable to start targets: -17 18:41:07:Lustre: Failing over t32fs-MDT0000 18:41:07:Lustre: Skipped 1 previous similar message 18:41:11:LustreError: 20714:0:(obd_mount.c:1949:lustre_mount()) Unable to mount (-17) 18:41:12:Lustre: DEBUG MARKER: conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT </pre> </div></div> LU-2201

conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT

Bug Minor Resolved Fixed Li Wei Li Wei Sun, 15 Apr 2012 23:32:36 +0000 Tue, 23 Apr 2013 23:09:23 +0000 Fri, 19 Apr 2013 20:36:50 +0000 Lustre 2.3.0 0 4 <p><a href="http://review.whamcloud.com/2546" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2546</a></p> <p>The patch has landed to Orion.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,client,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-2201" title="conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT" class="issue-link" data-issue-key="LU-2201"><del>ORI-635</del></a> tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)</p> <p> Result = SUCCESS<br/> Mikhail Pershin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=275eb2b56a6c3d52056e95a1e2a299f2ad73f076" class="external-link" target="_blank" rel="nofollow noopener">275eb2b56a6c3d52056e95a1e2a299f2ad73f076</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,client,el6,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-2201" title="conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT" class="issue-link" data-issue-key="LU-2201"><del>ORI-635</del></a> tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)</p> <p> Result = SUCCESS<br/> Mikhail Pershin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=275eb2b56a6c3d52056e95a1e2a299f2ad73f076" class="external-link" target="_blank" rel="nofollow noopener">275eb2b56a6c3d52056e95a1e2a299f2ad73f076</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,server,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-2201" title="conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT" class="issue-link" data-issue-key="LU-2201"><del>ORI-635</del></a> tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)</p> <p> Result = SUCCESS<br/> Mikhail Pershin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=275eb2b56a6c3d52056e95a1e2a299f2ad73f076" class="external-link" target="_blank" rel="nofollow noopener">275eb2b56a6c3d52056e95a1e2a299f2ad73f076</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,server,el6,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-2201" title="conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT" class="issue-link" data-issue-key="LU-2201"><del>ORI-635</del></a> tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)</p> <p> Result = SUCCESS<br/> Mikhail Pershin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=275eb2b56a6c3d52056e95a1e2a299f2ad73f076" class="external-link" target="_blank" rel="nofollow noopener">275eb2b56a6c3d52056e95a1e2a299f2ad73f076</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » i686,client,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-2201" title="conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT" class="issue-link" data-issue-key="LU-2201"><del>ORI-635</del></a> tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)</p> <p> Result = SUCCESS<br/> Mikhail Pershin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=275eb2b56a6c3d52056e95a1e2a299f2ad73f076" class="external-link" target="_blank" rel="nofollow noopener">275eb2b56a6c3d52056e95a1e2a299f2ad73f076</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,server,el5,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-2201" title="conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT" class="issue-link" data-issue-key="LU-2201"><del>ORI-635</del></a> tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)</p> <p> Result = SUCCESS<br/> Mikhail Pershin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=275eb2b56a6c3d52056e95a1e2a299f2ad73f076" class="external-link" target="_blank" rel="nofollow noopener">275eb2b56a6c3d52056e95a1e2a299f2ad73f076</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-dev/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/340/" class="external-link" target="_blank" rel="nofollow noopener">lustre-dev » x86_64,client,el6,inkernel #340</a><br/> <a href="https://jira.whamcloud.com/browse/LU-2201" title="conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT" class="issue-link" data-issue-key="LU-2201"><del>ORI-635</del></a> tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)</p> <p> Result = SUCCESS<br/> Mikhail Pershin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-dev.git;a=shortlog;h=refs/heads/$GIT_BRANCH&amp;a=commit&amp;h=275eb2b56a6c3d52056e95a1e2a299f2ad73f076" class="external-link" target="_blank" rel="nofollow noopener">275eb2b56a6c3d52056e95a1e2a299f2ad73f076</a><br/> Files : </p> <ul> <li>lustre/tests/conf-sanity.sh</li> </ul> <p>It appears that the conf-sanity.sh test_32 changes were lost during orion_head_sync merging. It would be great to land these on master again.</p> <p>Hi,</p> <p>Hit this issue on master <a href="https://maloo.whamcloud.com/test_sessions/378a61c6-17c0-11e2-a41f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/378a61c6-17c0-11e2-a41f-52540035b04c</a> </p> <p>Some logs:<br/> 06:47:49:Lustre: DEBUG MARKER: == conf-sanity test 32a: Upgrade (not live) == 06:47:42 (1350395262)<br/> 06:47:49:Lustre: DEBUG MARKER: which tunefs.lustre<br/> 06:47:49:Lustre: DEBUG MARKER: find /usr/lib64/lustre/tests -maxdepth 1 -name 'disk*-ldiskfs.tar.bz2'<br/> 06:47:49:Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids<br/> 06:47:49:Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/ost<br/> 06:47:49:Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_1-ldiskfs.tar.bz2 -S -C /tmp/t32<br/> 06:47:49:Lustre: DEBUG MARKER: cat /tmp/t32/commit<br/> 06:47:49:Lustre: DEBUG MARKER: cat /tmp/t32/kernel<br/> 06:47:50:Lustre: DEBUG MARKER: cat /tmp/t32/arch<br/> 06:47:50:Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1<br/> 06:47:50:Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt<br/> 06:47:50:Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust<br/> 06:48:02:Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt<br/> 06:48:02:LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=off. Opts: <br/> 06:48:02:Lustre: MGC192.168.4.20@o2ib: Reactivating import<br/> 06:48:02:Lustre: Found index 0 for t32fs-MDT0000, updating log<br/> 06:48:02:Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000<br/> 06:48:02:Lustre: Modifying parameter t32fs-clilov.lov.stripesize in log t32fs-client<br/> 06:48:02:Lustre: t32fs-MDT0000: used disk, loading<br/> 06:48:02:LustreError: 27989:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context<br/> 06:48:02:LustreError: 27989:0:(ldlm_lib.c:418:client_obd_setup()) can't add initial connection<br/> 06:48:02:LustreError: 27989:0:(osp_dev.c:493:osp_init0()) t32fs-OST0000-osc-MDT0000: can't setup obd: -2<br/> 06:48:02:LustreError: 27989:0:(obd_config.c:572:class_setup()) setup t32fs-OST0000-osc-MDT0000 failed (-2)<br/> 06:48:02:LustreError: 27989:0:(obd_config.c:1546:class_config_llog_handler()) MGC192.168.4.20@o2ib: cfg command failed: rc = -2<br/> 06:48:02:Lustre: cmd=cf003 0:t32fs-OST0000-osc-MDT0000 1:t32fs-OST0000_UUID 2:10.10.4.12@tcp <br/> 06:48:02:LustreError: 15c-8: MGC192.168.4.20@o2ib: The configuration from log 't32fs-MDT0000' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br/> 06:48:02:LustreError: 27939:0:(obd_mount.c:1850:server_start_targets()) failed to start server t32fs-MDT0000: -2<br/> 06:48:02:LustreError: 27939:0:(obd_mount.c:2401:server_fill_super()) Unable to start targets: -2<br/> 06:48:02:LustreError: 27939:0:(obd_mount.c:1350:lustre_disconnect_osp()) Can't end config log t32fs<br/> 06:48:02:LustreError: 27939:0:(obd_mount.c:2114:server_put_super()) t32fs-MDT0000: failed to disconnect osp-on-ost (rc=-2)!<br/> 06:48:02:Lustre: Failing over t32fs-MDT0000<br/> 06:48:02:LustreError: 27939:0:(obd_mount.c:1418:lustre_stop_osp()) Can not find osp-on-ost t32fs-MDT0000-osp-MDT0000<br/> 06:48:02:LustreError: 27939:0:(obd_mount.c:2159:server_put_super()) t32fs-MDT0000: Fail to stop osp-on-ost!<br/> 06:48:02:LustreError: 27939:0:(ldlm_request.c:1181:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway<br/> 06:48:02:LustreError: 27939:0:(ldlm_request.c:1811:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108<br/> 06:48:02:Lustre: 27939:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">&#91;sent 1350395276/real 1350395276&#93;</span> req@ffff88030f5bf800 x1415992054906891/t0(0) o251-&gt;MGC192.168.4.20@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1350395282 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1<br/> 06:48:02:Lustre: server umount t32fs-MDT0000 complete<br/> 06:48:02:LustreError: 27939:0:(obd_mount.c:2989:lustre_fill_super()) Unable to mount (-2)<br/> 06:48:02:Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity test_32a: @@@@@@ FAIL: Mounting the MDT </p> <p>I believe this has been resolved during llog and osd landings.</p> <p>One thing that is still missing is that there is no ZFS filesystem image for upgrade testing. There have been some issues hit by LLNL recently on ZFS due to small format changes of the config records, so it would be good to detect these during testing. That should be done in a separate bug, however.</p> Related LU-3198 Development Rank 1|hzuwuv: Rank (Obsolete) 2985 Severity Story Points 1.0