https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2200 Lustre <p>This issue was created by maloo for Oleg Drokin &lt;green@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/1e253cd6-17c2-11e2-a41f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/1e253cd6-17c2-11e2-a41f-52540035b04c</a>.</p> <p>The sub-test test_32a failed with the following error:</p> <blockquote> <p>CMD: client-20-ib mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt<br/> client-20-ib: mount.lustre: mount /dev/loop0 at /tmp/t32/mnt/mdt failed: No such file or directory<br/> client-20-ib: Is the MGS specification correct?<br/> client-20-ib: Is the filesystem name correct?<br/> client-20-ib: If upgrading, is the copied client log valid? (see upgrade docs)<br/> conf-sanity test_32a: @@@@@@ FAIL: Mounting the MDT <br/> test_32a failed with 1</p></blockquote> <p>Info required for matching: conf-sanity 32a</p> LU-2200

Test failure on test suite conf-sanity, subtest test_32a

Bug Critical Resolved Fixed Nathaniel Clark Maloo Tue, 16 Oct 2012 21:56:56 +0000 Thu, 15 Aug 2013 06:32:37 +0000 Thu, 11 Jul 2013 14:08:29 +0000 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.4.1 Lustre 2.5.0 0 11 <p>conf-sanity/32 does not work with IB.</p> <p>This bug seems to heavily affect our testing, so I am upgrading it to blocker status</p> <p>Patch to skip test <br/> <a href="http://review.whamcloud.com/4607" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4607</a></p> <p>Once the workaround patch lands, this bug needs to be kept open until the actual fix is landed to allow testing o2iblnd. This might be fixed by <a href="https://jira.whamcloud.com/browse/LU-1430" title="Changing network address without --writeconf option" class="issue-link" data-issue-key="LU-1430"><del>LU-1430</del></a> "lctl replace_nids" command (<a href="http://review.whamcloud.com/2896" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2896</a>) allowing it to replace any NIDs in the configuration. That would be a good test of both the "replace_nids" code, and would allow this upgrade test to be run again.</p> <p>Is the "exclude=t32fs-OST0000" mount option functioning correctly? I thought if it is the test should work with arbitrary NIDs.</p> <p>Landed for 2.4</p> <p>This test needs to be fixed so that it is able to run on IB. Unfortunately, skipping conf-sanity test_32<span class="error">&#91;ab&#93;</span> allowed patch <a href="http://review.whamcloud.com/4819" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4819</a> to land, but this caused <a href="https://jira.whamcloud.com/browse/LU-3198" title="conf-sanity test_32b: @@@@@@ FAIL: sha1sum verification failed" class="issue-link" data-issue-key="LU-3198"><del>LU-3198</del></a> to fail every test_32<span class="error">&#91;ab&#93;</span> that didn't run on IB.</p> <p><a href="https://maloo.whamcloud.com/test_sets/b38bfebe-a862-11e2-9f50-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b38bfebe-a862-11e2-9f50-52540035b04c</a></p> <p>Log from MDS console trying to initially mount:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>06:47:50:Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust 06:48:02:Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt 06:48:02:LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=off. Opts: 06:48:02:Lustre: MGC192.168.4.20@o2ib: Reactivating import 06:48:02:Lustre: Found index 0 for t32fs-MDT0000, updating log 06:48:02:Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 06:48:02:Lustre: Modifying parameter t32fs-clilov.lov.stripesize in log t32fs-client 06:48:02:Lustre: t32fs-MDT0000: used disk, loading 06:48:02:LustreError: 27989:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context 06:48:02:LustreError: 27989:0:(ldlm_lib.c:418:client_obd_setup()) can't add initial connection 06:48:02:LustreError: 27989:0:(osp_dev.c:493:osp_init0()) t32fs-OST0000-osc-MDT0000: can't setup obd: -2 06:48:02:LustreError: 27989:0:(obd_config.c:572:class_setup()) setup t32fs-OST0000-osc-MDT0000 failed (-2) 06:48:02:LustreError: 27989:0:(obd_config.c:1546:class_config_llog_handler()) MGC192.168.4.20@o2ib: cfg command failed: rc = -2 06:48:02:Lustre: cmd=cf003 0:t32fs-OST0000-osc-MDT0000 1:t32fs-OST0000_UUID 2:10.10.4.12@tcp 06:48:02:LustreError: 15c-8: MGC192.168.4.20@o2ib: The configuration from log 't32fs-MDT0000' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. 06:48:02:LustreError: 27939:0:(obd_mount.c:1850:server_start_targets()) failed to start server t32fs-MDT0000: -2 06:48:02:LustreError: 27939:0:(obd_mount.c:2401:server_fill_super()) Unable to start targets: -2 06:48:02:LustreError: 27939:0:(obd_mount.c:1350:lustre_disconnect_osp()) Can't end config log t32fs 06:48:02:LustreError: 27939:0:(obd_mount.c:2114:server_put_super()) t32fs-MDT0000: failed to disconnect osp-on-ost (rc=-2)! 06:48:02:Lustre: Failing over t32fs-MDT0000 06:48:02:LustreError: 27939:0:(obd_mount.c:1418:lustre_stop_osp()) Can not find osp-on-ost t32fs-MDT0000-osp-MDT0000 06:48:02:LustreError: 27939:0:(obd_mount.c:2159:server_put_super()) t32fs-MDT0000: Fail to stop osp-on-ost! 06:48:02:LustreError: 27939:0:(ldlm_request.c:1181:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 06:48:02:LustreError: 27939:0:(ldlm_request.c:1811:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 06:48:02:Lustre: 27939:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1350395276/real 1350395276] req@ffff88030f5bf800 x1415992054906891/t0(0) o251-&gt;MGC192.168.4.20@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1350395282 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 06:48:02:Lustre: server umount t32fs-MDT0000 complete 06:48:02:LustreError: 27939:0:(obd_mount.c:2989:lustre_fill_super()) Unable to mount (-2) 06:48:02:Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity test_32a: @@@@@@ FAIL: Mounting the MDT </pre> </div></div> <p>The exclude option to mount.lustre doesn't seem to be wired up.</p> <p>Correction, exclude is wired up, but not working correctly</p> <p>MDT debug_log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:00000001:0.0:1353824997.397533:0:15323:0:(obd_mount.c:2574:lmd_make_exclusion()) Process entered 00000020:00000010:0.0:1353824997.397535:0:15323:0:(obd_mount.c:2582:lmd_make_exclusion()) kmalloced 'exclude_list': 116 at ffff88032ec78cc0. 00000020:01000004:0.0:1353824997.397537:0:15323:0:(obd_mount.c:2597:lmd_make_exclusion()) ignoring exclude t32fs-O 00000020:00000010:0.0:1353824997.397538:0:15323:0:(obd_mount.c:2619:lmd_make_exclusion()) kfreed 'exclude_list': 116 at ffff88032ec78cc0. </pre> </div></div> <p><a href="http://review.whamcloud.com/6197" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6197</a></p> <p>With most recent patch (6197,5) exclude now works but I'm getting a different error (MDS Console log):<br/> <a href="https://maloo.whamcloud.com/test_sets/15ee3d6c-b3ce-11e2-b208-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/15ee3d6c-b3ce-11e2-b208-52540035b04c</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>12:40:27:Lustre: 5173:0:(obd_mount.c:837:lustre_check_exclusion()) Excluding t32fs-OST0000 (on exclusion list) 12:40:27:LustreError: 5173:0:(ldlm_lib.c:429:client_obd_setup()) can't add initial connection 12:40:27:LustreError: 5173:0:(osp_dev.c:686:osp_init0()) t32fs-OST0000-osc-MDT0000: can't setup obd: -2 12:40:27:LustreError: 5173:0:(obd_config.c:572:class_setup()) setup t32fs-OST0000-osc-MDT0000 failed (-2) 12:40:27:LustreError: 5173:0:(obd_config.c:1550:class_config_llog_handler()) MGC192.168.4.20@o2ib: cfg command failed: rc = -2 12:40:27:Lustre: cmd=cf003 0:t32fs-OST0000-osc-MDT0000 1:t32fs-OST0000_UUID 2:192.168.203.128@tcp 12:40:27:LustreError: 15c-8: MGC192.168.4.20@o2ib: The configuration from log 't32fs-MDT0000' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. 12:40:27:LustreError: 5123:0:(obd_mount_server.c:1257:server_start_targets()) failed to start server t32fs-MDT0000: -2 12:40:27:LustreError: 5123:0:(obd_mount_server.c:1699:server_fill_super()) Unable to start targets: -2 12:40:27:LustreError: 5123:0:(obd_mount_server.c:844:lustre_disconnect_lwp()) t32fs-MDT0000-lwp-MDT0000: Can't end config log t32fs-client. 12:40:27:LustreError: 5123:0:(obd_mount_server.c:1426:server_put_super()) t32fs-MDT0000: failed to disconnect lwp. (rc=-2) </pre> </div></div> <p>Adding an OST to the exlude list seems to only affect the LCFG_LOV_ADD_OBD command (turning it into LCFG_LOV_ADD_INA), the problem (from my rough reading) seems to stem from the fact that the command in question is LCFG_ADD_UUID, and the NID in question is 192.168.203.129@tcp which fails on IB.</p> <p>This seems to be the order that llog_process_thread sends LCFG commands through class_config_llog_handler().</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1) LCFG_MARKER (10) - find if excluded 2) LCFG_ADD_UUID (5) 3) LCFG_ATTACH (1) 4) LCFG_SETUP (3) - ERROR 5) LCFG_LOV_ADD_OBD (d) - Only interation with EXCLUDED flag: Change to LCFG_LOV_ADD_INA (13) </pre> </div></div> <p>Step 5 would be where the OST is excluded by adding it as inactive, but during step 4 OBD tries to setup a connection and that fails</p> <p>The call chain for LCFG_SETUP is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>class_setup obd_setup osp_device_alloc (as osp::ldto_device_alloc) osp_init0 client_obd_setup client_import_add_conn import_set_conn ptlrpc_uuid_to_connection ptrlrpc_uuid_to_peer -- Fails to find peer </pre> </div></div> <p>I opend a possible related LU. <a href="https://jira.whamcloud.com/browse/LU-3347" title="(local_storage.c:872:local_oid_storage_init()) ASSERTION( (*los)-&gt;los_last_oid &gt;= first_oid ) failed: 0 &lt; 1" class="issue-link" data-issue-key="LU-3347"><del>LU-3347</del></a> (local_storage.c:872:local_oid_storage_init()) ASSERTION( (*los)-&gt;los_last_oid &gt;= first_oid ) failed: 0 &lt; 1 </p> <p>It is a timeout error for test_32a. </p> <p>Because the tcp based nids are in the config logs, a writeconf is needed before the existing filesystem tar balls can be mounted on IB nodes. I hope that any conf-sanity/32 test that does writeconf will be able to work on IB, but the non-writeconf test (32a), I believe, has no chance of passing without major revisions to the lustre stack.</p> <p>What about using "lctl replace_nids" to fix up the configuration for IB testing?</p> <p>Testing lctl replace_nids now. Skipping non-writeconf tests will does allow 32b to be run successfully on IB nodes (See patch set 10)</p> <p>Because of <a href="https://jira.whamcloud.com/browse/LU-3357" title="lctl replace_nids loses Target UUID from llog header" class="issue-link" data-issue-key="LU-3357"><del>LU-3357</del></a>, using replace_nids causes the first lctl conf_param ($fsname-OST0000.osc.max_dirty_mb=15) to fail.</p> <p>Patch merged to master (post 2.4.51)</p> <p>reproduced again: <a href="https://maloo.whamcloud.com/test_sets/2b3a0a00-fcc8-11e2-9fdb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2b3a0a00-fcc8-11e2-9fdb-52540035b04c</a></p> <p>from the debug log of client 2: <a href="https://maloo.whamcloud.com/test_logs/b0e58a3a-fcc8-11e2-9fdb-52540035b04c/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_logs/b0e58a3a-fcc8-11e2-9fdb-52540035b04c/show_text</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>10000000:01000000:1.0:1375567124.192540:0:27905:0:(mgc_request.c:1820:mgc_process_log()) MGC192.168.4.20@o2ib: configuration from log 'lustre-sptlrpc' failed (-2). </pre> </div></div> <p>is it the same problem?</p> <p>Patch <a href="http://review.whamcloud.com/6197" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6197</a> was cherry-picked to Lustre b2_4 branch.</p> Blocker LU-3357 Related LU-1430 LU-1997 Development Rank 1|hzvajr: Rank (Obsolete) 5244 Severity