Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-2201

conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT

Details

    • Bug
    • Resolution: Fixed
    • Minor
    • Lustre 2.3.0
    • None
    • None
    • 1
    • 3
    • 2985

    Description

      https://maloo.whamcloud.com/test_sets/90a83b74-872a-11e1-aa0e-525400d2bfa6

      18:40:51:Lustre: DEBUG MARKER: == conf-sanity test 32a: Upgrade (not live) == 15:40:49 (1334443249)
18:40:52:LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=off. Opts: 
18:40:52:Lustre: 20270:0:(mdt_lproc.c:410:lprocfs_wr_identity_upcall()) t32fs-MDT0000: identity upcall set to /usr/sbin/l_getidentity
18:40:53:LDISKFS-fs (loop1): mounted filesystem with ordered data mode. quota=off. Opts: 
18:40:53:Lustre: 20424:0:(ofd_fs.c:257:ofd_groups_init()) t32fs-OST0000: 1 groups initialized
18:40:53:Lustre: Setting parameter t32fs-OST0000-osc.osc.max_dirty_mb in log t32fs-client
18:40:58:LustreError: 20580:0:(osp_dev.c:242:osp_process_config()) t32fs-OST0000-osc-MDT0000: unknown param osc.max_dirty_mb=15
18:40:58:LustreError: 20580:0:(obd_config.c:730:class_add_conn()) can't add connection on non-client dev
18:40:59:Lustre: Failing over t32fs-MDT0000
18:41:05:LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=off. Opts: 
18:41:06:Lustre: 20744:0:(mdt_lproc.c:410:lprocfs_wr_identity_upcall()) t32fs-MDT0000: identity upcall set to /usr/sbin/l_getidentity
18:41:06:LustreError: 20744:0:(genops.c:317:class_newdev()) Device t32fs-OST0000-osc-MDT0000 already exists at 6, won't add
18:41:06:LustreError: 20744:0:(obd_config.c:334:class_attach()) Cannot create device t32fs-OST0000-osc-MDT0000 of type osp : -17
18:41:06:LustreError: 20744:0:(obd_config.c:1407:class_config_llog_handler()) Err -17 on cfg command:
18:41:06:Lustre:    cmd=cf001 0:t32fs-OST0000-osc-MDT0000  1:osp  2:t32fs-MDT0000-mdtlov_UUID  
18:41:06:LustreError: 15c-8: MGC172.29.3.95@tcp: The configuration from log 't32fs-MDT0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
18:41:06:LustreError: 20714:0:(obd_mount.c:984:server_start_targets()) failed to start server t32fs-MDT0000: -17
18:41:06:LustreError: 20714:0:(obd_mount.c:1280:lustre_server_mount()) Unable to start targets: -17
18:41:07:Lustre: Failing over t32fs-MDT0000
18:41:07:Lustre: Skipped 1 previous similar message
18:41:11:LustreError: 20714:0:(obd_mount.c:1949:lustre_mount()) Unable to mount (-17)
18:41:12:Lustre: DEBUG MARKER: conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-3198 conf-sanity test_32b: @@@@@@ FAIL: sha1sum verification failed

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed
          Trackbacks

          [Confluence: Sequoia Project at LLNL] Developer Work Area Sequoia Blackboard

          Activity

            [LU-2201] conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT
            adilger Andreas Dilger added a comment -

            One thing that is still missing is that there is no ZFS filesystem image for upgrade testing. There have been some issues hit by LLNL recently on ZFS due to small format changes of the config records, so it would be good to detect these during testing. That should be done in a separate bug, however.

            adilger Andreas Dilger added a comment - One thing that is still missing is that there is no ZFS filesystem image for upgrade testing. There have been some issues hit by LLNL recently on ZFS due to small format changes of the config records, so it would be good to detect these during testing. That should be done in a separate bug, however.
            liwei Li Wei (Inactive) added a comment -

            I believe this has been resolved during llog and osd landings.

            liwei Li Wei (Inactive) added a comment - I believe this has been resolved during llog and osd landings.
            xuezhao Xuezhao Liu added a comment -

            Hi,

            Hit this issue on master https://maloo.whamcloud.com/test_sessions/378a61c6-17c0-11e2-a41f-52540035b04c

            Some logs:
            06:47:49:Lustre: DEBUG MARKER: == conf-sanity test 32a: Upgrade (not live) == 06:47:42 (1350395262)
            06:47:49:Lustre: DEBUG MARKER: which tunefs.lustre
            06:47:49:Lustre: DEBUG MARKER: find /usr/lib64/lustre/tests -maxdepth 1 -name 'disk*-ldiskfs.tar.bz2'
            06:47:49:Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids
            06:47:49:Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/ost
            06:47:49:Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_1-ldiskfs.tar.bz2 -S -C /tmp/t32
            06:47:49:Lustre: DEBUG MARKER: cat /tmp/t32/commit
            06:47:49:Lustre: DEBUG MARKER: cat /tmp/t32/kernel
            06:47:50:Lustre: DEBUG MARKER: cat /tmp/t32/arch
            06:47:50:Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1
            06:47:50:Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt
            06:47:50:Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust
            06:48:02:Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt
            06:48:02:LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=off. Opts:
            06:48:02:Lustre: MGC192.168.4.20@o2ib: Reactivating import
            06:48:02:Lustre: Found index 0 for t32fs-MDT0000, updating log
            06:48:02:Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000
            06:48:02:Lustre: Modifying parameter t32fs-clilov.lov.stripesize in log t32fs-client
            06:48:02:Lustre: t32fs-MDT0000: used disk, loading
            06:48:02:LustreError: 27989:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
            06:48:02:LustreError: 27989:0:(ldlm_lib.c:418:client_obd_setup()) can't add initial connection
            06:48:02:LustreError: 27989:0:(osp_dev.c:493:osp_init0()) t32fs-OST0000-osc-MDT0000: can't setup obd: -2
            06:48:02:LustreError: 27989:0:(obd_config.c:572:class_setup()) setup t32fs-OST0000-osc-MDT0000 failed (-2)
            06:48:02:LustreError: 27989:0:(obd_config.c:1546:class_config_llog_handler()) MGC192.168.4.20@o2ib: cfg command failed: rc = -2
            06:48:02:Lustre: cmd=cf003 0:t32fs-OST0000-osc-MDT0000 1:t32fs-OST0000_UUID 2:10.10.4.12@tcp
            06:48:02:LustreError: 15c-8: MGC192.168.4.20@o2ib: The configuration from log 't32fs-MDT0000' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
            06:48:02:LustreError: 27939:0:(obd_mount.c:1850:server_start_targets()) failed to start server t32fs-MDT0000: -2
            06:48:02:LustreError: 27939:0:(obd_mount.c:2401:server_fill_super()) Unable to start targets: -2
            06:48:02:LustreError: 27939:0:(obd_mount.c:1350:lustre_disconnect_osp()) Can't end config log t32fs
            06:48:02:LustreError: 27939:0:(obd_mount.c:2114:server_put_super()) t32fs-MDT0000: failed to disconnect osp-on-ost (rc=-2)!
            06:48:02:Lustre: Failing over t32fs-MDT0000
            06:48:02:LustreError: 27939:0:(obd_mount.c:1418:lustre_stop_osp()) Can not find osp-on-ost t32fs-MDT0000-osp-MDT0000
            06:48:02:LustreError: 27939:0:(obd_mount.c:2159:server_put_super()) t32fs-MDT0000: Fail to stop osp-on-ost!
            06:48:02:LustreError: 27939:0:(ldlm_request.c:1181:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
            06:48:02:LustreError: 27939:0:(ldlm_request.c:1811:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
            06:48:02:Lustre: 27939:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1350395276/real 1350395276] req@ffff88030f5bf800 x1415992054906891/t0(0) o251->MGC192.168.4.20@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1350395282 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
            06:48:02:Lustre: server umount t32fs-MDT0000 complete
            06:48:02:LustreError: 27939:0:(obd_mount.c:2989:lustre_fill_super()) Unable to mount (-2)
            06:48:02:Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity test_32a: @@@@@@ FAIL: Mounting the MDT

            xuezhao Xuezhao Liu added a comment - Hi, Hit this issue on master https://maloo.whamcloud.com/test_sessions/378a61c6-17c0-11e2-a41f-52540035b04c Some logs: 06:47:49:Lustre: DEBUG MARKER: == conf-sanity test 32a: Upgrade (not live) == 06:47:42 (1350395262) 06:47:49:Lustre: DEBUG MARKER: which tunefs.lustre 06:47:49:Lustre: DEBUG MARKER: find /usr/lib64/lustre/tests -maxdepth 1 -name 'disk*-ldiskfs.tar.bz2' 06:47:49:Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids 06:47:49:Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/ost 06:47:49:Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_1-ldiskfs.tar.bz2 -S -C /tmp/t32 06:47:49:Lustre: DEBUG MARKER: cat /tmp/t32/commit 06:47:49:Lustre: DEBUG MARKER: cat /tmp/t32/kernel 06:47:50:Lustre: DEBUG MARKER: cat /tmp/t32/arch 06:47:50:Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 06:47:50:Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt 06:47:50:Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust 06:48:02:Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt 06:48:02:LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=off. Opts: 06:48:02:Lustre: MGC192.168.4.20@o2ib: Reactivating import 06:48:02:Lustre: Found index 0 for t32fs-MDT0000, updating log 06:48:02:Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 06:48:02:Lustre: Modifying parameter t32fs-clilov.lov.stripesize in log t32fs-client 06:48:02:Lustre: t32fs-MDT0000: used disk, loading 06:48:02:LustreError: 27989:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context 06:48:02:LustreError: 27989:0:(ldlm_lib.c:418:client_obd_setup()) can't add initial connection 06:48:02:LustreError: 27989:0:(osp_dev.c:493:osp_init0()) t32fs-OST0000-osc-MDT0000: can't setup obd: -2 06:48:02:LustreError: 27989:0:(obd_config.c:572:class_setup()) setup t32fs-OST0000-osc-MDT0000 failed (-2) 06:48:02:LustreError: 27989:0:(obd_config.c:1546:class_config_llog_handler()) MGC192.168.4.20@o2ib: cfg command failed: rc = -2 06:48:02:Lustre: cmd=cf003 0:t32fs-OST0000-osc-MDT0000 1:t32fs-OST0000_UUID 2:10.10.4.12@tcp 06:48:02:LustreError: 15c-8: MGC192.168.4.20@o2ib: The configuration from log 't32fs-MDT0000' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. 06:48:02:LustreError: 27939:0:(obd_mount.c:1850:server_start_targets()) failed to start server t32fs-MDT0000: -2 06:48:02:LustreError: 27939:0:(obd_mount.c:2401:server_fill_super()) Unable to start targets: -2 06:48:02:LustreError: 27939:0:(obd_mount.c:1350:lustre_disconnect_osp()) Can't end config log t32fs 06:48:02:LustreError: 27939:0:(obd_mount.c:2114:server_put_super()) t32fs-MDT0000: failed to disconnect osp-on-ost (rc=-2)! 06:48:02:Lustre: Failing over t32fs-MDT0000 06:48:02:LustreError: 27939:0:(obd_mount.c:1418:lustre_stop_osp()) Can not find osp-on-ost t32fs-MDT0000-osp-MDT0000 06:48:02:LustreError: 27939:0:(obd_mount.c:2159:server_put_super()) t32fs-MDT0000: Fail to stop osp-on-ost! 06:48:02:LustreError: 27939:0:(ldlm_request.c:1181:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 06:48:02:LustreError: 27939:0:(ldlm_request.c:1811:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 06:48:02:Lustre: 27939:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1350395276/real 1350395276] req@ffff88030f5bf800 x1415992054906891/t0(0) o251->MGC192.168.4.20@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1350395282 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 06:48:02:Lustre: server umount t32fs-MDT0000 complete 06:48:02:LustreError: 27939:0:(obd_mount.c:2989:lustre_fill_super()) Unable to mount (-2) 06:48:02:Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity test_32a: @@@@@@ FAIL: Mounting the MDT
            adilger Andreas Dilger added a comment -

            It appears that the conf-sanity.sh test_32 changes were lost during orion_head_sync merging. It would be great to land these on master again.

            adilger Andreas Dilger added a comment - It appears that the conf-sanity.sh test_32 changes were lost during orion_head_sync merging. It would be great to land these on master again.
            hudson Build Master (Inactive) added a comment -

            Integrated in lustre-dev » x86_64,client,el6,inkernel #340
            ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

            Result = SUCCESS
            Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
            Files :

            • lustre/tests/conf-sanity.sh
            hudson Build Master (Inactive) added a comment - Integrated in lustre-dev » x86_64,client,el6,inkernel #340 ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076) Result = SUCCESS Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076 Files : lustre/tests/conf-sanity.sh
            hudson Build Master (Inactive) added a comment -

            Integrated in lustre-dev » x86_64,server,el5,inkernel #340
            ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

            Result = SUCCESS
            Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
            Files :

            • lustre/tests/conf-sanity.sh
            hudson Build Master (Inactive) added a comment - Integrated in lustre-dev » x86_64,server,el5,inkernel #340 ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076) Result = SUCCESS Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076 Files : lustre/tests/conf-sanity.sh
            hudson Build Master (Inactive) added a comment -

            Integrated in lustre-dev » i686,client,el5,inkernel #340
            ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

            Result = SUCCESS
            Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
            Files :

            • lustre/tests/conf-sanity.sh
            hudson Build Master (Inactive) added a comment - Integrated in lustre-dev » i686,client,el5,inkernel #340 ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076) Result = SUCCESS Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076 Files : lustre/tests/conf-sanity.sh
            hudson Build Master (Inactive) added a comment -

            Integrated in lustre-dev » x86_64,server,el6,inkernel #340
            ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

            Result = SUCCESS
            Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
            Files :

            • lustre/tests/conf-sanity.sh
            hudson Build Master (Inactive) added a comment - Integrated in lustre-dev » x86_64,server,el6,inkernel #340 ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076) Result = SUCCESS Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076 Files : lustre/tests/conf-sanity.sh
            hudson Build Master (Inactive) added a comment -

            Integrated in lustre-dev » i686,server,el5,inkernel #340
            ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

            Result = SUCCESS
            Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
            Files :

            • lustre/tests/conf-sanity.sh
            hudson Build Master (Inactive) added a comment - Integrated in lustre-dev » i686,server,el5,inkernel #340 ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076) Result = SUCCESS Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076 Files : lustre/tests/conf-sanity.sh

            People

              liwei Li Wei (Inactive)
              liwei Li Wei (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: