[LU-2201] conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.3.0
• Story Points: 1
• Severity: 3
• Assignee: Li Wei (Inactive)
• Reporter: Li Wei (Inactive)

Description

https://maloo.whamcloud.com/test_sets/90a83b74-872a-11e1-aa0e-525400d2bfa6

18:40:51:Lustre: DEBUG MARKER: == conf-sanity test 32a: Upgrade (not live) == 15:40:49 (1334443249)
18:40:52:LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=off. Opts: 
18:40:52:Lustre: 20270:0:(mdt_lproc.c:410:lprocfs_wr_identity_upcall()) t32fs-MDT0000: identity upcall set to /usr/sbin/l_getidentity
18:40:53:LDISKFS-fs (loop1): mounted filesystem with ordered data mode. quota=off. Opts: 
18:40:53:Lustre: 20424:0:(ofd_fs.c:257:ofd_groups_init()) t32fs-OST0000: 1 groups initialized
18:40:53:Lustre: Setting parameter t32fs-OST0000-osc.osc.max_dirty_mb in log t32fs-client
18:40:58:LustreError: 20580:0:(osp_dev.c:242:osp_process_config()) t32fs-OST0000-osc-MDT0000: unknown param osc.max_dirty_mb=15
18:40:58:LustreError: 20580:0:(obd_config.c:730:class_add_conn()) can't add connection on non-client dev
18:40:59:Lustre: Failing over t32fs-MDT0000
18:41:05:LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=off. Opts: 
18:41:06:Lustre: 20744:0:(mdt_lproc.c:410:lprocfs_wr_identity_upcall()) t32fs-MDT0000: identity upcall set to /usr/sbin/l_getidentity
18:41:06:LustreError: 20744:0:(genops.c:317:class_newdev()) Device t32fs-OST0000-osc-MDT0000 already exists at 6, won't add
18:41:06:LustreError: 20744:0:(obd_config.c:334:class_attach()) Cannot create device t32fs-OST0000-osc-MDT0000 of type osp : -17
18:41:06:LustreError: 20744:0:(obd_config.c:1407:class_config_llog_handler()) Err -17 on cfg command:
18:41:06:Lustre:    cmd=cf001 0:t32fs-OST0000-osc-MDT0000  1:osp  2:t32fs-MDT0000-mdtlov_UUID  
18:41:06:LustreError: 15c-8: MGC172.29.3.95@tcp: The configuration from log 't32fs-MDT0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
18:41:06:LustreError: 20714:0:(obd_mount.c:984:server_start_targets()) failed to start server t32fs-MDT0000: -17
18:41:06:LustreError: 20714:0:(obd_mount.c:1280:lustre_server_mount()) Unable to start targets: -17
18:41:07:Lustre: Failing over t32fs-MDT0000
18:41:07:Lustre: Skipped 1 previous similar message
18:41:11:LustreError: 20714:0:(obd_mount.c:1949:lustre_mount()) Unable to mount (-17)
18:41:12:Lustre: DEBUG MARKER: conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT

Issue Links

• is related to: LU-3198 conf-sanity test_32b: @@@@@@ FAIL: sha1sum verification failed
• Trackbacks: Developer Work Area Sequoia Blackboard

Comments

[Xuezhao Liu]

Hi,

Hit this issue on master https://maloo.whamcloud.com/test_sessions/378a61c6-17c0-11e2-a41f-52540035b04c

Some logs:
06:47:49:Lustre: DEBUG MARKER: == conf-sanity test 32a: Upgrade (not live) == 06:47:42 (1350395262)
06:47:49:Lustre: DEBUG MARKER: which tunefs.lustre
06:47:49:Lustre: DEBUG MARKER: find /usr/lib64/lustre/tests -maxdepth 1 -name 'disk*-ldiskfs.tar.bz2'
06:47:49:Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids
06:47:49:Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/ost
06:47:49:Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_1-ldiskfs.tar.bz2 -S -C /tmp/t32
06:47:49:Lustre: DEBUG MARKER: cat /tmp/t32/commit
06:47:49:Lustre: DEBUG MARKER: cat /tmp/t32/kernel
06:47:50:Lustre: DEBUG MARKER: cat /tmp/t32/arch
06:47:50:Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1
06:47:50:Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt
06:47:50:Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust
06:48:02:Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt
06:48:02:LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=off. Opts:
06:48:02:Lustre: MGC192.168.4.20@o2ib: Reactivating import
06:48:02:Lustre: Found index 0 for t32fs-MDT0000, updating log
06:48:02:Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000
06:48:02:Lustre: Modifying parameter t32fs-clilov.lov.stripesize in log t32fs-client
06:48:02:Lustre: t32fs-MDT0000: used disk, loading
06:48:02:LustreError: 27989:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
06:48:02:LustreError: 27989:0:(ldlm_lib.c:418:client_obd_setup()) can't add initial connection
06:48:02:LustreError: 27989:0:(osp_dev.c:493:osp_init0()) t32fs-OST0000-osc-MDT0000: can't setup obd: -2
06:48:02:LustreError: 27989:0:(obd_config.c:572:class_setup()) setup t32fs-OST0000-osc-MDT0000 failed (-2)
06:48:02:LustreError: 27989:0:(obd_config.c:1546:class_config_llog_handler()) MGC192.168.4.20@o2ib: cfg command failed: rc = -2
06:48:02:Lustre: cmd=cf003 0:t32fs-OST0000-osc-MDT0000 1:t32fs-OST0000_UUID 2:10.10.4.12@tcp
06:48:02:LustreError: 15c-8: MGC192.168.4.20@o2ib: The configuration from log 't32fs-MDT0000' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
06:48:02:LustreError: 27939:0:(obd_mount.c:1850:server_start_targets()) failed to start server t32fs-MDT0000: -2
06:48:02:LustreError: 27939:0:(obd_mount.c:2401:server_fill_super()) Unable to start targets: -2
06:48:02:LustreError: 27939:0:(obd_mount.c:1350:lustre_disconnect_osp()) Can't end config log t32fs
06:48:02:LustreError: 27939:0:(obd_mount.c:2114:server_put_super()) t32fs-MDT0000: failed to disconnect osp-on-ost (rc=-2)!
06:48:02:Lustre: Failing over t32fs-MDT0000
06:48:02:LustreError: 27939:0:(obd_mount.c:1418:lustre_stop_osp()) Can not find osp-on-ost t32fs-MDT0000-osp-MDT0000
06:48:02:LustreError: 27939:0:(obd_mount.c:2159:server_put_super()) t32fs-MDT0000: Fail to stop osp-on-ost!
06:48:02:LustreError: 27939:0:(ldlm_request.c:1181:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
06:48:02:LustreError: 27939:0:(ldlm_request.c:1811:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
06:48:02:Lustre: 27939:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1350395276/real 1350395276] req@ffff88030f5bf800 x1415992054906891/t0(0) o251->MGC192.168.4.20@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1350395282 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
06:48:02:Lustre: server umount t32fs-MDT0000 complete
06:48:02:LustreError: 27939:0:(obd_mount.c:2989:lustre_fill_super()) Unable to mount (-2)
06:48:02:Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity test_32a: @@@@@@ FAIL: Mounting the MDT

[Andreas Dilger]

It appears that the conf-sanity.sh test_32 changes were lost during orion_head_sync merging. It would be great to land these on master again.

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,client,el6,inkernel #340
ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

Result = SUCCESS
Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
Files :

• lustre/tests/conf-sanity.sh

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,server,el5,inkernel #340
ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

Result = SUCCESS
Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
Files :

• lustre/tests/conf-sanity.sh

[Build Master (Inactive)]

Integrated in lustre-dev » i686,client,el5,inkernel #340
ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

Result = SUCCESS
Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
Files :

• lustre/tests/conf-sanity.sh

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,server,el6,inkernel #340
ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

Result = SUCCESS
Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
Files :

• lustre/tests/conf-sanity.sh

[Build Master (Inactive)]

Integrated in lustre-dev » i686,server,el5,inkernel #340
ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

Result = SUCCESS
Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
Files :

• lustre/tests/conf-sanity.sh

[Build Master (Inactive)]

Integrated in lustre-dev » i686,client,el6,inkernel #340
ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

Result = SUCCESS
Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
Files :

• lustre/tests/conf-sanity.sh

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,client,el5,inkernel #340
ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

Result = SUCCESS
Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
Files :

• lustre/tests/conf-sanity.sh

[Li Wei (Inactive)]

The patch has landed to Orion.

[Li Wei (Inactive)]

http://review.whamcloud.com/2546