[LU-2201] conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.3.0
• Type: Bug
• Priority: Minor
• Reporter: Li Wei (Inactive)
• Assignee: Li Wei (Inactive)
• Resolution: Fixed
• Story Points: 1
• Severity: 3

Description

https://maloo.whamcloud.com/test_sets/90a83b74-872a-11e1-aa0e-525400d2bfa6

18:40:51:Lustre: DEBUG MARKER: == conf-sanity test 32a: Upgrade (not live) == 15:40:49 (1334443249)
18:40:52:LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=off. Opts: 
18:40:52:Lustre: 20270:0:(mdt_lproc.c:410:lprocfs_wr_identity_upcall()) t32fs-MDT0000: identity upcall set to /usr/sbin/l_getidentity
18:40:53:LDISKFS-fs (loop1): mounted filesystem with ordered data mode. quota=off. Opts: 
18:40:53:Lustre: 20424:0:(ofd_fs.c:257:ofd_groups_init()) t32fs-OST0000: 1 groups initialized
18:40:53:Lustre: Setting parameter t32fs-OST0000-osc.osc.max_dirty_mb in log t32fs-client
18:40:58:LustreError: 20580:0:(osp_dev.c:242:osp_process_config()) t32fs-OST0000-osc-MDT0000: unknown param osc.max_dirty_mb=15
18:40:58:LustreError: 20580:0:(obd_config.c:730:class_add_conn()) can't add connection on non-client dev
18:40:59:Lustre: Failing over t32fs-MDT0000
18:41:05:LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=off. Opts: 
18:41:06:Lustre: 20744:0:(mdt_lproc.c:410:lprocfs_wr_identity_upcall()) t32fs-MDT0000: identity upcall set to /usr/sbin/l_getidentity
18:41:06:LustreError: 20744:0:(genops.c:317:class_newdev()) Device t32fs-OST0000-osc-MDT0000 already exists at 6, won't add
18:41:06:LustreError: 20744:0:(obd_config.c:334:class_attach()) Cannot create device t32fs-OST0000-osc-MDT0000 of type osp : -17
18:41:06:LustreError: 20744:0:(obd_config.c:1407:class_config_llog_handler()) Err -17 on cfg command:
18:41:06:Lustre:    cmd=cf001 0:t32fs-OST0000-osc-MDT0000  1:osp  2:t32fs-MDT0000-mdtlov_UUID  
18:41:06:LustreError: 15c-8: MGC172.29.3.95@tcp: The configuration from log 't32fs-MDT0000' failed (-17). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
18:41:06:LustreError: 20714:0:(obd_mount.c:984:server_start_targets()) failed to start server t32fs-MDT0000: -17
18:41:06:LustreError: 20714:0:(obd_mount.c:1280:lustre_server_mount()) Unable to start targets: -17
18:41:07:Lustre: Failing over t32fs-MDT0000
18:41:07:Lustre: Skipped 1 previous similar message
18:41:11:LustreError: 20714:0:(obd_mount.c:1949:lustre_mount()) Unable to mount (-17)
18:41:12:Lustre: DEBUG MARKER: conf-sanity test_32a: @@@@@@ FAIL: Remounting the MDT

Issue Links

• is related to: LU-3198 conf-sanity test_32b: @@@@@@ FAIL: sh...

Comments

[Li Wei (Inactive)]

http://review.whamcloud.com/2546

[Li Wei (Inactive)]

The patch has landed to Orion.

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,client,el5,inkernel #340
ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

Result = SUCCESS
Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
Files :

• lustre/tests/conf-sanity.sh

[Build Master (Inactive)]

Integrated in lustre-dev » i686,client,el6,inkernel #340
ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

Result = SUCCESS
Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
Files :

• lustre/tests/conf-sanity.sh

[Build Master (Inactive)]

Integrated in lustre-dev » i686,server,el5,inkernel #340
ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

Result = SUCCESS
Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
Files :

• lustre/tests/conf-sanity.sh

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,server,el6,inkernel #340
ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

Result = SUCCESS
Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
Files :

• lustre/tests/conf-sanity.sh

[Build Master (Inactive)]

Integrated in lustre-dev » i686,client,el5,inkernel #340
ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

Result = SUCCESS
Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
Files :

• lustre/tests/conf-sanity.sh

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,server,el5,inkernel #340
ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

Result = SUCCESS
Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
Files :

• lustre/tests/conf-sanity.sh

[Build Master (Inactive)]

Integrated in lustre-dev » x86_64,client,el6,inkernel #340
ORI-635 tests: conf-sanity 32a MDT remount workaround (Revision 275eb2b56a6c3d52056e95a1e2a299f2ad73f076)

Result = SUCCESS
Mikhail Pershin : 275eb2b56a6c3d52056e95a1e2a299f2ad73f076
Files :

• lustre/tests/conf-sanity.sh

[Andreas Dilger]

It appears that the conf-sanity.sh test_32 changes were lost during orion_head_sync merging. It would be great to land these on master again.

[Xuezhao Liu]

Hi,

Hit this issue on master https://maloo.whamcloud.com/test_sessions/378a61c6-17c0-11e2-a41f-52540035b04c

Some logs:
06:47:49:Lustre: DEBUG MARKER: == conf-sanity test 32a: Upgrade (not live) == 06:47:42 (1350395262)
06:47:49:Lustre: DEBUG MARKER: which tunefs.lustre
06:47:49:Lustre: DEBUG MARKER: find /usr/lib64/lustre/tests -maxdepth 1 -name 'disk*-ldiskfs.tar.bz2'
06:47:49:Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids
06:47:49:Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/ost
06:47:49:Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_1-ldiskfs.tar.bz2 -S -C /tmp/t32
06:47:49:Lustre: DEBUG MARKER: cat /tmp/t32/commit
06:47:49:Lustre: DEBUG MARKER: cat /tmp/t32/kernel
06:47:50:Lustre: DEBUG MARKER: cat /tmp/t32/arch
06:47:50:Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1
06:47:50:Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt
06:47:50:Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests/../utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lust
06:48:02:Lustre: DEBUG MARKER: mount -t lustre -o loop,exclude=t32fs-OST0000 /tmp/t32/mdt /tmp/t32/mnt/mdt
06:48:02:LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=off. Opts:
06:48:02:Lustre: MGC192.168.4.20@o2ib: Reactivating import
06:48:02:Lustre: Found index 0 for t32fs-MDT0000, updating log
06:48:02:Lustre: Modifying parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000
06:48:02:Lustre: Modifying parameter t32fs-clilov.lov.stripesize in log t32fs-client
06:48:02:Lustre: t32fs-MDT0000: used disk, loading
06:48:02:LustreError: 27989:0:(sec_config.c:1024:sptlrpc_target_local_copy_conf()) missing llog context
06:48:02:LustreError: 27989:0:(ldlm_lib.c:418:client_obd_setup()) can't add initial connection
06:48:02:LustreError: 27989:0:(osp_dev.c:493:osp_init0()) t32fs-OST0000-osc-MDT0000: can't setup obd: -2
06:48:02:LustreError: 27989:0:(obd_config.c:572:class_setup()) setup t32fs-OST0000-osc-MDT0000 failed (-2)
06:48:02:LustreError: 27989:0:(obd_config.c:1546:class_config_llog_handler()) MGC192.168.4.20@o2ib: cfg command failed: rc = -2
06:48:02:Lustre: cmd=cf003 0:t32fs-OST0000-osc-MDT0000 1:t32fs-OST0000_UUID 2:10.10.4.12@tcp
06:48:02:LustreError: 15c-8: MGC192.168.4.20@o2ib: The configuration from log 't32fs-MDT0000' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.
06:48:02:LustreError: 27939:0:(obd_mount.c:1850:server_start_targets()) failed to start server t32fs-MDT0000: -2
06:48:02:LustreError: 27939:0:(obd_mount.c:2401:server_fill_super()) Unable to start targets: -2
06:48:02:LustreError: 27939:0:(obd_mount.c:1350:lustre_disconnect_osp()) Can't end config log t32fs
06:48:02:LustreError: 27939:0:(obd_mount.c:2114:server_put_super()) t32fs-MDT0000: failed to disconnect osp-on-ost (rc=-2)!
06:48:02:Lustre: Failing over t32fs-MDT0000
06:48:02:LustreError: 27939:0:(obd_mount.c:1418:lustre_stop_osp()) Can not find osp-on-ost t32fs-MDT0000-osp-MDT0000
06:48:02:LustreError: 27939:0:(obd_mount.c:2159:server_put_super()) t32fs-MDT0000: Fail to stop osp-on-ost!
06:48:02:LustreError: 27939:0:(ldlm_request.c:1181:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway
06:48:02:LustreError: 27939:0:(ldlm_request.c:1811:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108
06:48:02:Lustre: 27939:0:(client.c:1909:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1350395276/real 1350395276] req@ffff88030f5bf800 x1415992054906891/t0(0) o251->MGC192.168.4.20@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1350395282 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
06:48:02:Lustre: server umount t32fs-MDT0000 complete
06:48:02:LustreError: 27939:0:(obd_mount.c:2989:lustre_fill_super()) Unable to mount (-2)
06:48:02:Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity test_32a: @@@@@@ FAIL: Mounting the MDT

[Li Wei (Inactive)]

I believe this has been resolved during llog and osd landings.

[Andreas Dilger]

One thing that is still missing is that there is no ZFS filesystem image for upgrade testing. There have been some issues hit by LLNL recently on ZFS due to small format changes of the config records, so it would be good to detect these during testing. That should be done in a separate bug, however.