[LU-7123] sanity-scrub: OST shows unable to mount - Whamcloud Community JIRA

Details

Type: Bug
Resolution: Duplicate
Priority: Minor
Fix Version/s: None
Affects Version/s: Lustre 2.8.0
Labels:
None
Environment:
lustre-master build # 3175 RHEL7.1

Severity:
3
Rank (Obsolete):
9223372036854775807

Description

This issue was created by maloo for sarah_lw <wei3.liu@intel.com>

Please provide additional information about the failure here.

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/ff98b9de-5411-11e5-b12b-5254006e85c2.

OST console showen in the lustre-initialization

17:23:31:[ 9567.220671] Lustre: DEBUG MARKER: mkdir -p /mnt/ost1
17:23:31:[ 9567.529080] Lustre: 19295:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1441387392/real 1441387392]  req@ffff88005deb1e00 x1511395716989360/t0(0) o250->MGC10.1.4.162@tcp@10.1.4.162@tcp:26/25 lens 520/544 e 0 to 1 dl 1441387403 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
17:23:31:[ 9567.531521] Lustre: 19295:0:(client.c:2039:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
17:23:31:[ 9567.570845] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_OSS/P1
17:23:31:[ 9567.913313] Lustre: DEBUG MARKER: mkdir -p /mnt/ost1; mount -t lustre   		                   /dev/lvm-Role_OSS/P1 /mnt/ost1
17:23:31:[ 9568.305881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro
17:23:31:[ 9568.461183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache
17:23:31:[ 9568.480543] Lustre: Evicted from MGS (at 10.1.4.162@tcp) after server handle changed from 0xd1a6ba6be92fc62 to 0xd1a6ba6be9adf3a
17:23:31:[ 9568.482736] LustreError: 15f-b: lustre-OST0000: cannot register this server with the MGS: rc = -108. Is the MGS running?
17:23:31:[ 9568.482991] Lustre: MGC10.1.4.162@tcp: Connection restored to MGS (at 10.1.4.162@tcp)
17:23:31:[ 9568.482993] Lustre: Skipped 6 previous similar messages
17:23:31:[ 9568.485396] LustreError: 14130:0:(obd_mount_server.c:1794:server_fill_super()) Unable to start targets: -108
17:23:31:[ 9568.486579] LustreError: 14130:0:(obd_mount_server.c:1509:server_put_super()) no obd lustre-OST0000
17:23:31:[ 9568.487556] LustreError: 14130:0:(obd_mount_server.c:137:server_deregister_mount()) lustre-OST0000 not registered
17:23:31:[ 9568.620431] Lustre: server umount lustre-OST0000 complete
17:23:31:[ 9568.621660] Lustre: Skipped 3 previous similar messages
17:23:31:[ 9568.622489] LustreError: 14130:0:(obd_mount.c:1342:lustre_fill_super()) Unable to mount  (-108)

Attachments

Issue Links

duplicates

LU-7118 sanity-scrub: No sub tests failed in this test set

Resolved

Activity

[LU-7123] sanity-scrub: OST shows unable to mount

Andreas Dilger made changes - 11/Sep/15 3:36 AM

Resolution		New: Duplicate [ 3 ]
Status	Original: Open [ 1 ]	New: Resolved [ 5 ]

Andreas Dilger added a comment - 11/Sep/15 3:36 AM

Duplicate of ~~LU-7118~~.

Andreas Dilger added a comment - 11/Sep/15 3:36 AM Duplicate of LU-7118 .

Andreas Dilger made changes - 11/Sep/15 3:36 AM

Link

New: This issue duplicates ~~LU-7118~~ [ ~~LU-7118~~ ]

Sarah Liu made changes - 09/Sep/15 6:28 PM

Description	Original: This issue was created by maloo for sarah_lw <wei3.liu@intel.com> Please provide additional information about the failure here. This issue relates to the following test suite run: [https://testing.hpdd.intel.com/test_sets/ff98b9de-5411-11e5-b12b-5254006e85c2]. MDS console showen in the lustre-initialization {noformat} 17:23:00:[ 9529.925038] Lustre: DEBUG MARKER: /usr/sbin/lctl mark -----============= acceptance-small: sanity-scrub ============----- Fri Sep 4 17:22:53 UTC 2015 17:23:23:[ 9530.121771] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-scrub ============----- Fri Sep 4 17:22:53 UTC 2015 17:23:23:[ 9530.645738] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-scrub test complete, duration -o sec ======================================================= 17:22:53 $1441387373$ 17:23:23:[ 9530.809134] Lustre: DEBUG MARKER: == sanity-scrub test complete, duration -o sec ======================================================= 17:22:53 (1441387373) 17:23:23:[ 9532.667372] Lustre: DEBUG MARKER: grep -c /mnt/mds1' ' /proc/mounts 17:23:23:[ 9532.938577] Lustre: DEBUG MARKER: umount -d -f /mnt/mds1 17:23:23:[ 9533.081112] LustreError: 9058:0:(osp_sync.c:961:osp_sync_process_committed()) @@@ imp_committed = 25769804856 req@ffff88006a3fc600 x1511395704738308/t25769804884(25769804884) o6->lustre-OST0000-osc-MDT0000@10.1.4.165@tcp:28/4 lens 664/400 e 0 to 0 dl 1441387378 ref 1 fl Complete:R/4/0 rc 0/0 17:23:23:[ 9533.083806] LustreError: 9058:0:(osp_sync.c:961:osp_sync_process_committed()) Skipped 1481 previous similar messages 17:23:23:[ 9536.650523] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.1.4.165@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. 17:23:23:[ 9536.654278] LustreError: Skipped 7 previous similar messages 17:23:23:[ 9539.378194] Lustre: 10595:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1441387376/real 1441387376] req@ffff88005b459200 x1511395704739268/t0(0) o251->MGC10.1.4.162@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1441387382 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 17:23:23:[ 9539.382823] Lustre: 10595:0:(client.c:2039:ptlrpc_expire_one_request()) Skipped 7 previous similar messages 17:23:23:[ 9539.572786] Lustre: server umount lustre-MDT0000 complete 17:23:23:[ 9539.744469] Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && lctl dl \| grep ' ST ' 17:23:23:[ 9545.167612] Lustre: DEBUG MARKER: grep -c /mnt/mds1' ' /proc/mounts 17:23:23:[ 9545.516432] Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && lctl dl \| grep ' ST ' 17:23:23:[ 9545.915519] Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=lov.stripesize=1048576 --param=lov.stripecount=0 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=200000 --reformat /dev/lvm-Role_MDS/P1 17:23:23:[ 9546.340444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro 17:23:23:[ 9554.467960] Lustre: DEBUG MARKER: running=$(grep -c /mnt/mds1' ' /proc/mounts); 17:23:23:[ 9554.467960] mpts=$(mount \| grep -c /mnt/mds1' '); 17:23:23:[ 9554.467960] if [ $running -ne $mpts ]; then 17:23:23:[ 9554.467960] echo $(hostname) env are INSANE!; 17:23:23:[ 9554.467960] exit 1; 17:23:23:[ 9554.467960] fi 17:23:23:[ 9554.895067] Lustre: DEBUG MARKER: running=$(grep -c /mnt/mds1' ' /proc/mounts); 17:23:23:[ 9554.895067] mpts=$(mount \| grep -c /mnt/mds1' '); 17:23:23:[ 9554.895067] if [ $running -ne $mpts ]; then 17:23:23:[ 9554.895067] echo $(hostname) env are INSANE!; 17:23:23:[ 9554.895067] exit 1; 17:23:23:[ 9554.895067] fi 17:23:23:[ 9556.148844] Lustre: DEBUG MARKER: mkdir -p /mnt/mds1 17:23:23:[ 9556.528710] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1 17:23:23:[ 9556.836993] Lustre: DEBUG MARKER: mkdir -p /mnt/mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/mds1 17:23:23:[ 9557.023080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro 17:23:23:[ 9557.131187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache 17:23:23:[ 9557.167747] Lustre: Setting parameter lustre-MDT0000-mdtlov.lov.stripesize in log lustre-MDT0000 17:23:23:[ 9557.168569] Lustre: Skipped 5 previous similar messages 17:23:23:[ 9557.305864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space 17:23:23:[ 9557.306591] Lustre: Skipped 1 previous similar message 17:23:23:[ 9557.322246] Lustre: lustre-MDT0000: new disk, initializing 17:23:23:[ 9557.540613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400):0:mdt 17:24:04:[ 9557.984802] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u 17:24:04:[ 9558.575351] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre.enable_remote_dir=1 17:24:04:[ 9558.906996] Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2>/dev/null 17:24:04:[ 9559.234179] Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre.enable_remote_dir=1 {noformat}	New: This issue was created by maloo for sarah_lw <wei3.liu@intel.com> Please provide additional information about the failure here. This issue relates to the following test suite run: [https://testing.hpdd.intel.com/test_sets/ff98b9de-5411-11e5-b12b-5254006e85c2]. OST console showen in the lustre-initialization {noformat} 17:23:31:[ 9567.220671] Lustre: DEBUG MARKER: mkdir -p /mnt/ost1 17:23:31:[ 9567.529080] Lustre: 19295:0:(client.c:2039:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1441387392/real 1441387392] req@ffff88005deb1e00 x1511395716989360/t0(0) o250->MGC10.1.4.162@tcp@10.1.4.162@tcp:26/25 lens 520/544 e 0 to 1 dl 1441387403 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 17:23:31:[ 9567.531521] Lustre: 19295:0:(client.c:2039:ptlrpc_expire_one_request()) Skipped 3 previous similar messages 17:23:31:[ 9567.570845] Lustre: DEBUG MARKER: test -b /dev/lvm-Role_OSS/P1 17:23:31:[ 9567.913313] Lustre: DEBUG MARKER: mkdir -p /mnt/ost1; mount -t lustre /dev/lvm-Role_OSS/P1 /mnt/ost1 17:23:31:[ 9568.305881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro 17:23:31:[ 9568.461183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache 17:23:31:[ 9568.480543] Lustre: Evicted from MGS (at 10.1.4.162@tcp) after server handle changed from 0xd1a6ba6be92fc62 to 0xd1a6ba6be9adf3a 17:23:31:[ 9568.482736] LustreError: 15f-b: lustre-OST0000: cannot register this server with the MGS: rc = -108. Is the MGS running? 17:23:31:[ 9568.482991] Lustre: MGC10.1.4.162@tcp: Connection restored to MGS (at 10.1.4.162@tcp) 17:23:31:[ 9568.482993] Lustre: Skipped 6 previous similar messages 17:23:31:[ 9568.485396] LustreError: 14130:0:(obd_mount_server.c:1794:server_fill_super()) Unable to start targets: -108 17:23:31:[ 9568.486579] LustreError: 14130:0:(obd_mount_server.c:1509:server_put_super()) no obd lustre-OST0000 17:23:31:[ 9568.487556] LustreError: 14130:0:(obd_mount_server.c:137:server_deregister_mount()) lustre-OST0000 not registered 17:23:31:[ 9568.620431] Lustre: server umount lustre-OST0000 complete 17:23:31:[ 9568.621660] Lustre: Skipped 3 previous similar messages 17:23:31:[ 9568.622489] LustreError: 14130:0:(obd_mount.c:1342:lustre_fill_super()) Unable to mount (-108) {noformat}
Summary	Original: sanity-scrub: MDS unavailable	New: sanity-scrub: OST shows unable to mount

Sarah Liu made changes - 09/Sep/15 5:02 PM

Affects Version/s		New: Lustre 2.8.0 [ 11113 ]
Environment		New: lustre-master build # 3175 RHEL7.1

Maloo created issue - 09/Sep/15 5:01 PM

People

Assignee:: WC Triage

Reporter:: Maloo

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 09/Sep/15 5:01 PM

Updated:: 11/Sep/15 3:36 AM

Resolved:: 11/Sep/15 3:36 AM