Details
-
Bug
-
Resolution: Fixed
-
Major
-
Lustre 2.12.0
-
3
-
9223372036854775807
Description
This issue was created by maloo for sarah <sarah@whamcloud.com>
This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/44e5d976-9097-11e8-a9f7-52540065bddc
test_1a failed with the following error:
(5) Fail to mount lss_1a_0
server/client: lustre-master tag-2.11.53 ZFS DNE
VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity-lsnapshot test_1a - (5) Fail to mount lss_1a_0
Attachments
Issue Links
![[Whamcloud Community Wiki] Page](/images/icons/generic_link_16.png "[Whamcloud Community Wiki] Page"){kind=icon}
Activity
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33157/
Subject: LU-11193 llog: Do not write to read-only devices
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 12a695d5b2fea73ee5edb7a0a79d557c94cafa35
Nathaniel Clark (nclark@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/33157
Subject: LU-11193 llog: Do not write to read-only devices
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: de1e6bda2b4f04c74fe895ab88dfc9f6c329833a
First failure seems to be https://testing.whamcloud.com/test_sets/2f0b61ae-6be7-11e8-a522-52540065bddc
2.11.52.55 rev d66bba81fc641f8b3f7f8d01fe5da0d6406b7ed0
Though it it looks like all DNE on ZFS testing sanity-lsnaphot, fail with this error.
test log
== sanity-lsnapshot test 1a: mount/umount lustre snapshot ============================================ 01:59:42 (1532483982) Create lss_1a_0 CMD: trevis-3vm4 /usr/sbin/lctl snapshot_create -F lustre -n lss_1a_0 Check whether mounted (1) CMD: trevis-3vm4 /usr/sbin/lctl snapshot_list -F lustre -n lss_1a_0 -d Mount lss_1a_0 CMD: trevis-3vm4 /usr/sbin/lctl snapshot_mount -F lustre -n lss_1a_0 trevis-3vm4: mount.lustre: mount lustre-mdt2/mdt2@lss_1a_0 at /mnt/lss_1a_0_MDT0001 failed: Read-only file system trevis-3vm4: mount.lustre: mount lustre-mdt3/mdt3@lss_1a_0 at /mnt/lss_1a_0_MDT0002 failed: Read-only file system trevis-3vm4: mount.lustre: mount lustre-mdt4/mdt4@lss_1a_0 at /mnt/lss_1a_0_MDT0003 failed: Read-only file system trevis-3vm4: 3 of 12 pieces of the snapshot lss_1a_0 can't be mounted: Read-only file system CMD: trevis-3vm4 cat /var/log/lsnapshot.log Wed Jul 25 01:59:15 2018 (32588:jt_snapshot_create:1475:lustre:ssh): Create snapshot lss_0_0 successfully with comment <(null)>, barrier <enable>, timeout <30> Wed Jul 25 01:59:24 2018 (1080:jt_snapshot_create:1475:lustre:ssh): Create snapshot lss_0_1 successfully with comment <(null)>, barrier <disable>, timeout <-1> Wed Jul 25 01:59:30 2018 (2042:jt_snapshot_create:1475:lustre:ssh): Create snapshot lss_0_2 successfully with comment <This is test_0>, barrier <enable>, timeout <30> Wed Jul 25 01:59:38 2018 (2792:jt_snapshot_create:1475:lustre:ssh): Create snapshot lss_0_3 successfully with comment <Another one>, barrier <enable>, timeout <30> Wed Jul 25 01:59:40 2018 (3484:jt_snapshot_create:1468:lustre:ssh): Can't create snapshot lss_0_0 with comment <(null)> barrier <enable>, timeout <30>: -17 Wed Jul 25 01:59:49 2018 (3927:jt_snapshot_create:1475:lustre:ssh): Create snapshot lss_1a_0 successfully with comment <(null)>, barrier <enable>, timeout <30> Wed Jul 25 02:00:00 2018 (5079:snapshot_mount_target:2225:lustre:ssh): Can't execute "ssh trevis-3vm5 'PATH='/usr/lib64/qt-3.3/bin:/usr/lib64/compat-openmpi16/bin:/usr/bin:/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin' zpool import -d /dev/lvm-Role_MDS lustre-mdt2 > /dev/null 2>&1; mkdir -p /mnt/lss_1a_0_MDT0001 && mount -t lustre -o rdonly_dev lustre-mdt2/mdt2@lss_1a_0 /mnt/lss_1a_0_MDT0001'" on the target (trevis-3vm5:2:1): rc = -30 Wed Jul 25 02:00:00 2018 (5080:snapshot_mount_target:2225:lustre:ssh): Can't execute "ssh trevis-3vm4 'PATH='/usr/lib64/qt-3.3/bin:/usr/lib64/compat-openmpi16/bin:/usr/bin:/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin' zpool import -d /dev/lvm-Role_MDS lustre-mdt3 > /dev/null 2>&1; mkdir -p /mnt/lss_1a_0_MDT0002 && mount -t lustre -o rdonly_dev lustre-mdt3/mdt3@lss_1a_0 /mnt/lss_1a_0_MDT0002'" on the target (trevis-3vm4:2:2): rc = -30 Wed Jul 25 02:00:00 2018 (5081:snapshot_mount_target:2225:lustre:ssh): Can't execute "ssh trevis-3vm5 'PATH='/usr/lib64/qt-3.3/bin:/usr/lib64/compat-openmpi16/bin:/usr/bin:/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin' zpool import -d /dev/lvm-Role_MDS lustre-mdt4 > /dev/null 2>&1; mkdir -p /mnt/lss_1a_0_MDT0003 && mount -t lustre -o rdonly_dev lustre-mdt4/mdt4@lss_1a_0 /mnt/lss_1a_0_MDT0003'" on the target (trevis-3vm5:2:3): rc = -30 Wed Jul 25 02:00:08 2018 (4843:jt_snapshot_mount:2400:lustre:ssh): Can't mount snapshot lss_1a_0: -30 sanity-lsnapshot test_1a: @@@@@@ FAIL: (5) Fail to mount lss_1a_0
MDS dmesg
[43846.721497] Lustre: DEBUG MARKER: dmesg
[43847.506346] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lsnapshot test 1a: mount\/umount lustre snapshot ============================================ 01:59:42 \(1532483982\)
[43847.729856] Lustre: DEBUG MARKER: == sanity-lsnapshot test 1a: mount/umount lustre snapshot ============================================ 01:59:42 (1532483982)
[43848.191021] Lustre: DEBUG MARKER: /usr/sbin/lctl snapshot_create -F lustre -n lss_1a_0
[43853.437578] Lustre: DEBUG MARKER: /usr/sbin/lctl snapshot_list -F lustre -n lss_1a_0 -d
[43861.501650] Lustre: DEBUG MARKER: /usr/sbin/lctl snapshot_mount -F lustre -n lss_1a_0
[43863.279517] Lustre: 30b6b646-MDT0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180
[43864.230307] CPU: 1 PID: 5175 Comm: mount.lustre Kdump: loaded Tainted: P W OE ------------ 3.10.0-862.3.2.el7_lustre.x86_64 #1
[43864.232025] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[43864.232822] Call Trace:
[43864.233331] [<ffffffffac90e78e>] dump_stack+0x19/0x1b
[43864.234075] [<ffffffffc133a63a>] osd_trans_create+0x3ca/0x410 [osd_zfs]
[43864.235198] [<ffffffffc0e823a4>] llog_destroy+0x1f4/0x3f0 [obdclass]
[43864.235937] [<ffffffffc0e8ab76>] llog_cat_process_common+0x396/0x420 [obdclass]
[43864.236698] [<ffffffffc0e8bd67>] llog_cat_reverse_process_cb+0x57/0x200 [obdclass]
[43864.237722] [<ffffffffc0e8710c>] llog_reverse_process+0x38c/0xaa0 [obdclass]
[43864.238542] [<ffffffffc0e8bd10>] ? llog_cat_size_cb+0x210/0x210 [obdclass]
[43864.239292] [<ffffffffc0e87e29>] llog_cat_reverse_process+0x179/0x270 [obdclass]
[43864.240077] [<ffffffffc0e82675>] ? llog_init_handle+0xd5/0x9b0 [obdclass]
[43864.241018] [<ffffffffc0e83b88>] ? llog_open_create+0x78/0x320 [obdclass]
[43864.241961] [<ffffffffc1647800>] ? mdd_key_init+0xd0/0xd0 [mdd]
[43864.242612] [<ffffffffc164ccdc>] mdd_changelog_llog_init+0x75c/0xbc0 [mdd]
[43864.243495] [<ffffffffac26814e>] ? kvm_clock_get_cycles+0x1e/0x20
[43864.244276] [<ffffffffc164db08>] mdd_prepare+0x2b8/0x13b0 [mdd]
[43864.245190] [<ffffffffc14e0177>] mdt_prepare+0x57/0x3b0 [mdt]
[43864.246099] [<ffffffffc0ef4138>] server_start_targets+0x2298/0x2a30 [obdclass]
[43864.246996] [<ffffffffc0ec1ca0>] ? class_config_dump_handler+0x7e0/0x7e0 [obdclass]
[43864.248055] [<ffffffffc0ef59cc>] server_fill_super+0x10fc/0x18c0 [obdclass]
[43864.249055] [<ffffffffc0d79eb7>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[43864.249810] [<ffffffffc0ecb578>] lustre_fill_super+0x328/0x950 [obdclass]
[43864.250620] [<ffffffffc0ecb250>] ? lustre_common_put_super+0x270/0x270 [obdclass]
[43864.251465] [<ffffffffac41ef3f>] mount_nodev+0x4f/0xb0
[43864.252040] [<ffffffffc0ec36c8>] lustre_mount+0x38/0x60 [obdclass]
[43864.252695] [<ffffffffac41fabe>] mount_fs+0x3e/0x1b0
[43864.253277] [<ffffffffac43d097>] vfs_kern_mount+0x67/0x110
[43864.253945] [<ffffffffac43f6bf>] do_mount+0x1ef/0xce0
[43864.254571] [<ffffffffac3f77ac>] ? kmem_cache_alloc_trace+0x3c/0x200
[43864.255251] [<ffffffffac4404f3>] SyS_mount+0x83/0xd0
[43864.255810] [<ffffffffac92082f>] system_call_fastpath+0x1c/0x21
[43864.256455] [<ffffffffac92077b>] ? system_call_after_swapgs+0xc8/0x160
[43864.257216] Lustre: 5175:0:(llog_cat.c:855:llog_cat_process_common()) 30b6b646-MDD0002: can't destroy empty log [0x26:0x1:0x0]: rc = -30
[43864.262501] CPU: 1 PID: 5175 Comm: mount.lustre Kdump: loaded Tainted: P W OE ------------ 3.10.0-862.3.2.el7_lustre.x86_64 #1
[43864.263901] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[43864.264634] Call Trace:
[43864.265050] [<ffffffffac90e78e>] dump_stack+0x19/0x1b
[43864.265782] [<ffffffffc133a63a>] osd_trans_create+0x3ca/0x410 [osd_zfs]
[43864.266617] [<ffffffffc0e8472c>] llog_cancel_rec+0x1bc/0x850 [obdclass]
[43864.267378] [<ffffffffc0e8b02b>] llog_cat_cleanup+0xdb/0x380 [obdclass]
[43864.268247] [<ffffffffc0e8beb5>] llog_cat_reverse_process_cb+0x1a5/0x200 [obdclass]
[43864.269074] [<ffffffffc0e8710c>] llog_reverse_process+0x38c/0xaa0 [obdclass]
[43864.269811] [<ffffffffc0e8bd10>] ? llog_cat_size_cb+0x210/0x210 [obdclass]
[43864.270616] [<ffffffffc0e87e29>] llog_cat_reverse_process+0x179/0x270 [obdclass]
[43864.271403] [<ffffffffc0e82675>] ? llog_init_handle+0xd5/0x9b0 [obdclass]
[43864.272149] [<ffffffffc0e83b88>] ? llog_open_create+0x78/0x320 [obdclass]
[43864.272900] [<ffffffffc1647800>] ? mdd_key_init+0xd0/0xd0 [mdd]
[43864.273523] [<ffffffffc164ccdc>] mdd_changelog_llog_init+0x75c/0xbc0 [mdd]
[43864.274262] [<ffffffffac26814e>] ? kvm_clock_get_cycles+0x1e/0x20
[43864.275026] [<ffffffffc164db08>] mdd_prepare+0x2b8/0x13b0 [mdd]
[43864.275667] [<ffffffffc14e0177>] mdt_prepare+0x57/0x3b0 [mdt]
[43864.276303] [<ffffffffc0ef4138>] server_start_targets+0x2298/0x2a30 [obdclass]
[43864.277073] [<ffffffffc0ec1ca0>] ? class_config_dump_handler+0x7e0/0x7e0 [obdclass]
[43864.277930] [<ffffffffc0ef59cc>] server_fill_super+0x10fc/0x18c0 [obdclass]
[43864.278770] [<ffffffffc0d79eb7>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[43864.279487] [<ffffffffc0ecb578>] lustre_fill_super+0x328/0x950 [obdclass]
[43864.280216] [<ffffffffc0ecb250>] ? lustre_common_put_super+0x270/0x270 [obdclass]
[43864.281009] [<ffffffffac41ef3f>] mount_nodev+0x4f/0xb0
[43864.281560] [<ffffffffc0ec36c8>] lustre_mount+0x38/0x60 [obdclass]
[43864.282417] [<ffffffffac41fabe>] mount_fs+0x3e/0x1b0
[43864.283041] [<ffffffffac43d097>] vfs_kern_mount+0x67/0x110
[43864.283743] [<ffffffffac43f6bf>] do_mount+0x1ef/0xce0
[43864.284386] [<ffffffffac3f77ac>] ? kmem_cache_alloc_trace+0x3c/0x200
[43864.285062] [<ffffffffac4404f3>] SyS_mount+0x83/0xd0
[43864.285590] [<ffffffffac92082f>] system_call_fastpath+0x1c/0x21
[43864.286217] [<ffffffffac92077b>] ? system_call_after_swapgs+0xc8/0x160
[43864.287052] LustreError: 5175:0:(mdd_device.c:501:mdd_changelog_llog_init()) 30b6b646-MDD0002: changelog user init failed: rc = -30
[43864.289607] LustreError: 5175:0:(mdd_device.c:588:mdd_changelog_init()) 30b6b646-MDD0002: changelog setup during init failed: rc = -30
[43864.290911] LustreError: 5175:0:(mdd_device.c:1256:mdd_prepare()) 30b6b646-MDD0002: failed to initialize changelog: rc = -30
[43864.292143] LustreError: 5175:0:(obd_mount_server.c:1936:server_fill_super()) Unable to start targets: -30
[43864.297888] LustreError: 5248:0:(llog_osd.c:262:llog_osd_read_header()) 30b6b646-MDT0000-osp-MDT0002: bad log [0x200000402:0x1:0x0] header magic: 0x0 (expected 0x10645539)
[43864.299573] LustreError: 5248:0:(lod_dev.c:427:lod_sub_recovery_thread()) 30b6b646-MDT0000-osp-MDT0002 get update log failed: rc = -5
[43864.312054] LustreError: 5249:0:(lod_sub_object.c:934:lod_sub_prep_llog()) 30b6b646-MDT0002-mdtlov: can't get id from catalogs: rc = -5
[43864.313397] LustreError: 5249:0:(lod_dev.c:427:lod_sub_recovery_thread()) 30b6b646-MDT0001-osp-MDT0002 get update log failed: rc = -5
[43864.319552] LustreError: 5250:0:(lod_sub_object.c:934:lod_sub_prep_llog()) 30b6b646-MDT0002-mdtlov: can't get id from catalogs: rc = -5
[43864.320962] LustreError: 5250:0:(lod_dev.c:427:lod_sub_recovery_thread()) 30b6b646-MDT0003-osp-MDT0002 get update log failed: rc = -5
[43864.427856] LustreError: 5175:0:(obd_mount.c:1599:lustre_fill_super()) Unable to mount (-30)
[43870.310896] LustreError: 32242:0:(ldlm_lib.c:3235:target_bulk_io()) @@@ timeout on bulk WRITE after 6+0s req@ffff91d455464c50 x1606880080163600/t0(0) o1000->30b6b646-MDT0002-mdtlov_UUID@0@lo:106/0 lens 336/33016 e 0 to 0 dl 1532484006 ref 1 fl Interpret:/0/0 rc 0/0
[43873.074762] Lustre: DEBUG MARKER: cat /var/log/lsnapshot.log
[43873.482785] Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity-lsnapshot test_1a: @@@@@@ FAIL: \(5\) Fail to mount lss_1a_0
[43873.706267] Lustre: DEBUG MARKER: sanity-lsnapshot test_1a: @@@@@@ FAIL: (5) Fail to mount lss_1a_0
[43873.972000] Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /autotest/trevis/2018-07-24/lustre-master-el7-x86_64--full--1_4_1__3766___00d2c6de-3162-4c33-8dca-239543c570b0/sanity-lsnapshot.test_1a.debug_log.$(hostname -s).1532484009.log;
dmesg > /autotest/trevis/2018-07-24/lustre-master
The first failure seems on 2018-05-30 2.11.52.40
https://testing.whamcloud.com/test_sets/9b618996-6485-11e8-abc3-52540065bddc
Landed for 2.12