https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11193 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/44e5d976-9097-11e8-a9f7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/44e5d976-9097-11e8-a9f7-52540065bddc</a></p> <p>test_1a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(5) Fail to mount lss_1a_0 </pre> </div></div> <p>server/client: lustre-master tag-2.11.53 ZFS DNE</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-lsnapshot test_1a - (5) Fail to mount lss_1a_0</p> LU-11193

lsnapshot mount fails with DNE

Bug Major Resolved Fixed Nathaniel Clark Maloo DNE zfs Mon, 30 Jul 2018 17:15:12 +0000 Sat, 15 Dec 2018 18:06:18 +0000 Tue, 23 Oct 2018 05:36:19 +0000 Lustre 2.12.0 Lustre 2.12.0 0 5 <p>test log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity-lsnapshot test 1a: mount/umount lustre snapshot ============================================ 01:59:42 (1532483982) Create lss_1a_0 CMD: trevis-3vm4 /usr/sbin/lctl snapshot_create -F lustre -n lss_1a_0 Check whether mounted (1) CMD: trevis-3vm4 /usr/sbin/lctl snapshot_list -F lustre -n lss_1a_0 -d Mount lss_1a_0 CMD: trevis-3vm4 /usr/sbin/lctl snapshot_mount -F lustre -n lss_1a_0 trevis-3vm4: mount.lustre: mount lustre-mdt2/mdt2@lss_1a_0 at /mnt/lss_1a_0_MDT0001 failed: Read-only file system trevis-3vm4: mount.lustre: mount lustre-mdt3/mdt3@lss_1a_0 at /mnt/lss_1a_0_MDT0002 failed: Read-only file system trevis-3vm4: mount.lustre: mount lustre-mdt4/mdt4@lss_1a_0 at /mnt/lss_1a_0_MDT0003 failed: Read-only file system trevis-3vm4: 3 of 12 pieces of the snapshot lss_1a_0 can't be mounted: Read-only file system CMD: trevis-3vm4 cat /var/log/lsnapshot.log Wed Jul 25 01:59:15 2018 (32588:jt_snapshot_create:1475:lustre:ssh): Create snapshot lss_0_0 successfully with comment &lt;(null)&gt;, barrier &lt;enable&gt;, timeout &lt;30&gt; Wed Jul 25 01:59:24 2018 (1080:jt_snapshot_create:1475:lustre:ssh): Create snapshot lss_0_1 successfully with comment &lt;(null)&gt;, barrier &lt;disable&gt;, timeout &lt;-1&gt; Wed Jul 25 01:59:30 2018 (2042:jt_snapshot_create:1475:lustre:ssh): Create snapshot lss_0_2 successfully with comment &lt;This is test_0&gt;, barrier &lt;enable&gt;, timeout &lt;30&gt; Wed Jul 25 01:59:38 2018 (2792:jt_snapshot_create:1475:lustre:ssh): Create snapshot lss_0_3 successfully with comment &lt;Another one&gt;, barrier &lt;enable&gt;, timeout &lt;30&gt; Wed Jul 25 01:59:40 2018 (3484:jt_snapshot_create:1468:lustre:ssh): Can't create snapshot lss_0_0 with comment &lt;(null)&gt; barrier &lt;enable&gt;, timeout &lt;30&gt;: -17 Wed Jul 25 01:59:49 2018 (3927:jt_snapshot_create:1475:lustre:ssh): Create snapshot lss_1a_0 successfully with comment &lt;(null)&gt;, barrier &lt;enable&gt;, timeout &lt;30&gt; Wed Jul 25 02:00:00 2018 (5079:snapshot_mount_target:2225:lustre:ssh): Can't execute "ssh trevis-3vm5 'PATH='/usr/lib64/qt-3.3/bin:/usr/lib64/compat-openmpi16/bin:/usr/bin:/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin' zpool import -d /dev/lvm-Role_MDS lustre-mdt2 &gt; /dev/null 2&gt;&amp;1; mkdir -p /mnt/lss_1a_0_MDT0001 &amp;&amp; mount -t lustre -o rdonly_dev lustre-mdt2/mdt2@lss_1a_0 /mnt/lss_1a_0_MDT0001'" on the target (trevis-3vm5:2:1): rc = -30 Wed Jul 25 02:00:00 2018 (5080:snapshot_mount_target:2225:lustre:ssh): Can't execute "ssh trevis-3vm4 'PATH='/usr/lib64/qt-3.3/bin:/usr/lib64/compat-openmpi16/bin:/usr/bin:/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin' zpool import -d /dev/lvm-Role_MDS lustre-mdt3 &gt; /dev/null 2&gt;&amp;1; mkdir -p /mnt/lss_1a_0_MDT0002 &amp;&amp; mount -t lustre -o rdonly_dev lustre-mdt3/mdt3@lss_1a_0 /mnt/lss_1a_0_MDT0002'" on the target (trevis-3vm4:2:2): rc = -30 Wed Jul 25 02:00:00 2018 (5081:snapshot_mount_target:2225:lustre:ssh): Can't execute "ssh trevis-3vm5 'PATH='/usr/lib64/qt-3.3/bin:/usr/lib64/compat-openmpi16/bin:/usr/bin:/bin:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin' zpool import -d /dev/lvm-Role_MDS lustre-mdt4 &gt; /dev/null 2&gt;&amp;1; mkdir -p /mnt/lss_1a_0_MDT0003 &amp;&amp; mount -t lustre -o rdonly_dev lustre-mdt4/mdt4@lss_1a_0 /mnt/lss_1a_0_MDT0003'" on the target (trevis-3vm5:2:3): rc = -30 Wed Jul 25 02:00:08 2018 (4843:jt_snapshot_mount:2400:lustre:ssh): Can't mount snapshot lss_1a_0: -30 sanity-lsnapshot test_1a: @@@@@@ FAIL: (5) Fail to mount lss_1a_0 </pre> </div></div> <p>MDS dmesg</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[43846.721497] Lustre: DEBUG MARKER: dmesg [43847.506346] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanity-lsnapshot test 1a: mount\/umount lustre snapshot ============================================ 01:59:42 \(1532483982\) [43847.729856] Lustre: DEBUG MARKER: == sanity-lsnapshot test 1a: mount/umount lustre snapshot ============================================ 01:59:42 (1532483982) [43848.191021] Lustre: DEBUG MARKER: /usr/sbin/lctl snapshot_create -F lustre -n lss_1a_0 [43853.437578] Lustre: DEBUG MARKER: /usr/sbin/lctl snapshot_list -F lustre -n lss_1a_0 -d [43861.501650] Lustre: DEBUG MARKER: /usr/sbin/lctl snapshot_mount -F lustre -n lss_1a_0 [43863.279517] Lustre: 30b6b646-MDT0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [43864.230307] CPU: 1 PID: 5175 Comm: mount.lustre Kdump: loaded Tainted: P W OE ------------ 3.10.0-862.3.2.el7_lustre.x86_64 #1 [43864.232025] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [43864.232822] Call Trace: [43864.233331] [&lt;ffffffffac90e78e&gt;] dump_stack+0x19/0x1b [43864.234075] [&lt;ffffffffc133a63a&gt;] osd_trans_create+0x3ca/0x410 [osd_zfs] [43864.235198] [&lt;ffffffffc0e823a4&gt;] llog_destroy+0x1f4/0x3f0 [obdclass] [43864.235937] [&lt;ffffffffc0e8ab76&gt;] llog_cat_process_common+0x396/0x420 [obdclass] [43864.236698] [&lt;ffffffffc0e8bd67&gt;] llog_cat_reverse_process_cb+0x57/0x200 [obdclass] [43864.237722] [&lt;ffffffffc0e8710c&gt;] llog_reverse_process+0x38c/0xaa0 [obdclass] [43864.238542] [&lt;ffffffffc0e8bd10&gt;] ? llog_cat_size_cb+0x210/0x210 [obdclass] [43864.239292] [&lt;ffffffffc0e87e29&gt;] llog_cat_reverse_process+0x179/0x270 [obdclass] [43864.240077] [&lt;ffffffffc0e82675&gt;] ? llog_init_handle+0xd5/0x9b0 [obdclass] [43864.241018] [&lt;ffffffffc0e83b88&gt;] ? llog_open_create+0x78/0x320 [obdclass] [43864.241961] [&lt;ffffffffc1647800&gt;] ? mdd_key_init+0xd0/0xd0 [mdd] [43864.242612] [&lt;ffffffffc164ccdc&gt;] mdd_changelog_llog_init+0x75c/0xbc0 [mdd] [43864.243495] [&lt;ffffffffac26814e&gt;] ? kvm_clock_get_cycles+0x1e/0x20 [43864.244276] [&lt;ffffffffc164db08&gt;] mdd_prepare+0x2b8/0x13b0 [mdd] [43864.245190] [&lt;ffffffffc14e0177&gt;] mdt_prepare+0x57/0x3b0 [mdt] [43864.246099] [&lt;ffffffffc0ef4138&gt;] server_start_targets+0x2298/0x2a30 [obdclass] [43864.246996] [&lt;ffffffffc0ec1ca0&gt;] ? class_config_dump_handler+0x7e0/0x7e0 [obdclass] [43864.248055] [&lt;ffffffffc0ef59cc&gt;] server_fill_super+0x10fc/0x18c0 [obdclass] [43864.249055] [&lt;ffffffffc0d79eb7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [43864.249810] [&lt;ffffffffc0ecb578&gt;] lustre_fill_super+0x328/0x950 [obdclass] [43864.250620] [&lt;ffffffffc0ecb250&gt;] ? lustre_common_put_super+0x270/0x270 [obdclass] [43864.251465] [&lt;ffffffffac41ef3f&gt;] mount_nodev+0x4f/0xb0 [43864.252040] [&lt;ffffffffc0ec36c8&gt;] lustre_mount+0x38/0x60 [obdclass] [43864.252695] [&lt;ffffffffac41fabe&gt;] mount_fs+0x3e/0x1b0 [43864.253277] [&lt;ffffffffac43d097&gt;] vfs_kern_mount+0x67/0x110 [43864.253945] [&lt;ffffffffac43f6bf&gt;] do_mount+0x1ef/0xce0 [43864.254571] [&lt;ffffffffac3f77ac&gt;] ? kmem_cache_alloc_trace+0x3c/0x200 [43864.255251] [&lt;ffffffffac4404f3&gt;] SyS_mount+0x83/0xd0 [43864.255810] [&lt;ffffffffac92082f&gt;] system_call_fastpath+0x1c/0x21 [43864.256455] [&lt;ffffffffac92077b&gt;] ? system_call_after_swapgs+0xc8/0x160 [43864.257216] Lustre: 5175:0:(llog_cat.c:855:llog_cat_process_common()) 30b6b646-MDD0002: can't destroy empty log [0x26:0x1:0x0]: rc = -30 [43864.262501] CPU: 1 PID: 5175 Comm: mount.lustre Kdump: loaded Tainted: P W OE ------------ 3.10.0-862.3.2.el7_lustre.x86_64 #1 [43864.263901] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [43864.264634] Call Trace: [43864.265050] [&lt;ffffffffac90e78e&gt;] dump_stack+0x19/0x1b [43864.265782] [&lt;ffffffffc133a63a&gt;] osd_trans_create+0x3ca/0x410 [osd_zfs] [43864.266617] [&lt;ffffffffc0e8472c&gt;] llog_cancel_rec+0x1bc/0x850 [obdclass] [43864.267378] [&lt;ffffffffc0e8b02b&gt;] llog_cat_cleanup+0xdb/0x380 [obdclass] [43864.268247] [&lt;ffffffffc0e8beb5&gt;] llog_cat_reverse_process_cb+0x1a5/0x200 [obdclass] [43864.269074] [&lt;ffffffffc0e8710c&gt;] llog_reverse_process+0x38c/0xaa0 [obdclass] [43864.269811] [&lt;ffffffffc0e8bd10&gt;] ? llog_cat_size_cb+0x210/0x210 [obdclass] [43864.270616] [&lt;ffffffffc0e87e29&gt;] llog_cat_reverse_process+0x179/0x270 [obdclass] [43864.271403] [&lt;ffffffffc0e82675&gt;] ? llog_init_handle+0xd5/0x9b0 [obdclass] [43864.272149] [&lt;ffffffffc0e83b88&gt;] ? llog_open_create+0x78/0x320 [obdclass] [43864.272900] [&lt;ffffffffc1647800&gt;] ? mdd_key_init+0xd0/0xd0 [mdd] [43864.273523] [&lt;ffffffffc164ccdc&gt;] mdd_changelog_llog_init+0x75c/0xbc0 [mdd] [43864.274262] [&lt;ffffffffac26814e&gt;] ? kvm_clock_get_cycles+0x1e/0x20 [43864.275026] [&lt;ffffffffc164db08&gt;] mdd_prepare+0x2b8/0x13b0 [mdd] [43864.275667] [&lt;ffffffffc14e0177&gt;] mdt_prepare+0x57/0x3b0 [mdt] [43864.276303] [&lt;ffffffffc0ef4138&gt;] server_start_targets+0x2298/0x2a30 [obdclass] [43864.277073] [&lt;ffffffffc0ec1ca0&gt;] ? class_config_dump_handler+0x7e0/0x7e0 [obdclass] [43864.277930] [&lt;ffffffffc0ef59cc&gt;] server_fill_super+0x10fc/0x18c0 [obdclass] [43864.278770] [&lt;ffffffffc0d79eb7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [43864.279487] [&lt;ffffffffc0ecb578&gt;] lustre_fill_super+0x328/0x950 [obdclass] [43864.280216] [&lt;ffffffffc0ecb250&gt;] ? lustre_common_put_super+0x270/0x270 [obdclass] [43864.281009] [&lt;ffffffffac41ef3f&gt;] mount_nodev+0x4f/0xb0 [43864.281560] [&lt;ffffffffc0ec36c8&gt;] lustre_mount+0x38/0x60 [obdclass] [43864.282417] [&lt;ffffffffac41fabe&gt;] mount_fs+0x3e/0x1b0 [43864.283041] [&lt;ffffffffac43d097&gt;] vfs_kern_mount+0x67/0x110 [43864.283743] [&lt;ffffffffac43f6bf&gt;] do_mount+0x1ef/0xce0 [43864.284386] [&lt;ffffffffac3f77ac&gt;] ? kmem_cache_alloc_trace+0x3c/0x200 [43864.285062] [&lt;ffffffffac4404f3&gt;] SyS_mount+0x83/0xd0 [43864.285590] [&lt;ffffffffac92082f&gt;] system_call_fastpath+0x1c/0x21 [43864.286217] [&lt;ffffffffac92077b&gt;] ? system_call_after_swapgs+0xc8/0x160 [43864.287052] LustreError: 5175:0:(mdd_device.c:501:mdd_changelog_llog_init()) 30b6b646-MDD0002: changelog user init failed: rc = -30 [43864.289607] LustreError: 5175:0:(mdd_device.c:588:mdd_changelog_init()) 30b6b646-MDD0002: changelog setup during init failed: rc = -30 [43864.290911] LustreError: 5175:0:(mdd_device.c:1256:mdd_prepare()) 30b6b646-MDD0002: failed to initialize changelog: rc = -30 [43864.292143] LustreError: 5175:0:(obd_mount_server.c:1936:server_fill_super()) Unable to start targets: -30 [43864.297888] LustreError: 5248:0:(llog_osd.c:262:llog_osd_read_header()) 30b6b646-MDT0000-osp-MDT0002: bad log [0x200000402:0x1:0x0] header magic: 0x0 (expected 0x10645539) [43864.299573] LustreError: 5248:0:(lod_dev.c:427:lod_sub_recovery_thread()) 30b6b646-MDT0000-osp-MDT0002 get update log failed: rc = -5 [43864.312054] LustreError: 5249:0:(lod_sub_object.c:934:lod_sub_prep_llog()) 30b6b646-MDT0002-mdtlov: can't get id from catalogs: rc = -5 [43864.313397] LustreError: 5249:0:(lod_dev.c:427:lod_sub_recovery_thread()) 30b6b646-MDT0001-osp-MDT0002 get update log failed: rc = -5 [43864.319552] LustreError: 5250:0:(lod_sub_object.c:934:lod_sub_prep_llog()) 30b6b646-MDT0002-mdtlov: can't get id from catalogs: rc = -5 [43864.320962] LustreError: 5250:0:(lod_dev.c:427:lod_sub_recovery_thread()) 30b6b646-MDT0003-osp-MDT0002 get update log failed: rc = -5 [43864.427856] LustreError: 5175:0:(obd_mount.c:1599:lustre_fill_super()) Unable to mount (-30) [43870.310896] LustreError: 32242:0:(ldlm_lib.c:3235:target_bulk_io()) @@@ timeout on bulk WRITE after 6+0s req@ffff91d455464c50 x1606880080163600/t0(0) o1000-&gt;30b6b646-MDT0002-mdtlov_UUID@0@lo:106/0 lens 336/33016 e 0 to 0 dl 1532484006 ref 1 fl Interpret:/0/0 rc 0/0 [43873.074762] Lustre: DEBUG MARKER: cat /var/log/lsnapshot.log [43873.482785] Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity-lsnapshot test_1a: @@@@@@ FAIL: \(5\) Fail to mount lss_1a_0 [43873.706267] Lustre: DEBUG MARKER: sanity-lsnapshot test_1a: @@@@@@ FAIL: (5) Fail to mount lss_1a_0 [43873.972000] Lustre: DEBUG MARKER: /usr/sbin/lctl dk &gt; /autotest/trevis/2018-07-24/lustre-master-el7-x86_64--full--1_4_1__3766___00d2c6de-3162-4c33-8dca-239543c570b0/sanity-lsnapshot.test_1a.debug_log.$(hostname -s).1532484009.log; dmesg &gt; /autotest/trevis/2018-07-24/lustre-master </pre> </div></div> <p> </p> <p>The first failure seems on 2018-05-30 2.11.52.40<br/> <a href="https://testing.whamcloud.com/test_sets/9b618996-6485-11e8-abc3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/9b618996-6485-11e8-abc3-52540065bddc</a></p> <p>Nathaniel</p> <p>Can you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>First failure seems to be <a href="https://testing.whamcloud.com/test_sets/2f0b61ae-6be7-11e8-a522-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/2f0b61ae-6be7-11e8-a522-52540065bddc</a></p> <p>2.11.52.55 rev d66bba81fc641f8b3f7f8d01fe5da0d6406b7ed0</p> <p>Though it it looks like all DNE on ZFS testing sanity-lsnaphot, fail with this error.</p> <p>Nathaniel Clark (nclark@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33157" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33157</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11193" title="lsnapshot mount fails with DNE" class="issue-link" data-issue-key="LU-11193"><del>LU-11193</del></a> llog: Do not write to read-only devices<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: de1e6bda2b4f04c74fe895ab88dfc9f6c329833a</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33157/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33157/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11193" title="lsnapshot mount fails with DNE" class="issue-link" data-issue-key="LU-11193"><del>LU-11193</del></a> llog: Do not write to read-only devices<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 12a695d5b2fea73ee5edb7a0a79d557c94cafa35</p> <p>Landed for 2.12</p> Duplicate LU-11411 Development Rank 1|i00013: Rank (Obsolete) 9223372036854775807 Severity