https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7616 Lustre <p>trying to create a MDT snapshot via LVM (for backups) with the following command: </p> <p>lvcreate --size 20G --snapshot --name lv-mdt0_snap /dev/vg-3015/lv-mdt0</p> <p>immediately following, errors are logged like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Dec 28 09:24:32 mds01 kernel: ------------[ cut here ]------------ Dec 28 09:24:32 mds01 kernel: WARNING: at /var/lib/jenkins/workspace/lustre-b2_7/arch/x86_64/build_type/server/distro/el6.6/ib_stack/inkernel/BUILD/BUILD/lustre-2.7.0/ldiskfs/super.c:280 ldiskfs_journal_start_sb+0xce/0xe0 [ldiskfs]() (Tainted: P W --------------- ) Dec 28 09:24:32 mds01 kernel: Hardware name: ProLiant DL380p Gen8 Dec 28 09:24:32 mds01 kernel: Modules linked in: ext2 osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgs(U) mgc(U) osd_ldiskfs(U) lquota(U) ldiskfs(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) ip6table_filter ip6_tables ebtable_nat ebtables bridge stp llc ipt_REJECT ipt_LOG xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate dm_snapshot dm_bufio dm_round_robin dm_multipath vhost_net macvtap macvlan tun kvm_intel kvm iTCO_wdt iTCO_vendor_support microcode hpilo hpwdt power_meter acpi_ipmi ipmi_si ipmi_msghandler tg3 sg serio_raw lpc_ich mfd_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core hpsa(U) qla2xxx scsi_transport_fc scsi_tgt pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_ Dec 28 09:24:32 mds01 kernel: log dm_mod [last unloaded: scsi_wait_scan] Dec 28 09:24:32 mds01 kernel: Pid: 5300, comm: mdt00_005 Tainted: P W --------------- 2.6.32-504.8.1.el6_lustre.x86_64 #1 Dec 28 09:24:32 mds01 kernel: Call Trace: Dec 28 09:24:32 mds01 kernel: [&lt;ffffffff81074df7&gt;] ? warn_slowpath_common+0x87/0xc0 Dec 28 09:24:32 mds01 kernel: [&lt;ffffffff81074e4a&gt;] ? warn_slowpath_null+0x1a/0x20 Dec 28 09:24:32 mds01 kernel: [&lt;ffffffffa0e9234e&gt;] ? ldiskfs_journal_start_sb+0xce/0xe0 [ldiskfs] Dec 28 09:24:32 mds01 kernel: [&lt;ffffffffa0e6c23a&gt;] ? ldiskfs_dirty_inode+0x2a/0x60 [ldiskfs] Dec 28 09:24:32 mds01 kernel: [&lt;ffffffffa1050966&gt;] ? osd_attr_set+0x166/0x460 [osd_ldiskfs] Dec 28 09:25:03 mds01 kernel: ------------[ cut here ]------------ </pre> </div></div> <p>The lvcreate command does return if the MDT activity is halted (unmount clients). </p> CentOS release 6.5 (Final)<br/> <br/> 2.6.32-504.8.1.el6_lustre.x86_64 #1 SMP Thu Feb 26 22:39:27 PST 2015 x86_64 x86_64 x86_64 GNU/Linux<br/> <br/> lustre: 2.7.0<br/> kernel: patchless_client<br/> build: 2.7.0-RC4--PRISTINE-2.6.32-504.8.1.el6_lustre.x86_64<br/> LU-7616

creation of LVM snapshot on ldiskfs based MDT hangs until MDT activity/use is halted

Bug Major Open Unresolved WC Triage tom crowe Tue, 29 Dec 2015 18:25:39 +0000 Wed, 27 Apr 2016 18:40:09 +0000 Lustre 2.7.0 0 11 <p>We saw a very similar issue at NASA yesterday... and we believe there may be some bugs in journal locking when creating snapshots.</p> <p>Using CentOS-6.7, lustre-2.5.3, lvm2-2.02.118-3.el6_7.4. Also note that the MDT uses an external journal device.</p> <p>The MDS became extremely sluggish, and all MDT threads went into D state, when running the following command:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lvcreate -l95%FREE -s -p r -n mdt_snap /dev/nbp9-vg/mdt9 </pre> </div></div> <p> (the command never returned, and any further lv* commands hung as well)</p> <p>In the logs...</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Apr 25 17:09:35 nbp9-mds kernel: WARNING: at /usr/src/redhat/BUILD/lustre-2.5.3/ldiskfs/<span class="code-keyword">super</span>.c:280 ldiskfs_journal_start_sb+0xce/0xe0 [ldiskfs]() (Not tainted) Apr 25 17:14:45 nbp9-mds ] Apr 25 17:14:45 nbp9-mds kernel: [&lt;ffffffffa0e5c4e5&gt;] ? mds_readpage_handle+0x15/0x20 [mdt] Apr 25 17:14:45 nbp9-mds kernel: [&lt;ffffffffa08a90c5&gt;] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Apr 25 17:14:45 nbp9-mds kernel: [&lt;ffffffffa05d18d5&gt;] ? lc_watchdog_touch+0x65/0x170 [libcfs] Apr 25 17:14:45 nbp9-mds kernel: [&lt;ffffffffa08a1a69&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Apr 25 17:14:45 nbp9-mds kernel: [&lt;ffffffffa08ab89d&gt;] ? ptlrpc_main+0xafd/0x1780 [ptlrpc] Apr 25 17:14:45 nbp9-mds kernel: [&lt;ffffffff8100c28a&gt;] ? child_rip+0xa/0x20 Apr 25 17:14:45 nbp9-mds kernel: [&lt;ffffffffa08aada0&gt;] ? ptlrpc_main+0x0/0x1780 [ptlrpc] Apr 25 17:14:45 nbp9-mds kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 Apr 25 17:14:45 nbp9-mds kernel: ---[ end trace c9f3339c0e103edf ]--- Apr 25 17:14:57 nbp9-mds kernel: WARNING: at /usr/src/redhat/BUILD/lustre-2.5.3/ldiskfs/<span class="code-keyword">super</span>.c:280 ldiskfs_journal_start_sb+0xce/0xe0 [ldiskfs]() (Tainted: G W --------------- ) Apr 25 17:14:57 nbp9-mds kernel: Hardware name: AltixXE270 Apr 25 17:14:57 nbp9-mds kernel: Modules linked in: dm_snapshot dm_bufio osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) ldiskfs(U) jbd2 lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic crc32c_intel libcfs(U) sunrpc bonding ib_ucm(U) rdma_ucm(U) rdma_cm(U) iw_cm(U) configfs ib_ipoib(U) ib_cm(U) ib_uverbs(U) ib_umad(U) dm_round_robin scsi_dh_rdac dm_multipath microcode iTCO_wdt iTCO_vendor_support i2c_i801 lpc_ich mfd_core shpchp sg igb dca ptp pps_core tcp_bic ext3 jbd sd_mod crc_t10dif sr_mod cdrom ahci pata_acpi ata_generic pata_jmicron mptfc scsi_transport_fc scsi_tgt mptsas mptscsih mptbase scsi_transport_sas mlx4_ib(U) ib_sa(U) ib_mad(U) ib_core(U) ib_addr(U) ipv6 mlx4_core(U) mlx_compat(U) memtrack(U) usb_storage radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod gru [last unloaded: scsi_wait_scan] Apr 25 17:14:57 nbp9-mds kernel: Pid: 85906, comm: mdt_rdpg00_042 Tainted: G W --------------- 2.6.32-504.30.3.el6.20151008.x86_64.lustre253 #1 Apr 25 17:14:57 nbp9-mds kernel: Call Trace: Apr 25 17:14:57 nbp9-mds kernel: [&lt;ffffffff81074127&gt;] ? warn_slowpath_common+0x87/0xc0 Apr 25 17:14:57 nbp9-mds kernel: [&lt;ffffffff8107417a&gt;] ? warn_slowpath_null+0x1a/0x20 Apr 25 17:14:57 nbp9-mds kernel: [&lt;ffffffffa0a1c33e&gt;] ? ldiskfs_journal_start_sb+0xce/0xe0 [ldiskfs] Apr 25 17:14:57 nbp9-mds kernel: [&lt;ffffffffa0d6069f&gt;] ? osd_trans_start+0x1df/0x660 [osd_ldiskfs] Apr 25 17:15:06 nbp9-mds kernel: [&lt;ffffffffa0ef3619&gt;] ? lod_trans_start+0x1b9/0x250 [lod] Apr 25 17:15:06 nbp9-mds kernel: [&lt;ffffffffa0f7af07&gt;] ? mdd_trans_start+0x17/0x20 [mdd] Apr 25 17:15:06 nbp9-mds kernel: [&lt;ffffffffa0f61ece&gt;] ? mdd_close+0x6be/0xb80 [mdd] Apr 25 17:15:06 nbp9-mds kernel: [&lt;ffffffffa0e48be9&gt;] ? mdt_mfd_close+0x4a9/0x1bc0 [mdt] Apr 25 17:15:06 nbp9-mds kernel: [&lt;ffffffffa0899525&gt;] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Apr 25 17:15:06 nbp9-mds kernel: [&lt;ffffffffa08c07f6&gt;] ? __req_capsule_get+0x166/0x710 [ptlrpc] Apr 25 17:15:06 nbp9-mds kernel: [&lt;ffffffffa089a53e&gt;] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] Apr 25 17:15:06 nbp9-mds kernel: [&lt;ffffffffa06ebf05&gt;] ? class_handle2object+0x95/0x190 [obdclass] Apr 25 17:15:06 nbp9-mds kernel: [&lt;ffffffffa0e4b6a2&gt;] ? mdt_close+0x642/0xa80 [mdt] Apr 25 17:15:06 nbp9-mds kernel: [&lt;ffffffffa0e1fada&gt;] ? mdt_handle_common+0x52a/0x1470 [mdt] Apr 25 17:15:10 nbp9-mds multipathd: nbp9_MGS_MDS: sdc - rdac checker reports path is down Apr 25 17:15:10 nbp9-mds multipathd: checker failed path 8:32 in map nbp9_MGS_MDS Apr 25 17:15:10 nbp9-mds multipathd: nbp9_MGS_MDS: remaining active paths: 1 Apr 25 17:15:10 nbp9-mds multipathd: sdd: remove path (uevent) Apr 25 17:15:10 nbp9-mds multipathd: nbp9_MGS_MDS: failed in domap <span class="code-keyword">for</span> removal of path sdd Apr 25 17:15:10 nbp9-mds multipathd: uevent trigger error Apr 25 17:15:10 nbp9-mds multipathd: sdc: remove path (uevent) Apr 25 17:15:10 nbp9-mds kernel: [&lt;ffffffffa0e5c4e5&gt;] ? mds_readpage_handle+0x15/0x20 [mdt] Apr 25 17:15:10 nbp9-mds kernel: [&lt;ffffffffa08a90c5&gt;] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] Apr 25 17:15:10 nbp9-mds kernel: [&lt;ffffffffa05d18d5&gt;] ? lc_watchdog_touch+0x65/0x170 [libcfs] Apr 25 17:15:10 nbp9-mds kernel: [&lt;ffffffffa08a1a69&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Apr 25 17:15:10 nbp9-mds kernel: [&lt;ffffffffa08ab89d&gt;] ? ptlrpc_main+0xafd/0x1780 [ptlrpc] Apr 25 17:15:10 nbp9-mds kernel: [&lt;ffffffff8100c28a&gt;] ? child_rip+0xa/0x20 Apr 25 17:15:10 nbp9-mds kernel: [&lt;ffffffffa08aada0&gt;] ? ptlrpc_main+0x0/0x1780 [ptlrpc] Apr 25 17:15:10 nbp9-mds kernel: [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 Apr 25 17:15:16 nbp9-mds multipathd: nbp9_MGS_MDS: map in use Apr 25 17:15:16 nbp9-mds multipathd: nbp9_MGS_MDS: can't flush </pre> </div></div> <p>The server had to be rebooted and e2fsck run to get it back into production.</p> <p>Not sure how closely tied this is, but... after the reboot we ran e2fsck on the MDT. The fsck was able replay the journal but during the scan found lots of unattached inodes and moved them to /lost+found. Interestingly, most of those inodes were almost sequential:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Unattached inode 584092565 Connect to /lost+found&lt;y&gt;? yes Unattached inode 584092567 Connect to /lost+found&lt;y&gt;? yes Unattached inode 584092637 Connect to /lost+found&lt;y&gt;? yes Unattached inode 584092640 Connect to /lost+found&lt;y&gt;? yes Unattached inode 584092643 Connect to /lost+found&lt;y&gt;? yes Unattached inode 584092645 Connect to /lost+found&lt;y&gt;? yes Unattached inode 584092648 Connect to /lost+found&lt;y&gt;? yes Unattached inode 584092715 Connect to /lost+found&lt;y&gt;? yes Unattached inode 584092717 Connect to /lost+found&lt;y&gt;? yes Unattached inode 584092720 Connect to /lost+found&lt;y&gt;? yes Unattached inode 584092723 Connect to /lost+found&lt;y&gt;? yes Unattached inode 584092784 </pre> </div></div> <p>Mounted as ldiskfs and saw that all but about 20 files were owned by root and had surprisingly old date stamps:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">nbp9-mds ~ # ls -l /mnt/lustre/nbp9-mdt/lost+found/ total 6816 -rw------- 1 root root 5792 Jul 26 1970 #17831719 -rw------- 1 root root 5792 Jul 26 1970 #17831723 -rw------- 1 root root 5792 Feb 4 1988 #571005637 -rw------- 1 root root 5792 Feb 4 1988 #571016437 -rw------- 1 root root 5792 Feb 4 1988 #571029133 ... -rw------- 1 root root 4832 Jul 5 1988 #584109225 -rw------- 1 root root 4832 Jul 5 1988 #584109227 -rw------- 1 root root 4832 Jul 5 1988 #584109231 -rw------- 1 root root 4832 Jul 5 1988 #584109233 -rw------- 1 root root 4832 Jul 5 1988 #584109235 nbp9-mds ~ # ls -l /mnt/lustre/nbp9-mdt/lost+found | grep -v <span class="code-quote">" root "</span> total 47328 -rwxr----- 1 dmenemen g26209 0 Feb 10 2014 #570436464 -rw-r--r-- 1 dmenemen g26209 0 Feb 10 2014 #570436465 -rwxr----- 1 dmenemen g26209 0 Feb 10 2014 #570436466 -rw-r--r-- 1 dmenemen g26209 0 Feb 10 2014 #570436467 -rw-r--r-- 1 dmenemen g26209 0 Feb 10 2014 #570436468 -rw-r--r-- 1 dmenemen g26209 0 Feb 10 2014 #570436469 ... (another dozen or so more similar files) </pre> </div></div> <p>Ran "ll_recover_lost_found_objs" on the MDT mounted as ldiskfs, and it reported results like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">Object</span> /mnt/lustre/nbp9-mdt/O/22d0c13d/d23/cb410817 restored. <span class="code-object">Object</span> /mnt/lustre/nbp9-mdt/O/22d0c13f/d27/cb41083b restored. <span class="code-object">Object</span> /mnt/lustre/nbp9-mdt/O/22d0c145/d1/cb504661 restored. <span class="code-object">Object</span> /mnt/lustre/nbp9-mdt/O/22d0c147/d27/cb50467b restored. <span class="code-object">Object</span> /mnt/lustre/nbp9-mdt/O/22d0c153/d16/cb504810 restored. <span class="code-object">Object</span> /mnt/lustre/nbp9-mdt/O/22d0c155/d20/cb504814 restored. </pre> </div></div> <p> (It got all the non-root owned files)</p> <p>I believe that the rest of the files in lost+found were probably either useless or were never pointing to user data... they have no xattrs:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># getfattr -d -m <span class="code-quote">".*"</span> -e hex /mnt/lustre/nbp9-mdt/lost+found/* </pre> </div></div> <p> (returned nothing!)</p> <p>So, we ran another e2fsck (which came back clean!) and put the filesystem back into production.</p> Related LU-8071 Development Rank 1|hzxwzb: Rank (Obsolete) 9223372036854775807 Severity