<!-- 
RSS generated by JIRA (9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c) at Sat Feb 10 02:18:10 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>Whamcloud Community JIRA</title>
    <link>https://jira.whamcloud.com</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.4.14</version>
        <build-number>940014</build-number>
        <build-date>05-12-2023</build-date>
    </build-info>


<item>
            <title>[LU-8508] kernel:LustreError: 3842:0:(lu_object.c:1243:lu_device_fini()) ASSERTION( atomic_read(&amp;d-&gt;ld_ref) == 0 ) failed: Refcount is 1</title>
                <link>https://jira.whamcloud.com/browse/LU-8508</link>
                <project id="10000" key="LU">Lustre</project>
                    <description>&lt;p&gt;Lustre DNE2 Testing, noticed some issue with latest master builds. When mounting storage targets on servers other than ones with the MGT i get a kernel panic with the below; I have validated this is not (to the best of my ability) network, I have also tried and FE build which works and another master build (3419) which works:&lt;/p&gt;

&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt; 
[root@zlfs2-oss1 ~]# mount -vvv -t lustre /dev/nvme0n1 /mnt/MDT0000
arg[0] = /sbin/mount.lustre
arg[1] = -v
arg[2] = -o
arg[3] = rw
arg[4] = /dev/nvme0n1
arg[5] = /mnt/MDT0000
source = /dev/nvme0n1 (/dev/nvme0n1), target = /mnt/MDT0000
options = rw
checking for existing Lustre data: found
Reading CONFIGS/mountdata
Writing CONFIGS/mountdata
mounting device /dev/nvme0n1 at /mnt/MDT0000, flags=0x1000000 options=osd=osd-ldiskfs,user_xattr,errors=remount-ro,mgsnode=192.168.5.21@o2ib,virgin,update,param=mgsnode=192.168.5.21@o2ib,svname=zlfs2-MDT0000,device=/dev/nvme0n1
mount.lustre: cannot parse scheduler options for &apos;/sys/block/nvme0n1/queue/scheduler&apos;

Message from syslogd@zlfs2-oss1 at Aug 16 21:52:33 ...
 kernel:LustreError: 3842:0:(lu_object.c:1243:lu_device_fini()) ASSERTION( atomic_read(&amp;amp;d-&amp;gt;ld_ref) == 0 ) failed: Refcount is 1

Message from syslogd@zlfs2-oss1 at Aug 16 21:52:33 ...
 kernel:LustreError: 3842:0:(lu_object.c:1243:lu_device_fini()) LBUG

Message from syslogd@zlfs2-oss1 at Aug 16 21:52:33 ...
 kernel:Kernel panic - not syncing: LBUG
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt; 

&lt;p&gt;Attached is some debugging / more info. &lt;/p&gt;

&lt;p&gt;Builds Tried:&lt;br/&gt;
master b3424 - issues&lt;br/&gt;
master b3423 - issues&lt;br/&gt;
master b3420 - issues&lt;br/&gt;
master b3419 - works&lt;br/&gt;
fe 2.8 b18 - works&lt;/p&gt;</description>
                <environment>CentOS 7.2 (Kernel: Various)&lt;br/&gt;
lustre-master Build 3424 &amp;amp; 3423&lt;br/&gt;
&lt;br/&gt;
Hardware:&lt;br/&gt;
10x Lustre Servers (Intel Wildcat Pass, E5 v3 &amp;amp; 128GB)&lt;br/&gt;
Single LNET Network - o2ib0 (Omni-Path) IFS 10.1.1.0.9&lt;br/&gt;
All block devices for Lustre are NVMe, either DC P3700 or DC P3600 (Excluding MGT which is on normal SSD).&lt;br/&gt;
_____&lt;br/&gt;
&lt;br/&gt;
kernel-3.10.0-327.22.2.el7_lustre.x86_64&lt;br/&gt;
kernel-debuginfo-3.10.0-327.22.2.el7_lustre.x86_64&lt;br/&gt;
kernel-debuginfo-common-x86_64-3.10.0-327.22.2.el7_lustre.x86_64&lt;br/&gt;
kernel-devel-3.10.0-327.22.2.el7_lustre.x86_64&lt;br/&gt;
kernel-headers-3.10.0-327.22.2.el7_lustre.x86_64&lt;br/&gt;
kernel-tools-3.10.0-327.22.2.el7_lustre.x86_64&lt;br/&gt;
kernel-tools-debuginfo-3.10.0-327.22.2.el7_lustre.x86_64&lt;br/&gt;
kernel-tools-libs-3.10.0-327.22.2.el7_lustre.x86_64&lt;br/&gt;
kernel-tools-libs-devel-3.10.0-327.22.2.el7_lustre.x86_64&lt;br/&gt;
kmod-lustre-2.8.56_26_g6fad3ab-1.el7.x86_64&lt;br/&gt;
kmod-lustre-osd-ldiskfs-2.8.56_26_g6fad3ab-1.el7.x86_64&lt;br/&gt;
kmod-lustre-osd-zfs-2.8.56_26_g6fad3ab-1.el7.x86_64&lt;br/&gt;
kmod-lustre-tests-2.8.56_26_g6fad3ab-1.el7.x86_64&lt;br/&gt;
kmod-spl-3.10.0-327.22.2.el7_lustre.x86_64-0.6.5.7-1.el7.x86_64&lt;br/&gt;
kmod-spl-devel-3.10.0-327.22.2.el7_lustre.x86_64-0.6.5.7-1.el7.x86_64&lt;br/&gt;
kmod-zfs-3.10.0-327.22.2.el7_lustre.x86_64-0.6.5.7-1.el7.x86_64&lt;br/&gt;
kmod-zfs-devel-3.10.0-327.22.2.el7_lustre.x86_64-0.6.5.7-1.el7.x86_64&lt;br/&gt;
lustre-2.8.56_26_g6fad3ab-1.el7.x86_64&lt;br/&gt;
lustre-debuginfo-2.8.56_26_g6fad3ab-1.el7.x86_64&lt;br/&gt;
lustre-iokit-2.8.56_26_g6fad3ab-1.el7.x86_64&lt;br/&gt;
lustre-osd-ldiskfs-mount-2.8.56_26_g6fad3ab-1.el7.x86_64&lt;br/&gt;
lustre-osd-zfs-mount-2.8.56_26_g6fad3ab-1.el7.x86_64&lt;br/&gt;
lustre-tests-2.8.56_26_g6fad3ab-1.el7.x86_64&lt;br/&gt;
perf-3.10.0-327.22.2.el7_lustre.x86_64&lt;br/&gt;
perf-debuginfo-3.10.0-327.22.2.el7_lustre.x86_64&lt;br/&gt;
python-perf-3.10.0-327.22.2.el7_lustre.x86_64&lt;br/&gt;
python-perf-debuginfo-3.10.0-327.22.2.el7_lustre.x86_64</environment>
        <key id="38891">LU-8508</key>
            <summary>kernel:LustreError: 3842:0:(lu_object.c:1243:lu_device_fini()) ASSERTION( atomic_read(&amp;d-&gt;ld_ref) == 0 ) failed: Refcount is 1</summary>
                <type id="1" iconUrl="https://jira.whamcloud.com/secure/viewavatar?size=xsmall&amp;avatarId=11303&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.whamcloud.com/images/icons/priorities/critical.svg">Critical</priority>
                        <status id="5" iconUrl="https://jira.whamcloud.com/images/icons/statuses/resolved.png" description="A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.">Resolved</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="1">Fixed</resolution>
                                        <assignee username="kit.westneat">Kit Westneat</assignee>
                                    <reporter username="adam.j.roe">Adam Roe</reporter>
                        <labels>
                    </labels>
                <created>Tue, 16 Aug 2016 21:22:15 +0000</created>
                <updated>Fri, 16 Sep 2016 20:53:25 +0000</updated>
                            <resolved>Fri, 16 Sep 2016 03:36:15 +0000</resolved>
                                    <version>Lustre 2.8.0</version>
                                    <fixVersion>Lustre 2.9.0</fixVersion>
                                        <due></due>
                            <votes>0</votes>
                                    <watches>6</watches>
                                                                            <comments>
                            <comment id="162125" author="adam.j.roe" created="Tue, 16 Aug 2016 21:29:32 +0000"  >&lt;p&gt;Format Commands&lt;/p&gt;

&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt; 
[root@lfsmaster FORMAT_SCRIPTS]# cat zlfs2-dne2-ldiskfs_1MDT.sh
#!/bin/bash
MGSNID=192.168.5.21@o2ib0
FSNAME=zlfs2
RECORDSIZE=1024k

ssh zlfs2-mds1 zpool destroy MGT0000
ssh zlfs2-mds1 zpool create MGT0000 mirror -f /dev/sdc /dev/sdd
ssh zlfs2-mds1 mkfs.lustre --reformat --mgs --backfstype=zfs MGT0000/MGT0000
ssh zlfs2-mds1 mkfs.lustre --reformat --ost --fsname=${FSNAME} --mgsnode=${MGSNID} --index=0 /dev/nvme0n1
ssh zlfs2-mds1 mkfs.lustre --reformat --ost --fsname=${FSNAME} --mgsnode=${MGSNID} --index=1 /dev/nvme1n1
ssh zlfs2-mds1 mkfs.lustre --reformat --ost --fsname=${FSNAME} --mgsnode=${MGSNID} --index=2 /dev/nvme2n1
ssh zlfs2-mds1 mkfs.lustre --reformat --ost --fsname=${FSNAME} --mgsnode=${MGSNID} --index=3 /dev/nvme3n1
ssh zlfs2-mds2 mkfs.lustre --reformat --ost --fsname=${FSNAME} --mgsnode=${MGSNID} --index=4 /dev/nvme0n1
ssh zlfs2-mds2 mkfs.lustre --reformat --ost --fsname=${FSNAME} --mgsnode=${MGSNID} --index=5 /dev/nvme1n1
ssh zlfs2-mds2 mkfs.lustre --reformat --ost --fsname=${FSNAME} --mgsnode=${MGSNID} --index=6 /dev/nvme2n1
ssh zlfs2-mds2 mkfs.lustre --reformat --ost --fsname=${FSNAME} --mgsnode=${MGSNID} --index=7 /dev/nvme3n1
#
ssh zlfs2-oss1 mkfs.lustre --reformat --mdt --fsname=${FSNAME} --mgsnode=${MGSNID} --index=0 /dev/nvme0n1

ssh zlfs2-oss2 mkfs.lustre --reformat --mdt --fsname=${FSNAME} --mgsnode=${MGSNID} --index=1 /dev/nvme0n1

ssh zlfs2-oss3 mkfs.lustre --reformat --mdt --fsname=${FSNAME} --mgsnode=${MGSNID} --index=2 /dev/nvme0n1

ssh zlfs2-oss4 mkfs.lustre --reformat --mdt --fsname=${FSNAME} --mgsnode=${MGSNID} --index=3 /dev/nvme0n1

ssh zlfs2-oss5 mkfs.lustre --reformat --mdt --fsname=${FSNAME} --mgsnode=${MGSNID} --index=4 /dev/nvme0n1

ssh zlfs2-oss6 mkfs.lustre --reformat --mdt --fsname=${FSNAME} --mgsnode=${MGSNID} --index=5 /dev/nvme0n1

ssh zlfs2-oss7 mkfs.lustre --reformat --mdt --fsname=${FSNAME} --mgsnode=${MGSNID} --index=6 /dev/nvme0n1

ssh zlfs2-oss8 mkfs.lustre --reformat --mdt --fsname=${FSNAME} --mgsnode=${MGSNID} --index=7 /dev/nvme0n1
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Output&lt;/p&gt;

&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;[root@lfsmaster FORMAT_SCRIPTS]# ./zlfs2-dne2-ldiskfs_1MDT.sh

   Permanent disk data:
Target:     MGS
Index:      unassigned
Lustre FS:
Mount type: zfs
Flags:      0x64
              (MGS first_time update )
Persistent mount opts:
Parameters:

mkfs_cmd = zfs create -o canmount=off -o xattr=sa MGT0000/MGT0000
Writing MGT0000/MGT0000 properties
  lustre:version=1
  lustre:flags=100
  lustre:index=65535
  lustre:svname=MGS

   Permanent disk data:
Target:     zlfs2:OST0000
Index:      0
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 763097MB
formatting backing filesystem ldiskfs on /dev/nvme0n1
        target name   zlfs2:OST0000
        4k blocks     195353046
        options        -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:OST0000  -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F /dev/nvme0n1 195353046
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     zlfs2:OST0001
Index:      1
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 763097MB
formatting backing filesystem ldiskfs on /dev/nvme1n1
        target name   zlfs2:OST0001
        4k blocks     195353046
        options        -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:OST0001  -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F /dev/nvme1n1 195353046
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     zlfs2:OST0002
Index:      2
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 763097MB
formatting backing filesystem ldiskfs on /dev/nvme2n1
        target name   zlfs2:OST0002
        4k blocks     195353046
        options        -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:OST0002  -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F /dev/nvme2n1 195353046
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     zlfs2:OST0003
Index:      3
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 763097MB
formatting backing filesystem ldiskfs on /dev/nvme3n1
        target name   zlfs2:OST0003
        4k blocks     195353046
        options        -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:OST0003  -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F /dev/nvme3n1 195353046
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     zlfs2:OST0004
Index:      4
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 763097MB
formatting backing filesystem ldiskfs on /dev/nvme0n1
        target name   zlfs2:OST0004
        4k blocks     195353046
        options        -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:OST0004  -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F /dev/nvme0n1 195353046
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     zlfs2:OST0005
Index:      5
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 763097MB
formatting backing filesystem ldiskfs on /dev/nvme1n1
        target name   zlfs2:OST0005
        4k blocks     195353046
        options        -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:OST0005  -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F /dev/nvme1n1 195353046
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     zlfs2:OST0006
Index:      6
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 763097MB
formatting backing filesystem ldiskfs on /dev/nvme2n1
        target name   zlfs2:OST0006
        4k blocks     195353046
        options        -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:OST0006  -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F /dev/nvme2n1 195353046
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     zlfs2:OST0007
Index:      7
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: ,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 763097MB
formatting backing filesystem ldiskfs on /dev/nvme3n1
        target name   zlfs2:OST0007
        4k blocks     195353046
        options        -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:OST0007  -J size=400 -I 256 -i 69905 -q -O extents,uninit_bg,dir_nlink,quota,huge_file,flex_bg -G 256 -E resize=&quot;4290772992&quot;,lazy_journal_init -F /dev/nvme3n1 195353046
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     zlfs2:MDT0000
Index:      0
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 1907729MB
formatting backing filesystem ldiskfs on /dev/nvme0n1
        target name   zlfs2:MDT0000
        4k blocks     488378646
        options        -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:MDT0000  -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F /dev/nvme0n1 488378646
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     zlfs2:MDT0001
Index:      1
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 1907729MB
formatting backing filesystem ldiskfs on /dev/nvme0n1
        target name   zlfs2:MDT0001
        4k blocks     488378646
        options        -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:MDT0001  -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F /dev/nvme0n1 488378646
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     zlfs2:MDT0002
Index:      2
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 1907729MB
formatting backing filesystem ldiskfs on /dev/nvme0n1
        target name   zlfs2:MDT0002
        4k blocks     488378646
        options        -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:MDT0002  -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F /dev/nvme0n1 488378646
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     zlfs2:MDT0003
Index:      3
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 1907729MB
formatting backing filesystem ldiskfs on /dev/nvme0n1
        target name   zlfs2:MDT0003
        4k blocks     488378646
        options        -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:MDT0003  -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F /dev/nvme0n1 488378646
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     zlfs2:MDT0004
Index:      4
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 1907729MB
formatting backing filesystem ldiskfs on /dev/nvme0n1
        target name   zlfs2:MDT0004
        4k blocks     488378646
        options        -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:MDT0004  -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F /dev/nvme0n1 488378646
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     zlfs2:MDT0005
Index:      5
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 1907729MB
formatting backing filesystem ldiskfs on /dev/nvme0n1
        target name   zlfs2:MDT0005
        4k blocks     488378646
        options        -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:MDT0005  -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F /dev/nvme0n1 488378646
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     zlfs2:MDT0006
Index:      6
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 1907729MB
formatting backing filesystem ldiskfs on /dev/nvme0n1
        target name   zlfs2:MDT0006
        4k blocks     488378646
        options        -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:MDT0006  -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F /dev/nvme0n1 488378646
Writing CONFIGS/mountdata

   Permanent disk data:
Target:     zlfs2:MDT0007
Index:      7
Lustre FS:  zlfs2
Mount type: ldiskfs
Flags:      0x61
              (MDT first_time update )
Persistent mount opts: user_xattr,errors=remount-ro
Parameters: mgsnode=192.168.5.21@o2ib

device size = 1907729MB
formatting backing filesystem ldiskfs on /dev/nvme0n1
        target name   zlfs2:MDT0007
        4k blocks     488378646
        options        -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F
mkfs_cmd = mke2fs -j -b 4096 -L zlfs2:MDT0007  -J size=4096 -I 512 -i 2048 -q -O dirdata,uninit_bg,^extents,dir_nlink,quota,huge_file,flex_bg -E lazy_journal_init -F /dev/nvme0n1 488378646
Writing CONFIGS/mountdata
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;</comment>
                            <comment id="162126" author="adam.j.roe" created="Tue, 16 Aug 2016 21:30:34 +0000"  >&lt;p&gt;/var/log/messages&lt;/p&gt;

&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;[root@zlfs2-oss1 ~]# tail -f /var/log/messages
Aug 16 21:51:43 zlfs2-oss1 kernel: Lustre: Failing over zlfs2-MDT0000
Aug 16 21:51:49 zlfs2-oss1 kernel: Lustre: zlfs2-MDT0000: Not available for connect from 192.168.5.21@o2ib (stopping)
Aug 16 21:51:49 zlfs2-oss1 kernel: LustreError: 3732:0:(import.c:338:ptlrpc_invalidate_import()) zlfs2-OST0004_UUID: rc = -110 waiting for callback (1 != 0)
Aug 16 21:51:49 zlfs2-oss1 kernel: LustreError: 3732:0:(import.c:364:ptlrpc_invalidate_import()) @@@ still on sending list  req@ffff881fe3818300 x1542853838766384/t0(0) o8-&amp;gt;zlfs2-OST0004-osc-MDT0000@192.168.5.22@o2ib:28/4 lens 520/544 e 0 to 0 dl 1471380708 ref 2 fl UnregRPC:EN/0/ffffffff rc -5/-1
Aug 16 21:51:49 zlfs2-oss1 kernel: LustreError: 3732:0:(import.c:378:ptlrpc_invalidate_import()) zlfs2-OST0004_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting them to error out.
Aug 16 21:52:14 zlfs2-oss1 kernel: Lustre: zlfs2-MDT0000: Not available for connect from 192.168.5.21@o2ib (stopping)
Aug 16 21:52:14 zlfs2-oss1 kernel: Lustre: Skipped 3 previous similar messages
Aug 16 21:52:16 zlfs2-oss1 systemd-logind: New session 6 of user root.
Aug 16 21:52:16 zlfs2-oss1 systemd: Started Session 6 of user root.
Aug 16 21:52:16 zlfs2-oss1 systemd: Starting Session 6 of user root.
Aug 16 21:52:33 zlfs2-oss1 kernel: LustreError: 3842:0:(lu_object.c:1243:lu_device_fini()) ASSERTION( atomic_read(&amp;amp;d-&amp;gt;ld_ref) == 0 ) failed: Refcount is 1

Message from syslogd@zlfs2-oss1 at Aug 16 21:52:33 ...
 kernel:LustreError: 3842:0:(lu_object.c:1243:lu_device_fini()) ASSERTION( atomic_read(&amp;amp;d-&amp;gt;ld_ref) == 0 ) failed: Refcount is 1
Aug 16 21:52:33 zlfs2-oss1 kernel: LustreError: 3842:0:(lu_object.c:1243:lu_device_fini()) LBUG
Aug 16 21:52:33 zlfs2-oss1 kernel: Pid: 3842, comm: osp-syn-4-0
Aug 16 21:52:33 zlfs2-oss1 kernel: #012Call Trace:
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0ba27d3&amp;gt;] libcfs_debug_dumpstack+0x53/0x80 [libcfs]
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0ba2d75&amp;gt;] lbug_with_loc+0x45/0xc0 [libcfs]
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0cdd9d8&amp;gt;] lu_device_fini+0xb8/0xc0 [obdclass]
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0cc2b42&amp;gt;] ls_device_put+0x82/0x2a0 [obdclass]
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0cc2e3d&amp;gt;] local_oid_storage_fini+0xdd/0x210 [obdclass]
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0c9fa1c&amp;gt;] llog_osd_cleanup+0x3c/0x50 [obdclass]
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0c9cc23&amp;gt;] __llog_ctxt_put+0x93/0x140 [obdclass]
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0c9d113&amp;gt;] llog_cleanup+0xc3/0x490 [obdclass]
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0c9381d&amp;gt;] ? llog_handle_put+0x2d/0x70 [obdclass]
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0c938b9&amp;gt;] ? llog_close+0x59/0x1a0 [obdclass]
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffffa166830b&amp;gt;] osp_sync_thread+0x46b/0x990 [osp]

Message from syslogd@zlfs2-oss1 at Aug 16 21:52:33 ...
 kernel:LustreError: 3842:0:(lu_object.c:1243:lu_device_fini()) LBUG
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffffa1667ea0&amp;gt;] ? osp_sync_thread+0x0/0x990 [osp]
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffff810a5aef&amp;gt;] kthread+0xcf/0xe0
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffff810a5a20&amp;gt;] ? kthread+0x0/0xe0
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffff816469d8&amp;gt;] ret_from_fork+0x58/0x90
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffff810a5a20&amp;gt;] ? kthread+0x0/0xe0
Aug 16 21:52:33 zlfs2-oss1 kernel:
Aug 16 21:52:33 zlfs2-oss1 kernel: Kernel panic - not syncing: LBUG

Message from syslogd@zlfs2-oss1 at Aug 16 21:52:33 ...
 kernel:Kernel panic - not syncing: LBUG
Aug 16 21:52:33 zlfs2-oss1 kernel: CPU: 16 PID: 3842 Comm: osp-syn-4-0 Tainted: P           OE  ------------   3.10.0-327.22.2.el7_lustre.x86_64 #1
Aug 16 21:52:33 zlfs2-oss1 kernel: Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS SE5C610.86B.01.01.0016.033120161139 03/31/2016
Aug 16 21:52:33 zlfs2-oss1 kernel: ffffffffa0bbfdef 000000008386d797 ffff880ff7ba7bf8 ffffffff81636324
Aug 16 21:52:33 zlfs2-oss1 kernel: ffff880ff7ba7c78 ffffffff8162fb9a ffffffff00000008 ffff880ff7ba7c88
Aug 16 21:52:33 zlfs2-oss1 kernel: ffff880ff7ba7c28 000000008386d797 ffffffffa0d0d1d5 0000000000000000
Aug 16 21:52:33 zlfs2-oss1 kernel: Call Trace:
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffff81636324&amp;gt;] dump_stack+0x19/0x1b
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffff8162fb9a&amp;gt;] panic+0xd8/0x1e7
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0ba2ddb&amp;gt;] lbug_with_loc+0xab/0xc0 [libcfs]
Aug 16 21:52:33 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0cdd9d8&amp;gt;] lu_device_fini+0xb8/0xc0 [obdclass]
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;</comment>
                            <comment id="162127" author="pjones" created="Tue, 16 Aug 2016 21:31:08 +0000"  >&lt;p&gt;Fan Yong&lt;/p&gt;

&lt;p&gt;What do you advise here?&lt;/p&gt;

&lt;p&gt;Peter&lt;/p&gt;</comment>
                            <comment id="162155" author="adam.j.roe" created="Wed, 17 Aug 2016 07:49:45 +0000"  >&lt;p&gt;I have just tried this with Build 3420, and I get the below, I then re provisioned with Build 3419 to validate and all is working fine - I would suggest one of the b3420 patches is causing this.&lt;/p&gt;

&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;[root@zlfs2-oss1 ~]# mount -vvv -t lustre /dev/nvme0n1 /mnt/MDT0000
arg[0] = /sbin/mount.lustre
arg[1] = -v
arg[2] = -o
arg[3] = rw
arg[4] = /dev/nvme0n1
arg[5] = /mnt/MDT0000
source = /dev/nvme0n1 (/dev/nvme0n1), target = /mnt/MDT0000
options = rw
checking for existing Lustre data: found
Reading CONFIGS/mountdata
Writing CONFIGS/mountdata
mounting device /dev/nvme0n1 at /mnt/MDT0000, flags=0x1000000 options=osd=osd-ldiskfs,user_xattr,errors=remount-ro,mgsnode=192.168.5.21@o2ib,virgin,update,param=mgsnode=192.168.5.21@o2ib,svname=zlfs2-MDT0000,device=/dev/nvme0n1
mount.lustre: cannot parse scheduler options for &apos;/sys/block/nvme0n1/queue/scheduler&apos;

Message from syslogd@zlfs2-oss1 at Aug 17 08:06:45 ...
 kernel:LustreError: 3895:0:(lu_object.c:1243:lu_device_fini()) ASSERTION( atomic_read(&amp;amp;d-&amp;gt;ld_ref) == 0 ) failed: Refcount is 1

Message from syslogd@zlfs2-oss1 at Aug 17 08:06:45 ...
 kernel:LustreError: 3895:0:(lu_object.c:1243:lu_device_fini()) LBUG
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;/var/log/messages + syslogd&lt;/p&gt;

&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;Aug 17 08:06:44 zlfs2-oss1 kernel: LDISKFS-fs (nvme0n1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Aug 17 08:06:45 zlfs2-oss1 kernel: LDISKFS-fs (nvme0n1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache
Aug 17 08:06:45 zlfs2-oss1 kernel: LustreError: 3788:0:(mgc_request.c:257:do_config_log_add()) MGC192.168.5.21@o2ib: failed processing log, type 4: rc = -22
Aug 17 08:06:45 zlfs2-oss1 kernel: Lustre: ctl-zlfs2-MDT0000: No data found on store. Initialize space
Aug 17 08:06:45 zlfs2-oss1 kernel: Lustre: zlfs2-MDT0000: new disk, initializing
Aug 17 08:06:45 zlfs2-oss1 kernel: Lustre: ctl-zlfs2-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
Aug 17 08:06:45 zlfs2-oss1 kernel: LustreError: 3788:0:(nodemap_storage.c:368:nodemap_idx_nodemap_add_update()) cannot add nodemap config to non-existing MGS.
Aug 17 08:06:45 zlfs2-oss1 kernel: LustreError: 3788:0:(nodemap_storage.c:1313:nodemap_fs_init()) zlfs2-MDD0000: error loading nodemap config file, file must be removed via ldiskfs: rc = -22
Aug 17 08:06:45 zlfs2-oss1 kernel: LustreError: 3788:0:(obd_mount_server.c:1844:server_fill_super()) Unable to start targets: -22
Aug 17 08:06:45 zlfs2-oss1 kernel: Lustre: Failing over zlfs2-MDT0000
Aug 17 08:06:45 zlfs2-oss1 kernel: LustreError: 3895:0:(lu_object.c:1243:lu_device_fini()) ASSERTION( atomic_read(&amp;amp;d-&amp;gt;ld_ref) == 0 ) failed: Refcount is 1
Aug 17 08:06:45 zlfs2-oss1 kernel: LustreError: 3895:0:(lu_object.c:1243:lu_device_fini()) LBUG
Aug 17 08:06:45 zlfs2-oss1 kernel: Pid: 3895, comm: osp-syn-3-0

Message from syslogd@zlfs2-oss1 at Aug 17 08:06:45 ...
 kernel:LustreError: 3895:0:(lu_object.c:1243:lu_device_fini()) ASSERTION( atomic_read(&amp;amp;d-&amp;gt;ld_ref) == 0 ) failed: Refcount is 1

Message from syslogd@zlfs2-oss1 at Aug 17 08:06:45 ...
 kernel:LustreError: 3895:0:(lu_object.c:1243:lu_device_fini()) LBUG
Aug 17 08:06:45 zlfs2-oss1 kernel: #012Call Trace:
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0c017d3&amp;gt;] libcfs_debug_dumpstack+0x53/0x80 [libcfs]
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0c01d75&amp;gt;] lbug_with_loc+0x45/0xc0 [libcfs]
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0d239d8&amp;gt;] lu_device_fini+0xb8/0xc0 [obdclass]
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0d08b42&amp;gt;] ls_device_put+0x82/0x2a0 [obdclass]
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0d08e3d&amp;gt;] local_oid_storage_fini+0xdd/0x210 [obdclass]
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0ce5a1c&amp;gt;] llog_osd_cleanup+0x3c/0x50 [obdclass]
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0ce2c23&amp;gt;] __llog_ctxt_put+0x93/0x140 [obdclass]
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0ce3113&amp;gt;] llog_cleanup+0xc3/0x490 [obdclass]
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0cd981d&amp;gt;] ? llog_handle_put+0x2d/0x70 [obdclass]
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffffa0cd98b9&amp;gt;] ? llog_close+0x59/0x1a0 [obdclass]
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffffa166430b&amp;gt;] osp_sync_thread+0x46b/0x990 [osp]
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffffa1663ea0&amp;gt;] ? osp_sync_thread+0x0/0x990 [osp]
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffff810a5aef&amp;gt;] kthread+0xcf/0xe0
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffff810a5a20&amp;gt;] ? kthread+0x0/0xe0
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffff816469d8&amp;gt;] ret_from_fork+0x58/0x90
Aug 17 08:06:45 zlfs2-oss1 kernel: [&amp;lt;ffffffff810a5a20&amp;gt;] ? kthread+0x0/0xe0
Aug 17 08:06:45 zlfs2-oss1 kernel:
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;</comment>
                            <comment id="162158" author="gerrit" created="Wed, 17 Aug 2016 09:13:10 +0000"  >&lt;p&gt;Fan Yong (fan.yong@intel.com) uploaded a new patch: &lt;a href=&quot;http://review.whamcloud.com/21961&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/21961&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-8508&quot; title=&quot;kernel:LustreError: 3842:0:(lu_object.c:1243:lu_device_fini()) ASSERTION( atomic_read(&amp;amp;d-&amp;gt;ld_ref) == 0 ) failed: Refcount is 1&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-8508&quot;&gt;&lt;del&gt;LU-8508&lt;/del&gt;&lt;/a&gt; nodemap: release nodemap index obj if fail to register&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: 1f7fce9911cd5e52280ce959e585be19720b6f2b&lt;/p&gt;</comment>
                            <comment id="162247" author="adam.j.roe" created="Wed, 17 Aug 2016 17:31:30 +0000"  >&lt;p&gt;Okay I tested the build - to update. It didn&apos;t crash the system, but it fails to mount the target. See below:&lt;/p&gt;

&lt;p&gt;Some stuff mounts, some wont - strange behavior going on, will report back soon.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Mount&lt;/b&gt;&lt;/p&gt;

&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;[root@lfsmaster FORMAT_SCRIPTS]# ssh zlfs2-mds2
Last login: Wed Aug 17 08:46:40 2016 from 10.10.100.99
[root@zlfs2-mds2 ~]# mount -vvv -t lustre  /dev/nvme0n1 /mnt/OST0004
arg[0] = /sbin/mount.lustre
arg[1] = -v
arg[2] = -o
arg[3] = rw
arg[4] = /dev/nvme0n1
arg[5] = /mnt/OST0004
source = /dev/nvme0n1 (/dev/nvme0n1), target = /mnt/OST0004
options = rw
checking for existing Lustre data: found
Reading CONFIGS/mountdata
Writing CONFIGS/mountdata
mounting device /dev/nvme0n1 at /mnt/OST0004, flags=0x1000000 options=osd=osd-ldiskfs,,errors=remount-ro,mgsnode=192.168.5.21@o2ib,virgin,update,param=mgsnode=192.168.5.21@o2ib,svname=zlfs2-OST0004,device=/dev/nvme0n1
mount.lustre: cannot parse scheduler options for &apos;/sys/block/nvme0n1/queue/scheduler&apos;
mount.lustre: mount /dev/nvme0n1 at /mnt/OST0004 failed: Invalid argument retries left: 0
mount.lustre: mount /dev/nvme0n1 at /mnt/OST0004 failed: Invalid argument
This may have multiple causes.
Are the mount options correct?
Check the syslog for more info.
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;&lt;b&gt;Logs&lt;/b&gt;&lt;/p&gt;

&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;Aug 17 18:10:22 zlfs2-mds2 kernel: LDISKFS-fs (nvme0n1): file extents enabled, maximum tree depth=5
Aug 17 18:10:22 zlfs2-mds2 kernel: LDISKFS-fs (nvme0n1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Aug 17 18:10:23 zlfs2-mds2 kernel: LDISKFS-fs (nvme0n1): file extents enabled, maximum tree depth=5
Aug 17 18:10:23 zlfs2-mds2 kernel: LDISKFS-fs (nvme0n1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14037:0:(mgc_request.c:257:do_config_log_add()) MGC192.168.5.21@o2ib: failed processing log, type 4: rc = -22
Aug 17 18:10:23 zlfs2-mds2 kernel: Lustre: zlfs2-OST0004: new disk, initializing
Aug 17 18:10:23 zlfs2-mds2 kernel: Lustre: srv-zlfs2-OST0004: No data found on store. Initialize space
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(nodemap_storage.c:368:nodemap_idx_nodemap_add_update()) cannot add nodemap config to non-existing MGS.
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(nodemap_storage.c:1315:nodemap_fs_init()) zlfs2-OST0004: error loading nodemap config file, file must be removed via ldiskfs: rc = -22
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) header@ffff881ffb9ed5c0[0x0, 1, [0x1:0x0:0x0] hash exist]{
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) ....local_storage@ffff881ffb9ed610
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) ....osd-ldiskfs@ffff882022c59100osd-ldiskfs-object@ffff882022c59100(i:ffff881ff8e91e88:78/106428767)[plain]
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) } header@ffff881ffb9ed5c0
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) header@ffff881ffb9ed800[0x0, 1, [0x200000003:0x0:0x0] hash exist]{
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) ....local_storage@ffff881ffb9ed850
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) ....osd-ldiskfs@ffff882022c59e00osd-ldiskfs-object@ffff882022c59e00(i:ffff881ff8e88d88:77/106428733)[plain]
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) } header@ffff881ffb9ed800
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) header@ffff881ffb9ed2c0[0x0, 1, [0xa:0x0:0x0] hash exist]{
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) ....local_storage@ffff881ffb9ed310
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) ....osd-ldiskfs@ffff882022c58600osd-ldiskfs-object@ffff882022c58600(i:ffff881ff8e9af88:79/106428801)[plain]
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) } header@ffff881ffb9ed2c0
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) header@ffff882007239140[0x0, 1, [0xa:0x2:0x0] hash exist]{
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) ....local_storage@ffff882007239190
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) ....osd-ldiskfs@ffff88200f958800osd-ldiskfs-object@ffff88200f958800(i:ffff881ff8ea80c8:80/106428802)[plain]
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) } header@ffff882007239140
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) header@ffff881ffb9ed440[0x0, 1, [0x200000001:0x1017:0x0] hash exist]{
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) ....local_storage@ffff881ffb9ed490
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) ....osd-ldiskfs@ffff882022c59600osd-ldiskfs-object@ffff882022c59600(i:ffff881ff8e71e88:5898241/3450894875)[plain]
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(ofd_dev.c:248:ofd_stack_fini()) } header@ffff881ffb9ed440
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(obd_config.c:578:class_setup()) setup zlfs2-OST0004 failed (-22)
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14203:0:(obd_config.c:1671:class_config_llog_handler()) MGC192.168.5.21@o2ib: cfg command failed: rc = -22
Aug 17 18:10:23 zlfs2-mds2 kernel: Lustre:    cmd=cf003 0:zlfs2-OST0004  1:dev  2:0  3:f
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 15b-f: MGC192.168.5.21@o2ib: The configuration from log &apos;zlfs2-OST0004&apos;failed from the MGS (-22).  Make sure this client and the MGS are running compatible versions of Lustre.
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14037:0:(obd_mount_server.c:1352:server_start_targets()) failed to start server zlfs2-OST0004: -22
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14037:0:(obd_mount_server.c:1844:server_fill_super()) Unable to start targets: -22
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14037:0:(obd_config.c:625:class_cleanup()) Device 3 not setup
Aug 17 18:10:23 zlfs2-mds2 kernel: Lustre: server umount zlfs2-OST0004 complete
Aug 17 18:10:23 zlfs2-mds2 kernel: LustreError: 14037:0:(obd_mount.c:1453:lustre_fill_super()) Unable to mount /dev/nvme0n1 (-22)
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;</comment>
                            <comment id="162253" author="adam.j.roe" created="Wed, 17 Aug 2016 18:28:06 +0000"  >&lt;p&gt;Okay so the strange behavior: &lt;/p&gt;

&lt;p&gt;The first target I try to mount which isn&apos;t on the same server as the MGT will fail and get stuck in this state. Not mounted but in a lock somewhere, its like it starts the service without a target.&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;mount.lustre: mount /dev/nvme1n1 at /mnt/OST0005 failed: Operation already in progress
The target service is already running. (/dev/nvme1n1)
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;All other targets after the first will error out, but not get stuck:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;mount.lustre: mount /dev/nvme0n1 at /mnt/MDT0004 failed: Invalid argument
This may have multiple causes.
Are the mount options correct?
Check the syslog for more info.
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;If I then run the mount command for a second time it will mount. But I have not found a way to recover the first locked target. I have to reboot and remount. &lt;/p&gt;</comment>
                            <comment id="162273" author="adam.j.roe" created="Wed, 17 Aug 2016 21:12:18 +0000"  >&lt;p&gt;Some more verbose information on the failed first mount:&lt;/p&gt;

&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;[root@zlfs2-mds2 ~]# mount -vvv -t lustre  /dev/nvme0n1 /mnt/OST0004
arg[0] = /sbin/mount.lustre
arg[1] = -v
arg[2] = -o
arg[3] = rw
arg[4] = /dev/nvme0n1
arg[5] = /mnt/OST0004
source = /dev/nvme0n1 (/dev/nvme0n1), target = /mnt/OST0004
options = rw
checking for existing Lustre data: found
Reading CONFIGS/mountdata
Writing CONFIGS/mountdata
mounting device /dev/nvme0n1 at /mnt/OST0004, flags=0x1000000 options=osd=osd-ldiskfs,,errors=remount-ro,mgsnode=192.168.5.21@o2ib,virgin,update,param=mgsnode=192.168.5.21@o2ib,svname=zlfs2-OST0004,device=/dev/nvme0n1
mount.lustre: cannot parse scheduler options for &apos;/sys/block/nvme0n1/queue/scheduler&apos;
mount.lustre: mount /dev/nvme0n1 at /mnt/OST0004 failed: Invalid argument retries left: 0
mount.lustre: mount /dev/nvme0n1 at /mnt/OST0004 failed: Invalid argument
This may have multiple causes.
Are the mount options correct?
Check the syslog for more info.
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;Aug 17 22:10:37 zlfs2-mds2 kernel: LDISKFS-fs (nvme0n1): file extents enabled, maximum tree depth=5
Aug 17 22:10:37 zlfs2-mds2 kernel: LDISKFS-fs (nvme0n1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
Aug 17 22:10:37 zlfs2-mds2 kernel: LDISKFS-fs (nvme0n1): file extents enabled, maximum tree depth=5
Aug 17 22:10:37 zlfs2-mds2 kernel: LDISKFS-fs (nvme0n1): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro,no_mbcache
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4148:0:(mgc_request.c:257:do_config_log_add()) MGC192.168.5.21@o2ib: failed processing log, type 4: rc = -22
Aug 17 22:10:38 zlfs2-mds2 kernel: Lustre: zlfs2-OST0004: new disk, initializing
Aug 17 22:10:38 zlfs2-mds2 kernel: Lustre: srv-zlfs2-OST0004: No data found on store. Initialize space
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(nodemap_storage.c:368:nodemap_idx_nodemap_add_update()) cannot add nodemap config to non-existing MGS.
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(nodemap_storage.c:1315:nodemap_fs_init()) zlfs2-OST0004: error loading nodemap config file, file must be removed via ldiskfs: rc = -22
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) header@ffff881ffa035080[0x0, 1, [0x1:0x0:0x0] hash exist]{
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) ....local_storage@ffff881ffa0350d0
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) ....osd-ldiskfs@ffff882006418700osd-ldiskfs-object@ffff882006418700(i:ffff881ff9da1e88:78/1354905553)[plain]
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) } header@ffff881ffa035080
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) header@ffff881ffa034fc0[0x0, 1, [0x200000003:0x0:0x0] hash exist]{
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) ....local_storage@ffff881ffa035010
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) ....osd-ldiskfs@ffff88200641b700osd-ldiskfs-object@ffff88200641b700(i:ffff881ff9d98d88:77/1354905519)[plain]
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) } header@ffff881ffa034fc0
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) header@ffff881ffa034c00[0x0, 1, [0xa:0x0:0x0] hash exist]{
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) ....local_storage@ffff881ffa034c50
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) ....osd-ldiskfs@ffff88200641b100osd-ldiskfs-object@ffff88200641b100(i:ffff881ff9daaf88:79/1354905587)[plain]
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) } header@ffff881ffa034c00
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) header@ffff88202684cd80[0x0, 1, [0xa:0x2:0x0] hash exist]{
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) ....local_storage@ffff88202684cdd0
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) ....osd-ldiskfs@ffff881ffa237e00osd-ldiskfs-object@ffff881ffa237e00(i:ffff882024048948:80/1354905588)[plain]
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) } header@ffff88202684cd80
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) header@ffff881ffa035380[0x0, 1, [0x200000001:0x1017:0x0] hash exist]{
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) ....local_storage@ffff881ffa0353d0
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) ....osd-ldiskfs@ffff88200641be00osd-ldiskfs-object@ffff88200641be00(i:ffff882024309a48:7864321/2679038361)[plain]
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(ofd_dev.c:248:ofd_stack_fini()) } header@ffff881ffa035380
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(obd_config.c:578:class_setup()) setup zlfs2-OST0004 failed (-22)
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4314:0:(obd_config.c:1671:class_config_llog_handler()) MGC192.168.5.21@o2ib: cfg command failed: rc = -22
Aug 17 22:10:38 zlfs2-mds2 kernel: Lustre:    cmd=cf003 0:zlfs2-OST0004  1:dev  2:0  3:f
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 15b-f: MGC192.168.5.21@o2ib: The configuration from log &apos;zlfs2-OST0004&apos;failed from the MGS (-22).  Make sure this client and the MGS are running compatible versions of Lustre.
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4148:0:(obd_mount_server.c:1352:server_start_targets()) failed to start server zlfs2-OST0004: -22
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4148:0:(obd_mount_server.c:1844:server_fill_super()) Unable to start targets: -22
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4148:0:(obd_config.c:625:class_cleanup()) Device 3 not setup
Aug 17 22:10:38 zlfs2-mds2 kernel: Lustre: server umount zlfs2-OST0004 complete
Aug 17 22:10:38 zlfs2-mds2 kernel: LustreError: 4148:0:(obd_mount.c:1453:lustre_fill_super()) Unable to mount /dev/nvme0n1 (-22)
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;I then try to mount again and get the same as above with this extra:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;The target service is already running. (/dev/nvme0n1)
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;I reboot the server, mounting that target still fails, however If I mount a different target on the same server beforehand, say nvme1n1 this I am then able to mount nvme0n1 without issue.  &lt;/p&gt;
</comment>
                            <comment id="162337" author="yong.fan" created="Thu, 18 Aug 2016 13:59:51 +0000"  >&lt;p&gt;Fail to mount the OST is another issue that is different from the original &quot;ASSERTION( atomic_read(&amp;amp;d-&amp;gt;ld_ref) == 0 )&quot;.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The first target I try to mount which isn&apos;t on the same server as the MGT will fail and get stuck in this state. Not mounted but in a lock somewhere, its like it starts the service without a target.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Have you mounted up the MGS before mounting the MDT or OTS? If no, please mount up the MGS (or say MGT on the MGS node) firstly. Otherwise, please enable -1 level Lustre kernel debug on both the MGS and OSS/MDS, then try again and attach the Lustre debug logs. Thanks!&lt;/p&gt;</comment>
                            <comment id="162387" author="gerrit" created="Thu, 18 Aug 2016 16:42:06 +0000"  >&lt;p&gt;Kit Westneat (kit.westneat@gmail.com) uploaded a new patch: &lt;a href=&quot;http://review.whamcloud.com/22004&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/22004&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-8508&quot; title=&quot;kernel:LustreError: 3842:0:(lu_object.c:1243:lu_device_fini()) ASSERTION( atomic_read(&amp;amp;d-&amp;gt;ld_ref) == 0 ) failed: Refcount is 1&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-8508&quot;&gt;&lt;del&gt;LU-8508&lt;/del&gt;&lt;/a&gt; nodemap: improve object handling in cache saving&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: 6753d578f44195ff6e4476266538887f6cd07712&lt;/p&gt;</comment>
                            <comment id="162392" author="kit.westneat" created="Thu, 18 Aug 2016 16:53:58 +0000"  >&lt;p&gt;This patch is still a work in progress, but addresses both these issues. &lt;/p&gt;</comment>
                            <comment id="162395" author="kit.westneat" created="Thu, 18 Aug 2016 17:03:40 +0000"  >&lt;p&gt;BTW the cause of the second bug is that if a new OST mounts before the MGC has pulled the nodemap config from the MGS, it creates a new blank config on disk. Part of that code was erroneously assuming that it was in the MGS, as normally all new records are created there and then sent to the OSTs, so it was returning an error. That&apos;s why the first OST failed to mount. When the other OSTs were mounted, the MGC was already connected to the MGS, so it was able to pull the config and save it properly. That&apos;s why the other OSTs were able to mount after rebooting, but nvme0n1 wasn&apos;t able to until the others were mounted.&lt;/p&gt;</comment>
                            <comment id="164590" author="pjones" created="Thu, 1 Sep 2016 14:32:43 +0000"  >&lt;p&gt;Kit&lt;/p&gt;

&lt;p&gt;Will you be refreshing the patch in light of Andreas&apos;s review feedback?&lt;/p&gt;

&lt;p&gt;Peter&lt;/p&gt;</comment>
                            <comment id="164597" author="kit.westneat" created="Thu, 1 Sep 2016 14:46:55 +0000"  >&lt;p&gt;Hey Peter,&lt;/p&gt;

&lt;p&gt;I wasn&apos;t planning on it since he +1&apos;d it, unless there were other issues found, but I can if that&apos;s desired.&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Kit&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="164599" author="pjones" created="Thu, 1 Sep 2016 14:50:58 +0000"  >&lt;p&gt;Kit&lt;/p&gt;

&lt;p&gt;I think that at the moment a second reviewer is holding off in anticipation of another version being forthcoming given that there are quite a number of comments so I tihnk that it would be good to refresh it&lt;/p&gt;

&lt;p&gt;Peter&lt;/p&gt;</comment>
                            <comment id="164619" author="kit.westneat" created="Thu, 1 Sep 2016 15:34:32 +0000"  >&lt;p&gt;Hey Peter, &lt;/p&gt;

&lt;p&gt;Are we talking about change 22004? I only see two style comments from Andreas. There are a few over 80 chars autocomments as well, but I thought we were ignoring those now to match the Linux style guide. I&apos;ll refresh it, but I want to make sure I&apos;m not missing something.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Kit &lt;/p&gt;</comment>
                            <comment id="164628" author="pjones" created="Thu, 1 Sep 2016 15:54:19 +0000"  >&lt;p&gt;Hi Kit&lt;/p&gt;

&lt;p&gt;I checked with Oleg and you are right - sorry about that - so I have requested a second reviewer so that we can get this landed&lt;/p&gt;

&lt;p&gt;Peter&lt;/p&gt;</comment>
                            <comment id="164631" author="kit.westneat" created="Thu, 1 Sep 2016 15:58:49 +0000"  >&lt;p&gt;Hey Peter,&lt;/p&gt;

&lt;p&gt;No problem. I made the changes, would it be better to upload them and face the tests again, or leave it as is?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Kit&lt;/p&gt;</comment>
                            <comment id="164632" author="pjones" created="Thu, 1 Sep 2016 15:59:39 +0000"  >&lt;p&gt;Let&apos;s see how the second review goes to see whether the refresh is needed&lt;/p&gt;</comment>
                            <comment id="166220" author="gerrit" created="Thu, 15 Sep 2016 23:31:41 +0000"  >&lt;p&gt;Oleg Drokin (oleg.drokin@intel.com) merged in patch &lt;a href=&quot;http://review.whamcloud.com/22004/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/22004/&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-8508&quot; title=&quot;kernel:LustreError: 3842:0:(lu_object.c:1243:lu_device_fini()) ASSERTION( atomic_read(&amp;amp;d-&amp;gt;ld_ref) == 0 ) failed: Refcount is 1&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-8508&quot;&gt;&lt;del&gt;LU-8508&lt;/del&gt;&lt;/a&gt; nodemap: improve object handling in cache saving&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: &lt;br/&gt;
Commit: 45cb603b4352a73077dcc45ec2cdea403837a7ba&lt;/p&gt;</comment>
                            <comment id="166224" author="pjones" created="Fri, 16 Sep 2016 03:36:15 +0000"  >&lt;p&gt;Landed for 2.9&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="39830">LU-8611</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Related</name>
                                            <outwardlinks description="is related to ">
                                        <issuelink>
            <issuekey id="18740">LU-3291</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="22594" name="master-b3423_LBUG.txt" size="22927" author="adam.j.roe" created="Tue, 16 Aug 2016 21:22:15 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                            <customfield id="customfield_10890" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10390" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzykxj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10090" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10060" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Severity</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10022"><![CDATA[3]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        </customfields>
    </item>
</channel>
</rss>