<!-- 
RSS generated by JIRA (9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c) at Sat Feb 10 01:05:39 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>Whamcloud Community JIRA</title>
    <link>https://jira.whamcloud.com</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.4.14</version>
        <build-number>940014</build-number>
        <build-date>05-12-2023</build-date>
    </build-info>


<item>
            <title>[LU-297] llverfs: Open &apos;/mnt/lustre/dir00007/file015&apos; failed:Input/output error</title>
                <link>https://jira.whamcloud.com/browse/LU-297</link>
                <project id="10000" key="LU">Lustre</project>
                    <description>&lt;p&gt;While running llverfs in full mode on Lustre filesystem with 1TB OST, it hit -EIO error after writing (7*32+15)*4G=956G data into the filesystem as follows:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;llverfs: Open &apos;/mnt/lustre/dir00007/file015&apos; failed:Input/output error
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Dmesg log showed that:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;Lustre: DEBUG MARKER: ======== run llverfs in full mode on the Lustre filesystem /mnt/lustre ========
LustreError: 21145:0:(mds_open.c:441:mds_create_objects()) error creating objects for inode 41943290: rc = -5
LustreError: 21145:0:(mds_open.c:826:mds_finish_open()) mds_create_objects: rc = -5
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Before running llverfs, the Lustre disk space and inode usage info are as follows:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;# lfs df -h /mnt/lustre
UUID                       bytes        Used   Available Use% Mounted on
largefs-MDT0000_UUID      224.0G      461.0M      210.7G   0% /mnt/lustre[MDT:0]
largefs-OST0000_UUID     1007.9G      472.6M      956.3G   0% /mnt/lustre[OST:0]

filesystem summary:      1007.9G      472.6M      956.3G   0% /mnt/lustre


# lfs df -i /mnt/lustre
UUID                      Inodes       IUsed       IFree IUse% Mounted on
largefs-MDT0000_UUID    58597923          26    58597897   0% /mnt/lustre[MDT:0]
largefs-OST0000_UUID    67108864          72    67108792   0% /mnt/lustre[OST:0]

filesystem summary:     58597923          26    58597897   0% /mnt/lustre
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;After running llverfs, the Lustre disk space and inode usage info are as follows:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;# lfs df -h /mnt/lustre
UUID                       bytes        Used   Available Use% Mounted on
largefs-MDT0000_UUID      224.0G      461.0M      210.7G   0% /mnt/lustre[MDT:0]
largefs-OST0000_UUID     1007.9G      956.5G      276.8M 100% /mnt/lustre[OST:0]

filesystem summary:      1007.9G      956.5G      276.8M 100% /mnt/lustre


# lfs df -i /mnt/lustre
UUID                      Inodes       IUsed       IFree IUse% Mounted on
largefs-MDT0000_UUID    58598163         274    58597889   0% /mnt/lustre[MDT:0]
largefs-OST0000_UUID    13492969         328    13492641   0% /mnt/lustre[OST:0]

filesystem summary:     58598163         274    58597889   0% /mnt/lustre
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Maloo report: &lt;a href=&quot;https://maloo.whamcloud.com/test_sets/4b6fc288-7a43-11e0-a1b3-52540025f9af&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://maloo.whamcloud.com/test_sets/4b6fc288-7a43-11e0-a1b3-52540025f9af&lt;/a&gt;&lt;/p&gt;</description>
                <environment>Lustre Branch: b1_8&lt;br/&gt;
Lustre Build: &lt;a href=&quot;http://newbuild.whamcloud.com/job/lustre-reviews/363/arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/&quot;&gt;http://newbuild.whamcloud.com/job/lustre-reviews/363/arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/&lt;/a&gt;&lt;br/&gt;
Test System: DDN SFA10000E (1Client+1MGS/MDT+1OST)&lt;br/&gt;
Network: tcp</environment>
        <key id="10770">LU-297</key>
            <summary>llverfs: Open &apos;/mnt/lustre/dir00007/file015&apos; failed:Input/output error</summary>
                <type id="1" iconUrl="https://jira.whamcloud.com/secure/viewavatar?size=xsmall&amp;avatarId=11303&amp;avatarType=issuetype">Bug</type>
                                            <priority id="4" iconUrl="https://jira.whamcloud.com/images/icons/priorities/minor.svg">Minor</priority>
                        <status id="5" iconUrl="https://jira.whamcloud.com/images/icons/statuses/resolved.png" description="A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.">Resolved</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="1">Fixed</resolution>
                                        <assignee username="yujian">Jian Yu</assignee>
                                    <reporter username="yujian">Jian Yu</reporter>
                        <labels>
                    </labels>
                <created>Mon, 9 May 2011 07:51:59 +0000</created>
                <updated>Fri, 13 May 2016 02:06:39 +0000</updated>
                            <resolved>Fri, 13 May 2016 02:06:39 +0000</resolved>
                                    <version>Lustre 2.0.0</version>
                    <version>Lustre 1.8.6</version>
                                    <fixVersion>Lustre 2.1.0</fixVersion>
                    <fixVersion>Lustre 1.8.7</fixVersion>
                                        <due></due>
                            <votes>0</votes>
                                    <watches>1</watches>
                                                                            <comments>
                            <comment id="14011" author="yujian" created="Mon, 9 May 2011 08:17:06 +0000"  >&lt;p&gt;The failure occurred while performing the following codes of lustre/utils/llverfs.c:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;static int dir_write(char *chunk_buf, size_t chunksize,
             time_t time_st, unsigned long dir_num)
{
    ......
    for (; dir_num &amp;lt; num_dirs; num_files++, file_num++) {
        ......

        fd = open_file(new_file(tempfile, tempdir, file_num),
                   O_WRONLY | O_CREAT | O_TRUNC | O_LARGEFILE);
        ......
    }
    ......
}
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;I&apos;ll investigate more in mds_create_objects() which returned -EIO instead of -ENOSPC while opening/creating a file with OST out of space.&lt;/p&gt;</comment>
                            <comment id="14018" author="adilger" created="Mon, 9 May 2011 10:02:58 +0000"  >&lt;p&gt;This EIO error is likely involving the alloc_rr-&amp;gt;osc_precreate() codepath, where it is checking whether the OST is full (OSCC_FLAG_ENOSPC) before allocating more objects there.  I agree that returning -ENOSPC is the better thing to do in this case, and -EIO is not correct for OSCC_FLAG_ENOSPC.  I think it is still correct to return -EIO in the case where the OSTs are unavailable due to inactive devices, instead of the case where the OSTs are full.&lt;/p&gt;


&lt;p&gt;Because running llverfs takes such a long time, it does not stop the test when it hits this EIO error, just to allow testing to make progress.  Looking at this code and the test results, llverfs doesn&apos;t consider an error from open_file() fatal (it just breaks out of the loop and begin the read test), and write_chunks() calls write_retry() that should record the error but continue trying to write until it hit too many errors, then moves on to the read verification.  It only returns an error at the end of the test.&lt;/p&gt;

&lt;p&gt;Looking at &lt;a href=&quot;https://maloo.whamcloud.com/test_sets/4b6fc288-7a43-11e0-a1b3-52540025f9af&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://maloo.whamcloud.com/test_sets/4b6fc288-7a43-11e0-a1b3-52540025f9af&lt;/a&gt; it does appear that the full test on Lustre was run until the filesystem was full, and then the read verification completed.  This means it should be possible to start running llverfs on a 24TB ldiskfs filesystem while debugging this problem in Lustre, and if possible also start running on a 128TB+ ldiskfs filesystem in order to speed up this process.  My calculations show a 24TB test will take about 88 hours, and a 128TB test will take about 473 hours = 19 days, so it makes sense to start those tests as soon as possible after making script adjustments (as described below) and verifying them one last time on the 1TB filesystem.&lt;/p&gt;


&lt;p&gt;I noticed something incorrect in the test script - after the partial llverfs run on ldiskfs the test files were deleted &lt;em&gt;before&lt;/em&gt; the e2fsck was run.  This makes the e2fsck check much less valuable, because it hides any kernel or e2fsck bugs related to how the files are allocated on disk.  The e2fsck should be run after &lt;em&gt;each&lt;/em&gt; llverfs test is finished, but before the test files are deleted.  There is currently no e2fsck check at all after the partial llverfs run on Lustre, which needs to be added.&lt;/p&gt;

&lt;p&gt;Also, for proper problem isolation there should be a full run of llverfs on the ldiskfs-mounted OST filesystem after the partial ldiskfs test, because I suspect there may be some bugs in ext4 or the ldiskfs patches themselves, even before we start testing obdfilter running on large filesystems.  Again, there should be a full e2fsck run after llverfs is run.&lt;/p&gt;</comment>
                            <comment id="14132" author="adilger" created="Tue, 10 May 2011 12:15:57 +0000"  >&lt;p&gt;If you are modifying llverfs, it might be convenient to print out the current MB/s with each status line, and in a summary at the end, along with an estimate of how long the test will continue (based on the number of total_size = file_size * num_files to create for partial mode, or the total_size = free space in full mode).&lt;/p&gt;

&lt;p&gt;The best way to do this, I think, is to keep a global &quot;curr_bytes&quot; counter for the number of bytes read or written since the start of the the dir_write() or dir_read() phase.  Inside show_filename() it should use gettimeofday() for the last_time a message was printed (instead of the simple time() value used now) and it should track the last_bytes value:&lt;/p&gt;

&lt;p&gt;    static struct timeval last_time;&lt;br/&gt;
    static __u64 last_bytes;&lt;br/&gt;
    static char last_op;&lt;br/&gt;
    struct timeval curr_time;&lt;br/&gt;
    double curr_delta, overall_delta, total_time, remain_time;&lt;br/&gt;
    int remain_hours, remain_minutes, remain_seconds;&lt;/p&gt;

&lt;p&gt;    if (last_op != op&lt;span class=&quot;error&quot;&gt;&amp;#91;0&amp;#93;&lt;/span&gt;) &lt;/p&gt;
{
        last_bytes = 0;
        last_time = start_time;
        last_op = op[0];
    }

&lt;p&gt;    curr_delta = (curr_time.tv_sec - last_time.tv_sec) +&lt;br/&gt;
                 (double)(curr_time.tv_usec - last_time.tv_usec) / 1000000;&lt;br/&gt;
    overall_delta =  (curr_time.tv_sec - start_time.tv_sec) +&lt;br/&gt;
                 (double)(curr_time.tv_usec - start_time.tv_usec) / 1000000;&lt;br/&gt;
    curr_rate = (curr_bytes - last_bytes) / curr_delta;&lt;br/&gt;
    overall_rate = curr_bytes / overall_delta;&lt;/p&gt;

&lt;p&gt;    total_time = total_size / curr_rate;&lt;br/&gt;
    remain_time = total_time - curr_time;&lt;br/&gt;
    remain_hours = (remain_time / 3600);&lt;br/&gt;
    remain_minutes = (remain_time - remain_hours * 3600)/60;&lt;br/&gt;
    remain_seconds = (remain_time - remain_hours * 3600 - remain_minutes*60);&lt;/p&gt;

&lt;p&gt;    &quot;%s Filename %s   current %g MB/s, overall %g MB/s, est %u:%u:%u left&quot;, op,&lt;br/&gt;
    filename, curr_rate, total_rate, remain_hours, remain_minutes, remain_seconds&lt;/p&gt;

&lt;p&gt;    last_time = curr_time;&lt;br/&gt;
    last_bytes = curr_bytes;&lt;/p&gt;
</comment>
                            <comment id="15015" author="yujian" created="Wed, 25 May 2011 04:49:35 +0000"  >&lt;p&gt;Status: I would work on this ticket after Lustre 1.8.6 pre-release/release testing.&lt;/p&gt;</comment>
                            <comment id="17943" author="yujian" created="Sun, 17 Jul 2011 10:14:03 +0000"  >&lt;blockquote&gt;&lt;p&gt;If you are modifying llverfs, it might be convenient to print out the current MB/s with each status line, and in a summary at the end, along with an estimate of how long the test will continue (based on the number of total_size = file_size * num_files to create for partial mode, or the total_size = free space in full mode).&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;The patch for master branch is in &lt;a href=&quot;http://review.whamcloud.com/1110&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/1110&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="17944" author="yujian" created="Sun, 17 Jul 2011 11:10:54 +0000"  >&lt;p&gt;For the EIO error while opening/creating a new file with a full OST, the lctl debug log on master branch showed that:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;00020000:00000001:1.0:1310895705.290808:0:7325:0:(obd_class.h:742:obd_precreate()) Process entered
00000008:00000001:1.0:1310895705.290809:0:7325:0:(osc_create.c:354:osc_precreate()) Process entered
00000008:00000001:1.0:1310895705.290810:0:7325:0:(osc_create.c:365:osc_precreate()) Process leaving via out (rc=1000 : 1000 : 0x3e8)
00020000:00000001:1.0:1310895705.290811:0:7325:0:(obd_class.h:748:obd_precreate()) Process leaving (rc=1000 : 1000 : 3e8)
00020000:00000001:1.0:1310895705.290812:0:7325:0:(lov_qos.c:685:alloc_rr()) Process leaving (rc=0 : 0 : 0)
00020000:00000001:1.0:1310895705.290813:0:7325:0:(obd_class.h:1101:obd_putref()) Process entered
00020000:00000001:1.0:1310895705.290814:0:7325:0:(obd_class.h:1106:obd_putref()) Process leaving
00020000:00000001:1.0:1310895705.290815:0:7325:0:(lov_qos.c:952:alloc_qos()) Process leaving (rc=0 : 0 : 0)
00020000:00000001:1.0:1310895705.290816:0:7325:0:(lov_qos.c:983:alloc_idx_array()) Process leaving (rc=0 : 0 : 0)
00020000:00000001:1.0:1310895705.290819:0:7325:0:(lov_qos.c:1062:qos_prep_create()) Process leaving via out_err (rc=18446744073709551611 : -5 : 0xfffffffffffffffb)
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;So, the code path is qos_prep_create()&amp;#45;&amp;gt;alloc_idx_array()&amp;#45;&amp;gt;alloc_qos()&amp;#45;&amp;gt;alloc_rr()&amp;#45;&amp;gt;obd_precreate()&amp;#45;&amp;gt;osc_precreate(). In osc_precreate(), it did check that the OSCC_FLAG_NOSPC was set and returned &quot;rc=1000&quot;:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;int osc_precreate(struct obd_export *exp)
{
        //......
        if (oscc-&amp;gt;oscc_flags &amp;amp; OSCC_FLAG_NOSPC ||
            oscc-&amp;gt;oscc_flags &amp;amp; OSCC_FLAG_RDONLY ||
            oscc-&amp;gt;oscc_flags &amp;amp; OSCC_FLAG_EXITING)
                GOTO(out, rc = 1000);
        //......
}
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;However, it did not return the corresponding error code for each flag. So, the caller functions did not know what the real error was.&lt;/p&gt;

&lt;p&gt;And the EIO error was actually generated from here:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;int qos_prep_create(struct obd_export *exp, struct lov_request_set *set)
{
        //......
        stripes = alloc_idx_array(exp, lsm, newea, &amp;amp;idx_arr, &amp;amp;idx_cnt, flag);
        if (stripes &amp;lt;= 0)
                GOTO(out_err, rc = stripes ? stripes : -EIO);
        //......
}
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The alloc_idx_array() returned 0 because alloc_rr() did not handle the return value of osc_precreate() properly.&lt;/p&gt;

&lt;p&gt;Andreas, should I improve osc_precreate() to return specific error codes and make alloc_rr() to handle them correctly?&lt;/p&gt;</comment>
                            <comment id="17945" author="adilger" created="Sun, 17 Jul 2011 21:48:53 +0000"  >&lt;p&gt;One possibility is to return 1000+(errno) so that it can be decoded by the caller, but does not affect the logic too much.  The difficulty is what to do if different stripes return different errors, but e.g. taking the first one is ok too. &lt;/p&gt;

&lt;p&gt;It isn&apos;t a high priority if you have other tasks however. &lt;/p&gt;</comment>
                            <comment id="17946" author="yujian" created="Mon, 18 Jul 2011 00:00:57 +0000"  >&lt;blockquote&gt;
&lt;p&gt;One possibility is to return 1000+(errno) so that it can be decoded by the caller, but does not affect the logic too much. The difficulty is what to do if different stripes return different errors, but e.g. taking the first one is ok too.&lt;/p&gt;

&lt;p&gt;It isn&apos;t a high priority if you have other tasks however.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;OK, I see. I&apos;d work on my other tasks which have higher priorities then.&lt;/p&gt;</comment>
                            <comment id="19144" author="hudson" created="Thu, 11 Aug 2011 22:12:44 +0000"  >&lt;p&gt;Integrated in &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;http://build.whamcloud.com/images/16x16/blue.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;a href=&quot;http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/254/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;lustre-master &#187; i686,client,el6,inkernel #254&lt;/a&gt;&lt;br/&gt;
     &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-297&quot; title=&quot;llverfs: Open &amp;#39;/mnt/lustre/dir00007/file015&amp;#39; failed:Input/output error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-297&quot;&gt;&lt;del&gt;LU-297&lt;/del&gt;&lt;/a&gt; improve llverfs to display the current read/write performance&lt;/p&gt;

&lt;p&gt;Oleg Drokin : &lt;a href=&quot;http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;amp;a=commit&amp;amp;h=64f0f965dc80556739a3f86c15c0bb158b6289a2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;64f0f965dc80556739a3f86c15c0bb158b6289a2&lt;/a&gt;&lt;br/&gt;
Files : &lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lustre/utils/llverdev.c&lt;/li&gt;
	&lt;li&gt;lustre/utils/llverfs.c&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="19145" author="hudson" created="Thu, 11 Aug 2011 22:12:46 +0000"  >&lt;p&gt;Integrated in &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;http://build.whamcloud.com/images/16x16/blue.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;a href=&quot;http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/254/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;lustre-master &#187; x86_64,client,el5,ofa #254&lt;/a&gt;&lt;br/&gt;
     &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-297&quot; title=&quot;llverfs: Open &amp;#39;/mnt/lustre/dir00007/file015&amp;#39; failed:Input/output error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-297&quot;&gt;&lt;del&gt;LU-297&lt;/del&gt;&lt;/a&gt; improve llverfs to display the current read/write performance&lt;/p&gt;

&lt;p&gt;Oleg Drokin : &lt;a href=&quot;http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;amp;a=commit&amp;amp;h=64f0f965dc80556739a3f86c15c0bb158b6289a2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;64f0f965dc80556739a3f86c15c0bb158b6289a2&lt;/a&gt;&lt;br/&gt;
Files : &lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lustre/utils/llverdev.c&lt;/li&gt;
	&lt;li&gt;lustre/utils/llverfs.c&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="19146" author="hudson" created="Thu, 11 Aug 2011 22:16:04 +0000"  >&lt;p&gt;Integrated in &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;http://build.whamcloud.com/images/16x16/blue.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;a href=&quot;http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/254/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;lustre-master &#187; x86_64,client,el6,inkernel #254&lt;/a&gt;&lt;br/&gt;
     &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-297&quot; title=&quot;llverfs: Open &amp;#39;/mnt/lustre/dir00007/file015&amp;#39; failed:Input/output error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-297&quot;&gt;&lt;del&gt;LU-297&lt;/del&gt;&lt;/a&gt; improve llverfs to display the current read/write performance&lt;/p&gt;

&lt;p&gt;Oleg Drokin : &lt;a href=&quot;http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;amp;a=commit&amp;amp;h=64f0f965dc80556739a3f86c15c0bb158b6289a2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;64f0f965dc80556739a3f86c15c0bb158b6289a2&lt;/a&gt;&lt;br/&gt;
Files : &lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lustre/utils/llverdev.c&lt;/li&gt;
	&lt;li&gt;lustre/utils/llverfs.c&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="19147" author="hudson" created="Thu, 11 Aug 2011 22:17:57 +0000"  >&lt;p&gt;Integrated in &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;http://build.whamcloud.com/images/16x16/blue.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;a href=&quot;http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/254/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;lustre-master &#187; x86_64,server,el6,inkernel #254&lt;/a&gt;&lt;br/&gt;
     &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-297&quot; title=&quot;llverfs: Open &amp;#39;/mnt/lustre/dir00007/file015&amp;#39; failed:Input/output error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-297&quot;&gt;&lt;del&gt;LU-297&lt;/del&gt;&lt;/a&gt; improve llverfs to display the current read/write performance&lt;/p&gt;

&lt;p&gt;Oleg Drokin : &lt;a href=&quot;http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;amp;a=commit&amp;amp;h=64f0f965dc80556739a3f86c15c0bb158b6289a2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;64f0f965dc80556739a3f86c15c0bb158b6289a2&lt;/a&gt;&lt;br/&gt;
Files : &lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lustre/utils/llverdev.c&lt;/li&gt;
	&lt;li&gt;lustre/utils/llverfs.c&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="19148" author="hudson" created="Thu, 11 Aug 2011 22:18:00 +0000"  >&lt;p&gt;Integrated in &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;http://build.whamcloud.com/images/16x16/blue.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;a href=&quot;http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/254/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;lustre-master &#187; x86_64,client,el5,inkernel #254&lt;/a&gt;&lt;br/&gt;
     &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-297&quot; title=&quot;llverfs: Open &amp;#39;/mnt/lustre/dir00007/file015&amp;#39; failed:Input/output error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-297&quot;&gt;&lt;del&gt;LU-297&lt;/del&gt;&lt;/a&gt; improve llverfs to display the current read/write performance&lt;/p&gt;

&lt;p&gt;Oleg Drokin : &lt;a href=&quot;http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;amp;a=commit&amp;amp;h=64f0f965dc80556739a3f86c15c0bb158b6289a2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;64f0f965dc80556739a3f86c15c0bb158b6289a2&lt;/a&gt;&lt;br/&gt;
Files : &lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lustre/utils/llverfs.c&lt;/li&gt;
	&lt;li&gt;lustre/utils/llverdev.c&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="19149" author="hudson" created="Thu, 11 Aug 2011 22:19:40 +0000"  >&lt;p&gt;Integrated in &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;http://build.whamcloud.com/images/16x16/blue.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;a href=&quot;http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/254/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;lustre-master &#187; i686,client,el5,inkernel #254&lt;/a&gt;&lt;br/&gt;
     &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-297&quot; title=&quot;llverfs: Open &amp;#39;/mnt/lustre/dir00007/file015&amp;#39; failed:Input/output error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-297&quot;&gt;&lt;del&gt;LU-297&lt;/del&gt;&lt;/a&gt; improve llverfs to display the current read/write performance&lt;/p&gt;

&lt;p&gt;Oleg Drokin : &lt;a href=&quot;http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;amp;a=commit&amp;amp;h=64f0f965dc80556739a3f86c15c0bb158b6289a2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;64f0f965dc80556739a3f86c15c0bb158b6289a2&lt;/a&gt;&lt;br/&gt;
Files : &lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lustre/utils/llverfs.c&lt;/li&gt;
	&lt;li&gt;lustre/utils/llverdev.c&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="19150" author="hudson" created="Thu, 11 Aug 2011 22:20:11 +0000"  >&lt;p&gt;Integrated in &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;http://build.whamcloud.com/images/16x16/blue.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;a href=&quot;http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/254/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;lustre-master &#187; i686,server,el6,inkernel #254&lt;/a&gt;&lt;br/&gt;
     &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-297&quot; title=&quot;llverfs: Open &amp;#39;/mnt/lustre/dir00007/file015&amp;#39; failed:Input/output error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-297&quot;&gt;&lt;del&gt;LU-297&lt;/del&gt;&lt;/a&gt; improve llverfs to display the current read/write performance&lt;/p&gt;

&lt;p&gt;Oleg Drokin : &lt;a href=&quot;http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;amp;a=commit&amp;amp;h=64f0f965dc80556739a3f86c15c0bb158b6289a2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;64f0f965dc80556739a3f86c15c0bb158b6289a2&lt;/a&gt;&lt;br/&gt;
Files : &lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lustre/utils/llverdev.c&lt;/li&gt;
	&lt;li&gt;lustre/utils/llverfs.c&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="19151" author="hudson" created="Thu, 11 Aug 2011 22:22:18 +0000"  >&lt;p&gt;Integrated in &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;http://build.whamcloud.com/images/16x16/blue.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;a href=&quot;http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/254/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;lustre-master &#187; x86_64,client,sles11,inkernel #254&lt;/a&gt;&lt;br/&gt;
     &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-297&quot; title=&quot;llverfs: Open &amp;#39;/mnt/lustre/dir00007/file015&amp;#39; failed:Input/output error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-297&quot;&gt;&lt;del&gt;LU-297&lt;/del&gt;&lt;/a&gt; improve llverfs to display the current read/write performance&lt;/p&gt;

&lt;p&gt;Oleg Drokin : &lt;a href=&quot;http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;amp;a=commit&amp;amp;h=64f0f965dc80556739a3f86c15c0bb158b6289a2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;64f0f965dc80556739a3f86c15c0bb158b6289a2&lt;/a&gt;&lt;br/&gt;
Files : &lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lustre/utils/llverdev.c&lt;/li&gt;
	&lt;li&gt;lustre/utils/llverfs.c&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="19152" author="hudson" created="Thu, 11 Aug 2011 22:22:33 +0000"  >&lt;p&gt;Integrated in &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;http://build.whamcloud.com/images/16x16/blue.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;a href=&quot;http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/254/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;lustre-master &#187; x86_64,server,el5,inkernel #254&lt;/a&gt;&lt;br/&gt;
     &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-297&quot; title=&quot;llverfs: Open &amp;#39;/mnt/lustre/dir00007/file015&amp;#39; failed:Input/output error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-297&quot;&gt;&lt;del&gt;LU-297&lt;/del&gt;&lt;/a&gt; improve llverfs to display the current read/write performance&lt;/p&gt;

&lt;p&gt;Oleg Drokin : &lt;a href=&quot;http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;amp;a=commit&amp;amp;h=64f0f965dc80556739a3f86c15c0bb158b6289a2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;64f0f965dc80556739a3f86c15c0bb158b6289a2&lt;/a&gt;&lt;br/&gt;
Files : &lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lustre/utils/llverfs.c&lt;/li&gt;
	&lt;li&gt;lustre/utils/llverdev.c&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="19153" author="hudson" created="Thu, 11 Aug 2011 22:22:49 +0000"  >&lt;p&gt;Integrated in &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;http://build.whamcloud.com/images/16x16/blue.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;a href=&quot;http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/254/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;lustre-master &#187; i686,server,el5,ofa #254&lt;/a&gt;&lt;br/&gt;
     &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-297&quot; title=&quot;llverfs: Open &amp;#39;/mnt/lustre/dir00007/file015&amp;#39; failed:Input/output error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-297&quot;&gt;&lt;del&gt;LU-297&lt;/del&gt;&lt;/a&gt; improve llverfs to display the current read/write performance&lt;/p&gt;

&lt;p&gt;Oleg Drokin : &lt;a href=&quot;http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;amp;a=commit&amp;amp;h=64f0f965dc80556739a3f86c15c0bb158b6289a2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;64f0f965dc80556739a3f86c15c0bb158b6289a2&lt;/a&gt;&lt;br/&gt;
Files : &lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lustre/utils/llverfs.c&lt;/li&gt;
	&lt;li&gt;lustre/utils/llverdev.c&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="19154" author="hudson" created="Thu, 11 Aug 2011 22:25:01 +0000"  >&lt;p&gt;Integrated in &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;http://build.whamcloud.com/images/16x16/blue.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;a href=&quot;http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/254/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;lustre-master &#187; x86_64,server,el5,ofa #254&lt;/a&gt;&lt;br/&gt;
     &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-297&quot; title=&quot;llverfs: Open &amp;#39;/mnt/lustre/dir00007/file015&amp;#39; failed:Input/output error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-297&quot;&gt;&lt;del&gt;LU-297&lt;/del&gt;&lt;/a&gt; improve llverfs to display the current read/write performance&lt;/p&gt;

&lt;p&gt;Oleg Drokin : &lt;a href=&quot;http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;amp;a=commit&amp;amp;h=64f0f965dc80556739a3f86c15c0bb158b6289a2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;64f0f965dc80556739a3f86c15c0bb158b6289a2&lt;/a&gt;&lt;br/&gt;
Files : &lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lustre/utils/llverdev.c&lt;/li&gt;
	&lt;li&gt;lustre/utils/llverfs.c&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="19155" author="hudson" created="Thu, 11 Aug 2011 22:25:35 +0000"  >&lt;p&gt;Integrated in &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;http://build.whamcloud.com/images/16x16/blue.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;a href=&quot;http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/254/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;lustre-master &#187; x86_64,client,ubuntu1004,inkernel #254&lt;/a&gt;&lt;br/&gt;
     &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-297&quot; title=&quot;llverfs: Open &amp;#39;/mnt/lustre/dir00007/file015&amp;#39; failed:Input/output error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-297&quot;&gt;&lt;del&gt;LU-297&lt;/del&gt;&lt;/a&gt; improve llverfs to display the current read/write performance&lt;/p&gt;

&lt;p&gt;Oleg Drokin : &lt;a href=&quot;http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;amp;a=commit&amp;amp;h=64f0f965dc80556739a3f86c15c0bb158b6289a2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;64f0f965dc80556739a3f86c15c0bb158b6289a2&lt;/a&gt;&lt;br/&gt;
Files : &lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lustre/utils/llverfs.c&lt;/li&gt;
	&lt;li&gt;lustre/utils/llverdev.c&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="19156" author="hudson" created="Thu, 11 Aug 2011 22:32:27 +0000"  >&lt;p&gt;Integrated in &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;http://build.whamcloud.com/images/16x16/blue.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;a href=&quot;http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/254/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;lustre-master &#187; i686,server,el5,inkernel #254&lt;/a&gt;&lt;br/&gt;
     &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-297&quot; title=&quot;llverfs: Open &amp;#39;/mnt/lustre/dir00007/file015&amp;#39; failed:Input/output error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-297&quot;&gt;&lt;del&gt;LU-297&lt;/del&gt;&lt;/a&gt; improve llverfs to display the current read/write performance&lt;/p&gt;

&lt;p&gt;Oleg Drokin : &lt;a href=&quot;http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;amp;a=commit&amp;amp;h=64f0f965dc80556739a3f86c15c0bb158b6289a2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;64f0f965dc80556739a3f86c15c0bb158b6289a2&lt;/a&gt;&lt;br/&gt;
Files : &lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lustre/utils/llverdev.c&lt;/li&gt;
	&lt;li&gt;lustre/utils/llverfs.c&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="19157" author="hudson" created="Thu, 11 Aug 2011 22:32:40 +0000"  >&lt;p&gt;Integrated in &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;http://build.whamcloud.com/images/16x16/blue.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;a href=&quot;http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/254/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;lustre-master &#187; i686,client,el5,ofa #254&lt;/a&gt;&lt;br/&gt;
     &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-297&quot; title=&quot;llverfs: Open &amp;#39;/mnt/lustre/dir00007/file015&amp;#39; failed:Input/output error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-297&quot;&gt;&lt;del&gt;LU-297&lt;/del&gt;&lt;/a&gt; improve llverfs to display the current read/write performance&lt;/p&gt;

&lt;p&gt;Oleg Drokin : &lt;a href=&quot;http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;amp;a=commit&amp;amp;h=64f0f965dc80556739a3f86c15c0bb158b6289a2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;64f0f965dc80556739a3f86c15c0bb158b6289a2&lt;/a&gt;&lt;br/&gt;
Files : &lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;lustre/utils/llverfs.c&lt;/li&gt;
	&lt;li&gt;lustre/utils/llverdev.c&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="152146" author="adilger" created="Fri, 13 May 2016 02:06:39 +0000"  >&lt;p&gt;Patch was landed for Lustre 2.0.67 and the OST object precreate was totally rewritten in 2.4.0.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                    <customfield id="customfield_10020" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>Bugzilla ID</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>24017.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10890" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10390" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzv8jj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10090" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4918</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10060" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Severity</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10022"><![CDATA[3]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        </customfields>
    </item>
</channel>
</rss>