<!-- 
RSS generated by JIRA (9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c) at Sat Feb 10 01:53:07 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>Whamcloud Community JIRA</title>
    <link>https://jira.whamcloud.com</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.4.14</version>
        <build-number>940014</build-number>
        <build-date>05-12-2023</build-date>
    </build-info>


<item>
            <title>[LU-5626] Corruption of MDT &#8220;..&#8221; entry in non-HTree ldiskfs directories</title>
                <link>https://jira.whamcloud.com/browse/LU-5626</link>
                <project id="10000" key="LU">Lustre</project>
                    <description>&lt;p&gt;&lt;a href=&quot;https://jira.whamcloud.com/browse/LU-2638&quot; title=&quot;corruption of MDT &amp;quot;..&amp;quot; entry in some ldiskfs directories&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-2638&quot;&gt;&lt;del&gt;LU-2638&lt;/del&gt;&lt;/a&gt; reported directory entry corruption related to FID-in-dirent and the &#8220;..&#8221; entry in HTree directories.&lt;br/&gt;
We have since discovered an identical problem in non-HTree directories.&lt;/p&gt;

&lt;p&gt;This is essentially exactly the same problem, but it manifests itself slightly different in non-HTree directories.  The &#8220;..&#8221; entry must remain as the second entry in the directory block (FSCK demands this), and when a directory created under 1.8 (now on a 2.4+ server with dirdata enabled) is moved to a new parent, the &#8220;..&#8221; entry is updated.  Exactly as happened in &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-2638&quot; title=&quot;corruption of MDT &amp;quot;..&amp;quot; entry in some ldiskfs directories&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-2638&quot;&gt;&lt;del&gt;LU-2638&lt;/del&gt;&lt;/a&gt;, the FID is added to the &#8220;..&#8221; entry without regards to whether or not there is sufficient space in the second position in the directory block.&lt;/p&gt;

&lt;p&gt;In the lucky case where space is already available in the second entry in a directory, the &#8220;..&#8221; entry is -recreated in the same place, FID attached.  If not, it is created in the next available space of sufficient size.  This causes complaints from FSCK, and when FSCK repairs this, it places the updated &#8220;..&#8221; immediately after &#8220;.&#8221; again, which causes it to overlap the next entry in the directory block.  This entry - which is for a real user created file, not . or .. - is moved to Lost + found.&lt;/p&gt;

&lt;p&gt;This is because add_dirent_to_buf (used when not in a dx directory) has the same bug as &#8220;ldiskfs_update_dotdot&#8221;, which was fixed in &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-2638&quot; title=&quot;corruption of MDT &amp;quot;..&amp;quot; entry in some ldiskfs directories&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-2638&quot;&gt;&lt;del&gt;LU-2638&lt;/del&gt;&lt;/a&gt;.  Because the structure of add_dirent_to_buf is a bit different, the fix looks different as well.&lt;/p&gt;

&lt;p&gt;I don&#8217;t have time at the moment to commit &amp;amp; update the new ldiskfs patch file to Gerrit, but I will do so shortly.  In the meantime, I&#8217;m attaching the new patch file &amp;amp; the resulting namei.c to this bug.&lt;/p&gt;

&lt;p&gt;The patch is a bit ugly and could probably use improvement, but in my testing, it does fix the bug.&lt;/p&gt;

&lt;p&gt;I&apos;ll share replication details in a comment.&lt;/p&gt;

&lt;p&gt;One &#8216;technical debt&#8217; problem with this patch:&lt;br/&gt;
This patch, and the one for &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-2638&quot; title=&quot;corruption of MDT &amp;quot;..&amp;quot; entry in some ldiskfs directories&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-2638&quot;&gt;&lt;del&gt;LU-2638&lt;/del&gt;&lt;/a&gt;, do not simply avoid writing the FID in to the &#8220;..&#8221; entry.  In fact, they avoid writing the entire data section on to the &#8220;..&#8221; entry, so if there were a pre-existing &#8220;..&#8221; entry with something else in data other than the FID, that would be lost on directory moves.  Currently, it appears that FID-in-dirent is the only user of this extra section.&lt;/p&gt;</description>
                <environment>Lustre 2.4 or newer, file system upgraded from 1.8</environment>
        <key id="26582">LU-5626</key>
            <summary>Corruption of MDT &#8220;..&#8221; entry in non-HTree ldiskfs directories</summary>
                <type id="1" iconUrl="https://jira.whamcloud.com/secure/viewavatar?size=xsmall&amp;avatarId=11303&amp;avatarType=issuetype">Bug</type>
                                            <priority id="1" iconUrl="https://jira.whamcloud.com/images/icons/priorities/blocker.svg">Blocker</priority>
                        <status id="5" iconUrl="https://jira.whamcloud.com/images/icons/statuses/resolved.png" description="A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.">Resolved</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="1">Fixed</resolution>
                                        <assignee username="bogl">Bob Glossman</assignee>
                                    <reporter username="paf">Patrick Farrell</reporter>
                        <labels>
                            <label>e2fsck</label>
                            <label>e2fsprogs</label>
                            <label>patch</label>
                    </labels>
                <created>Mon, 15 Sep 2014 21:20:49 +0000</created>
                <updated>Wed, 2 Dec 2015 13:51:29 +0000</updated>
                            <resolved>Fri, 7 Nov 2014 19:55:17 +0000</resolved>
                                    <version>Lustre 2.7.0</version>
                    <version>Lustre 2.4.3</version>
                    <version>Lustre 2.5.3</version>
                                    <fixVersion>Lustre 2.7.0</fixVersion>
                    <fixVersion>Lustre 2.5.4</fixVersion>
                                        <due></due>
                            <votes>0</votes>
                                    <watches>12</watches>
                                                                            <comments>
                            <comment id="94101" author="paf" created="Mon, 15 Sep 2014 21:23:12 +0000"  >&lt;p&gt;This problem can be reproduced by formatting a file system under 1.8 (or, probably, earlier versions of 2.x), creating a directory with at least one file in it, stopping the file system &amp;amp; adding the dirdata attribute to the MDT, then starting the same file system with 2.4 or newer (bug exists in master as well) and moving that directory to a new location.&lt;/p&gt;

&lt;p&gt;Running fsck will show errors similar to those reported in &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-2638&quot; title=&quot;corruption of MDT &amp;quot;..&amp;quot; entry in some ldiskfs directories&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-2638&quot;&gt;&lt;del&gt;LU-2638&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;This is an example of a directory block AFTER the problem has occurred. Note the presence of the first entry, &quot;.&quot;. Its length (24 decimal) includes the old &quot;..&quot; entry, which is seen in the second 12 bytes, but is not read because it&apos;s inside the rec_len of the first entry. Looking further down the directory block, we see a normal file dentry, then after that, we see the new &quot;..&quot; entry (look for &apos;2e2e&apos;), which includes the FID of the new parent. (It is followed by other file dentries.)&lt;br/&gt;
debugfs: bd 237582&lt;br/&gt;
0000 b782 0300 1800 0102 2e00 0000 8f8b cb02 ................&lt;br/&gt;
^^^^^^^^^ ^^ ^^ ^^&lt;br/&gt;
inode | | &quot;.&quot;&lt;br/&gt;
reclen |&lt;br/&gt;
namelen&lt;br/&gt;
0020 0c00 0202 2e2e 0000 b982 0300 2400 0812 ............$...&lt;br/&gt;
^^^^^^^^^ ^^ ^^&lt;br/&gt;
inode | |&lt;br/&gt;
reclen |&lt;br/&gt;
namelen&lt;br/&gt;
0040 4361 7461 6c79 7374 0011 0000 0000 0003 Catalyst........&lt;br/&gt;
^^^^^^^^^^^^^^^^^^^&lt;br/&gt;
&quot;Catalyst&quot;&lt;br/&gt;
0060 82b9 2475 25ab 0000 0000 7374 1f80 0702 ..$u%.....st....&lt;br/&gt;
^^^^^^^^^&lt;br/&gt;
inode&lt;br/&gt;
0100 2000 0212 2e2e 0011 0000 0002 0000 6dd8 .............m.&lt;br/&gt;
^^ ^^ ^^ ^^ ^^^^^^^^^^^^^^^^^^^&lt;br/&gt;
reclen | &quot;..&quot; LEN ^^^^^^^^^^^^^^^^^^^&lt;br/&gt;
namelen new parent fid ****&lt;br/&gt;
0120 0001 9551 0000 0000 6500 0000 bc82 0300 ...Q....e.......&lt;br/&gt;
^^^^^^^^^^^^^^^^^^^^^^&lt;br/&gt;
**********************&lt;br/&gt;
0140 2800 0e11 434d 616b 654c 6973 7473 2e74 (...CMakeLists.t&lt;br/&gt;
...&lt;/p&gt;</comment>
                            <comment id="94102" author="paf" created="Mon, 15 Sep 2014 21:27:50 +0000"  >&lt;p&gt;Patch file &amp;amp; resulting namei.c from ldiskfs &quot;make&quot; of current master+this patch.&lt;/p&gt;</comment>
                            <comment id="94103" author="paf" created="Mon, 15 Sep 2014 21:30:39 +0000"  >&lt;p&gt;The attached patch attempts to resolve the issue by special casing &quot;..&quot;.  A special, alternate length for &quot;..&quot; is calculated, which does not include the data section.  When a dotdot entry is identified, the space checking code first checks to see if there is sufficient space for the data secton; if there is not, it then checks for space for the special alternate length.  This guarantees &quot;..&quot; will be placed on top of the pre-existing &quot;..&quot; entry, even when there is not additional space for the FID.&lt;/p&gt;

&lt;p&gt;The result of this space check is recorded and is used to determine whether or not to write the data section.&lt;/p&gt;</comment>
                            <comment id="94156" author="paf" created="Tue, 16 Sep 2014 15:45:32 +0000"  >&lt;p&gt;Patch is here:&lt;br/&gt;
&lt;a href=&quot;http://review.whamcloud.com/11939&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/11939&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Local testing suggests this resolves the issue.&lt;/p&gt;</comment>
                            <comment id="94321" author="paf" created="Wed, 17 Sep 2014 22:05:04 +0000"  >&lt;p&gt;We also saw file systems taking errors and getting remounted read-only, and we were initially unable to figure out why.  It turns out that when an incorrect/damaged (IE, &quot;..&quot; entry in the wrong place) non-HTree directory is converted to an HTree directory, the conversion goes badly wrong, and the resulting directory is badly corrupt.&lt;/p&gt;

&lt;p&gt;With the patch for this bug, it&apos;s no longer possible to get in the bad state.  I thought I&apos;d share the errors here so others who hit this bug have a better chance of finding this JIRA ticket.&lt;/p&gt;

&lt;p&gt;Here&apos;s what the resulting errors look like - The key thing is &quot;rec_len=2049&quot;, which we&apos;ve always seen in this situation:&lt;br/&gt;
LDISKFS-fs error (device sdi): ldiskfs_dx_find_entry: bad entry in directory #32789: rec_len % 4 != 0 - block=16755offset=24(24), inode=0, rec_len=2049, name_len=0&lt;br/&gt;
Aborting journal on device sdi-8.&lt;br/&gt;
LDISKFS-fs (sdi): Remounting filesystem read-only&lt;br/&gt;
LDISKFS-fs error (device sdi): ldiskfs_dx_find_entry: bad entry in directory #32789: rec_len % 4 != 0 - block=16755offset=24(24), inode=0, rec_len=2049, name_len=0&lt;br/&gt;
Lustre: 2208:0:(mdd_dir.c:2926:mdd_rename()) cent5602-MDD0000: sp obj dotdot delete error: rc = -2&lt;br/&gt;
Lustre: 2208:0:(mdd_dir.c:2933:mdd_rename()) cent5602-MDD0000: sp obj dotdot insert error: rc = -30&lt;br/&gt;
LDISKFS-fs error (device sdi) in add_dirent_to_buf: Journal has aborted&lt;br/&gt;
Lustre: 2208:0:(mdd_dir.c:2942:mdd_rename()) sp obj fix error: rc = -30&lt;br/&gt;
LustreError: 2208:0:(osd_io.c:1595:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30&lt;br/&gt;
LustreError: 2208:0:(osd_handler.c:1126:osd_trans_stop()) Failure in transaction hook: -30&lt;br/&gt;
LustreError: 2208:0:(osd_handler.c:1135:osd_trans_stop()) Failure to stop transaction: -30&lt;br/&gt;
LustreError: 2205:0:(osd_handler.c:910:osd_trans_commit_cb()) transaction @0xffff88022e004c00 commit error: 2&lt;/p&gt;</comment>
                            <comment id="97035" author="simmonsja" created="Wed, 22 Oct 2014 16:39:19 +0000"  >&lt;p&gt;Once this lands the ldiskfs patches for SLES12 and RHEL7 will need to be updated.&lt;/p&gt;</comment>
                            <comment id="97907" author="yong.fan" created="Thu, 30 Oct 2014 03:10:28 +0000"  >&lt;p&gt;James, you mean the ldiskfs/kernel_patches/patches/sles11sp2/ext4-data-in-dirent.patch should be updated? or anything else?&lt;/p&gt;</comment>
                            <comment id="97919" author="simmonsja" created="Thu, 30 Oct 2014 10:23:23 +0000"  >&lt;p&gt;Now these changes need to be integrated into the upcoming SLES12 and RHEL7 ldiskfs work.&lt;/p&gt;</comment>
                            <comment id="98073" author="simmonsja" created="Fri, 31 Oct 2014 17:40:27 +0000"  >&lt;p&gt;To clarify RHEL7 and SLES12 server side support are both 2.8 feature so this ticket is safe to close. Just need to integrate this work into those distros for 2.8.&lt;/p&gt;</comment>
                            <comment id="98193" author="simmonsja" created="Mon, 3 Nov 2014 17:29:05 +0000"  >&lt;p&gt;Looks like we will need this for SLES11 SP3 as well.&lt;/p&gt;</comment>
                            <comment id="98436" author="bogl" created="Wed, 5 Nov 2014 18:17:23 +0000"  >&lt;p&gt;for sles11sp3:&lt;br/&gt;
&lt;a href=&quot;http://review.whamcloud.com/12585&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/12585&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="98539" author="bogl" created="Thu, 6 Nov 2014 17:37:13 +0000"  >&lt;p&gt;patched namei.c from sles11sp3 requested by comment in &lt;a href=&quot;http://review.whamcloud.com/#/c/12585&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/#/c/12585&lt;/a&gt;.  to be absolutely clear this is the ldiskfs/namei.c after the full, updated patch series is applied, not just the 1 refreshed data-in-dirent patch.&lt;/p&gt;</comment>
                            <comment id="98682" author="adilger" created="Fri, 7 Nov 2014 18:59:50 +0000"  >&lt;p&gt;SLES 11 SP3 patch landed, need one for RHEL 7 ldiskfs series in &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-5022&quot; title=&quot;support for 3.10 rhel7 linux kernel&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-5022&quot;&gt;&lt;del&gt;LU-5022&lt;/del&gt;&lt;/a&gt;. I think it is fine to land the current ldiskfs series and then add this patch separately. &lt;/p&gt;</comment>
                            <comment id="98684" author="bogl" created="Fri, 7 Nov 2014 19:03:34 +0000"  >&lt;p&gt;Already talked to Yang Sheng about el7.  The latest refresh of &lt;a href=&quot;http://review.whamcloud.com/#/c/10249&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/#/c/10249&lt;/a&gt; includes changes that are supposed to address the issue.  Hopefully the el7 ldiskfs patches will be landing soon.&lt;/p&gt;</comment>
                            <comment id="98693" author="pjones" created="Fri, 7 Nov 2014 19:55:18 +0000"  >&lt;p&gt;ok then it sounds like the recent landing to master of the SLES fix means that this can be marked as resolved for 2.7&lt;/p&gt;</comment>
                            <comment id="98700" author="simmonsja" created="Fri, 7 Nov 2014 21:32:29 +0000"  >&lt;p&gt;We should wait until 2.8 to land the ldiskfs series for RHEL7.&lt;/p&gt;</comment>
                            <comment id="100614" author="gerrit" created="Wed, 3 Dec 2014 23:28:20 +0000"  >&lt;p&gt;Jian Yu (jian.yu@intel.com) uploaded a new patch: &lt;a href=&quot;http://review.whamcloud.com/12926&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/12926&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-5626&quot; title=&quot;Corruption of MDT &#8220;..&#8221; entry in non-HTree ldiskfs directories&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-5626&quot;&gt;&lt;del&gt;LU-5626&lt;/del&gt;&lt;/a&gt; ldiskfs: update non-htree dotdot in rename&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: b2_5&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: 2caf4d3a14df0974e1d96716f0f376f94be3a4ab&lt;/p&gt;</comment>
                            <comment id="100615" author="gerrit" created="Wed, 3 Dec 2014 23:34:48 +0000"  >&lt;p&gt;Jian Yu (jian.yu@intel.com) uploaded a new patch: &lt;a href=&quot;http://review.whamcloud.com/12927&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/12927&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-5626&quot; title=&quot;Corruption of MDT &#8220;..&#8221; entry in non-HTree ldiskfs directories&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-5626&quot;&gt;&lt;del&gt;LU-5626&lt;/del&gt;&lt;/a&gt; ldiskfs: update non-htree dotdot in rename&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: b2_5&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: 2b4f7f9612742ce7846646f6f50276080fdf398c&lt;/p&gt;</comment>
                            <comment id="101032" author="gerrit" created="Tue, 9 Dec 2014 00:41:11 +0000"  >&lt;p&gt;Oleg Drokin (oleg.drokin@intel.com) merged in patch &lt;a href=&quot;http://review.whamcloud.com/12926/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/12926/&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-5626&quot; title=&quot;Corruption of MDT &#8220;..&#8221; entry in non-HTree ldiskfs directories&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-5626&quot;&gt;&lt;del&gt;LU-5626&lt;/del&gt;&lt;/a&gt; ldiskfs: update non-htree dotdot in rename&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: b2_5&lt;br/&gt;
Current Patch Set: &lt;br/&gt;
Commit: 76a7bae58006e4f6d1c13216df8cedda85e5e911&lt;/p&gt;</comment>
                            <comment id="101033" author="gerrit" created="Tue, 9 Dec 2014 00:41:18 +0000"  >&lt;p&gt;Oleg Drokin (oleg.drokin@intel.com) merged in patch &lt;a href=&quot;http://review.whamcloud.com/12927/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/12927/&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-5626&quot; title=&quot;Corruption of MDT &#8220;..&#8221; entry in non-HTree ldiskfs directories&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-5626&quot;&gt;&lt;del&gt;LU-5626&lt;/del&gt;&lt;/a&gt; ldiskfs: update non-htree dotdot in rename&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: b2_5&lt;br/&gt;
Current Patch Set: &lt;br/&gt;
Commit: 639aca79b2c87aae2adf16463d50b9318f7429e5&lt;/p&gt;</comment>
                            <comment id="132579" author="adilger" created="Wed, 4 Nov 2015 02:53:09 +0000"  >&lt;p&gt;Attached is a script that can be used to recover the files from lost+found in this case, when the output from e2fsck is available.  The script itself is not ready for distribution use because it hard-codes pathnames to log files and such, but was useful in recovering problematic files.&lt;/p&gt;

&lt;p&gt;This could theoretically also be handled within e2fsck by moving the second entry (if otherwise valid) to another spot in the directory before clobbering it with &quot;..&quot;, but I haven&apos;t looked at e2fsck to know of the complexity of this (it would have to know where to put the entry to avoid making it unreachable in htree directories, or if the directory doesn&apos;t have enough space).&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Related</name>
                                            <outwardlinks description="is related to ">
                                        <issuelink>
            <issuekey id="24605">LU-5022</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="33030">LU-7399</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="15800" name="ext4-data-in-dirent-dotdot-fixes.patch" size="2061" author="paf" created="Mon, 15 Sep 2014 21:27:50 +0000"/>
                            <attachment id="19518" name="ll_fix_mdt_lost_found.sh" size="1145" author="adilger" created="Wed, 4 Nov 2015 02:53:09 +0000"/>
                            <attachment id="16323" name="namei.c" size="96187" author="bogl" created="Thu, 6 Nov 2014 17:37:13 +0000"/>
                            <attachment id="15801" name="namei.c" size="94205" author="paf" created="Mon, 15 Sep 2014 21:27:50 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                            <customfield id="customfield_10890" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10390" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzwwbr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10090" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>15741</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10060" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Severity</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10022"><![CDATA[3]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        </customfields>
    </item>
</channel>
</rss>