<!-- 
RSS generated by JIRA (9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c) at Sat Feb 10 02:34:21 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>Whamcloud Community JIRA</title>
    <link>https://jira.whamcloud.com</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.4.14</version>
        <build-number>940014</build-number>
        <build-date>05-12-2023</build-date>
    </build-info>


<item>
            <title>[LU-10360] use Imperative Recovery logs for client-&gt;MDT/OST connections</title>
                <link>https://jira.whamcloud.com/browse/LU-10360</link>
                <project id="10000" key="LU">Lustre</project>
                    <description>&lt;p&gt;The Imperative Recovery (IR) feature landed in &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-19&quot; title=&quot;imperative recovery&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-19&quot;&gt;&lt;del&gt;LU-19&lt;/del&gt;&lt;/a&gt; created a dynamic list of active server NIDs on the MGS for purposes of speeding up client recovery when a target failed over to another server node.  A server failure triggered a notification from the MGS to the client to update its target NIDs to reconnect to the recovered server more quickly.&lt;/p&gt;

&lt;p&gt;It would be possible to extend this mechanism to also use the MGS IR log to do initial client mount, so that the MGS did not need to store the OST/MDT NIDs statically in the config log, but rather get the current NIDs directly from the dynamic MGS log.  This would facilitate Lustre running in configurations where the server NIDs are not static (e.g. cloud, DHCP, etc).  The initial connection to the MGS node(s) can already be done using the MGS hostname, since &lt;tt&gt;mount.lustre&lt;/tt&gt; will do DNS name resolution.&lt;/p&gt;

&lt;p&gt;Some care would be needed when OSTs are being registered with the MGS, especially in testing environments where OSTs are reformatted regularly and often use the same fsname, since this may allow OSTs to register with the MGS that do not actually belong to the same filesystem.&lt;/p&gt;
</description>
                <environment></environment>
        <key id="49665">LU-10360</key>
            <summary>use Imperative Recovery logs for client-&gt;MDT/OST connections</summary>
                <type id="2" iconUrl="https://jira.whamcloud.com/secure/viewavatar?size=xsmall&amp;avatarId=11311&amp;avatarType=issuetype">New Feature</type>
                                            <priority id="4" iconUrl="https://jira.whamcloud.com/images/icons/priorities/minor.svg">Minor</priority>
                        <status id="1" iconUrl="https://jira.whamcloud.com/images/icons/statuses/open.png" description="The issue is open and ready for the assignee to start work on it.">Open</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="wc-triage">WC Triage</assignee>
                                    <reporter username="adilger">Andreas Dilger</reporter>
                        <labels>
                    </labels>
                <created>Fri, 8 Dec 2017 21:08:57 +0000</created>
                <updated>Mon, 8 Jan 2024 20:57:51 +0000</updated>
                                                                                <due></due>
                            <votes>1</votes>
                                    <watches>13</watches>
                                                                            <comments>
                            <comment id="236982" author="nrutman" created="Wed, 14 Nov 2018 15:52:53 +0000"  >&lt;p&gt;What happens today if a server restarts on an unregistered NID? Does IR still work, and clients add the new NID to the failover list? If not true, it seems that could be a useful first step. In fact, if we can count on IR, then we can actually replace the entire NID list with whatever the IR NID is - a failover list of 1, which is the latest place the server started.&lt;/p&gt;

&lt;p&gt;If we can&apos;t count on IR (e.g. MGS is unavailable / unreachable), then a client would continue to use last known location, so maybe IR should include a list of failover NIDs provided by the newly restarting server. NIDs (and failovers) just become dynamic (last reported by server startup) rather than statically defined by the first registration. IIRC back in the day we decided not to add new NIDs to the config log (statically), but I think the dynamic path with IR makes much more sense.&#160;&#160;&lt;/p&gt;</comment>
                            <comment id="264246" author="adilger" created="Fri, 28 Feb 2020 12:22:28 +0000"  >&lt;p&gt;The IPv6 page discusses the use of IR for peer NID configuration.  The &lt;tt&gt;mgs_nidtbl_entry&lt;/tt&gt; already contains a list of all NIDs for a client:&lt;/p&gt;
&lt;div class=&quot;code panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;codeContent panelContent&quot;&gt;
&lt;pre class=&quot;code-java&quot;&gt;
struct mgs_nidtbl_entry {
        __u64           mne_version;    &lt;span class=&quot;code-comment&quot;&gt;/* table version of &lt;span class=&quot;code-keyword&quot;&gt;this&lt;/span&gt; entry */&lt;/span&gt;
        __u32           mne_instance;   &lt;span class=&quot;code-comment&quot;&gt;/* target instance # */&lt;/span&gt;
        __u32           mne_index;      &lt;span class=&quot;code-comment&quot;&gt;/* target index */&lt;/span&gt;
        __u32           mne_length;     &lt;span class=&quot;code-comment&quot;&gt;/* length of &lt;span class=&quot;code-keyword&quot;&gt;this&lt;/span&gt; entry - by bytes */&lt;/span&gt;
        __u8            mne_type;       &lt;span class=&quot;code-comment&quot;&gt;/* target type LDD_F_SV_TYPE_OST/MDT */&lt;/span&gt;
        __u8            mne_nid_type;   &lt;span class=&quot;code-comment&quot;&gt;/* type of nid(mbz). &lt;span class=&quot;code-keyword&quot;&gt;for&lt;/span&gt; ipv6. */&lt;/span&gt;
        __u8            mne_nid_size;   &lt;span class=&quot;code-comment&quot;&gt;/* size of each NID, by bytes */&lt;/span&gt;
        __u8            mne_nid_count;  &lt;span class=&quot;code-comment&quot;&gt;/* # of NIDs in buffer */&lt;/span&gt;
        union {
                lnet_nid_t nids[0];     &lt;span class=&quot;code-comment&quot;&gt;/* variable size buffer &lt;span class=&quot;code-keyword&quot;&gt;for&lt;/span&gt; NIDs. */&lt;/span&gt;
        } u;
};
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Since the MGS is &lt;em&gt;already&lt;/em&gt; needed at initial client mount time, not being able to access the MGS IR service at mount would not be a reduction in functionality compared to needing the MGS to fetch the config logs.&lt;/p&gt;

&lt;p&gt;Using MGS IR to announce server NIDs to clients would also remove the complexity of changing NIDs in the configuration logs, which currently requires a full filesystem shutdown (stop all clients and unmount servers) and rewriting the config logs.&lt;/p&gt;

&lt;p&gt;One improvement that would be needed is for the servers to re-announce their NIDs if they are changed &lt;b&gt;while the OST is mounted&lt;/b&gt; (e.g. expired DHCP lease, as opposed to the OST starting up on a new OSS).  That wouldn&apos;t be much different than handling a target failover to another server, but would be noticeable on the clients.&lt;/p&gt;</comment>
                            <comment id="275824" author="adilger" created="Mon, 20 Jul 2020 22:43:42 +0000"  >&lt;blockquote&gt;
&lt;p&gt;What happens today if a server restarts on an unregistered NID? Does IR still work, and clients add the new NID to the failover list? If not true, it seems that could be a useful first step.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;The current case is that the client will drop any NID that it receives that is not in the list of configured NIDs in the import.  This (AFAIK) is in &lt;tt&gt;mgc_apply_recover_logs()&lt;/tt&gt; where it checks for any existing NID on the import matching the NIDs in the IR entry:&lt;/p&gt;
&lt;div class=&quot;code panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;codeContent panelContent&quot;&gt;
&lt;pre class=&quot;code-java&quot;&gt;
                &lt;span class=&quot;code-comment&quot;&gt;/* iterate all nids to find one */&lt;/span&gt;
                &lt;span class=&quot;code-comment&quot;&gt;/* find uuid by nid */&lt;/span&gt;
                rc = -ENOENT;
                &lt;span class=&quot;code-keyword&quot;&gt;for&lt;/span&gt; (i = 0; i &amp;lt; entry-&amp;gt;mne_nid_count; i++) {
                        rc = client_import_find_conn(obd-&amp;gt;u.cli.cl_import,
                                                     entry-&amp;gt;u.nids[i],
                                                     (struct obd_uuid *)uuid);
                        &lt;span class=&quot;code-keyword&quot;&gt;if&lt;/span&gt; (rc == 0)
                                &lt;span class=&quot;code-keyword&quot;&gt;break&lt;/span&gt;;
                }
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;It was done this way to prevent misconfigured/rogue OSTs from connecting to the MGS and advertising &quot;&lt;tt&gt;lustre-OSTxxxx&lt;/tt&gt;&quot; as a target, but it is for a &lt;b&gt;different&lt;/b&gt; filesystem named &lt;tt&gt;lustre&lt;/tt&gt;.  This has happened in the test environment because of many concurrent &lt;tt&gt;lustre&lt;/tt&gt; filesystems and re-use of IP addresses for different test runs, and this causes very hard to diagnose problems.  At a minimum, there should be a tunable parameter that enables/disables the ability to connect an import to &quot;unknown&quot; NIDs.  A more complete solution to restrict connections to a specific filesystem UUID stored on MDT0000 would be very desirable, but would be for a separate ticket.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;In fact, if we can count on IR, then we can actually replace the entire NID list with whatever the IR NID is - a failover list of 1, which is the latest place the server started.  If we can&apos;t count on IR (e.g. MGS is unavailable / unreachable), then a client would continue to use last known location, so maybe IR should include a list of failover NIDs provided by the newly restarting server.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;The MGS is &lt;b&gt;required&lt;/b&gt; for initial mount, and is &lt;b&gt;desirable&lt;/b&gt; for normal operation, but not strictly required since the client stores its own failover list.  The &lt;tt&gt;mgs_nidtbl_entry&lt;/tt&gt; allows space for multiple NIDs, but these are intended to be the &lt;em&gt;current&lt;/em&gt; NID(s) of the target (i.e. if there are multiple interfaces for different LNets), but not the failover NIDs.  In a dynamic environment, it isn&apos;t necessarily even &lt;em&gt;possible&lt;/em&gt; to know what the failover NID is going to be in advance, so it isn&apos;t clear whether it is worthwhile to add the ability to specify failover NIDs via the IR NID table.&lt;/p&gt;

&lt;p&gt;It would make sense for the client to still parse the MGS config log (if NIDs are present) for any failover NIDs to handle the case of MGS failure.  It could also store any previously sent dynamic target NIDs for that target in its import list, for the case where the MGS is not working, it can try them as it does today, but it would be more desirable to have a real UUID for the filesystem beyond just &quot;$fsname-OSTxxxx&quot; to avoid errors during testing if that IP has been reassigned to another filesystem of the same name.&lt;/p&gt;</comment>
                            <comment id="277159" author="gerrit" created="Tue, 11 Aug 2020 00:08:36 +0000"  >&lt;p&gt;Amir Shehata (ashehata@whamcloud.com) uploaded a new patch: &lt;a href=&quot;https://review.whamcloud.com/39613&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/39613&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-10360&quot; title=&quot;use Imperative Recovery logs for client-&amp;gt;MDT/OST connections&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-10360&quot;&gt;LU-10360&lt;/a&gt; mgc: Use IR for client-&amp;gt;MDS/OST connections&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: 07b3c5e527ba6fe86d164d921acec8caafa5d757&lt;/p&gt;</comment>
                            <comment id="277920" author="gerrit" created="Sat, 22 Aug 2020 01:22:31 +0000"  >&lt;p&gt;Amir Shehata (ashehata@whamcloud.com) uploaded a new patch: &lt;a href=&quot;https://review.whamcloud.com/39709&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/39709&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-10360&quot; title=&quot;use Imperative Recovery logs for client-&amp;gt;MDT/OST connections&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-10360&quot;&gt;LU-10360&lt;/a&gt; mgs: Dynamic network updates&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: 09cc831918a3d661055ccfbf8f12ee8f13d91ac2&lt;/p&gt;</comment>
                            <comment id="279577" author="gerrit" created="Tue, 15 Sep 2020 01:19:51 +0000"  >&lt;p&gt;Amir Shehata (ashehata@whamcloud.com) uploaded a new patch: &lt;a href=&quot;https://review.whamcloud.com/39911&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/39911&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-10360&quot; title=&quot;use Imperative Recovery logs for client-&amp;gt;MDT/OST connections&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-10360&quot;&gt;LU-10360&lt;/a&gt; tests: test dynamic NIDs feature&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: d0bfbcb3bb643ce6dc33590bd937cb3c935ac88a&lt;/p&gt;</comment>
                            <comment id="280045" author="gerrit" created="Sat, 19 Sep 2020 14:11:46 +0000"  >&lt;p&gt;Oleg Drokin (green@whamcloud.com) merged in patch &lt;a href=&quot;https://review.whamcloud.com/39613/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/39613/&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-10360&quot; title=&quot;use Imperative Recovery logs for client-&amp;gt;MDT/OST connections&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-10360&quot;&gt;LU-10360&lt;/a&gt; mgc: Use IR for client-&amp;gt;MDS/OST connections&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: &lt;br/&gt;
Commit: 37be05eca3f4aee15c946656a77f56967c15253d&lt;/p&gt;</comment>
                            <comment id="285836" author="gerrit" created="Mon, 23 Nov 2020 23:50:53 +0000"  >&lt;p&gt;Amir Shehata (ashehata@whamcloud.com) uploaded a new patch: &lt;a href=&quot;https://review.whamcloud.com/40736&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/40736&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-10360&quot; title=&quot;use Imperative Recovery logs for client-&amp;gt;MDT/OST connections&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-10360&quot;&gt;LU-10360&lt;/a&gt; mgs: Mount to dynamically added networks&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: 4c68340088f2f56d16f6b1392de5ad7f7d139ff4&lt;/p&gt;</comment>
                            <comment id="321279" author="gerrit" created="Tue, 21 Dec 2021 10:52:39 +0000"  >&lt;p&gt;&quot;Etienne AUJAMES &amp;lt;eaujames@ddn.com&amp;gt;&quot; uploaded a new patch: &lt;a href=&quot;https://review.whamcloud.com/45905&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/45905&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-10360&quot; title=&quot;use Imperative Recovery logs for client-&amp;gt;MDT/OST connections&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-10360&quot;&gt;LU-10360&lt;/a&gt; mgc: Use IR for client-&amp;gt;MDS/OST connections&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: b2_12&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: b1c09656513f3198adf849182617e6eafef76954&lt;/p&gt;</comment>
                            <comment id="348685" author="gerrit" created="Tue, 4 Oct 2022 19:33:45 +0000"  >&lt;p&gt;&quot;Oleg Drokin &amp;lt;green@whamcloud.com&amp;gt;&quot; merged in patch &lt;a href=&quot;https://review.whamcloud.com/c/fs/lustre-release/+/39911/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/c/fs/lustre-release/+/39911/&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-10360&quot; title=&quot;use Imperative Recovery logs for client-&amp;gt;MDT/OST connections&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-10360&quot;&gt;LU-10360&lt;/a&gt; tests: test dynamic NIDs feature&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: &lt;br/&gt;
Commit: 2553f2fc8630061a8b6dbc5504d3f5277cb1cecf&lt;/p&gt;</comment>
                            <comment id="362844" author="gerrit" created="Wed, 15 Feb 2023 05:32:17 +0000"  >&lt;p&gt;&quot;Neil Brown &amp;lt;neilb@suse.de&amp;gt;&quot; uploaded a new patch: &lt;a href=&quot;https://review.whamcloud.com/c/fs/lustre-release/+/50000&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/c/fs/lustre-release/+/50000&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-10360&quot; title=&quot;use Imperative Recovery logs for client-&amp;gt;MDT/OST connections&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-10360&quot;&gt;LU-10360&lt;/a&gt; ldlm: remove client_import_find_conn()&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: 12cbeaf1fb7bc83d7a842b71d6e8a33601e085ce&lt;/p&gt;</comment>
                            <comment id="365195" author="gerrit" created="Wed, 8 Mar 2023 03:28:52 +0000"  >&lt;p&gt;&quot;Oleg Drokin &amp;lt;green@whamcloud.com&amp;gt;&quot; merged in patch &lt;a href=&quot;https://review.whamcloud.com/c/fs/lustre-release/+/50000/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/c/fs/lustre-release/+/50000/&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-10360&quot; title=&quot;use Imperative Recovery logs for client-&amp;gt;MDT/OST connections&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-10360&quot;&gt;LU-10360&lt;/a&gt; ldlm: remove client_import_find_conn()&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: &lt;br/&gt;
Commit: 14544bdca5cc42a3ea80fe665e332fe4c88b081a&lt;/p&gt;</comment>
                            <comment id="398879" author="adilger" created="Mon, 8 Jan 2024 20:50:21 +0000"  >&lt;p&gt;I think in conjunction with &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-10359&quot; title=&quot;remove NIDs from config llogs&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-10359&quot;&gt;LU-10359&lt;/a&gt; it should be possible to test a configuration that doesn&apos;t have server NIDs in the configuration at all, or totally incorrect NIDs in the config llog, to confirm that this is working properly.&lt;/p&gt;

&lt;p&gt;A conf-sanity test case should be added to confirm that this is working properly.&lt;/p&gt;</comment>
                            <comment id="398880" author="adilger" created="Mon, 8 Jan 2024 20:56:55 +0000"  >&lt;p&gt;It seems possible to transition systems to using dynamic NIDs by putting &quot;&lt;tt&gt;lctl set_param mgc.*.dynamic_nids=1&lt;/tt&gt;&quot; as one of the first records in the config llog, so that clients will allow IR to determine where the MDTs and OSTs are located.  After that is done, it would just be a matter of how long to create config llog records with the NIDs in them for backward compatibility.  The original patch landed as v2_13_55-106-g37be05eca3, so it is in all 2.14.0 and later releases.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10324">
                    <name>Cloners</name>
                                                                <inwardlinks description="is cloned by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Related</name>
                                            <outwardlinks description="is related to ">
                                        <issuelink>
            <issuekey id="13182">LU-10391</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="10116">LU-19</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="27500">LU-5881</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="61424">LU-14090</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="49664">LU-10359</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="52532">LU-11077</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="71793">LU-16086</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="75489">LU-16722</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="58213">LU-13306</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="58307">LU-13340</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="64039">LU-14668</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="63748">LU-14608</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                <customfield id="customfield_10890" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10390" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzzp2f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10090" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                </customfields>
    </item>
</channel>
</rss>