<!-- 
RSS generated by JIRA (9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c) at Sat Feb 10 01:32:07 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>Whamcloud Community JIRA</title>
    <link>https://jira.whamcloud.com</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.4.14</version>
        <build-number>940014</build-number>
        <build-date>05-12-2023</build-date>
    </build-info>


<item>
            <title>[LU-3233] tgt_cb_last_committed()) ASSERTION( ccb-&gt;llcc_exp-&gt;exp_obd == ccb-&gt;llcc_tgt-&gt;lut_obd ) failed: </title>
                <link>https://jira.whamcloud.com/browse/LU-3233</link>
                <project id="10000" key="LU">Lustre</project>
                    <description>&lt;p&gt;I see this running racer on a single node MDSCOUNT=2 llmount.sh filesystem. (I upgraded the LASSERT() to an LASSERTF().)&lt;/p&gt;

&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;LustreError: 10307:0:(mdt_recovery.c:390:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 4295820775, new: 4295609707 replay: 0. see LU-617.
LustreError: 10277:0:(tgt_lastrcvd.c:412:tgt_cb_last_committed()) ASSERTION( ccb-&amp;gt;llcc_exp-&amp;gt;exp_obd == ccb-&amp;gt;llcc_tgt-&amp;gt;lut_obd ) failed: exp_obd ffff88014fb182f8 != lut_obd ffff88016c536178

Breakpoint 2, lbug_with_loc (msgdata=0xffffffffa12c9360) at /root/lustre-release/libcfs/libcfs/linux/linux-debug.c:167
167	        libcfs_debug_msg(msgdata, &quot;LBUG\n&quot;);
(gdb) bt
#0  lbug_with_loc (msgdata=0xffffffffa12c9360)
    at /root/lustre-release/libcfs/libcfs/linux/linux-debug.c:167
#1  0xffffffffa12613e9 in tgt_cb_last_committed (env=&amp;lt;value optimized out&amp;gt;, 
    th=&amp;lt;value optimized out&amp;gt;, cb=0xffff88017377cb40, err=&amp;lt;value optimized out&amp;gt;)
    at /root/lustre-release/lustre/ptlrpc/../../lustre/target/tgt_lastrcvd.c:410
#2  0xffffffffa0874a24 in osd_trans_commit_cb (sb=&amp;lt;value optimized out&amp;gt;, 
    jcb=&amp;lt;value optimized out&amp;gt;, error=0)
    at /root/lustre-release/lustre/osd-ldiskfs/osd_handler.c:646
#3  0xffffffffa082610a in ldiskfs_journal_commit_callback (
    journal=&amp;lt;value optimized out&amp;gt;, txn=0xffff880177622a80)
    at /root/lustre-release/ldiskfs/ldiskfs/super.c:329
#4  0xffffffffa007584f in jbd2_journal_commit_transaction (journal=0xffff880150255800)
    at fs/jbd2/commit.c:1049
#5  0xffffffffa007b698 in kjournald2 (arg=0xffff880150255800) at fs/jbd2/journal.c:163
#6  0xffffffff81090626 in kthread (_create=0xffff880160ab15c8) at kernel/kthread.c:78
#7  0xffffffff8100c0ca in ?? () at arch/x86/kernel/entry_64.S:1211
#8  0x0000000000000000 in ?? ()
(gdb) p *((struct obd_device *) 0xffff88014fb182f8)
$2 = {
  obd_type = 0xffff88019705ca40, 
  obd_magic = 2874988271, 
  obd_name = &quot;lustre-MDT0001&quot;, &apos;\000&apos; &amp;lt;repeats 113 times&amp;gt;, 
  obd_uuid = {
    uuid = &quot;lustre-MDT0001_UUID&quot;, &apos;\000&apos; &amp;lt;repeats 20 times&amp;gt;
  }, 
...
(gdb) p *((struct obd_device *) 0xffff88016c536178)
$3 = {
  obd_type = 0xffff88019705ca40, 
  obd_magic = 2874988271, 
  obd_name = &quot;lustre-MDT0000&quot;, &apos;\000&apos; &amp;lt;repeats 113 times&amp;gt;, 
  obd_uuid = {
    uuid = &quot;lustre-MDT0000_UUID&quot;, &apos;\000&apos; &amp;lt;repeats 20 times&amp;gt;
  }, 
...
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;</description>
                <environment></environment>
        <key id="18586">LU-3233</key>
            <summary>tgt_cb_last_committed()) ASSERTION( ccb-&gt;llcc_exp-&gt;exp_obd == ccb-&gt;llcc_tgt-&gt;lut_obd ) failed: </summary>
                <type id="1" iconUrl="https://jira.whamcloud.com/secure/viewavatar?size=xsmall&amp;avatarId=11303&amp;avatarType=issuetype">Bug</type>
                                            <priority id="4" iconUrl="https://jira.whamcloud.com/images/icons/priorities/minor.svg">Minor</priority>
                        <status id="5" iconUrl="https://jira.whamcloud.com/images/icons/statuses/resolved.png" description="A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.">Resolved</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="1">Fixed</resolution>
                                        <assignee username="jhammond">John Hammond</assignee>
                                    <reporter username="jhammond">John Hammond</reporter>
                        <labels>
                            <label>mdt</label>
                    </labels>
                <created>Fri, 26 Apr 2013 02:16:51 +0000</created>
                <updated>Thu, 12 Sep 2013 23:59:35 +0000</updated>
                            <resolved>Thu, 12 Sep 2013 23:59:35 +0000</resolved>
                                    <version>Lustre 2.4.0</version>
                                    <fixVersion>Lustre 2.4.1</fixVersion>
                    <fixVersion>Lustre 2.5.0</fixVersion>
                                        <due></due>
                            <votes>0</votes>
                                    <watches>7</watches>
                                                                            <comments>
                            <comment id="57142" author="tappro" created="Fri, 26 Apr 2013 16:59:25 +0000"  >&lt;p&gt;so callback from second MDT comes to the first one, how intriguing. Jonh, I suspect that above message from mdt_last_rcvd_update() might be also because of that, can you add dump_stack() there to see call trace when that happens?&lt;/p&gt;</comment>
                            <comment id="57143" author="jhammond" created="Fri, 26 Apr 2013 17:33:51 +0000"  >&lt;p&gt;I tried something similar running with the patch from &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-3231&quot; title=&quot;fld_cache_lookup() copies fld_cache_entry onto lu_seq_range&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-3231&quot;&gt;&lt;del&gt;LU-3231&lt;/del&gt;&lt;/a&gt; &lt;a href=&quot;http://review.whamcloud.com/6171&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/6171&lt;/a&gt;. (Please see the comments on that patch.) I added an assertion like the one above to tgt_last_commit_cb_add():&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;@@ -435,6 +437,9 @@ int tgt_last_commit_cb_add(struct thandle *th, struct lu_target *tgt,
        struct dt_txn_commit_cb                 *dcb;
        int                                      rc;
 
+       LASSERTF(exp-&amp;gt;exp_obd == tgt-&amp;gt;lut_obd, 
+                &quot;exp_obd %p != lut_obd %p\n&quot;, exp-&amp;gt;exp_obd, tgt-&amp;gt;lut_obd);
+
        OBD_ALLOC_PTR(ccb);
        if (ccb == NULL)
                return -ENOMEM;
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;I got the following:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;LustreError: 7895:0:(tgt_lastrcvd.c:441:tgt_last_commit_cb_add()) ASSERTION( exp-&amp;gt;exp_obd == tgt-&amp;gt;lut_obd ) failed: exp_obd ffff88016a8842f8 != lut_obd ffff88016ec42178

(gdb) bt
#0  lbug_with_loc (msgdata=0xffffffffa091b300)
    at /root/lustre-release/libcfs/libcfs/linux/linux-debug.c:167
#1  0xffffffffa08b4f95 in tgt_last_commit_cb_add (th=0xffff880151342e80, 
    tgt=0xffff88016ec28128, exp=0xffff880162bc5000, transno=4294978364)
    at /root/lustre-release/lustre/ptlrpc/../../lustre/target/tgt_lastrcvd.c:440
#2  0xffffffffa0ccff92 in mdt_txn_stop_cb (env=0xffff880170ce5480, txn=0xffff880151342e80, 
    cookie=0xffff88016ec28000) at /root/lustre-release/lustre/mdt/mdt_recovery.c:568
#3  0xffffffffa06da313 in dt_txn_hook_stop (env=0xffff880170ce5480, txn=0xffff880151342e80)
    at /root/lustre-release/lustre/obdclass/dt_object.c:114
#4  0xffffffffa0c22d15 in osd_trans_stop (env=0xffff880170ce5480, th=0xffff880151342e80)
    at /root/lustre-release/lustre/osd-ldiskfs/osd_handler.c:835
#5  0xffffffffa0d88e34 in lod_trans_stop ()
#6  0xffffffffa07db07d in mdd_trans_stop (env=&amp;lt;value optimized out&amp;gt;, 
    mdd=&amp;lt;value optimized out&amp;gt;, result=&amp;lt;value optimized out&amp;gt;, handle=&amp;lt;value optimized out&amp;gt;)
    at /root/lustre-release/lustre/mdd/mdd_trans.c:67
#7  0xffffffffa07c1ad2 in mdd_attr_set (env=0xffff880170ce5480, obj=0xffff8801505ee3d0, 
    ma=&amp;lt;value optimized out&amp;gt;) at /root/lustre-release/lustre/mdd/mdd_object.c:884
#8  0xffffffffa0cd5f72 in mo_attr_set (info=0xffff88016fb80000, mfd=0xffff880152b608c0)
    at /root/lustre-release/lustre/include/md_object.h:569
#9  mdt_mfd_close (info=0xffff88016fb80000, mfd=0xffff880152b608c0)
    at /root/lustre-release/lustre/mdt/mdt_open.c:1769
#10 0xffffffffa0cd67d2 in mdt_close (info=0xffff88016fb80000)
    at /root/lustre-release/lustre/mdt/mdt_open.c:1898
#11 0xffffffffa0cad118 in mdt_req_handle (req=0xffff8801518b0000, 
    supported=0xffffffffa0d11080) at /root/lustre-release/lustre/mdt/mdt_handler.c:2983
#12 mdt_handle0 (req=0xffff8801518b0000, supported=0xffffffffa0d11080)
    at /root/lustre-release/lustre/mdt/mdt_handler.c:3335
#13 mdt_handle_common (req=0xffff8801518b0000, supported=0xffffffffa0d11080)
    at /root/lustre-release/lustre/mdt/mdt_handler.c:3377
#14 0xffffffffa0ce94c5 in mds_readpage_handle (req=&amp;lt;value optimized out&amp;gt;)
    at /root/lustre-release/lustre/mdt/mdt_mds.c:358
#15 0xffffffffa08755b8 in ptlrpc_server_handle_request (svcpt=0xffff88016fbfec00, 
    thread=&amp;lt;value optimized out&amp;gt;) at /root/lustre-release/lustre/ptlrpc/service.c:2016
#16 0xffffffffa087693e in ptlrpc_main (arg=0xffff88016f49a6c0)
    at /root/lustre-release/lustre/ptlrpc/service.c:2522
#17 0xffffffff8100c0ca in ?? () at arch/x86/kernel/entry_64.S:1211
#18 0x0000000000000000 in ?? ()
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;10 mdt_close (info=0xffff88016fb80000)

        (gdb) p o
        $43 = (struct mdt_object *) 0xffff880154e60c10
        (gdb) p/x o-&amp;gt;mot_header.loh_fid
        $47 = {
          f_seq = 0x280000401,
          f_oid = 0xccc,
          f_ver = 0x0
        }
        (gdb) p o-&amp;gt;mot_obj.mo_lu.lo_dev
        $51 = (struct lu_device *) 0xffff88016ec28000 ### lustre-MDT0000


9 mdt_mfd_close (info=0xffff88016fb80000, mfd=0xffff880152b608c0)

8 mo_attr_set(...)

7 mdd_attr_set mdd_attr_set (env=0xffff880170ce5480, obj=0xffff8801505ee3d0, ma=&amp;lt;value optimized out&amp;gt;)

        p obj-&amp;gt;mo_lu.lo_dev-&amp;gt;ld_obd
        $8 = (struct obd_device *) 0xffff88016ec441b8 ### lustre-MDD0000

        p *mdd_obj-&amp;gt;mod_obj.mo_lu.lo_dev-&amp;gt;ld_obd
        $15 = (struct obd_device *) 0xffff88016ec441b8 ### lustre-MDD0000

        p (struct mdd_device *)mdd_obj-&amp;gt;mod_obj.mo_lu.lo_dev
        $20 = (struct mdd_device *) 0xffff88016ec2bc00 ### lustre-MDD0000


6 mdd_trans_stop(env, mdd, result, handle)
5 lod_trans_stop(...)
4 osd_trans_stop(env, th)
3 dt_txn_hook_stop(env, txn)
2 mdt_txn_stop_cb (env=0xffff880170ce5480, txn=0xffff880151342e80, cookie=0xffff88016ec28000)

        ### mti == info from f10

        p (struct mdt_device *) cookie
        $23 = (struct mdt_device *) 0xffff88016ec28000    ### lustre-MDT0000

        (gdb) p mdt
        $28 = (struct mdt_device *) 0xffff88016ec28000    ### lustre-MDT0000

        (gdb) p mdt-&amp;gt;mdt_md_dev.md_lu_dev-&amp;gt;ld_obd
        $29 = (struct obd_device *) 0xffff88016ec42178    ### lustre-MDT0000

        (gdb) p *mdt-&amp;gt;mdt_md_dev.md_lu_dev-&amp;gt;ld_obd        ### lustre-MDT0000



        (gdb) p mti-&amp;gt;mti_mdt
        $39 = (struct mdt_device *) 0xffff88016cd03000    ### lustre-MDT0001

        (gdb) p mti-&amp;gt;mti_mdt-&amp;gt;mdt_md_dev.md_lu_dev-&amp;gt;ld_obd
        $40 = (struct obd_device *) 0xffff88016a8842f8    ### lustre-MDT0001


        (gdb) p mti-&amp;gt;mti_exp
        $38 = (struct obd_export *) 0xffff880162bc5000

        (gdb) p mti-&amp;gt;mti_pill-&amp;gt;rc_req
        $42 = (struct ptlrpc_request *) 0xffff8801518b0000

        (gdb) p mti-&amp;gt;mti_pill-&amp;gt;rc_req-&amp;gt;rq_export
        $37 = (struct obd_export *) 0xffff880162bc5000

        (gdb) p mti-&amp;gt;mti_pill-&amp;gt;rc_req-&amp;gt;rq_export-&amp;gt;exp_obd
        $35 = (struct obd_device *) 0xffff88016a8842f8    ### lustre-MDT0001

1 tgt_last_commit_cb_add (th=0xffff880151342e80, tgt=0xffff88016ec28128, exp=0xffff880162bc5000, transno=4294978364)
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;I have tuned racer to reproduce this more readily so please let me know if you to see more of the stack.&lt;/p&gt;</comment>
                            <comment id="57182" author="tappro" created="Sat, 27 Apr 2013 07:29:23 +0000"  >&lt;p&gt;John, it is clear that mdt from cookie is not the same as one in mti-&amp;gt;mti_mdt. I think that mti_mdt must be correct one, it is taken from request export, and I see that wrong mdt is taken from object which in turn was taken in mdt_close() as mfd_object. So I suspect we have found wrong mfd due to hash collision:&lt;br/&gt;
    mfd = mdt_handle2mfd(info, &amp;amp;info-&amp;gt;mti_ioepoch-&amp;gt;handle);&lt;/p&gt;

&lt;p&gt;To check that you can add extra check in mdt_handle2mfd() and check that mfd_object has the same device as info-&amp;gt;mti_mdt. Such mfd must be ignored and not used. &lt;/p&gt;</comment>
                            <comment id="57214" author="jhammond" created="Mon, 29 Apr 2013 00:21:32 +0000"  >&lt;p&gt;From my read of the handles code, it doesn&apos;t seem likely that mdt_mfd_new() would issue the same cookie twice, as it&apos;s not really a hash value but a counter.&lt;/p&gt;</comment>
                            <comment id="57218" author="tappro" created="Mon, 29 Apr 2013 02:53:25 +0000"  >&lt;p&gt;we need to trace down where this cookie came from, maybe this is client side issue and wrong cookie is sent to the server or something similar.&lt;/p&gt;</comment>
                            <comment id="57411" author="jhammond" created="Wed, 1 May 2013 02:15:06 +0000"  >&lt;p&gt;In mtd_reint_open() should we be doing lookup if the MDS_OPEN_BY_FID flag is set? We already have what should be the child FID in that case. Moreover if we trust that FID when MDS_OPEN_BY_FID is set then I can no longer reproduce this. But I cannot say I fully understand this code.&lt;/p&gt;</comment>
                            <comment id="60826" author="jhammond" created="Tue, 18 Jun 2013 18:09:42 +0000"  >&lt;p&gt;I think this is because ll_och_fill() may use the wrong FID under certain circumstances. Consider the following:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;ll_intent_file_open()
        lmv_intent_lock()
                mdc_intent_lock()
                        mdc_finish_intent_lock()
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;And assume that mdc_finish_intent_lock() returns -ESTALE because the FID in body does not match either of the FIDs in op_data. Then ll_intent_file_open() see -ESTALE and we have:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;ll_intent_file_open(file, ..., it)
        ll_release_openhandle(dentry, it)
                OBD_ALLOC(och)
                ll_och_fill(md_exp, lli, it, och)
                        req = it-&amp;gt;d.lustre.it_data
                        body = req_capsule_server_get(req...)
                        och-&amp;gt;och_fh = body-&amp;gt;handle
                        och-&amp;gt;och_fid = lli-&amp;gt;lli_fid
                ll_close_inode_openhandle(md_exp, inode, och)
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;But ll_och_fill() should be using the FID from body for this close, not from the lli. Then lmv_set_open_replay_data() uses this bad FID to choose the wrong target, and so on.&lt;/p&gt;

&lt;p&gt;It seems like the fix is to have ll_och_fill() use the FID from body in general. Perhaps mdt_close() should also check that the FID sent matched the FID of the object in mfd. Now it totally trusts the cookie/mfd, and we only notice the mismatch when DNE is enabled and only because of the assertion in tgt_cb_last_committed().&lt;/p&gt;</comment>
                            <comment id="60840" author="jhammond" created="Wed, 19 Jun 2013 00:56:32 +0000"  >&lt;p&gt;Please see &lt;a href=&quot;http://review.whamcloud.com/6695&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/6695&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="61998" author="green" created="Wed, 10 Jul 2013 03:23:19 +0000"  >&lt;p&gt;So, do I get this right, a data from client crashes server?&lt;br/&gt;
Let&apos;s hold on on the client patch and fix the server first so that it does not crash!&lt;/p&gt;</comment>
                            <comment id="62044" author="jhammond" created="Wed, 10 Jul 2013 17:58:54 +0000"  >&lt;p&gt;&amp;gt; So, do I get this right, a data from client crashes server?&lt;/p&gt;

&lt;p&gt;Yes, the och gets the wrong FID, so LMV routes the right handle to the wrong MDT (on the same MDS as the right MDT). The MDT totally trusts the handle (until we hit the assertion above).&lt;/p&gt;

&lt;p&gt;&amp;gt; Let&apos;s hold on on the client patch and fix the server first so that it does not crash!&lt;/p&gt;

&lt;p&gt;OK good.&lt;/p&gt;

&lt;p&gt;I&apos;m concerned that once we don&apos;t trust handles then class_handle2object() and the synchronization (or lack of) around it needs a total rethink. Can someone suggest a less drastic solution?&lt;/p&gt;</comment>
                            <comment id="62052" author="green" created="Wed, 10 Jul 2013 20:58:38 +0000"  >&lt;p&gt;Right, there&apos;s normally an overtrust to those handles and underchecking their validness (mostly in locks), but granted most of the time it does not end up with such drastic crashes on server if you get the handle wrong.&lt;/p&gt;</comment>
                            <comment id="62054" author="jhammond" created="Wed, 10 Jul 2013 21:03:55 +0000"  >&lt;p&gt;Indeed. In any case I think I may have a reasonable fix for the MDT case without touching too much else.&lt;/p&gt;

&lt;p&gt;Please see &lt;a href=&quot;http://review.whamcloud.com/6938&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/6938&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="62612" author="jhammond" created="Fri, 19 Jul 2013 14:30:32 +0000"  >&lt;p&gt;Both patches landed to master.&lt;/p&gt;

&lt;p&gt;Dropping to minor as Jinshan and I had discussed whether the logic of using M_CHECK_STALE and having the server return the FID of the new file (along with an open handle) really made any sense here. It appeared that someone planned an optimization around returning the new FID but it never got implemented. I believe that open might be made more robust by having the server ignore any supplied name if MDS_OPEN_BY_FID is set, and perhaps by having the client not supply a name when it sets MDS_OPEN_BY_FID. More analysis is needed however.&lt;/p&gt;</comment>
                            <comment id="66403" author="jhammond" created="Wed, 11 Sep 2013 19:40:24 +0000"  >&lt;p&gt;See Niu&apos;s patches to remove M_CHECK_STALE and make open by FID more sane.&lt;/p&gt;</comment>
                            <comment id="66556" author="pjones" created="Thu, 12 Sep 2013 23:59:35 +0000"  >&lt;p&gt;Landed for 2.4.1 and 2.5.0&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Related</name>
                                            <outwardlinks description="is related to ">
                                        <issuelink>
            <issuekey id="18548">LU-3231</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="20358">LU-3765</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                            <customfield id="customfield_10890" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10390" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzvp3r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10090" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7934</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10060" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Severity</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10022"><![CDATA[3]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        </customfields>
    </item>
</channel>
</rss>