Production, Lustre 2.12.7 on router and computes, Lustre 2.12.9 + patches on servers
peer_credits = 42
infiniband (mofed 5.4 on router and on computes, mofed 4.7 on servers)
3
9223372036854775807
Description
A LNet router crash regularly with OOM on a compute partition at the CEA.
Each time, the router complains about a compute node (with RDMA timeout) and then crash with OOM.
This issue seems to be linked to a defective compute rack or infiniband interface, but this should not cause the LNet router to crash.
Kdumps information:
On the peer interface (lnet_peer_ni) to the faulty compute:
tx credits: ~ -4500
I read the msg tx queue (lpni_txq) and sort the messages by NID sources: for 69 NIDs I count 42 messages (peer_credits value) blocked in the tx queue.
I found the peer interface with a server NID that have 42 msg blocked on tx:
peer buffer credit: ~ -17000
On the peer router queue (lpni_rtrq), messages seems to be linked to different kib_conn (kib_rx.rx_conn) every 42 messages.
These connections are in disconnected state, with ibc_list and ibc_sched_list not linked (poison value inside). But qp and cq are not freed.
A QP take 512 pages and a CQ take 256 pages, ~ 3 MB per connection.
So it seems to be a connections leak.
Analyze
Here what I understood with the lnet/ko2iblnd sources:
1. Compute node have an issue and do not answer (or partially) to the router.
2. Messages from the servers to the compute node are queued and the tx peer credits is negative.
3. When a server peer interface have more than 42 messages blocked on tx, peer_buffer_credits is negative (by default, peer_credits == peer_buffer_credits). In that case, new message from server are queued in lpni_rtrq.
4. After that, the server is not able to send any messages to the router because peer_buffer_credits < 0. All messages from the server sent to the router timeout (RDMA timeout).
5. The server disconnects/reconnects to the routers and cleans its tx credits and resend its messages.
6. On the router, the old connection is set to disconnect but not freed because old Rx messages are not cleaned and still reference the old connection.
Can someone help me with this ?
I am not used to debug LNet/ko2iblnd.
Etienne Aujames
added a comment - Here some context for logs:
o2ibxx: storage network
o2ibyy: compute network
for vmcore-dmesg_hide_router272a_20221220_213634_1.txt : BB.BB.ID8@o2ibyy is the faulty client node
for vmcore-dmesg_hide_router272a_20221209_152308_1.txt : BB.BB.ID17@o2ibyy is the faulty client node
Those are logs from 2 crashes of the router in production.
The situation was stabilized by changing the CPU of the faulty client node.
Etienne Aujames
added a comment - Hi Cyril,
Sorry for the delay.
I have submitted 2 dmesg logs:
vmcore-dmesg_hide_router272a_20221209_152308_1.txt
vmcore-dmesg_hide_router272a_20221220_213634_1.txt
Those are logs from 2 crashes of the router in production.
The situation was stabilized by changing the CPU of the faulty client node.
I can't get you debug_log (maybe some dmesg if you want).
The setup is not available because it was reproduced on a router from the cluster (to reproduce this it needs a node with 2 ib interfaces on different networks).
I tried to reproduce this with tcp <-> ib but unsuccessfully.
Etienne Aujames
added a comment - Hi Cyril,
I can't get you debug_log (maybe some dmesg if you want).
The setup is not available because it was reproduced on a router from the cluster (to reproduce this it needs a node with 2 ib interfaces on different networks).
I tried to reproduce this with tcp <-> ib but unsuccessfully.
I did take a look but then was on something else… Sorry about that. I plan to work on it again very soon.
Thank you.
Cyril Bordage
added a comment - Hello Etienne,
I did take a look but then was on something else… Sorry about that. I plan to work on it again very soon.
Thank you.
tx credits value is minimum for client1 peer_ni (available_tx_credits = - peer_buffer_credit_param * server_nodes = - 42 * 5 = -210)
The available_rtr_credits (peer_buffer_credit) keep decreasing on all the peer_ni of servers
The number of QP keep increasing
The client2 is not able to communicate with servers. All the Rx peers_ni of the servers are saturated on the router (peer_buffer_credit < 0).
The servers keep trying to reconnect to the clients.
Remarks
A drop rule or an Infiniband device reset on the client1 do not produce the issue: communication errors are detected by the router and the peer_ni on the router is set to down (see auto_down feature), messages are dropped.
I try to increase peer_buffer_credit and set lnet_health_sensitivity, this does not change the behavior.
Etienne Aujames
added a comment - - edited Hi,
We successfully reproduce the issue on a test filesystem with Infiniband:
Configuration
1 MDT/MDS
5 OSS/OST
2 Clients
1 router IB <--> IB
Lustre 2.12.7 LTS on all nodes.
LNet configuration:
options lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0
options ko2iblnd peer_credits=42
servers use o2ib50
clients use o2ib51
Reproducer
Mount clients
Do some IO with clients (I used multithread fio)
Spam client1 with lnet ping from all the servers
clush -w@servers "while true; do seq 1 100 | xargs -P100 -I{} lnetctl ping client1@o2ib51; done"
Add a delay (2s) rule on the client1 for the incoming traffic from *@o2ib50
ssh client1 lctl net_delay_add -s "*.o2ib50" -d "o2ib51" -l 2 --rate=1
That's it!
On the router:
tx credits value is minimum for client1 peer_ni (available_tx_credits = - peer_buffer_credit_param * server_nodes = - 42 * 5 = -210)
The available_rtr_credits (peer_buffer_credit) keep decreasing on all the peer_ni of servers
The number of QP keep increasing
The client2 is not able to communicate with servers. All the Rx peers_ni of the servers are saturated on the router (peer_buffer_credit < 0).
The servers keep trying to reconnect to the clients.
Remarks
A drop rule or an Infiniband device reset on the client1 do not produce the issue: communication errors are detected by the router and the peer_ni on the router is set to down (see auto_down feature), messages are dropped.
I try to increase peer_buffer_credit and set lnet_health_sensitivity, this does not change the behavior.
Peter Jones
added a comment - Cyril
Can you please advise?
Thanks
Peter
People
Cyril Bordage
Etienne Aujames
Votes:
0Vote for this issue
Watchers:
5Start watching this issue
Dates
Created:
Updated:
1 of 1
{"searchers":{"groups":[{"searchers":[{"name":"Project","id":"project","key":"issue.field.project","isShown":true,"lastViewed":1750060563285},{"name":"Summary","id":"summary","key":"issue.field.summary","isShown":true},{"name":"Type","id":"issuetype","key":"issue.field.issuetype","isShown":true,"lastViewed":1750060563286},{"name":"Status","id":"status","key":"issue.field.status","isShown":true,"lastViewed":1750060563294},{"name":"Priority","id":"priority","key":"issue.field.priority","isShown":true},{"name":"Resolution","id":"resolution","key":"issue.field.resolution","isShown":true},{"name":"Creator","id":"creator","key":"issue.field.creator","isShown":true},{"name":"Affects Version","id":"version","key":"issue.field.affectsversions","isShown":true},{"name":"Fix Version","id":"fixfor","key":"issue.field.fixversions","isShown":true},{"name":"Component","id":"component","key":"issue.field.components","isShown":false},{"name":"% Limits","id":"workratio","key":"issue.field.workratio","isShown":true},{"name":"Link types","id":"issue_link_type","key":"issue.field.issuelinks","isShown":true},{"name":"Environment","id":"environment","key":"issue.field.environment","isShown":true},{"name":"Description","id":"description","key":"issue.field.description","isShown":true},{"name":"Comment","id":"comment","key":"issue.field.comment","isShown":true},{"name":"Label","id":"labels","key":"issue.field.labels","isShown":true,"lastViewed":1750060563298},{"name":"Query","id":"text","key":"text","isShown":true},{"name":"Bugzilla ID","id":"customfield_10020","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Business Value","id":"customfield_10003","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Development","id":"customfield_10890","key":"com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary","isShown":true},{"name":"Epic","id":"customfield_10040","key":"com.atlassian.jira.plugin.system.customfieldtypes:labels","isShown":true},{"name":"Epic Colour","id":"customfield_10095","key":"com.pyxis.greenhopper.jira:gh-epic-color","isShown":false},{"name":"Epic Link","id":"customfield_10092","key":"com.pyxis.greenhopper.jira:gh-epic-link","isShown":true},{"name":"Epic Name","id":"customfield_10093","key":"com.pyxis.greenhopper.jira:gh-epic-label","isShown":true},{"name":"Epic Status","id":"customfield_10094","key":"com.pyxis.greenhopper.jira:gh-epic-status","isShown":false},{"name":"Epic/Theme","id":"customfield_10030","key":"com.atlassian.jira.plugin.system.customfieldtypes:labels","isShown":true},{"name":"Flagged","id":"customfield_10000","key":"com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes","isShown":true},{"name":"IEEL Options","id":"customfield_10191","key":"com.atlassian.jira.plugin.system.customfieldtypes:multiselect","isShown":true},{"name":"Original story points","id":"customfield_11094","key":"com.atlassian.jpo:jpo-custom-field-original-story-points","isShown":true},{"name":"Parent Link","id":"customfield_11091","key":"com.atlassian.jpo:jpo-custom-field-parent","isShown":false},{"name":"Project","id":"customfield_10070","key":"com.atlassian.jira.plugin.system.customfieldtypes:select","isShown":true},{"name":"Rank","id":"customfield_10390","key":"com.pyxis.greenhopper.jira:gh-lexo-rank","isShown":true},{"name":"Rank (Obsolete)","id":"customfield_10001","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Rank (Obsolete)","id":"customfield_10090","key":"com.pyxis.greenhopper.jira:gh-global-rank","isShown":true},{"name":"Release Version History","id":"customfield_10050","key":"com.pyxis.greenhopper.jira:greenhopper-releasedmultiversionhistory","isShown":true},{"name":"Severity","id":"customfield_10060","key":"com.atlassian.jira.plugin.system.customfieldtypes:select","isShown":true},{"name":"Site Affected:","id":"customfield_10190","key":"com.atlassian.jira.plugin.system.customfieldtypes:textfield","isShown":true},{"name":"Sprint","id":"customfield_10091","key":"com.pyxis.greenhopper.jira:gh-sprint","isShown":true},{"name":"Story Points","id":"customfield_10002","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":true},{"name":"Support Region","id":"customfield_10990","key":"com.atlassian.jira.plugin.system.customfieldtypes:select","isShown":true},{"name":"Team","id":"customfield_11090","key":"com.atlassian.teams:rm-teams-custom-field-team","isShown":true},{"name":"Upstreaming","id":"customfield_10290","key":"com.atlassian.jira.plugin.system.customfieldtypes:select","isShown":true},{"name":"Whiteboard","id":"customfield_10591","key":"com.atlassian.jira.plugin.system.customfieldtypes:textfield","isShown":true},{"name":"issueFunction","id":"customfield_10590","key":"com.onresolve.jira.groovy.groovyrunner:jqlFunctionsCustomFieldType","isShown":true}],"type":"DETAILS","title":"Details"},{"searchers":[{"name":"Created Date","id":"created","key":"issue.field.created","isShown":true},{"name":"Updated Date","id":"updated","key":"issue.field.updated","isShown":true},{"name":"Resolution Date","id":"resolutiondate","key":"issue.field.resolution.date","isShown":true},{"name":"Due Date","id":"duedate","key":"issue.field.duedate","isShown":true},{"name":"Baseline end date","id":"customfield_10494","key":"com.atlassian.jira.plugin.system.customfieldtypes:datepicker","isShown":true},{"name":"Baseline start date","id":"customfield_10492","key":"com.atlassian.jira.plugin.system.customfieldtypes:datepicker","isShown":true},{"name":"Baseline start date","id":"customfield_10790","key":"com.atlassian.jira.plugin.system.customfieldtypes:datepicker","isShown":true},{"name":"End date","id":"customfield_10490","key":"com.atlassian.jira.plugin.system.customfieldtypes:datepicker","isShown":true},{"name":"Start date","id":"customfield_10493","key":"com.atlassian.jira.plugin.system.customfieldtypes:datepicker","isShown":true},{"name":"Target end","id":"customfield_11093","key":"com.atlassian.jpo:jpo-custom-field-baseline-end","isShown":true},{"name":"Target start","id":"customfield_11092","key":"com.atlassian.jpo:jpo-custom-field-baseline-start","isShown":true}],"type":"DATES","title":"Dates"},{"searchers":[{"name":"Assignee","id":"assignee","key":"issue.field.assignee","isShown":true,"lastViewed":1750060563297},{"name":"Reporter","id":"reporter","key":"issue.field.reporter","isShown":true}],"type":"PEOPLE","title":"People"}]},"values":{"issuetype":{"name":"Type","editHtml":"\n\n\n\n <div class=\"field-group aui-field-issuetype\" >\n <label for=\"searcher-type\">Type</label> <select class=\"select js-default-checkboxmultiselect\"\n id=\"searcher-type\"\n multiple=\"multiple\"\n name=\"type\"\n data-max-inline-results-displayed=\"100\"\n data-placeholder-text=\"Find Issue Types...\">\n <optgroup>\n \n <option class=\" \"\n id=\"type_-2\"\n title=\"All Standard Issue Types\"\n value=\"-2\">All Standard Issue Types</option>\n \n <option class=\" \"\n id=\"type_-3\"\n title=\"All Sub-Task Issue Types\"\n value=\"-3\">All Sub-Task Issue Types</option>\n </optgroup>\n\n <optgroup label=\"Standard Issue Types\">\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11303&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_1\"\n title=\"Bug\"\n value=\"1\">Bug</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/images/icons/issuetypes/epic.png\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_5\"\n title=\"Epic\"\n value=\"5\">Epic</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11310&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_4\"\n title=\"Improvement\"\n value=\"4\">Improvement</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11311&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_2\"\n title=\"New Feature\"\n value=\"2\">New Feature</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/images/icons/issuetypes/undefined.png\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_9\"\n title=\"Question/Request\"\n value=\"9\">Question/Request</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11300&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_10200\"\n title=\"Requirement\"\n value=\"10200\">Requirement</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11315&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_6\"\n title=\"Story\"\n value=\"6\">Story</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11318&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_3\"\n title=\"Task\"\n value=\"3\">Task</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/download/resources/com.thed.zephyr.je/images/icons/ico_zephyr_issuetype.png\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_10100\"\n title=\"Test\"\n value=\"10100\">Test</option>\n </optgroup>\n\n <optgroup label=\"Sub-Task Issue Types\">\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11316&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_10300\"\n title=\"Requirement task\"\n value=\"10300\">Requirement task</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11300&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_8\"\n title=\"Review task\"\n value=\"8\">Review task</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/images/icons/issuetypes/task_agile.png\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_7\"\n title=\"Technical task\"\n value=\"7\">Technical task</option>\n </optgroup>\n </select>\n </div>\n ","validSearcher":true,"isShown":true},"project":{"name":"Project","editHtml":" \n <div class=\"field-group aui-field-project\" >\n <label for=\"searcher-pid\">Project</label> <select class=\"js-project-checkboxmultiselect\"\n data-placeholder-text=\"Find Projects...\"\n id=\"searcher-pid\"\n multiple=\"multiple\"\n name=\"pid\">\n <optgroup label=\"Recent Projects\">\n </optgroup>\n <optgroup label=\"All Projects\" >\n \n <option data-icon=\"/secure/projectavatar?pid=11910&size=small\"\n title=\"Lemur\"\n value=\"11910\">\n Lemur (LMR)\n </option>\n <option data-icon=\"/secure/projectavatar?pid=10000&size=small\"\n title=\"Lustre\"\n value=\"10000\">\n Lustre (LU)\n </option>\n <option data-icon=\"/secure/projectavatar?pid=10070&size=small\"\n title=\"Lustre Documentation\"\n value=\"10070\">\n Lustre Documentation (LUDOC)\n </option>\n </optgroup>\n </select>\n </div>\n \n\n","validSearcher":true,"isShown":true},"assignee":{"name":"Assignee","editHtml":"\n \n <div class=\"field-group aui-field-userlist\" >\n <label for=\"searcher-assigneeSelect\">Assignee</label> <fieldset rel=\"assignee\" class=\"hidden user-group-searcher-params\">\n </fieldset>\n <select class=\"js-usergroup-checkboxmultiselect\" multiple=\"multiple\" id=\"assignee\" name=\"assignee\" data-placeholder-text=\"Enter username or group\">\n <optgroup>\n <option class=\"headerOption\" data-icon=\"https://jira.whamcloud.com/secure/useravatar?size=xsmall&avatarId=10123\" value=\"empty\" title=\"Unassigned\">Unassigned</option>\n </optgroup>\n <optgroup>\n </optgroup>\n </select>\n <input type=\"hidden\" name=\"check_prev_assignee\" value=\"true\">\n </div>\n \n","validSearcher":true,"isShown":true},"status":{"name":"Status","editHtml":"\n <div class=\"field-group aui-field-constants\" >\n <label for=\"searcher-status\">Status</label> <select class=\"select js-default-checkboxmultiselectstatuslozenge\"\n data-placeholder-text=\"Find Statuses...\"\n id=\"searcher-status\"\n multiple=\"multiple\"\n name=\"status\"\n data-max-inline-results-displayed=\"100\"\n data-footer-text=\"-78 more options. Continue typing to refine further.\" data-status-lozenge=\"true\">\n <optgroup >\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/open.png\" value=\"1\" title=\"Open\" data-simple-status=\"{"id":"1","name":"Open","description":"The issue is open and ready for the assignee to start work on it.","iconUrl":"/images/icons/statuses/open.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Open</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/inprogress.png\" value=\"3\" title=\"In Progress\" data-simple-status=\"{"id":"3","name":"In Progress","description":"This issue is being actively worked on at the moment by the assignee.","iconUrl":"/images/icons/statuses/inprogress.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">In Progress</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/reopened.png\" value=\"4\" title=\"Reopened\" data-simple-status=\"{"id":"4","name":"Reopened","description":"This issue was once resolved, but the resolution was deemed incorrect. From here issues are either marked assigned or resolved.","iconUrl":"/images/icons/statuses/reopened.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Reopened</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/resolved.png\" value=\"5\" title=\"Resolved\" data-simple-status=\"{"id":"5","name":"Resolved","description":"A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.","iconUrl":"/images/icons/statuses/resolved.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Resolved</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/closed.png\" value=\"6\" title=\"Closed\" data-simple-status=\"{"id":"6","name":"Closed","description":"The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.","iconUrl":"/images/icons/statuses/closed.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Closed</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10000\" title=\"Accepted\" data-simple-status=\"{"id":"10000","name":"Accepted","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Accepted</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10001\" title=\"In Backlog\" data-simple-status=\"{"id":"10001","name":"In Backlog","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">In Backlog</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10002\" title=\"Blocked External\" data-simple-status=\"{"id":"10002","name":"Blocked External","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Blocked External</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10003\" title=\"Blocked Internal\" data-simple-status=\"{"id":"10003","name":"Blocked Internal","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Blocked Internal</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10004\" title=\"Pending Review\" data-simple-status=\"{"id":"10004","name":"Pending Review","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Pending Review</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10105\" title=\"Waiting On Gatekeeper\" data-simple-status=\"{"id":"10105","name":"Waiting On Gatekeeper","description":"Ticket is waiting on the gate keeper for code to land.","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Waiting On Gatekeeper</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/closed.png\" value=\"10205\" title=\"Done\" data-simple-status=\"{"id":"10205","name":"Done","description":"","iconUrl":"/images/icons/statuses/closed.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Done</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/open.png\" value=\"10206\" title=\"To Do\" data-simple-status=\"{"id":"10206","name":"To Do","description":"","iconUrl":"/images/icons/statuses/open.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">To Do</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/information.png\" value=\"10305\" title=\"In Review\" data-simple-status=\"{"id":"10305","name":"In Review","description":"","iconUrl":"/images/icons/statuses/information.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">In Review</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10505\" title=\"Waiting\" data-simple-status=\"{"id":"10505","name":"Waiting","description":"Waiting for a response or something else that is required to complete the issue.","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Waiting</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10605\" title=\"Committed\" data-simple-status=\"{"id":"10605","name":"Committed","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Committed</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10606\" title=\"Rejected\" data-simple-status=\"{"id":"10606","name":"Rejected","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Rejected</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10607\" title=\"Completed\" data-simple-status=\"{"id":"10607","name":"Completed","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Completed</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10608\" title=\"New\" data-simple-status=\"{"id":"10608","name":"New","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">New</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10705\" title=\"Awaiting Verification\" data-simple-status=\"{"id":"10705","name":"Awaiting Verification","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Awaiting Verification</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10706\" title=\"Fix Verified\" data-simple-status=\"{"id":"10706","name":"Fix Verified","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Fix Verified</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10806\" title=\"Need Information\" data-simple-status=\"{"id":"10806","name":"Need Information","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Need Information</option>\n </optgroup>\n</select>\n </div>\n \n","validSearcher":true,"isShown":true},"labels":{"name":"Label","viewHtml":" <div class=\"searcherValue\">\n \n <label class=\"fieldLabel\" for=\"fieldlabels\">Label:</label><span id=\"fieldlabels\" class=\"fieldValue\">\n \n router\n</span></div>\n","editHtml":"\n <div class=\"field-group aui-field-labels\" >\n <label for=\"searcher-labels\">Labels</label> <select class=\"js-label-checkboxmultiselect\" multiple=\"multiple\" id=\"searcher-labels\" name=\"labels\" data-placeholder-text=\"Find Labels...\">\n <option value=\"router\" title=\"router\" selected=\"selected\">router</option>\n </select>\n </div>\n \n","jql":"labels = router","validSearcher":true,"isShown":true}}}
[{"id":-1,"name":"My open issues","jql":"assignee = currentUser() AND resolution = Unresolved order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-2,"name":"Reported by me","jql":"reporter = currentUser() order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-4,"name":"All issues","jql":"order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-5,"name":"Open issues","jql":"resolution = Unresolved order by priority DESC,updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-9,"name":"Done issues","jql":"statusCategory = Done order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-3,"name":"Viewed recently","jql":"issuekey in issueHistory() order by lastViewed DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-6,"name":"Created recently","jql":"created >= -1w order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-7,"name":"Resolved recently","jql":"resolutiondate >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-8,"name":"Updated recently","jql":"updated >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false}]
Major 
Here some context for logs: