Not easy to provide you with a vmcore, but here a 'foreach bt' log. (This is a new occurence of the problem, do not compare this output with the exact output I posted above)
Aurelien Degremont (Inactive)
added a comment - foreach_bt.log
Not easy to provide you with a vmcore, but here a 'foreach bt' log. (This is a new occurence of the problem, do not compare this output with the exact output I posted above)
Have possible to share the vmcore with us? Or only paste output of 'foreach bt' in crash. Many thanks for your help.
Thanks,
YangSheng
Yang Sheng
added a comment - Hi, Aurelien,
Have possible to share the vmcore with us? Or only paste output of 'foreach bt' in crash. Many thanks for your help.
Thanks,
YangSheng
We are seeing an unresponsive MDS, all CPUs at 100% with lots of ldlm_cn_xxx threads and few ldlm_bl threads consumming all the CPUs.
ldlm_cn threads have this kind of strack trace:
Aug 26 13:38:31 ip-198-19-95-73 kernel: Lustre: 79258:0:(service.c:2114:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:136s); client may timeout. req@ffff88d6dfb47850 x1642929689998160/t0(0) o103->0e4989ad-ae70-2c70-9db7-05f5fafae50e@10.26.86.9@tcp:745/0 lens 3584/0 e 0 to 0 dl 1566826575 ref 1 fl Interpret:/2/ffffffff rc 0/-1
Aug 26 13:38:31 ip-198-19-95-73 kernel: Lustre: 79258:0:(service.c:2114:ptlrpc_server_handle_request()) Skipped 11126 previous similar messages
Aug 26 13:39:11 ip-198-19-95-73 kernel: LNet: Service thread pid 14956 was inactive for 204.11s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Aug 26 13:39:11 ip-198-19-95-73 kernel: Pid: 14956, comm: mdt01_094 4.14.123-111.109.amzn2.x86_64 #1 SMP Mon Jun 10 19:37:57 UTC 2019
Aug 26 13:39:11 ip-198-19-95-73 kernel: Call Trace:
Aug 26 13:39:11 ip-198-19-95-73 kernel: ldlm_completion_ast+0x63e/0x910 [ptlrpc]
Aug 26 13:39:11 ip-198-19-95-73 kernel: ldlm_cli_enqueue_local+0x27c/0x880 [ptlrpc]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_object_local_lock+0x759/0xa00 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_object_lock_internal+0x5b/0x2e0 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_reint_open+0xb4e/0x3020 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_reint_rec+0x7f/0x250 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_reint_internal+0x5ee/0x680 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_intent_reint+0x173/0x3e0 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_intent_opc+0x385/0xa00 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_intent_policy+0x1c5/0x370 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: ldlm_lock_enqueue+0x234/0x950 [ptlrpc]
Aug 26 13:39:11 ip-198-19-95-73 kernel: ldlm_handle_enqueue0+0xa0b/0x1750 [ptlrpc]
A couple of ldlm_bl_NN have this kind of stack, obtained from crash bt -T:
[ffffc9001eecfa58] try_to_wake_up at ffffffff810b3b74
[ffffc9001eecfac8] __update_load_avg_se at ffffffff810b9258
[ffffc9001eecfad8] __update_load_avg_se at ffffffff810b9258
[ffffc9001eecfb10] update_load_avg at ffffffff810bbacd
[ffffc9001eecfb30] account_entity_enqueue at ffffffff810bdb45
[ffffc9001eecfb68] enqueue_entity at ffffffff810bfa63
[ffffc9001eecfbf8] tgt_blocking_ast at ffffffffa0c78a28 [ptlrpc]
[ffffc9001eecfc40] ldlm_export_lock_hash at ffffffffa0bea6d0 [ptlrpc]
--------------
[ffffc9001eecfc48] cfs_hash_bd_from_key at ffffffffa084fad1 [libcfs]
[ffffc9001eecfc68] cfs_hash_rw_lock at ffffffffa084f370 [libcfs]
[ffffc9001eecfc70] cfs_hash_hh_hnode_del at ffffffffa084f3e0 [libcfs]
[ffffc9001eecfc78] cfs_hash_bd_del_locked at ffffffffa084f7df [libcfs]
[ffffc9001eecfc90] cfs_hash_rw_unlock at ffffffffa084ff20 [libcfs]
[ffffc9001eecfc98] cfs_hash_del at ffffffffa085305e [libcfs]
[ffffc9001eecfcd8] cfs_hash_bd_from_key at ffffffffa084fad1 [libcfs]
[ffffc9001eecfd18] cfs_hash_bd_from_key at ffffffffa084fad1 [libcfs]
[ffffc9001eecfd80] ldlm_lock_put at ffffffffa0bc3f73 [ptlrpc]
[ffffc9001eecfd90] cfs_hash_hh_hhead at ffffffffa084f390 [libcfs]
[ffffc9001eecfd98] cfs_hash_for_each_relax at ffffffffa0850458 [libcfs]
[ffffc9001eecfda0] ldlm_cancel_locks_for_export_cb at ffffffffa0bcbe50 [ptlrpc]
[ffffc9001eecfdb8] ldlm_export_lock_put at ffffffffa0bea740 [ptlrpc]
[ffffc9001eecfdf0] ldlm_cancel_locks_for_export_cb at ffffffffa0bcbe50 [ptlrpc]
[ffffc9001eecfe18] cfs_hash_for_each_empty at ffffffffa08538b8 [libcfs]
[ffffc9001eecfe58] ldlm_export_cancel_locks at ffffffffa0bcc10f [ptlrpc]
[ffffc9001eecfe80] ldlm_bl_thread_main at ffffffffa0bf5147 [ptlrpc]
[ffffc9001eecfec0] default_wake_function at ffffffff810b4040
[ffffc9001eecff08] ldlm_bl_thread_main at ffffffffa0bf4c60 [ptlrpc]
[ffffc9001eecff10] kthread at ffffffff810a90fa
[ffffc9001eecff20] kthread at ffffffff810a8fe0
[ffffc9001eecff50] ret_from_fork at ffffffff81800205
MDT is receiving plenty of LDLM_CANCEL from clients.
As we have 747 GB of RAM, LDLM computes a LDLM pool limit to 36 millions locks.
When number of active locks get over this number the MDT can't cope with that and the problem starts appearing
If we control the ldlm tuning on client side to limit the number of locks, we don't see the problem.
Based on the MDT HW we are using, the threshold is around 10 millions locks.
For anybody hitting a similar issue I would recommend checking the number of locks on server side and limit it on client side
ldlm.namespace..lru_max_age and ldlm.namespace..lru_size
Is this a scalibility issue of the LDLM code? Too many active locks?
Or do you think the problem is elsewhere?
Aurelien Degremont (Inactive)
added a comment - We were able to reproduce a very similar issue.
We are seeing an unresponsive MDS, all CPUs at 100% with lots of ldlm_cn_xxx threads and few ldlm_bl threads consumming all the CPUs.
ldlm_cn threads have this kind of strack trace:
Aug 26 13:38:31 ip-198-19-95-73 kernel: Lustre: 79258:0:(service.c:2114:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (6:136s); client may timeout. req@ffff88d6dfb47850 x1642929689998160/t0(0) o103->0e4989ad-ae70-2c70-9db7-05f5fafae50e@10.26.86.9@tcp:745/0 lens 3584/0 e 0 to 0 dl 1566826575 ref 1 fl Interpret:/2/ffffffff rc 0/-1
Aug 26 13:38:31 ip-198-19-95-73 kernel: Lustre: 79258:0:(service.c:2114:ptlrpc_server_handle_request()) Skipped 11126 previous similar messages
Aug 26 13:39:11 ip-198-19-95-73 kernel: LNet: Service thread pid 14956 was inactive for 204.11s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Aug 26 13:39:11 ip-198-19-95-73 kernel: Pid: 14956, comm: mdt01_094 4.14.123-111.109.amzn2.x86_64 #1 SMP Mon Jun 10 19:37:57 UTC 2019
Aug 26 13:39:11 ip-198-19-95-73 kernel: Call Trace:
Aug 26 13:39:11 ip-198-19-95-73 kernel: ldlm_completion_ast+0x63e/0x910 [ptlrpc]
Aug 26 13:39:11 ip-198-19-95-73 kernel: ldlm_cli_enqueue_local+0x27c/0x880 [ptlrpc]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_object_local_lock+0x759/0xa00 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_object_lock_internal+0x5b/0x2e0 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_reint_open+0xb4e/0x3020 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_reint_rec+0x7f/0x250 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_reint_internal+0x5ee/0x680 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_intent_reint+0x173/0x3e0 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_intent_opc+0x385/0xa00 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: mdt_intent_policy+0x1c5/0x370 [mdt]
Aug 26 13:39:11 ip-198-19-95-73 kernel: ldlm_lock_enqueue+0x234/0x950 [ptlrpc]
Aug 26 13:39:11 ip-198-19-95-73 kernel: ldlm_handle_enqueue0+0xa0b/0x1750 [ptlrpc]
A couple of ldlm_bl_NN have this kind of stack, obtained from crash bt -T:
[ffffc9001eecfa58] try_to_wake_up at ffffffff810b3b74
[ffffc9001eecfac8] __update_load_avg_se at ffffffff810b9258
[ffffc9001eecfad8] __update_load_avg_se at ffffffff810b9258
[ffffc9001eecfb10] update_load_avg at ffffffff810bbacd
[ffffc9001eecfb30] account_entity_enqueue at ffffffff810bdb45
[ffffc9001eecfb68] enqueue_entity at ffffffff810bfa63
[ffffc9001eecfbf8] tgt_blocking_ast at ffffffffa0c78a28 [ptlrpc]
[ffffc9001eecfc40] ldlm_export_lock_hash at ffffffffa0bea6d0 [ptlrpc]
--------------
[ffffc9001eecfc48] cfs_hash_bd_from_key at ffffffffa084fad1 [libcfs]
[ffffc9001eecfc68] cfs_hash_rw_lock at ffffffffa084f370 [libcfs]
[ffffc9001eecfc70] cfs_hash_hh_hnode_del at ffffffffa084f3e0 [libcfs]
[ffffc9001eecfc78] cfs_hash_bd_del_locked at ffffffffa084f7df [libcfs]
[ffffc9001eecfc90] cfs_hash_rw_unlock at ffffffffa084ff20 [libcfs]
[ffffc9001eecfc98] cfs_hash_del at ffffffffa085305e [libcfs]
[ffffc9001eecfcd8] cfs_hash_bd_from_key at ffffffffa084fad1 [libcfs]
[ffffc9001eecfd18] cfs_hash_bd_from_key at ffffffffa084fad1 [libcfs]
[ffffc9001eecfd80] ldlm_lock_put at ffffffffa0bc3f73 [ptlrpc]
[ffffc9001eecfd90] cfs_hash_hh_hhead at ffffffffa084f390 [libcfs]
[ffffc9001eecfd98] cfs_hash_for_each_relax at ffffffffa0850458 [libcfs]
[ffffc9001eecfda0] ldlm_cancel_locks_for_export_cb at ffffffffa0bcbe50 [ptlrpc]
[ffffc9001eecfdb8] ldlm_export_lock_put at ffffffffa0bea740 [ptlrpc]
[ffffc9001eecfdf0] ldlm_cancel_locks_for_export_cb at ffffffffa0bcbe50 [ptlrpc]
[ffffc9001eecfe18] cfs_hash_for_each_empty at ffffffffa08538b8 [libcfs]
[ffffc9001eecfe58] ldlm_export_cancel_locks at ffffffffa0bcc10f [ptlrpc]
[ffffc9001eecfe80] ldlm_bl_thread_main at ffffffffa0bf5147 [ptlrpc]
[ffffc9001eecfec0] default_wake_function at ffffffff810b4040
[ffffc9001eecff08] ldlm_bl_thread_main at ffffffffa0bf4c60 [ptlrpc]
[ffffc9001eecff10] kthread at ffffffff810a90fa
[ffffc9001eecff20] kthread at ffffffff810a8fe0
[ffffc9001eecff50] ret_from_fork at ffffffff81800205
perf top :
Samples: 10M of event 'cpu-clock', Event count (approx.): 118365912201
Overhead Shared Object Symbol
45.75% [libcfs] [k] cfs_hash_hh_hhead
21.23% [libcfs] [k] cfs_hash_for_each_relax
6.93% [kernel] [k] __x86_indirect_thunk_rax
3.85% [obdclass] [k] class_handle2object
2.72% [kernel] [k] _raw_read_lock
2.46% [libcfs] [k] cfs_hash_rw_unlock
This issue appears when there are lots of locks in the MDT namespace.
ldlm.namespaces.mdt-lustre-MDT0000_UUID.contended_locks=32
ldlm.namespaces.mdt-lustre-MDT0000_UUID.contention_seconds=2
ldlm.namespaces.mdt-lustre-MDT0000_UUID.ctime_age_limit=10
ldlm.namespaces.mdt-lustre-MDT0000_UUID.early_lock_cancel=0
ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=46344331
ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_timeouts=55
ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_unused_count=0
ldlm.namespaces.mdt-lustre-MDT0000_UUID.lru_max_age=3900000
ldlm.namespaces.mdt-lustre-MDT0000_UUID.lru_size=9600
ldlm.namespaces.mdt-lustre-MDT0000_UUID.max_nolock_bytes=0
ldlm.namespaces.mdt-lustre-MDT0000_UUID.max_parallel_ast=1024
ldlm.namespaces.mdt-lustre-MDT0000_UUID.resource_count=14281836
ldlm.namespaces.mdt-lustre-MDT0000_UUID.pool.stats=
snapshot_time 1568643075.466635065 secs.nsecs
granted 59817 samples [locks] 0 55589762 171302800233 8049595240895311875
grant 153081925 samples [locks] 1 1 153081925 153081925
cancel 106817757 samples [locks] 1 1 106817757 106817757
grant_rate 59817 samples [locks/s] 0 527737 153081896 36151088814526
cancel_rate 59817 samples [locks/s] 0 315229 106817738 15370244224014
grant_plan 59817 samples [locks/s] 382861 47857748 189705106051 7747978793807775269
slv 59817 samples [slv] 1 1378303200000 77899468648378585 6008560669154165631
recalc_freed 259959295 samples [locks] 0 0 0 0
recalc_timing 59791 samples [sec] 1 52689 140658 3151201290
MDT is receiving plenty of LDLM_CANCEL from clients.
As we have 747 GB of RAM, LDLM computes a LDLM pool limit to 36 millions locks.
When number of active locks get over this number the MDT can't cope with that and the problem starts appearing
If we control the ldlm tuning on client side to limit the number of locks, we don't see the problem.
Based on the MDT HW we are using, the threshold is around 10 millions locks.
For anybody hitting a similar issue I would recommend checking the number of locks on server side and limit it on client side
ldlm.namespace. .lru_max_age and ldlm.namespace. .lru_size
Is this a scalibility issue of the LDLM code? Too many active locks?
Or do you think the problem is elsewhere?
I am seeing the same issue as well however on the oss's, the similarities I am seeing are the ldlm threads consuming all except 2 cores (libcfs cpu_pattern) on the oss's (DNE).
Ian Costello
added a comment - I am seeing the same issue as well however on the oss's, the similarities I am seeing are the ldlm threads consuming all except 2 cores (libcfs cpu_pattern) on the oss's (DNE).
People
Yang Sheng
anhua (Inactive)
Votes:
0Vote for this issue
Watchers:
6Start watching this issue
Dates
Created:
Updated:
Resolved:
1 of 2
{"searchers":{"groups":[{"searchers":[{"name":"Project","id":"project","key":"issue.field.project","isShown":true,"lastViewed":1753204869695},{"name":"Summary","id":"summary","key":"issue.field.summary","isShown":true},{"name":"Type","id":"issuetype","key":"issue.field.issuetype","isShown":true,"lastViewed":1753204869696},{"name":"Status","id":"status","key":"issue.field.status","isShown":true,"lastViewed":1753204869708},{"name":"Priority","id":"priority","key":"issue.field.priority","isShown":true},{"name":"Resolution","id":"resolution","key":"issue.field.resolution","isShown":true},{"name":"Creator","id":"creator","key":"issue.field.creator","isShown":true},{"name":"Affects Version","id":"version","key":"issue.field.affectsversions","isShown":true},{"name":"Fix Version","id":"fixfor","key":"issue.field.fixversions","isShown":true},{"name":"Component","id":"component","key":"issue.field.components","isShown":false},{"name":"% Limits","id":"workratio","key":"issue.field.workratio","isShown":true},{"name":"Link types","id":"issue_link_type","key":"issue.field.issuelinks","isShown":true},{"name":"Environment","id":"environment","key":"issue.field.environment","isShown":true},{"name":"Description","id":"description","key":"issue.field.description","isShown":true},{"name":"Comment","id":"comment","key":"issue.field.comment","isShown":true},{"name":"Label","id":"labels","key":"issue.field.labels","isShown":true},{"name":"Query","id":"text","key":"text","isShown":true},{"name":"Bugzilla ID","id":"customfield_10020","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Business Value","id":"customfield_10003","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Development","id":"customfield_10890","key":"com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary","isShown":true},{"name":"Epic","id":"customfield_10040","key":"com.atlassian.jira.plugin.system.customfieldtypes:labels","isShown":true},{"name":"Epic Colour","id":"customfield_10095","key":"com.pyxis.greenhopper.jira:gh-epic-color","isShown":false},{"name":"Epic Link","id":"customfield_10092","key":"com.pyxis.greenhopper.jira:gh-epic-link","isShown":true},{"name":"Epic Name","id":"customfield_10093","key":"com.pyxis.greenhopper.jira:gh-epic-label","isShown":true},{"name":"Epic Status","id":"customfield_10094","key":"com.pyxis.greenhopper.jira:gh-epic-status","isShown":false},{"name":"Epic/Theme","id":"customfield_10030","key":"com.atlassian.jira.plugin.system.customfieldtypes:labels","isShown":true,"lastViewed":1753204869712},{"name":"Flagged","id":"customfield_10000","key":"com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes","isShown":true},{"name":"IEEL Options","id":"customfield_10191","key":"com.atlassian.jira.plugin.system.customfieldtypes:multiselect","isShown":true},{"name":"Original story points","id":"customfield_11094","key":"com.atlassian.jpo:jpo-custom-field-original-story-points","isShown":true},{"name":"Parent Link","id":"customfield_11091","key":"com.atlassian.jpo:jpo-custom-field-parent","isShown":false},{"name":"Project","id":"customfield_10070","key":"com.atlassian.jira.plugin.system.customfieldtypes:select","isShown":true},{"name":"Rank","id":"customfield_10390","key":"com.pyxis.greenhopper.jira:gh-lexo-rank","isShown":true},{"name":"Rank (Obsolete)","id":"customfield_10001","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":false},{"name":"Rank (Obsolete)","id":"customfield_10090","key":"com.pyxis.greenhopper.jira:gh-global-rank","isShown":true},{"name":"Release Version History","id":"customfield_10050","key":"com.pyxis.greenhopper.jira:greenhopper-releasedmultiversionhistory","isShown":true},{"name":"Severity","id":"customfield_10060","key":"com.atlassian.jira.plugin.system.customfieldtypes:select","isShown":true},{"name":"Site Affected:","id":"customfield_10190","key":"com.atlassian.jira.plugin.system.customfieldtypes:textfield","isShown":true},{"name":"Sprint","id":"customfield_10091","key":"com.pyxis.greenhopper.jira:gh-sprint","isShown":true},{"name":"Story Points","id":"customfield_10002","key":"com.atlassian.jira.plugin.system.customfieldtypes:float","isShown":true},{"name":"Support Region","id":"customfield_10990","key":"com.atlassian.jira.plugin.system.customfieldtypes:select","isShown":true},{"name":"Team","id":"customfield_11090","key":"com.atlassian.teams:rm-teams-custom-field-team","isShown":true},{"name":"Upstreaming","id":"customfield_10290","key":"com.atlassian.jira.plugin.system.customfieldtypes:select","isShown":true},{"name":"Whiteboard","id":"customfield_10591","key":"com.atlassian.jira.plugin.system.customfieldtypes:textfield","isShown":true},{"name":"issueFunction","id":"customfield_10590","key":"com.onresolve.jira.groovy.groovyrunner:jqlFunctionsCustomFieldType","isShown":true}],"type":"DETAILS","title":"Details"},{"searchers":[{"name":"Created Date","id":"created","key":"issue.field.created","isShown":true},{"name":"Updated Date","id":"updated","key":"issue.field.updated","isShown":true},{"name":"Resolution Date","id":"resolutiondate","key":"issue.field.resolution.date","isShown":true},{"name":"Due Date","id":"duedate","key":"issue.field.duedate","isShown":true},{"name":"Baseline end date","id":"customfield_10494","key":"com.atlassian.jira.plugin.system.customfieldtypes:datepicker","isShown":true},{"name":"Baseline start date","id":"customfield_10492","key":"com.atlassian.jira.plugin.system.customfieldtypes:datepicker","isShown":true},{"name":"Baseline start date","id":"customfield_10790","key":"com.atlassian.jira.plugin.system.customfieldtypes:datepicker","isShown":true},{"name":"End date","id":"customfield_10490","key":"com.atlassian.jira.plugin.system.customfieldtypes:datepicker","isShown":true},{"name":"Start date","id":"customfield_10493","key":"com.atlassian.jira.plugin.system.customfieldtypes:datepicker","isShown":true},{"name":"Target end","id":"customfield_11093","key":"com.atlassian.jpo:jpo-custom-field-baseline-end","isShown":true},{"name":"Target start","id":"customfield_11092","key":"com.atlassian.jpo:jpo-custom-field-baseline-start","isShown":true}],"type":"DATES","title":"Dates"},{"searchers":[{"name":"Assignee","id":"assignee","key":"issue.field.assignee","isShown":true,"lastViewed":1753204869711},{"name":"Reporter","id":"reporter","key":"issue.field.reporter","isShown":true}],"type":"PEOPLE","title":"People"}]},"values":{"issuetype":{"name":"Type","editHtml":"\n\n\n\n <div class=\"field-group aui-field-issuetype\" >\n <label for=\"searcher-type\">Type</label> <select class=\"select js-default-checkboxmultiselect\"\n id=\"searcher-type\"\n multiple=\"multiple\"\n name=\"type\"\n data-max-inline-results-displayed=\"100\"\n data-placeholder-text=\"Find Issue Types...\">\n <optgroup>\n \n <option class=\" \"\n id=\"type_-2\"\n title=\"All Standard Issue Types\"\n value=\"-2\">All Standard Issue Types</option>\n \n <option class=\" \"\n id=\"type_-3\"\n title=\"All Sub-Task Issue Types\"\n value=\"-3\">All Sub-Task Issue Types</option>\n </optgroup>\n\n <optgroup label=\"Standard Issue Types\">\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11303&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_1\"\n title=\"Bug\"\n value=\"1\">Bug</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/images/icons/issuetypes/epic.png\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_5\"\n title=\"Epic\"\n value=\"5\">Epic</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11310&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_4\"\n title=\"Improvement\"\n value=\"4\">Improvement</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11311&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_2\"\n title=\"New Feature\"\n value=\"2\">New Feature</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/images/icons/issuetypes/undefined.png\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_9\"\n title=\"Question/Request\"\n value=\"9\">Question/Request</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11300&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_10200\"\n title=\"Requirement\"\n value=\"10200\">Requirement</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11315&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_6\"\n title=\"Story\"\n value=\"6\">Story</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11318&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_3\"\n title=\"Task\"\n value=\"3\">Task</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/download/resources/com.thed.zephyr.je/images/icons/ico_zephyr_issuetype.png\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_10100\"\n title=\"Test\"\n value=\"10100\">Test</option>\n </optgroup>\n\n <optgroup label=\"Sub-Task Issue Types\">\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11316&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_10300\"\n title=\"Requirement task\"\n value=\"10300\">Requirement task</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/secure/viewavatar?size=xsmall&avatarId=11300&avatarType=issuetype\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_8\"\n title=\"Review task\"\n value=\"8\">Review task</option>\n \n <option class=\" imagebacked 10000 \"\n data-icon=\"/images/icons/issuetypes/task_agile.png\"\n data-fallback-icon=\"/images/icons/issuetypes/blank.png\"\n id=\"type_7\"\n title=\"Technical task\"\n value=\"7\">Technical task</option>\n </optgroup>\n </select>\n </div>\n ","validSearcher":true,"isShown":true},"customfield_10030":{"name":"Epic/Theme","viewHtml":"\n\n\n <div class=\"searcherValue\">\n \n <label class=\"fieldLabel\" for=\"fieldcustomfield_10030\">Epic/Theme:</label><span id=\"fieldcustomfield_10030\" class=\"fieldValue \">\n \nldlm\n</span></div>\n","editHtml":"\n <div class=\"field-group aui-field-text\" >\n <label for=\"searcher-customfield_10030\">Epic/Theme</label> <input class=\"text\" id=\"searcher-customfield_10030\" name=\"customfield_10030\" type=\"text\" value=\"ldlm\" />\n <div class=\"description\" id=\"customfield_10030-description\"><p>Field that will help you regroup issues under an Epic or under a theme.</p></div>\n </div>\n ","jql":"\"Epic/Theme\" = ldlm","validSearcher":true,"isShown":true},"project":{"name":"Project","editHtml":" \n <div class=\"field-group aui-field-project\" >\n <label for=\"searcher-pid\">Project</label> <select class=\"js-project-checkboxmultiselect\"\n data-placeholder-text=\"Find Projects...\"\n id=\"searcher-pid\"\n multiple=\"multiple\"\n name=\"pid\">\n <optgroup label=\"Recent Projects\">\n </optgroup>\n <optgroup label=\"All Projects\" >\n \n <option data-icon=\"/secure/projectavatar?pid=11910&size=small\"\n title=\"Lemur\"\n value=\"11910\">\n Lemur (LMR)\n </option>\n <option data-icon=\"/secure/projectavatar?pid=10000&size=small\"\n title=\"Lustre\"\n value=\"10000\">\n Lustre (LU)\n </option>\n <option data-icon=\"/secure/projectavatar?pid=10070&size=small\"\n title=\"Lustre Documentation\"\n value=\"10070\">\n Lustre Documentation (LUDOC)\n </option>\n </optgroup>\n </select>\n </div>\n \n\n","validSearcher":true,"isShown":true},"assignee":{"name":"Assignee","editHtml":"\n \n <div class=\"field-group aui-field-userlist\" >\n <label for=\"searcher-assigneeSelect\">Assignee</label> <fieldset rel=\"assignee\" class=\"hidden user-group-searcher-params\">\n </fieldset>\n <select class=\"js-usergroup-checkboxmultiselect\" multiple=\"multiple\" id=\"assignee\" name=\"assignee\" data-placeholder-text=\"Enter username or group\">\n <optgroup>\n <option class=\"headerOption\" data-icon=\"https://jira.whamcloud.com/secure/useravatar?size=xsmall&avatarId=10123\" value=\"empty\" title=\"Unassigned\">Unassigned</option>\n </optgroup>\n <optgroup>\n </optgroup>\n </select>\n <input type=\"hidden\" name=\"check_prev_assignee\" value=\"true\">\n </div>\n \n","validSearcher":true,"isShown":true},"status":{"name":"Status","editHtml":"\n <div class=\"field-group aui-field-constants\" >\n <label for=\"searcher-status\">Status</label> <select class=\"select js-default-checkboxmultiselectstatuslozenge\"\n data-placeholder-text=\"Find Statuses...\"\n id=\"searcher-status\"\n multiple=\"multiple\"\n name=\"status\"\n data-max-inline-results-displayed=\"100\"\n data-footer-text=\"-78 more options. Continue typing to refine further.\" data-status-lozenge=\"true\">\n <optgroup >\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/open.png\" value=\"1\" title=\"Open\" data-simple-status=\"{"id":"1","name":"Open","description":"The issue is open and ready for the assignee to start work on it.","iconUrl":"/images/icons/statuses/open.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Open</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/inprogress.png\" value=\"3\" title=\"In Progress\" data-simple-status=\"{"id":"3","name":"In Progress","description":"This issue is being actively worked on at the moment by the assignee.","iconUrl":"/images/icons/statuses/inprogress.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">In Progress</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/reopened.png\" value=\"4\" title=\"Reopened\" data-simple-status=\"{"id":"4","name":"Reopened","description":"This issue was once resolved, but the resolution was deemed incorrect. From here issues are either marked assigned or resolved.","iconUrl":"/images/icons/statuses/reopened.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Reopened</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/resolved.png\" value=\"5\" title=\"Resolved\" data-simple-status=\"{"id":"5","name":"Resolved","description":"A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.","iconUrl":"/images/icons/statuses/resolved.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Resolved</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/closed.png\" value=\"6\" title=\"Closed\" data-simple-status=\"{"id":"6","name":"Closed","description":"The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.","iconUrl":"/images/icons/statuses/closed.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Closed</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10000\" title=\"Accepted\" data-simple-status=\"{"id":"10000","name":"Accepted","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Accepted</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10001\" title=\"In Backlog\" data-simple-status=\"{"id":"10001","name":"In Backlog","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">In Backlog</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10002\" title=\"Blocked External\" data-simple-status=\"{"id":"10002","name":"Blocked External","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Blocked External</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10003\" title=\"Blocked Internal\" data-simple-status=\"{"id":"10003","name":"Blocked Internal","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Blocked Internal</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10004\" title=\"Pending Review\" data-simple-status=\"{"id":"10004","name":"Pending Review","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Pending Review</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10105\" title=\"Waiting On Gatekeeper\" data-simple-status=\"{"id":"10105","name":"Waiting On Gatekeeper","description":"Ticket is waiting on the gate keeper for code to land.","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Waiting On Gatekeeper</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/closed.png\" value=\"10205\" title=\"Done\" data-simple-status=\"{"id":"10205","name":"Done","description":"","iconUrl":"/images/icons/statuses/closed.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Done</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/open.png\" value=\"10206\" title=\"To Do\" data-simple-status=\"{"id":"10206","name":"To Do","description":"","iconUrl":"/images/icons/statuses/open.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">To Do</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/information.png\" value=\"10305\" title=\"In Review\" data-simple-status=\"{"id":"10305","name":"In Review","description":"","iconUrl":"/images/icons/statuses/information.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">In Review</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10505\" title=\"Waiting\" data-simple-status=\"{"id":"10505","name":"Waiting","description":"Waiting for a response or something else that is required to complete the issue.","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Waiting</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10605\" title=\"Committed\" data-simple-status=\"{"id":"10605","name":"Committed","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Committed</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10606\" title=\"Rejected\" data-simple-status=\"{"id":"10606","name":"Rejected","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Rejected</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10607\" title=\"Completed\" data-simple-status=\"{"id":"10607","name":"Completed","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Completed</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10608\" title=\"New\" data-simple-status=\"{"id":"10608","name":"New","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">New</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10705\" title=\"Awaiting Verification\" data-simple-status=\"{"id":"10705","name":"Awaiting Verification","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":4,"key":"indeterminate","colorName":"inprogress"}}\">Awaiting Verification</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10706\" title=\"Fix Verified\" data-simple-status=\"{"id":"10706","name":"Fix Verified","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":3,"key":"done","colorName":"success"}}\">Fix Verified</option>\n <option class=\"imagebacked\" data-icon=\"/images/icons/statuses/generic.png\" value=\"10806\" title=\"Need Information\" data-simple-status=\"{"id":"10806","name":"Need Information","description":"","iconUrl":"/images/icons/statuses/generic.png","statusCategory":{"id":2,"key":"new","colorName":"default"}}\">Need Information</option>\n </optgroup>\n</select>\n </div>\n \n","validSearcher":true,"isShown":true}}}
[{"id":-1,"name":"My open issues","jql":"assignee = currentUser() AND resolution = Unresolved order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-2,"name":"Reported by me","jql":"reporter = currentUser() order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":true},{"id":-4,"name":"All issues","jql":"order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-5,"name":"Open issues","jql":"resolution = Unresolved order by priority DESC,updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-9,"name":"Done issues","jql":"statusCategory = Done order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-3,"name":"Viewed recently","jql":"issuekey in issueHistory() order by lastViewed DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-6,"name":"Created recently","jql":"created >= -1w order by created DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-7,"name":"Resolved recently","jql":"resolutiondate >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false},{"id":-8,"name":"Updated recently","jql":"updated >= -1w order by updated DESC","isSystem":true,"sharePermissions":[],"requiresLogin":false}]
Major 
Please feel free to reopen it once have new info.