Details
-
Bug
-
Resolution: Won't Fix
-
Major
-
None
-
Lustre 2.4.2
-
Server: CentOS 6.3/Lustre 2.4.2
Client: CentOS 6.3/Lustre 2.5.2
-
3
-
15353
Description
We are getting increasing number of MDS service inactive like below. Whenever this happens, clients get evicted and file-system hang.
LNet: Service thread pid 3766 was inactive for 334.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 3766, comm: mdt_rdpg00_003
Call Trace:
[<ffffffff81133281>] ? shrink_inactive_list+0x191/0x830
[<ffffffff81133017>] ? shrink_active_list+0x297/0x370
[<ffffffff81133cce>] ? shrink_mem_cgroup_zone+0x3ae/0x610
[<ffffffff8117289d>] ? mem_cgroup_iter+0xfd/0x280
[<ffffffff81133f93>] ? shrink_zone+0x63/0xb0
[<ffffffff81134b99>] ? zone_reclaim+0x349/0x400
[<ffffffff8112a6ac>] ? get_page_from_freelist+0x69c/0x830
[<ffffffff811b5a07>] ? __find_get_block+0x97/0xe0
[<ffffffff8112bc43>] ? __alloc_pages_nodemask+0x113/0x8d0
[<ffffffffa0b2d46e>] ? ldiskfs_ind_get_blocks+0x35e/0x610 [ldiskfs]
[<ffffffff81170b10>] ? mem_cgroup_get_reclaim_stat_from_page+0x20/0x70
[<ffffffff8112f18d>] ? update_page_reclaim_stat+0x2d/0x70
[<ffffffff81166dc2>] ? kmem_getpages+0x62/0x170
[<ffffffff8116742f>] ? cache_grow+0x2cf/0x320
[<ffffffff8116774a>] ? ____cache_alloc_node+0x8a/0x160
[<ffffffff811686db>] ? kmem_cache_alloc+0x11b/0x190
[<ffffffffa0b526b0>] ? ldiskfs_alloc_inode+0x20/0x150 [ldiskfs]
[<ffffffff8119d257>] ? alloc_inode+0x27/0xa0
[<ffffffff8119e168>] ? iget_locked+0x78/0x170
[<ffffffffa0b2f6e7>] ? ldiskfs_iget+0x37/0x800 [ldiskfs]
[<ffffffffa0791ad2>] ? fld_server_lookup+0x72/0x3d0 [fld]
[<ffffffff8119cbbe>] ? generic_detach_inode+0x18e/0x1f0
[<ffffffffa0b9e4ae>] ? osd_iget+0x2e/0x2c0 [osd_ldiskfs]
[<ffffffffa0ba9546>] ? osd_ea_fid_get+0x176/0x2c0 [osd_ldiskfs]
[<ffffffffa0b9ff8a>] ? osd_remote_fid+0x9a/0x280 [osd_ldiskfs]
[<ffffffffa0bb00f5>] ? osd_it_ea_rec+0xb45/0x1470 [osd_ldiskfs]
[<ffffffffa0b1f3a5>] ? call_filldir+0xb5/0x150 [ldiskfs]
[<ffffffffa0bad170>] ? osd_ldiskfs_filldir+0x0/0x480 [osd_ldiskfs]
[<ffffffffa0b1fd09>] ? ldiskfs_readdir+0x5a9/0x730 [ldiskfs]
[<ffffffffa0bad170>] ? osd_ldiskfs_filldir+0x0/0x480 [osd_ldiskfs]
[<ffffffffa0b5f1fd>] ? htree_unlock+0x3d/0x2c6 [ldiskfs]
[<ffffffffa0d70341>] ? lod_it_rec+0x21/0x90 [lod]
[<ffffffffa0a241dc>] ? mdd_dir_page_build+0xfc/0x210 [mdd]
[<ffffffffa0435b22>] ? dt_index_walk+0x162/0x3d0 [obdclass]
[<ffffffff815101e6>] ? down_read+0x16/0x30
[<ffffffffa0a240e0>] ? mdd_dir_page_build+0x0/0x210 [mdd]
[<ffffffffa0a25e2b>] ? mdd_readpage+0x38b/0x5a0 [mdd]
[<ffffffffa0cb727f>] ? mdt_readpage+0x47f/0x960 [mdt]
[<ffffffffa0ca7a97>] ? mdt_handle_common+0x647/0x16d0 [mdt]
[<ffffffffa0ce1685>] ? mds_readpage_handle+0x15/0x20 [mdt]
[<ffffffffa05cf3a8>] ? ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
[<ffffffffa02e95de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
[<ffffffffa02fad9f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
[<ffffffffa05c6709>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
[<ffffffff81055ad3>] ? __wake_up+0x53/0x70
[<ffffffffa05d073e>] ? ptlrpc_main+0xace/0x1700 [ptlrpc]
[<ffffffffa05cfc70>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa05cfc70>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
[<ffffffffa05cfc70>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
LustreError: dumping log to /tmp/lustre-log.1408400803.3766
Attachments
Issue Links
- is related to
-
-
- Resolved
-
Activity
what is the timespan of lustre-log.1408400803.3766.gz, I didn't see client eviction messages in it.
since
Whenever this happens, clients get evicted and file-system hang.
Zheng,
On MDS, here is listing of the /tmp
Note that /tmp was filled up over the weekend hence some files had size 0.
[root@dolphin-mds-9-2 ~]# ls -ltr /tmp/
total 13449800
drwx------ 2 root root 4096 Mar 10 11:21 RCS
rw-rr- 1 root root 88189 Aug 20 12:23 dolphin-mds-9-2-20140820-122306.raw.gz
rw-rr- 1 root root 109622 Aug 20 12:24 dolphin-mds-9-2-20140820-122335.raw.gz
rw-rr- 1 root root 331058 Aug 20 12:32 dolphin-mds-9-2-20140820-122837.raw.gz
rw------ 1 root root 64856378 Aug 20 14:52 lustre-log.1408571551.15394
rw------ 1 root root 16733296 Aug 20 14:54 lustre-log.1408571641.3653
rw------ 1 root root 29240045 Aug 20 14:56 lustre-log.1408571811.3789
rw------ 1 root root 45241466 Aug 20 15:06 lustre-log.1408572214.3396
rw------ 1 root root 64718952 Aug 20 16:59 lustre-log.1408578937.15394
rw------ 1 root root 64952001 Aug 20 17:25 lustre-log.1408580070.15394
rw------ 1 root root 64934497 Aug 20 17:42 lustre-log.1408581306.3396
rw------ 1 root root 65034904 Aug 20 18:28 lustre-log.1408584513.3683
rw------ 1 root root 64831447 Aug 20 20:35 lustre-log.1408591802.3802
rw------ 1 root root 64929739 Aug 20 23:17 lustre-log.1408601860.15394
rw------ 1 root root 64787617 Aug 20 23:27 lustre-log.1408602479.3683
rw------ 1 root root 55597661 Aug 20 23:36 lustre-log.1408602992.3753
rw------ 1 root root 65060039 Aug 21 02:23 lustre-log.1408613037.3753
rw------ 1 root root 65186948 Aug 21 08:11 lustre-log.1408633864.3395
rw------ 1 root root 65060435 Aug 21 09:29 lustre-log.1408638543.15394
rw------ 1 root root 64968873 Aug 21 09:51 lustre-log.1408639881.3683
rw------ 1 root root 64927809 Aug 21 10:34 lustre-log.1408642444.3683
rw------ 1 root root 64975153 Aug 21 11:24 lustre-log.1408645453.3683
rw------ 1 root root 65036350 Aug 21 12:28 lustre-log.1408649338.15394
rw------ 1 root root 64867969 Aug 21 13:00 lustre-log.1408651201.15394
rw------ 1 root root 32472966 Aug 21 13:06 lustre-log.1408651435.3710
rw------ 1 root root 60896234 Aug 21 13:43 lustre-log.1408653790.3794
rw------ 1 root root 60851596 Aug 21 14:03 lustre-log.1408655038.3794
rw------ 1 root root 61772844 Aug 21 14:26 lustre-log.1408656414.15394
rw------ 1 root root 64798604 Aug 21 14:50 lustre-log.1408657840.3794
rw------ 1 root root 64765560 Aug 21 15:05 lustre-log.1408658557.3683
rw------ 1 root root 64763307 Aug 21 15:30 lustre-log.1408660232.3794
rw------ 1 root root 39852856 Aug 21 15:32 lustre-log.1408660334.3778
rw------ 1 root root 64871356 Aug 21 15:54 lustre-log.1408661611.3396
rw------ 1 root root 64788431 Aug 21 16:39 lustre-log.1408664371.3796
rw------ 1 root root 64705388 Aug 21 17:12 lustre-log.1408666325.3754
rw------ 1 root root 290166 Aug 21 17:12 lustre-log.1408666334.3676
rw------ 1 root root 12951514 Aug 21 17:12 lustre-log.1408666334.15394
rw------ 1 root root 11861616 Aug 21 17:12 lustre-log.1408666343.3749
rw------ 1 root root 189345 Aug 21 17:12 lustre-log.1408666347.3743
rw------ 1 root root 4508540 Aug 21 17:12 lustre-log.1408666347.3385
rw------ 1 root root 8101615 Aug 21 17:12 lustre-log.1408666354.3599
rw------ 1 root root 12960455 Aug 21 17:12 lustre-log.1408666362.3778
rw------ 1 root root 6265040 Aug 21 17:12 lustre-log.1408666363.3716
rw------ 1 root root 14125256 Aug 21 17:12 lustre-log.1408666374.3756
rw------ 1 root root 19298834 Aug 21 17:13 lustre-log.1408666396.3703
rw------ 1 root root 48790339 Aug 21 17:16 lustre-log.1408666579.3758
rw------ 1 root root 1751751 Aug 21 17:16 lustre-log.1408666584.3396
rw------ 1 root root 62297722 Aug 21 17:27 lustre-log.1408667268.3740
rw------ 1 root root 11156692 Aug 21 17:29 lustre-log.1408667283.3715
rw------ 1 root root 64819776 Aug 21 17:49 lustre-log.1408668599.15393
rw------ 1 root root 60913323 Aug 21 18:38 lustre-log.1408671483.3794
rw------ 1 root root 64637863 Aug 21 19:30 lustre-log.1408674629.3395
rw------ 1 root root 64603682 Aug 21 20:01 lustre-log.1408676478.3400
rw------ 1 root root 60697956 Aug 21 20:09 lustre-log.1408676946.3768
rw------ 1 root root 6962681 Aug 21 20:09 lustre-log.1408676959.3743
rw------ 1 root root 90393 Aug 21 20:09 lustre-log.1408676959.13615
rw------ 1 root root 3434802 Aug 21 20:09 lustre-log.1408676964.3385
rw------ 1 root root 2592231 Aug 21 20:09 lustre-log.1408676965.3722
rw------ 1 root root 13436318 Aug 21 20:09 lustre-log.1408676971.3673
rw------ 1 root root 9829660 Aug 21 20:09 lustre-log.1408676975.3750
rw------ 1 root root 9758539 Aug 21 20:10 lustre-log.1408676981.3729
rw------ 1 root root 34013365 Aug 21 20:13 lustre-log.1408677015.3747
rw------ 1 root root 50949684 Aug 21 20:13 lustre-log.1408677223.3702
rw------ 1 root root 56933286 Aug 21 20:18 lustre-log.1408677393.3730
rw------ 1 root root 43362181 Aug 21 20:18 lustre-log.1408677505.3724
rw------ 1 root root 29101077 Aug 21 20:19 lustre-log.1408677548.3396
rw------ 1 root root 8979661 Aug 21 20:19 lustre-log.1408677563.3768
rw------ 1 root root 60274601 Aug 21 20:21 lustre-log.1408677714.3707
rw------ 1 root root 33697912 Aug 21 20:23 lustre-log.1408677780.13615
rw------ 1 root root 9178954 Aug 21 20:23 lustre-log.1408677787.3758
rw------ 1 root root 6869833 Aug 21 20:23 lustre-log.1408677797.3722
rw------ 1 root root 987487 Aug 21 20:23 lustre-log.1408677798.3673
rw------ 1 root root 26307200 Aug 21 20:23 lustre-log.1408677837.3764
rw------ 1 root root 20859592 Aug 21 20:24 lustre-log.1408677866.3743
rw------ 1 root root 58792419 Aug 21 20:28 lustre-log.1408678095.3391
rw------ 1 root root 60804973 Aug 21 20:39 lustre-log.1408678742.3575
rw------ 1 root root 39132873 Aug 21 20:42 lustre-log.1408678823.3789
rw------ 1 root root 45425439 Aug 21 20:43 lustre-log.1408678998.3599
rw------ 1 root root 58505400 Aug 21 20:51 lustre-log.1408679504.3731
rw------ 1 root root 57241760 Aug 21 21:01 lustre-log.1408680116.3716
rw------ 1 root root 10788723 Aug 21 21:02 lustre-log.1408680120.3692
rw------ 1 root root 22539688 Aug 21 21:02 lustre-log.1408680146.13615
rw------ 1 root root 19105938 Aug 21 21:03 lustre-log.1408680175.3758
rw------ 1 root root 57213533 Aug 21 21:11 lustre-log.1408680718.3778
rw------ 1 root root 22652767 Aug 21 21:12 lustre-log.1408680741.3774
rw------ 1 root root 12289309 Aug 21 21:12 lustre-log.1408680767.3764
rw------ 1 root root 15409578 Aug 21 21:13 lustre-log.1408680809.3767
rw------ 1 root root 7735637 Aug 21 21:13 lustre-log.1408680813.3707
rw------ 1 root root 58640898 Aug 21 21:21 lustre-log.1408681261.3576
rw------ 1 root root 22421275 Aug 21 21:22 lustre-log.1408681349.14007
rw------ 1 root root 64906096 Aug 21 23:05 lustre-log.1408687515.3683
rw------ 1 root root 40284768 Aug 21 23:08 lustre-log.1408687717.3712
rw------ 1 root root 45396155 Aug 21 23:12 lustre-log.1408687936.14007
rw------ 1 root root 60719228 Aug 21 23:34 lustre-log.1408689276.15390
rw------ 1 root root 60553827 Aug 22 00:35 lustre-log.1408692911.3576
rw------ 1 root root 60577749 Aug 22 01:06 lustre-log.1408694774.15393
rw------ 1 root root 60718812 Aug 22 01:25 lustre-log.1408695934.3698
rw------ 1 root root 64743923 Aug 22 04:34 lustre-log.1408707288.3794
rw------ 1 root root 33443900 Aug 22 04:35 lustre-log.1408707339.3730
rw------ 1 root root 25170805 Aug 22 04:35 lustre-log.1408707358.3575
rw------ 1 root root 9486559 Aug 22 04:36 lustre-log.1408707360.3767
rw------ 1 root root 64743275 Aug 22 05:01 lustre-log.1408708891.15393
rw------ 1 root root 16648693 Aug 22 05:01 lustre-log.1408708906.3394
rw------ 1 root root 64634402 Aug 22 05:48 lustre-log.1408711692.3400
rw------ 1 root root 42292633 Aug 22 05:49 lustre-log.1408711798.3646
rw------ 1 root root 9538191 Aug 22 05:50 lustre-log.1408711801.3576
rw------ 1 root root 15850360 Aug 22 05:50 lustre-log.1408711805.13179
rw------ 1 root root 9531803 Aug 22 05:50 lustre-log.1408711809.3391
rw------ 1 root root 3511794 Aug 22 05:50 lustre-log.1408711813.3749
rw------ 1 root root 20356016 Aug 22 05:50 lustre-log.1408711828.3736
rw------ 1 root root 24174089 Aug 22 05:50 lustre-log.1408711849.3673
rw------ 1 root root 52253899 Aug 22 05:53 lustre-log.1408711989.3707
rw------ 1 root root 22879004 Aug 22 05:53 lustre-log.1408712016.3723
rw------ 1 root root 62811752 Aug 22 05:56 lustre-log.1408712218.3757
rw------ 1 root root 50146596 Aug 22 05:58 lustre-log.1408712294.3673
rw------ 1 root root 15040450 Aug 22 05:58 lustre-log.1408712303.3767
rw------ 1 root root 51602345 Aug 22 06:04 lustre-log.1408712403.3391
rw------ 1 root root 60881326 Aug 22 06:08 lustre-log.1408712862.3676
rw------ 1 root root 47201845 Aug 22 06:14 lustre-log.1408713296.3723
rw------ 1 root root 50482970 Aug 22 06:27 lustre-log.1408713853.3754
rw------ 1 root root 53853135 Aug 22 06:33 lustre-log.1408714350.3744
rw------ 1 root root 49973094 Aug 22 06:39 lustre-log.1408714750.3676
rw------ 1 root root 3125898 Aug 22 06:39 lustre-log.1408714760.3395
rw------ 1 root root 64389798 Aug 22 06:46 lustre-log.1408715054.3774
rw------ 1 root root 47722048 Aug 22 06:50 lustre-log.1408715450.3754
rw------ 1 root root 9108016 Aug 22 06:50 lustre-log.1408715454.3707
rw------ 1 root root 58776608 Aug 22 07:00 lustre-log.1408716054.3700
rw------ 1 root root 50582444 Aug 22 07:04 lustre-log.1408716256.3676
rw------ 1 root root 58281329 Aug 22 07:10 lustre-log.1408716641.3396
rw------ 1 root root 40277168 Aug 22 07:11 lustre-log.1408716678.3723
rw------ 1 root root 15202336 Aug 22 07:11 lustre-log.1408716685.3757
rw------ 1 root root 48992510 Aug 22 07:15 lustre-log.1408716910.15394
rw------ 1 root root 54145954 Aug 22 07:20 lustre-log.1408717243.3577
rw------ 1 root root 5820274 Aug 22 07:20 lustre-log.1408717252.3395
rw------ 1 root root 604895 Aug 22 07:20 lustre-log.1408717256.3789
rw------ 1 root root 1610826 Aug 22 07:21 lustre-log.1408717266.3774
rw------ 1 root root 58552048 Aug 22 07:29 lustre-log.1408717776.3794
rw------ 1 root root 38277962 Aug 22 07:30 lustre-log.1408717856.3749
rw------ 1 root root 8505256 Aug 22 07:30 lustre-log.1408717859.3673
rw------ 1 root root 4976044 Aug 22 07:31 lustre-log.1408717865.3599
rw------ 1 root root 161057 Aug 22 07:31 lustre-log.1408717866.3730
rw------ 1 root root 60732351 Aug 22 07:41 lustre-log.1408718476.3738
rw------ 1 root root 23163397 Aug 22 07:41 lustre-log.1408718506.3757
rw------ 1 root root 41134806 Aug 22 07:43 lustre-log.1408718616.3707
rw------ 1 root root 7572039 Aug 22 07:43 lustre-log.1408718624.13179
rw------ 1 root root 2712914 Aug 22 07:43 lustre-log.1408718634.3676
rw------ 1 root root 13279300 Aug 22 07:44 lustre-log.1408718640.3395
rw------ 1 root root 54486283 Aug 22 07:47 lustre-log.1408718846.3394
rw------ 1 root root 11346621 Aug 22 07:47 lustre-log.1408718859.3700
rw------ 1 root root 31824277 Aug 22 07:48 lustre-log.1408718937.3706
rw------ 1 root root 41548331 Aug 22 07:53 lustre-log.1408719238.3388
rw------ 1 root root 796564 Aug 22 07:54 lustre-log.1408719242.30684
rw------ 1 root root 2130563 Aug 22 07:55 lustre-log.1408719322.3736
rw------ 1 root root 29835246 Aug 22 07:55 lustre-log.1408719322.3730
rw------ 1 root root 53319 Aug 22 07:55 lustre-log.1408719337.3754
rw------ 1 root root 5883371 Aug 22 07:55 lustre-log.1408719337.3575
rw------ 1 root root 8244155 Aug 22 07:55 lustre-log.1408719342.3764
rw------ 1 root root 58395097 Aug 22 07:58 lustre-log.1408719532.3757
rw------ 1 root root 60638656 Aug 22 08:05 lustre-log.1408719922.3723
rw------ 1 root root 15158643 Aug 22 08:05 lustre-log.1408719932.3738
rw------ 1 root root 21232423 Aug 22 08:05 lustre-log.1408719947.3599
rw------ 1 root root 60743805 Aug 22 08:15 lustre-log.1408720543.3698
rw------ 1 root root 31467293 Aug 22 08:17 lustre-log.1408720621.3395
rw------ 1 root root 1108129 Aug 22 08:17 lustre-log.1408720627.3771
rw------ 1 root root 45879 Aug 22 08:17 lustre-log.1408720627.3703
rw------ 1 root root 64748208 Aug 22 09:14 lustre-log.1408723656.15393
rw------ 1 root root 50645018 Aug 22 09:14 lustre-log.1408724073.3787
rw------ 1 root root 18836380 Aug 22 09:15 lustre-log.1408724103.3697
rw------ 1 root root 49984648 Aug 22 09:21 lustre-log.1408724462.3794
rw------ 1 root root 37824019 Aug 22 09:25 lustre-log.1408724723.3721
rw------ 1 root root 19603501 Aug 22 09:28 lustre-log.1408724880.3698
rw------ 1 root root 1204398 Aug 22 09:28 lustre-log.1408724890.3700
rw------ 1 root root 56032439 Aug 22 09:37 lustre-log.1408725437.3740
rw------ 1 root root 37984020 Aug 22 09:39 lustre-log.1408725595.3604
rw------ 1 root root 35372299 Aug 22 09:42 lustre-log.1408725764.3673
rw------ 1 root root 7990275 Aug 22 09:42 lustre-log.1408725777.3653
rw------ 1 root root 9435481 Aug 22 09:44 lustre-log.1408725861.3391
rw------ 1 root root 440314 Aug 22 09:44 lustre-log.1408725866.3737
rw------ 1 root root 205413 Aug 22 09:44 lustre-log.1408725868.3716
rw------ 1 root root 406941 Aug 22 09:44 lustre-log.1408725871.3676
rw------ 1 root root 18770650 Aug 22 09:45 lustre-log.1408725933.3703
rw------ 1 root root 60878253 Aug 22 10:03 lustre-log.1408727021.3769
rw------ 1 root root 9560337 Aug 22 10:03 lustre-log.1408727028.3754
rw------ 1 root root 26935844 Aug 22 10:05 lustre-log.1408727101.3757
rw------ 1 root root 41367936 Aug 22 10:09 lustre-log.1408727385.3739
rw------ 1 root root 34292016 Aug 22 10:13 lustre-log.1408727611.3388
rw------ 1 root root 57359594 Aug 22 10:22 lustre-log.1408728179.3759
rw------ 1 root root 64754668 Aug 22 11:21 lustre-log.1408731675.3693
rw------ 1 root root 8107691 Aug 22 11:21 lustre-log.1408731677.3757
rw------ 1 root root 13472321 Aug 22 11:21 lustre-log.1408731680.3754
rw------ 1 root root 64035819 Aug 22 11:31 lustre-log.1408732269.3673
rw------ 1 root root 4188224 Aug 22 11:31 lustre-log.1408732270.3604
rw------ 1 root root 5765992 Aug 22 11:31 lustre-log.1408732274.3716
rw------ 1 root root 4631322 Aug 22 11:31 lustre-log.1408732280.3750
rw------ 1 root root 1036323 Aug 22 11:31 lustre-log.1408732283.3708
rw------ 1 root root 1175404 Aug 22 11:31 lustre-log.1408732293.3396
rw------ 1 root root 64779862 Aug 22 12:22 lustre-log.1408735353.3794
rw------ 1 root root 2034092 Aug 22 12:22 lustre-log.1408735373.3787
rw------ 1 root root 64787161 Aug 22 13:18 lustre-log.1408738699.15393
rw------ 1 root root 64826869 Aug 22 13:27 lustre-log.1408739276.3794
rw------ 1 root root 64756422 Aug 22 14:21 lustre-log.1408742478.3400
rw------ 1 root root 2078088 Aug 22 14:21 lustre-log.1408742488.3723
rw------ 1 root root 35478 Aug 22 14:21 lustre-log.1408742489.3754
rw------ 1 root root 1515987 Aug 22 14:21 lustre-log.1408742498.3762
rw------ 1 root root 64674592 Aug 22 14:38 lustre-log.1408743500.3796
rw------ 1 root root 64635472 Aug 22 15:00 lustre-log.1408744842.15394
rw------ 1 root root 64779849 Aug 22 15:37 lustre-log.1408747025.3683
rw------ 1 root root 3972319 Aug 22 15:37 lustre-log.1408747045.3708
rw------ 1 root root 34625007 Aug 22 15:39 lustre-log.1408747194.3762
rw------ 1 root root 17951688 Aug 22 15:40 lustre-log.1408747220.14007
rw------ 1 root root 1054873 Aug 22 15:40 lustre-log.1408747223.3634
rw------ 1 root root 21557428 Aug 22 15:41 lustre-log.1408747248.3592
rw------ 1 root root 23923042 Aug 22 15:42 lustre-log.1408747279.3757
rw------ 1 root root 51309095 Aug 22 15:44 lustre-log.1408747445.3697
rw------ 1 root root 26545882 Aug 22 15:45 lustre-log.1408747488.3708
rw------ 1 root root 64657388 Aug 22 15:53 lustre-log.1408747984.3576
rw------ 1 root root 456098 Aug 22 15:53 lustre-log.1408747984.3391
rw------ 1 root root 22805226 Aug 22 15:53 lustre-log.1408747998.3710
rw------ 1 root root 53441585 Aug 22 15:54 lustre-log.1408748059.3762
rw------ 1 root root 25216807 Aug 22 15:54 lustre-log.1408748074.3592
rw------ 1 root root 26187245 Aug 22 15:54 lustre-log.1408748085.3753
rw------ 1 root root 21366388 Aug 22 15:54 lustre-log.1408748092.3758
rw------ 1 root root 52693746 Aug 22 15:56 lustre-log.1408748199.3712
rw------ 1 root root 32627138 Aug 22 15:57 lustre-log.1408748226.3599
rw------ 1 root root 15698462 Aug 22 15:57 lustre-log.1408748234.3754
rw------ 1 root root 35504606 Aug 22 15:58 lustre-log.1408748283.3395
rw------ 1 root root 50724815 Aug 22 16:01 lustre-log.1408748475.3701
rw------ 1 root root 64693911 Aug 22 16:10 lustre-log.1408749046.3683
rw------ 1 root root 64726349 Aug 22 16:24 lustre-log.1408749864.15393
rw------ 1 root root 64778342 Aug 22 16:50 lustre-log.1408751443.15394
rw------ 1 root root 11538453 Aug 22 16:50 lustre-log.1408751452.3787
rw------ 1 root root 9836518 Aug 22 16:51 lustre-log.1408751471.3576
rw------ 1 root root 14286259 Aug 22 16:51 lustre-log.1408751491.3732
rw------ 1 root root 25117734 Aug 22 16:51 lustre-log.1408751511.3738
rw------ 1 root root 64585793 Aug 22 17:04 lustre-log.1408752251.3683
rw------ 1 root root 64682564 Aug 22 17:44 lustre-log.1408754666.3576
rw------ 1 root root 63119417 Aug 22 17:50 lustre-log.1408755041.3683
rw------ 1 root root 64862670 Aug 22 18:39 lustre-log.1408757979.3608
rw------ 1 root root 1480254 Aug 22 18:39 lustre-log.1408757980.3700
rw------ 1 root root 8559630 Aug 22 18:39 lustre-log.1408757983.3698
rw------ 1 root root 12023244 Aug 22 18:39 lustre-log.1408757998.3697
rw------ 1 root root 1016461 Aug 22 18:40 lustre-log.1408758002.13180
rw------ 1 root root 1993554 Aug 22 18:40 lustre-log.1408758010.30684
rw------ 1 root root 25806319 Aug 22 18:43 lustre-log.1408758157.13615
rw------ 1 root root 27000779 Aug 22 18:45 lustre-log.1408758284.3731
rw------ 1 root root 30579350 Aug 22 18:47 lustre-log.1408758456.3697
rw------ 1 root root 11438811 Aug 22 18:48 lustre-log.1408758486.3756
rw------ 1 root root 16846640 Aug 22 18:49 lustre-log.1408758534.3592
rw------ 1 root root 65220434 Aug 22 19:00 lustre-log.1408759194.13615
rw------ 1 root root 27442850 Aug 22 19:02 lustre-log.1408759299.3608
rw------ 1 root root 60997600 Aug 22 19:57 lustre-log.1408762651.3794
rw------ 1 root root 9534558 Aug 22 19:57 lustre-log.1408762676.3653
rw------ 1 root root 64992300 Aug 22 20:29 lustre-log.1408764552.3781
rw------ 1 root root 63758308 Aug 22 20:38 lustre-log.1408765119.3794
rw------ 1 root root 64895669 Aug 22 20:53 lustre-log.1408766000.3794
rw------ 1 root root 9269953 Aug 22 20:53 lustre-log.1408766008.13179
rw------ 1 root root 35790216 Aug 22 20:56 lustre-log.1408766215.3750
rw------ 1 root root 13520228 Aug 22 20:57 lustre-log.1408766243.3740
rw------ 1 root root 8549093 Aug 22 20:57 lustre-log.1408766260.3712
rw------ 1 root root 26426527 Aug 22 20:59 lustre-log.1408766361.3608
rw------ 1 root root 5917210 Aug 22 20:59 lustre-log.1408766374.3707
rw------ 1 root root 37546940 Aug 22 21:03 lustre-log.1408766592.3701
rw------ 1 root root 50638878 Aug 22 21:09 lustre-log.1408766989.3704
rw------ 1 root root 64881116 Aug 22 21:48 lustre-log.1408769305.15390
rw------ 1 root root 8158920 Aug 22 21:48 lustre-log.1408769330.13179
rw------ 1 root root 19175772 Aug 22 21:50 lustre-log.1408769415.3764
rw------ 1 root root 19294268 Aug 22 21:50 lustre-log.1408769452.3720
rw------ 1 root root 5194092 Aug 22 21:50 lustre-log.1408769453.3576
rw------ 1 root root 21405735 Aug 22 21:52 lustre-log.1408769535.3645
rw------ 1 root root 40400067 Aug 22 21:57 lustre-log.1408769781.3708
rw------ 1 root root 35795960 Aug 22 22:00 lustre-log.1408770003.3704
rw------ 1 root root 35577395 Aug 22 22:03 lustre-log.1408770213.3767
rw------ 1 root root 18770972 Aug 22 22:04 lustre-log.1408770240.3764
rw------ 1 root root 22927633 Aug 22 22:05 lustre-log.1408770303.3707
rw------ 1 root root 38442591 Aug 22 22:10 lustre-log.1408770534.3386
rw------ 1 root root 52339332 Aug 22 22:19 lustre-log.1408771161.3698
rw------ 1 root root 18858413 Aug 22 22:20 lustre-log.1408771232.3730
rw------ 1 root root 13256469 Aug 22 22:20 lustre-log.1408771238.3645
rw------ 1 root root 8298736 Aug 22 22:20 lustre-log.1408771242.13179
rw------ 1 root root 1392430 Aug 22 22:20 lustre-log.1408771246.3704
rw------ 1 root root 40817057 Aug 22 22:24 lustre-log.1408771496.3746
rw------ 1 root root 41879587 Aug 22 22:30 lustre-log.1408771843.3764
rw------ 1 root root 51337638 Aug 22 22:39 lustre-log.1408772383.3767
rw------ 1 root root 16212869 Aug 22 22:40 lustre-log.1408772425.3707
rw------ 1 root root 33845894 Aug 22 22:44 lustre-log.1408772561.3577
rw------ 1 root root 47281454 Aug 22 22:50 lustre-log.1408773025.3730
rw------ 1 root root 17252912 Aug 22 22:51 lustre-log.1408773038.3708
rw------ 1 root root 34191262 Aug 22 22:55 lustre-log.1408773227.3701
rw------ 1 root root 60803358 Aug 22 23:00 lustre-log.1408773627.3576
rw------ 1 root root 16322671 Aug 22 23:00 lustre-log.1408773636.3764
rw------ 1 root root 2628318 Aug 22 23:00 lustre-log.1408773638.3767
rw------ 1 root root 12120076 Aug 22 23:00 lustre-log.1408773648.3746
rw------ 1 root root 47758846 Aug 22 23:05 lustre-log.1408773931.3711
rw------ 1 root root 56376249 Aug 22 23:16 lustre-log.1408774236.3731
rw------ 1 root root 62818422 Aug 22 23:24 lustre-log.1408774946.3720
rw------ 1 root root 60036451 Aug 22 23:35 lustre-log.1408775733.3711
rw------ 1 root root 5008166 Aug 22 23:35 lustre-log.1408775752.3653
rw------ 1 root root 54720619 Aug 22 23:45 lustre-log.1408776353.13615
rw------ 1 root root 64884718 Aug 23 00:43 lustre-log.1408779837.3794
rw------ 1 root root 395025 Aug 23 00:44 lustre-log.1408779841.3703
rw------ 1 root root 65646 Aug 23 00:44 lustre-log.1408779842.3750
rw------ 1 root root 1386096 Aug 23 00:44 lustre-log.1408779850.3704
rw------ 1 root root 1993056 Aug 23 00:44 lustre-log.1408779854.3693
rw------ 1 root root 57827 Aug 23 00:44 lustre-log.1408779855.3789
rw------ 1 root root 602862 Aug 23 00:44 lustre-log.1408779860.3720
rw------ 1 root root 56855 Aug 23 00:44 lustre-log.1408779861.3787
rw------ 1 root root 12708303 Aug 23 00:45 lustre-log.1408779938.13180
rw------ 1 root root 8731398 Aug 23 00:45 lustre-log.1408779945.3645
rw------ 1 root root 1555547 Aug 23 00:45 lustre-log.1408779956.3777
rw------ 1 root root 49164102 Aug 23 00:52 lustre-log.1408780319.3693
rw------ 1 root root 655226 Aug 23 00:52 lustre-log.1408780325.3789
rw------ 1 root root 49094157 Aug 23 00:56 lustre-log.1408780607.3394
rw------ 1 root root 10948352 Aug 23 00:57 lustre-log.1408780636.3751
rw------ 1 root root 3613338 Aug 23 00:57 lustre-log.1408780653.3762
rw------ 1 root root 46768554 Aug 23 01:05 lustre-log.1408780949.3777
rw------ 1 root root 64937275 Aug 23 01:31 lustre-log.1408782691.3683
rw------ 1 root root 53574802 Aug 23 01:39 lustre-log.1408783146.3704
rw------ 1 root root 54144234 Aug 23 01:47 lustre-log.1408783643.3395
rw------ 1 root root 796308 Aug 23 01:47 lustre-log.1408783650.3789
rw------ 1 root root 11307208 Aug 23 01:48 lustre-log.1408783739.3767
rw------ 1 root root 7365262 Aug 23 01:49 lustre-log.1408783781.3731
rw------ 1 root root 1318671 Aug 23 01:49 lustre-log.1408783793.3778
rw------ 1 root root 23451796 Aug 23 01:51 lustre-log.1408783863.3396
rw------ 1 root root 3319 Aug 23 01:51 lustre-log.1408783875.3764
rw------ 1 root root 1348810 Aug 23 01:51 lustre-log.1408783875.3750
rw------ 1 root root 60502821 Aug 23 02:00 lustre-log.1408784411.3700
rw------ 1 root root 11227248 Aug 23 02:01 lustre-log.1408784461.3653
rw------ 1 root root 5363606 Aug 23 02:01 lustre-log.1408784500.3751
rw------ 1 root root 4538412 Aug 23 02:02 lustre-log.1408784535.3704
rw------ 1 root root 5771930 Aug 23 02:02 lustre-log.1408784577.3693
rw------ 1 root root 1533573 Aug 23 02:03 lustre-log.1408784586.3762
rw------ 1 root root 63217425 Aug 23 02:17 lustre-log.1408785471.13179
rw------ 1 root root 991259 Aug 23 02:17 lustre-log.1408785479.3673
rw------ 1 root root 7444865 Aug 23 02:18 lustre-log.1408785500.3750
rw------ 1 root root 1208186 Aug 23 02:18 lustre-log.1408785509.3767
rw------ 1 root root 57158030 Aug 23 02:28 lustre-log.1408786084.3700
rw------ 1 root root 9953496 Aug 23 02:28 lustre-log.1408786089.3746
rw------ 1 root root 3871280 Aug 23 02:28 lustre-log.1408786101.3777
rw------ 1 root root 338827 Aug 23 02:28 lustre-log.1408786101.3394
rw------ 1 root root 1438199 Aug 23 02:28 lustre-log.1408786104.3604
rw------ 1 root root 65139525 Aug 23 02:38 lustre-log.1408786685.3673
rw------ 1 root root 4519793 Aug 23 02:38 lustre-log.1408786689.3704
rw------ 1 root root 5817800 Aug 23 02:39 lustre-log.1408786741.3391
rw------ 1 root root 56583947 Aug 23 02:48 lustre-log.1408787285.3750
rw------ 1 root root 5425258 Aug 23 02:48 lustre-log.1408787291.3762
rw------ 1 root root 25025917 Aug 23 02:50 lustre-log.1408787390.3700
rw------ 1 root root 55041159 Aug 23 03:00 lustre-log.1408787990.3754
rw------ 1 root root 30021206 Aug 23 03:02 lustre-log.1408788095.3739
rw------ 1 root root 23077909 Aug 23 03:05 lustre-log.1408788291.3604
rw------ 1 root root 47473368 Aug 23 03:11 lustre-log.1408788695.3653
rw------ 1 root root 5241930 Aug 23 03:11 lustre-log.1408788702.3693
rw------ 1 root root 41740506 Aug 23 03:16 lustre-log.1408788990.3653
rw------ 1 root root 11515002 Aug 23 03:16 lustre-log.1408789001.3394
rw------ 1 root root 60772495 Aug 23 03:48 lustre-log.1408790911.3400
rw------ 1 root root 64958373 Aug 23 04:43 lustre-log.1408794211.3683
rw------ 1 root root 3095867 Aug 23 04:44 lustre-log.1408794240.3707
rw------ 1 root root 80341 Aug 23 04:44 lustre-log.1408794241.3767
rw------ 1 root root 65071229 Aug 23 05:40 lustre-log.1408797603.3604
rw------ 1 root root 65254901 Aug 23 09:55 lustre-log.1408812948.3796
rw------ 1 root root 65226790 Aug 23 13:25 lustre-log.1408825546.3400
rw------ 1 root root 60104934 Aug 23 13:38 lustre-log.1408826310.15393
rw------ 1 root root 2885683 Aug 23 13:38 lustre-log.1408826337.3778
rw------ 1 root root 65076812 Aug 23 13:55 lustre-log.1408827230.13179
rw------ 1 root root 63071055 Aug 23 14:07 lustre-log.1408828026.3796
rw------ 1 root root 60864561 Aug 23 14:23 lustre-log.1408828888.3394
rw------ 1 root root 63139349 Aug 23 14:31 lustre-log.1408829513.3794
rw------ 1 root root 65137799 Aug 23 15:14 lustre-log.1408832069.3720
rw------ 1 root root 62311273 Aug 23 15:42 lustre-log.1408833757.3794
rw------ 1 root root 62428073 Aug 23 15:52 lustre-log.1408834349.3712
rw------ 1 root root 15144561 Aug 23 15:53 lustre-log.1408834380.3750
rw------ 1 root root 53918819 Aug 23 16:00 lustre-log.1408834763.3653
rw------ 1 root root 27672103 Aug 23 16:01 lustre-log.1408834854.3698
rw------ 1 root root 18825101 Aug 23 16:01 lustre-log.1408834892.3396
rw------ 1 root root 1977064 Aug 23 16:01 lustre-log.1408834893.3708
rw------ 1 root root 59853229 Aug 23 16:10 lustre-log.1408835419.3693
rw------ 1 root root 20904834 Aug 23 16:11 lustre-log.1408835486.3751
rw------ 1 root root 63884939 Aug 23 16:28 lustre-log.1408836488.3778
rw------ 1 root root 3920811 Aug 23 16:28 lustre-log.1408836492.3645
rw------ 1 root root 7675644 Aug 23 16:28 lustre-log.1408836499.3653
rw------ 1 root root 64791600 Aug 23 16:46 lustre-log.1408837577.3396
rw------ 1 root root 32074523 Aug 23 16:48 lustre-log.1408837685.3754
rw------ 1 root root 24760881 Aug 23 16:49 lustre-log.1408837736.3702
rw------ 1 root root 19097503 Aug 23 16:50 lustre-log.1408837816.3599
rw------ 1 root root 34196679 Aug 23 16:52 lustre-log.1408837932.3751
rw------ 1 root root 40269222 Aug 23 16:55 lustre-log.1408838048.3653
rw------ 1 root root 35588482 Aug 23 16:56 lustre-log.1408838178.3754
rw------ 1 root root 64755936 Aug 23 17:13 lustre-log.1408839201.15393
rw------ 1 root root 7599303 Aug 23 17:13 lustre-log.1408839204.15390
rw------ 1 root root 62680300 Aug 23 17:23 lustre-log.1408839791.3683
rw------ 1 root root 63160106 Aug 23 17:37 lustre-log.1408840577.3703
rw------ 1 root root 12730177 Aug 23 17:37 lustre-log.1408840676.3724
rw------ 1 root root 1708834 Aug 23 17:38 lustre-log.1408840689.13179
rw------ 1 root root 7872216 Aug 23 17:38 lustre-log.1408840739.3750
rw------ 1 root root 6912843 Aug 23 17:39 lustre-log.1408840783.3653
rw------ 1 root root 5908412 Aug 23 17:40 lustre-log.1408840823.3645
rw------ 1 root root 4630069 Aug 23 17:40 lustre-log.1408840855.3720
rw------ 1 root root 10438816 Aug 23 17:42 lustre-log.1408840928.3599
rw------ 1 root root 27833485 Aug 23 17:45 lustre-log.1408841125.3751
rw------ 1 root root 22171833 Aug 23 17:48 lustre-log.1408841293.3778
rw------ 1 root root 19588 Aug 23 17:48 lustre-log.1408841293.3698
rw------ 1 root root 4317284 Aug 23 17:48 lustre-log.1408841328.3395
rw------ 1 root root 21359549 Aug 23 17:51 lustre-log.1408841484.3697
rw------ 1 root root 13806110 Aug 23 17:53 lustre-log.1408841586.3708
rw------ 1 root root 19891974 Aug 23 17:56 lustre-log.1408841760.3744
rw------ 1 root root 7083647 Aug 23 17:57 lustre-log.1408841828.3724
rw-rr- 1 root root 554 Aug 23 18:01 411-alert-handler.pkl
rw------ 1 root root 38334417 Aug 23 18:03 lustre-log.1408842192.3394
rw------ 1 root root 41359558 Aug 23 18:11 lustre-log.1408842626.3720
rw------ 1 root root 50591073 Aug 23 18:15 lustre-log.1408842835.3599
rw------ 1 root root 63157513 Aug 23 18:20 lustre-log.1408843232.3708
rw------ 1 root root 8133368 Aug 23 18:20 lustre-log.1408843235.3700
rw------ 1 root root 19610263 Aug 23 18:21 lustre-log.1408843249.3711
rw------ 1 root root 42515217 Aug 23 18:24 lustre-log.1408843440.13179
rw------ 1 root root 58921227 Aug 23 18:30 lustre-log.1408843837.3576
rw------ 1 root root 36620316 Aug 23 18:32 lustre-log.1408843921.3577
rw------ 1 root root 59869130 Aug 23 18:41 lustre-log.1408844432.3720
rw------ 1 root root 63813462 Aug 23 18:51 lustre-log.1408844959.3744
rw------ 1 root root 42228112 Aug 23 18:54 lustre-log.1408845237.3754
rw------ 1 root root 50643783 Aug 23 19:00 lustre-log.1408845630.3711
rw------ 1 root root 7702936 Aug 23 19:00 lustre-log.1408845637.3386
rw------ 1 root root 215759 Aug 23 19:00 lustre-log.1408845638.3732
rw------ 1 root root 3423279 Aug 23 19:00 lustre-log.1408845645.3738
rw------ 1 root root 48639259 Aug 23 19:06 lustre-log.1408845946.3391
rw------ 1 root root 54992629 Aug 23 19:13 lustre-log.1408846346.13179
rw------ 1 root root 50854253 Aug 23 19:17 lustre-log.1408846638.3720
rw------ 1 root root 13400896 Aug 23 19:19 lustre-log.1408846755.3608
rw------ 1 root root 21608267 Aug 23 19:22 lustre-log.1408846940.3740
rw------ 1 root root 15049858 Aug 23 19:23 lustre-log.1408847038.3702
rw------ 1 root root 7274992 Aug 23 19:24 lustre-log.1408847057.3763
rw------ 1 root root 50345662 Aug 23 19:30 lustre-log.1408847444.3730
rw------ 1 root root 2679389 Aug 23 19:31 lustre-log.1408847458.3739
rw------ 1 root root 15609833 Aug 23 19:32 lustre-log.1408847557.3673
rw------ 1 root root 30518544 Aug 23 19:37 lustre-log.1408847836.3754
rw------ 1 root root 1402650 Aug 23 19:37 lustre-log.1408847851.3738
rw------ 1 root root 1286288 Aug 23 19:37 lustre-log.1408847863.3787
rw------ 1 root root 62344802 Aug 23 19:52 lustre-log.1408848762.15390
rw------ 1 root root 39743488 Aug 23 20:06 lustre-log.1408849585.3683
rw------ 1 root root 0 Aug 23 20:06 lustre-log.1408849587.3599
rw------ 1 root root 0 Aug 23 20:06 lustre-log.1408849590.3698
rw------ 1 root root 0 Aug 23 20:06 lustre-log.1408849592.3738
rw------ 1 root root 0 Aug 23 20:06 lustre-log.1408849594.3730
rw------ 1 root root 0 Aug 23 20:06 lustre-log.1408849595.3394
rw------ 1 root root 0 Aug 23 20:06 lustre-log.1408849601.3673
rw------ 1 root root 0 Aug 23 20:06 lustre-log.1408849602.3763
rw------ 1 root root 0 Aug 23 20:06 lustre-log.1408849604.3787
rw------ 1 root root 0 Aug 23 20:06 lustre-log.1408849606.3720
rw------ 1 root root 0 Aug 23 20:06 lustre-log.1408849610.3753
rw------ 1 root root 0 Aug 23 20:06 lustre-log.1408849615.3604
rw------ 1 root root 0 Aug 23 20:07 lustre-log.1408849620.13179
rw------ 1 root root 0 Aug 23 20:07 lustre-log.1408849621.3391
rw------ 1 root root 0 Aug 23 20:07 lustre-log.1408849622.3778
rw------ 1 root root 0 Aug 23 20:07 lustre-log.1408849639.3386
rw------ 1 root root 0 Aug 23 20:07 lustre-log.1408849651.3768
rw------ 1 root root 0 Aug 23 20:08 lustre-log.1408849696.3700
rw------ 1 root root 0 Aug 23 20:11 lustre-log.1408849881.3577
rw------ 1 root root 0 Aug 23 20:16 lustre-log.1408850177.3739
rw------ 1 root root 0 Aug 23 20:33 lustre-log.1408851226.3763
rw------ 1 root root 0 Aug 23 20:33 lustre-log.1408851228.3394
rw------ 1 root root 0 Aug 23 20:34 lustre-log.1408851244.3787
rw------ 1 root root 0 Aug 23 20:43 lustre-log.1408851826.3753
rw------ 1 root root 0 Aug 23 20:43 lustre-log.1408851828.3608
rw------ 1 root root 0 Aug 23 20:48 lustre-log.1408852083.3386
rw------ 1 root root 0 Aug 23 20:53 lustre-log.1408852439.3711
rw------ 1 root root 0 Aug 23 21:04 lustre-log.1408853046.3576
rw------ 1 root root 0 Aug 23 21:04 lustre-log.1408853046.13179
rw------ 1 root root 0 Aug 23 21:04 lustre-log.1408853047.3575
rw------ 1 root root 0 Aug 23 21:12 lustre-log.1408853561.3790
rw------ 1 root root 0 Aug 23 21:13 lustre-log.1408853629.3739
rw------ 1 root root 0 Aug 23 21:13 lustre-log.1408853632.3693
rw------ 1 root root 0 Aug 23 21:14 lustre-log.1408853643.3744
rw------ 1 root root 0 Aug 23 21:23 lustre-log.1408854232.3787
rw------ 1 root root 0 Aug 23 21:24 lustre-log.1408854260.3713
rw------ 1 root root 0 Aug 23 21:28 lustre-log.1408854518.13487
rw------ 1 root root 0 Aug 23 21:33 lustre-log.1408854834.3730
rw------ 1 root root 0 Aug 23 21:34 lustre-log.1408854843.3753
rw------ 1 root root 0 Aug 23 21:34 lustre-log.1408854850.3723
rw------ 1 root root 0 Aug 23 21:43 lustre-log.1408855434.3707
rw------ 1 root root 0 Aug 23 21:44 lustre-log.1408855445.3711
rw------ 1 root root 0 Aug 23 21:51 lustre-log.1408855896.3386
rw------ 1 root root 0 Aug 23 21:52 lustre-log.1408855900.3700
rw------ 1 root root 0 Aug 23 21:54 lustre-log.1408856047.3673
rw------ 1 root root 0 Aug 23 22:05 lustre-log.1408856724.3787
rw------ 1 root root 0 Aug 23 22:05 lustre-log.1408856735.3576
rw------ 1 root root 0 Aug 23 22:05 lustre-log.1408856733.3604
rw------ 1 root root 0 Aug 23 22:17 lustre-log.1408857438.3730
rw------ 1 root root 0 Aug 23 22:27 lustre-log.1408858038.3769
rw------ 1 root root 0 Aug 23 22:29 lustre-log.1408858140.3707
rw------ 1 root root 0 Aug 23 22:38 lustre-log.1408858731.3711
rw------ 1 root root 0 Aug 23 22:39 lustre-log.1408858740.3723
rw------ 1 root root 0 Aug 23 22:49 lustre-log.1408859331.3763
rw------ 1 root root 0 Aug 23 22:50 lustre-log.1408859429.3386
rw------ 1 root root 0 Aug 23 22:50 lustre-log.1408859433.3700
rw------ 1 root root 0 Aug 23 22:50 lustre-log.1408859444.3714
rw------ 1 root root 0 Aug 23 23:23 lustre-log.1408861422.3400
rw------ 1 root root 0 Aug 23 23:23 lustre-log.1408861435.3769
rw------ 1 root root 0 Aug 23 23:30 lustre-log.1408861816.3756
rw------ 1 root root 0 Aug 23 23:30 lustre-log.1408861859.13179
rw------ 1 root root 0 Aug 23 23:31 lustre-log.1408861907.3386
rw------ 1 root root 0 Aug 23 23:33 lustre-log.1408861982.3738
rw------ 1 root root 0 Aug 23 23:41 lustre-log.1408862506.3710
rw------ 1 root root 0 Aug 24 00:15 lustre-log.1408864535.3599
rw------ 1 root root 0 Aug 24 00:41 lustre-log.1408866085.3394
rw------ 1 root root 0 Aug 24 00:42 lustre-log.1408866131.3739
rw------ 1 root root 0 Aug 24 00:42 lustre-log.1408866148.3723
rw------ 1 root root 0 Aug 24 00:54 lustre-log.1408866748.3730
rw------ 1 root root 0 Aug 24 01:07 lustre-log.1408867637.3756
rw------ 1 root root 0 Aug 24 01:17 lustre-log.1408868239.15390
rw------ 1 root root 0 Aug 24 01:17 lustre-log.1408868262.3757
rw------ 1 root root 0 Aug 24 01:24 lustre-log.1408868649.3789
rw------ 1 root root 0 Aug 24 01:44 lustre-log.1408869893.3794
rw------ 1 root root 0 Aug 24 01:46 lustre-log.1408869967.31830
rw------ 1 root root 0 Aug 24 01:46 lustre-log.1408869992.3701
rw------ 1 root root 0 Aug 24 01:46 lustre-log.1408869995.3646
rw------ 1 root root 0 Aug 24 01:46 lustre-log.1408869999.13180
rw------ 1 root root 0 Aug 24 01:46 lustre-log.1408870003.3704
rw------ 1 root root 0 Aug 24 01:54 lustre-log.1408870472.3599
rw------ 1 root root 0 Aug 24 01:54 lustre-log.1408870493.3697
rw------ 1 root root 0 Aug 24 02:14 lustre-log.1408871649.3794
rw------ 1 root root 0 Aug 24 02:20 lustre-log.1408872011.3599
rw------ 1 root root 0 Aug 24 02:20 lustre-log.1408872050.3653
rw------ 1 root root 0 Aug 24 02:21 lustre-log.1408872064.3717
rw------ 1 root root 0 Aug 24 02:21 lustre-log.1408872096.3751
rw------ 1 root root 0 Aug 24 02:42 lustre-log.1408873370.3787
rw------ 1 root root 0 Aug 24 02:52 lustre-log.1408873970.3736
rw------ 1 root root 0 Aug 24 03:02 lustre-log.1408874568.3741
rw------ 1 root root 0 Aug 24 03:03 lustre-log.1408874587.3732
rw------ 1 root root 0 Aug 24 03:44 lustre-log.1408877070.3577
rw------ 1 root root 0 Aug 24 04:56 lustre-log.1408881393.3753
rw------ 1 root root 0 Aug 24 04:56 lustre-log.1408881401.3758
rw------ 1 root root 0 Aug 24 04:56 lustre-log.1408881417.3720
rw------ 1 root root 0 Aug 24 04:57 lustre-log.1408881445.3396
rw------ 1 root root 0 Aug 24 04:57 lustre-log.1408881464.3741
rw------ 1 root root 0 Aug 24 04:57 lustre-log.1408881471.3710
rw------ 1 root root 0 Aug 24 04:57 lustre-log.1408881472.3777
rw------ 1 root root 0 Aug 24 04:57 lustre-log.1408881477.3778
rw------ 1 root root 0 Aug 24 04:58 lustre-log.1408881511.3754
rw------ 1 root root 0 Aug 24 04:58 lustre-log.1408881514.3646
rw------ 1 root root 0 Aug 24 05:15 lustre-log.1408882512.3713
rw------ 1 root root 0 Aug 24 05:30 lustre-log.1408883447.3711
rw------ 1 root root 0 Aug 24 05:30 lustre-log.1408883454.3710
rw------ 1 root root 0 Aug 24 06:14 lustre-log.1408886041.3714
rw------ 1 root root 0 Aug 24 06:18 lustre-log.1408886306.3653
rw------ 1 root root 0 Aug 24 06:35 lustre-log.1408887351.3796
rw------ 1 root root 0 Aug 24 06:39 lustre-log.1408887573.3740
rw------ 1 root root 0 Aug 24 06:39 lustre-log.1408887576.3758
rw------ 1 root root 0 Aug 24 06:39 lustre-log.1408887584.14007
rw------ 1 root root 0 Aug 24 06:39 lustre-log.1408887587.3753
rw------ 1 root root 0 Aug 24 06:39 lustre-log.1408887595.13615
rw------ 1 root root 0 Aug 24 06:39 lustre-log.1408887599.3713
rw------ 1 root root 0 Aug 24 06:52 lustre-log.1408888356.3741
rw------ 1 root root 0 Aug 24 08:01 lustre-log.1408892465.3713
rw------ 1 root root 0 Aug 24 08:06 lustre-log.1408892778.3790
rw------ 1 root root 0 Aug 24 08:26 lustre-log.1408893975.3794
rw------ 1 root root 0 Aug 24 08:29 lustre-log.1408894162.13615
rw------ 1 root root 0 Aug 24 08:36 lustre-log.1408894607.3768
rw------ 1 root root 0 Aug 24 08:57 lustre-log.1408895828.3762
rw------ 1 root root 0 Aug 24 11:15 lustre-log.1408904103.3731
rw------ 1 root root 0 Aug 24 11:15 lustre-log.1408904105.3653
rw------ 1 root root 0 Aug 24 11:15 lustre-log.1408904119.13615
rw------ 1 root root 0 Aug 24 11:16 lustre-log.1408904205.13487
rw------ 1 root root 0 Aug 24 11:16 lustre-log.1408904209.3634
rw------ 1 root root 0 Aug 24 11:16 lustre-log.1408904213.3740
rw------ 1 root root 0 Aug 24 11:17 lustre-log.1408904232.3757
rw------ 1 root root 0 Aug 24 11:28 lustre-log.1408904904.3768
rw------ 1 root root 0 Aug 24 11:48 lustre-log.1408906117.3700
rw------ 1 root root 0 Aug 24 12:44 lustre-log.1408909461.3741
what the timespan of the MDS log? I haven't seen client eviction messages in it.
Hi Zhengyu,
Our client has debug flags set to
lctl > get_param debug
debug=ioctl neterror warning error emerg ha config console
If the client messages log attached in this ticket doesn't have it, how can I get it?
We have the problem occurring continuously, so it won't be too hard for me to get it.
thanks,
Haisong
since it has LU-1128 patch, so the server should have set SLV and bail out the loop, and clients should honor the SLV change and cancel locks hence that MDS release some locks, do you have some clients debug log that I can check whether clients honor the SLV change and cancel locks voluntarily? Thanks.
Hi Zhenyu,
We are running binary RPM from Intel.
[root@dolphin-mds-9-2 ~]# uname -a
Linux dolphin-mds-9-2.local 2.6.32-358.23.2.el6_lustre.x86_64 #1 SMP Thu Dec 19 19:57:45 PST 2013 x86_64 x86_64 x86_64 GNU/Linux
update for what I found till now (haven't figured out the root cause yet)
in the lustre-log.1408400803.3766.3766, at time of 1408400290.481213, mdt thread 3766 is serving a readpage request, and found short of memory, it runs into memory reclaim path, and goes down through to ldlm_pools_shrink(), and in server's pools shrinker, it just adjust its SLV (Server Lock Volume) and does not release memory, so LU-1128 patch makes ldlm_pools_shrink() return -1 so that kernel code will not loop calling shrink_slab().
2.4.2 code should have this patch, but the log shows since 1408400290.481213, thread 3766 did nothing but keep looping ldlm_pools_shrink(), that is unexpected.
Haisong, what is the kernel version you are using?
Clients (2.5.2) are still reporting errors LU_3487 claimed that should have been fixed:
Aug 18 11:32:12 tscc-login1 kernel: LustreError: 55183:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file yout [0x20000518e:0x2f:0x0] error -5.
Aug 18 14:13:17 tscc-login1 kernel: LustreError: 56399:0:(vvp_io.c:1229:vvp_io_init()) dolphin: refresh file yout [0x200005186:0xc:0x0] error -108.
Aug 18 14:13:17 tscc-login1 kernel: LustreError: 56399:0:(vvp_io.c:1229:vvp_io_init()) Skipped 1 previous simar message
SDSC have moved onto more current releases so I do not think any further work is needed here