Details
-
Bug
-
Resolution: Fixed
-
Major
-
Lustre 2.4.0
-
3
-
5689
Description
While running an IOR on Vulcan I see a single Lustre client (IO node) with the following messages in its log:
2012-11-27 14:42:32.932604 {DefaultControlEventListener} [mmcs]{52}.5.3: cannot allocate a tage (0)
2012-11-27 14:42:32.933545 {DefaultControlEventListener} [mmcs]{52}.5.3: Lustre: 3793:0:(file.c:2880:ll_layout_refresh()) maximum lustre stack 12368
2012-11-27 14:42:32.934066 {DefaultControlEventListener} [mmcs]{52}.5.3: Call Trace:
2012-11-27 14:42:32.934568 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e10f00] [c000000000008160] .show_stack+0x7c/0x184 (unreliable)
2012-11-27 14:42:32.934997 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e10fb0] [800000000694e518] .ll_layout_refresh+0x1418/0x1cb0 [lustre]
2012-11-27 14:42:32.935436 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11180] [80000000069cde7c] .vvp_io_init+0x25c/0x370 [lustre]
2012-11-27 14:42:32.935941 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11240] [800000000249b4a4] .cl_io_init0+0x104/0x260 [obdclass]
2012-11-27 14:42:32.936453 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e112f0] [80000000045f9e50] .osc_lru_shrink+0x540/0x1390 [osc]
2012-11-27 14:42:32.936972 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11490] [80000000045fb14c] .osc_lru_del+0x3cc/0x760 [osc]
2012-11-27 14:42:32.937522 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e115c0] [80000000045fcb30] .osc_page_delete+0x150/0x550 [osc]
2012-11-27 14:42:32.938101 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11690] [800000000248a540] .cl_page_delete0+0x150/0x820 [obdclass]
2012-11-27 14:42:32.938606 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11770] [800000000248ac94] .cl_page_delete+0x84/0x250 [obdclass]
2012-11-27 14:42:32.939292 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11820] [80000000069b17a0] .ll_releasepage+0x170/0x1b0 [lustre]
2012-11-27 14:42:32.939924 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e118d0] [c0000000000932dc] .try_to_release_page+0x68/0x8c
2012-11-27 14:42:32.940426 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11940] [c0000000000a3440] .shrink_page_list.clone.0+0x3b0/0x614
2012-11-27 14:42:32.940985 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11af0] [c0000000000a3ab0] .shrink_inactive_list+0x40c/0x71c
2012-11-27 14:42:32.941743 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11cd0] [c0000000000a42d0] .shrink_zone+0x510/0x514
2012-11-27 14:42:32.942242 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11e00] [c0000000000a4dc0] .try_to_free_pages+0x204/0x3d0
2012-11-27 14:42:32.942770 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11f60] [c00000000009c184] .__alloc_pages_nodemask+0x42c/0x6ec
2012-11-27 14:42:32.943289 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e120e0] [c0000000000c917c] .__slab_alloc+0x1f4/0x68c
2012-11-27 14:42:32.943707 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e121b0] [c0000000000ca1c8] .__kmalloc+0xd4/0x150
2012-11-27 14:42:32.944275 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12260] [8000000000aa2504] .cfs_alloc+0x34/0x80 [libcfs]
2012-11-27 14:42:32.944761 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e122f0] [8000000003a31fdc] .ptlrpc_request_alloc_internal+0x12c/0x550 [ptlrpc]
2012-11-27 14:42:32.945386 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e123d0] [80000000045d5744] .osc_brw_prep_request+0x314/0x1e00 [osc]
2012-11-27 14:42:32.945919 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12590] [80000000045e9024] .osc_build_rpc+0x964/0x1f60 [osc]
2012-11-27 14:42:32.946397 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12760] [80000000046148dc] .osc_check_rpcs+0x8bc/0x1320 [osc]
2012-11-27 14:42:32.946851 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e128e0] [8000000004615770] .osc_io_unplug0+0x430/0x6a0 [osc]
2012-11-27 14:42:32.947423 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12a00] [8000000004617ddc] .osc_enter_cache+0x79c/0x12c0 [osc]
2012-11-27 14:42:32.947856 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12bd0] [800000000461fb70] .osc_queue_async_io+0xd10/0x1a40 [osc]
2012-11-27 14:42:32.948252 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12df0] [80000000045fbb88] .osc_page_cache_add+0xf8/0x2a0 [osc]
2012-11-27 14:42:32.948798 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12eb0] [8000000002486a58] .cl_page_cache_add+0xe8/0x3b0 [obdclass]
2012-11-27 14:42:32.949181 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12fe0] [800000000516c438] .lov_page_cache_add+0xc8/0x340 [lov]
2012-11-27 14:42:32.949654 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e130b0] [8000000002486a58] .cl_page_cache_add+0xe8/0x3b0 [obdclass]
2012-11-27 14:42:32.950164 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e131e0] [80000000069ccf94] .vvp_io_commit_write+0x474/0x8a0 [lustre]
2012-11-27 14:42:32.950677 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13300] [80000000024a0aec] .cl_io_commit_write+0x11c/0x2d0 [obdclass]
2012-11-27 14:42:32.951110 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e133c0] [800000000698f6f0] .ll_commit_write+0x120/0x3e0 [lustre]
2012-11-27 14:42:32.951619 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13490] [80000000069b1814] .ll_write_end+0x34/0x80 [lustre]
2012-11-27 14:42:32.952104 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13520] [c0000000000959a8] .generic_file_buffered_write+0x1e4/0x354
2012-11-27 14:42:32.952593 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13660] [c000000000096074] .__generic_file_aio_write+0x374/0x3d8
2012-11-27 14:42:32.953076 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13760] [c000000000096150] .generic_file_aio_write+0x78/0xe8
2012-11-27 14:42:32.953588 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13810] [80000000069cfd9c] .vvp_io_write_start+0xfc/0x3e0 [lustre]
2012-11-27 14:42:32.954050 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e138e0] [800000000249b03c] .cl_io_start+0xcc/0x220 [obdclass]
2012-11-27 14:42:32.954590 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13980] [80000000024a2e54] .cl_io_loop+0x194/0x2c0 [obdclass]
2012-11-27 14:42:32.955062 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13a30] [800000000694a158] .ll_file_io_generic+0x498/0x670 [lustre]
2012-11-27 14:42:32.955499 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13b30] [800000000694a7b4] .ll_file_aio_write+0x1d4/0x3a0 [lustre]
2012-11-27 14:42:32.955995 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13c00] [800000000694aad0] .ll_file_write+0x150/0x320 [lustre]
2012-11-27 14:42:32.956478 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13ce0] [c0000000000d1e9c] .vfs_write+0xd0/0x1c4
2012-11-27 14:42:32.956965 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13d80] [c0000000000d208c] .SyS_write+0x54/0x98
2012-11-27 14:42:32.957488 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13e30] [c000000000000580] syscall_exit+0x0/0x2c
2012-11-27 14:42:32.957878 {DefaultControlEventListener} [mmcs]{52}.5.3: cannot allocate a tage (0)
2012-11-27 14:42:32.958298 {DefaultControlEventListener} [mmcs]{52}.5.3: Lustre: 3793:0:(lov_io.c:252:lov_page_stripe()) maximum lustre stack 12464
2012-11-27 14:42:32.958849 {DefaultControlEventListener} [mmcs]{52}.5.3: Call Trace:
2012-11-27 14:42:32.959296 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e10ea0] [c000000000008160] .show_stack+0x7c/0x184 (unreliable)
2012-11-27 14:42:32.959794 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e10f50] [80000000051746dc] .lov_page_stripe+0x20c/0x220 [lov]
2012-11-27 14:42:32.960305 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e10ff0] [8000000005178d18] .lov_page_subio+0xf8/0x2d0 [lov]
2012-11-27 14:42:32.960799 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e110a0] [800000000516c790] .lov_page_own+0xe0/0x250 [lov]
2012-11-27 14:42:32.961312 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11150] [800000000248b928] .cl_page_own0+0x188/0x540 [obdclass]
2012-11-27 14:42:32.961789 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11230] [80000000045f9834] .discard_pagevec+0x74/0x150 [osc]
2012-11-27 14:42:32.962278 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e112f0] [80000000045f9e04] .osc_lru_shrink+0x4f4/0x1390 [osc]
2012-11-27 14:42:32.962715 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11490] [80000000045fb14c] .osc_lru_del+0x3cc/0x760 [osc]
2012-11-27 14:42:32.963263 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e115c0] [80000000045fcb30] .osc_page_delete+0x150/0x550 [osc]
2012-11-27 14:42:32.963630 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11690] [800000000248a540] .cl_page_delete0+0x150/0x820 [obdclass]
2012-11-27 14:42:32.964208 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11770] [800000000248ac94] .cl_page_delete+0x84/0x250 [obdclass]
2012-11-27 14:42:32.964559 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11820] [80000000069b17a0] .ll_releasepage+0x170/0x1b0 [lustre]
2012-11-27 14:42:32.965037 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e118d0] [c0000000000932dc] .try_to_release_page+0x68/0x8c
2012-11-27 14:42:32.965517 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11940] [c0000000000a3440] .shrink_page_list.clone.0+0x3b0/0x614
2012-11-27 14:42:32.965997 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11af0] [c0000000000a3ab0] .shrink_inactive_list+0x40c/0x71c
2012-11-27 14:42:32.966558 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11cd0] [c0000000000a42d0] .shrink_zone+0x510/0x514
2012-11-27 14:42:32.975334 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11e00] [c0000000000a4dc0] .try_to_free_pages+0x204/0x3d0
2012-11-27 14:42:32.975849 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11f60] [c00000000009c184] .__alloc_pages_nodemask+0x42c/0x6ec
2012-11-27 14:42:32.976254 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e120e0] [c0000000000c917c] .__slab_alloc+0x1f4/0x68c
2012-11-27 14:42:32.976720 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e121b0] [c0000000000ca1c8] .__kmalloc+0xd4/0x150
2012-11-27 14:42:32.977232 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12260] [8000000000aa2504] .cfs_alloc+0x34/0x80 [libcfs]
2012-11-27 14:42:32.977747 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e122f0] [8000000003a31fdc] .ptlrpc_request_alloc_internal+0x12c/0x550 [ptlrpc]
2012-11-27 14:42:32.978173 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e123d0] [80000000045d5744] .osc_brw_prep_request+0x314/0x1e00 [osc]
2012-11-27 14:42:32.978690 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12590] [80000000045e9024] .osc_build_rpc+0x964/0x1f60 [osc]
2012-11-27 14:42:32.979247 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12760] [80000000046148dc] .osc_check_rpcs+0x8bc/0x1320 [osc]
2012-11-27 14:42:32.979743 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e128e0] [8000000004615770] .osc_io_unplug0+0x430/0x6a0 [osc]
2012-11-27 14:42:32.980314 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12a00] [8000000004617ddc] .osc_enter_cache+0x79c/0x12c0 [osc]
2012-11-27 14:42:32.980838 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12bd0] [800000000461fb70] .osc_queue_async_io+0xd10/0x1a40 [osc]
2012-11-27 14:42:32.981467 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12df0] [80000000045fbb88] .osc_page_cache_add+0xf8/0x2a0 [osc]
2012-11-27 14:42:32.981970 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12eb0] [8000000002486a58] .cl_page_cache_add+0xe8/0x3b0 [obdclass]
2012-11-27 14:42:32.982461 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12fe0] [800000000516c438] .lov_page_cache_add+0xc8/0x340 [lov]
2012-11-27 14:42:32.982983 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e130b0] [8000000002486a58] .cl_page_cache_add+0xe8/0x3b0 [obdclass]
2012-11-27 14:42:32.983542 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e131e0] [80000000069ccf94] .vvp_io_commit_write+0x474/0x8a0 [lustre]
2012-11-27 14:42:32.984029 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13300] [80000000024a0aec] .cl_io_commit_write+0x11c/0x2d0 [obdclass]
2012-11-27 14:42:32.984556 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e133c0] [800000000698f6f0] .ll_commit_write+0x120/0x3e0 [lustre]
2012-11-27 14:42:32.985023 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13490] [80000000069b1814] .ll_write_end+0x34/0x80 [lustre]
2012-11-27 14:42:32.985512 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13520] [c0000000000959a8] .generic_file_buffered_write+0x1e4/0x354
2012-11-27 14:42:32.985926 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13660] [c000000000096074] .__generic_file_aio_write+0x374/0x3d8
2012-11-27 14:42:32.986408 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13760] [c000000000096150] .generic_file_aio_write+0x78/0xe8
2012-11-27 14:42:32.988610 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13810] [80000000069cfd9c] .vvp_io_write_start+0xfc/0x3e0 [lustre]
2012-11-27 14:42:32.989163 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e138e0] [800000000249b03c] .cl_io_start+0xcc/0x220 [obdclass]
2012-11-27 14:42:32.989645 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13980] [80000000024a2e54] .cl_io_loop+0x194/0x2c0 [obdclass]
2012-11-27 14:42:32.990172 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13a30] [800000000694a158] .ll_file_io_generic+0x498/0x670 [lustre]
2012-11-27 14:42:32.990667 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13b30] [800000000694a7b4] .ll_file_aio_write+0x1d4/0x3a0 [lustre]
2012-11-27 14:42:32.991176 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13c00] [800000000694aad0] .ll_file_write+0x150/0x320 [lustre]
2012-11-27 14:42:32.991656 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13ce0] [c0000000000d1e9c] .vfs_write+0xd0/0x1c4
2012-11-27 14:42:32.992185 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13d80] [c0000000000d208c] .SyS_write+0x54/0x98
2012-11-27 14:42:32.992682 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13e30] [c000000000000580] syscall_exit+0x0/0x2c
2012-11-27 14:42:32.993218 {DefaultControlEventListener} [mmcs]{52}.5.3: cannot allocate a tage (0)
2012-11-27 14:42:32.993684 {DefaultControlEventListener} [mmcs]{52}.5.3: Lustre: 3793:0:(cl_object.c:851:cl_env_peek()) maximum lustre stack 12752
2012-11-27 14:42:32.994215 {DefaultControlEventListener} [mmcs]{52}.5.3: Call Trace:
2012-11-27 14:42:32.994799 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e10d80] [c000000000008160] .show_stack+0x7c/0x184 (unreliable)
2012-11-27 14:42:32.995344 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e10e30] [8000000002481e88] .cl_env_peek+0x198/0x210 [obdclass]
2012-11-27 14:42:32.995879 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e10ee0] [8000000002483dfc] .cl_env_get+0x4c/0x4a0 [obdclass]
2012-11-27 14:42:32.996364 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e10fb0] [80000000051768a4] .lov_sub_get+0x3f4/0x900 [lov]
2012-11-27 14:42:32.996906 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e110a0] [800000000516c790] .lov_page_own+0xe0/0x250 [lov]
2012-11-27 14:42:32.997421 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11150] [800000000248b928] .cl_page_own0+0x188/0x540 [obdclass]
2012-11-27 14:42:32.997999 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11230] [80000000045f9834] .discard_pagevec+0x74/0x150 [osc]
2012-11-27 14:42:32.998561 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e112f0] [80000000045f9e04] .osc_lru_shrink+0x4f4/0x1390 [osc]
2012-11-27 14:42:32.999146 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11490] [80000000045fb14c] .osc_lru_del+0x3cc/0x760 [osc]
2012-11-27 14:42:32.999645 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e115c0] [80000000045fcb30] .osc_page_delete+0x150/0x550 [osc]
2012-11-27 14:42:33.000181 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11690] [800000000248a540] .cl_page_delete0+0x150/0x820 [obdclass]
2012-11-27 14:42:33.000696 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11770] [800000000248ac94] .cl_page_delete+0x84/0x250 [obdclass]
2012-11-27 14:42:33.001237 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11820] [80000000069b17a0] .ll_releasepage+0x170/0x1b0 [lustre]
2012-11-27 14:42:33.001759 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e118d0] [c0000000000932dc] .try_to_release_page+0x68/0x8c
2012-11-27 14:42:33.002342 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11940] [c0000000000a3440] .shrink_page_list.clone.0+0x3b0/0x614
2012-11-27 14:42:33.002886 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11af0] [c0000000000a3ab0] .shrink_inactive_list+0x40c/0x71c
2012-11-27 14:42:33.003416 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11cd0] [c0000000000a42d0] .shrink_zone+0x510/0x514
2012-11-27 14:42:33.003967 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11e00] [c0000000000a4dc0] .try_to_free_pages+0x204/0x3d0
2012-11-27 14:42:33.004521 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11f60] [c00000000009c184] .__alloc_pages_nodemask+0x42c/0x6ec
2012-11-27 14:42:33.005093 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e120e0] [c0000000000c917c] .__slab_alloc+0x1f4/0x68c
2012-11-27 14:42:33.005653 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e121b0] [c0000000000ca1c8] .__kmalloc+0xd4/0x150
2012-11-27 14:42:33.006904 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12260] [8000000000aa2504] .cfs_alloc+0x34/0x80 [libcfs]
2012-11-27 14:42:33.007480 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e122f0] [8000000003a31fdc] .ptlrpc_request_alloc_internal+0x12c/0x550 [ptlrpc]
2012-11-27 14:42:33.008028 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e123d0] [80000000045d5744] .osc_brw_prep_request+0x314/0x1e00 [osc]
2012-11-27 14:42:33.008534 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12590] [80000000045e9024] .osc_build_rpc+0x964/0x1f60 [osc]
2012-11-27 14:42:33.009027 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12760] [80000000046148dc] .osc_check_rpcs+0x8bc/0x1320 [osc]
2012-11-27 14:42:33.009560 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e128e0] [8000000004615770] .osc_io_unplug0+0x430/0x6a0 [osc]
2012-11-27 14:42:33.010126 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12a00] [8000000004617ddc] .osc_enter_cache+0x79c/0x12c0 [osc]
2012-11-27 14:42:33.010528 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12bd0] [800000000461fb70] .osc_queue_async_io+0xd10/0x1a40 [osc]
2012-11-27 14:42:33.010983 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12df0] [80000000045fbb88] .osc_page_cache_add+0xf8/0x2a0 [osc]
2012-11-27 14:42:33.011568 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12eb0] [8000000002486a58] .cl_page_cache_add+0xe8/0x3b0 [obdclass]
2012-11-27 14:42:33.012988 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12fe0] [800000000516c438] .lov_page_cache_add+0xc8/0x340 [lov]
2012-11-27 14:42:33.014346 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e130b0] [8000000002486a58] .cl_page_cache_add+0xe8/0x3b0 [obdclass]
2012-11-27 14:42:33.020612 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e131e0] [80000000069ccf94] .vvp_io_commit_write+0x474/0x8a0 [lustre]
2012-11-27 14:42:33.021195 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13300] [80000000024a0aec] .cl_io_commit_write+0x11c/0x2d0 [obdclass]
2012-11-27 14:42:33.021701 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e133c0] [800000000698f6f0] .ll_commit_write+0x120/0x3e0 [lustre]
2012-11-27 14:42:33.022264 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13490] [80000000069b1814] .ll_write_end+0x34/0x80 [lustre]
2012-11-27 14:42:33.022790 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13520] [c0000000000959a8] .generic_file_buffered_write+0x1e4/0x354
2012-11-27 14:42:33.023391 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13660] [c000000000096074] .__generic_file_aio_write+0x374/0x3d8
2012-11-27 14:42:33.023947 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13760] [c000000000096150] .generic_file_aio_write+0x78/0xe8
2012-11-27 14:42:33.024507 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13810] [80000000069cfd9c] .vvp_io_write_start+0xfc/0x3e0 [lustre]
2012-11-27 14:42:33.025047 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e138e0] [800000000249b03c] .cl_io_start+0xcc/0x220 [obdclass]
2012-11-27 14:42:33.025546 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13980] [80000000024a2e54] .cl_io_loop+0x194/0x2c0 [obdclass]
2012-11-27 14:42:33.026049 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13a30] [800000000694a158] .ll_file_io_generic+0x498/0x670 [lustre]
2012-11-27 14:42:33.026496 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13b30] [800000000694a7b4] .ll_file_aio_write+0x1d4/0x3a0 [lustre]
2012-11-27 14:42:33.026949 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13c00] [800000000694aad0] .ll_file_write+0x150/0x320 [lustre]
2012-11-27 14:42:33.027401 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13ce0] [c0000000000d1e9c] .vfs_write+0xd0/0x1c4
2012-11-27 14:42:33.027830 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13d80] [c0000000000d208c] .SyS_write+0x54/0x98
2012-11-27 14:42:33.028349 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e13e30] [c000000000000580] syscall_exit+0x0/0x2c
2012-11-27 14:42:33.028832 {DefaultControlEventListener} [mmcs]{52}.5.3: cannot allocate a tage (0)
2012-11-27 14:42:33.029354 {DefaultControlEventListener} [mmcs]{52}.5.3: Lustre: 3793:0:(cl_io.c:1073:cl_io_slice_add()) maximum lustre stack 12896
2012-11-27 14:42:33.029844 {DefaultControlEventListener} [mmcs]{52}.5.3: Call Trace:
2012-11-27 14:42:33.030339 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e10cf0] [c000000000008160] .show_stack+0x7c/0x184 (unreliable)
2012-11-27 14:42:33.030866 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e10da0] [800000000249a5b8] .cl_io_slice_add+0x288/0x2d0 [obdclass]
2012-11-27 14:42:33.031344 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e10e70] [8000000004603e2c] .osc_io_init+0x6c/0xd0 [osc]
2012-11-27 14:42:33.031863 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e10f00] [800000000249b4a4] .cl_io_init0+0x104/0x260 [obdclass]
2012-11-27 14:42:33.032387 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e10fb0] [8000000005176774] .lov_sub_get+0x2c4/0x900 [lov]
2012-11-27 14:42:33.032932 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e110a0] [800000000516c790] .lov_page_own+0xe0/0x250 [lov]
2012-11-27 14:42:33.033462 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11150] [800000000248b928] .cl_page_own0+0x188/0x540 [obdclass]
2012-11-27 14:42:33.034017 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11230] [80000000045f9834] .discard_pagevec+0x74/0x150 [osc]
2012-11-27 14:42:33.034545 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e112f0] [80000000045f9e04] .osc_lru_shrink+0x4f4/0x1390 [osc]
2012-11-27 14:42:33.035124 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11490] [80000000045fb14c] .osc_lru_del+0x3cc/0x760 [osc]
2012-11-27 14:42:33.035644 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e115c0] [80000000045fcb30] .osc_page_delete+0x150/0x550 [osc]
2012-11-27 14:42:33.036173 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11690] [800000000248a540] .cl_page_delete0+0x150/0x820 [obdclass]
2012-11-27 14:42:33.036578 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11770] [800000000248ac94] .cl_page_delete+0x84/0x250 [obdclass]
2012-11-27 14:42:33.036978 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11820] [80000000069b17a0] .ll_releasepage+0x170/0x1b0 [lustre]
2012-11-27 14:42:33.037382 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e118d0] [c0000000000932dc] .try_to_release_page+0x68/0x8c
2012-11-27 14:42:33.037797 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11940] [c0000000000a3440] .shrink_page_list.clone.0+0x3b0/0x614
2012-11-27 14:42:33.038197 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11af0] [c0000000000a3ab0] .shrink_inactive_list+0x40c/0x71c
2012-11-27 14:42:33.038699 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11cd0] [c0000000000a42d0] .shrink_zone+0x510/0x514
2012-11-27 14:42:33.039159 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11e00] [c0000000000a4dc0] .try_to_free_pages+0x204/0x3d0
2012-11-27 14:42:33.039656 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e11f60] [c00000000009c184] .__alloc_pages_nodemask+0x42c/0x6ec
2012-11-27 14:42:33.040128 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e120e0] [c0000000000c917c] .__slab_alloc+0x1f4/0x68c
2012-11-27 14:42:33.040638 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e121b0] [c0000000000ca1c8] .__kmalloc+0xd4/0x150
2012-11-27 14:42:33.041872 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12260] [8000000000aa2504] .cfs_alloc+0x34/0x80 [libcfs]
2012-11-27 14:42:33.042817 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e122f0] [8000000003a31fdc] .ptlrpc_request_alloc_internal+0x12c/0x550 [ptlrpc]
2012-11-27 14:42:33.043583 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e123d0] [80000000045d5744] .osc_brw_prep_request+0x314/0x1e00 [osc]
2012-11-27 14:42:33.044089 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12590] [80000000045e9024] .osc_build_rpc+0x964/0x1f60 [osc]
2012-11-27 14:42:33.044762 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12760] [80000000046148dc] .osc_check_rpcs+0x8bc/0x1320 [osc]
2012-11-27 14:42:33.045312 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e128e0] [8000000004615770] .osc_io_unplug0+0x430/0x6a0 [osc]
2012-11-27 14:42:33.045854 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12a00] [8000000004617ddc] .osc_enter_cache+0x79c/0x12c0 [osc]
2012-11-27 14:42:33.046343 {DefaultControlEventListener} [mmcs]{52}.5.3: [c0000003e4e12bd0] [800000000461fb70] .osc_queue_async_io+0xd10/0x1a40 [osc]
These all look troubling as we've run into stack overrun issues in the past.