|
Hello Seb! I have talked with CEA people about this problem today and it seems that, very likely due to memory/cgroup limit (kmem), a Slub allocation failed that may than have caused to trigger a buggy error handling path in Lustre code.
If necessary I can go on-site and analyze the crash-dump to confirm our previous assumptions.
|
|
I have been on-site to analyze crash-dumps for this problem and also to discuss with people about how/when problem occurs and try to understand its root cause.
Finally we have discovered that the SLUB allocation failure, likely to further lead to the LBUG during error processing code path, is due to the memory cgroup limit also taking Kernel memory volume into account on a per thread basis. Thus, the obvious work-around has been to avoid kmem accounting via "cgroup.memory=nokmem" boot parameter.
Concerning the LBUG, I am presently reviewing the different and possibly involved routines/code-paths to understand how/why the expected flag is missing. One of the interesting discovery I presently investigate is that the current cl_page is the 1st one of the 256 for a 1MB read and also that its underlying VM-page is not locked (!PG_locked) like the others.
|
|
Actually, cgroup.memory=nokmem is not the correct workaround as this option was introduced recently in Linux kernel. This is not supported by RHEL7.
However, it still possible to diable kernel memory accounting with cgroup, using a different method, but more difficult to deploy as it requires to patch cgroup code in Slurm.
I will try to post a reproducer which does not require slurm at all.
|
|
Well sorry Aurelien, usually I am late about features but this time I have been too early! I must have been tired because I thought that this was this kind of parameter you had found to disable kmem accounting in cgroup and that we have discussed briefly about...
So thanks for the clarification.
|
|
No problem Bruno. We had a better understanding of these params after you left 
Here is a shell script to help you reproduce the problem.
System is RHEL 7.2.
It is easier to reproduce the problem when reading the file. The file should not be in cache (if it is, there won't be I/O going through osc to OST)
#!/bin/bash
# usage: $0 LUSTREFILE CGROUPDIR
# ie: $0 /path/to/file/in/lustre /sys/fs/cgroup/memory/foo/test
#
# LUSTREFILE should exist and be bigger than 100MB
#
# First, root should create the parent cgroup
# mkdir /sys/fs/cgroup/memory/foo
# chown -R foo /sys/fs/cgroup/memory/foo
#
[[ -n "$1" ]] && OUTPUT_FILE="$1"
[[ -n "$2" ]] && CGROUP_DIR="$2"
[[ ! -d $CGROUP_DIR ]] && mkdir -p $CGROUP_DIR
echo 3G > ${CGROUP_DIR}/memory.kmem.limit_in_bytes
echo $(( $(cat ${CGROUP_DIR}/memory.usage_in_bytes) + 5000000 )) > ${CGROUP_DIR}/memory.limit_in_bytes
echo $$ > ${CGROUP_DIR}/tasks
dd if=${OUTPUT_FILE} of=/dev/null bs=1M
|
|
Cray has also seen this, most often when running SLURM with multiple jobs per node enabled.
|
|
Could you verify memory cgroup is enabled in your SLURM environment? See /etc/slurm/cgroup.conf
SLURM is tuning these files.
https://github.com/SchedMD/slurm/blob/slurm-16.05/src/plugins/task/cgroup/task_cgroup_memory.c#L305
|
|
the cgroup.conf looks like this:
CgroupAutomount=yes
CgroupMountpoint="/dev"
ConstrainCores=yes
ConstrainRAMSpace=yes
TaskAffinity=no
|
|
ConstrainRAMSpace=yes is the line. That's the same issue than us. If you can afford to turn it off, this should work around the crash.
|
|
Well having a better look to the code involved in error path, my guess is that the problem may be caused by the following piece:
the list_del_init() should cause the current oap to have its next/prev oap_pending_item list pointers to point on itself and thus to be again selected by list-for_each_entry() to be passed for a 2nd call to osc_ap_completion() when its processing has already been done during 1st call to osc_ap_completion() for it, and particularly its associated cl_page->cp_state has already been set to CPS_CACHED in osc_ap_completion()>osc_completion()>cl_page_completion(), hence the Assert du to no longer CPS_PAGEIN.
I am working on a patch to fix this and to be pushed soon.
|
|
I agree with this analysis. cp_state is CPS_PAGEIN in almost all our crashdumps. Except for 1 or 2 but this could means that the page was freed between the 2 iterations IMO?
list_for_each_entry + list_del_init seems a really bad idea.
|
|
And use of list_for_each_entry_safe() + list_del() looks like a better one 
This will be my first patch attempt to fix this, I will then try to expose it to your reproducer to see how this works then.
|
|
Well in fact, I just found that this has already been fixed in master by Andreas fix for LU-6215 (http://review.whamcloud.com/13956, commit eaba382df06cd6f570b16c5f759f612b057e5324). Thus I will simply back-port it to b2_7!
|
|
Aurelien Degremont (aurelien.degremont@cea.fr) uploaded a new patch: http://review.whamcloud.com/21745
Subject: LU-8435 tests: slab alloc error does not LBUG
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0e8a912a7dfc65d14d1f252efea043637620f518
|
|
My bad, I miss your last comment . That's the problem working too often with 2.7.2 only, you sometimes miss what was added in master.
I've pushed a reproducer for this in sanity. Do whatever you want with it.
|
|
I'm running a test on this now. Seems good, no crashes ... yet.
|
|
Hello Sebastien,
Patch has been back-ported and has landed on b2_7_fe branch, do you agree to close ?
|
|
Hi Bruno,
Thanks for the backport.
You can close.
Regards.
|
|
I think that we need the ticket to remain open until the original patch has landed to master
|
|
Peter the original fix https://review.whamcloud.com/#/c/13956 has already landed to master. I think this is safe to close. Or do you mean https://review.whamcloud.com/#/c/21745 ?
|
|
Yes I meant the testing patch
|
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/21745/
Subject: LU-8435 tests: slab alloc error does not LBUG
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 15dac618aabf2d5611a280bce13ca79c673f4f6d
|
|
Landed for 2.11
|
|
Hm I just hada failure in a test introduced by this patch:
I have a crashdump if anybody is interested.
|
|
Oleg,
my guess is that this new sub-test sanity/test_411, introduced by change #21745, is setting a highly constraining Kernel memory limit that is very likely to trigger some memcg/slab bug.
But I am ok to have a look to the crash dump to try to confirm.
|
|
Ok, thanks.
I had 4 more failures in the past 24 hours, btw.
The crashdumps are on onyx-68 in /export/crashdumps.
they are:
192.168.123.199-2017-09-01-10:34:*
192.168.123.111-2017-09-02-15:06:*
192.168.123.195-2017-09-03-13:*
192.168.123.151-2017-09-03-14:06:*
192.168.123.135-2017-09-03-14:11:*
build tree is currently in /export/centos7-nfsroot/home/green/git/lustre-release with all the modules (I'll update it on Tuesday ,but should be good for the next 30 or so hours).
|
|
Bruno, this was exactly the purpose of this test. It seems it discover other memory management issues in client code. I/O is not really expected to succeed under such constraints, but only returing EIO or ENOMEM, not crashing
|
|
On that note, Aurelien, I think we should add a write component to the test after the memory limit is set... Or perhaps a separate test. But either way - write under pressure would be good to have as well.
|
|
Aurelien, I know that you posted this new test to ensure memcg limit do not cause crashes in Lustre code, but with this new kind of crash in Kernel/memcg layer, it seems you should also propose it for the kernel regression tests suite!!
My first crash-dumps analysis results poins to a possible race during memcg lazy registration to current kmem_caches and concurrent Slab allocations, that triggers the unexpected situation, in __memcg_kmem_get_cache(), where memcg_params has still not been initialized in "ptlrpc_cache" kmem_cache.
It is also interesting to note that recent auto-tests results of sanity/tests_411 are all success, when all these crashes have occurred during single-node sessions, and that the only kmem_cache in the system that do not have memcg_params initialized are those that have been created in Lustre code.
More to come.
|
|
noet that I am running unsupported redhat kernel config with a lot of debugging and I also enable SLAB, where as redhat defaults to SLUB.
This does not make the bugs any less real of course, but at times it takes some time to convince the RH people that it's ok to have unsupported config because it just helps to detect the bugs earlier vs random crashes elsewhere later.
|
|
Well, I have been quite puzzled with these crash-dumps, because according to the Kernel source code, the only case where memcg_params could be NULL is when a kmem_cache is being created and !memcg_kmem_enabled() (tested in memcg_register_cache(), this is when no memory group with kmem limit is still active), but then all kmem_cache should have their memcg_params==NULL, and not only the Lustre ones!! I have tested this on a fresh system after reboot with no memory cgroup configured, all kmem_cache, and even the Lustre ones after a delayed Lustre start and modules load, have their memcg_params==NULL.
Finally, I found a possible scenario where this can happen, and this is confirmed by/in all Kernel logs from the crash-dumps, the sanity/test_411 test (likely to be the only one to create+use a memory group), has to be run at least 2 times with Lustre modules unload in between (causing Lustre kmem_cache to be destroyed and re-created).
Then, trying to reproduce by doing so on a test system I was not able to make it since !memcg_kmem_enabled() could never be satisfied because of "child/memcg" kmem_cache (created during 1st sub-test run and memory group creation) still being present, and this even being fully empty/freed. But this could be where the SLAB vs SLUB config should have made the difference, because it appears that SLAB mechanism include a regular kmem_cache shrink/drain cleanup that could have caused these kind of leaked kmem_cache to be garbage collected, when this never (or may be a lot later) could happen running with SLUB.
But anyway, the fact that memcg_params could be NULL for a recently created kmem_cache when no kmem memory cgroup is still active (but some had already existed before!), needs to be handled by current code. I will check if this is already the case in recent Kernels, and push a fix if not.
Oleg, this looks like a new argument to confirm the need to test Kernels with "unsupported" configs!
|
|
This problem should have already been addressed by patch at https://patchwork.kernel.org/patch/2850152/.
But since the RHEL7 Kernel version we have experienced these crashes with already includes it, after digging into the Kernel related source code, I think the following one is required to fully fix this issue :
Oleg, you may want to try it ? And on my side I will check if this problem is still present in recent Kernels and report it if not.
|
|
After browsing Kernel sources in kernel.org, I think this issue is fixed in 4.x and 3.x more recent Kernels, mainly with changes that have been introduced in patch/commit "memcg: don't call memcg_update_all_caches if new cache id fits" available at
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/patch/mm/memcontrol.c?id=f3bb3043a092368a255bca5d1c6f4352c96a3b2d, and where ALL kmem_cache are systematically parsed to re-size/initialize a new memcg_params when a kmem mermory-cgroup is being started.
And again, I need to insist that this bug is very unlikely to be triggered if we keep running with SLUB configured instead to switch back to SLAB.
|
|
Hmm, in fact I think the kernel patch ("memcg: don't call memcg_update_all_caches if new cache id fits") that I have indicated in my just previous comment is not likely to fix finally.
After I have browsed again more recent Kernels source code, problem appears to be fixed in 4.x kernels because now memcg_params is no longer a pointer to but an embedded struct memcg_cache_params inside the kmem_cache! But looks like problem is still in 3.x.
BTW, I have found a possible other/alternate way to fix it with the following patch :
where memcg_register_cache() routine has been changed by memcg_alloc_cache_params() in some 3.x versions but where same fix applies.
I will try to propose either patch to the Kernel mailing list.
|
|
Problem description and 2x patch proposals have been posted/emailed to cgroups@vger.kernel.org.
|
|
Hmm, still not a lot of answer to my report at http://www.spinics.net/lists/cgroups/msg18187.html ....
So, I have submitted #197075 bug report to bugzilla.kernel.org, https://bugzilla.kernel.org/show_bug.cgi?id=197075.
|
|
And here is the answer I got for BZ #197075 !!... :
|
|
I think it makes sense to disable cgroup testing if client is running version < 4.4, until we know there is a fix in RHEL and what version it is in.
|
Generated at Sat Feb 10 02:17:30 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
crash_output.txt