|
can you try patch https://review.whamcloud.com/#/c/36229/
|
can you try patch https://review.whamcloud.com/#/c/36229/
tested this patch. although it's still problem, but it was another call trace at __write_lock_failed after patch.
|
|
Shilong
Could you please advise
Thanks
Peter
|
|
it enabled kernel.softlockup_panic and we got crash and crash dump when softlockup happens on client.
|
|
Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/36413
Subject: LU-12832 osc: invalidate kms upon blocking ast
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 92f87e3a4b25392930e3b6df9b5c6a46eb6191ff
|
|
Just re-think the problem, IOR seems try to write file firstly stridely and in this case we might have every
client full with many PW extents, more than 60K each client.
And before Read starts, lock cancel might happen which will try to cancel tons of cache? which might make our spinlock very hot.
|
|
So to walk around the problem, maybe use a smaller LRU SIZE shall be fine.
But we might need add some debug information to extent tree locks to understand details etc.
|
|
yes, fixed lru_size or short age for lru cache could be workaround, but the other hand, some application prefer keeping locks in caches as much as possible. For just an benchmark with ior for strided write/read is fine, but we need a sophisticate logic for both workload reasonably works properly 
|
|
Wang Shilong (wshilong@ddn.com) uploaded a new patch: https://review.whamcloud.com/36419
Subject: LU-12832 ldlm: debug extent tree
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ec320272aa8d3f3b6353aff2f556b9641f9245a9
|
|
dongyang found following problem might fix the problem:
The point is It is very possible that arg->old_kms = last extent end. and if we fail last one check,
we will have to iterate all PR extent locks in tree which is O(N).
|
|
It definitely seems possible that the above change could fix the problem, as the comment before ldlm_extent_shift_kms() says:
.
That said, the above-referenced code in ldlm_kms_shift_cb() hasn't changed since patch https://review.whamcloud.com/20779 "LU-8272 ldlm: Use interval tree to update kms" in 2016-10-17, so it shouldn't be the immediate cause of a regression since the last time this code was working, so it is at least going to be a bad interaction with a patch that landed more recently. I'm not saying the above code is not the source of the problem, just that it would be good to know why this problem was suddenly being hit. It doesn't look like it is directly related to lock conversion, since that patch landed as v2_11_52_0-78-g541902a3f9 in 2018-07-03 before the 2.12.0 release and we haven't been seeing this problem very long.
It looks like there are 3 patches to ldlm/ldlm_extent.c in the 2.13 development cycle:
https://review.whamcloud.com/35057 2019-08-27 "LU-12017 ldlm: DoM truncate deadlock"
https://review.whamcloud.com/35486 2019-07-20 "LU-12542 ldlm: simplify lock_mode_to_index()"
https://review.whamcloud.com/19665 2019-03-21 "LU-8047 llite: optimizations for not granted lock processing"
There are also a few other patches in 2.13 that are related to KMS handling:
https://review.whamcloud.com/36200 2019-09-30 "LU-12681 osc: wrong cache of LVB attrs, part2"
https://review.whamcloud.com/36199 2019-09-30 "LU-12681 osc: wrong cache of LVB attrs"
https://review.whamcloud.com/34095 2019-06-25 "LU-12043 llite: improve single-thread read performance"
which would be a good place to start to see if reverting those patches, or testing the commit immediately before each patch avoids this problem. If we know the first time this problem was seen it might make it easy to track down the problematic patch.
If there are problems tracking down the source of this bug then it would be possible to use "git bisect" at the cost of running more tests. There are currently 1008 patches from 2.12.50..fcf219d so if the problem is reliably hit during testing then it would be possible to find the problem in 10 test runs or fewer. Since it is likely to be a client-side problem then it (probably) wouldn't be necessary to update the server version each time if that makes it faster for each test run.
|
|
Yup, it is intresting to know why we did not hit the problem before, this could potentially related to LRU SIZE? or testing file size?
or some locks changes regression raised the problem.
|
|
I've tested above change with same workload, but it caused another regression at write (not read). Please see console messages whne client crashed.
|
|
I wonder if overstriping could be related. I have been enabling overstriping for this testing, but will test without overstriping in master. If the problem is still exist, we can test old commit. It takes too long time to finish this ior job without overstriping though...
|
|
background aggressive lock canceling in advance helps to avoid this problem.
e.g. setting 30sec to lru_max_age 'lctl set_param ldlm.namespaces.*.lru_max_age=30000'
After this change, I didn't see see problem.
However, now I'm reproducing LU-12786 again.. I will re-open LU-12786.
|
|
seems to be hw issue
|
Generated at Sat Feb 10 02:56:02 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
vmcore-dmesg.txt