|
Mike
Could you please investigate?
Peter
|
|
Sarah, it would be useful to know which was the last 2.14.5x version where this test was still passing? Have you restarted soak with the current build to see if this is repeatable, or if it was a one-time failure?
According to https://wiki.whamcloud.com/display/Releases/Soak+Testing+on+Soak it looks like 2.14.56 was run on 20211223, and 2.15.0-RC2-2-g94f4e1f was run on 20220302. Did those builds work without error?
The number of patches that landed since 2.15.0-RC2-2-g94f4e1f is quite small (excluding test patches):
|
|
Sarah, I didn't notice that you have already hit this problem twice. Could you please test with patch https://review.whamcloud.com/46837 "LU-15645 obdclass: llog to handle gaps" (already based on 2.15.0-RC3) to see if this solves the recovery problem?
|
|
Hi Andreas, I didn't notice this problem on the previous master tags, I will reload soak tomorrow to try the patch, thank you.
|
|
Hi sarah
Any luck reproducing?
-cf
|
|
Hi Colin,
soak doesn't hit the same issue, instead it has another issue on the other pair of MDSs(soak-8 and soak-9) during failback.
on soak-8, umount hung when it tried to failback MDT0001
|
|
sarah would it be possible to get all traces for the case?
|
|
bzzz, please check the attached.
|
|
still looking at the logs, but haven't got a good idea yet. is it reproducable?
|
|
bzzz, the same umount hang issue happened again on a different node, if you want to check the status, please let me know. I will also upload the trace from soak-10 which the hang happened
|
|
can you grab all the traces with echo t >/proc/sysrq-trigger please?
|
|
sarah thank you, checking..
|
|
and the kernel logs from the servers please
|
|
"Alex Zhuravlev <bzzz@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/47185
Subject: LU-15761 obdclass: fix locking in llog_cat_refresh()
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f161c9dd6fcc11dbb044225a05d6d283d665424f
|
|
It looks like this locking imbalance was introduced by patch https://review.whamcloud.com/33401 "LU-11418 llog: refresh remote llog upon -ESTALE".
The very strange thing is that patch was landed in 2.12.0-RC1, but has not been noticed until now. It is possible that some other recent landing has made this codepath more likely to be hit, because otherwise it seems like the code would always be broken.
|
|
This may be related to several llog-related test failures that were previously reported, all of them with the "retry remote llog process" message printed.
Looking at the callers of llog_cat_refresh(), it looks like this could happen in combination with the patch https://review.whamcloud.com/40742 "LU-13974 llog: check stale osp object" landed in 2.15.0 that returns -ESTALE more frequently:
int llog_cat_declare_add_rec()
{
:
start:
down_read_nested(&cathandle->lgh_lock, LLOGH_CAT);
:
rc = llog_declare_write_rec(env, cathandle->u.chd.chd_current_log,
rec, -1, th);
if (rc == -ESTALE && dt_object_remote(cathandle->lgh_obj)) {
up_read(&cathandle->lgh_lock);
rc = llog_cat_refresh(env, cathandle);
if (rc)
RETURN(rc);
goto start;
}
|
|
Sarah, are you able to restart Soak with Alex's patch on master? You were previously able to hit this issue easily, so hopefully it will tell us quickly if the patch fixes it.
|
|
Sarah replied on Skype: I have loaded soak with it yesterday, will check the status a little later and let you know
|
|
Hi Andreas, last time the failure occurred after 6/7 hours running, so far soak has been up over 10 hours and it is doing well.
|
|
A quick update on soak, it doesn't hit the umount issue during failover, but it did hit LU-14495 on MDS0, test is still ongoing since disabled panic-on-lbug. I will upload the trace and debug log here since LU-14495 is closed.
|
|
thanks for the update. I'm going to check the logs. in the mean time, can you please restart?
|
|
I don't understand where this problem comes from:
[ 5814.198337] LustreError: 12981:0:(out_handler.c:910:out_tx_end()) soaked-MDT0000-osd: undo for /tmp/rpmbuild-lustre-jenkins-ZEMyr9OQ/BUILD/lustre-2.15.0_RC3_3_gf161c9d/lustre/ptlrpc/../../lustre/target/out_handler.c:445: rc = -524
want to debug this..
would it be possible to run the test with full debug enabled and dump Lustre upon this message?
|
|
okay, I will restart with full debug
|
|
bzzz, please check the log lustre-log.1651732904.9736, this is with full debug, hope it helps
|
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/47185/
Subject: LU-15761 obdclass: fix locking in llog_cat_refresh()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 69cdcbe3cf732124b3b5be1a1f235aa1b78f6c85
|
|
LU-15761 fix has landed for 2.15. I have reopened LU-14495 to track the remaining issue
|
Alex Zhuravlev, please check the log lustre-log.1651732904.9736, this is with full debug, hope it helps
yes, it did, thanks a lot. I've got rough understanding, trying to construct a reproducer.
|
|
"Etienne AUJAMES <eaujames@ddn.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49470
Subject: LU-15761 obdclass: fix locking in llog_cat_refresh()
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: cb2086c2f9268db6f65ed2d31384806057a0ee30
|
Generated at Sat Feb 10 03:21:04 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
lu-14495-soak-8debug
lustre-log.1651263727.22864