|
Mike believes that this issue is a duplicate of LU-11358
|
|
Here is an update on trying to isolate the issue
b2_10-ib branch, system is configured with PFL and striped dir, fault injection interval is 7200s, running overnight, fail/pass rate:
1. build https://build.whamcloud.com/job/lustre-b2_10-ib/92/ 37/301
2. build https://build.whamcloud.com/job/lustre-b2_10-ib/95/ 65/563
3. build https://build.whamcloud.com/job/lustre-b2_10-ib/98/ 58/593
Then I decreased the fault interval from 7200s to 3600s, on build #98 again, the result is 202/636, the failure rate rises about 4 times.
Clients got evicted
Around the same time check MDS 0, recovery had some problem
MDT0 dmesg around the time
I will reload build #92 with 3600s interval and see how it goes
----Changes from #92 to #95 are------
LU-10527 obdclass: don't recycle loghandle upon ENOSPC (detail)
LU-11582 llite: protect reading inode->i_data.nrpages (detail)
LU-11461 scripts: Support symlink target (detail)
LU-9230 ldlm: speed up preparation for list of lock cancel (detail)
LU-7631 tests: wait_osts_up waits for MDS precreates (detail)
LU-11187 ldiskfs: don't mark mmp buffer head dirty (detail)
LU-10055 mdt: use max_mdsize in reply for layout intent (detail)
LU-10818 obdecho: don't set ma_need in echo_attr_get_complex() (detail)
LU-10612 tests: reply_single.sh,test_48: No space left (detail)
LU-11195 lod: Mark comps cached on replay of layout change (detail)
LU-11536 ofd: ofd_create_hdl may return 0 in case of ENOSPC (detail)
LU-10629 lod: Clear OST pool with setstripe (detail)
LU-11068 build: remove invalid kernel srpm location (detail)
----Changes from #95 to #98 are------
LU-8999 quota: fix issue of multiple call of seq start (detail)
LU-11186 ofd: fix for a final oid at sequence (detail)
LU-11658 lov: cl_cache could miss initialize (detail)
LU-11567 utils: llog_reader print changelog index (detail)
LU-10985 mdt: properly handle unknown intent requests (detail)
LU-11056 lwp: fix lwp reconnection issue (detail)
LU-9874 osd-ldiskfs: simplify project transfer codes (detail)
LU-10461 tests: call exit in the skip routine (detail)
LU-9810 lnd: use less CQ entries for each connection (detail)
LU-10213 lnd: calculate qp max_send_wrs properly (detail)
LU-8895 target: limit grant allocation (detail)
LU-7988 hsm: added coordinator housekeeping flag (detail)
LU-10516 misc: require ldiskfsprogs-1.42.13.wc6 or later (detail)
LU-11049 ssk: correctly handle null byte by lgss_sk (detail)
LU-11196 tests: clean up after conf-sanity test_101 (detail)
LU-11736 utils: don't set max_sectors_kb on MDT/MGT (detail)
LU-11757 lod: use calculated stripe count (detail)
LU-11006 lnet: fix show peer yaml tree with no peer (detail)
LU-11737 lfsck: do not ignore dryrun (detail)
LU-10565 osd: unify interface for vfs (detail)
LU-11187 ldiskfs: update rhel7.6 series (detail)
LU-8402 ldlm: assume OBD_CONNECT_IBITS (detail)
|
|
With the same fault injection time(3600s) and same build (#98), after clearing the update log on all MDS, the fail/pass number change to 82 fail / 530 pass, so it helps with the case. Will try on earlier build to see how it does
|
|
Just want to update the ticket to clarify a bit.
The current plan (per the call this morning) is to go to the tip of b2_10 and run the tests on that codebase.
- What is the approximate ETA on that?
- Is it possible to stop testing on failure? One problem with the existing logs is the volume of information.
|
|
By the way, what is our fault injection? What exactly are we doing? I know failover, but of which components, and how?
|
|
hi Patrick, please see below
What is the approximate ETA on that?
I will work on moving soak to tip of b2_10 tomorrow, and can get some data later tomorrow
Is it possible to stop testing on failure? One problem with the existing logs is the volume of information.
As I know soak cannot stop on its own when hitting problem, but I can stop it manually
what is our fault injection? What exactly are we doing? I know failover, but of which components, and how?
soak does 4 kinds of faults for now: MDS restart, MDS failover, OSS restart, OSS failover
for MDS/OSS restart, it will reboot the node(power cycle) and wait the node come back to mount
for MDS/OSS failover, it will reboot the node as above, and do failover and failback to/from secondary node configured.
|
|
Thanks Sarah!
So James was saying Soak is up and running today, and we've got some failures.
Could you help me out by giving me an example with this information:
- Application information for failing application
Application, command invocation, nodes, any logs or error messages specifically from the application
- Whatever fault injection we did around that time (what it was, which nodes)
Pointers towards logs from these nodes
I will try to see about getting on the system some time later today.
|
|
Here is an example failure, from 240435-mdtestfpp.out
console and syslog and application log are attached.
1. Application failure error, soak-* are test nodes involved. I picked 1 node soak-23 to find more info listed below. The fail occurred around 2/22 16:04:36
command used:
2. syslog of soak-23(soak-23.log-20190224.) around 2/22 16:04:36, the client was evicted, cannot stat MDS #2, which is soak-10.
3. soak-10 syslog (soak-10.log-20190224) around the same time, it just came back from reboot, check the soak.log and found soak-10 was triggered mds_failover right before this
soak.log
|
|
Hmm, OK.
Well, this error and the associated evictions are enough to cause job failure:
I'm trying to find a report of an issue like this, nothing is coming up immediately. I'll see what, if anything else, I can find...
|
|
And, thank you! Very helpful.
I need to go on soak and look around, but if you're able, can you check how often/if we've seen this set of messages? Specifically the Invalid record message - something like:
|
|
In about 19h running, found 3 messages
I will move soak to tip of b2_10 later today, you can still get these logs on soak
|
|
I'm sorry if I'm asking questions that have been answered elsewhere, but I've lost track of all the info.
So, the application failure rate on soak is pretty high right now. The primary culprit - for the large majority of failures - is various different errors coming out of the llog code on MDS/MDT failover/failback (all of which point to, essentially, the log being incorrect/damaged). It's causing aborted recoveries, which cause a lot of evictions, which lead (sometimes) to application failures.
A few questions.
- Have we tried master on soak recently? Did it have a similar failure rate to 2.10?
- What earlier versions of 2.10 have we tried? Did any of them have significantly lower failure rates?
- What is the most recent version of 2.10 we are confident had lower failure rates?
|
|
I've identified two candidates for possible changes that could've introduced these issues. I have not identified any llog related changes in master that are missing in b2_10 currently.
These are the changes:
commit 51e962be60cf599ecf154ea3a6b1c0f9882daac2
Author: Bruno Faccini <bruno.faccini@intel.com>
Date: Wed Jan 17 16:22:58 2018 +0100
LU-10527 obdclass: don't recycle loghandle upon ENOSPC
In llog_cat_add_rec(), upon -ENOSPC error being returned from
llog_cat_new_log(), don't reset "cathandle->u.chd.chd_current_log"
to NULL.
Not doing so will avoid to have llog_cat_declare_add_rec() repeatedly
and unnecessarily create new+partially initialized LLOGs/llog_handle
and assigned to "cathandle->u.chd.chd_current_log", this without
llog_init_handle() never being called to initialize
"loghandle->lgh_hdr".
Also, unnecessary LASSERT(llh) has been removed in
llog_cat_current_log() as it prevented to gracefully handle this
case by simply returning the loghandle.
Thanks to S.Cheremencev (Cray) to report this.
Both ways to fix have been kept in patch as the 1st part allows for
better performance in terms of number of FS operations being done
with permanent changelog's ENOSPC condition, even if this covers
a somewhat unlikely situation.
That change first appeared in 2.10.6.
The other change:
-------
commit 10cc97e3c1487692b460702bf46220b1acb452ee
Author: Alexander Boyko <alexander.boyko@seagate.com>
Date: Wed Mar 22 14:39:48 2017 +0300
LU-7001 osp: fix llog processing
The osp_sync_thread base on fact that llog_cat_process
will not end until umount. This is worng when processing reaches
bottom of catalog, or if catalog is wrapped.
The patch fixes this issue.
For wrapped catalog llog_process_thread could process old
record.
1 thread llog_process_thread read chunk and proccing first record
2 thread add rec to this catalog at this chunk and
update bitmap
1 check bitmap for next idx and process old record
Test conf-sanity 106 was added.
Lustre-change: https://review.whamcloud.com/26132
Lustre-commit: 8da9fb0cf14cc79bf1985d144d0a201e136dfe51
Signed-off-by: Alexander Boyko <alexander.boyko@seagate.com>
Seagate-bug-id: MRP-4235
Change-Id: Ifc983018e3a325622ef3215bec4b69f5c9ac2ba2
Reviewed-by: Andriy Skulysh
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Signed-off-by: Minh Diep <minh.diep@intel.com>
Reviewed-on: https://review.whamcloud.com/32097
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
------
This change first appeared in 2.10.4. (Which had a lot of changes from 2.10.3.)
|
Have we tried master on soak recently? Did it have a similar failure rate to 2.10?
Last time tested master was on 1/11 and that was running without PFL and DoM stripe. When we did the 2.12 testing, it had similar failure rate as current 2.10 with PFl and DoM stripe(LU-11818); but if disable PFL and DoM, it did better
What earlier versions of 2.10 have we tried? Did any of them have significantly lower failure rates?
The earlier build tested are: 2.10.6 (b2_10-ib build #92), #95 #98, build #98 seems have higher failure rates
What is the most recent version of 2.10 we are confident had lower failure rates?
I think 2.10.6 RC build(#92) is in good shape
|
|
I see you said we were going to run build #92 recently:
"I will reload build #92 with 3600s interval and see how it goes"
But I don't see the results. Did we do that? How did it go?
If #92 is indeed OK, then that suggests:
LU-10527 obdclass: don't recycle loghandle upon ENOSPC (detail)
Which is in your list of changes from 92 to 95.
If the recent run of #92 was indeed OK, can we remove that patch and try tip of b2_10?
|
|
No, I didn't rerun #92 with 3600s inverval, instead I tried #98 first with removal of update log and it helped the pass rate, then triage team suggested move to tip of b2_10
Let me first try the review build, if it doesn't help with the pass rate, then I could move to #92 to make sure it is the latest good one.
|
|
Sounds perfect, thanks!
|
|
Quick early readout. I don't have application failure rates (Sarah will have to provide that), but I did look at the failovers and evictions today. Overall failover still looks pretty choppy - quite a few have evictions - but the MDT failover is (tentatively) much improved.
Over the last month or so, about 50% of the ~320 MDT failovers we did resulted in an aborted recovery. All or almost all of those were from llog problems. So far today we've done 9 MDT failovers, with 0 aborts.
So... Tentatively, much improved. We'll have to see what the failure rate is like, and how things shake out overnight.
|
|
A quick update: soak has been running with Patrick's patch(https://build.whamcloud.com/job/lustre-reviews-ib/4213/) for about 6 hours, with only mds_failover and mds_restart, 105 pass / 4 fail(simul only). It seems the revert works. I will leave it running overnight and update the ticket tomorrow
|
|
review build #4213 has been running 20 hours, the fail/pass number is 19 fail/384 pass, only simul failed due to time out.
no crash on servers, but 1 client hit kernel NULL pointer
soak-40 console
|
|
We are now up to 54 failovers without an abort. I'll update the other bug - I'm not sure how, but that patch is causing a lot of trouble.
|
|
Suggested fix is here:
https://review.whamcloud.com/34346
I'll push it for master shortly.
|
|
Changed fixes to associate with this ticket.
Master:
https://review.whamcloud.com/#/c/34347/
b2_10:
https://review.whamcloud.com/#/c/34346/
|
|
update of lustre-reviews-ib build #4241 (https://review.whamcloud.com/34346)
After 17 hours running with only mds_failover and mds_restart, the fail/pass rate is: 97 fail / 453 pass. With this modified patch, the pass rate were still worse than the previous revert patch.
Check one of the failed application mdtestfpp.out
then check the client soak-28 syslog around 3/1 12:49, it was evicted by MDS #2(soak-10)
Check soak-10 around the same time
|
|
Woah. That's very confusing. Unless we made a mistake with the build, that should be impossible. But I do see the errors. That's extremely confusing.
I'll see what I can figure out.
|
|
Would we know if the MDS were asserting during shutdown/failover? How do we cause the MDS failover - What do we do to it exactly?
|
|
the lustre version is
lustre-reviews-ib build #4241 shows the same version, unfortunately the 4241 build was gone from jenkins.
We don't assert server, failover by hard reboot. If you want to check the soak code, it is under
|
|
Sarah,
Just reviewed with Minh.
He made a mistake yesterday. 4241 is an unrelated build.
The one we need is:
https://build.whamcloud.com/job/lustre-reviews-ib/4248/
Sorry. Are you able to install and test that on Soak?
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34346/
Subject: LU-11943 llog: Reset current log on ENOSPC
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 0cba3cc14fa68b8f43833abfb6f7bc3e547bce03
|
|
Here is an update of patch #34346 tested on soak(running about 48h) 200 fail / 1192 pass
I did see some application failures with input/output error, here is an example failure:
application mdtestfpp failed as
check the syslog of client node soak-24 and it was evicted by MDS1 which is soak-9
check soak-9, around the time
|
|
After system running about 48 hours, 1 MDS crashed after it was restarted. and I think it hit LU-11781. Also attach the console log of soak-10, soak-10-console
soak-10
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34347/
Subject: LU-11943 llog: Reset current log on ENOSPC
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 7f4a1dd92a900a55cbb980bac3d20c32968334ec
|
|
Landed for 2.13
|
|
Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34459
Subject: LU-11943 llog: Reset current log on ENOSPC
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 374a1414097fcaf83a27e7b107aa984809dba58a
|
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34459/
Subject: LU-11943 llog: Reset current log on ENOSPC
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: 4fd6fbbd0f0a0f45cd41b560c4711002e86747b6
|
Generated at Sat Feb 10 02:48:17 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
240435-mdtestfpp.out