|
This only tells us that the client reexporting Lustre as NFS was evicted, but not why. Can you please provide MDT side log that lists the reason for eviction?
|
|
Hi,
Following is logs from MDT server.
|
|
Thank you, hm, this looksl like a lock timeout.
Can you please collect lustre debug logs on the MDS and nfs-reexporting client when this is happening?
Stop all other activity, then run this on both nodes:
then start the nfsd on the client and wait until eviction, then run this right away on both nodes:
collect the lustre.log from both nodes and make them available for download somehow.
|
|
Hi,
I have collected logs and uploaded to our ftp server.
Could you please connect to our ftp.dneg.com server using filezilla and download logs
Host= ftp.dneg.com
Port = 2222
Server Type = SFTP-SSH File Transfer Protocol
Logon Type = Normal
Login Name = lustre_log
Password = L@8Ki07g
Once connected go into From_Dneg folder.
Please let us know if you are able to download logs from ftp.dneg.com ftp server .
|
|
I am (somewaht slowly) getting these files so access is ok, thanks.
Can you please include server side mds log excerpt that lists the evicted log handle?
|
|
Thanks Oleg for looking into it.
I checked the mds and client logs for 4th May, it didn't evict the client at that time even if mount was not accessible.
Have reproduced this issue again today when client got evicted. Today's logs are uploaded again on ftp under /from_dneg/06May2020/
mds logs evicting client
|
|
Any update with this please.
|
|
I got the logs and they have some holes in them due to a lot of ongoing activity so the understanding is still limited.
Basically what I pieced together is:
as the lock BL ast appears for an OPEN lock, we are sending a CLOSE RPC to the MDS. then there's a gap until server evicts us 100 seconds later:
no interim state from this process got into the logs unfortunately. But on the server messages log there are messages of long processing of messages, so I wonder if the message is delayed somewhere?
Also you did not include the lustre-log.1588746168.17655 the second time around so all that state until the timeout hit is lost and the first mentioning of the lock in question is the eviction message.
Also the messages about processing taking 80000+ seconds is highly unusual.
Can you please describe what's the workload like on the fs? any non-nfs access in shared namespace with nfs access? what do the nfs clients roughly do?
I'd really like to get the full logs but with the sheer amount of activity going on that appears to be not very practical, does the nfs reexport node has local (and fast) storage by any chance so we can try running debug daemon there? Absent that
can you rerun the reproducer but instead of setting debug to -1 set it to: "inode super info warning other ha dlmtrace error emerg rpctrace vfstrace console".
If any of the /tmp/lustre.log files are generated, we need them too.
Additionally please compress them (xz is best it seems though is sort of slowish) as otherwise downloading is really slow from your location for me. Please include dmesg excerpt that shows the first eviction.
Additionally I tried to reproduce this on my test system applying what I was able to gather from existing logs and it seems to be working fine (but I am on a current b2_12), would not be a bad idea if you are able to update to 2.12.4 for this as well, I see there are some potentially relevant changes (e.g. the LU-4398 patch)
|
|
lustre-log.1588746168.17655.xz is uploaded under same folder, the original lustre-log.1588746168.17655 is uploaded too.
The namespace is accessed for data backups using rsyncs locally. Time machine backups over afp protocol. The namespace gets mounted on other server over nfs to copy data using rsync.
In parallel we are exploring if we can get a downtime to upgrade it to 2.12.4.
|
|
We have upgraded golf1,2,gmds1,goss1 to goss6 to 2.12.4
|
|
ok. Please let us know whether this issue is still impacting you.
|
|
We have upgraded hotel servers as well to the Lustre version 2.12.4 . But, We are facing the issues with evicting clients in the 2.12.4 version as well.The NFS reexport is enabled in the hotel1 only in which we are having issues with client evictions.
We Have uploaded the logs to our ftp server from MDS and NFS client servers to check this client evictions and the associated NFS re-export issues.
Log location : /from_dneg/31Aug2020
Adding the details of the hotel servers
client (hotel1) kernel version : 3.10.0-1062.9.1.el7_lustre.x86_64
client (hotel1) lustre version : lustre-2.12.4-1.el7.x86_64
mds (hmds1) kernel version : 3.10.0-1062.9.1.el7_lustre.x86_64
mds (hmds1) lustre version : lustre-2.12.4-1.el7.x86_64
oss (hoss1-hoss6) kernel version : 3.10.0-1062.9.1.el7_lustre.x86_64
oss (hoss1-hoss6) lustre version : lustre-2.12.4-1.el7.x86_64
/etc/exports on hotel1 :
/user_data 10.27.0.0/16(fsid=123456789,rw,anonuid=0,insecure,no_subtree_check,insecure_locks,async)
|
|
Hello Team,
Can we have an update on this ? Also please let us know if we need any additional info to check this further.
|
|
Really appreciate if we get an update on this at the earliest .
|
|
It looks like your logs cover different regions.
The dmesgs covering the eviction are at 17:00 but the messages at 17:07 to 17:10 so we only know there's an eviction but nothing to corellate back to the logs
|
|
Yes, the eviction happened at aug 29 17:00 IST . The dmesg and messages logs having the same time but not really sure why the Lustre logs don't cover the logs at 17:00 . The evictions are keep happening.
I had generated the logs again and uploaded to the FTP server . Please have a look on these new evictions logs as well to investigate this further .
Eviction time : Aug 31 at 19:23 and 19:35
Log location : /from_dneg/05Sept2020
|
|
Well, this still seems to be very disconnected between the two sets of logs. the debug logs cover from 1598883056.697070 to 1598883239.198867 (these are seconds since unix epoch), I am not exactly sure what your timezone is but it's XX:10:56 to XX:13:59 which seems to be far away from the eviction event.
Did it take you a long time between the eviction and stopping the log? The log data overflows super fast so it's important to stop the lustre logging very quickly after the eviction was recorded.
|
|
Both hmds1 and hotel1 running with the same timezone (Asia/Calcutta (IST, +0530)).
I have collected the lustre debug logs right after (approx 19:45 IST) the reported evictions which are at "Mon Aug 31 19:23:03 2020" and ""Mon Aug 31 19:35:55 2020" . The debug logs writing finished at 19:50 IST (approx) on both hmds1 and hotel servers.
As checked the lustre debug logs, there are some difference in the hmds1 lustre debug log write timings. hotel1 seems to be having the logs of eviction at 19:35 IST.
I am not sure why hmds1 is not captured the logs of evictions. Why this difference is coming in hmds1 ? Is there any other settings ?
The connection restorations details also available in the logs which are happening quickly but the evictions are making more trouble with the backup jobs.
|
|
Any update on this please ?
|
|
Hello Team,
Any news here ?
|
|
Ok, so did nto realize you had a fraction of hour timezone.
I think everything checks out then. The debug logs at your debug_mb setting only allow ~3 minutes of detailed logs and you generated them 5 and 15 minutes away from the time eviction happened.
It sounds like you might want to set dump_on_eviction to 1 on your clients to cause the client to dump current copy of the log the moment it realizes it's evicted and also dump_on_timeout on the server to do the same when it encounters lock timeout and evicts a client.
|
|
OK, We set these parameters on both MDS and client currently to the get the proper logs after evictions .
On both client and MDS
On Client
On MDS
|
|
Have uploaded the log dump files to FTP server which is generated on MDS and client right after the eviction. Hope that would help to investigate more on it.
Evictions happened at "Sun Sep 27 22:57:23 2020 IST"
Log Location: /from_dneg/29Sept2020
hmds1 (MDS) log file :lustre-log.1601227748.5992.xz
hotel (Client) log file : lustre-log.1601227749.1886.xz
|
|
We had eviction today as well which was happened on Tue Sep 29 09:20:16 2020 IST and Tue Sep 29 09:29:16 2020 IST , Uploaded the logs for the same as well to investigate further.
Evictions at : Tue Sep 29 09:20:16 2020 IST and Tue Sep 29 09:29:16 2020 IST
Log Location: /from_dneg/30Sept2020
|
|
Any updates here please ?
|
|
it seems you have way too much stuff going on your nodes even for 10G of ram logs, so I guess we really need to reduce debug level and hope it all fits in and the debug level catches all the interesting bits.
lctl set_param debug="+rpctrace +dlmtrace +neterror +net"
please replace the debug=-1 with the above and otherwise repeat the exercise and upload the logs.
Sorry it's taking this long with the iterative process.
|
|
OK, Do I need to stop all the IO/NFS connections on it to apply this change or apply it directly ?
So this will be the final config on both MDS and client or any other changes ? Please confirm
|
|
you can apply it directly.
basically you run it like before, jsut changing the lctl set_param debug=-1 line to this one, but the rest of it stays the same
|
|
Have uploaded eviction logs after above changes. Please have a look on this
Eviction at : Oct 7 19:22 IST
Log location : /from_dneg/08Oct2020
|
|
I got the logs and the are still full debug and then it dawned on me that since you are not restarting the nodes - the instruction I gave you would not disable excess logic. Sorry about that.
So to manually set the desired debug level and override all the bits you currently have you need to do this instead:
lctl set_param debug="vfstrace info inode warning error dentry emerg console rpctrace dlmtrace neterror net"
|
|
OK , Have set these parameters on both hmds1 and client. However the "get_param" still showing blank for the debug level, Hope that won't be a problem.
|
|
"still" showing blank? it was not supposed to show blank at any time past or present. are there any errors in dmesg?
|
|
Yes, Still it is blank and no errors in dmesg as well 
|
|
I mean was it is supposed to be something like this:
does "cat /sys/kernel/debug/lnet/debug" output anything?
|
|
OK, it's giving the output for debug on both and hope all good with that.
|
|
There were couple of evictions happened after the change in the debug level and have uploaded the latest logs . Please have a look on this
Evictions at : Oct 14 04:11 and 04:28 IST
Location : /from_dneg/14Oct2020
|
|
Whether the latest logs contains sufficient data ?
|
|
Any updates here ?
|
|
the 1602629930.5992/1602629931.16759 set of logs seems to be promising.
I see there's an attempt to bunch some other locks that all happen to be open locks that are sort of expensive in the sense that they cause an extra RPC to close the file descriptor first this is not ideal but not fatal.
The real trouble is eventually we run out of "slots" for the close RPC and nothing is waking us up until eviction 100 seconds later:
The reason seems to be some RPCs from this client being stuck on the server and so we run out of slots.
checking server side for one of such requests it looks like LU-13692 might be what you are currently hitting. NASA seems to be having a good success with the patch there now.
|
|
Thanks for the update and just gone thorough that case . Could you please explain which patch need to apply here in our system ? What exactly the patch is doing ? How we can implement the patch ? It would be good if you can share some docs on for that too.
|
|
Campbell
The fix Oleg mentioned is one that we are hoping to include in our upcoming 2.12.6 release. Would you prefer to wait until that so that you are running something well-tested or are things dire enough that you need to attempt a hotfix ASAP?
Peter
|
|
Hi Peter,
When we are planning to release 2.12.6 version with the fix ? Whether there will be more delay in releasing this minor version ?
|
|
Meantime, Could you please provide the patch as well with the documentation to apply it ? We will also try it out in our servers to see if that fixing the eviction issue.
|
|
Hi Peter/Oleg,
As requested earlier, We would like to apply the patch first in our servers .Could you please share the patches for the same ASAP ?
|
|
The patch is at https://review.whamcloud.com/#/c/39598/
|
|
Hi Peter,
Can you please tell me what needs to be done here to apply the patch? I downloaded ldlm_lock_new-83331795f2550e9e0bf15c3d3832f3d3.c, created a patch file from a diff between it and the version of ldm_lock.c in the 2.12.4 src rpm, and it patched successfully but failed to build. I was able to build rpms from the 142fe73 tree, but they were version 2.13.56, and the rpms were kmod-lustre-client and lustre-client - so is this patch for the client packages? We use the server packages on the affected client, so have the lustre kernel and lustre rpm on our client. Should I build the 2.13.56 rpms against a generic kernel and use that?
Thanks,
Campbell
|
|
you can get already built RPMs from this patch: https://review.whamcloud.com/40412 there's a build link in the comments.
The patch only affects MDS so technically that's the only node you need to update with it (This is tip of the b2_!2 tree so it has all the other fixes too though)
|
|
Hi Oleg,
Apologies, I have not been able to find the link to the RPMs. I've found a kernel image (kernel-centos7-x86_64) and a source tree (ac40c31.tar.gz), but could not find a link to any rpms - can you please provide more detail? Sorry, I'm clearly missing something obvious.
Kind regards,
Campbell
|
|
Campbell
The build link from the comments Oleg was referring to is https://build.whamcloud.com/job/lustre-reviews/77274/
Hopefully it is intuitive enough how to drill in to get the distro/arch version relevant for you but let us know if not.
Peter
|
|
Hi Peter,
Thanks, I found them at https://build.whamcloud.com/job/lustre-reviews/arch=x86_64,build_type=server,distro=el7.7,ib_stack=inkernel/lastSuccessfulBuild/artifact/artifacts/RPMS/x86_64/
Regards,
Campbell
|
|
Since upgrading to 2.13.56 on the MDS, we've found a strange issue whereby we cannot create files over NFS to the lustre client on the lustre filesystem. We can remove files, and chown and chmod. Permissions make no difference. I can't see anything in the logs. Any ideas why this might happen? Do we need to update packages on anything else besides the MDS perhaps?
|
|
Wait - 2.13.56?! I hope that this is a misunderstanding as that is a 2.14 development tag and not something suitable for running in production.
|
|
Ah I see - you navigated to the most recent development build rather than the specific one highlighted above. The link I expected you to navigate to was https://build.whamcloud.com/job/lustre-reviews/77274/arch=x86_64,build_type=server,distro=el7.7,ib_stack=inkernel/
|
|
ok, thanks Peter. Just to make sure: I should be installing version 2.12.5_62_ga40c31-1?
|
|
Correct
|
|
HI Peter / Oleg,
We installed the version "2.12.5_62_ga40c31-1" , But unfortunately that didn't help us to fix the eviction issue. We do still having client evictions and have uploaded the latest logs to investigate further on this.
Eviction time : Sun Nov 1 02:12:09 2020 IST
Log location :/from_dneg/01Nov2020/
|
|
Any thoughts here ?
|
|
the traces are ramarkably similar, but this time as the client is forming early-cancel-requests we can trace them all on the server and the last one sent is handled on the server real fast, the reply is formed and sent, but somehow it is never received on the client (or if it is, there are no traces of it)
client side:
no activity from this thread from this moment on highlighting it's still waiting for reply
server side:
Now if we switch back to the client we actually can see the reply come from the server - the next message received from MDS is for xid 1682064348976448
But while it was sent a small fraction of a second before the other one - we don't see the message being sent in the client logs (we do see it in the server logs) - which I guess might explain why we never see the other one in logs either - it was just dropped as log bucket was full (we have log buckets divider per cpu and priority so some might be dropped before the others leaving such gaps seemingly at random when you have high level of activity).
It is still kind of strange we never see any more logs from the thread that sent the ELC request since if it did get the reply it is supposed to be sending more requests to the server. In fact checking server logs we do see more of requests that thread sent that were again timely handled on the server until that activity stops, we just don't know why did it stop.
Do you have any storage on your nfs reexport node (like real fast local storage)?
I wonder if you would be able to run in debug daemon mode there (just on the lustre client since we appear to be getting good logs on the server).
Basically you set the debug level as before, but you run this on the client:
lctl debug_daemon start /some/path/on/real/local/storage/lustre.log 40000
this will create up to 40G file at the place.
and once the condition is reproduced you then do
you don't do the dump on eviction setting in this mode operation so it's important
to still do the debug daemon stop somewhat quickly after the eviction happens.
Do you think you would be able to do something like this?
|
|
OK, I have disabled the dump on eviction on client and trying to start the debug daemon but it seems there is some limit. But not sure whether 20GB is enough for this to capture it ? Please suggest
|
|
well then please use 20480 instead of the 40000
We won't know if it's enough or not until we look inside, here's hoping it is enough but if you can reproduce with less activity that certainly should help.
|
|
I have uploaded the latest logs after starting the debug_daemon to investigate further on this.
Log Location : /from_dneg/04Nov2020/
Eviction time : Wed Nov 4 02:12:55 2020 IST
|
|
Those logs are helping out us to look more on this ?
|
|
please include server side dmesg -it helps me see evicted logs without poring through huge debug logs.
anyway I see you included a number of server side logs, but all of them were dumped an hour before the client log begins?
|
|
I also noticed that the client log only covers 15 minutes of time - this is enough to go back for a 100 second eviction, but you really need to be able to stop client side logging pretty quickly after server side eviction hits.
|
|
Unfortunately it took a while for me to stop the debug daemon after the eviction happened . I think I had stopped the service after 1-2 hr (approx) of the eviction since it is happening randomly. I could see the evictions happened at "Wed Nov 4 02:12:55 2020 IST", But it dumped other files as well , Just added the created time details below.
Client debug log seems to be stopped writing once its reached the file size threshold?, Added the time details as well for that too.
I have uploaded the dmesg of the server and client as requested. If it is making more trouble to dig more into the log file then I can upload another set of new evictions logs but this time will try to stop the service right after it reported, Let me know if that is really required here.
Log location : /from_dneg/04Nov2020/hmds1-logs
Filename : hmds1-dmesg04112020
|
|
basically for the client log to contain enough information you must stop it within ~10 minutes of the eviction happening.
the max size of 20GB is not used to stop writing but basically makes the file to act as a circular buffer so once the size is reached writing continues from offset zero overwriting old records.
So please try to catch the eviction and stop the client logging right after reporting (since there's a very characteristic eviction record in the logs on the server side you might be able to script something up to catch the event and stop the writing client-side)
|
|
OK, Understood. So have uploaded latest set of logs to check further . This time have stopped logging at client side immediately after seeing the evictions.
Log Location : /from_dneg/10Nov2020/
Eviction Time :Tue Nov 10 02:27:42 2020
There are couple of files got generated at server end , Adding the file creation details as well
hmds1
Hotel1- Eviction happened at Nov 10 02:27:42 2020 and had stopped client logging writing at Nov 10 02:29
|
|
Oleg Drokin (green@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/40602
Subject: LU-13500 ldlm: Do not LRU-cancel "expensive" locks for in bl-ast
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6afaa40254cb6c0431ea05a72fc0fd076ec0bb81
|
|
Oleg Drokin (green@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/40603
Subject: LU-13500 ldlm: Do not LRU-cancel "expensive" locks for in bl-ast
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 3b5eb8eb50885dd690307d4ba4178ee55a5bf970
|
|
Thank you. this is a good log but it's very big to sift through all of it fast.
Meanwhile some suspicions I had from the previous logs seem to be confirming - the "proactive lru lock cancelling" we do when preparing to send a cancel anyway seems to be a bit overezalous and does some pretty heavy processing.
I am going to post a patch that should help here while I am digging out some other suspicious stuff I also see. https://review.whamcloud.com/#/c/40603/
This patch you need to apply on the clients (nfs exporting ones at least) (or just use out build from the builders: https://build.whamcloud.com/job/lustre-reviews/77636/ - select suitable arch and distro type in the long list)
|
|
Looks like we are seeing some more stuffs here to rectify. Should we try this patch now or we will wait to dig more on the undergoing case ?
|
|
I think it makes sense to try the patch now - if it helps - great.
Whenever any additional patches would be needed is not yet clear and if yes and the condition would necessitate reversal of a previous patch to reproduce - that could be decided on later.
|
|
Oleg,
We have applied the provided patch but unfortunately that seems to be not helping much to reduce the evictions . Your thoughts on this please.
|
|
nfs exports stopped working. Couldn't see anything in the client log.
A restart of nfs did not fix the issue. So we had to downgrade it back to 2.12.4.
|
|
After downgrading, back to 2.12.4, the lustre filesystem was hanging on the client (the other client we have was fine). It would work intermittently but mostly not (the other client was fine during this time). Is this due to lock recovery? I tried mounting with 'abort_recov', but same issue. It eventually resolved itself however.
There's still a large number of stack traces in the MDS log, e.g.:
Nov 16 02:58:40 hmds1 kernel: LNet: Service thread pid 16267 was inactive for 278.13s. The thread might be hung, or it might only be slow and will res
ume later. Dumping the stack trace for debugging purposes:
Nov 16 02:58:40 hmds1 kernel: LNet: Skipped 4 previous similar messages
Nov 16 02:58:40 hmds1 kernel: Pid: 16267, comm: mdt01_051 3.10.0-1062.18.1.el7_lustre.x86_64 #1 SMP Mon Jun 8 13:47:48 UTC 2020
Nov 16 02:58:40 hmds1 kernel: Call Trace:
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc12b8070>] ldlm_completion_ast+0x430/0x860 [ptlrpc]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc12ba0a1>] ldlm_cli_enqueue_local+0x231/0x830 [ptlrpc]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc15a817b>] mdt_rename_lock+0x24b/0x4b0 [mdt]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc15aa350>] mdt_reint_rename+0x2c0/0x2900 [mdt]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc15b31b3>] mdt_reint_rec+0x83/0x210 [mdt]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc158f383>] mdt_reint_internal+0x6e3/0xaf0 [mdt]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc159b0f7>] mdt_reint+0x67/0x140 [mdt]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc1356e8a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc12fb83b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Nov 16 02:58:40 hmds1 kernel: [<ffffffffc12ff1a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
Nov 16 02:58:40 hmds1 kernel: [<ffffffff8dac6321>] kthread+0xd1/0xe0
Nov 16 02:58:40 hmds1 kernel: [<ffffffff8e18ed37>] ret_from_fork_nospec_end+0x0/0x39
Nov 16 02:58:40 hmds1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Nov 16 02:58:40 hmds1 kernel: LustreError: dumping log to /tmp/lustre-log.1605475720.16267
We've uploaded a log (5.1GB) from the MDS
|
|
Any findings here please ?
|
|
Hello Team,
It would be really good if you can provide a solution for the same at the earliest as this is really affecting the production backup which is really putting us in a panic situations. Requesting you to check this on priority please.
|
|
Sorry that I have no immediate answers to this, I am still thinking about the whole thing.
|
|
So I wonder if reducing amount of mdt locks would at least help you temporarily since the problem is clearly related to parallel cancels.
can you please run this on your nfs export node(s)? The setting is not permanent and would reset on node reboot/lustre client remount.
|
|
Hi Oleg,
Just some feedback on this setting: since the change has been made, there have been no stack traces, and the backups that run on that node have not been finishing early (so far). We'll see what happens when they finish, but looking much better so far!
Thanks,
Campbell
|
|
HI Oleg,
We have modified the config for limit the number of nfsd threads a single client can use on NFS gateways which helped to avoid the further client evictions. We are not seeing any stack traces and the backups are running smoothly so far.
Also we are having the below lru size configs are in place too on the clients
Please check and suggest .
|
|
Any updates here ?
|
|
these ldlm settings look fine to me if they work for you. You can shorten the max age if you want depending on your workload.
I can't comment on the "svc_rpc_per_connection_limit" parameter as I am not familiar with that part of the NFS stack.
|
|
Hi Oleg Drokin,
As we monitored the system with below suggested change , it's noticed that the Metadata server (hmds1) is getting rebooted randomly and the both the clients are evicting from the cluster which is again making issues. Could you please look into that more and suggest ?
|
|
is there any more information about the MDS crashes/reboots? Is there an oops/lbug/assert (what's the stacktrace)? a HA-induced reboot due to high load/whatever? something else?
|
|
I have uploaded the crash logs for couple of reboot instances to check this further.
Uploaded file name: vmcore-dmesg-2021-06-17.txt - Restarted on 17th June
vmcore-dmesg-2021-05-11.txt - Restarted on 11th May
Last restart instance
Load usage seems to be normal during that time
Most of the restart, could see the message "blk_update_request: critical medium error" always.
|
|
The last one looks like LU-13054 to me and this patch seems to be missing from the tree you are running (git hash ac40c31)
|
|
Hi Oleg,
Whether this patch(for the reboot issue) included in 2.12.6 version or do we need to apply this manually in 2.12.6 ?
|
|
The fix is queued up for the upcoming 2.12.8 release but, in the meantime, you could apply the patch - see https://review.whamcloud.com/#/c/44121/
|
|
Hi Peter,
Thanks for the update, We have applied above patch on hotel and monitoring it further.
When are we planning to release the version 2.12.8 ?
|
|
The timeline will be affected by third party software releases so is a little unclear but seems very likely to be before the year is out.
|
Generated at Sat Feb 10 03:01:48 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
hmds1-log-write.png
vmcore-dmesg-2021-05-11.txt
