|
Can you check "lfs getstripe" on a few of the broken files, to see if the FIDs of the IST objects are unusual? I suspect that the directory is OK, but the error is coming from the OST which does not have the objects in the MDT file's layout. That may still indicate a problem with the MDT or OST, but will give a starting point.
|
|
Can you please check "lfs getstripe" on a few of the broken files. It may be that the error is coming from the OST and not the directory at all.
|
|
Assigning to Fan Yong for further investigation
|
|
Here is the result of lfs getstripe for files in that directory:
|
|
Would you please to collect the -1 level Lustre debug log on both the client and MDT when you hit "lfs getstripe simul_link.2280" failure? Since we do NOT know (if you know, that is better) on which MDT the file "lfs getstripe simul_link.2280" resides, then have to collect the logs on all MDTs.
Thanks!
|
|
I collected -1 level Lustre logs on the client and for each MDT. They are in the tar file 'getstripelogs.tar.gz' which I attached to this issue.
The command I logged is:
The output of the command was:
A grep seems to indicate that jet2 may be the log of interest, but I included all of them for completeness. Let me know if you need any other information.
|
|
The log on the client (client-getstripe.log) shows that:
Means client intent open ([0x30000cf20:0x1:0x0]/simul_link.898) RPC to the mds#1
The log on the mds1 (jet2-getstripe.log) shows that:
Means the mds#1 received the intent open RPC. It did lookup firstly and found the name entry "simul_link.898" existed on this MDT, but its FID is remote, then triggered osp_attr_get() to fetch the object's attribute when initialise the object. Unfortunately, the remote MDT returned -2 (-ENOENT) to this MDT. That the "simul_link.898" is dangling name entry. That is why the subsequent operation got -14 (-EFAULT) failure.
Currently, I do not know what caused the dangling name entry. But I would suggest to run namespace LFSCK to fix related Lustre inconsistency. To be safe, you can run namespace LFSCK without "-C" option firstly, that will detect how many dangling name entries in the system but NOT auto repair them. Then you can check whether need to fix them. If you think it is necessary to re-create related lost MDT-objects, then re-run the namespace LFSCK with "-C" specified.
|
|
We came up with an easier reproducer for this issue in case you need to collect more information. Details are below.
Create a striped directory for this test. cd to that directory and create a simple file:
From there, create a script called 'linkme.sh' with the following contents:
Now, using srun, we can run the script across many nodes/cores w/ no timeout. Example below:
The script ran for a bit, but eventually we started seeing "bad address" errors. I'll continue to try and collect more information.
|
|
Ran an lfsck namespace with -C and got the following LBUG on multiple MDTs.
Have the following call stack on two MDTs.
Filesystem info:
16 MDS, 4 OSS, running ZFS 0.7.0-0.3llnl and lustre 2.8.0 on a RHEL 7.2 based operating system (3.10.0-327.28.2.1chaos.ch6.x86_64).
Also worth noting, once we have a directory with files that exhibit this "bad address" error, the directory cannot be removed.
Let me know if you need more info.
|
|
I'm going to attempt to bring our filesystem back up this afternoon, if you could let me know if you have everything you need, that'd be great! Thanks!
|
|
Joe
Fan Yong is based in China so mayl not see this question until Sunday evening by this time of day
Peter
|
|
Ah, thanks for letting me know, Peter. We are able to reproduce it if necessary, so I think it's safe to reboot our filesystem.
|
|
I will make patch to fix the namespace LFSCK assertion.
Also worth noting, once we have a directory with files that exhibit this "bad address" error, the directory cannot be removed.
That is because there are dangling name entries under the parent directory, the dangling name entries cannot be removed via normal unlink/rmdir command, as to the parent directory are not empty. That is why the parent directory cannot be removed under such case.
To resolve such trouble, you have to use the namespace LFSCK with "-C" option to fix the dangling name entries firstly, then removed them.
|
|
Fan Yong (fan.yong@intel.com) uploaded a new patch: http://review.whamcloud.com/22723
Subject: LU-8569 lfsck: cleanup lfsck requests list before exit
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 46dd7d98cb262fbbe1285b447cd763bfc80b27d4
|
But I would suggest to run namespace LFSCK to fix related Lustre inconsistency.
I had done that, but it did not fix the problem. lfsck ran to completion and did not assert when I ran it. The assertion is a new thing. I have no idea why it crashed this time around.
|
|
I think you have specified "-C" option when run the namespace LFSCK completely, right?
Do you have the Lustre debug log (with "lfsck" debug enabled) when LFSCK ran? That will record which inconsistency have been detected and repaired (or failure). If you have not collected related information, can you show the me the lfsck status lproc output?
Thanks!
|
|
I went ahead and attached a log file called "lfsck_namespace_state-9-28-2016.log" which was obtained by running the following on each MDS:
Worth noting, when I restarted the filesystem, I had to stop the lfsck namespace check because the kernel panics would continue to occur because lfsck tried picking up where it left off.
Also, we are creating the name dangling issue at will at the moment with the reproduction steps I provided in my Sept 21, 2016 comment (the one with the linkme.sh). I think that still needs to be addressed.
I'm also going to break out the lfsck call stack issue to a separate ticket, it is unclear whether or not it is related.
|
|
LU-8647 for lfsck assertion which we started discussing in this ticket.
|
|
According to namespace LFSCK status, some dangling name entry should have been fixed:
But still some failures when try to repair the striped directories:
Unfortunately, if without the detailed LFSCK Lustre kernel debug logs, we cannot know what caused the LFSCK failure. If you can re-run the namespace LFSCK, then please enable "lfsck" debug on the MDTs, and collect the Lustre kernel debug logs.
Currently, since the the namespace LFSCK failed to fix some inconsistency, if you have to remove those dangling entries some soon, then one possible solution is that: mount the backend as "ZFS" and remove those entries under "ZFS" mode directly. That will leave some stale OI mappings in the system, but it is almost harmless but space waste.
|
|
It may also be possible to use "lfs rm" to remove dangling remote directory entries without trying to unlink the remote inode. That is intended for use in case of an MDT becoming permanently unavailable, but should also work in this case.
|
#!/bin/bash
filename=$(hostname)_${RANDOM}
ln afile $filename
I have ever tried above scripts with multiple clients run in parallel, but cannot reproduce the trouble.
On the other hand, analysis your scripts: it creates a regular file 'afile', then repeatedly hardlink to it. If the hardlink triggers the issue finally, since nobody unlink the source object 'afile', then the unique possible case (for dangling name entry) is that the FID stored in the hardlink name entry is wrong. But I cannot imagine how this can happen.
Giuseppe, would you please to reproduce the issue as the way you mentioned with "-1" level Lustre kernel debug logs collected on the MDTs? Thanks!
|
|
I can get you some logs soon. Our test system isn't happy right now. Working on getting it back up so I can reproduce this to get those logs. Stay tuned.
|
|
Hi dinatale2,
Were you able to get the test system up and reproduce?
|
|
Still having issues with it. Will attempt to reproduce this ASAP.
|
|
Logs are now attached to this incident. The file names are jet-link-logs-part[1-4].tar.gz. The part 1 gzip has errors.log in it which has a sampling of what shows up in the console so you can use that to track down a specific file in the logs. Let me know if you need anything else.
|
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/22723/
Subject: LU-8569 lfsck: cleanup lfsck requests list before exit
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 445da16c2ac0475b1c1077c822800b68cdbb7ce3
|
|
Landed for 2.9
|
|
Actually perhaps I was premature to mark as resolved here. Fan Yong, what did the patch tracked under this ticket that jus tlanded to master address? Is there still work to be tracked under this ticket?
|
|
Peter,
There is still work being tracked under this ticket. The logs I posted last week are to help find a resolution to this issue.
The patch that landed was for LU-8647.
|
|
So LU-8647 was fixed by http://git.whamcloud.com/fs/lustre-release.git/commit/445da16c2ac0475b1c1077c822800b68cdbb7ce3 even though it used the LU-8569 JIRA reference in the commit message?
|
|
Peter,
As you can see in the comment history, to make LU-8569 original issues to be clear, the new test failure about the LFSCK was split from LU-8569 description with new ticket LU-8647. The patch http://review.whamcloud.com/22723/ was used for resolving LU-8647 issue, but because the patch http://review.whamcloud.com/22723/ was push to Gerrit before LU-8647 created, then such patch still used the old ticket number.
So we can close the ticket LU-8647 as resolved. There are still some work to be done for LU-8569. I am investigating the huge logs.
|
|
Got it. For future reference it is possible to make adjustments to git commit messages when landing, so it would have been possible to use the correct JIRA reference without delaying things.
|
|
Just looked the debug log, it looks like update log is too long, which seems not right.
too much log cookies ( > 1k) for this transaction, each cookie can hold 32k update records. So I do not understand why link can generate such big record size. Hmm, even though the linkea size might be big in your test. (Do we limit linkea size for zfs?) the problem might be in
sub_updates_write. and related with this patch http://review.whamcloud.com/21334 , I will check.
I suspect this test might reproduce the problem, sigh, I do not have zfs environment here,
|
|
Just did some tests on ZFS and it looks like the problem is because the linkEA on ZFS reach above the llog chunk size (32768), which our current update llog system can not handle. i.e. one update operation (update op + its parameter) size can not > llog chunk size (32KB).
So is it ok to limit the linkea size here?
|
|
Yes, I think it is reasonable to limit linkEA size in this case. The Linux kernel xattr API is also similarly limited by the size of individual xattrs, and ldiskfs has a 4KB limit for xattrs, so the Lustre code is already expecting that not all links will be stored for a given file.
|
|
Fan Yong (fan.yong@intel.com) uploaded a new patch: http://review.whamcloud.com/23500
Subject: LU-8569 linkea: linkEA size limitation
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0d8fe108f7b7f267fa790320954fc55e996af964
|
|
Fan Yong (fan.yong@intel.com) uploaded a new patch: http://review.whamcloud.com/23741
Subject: LU-8569 lfsck: handle linkEA overflow
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 94f5d2fec9edb6e1e5359ceebea9882cb5bb2719
|
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/23500/
Subject: LU-8569 linkea: linkEA size limitation
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e760042016bb5b12f9b21568304c02711930720f
|
|
Before this closes, can these patches also be ported to the 2.8FE branch?
|
|
Giuseppe
The ticket will be marked resolved when the patches land to master but the ticket will remain on the LLNL prority list until the equivalent patches have been ported and landed to the 2.8 FE branch
Peter
|
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/23741/
Subject: LU-8569 lfsck: handle linkEA overflow
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 048a8740ae26e3406a7eab3bca383a90490cef93
|
|
All patches landed to master for 2.10. Ports to 2.8 and 2.9 FE branches will be tracked separately.
|
|
Peter,
Are there tasks created so I can keep track of the 2.8 FE port?
Joe
|
|
We'll post the links on the ticket and mark with llnlfixready when it's ready for you to pick up
|
|
Apologies Peter, I went ahead and created LU-9037 to keep track of the porting so those who are interested can keep track of it's progress.
|
Generated at Sat Feb 10 02:18:42 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
getstripelogs.tar.gz