|
I hit this same LASSERT running racer on my single-node test system (2x1GB MDT, 3x4GB OST) as part of acceptance-small.sh using v2_6_54_0-16-g0024956:
|
|
I've pushed a cleanup/debug patch http://review.whamcloud.com/12398 to at least split this LASSERT() into two separate ones, so that it is easier to see which one is failing. This is not expected to fix the problem here.
|
|
dt_object_exists() is failing.
|
|
I hit this problem again, with the same stack in out_handle() as my previous occurrence.
|
|
Can we replace this with error handling?
|
|
yes, in out_tx_attr_set_exec() once the lock is taken we should check whether the object still exists.
|
|
Please see http://review.whamcloud.com/12608.
|
|
"Note that the setattr is for the ctime on d0. Should this ctime update really go to each stripe? If so then shouldn't we lock the stripes as we do from mdt_attr_set()?"
John, I think you are right, we should lock the object both for setting xattr and attr. And ctime does not need to go every stripe, but
1. lod_attr_set needs to be fixed, i.e. set_attr time only goes to master object.
2. lmv_merge_attr needs to be fixed, i.e. it needs to take count in master object time as well, right now it only merge the time attributes from all of the sub-stripes.
|
|
Can we replace this with error handling?
Please do it in OSD, then local object and remote object are getting equal treatment.
|
|
The real reason of this LBUG is because of the race between close(unlink orphan) and setxattr(or setattr). Because close is not protected by ldlm lock. In current implementation(mdd_close()), we use a local lock(mdd_write_lock) to protect the unlink orphan process in the MDD layer, which is fine if the directory is a local directory. But if the directory is a striped directory, the local lock might not be enough, because the unlink orphan (triggered by close) will delete the stripe on the other MDT as well. If other threads do setxattr(or setattr) at the same time on this striped directory. it causes this LBUG.
There are a few options to fix this problem,
1. protect close process with ldlm lock.
2. "Open" the file before setxattr (or setattr), of course only on the server side in MDD layer, then "close" the file after setxattr (or setattr), which might trigger unlink orphan as well.
Either of them might need some changes (not tiny), probably not a good choice for 2.7. Any other suggestions?
|
|
Isn't there a (already broken) rule against taking LDLM locks in close. From mdc_close()
/* To avoid a livelock (bug 7034), we need to send CLOSE RPCs to a
* portal whose threads are not taking any DLM locks and are therefore
* always progressing */
req->rq_request_portal = MDS_READPAGE_PORTAL;
ptlrpc_at_set_req_timeout(req);
Why not fix this by handling the race?
|
|
using LDLM lock has additional benefits as an object can be accessed by another node and LDLM is the natural mechanism to control access.
|
|
Couldn't we just return -ENOENT in this case? The setxattr() call is path-based, and not file descriptor based, so there is no expectation that it is free of races. If the object is deleted then it is perfectly fine to return an error to the caller.
|
|
I had thought the same thing, then I saw LU-6027, and discussed with Li Wei. The problem is if the application open the file and hold the file handle, do setxattr, should we expect succeeds or failure? And it does succeed on ext4? I do not know if posix require this or not? And also if we return -ENOENT here, Li Wei needs to change his new "racer" program and disable sanityn.sh 82.
|
|
> I had thought the same thing, then I saw LU-6027, and discussed with Li Wei. The problem is if the application open the file and hold the file handle, do setxattr, should we expect succeeds or failure? And it does succeed on ext4? I do not know if posix require this or not? And also if we return -ENOENT here, Li Wei needs to change his new "racer" program and disable sanityn.sh 82.
setxattr() or fsetxattr()?
|
|
I meant Li Wei used this file handle thing to set xattr in his new "racer" test program. I am not sure which one he used. But if I returned ENOENT here, it will definitely cause sanityn.sh 82 failed.
|
|
We should be clear about that before we add LDLM locks to close because of it.
Note that http://review.whamcloud.com/#/c/12608/ does not cause -ENOENT to be returned under the following situation.
- lfs mkdir -c4 d0
- fd = open("d0", O_RDONLY)
- rmdir("d0")
- fsetxattr(fd, "user.foo", "bar", 3, 0)
|
|
Yes, right now, it does not return ENOENT, because of the file is still open, and we allow this in current implementation. So is it bad we just return ENOENT for fsetxattr in this case?
|
|
> Yes, right now, it does not return ENOENT, because of the file is still open, and we allow this in current implementation. So is it bad we just return ENOENT for fsetxattr in this case?
Why would we do that?
|
|
I mean If it return ENOENT for unlinked object during setattr or setxattr, then we do not need worry about orphans.
|
|
Why not use http://review.whamcloud.com/#/c/12608/?
|
|
It is different problem. I thought we are talking about about resolving the race, and http://review.whamcloud.com/#/c/12608 is to avoid panic, which actually is covering the problem somehow. IMHO
|
|
It is the same problem. http://review.whamcloud.com/#/c/12608 handles the race.
|
|
Hmm, I thought we suppose to use ldlm lock to protect the race like this, or just not get into setattr or set xattr, once the object is being unlinked.
And also I thought the object should exist once the modification getting into OSD/OSP layer, because it suppose to do sanity check and hold ldlm lock in MDT layer. Though there are some exception for now, like close. I thought we should resolve this in MDT layer, instead of "covering" it in OSD/OSP.
|
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/12608/
Subject: LU-5565 osd-ldiskfs: handle non-existing objects
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 30b0b011dfd53faf8f903401fa0539a8d625b0e5
|
|
Note that osd-zfs still needs to be fixed.
|
|
Patch landed to Master.
|
|
John L. Hammond (john.hammond@intel.com) uploaded a new patch: http://review.whamcloud.com/13496
Subject: LU-5565 osd-ldiskfs: handle nonexisting objects
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0b4fd760b34b1ded5e8fee6db07a5a0f0291389d
|
|
We hit the same trouble with ZFS backend on lola. We need enhance the patch http://review.whamcloud.com/#/c/12608/ for ZFS case.
|
|
this is done in http://review.whamcloud.com/#/c/18024/ and http://review.whamcloud.com/#/c/18155/
|
|
For build '20160201' (master) executed during soaktest (see: https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160201)
the error happened again and persists even after reboot of the node so that the affected
OSS node and OSTs become unusable.
The MDTs have been formated with ldiskfs, OSTs with zfs
Sequence of events:
- Feb 2 01:32:18 2016: lfsck started on mds-0 (lola-8 using the command {{lctl lfsck_start -M soaked-MDT0000 -A }}
OSS nodes lola-2 crashed with LBUG (See see error message after time stamp in console log 'Feb 2 01:30:01')
- Feb 2 01:50:01 2016: lola-2 rebooted; remounted OSTs node crashed with same LBUG
(see error message after 'Feb 2 01:50:01 lola-2 TIME: Time stamp for console' in console log)
- Several tries to simply reboot the node and remount again
- Feb 2 02:13:35 PST 2016: Execution of lctl lfsck_stop -M soaked-MDT0000 -A }} end successfull. No OSTs mounted on {{lola-2
- Feb 2 02:15:01 2016: Mounted OSTs on lola-2. Node crashed almost immediately with same LBUG again. Forced creation of debug log with 'lfsck' enable and buffer size 512M
(see attached file: lustre-log-lu-5565)
- Feb 2 02:31 2016: Rebooted and re-mounted OSTs one more with 'panic_on_lbug', lfsck enabled, debug buffer 512M
Explicitly forced stack trace.
Also debug logs were written after LBUG appear again (almost immediately). See:
lustre-log.1454409077.7931
lustre-log.1454409090.7841
- OSS node can't be used as FS resource anymore
Attached files:
- lola-2 message and console log
- debug-log after lfsck stopped and remount of OSTs
- lustre-log.1454409077.7931, lustre-log.1454409090.7841 - Debug logs of server in current status
when mounting the OSTs
|
|
Please open a new ticket to track any further work that is needed in this area for 2.8
|
Generated at Sat Feb 10 01:52:35 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
console-lola-2.log.bz2