so OBD_RACE() is supposed to stop for the first caller and then continue when somebody else reaches there.
Yet we see both threads stuck in the same place in OBD_RACE? Something strange is going on here. Also there's only one print in the logs about the OBD_RACE actually being hit
There's a bug in the test:
#define OBD_FAIL_TGT_MOUNT_RACE 0x716
do_facet ost1 "$LCTL set_param fail_loc=0x716"
start ost1 $(ostdevname 1) $OST_MOUNT_OPTS &
pid=$!
start ost1 $(ostdevname 1) $OST_MOUNT_OPTS &
do_facet ost1 "$LCTL set_param fail_loc=0x0"
pid2=$!
Since both mount commands are asynchronously, we may clear fail_loc between the two invocations of OBD_RACE(). In this case both threads will hang.
> Also there's only one print in the logs about the OBD_RACE actually being hit
Shouldn't we expect that CDEBUG rate limitation will prevent the second message from being printed.
And we need CFS_FAIL_ONCE along with OBD_FAIL_TGT_MOUNT_RACE when setting fail_loc.
John, since I am the "responsible" of this test and associated code, I feel a bit interested by this problem and ticket 
My understanding was that OBD_RACE()/CFS_RACE()/cfs_race() only/safely work with 2 threads, the 1st stops waiting for the 2nd to awake it.
And I think, like you, that for this to work as expected, fail_loc must not be cleared in between, when there is a potential window for this to happen due to the second mount/thread to be run asynchronously, with the clear of fail_loc being launched simultaneously.
On the other hand, I thought before that CFS_FAIL_ONCE was useless for OBD_RACE() to work, but after better regarding the code in __cfs_fail_check_set() core routine this may also, as you point, be the reason for both threads being stuck if they are closely racing and have entered the routine at around the same time, and particularly if CFS_FAILED bit has still not been set.
Do you want me to push a patch to fix these 2 holes (4 in fact since fail_loc is wrongly set/cleared for MDSs and OSSs cases) in test ?
> Do you want me to push a patch to fix these 2 holes (4 in fact since fail_loc is wrongly set/cleared for MDSs and OSSs cases) in test ?
Yes, please.
I'm still not confident that I understand how this is working but I can see several potential issues with this code: cfs_race_state should be declared volatile. There is a race between the two threads setting cfs_race_state and one sleeping the other waking. proc_fail_loc() calls wake_up(&cfs_race_waitq) but it does not set cfs_race_state to 1 so it seems like the sleeping task would just go back to sleep.
My best guess is that usually "do_facet ost1 $LCTL set_param fail_loc=0x0" is running before either mount command so the CFS_RACE() is never triggered.
Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: http://review.whamcloud.com/21117
Subject: LU-8295 tests: comply with OBD_RACE() behavior
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e3050d4414667a941c501728de2bfb0b78e8b097
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/21117/
Subject: LU-8295 tests: comply with OBD_RACE() behavior
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a3becc1092524366e13b3c72d61c6b0ea82709f7
Landed for 2.10