It seems that tests in ALWAYS_EXCEPT are reported in maloo as "skip ALWAYS excluded test NN". However, if some test is skipped due to an environmental condition (e.g. sanity.sh test_27m, which is testing out-of-space conditions and is skipped during review runs when the filesystem is too large, as in https://maloo.whamcloud.com/sub_tests/78d60ce0-9dc0-11e1-a1d8-52540035b04c) this will print a "SKIP" message in the test, but the test itself is reported as "PASS" Maloo.
It would be more useful if the test were reported in Maloo as "SKIP: sanity test_27m 892015396 > 1400000 skipping out-of-space test on OST0". What does auster/Maloo key on to determine whether a test is SKIP or PASS, and what needs to be done to make the "skip" directive inside a test cause Maloo to report the test as skipped?
The reason this is of interest is because I want the tests skipped because of
LU-482 to be clearly flagged as not having been run, rather than pretending the test has passed. The patch in http://review.whamcloud.com/2731 will mark the test with "skip", but until this issue is fixed then the whole replay-dual will appear to be a PASS, even though nothing was run.