Detect always-skipped test functions due to auth mismatches
!3 (merged) introduced support for per-file/per-directory authentication information.
Some tests use multiple entities (e.g. copying a file to a collection, test_copy_accessible
) and do so by performing a single request, meaning only a single set of authentication information can be part of the request.
Thus, the entities - i.e. info objects - must have the same authentication attached to describe a valid test scenario.
However, for parameterised fixtures, pytest will run all combinations (cartesian product) of parameter values.
The single_auth_info
fixture will automatically discard combinations of parameter values containing different authentications.
It does so by marking the test as skipped, see Mismatching auth infos
in the output.
The skipping behaviour makes reviewing tests complicated:
If there happens to be no single combination of parameter values with the same auth, the underlying test (e.g. test_copy_accessible
) is never executed and this is only visible when looking at passed vs skipped tests.
This is even more complicated, because for the particular trick with the common suite, not all information is shown at the same time (try identifying which function is skipped due to mismatching auths purely from output...).
A possible fix could be to check after the testing completes whether a specific function was always skipped with a mismatching auth message.
AFAIU the way to implement this is through another pytest plugin which registers (pytest docs: writing hook functions) one of the reporting hooks. In the hook, all test results can be scanned to see if for a single test all parameterisations were skipped. If so, at least a warning could be added to the report (what else would be the expected result?).
This would still require recognising whether a test execution was skipped because of a unsuitable parameter combination. This information would have to be encoded in the user facing info string (or perhaps it is possible to attach it to the string instance).
/cc @cw296