4.4 KiB
Canonical harness tests
The canonical runner is anchored to its own source tree and works from any current directory. From the repository root:
python3 -B harness/tests/run.py
python3 -B harness/tests/run.py --collect-only
From outside the repository, use the absolute command:
python3 -B /opt/data/gitea-agent/harness/tests/run.py --collect-only
That absolute command is intentionally literal and may be run from / or any
other working directory; do not replace it with a relative path. The canonical
source paths are /opt/data/gitea-agent/harness/tests/run.py and the four role
trees under /opt/data/gitea-agent/harness/tests/{common,spec,impl,review}.
This document is owned by the canonical harness tests and must be updated in
place when their supported discovery contract or command paths change.
Broker fixture contract: in-process broker fixtures bind and connect through
/proc/self/fd/{parent_fd}/{socket_name}, matching the production broker. Long
UTF-8 roots and socket paths are supported; the fixture must not shorten them or
reject them merely because the conventional Unix-domain path encoding exceeds
107 bytes. Fixture setup failures clean up only resources they own, preserve
foreign/replaced paths, close every client and directory descriptor, and retain
unauthenticated-token rejection behavior.
It validates all test*.py files recursively under exactly these required
roles: common, spec, impl, and review. harness/tests, every role, and
every nested test package must have an __init__.py; every role and every
individual test source file must contain at least one statically inventoried
test. Files matching test*.py outside a role are rejected.
Before executing tests, the runner independently inventories source with the
Python AST, collects with standard unittest, and requires exact qualified
ID-set equality. It first preserves the loader's raw ID list so duplicate
loader results cannot disappear through set conversion. It rejects missing or
extra loader IDs, import failures and _FailedTest entries, duplicate class or
method declarations, zero-test files/roles, module/package load_tests,
unsupported conditional discovery declarations, unsupported bases, cyclic
local inheritance, free-function tests, missing package markers, and raw-byte
source/package-manifest drift during collection. The package manifest covers
required, nested, and root markers; it is rescanned after collection so added
or removed files/markers and initializer changes fail closed.
As the first admission step, the runner rejects every symbolic link anywhere
under harness/tests (including test sources, package markers, role
directories, and nested directories) with a canonical exit-2 diagnostic. This
boundary is checked before source bytes are read or test modules are imported;
it prevents source-tree path escape through links. It is not a sandbox: code
that is admitted and imported may still perform ordinary external filesystem,
process, network, or other side effects.
--collect-only emits JSON containing collection_only: true, per-role test
counts, sorted exact unittest IDs, and source SHA-256 hashes. Collection-only
success is not runtime success: normal unittest assertion failures and errors
remain visible and return a nonzero status. The runner intentionally accepts
standard unittest.TestCase, aliases, local mixins/inheritance, inherited
test* methods, and normal decorators such as @unittest.skip.
The inventory is deliberately conservative, not a Python semantic evaluator or security sandbox. Test classes/functions declared inside module/class control flow are unsupported and rejected; helper functions/classes nested inside an ordinary test method are runtime fixture code and are not inspected as canonical discovery declarations. Tests supplied solely by dynamic imports, metaclasses, or custom discovery protocols are unsupported: their collected IDs must not substitute for or extend the statically declared inventory. Imports used by ordinary fixture helpers are not prohibited. The harness does not claim to prevent side effects performed by imported test code; it detects source drift and validates collection before execution.
The regression tests use bounded subprocess timeouts and clean test-owned
TemporaryDirectory trees under /opt/data/cache/scratch/canonical-tests-main/fix2/.
Run them with:
python3 -m unittest harness.tests.common.test_canonical_test_runner -v