[verified] Add canonical harness test runner and enforce role collection

2026-10-11 13:46:34 +00:00
parent d46f5c4a09
commit 5c3bffaf71
8 changed files with 696 additions and 23 deletions
@@ -0,0 +1,74 @@
+# Canonical harness tests
+
+The canonical runner is anchored to its own source tree and works from any
+current directory. From the repository root:
+
+```sh
+python3 -B harness/tests/run.py
+python3 -B harness/tests/run.py --collect-only
+```
+
+From outside the repository, use the absolute command:
+
+```sh
+python3 -B /opt/data/gitea-agent/harness/tests/run.py --collect-only
+```
+
+That absolute command is intentionally literal and may be run from `/` or any
+other working directory; do not replace it with a relative path. The canonical
+source paths are `/opt/data/gitea-agent/harness/tests/run.py` and the four role
+trees under `/opt/data/gitea-agent/harness/tests/{common,spec,impl,review}`.
+This document is owned by the canonical harness tests and must be updated in
+place when their supported discovery contract or command paths change.
+
+It validates all `test*.py` files recursively under exactly these required
+roles: `common`, `spec`, `impl`, and `review`. `harness/tests`, every role, and
+every nested test package must have an `__init__.py`; every role and every
+individual test source file must contain at least one statically inventoried
+test. Files matching `test*.py` outside a role are rejected.
+
+Before executing tests, the runner independently inventories source with the
+Python AST, collects with standard `unittest`, and requires exact qualified
+ID-set equality. It first preserves the loader's raw ID list so duplicate
+loader results cannot disappear through set conversion. It rejects missing or
+extra loader IDs, import failures and `_FailedTest` entries, duplicate class or
+method declarations, zero-test files/roles, module/package `load_tests`,
+unsupported conditional discovery declarations, unsupported bases, cyclic
+local inheritance, free-function tests, missing package markers, and raw-byte
+source/package-manifest drift during collection. The package manifest covers
+required, nested, and root markers; it is rescanned after collection so added
+or removed files/markers and initializer changes fail closed.
+
+As the first admission step, the runner rejects every symbolic link anywhere
+under `harness/tests` (including test sources, package markers, role
+directories, and nested directories) with a canonical exit-2 diagnostic. This
+boundary is checked before source bytes are read or test modules are imported;
+it prevents source-tree path escape through links. It is not a sandbox: code
+that is admitted and imported may still perform ordinary external filesystem,
+process, network, or other side effects.
+
+`--collect-only` emits JSON containing `collection_only: true`, per-role test
+counts, sorted exact unittest IDs, and source SHA-256 hashes. Collection-only
+success is not runtime success: normal unittest assertion failures and errors
+remain visible and return a nonzero status. The runner intentionally accepts
+standard `unittest.TestCase`, aliases, local mixins/inheritance, inherited
+`test*` methods, and normal decorators such as `@unittest.skip`.
+
+The inventory is deliberately conservative, not a Python semantic evaluator or
+security sandbox. Test classes/functions declared inside module/class control
+flow are unsupported and rejected; helper functions/classes nested inside an
+ordinary test method are runtime fixture code and are not inspected as
+canonical discovery declarations. Tests supplied solely by dynamic imports,
+metaclasses, or custom discovery protocols are unsupported: their collected
+IDs must not substitute for or extend the statically declared inventory.
+Imports used by ordinary fixture helpers are not prohibited. The harness does
+not claim to prevent side effects performed by imported test code; it detects
+source drift and validates collection before execution.
+
+The regression tests use bounded subprocess timeouts and clean test-owned
+`TemporaryDirectory` trees under `/opt/data/cache/scratch/canonical-tests-main/fix2/`.
+Run them with:
+
+```sh
+python3 -m unittest harness.tests.common.test_canonical_test_runner -v
+```