95 lines
5.2 KiB
Markdown
95 lines
5.2 KiB
Markdown
# Canonical harness tests
|
|
|
|
The canonical runner is anchored to its own source tree and works from any
|
|
current directory. From the repository root:
|
|
|
|
```sh
|
|
python3 -B harness/tests/run.py
|
|
python3 -B harness/tests/run.py --collect-only
|
|
```
|
|
|
|
From outside the repository, use the absolute command:
|
|
|
|
```sh
|
|
python3 -B /opt/data/gitea-agent/harness/tests/run.py --collect-only
|
|
```
|
|
|
|
That absolute command is intentionally literal and may be run from `/` or any
|
|
other working directory; do not replace it with a relative path. The canonical
|
|
source paths are `/opt/data/gitea-agent/harness/tests/run.py` and the four role
|
|
trees under `/opt/data/gitea-agent/harness/tests/{common,spec,impl,review}`.
|
|
This document is owned by the canonical harness tests and must be updated in
|
|
place when their supported discovery contract or command paths change.
|
|
|
|
Broker fixture contract: in-process broker fixtures bind and connect through
|
|
`/proc/self/fd/{parent_fd}/{socket_name}`, matching the production broker. Long
|
|
UTF-8 roots and socket paths are supported; the fixture must not shorten them or
|
|
reject them merely because the conventional Unix-domain path encoding exceeds
|
|
107 bytes. Fixture setup failures clean up only resources they own, preserve
|
|
foreign/replaced paths, close every client and directory descriptor, and retain
|
|
unauthenticated-token rejection behavior.
|
|
|
|
Rework readback contract: the impl role's canonical `rework_comment.py snapshot`
|
|
and `verify-live` commands collect authenticated GET-only provider state through
|
|
`rework_readback.py` and `rework_readback_cli.py`. Focused coverage lives in
|
|
`harness/tests/impl/test_rework_readback.py` and
|
|
`harness/tests/impl/test_rework_readback_transport.py`; run them with the existing
|
|
`harness/tests/impl/test_rework_comment.py` regressions. They exercise real
|
|
renderer/verifier integration and a local HTTP fixture, not live Gitea access.
|
|
Provider/baseline fingerprints, exact identity/endpoints, complete pagination,
|
|
create/reuse deltas, metadata causality and failed-receipt preservation are
|
|
fail-closed. The workflow contract and canonical runtime artifact paths are in
|
|
`harness/agents/impl/skills/gitea-issue-rework/references/readback.md`.
|
|
|
|
It validates all `test*.py` files recursively under exactly these required
|
|
roles: `common`, `spec`, `impl`, and `review`. `harness/tests`, every role, and
|
|
every nested test package must have an `__init__.py`; every role and every
|
|
individual test source file must contain at least one statically inventoried
|
|
test. Files matching `test*.py` outside a role are rejected.
|
|
|
|
Before executing tests, the runner independently inventories source with the
|
|
Python AST, collects with standard `unittest`, and requires exact qualified
|
|
ID-set equality. It first preserves the loader's raw ID list so duplicate
|
|
loader results cannot disappear through set conversion. It rejects missing or
|
|
extra loader IDs, import failures and `_FailedTest` entries, duplicate class or
|
|
method declarations, zero-test files/roles, module/package `load_tests`,
|
|
unsupported conditional discovery declarations, unsupported bases, cyclic
|
|
local inheritance, free-function tests, missing package markers, and raw-byte
|
|
source/package-manifest drift during collection. The package manifest covers
|
|
required, nested, and root markers; it is rescanned after collection so added
|
|
or removed files/markers and initializer changes fail closed.
|
|
|
|
As the first admission step, the runner rejects every symbolic link anywhere
|
|
under `harness/tests` (including test sources, package markers, role
|
|
directories, and nested directories) with a canonical exit-2 diagnostic. This
|
|
boundary is checked before source bytes are read or test modules are imported;
|
|
it prevents source-tree path escape through links. It is not a sandbox: code
|
|
that is admitted and imported may still perform ordinary external filesystem,
|
|
process, network, or other side effects.
|
|
|
|
`--collect-only` emits JSON containing `collection_only: true`, per-role test
|
|
counts, sorted exact unittest IDs, and source SHA-256 hashes. Collection-only
|
|
success is not runtime success: normal unittest assertion failures and errors
|
|
remain visible and return a nonzero status. The runner intentionally accepts
|
|
standard `unittest.TestCase`, aliases, local mixins/inheritance, inherited
|
|
`test*` methods, and normal decorators such as `@unittest.skip`.
|
|
|
|
The inventory is deliberately conservative, not a Python semantic evaluator or
|
|
security sandbox. Test classes/functions declared inside module/class control
|
|
flow are unsupported and rejected; helper functions/classes nested inside an
|
|
ordinary test method are runtime fixture code and are not inspected as
|
|
canonical discovery declarations. Tests supplied solely by dynamic imports,
|
|
metaclasses, or custom discovery protocols are unsupported: their collected
|
|
IDs must not substitute for or extend the statically declared inventory.
|
|
Imports used by ordinary fixture helpers are not prohibited. The harness does
|
|
not claim to prevent side effects performed by imported test code; it detects
|
|
source drift and validates collection before execution.
|
|
|
|
The regression tests use bounded subprocess timeouts and clean test-owned
|
|
`TemporaryDirectory` trees under `/opt/data/cache/scratch/canonical-tests-main/fix2/`.
|
|
Run them with:
|
|
|
|
```sh
|
|
python3 -m unittest harness.tests.common.test_canonical_test_runner -v
|
|
```
|