7.8 KiB
Canonical harness tests
Installation-root prerequisite
Before executing the suite, provide HARNESS_ROOT in the process environment.
It names the installation root containing harness/ and repositories/, not
the harness/ directory itself. For the current installation, configure the
current agent's $HERMES_HOME/.env with:
HARNESS_ROOT=/opt/data/gitea-agent
The tests consume the environment supplied by Hermes; they do not open or
source .env files, search other profiles, or implement a second configuration
loader. When running outside Hermes, supply the variable explicitly:
export HARNESS_ROOT=/opt/data/gitea-agent
python3 -B "$HARNESS_ROOT/harness/tests/run.py"
On another system, replace the value with that installation's absolute root.
The managed-lock subprocess contention test requires a nonempty, existing
absolute directory and compares its resolved location with the installation
containing the test file. Missing, empty, relative, nonexistent, non-directory,
or different-installation values raise an explicit HARNESS_ROOT error before
creating a lock or starting the contention subprocess. There is no fallback to
Path.cwd() or to the test-file-derived root. Only the validated import root
is passed as a subprocess argument; no .env contents are read or serialized.
This test-only prerequisite does not change the production lock implementation
or the canonical runner's source-bound collection behavior; --collect-only
does not require the variable.
An external-cwd regression runs only the contention test from a temporary
working directory, with PYTHONPATH removed and an explicit fixture
HARNESS_ROOT, using isolated Python and bounded subprocess timeouts. Invalid
configuration coverage checks that neither lock acquisition nor subprocess
execution occurs. It does not recursively invoke the entire suite.
Running the canonical suite
The canonical runner is anchored to its own source tree and works from any current directory. From the repository root:
python3 -B harness/tests/run.py
python3 -B harness/tests/run.py --collect-only
From outside the repository, use the absolute command:
python3 -B /opt/data/gitea-agent/harness/tests/run.py --collect-only
That absolute command is intentionally literal and may be run from / or any
other working directory; do not replace it with a relative path. The canonical
source paths are /opt/data/gitea-agent/harness/tests/run.py and the four role
trees under /opt/data/gitea-agent/harness/tests/{common,spec,impl,review}.
This document is owned by the canonical harness tests and must be updated in
place when their supported discovery contract or command paths change.
Broker fixture contract: in-process broker fixtures bind and connect through
/proc/self/fd/{parent_fd}/{socket_name}, matching the production broker. Long
UTF-8 roots and socket paths are supported; the fixture must not shorten them or
reject them merely because the conventional Unix-domain path encoding exceeds
107 bytes. Fixture setup failures clean up only resources they own, preserve
foreign/replaced paths, close every client and directory descriptor, and retain
unauthenticated-token rejection behavior.
Rework readback contract: the impl role's canonical rework_comment.py snapshot
and verify-live commands collect authenticated GET-only provider state through
rework_readback.py and rework_readback_cli.py. Focused coverage lives in
harness/tests/impl/test_rework_readback.py and
harness/tests/impl/test_rework_readback_transport.py; run them with the existing
harness/tests/impl/test_rework_comment.py regressions. They exercise real
renderer/verifier integration and a local HTTP fixture, not live Gitea access.
Provider/baseline fingerprints, exact identity/endpoints, complete pagination,
create/reuse deltas, metadata causality and failed-receipt preservation are
fail-closed. The workflow contract and canonical runtime artifact paths are in
harness/agents/impl/skills/gitea-issue-rework/references/readback.md.
Initial input-gate coverage is in harness/tests/impl/test_rework_input_gate.py.
It exercises the real public CLI and canonical spec renderer, mixed-role/permutation
and same-role supersession decisions, precise timestamp/tie handling, marker/state/
source binding, complete raw validation/drift, immutable capture and atomic output
races, and new/legacy output budgets. Synthetic transport is GET-only; these
regressions do not claim live Gitea or native deployment execution.
The review native adapter's registered-worktree fixture accepts only exact
gitea-impl-agent or read-only legacy impl creator-role literals. It preserves
state.json bytes/mode/mtime/inode and rejects malformed role values without
relaxing the Issue lease, Git, target, or security guards.
It validates all test*.py files recursively under exactly these required
roles: common, spec, impl, and review. harness/tests, every role, and
every nested test package must have an __init__.py; every role and every
individual test source file must contain at least one statically inventoried
test. Files matching test*.py outside a role are rejected.
Before executing tests, the runner independently inventories source with the
Python AST, collects with standard unittest, and requires exact qualified
ID-set equality. It first preserves the loader's raw ID list so duplicate
loader results cannot disappear through set conversion. It rejects missing or
extra loader IDs, import failures and _FailedTest entries, duplicate class or
method declarations, zero-test files/roles, module/package load_tests,
unsupported conditional discovery declarations, unsupported bases, cyclic
local inheritance, free-function tests, missing package markers, and raw-byte
source/package-manifest drift during collection. The package manifest covers
required, nested, and root markers; it is rescanned after collection so added
or removed files/markers and initializer changes fail closed.
As the first admission step, the runner rejects every symbolic link anywhere
under harness/tests (including test sources, package markers, role
directories, and nested directories) with a canonical exit-2 diagnostic. This
boundary is checked before source bytes are read or test modules are imported;
it prevents source-tree path escape through links. It is not a sandbox: code
that is admitted and imported may still perform ordinary external filesystem,
process, network, or other side effects.
--collect-only emits JSON containing collection_only: true, per-role test
counts, sorted exact unittest IDs, and source SHA-256 hashes. Collection-only
success is not runtime success: normal unittest assertion failures and errors
remain visible and return a nonzero status. The runner intentionally accepts
standard unittest.TestCase, aliases, local mixins/inheritance, inherited
test* methods, and normal decorators such as @unittest.skip.
The inventory is deliberately conservative, not a Python semantic evaluator or security sandbox. Test classes/functions declared inside module/class control flow are unsupported and rejected; helper functions/classes nested inside an ordinary test method are runtime fixture code and are not inspected as canonical discovery declarations. Tests supplied solely by dynamic imports, metaclasses, or custom discovery protocols are unsupported: their collected IDs must not substitute for or extend the statically declared inventory. Imports used by ordinary fixture helpers are not prohibited. The harness does not claim to prevent side effects performed by imported test code; it detects source drift and validates collection before execution.
The regression tests use bounded subprocess timeouts and clean test-owned
TemporaryDirectory trees under /opt/data/cache/scratch/canonical-tests-main/fix2/.
Run them with:
python3 -m unittest harness.tests.common.test_canonical_test_runner -v