gitea-agent/harness/tests/README.md

# Canonical harness tests

## Installation-root prerequisite

Before executing the suite, provide `HARNESS_ROOT` in the process environment.
It names the installation root containing `harness/` and `repositories/`, not
the `harness/` directory itself. For the current installation, configure the
current agent's `$HERMES_HOME/.env` with:

```dotenv
HARNESS_ROOT=/opt/data/gitea-agent
```

The tests consume the environment supplied by Hermes; they do not open or
source `.env` files, search other profiles, or implement a second configuration
loader. When running outside Hermes, supply the variable explicitly:

```sh
export HARNESS_ROOT=/opt/data/gitea-agent
python3 -B "$HARNESS_ROOT/harness/tests/run.py"
```

On another system, replace the value with that installation's absolute root.
The managed-lock subprocess contention test requires a nonempty, existing
absolute directory and compares its resolved location with the installation
containing the test file. Missing, empty, relative, nonexistent, non-directory,
or different-installation values raise an explicit `HARNESS_ROOT` error before
creating a lock or starting the contention subprocess. There is no fallback to
`Path.cwd()` or to the test-file-derived root. Only the validated import root
is passed as a subprocess argument; no `.env` contents are read or serialized.
This test-only prerequisite does not change the production lock implementation
or the canonical runner's source-bound collection behavior; `--collect-only`
does not require the variable.

An external-cwd regression runs only the contention test from a temporary
working directory, with `PYTHONPATH` removed and an explicit fixture
`HARNESS_ROOT`, using isolated Python and bounded subprocess timeouts. Invalid
configuration coverage checks that neither lock acquisition nor subprocess
execution occurs. It does not recursively invoke the entire suite.

## Running the canonical suite

The canonical runner is anchored to its own source tree and works from any
current directory. From the repository root:

```sh
python3 -B harness/tests/run.py
python3 -B harness/tests/run.py --collect-only
```

From outside the repository, use the absolute command:

```sh
python3 -B /opt/data/gitea-agent/harness/tests/run.py --collect-only
```

That absolute command is intentionally literal and may be run from `/` or any
other working directory; do not replace it with a relative path. The canonical
source paths are `/opt/data/gitea-agent/harness/tests/run.py` and the four role
trees under `/opt/data/gitea-agent/harness/tests/{common,spec,impl,review}`.
This document is owned by the canonical harness tests and must be updated in
place when their supported discovery contract or command paths change.

Broker fixture contract: in-process broker fixtures bind and connect through
`/proc/self/fd/{parent_fd}/{socket_name}`, matching the production broker. Long
UTF-8 roots and socket paths are supported; the fixture must not shorten them or
reject them merely because the conventional Unix-domain path encoding exceeds
107 bytes. Fixture setup failures clean up only resources they own, preserve
foreign/replaced paths, close every client and directory descriptor, and retain
unauthenticated-token rejection behavior.

Rework readback contract: the impl role's canonical `rework_comment.py snapshot`
and `verify-live` commands collect authenticated GET-only provider state through
`rework_readback.py` and `rework_readback_cli.py`. Focused coverage lives in
the provider/cache/bridge modules under `harness/tests/impl/rework/` and
`harness/tests/impl/rework/test_rework_readback_transport.py`; run them with the
`harness/tests/impl/rework/test_rework_comment.py` regressions. They exercise real
renderer/verifier integration and a local HTTP fixture, not live Gitea access.
Provider/baseline fingerprints, exact identity/endpoints, complete pagination,
create/reuse deltas, metadata causality and failed-receipt preservation are
fail-closed. The workflow contract and canonical runtime artifact paths are in
`harness/agents/impl/skills/gitea-issue-rework/references/readback.md`.

Initial input-gate coverage is in the `test_rework_input_gate_*.py` modules
under `harness/tests/impl/rework/`.
It exercises the real public CLI and canonical spec renderer, mixed-role/permutation
and same-role supersession decisions, precise timestamp/tie handling, marker/state/
source binding, complete raw validation/drift, immutable capture and atomic output
races, and new/legacy output budgets. Synthetic transport is GET-only; these
regressions do not claim live Gitea or native deployment execution.

The review native adapter's registered-worktree fixture accepts only exact
`gitea-impl-agent` or read-only legacy `impl` creator-role literals. It preserves
`state.json` bytes/mode/mtime/inode and rejects malformed role values without
relaxing the Issue lease, Git, target, or security guards.

It validates all `test*.py` files recursively under exactly these required
roles: `common`, `spec`, `impl`, and `review`. `harness/tests`, every role, and
every nested test package must have an `__init__.py`; every role and every
individual test source file must contain at least one statically inventoried
test. Files matching `test*.py` outside a role are rejected.

Before executing tests, the runner independently inventories source with the
Python AST, collects with standard `unittest`, and requires exact qualified
ID-set equality. It first preserves the loader's raw ID list so duplicate
loader results cannot disappear through set conversion. It rejects missing or
extra loader IDs, import failures and `_FailedTest` entries, duplicate class or
method declarations, zero-test files/roles, module/package `load_tests`,
unsupported conditional discovery declarations, unsupported bases, cyclic
local inheritance, free-function tests, missing package markers, and raw-byte
source/package-manifest drift during collection. The package manifest covers
required, nested, and root markers; it is rescanned after collection so added
or removed files/markers and initializer changes fail closed.

As the first admission step, the runner rejects every symbolic link anywhere
under `harness/tests` (including test sources, package markers, role
directories, and nested directories) with a canonical exit-2 diagnostic. This
boundary is checked before source bytes are read or test modules are imported;
it prevents source-tree path escape through links. It is not a sandbox: code
that is admitted and imported may still perform ordinary external filesystem,
process, network, or other side effects.

`--collect-only` emits JSON containing `collection_only: true`, per-role test
counts, sorted exact unittest IDs, and source SHA-256 hashes. Collection-only
success is not runtime success: normal unittest assertion failures and errors
remain visible and return a nonzero status. The runner intentionally accepts
standard `unittest.TestCase`, aliases, local mixins/inheritance, inherited
`test*` methods, and normal decorators such as `@unittest.skip`.

The inventory is deliberately conservative, not a Python semantic evaluator or
security sandbox. Test classes/functions declared inside module/class control
flow are unsupported and rejected; helper functions/classes nested inside an
ordinary test method are runtime fixture code and are not inspected as
canonical discovery declarations. Tests supplied solely by dynamic imports,
metaclasses, or custom discovery protocols are unsupported: their collected
IDs must not substitute for or extend the statically declared inventory.
Imports used by ordinary fixture helpers are not prohibited. The harness does
not claim to prevent side effects performed by imported test code; it detects
source drift and validates collection before execution.

The regression tests use bounded subprocess timeouts and clean test-owned
`TemporaryDirectory` trees under `/opt/data/cache/scratch/canonical-tests-main/fix2/`.
Run them with:

```sh
python3 -B -m unittest harness.tests.common.runner.test_canonical_test_runner -v
```

## Domain ownership and verification levels

The canonical output of this documentation is `harness/tests/README.md` in this
repository. Update this file in place rather than publishing a parallel test
inventory document. The canonical runner remains `harness/tests/run.py`;
there is no alternate discovery framework or hand-maintained source-hash list.

- `common/runner`: source admission and exact discovery/exit-status contracts.
- `common/locking`: primitive flock and authenticated broker lifecycle remain
  separate modules; neither substitutes for the other.
- `common/repository`: prefetch, runtime/verification roots, and Git source binding.
- `common/delegation`: evidence, adoption, and real-Git consumer integration.
- `common/evidence`: workflow evidence admission and conditional pytest recipe.
- `common/access`: local credential protocol recipe, never real credentials.
- `common/contracts`: cross-role documentation wiring, not live workflow proof.
- `impl/delivery`: coordinator, initial delivery, reconciliation, and response.
- `impl/rework`: intake, decision, rendering, provider readback, publication,
  and separately authorized correction.
- `impl/evidence`: manifest/schema, filesystem/Git admission, and bounded
  command/process-group execution (distinct verification boundaries).
- `impl/contracts`: documentation wiring and separately named comparator
  execution extracted from the actual canonical documentation.
- `review/native`: native publication/plan, transport, lease/worktree admission,
  and durable resume; GET/POST loopback fixtures are not live Gitea publication.
- `review/journal`: binding/presence safety, raw-byte legacy import, and actual
  subprocess crash/locking/race tests. Archive import is not fresh readback.
- `review/evidence`: execution of the actual native evidence recipe.
- `review/contracts`: policy-specific classes consolidated in one documentation
  module; string/link checks do not certify production behavior.
- `spec/support`: result codec, cache admission, publication/readback, and AC checker.
- `spec/projection`: source-Issue reference and requested-reviewer projection
  contracts remain independent modules with their own strict input matrices.
- `spec/contracts`: documentation contracts plus the separately named
  `test_actual_collector.py` executable collector regression. That regression
  calls the real collector; it is not a local-model documentation example.

### Documentation regression boundary

General workflow guidance is not an exact-prose regression interface. Do not
pin advisory sentences or generic review/reporting guidance with substring
assertions. Keep executable behavior and explicit technical contracts: CLI
options, schema fields, canonical paths/links, recipe structure, and required
procedure ordering. In mixed tests, remove only the prose assertions rather
than deleting the executable or structural checks.

The earlier `20261015T232331Z-final` traceability below is historical evidence
for that source snapshot, not an index of the current tree. Subsequent removal
runs under `logs/test-removal/` preserve their own baseline, exact dispositions,
source hashes and observed outcomes; never relabel old outcomes as current.
The general-guidance cleanup is recorded in
`logs/test-removal/general-guidance-20261016T054315Z/`.

### Shared fixture boundaries

`_support/` holds ordinary data/transport/resource builders only. It must not
contain or re-export TestCase classes, imported test bases, generated tests, or
`load_tests`. Root derivation is centralized in `_support/paths.py`; helpers do
not use `HARNESS_ROOT` or cwd to select canonical test sources. Resource builders
own explicit cleanup. Meaningful expected results, no-call/no-write/no-retry,
identity/source/stage and mode/inode/bytes assertions stay in the tests. Import
fixture helpers rather than importing another test module or manually invoking
another TestCase's `setUp()`/`tearDown()`.

### Focused execution and evidence limits

From the repository root, for example:

```sh
HARNESS_ROOT=/opt/data/gitea-agent python3 -B -m unittest discover -s harness/tests/common -t .
HARNESS_ROOT=/opt/data/gitea-agent python3 -B -m unittest discover -s harness/tests/impl -t .
HARNESS_ROOT=/opt/data/gitea-agent python3 -B -m unittest discover -s harness/tests/review -t .
HARNESS_ROOT=/opt/data/gitea-agent python3 -B -m unittest discover -s harness/tests/spec -t .
```

These are focused runtime checks, not substitutes for the canonical admission
and full execution command above. Compilation is not test execution;
`--collect-only` is not runtime success. Source hashes and counts are emitted
from the actual current source tree, never manually adjusted to admit a move.

- Documentation contracts read canonical documents (and required workflow
  references through `_support/contracts.py`); they emit assertions, not a live
  deployment/publication receipt.
- Unit tests call actual helpers with bounded synthetic inputs; they certify
  their stated input/side-effect boundaries, not provider behavior in general.
- Git/HTTP integration tests create local temporary repositories or loopback
  servers. They exercise real Git or transport code, not authenticated Gitea.
- Process tests exercise actual child processes/crash/locks and preserve durable
  state assertions. In-process fakes or archive replay do not replace these.
- Evidence recipe tests execute the canonical recipe in test-owned temporary
  roots. Their manifests/logs are local synthetic evidence, not product delivery.
- Two pytest recipe execution cases skip when this interpreter has no admitted
  pytest installation (`admitted pytest interpreter required`). The documentation
  case still runs. No dependency is installed, skip removed, or warning/failure
  summary suppressed merely to obtain a green run.

### Test-to-artifact scope

Artifact scope is local and test-owned unless explicitly stated otherwise.
Temporary files, synthetic repositories, loopback traffic and child-process
journals are cleaned up by fixtures; they are not retained delivery evidence.
The retained canonical runtime log records each exact test ID and outcome.
The following groups specify what each module's tests may produce or inspect;
a passing group does not certify every stronger evidence layer listed elsewhere.

- `common/access/test_git_credential_recipe.py`: local helper/askpass scripts,
  permissions and protocol stdout; no stored or real provider credentials.
- `common/contracts/test_*.py`: canonical Markdown sections, workflow links and
  documentation assertions; no publication receipt or workflow execution.
- `common/delegation/test_delegation_evidence*.py`: test-owned declarations,
  source snapshots, bundle hashes and verifier decisions. Adoption tests add
  fresh admission results; consumer unit tests add adapter-call records.
  `test_delegation_consumers_integration.py` adds actual temporary Git evidence,
  CLI stdout/stderr and fake consumer effects, not live Gitea writes.
- `common/evidence/test_workflow_guard.py`: local contract/test inventories,
  snapshots, change decisions and fault inputs. `test_pytest_evidence_recipe.py`
  inspects the canonical recipe and conditionally creates actual pytest
  logs/summaries; skipped execution produces no recipe execution evidence.
- `common/locking/test_managed_lock.py`: local lock files/descriptors and
  contention subprocess results. `test_managed_lock_broker.py`: private socket,
  lease/token/lifecycle state and process results; not a remote Issue lock.
- `common/repository/test_repository_prefetch.py`: temporary local Git/cache
  fixtures. Root tests produce local ownership/path/cleanup state;
  `test_test_inventory_binding.py` produces source-bound inventories and binding
  decisions, not semantic test-coverage proof.
- `common/runner/test_canonical_test_runner.py`: temporary source trees,
  canonical collection JSON/diagnostics and subprocess exit codes, including
  negative admission inputs. This is runner behavior, not product delivery.
- `impl/contracts/test_impl_architecture_wiring.py`: canonical document/link
  assertions only. `test_impl_architecture_comparator.py`: comparator code
  extracted from that document, synthetic inputs and comparator decisions.
- `impl/delivery/test_impl_operation.py`, `test_implementation_delivery.py`
  and `test_implementation_operation_integration.py`: local operation plans,
  evidence/adoption records and adapter call/effect ordering.
  `test_delivery_response.py` and `test_delivery_reconciliation.py` add raised
  HTTP errors or loopback response/reconciliation fixtures and completion
  decisions; neither authorizes retrying an uncertain remote write.
- `impl/evidence/test_evidence_manifest_schema.py`: plan/manifest JSON,
  command-admission diagnostics and bounded command outputs.
  `test_evidence_manifest_filesystem_git.py`: real temporary Git identities,
  artifact bytes/paths and manifest validation.
  `test_evidence_command_deadlines.py`: actual child/process-group deadline
  behavior and local command logs; separate from Git topology validation.
- `impl/rework/test_rework_input_gate_*.py`: schema inputs, CLI output files,
  role/timestamp decisions and counter/publication-phase diagnostics using
  canonical production entry points and synthetic provider data.
- `impl/rework/test_rework_readback_provider.py`: provider-shaped snapshots,
  pagination and normalization decisions. Cache tests add strict-cache records;
  bridge/transport tests add renderer/CLI or loopback HTTP artifacts. These are
  not authenticated fresh Gitea readbacks.
- `impl/rework/test_rework_comment.py`, `test_rework_correction.py` and
  `test_rework_publication.py`: local rendered comments, correction/publication
  plans, synthetic readback/receipt and no-write/no-retry call records.
- `review/contracts/test_review_document_contracts.py`: actual policy document
  assertions only; no native review publication or observability run receipt.
- `review/evidence/test_native_evidence_recipe.py`: actual recipe execution in
  temporary roots, evidence plans/manifests/logs and archive-path rejection;
  not a product delivery manifest or live provider verification.
- `review/journal/test_journal_safety.py` and
  `test_journal_filesystem_safety.py`: private binding/presence records,
  byte/mode/path integrity and corruption decisions.
  `test_legacy_journal_contract.py`: source raw bytes, registration receipt,
  imported legacy archive and zero-POST assertions; imported archives do not
  establish fresh verification. `test_publication_processes.py`: actual child
  crashes, lock contention and durable POST-counter/journal state.
- `review/native/test_native_publication.py`: synthetic plans/snapshots,
  journal/receipt and fake API call counts. Lease tests add local worktree/lease
  admission state; resume tests add durable pending-intent/readback failures;
  transport tests add local HTTP request/response and adapter artifacts.
- `spec/contracts/test_priority_review_contracts.py`: actual canonical documents
  and links, not execution of local-model examples.
  `test_actual_collector.py`: actual collector callback arguments/results.
- `spec/projection/test_requested_reviewer.py` and
  `test_source_issue_reference.py`: distinct snapshot/projection inputs,
  production comparison results, CLI stdout/stderr and unchanged-file checks.
- `spec/support/test_codec.py`: rendered/parsed result bytes and CLI decisions;
  `test_cache.py`: private cache/path/ownership admission;
  `test_publication.py`: synthetic exact readback/publication comparisons;
  `test_ac_checker.py`: actual classification CLI/input digest decisions and
  canonical documentation assertions. None is live authenticated publication.

Exact per-test IDs, baseline predecessors, source anchors, artifact scope,
verification layers and actual execution outcomes are retained in the run's
`traceability.json`; `inventory-summary.json` distinguishes documentation,
production behavior and overlapping process/integration evidence rather than
adding incompatible counts. For this cleanup the run directory is
`logs/regression-test-implementation/20261015T232331Z-final/`.
Rebuild that index after source/test-ID changes; fail closed on an unmapped ID,
missing outcome or source-hash mismatch. Primary classifications come from
source assertions/calls and fixture ownership. Detailed Git/HTTP/process/archive
labels are conservative source-confirmed subsets, not exhaustive dynamic
coverage counts; indirect helper calls may add work beyond those labels. Layer
counts overlap, and a declared execution layer is not executed evidence when
its recorded outcome is skipped.

The final raw collection/runtime inputs are preserved in that run directory's
`acceptance-final/` subdirectory. After reviewing any classification changes,
rebuild the exact join with:

```sh
python3 -B logs/regression-test-implementation/20261015T232331Z-final/refine_classification.py
python3 -B logs/regression-test-implementation/20261015T232331Z-final/build_traceability.py
```

These are run-specific evidence tools, not a replacement canonical runner.
Do not infer coverage from a filename, a successful collection or the mere
existence of an artifact.

The cleanup's original-to-current exact ID dispositions and run-specific counts,
verification-level inventory, collection JSON, fault-injection controls and full
execution logs are preserved under
`logs/regression-test-implementation/20261015/`. The original 659-ID baseline
under `logs/regression-test-audit/20261015/` is immutable. Counts in run evidence
refer to actual methods (subTest branches are not additional tests); domain and
verification-level labels do not make one test a substitute for another.