test(harness): reorganize regression suites and preserve contract coverage
This commit is contained in:
+221
-5
@@ -72,16 +72,17 @@ unauthenticated-token rejection behavior.
|
||||
Rework readback contract: the impl role's canonical `rework_comment.py snapshot`
|
||||
and `verify-live` commands collect authenticated GET-only provider state through
|
||||
`rework_readback.py` and `rework_readback_cli.py`. Focused coverage lives in
|
||||
`harness/tests/impl/test_rework_readback.py` and
|
||||
`harness/tests/impl/test_rework_readback_transport.py`; run them with the existing
|
||||
`harness/tests/impl/test_rework_comment.py` regressions. They exercise real
|
||||
the provider/cache/bridge modules under `harness/tests/impl/rework/` and
|
||||
`harness/tests/impl/rework/test_rework_readback_transport.py`; run them with the
|
||||
`harness/tests/impl/rework/test_rework_comment.py` regressions. They exercise real
|
||||
renderer/verifier integration and a local HTTP fixture, not live Gitea access.
|
||||
Provider/baseline fingerprints, exact identity/endpoints, complete pagination,
|
||||
create/reuse deltas, metadata causality and failed-receipt preservation are
|
||||
fail-closed. The workflow contract and canonical runtime artifact paths are in
|
||||
`harness/agents/impl/skills/gitea-issue-rework/references/readback.md`.
|
||||
|
||||
Initial input-gate coverage is in `harness/tests/impl/test_rework_input_gate.py`.
|
||||
Initial input-gate coverage is in the `test_rework_input_gate_*.py` modules
|
||||
under `harness/tests/impl/rework/`.
|
||||
It exercises the real public CLI and canonical spec renderer, mixed-role/permutation
|
||||
and same-role supersession decisions, precise timestamp/tie handling, marker/state/
|
||||
source binding, complete raw validation/drift, immutable capture and atomic output
|
||||
@@ -142,5 +143,220 @@ The regression tests use bounded subprocess timeouts and clean test-owned
|
||||
Run them with:
|
||||
|
||||
```sh
|
||||
python3 -m unittest harness.tests.common.test_canonical_test_runner -v
|
||||
python3 -B -m unittest harness.tests.common.runner.test_canonical_test_runner -v
|
||||
```
|
||||
|
||||
## Domain ownership and verification levels
|
||||
|
||||
The canonical output of this documentation is `harness/tests/README.md` in this
|
||||
repository. Update this file in place rather than publishing a parallel test
|
||||
inventory document. The canonical runner remains `harness/tests/run.py`;
|
||||
there is no alternate discovery framework or hand-maintained source-hash list.
|
||||
|
||||
- `common/runner`: source admission and exact discovery/exit-status contracts.
|
||||
- `common/locking`: primitive flock and authenticated broker lifecycle remain
|
||||
separate modules; neither substitutes for the other.
|
||||
- `common/repository`: prefetch, runtime/verification roots, and Git source binding.
|
||||
- `common/delegation`: evidence, adoption, and real-Git consumer integration.
|
||||
- `common/evidence`: workflow evidence admission and conditional pytest recipe.
|
||||
- `common/access`: local credential protocol recipe, never real credentials.
|
||||
- `common/contracts`: cross-role documentation wiring, not live workflow proof.
|
||||
- `impl/delivery`: coordinator, initial delivery, reconciliation, and response.
|
||||
- `impl/rework`: intake, decision, rendering, provider readback, publication,
|
||||
and separately authorized correction.
|
||||
- `impl/evidence`: manifest/schema, filesystem/Git admission, and bounded
|
||||
command/process-group execution (distinct verification boundaries).
|
||||
- `impl/contracts`: documentation wiring and separately named comparator
|
||||
execution extracted from the actual canonical documentation.
|
||||
- `review/native`: native publication/plan, transport, lease/worktree admission,
|
||||
and durable resume; GET/POST loopback fixtures are not live Gitea publication.
|
||||
- `review/journal`: binding/presence safety, raw-byte legacy import, and actual
|
||||
subprocess crash/locking/race tests. Archive import is not fresh readback.
|
||||
- `review/evidence`: execution of the actual native evidence recipe.
|
||||
- `review/contracts`: policy-specific classes consolidated in one documentation
|
||||
module; string/link checks do not certify production behavior.
|
||||
- `spec/support`: result codec, cache admission, publication/readback, and AC checker.
|
||||
- `spec/projection`: source-Issue reference and requested-reviewer projection
|
||||
contracts remain independent modules with their own strict input matrices.
|
||||
- `spec/contracts`: documentation contracts plus the separately named
|
||||
`test_actual_collector.py` executable collector regression. That regression
|
||||
calls the real collector; it is not a local-model documentation example.
|
||||
|
||||
### Documentation regression boundary
|
||||
|
||||
General workflow guidance is not an exact-prose regression interface. Do not
|
||||
pin advisory sentences or generic review/reporting guidance with substring
|
||||
assertions. Keep executable behavior and explicit technical contracts: CLI
|
||||
options, schema fields, canonical paths/links, recipe structure, and required
|
||||
procedure ordering. In mixed tests, remove only the prose assertions rather
|
||||
than deleting the executable or structural checks.
|
||||
|
||||
The earlier `20261015T232331Z-final` traceability below is historical evidence
|
||||
for that source snapshot, not an index of the current tree. Subsequent removal
|
||||
runs under `logs/test-removal/` preserve their own baseline, exact dispositions,
|
||||
source hashes and observed outcomes; never relabel old outcomes as current.
|
||||
The general-guidance cleanup is recorded in
|
||||
`logs/test-removal/general-guidance-20261016T054315Z/`.
|
||||
|
||||
### Shared fixture boundaries
|
||||
|
||||
`_support/` holds ordinary data/transport/resource builders only. It must not
|
||||
contain or re-export TestCase classes, imported test bases, generated tests, or
|
||||
`load_tests`. Root derivation is centralized in `_support/paths.py`; helpers do
|
||||
not use `HARNESS_ROOT` or cwd to select canonical test sources. Resource builders
|
||||
own explicit cleanup. Meaningful expected results, no-call/no-write/no-retry,
|
||||
identity/source/stage and mode/inode/bytes assertions stay in the tests. Import
|
||||
fixture helpers rather than importing another test module or manually invoking
|
||||
another TestCase's `setUp()`/`tearDown()`.
|
||||
|
||||
### Focused execution and evidence limits
|
||||
|
||||
From the repository root, for example:
|
||||
|
||||
```sh
|
||||
HARNESS_ROOT=/opt/data/gitea-agent python3 -B -m unittest discover -s harness/tests/common -t .
|
||||
HARNESS_ROOT=/opt/data/gitea-agent python3 -B -m unittest discover -s harness/tests/impl -t .
|
||||
HARNESS_ROOT=/opt/data/gitea-agent python3 -B -m unittest discover -s harness/tests/review -t .
|
||||
HARNESS_ROOT=/opt/data/gitea-agent python3 -B -m unittest discover -s harness/tests/spec -t .
|
||||
```
|
||||
|
||||
These are focused runtime checks, not substitutes for the canonical admission
|
||||
and full execution command above. Compilation is not test execution;
|
||||
`--collect-only` is not runtime success. Source hashes and counts are emitted
|
||||
from the actual current source tree, never manually adjusted to admit a move.
|
||||
|
||||
- Documentation contracts read canonical documents (and required workflow
|
||||
references through `_support/contracts.py`); they emit assertions, not a live
|
||||
deployment/publication receipt.
|
||||
- Unit tests call actual helpers with bounded synthetic inputs; they certify
|
||||
their stated input/side-effect boundaries, not provider behavior in general.
|
||||
- Git/HTTP integration tests create local temporary repositories or loopback
|
||||
servers. They exercise real Git or transport code, not authenticated Gitea.
|
||||
- Process tests exercise actual child processes/crash/locks and preserve durable
|
||||
state assertions. In-process fakes or archive replay do not replace these.
|
||||
- Evidence recipe tests execute the canonical recipe in test-owned temporary
|
||||
roots. Their manifests/logs are local synthetic evidence, not product delivery.
|
||||
- Two pytest recipe execution cases skip when this interpreter has no admitted
|
||||
pytest installation (`admitted pytest interpreter required`). The documentation
|
||||
case still runs. No dependency is installed, skip removed, or warning/failure
|
||||
summary suppressed merely to obtain a green run.
|
||||
|
||||
### Test-to-artifact scope
|
||||
|
||||
Artifact scope is local and test-owned unless explicitly stated otherwise.
|
||||
Temporary files, synthetic repositories, loopback traffic and child-process
|
||||
journals are cleaned up by fixtures; they are not retained delivery evidence.
|
||||
The retained canonical runtime log records each exact test ID and outcome.
|
||||
The following groups specify what each module's tests may produce or inspect;
|
||||
a passing group does not certify every stronger evidence layer listed elsewhere.
|
||||
|
||||
- `common/access/test_git_credential_recipe.py`: local helper/askpass scripts,
|
||||
permissions and protocol stdout; no stored or real provider credentials.
|
||||
- `common/contracts/test_*.py`: canonical Markdown sections, workflow links and
|
||||
documentation assertions; no publication receipt or workflow execution.
|
||||
- `common/delegation/test_delegation_evidence*.py`: test-owned declarations,
|
||||
source snapshots, bundle hashes and verifier decisions. Adoption tests add
|
||||
fresh admission results; consumer unit tests add adapter-call records.
|
||||
`test_delegation_consumers_integration.py` adds actual temporary Git evidence,
|
||||
CLI stdout/stderr and fake consumer effects, not live Gitea writes.
|
||||
- `common/evidence/test_workflow_guard.py`: local contract/test inventories,
|
||||
snapshots, change decisions and fault inputs. `test_pytest_evidence_recipe.py`
|
||||
inspects the canonical recipe and conditionally creates actual pytest
|
||||
logs/summaries; skipped execution produces no recipe execution evidence.
|
||||
- `common/locking/test_managed_lock.py`: local lock files/descriptors and
|
||||
contention subprocess results. `test_managed_lock_broker.py`: private socket,
|
||||
lease/token/lifecycle state and process results; not a remote Issue lock.
|
||||
- `common/repository/test_repository_prefetch.py`: temporary local Git/cache
|
||||
fixtures. Root tests produce local ownership/path/cleanup state;
|
||||
`test_test_inventory_binding.py` produces source-bound inventories and binding
|
||||
decisions, not semantic test-coverage proof.
|
||||
- `common/runner/test_canonical_test_runner.py`: temporary source trees,
|
||||
canonical collection JSON/diagnostics and subprocess exit codes, including
|
||||
negative admission inputs. This is runner behavior, not product delivery.
|
||||
- `impl/contracts/test_impl_architecture_wiring.py`: canonical document/link
|
||||
assertions only. `test_impl_architecture_comparator.py`: comparator code
|
||||
extracted from that document, synthetic inputs and comparator decisions.
|
||||
- `impl/delivery/test_impl_operation.py`, `test_implementation_delivery.py`
|
||||
and `test_implementation_operation_integration.py`: local operation plans,
|
||||
evidence/adoption records and adapter call/effect ordering.
|
||||
`test_delivery_response.py` and `test_delivery_reconciliation.py` add raised
|
||||
HTTP errors or loopback response/reconciliation fixtures and completion
|
||||
decisions; neither authorizes retrying an uncertain remote write.
|
||||
- `impl/evidence/test_evidence_manifest_schema.py`: plan/manifest JSON,
|
||||
command-admission diagnostics and bounded command outputs.
|
||||
`test_evidence_manifest_filesystem_git.py`: real temporary Git identities,
|
||||
artifact bytes/paths and manifest validation.
|
||||
`test_evidence_command_deadlines.py`: actual child/process-group deadline
|
||||
behavior and local command logs; separate from Git topology validation.
|
||||
- `impl/rework/test_rework_input_gate_*.py`: schema inputs, CLI output files,
|
||||
role/timestamp decisions and counter/publication-phase diagnostics using
|
||||
canonical production entry points and synthetic provider data.
|
||||
- `impl/rework/test_rework_readback_provider.py`: provider-shaped snapshots,
|
||||
pagination and normalization decisions. Cache tests add strict-cache records;
|
||||
bridge/transport tests add renderer/CLI or loopback HTTP artifacts. These are
|
||||
not authenticated fresh Gitea readbacks.
|
||||
- `impl/rework/test_rework_comment.py`, `test_rework_correction.py` and
|
||||
`test_rework_publication.py`: local rendered comments, correction/publication
|
||||
plans, synthetic readback/receipt and no-write/no-retry call records.
|
||||
- `review/contracts/test_review_document_contracts.py`: actual policy document
|
||||
assertions only; no native review publication or observability run receipt.
|
||||
- `review/evidence/test_native_evidence_recipe.py`: actual recipe execution in
|
||||
temporary roots, evidence plans/manifests/logs and archive-path rejection;
|
||||
not a product delivery manifest or live provider verification.
|
||||
- `review/journal/test_journal_safety.py` and
|
||||
`test_journal_filesystem_safety.py`: private binding/presence records,
|
||||
byte/mode/path integrity and corruption decisions.
|
||||
`test_legacy_journal_contract.py`: source raw bytes, registration receipt,
|
||||
imported legacy archive and zero-POST assertions; imported archives do not
|
||||
establish fresh verification. `test_publication_processes.py`: actual child
|
||||
crashes, lock contention and durable POST-counter/journal state.
|
||||
- `review/native/test_native_publication.py`: synthetic plans/snapshots,
|
||||
journal/receipt and fake API call counts. Lease tests add local worktree/lease
|
||||
admission state; resume tests add durable pending-intent/readback failures;
|
||||
transport tests add local HTTP request/response and adapter artifacts.
|
||||
- `spec/contracts/test_priority_review_contracts.py`: actual canonical documents
|
||||
and links, not execution of local-model examples.
|
||||
`test_actual_collector.py`: actual collector callback arguments/results.
|
||||
- `spec/projection/test_requested_reviewer.py` and
|
||||
`test_source_issue_reference.py`: distinct snapshot/projection inputs,
|
||||
production comparison results, CLI stdout/stderr and unchanged-file checks.
|
||||
- `spec/support/test_codec.py`: rendered/parsed result bytes and CLI decisions;
|
||||
`test_cache.py`: private cache/path/ownership admission;
|
||||
`test_publication.py`: synthetic exact readback/publication comparisons;
|
||||
`test_ac_checker.py`: actual classification CLI/input digest decisions and
|
||||
canonical documentation assertions. None is live authenticated publication.
|
||||
|
||||
Exact per-test IDs, baseline predecessors, source anchors, artifact scope,
|
||||
verification layers and actual execution outcomes are retained in the run's
|
||||
`traceability.json`; `inventory-summary.json` distinguishes documentation,
|
||||
production behavior and overlapping process/integration evidence rather than
|
||||
adding incompatible counts. For this cleanup the run directory is
|
||||
`logs/regression-test-implementation/20261015T232331Z-final/`.
|
||||
Rebuild that index after source/test-ID changes; fail closed on an unmapped ID,
|
||||
missing outcome or source-hash mismatch. Primary classifications come from
|
||||
source assertions/calls and fixture ownership. Detailed Git/HTTP/process/archive
|
||||
labels are conservative source-confirmed subsets, not exhaustive dynamic
|
||||
coverage counts; indirect helper calls may add work beyond those labels. Layer
|
||||
counts overlap, and a declared execution layer is not executed evidence when
|
||||
its recorded outcome is skipped.
|
||||
|
||||
The final raw collection/runtime inputs are preserved in that run directory's
|
||||
`acceptance-final/` subdirectory. After reviewing any classification changes,
|
||||
rebuild the exact join with:
|
||||
|
||||
```sh
|
||||
python3 -B logs/regression-test-implementation/20261015T232331Z-final/refine_classification.py
|
||||
python3 -B logs/regression-test-implementation/20261015T232331Z-final/build_traceability.py
|
||||
```
|
||||
|
||||
These are run-specific evidence tools, not a replacement canonical runner.
|
||||
Do not infer coverage from a filename, a successful collection or the mere
|
||||
existence of an artifact.
|
||||
|
||||
The cleanup's original-to-current exact ID dispositions and run-specific counts,
|
||||
verification-level inventory, collection JSON, fault-injection controls and full
|
||||
execution logs are preserved under
|
||||
`logs/regression-test-implementation/20261015/`. The original 659-ID baseline
|
||||
under `logs/regression-test-audit/20261015/` is immutable. Counts in run evidence
|
||||
refer to actual methods (subTest branches are not additional tests); domain and
|
||||
verification-level labels do not make one test a substitute for another.
|
||||
|
||||
Reference in New Issue
Block a user