[verified] Add executable reviewer evidence manifest recipe

This commit is contained in:
2026-10-12 06:49:44 +00:00
parent 4acc28563c
commit d4db3f6a8e
2 changed files with 137 additions and 0 deletions
@@ -137,6 +137,33 @@ PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s harness/tests/impl -v
Run the test commands from the canonical harness repository root. The existing spec suite requires `/opt/data/cache/scratch`: prepare that directory only if absent after confirming a nonsymlink, role-owned, non-group/world-writable parent, and require mode 0700 for the scratch child. Preserve existing shared parent permissions; do not chmod/chown them to satisfy a test. Use isolated/local runner variables and properly quoted command arguments rather than reusing mutable interpreter globals across review steps. Keep local stub/loopback transport checks, real temporary Git/worktree/lease checks, and saved production-wire replay clearly distinguished from a live Gitea mutation test. Introducing this helper does not authorize changing any existing product PR verdict, source, Issue state, merge or deployment. Run the test commands from the canonical harness repository root. The existing spec suite requires `/opt/data/cache/scratch`: prepare that directory only if absent after confirming a nonsymlink, role-owned, non-group/world-writable parent, and require mode 0700 for the scratch child. Preserve existing shared parent permissions; do not chmod/chown them to satisfy a test. Use isolated/local runner variables and properly quoted command arguments rather than reusing mutable interpreter globals across review steps. Keep local stub/loopback transport checks, real temporary Git/worktree/lease checks, and saved production-wire replay clearly distinguished from a live Gitea mutation test. Introducing this helper does not authorize changing any existing product PR verdict, source, Issue state, merge or deployment.
## Reviewer verification evidence recipe
This is the existing common evidence helper, not the `native_review.py preflight/verify` publisher replay interface. Use the canonical helper at `$HARNESS_ROOT/harness/agents/impl/skills/gitea-issue-rework/scripts/evidence_manifest.py`; do not create a reviewer wrapper or copy it into another profile. The historical v2 recipe gap is addressed using the current `evidence-manifest/v3` contract.
First complete authenticated cache/worktree admission and retain the current review-role Issue broker lease. Prepare a fresh stage/run directory using the existing `verification_root.py prepare` procedure in the [canonical repository-cache contract](../../../../../operations/repository-cache.md). That procedure authenticates the live handle and Issue identity; the evidence helper does not acquire or certify a lease. Derive these absolute paths from the admitted harness/Issue identity, not from an arbitrary output directory:
- `HARNESS_ROOT`: the admitted harness root containing `harness/operations/repository-cache.md`.
- `REPO_ROOT`: `$HARNESS_ROOT/repositories/$OWNER/$REPOSITORY/worktrees/issue-$ISSUE/checkout`.
- `VERIFICATION_RUN_ROOT`: `$HARNESS_ROOT/repositories/$OWNER/$REPOSITORY/worktrees/issue-$ISSUE/verification/$RUN_ID`.
Require a **clean checkout**, **no source/evidence exclusions**, and prepared owner-only `0700` verification/run directories. Place the reviewed command plan at `$VERIFICATION_RUN_ROOT/evidence-plan.json` with mode `0600` before running; adapt the existing [evidence-plan template](../../../../impl/skills/gitea-issue-rework/templates/evidence-plan.json) to the actual reviewer commands, finite `timeout_seconds`, output bounds, source `hash_paths`, claims and limitations. `hash_paths` are checkout-relative, while plan/output/manifest CLI arguments are **run-relative**. Do not put artifacts in the checkout, weaken permissions, change the clean-tree check, or normalize rejected absolute artifact filenames. All roots/contract paths below are absolute; the three artifact filenames are not:
```sh
python3 -B "$HARNESS_ROOT/harness/agents/impl/skills/gitea-issue-rework/scripts/evidence_manifest.py" run \
--contract-path "$HARNESS_ROOT/harness/operations/repository-cache.md" \
--repo-root "$REPO_ROOT" --artifact-root "$VERIFICATION_RUN_ROOT" \
--plan evidence-plan.json --output evidence-manifest.json
python3 -B "$HARNESS_ROOT/harness/agents/impl/skills/gitea-issue-rework/scripts/evidence_manifest.py" validate \
--contract-path "$HARNESS_ROOT/harness/operations/repository-cache.md" \
--repo-root "$REPO_ROOT" --artifact-root "$VERIFICATION_RUN_ROOT" \
--manifest evidence-manifest.json
```
Run `validate` only after `run` succeeds, using the same roots and exact source HEAD. A nonzero result, timeout or overflow blocks adoption; retain bounded failure evidence and do not automatically rerun the suite. Existing manifests are not overwritten. Local validation binds the current source, plan and actual command records; it is not live Gitea readback, reviewer approval, Native publication verification or cleanup completion. Preserve the fresh stage/run ID, source SHA, plan/manifest digests, actual results/limitations and verified reviewer-access method under the existing reporting/cleanup contract.
Archived v2 uses only `validate-archived-v2` read-only compatibility; archived v1 uses only `validate-legacy`. Neither upgrades old records to v3 deadline/cleanup guarantees nor creates fresh publication evidence. Do not migrate archived manifests, add exclusions or change schemas to make the example pass.
## Canonical native-review journal ## Canonical native-review journal
Native code-review publication의 정본 journal root는 `<harness-root>/harness/.runtime/native-review-journals`이다. Canonical publisher의 물리적인 harness 위치에서 결정하며 caller의 run-id, 임의 journal 경로, 환경 변수 변경으로 다른 registry를 선택하지 않는다. `canonical_journal_path(plan)`은 검증된 plan의 표시용 경로 계산일 뿐 인증·게시 권한의 증명이 아니다. 실제 registry claim 전에 현재 role identity와 held Issue lease 및 source/delegation gate를 검증한다. Native code-review publication의 정본 journal root는 `<harness-root>/harness/.runtime/native-review-journals`이다. Canonical publisher의 물리적인 harness 위치에서 결정하며 caller의 run-id, 임의 journal 경로, 환경 변수 변경으로 다른 registry를 선택하지 않는다. `canonical_journal_path(plan)`은 검증된 plan의 표시용 경로 계산일 뿐 인증·게시 권한의 증명이 아니다. 실제 registry claim 전에 현재 role identity와 held Issue lease 및 source/delegation gate를 검증한다.
@@ -0,0 +1,110 @@
"""Reviewer recipe contracts; no live provider or publication calls."""
import json
import shlex
import stat
import string
import subprocess
import sys
import unittest
from pathlib import Path
from harness.tests.impl import test_evidence_manifest as fixtures
ROOT = Path(__file__).resolve().parents[3]
REFERENCE = ROOT / "harness/agents/review/skills/gitea-code-review/references/native-review.md"
HELPER_RELATIVE = "harness/agents/impl/skills/gitea-issue-rework/scripts/evidence_manifest.py"
HEADING = "## Reviewer verification evidence recipe"
class NativeEvidenceRecipeTests(unittest.TestCase):
def recipe(self):
text = REFERENCE.read_text(encoding="utf-8")
self.assertTrue(HEADING in text, "reviewer evidence-manifest recipe is missing")
return text.split(HEADING, 1)[1].split("\n## ", 1)[0]
def test_current_recipe_preserves_canonical_path_and_archive_contracts(self):
section = self.recipe()
for required in (HELPER_RELATIVE, "verification_root.py prepare", "evidence-manifest/v3",
"validate-archived-v2", "read-only", "run-relative", "clean checkout",
"no source/evidence exclusions", "0700", "0600"):
self.assertIn(required, section)
block = section.split("```sh\n", 1)[1].split("```", 1)[0]
lines = block.replace("\\\n", "").splitlines()
commands = [shlex.split(line) for line in lines if line.strip() and not line.lstrip().startswith("#")]
self.assertEqual(len(commands), 2)
self.assertEqual([command[:4] for command in commands],
[["python3", "-B", "$HARNESS_ROOT/" + HELPER_RELATIVE, "run"],
["python3", "-B", "$HARNESS_ROOT/" + HELPER_RELATIVE, "validate"]])
for command in commands:
self.assertEqual(command[command.index("--contract-path") + 1],
"$HARNESS_ROOT/harness/operations/repository-cache.md")
self.assertEqual(command[command.index("--repo-root") + 1], "$REPO_ROOT")
self.assertEqual(command[command.index("--artifact-root") + 1], "$VERIFICATION_RUN_ROOT")
self.assertEqual(commands[0][commands[0].index("--plan") + 1], "evidence-plan.json")
self.assertEqual(commands[0][commands[0].index("--output") + 1], "evidence-manifest.json")
self.assertEqual(commands[1][commands[1].index("--manifest") + 1], "evidence-manifest.json")
def test_documented_commands_run_validate_and_reject_absolute_artifact_paths(self):
section = self.recipe()
fixture = fixtures.EvidenceManifestTests()
try:
fixture.setUp()
# Install exact current helper bytes into an isolated canonical-shaped
# synthetic harness. No fork, wrapper, live lease or provider is used.
source = ROOT / HELPER_RELATIVE
helper = fixture.root / HELPER_RELATIVE
helper.parent.mkdir(parents=True)
helper.write_bytes(source.read_bytes())
self.assertEqual(helper.read_bytes(), source.read_bytes())
fixture.plan().rename(fixture.artifacts / "evidence-plan.json")
variables = {"HARNESS_ROOT": str(fixture.root), "REPO_ROOT": str(fixture.repo),
"VERIFICATION_RUN_ROOT": str(fixture.artifacts)}
block = section.split("```sh\n", 1)[1].split("```", 1)[0]
lines = string.Template(block).substitute(variables).replace("\\\n", "").splitlines()
commands = [shlex.split(line) for line in lines if line.strip() and not line.lstrip().startswith("#")]
self.assertEqual(len(commands), 2)
self.assertEqual([command[:4] for command in commands],
[["python3", "-B", str(helper), "run"],
["python3", "-B", str(helper), "validate"]])
for command in commands:
command[0] = sys.executable
def invoke(command):
return subprocess.run(command, cwd=fixture.repo, text=True, capture_output=True, timeout=15)
run = invoke(commands[0])
self.assertEqual(run.returncode, 0, run.stderr)
manifest_path = fixture.artifacts / "evidence-manifest.json"
manifest = json.loads(manifest_path.read_text())
self.assertEqual(manifest["schema"], "evidence-manifest/v3")
self.assertEqual(manifest["source_revision"], fixtures.run_git(fixture.repo, "rev-parse", "HEAD").stdout.strip())
self.assertEqual(manifest["plan"]["path"], "evidence-plan.json")
self.assertEqual(manifest["commands"][0]["argv"], ["git", "show", "HEAD:input.txt"])
self.assertEqual(manifest["commands"][0]["exit_code"], 0)
self.assertEqual(manifest["commands"][0]["output"], "hello\n")
validated = invoke(commands[1])
self.assertEqual(validated.returncode, 0, validated.stderr)
before = manifest_path.read_bytes()
mode = stat.S_IMODE(manifest_path.stat().st_mode)
self.assertEqual(mode, 0o600)
for flag, base_command, filename in (
("--plan", commands[0], "evidence-plan.json"),
("--output", commands[0], "bad-output.json"),
("--manifest", commands[1], "evidence-manifest.json")):
with self.subTest(flag=flag):
invalid = list(base_command)
invalid[invalid.index(flag) + 1] = str(fixture.artifacts / filename)
if flag == "--plan":
# Use an absent output so collision cannot mask the path rejection.
invalid[invalid.index("--output") + 1] = "bad-plan.json"
rejected = invoke(invalid)
self.assertEqual(rejected.returncode, 2)
self.assertIn("artifact path must be run-relative", rejected.stderr)
self.assertEqual(manifest_path.read_bytes(), before)
self.assertEqual(stat.S_IMODE(manifest_path.stat().st_mode), mode)
self.assertFalse((fixture.artifacts / "bad-plan.json").exists())
self.assertFalse((fixture.artifacts / "bad-output.json").exists())
self.assertEqual(fixtures.run_git(fixture.repo, "status", "--porcelain").stdout, "")
finally:
if hasattr(fixture, "tmp"):
fixture.tearDown()