feat: enforce import provenance exposure policy (#2)

Merge BAT-017 / REQ-021 implementation after delivery-runner implementation, independent QC, and passing GitHub Actions CI.
2026-07-09 18:44:19 +09:00
parent c9231c2685
commit d9182f28f2
13 changed files with 1168 additions and 47 deletions
@@ -1,16 +1,123 @@
 # BAT-017 Implementation Evidence

- Status: not-started
+- Status: implemented-awaiting-qc
 - Created At (KST): 2026-07-08 11:33 KST
+- Last Updated (KST): 2026-07-09T13:16:10+09:00

 ## Implementation Summary

-Pending.
+### Slice 01 — provenance foundation, import preview/write sanitization
+- Added a shared provenance policy module for source-hash labeling, allowlisted raw-payload JSON serialization, and CSV preview column filtering.
+- Stopped new import preview/import write paths from exposing or persisting full local source paths for:
+  - KB Bank PDF
+  - Toss Bank PDF
+  - Toss Securities PDF
+  - SeoulPay XLSX
+  - CSV preview
+- Changed `import_batches.filename` new writes to `masked-basename#source_hash_prefix`.
+- Minimized `transactions.raw_payload` for current supported import sources through explicit allowlists.
+- Added targeted regression tests for path redaction and forbidden-field non-leak assertions.

 ## Changed Files
+- `jarca_ledger_mcp/application/imports_service.py`
+- `jarca_ledger_mcp/kb_bank_pdf.py`
+- `jarca_ledger_mcp/seoulpay_xlsx.py`
+- `jarca_ledger_mcp/toss_bank_pdf.py`
+- `jarca_ledger_mcp/toss_securities_pdf.py`
+- `jarca_ledger_mcp/provenance_policy.py`
+- `tests/test_import_provenance_policy.py`

-Pending.
+## Execution Evidence
+- `python -m pytest -q tests/test_import_provenance_policy.py`
+  - result: pass (`.....`)
+- `python -m ruff check jarca_ledger_mcp/application/imports_service.py jarca_ledger_mcp/kb_bank_pdf.py jarca_ledger_mcp/seoulpay_xlsx.py jarca_ledger_mcp/toss_bank_pdf.py jarca_ledger_mcp/toss_securities_pdf.py jarca_ledger_mcp/provenance_policy.py tests/test_import_provenance_policy.py`
+  - result: pass (`All checks passed!`)
+- `pytest -q tests/test_import_provenance_policy.py tests/test_kb_bank_pdf.py tests/test_seoulpay_xlsx.py tests/test_toss_securities_pdf.py`
+  - result: pass via `dev-impl`
+- `pytest -q tests/test_kb_bank_pdf.py tests/test_seoulpay_xlsx.py tests/test_toss_securities_pdf.py tests/test_service.py`
+  - result: pass via `dev-impl`
+- `pytest -q tests/test_import_provenance_policy.py tests/test_kb_bank_pdf.py tests/test_seoulpay_xlsx.py tests/test_toss_securities_pdf.py tests/test_service.py`
+  - result: pass via `dev-impl`
+- `pytest -q`
+  - result: pass via `dev-impl`
+
+## Evidence Notes
+- Targeted regression confirms preview responses no longer emit full absolute paths.
+- CSV preview regression confirms arbitrary raw columns such as `secret_note` and `local_path` are suppressed by default.
+- Import persistence regression confirms `import_batches.filename` stores masked basename plus source-hash prefix rather than full paths.
+- Import persistence regression confirms `transactions.raw_payload` stores only the source-specific allowlisted minimum fields for the implemented sources.
+
+## Residual Risks / Follow-up
+- Slice 01 does not yet complete the MCP provenance exposure contract (`minimal` / `operator-summary` / `debug-gated`).
+- Import/read surfaces beyond the new-write preview/import boundary still need review for finance-agent privacy alignment.
+- Baseline docs still need BAT-017-specific updates once the exposure contract and final runtime boundary are settled.
+- Existing operating DB sanitization remains intentionally out of scope and must stay in the separate follow-up Todo.

 ## Notes
+- Implementation must not use Kanban unless the user explicitly re-allows it.

-Implementation must not use Kanban unless the user explicitly re-allows it.
+### Slice 02A — provenance exposure helper + tests
+- Added explicit provenance exposure constants and helper APIs in `jarca_ledger_mcp/provenance_policy.py` for `minimal`, `operator-summary`, and `debug-gated`.
+- Added sanitizer behavior that keeps normalized provenance fields while suppressing full paths, raw document/raw text dumps, passwords, tokens, secrets, and non-allowlisted raw payload keys.
+- Added alias helpers `sanitize_provenance_envelope(...)` and `sanitize_provenance_for_exposure(...)` for bounded downstream adoption.
+- Added helper-level regression coverage in `tests/test_import_provenance_policy.py` for exposure-level behavior and forbidden-field suppression.
+
+## Additional Changed Files (Slice 02A)
+- `jarca_ledger_mcp/provenance_policy.py`
+- `tests/test_import_provenance_policy.py`
+
+## Additional Execution Evidence (Slice 02A)
+- Supervised impl child result
+  - artifact: `.stagepilot/worker-supervision/impl-20260709T030219Z/final-result.json`
+  - result: `done` (`child_exit_code=0`)
+- `python -m pytest -q tests/test_import_provenance_policy.py`
+  - result: pass (`7 passed`)
+- `python -m py_compile jarca_ledger_mcp/provenance_policy.py tests/test_import_provenance_policy.py`
+  - result: pass
+- `python -m ruff check jarca_ledger_mcp/provenance_policy.py tests/test_import_provenance_policy.py`
+  - result: initially blocked in runner environment (`No module named ruff` from current Python)
+- Lead follow-up verification after environment check:
+  - `. .venv/bin/activate && python -m ruff check jarca_ledger_mcp/provenance_policy.py tests/test_import_provenance_policy.py`
+    - result: pass (`All checks passed!`)
+  - `. .venv/bin/activate && python -m pytest -q tests/test_import_provenance_policy.py`
+    - result: pass (`7 passed`)
+  - `. .venv/bin/activate && python -m py_compile jarca_ledger_mcp/provenance_policy.py tests/test_import_provenance_policy.py`
+    - result: pass
+  - `. .venv/bin/activate && python -m ruff check .`
+    - result: pass (`All checks passed!`)
+  - `. .venv/bin/activate && python -m pytest -q`
+    - result: pass (`55 passed, 4 skipped`)
+
+## Slice 02B — import batch read-surface wiring + baseline docs
+- Wired `list_ledger_import_batches(...)` through the BAT-017 provenance sanitizer so default responses now return `minimal` provenance and scrub the raw `memo` field from user-facing payloads.
+- Added bounded operator-summary support for import batch listing so masked filename, source-hash prefix, sheet reference, and validation summary are available only when explicitly requested.
+- Added MCP adapter pass-through for `provenance_exposure` on `list_ledger_import_batches(...)`.
+- Added explicit `debug_provenance_gate=True` enforcement before `debug-gated` provenance can be requested from the import batch listing surface.
+- Extended provenance regression coverage for:
+  - default/minimal import batch responses
+  - explicit operator-summary import batch responses
+  - debug-gated access denial without the explicit gate and allow-path with the gate
+  - MCP adapter pass-through of the exposure level
+- Updated baseline docs (`docs/data-model.md`, `docs/project-structure.md`, `docs/runtime-flows.md`) to record the provenance envelope boundary, `raw_payload=minimize`, masked filename policy, and read-time exposure split.
+
+## Additional Changed Files (Slice 02B)
+- `docs/data-model.md`
+- `docs/project-structure.md`
+- `docs/runtime-flows.md`
+- `jarca_ledger_mcp/adapters/mcp/server.py`
+- `jarca_ledger_mcp/application/imports_service.py`
+- `tests/test_import_provenance_policy.py`
+
+## Additional Execution Evidence (Slice 02B)
+- `. .venv/bin/activate && python -m ruff check jarca_ledger_mcp/application/imports_service.py jarca_ledger_mcp/adapters/mcp/server.py tests/test_import_provenance_policy.py`
+  - result: pass (`All checks passed!`)
+- `. .venv/bin/activate && python -m pytest -q tests/test_import_provenance_policy.py`
+  - result: pass (`11 passed`)
+- `. .venv/bin/activate && python -m ruff check .`
+  - result: pass (`All checks passed!`)
+- `. .venv/bin/activate && python -m pytest -q`
+  - result: pass (`59 passed, 4 skipped`)
+
+## Updated Residual Risks / Follow-up
+- Independent QC, PR creation/push, GitHub CI, and post-merge `confirm-req-implemented` remain pending.
+- StagePilot doctor entrypoint is not present in the current workspace/runtime; this should be classified during QC as environment/tooling debt unless the lead requires a separate setup action before closure.
@@ -1,11 +1,20 @@
 # BAT-017 Verification Evidence

- Status: not-started
+- Status: ready-for-qc
 - Created At (KST): 2026-07-08 11:33 KST

 ## Verification Commands

-Pending.
+- `. .venv/bin/activate && python -m ruff check jarca_ledger_mcp/application/imports_service.py jarca_ledger_mcp/adapters/mcp/server.py tests/test_import_provenance_policy.py`
+  - result: pass (`All checks passed!`)
+- `. .venv/bin/activate && python -m pytest -q tests/test_import_provenance_policy.py`
+  - result: pass (`11 passed`)
+- `. .venv/bin/activate && python -m ruff check .`
+  - result: pass (`All checks passed!`)
+- `. .venv/bin/activate && python -m pytest -q`
+  - result: pass (`59 passed, 4 skipped`)
+- StagePilot doctor
+  - result: blocked in current workspace/runtime because no discoverable `stagepilot-doctor` entrypoint or package was present; needs QC classification as pre-existing environment/tooling debt vs. closure blocker.

 ## Expected Gates

@@ -16,3 +25,9 @@ Pending.
 - fixture/log sensitive-field scan
 - affected dashboard smoke, if applicable
 - StagePilot doctor with pre-existing debt separated from BAT-017 findings
+
+## Batch-specific verification notes
+- Full pytest includes the affected import/privacy regression suite and leaves 4 skips in place without introducing new failures.
+- Import batch debug-gated exposure now requires explicit `debug_provenance_gate=True`; the regression suite covers both deny-without-gate and allow-with-gate behavior.
+- BAT-017 touched MCP import batch exposure behavior but did not modify dashboard code paths directly; no additional dashboard-only code change was required in this slice.
+- QC still needs to independently inspect the changed surfaces, validate the documentation updates, and classify the missing StagePilot doctor entrypoint.
@@ -4,8 +4,8 @@
 - Baseline Seed: `.stage-pilot/bootstrap/baseline.yaml`
 - Status: draft
 - Owner: Jarca / project maintainer
- Last Updated (KST): 2026-07-01
- Source Discovery / Batch: bootstrap-baseline, BAT-008, BAT-009
+- Last Updated (KST): 2026-07-09
+- Source Discovery / Batch: bootstrap-baseline, BAT-008, BAT-009, REQ-021/BAT-017

 ## Project Summary
 - Jarca 가계부 전용 MCP 서버와 Streamlit 대시보드로 계좌/카드 거래, 예산, 분류 규칙, import, 현금흐름 분석을 SQLite에 저장하고 운영한다.
@@ -107,6 +107,7 @@
  - `category` / `subcategory` :: text :: household classification
  - `merchant`, `counterparty`, `description`, `payment_method` :: text :: evidence and display fields
  - `source`, `external_id`, `import_batch_id`, `raw_summary`, `raw_payload` :: text :: provenance and audit fields
+  - `raw_payload` write policy is `minimize`: only source-specific allowlisted normalized keys may be persisted; raw document dumps, full file paths, passwords, tokens/secrets, and arbitrary parser columns are excluded.
  - `memo` :: text :: operator notes and reconciliation tags
  - `status` :: text :: active/deleted/review lifecycle state where used
 - Relationships:
@@ -160,13 +161,15 @@
 - Key Fields:
  - `id` :: text :: import batch identifier
  - `source` :: text :: source kind
-  - `file_path` / source metadata where stored
+  - `filename` :: text :: masked basename plus source-hash prefix for operator correlation; full local source paths must not be stored on new writes
+  - `memo` :: text/json :: compact batch metadata such as source hash, validation summary, or source-specific sheet/page references used for read-time provenance shaping
  - `inserted_count`, `skipped_duplicate_count`, `created_at`, `rolled_back_at`
 - Relationships:
  - Transaction rows reference `import_batch_id`.
 - State Rules:
  - Rollback should soft-delete by default.
  - Duplicate imports with `skip_duplicates=True` may insert zero rows; that is not automatically an error.
+  - Read surfaces should expose sanitized provenance envelopes: default `minimal`, explicit operator-facing `operator-summary`, and explicit debug-gated diagnostics without full document/path leakage.

 ## Entity: Budget
 - Source of Truth: budget table/service functions
@@ -224,7 +227,7 @@
 - Operational backup/status evidence may include only path, size, mtime, checksum, integrity, row counts, and aggregate totals.
 - Validation DB copies should live outside the repository and should be selected explicitly via `JARCA_LEDGER_DB_PATH` when used.
 - Passwords for encrypted imports are runtime-only inputs and are not part of the persisted model.
- Raw payloads may contain sensitive financial evidence and should not be committed to git.
+- Raw payloads may contain sensitive financial evidence and should not be committed to git; BAT-017 constrains new writes to minimized allowlisted payloads only and leaves legacy row cleanup to a separate approved follow-up.
 - ReviewDecision `target_payload` is sanitized to omit `raw_payload`, passwords, tokens, secrets, and file paths before persistence.

 ## Current Gaps / Planned Changes
@@ -234,6 +237,7 @@
 - BAT-009 implements manual asset CRUD mutation and month-end snapshot preview/close/correction service/MCP/dashboard workflow. Future work may add automatic market/real-estate valuation or scheduled month-end close.

 ## Change Log
+- 2026-07-09 KST: REQ-021/BAT-017 기준으로 import provenance exposure policy, `transactions.raw_payload=minimize`, `import_batches.filename` masked-basename persistence, and explicit debug gate expectations를 반영.
 - 2026-06-26 22:24 KST: Created by `bootstrap-baseline` from repository observation.
 - 2026-06-27 10:17 KST: BAT-002/REQ-002 기준으로 LedgerBackupArtifact와 metadata-only evidence 정책 추가.
 - 2026-06-29 KST: REQ-003/BAT-003 기준으로 `accounts.statement_status`와 수동 잔액대조 preview 모델 추가.
@@ -4,8 +4,8 @@
 - Baseline Seed: `.stage-pilot/bootstrap/baseline.yaml`
 - Status: draft
 - Owner: Jarca / project maintainer
- Last Updated (KST): 2026-07-08
- Source Discovery / Batch: bootstrap-baseline, BAT-006, BAT-007, BAT-010, BAT-012, BAT-013, BAT-014, dcy-013/BAT-015, dcy-012/REQ-020/BAT-016
+- Last Updated (KST): 2026-07-09
+- Source Discovery / Batch: bootstrap-baseline, BAT-006, BAT-007, BAT-010, BAT-012, BAT-013, BAT-014, dcy-013/BAT-015, dcy-012/REQ-020/BAT-016, REQ-021/BAT-017

 ## Project Summary
 - Jarca 가계부 전용 MCP 서버와 Streamlit 대시보드로 계좌/카드 거래, 예산, 분류 규칙, import, 자산/스냅샷, 현금흐름 분석을 SQLite에 저장하고 운영한다.
@@ -221,7 +221,8 @@ jarca-ledger-mcp/
 ## Shared Boundaries
 - `cashflow_kind` is the boundary between actual household income/expense and account movement/exclusion semantics.
 - `liquidity_bucket` is the account/manual-asset boundary for liquidity/composition/debt/net-worth views.
- Import evidence fields (`source`, `external_id`, `raw_summary`, `raw_payload`, `counterparty`, `import_batch_id`) are audit boundaries and should be preserved.
+- Import evidence fields (`source`, `external_id`, `raw_summary`, `raw_payload`, `counterparty`, `import_batch_id`) are audit boundaries and should be preserved, but BAT-017 constrains new persistence/response behavior to normalized provenance plus source-specific minimized payload allowlists.
+- `import_batches.filename` is an operator-correlation field, not a raw path cache: new writes must use masked basename plus source-hash prefix, and read surfaces must expose provenance at `minimal` by default unless an explicit operator/debug path is invoked.
 - Manual reconciliation is represented through memo tags and narrow transaction updates, not broad rewrites.
 - Browser evidence artifacts are generated from synthetic/validation DBs and should not embed real financial identifiers or raw source payloads.
 - CI gate fixtures are generated at runtime under temp space by `scripts/ci_fixture_db.py`; CI/AppTest/browser smoke must not read the default operating DB path.
@@ -237,6 +238,7 @@ jarca-ledger-mcp/
 - Update this document when a batch changes the approved structure baseline.

 ## Change Log
+- 2026-07-09 KST: REQ-021/BAT-017 기준으로 import provenance sanitizer 경계, masked filename persistence, explicit debug gate wiring, and baseline doc ownership을 반영.
 - 2026-06-26 22:24 KST: Created by `bootstrap-baseline` from repository observation.
 - 2026-06-29 KST: REQ-004/BAT-004 기준으로 `scripts/streamlit_browser_evidence.py` 및 browser evidence artifact 구조 추가.
 - 2026-06-29 KST: REQ-005/BAT-005 기준으로 dashboard operations hub와 service asset liquidity aggregation 책임 추가.
@@ -4,8 +4,8 @@
 - Baseline Seed: `.stage-pilot/bootstrap/baseline.yaml`
 - Status: draft
 - Owner: Jarca / project maintainer
- Last Updated (KST): 2026-07-08
- Source Discovery / Batch: bootstrap-baseline, BAT-006, BAT-007, BAT-008, BAT-010, BAT-012, BAT-013, BAT-014, dcy-013/BAT-015, dcy-012/REQ-020/BAT-016
+- Last Updated (KST): 2026-07-09
+- Source Discovery / Batch: bootstrap-baseline, BAT-006, BAT-007, BAT-008, BAT-010, BAT-012, BAT-013, BAT-014, dcy-013/BAT-015, dcy-012/REQ-020/BAT-016, REQ-021/BAT-017
 - Primary Runtime: mixed

 ## Project Summary
@@ -55,9 +55,14 @@
 2. MCP adapter forwards arguments to the matching callable exposed by `adapters/mcp/tool_contracts.py`.
 3. Service facade routes into the relevant application module.
 4. Application code opens SQLite through `infrastructure/sqlite` helpers, validates inputs, and performs the operation.
-5. Application returns a dictionary payload with success data or error information.
+5. Application returns a dictionary payload with success data or error information; import-related provenance is expected to be sanitized for the requested exposure level.
 6. Hermes surfaces the tool result to the user or uses it in a follow-up workflow.

+### MCP provenance exposure boundary
+- Default finance-agent/tool responses use `minimal` provenance exposure.
+- Operator-facing surfaces may request `operator-summary` when masked filename, source-hash prefix, or validation summary are needed for audit triage.
+- Debug details require an explicit debug-gated path and still exclude passwords, tokens/secrets, and full source document dumps by default.
+
 ## Flow: `Streamlit dashboard operation`
 1. Operator starts `streamlit run dashboard/app.py --server.address 0.0.0.0 --server.port 8501`.
 2. Dashboard reads the configured/default SQLite database.
@@ -84,8 +89,8 @@
 ## Flow: `Import preview and import`
 1. Operator chooses a supported source file: KB Bank PDF, Toss Bank PDF, Toss Securities PDF, SeoulPay XLSX, or CSV.
 2. Preview function routes through `imports_service.py`, which calls the parser seam under `infrastructure/parsers/*` and reports counts, sample rows, duplicate status, and classification preview.
-3. On explicit import, `imports_service.py` creates/fetches the target account, inserts non-duplicate transactions, writes provenance, and records an import batch.
-4. Reports and dashboard views read the inserted rows using shared category/cashflow semantics.
+3. On explicit import, `imports_service.py` creates/fetches the target account, inserts non-duplicate transactions, writes minimized/raw-payload-allowlisted provenance, and records an import batch with masked filename plus source-hash prefix instead of a full local path.
+4. Reports, MCP tools, and dashboard views read the inserted rows using shared category/cashflow semantics while returning normalized provenance metadata rather than raw document evidence by default.
 5. If an import is wrong, rollback uses import batch data and soft-deletes by default.

 ## Flow: `Categorization rule lifecycle`
@@ -155,6 +160,7 @@
 - Update this document when a batch changes the approved flow baseline.

 ## Change Log
+- 2026-07-09 KST: REQ-021/BAT-017 기준으로 import provenance exposure flow, operator-summary handling, and explicit debug gate requirement를 반영.
 - 2026-06-26 22:24 KST: Created by `bootstrap-baseline` from repository observation.
 - 2026-06-27 10:17 KST: BAT-002/REQ-002 기준으로 운영 backup/status/list/hash 및 restore 제한 flow 추가.
 - 2026-06-29 KST: REQ-003/BAT-003 기준으로 statement-unavailable 수동 잔액대조 flow 추가.