# req-021: Import provenance and sensitive-data policy

- Status: Implemented
- Type: Data
- Priority: High
- Owner: 렛저장인

## Intent
- `dcy-014`에서 확정된 import provenance 및 민감자료 저장 정책을 구현 가능한 requirement로 정규화한다.
- Jarca Ledger는 import rollback, duplicate detection, audit investigation에 필요한 lineage를 보존하되, 운영 DB·repo fixture·logs·MCP/Streamlit responses에 원본 금융 증빙을 과도하게 저장하거나 노출하지 않아야 한다.
- finance agent와 operator-facing surface는 raw evidence가 아니라 normalized provenance metadata를 소비해야 한다.

## Requirement
- 시스템은 모든 import source에 대해 표준 provenance envelope를 정의하고, source kind, parser identity, source hash, import batch id, row/page/sheet reference, external id, dedupe key, validation summary를 일관되게 보존해야 한다.
- `transactions.raw_payload` 저장 정책은 기본 `minimize`로 한다. 원본 문서/row 덤프를 저장하지 않고 source별 allowlist에 포함된 normalized 최소 필드만 저장하거나, allowlist 밖 key는 제거/마스킹해야 한다.
- `import_batches.filename`은 전체 file path를 저장하지 않고 `masked-basename + source_hash` 정책을 사용해야 한다.
- MCP provenance exposure는 기본 `minimal`, operator-facing `operator-summary`, explicit debug gate가 필요한 `debug-gated` 수준으로 분리해야 한다.
- finance agent는 raw financial evidence를 직접 받지 않고 normalized provenance metadata만 받아야 한다.
- 기존 운영 DB에 이미 저장된 raw payload/file path sanitization은 본 REQ의 destructive migration 범위에 포함하지 않는다. 별도 GitHub Project Todo/후속 SDLC에서 scan/preview/backup/approval/migration 순서로 다룬다.
- `docs/data-model.md`, `docs/project-structure.md`, `docs/runtime-flows.md`는 import provenance envelope, sensitive-data boundary, MCP exposure level, existing DB sanitization split을 반영해야 한다.

## Acceptance Criteria
1. Given synthetic import inputs for KB Bank PDF, Toss Bank PDF, Toss Securities PDF, SeoulPay XLSX, and CSV preview/import paths, When import preview or import is executed, Then each result exposes or stores a normalized provenance envelope with source kind, parser identity, source hash, import batch id where applicable, row/page/sheet reference where available, external id/dedupe key where applicable, and validation summary.
2. Given an import source file path containing personal/environmental path segments, When an import batch is recorded, Then `import_batches.filename` or its successor field contains no full path and uses masked basename plus source hash for operator identification and audit correlation.
3. Given source transactions contain raw parser payloads, file paths, account/card identifiers, passwords, tokens, secrets, or arbitrary CSV columns, When data is persisted to `transactions.raw_payload` or returned through preview/response payloads, Then only the source-specific allowlisted minimized fields remain and forbidden fields are removed or masked.
4. Given MCP default tools return transaction/import provenance, When called by a normal finance-agent client, Then the response uses the `minimal` level and excludes raw financial evidence, full file paths, passwords, tokens/secrets, raw document text, and non-allowlisted payload fields.
5. Given an operator-facing view or tool explicitly requests operator summary, When provenance is shown, Then masked filename, parser identity, source hash prefix, row/page/sheet reference, and validation summary may be included without exposing raw document text or full paths.
6. Given debug diagnostics are needed, When a debug-gated path is used, Then debug details are available only through an explicit gate and still exclude passwords, tokens/secrets, and full source document dumps by default.
7. Given existing operating DB rows may contain legacy raw payload/file path data, When REQ-021 delivery runs, Then it must not destructively rewrite existing operating DB content; it may add read-time sanitization or new-write policy only. Existing-data cleanup must remain in the separate Todo follow-up.
8. Given CI and test fixtures run import-related tests, When pytest/AppTest/HTTP smoke/contract tests execute, Then they use synthetic/temp DBs and static/contract checks verify that forbidden sensitive fields are not emitted in default MCP responses or committed fixtures.
9. Given baseline documentation is updated, When `docs/data-model.md`, `docs/project-structure.md`, and `docs/runtime-flows.md` are reviewed, Then they describe the provenance envelope, `raw_payload=minimize`, masked filename + source hash, MCP exposure levels, and the explicit split of existing DB sanitization.
10. Given implementation is complete, When `python -m ruff check .`, full pytest, contract regression, dashboard tests/smoke where affected, and StagePilot doctor are run, Then current-batch regressions must be clear; pre-existing unrelated doctor debt may be reported separately.

## Impacted Area
- Docs:
  - `docs/discovery/dcy-014_20260708_import-provenance-sensitive-data-policy.md`
  - `docs/data-model.md`
  - `docs/project-structure.md`
  - `docs/runtime-flows.md`
  - 후속 batch planning/design/verification 문서
- Modules:
  - `jarca_ledger_mcp/application/imports_service.py`
  - `jarca_ledger_mcp/infrastructure/parsers/*`
  - `jarca_ledger_mcp/infrastructure/sqlite/schema.py`
  - `jarca_ledger_mcp/infrastructure/sqlite/migrations.py`
  - `jarca_ledger_mcp/adapters/mcp/server.py`
  - `jarca_ledger_mcp/adapters/mcp/tool_contracts.py`
  - Streamlit import/operations gateway/page surfaces where provenance is displayed
- Tests:
  - source-specific import regression tests
  - sensitive-field/static scan tests
  - MCP contract/default response tests
  - fixture/log safety checks where applicable

## Notes
- Source Discovery: `docs/discovery/dcy-014_20260708_import-provenance-sensitive-data-policy.md`.
- Confirmed decisions from dcy-014:
  - `raw_payload`: `minimize`.
  - `import_batches.filename`: `masked-basename + source_hash`, no full path.
  - MCP exposure: default `minimal`, operator-facing `operator-summary`, debug `debug-gated`.
  - existing operating DB sanitization: split to separate GitHub Project Todo.
  - finance agent privacy boundary: normalized provenance only.
  - destructive migration: prohibited without preview/backup/explicit approval.
- Code-derived source matrix from dcy-014:
  - KB Bank PDF: source hash, page count, metadata, validation, parsed row count, file path, raw payload/summary, branch/balance, external id, dedupe key currently appear in import flow.
  - Toss Bank PDF: source hash, page count, metadata, validation, parsed row count, file path, action/counterparty/balance/raw payload/summary, external id, dedupe key currently appear in import flow.
  - Toss Securities PDF: source hash, page count, metadata, validation, parsed row count, file path, section/action/code/name/quantity/unit price/fee/tax/balance/raw payload currently appear in import flow.
  - SeoulPay XLSX: source hash, sheet name, metadata, validation, parsed row count, file path, transaction kind/payment kind/no/merchant id/channel/raw payload currently appear in import flow.
  - CSV preview: current preview can echo raw CSV rows and full local path, so it requires allowlist/masking policy before broader exposure.
- Out of scope:
  - Existing operating DB sanitization migration.
  - External legal review or third-party security audit.
  - File vault/encryption system for original source documents.
  - External public API auth/TLS design.

## Change Log

### CHG-20260708-01
- Date: 2026-07-08
- Author: 가웨인

#### Change Summary
- dcy-014 confirmed 범위를 REQ-021 Proposed 요구사항으로 생성했다.

#### Intent
- import provenance와 민감자료 저장/노출 정책을 delivery input으로 사용할 수 있는 단일 Data requirement로 정규화하기 위해 생성했다.

#### Acceptance Criteria Delta
- Added:
  - provenance envelope, raw_payload minimize, masked filename + source hash, MCP exposure levels, finance-agent privacy boundary, existing DB sanitization split, fixture/log/MCP sensitive-data checks를 acceptance criteria로 추가했다.
- Updated:
  - none
- Removed:
  - none

#### Impacted Area Delta
- Docs:
  - `docs/srs/index.md`
  - `docs/srs/Data/req-021_import-provenance-sensitive-data-policy.md`
  - `docs/discovery/dcy-014_20260708_import-provenance-sensitive-data-policy.md`
- Modules:
  - none yet; implementation pending
- Tests:
  - none yet; implementation pending

#### Delivery Trace Delta
- Batches:
  - none yet
- Releases:
  - none

#### Revalidation Impact
- Reverification Needed: yes
- Existing Implementation Invalidated: no
- Status Recommendation: keep-current
- Follow-up Action: run `confirm-req` for REQ-021 before batch selection/drafting.

#### Validation Plan
- Before implementation approval, verify source-specific provenance envelope behavior, sensitive-field filtering, MCP exposure-level defaults, synthetic fixture safety, full pytest, lint, relevant dashboard/MCP smoke, DB safety smoke, and baseline document updates.

### CHG-20260708-02
- Date: 2026-07-08
- Author: 가웨인

#### Change Summary
- REQ-021 승인 게이트를 통과해 Status를 Approved로 전환했다.

#### Intent
- dcy-014 결정사항이 모두 해소되었고, raw_payload minimize, masked filename + source hash, MCP exposure boundary, existing DB sanitization split, finance-agent privacy boundary, destructive migration 금지 원칙이 delivery input으로 충분히 구체화되어 승인했다.

#### Acceptance Criteria Delta
- Added:
  - none
- Updated:
  - none
- Removed:
  - none

#### Impacted Area Delta
- Docs:
  - `docs/srs/Data/req-021_import-provenance-sensitive-data-policy.md`
  - `docs/srs/index.md`
- Modules:
  - none
- Tests:
  - none

#### Delivery Trace Delta
- Batches:
  - none yet
- Releases:
  - none

#### Revalidation Impact
- Reverification Needed: yes
- Existing Implementation Invalidated: no
- Status Recommendation: keep-current
- Follow-up Action: draft batch for REQ-021 before implementation.

#### Validation Plan
- Validate through source-specific import tests, sensitive-field filtering/static scan, MCP exposure-level contract checks, fixture/log safety checks, full pytest, lint, relevant dashboard/MCP smoke, and baseline document updates.

### CHG-20260709-01
- Date: 2026-07-09
- Author: 가웨인

#### Change Summary
- PR #2 merge 및 GitHub Actions CI 통과 근거로 REQ-021을 Implemented 상태로 전환했다.

#### Intent
- BAT-017 구현이 main canonical codebase에 반영되었고, import provenance/sensitive-data policy acceptance 기준이 merge-backed evidence로 충족되었음을 문서 상태에 반영한다.

#### Acceptance Criteria Delta
- Added:
  - none
- Updated:
  - none
- Removed:
  - none

#### Impacted Area Delta
- Docs:
  - `docs/srs/Data/req-021_import-provenance-sensitive-data-policy.md`
  - `docs/srs/index.md`
  - `docs/batches/index.md`
  - `docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/index.md`
  - `docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/verification.md`
- Modules:
  - merge-backed BAT-017 implementation from PR #2
- Tests:
  - GitHub Actions CI: contract-regression, dashboard-app-test, dashboard-http-smoke, db-safety-smoke, lint, pytest-regression all passed.

#### Delivery Trace Delta
- Batches:
  - BAT-017 moved to release-candidate after PR #2 squash merge (`d9182f28f286667fb7ff52115c33ab2e5003b269`).
- Releases:
  - none yet

#### Revalidation Impact
- Reverification Needed: no for current implementation gate; release grouping remains separate.
- Existing Implementation Invalidated: no
- Status Recommendation: implemented
- Follow-up Action: keep existing operating DB import evidence sanitization as separate follow-up Todo; do not treat it as part of destructive migration scope for REQ-021.

#### Validation Plan
- Merge-backed evidence verified through PR #2 CI plus local main ruff/full pytest after merge.