207 lines
12 KiB
Markdown
207 lines
12 KiB
Markdown
# req-021: Import provenance and sensitive-data policy
|
|
|
|
- Status: Implemented
|
|
- Type: Data
|
|
- Priority: High
|
|
- Owner: 렛저장인
|
|
|
|
## Intent
|
|
- `dcy-014`에서 확정된 import provenance 및 민감자료 저장 정책을 구현 가능한 requirement로 정규화한다.
|
|
- Jarca Ledger는 import rollback, duplicate detection, audit investigation에 필요한 lineage를 보존하되, 운영 DB·repo fixture·logs·MCP/Streamlit responses에 원본 금융 증빙을 과도하게 저장하거나 노출하지 않아야 한다.
|
|
- finance agent와 operator-facing surface는 raw evidence가 아니라 normalized provenance metadata를 소비해야 한다.
|
|
|
|
## Requirement
|
|
- 시스템은 모든 import source에 대해 표준 provenance envelope를 정의하고, source kind, parser identity, source hash, import batch id, row/page/sheet reference, external id, dedupe key, validation summary를 일관되게 보존해야 한다.
|
|
- `transactions.raw_payload` 저장 정책은 기본 `minimize`로 한다. 원본 문서/row 덤프를 저장하지 않고 source별 allowlist에 포함된 normalized 최소 필드만 저장하거나, allowlist 밖 key는 제거/마스킹해야 한다.
|
|
- `import_batches.filename`은 전체 file path를 저장하지 않고 `masked-basename + source_hash` 정책을 사용해야 한다.
|
|
- MCP provenance exposure는 기본 `minimal`, operator-facing `operator-summary`, explicit debug gate가 필요한 `debug-gated` 수준으로 분리해야 한다.
|
|
- finance agent는 raw financial evidence를 직접 받지 않고 normalized provenance metadata만 받아야 한다.
|
|
- 기존 운영 DB에 이미 저장된 raw payload/file path sanitization은 본 REQ의 destructive migration 범위에 포함하지 않는다. 별도 GitHub Project Todo/후속 SDLC에서 scan/preview/backup/approval/migration 순서로 다룬다.
|
|
- `docs/data-model.md`, `docs/project-structure.md`, `docs/runtime-flows.md`는 import provenance envelope, sensitive-data boundary, MCP exposure level, existing DB sanitization split을 반영해야 한다.
|
|
|
|
## Acceptance Criteria
|
|
1. Given synthetic import inputs for KB Bank PDF, Toss Bank PDF, Toss Securities PDF, SeoulPay XLSX, and CSV preview/import paths, When import preview or import is executed, Then each result exposes or stores a normalized provenance envelope with source kind, parser identity, source hash, import batch id where applicable, row/page/sheet reference where available, external id/dedupe key where applicable, and validation summary.
|
|
2. Given an import source file path containing personal/environmental path segments, When an import batch is recorded, Then `import_batches.filename` or its successor field contains no full path and uses masked basename plus source hash for operator identification and audit correlation.
|
|
3. Given source transactions contain raw parser payloads, file paths, account/card identifiers, passwords, tokens, secrets, or arbitrary CSV columns, When data is persisted to `transactions.raw_payload` or returned through preview/response payloads, Then only the source-specific allowlisted minimized fields remain and forbidden fields are removed or masked.
|
|
4. Given MCP default tools return transaction/import provenance, When called by a normal finance-agent client, Then the response uses the `minimal` level and excludes raw financial evidence, full file paths, passwords, tokens/secrets, raw document text, and non-allowlisted payload fields.
|
|
5. Given an operator-facing view or tool explicitly requests operator summary, When provenance is shown, Then masked filename, parser identity, source hash prefix, row/page/sheet reference, and validation summary may be included without exposing raw document text or full paths.
|
|
6. Given debug diagnostics are needed, When a debug-gated path is used, Then debug details are available only through an explicit gate and still exclude passwords, tokens/secrets, and full source document dumps by default.
|
|
7. Given existing operating DB rows may contain legacy raw payload/file path data, When REQ-021 delivery runs, Then it must not destructively rewrite existing operating DB content; it may add read-time sanitization or new-write policy only. Existing-data cleanup must remain in the separate Todo follow-up.
|
|
8. Given CI and test fixtures run import-related tests, When pytest/AppTest/HTTP smoke/contract tests execute, Then they use synthetic/temp DBs and static/contract checks verify that forbidden sensitive fields are not emitted in default MCP responses or committed fixtures.
|
|
9. Given baseline documentation is updated, When `docs/data-model.md`, `docs/project-structure.md`, and `docs/runtime-flows.md` are reviewed, Then they describe the provenance envelope, `raw_payload=minimize`, masked filename + source hash, MCP exposure levels, and the explicit split of existing DB sanitization.
|
|
10. Given implementation is complete, When `python -m ruff check .`, full pytest, contract regression, dashboard tests/smoke where affected, and StagePilot doctor are run, Then current-batch regressions must be clear; pre-existing unrelated doctor debt may be reported separately.
|
|
|
|
## Impacted Area
|
|
- Docs:
|
|
- `docs/discovery/dcy-014_20260708_import-provenance-sensitive-data-policy.md`
|
|
- `docs/data-model.md`
|
|
- `docs/project-structure.md`
|
|
- `docs/runtime-flows.md`
|
|
- 후속 batch planning/design/verification 문서
|
|
- Modules:
|
|
- `jarca_ledger_mcp/application/imports_service.py`
|
|
- `jarca_ledger_mcp/infrastructure/parsers/*`
|
|
- `jarca_ledger_mcp/infrastructure/sqlite/schema.py`
|
|
- `jarca_ledger_mcp/infrastructure/sqlite/migrations.py`
|
|
- `jarca_ledger_mcp/adapters/mcp/server.py`
|
|
- `jarca_ledger_mcp/adapters/mcp/tool_contracts.py`
|
|
- Streamlit import/operations gateway/page surfaces where provenance is displayed
|
|
- Tests:
|
|
- source-specific import regression tests
|
|
- sensitive-field/static scan tests
|
|
- MCP contract/default response tests
|
|
- fixture/log safety checks where applicable
|
|
|
|
## Notes
|
|
- Source Discovery: `docs/discovery/dcy-014_20260708_import-provenance-sensitive-data-policy.md`.
|
|
- Confirmed decisions from dcy-014:
|
|
- `raw_payload`: `minimize`.
|
|
- `import_batches.filename`: `masked-basename + source_hash`, no full path.
|
|
- MCP exposure: default `minimal`, operator-facing `operator-summary`, debug `debug-gated`.
|
|
- existing operating DB sanitization: split to separate GitHub Project Todo.
|
|
- finance agent privacy boundary: normalized provenance only.
|
|
- destructive migration: prohibited without preview/backup/explicit approval.
|
|
- Code-derived source matrix from dcy-014:
|
|
- KB Bank PDF: source hash, page count, metadata, validation, parsed row count, file path, raw payload/summary, branch/balance, external id, dedupe key currently appear in import flow.
|
|
- Toss Bank PDF: source hash, page count, metadata, validation, parsed row count, file path, action/counterparty/balance/raw payload/summary, external id, dedupe key currently appear in import flow.
|
|
- Toss Securities PDF: source hash, page count, metadata, validation, parsed row count, file path, section/action/code/name/quantity/unit price/fee/tax/balance/raw payload currently appear in import flow.
|
|
- SeoulPay XLSX: source hash, sheet name, metadata, validation, parsed row count, file path, transaction kind/payment kind/no/merchant id/channel/raw payload currently appear in import flow.
|
|
- CSV preview: current preview can echo raw CSV rows and full local path, so it requires allowlist/masking policy before broader exposure.
|
|
- Out of scope:
|
|
- Existing operating DB sanitization migration.
|
|
- External legal review or third-party security audit.
|
|
- File vault/encryption system for original source documents.
|
|
- External public API auth/TLS design.
|
|
|
|
## Change Log
|
|
|
|
### CHG-20260708-01
|
|
- Date: 2026-07-08
|
|
- Author: 가웨인
|
|
|
|
#### Change Summary
|
|
- dcy-014 confirmed 범위를 REQ-021 Proposed 요구사항으로 생성했다.
|
|
|
|
#### Intent
|
|
- import provenance와 민감자료 저장/노출 정책을 delivery input으로 사용할 수 있는 단일 Data requirement로 정규화하기 위해 생성했다.
|
|
|
|
#### Acceptance Criteria Delta
|
|
- Added:
|
|
- provenance envelope, raw_payload minimize, masked filename + source hash, MCP exposure levels, finance-agent privacy boundary, existing DB sanitization split, fixture/log/MCP sensitive-data checks를 acceptance criteria로 추가했다.
|
|
- Updated:
|
|
- none
|
|
- Removed:
|
|
- none
|
|
|
|
#### Impacted Area Delta
|
|
- Docs:
|
|
- `docs/srs/index.md`
|
|
- `docs/srs/Data/req-021_import-provenance-sensitive-data-policy.md`
|
|
- `docs/discovery/dcy-014_20260708_import-provenance-sensitive-data-policy.md`
|
|
- Modules:
|
|
- none yet; implementation pending
|
|
- Tests:
|
|
- none yet; implementation pending
|
|
|
|
#### Delivery Trace Delta
|
|
- Batches:
|
|
- none yet
|
|
- Releases:
|
|
- none
|
|
|
|
#### Revalidation Impact
|
|
- Reverification Needed: yes
|
|
- Existing Implementation Invalidated: no
|
|
- Status Recommendation: keep-current
|
|
- Follow-up Action: run `confirm-req` for REQ-021 before batch selection/drafting.
|
|
|
|
#### Validation Plan
|
|
- Before implementation approval, verify source-specific provenance envelope behavior, sensitive-field filtering, MCP exposure-level defaults, synthetic fixture safety, full pytest, lint, relevant dashboard/MCP smoke, DB safety smoke, and baseline document updates.
|
|
|
|
### CHG-20260708-02
|
|
- Date: 2026-07-08
|
|
- Author: 가웨인
|
|
|
|
#### Change Summary
|
|
- REQ-021 승인 게이트를 통과해 Status를 Approved로 전환했다.
|
|
|
|
#### Intent
|
|
- dcy-014 결정사항이 모두 해소되었고, raw_payload minimize, masked filename + source hash, MCP exposure boundary, existing DB sanitization split, finance-agent privacy boundary, destructive migration 금지 원칙이 delivery input으로 충분히 구체화되어 승인했다.
|
|
|
|
#### Acceptance Criteria Delta
|
|
- Added:
|
|
- none
|
|
- Updated:
|
|
- none
|
|
- Removed:
|
|
- none
|
|
|
|
#### Impacted Area Delta
|
|
- Docs:
|
|
- `docs/srs/Data/req-021_import-provenance-sensitive-data-policy.md`
|
|
- `docs/srs/index.md`
|
|
- Modules:
|
|
- none
|
|
- Tests:
|
|
- none
|
|
|
|
#### Delivery Trace Delta
|
|
- Batches:
|
|
- none yet
|
|
- Releases:
|
|
- none
|
|
|
|
#### Revalidation Impact
|
|
- Reverification Needed: yes
|
|
- Existing Implementation Invalidated: no
|
|
- Status Recommendation: keep-current
|
|
- Follow-up Action: draft batch for REQ-021 before implementation.
|
|
|
|
#### Validation Plan
|
|
- Validate through source-specific import tests, sensitive-field filtering/static scan, MCP exposure-level contract checks, fixture/log safety checks, full pytest, lint, relevant dashboard/MCP smoke, and baseline document updates.
|
|
|
|
### CHG-20260709-01
|
|
- Date: 2026-07-09
|
|
- Author: 가웨인
|
|
|
|
#### Change Summary
|
|
- PR #2 merge 및 GitHub Actions CI 통과 근거로 REQ-021을 Implemented 상태로 전환했다.
|
|
|
|
#### Intent
|
|
- BAT-017 구현이 main canonical codebase에 반영되었고, import provenance/sensitive-data policy acceptance 기준이 merge-backed evidence로 충족되었음을 문서 상태에 반영한다.
|
|
|
|
#### Acceptance Criteria Delta
|
|
- Added:
|
|
- none
|
|
- Updated:
|
|
- none
|
|
- Removed:
|
|
- none
|
|
|
|
#### Impacted Area Delta
|
|
- Docs:
|
|
- `docs/srs/Data/req-021_import-provenance-sensitive-data-policy.md`
|
|
- `docs/srs/index.md`
|
|
- `docs/batches/index.md`
|
|
- `docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/index.md`
|
|
- `docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/verification.md`
|
|
- Modules:
|
|
- merge-backed BAT-017 implementation from PR #2
|
|
- Tests:
|
|
- GitHub Actions CI: contract-regression, dashboard-app-test, dashboard-http-smoke, db-safety-smoke, lint, pytest-regression all passed.
|
|
|
|
#### Delivery Trace Delta
|
|
- Batches:
|
|
- BAT-017 moved to release-candidate after PR #2 squash merge (`d9182f28f286667fb7ff52115c33ab2e5003b269`).
|
|
- Releases:
|
|
- none yet
|
|
|
|
#### Revalidation Impact
|
|
- Reverification Needed: no for current implementation gate; release grouping remains separate.
|
|
- Existing Implementation Invalidated: no
|
|
- Status Recommendation: implemented
|
|
- Follow-up Action: keep existing operating DB import evidence sanitization as separate follow-up Todo; do not treat it as part of destructive migration scope for REQ-021.
|
|
|
|
#### Validation Plan
|
|
- Merge-backed evidence verified through PR #2 CI plus local main ruff/full pytest after merge.
|