From c9231c2685c5e8fc05f9ce60da40f5ab7f409d8c Mon Sep 17 00:00:00 2001 From: Jarca Date: Wed, 8 Jul 2026 11:34:37 +0900 Subject: [PATCH] docs: confirm import provenance req and batch --- .../design.md | 49 ++++++ .../implementation.md | 16 ++ .../index.md | 35 ++++ .../planning.md | 57 ++++++ .../verification.md | 18 ++ docs/batches/index.md | 2 + ...import-provenance-sensitive-data-policy.md | 100 ++++++++--- docs/discovery/index.md | 2 +- ...import-provenance-sensitive-data-policy.md | 162 ++++++++++++++++++ docs/srs/index.md | 5 +- 10 files changed, 421 insertions(+), 25 deletions(-) create mode 100644 docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/design.md create mode 100644 docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/implementation.md create mode 100644 docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/index.md create mode 100644 docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/planning.md create mode 100644 docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/verification.md create mode 100644 docs/srs/Data/req-021_import-provenance-sensitive-data-policy.md diff --git a/docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/design.md b/docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/design.md new file mode 100644 index 0000000..5992daf --- /dev/null +++ b/docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/design.md @@ -0,0 +1,49 @@ +# BAT-017 Design: Import provenance and sensitive-data policy + +- Status: draft +- Created At (KST): 2026-07-08 11:33 KST +- Included REQ: REQ-021 + +## Design Direction + +Introduce a narrow provenance/sanitization layer used by import services and response adapters rather than scattering per-source masking rules across UI/MCP code. + +## Proposed Components + +- `provenance` helper functions under application/import boundary or a small shared module: + - `mask_source_filename(path_or_name)` + - `build_source_identity(source, file_path, source_hash, parser_name, parser_version)` + - `minimize_raw_payload(source, payload)` + - `sanitize_provenance_for_exposure(payload, level)` +- Source-specific allowlists: + - KB Bank PDF: date/time, amount, direction, masked account hint, page/row reference, validation keys. + - Toss Bank PDF: action category, normalized counterparty, masked account hint, page/row reference, validation keys. + - Toss Securities PDF: section/action/code, normalized security/counterparty summary, numeric audit fields when required. + - SeoulPay XLSX: sheet name, row reference, payment kind/category summary, minimized merchant/transaction reference. + - CSV: explicit preview/import allowlist and sensitive-column suppression. + +## Storage Policy + +- Do not persist full local source paths. +- Store masked basename and source hash for operator correlation. +- Keep `source_hash` stable for dedupe/audit. +- `raw_payload` stores only minimized allowlisted fields; forbidden fields are removed/masked. +- Existing rows are not destructively rewritten by this batch. + +## Interface Policy + +- Default MCP responses use `minimal` exposure. +- Operator summary may include masked filename, parser identity, row/page/sheet reference, validation summary. +- Debug diagnostics require explicit gate and still exclude passwords/tokens/secrets/full path/raw document dump. + +## Test Strategy + +- Use synthetic fixtures only. +- Include negative assertions for forbidden keys and full path patterns. +- Verify duplicate/rollback metadata remains usable after minimization. + +## Open Design Checks Before Implementation + +- Decide whether provenance helpers live in `application/imports_service.py` initially or a new module. +- Confirm whether schema version bump is required or policy can be implemented through current columns plus memo JSON shape. +- Confirm MCP contract change is additive and does not break existing consumers. diff --git a/docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/implementation.md b/docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/implementation.md new file mode 100644 index 0000000..045fa64 --- /dev/null +++ b/docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/implementation.md @@ -0,0 +1,16 @@ +# BAT-017 Implementation Evidence + +- Status: not-started +- Created At (KST): 2026-07-08 11:33 KST + +## Implementation Summary + +Pending. + +## Changed Files + +Pending. + +## Notes + +Implementation must not use Kanban unless the user explicitly re-allows it. diff --git a/docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/index.md b/docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/index.md new file mode 100644 index 0000000..315540c --- /dev/null +++ b/docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/index.md @@ -0,0 +1,35 @@ +# BAT-017: Import provenance and sensitive-data policy + +- Profile: standard +- Status: draft +- Included REQ: REQ-021 +- Source Discovery: dcy-014 +- Created At (KST): 2026-07-08 11:33 KST + +## Objective +Deliver the approved REQ-021 scope for normalized import provenance, `raw_payload=minimize`, masked filename + source hash, MCP provenance exposure levels, finance-agent privacy boundary, destructive migration guardrails, and baseline documentation. + +## Included Scope +- Source-specific provenance envelope for KB Bank PDF, Toss Bank PDF, Toss Securities PDF, SeoulPay XLSX, and CSV preview/import surfaces. +- `transactions.raw_payload` minimize policy using explicit allowlists and forbidden-field filtering. +- `import_batches.filename` policy change away from full paths toward masked basename + source hash. +- MCP/Streamlit provenance exposure levels: default `minimal`, operator `operator-summary`, debug `debug-gated`. +- Sensitive-field tests/static checks for raw payload, full paths, password/token/secret, raw document text, and arbitrary CSV column leakage. +- Baseline updates in `docs/data-model.md`, `docs/project-structure.md`, and `docs/runtime-flows.md`. + +## Excluded Scope +- Existing operating DB sanitization migration; tracked separately in GitHub Project 4 Todo `기존 운영 DB import evidence sanitization preview 및 정리`. +- External legal/security audit. +- Original source document vault/encryption system. +- Public API auth/TLS design. +- Kanban-based delivery execution. + +## Documents +- [planning.md](planning.md) +- [design.md](design.md) +- [implementation.md](implementation.md) +- [verification.md](verification.md) + +## Delivery Notes +- User explicitly prohibited Kanban; delivery execution must use direct branch/PR or a dedicated non-Kanban worktree if delegated. +- Implementation should be sliced to avoid the previous BAT-016 large-card failure pattern: provenance helpers first, source adapters second, MCP/Streamlit exposure third, tests/docs last. diff --git a/docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/planning.md b/docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/planning.md new file mode 100644 index 0000000..cd70181 --- /dev/null +++ b/docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/planning.md @@ -0,0 +1,57 @@ +# BAT-017 Planning: Import provenance and sensitive-data policy + +- Status: draft +- Created At (KST): 2026-07-08 11:33 KST +- Included REQ: REQ-021 + +## Planning Summary + +BAT-017 is a standard batch because REQ-021 affects data persistence policy, import adapters, MCP/Streamlit response contracts, tests, and baseline documentation. + +## Work Slices + +1. Define provenance policy helpers + - masked basename helper + - source hash propagation helper + - raw payload minimization/filter helper + - exposure-level sanitizer for `minimal`, `operator-summary`, `debug-gated` +2. Apply policy to import write paths + - KB Bank PDF + - Toss Bank PDF + - Toss Securities PDF + - SeoulPay XLSX + - CSV preview/import boundary +3. Apply policy to read/response surfaces + - import batch listing + - preview responses + - MCP tool contracts/default output + - Streamlit/operator summary where applicable +4. Add regression checks + - source-specific import tests + - forbidden-field filtering tests + - default MCP response privacy checks + - fixture/log/static scan checks +5. Update baseline documents + - `docs/data-model.md` + - `docs/project-structure.md` + - `docs/runtime-flows.md` + +## Design Gate + +Design is required. The batch changes persistence semantics and interface response boundaries, so `design.md` must be completed before implementation. + +## Risks + +- Over-sanitizing may weaken duplicate/rollback/audit traceability. +- Under-sanitizing may leak file paths, raw financial evidence, or arbitrary CSV columns. +- Existing operating DB rows may contain legacy data; this batch must avoid destructive cleanup and keep the follow-up Todo separate. + +## Verification Plan + +- `python -m ruff check .` +- source-specific pytest for import provenance behavior +- full `python -m pytest -q` +- MCP contract/default privacy checks +- dashboard/AppTest/HTTP smoke if affected +- DB safety smoke if schema/migration changes are included +- StagePilot doctor with current-batch findings separated from pre-existing debt diff --git a/docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/verification.md b/docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/verification.md new file mode 100644 index 0000000..2fb7529 --- /dev/null +++ b/docs/batches/bat-017_20260708_import-provenance-sensitive-data-policy/verification.md @@ -0,0 +1,18 @@ +# BAT-017 Verification Evidence + +- Status: not-started +- Created At (KST): 2026-07-08 11:33 KST + +## Verification Commands + +Pending. + +## Expected Gates + +- `python -m ruff check .` +- `python -m pytest -q` +- source-specific import provenance tests +- MCP privacy/default exposure tests +- fixture/log sensitive-field scan +- affected dashboard smoke, if applicable +- StagePilot doctor with pre-existing debt separated from BAT-017 findings diff --git a/docs/batches/index.md b/docs/batches/index.md index d8ec54c..51df852 100644 --- a/docs/batches/index.md +++ b/docs/batches/index.md @@ -24,5 +24,7 @@ | BAT-014 | standard | release-candidate | REQ-018 | dcy-011 | [bat-014_20260703_adapter-surface-narrowing-baseline-refresh](bat-014_20260703_adapter-surface-narrowing-baseline-refresh/) | | BAT-015 | standard | release-candidate | REQ-019 | dcy-013 | [bat-015_20260707_docker-network-mcp-deployment](bat-015_20260707_docker-network-mcp-deployment/) | | BAT-016 | standard | release-candidate | REQ-020 | dcy-012 | [bat-016_20260708_review-decision-reconciliation-workbench](bat-016_20260708_review-decision-reconciliation-workbench/) | +| BAT-017 | standard | draft | REQ-021 | dcy-014 | [bat-017_20260708_import-provenance-sensitive-data-policy](bat-017_20260708_import-provenance-sensitive-data-policy/) | - 2026-07-08 07:11 KST: BAT-016을 local verification 통과 근거로 release-candidate 상태로 전환했다. Kanban은 사용자 명시 지시로 사용하지 않았다. +- 2026-07-08 11:33 KST: BAT-017을 REQ-021 delivery batch로 draft 생성했다. Kanban은 사용자 명시 지시로 사용하지 않는다. diff --git a/docs/discovery/dcy-014_20260708_import-provenance-sensitive-data-policy.md b/docs/discovery/dcy-014_20260708_import-provenance-sensitive-data-policy.md index 576db1f..c1d005b 100644 --- a/docs/discovery/dcy-014_20260708_import-provenance-sensitive-data-policy.md +++ b/docs/discovery/dcy-014_20260708_import-provenance-sensitive-data-policy.md @@ -1,14 +1,14 @@ # 0. 문서 상태 이 문서의 현재 상태와 식별 정보를 기록한다. 문서 추적과 승인 흐름에 필요한 최소 메타데이터를 적는다. -- 상태: draft +- 상태: confirmed - 문서 ID: dcy-014_20260708_import-provenance-sensitive-data-policy - 이슈명: Import provenance 및 민감자료 저장 정책 정리 - 작성 시각(KST): 2026-07-08 10:48 KST -- 마지막 갱신 시각(KST): 2026-07-08 10:48 KST +- 마지막 갱신 시각(KST): 2026-07-08 11:26 KST - 대체됨: 없음 - 후속 Discovery 참조: 없음 -- 생성된 REQ 참조: 없음 +- 생성된 REQ 참조: `REQ-021` # 1. 계획 상태 요약 @@ -92,6 +92,64 @@ - NFR-3 (backward compatibility): 기존 운영 DB의 import rows는 additive migration 또는 compatibility adapter로 읽을 수 있어야 하며, destructive rewrite는 별도 승인 없이는 금지한다. - NFR-4 (test safety): CI/AppTest/HTTP smoke fixtures는 synthetic/temp DB만 사용하고 실제 원본 파일명·계좌 식별자·raw payload를 포함하지 않아야 한다. +## 정책 결정 결과 + +### raw_payload `minimize`의 의미 + +`minimize`는 raw evidence를 전부 삭제한다는 뜻이 아니라, **원본 문서/row 덤프를 저장하지 않고 재현성에 필요한 정규화된 최소 provenance만 저장**한다는 뜻이다. + +- 저장 허용: + - source kind: `kb_bank_pdf`, `toss_bank_pdf`, `toss_securities_pdf`, `seoulpay_xlsx`, `csv_generic` + - parser name/version 또는 parser identity + - `source_hash`와 필요 시 짧은 prefix + - import batch id + - source row/page/sheet reference: page number, sheet name, row index, section/action code 등 원본 위치를 좁힐 수 있는 값 + - normalized transaction evidence: date/time, amount, direction/type, masked account hint, normalized merchant/counterparty, validation summary + - duplicate/rollback에 필요한 `external_id`, `dedupe_key`, validation result, parsed row count +- 저장 금지: + - password, token, secret + - 전체 파일 경로 + - 원본 PDF text 전체 + - XLSX/CSV row 전체 dump + - 계좌번호/카드번호 원문 + - 불필요한 raw payload 전체 JSON +- source별 예외: + - source별 parser가 audit에 꼭 필요한 작은 필드를 갖는 경우 REQ-021에서 allowlist로 명시한다. + - allowlist 밖 key는 기본 저장하지 않거나 masking한다. + +### filename/path 정책 + +- `import_batches.filename`에는 전체 path를 저장하지 않는다. +- 운영자 표시에는 `masked-basename`을 사용하고, 실제 동일성/추적성은 `source_hash`로 판정한다. +- 예: `/volume1/private/국민은행_고진환_2026년06월.pdf` → `국민은행_***_2026년06월.pdf` + `source_hash`. + +### MCP provenance 노출 정책 + +- `minimal`: agent 기본값. `source`, `import_batch_id`, `external_id`, `dedupe_key`, `source_hash_prefix` 정도만 제공한다. +- `operator-summary`: 운영자 화면/도구용. masked filename, parser identity, row/page/sheet reference, validation summary를 포함할 수 있다. +- `debug-gated`: 명시 debug gate가 있을 때만 parser diagnostics를 제공한다. raw 원문·password·전체 path는 여전히 제외한다. + +### 기존 운영 DB sanitization 분리 + +- 기존 운영 DB의 raw payload/file path 정리는 REQ-021에 섞지 않는다. +- 별도 GitHub Project Todo `기존 운영 DB import evidence sanitization preview 및 정리`로 분리했다. +- 해당 후속 작업은 scan → preview report → backup → explicit approval → narrow migration → verification 순서로만 진행한다. + +### finance agent privacy boundary 및 destructive migration 원칙 + +- finance agent는 raw financial evidence를 소비하지 않고 normalized provenance metadata만 소비한다. +- destructive migration은 preview/backup/명시 승인 없이는 금지한다. + +### 코드 기준 import source matrix + +| Source | 현재 코드 기준 저장/응답 항목 | 민감 위험 | REQ-021 권고 저장/노출 | +|---|---|---|---| +| KB Bank PDF | `source_hash`, `page_count`, `metadata`, `validation`, `parsed_row_count`, `parsed["file_path"]`, `transactions.raw_payload`, `raw_summary`, `branch`, `balance_after`, `external_id`, `dedupe_key` | file path, holder/account metadata, PDF-derived raw payload, branch/balance detail | masked filename + source_hash, page/row reference, parser identity, validation summary, minimized payload allowlist | +| Toss Bank PDF | `source_hash`, `page_count`, `metadata`, `validation`, `parsed_row_count`, `parsed["file_path"]`, action/counterparty/balance/raw_summary/raw_payload, `external_id`, `dedupe_key` | file path, account masked metadata, counterparty/action raw text, balance detail | masked filename + source_hash, action category, row/page reference, normalized counterparty, minimized payload allowlist | +| Toss Securities PDF | `source_hash`, `page_count`, `metadata`, `validation`, `parsed_row_count`, `parsed["file_path"]`, section/action/code/name/quantity/unit_price/fee/tax/balance/raw_payload | securities detail and counterparty/security raw text, file path, account metadata | masked filename + source_hash, section/action/code where needed, normalized security/counterparty summary, numeric details only when audit-required | +| SeoulPay XLSX | `source_hash`, `sheet_name`, `metadata`, `validation`, `parsed_row_count`, `parsed["file_path"]`, transaction kind/payment kind/no/merchant id/channel/raw_payload | XLSX file path, merchant id, transaction no, raw row payload | masked filename + source_hash, sheet name, row index/reference, payment kind/category summary, minimized merchant/transaction reference | +| CSV preview | `filename=str(path)`, raw preview rows from `csv.DictReader` | arbitrary user columns may include sensitive data; preview currently can echo raw CSV rows | preview output should mask/limit columns by allowlist; import policy should require source profile and sensitive-column filtering | + ## 범위 경계 - In Scope @@ -126,13 +184,13 @@ ## 오픈 질문 - Q-1: `transactions.raw_payload`의 장기 정책은 source별 유지/축소/마스킹/미저장 중 무엇이어야 하는가? - - 상태: Open + - 상태: Answered — 기본 정책은 `minimize`다. raw 원문 전체를 저장하지 않고, rollback/dedupe/audit에 필요한 normalized 최소 필드만 보존한다. - Q-2: `import_batches.filename`은 전체 경로, basename, masked basename, hash 중 무엇을 저장해야 하는가? - - 상태: Open + - 상태: Answered — `masked-basename + source_hash`를 사용한다. 전체 경로는 저장하지 않는다. - Q-3: MCP default response에서 provenance detail을 어느 수준까지 노출할 것인가? - - 상태: Open + - 상태: Answered — 기본은 `minimal`, 운영자용은 `operator-summary`, debug 상세는 별도 gate로 분리한다. - Q-4: 기존 운영 DB에 이미 저장된 raw payload/file path를 정리하는 migration을 이번 범위에 포함할 것인가, 별도 follow-up으로 둘 것인가? - - 상태: Open + - 상태: Answered — 이번 REQ에서는 정책/신규 동작을 정의하고, 기존 운영 DB sanitization은 별도 GitHub Project Todo/후속 SDLC로 분리한다. # 7. 초기 성공 기준 @@ -152,10 +210,10 @@ # 8. REQ로 넘기기 전 확인 체크 -- [ ] source별 raw evidence 보존 정책 결정 -- [ ] filename/path 저장 정책 결정 -- [ ] MCP default/debug/operator response boundary 결정 -- [ ] 기존 운영 DB sanitization을 이번 delivery에 포함할지 결정 +- [x] source별 raw evidence 보존 정책 결정 — `raw_payload=minimize` +- [x] filename/path 저장 정책 결정 — `masked-basename + source_hash` +- [x] MCP default/debug/operator response boundary 결정 — `minimal`/`operator-summary`/`debug-gated` +- [x] 기존 운영 DB sanitization을 이번 delivery에 포함할지 결정 — 별도 GitHub Project Todo로 분리 - [x] 관련 현재 코드와 baseline 문서 확인 - [x] baseline 문서 갱신 필요성 식별 @@ -179,17 +237,13 @@ # 10. 사용자 결정 필요 항목 요약 ## DECIDE -- `{{DECIDE_RAW_PAYLOAD_POLICY}}`: source별 `transactions.raw_payload` 보존 정책을 `retain|minimize|mask|do-not-store` 중 선택한다. -- `{{DECIDE_FILENAME_PATH_POLICY}}`: `import_batches.filename` 저장 정책을 `full-path|basename|masked-basename|hash-only` 중 선택한다. -- `{{DECIDE_MCP_PROVENANCE_EXPOSURE}}`: MCP default response의 provenance 노출 수준을 `minimal|operator-summary|debug-gated` 중 선택한다. -- `{{DECIDE_EXISTING_DB_SANITIZATION_SCOPE}}`: 기존 운영 DB에 저장된 raw payload/file path 정리를 이번 REQ에 포함할지, 별도 후속 Discovery/REQ로 분리할지 선택한다. +- 없음 — 사용자 결정 반영 완료. ## CONFIRM -- `{{CONFIRM_AGENT_PRIVACY_BOUNDARY}}`: finance agent가 raw financial evidence 없이 normalized provenance metadata만 소비한다는 원칙을 확인한다. -- `{{CONFIRM_NO_DESTRUCTIVE_MIGRATION_WITHOUT_APPROVAL}}`: 기존 import evidence 정리는 additive/preview-first로 설계하고 destructive rewrite는 별도 사용자 승인 없이는 금지한다. +- 없음 — finance agent privacy boundary 및 destructive migration 금지 원칙 확인 완료. ## DATA -- `{{DATA_IMPORT_SOURCE_MATRIX}}`: 현재 지원 source별로 저장 중인 raw evidence, metadata, parser output 목록을 REQ 작성 시 코드 기준으로 확정한다. +- 없음 — import source matrix를 코드 기준으로 작성 완료. # 11. Discovery Freeze @@ -199,8 +253,8 @@ - 성공 기준: [# 7. 초기 성공 기준](#7-초기-성공-기준) ## Freeze 플래그 -- Handoff Decision: 보류 -- Handoff Rationale: 초기 Discovery 초안 생성 완료. REQ-021 초안 작성 전 raw payload, filename/path, MCP exposure, 기존 DB sanitization 범위에 대한 사용자 결정이 필요하다. -- Ready for REQ Drafting: false -- Confirmed By: 없음 -- Confirmed At (KST): 없음 +- Handoff Decision: confirmed +- Handoff Rationale: 사용자 결정으로 raw_payload minimize, masked-basename + source_hash, MCP minimal/operator-summary/debug-gated, 기존 DB sanitization 후속 분리, finance agent privacy boundary, destructive migration 금지 원칙이 확정되어 REQ-021 초안 작성으로 인계 가능하다. +- Ready for REQ Drafting: true +- Confirmed By: 고진환 +- Confirmed At (KST): 2026-07-08 10:58 KST diff --git a/docs/discovery/index.md b/docs/discovery/index.md index ccef415..6710a47 100644 --- a/docs/discovery/index.md +++ b/docs/discovery/index.md @@ -19,4 +19,4 @@ | dcy-011_20260703_backend-refactor-phase4-write-import-workflow-adapter-narrowing | 20260703 | Jarca Ledger 백엔드 리팩터링 Phase 4+ 후속 주기 (write service 분리 → import pipeline 분리 → advanced finance workflow 분리 → MCP/Streamlit adapter 표면 축소) | [Discovery](./dcy-011_20260703_backend-refactor-phase4-write-import-workflow-adapter-narrowing.md) | confirmed | | dcy-012_20260703_review-queue-reconciliation-workbench-persistence | 20260703 | 확인필요 큐 / 잔액대조 워크벤치 영속화 및 운영 모델 정립 | [Discovery](./dcy-012_20260703_review-queue-reconciliation-workbench-persistence.md) | confirmed | | dcy-013_20260707_docker-network-mcp-deployment | 20260707 | Jarca Ledger 독립 Docker 배포 및 network MCP endpoint 제공 | [Discovery](./dcy-013_20260707_docker-network-mcp-deployment.md) | confirmed | -| dcy-014_20260708_import-provenance-sensitive-data-policy | 20260708 | Import provenance 및 민감자료 저장 정책 정리 | [Discovery](./dcy-014_20260708_import-provenance-sensitive-data-policy.md) | draft | +| dcy-014_20260708_import-provenance-sensitive-data-policy | 20260708 | Import provenance 및 민감자료 저장 정책 정리 | [Discovery](./dcy-014_20260708_import-provenance-sensitive-data-policy.md) | confirmed | diff --git a/docs/srs/Data/req-021_import-provenance-sensitive-data-policy.md b/docs/srs/Data/req-021_import-provenance-sensitive-data-policy.md new file mode 100644 index 0000000..ece0f58 --- /dev/null +++ b/docs/srs/Data/req-021_import-provenance-sensitive-data-policy.md @@ -0,0 +1,162 @@ +# req-021: Import provenance and sensitive-data policy + +- Status: Approved +- Type: Data +- Priority: High +- Owner: 렛저장인 + +## Intent +- `dcy-014`에서 확정된 import provenance 및 민감자료 저장 정책을 구현 가능한 requirement로 정규화한다. +- Jarca Ledger는 import rollback, duplicate detection, audit investigation에 필요한 lineage를 보존하되, 운영 DB·repo fixture·logs·MCP/Streamlit responses에 원본 금융 증빙을 과도하게 저장하거나 노출하지 않아야 한다. +- finance agent와 operator-facing surface는 raw evidence가 아니라 normalized provenance metadata를 소비해야 한다. + +## Requirement +- 시스템은 모든 import source에 대해 표준 provenance envelope를 정의하고, source kind, parser identity, source hash, import batch id, row/page/sheet reference, external id, dedupe key, validation summary를 일관되게 보존해야 한다. +- `transactions.raw_payload` 저장 정책은 기본 `minimize`로 한다. 원본 문서/row 덤프를 저장하지 않고 source별 allowlist에 포함된 normalized 최소 필드만 저장하거나, allowlist 밖 key는 제거/마스킹해야 한다. +- `import_batches.filename`은 전체 file path를 저장하지 않고 `masked-basename + source_hash` 정책을 사용해야 한다. +- MCP provenance exposure는 기본 `minimal`, operator-facing `operator-summary`, explicit debug gate가 필요한 `debug-gated` 수준으로 분리해야 한다. +- finance agent는 raw financial evidence를 직접 받지 않고 normalized provenance metadata만 받아야 한다. +- 기존 운영 DB에 이미 저장된 raw payload/file path sanitization은 본 REQ의 destructive migration 범위에 포함하지 않는다. 별도 GitHub Project Todo/후속 SDLC에서 scan/preview/backup/approval/migration 순서로 다룬다. +- `docs/data-model.md`, `docs/project-structure.md`, `docs/runtime-flows.md`는 import provenance envelope, sensitive-data boundary, MCP exposure level, existing DB sanitization split을 반영해야 한다. + +## Acceptance Criteria +1. Given synthetic import inputs for KB Bank PDF, Toss Bank PDF, Toss Securities PDF, SeoulPay XLSX, and CSV preview/import paths, When import preview or import is executed, Then each result exposes or stores a normalized provenance envelope with source kind, parser identity, source hash, import batch id where applicable, row/page/sheet reference where available, external id/dedupe key where applicable, and validation summary. +2. Given an import source file path containing personal/environmental path segments, When an import batch is recorded, Then `import_batches.filename` or its successor field contains no full path and uses masked basename plus source hash for operator identification and audit correlation. +3. Given source transactions contain raw parser payloads, file paths, account/card identifiers, passwords, tokens, secrets, or arbitrary CSV columns, When data is persisted to `transactions.raw_payload` or returned through preview/response payloads, Then only the source-specific allowlisted minimized fields remain and forbidden fields are removed or masked. +4. Given MCP default tools return transaction/import provenance, When called by a normal finance-agent client, Then the response uses the `minimal` level and excludes raw financial evidence, full file paths, passwords, tokens/secrets, raw document text, and non-allowlisted payload fields. +5. Given an operator-facing view or tool explicitly requests operator summary, When provenance is shown, Then masked filename, parser identity, source hash prefix, row/page/sheet reference, and validation summary may be included without exposing raw document text or full paths. +6. Given debug diagnostics are needed, When a debug-gated path is used, Then debug details are available only through an explicit gate and still exclude passwords, tokens/secrets, and full source document dumps by default. +7. Given existing operating DB rows may contain legacy raw payload/file path data, When REQ-021 delivery runs, Then it must not destructively rewrite existing operating DB content; it may add read-time sanitization or new-write policy only. Existing-data cleanup must remain in the separate Todo follow-up. +8. Given CI and test fixtures run import-related tests, When pytest/AppTest/HTTP smoke/contract tests execute, Then they use synthetic/temp DBs and static/contract checks verify that forbidden sensitive fields are not emitted in default MCP responses or committed fixtures. +9. Given baseline documentation is updated, When `docs/data-model.md`, `docs/project-structure.md`, and `docs/runtime-flows.md` are reviewed, Then they describe the provenance envelope, `raw_payload=minimize`, masked filename + source hash, MCP exposure levels, and the explicit split of existing DB sanitization. +10. Given implementation is complete, When `python -m ruff check .`, full pytest, contract regression, dashboard tests/smoke where affected, and StagePilot doctor are run, Then current-batch regressions must be clear; pre-existing unrelated doctor debt may be reported separately. + +## Impacted Area +- Docs: + - `docs/discovery/dcy-014_20260708_import-provenance-sensitive-data-policy.md` + - `docs/data-model.md` + - `docs/project-structure.md` + - `docs/runtime-flows.md` + - 후속 batch planning/design/verification 문서 +- Modules: + - `jarca_ledger_mcp/application/imports_service.py` + - `jarca_ledger_mcp/infrastructure/parsers/*` + - `jarca_ledger_mcp/infrastructure/sqlite/schema.py` + - `jarca_ledger_mcp/infrastructure/sqlite/migrations.py` + - `jarca_ledger_mcp/adapters/mcp/server.py` + - `jarca_ledger_mcp/adapters/mcp/tool_contracts.py` + - Streamlit import/operations gateway/page surfaces where provenance is displayed +- Tests: + - source-specific import regression tests + - sensitive-field/static scan tests + - MCP contract/default response tests + - fixture/log safety checks where applicable + +## Notes +- Source Discovery: `docs/discovery/dcy-014_20260708_import-provenance-sensitive-data-policy.md`. +- Confirmed decisions from dcy-014: + - `raw_payload`: `minimize`. + - `import_batches.filename`: `masked-basename + source_hash`, no full path. + - MCP exposure: default `minimal`, operator-facing `operator-summary`, debug `debug-gated`. + - existing operating DB sanitization: split to separate GitHub Project Todo. + - finance agent privacy boundary: normalized provenance only. + - destructive migration: prohibited without preview/backup/explicit approval. +- Code-derived source matrix from dcy-014: + - KB Bank PDF: source hash, page count, metadata, validation, parsed row count, file path, raw payload/summary, branch/balance, external id, dedupe key currently appear in import flow. + - Toss Bank PDF: source hash, page count, metadata, validation, parsed row count, file path, action/counterparty/balance/raw payload/summary, external id, dedupe key currently appear in import flow. + - Toss Securities PDF: source hash, page count, metadata, validation, parsed row count, file path, section/action/code/name/quantity/unit price/fee/tax/balance/raw payload currently appear in import flow. + - SeoulPay XLSX: source hash, sheet name, metadata, validation, parsed row count, file path, transaction kind/payment kind/no/merchant id/channel/raw payload currently appear in import flow. + - CSV preview: current preview can echo raw CSV rows and full local path, so it requires allowlist/masking policy before broader exposure. +- Out of scope: + - Existing operating DB sanitization migration. + - External legal review or third-party security audit. + - File vault/encryption system for original source documents. + - External public API auth/TLS design. + +## Change Log + +### CHG-20260708-01 +- Date: 2026-07-08 +- Author: 가웨인 + +#### Change Summary +- dcy-014 confirmed 범위를 REQ-021 Proposed 요구사항으로 생성했다. + +#### Intent +- import provenance와 민감자료 저장/노출 정책을 delivery input으로 사용할 수 있는 단일 Data requirement로 정규화하기 위해 생성했다. + +#### Acceptance Criteria Delta +- Added: + - provenance envelope, raw_payload minimize, masked filename + source hash, MCP exposure levels, finance-agent privacy boundary, existing DB sanitization split, fixture/log/MCP sensitive-data checks를 acceptance criteria로 추가했다. +- Updated: + - none +- Removed: + - none + +#### Impacted Area Delta +- Docs: + - `docs/srs/index.md` + - `docs/srs/Data/req-021_import-provenance-sensitive-data-policy.md` + - `docs/discovery/dcy-014_20260708_import-provenance-sensitive-data-policy.md` +- Modules: + - none yet; implementation pending +- Tests: + - none yet; implementation pending + +#### Delivery Trace Delta +- Batches: + - none yet +- Releases: + - none + +#### Revalidation Impact +- Reverification Needed: yes +- Existing Implementation Invalidated: no +- Status Recommendation: keep-current +- Follow-up Action: run `confirm-req` for REQ-021 before batch selection/drafting. + +#### Validation Plan +- Before implementation approval, verify source-specific provenance envelope behavior, sensitive-field filtering, MCP exposure-level defaults, synthetic fixture safety, full pytest, lint, relevant dashboard/MCP smoke, DB safety smoke, and baseline document updates. + +### CHG-20260708-02 +- Date: 2026-07-08 +- Author: 가웨인 + +#### Change Summary +- REQ-021 승인 게이트를 통과해 Status를 Approved로 전환했다. + +#### Intent +- dcy-014 결정사항이 모두 해소되었고, raw_payload minimize, masked filename + source hash, MCP exposure boundary, existing DB sanitization split, finance-agent privacy boundary, destructive migration 금지 원칙이 delivery input으로 충분히 구체화되어 승인했다. + +#### Acceptance Criteria Delta +- Added: + - none +- Updated: + - none +- Removed: + - none + +#### Impacted Area Delta +- Docs: + - `docs/srs/Data/req-021_import-provenance-sensitive-data-policy.md` + - `docs/srs/index.md` +- Modules: + - none +- Tests: + - none + +#### Delivery Trace Delta +- Batches: + - none yet +- Releases: + - none + +#### Revalidation Impact +- Reverification Needed: yes +- Existing Implementation Invalidated: no +- Status Recommendation: keep-current +- Follow-up Action: draft batch for REQ-021 before implementation. + +#### Validation Plan +- Validate through source-specific import tests, sensitive-field filtering/static scan, MCP exposure-level contract checks, fixture/log safety checks, full pytest, lint, relevant dashboard/MCP smoke, and baseline document updates. + diff --git a/docs/srs/index.md b/docs/srs/index.md index 29465b0..7a54a09 100644 --- a/docs/srs/index.md +++ b/docs/srs/index.md @@ -33,7 +33,7 @@ ## Next Requirement ID -- Current: `REQ-021` +- Current: `REQ-022` - Rule: `Requirement Register`에 있는 가장 큰 REQ 번호보다 정확히 1 커야 한다. ## Requirement Register @@ -60,6 +60,7 @@ | REQ-018 | MCP/Streamlit adapter surface narrowing and baseline refresh for dcy-011 completion | Interface | Implemented | High | 렛저장인 | [req-018_adapter-surface-narrowing-baseline-refresh.md](Interface/req-018_adapter-surface-narrowing-baseline-refresh.md) | | REQ-019 | Docker network MCP deployment and operating DB migration | Deployment | Implemented | High | 렛저장인 | [req-019_docker-network-mcp-deployment.md](Deployment/req-019_docker-network-mcp-deployment.md) | | REQ-020 | Review decision persistence and reconciliation workbench | Interface | Implemented | High | 렛저장인 | [req-020_review-decision-persistence-reconciliation-workbench.md](Interface/req-020_review-decision-persistence-reconciliation-workbench.md) | +| REQ-021 | Import provenance and sensitive-data policy | Data | Approved | High | 렛저장인 | [req-021_import-provenance-sensitive-data-policy.md](Data/req-021_import-provenance-sensitive-data-policy.md) | ## Recent Change Log Summary @@ -99,3 +100,5 @@ - 2026-07-08 00:00 KST: REQ-020을 승인 게이트 검토 후 Approved 상태로 전환했다. - 2026-07-08 07:11 KST: REQ-020을 BAT-016 local verification 근거로 Implemented 상태로 전환했다. CI/PR review는 후속 외부 gate로 남아 있다. +- 2026-07-08 11:26 KST: dcy-014 confirmed 범위를 `REQ-021` Proposed 상태로 생성하고 import provenance 및 민감자료 저장 정책 요구사항으로 등록했다. +- 2026-07-08 11:31 KST: REQ-021을 승인 게이트 검토 후 Approved 상태로 전환했다.