docs: confirm import provenance req and batch
This commit is contained in:
Jarca
@@ -0,0 +1,49 @@
|
||||
# BAT-017 Design: Import provenance and sensitive-data policy
|
||||
|
||||
- Status: draft
|
||||
- Created At (KST): 2026-07-08 11:33 KST
|
||||
- Included REQ: REQ-021
|
||||
|
||||
## Design Direction
|
||||
|
||||
Introduce a narrow provenance/sanitization layer used by import services and response adapters rather than scattering per-source masking rules across UI/MCP code.
|
||||
|
||||
## Proposed Components
|
||||
|
||||
- `provenance` helper functions under application/import boundary or a small shared module:
|
||||
- `mask_source_filename(path_or_name)`
|
||||
- `build_source_identity(source, file_path, source_hash, parser_name, parser_version)`
|
||||
- `minimize_raw_payload(source, payload)`
|
||||
- `sanitize_provenance_for_exposure(payload, level)`
|
||||
- Source-specific allowlists:
|
||||
- KB Bank PDF: date/time, amount, direction, masked account hint, page/row reference, validation keys.
|
||||
- Toss Bank PDF: action category, normalized counterparty, masked account hint, page/row reference, validation keys.
|
||||
- Toss Securities PDF: section/action/code, normalized security/counterparty summary, numeric audit fields when required.
|
||||
- SeoulPay XLSX: sheet name, row reference, payment kind/category summary, minimized merchant/transaction reference.
|
||||
- CSV: explicit preview/import allowlist and sensitive-column suppression.
|
||||
|
||||
## Storage Policy
|
||||
|
||||
- Do not persist full local source paths.
|
||||
- Store masked basename and source hash for operator correlation.
|
||||
- Keep `source_hash` stable for dedupe/audit.
|
||||
- `raw_payload` stores only minimized allowlisted fields; forbidden fields are removed/masked.
|
||||
- Existing rows are not destructively rewritten by this batch.
|
||||
|
||||
## Interface Policy
|
||||
|
||||
- Default MCP responses use `minimal` exposure.
|
||||
- Operator summary may include masked filename, parser identity, row/page/sheet reference, validation summary.
|
||||
- Debug diagnostics require explicit gate and still exclude passwords/tokens/secrets/full path/raw document dump.
|
||||
|
||||
## Test Strategy
|
||||
|
||||
- Use synthetic fixtures only.
|
||||
- Include negative assertions for forbidden keys and full path patterns.
|
||||
- Verify duplicate/rollback metadata remains usable after minimization.
|
||||
|
||||
## Open Design Checks Before Implementation
|
||||
|
||||
- Decide whether provenance helpers live in `application/imports_service.py` initially or a new module.
|
||||
- Confirm whether schema version bump is required or policy can be implemented through current columns plus memo JSON shape.
|
||||
- Confirm MCP contract change is additive and does not break existing consumers.
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
# BAT-017 Implementation Evidence
|
||||
|
||||
- Status: not-started
|
||||
- Created At (KST): 2026-07-08 11:33 KST
|
||||
|
||||
## Implementation Summary
|
||||
|
||||
Pending.
|
||||
|
||||
## Changed Files
|
||||
|
||||
Pending.
|
||||
|
||||
## Notes
|
||||
|
||||
Implementation must not use Kanban unless the user explicitly re-allows it.
|
||||
@@ -0,0 +1,35 @@
|
||||
# BAT-017: Import provenance and sensitive-data policy
|
||||
|
||||
- Profile: standard
|
||||
- Status: draft
|
||||
- Included REQ: REQ-021
|
||||
- Source Discovery: dcy-014
|
||||
- Created At (KST): 2026-07-08 11:33 KST
|
||||
|
||||
## Objective
|
||||
Deliver the approved REQ-021 scope for normalized import provenance, `raw_payload=minimize`, masked filename + source hash, MCP provenance exposure levels, finance-agent privacy boundary, destructive migration guardrails, and baseline documentation.
|
||||
|
||||
## Included Scope
|
||||
- Source-specific provenance envelope for KB Bank PDF, Toss Bank PDF, Toss Securities PDF, SeoulPay XLSX, and CSV preview/import surfaces.
|
||||
- `transactions.raw_payload` minimize policy using explicit allowlists and forbidden-field filtering.
|
||||
- `import_batches.filename` policy change away from full paths toward masked basename + source hash.
|
||||
- MCP/Streamlit provenance exposure levels: default `minimal`, operator `operator-summary`, debug `debug-gated`.
|
||||
- Sensitive-field tests/static checks for raw payload, full paths, password/token/secret, raw document text, and arbitrary CSV column leakage.
|
||||
- Baseline updates in `docs/data-model.md`, `docs/project-structure.md`, and `docs/runtime-flows.md`.
|
||||
|
||||
## Excluded Scope
|
||||
- Existing operating DB sanitization migration; tracked separately in GitHub Project 4 Todo `기존 운영 DB import evidence sanitization preview 및 정리`.
|
||||
- External legal/security audit.
|
||||
- Original source document vault/encryption system.
|
||||
- Public API auth/TLS design.
|
||||
- Kanban-based delivery execution.
|
||||
|
||||
## Documents
|
||||
- [planning.md](planning.md)
|
||||
- [design.md](design.md)
|
||||
- [implementation.md](implementation.md)
|
||||
- [verification.md](verification.md)
|
||||
|
||||
## Delivery Notes
|
||||
- User explicitly prohibited Kanban; delivery execution must use direct branch/PR or a dedicated non-Kanban worktree if delegated.
|
||||
- Implementation should be sliced to avoid the previous BAT-016 large-card failure pattern: provenance helpers first, source adapters second, MCP/Streamlit exposure third, tests/docs last.
|
||||
@@ -0,0 +1,57 @@
|
||||
# BAT-017 Planning: Import provenance and sensitive-data policy
|
||||
|
||||
- Status: draft
|
||||
- Created At (KST): 2026-07-08 11:33 KST
|
||||
- Included REQ: REQ-021
|
||||
|
||||
## Planning Summary
|
||||
|
||||
BAT-017 is a standard batch because REQ-021 affects data persistence policy, import adapters, MCP/Streamlit response contracts, tests, and baseline documentation.
|
||||
|
||||
## Work Slices
|
||||
|
||||
1. Define provenance policy helpers
|
||||
- masked basename helper
|
||||
- source hash propagation helper
|
||||
- raw payload minimization/filter helper
|
||||
- exposure-level sanitizer for `minimal`, `operator-summary`, `debug-gated`
|
||||
2. Apply policy to import write paths
|
||||
- KB Bank PDF
|
||||
- Toss Bank PDF
|
||||
- Toss Securities PDF
|
||||
- SeoulPay XLSX
|
||||
- CSV preview/import boundary
|
||||
3. Apply policy to read/response surfaces
|
||||
- import batch listing
|
||||
- preview responses
|
||||
- MCP tool contracts/default output
|
||||
- Streamlit/operator summary where applicable
|
||||
4. Add regression checks
|
||||
- source-specific import tests
|
||||
- forbidden-field filtering tests
|
||||
- default MCP response privacy checks
|
||||
- fixture/log/static scan checks
|
||||
5. Update baseline documents
|
||||
- `docs/data-model.md`
|
||||
- `docs/project-structure.md`
|
||||
- `docs/runtime-flows.md`
|
||||
|
||||
## Design Gate
|
||||
|
||||
Design is required. The batch changes persistence semantics and interface response boundaries, so `design.md` must be completed before implementation.
|
||||
|
||||
## Risks
|
||||
|
||||
- Over-sanitizing may weaken duplicate/rollback/audit traceability.
|
||||
- Under-sanitizing may leak file paths, raw financial evidence, or arbitrary CSV columns.
|
||||
- Existing operating DB rows may contain legacy data; this batch must avoid destructive cleanup and keep the follow-up Todo separate.
|
||||
|
||||
## Verification Plan
|
||||
|
||||
- `python -m ruff check .`
|
||||
- source-specific pytest for import provenance behavior
|
||||
- full `python -m pytest -q`
|
||||
- MCP contract/default privacy checks
|
||||
- dashboard/AppTest/HTTP smoke if affected
|
||||
- DB safety smoke if schema/migration changes are included
|
||||
- StagePilot doctor with current-batch findings separated from pre-existing debt
|
||||
@@ -0,0 +1,18 @@
|
||||
# BAT-017 Verification Evidence
|
||||
|
||||
- Status: not-started
|
||||
- Created At (KST): 2026-07-08 11:33 KST
|
||||
|
||||
## Verification Commands
|
||||
|
||||
Pending.
|
||||
|
||||
## Expected Gates
|
||||
|
||||
- `python -m ruff check .`
|
||||
- `python -m pytest -q`
|
||||
- source-specific import provenance tests
|
||||
- MCP privacy/default exposure tests
|
||||
- fixture/log sensitive-field scan
|
||||
- affected dashboard smoke, if applicable
|
||||
- StagePilot doctor with pre-existing debt separated from BAT-017 findings
|
||||
@@ -24,5 +24,7 @@
|
||||
| BAT-014 | standard | release-candidate | REQ-018 | dcy-011 | [bat-014_20260703_adapter-surface-narrowing-baseline-refresh](bat-014_20260703_adapter-surface-narrowing-baseline-refresh/) |
|
||||
| BAT-015 | standard | release-candidate | REQ-019 | dcy-013 | [bat-015_20260707_docker-network-mcp-deployment](bat-015_20260707_docker-network-mcp-deployment/) |
|
||||
| BAT-016 | standard | release-candidate | REQ-020 | dcy-012 | [bat-016_20260708_review-decision-reconciliation-workbench](bat-016_20260708_review-decision-reconciliation-workbench/) |
|
||||
| BAT-017 | standard | draft | REQ-021 | dcy-014 | [bat-017_20260708_import-provenance-sensitive-data-policy](bat-017_20260708_import-provenance-sensitive-data-policy/) |
|
||||
|
||||
- 2026-07-08 07:11 KST: BAT-016을 local verification 통과 근거로 release-candidate 상태로 전환했다. Kanban은 사용자 명시 지시로 사용하지 않았다.
|
||||
- 2026-07-08 11:33 KST: BAT-017을 REQ-021 delivery batch로 draft 생성했다. Kanban은 사용자 명시 지시로 사용하지 않는다.
|
||||
|
||||
Reference in New Issue
Block a user