Merge pull request #21 from fourblocks2020/feat/dcy-012-portfolio-ledger-alembic-migration

feat(migrations): add BAT-025 ledger metadata Alembic migration
This commit is contained in:
2026-07-15 20:26:35 +09:00
committed by GitHub
15 changed files with 937 additions and 5 deletions
+11 -1
View File
@@ -16,6 +16,10 @@ def get_alembic_config(settings: Settings) -> Config:
config = Config(str(project_root / "alembic.ini"))
config.set_main_option("script_location", str(project_root / "migrations"))
config.set_main_option("sqlalchemy.url", settings.active_database_url.replace("%", "%%"))
# migrations/env.py must use this explicit settings object instead of the
# process-global cached Settings. This keeps programmatic callers (notably
# isolated integration tests and runtime bootstrap) on their requested DB.
config.attributes["portfoliohub_settings"] = settings
return config
@@ -29,7 +33,13 @@ def bootstrap_or_upgrade_to_head(settings: Settings) -> str:
return "head"
if _has_legacy_portfolio_schema(settings):
command.stamp(get_alembic_config(settings), "head")
# Runtime ensure_schema() can create the BAT-025 columns before Alembic is
# introduced. Stamp only the revision immediately before BAT-025, then run
# the real upgrade so 0028 backfills a nullable pre-created `source` and
# converges constraints/defaults without replaying the older DDL chain.
config = get_alembic_config(settings)
command.stamp(config, "20260628_0027")
command.upgrade(config, "head")
return "head"
upgrade_to_head(settings)
@@ -0,0 +1,40 @@
# BAT-025 Design
## D-1. Revision graph and parent
구현 시작 직전 `uv run alembic heads` 및 `uv run alembic history`로 유효 head를 재확인한다. 신규 revision은 그 실제 head 하나를 `down_revision`으로 사용하며, 문서/코드에 20260628_0027을 선결 가정으로 고정하지 않는다.
## D-2. Migration idempotency
신규 revision은 inspector/information_schema 기반으로 table/column 존재를 확인한다.
- `portfolio_ledger`가 정상적으로 존재하는 일반 경로에서는 네 컬럼을 추가한다.
- runtime `ensure_schema()`가 이미 컬럼을 생성한 경로에서는 컬럼 재생성 없이 nullable/default를 검증·보정한다.
- `source`는 upgrade 전에 NULL을 `'broker'`로 backfill하고 이후 `NOT NULL DEFAULT 'broker'`를 보장한다.
- 기존 행은 삭제·재작성하지 않으며 upgrade 전후 row count와 business field를 비교한다.
- down migration은 운영 forward-only 정책과 충돌하지 않도록 구현 범위를 명시하고, 운영 runbook은 column deletion을 rollback으로 제시하지 않는다.
## D-3. Runtime boundary
`app/repositories/postgres.py:ensure_schema()`는 bootstrap safety 경로로 유지한다. Alembic revision은 persistent schema history의 source of truth로 둔다. migration 성공만으로 version table을 임의 stamp하지 않으며, schema와 Alembic current가 함께 일치하는지 검증한다.
## D-4. Test matrix
| 시작 상태 | 검증 |
| --- | --- |
| fresh Alembic schema | upgrade, 네 컬럼 타입/nullable/default, current/head |
| 기존 ledger rows | row count/business data 보존, source broker backfill |
| runtime-precreated columns | 중복 DDL 부재, constraints/default 정합, current/head |
| broker write | 기존 `/ledger/trades` 회귀 없음 |
| manual API | source/manual metadata 및 REQ-056 핵심 regression |
| runbook | preflight 조건, 실패 중단, forward-only recovery 문서 정합 |
## D-5. Operational gate
이번 batch는 local/non-production verification만 수행한다. 운영 DB backup/upgrade, service restart, 신규 API activation, deploy/push/merge는 M5 Release의 별도 결재 없이는 실행하지 않는다. runbook은 실행 가능한 명령 템플릿과 중단 조건을 제공하되 실제 운영 결과를 가장하지 않는다.
## Design decisions / open findings
- 실제 revision ID는 구현 preflight 결과로 결정한다.
- 실제 DB driver availability에 따라 migration integration test harness는 repository 표준 fixture 또는 isolated PostgreSQL 경로를 사용하되, 사용하지 못한 환경은 QC evidence에 명시한다.
- stamp 필요성은 schema inspection과 version table evidence 이후에만 판단하며 자동 조치하지 않는다.
@@ -0,0 +1,55 @@
# BAT-025 Implementation Log
- Date: 2026-07-15
- Assignee: dev-impl
- Scope: REQ-057 `portfolio_ledger` 수동 장부 메타데이터의 명시 Alembic migration
- Implementation state: remediation complete; independent QC pending
## Baseline and remediation provenance
- Initial implementation baseline: `f6195ff8090d8761d467e15f44255642ca350e80`.
- The previously recorded SHA `850e970` does **not** identify the reviewed implementation and is superseded by the baseline above.
- The remediation is committed locally as `779d6f7711d700d6f56be22023cb826cc6f2fe28` on the isolated worktree branch. No push, PR, main merge, production migration, deployment, or service activation was performed.
## Implemented changes
1. Explicit revision `20260715_0028` remains the REQ-057 head.
- Parent: `20260628_0027`.
- Adds `source`, `target_ledger_entry_id`, `confirmation_evidence`, and `actor_user_id` additively.
- Handles runtime-precreated columns by backfilling nullable `source`, then converging its default and `NOT NULL` constraint.
- Downgrade remains a forward-only no-op; no column deletion was added.
2. Remediation of actual PostgreSQL migration-chain/runtime behavior.
- `migrations/versions/20260605_0025_ledger_consolidation.py`: makes the identity `ledger_entry_id` explicitly `nullable=False`. PostgreSQL identity columns are implicitly non-null; the prior explicit trailing `NULL` was incompatible with fresh real-PostgreSQL migration.
- `app/db/migrations.py`: programmatic Alembic configuration now carries the caller-provided `Settings`. For a legacy/runtime-precreated schema, bootstrap stamps `20260628_0027` and then executes `upgrade head`, so revision `0028` performs its required source backfill/default/constraint convergence instead of being skipped by a direct head stamp.
- `migrations/env.py`: consumes the explicit settings when supplied and preserves existing application loggers when Alembic runs in-process.
3. Real isolated PostgreSQL integration coverage.
- `tests/test_bat025_portfolio_ledger_manual_metadata_migration.py` creates a UUID-named temporary database only through an explicitly supplied loopback `postgres` maintenance DSN.
- The test rejects non-loopback hosts and any maintenance database other than `postgres`; each test terminates connections and drops only its own UUID-named database.
- Coverage includes pre-0028 upgrade/data preservation, runtime-precreated nullable `source` convergence and Alembic current-head state, plus representative broker/manual ledger writes after upgrade.
4. Existing operation runbook is retained.
- `docs/runbooks/bat-025-req057-portfolio-ledger-manual-metadata-runbook.md`
- It remains a runbook only; no production command in it was executed.
## Executed evidence
All commands below used the local Docker PostgreSQL service only. The DSN was supplied from an environment variable and is intentionally omitted here.
- `BAT025_LOCAL_POSTGRES_DSN=[REDACTED local loopback maintenance DSN] PYTHONPATH=. uv run --extra dev --with httpx2 python -m pytest -q --override-ini addopts='' -p no:cacheprovider tests/test_bat025_portfolio_ledger_manual_metadata_migration.py`
- Result: **3 passed in 6.10s**.
- Verifies real 0027→0028 data preservation/source backfill, runtime-precreated compatibility/current revision, and broker/manual write regression in throwaway DBs.
- `PYTHONPATH=. uv run --extra dev --with httpx2 python -m pytest -q --override-ini addopts='' -p no:cacheprovider`
- Result: **597 passed, 11 warnings in 41.41s**.
- Warnings are pre-existing Starlette `HTTP_422_UNPROCESSABLE_ENTITY` deprecations in endpoint tests; no test failed.
- `uv run --extra dev ruff check app/db/migrations.py migrations/env.py migrations/versions/20260605_0025_ledger_consolidation.py tests/test_bat025_portfolio_ledger_manual_metadata_migration.py`
- Result: **All checks passed**.
- `python3 -m compileall -q app migrations` and `git diff --check`
- Result: exit 0.
## Limits / non-claims
- This evidence is for isolated local PostgreSQL only; it is not an operational DB upgrade.
- Lock contention, backup restore, production access control, deployment, service activation, and M5 release remain unexecuted.
- This log is developer evidence only. It does not mark BAT-025 verified or approved; independent QC remains required.
@@ -0,0 +1,51 @@
# BAT-025 — portfolio_ledger 명시 Alembic migration
- Batch ID: BAT-025
- Profile: standard
- Status: release-candidate (PR #21 independent QC PASS; lead confirmation recorded)
- Discovery: dcy-012_20260714_manual-ledger-write-api-owner-scoped (REQ-057 follow-on)
- Included REQ: REQ-057
- Folder: docs/batches/bat-025_20260715_portfolio-ledger-alembic-migration/
- Created: 2026-07-15
## Summary
REQ-056/BAT-024가 사용한 `portfolio_ledger` additive metadata 네 컬럼을 runtime `ensure_schema()` 자동 DDL에만 의존하지 않고 명시 Alembic revision으로 추적·검증한다. 기존 broker/manual 데이터 보존, runtime-precreated schema 호환성, 운영 migration runbook 및 독립 QC를 delivery 범위로 둔다.
## Scope
- `source TEXT NOT NULL DEFAULT 'broker'`
- `target_ledger_entry_id BIGINT NULL`
- `confirmation_evidence TEXT NULL`
- `actor_user_id TEXT NULL`
- fresh/기존/runtime-precreated schema migration compatibility
- broker ledger regression 및 manual ledger regression
- backup/preflight/current/schema/lock/smoke/forward-only recovery runbook
## DoR
- [x] REQ-057 Approved (Justin Ko)
- [x] REQ-056 Implemented 및 BAT-024 release-candidate 유지 확인
- [x] implementation 시점 `alembic heads`/graph 재확인 요구사항 확인
- [x] 운영 DB upgrade·service activation은 별도 M5 Release gate로 분리
## DoD
- [x] confirmed planning/design 문서 local main commit (`600375b`)
- [x] dev-impl의 migration/code/test/runbook commit (`f6195ff`, remediation `779d6f7`)
- [x] PR #21 committed-diff independent QC PASS (`deleg_ae0c6f27`; patch-id mapping, loopback PostgreSQL, Ruff, hygiene)
- [x] lead verification confirmation re-recorded after PR #21 QC PASS (2026-07-15)
- [x] M5 Release 미착수 — 별도 release approval 전 운영 migration/deploy/service activation 금지
## Gate / Constraints
- 운영 DB, deploy, service activation, push/merge, credential/account disclosure 금지.
- down migration/column deletion은 운영 rollback 경로로 사용하지 않는다. 복구는 호환 코드 rollback + additive schema 유지의 forward-only 정책이다.
- migration 구현 전 실제 단일 head와 graph를 재확인하고, 결과를 implementation/verification evidence에 기록한다.
## Documents
- Planning: [planning.md](./planning.md)
- Design: [design.md](./design.md)
- Implementation: [implementation.md](./implementation.md) (dev-impl 산출)
- Verification: [verification.md](./verification.md) (dev-qc/lead 산출)
@@ -0,0 +1,49 @@
# BAT-025 Planning
## Objective
REQ-057의 명시 Alembic migration과 운영 runbook을 구현·검증한다. BAT-024의 API 계약과 Implemented 상태는 변경하지 않는다.
## Delivery sequence
1. **Preflight** — `uv run alembic heads`, `uv run alembic history`, revision graph 및 `ensure_schema()` DDL 소유 경계를 확인하고 실제 head를 evidence로 고정한다.
2. **Migration** — 실제 head를 parent로 하는 신규 revision을 작성한다. fresh/기존/runtime-precreated schema에서 중복 DDL 없이 네 컬럼, source backfill, NOT NULL/default를 보장한다.
3. **Runtime compatibility** — `ensure_schema()` 선행 생성 경로와 Alembic version tracking의 상호작용을 테스트한다. 임의 stamp는 schema/version evidence와 승인된 절차 없이 사용하지 않는다.
4. **Regression tests** — 기존 broker ledger write, manual ledger API, row/data preservation, schema metadata/current 정합을 검증한다.
5. **Runbook** — backup, current, schema/row count, active query/lock, upgrade, post-check, smoke, 실패 대응 및 forward-only recovery를 문서화하고 dry-run 검토한다.
6. **Independent QC** — dev-qc가 committed diff를 직접 검토하고 표준 테스트 및 migration-focused 테스트를 재실행한다.
## Work packets
### WP-1: Batch artifacts (lead)
- 이 문서와 `design.md`, `index.md`를 confirmed 상태로 작성한다.
- `docs/batches/index.md`에 BAT-025를 `confirmed`로 등록한다.
- local main에 docs-only commit을 남긴다.
### WP-2: Implementation (dev-impl)
- 신규 Alembic revision, 필요한 runtime compatibility 수정, tests, `docs/runbooks/` runbook을 작성한다.
- implementation log와 실행 명령/결과를 기록하고 implementation commit을 반환한다.
- 운영 DB나 서비스 activation은 실행하지 않는다.
### WP-3: QC (dev-qc)
- dev-impl committed diff와 문서/REQ 정합성을 독립 검토한다.
- `uv run pytest` 및 가능한 migration-focused checks를 재실행한다.
- blocker가 있으면 PASS가 아닌 명시적 finding으로 반환한다.
## Acceptance evidence targets
- AC-1: head/graph preflight 출력
- AC-2~5: fresh/legacy/runtime-precreated schema upgrade와 source backfill/중복 오류 부재
- AC-6: `alembic current`, information_schema, row count 결과
- AC-7~9: runbook preflight/실패 대응/forward-only recovery 검토 evidence
- AC-10: 운영 실행을 하지 않았다는 scope evidence 및 release gate 기록
- AC-11: implementation + tests + QC committed evidence
## Risks / blockers
- local repository의 schema graph와 runtime-created schema가 서로 다른 시작 상태를 가질 수 있다.
- `ensure_schema()`와 Alembic 양쪽 소유 경계 변경은 기존 broker flow 회귀 위험이 있다.
- 실제 PostgreSQL 운영 DB evidence는 이번 batch 밖이며 M5 Release 승인 없이는 확보하지 않는다.
@@ -0,0 +1,65 @@
# BAT-025 Developer Self-Verification
- Status: **release-candidate — PR #21 independent QC PASS; lead confirmation recorded**
- Batch: BAT-025
- Requirement: REQ-057
- Date: 2026-07-15
- Scope boundary: local isolated PostgreSQL evidence only. No production DB, deployment, service activation, push, merge, or release action was performed.
## Acceptance mapping
| REQ-057 acceptance area | Evidence | Result |
| --- | --- | --- |
| Explicit `0028` revision is the chain head | `alembic_version` queried in a throwaway PostgreSQL DB after upgrade | Pass — current revision `20260715_0028` |
| Existing broker/manual business data survives upgrade | Real `20260628_0027 → head` upgrade after a representative pre-existing ledger row is inserted; row fields and metadata are asserted after upgrade | Pass |
| `source` is backfilled/defaulted/not-null | Real upgrade asserts preserved row `source='broker'`; schema inspection checks text type, non-null constraint, and default | Pass |
| Runtime-precreated metadata schema converges | Real test creates all four columns with nullable `source`, invokes bootstrap, and asserts source backfill/default/not-null plus `current=head` | Pass |
| Representative broker/manual writes still work | Post-upgrade synthetic repository writes assert broker source and manual metadata fields | Pass |
| Earlier fresh PostgreSQL chain can reach 0028 | `0025` identity-column PostgreSQL incompatibility corrected and exercised by fresh throwaway DB migration path | Pass |
| API/manual-ledger regression | `tests/test_v1_portfolios_manual_ledger.py`, `tests/test_v1_portfolios_trades.py`, `tests/test_v1_portfolios_ledger.py` | Pass — 33 tests |
| Whole-repository regression | pytest with project dev extra plus temporary TestClient dependency overlay | Pass — 597 tests, 11 warnings |
| Lint/compile/diff hygiene | Ruff, compileall, `git diff --check` | Pass |
## Executed commands and results
- `BAT025_LOCAL_POSTGRES_DSN=[REDACTED local loopback maintenance DSN] PYTHONPATH=. uv run --extra dev --with httpx2 python -m pytest -q --override-ini addopts='' -p no:cacheprovider tests/test_bat025_portfolio_ledger_manual_metadata_migration.py`
- **3 passed in 6.10s**.
- Each test creates and removes a UUID-named database on local loopback PostgreSQL; it refuses non-loopback hosts and non-`postgres` maintenance databases.
- `PYTHONPATH=. uv run --extra dev --with httpx2 python -m pytest -q --override-ini addopts='' -p no:cacheprovider`
- **597 passed, 11 warnings in 41.41s**.
- Warnings are Starlette deprecations for `HTTP_422_UNPROCESSABLE_ENTITY`; they are not BAT-025 failures.
- `uv run --extra dev ruff check app/db/migrations.py migrations/env.py migrations/versions/20260605_0025_ledger_consolidation.py tests/test_bat025_portfolio_ledger_manual_metadata_migration.py`
- **All checks passed**.
- `python3 -m compileall -q app migrations` and `git diff --check`
- exit 0.
## Evidence boundaries and blockers
- The local container is an isolated test target, not a staging or production database.
- No production backup/restore, lock-duration measurement under production load, migration run against an operational DB, deploy, or service activation was attempted.
- The test command uses a transient `httpx2` overlay because this environment's resolved Starlette TestClient requires it while the project declares `httpx`; no dependency source change was made for this batch.
- The initial independent QC reproduced the technical evidence but returned FAIL because the then-committed document described the remediation as uncommitted and its test run produced an untracked `uv.lock`. Follow-up commit `71a0312e1efb54772b94fd7de0d8add5943f05af` corrected provenance/hygiene only; independent re-QC inspected that committed diff and passed it.
## PR #21 provenance reconciliation
The delivery was implemented and initially verified on the isolated worktree chain. PR #21 was then assembled by cherry-picking that chain onto the confirmed local-main baseline, which created different commit IDs. The mappings are:
| Original worktree commit | PR #21 commit | Purpose |
|---|---|---|
| `f6195ff8090d8761d467e15f44255642ca350e80` | `1909caa22ce3f7d271dfb619b9095f998d0566cc` | BAT-025 implementation |
| `779d6f7711d700d6f56be22023cb826cc6f2fe28` | `d48279f7abdbabe1eb59a8d85703cf21f28d39d0` | PostgreSQL verification hardening |
| `71a0312e1efb54772b94fd7de0d8add5943f05af` | `2f82251e57cfefacec3dd39c49bfe40d605fe145` | provenance/hygiene correction |
| `da731b554a2cb7823100f32c85e8cab578e0ea36` | `5f8097947c1933f6b5eadeab94e5528680effc10` | lead confirmation state at PR creation |
- `git diff --exit-code da731b55..5f809794` confirmed that the original verified worktree state and PR #21 initial head have identical file content.
- Lead reproduction after the PR QC environment correction used the tracked local-test loopback defaults without exposing the DSN. The migration suite passed **3 passed in 6.16s**.
- Cleanup verification targets only the fixture contract `portfolio_hub_bat025_<32 lowercase hex UUID>`; its catalog count was **0**. The separately existing `portfolio_hub_bat025_test` database does not match that generated-name contract and was not modified.
## Gate result
- Developer self-verification: **complete** on the original worktree remediation chain, with the PR #21 commit mapping above.
- Initial independent QC: **FAIL** — documentation provenance and untracked `uv.lock` hygiene only; technical ACs reproduced.
- Original-worktree independent re-QC: **PASS** (`deleg_6b805aa0`, 2026-07-15), including loopback PostgreSQL and direct Ruff evidence.
- PR #21 committed-diff QC: **PASS** (`deleg_ae0c6f27`, 2026-07-15). It verified the cherry-pick patch-id mapping, PR head `bd51330`, migration semantics, loopback PostgreSQL suite (**3 passed**), fixture-generated UUID catalog cleanup (**0**), direct Ruff checks, clean tested worktree, and PR diff hygiene.
- `confirm-batch-verification`: **PASS accepted by TREX lead (2026-07-15)**. This approval is limited to merging the verified PR; it does not authorize release, deployment, operational database migration, or service restart.
- M5 Release: **not started**. Release preparation requires separate scope/approval after merge; production migration requires a separate operational decision and release approval.
+1
View File
@@ -32,3 +32,4 @@
| BAT-022 | standard | release-candidate | REQ-052, REQ-053, REQ-054 | dcy-011 | [bat-022](./bat-022_20260628_daemon-crud-self-service-credential-api/) |
| BAT-023 | batch-lite | in-progress | REQ-055 | trex-tvr-v1 dcy-001 | [bat-023](./bat-023_20260704_portfolios-strategy-market-meta/) |
| BAT-024 | standard | release-candidate | REQ-056 | dcy-012 | [bat-024](./bat-024_20260714_manual-ledger-api/) |
| BAT-025 | standard | release-candidate | REQ-057 | dcy-012 follow-on | [bat-025](./bat-025_20260715_portfolio-ledger-alembic-migration/) |
@@ -0,0 +1,130 @@
# 운영 적용 런북: REQ-057 — portfolio_ledger 수동 장부 메타데이터 migration
- Batch: BAT-025
- Discovery: dcy-012
- Date: 2026-07-15
- 결재 게이트: REQ-057 완료 승인 및 release 결재 필수
- 작업 예상 시간: 약 20~40분
---
## 전제 조건
- DB 백업 완료
- `PORTFOLIOHUB_DB_ENABLED=true` 및 DB 접속 환경변수 설정
- 대상 DB는 `alembic current`로 `20260628_0027` 상태(또는 그 이전 배포 라인)에서 진행
- 새 코드 배포는 `alembic upgrade 20260715_0028` 완료 + 검증 성공 및 `M5 Release` 결재 후에만 수행
> 주의: 운영 DB는 본 배치에서 실제 upgrade/runbook 실행 대상을 만들지 않는다. 아래는 실운영 적용 체크리스트이다.
---
## 0) 사전 점검
```bash
export TIMESCALEDB_HOST=<DB_HOST>
export TIMESCALEDB_PORT=<DB_PORT>
export TIMESCALEDB_NAME=<DB_NAME>
export TIMESCALEDB_USER=<DB_USER>
export TIMESCALEDB_PASSWORD=<DB_PASS>
export PORTFOLIOHUB_DB_ENABLED=true
alembic heads
alembic current
```
- `alembic heads`에는 구현 시점 head `20260715_0028`가 확인되어야 한다.
- `alembic current` 값이 `20260715_0028`이면 이미 마이그레이션 적용 완료 상태다.
## 1) 프리체크
### 1-1. 대기 중인 락/액티브 커넥션
```sql
SELECT pid, usename, application_name, state, wait_event_type, wait_event, query
FROM pg_stat_activity
WHERE query ILIKE '%portfolio_ledger%'
AND state != 'idle';
```
- 결과는 반드시 비어 있거나 승인된 윈도우에서만 허용.
### 1-2. 대상 스키마 및 데이터 건수
```sql
-- 현재 컬럼 상태
SELECT column_name, data_type, is_nullable, column_default
FROM information_schema.columns
WHERE table_name = 'portfolio_ledger'
AND column_name IN ('source', 'target_ledger_entry_id', 'confirmation_evidence', 'actor_user_id')
ORDER BY column_name;
-- 기존 행 수(증적)
SELECT COUNT(*) AS rows FROM portfolio_ledger;
```
- `source`가 이미 없거나 NULL 허용 상태여도 업그레이드 실행 전/후로 수렴 가능.
---
## 2) 마이그레이션 실행
```bash
alembic upgrade 20260715_0028
```
### 수신 메시지 예시
- 정상: `Running upgrade ...` 형태로 revision 로그 출력
- 실패: 즉시 종료되며 partial 변경 여부는 로그로 확인
---
## 3) 사후 검증
```sql
SELECT column_name, data_type, is_nullable, column_default
FROM information_schema.columns
WHERE table_name = 'portfolio_ledger'
AND column_name IN ('source', 'target_ledger_entry_id', 'confirmation_evidence', 'actor_user_id')
ORDER BY column_name;
-- source backfill 결과(0이어야 함)
SELECT COUNT(*) AS null_source_rows
FROM portfolio_ledger
WHERE source IS NULL;
SELECT COUNT(*) AS rows_after FROM portfolio_ledger;
SELECT column_name, is_nullable
FROM information_schema.columns
WHERE table_name = 'portfolio_ledger' AND column_name = 'source';
SELECT column_default
FROM information_schema.columns
WHERE table_name = 'portfolio_ledger'
AND column_name = 'source';
```
- 기대치: `source`가 `NOT NULL`, `DEFAULT 'broker'`, `null_source_rows = 0`
- `rows_after` 값은 프리체크 시점과 일치해야 한다.
- `alembic current`가 `20260715_0028`로 일치해야 한다.
```bash
alembic current
```
---
## 4) 실패 대응 (Forward-only)
- 업그레이드 실패 시 바로 이전 코드로 rollback은 허용하되, 본 migration에서 추가된 4개 컬럼은 운영에서 자동 삭제하지 않는다.
- 필요한 경우 운영 담당과 협의하여 수동 복구 계획을 수립한다.
- 필수 검토 항목: 대상 DB 백업 상태, 실행 로그, 에러 스택, 락/동시 접속 상태
---
## 5) 운영 활성화 조건
- 위 사후 검증(`alembic current`/schema/row count/source null 없음) 완료
- API smoke(REQ-056 manual ledger) 통과
- release 결재 완료
- 위 조건 충족 시 신규 코드 재시작 및 API 활성화 수행
@@ -182,6 +182,39 @@
#### Validation Plan
- AC-1~19를 batch verification에서 API/repository/concurrency/regression evidence로 검증한다.
### CHG-20260715-03
- Date: 2026-07-15
- Author: TREX lead (Hermes)
#### Change Summary
- 운영 적용 방식 보완을 REQ-057로 분리 추적한다. REQ-056의 기존 API 계약, AC-1~19, `Implemented` status 및 BAT-024 `release-candidate` 상태는 변경하거나 되돌리지 않는다.
#### Intent
- REQ-056 구현에서 사용된 additive `portfolio_ledger` 컬럼의 운영 DB 적용을 runtime `ensure_schema()` 자동 DDL에만 의존하지 않도록 명시 Alembic migration과 운영 runbook 요구사항으로 후속 관리한다.
#### Acceptance Criteria Delta
- Added: 없음 (REQ-056 AC 변경 없음)
- Updated: 없음
- Removed: 없음
- Follow-on: REQ-057 AC-1~11이 migration, schema/Alembic revision 정합, 운영 preflight/runbook, forward-only recovery, release 결재 게이트를 별도로 정의한다.
#### Impacted Area Delta
- REQ-056 구현 영향 범위는 변경 없음.
- 후속 REQ-057 영향: `migrations/versions/`, runtime `ensure_schema()` 호환성, migration/runbook 및 persistence baseline 문서. 실제 변경은 BAT-025 standard에서 수행한다.
#### Delivery Trace Delta
- Batches: BAT-024는 REQ-056 구현/검증 release-candidate 상태로 유지. BAT-025 standard가 REQ-057 migration implementation/test/QC/runbook을 담당한다.
- Releases: 운영 DB upgrade, 서비스 재시작 및 신규 manual ledger API 활성화는 별도 명시 release 결재 전 실행하지 않는다.
#### Revalidation Impact
- Reverification Needed: no — REQ-056의 기존 implementation/QC evidence는 무효화하지 않는다.
- Existing Implementation Invalidated: no
- Status Recommendation: keep-Implemented
- Follow-up Action: REQ-057 Proposed 검토·승인 후 BAT-025에서 migration compatibility/data preservation/runbook evidence를 확보한다.
#### Validation Plan
- BAT-025에서 fresh DB와 runtime-precreated-schema 환경의 Alembic upgrade, 기존 row/source 보존, `alembic current` 정합, broker/manual ledger regression, runbook preflight 및 service smoke를 검증한다. 운영 DB 실행은 별도 release 결재 범위다.
## Implementation Confirmation
@@ -0,0 +1,124 @@
# REQ-057: portfolio_ledger 수동 장부 메타데이터 명시 Alembic 마이그레이션
- Status: Approved
- Type: Migration
- Priority: High
- Owner: Justin Ko
## Intent
- REQ-056 수동 장부 API가 사용하는 `portfolio_ledger`의 additive 메타데이터 컬럼을 runtime `ensure_schema()` 자동 DDL에만 의존하지 않고, 추적 가능한 명시적 Alembic revision으로 운영 적용 가능하게 한다.
- 운영 DB 스키마와 Alembic revision 상태를 일치시키고, migration 검증 전에는 해당 신규 API 코드를 활성화하지 않는 배포 선행조건을 정의한다.
## Requirement
- `portfolio_ledger`에 아래 네 컬럼을 적용하는 신규 Alembic revision을 작성한다.
| 컬럼 | SQL 타입 | NULL 정책 | 기본값/의미 |
| --- | --- | --- | --- |
| `source` | `TEXT` | `NOT NULL` | `DEFAULT 'broker'`; broker 기존 행과 broker write의 출처 |
| `target_ledger_entry_id` | `BIGINT` | `NULL` 허용 | `REVERSAL` target ledger entry 참조 |
| `confirmation_evidence` | `TEXT` | `NULL` 허용 | daemon-managed manual write의 명시적 확인 근거 |
| `actor_user_id` | `TEXT` | `NULL` 허용 | manual write를 수행한 사용자 추적 값 |
- revision의 `down_revision`은 구현 시점에 repository의 실제 Alembic head를 `alembic heads` 및 migration graph로 사전 확인하여 결정한다. 요구사항 문서에서 특정 revision ID를 하드코드 전제로 단정하지 않는다.
- upgrade는 기존 `portfolio_ledger` 행을 삭제·재작성하지 않고 보존해야 하며, 기존 행과 `source`가 비어 있는 선행 상태의 행은 `source='broker'`를 만족하도록 처리한 뒤 `source NOT NULL DEFAULT 'broker'`를 보장해야 한다.
- runtime `ensure_schema()` 자동 DDL로 위 컬럼이 이미 선행 생성된 DB에서도 migration이 안전하게 완료되어야 한다. 구현은 컬럼/제약 존재 여부, 기존 행의 `source` 값, Alembic version table을 확인해 중복 DDL 오류 없이 revision을 적용 또는 필요한 상태 정합 절차를 수행해야 한다.
- migration 완료 후 `alembic current`가 구현 시 확정한 revision/head와 일치해야 한다. 스키마만 존재하고 Alembic revision이 불일치한 상태는 완료로 간주하지 않는다.
- 이 migration의 검증 완료는 운영 배포에서 REQ-056 신규 manual ledger API 코드를 활성화하기 위한 선행조건이다. 서비스 재시작·신규 코드 활성화는 migration upgrade와 schema/current 검증이 성공한 뒤에만 허용한다.
- 운영 runbook을 작성한다. runbook은 최소한 백업, `alembic current`, 대상 테이블 schema/row count, 활성 쿼리·lock 확인, upgrade, schema/current 재검증, service smoke, 실패 대응을 포함해야 한다.
- rollback은 code rollback + additive schema 유지의 forward-only 정책으로 정의한다. 운영 환경에서 down migration 또는 컬럼 삭제를 자동 rollback 경로로 사용하지 않는다.
- 운영 DB 실행은 이 REQ/BAT 범위에 포함하지 않는다. 실제 upgrade·서비스 활성화는 증적 검토 후 별도 명시 release 결재가 있어야 한다.
- migration implementation, test, 독립 QC 및 batch verification은 표준 배치 `BAT-025`에서 수행한다. 이 REQ 초안에서는 BAT-025 batch artifact 또는 migration/code 파일을 생성하지 않는다.
## Acceptance Criteria
1. Given implementation preflight, When Alembic migration을 생성하기 전 `alembic heads`와 revision graph를 확인하면, Then 실제 단일/유효 head를 근거로 신규 revision의 `down_revision`을 결정하며 REQ 본문에 특정 기존 revision ID를 선결 조건으로 하드코드하지 않는다.
2. Given migration이 적용되지 않은 DB의 `portfolio_ledger`, When 신규 revision을 upgrade하면, Then `source TEXT NOT NULL DEFAULT 'broker'`, `target_ledger_entry_id BIGINT NULL`, `confirmation_evidence TEXT NULL`, `actor_user_id TEXT NULL` 네 컬럼이 모두 존재한다.
3. Given upgrade 전 기존 `portfolio_ledger` 행이 존재하면, When 신규 revision을 upgrade하면, Then 기존 행 수와 기존 business data가 보존되고 모든 기존 행의 `source`는 `'broker'`이다.
4. Given 기존 행 또는 runtime 선행 DDL 상태에서 `source`가 NULL 또는 미설정이면, When migration이 완료되면, Then NULL `source`가 남지 않고 `source`는 `'broker'`로 backfill되며 이후 컬럼은 `NOT NULL DEFAULT 'broker'` 제약을 만족한다.
5. Given runtime `ensure_schema()`가 네 컬럼을 이미 생성한 DB, When Alembic migration을 적용하면, Then 중복 컬럼/제약 오류 없이 안전하게 완료되고 schema와 Alembic version 상태가 revision/head와 일치한다.
6. Given migration upgrade가 완료되면, When `alembic current`와 information schema/row count 검증을 실행하면, Then current revision은 구현 시 확정한 head이고 네 컬럼의 타입·nullable/default 및 row count 보존 결과가 기대값과 일치한다.
7. Given migration 전 preflight, When 운영 runbook을 수행하면, Then DB backup 성공, `alembic current` 기록, `portfolio_ledger` schema/row count 기록, 대상 테이블 활성 쿼리·lock 부재 또는 승인된 maintenance window가 모두 확인되지 않으면 upgrade 전에 중단한다.
8. Given upgrade와 사후 schema/current 검증이 성공하면, When 서비스 smoke를 실행하면, Then DB-enabled health와 REQ-056 manual ledger API의 승인된 non-production/운영 smoke 시나리오가 성공한다. migration 검증 실패 시 신규 API 코드 활성화 또는 서비스 재시작을 진행하지 않는다.
9. Given migration 또는 service smoke 이후 코드 문제가 발생하면, When rollback 판단을 내리면, Then 직전 호환 코드로만 rollback하고 적용된 네 additive 컬럼과 Alembic revision은 유지하는 forward-only 복구 정책을 따른다. 운영 down migration/컬럼 삭제는 자동 조치가 아니다.
10. Given 운영 DB 적용을 요청할 때, When 별도 release 결재가 없으면, Then 실제 DB upgrade, 서비스 재시작, 신규 API 활성화를 실행하지 않는다.
11. Given `BAT-025` delivery, When implementation/test/QC를 완료하면, Then Alembic upgrade 및 runtime-precreated-schema 호환성, data preservation, runbook dry-run/검증, 기존 broker ledger 흐름 회귀를 evidence로 기록하고 독립 QC를 거친다.
## Impacted Area
- Docs:
- `docs/runbooks/` — BAT-025 운영 migration runbook 신규 작성 예정.
- `docs/data-model.md` — `portfolio_ledger` 컬럼과 revision 기록 갱신 예정.
- `docs/releases/` — 운영 적용 시 별도 release 결재와 실행 증적 기록 예정.
- Modules:
- `migrations/versions/` — 네 컬럼을 위한 신규 Alembic revision 예정.
- `app/repositories/postgres.py` 또는 `ensure_schema()` 소유 모듈 — runtime 자동 DDL과 Alembic migration의 idempotent 호환성 검증/보강 예정.
- migration bootstrap/CLI 경로 — `alembic current` 및 upgrade 사전·사후 검증 경로 확인 예정.
- Tests:
- fresh/upgraded DB schema, 기존 행 보존과 `source='broker'` backfill 검증.
- runtime-precreated-schema → Alembic upgrade → current/head 정합 integration 검증.
- 기존 broker ledger write 및 REQ-056 manual ledger smoke/regression 검증.
- runbook preflight/forward-only recovery 절차의 문서 검증.
## Notes
- Source: REQ-056 Implementation Confirmation 및 `docs/srs/index.md` 2026-07-14 release caveat. REQ-056은 이미 Implemented/BAT-024 release-candidate이므로 본 REQ는 REQ-056 AC 또는 status를 되돌리지 않고 운영 적용 방식만 별도 추적한다.
- Repository 기록에는 과거 Alembic head로 보이는 revision이 있으나, 새 migration의 parent는 구현 착수 시점의 실제 head로 재검증해야 한다.
- 기존 runtime `ensure_schema()` 자동 DDL은 선행 스키마 생성 가능성을 만들 수 있으므로, 그 DB에서도 Alembic revision 추적을 회복·일치시키는 것이 핵심이다. 임의 `stamp` 여부와 조건은 BAT-025 design/implementation에서 실제 schema와 migration graph 증거로 결정한다.
- 운영 DB 실행, 배포, 서비스 재시작, commit/push는 이번 요구사항 초안 범위에 포함하지 않는다.
## Change Log
### CHG-20260715-01
- Date: 2026-07-15
- Author: portfolio-hub-dev (Hermes)
#### Change Summary
- 신규 생성 (Proposed) — REQ-056 운영 release caveat에 대한 change requirement. `portfolio_ledger` manual ledger 메타데이터 네 컬럼을 명시 Alembic migration과 운영 runbook으로 관리하는 범위를 정의한다.
#### Intent
- runtime 자동 DDL만으로 생성될 수 있는 additive 컬럼을 Alembic revision 및 운영 증적으로 관리해, 스키마와 version history의 불일치를 방지한다.
#### Acceptance Criteria Delta
- Added:
- AC-1~11 최초 작성
- Updated: 없음
- Removed: 없음
#### Impacted Area Delta
- Docs: BAT-025 runbook, data model/persistence baseline, release evidence 갱신 예정.
- Modules: 신규 Alembic revision, runtime `ensure_schema()` 호환 경로, migration bootstrap/CLI 검증 경로.
- Tests: fresh DB, runtime-precreated schema, data preservation, broker/manual ledger regression, runbook preflight 검증.
#### Delivery Trace Delta
- Batches: BAT-025 standard 예정 — migration implementation/test/QC/runbook 작성.
- Releases: 운영 DB upgrade와 신규 API 활성화는 별도 release 결재 필요.
#### Revalidation Impact
- Reverification Needed: yes — migration 구현 후 AC-1~11 evidence 및 운영 실행 시 별도 실운영 evidence 필요.
- Existing Implementation Invalidated: no — REQ-056 Implemented status와 BAT-024 release-candidate 상태는 유지한다.
- Status Recommendation: keep-proposed
- Follow-up Action: confirm-req → BAT-025 standard batch drafting/planning/design → implementation/QC. 운영 실행은 별도 release approval 후에만 검토한다.
#### Validation Plan
- BAT-025에서 Alembic graph/head preflight, fresh/upgraded DB 및 runtime-precreated-schema integration 검증, data preservation/source backfill, `alembic current` 정합, broker/manual ledger smoke/regression을 실행한다. 운영 runbook은 backup·lock·schema/current·smoke·forward-only recovery 체크리스트로 검토하며, 실 운영 DB 실행은 별도 결재 전 수행하지 않는다.
### CHG-20260715-02
- Date: 2026-07-15
- Author: TREX lead (Hermes)
- confirm-req: `Proposed` → `Approved`.
#### Change Summary
- REQ-056의 이미 검증된 API 계약을 다시 열지 않고, `portfolio_ledger` 운영 스키마 적용을 독립 Migration REQ로 승인한다.
#### Approval Basis
- dcy-012의 `source` 분리·REVERSAL target·daemon-managed confirmation audit 결정과 REQ-056 Implementation Confirmation의 운영 release caveat를 추적했다.
- 실제 `uv run alembic heads` 결과는 단일 head `20260628_0027`이었다. 단, 병렬 revision 가능성을 포함한 구현 시점 graph 재확인을 AC-1의 필수 preflight로 유지한다.
- AC-1~11은 fresh/기존 데이터/runtime-precreated schema, schema-version 정합, rollback 경계, 운영 runbook을 검증 가능하게 정의한다.
#### Revalidation Impact
- Reverification Needed: yes — BAT-025에서 migration 및 runbook evidence를 확보해야 한다.
- Existing Implementation Invalidated: no — REQ-056 `Implemented` 및 BAT-024 `release-candidate`는 유지한다.
- Status Recommendation: proceed-to-batch.
- Follow-up Action: suggest-batch-reqs → BAT-025 standard drafting/planning/design/implementation/verification → independent QC. 운영 DB 실행은 별도 release 결재 후에만 가능하다.
+3 -1
View File
@@ -74,7 +74,7 @@
## Next Requirement ID
- Current: `REQ-057`
- Current: `REQ-058`
- Rule: `Requirement Register`에 있는 가장 큰 REQ 번호보다 정확히 1 커야 한다.
## Requirement Register
@@ -137,9 +137,11 @@
| REQ-054 | 인증정보 관리 보안 격리 — write-only·secret 미반환·로그 미노출 | Non-Functional | Implemented | High | Justin Ko | [req-054](./Non-Functional/req-054_credential-management-security-write-only.md) |
| REQ-055 | GET /v1/portfolios 응답에 전략·시장 메타데이터 노출 (additive) | Interface | Implemented | High | Justin Ko | [req-055](./Interface/req-055_v1-portfolios-strategy-market-meta-exposure.md) |
| REQ-056 | owner-scoped 수동 장부 입력 HTTP API | Interface | Implemented | High | Justin Ko | [req-056](./Interface/req-056_v1-portfolios-manual-ledger-api.md) |
| REQ-057 | portfolio_ledger 수동 장부 메타데이터 명시 Alembic 마이그레이션 | Migration | Approved | High | Justin Ko | [req-057](./Migration/req-057_portfolio-ledger-manual-metadata-alembic-migration.md) |
## Recent Change Log Summary
- 2026-07-15: REQ-057 `Proposed` → `Approved` (confirm-req) — REQ-056/BAT-024의 검증 완료 API 계약은 유지하고, `portfolio_ledger` 네 컬럼의 명시 Alembic migration을 독립 추적한다. 실제 `alembic heads` 단일 head `20260628_0027` 확인, 구현 시점 graph 재확인(AC-1) 유지. BAT-025 standard에서 fresh/기존/runtime-precreated schema 검증, data preservation, runbook, 독립 QC를 수행한다. 운영 DB upgrade·서비스 활성화는 별도 release 결재 전 실행하지 않는다.
- 2026-07-14: REQ-056 `Approved` → `Implemented` (confirm-req-implemented) — BAT-024 local implementation commit `b3447f6`, independent QC PASS (`deleg_f2337a39`). Manual ledger API endpoints added (`POST/GET /v1/portfolios/{portfolio_id}/ledger/manual`, `GET /v1/portfolios/{portfolio_id}/returns/summary`), strict owner-scope without admin bypass, service-token fail-closed, insufficient state 409, daemon-managed confirmation, `total_value_after`, `cumulative_profit_withdrawn`. Verification: full `594 passed`, focused remediation `4 passed`. Release caveat: production DB rollout should address additive `portfolio_ledger` columns with migration/runbook at release gate.
- 2026-07-14: REQ-056 `Proposed` → `Approved` (confirm-req) — dcy-012 confirmed Discovery 근거. 신규 `POST /v1/portfolios/{portfolio_id}/ledger/manual` + action enum(`INVEST`, `BUFFER_OUT`, `BUFFER_IN`, `BUY`, `SELL`, `WITHDRAW`, `PROFIT`, `REVERSAL`) owner-scoped user/session API 계약. 기존 `/ledger/trades`는 daemon/service-token broker write 전용으로 보존. 부족 cash/quantity/buffer 409, daemon-managed confirmation 409 `MANUAL_WRITE_REQUIRES_CONFIRMATION`, after-snapshot/history fields, `cumulative_profit_withdrawn` 및 `PROFIT` 순입금액 미포함 의미 포함. BAT-024 delivery 후보로 진행.
- 2026-07-04: REQ-055 신규 생성 `Implemented` (draft-req) — trex-tvr-v1 dcy-001 confirmed Discovery 근거. `GET /v1/portfolios` 응답에 `active_strategy_id`·`strategy_name`·`strategy_params`·`markets` additive 노출. DB 마이그레이션 불필요(기존 컬럼·테이블 활용), N+1 금지·파싱 에러 격리(500 금지) 포함. BAT-024 delivery 후보로 진행.
+3 -2
View File
@@ -10,13 +10,14 @@ from app.core.config import get_settings
config = context.config
if config.config_file_name is not None:
fileConfig(config.config_file_name)
# Alembic may run in-process (bootstrap/tests); preserve application loggers.
fileConfig(config.config_file_name, disable_existing_loggers=False)
target_metadata = None
def _configure_database_url() -> str:
settings = get_settings()
settings = config.attributes.get("portfoliohub_settings") or get_settings()
database_url = settings.active_database_url
config.set_main_option("sqlalchemy.url", database_url.replace("%", "%%"))
return database_url
@@ -18,7 +18,9 @@ depends_on: str | Sequence[str] | None = None
def upgrade() -> None:
op.add_column("portfolio_ledger", sa.Column("ledger_entry_id", sa.BigInteger(), sa.Identity(always=False), nullable=True))
# PostgreSQL identity columns are implicitly NOT NULL. ``nullable=True``
# renders a conflicting trailing ``NULL`` clause on a real PostgreSQL DB.
op.add_column("portfolio_ledger", sa.Column("ledger_entry_id", sa.BigInteger(), sa.Identity(always=False), nullable=False))
op.add_column("portfolio_ledger", sa.Column("order_id", sa.Text(), nullable=True))
op.add_column("portfolio_ledger", sa.Column("execution_id", sa.Text(), nullable=True))
op.add_column("portfolio_ledger", sa.Column("side", sa.Text(), nullable=True))
@@ -0,0 +1,96 @@
"""portfolio_ledger: manual ledger metadata columns backfill-safe upgrade
Revision ID: 20260715_0028
Revises: 20260628_0027
Create Date: 2026-07-15 00:00:00
This migration adds the portfolio_ledger additive metadata columns introduced by
REQ-056. It is safe for both fresh DBs and legacy DBs where ensure_schema() already
created columns at runtime.
- source: TEXT NOT NULL DEFAULT 'broker'
- target_ledger_entry_id: BIGINT NULL
- confirmation_evidence: TEXT NULL
- actor_user_id: TEXT NULL
Runtime behavior:
- if source exists, backfill NULL -> 'broker', then enforce NOT NULL/default.
- if source/other columns are absent, add them as idempotent ALTERs.
- existing rows are preserved; only metadata defaults/backfill are adjusted.
- downgrade is forward-only (columns are not removed).
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
from sqlalchemy import inspect, text
revision = "20260715_0028"
down_revision = "20260628_0027"
branch_labels = None
depends_on = None
def _table_exists(inspector: sa.engine.Inspector, table_name: str) -> bool:
return table_name in inspector.get_table_names()
def _column_exists(inspector: sa.engine.Inspector, table_name: str, column_name: str) -> bool:
if not _table_exists(inspector, table_name):
return False
return any(c["name"] == column_name for c in inspector.get_columns(table_name))
def upgrade() -> None:
bind = op.get_bind()
inspector = inspect(bind)
if not _table_exists(inspector, "portfolio_ledger"):
return
if not _column_exists(inspector, "portfolio_ledger", "source"):
op.add_column(
"portfolio_ledger",
sa.Column(
"source",
sa.Text(),
nullable=False,
server_default=sa.text("'broker'"),
),
)
else:
# runtime-precreated 환경: source가 NULL이면 명시적으로 backfill 후 NOT NULL + 기본값 보장
bind.execute(text("UPDATE portfolio_ledger SET source = 'broker' WHERE source IS NULL"))
op.alter_column(
"portfolio_ledger",
"source",
existing_type=sa.Text(),
server_default=sa.text("'broker'"),
nullable=False,
)
if not _column_exists(inspector, "portfolio_ledger", "target_ledger_entry_id"):
op.add_column(
"portfolio_ledger",
sa.Column("target_ledger_entry_id", sa.BigInteger(), nullable=True),
)
if not _column_exists(inspector, "portfolio_ledger", "confirmation_evidence"):
op.add_column(
"portfolio_ledger",
sa.Column("confirmation_evidence", sa.Text(), nullable=True),
)
if not _column_exists(inspector, "portfolio_ledger", "actor_user_id"):
op.add_column(
"portfolio_ledger",
sa.Column("actor_user_id", sa.Text(), nullable=True),
)
# Forward-only policy: 운영 rollback은 컬럼 삭제/재적용 경로가 아니다.
def downgrade() -> None:
"""Forward-only migration: 컬럼 삭제 downgrade는 수행하지 않는다."""
return
@@ -0,0 +1,273 @@
"""Real PostgreSQL integration tests for BAT-025 / REQ-057.
These tests deliberately require an explicit local maintenance DSN and create a
unique throwaway database per test. They reject non-loopback hosts and any
non-``postgres`` maintenance database so test execution cannot target a shared
or production application database.
"""
from __future__ import annotations
import os
from datetime import datetime, timezone
from urllib.parse import unquote, urlparse
from uuid import uuid4
import psycopg
import pytest
from psycopg import sql
from app.core.config import Settings
from app.db.migrations import (
bootstrap_or_upgrade_to_head,
get_alembic_config,
upgrade_to_head,
)
from app.repositories.postgres import PortfolioDBConfig, PostgresPortfolioRepository
_LOCAL_DSN_ENV = "BAT025_LOCAL_POSTGRES_DSN"
_LOOPBACK_HOSTS = {"127.0.0.1", "localhost", "::1"}
_HEAD = "20260715_0028"
_PRE_BAT025 = "20260628_0027"
def _maintenance_dsn() -> str:
dsn = os.environ.get(_LOCAL_DSN_ENV)
if not dsn:
pytest.skip(f"{_LOCAL_DSN_ENV} is required for real PostgreSQL integration tests")
parsed = urlparse(dsn)
if parsed.scheme not in {"postgres", "postgresql"}:
pytest.fail("BAT-025 test DSN must use postgres/postgresql")
if parsed.hostname not in _LOOPBACK_HOSTS:
pytest.fail("BAT-025 tests refuse a non-loopback PostgreSQL host")
if parsed.path.lstrip("/") != "postgres":
pytest.fail("BAT-025 test DSN must connect to local maintenance database 'postgres'")
if not parsed.username:
pytest.fail("BAT-025 test DSN must include a local database user")
return dsn
def _settings_for_database(maintenance_dsn: str, database: str) -> Settings:
parsed = urlparse(maintenance_dsn)
return Settings(
db_enabled=True,
timescaledb_host=parsed.hostname,
timescaledb_port=parsed.port or 5432,
timescaledb_name=database,
timescaledb_user=unquote(parsed.username or ""),
timescaledb_password=unquote(parsed.password or ""),
timescaledb_sslmode="disable",
timescaledb_connect_timeout_sec=3,
timescaledb_application_name="bat025-real-postgres-test",
)
@pytest.fixture()
def isolated_settings() -> Settings:
maintenance_dsn = _maintenance_dsn()
database = f"portfolio_hub_bat025_{uuid4().hex}"
with psycopg.connect(maintenance_dsn, autocommit=True) as conn:
with conn.cursor() as cur:
cur.execute(sql.SQL("CREATE DATABASE {}").format(sql.Identifier(database)))
try:
yield _settings_for_database(maintenance_dsn, database)
finally:
with psycopg.connect(maintenance_dsn, autocommit=True) as conn:
with conn.cursor() as cur:
cur.execute(
"SELECT pg_terminate_backend(pid) FROM pg_stat_activity "
"WHERE datname = %s AND pid <> pg_backend_pid()",
(database,),
)
cur.execute(sql.SQL("DROP DATABASE IF EXISTS {}").format(sql.Identifier(database)))
def _repo(settings: Settings) -> PostgresPortfolioRepository:
return PostgresPortfolioRepository(
PortfolioDBConfig(
host=settings.active_db_host,
port=settings.active_db_port,
dbname=settings.active_db_name,
user=settings.active_db_user,
password=settings.active_db_password,
sslmode=settings.active_db_sslmode,
connect_timeout_sec=settings.active_db_connect_timeout_sec,
application_name=settings.active_db_application_name,
)
)
def _current_revision(settings: Settings) -> str:
with psycopg.connect(
host=settings.active_db_host,
port=settings.active_db_port,
dbname=settings.active_db_name,
user=settings.active_db_user,
password=settings.active_db_password,
sslmode=settings.active_db_sslmode,
) as conn:
with conn.cursor() as cur:
cur.execute("SELECT version_num FROM alembic_version")
return str(cur.fetchone()[0])
def _metadata_columns(settings: Settings) -> dict[str, tuple[str, str, str | None]]:
with psycopg.connect(
host=settings.active_db_host,
port=settings.active_db_port,
dbname=settings.active_db_name,
user=settings.active_db_user,
password=settings.active_db_password,
sslmode=settings.active_db_sslmode,
) as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT column_name, data_type, is_nullable, column_default
FROM information_schema.columns
WHERE table_schema = 'public' AND table_name = 'portfolio_ledger'
AND column_name IN ('source', 'target_ledger_entry_id',
'confirmation_evidence', 'actor_user_id')
"""
)
return {row[0]: (row[1], row[2], row[3]) for row in cur.fetchall()}
def _seed_portfolio(repo: PostgresPortfolioRepository, suffix: str) -> str:
portfolio_id = f"bat025-{suffix}"
user_id = f"bat025-user-{suffix}"
symbol = f"BAT025-{suffix}"
with repo._connect().cursor() as cur:
cur.execute(
"INSERT INTO users (id, username) VALUES (%s, %s)",
(user_id, user_id),
)
cur.execute(
"INSERT INTO assets (symbol, markets) VALUES (%s, ARRAY[]::text[]) RETURNING asset_id",
(symbol,),
)
asset_id = cur.fetchone()[0]
cur.execute(
"""
INSERT INTO portfolios (
portfolio_id, name, owner_user_id, cash_balance, asset_quantity, asset_price,
profit_buffer, asset_id, broker, status
) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
""",
(portfolio_id, portfolio_id, user_id, 1_000_000, 0, 0, 0, asset_id, "bat025", "enabled"),
)
return portfolio_id
def test_upgrade_from_pre_0028_preserves_business_data_and_backfills_source(
isolated_settings: Settings,
) -> None:
"""A real 0027 -> head upgrade preserves a representative ledger row."""
from alembic import command
command.upgrade(get_alembic_config(isolated_settings), _PRE_BAT025)
repo = _repo(isolated_settings)
portfolio_id = _seed_portfolio(repo, "preserve")
recorded_at = datetime(2026, 7, 15, tzinfo=timezone.utc)
with repo._connect().cursor() as cur:
cur.execute(
"""
INSERT INTO portfolio_ledger (
portfolio_id, recorded_at, side, price, quantity, fee_amount, tax_amount,
cash_balance_after, asset_quantity_after, total_value_after,
profit_buffer_after, strategy_set_id, note, executed_at
) VALUES (%s, %s, 'buy', 123.45, 7, 1, 0, 998_000, 7, 998_864.15, 0,
'business-strategy', 'preserve-me', %s)
""",
(portfolio_id, recorded_at, recorded_at),
)
cur.execute("SELECT COUNT(*) FROM portfolio_ledger")
before_count = cur.fetchone()[0]
upgrade_to_head(isolated_settings)
assert _current_revision(isolated_settings) == _HEAD
assert _metadata_columns(isolated_settings) == {
"source": ("text", "NO", "'broker'::text"),
"target_ledger_entry_id": ("bigint", "YES", None),
"confirmation_evidence": ("text", "YES", None),
"actor_user_id": ("text", "YES", None),
}
with repo._connect().cursor() as cur:
cur.execute(
"SELECT COUNT(*), side, price::text, quantity::text, note, source "
"FROM portfolio_ledger GROUP BY side, price, quantity, note, source"
)
count, side, price, quantity, note, source = cur.fetchone()
assert count == before_count
assert (side, price, quantity, note, source) == ("buy", "123.45000000", "7.00000000", "preserve-me", "broker")
repo.close()
def test_runtime_precreated_schema_with_null_source_converges_without_duplicate_ddl(
isolated_settings: Settings,
) -> None:
"""ensure_schema-first DB stamps 0027 then truly executes 0028 to head."""
repo = _repo(isolated_settings)
repo.ensure_schema()
portfolio_id = _seed_portfolio(repo, "runtime")
with repo._connect().cursor() as cur:
cur.execute("ALTER TABLE portfolio_ledger ALTER COLUMN source DROP NOT NULL")
cur.execute(
"""
INSERT INTO portfolio_ledger (
portfolio_id, side, price, quantity, fee_amount, tax_amount,
cash_balance_after, asset_quantity_after, total_value_after,
profit_buffer_after, strategy_set_id, note, source
) VALUES (%s, 'buy', 10, 1, 0, 0, 990, 1, 1000, 0, 'runtime', 'null-source', NULL)
""",
(portfolio_id,),
)
assert bootstrap_or_upgrade_to_head(isolated_settings) == "head"
assert _current_revision(isolated_settings) == _HEAD
assert _metadata_columns(isolated_settings)["source"] == ("text", "NO", "'broker'::text")
with repo._connect().cursor() as cur:
cur.execute("SELECT source FROM portfolio_ledger WHERE portfolio_id = %s", (portfolio_id,))
assert cur.fetchone()[0] == "broker"
repo.close()
def test_broker_and_manual_ledger_writes_regress_after_real_upgrade(
isolated_settings: Settings,
) -> None:
"""Synthetic broker and manual writes use the migrated columns on real PostgreSQL."""
upgrade_to_head(isolated_settings)
repo = _repo(isolated_settings)
portfolio_id = _seed_portfolio(repo, "writes")
broker, created = repo.insert_ledger_trade(
portfolio_id=portfolio_id,
order_id="bat025-order",
execution_id="bat025-execution",
side="buy",
quantity=2,
price=100,
executed_at=datetime.now(timezone.utc),
)
manual = repo.record_manual_ledger(
portfolio_id,
action="INVEST",
amount=50,
actor_user_id="bat025-user",
confirmation_evidence="synthetic-local-fixture",
)
assert created is True
assert broker.source == "broker"
assert manual.source == "manual"
assert manual.actor_user_id == "bat025-user"
assert manual.confirmation_evidence == "synthetic-local-fixture"
with repo._connect().cursor() as cur:
cur.execute(
"SELECT source, actor_user_id, confirmation_evidence FROM portfolio_ledger "
"WHERE ledger_entry_id = %s",
(manual.ledger_entry_id,),
)
assert cur.fetchone() == ("manual", "bat025-user", "synthetic-local-fixture")
repo.close()