portfolio-hub/docs/runbooks/rel-004-m5-manual-ledger-deployment-runbook.md

# REL-004 운영 배포 런북 — M5 수동 장부 API 및 ledger metadata migration

- Service: `portfolio-hub`
- Release: [REL-004](../releases/rel-004_20260715_manual-ledger-api-and-metadata-migration.md)
- Target source baseline: `origin/main@914b3dc6e6b9e2cbd7068fb30435c83f0581eab5`
- Database revision: `20260715_0028`
- Classification: production DB migration + app-service deployment
- Required approver: Justin Ko
- Required operator: trex-op
- Estimated active window: 20–40 minutes plus observation

> **Do not begin from this document alone.** Production execution needs an explicit REL-004 approval, a maintenance window, and the exact deployment-platform command approved by trex-op. This repository contains no production Docker Compose, systemd, Kubernetes, or Helm manifest; do not infer one from `docker-compose.local-test.yml`, which is local-test infrastructure only.

## 1. Safety model

- The migration is additive. It adds/normalizes four `portfolio_ledger` metadata columns and records Alembic revision `20260715_0028`.
- The permitted recovery path after a successful migration is **compatible code rollback while retaining the schema**.
- `alembic downgrade`, column deletion, and automatic `pg_restore` are prohibited in this runbook. Any restore is a separately approved incident recovery action.
- The new API can mutate ledger state. A production write smoke requires an explicitly designated test portfolio, a named owner, and approval; otherwise stop after read-only health and schema checks.
- Do not paste DB passwords, bearer tokens, credential payloads, or backups into this document, chat, CI logs, or release evidence. Record secret-free command outcomes only.

## 2. Roles and stop authority

| Role | Responsibility |
| --- | --- |
| Justin Ko | grants release-window approval; decides whether production mutation smoke is allowed; approves exceptional restore |
| trex-op | captures evidence, performs backup/preflight/migration/deployment using the real platform tooling, stops on any failed gate |
| TREX lead | reviews completed evidence and records final release decision; does not perform production commands under this draft |

Any operator may stop the rollout for a failed gate, unexpected active workload, missing backup, wrong baseline, migration error, or unexpected API behavior.

## 3. Inputs to obtain before window

- [ ] approved window and approver/time recorded in REL-004
- [ ] immutable deployable artifact built from `914b3dc` or reviewed descendant, with image/package digest
- [ ] exact production deployment command supplied by trex-op’s platform (for example, its existing rollout command); do not substitute a local-test Compose command
- [ ] secure database environment available to the migration runner:
  - `TIMESCALEDB_HOST`, `TIMESCALEDB_PORT`, `TIMESCALEDB_NAME`, `TIMESCALEDB_USER`, `TIMESCALEDB_PASSWORD`
  - `PORTFOLIOHUB_DB_ENABLED=true`
- [ ] confirmed DB backup destination and restore owner
- [ ] approved health base URL
- [ ] optional, separately approved test portfolio ID and owner-token handling plan for mutation smoke

## 4. Preflight — capture, then decide go/no-go

Run from the approved artifact or a clean checkout at the approved source baseline, with the secured DB environment loaded.

```bash
# Identity: must resolve to the approved release source or its artifact provenance.
git rev-parse HEAD
alembic heads
alembic current
```

Expected head is `20260715_0028`. `current` may be an earlier valid deployed revision before the upgrade, but an unknown/branched state is a **stop** condition.

Capture active work and locks before touching schema:

```sql
SELECT pid, usename, application_name, state, wait_event_type, wait_event, query
FROM pg_stat_activity
WHERE query ILIKE '%portfolio_ledger%'
  AND state <> 'idle';

SELECT locktype, mode, granted, pid, relation::regclass
FROM pg_locks
WHERE relation = 'portfolio_ledger'::regclass;
```

Capture schema and row-count baseline:

```sql
SELECT column_name, data_type, is_nullable, column_default
FROM information_schema.columns
WHERE table_name = 'portfolio_ledger'
  AND column_name IN ('source', 'target_ledger_entry_id', 'confirmation_evidence', 'actor_user_id')
ORDER BY column_name;

SELECT COUNT(*) AS rows_before FROM portfolio_ledger;
SELECT COUNT(*) AS null_source_before FROM portfolio_ledger WHERE source IS NULL;
```

**Go only when:** backup is complete, active workload/locks are absent or the window owner explicitly accepts them, `heads` has one expected head, and the current state is understood. Otherwise stop and record the result.

## 5. Backup gate

Perform the established production backup procedure. The following is an interface, not a command to run blindly:

```text
<ops-approved pg_dump/snapshot procedure> → <backup reference>
```

Record in REL-004:

- backup reference/path (not credentials), timestamp, byte size/checksum if available;
- DB/database identifier;
- restore owner and how restore viability is known.

**Stop** if the backup has not completed or cannot be located by the restore owner.

## 6. Migration gate

Execute exactly the reviewed target revision:

```bash
alembic upgrade 20260715_0028
alembic current
```

Then execute and retain secret-free output for:

```sql
SELECT column_name, data_type, is_nullable, column_default
FROM information_schema.columns
WHERE table_name = 'portfolio_ledger'
  AND column_name IN ('source', 'target_ledger_entry_id', 'confirmation_evidence', 'actor_user_id')
ORDER BY column_name;

SELECT COUNT(*) AS rows_after FROM portfolio_ledger;
SELECT COUNT(*) AS null_source_rows
FROM portfolio_ledger
WHERE source IS NULL;

SELECT column_name, is_nullable
FROM information_schema.columns
WHERE table_name = 'portfolio_ledger' AND column_name = 'source';

SELECT column_default
FROM information_schema.columns
WHERE table_name = 'portfolio_ledger' AND column_name = 'source';
```

Pass criteria:

- `alembic current` reports `20260715_0028`;
- all four columns exist with the REL-004/REQ-057 types;
- `source` is `NOT NULL`, default is `'broker'`, and `null_source_rows = 0`;
- `rows_after = rows_before`.

**Stop** on a migration error, mismatched revision, missing/incorrect column, remaining null `source`, or row-count mismatch. Do not start/restart the new service code.

## 7. Artifact deployment and activation gate

Only after Section 6 passes:

1. Record the exact immutable artifact ref/digest and source provenance in REL-004.
2. Use the already-approved **production** deploy/rolling-restart procedure maintained by trex-op.
3. Capture the deployment command class, rollout/restart result, active artifact digest, and service logs/status without secrets.
4. Do not invoke `ensure_schema()` as a substitute for this reviewed Alembic preflight/migration flow.

There is no repo-owned production rollout command. `docker-compose.local-test.yml` must not be used for a production service replacement.

## 8. Post-deploy verification

### 8.1 Mandatory read-only checks

```bash
curl --fail-with-body --silent --show-error "$PORTFOLIO_HUB_BASE_URL/health"
```

Pass: HTTP 200 and JSON includes `db_enabled: true`.

Use authorized existing read probes to check that the service is reachable and that existing portfolio reads still work. Do not log bearer tokens or response PII in REL-004.

### 8.2 Manual-ledger API authorization checks

With an approved non-sensitive test identity or non-production endpoint:

- owner user/session token: `GET /v1/portfolios/{portfolio_id}/ledger/manual` succeeds with HTTP 200;
- service/daemon token calling `POST .../ledger/manual` is rejected HTTP 403;
- non-owner user/session token is rejected HTTP 403.

### 8.3 Optional mutation smoke — only with explicit approval

Only after Justin Ko explicitly authorizes the named test portfolio and operation:

- submit one bounded `INVEST` entry through the new manual endpoint;
- verify HTTP 201, returned `source="manual"`, and expected after-snapshot;
- retrieve the entry via manual history;
- if business policy permits, exercise approved reversal of that exact test entry and verify append-only behavior.

This is business-data mutation. Never use an unapproved customer, live strategy-managed, or production trading portfolio as a convenience test target. A daemon-managed portfolio needs the REQ-056 confirmation evidence flow.

## 9. Observation and stop/rollback decision

Observe the service for the agreed window. Stop and roll back code if health fails, 5xx/authorization anomalies appear, or existing `/ledger/trades`/portfolio reads regress.

- **Migration not successful:** do not deploy the new artifact; preserve command output and consult DB owner.
- **Migration successful, code problem:** use the platform’s approved rollback to a reviewed compatible prior artifact; retain revision `20260715_0028` and all four columns.
- **Restore needed:** halt and obtain separate incident/recovery approval. Do not self-initiate downgrade, column deletion, or restore.

## 10. Evidence handoff template

Send the following completed facts to the TREX lead for REL-004 decision:

```text
REL-004 execution
- approval: <approver, time, window>
- operator: <name>
- source/artifact: <commit>, <immutable digest>
- backup: <reference, time, size/checksum, restore owner>
- preflight: <heads/current>, <locks result>, <rows_before>, <null_source_before>
- migration: <command exit/result>, <current_after>, <rows_after>, <null_source_rows>, <schema/default evidence>
- deployment: <platform procedure/result>, <active artifact digest>
- read-only smoke: <health result>, <existing read probes>
- authorization smoke: <owner/read, service 403, non-owner 403>
- mutation smoke: <not authorized | test portfolio approval + result>
- observation: <duration, alerts/errors>
- rollback: <not needed | reason/action>
```