Files
trex-core/docs/runbooks/rel-011-m5-operator-broker-truth-hotfix-deployment.md
T

6.6 KiB
Raw Blame History

REL-011 M5 trex-core operator broker truth hotfix — deployment runbook

  • Release: REL-011
  • Target source: 6d90f8951dbf354eb41301492d0bcaf0af24d882
  • Runtime implementation: BAT-057 / PR #42 merge 94e148940adb008db0bbd215052a1e984f7c566b
  • Execution owner: 운영팀
  • Approval required: team lead review + user release approval
  • Safety classification: production daemon restart; no DB migration; broker-read smoke only

Do not paste credentials, tokens, account numbers, DSNs, .env contents, raw KIS responses, or full command lines containing secrets into the evidence log. Use [REDACTED].

1. Scope and hard stops

Allowed after release approval:

  • Pinned source checkout, existing trex-core daemon stop/start, startup log inspection.
  • Existing approved instance configuration used without semantic changes.
  • sync --all --dry-run and verify as broker-read/CLI smoke checks.

Not allowed in this runbook:

  • sync --live, --yes with a live write, enter, exit, cancel, or any automated remediation.
  • Changing credential_fetch_enabled, TREX_KIS_BROKER_BREAK_GLASS, broker mode, instance key, portfolio-hub endpoint, or secrets.
  • Database schema migration, manual DB update/delete, AMQP topology change, or strategy daemon deployment.

If any prohibited action appears necessary, stop and return to the release approver.

2. Preflight — record before modifying the host

  1. Confirm the approved release, target SHA, and approved 장외 window.

  2. From the operating checkout, capture only redacted metadata:

    pwd
    git status --short
    git rev-parse HEAD
    git rev-parse --abbrev-ref HEAD
    
    • Record the exact pre-deploy revision as ROLLBACK_REVISION.
    • If the checkout is dirty or target source cannot be verified, stop.
  3. Identify the active instance configuration and process supervisor without printing secrets. Confirm the existing config’s credential_fetch_enabled and break-glass state will remain unchanged.

  4. Confirm the expected process command remains the established runtime form:

    .venv/bin/trex-core-daemon --config <approved-instance-config>
    
  5. Confirm target/source characteristics from a clean checkout:

    git fetch origin
    git show -s --format='%H %s' 6d90f8951dbf354eb41301492d0bcaf0af24d882
    git diff --name-only "$ROLLBACK_REVISION" 6d90f8951dbf354eb41301492d0bcaf0af24d882 -- pyproject.toml trex_core/adapters/persistence/migrations
    

    Expected for the reviewed M4 baseline: no dependency or migration path. If actual output differs, stop and request lead review; do not infer safety from this draft.

  6. Ensure the deployment host’s virtual environment has the existing runtime dependencies. Do not run dependency upgrades as an implicit part of this release.

3. Deploy source and restart daemon

Use the host’s approved supervisor procedure. The following expresses the source boundary, not a replacement for host-specific service management:

git fetch origin
git checkout --detach 6d90f8951dbf354eb41301492d0bcaf0af24d882
# Stop the existing trex-core daemon through the approved host supervisor.
# Start exactly one daemon with the unchanged approved instance config.
.venv/bin/trex-core-daemon --config <approved-instance-config>

Record, with sensitive values redacted:

  • checkout SHA;
  • prior process stop confirmation;
  • new process identifier or supervisor status;
  • selected instance key (not account details);
  • startup log interval and log location.

4. Startup acceptance

Before any CLI smoke, confirm:

  • the daemon completed startup and RabbitMQ topology declaration;
  • worker threads are running according to the instance profile;
  • no ERROR, CRITICAL, or traceback appears in the post-restart interval;
  • the daemon is using the unchanged configuration mode. Source evidence may show only safe mode/label (fetched, env, or break_glass), never credential payload or account number.

Any KIS credential-resolution or balance-read failure must be treated as a failure; BAT-057 intentionally prevents treating it as a normal empty broker balance.

5. Read-only/dry-run smoke

Run against the same explicit approved configuration. These calls may read the configured broker and database but must not write a slot, warning, broker order, or ledger record.

.venv/bin/trex-core sync --all --dry-run --config <approved-instance-config> --json
.venv/bin/trex-core verify --config <approved-instance-config> --json

Acceptance:

  • Commands either succeed with sanitized source/balance evidence, or fail loudly with safe error code/message.
  • A failure must not be reinterpreted as an empty balance or “zero quantity” condition.
  • For unmanaged_position or managed_non_holding_divergence, record classification only and stop for triage; no remediation is performed.
  • Never add --live or --yes in this runbook.

6. Rollback

Trigger rollback for any daemon startup failure, unexpected source-mode change, unexpected credential fetch/fallback behavior, broker-read failure not safely surfaced, AMQP connectivity regression, or smoke behavior inconsistent with BAT-057’s fail-loud/no-auto-remediation contract.

# Stop the new daemon via the approved host supervisor.
git checkout --detach "$ROLLBACK_REVISION"
# Restart with the same unchanged approved configuration via the supervisor.

Then confirm the previous daemon revision is running and collect redacted logs. Do not run database rollback commands; REL-011 introduces no migration.

7. Evidence record template

Release: REL-011
Approved by / timestamp: [REDACTED AS NEEDED]
Window: [REDACTED]
Host identity: [REDACTED]
Instance key: [REDACTED]
Pre-deploy revision (ROLLBACK_REVISION): <sha>
Target revision: 6d90f8951dbf354eb41301492d0bcaf0af24d882
Configuration mode unchanged: yes/no (credential_fetch_enabled / break-glass values not recorded)
Daemon stop/start evidence: <supervisor status or PID, redacted>
Startup log interval: <path/time range, redacted>
Startup acceptance: pass/fail
sync --dry-run result: pass/fail + sanitized exit code/output summary
verify result: pass/fail + sanitized exit code/output summary
Unexpected divergence classification: none / unmanaged_position / managed_non_holding_divergence
Rollback performed: no / yes (reason and restored SHA)
Operator: [REDACTED]

8. Completion criteria

REL-011 may move from Release-Pending only after the exact target revision, restart evidence, startup acceptance, dry-run smoke result, verify result, and rollback disposition are reviewed by the team lead. Production live sync remains a separate approval gate even after REL-011 is released.