---
name: run-trex-core-daemon
description: "Launch the trex-core daemon against the real PostgreSQL + RabbitMQ infrastructure and drive it via the messaging surfaces (snapshot RPC, intent admission RPC, slot events). Use when asked to run trex-core, smoke-test the daemon, confirm a change works end-to-end, or reproduce the BAT-007 transport lifecycle."
---

# Run trex-core (sync + thread daemon over RabbitMQ)

The daemon (BAT-007) has **no HTTP surface**. All strategy-facing traffic goes
over RabbitMQ — snapshot RPC on `trex.snapshot.req`, intent admission RPC on
`trex.intent.req`, slot lifecycle events on the `trex.events` topic exchange.
"Run" means: boot the daemon in the background, drive it with messages, look
at the responses + the events.

## Prerequisites

The daemon needs three live services. Verify before launching:

```bash
.venv/bin/python - <<'PY'
import os, psycopg, pika
from dotenv import load_dotenv
load_dotenv("config/.env")  # BAT-026: .env lives in the config dir
psycopg.connect(os.environ["TREX_POSTGRESQL_DSN"], connect_timeout=3).close()
print("postgres OK")
p = pika.URLParameters(os.environ["TREX_RABBITMQ_URL"]); p.socket_timeout = 3
pika.BlockingConnection(p).close()
print("rabbitmq OK")
PY
```

If either fails, fix the infra first — the daemon will not boot without
`TREX_POSTGRESQL_DSN` and will boot silently degraded if `TREX_RABBITMQ_URL`
is missing (stub `InMemoryTransport` — strategy modules can't reach it).

### Required env (`config/.env`, gitignored) + per-instance TOML

| Variable | Required | Notes |
|---|---|---|
| `TREX_POSTGRESQL_DSN` | yes | psycopg DSN |
| `TREX_RABBITMQ_URL` | yes for real use | e.g. `amqp://admin:admin@localhost:5672/` |
| `TREX_BROKER` | no | `paper` (default) or `kis` |
| `TREX_EXECUTION_INTERVAL_SEC` | no | default 1.0 |
| `TREX_LEDGER_INTERVAL_SEC` | no | default 1.0 |
| `TREX_PORTFOLIO_HUB_BASE_URL` | no | if set, real portfolio-hub universe+ledger; else stub universe (pf-005930/pf-000660) + in-memory ledger |

`--config <name>` (required since BAT-013) resolves `config/<name>.toml`:
`instance_key` (queue names: `trex.intent.req.<key>` etc.; `default` keeps
the literal names) and `max_active_slots` (M cap; unset = no cap).

**BAT-030 (asset-centric)**: there is NO daily preallocation. Permanent asset
slots (one per universe portfolio, status `inactive`) are created by the
snapshot scheduler's universe reconcile right after boot. Slot states are
`inactive / pending_buy / holding / pending_sell / error` — `idle`/`closed`
no longer exist.

## Launch

Background, redirect log (smoke against the test DB: point
TREX_POSTGRESQL_DSN at `trex_core_test` via `--env-file`, never the live DB):

```bash
.venv/bin/trex-core-daemon --config <instance> &> /tmp/trex-core.log &
DAEMON_PID=$!
```

The daemon does not bind any TCP port; "ready" means **four worker threads
running and all RabbitMQ topology declared**. Poll for the topology to
appear:

```bash
for i in {1..15}; do
  .venv/bin/python - <<'PY' && break
import pika, sys
p = pika.URLParameters("amqp://admin:admin@localhost:5672/"); p.socket_timeout = 2
c = pika.BlockingConnection(p)
try:
    ch = c.channel()
    # passive=True: succeeds only if the daemon already declared it
    ch.queue_declare("trex.intent.req", durable=True, passive=True)
    ch.queue_declare("trex.snapshot.req", durable=True, passive=True)
    ch.exchange_declare("trex.events", exchange_type="topic", durable=True, passive=True)
    print("daemon ready")
    sys.exit(0)
except Exception as exc:
    print(f"not ready: {exc}"); sys.exit(1)
finally:
    c.close()
PY
  sleep 0.5
done
```

`/tmp/trex-core.log` has the pika INFO chatter — first connection lines mean
boot succeeded; absence of `ERROR`/exception traceback means topology declare
worked.

## Drive

The canonical end-to-end smoke is in `smoke.py` — copy/adapt it. It exercises:

1. **snapshot RPC** (view=all) — daemon answers; `slots[]` holds the permanent
   asset slots from the universe reconcile (status `inactive`).
2. **buy intent RPC keyed by `slot_id`** (REQ-061 canonical; `portfolio_id`
   dual-accept) — admitted, slot → `pending_buy`. Rejections: `unknown_slot`,
   `already_active`, `capacity_full` (M cap).
3. **duplicate buy** with the *same* `intent_id` — identical response,
   no double transition (intent_log dedup gate, REQ-023 AC-1).
4. **wait one execution loop tick** — paper broker fills, slot → `holding`.
5. **sell intent RPC** on the holding slot — fill returns the slot to
   `inactive` (occupancy cleared, asset binding kept — NOT `closed`).
   Sell dispatch quantity is the **broker balance** (REQ-062); broker flat →
   the sell is skipped with a WARN.

Run it (instance key optional, default `default`):

```bash
.venv/bin/python .claude/skills/run-trex-core-daemon/smoke.py [instance_key]
```

Sample output (paper broker, stub universe):

```
=== 1) snapshot RPC ===
assets=1 slots=2 available_slots=1
target slot_id=8a160254-… portfolio_id=pf-005930
=== 2) buy by slot_id ===
-> admitted reason=None
=== 3) duplicate buy (same intent_id) ===
identical: True
=== 4) wait for fill ===
slot status=holding qty=10.0 avg_price=1000.0
=== 5) sell intent ===
-> admitted
slot status=inactive qty=None portfolio_id=pf-005930 (asset binding kept)
DONE
```

### Subscribing to events (timing caveat)

If you subscribe to `trex.events` *after* the publish happens, the message
is gone — topic exchanges drop unroutable publishes. Production strategy
modules have their per-strategy queue bound from boot, so they don't miss
anything; ad-hoc subscribers need to declare + bind their queue **before**
triggering the intent. `smoke.py` does this correctly (declares queue, then
sleeps 0.6s, then runs the RPC).

## Stop

```bash
kill -TERM $DAEMON_PID
# or if you've lost the PID:
pkill -f trex-core-daemon
```

The daemon installs SIGINT/SIGTERM handlers that set the stop event,
join all worker threads, and `close()` every RabbitMQ connection — last
log line should be `Normal shutdown` from pika. Exit code 0.

## Inspecting state

The daemon's source of truth is PostgreSQL `daily_slot`:

```bash
.venv/bin/python - <<'PY'
import os, psycopg
from dotenv import load_dotenv
load_dotenv("config/.env")
with psycopg.connect(os.environ["TREX_POSTGRESQL_DSN"]) as c:
    with c.cursor() as cur:
        cur.execute("""
          SELECT instance_key, status, COUNT(*)
          FROM daily_slot GROUP BY instance_key, status ORDER BY 1, 2
        """)
        for r in cur.fetchall(): print(r)
PY
```

Ledger outbox state (BAT-006):

```bash
.venv/bin/python - <<'PY'
import os, psycopg
from dotenv import load_dotenv
load_dotenv("config/.env")
with psycopg.connect(os.environ["TREX_POSTGRESQL_DSN"]) as c:
    with c.cursor() as cur:
        cur.execute("SELECT status, last_outcome, COUNT(*) FROM ledger_outbox GROUP BY 1,2 ORDER BY 1,2")
        for r in cur.fetchall(): print(r)
PY
```

## Troubleshooting

| Symptom | Cause | Fix |
|---|---|---|
| `slots returned: 0` from snapshot | universe reconcile hasn't run (portfolio-hub down / boot race) | check the scheduler log line `slot reconcile: {'created': N, ...}`; with no hub config the stub universe gives 2 slots |
| intent gets `rejected: unknown_slot` / `unknown_portfolio` | slot_id/portfolio_id not in this instance's slot set | read `slots[]` from a snapshot RPC first and reuse its `slot_id` |
| intent gets `rejected: capacity_full` | `max_active_slots` (M) reached — FCFS, no eviction | sell something first or raise M in the instance TOML |
| sell never dispatches, WARN `sell dispatch skipped` | broker balance 0 for the symbol (REQ-062 divergence) | reconcile the divergence (operator); no mirror fallback by design |
| `RpcTimeoutError` on the client | daemon not up, or consumer thread crashed | check `/tmp/trex-core.log` for exception trace |
| `TransportUnavailableError` | RabbitMQ down | the daemon does NOT buffer locally (REQ-024) — fix the broker, then re-issue |
| fill event missed by ad-hoc subscriber | queue bound *after* publish | bind queue + sleep ≥0.5s before triggering the intent |
| import-time `ModuleNotFoundError: pika` | dev deps not installed | `.venv/bin/pip install -e ".[dev]"` |

## Related docs

- `docs/interface-contract.md` — full RabbitMQ topology + message payloads
- `docs/runtime-flows.md` — flow diagrams (snapshot RPC, intent RPC, events, degrade branch)
- `docs/batches/bat-007_20260605_strategy-module-rabbitmq-transport/` — the batch that introduced this runtime shape