Operator Runbook

This page is for self-hosted operators. Hosted end users should use the dashboard and axon-op, not direct database mutation.

Two-Surface Support Note

If a user reports TOOL_NOT_AVAILABLE_IN_RUNTIME_SURFACE, they are on the runtime surface and tried to use an operator-only tool. The remediation is to move the workflow to the dashboard, @axon402/cli (axon-op), or the explicit operator/manage MCP surface. Do not ask them to broaden the runtime key or re-enable a mixed default surface.

Launch Guardrails

Keep provider-search feedback publishing and retention flags off unless you have deployed the receiver URL, signing key, and monitoring for that pipeline.
Treat SQL mutation paths below as break-glass procedures for self-hosted environments, not normal product workflows.

Triage table

Symptom	Likely cause	Action
`TOOL_NOT_AVAILABLE_IN_RUNTIME_SURFACE` with `required_surface: "operator"`	Wallet-scoped runtime key hit a control-plane tool name	Point them to Two-Surface Migration; mint an operator key, mount the operator entry, retry on that surface
`REJECTED_POLICY` on `POST /v1/operator/mcp`	Runtime key or platform session hit the operator entrypoint	Confirm the caller’s key shape via `GET /v1/me`; operator entry requires an org-wide key or static operator key
`tools/list` on the runtime MCP shows a control-plane tool (e.g. `create_wallet`)	Surface leak — should not happen on current builds	File a bug against `axon-402-mcp`; do not suppress on the client
IDE had one MCP server that used to do everything	Legacy mixed setup	Walk them through the step-by-step in Two-Surface Migration — never advise collapsing the two entries back together
User unsure which surface they are on	Unclear credential origin	`curl -H "Authorization: Bearer $KEY" $AXON_API_URL/v1/me` and read `surface` + `wallet_id`

Do not recommend broadening a runtime key, issuing a wildcard-scope key, or bridging the surfaces. The split is the product model; support guidance should reinforce it, not soften it.

Startup Checklist

Confirm the daemon is running on port 3030
Confirm runtime config from the host environment or Secret Manager. For local development, the canonical file is axon-402-mcp/.env.
Run curl $AXON_API_URL/health
Run curl $AXON_API_URL/ready — confirm ready: true
Confirm one low-risk governed fetch succeeds
Confirm one approval-required request can be approved and retried
Confirm /v1/orgs/<org_id>/vault shows real on-chain USDC balance
Confirm /v1/orgs/<org_id>/workforce shows non-negative spendable balances
Confirm axon-op auth whoami succeeds with an org-wide operator key

Approval Handling

Dashboard Approval

Open the Approvals page
Review intent, host, amount, risk_score, and alignment_score
Click Approve or Deny

Headless Approval

This repo does not expose a dedicated REST approval endpoint. For headless operation in a self-hosted environment, the break-glass path is direct database mutation.

UPDATE spend_approvals
SET status = 'approved',
    approved_by = '<operator-user-id>',
    approved_at = now()
WHERE id = '<approval_id>'
  AND status = 'pending';

Then consume the token:

curl -s -X POST \
  -H "Authorization: Bearer $AXON_DECISION_API_KEY" \
  "$AXON_API_URL/v1/approvals/$APPROVAL_ID/consume"

Approvals expire after AXON_APPROVAL_EXPIRY_MINUTES (default 5).

Manual Freeze Procedure

Dashboard
MCP Tool
SQL (Last Resort)

Click Emergency Stop in the app shell
Confirm the freeze action

The current UI is a global freeze control. Scoped freeze is available through MCP or SQL.

{
  "tool": "emergency_stop",
  "args": {
    "scope": "pocket",
    "target_id": "<wallet_id>",
    "reason": "Operator freeze: <reason>"
  }
}

-- Freeze a single agent pocket
UPDATE wallets SET is_active = false WHERE id = '<wallet_id>';

-- Freeze all wallets in an organization
UPDATE wallets SET is_active = false WHERE organization_id = '<org_id>';

-- Cancel pending transactions
UPDATE transactions
SET status = 'failed', failure_reason = 'Manual operator freeze'
WHERE wallet_id = '<wallet_id>' AND status = 'pending';

-- Unfreeze
UPDATE wallets SET is_active = true WHERE id = '<wallet_id>';

Target: complete freeze in under 5 minutes.

Incident Triage Queries

Recent Denied Decisions

SELECT de.created_at, de.reason_code, de.reason_text, de.risk_score,
       sr.origin_host, sr.url, sr.intent
FROM decision_events de
JOIN spend_requests sr ON sr.id = de.spend_request_id
WHERE de.outcome = 'denied'
ORDER BY de.created_at DESC
LIMIT 20;

Spending Velocity (Last Hour)

SELECT w.name, COUNT(*) as requests,
       SUM(t.amount_usdc) as total_usdc
FROM transactions t
JOIN wallets w ON w.id = t.wallet_id
WHERE t.created_at > now() - interval '1 hour'
GROUP BY w.name
ORDER BY total_usdc DESC;

Escalated Approvals Waiting

SELECT sa.id, sa.status, sa.requested_at, sa.expires_at,
       sr.origin_host, sr.intent, sr.url
FROM spend_approvals sa
JOIN spend_requests sr ON sr.id = sa.spend_request_id
WHERE sa.status = 'pending'
ORDER BY sa.requested_at ASC;

Webhook Delivery Health

SELECT status, COUNT(*) as count
FROM event_outbox
WHERE created_at > now() - interval '1 hour'
GROUP BY status;

Dead-Letter Webhooks

SELECT eo.event_name, eo.last_error, eo.attempts, wc.url
FROM event_outbox eo
JOIN webhook_configs wc ON wc.id = eo.webhook_config_id
WHERE eo.status = 'dead_letter'
ORDER BY eo.created_at DESC
LIMIT 10;

Rollback Checklist

Level 1: Freeze All Agents

UPDATE wallets SET is_active = false WHERE role = 'agent_pocket';

Level 2: Disable Background Workers

AXON_FORENSIC_WEBHOOKS_ENABLED=false
AXON_PROVIDER_SYNC_ENABLED=false
AXON_PROVIDER_HEALTH_CHECK_ENABLED=false
AXON_APPROVAL_EXPIRY_WORKER_ENABLED=false
AXON_CONFIRMATION_WORKER_ENABLED=false
AXON_TREASURY_RECONCILIATION_ENABLED=false
AXON_REPAIR_QUEUE_WORKER_ENABLED=false
AXON_PROVIDER_SEARCH_FEEDBACK_PUBLISH_ENABLED=false
AXON_PROVIDER_SEARCH_FEEDBACK_RETENTION_ENABLED=false

Restart the daemon after changing these values.

Level 3: Zero-Out Mandate Limits

UPDATE mandates SET
  per_transaction_limit_usdc = 0,
  daily_limit_usdc = 0
WHERE wallet_id IN (SELECT id FROM wallets WHERE role = 'agent_pocket');

Level 4: Shutdown the Daemon

kill -TERM $DAEMON_PID

The daemon handles SIGTERM gracefully.

Getting Started

CLI

Core Concepts

Integrations

Operations

Operator Runbook

Two-Surface Support Note

Launch Guardrails

Triage table

Startup Checklist

Approval Handling

Dashboard Approval

Headless Approval

Manual Freeze Procedure

Incident Triage Queries

Recent Denied Decisions

Spending Velocity (Last Hour)

Escalated Approvals Waiting

Webhook Delivery Health

Dead-Letter Webhooks

Rollback Checklist

Level 1: Freeze All Agents

Level 2: Disable Background Workers

Level 3: Zero-Out Mandate Limits

Level 4: Shutdown the Daemon

​Two-Surface Support Note

​Launch Guardrails

​Triage table

​Startup Checklist

​Approval Handling

​Dashboard Approval

​Headless Approval

​Manual Freeze Procedure

​Incident Triage Queries

​Recent Denied Decisions

​Spending Velocity (Last Hour)

​Escalated Approvals Waiting

​Webhook Delivery Health

​Dead-Letter Webhooks

​Rollback Checklist

​Level 1: Freeze All Agents

​Level 2: Disable Background Workers

​Level 3: Zero-Out Mandate Limits

​Level 4: Shutdown the Daemon

Two-Surface Support Note

Launch Guardrails

Triage table

Startup Checklist

Approval Handling

Dashboard Approval

Headless Approval

Manual Freeze Procedure

Incident Triage Queries

Recent Denied Decisions

Spending Velocity (Last Hour)

Escalated Approvals Waiting

Webhook Delivery Health

Dead-Letter Webhooks

Rollback Checklist

Level 1: Freeze All Agents

Level 2: Disable Background Workers

Level 3: Zero-Out Mandate Limits

Level 4: Shutdown the Daemon