> ## Documentation Index
> Fetch the complete documentation index at: https://docs.axon402.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Operator Runbook

> Operational procedures for running the Decision daemon and dashboard

> [!NOTE]
>
> This page is for self-hosted operators. Hosted end users should use the dashboard and `axon-op`, not direct database mutation.

## Two-Surface Support Note

If a user reports `TOOL_NOT_AVAILABLE_IN_RUNTIME_SURFACE`, they are on the runtime surface and tried to use an operator-only tool. The remediation is to move the workflow to the dashboard, `@axon402/cli` (`axon-op`), or the explicit operator/manage MCP surface. Do not ask them to broaden the runtime key or re-enable a mixed default surface.

## Launch Guardrails

* Keep provider-search feedback publishing and retention flags off unless you have deployed the receiver URL, signing key, and monitoring for that pipeline.
* Treat SQL mutation paths below as break-glass procedures for self-hosted environments, not normal product workflows.

### Triage table

| Symptom | Likely cause | Action |
| :- | :- | :- |
| `TOOL_NOT_AVAILABLE_IN_RUNTIME_SURFACE` with `required_surface: "operator"` | Wallet-scoped runtime key hit a control-plane tool name | Point them to [Two-Surface Migration](https://docs.axon402.com/docs/integrations/two-surface-migration.md); mint an operator key, mount the operator entry, retry on that surface |
| `REJECTED_POLICY` on `POST /v1/operator/mcp` | Runtime key or platform session hit the operator entrypoint | Confirm the caller's key shape via `GET /v1/me`; operator entry requires an org-wide key or static operator key |
| `tools/list` on the runtime MCP shows a control-plane tool (e.g. `create_wallet`) | Surface leak — should not happen on current builds | File a bug against `axon-402-mcp`; do not suppress on the client |
| IDE had one MCP server that used to do everything | Legacy mixed setup | Walk them through the step-by-step in [Two-Surface Migration](https://docs.axon402.com/docs/integrations/two-surface-migration.md#step-by-step-migration) — never advise collapsing the two entries back together |
| User unsure which surface they are on | Unclear credential origin | `curl -H "Authorization: Bearer $KEY" $AXON_API_URL/v1/me` and read `surface` + `wallet_id` |

Do not recommend broadening a runtime key, issuing a wildcard-scope key, or bridging the surfaces. The split is the product model; support guidance should reinforce it, not soften it.

## Startup Checklist

1. Confirm the daemon is running on port `3030`
2. Confirm runtime config from the host environment or Secret Manager. For local development, the canonical file is `axon-402-mcp/.env`.
3. Run `curl $AXON_API_URL/health`
4. Run `curl $AXON_API_URL/ready` — confirm `ready: true`
5. Confirm one low-risk governed fetch succeeds
6. Confirm one approval-required request can be approved and retried
7. Confirm `/v1/orgs/<org_id>/vault` shows real on-chain USDC balance
8. Confirm `/v1/orgs/<org_id>/workforce` shows non-negative spendable balances
9. Confirm `axon-op auth whoami` succeeds with an org-wide operator key

## Approval Handling

### Dashboard Approval

1. Open the **Approvals** page
2. Review `intent`, `host`, `amount`, `risk_score`, and `alignment_score`
3. Click **Approve** or **Deny**

### Headless Approval

> [!WARNING]
>
> This repo does not expose a dedicated REST approval endpoint. For headless operation in a self-hosted environment, the break-glass path is direct database mutation.

```sql theme={null}
UPDATE spend_approvals
SET status = 'approved',
    approved_by = '<operator-user-id>',
    approved_at = now()
WHERE id = '<approval_id>'
  AND status = 'pending';
```

Then consume the token:

```bash theme={null}
curl -s -X POST \
  -H "Authorization: Bearer $AXON_DECISION_API_KEY" \
  "$AXON_API_URL/v1/approvals/$APPROVAL_ID/consume"
```

Approvals expire after `AXON_APPROVAL_EXPIRY_MINUTES` (default `5`).

## Manual Freeze Procedure

**Dashboard**

1. Click **Emergency Stop** in the app shell
2. Confirm the freeze action

The current UI is a global freeze control. Scoped freeze is available through MCP or SQL.

**MCP Tool**

```json theme={null}
{
  "tool": "emergency_stop",
  "args": {
    "scope": "pocket",
    "target_id": "<wallet_id>",
    "reason": "Operator freeze: <reason>"
  }
}
```

**SQL (Last Resort)**

```sql theme={null}
-- Freeze a single agent pocket
UPDATE wallets SET is_active = false WHERE id = '<wallet_id>';

-- Freeze all wallets in an organization
UPDATE wallets SET is_active = false WHERE organization_id = '<org_id>';

-- Cancel pending transactions
UPDATE transactions
SET status = 'failed', failure_reason = 'Manual operator freeze'
WHERE wallet_id = '<wallet_id>' AND status = 'pending';

-- Unfreeze
UPDATE wallets SET is_active = true WHERE id = '<wallet_id>';
```

Target: complete freeze in under 5 minutes.

## Incident Triage Queries

### Recent Denied Decisions

```sql theme={null}
SELECT de.created_at, de.reason_code, de.reason_text, de.risk_score,
       sr.origin_host, sr.url, sr.intent
FROM decision_events de
JOIN spend_requests sr ON sr.id = de.spend_request_id
WHERE de.outcome = 'denied'
ORDER BY de.created_at DESC
LIMIT 20;
```

### Spending Velocity (Last Hour)

```sql theme={null}
SELECT w.name, COUNT(*) as requests,
       SUM(t.amount_usdc) as total_usdc
FROM transactions t
JOIN wallets w ON w.id = t.wallet_id
WHERE t.created_at > now() - interval '1 hour'
GROUP BY w.name
ORDER BY total_usdc DESC;
```

### Escalated Approvals Waiting

```sql theme={null}
SELECT sa.id, sa.status, sa.requested_at, sa.expires_at,
       sr.origin_host, sr.intent, sr.url
FROM spend_approvals sa
JOIN spend_requests sr ON sr.id = sa.spend_request_id
WHERE sa.status = 'pending'
ORDER BY sa.requested_at ASC;
```

### Webhook Delivery Health

```sql theme={null}
SELECT status, COUNT(*) as count
FROM event_outbox
WHERE created_at > now() - interval '1 hour'
GROUP BY status;
```

### Dead-Letter Webhooks

```sql theme={null}
SELECT eo.event_name, eo.last_error, eo.attempts, wc.url
FROM event_outbox eo
JOIN webhook_configs wc ON wc.id = eo.webhook_config_id
WHERE eo.status = 'dead_letter'
ORDER BY eo.created_at DESC
LIMIT 10;
```

## Rollback Checklist

### Level 1: Freeze All Agents

```sql theme={null}
UPDATE wallets SET is_active = false WHERE role = 'agent_pocket';
```

### Level 2: Disable Background Workers

```env theme={null}
AXON_FORENSIC_WEBHOOKS_ENABLED=false
AXON_PROVIDER_SYNC_ENABLED=false
AXON_PROVIDER_HEALTH_CHECK_ENABLED=false
AXON_APPROVAL_EXPIRY_WORKER_ENABLED=false
AXON_CONFIRMATION_WORKER_ENABLED=false
AXON_TREASURY_RECONCILIATION_ENABLED=false
AXON_REPAIR_QUEUE_WORKER_ENABLED=false
AXON_PROVIDER_SEARCH_FEEDBACK_PUBLISH_ENABLED=false
AXON_PROVIDER_SEARCH_FEEDBACK_RETENTION_ENABLED=false
```

Restart the daemon after changing these values.

### Level 3: Zero-Out Mandate Limits

```sql theme={null}
UPDATE mandates SET
  per_transaction_limit_usdc = 0,
  daily_limit_usdc = 0
WHERE wallet_id IN (SELECT id FROM wallets WHERE role = 'agent_pocket');
```

### Level 4: Shutdown the Daemon

```bash theme={null}
kill -TERM $DAEMON_PID
```

The daemon handles `SIGTERM` gracefully.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.