Troubleshooting

Common Issues

Symptom	Cause	Fix
Paid chat requests return 500 with `error.type = "internal_error"` and `"all providers failed"`	No upstream provider API key configured for the resolved model's provider	Set at least one matching provider key in `.env` (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`, `XAI_API_KEY`, `DEEPSEEK_API_KEY`) and restart the gateway
402 response but SDK does not auto-pay	Wallet key env var not set, or read under the wrong name (the CLI reads `SOLANA_WALLET_KEY`; the TypeScript/Python SDK examples read `SOLANA_PRIVATE_KEY` — the env var name is whatever you pass to `Wallet.fromEnv()` / `Wallet.from_env()`)	Set the env var your code reads and verify the value is a base58-encoded Solana keypair
402 returned with `accepts[0].amount: "0"`	Using a free model (`gpt-oss-120b`)	Free models do not require payment; remove the `PAYMENT-SIGNATURE` header
Redis connection refused	Redis not running	Run `docker compose up -d redis`
402 `"transaction has already been used; each payment signature may only be submitted once"`	Same transaction signature submitted twice	Generate a new transaction for each request
400 `"payment amount insufficient: paid N but cost is M atomic USDC"`	Signed amount less than the quoted price	Fetch a fresh 402, use `accepts[0].amount` exactly (atomic units: 1 USDC = 1,000,000)
400 `"Payment recipient does not match. Use the pay_to advertised in the 402 response."`	Payment sent to wrong wallet	Send to `accepts[0].pay_to` from the 402 response
429 Too Many Requests	Per-client rate limit exceeded (payer wallet → IP fallback)	Wait until `X-RateLimit-Reset` (also surfaced as `Retry-After`), then retry
502 Bad Gateway with `error.type = "provider_error"`	Upstream LLM provider returned an error	Check `RUST_LOG=gateway=debug` for sanitised details (raw provider errors are redacted from the response)
404 `model not found: …` (`error.type = "model_not_found"`)	Model ID not in registry	Use `GET /v1/models` to list valid model IDs
Escrow claim failures	Fee payer SOL balance too low	Fund fee payer wallets; check `GET /v1/escrow/health`
Metrics endpoint returns 404	`SOLVELA_ADMIN_TOKEN` not set — endpoint is hidden when no token is configured	Set `SOLVELA_ADMIN_TOKEN` (or `RCR_ADMIN_TOKEN` for backward compat) and use `Authorization: Bearer <token>`
Metrics endpoint returns 401	Token configured but request omitted/sent wrong `Authorization: Bearer …` header	Send `Authorization: Bearer $SOLVELA_ADMIN_TOKEN`
Wallet stats returns 503	PostgreSQL not configured	Set `DATABASE_URL` in `.env`
CORS errors in browser	Origin not in allowlist	Add your domain to `SOLVELA_CORS_ORIGINS` (or `RCR_CORS_ORIGINS` for backward compat)
Slow responses (>10s)	Provider latency or network issues	Check `solvela_provider_request_duration_seconds` metric; try a different provider
"circuit breaker open" in escrow health	>50% claim failures in 5-minute window	Wait 1 minute for auto-reset; check fee payer balance and RPC connectivity

Debug Headers

Enable debug headers to diagnose routing and payment issues:

curl -X POST http://localhost:8402/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Solvela-Debug: true" \
  -H "PAYMENT-SIGNATURE: <payment>" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "Hello"}]}'

The response includes:

X-Solvela-Request-Id: 550e8400-e29b-41d4-a716-446655440000
X-Solvela-Model: google/gemini-2.5-flash
X-Solvela-Profile: auto
X-Solvela-Tier: simple
X-Solvela-Score: -0.3500
X-Solvela-Provider: google
X-Solvela-Cache: miss
X-Solvela-Payment-Status: verified
X-Solvela-Token-Estimate-In: 10
X-Solvela-Token-Estimate-Out: 1000
X-Solvela-Latency-Ms: 1247

X-Solvela-Model is the model the request actually routed to (already resolved from an alias or a profile like auto); the routing decision is broken out across X-Solvela-Profile, X-Solvela-Tier, and X-Solvela-Score. Input and output token estimates are separate headers (-In / -Out).

The legacy X-RCR-* header set (the X-RCR-Debug request flag plus an X-RCR- mirror of every response header above) is still emitted for backward compatibility with pre-rebrand clients. X-Solvela-Request-Id (and its X-RCR-Request-Id mirror) is always returned, not gated by the debug flag.

Health Check Interpretation

curl -s http://localhost:8402/health | jq

Response	Meaning
`{"status": "ok"}`	Gateway is running and ready to serve requests
Connection refused	Gateway process is not running
Timeout	Gateway is starting up or overloaded

For deeper diagnostics, check the escrow health endpoint:

curl -s http://localhost:8402/v1/escrow/health \
  -H "Authorization: Bearer <admin-token>" | jq

Key fields to check:

claims_failed > 0: some claims are failing; check fee payer balance
circuit_breaker_open: true: claiming is paused; check RPC connectivity
queue_depth > 50: claim processing is falling behind
fee_payers_healthy < fee_payers_total: some fee payer keys are in cooldown

CLI Diagnostics

The solvela doctor command runs a comprehensive check:

cargo run -p solvela-cli -- doctor

Checks performed:

Wallet loaded -- verifies SOLANA_WALLET_KEY is valid
Gateway reachable -- connects to the gateway's /health endpoint
Models available -- fetches model list from /v1/models
RPC connected -- verifies Solana RPC connectivity
Balance check -- queries USDC-SPL balance
Payment flow -- sends a test request and verifies the 402/200 cycle

Each check reports PASS, FAIL, WARN, or SKIP.

Log Levels

Set via RUST_LOG:

# Recommended for production
RUST_LOG=gateway=info,tower_http=info

# For debugging request flow
RUST_LOG=gateway=debug,tower_http=debug

# For debugging payment verification
RUST_LOG=gateway=debug,x402=debug

# For debugging smart routing
RUST_LOG=gateway=debug,router=debug

# Maximum verbosity (noisy)
RUST_LOG=debug

The gateway uses structured logging via tracing:

INFO gateway::routes::chat: processing request wallet=7YkAz... model=openai/gpt-4o
INFO gateway::providers::openai: forwarding to OpenAI model=gpt-4o tokens_est=150
INFO gateway::usage: spend logged wallet=7YkAz... cost_usdc=0.006563 model=openai/gpt-4o

Database Troubleshooting

If PostgreSQL-dependent features (stats, spend logging) are not working:

# Check PostgreSQL is running
docker compose ps postgres

# Check connectivity
psql postgres://solvela:solvela_dev_password@localhost:5432/solvela -c "SELECT 1"

# Check migrations ran
psql postgres://solvela:solvela_dev_password@localhost:5432/solvela \
  -c "\dt"

# Check spend logs
psql postgres://solvela:solvela_dev_password@localhost:5432/solvela \
  -c "SELECT COUNT(*) FROM spend_log"

Redis Troubleshooting

If caching or replay protection is not working:

# Check Redis is running
docker compose ps redis

# Check connectivity
redis-cli -h localhost ping

# Check cache keys (prefixes: solvela:cache: for response cache, solvela:txn: for replay protection)
redis-cli -h localhost keys "solvela:*" | head -20

# Check memory usage
redis-cli -h localhost info memory | grep used_memory_human