Troubleshooting

Common Issues

Symptom	Cause	Fix
Paid chat requests return 500 with `error.type = "internal_error"` and `"all providers failed"`	No upstream provider API key configured for the resolved model's provider	Set at least one matching provider key in `.env` (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`, `XAI_API_KEY`, `DEEPSEEK_API_KEY`, `NVIDIA_API_KEY`) and restart the gateway
402 response but SDK does not auto-pay	Wallet key env var not set, or read under the wrong name (the CLI reads `SOLANA_WALLET_KEY`; the TypeScript/Python SDK examples read `SOLANA_PRIVATE_KEY` — the env var name is whatever you pass to `Wallet.fromEnv()` / `Wallet.from_env()`)	Set the env var your code reads and verify the value is a base58-encoded Solana keypair
Free-model request served at $0 even with a `PAYMENT-SIGNATURE` header	Using a free model (`google/gemini-3.1-flash-lite`) — zero-cost requests skip the 402 challenge entirely and any payment header is ignored	Expected behavior; no payment is needed for free models
Redis connection refused	Redis not running	Run `docker compose up -d redis`
402 `"transaction has already been used; each payment signature may only be submitted once"`	Same transaction signature submitted twice	Generate a new transaction for each request
400 `"payment amount insufficient: paid N but cost is M atomic USDC"`	Signed amount less than the quoted price	Fetch a fresh 402, use `accepts[0].amount` exactly (atomic units: 1 USDC = 1,000,000)
400 `"Payment recipient does not match. Use the pay_to advertised in the 402 response."`	Payment sent to wrong wallet	Send to `accepts[0].pay_to` from the 402 response
429 Too Many Requests	Per-client rate limit exceeded (payer wallet → IP fallback)	Wait until `X-RateLimit-Reset` (also surfaced as `Retry-After`), then retry
502 Bad Gateway with `error.type = "provider_error"`	Upstream LLM provider returned an error	Check `RUST_LOG=gateway=debug` for sanitised details (raw provider errors are redacted from the response)
404 `model not found: …` (`error.type = "model_not_found"`)	Model ID not in registry	Use `GET /v1/models` to list valid model IDs
Escrow claim failures	Fee payer SOL balance too low	Fund fee payer wallets; check `GET /v1/escrow/health`
Metrics endpoint returns 404	`SOLVELA_ADMIN_TOKEN` not set — endpoint is hidden when no token is configured	Set `SOLVELA_ADMIN_TOKEN` (or `RCR_ADMIN_TOKEN` for backward compat) and use `Authorization: Bearer <token>`
Metrics endpoint returns 401	Token configured but request omitted/sent wrong `Authorization: Bearer …` header	Send `Authorization: Bearer $SOLVELA_ADMIN_TOKEN`
Wallet stats returns 503	PostgreSQL not configured	Set `DATABASE_URL` in `.env`
CORS errors in browser	Origin not in allowlist	Add your domain to `SOLVELA_CORS_ORIGINS` (or `RCR_CORS_ORIGINS` for backward compat)
Slow responses (>10s)	Provider latency or network issues	Check `solvela_provider_request_duration_seconds` metric; try a different provider
"claim circuit breaker OPENED" in gateway logs	>50% claim failures in 5-minute window (minimum 4 samples)	Wait 1 minute for auto-reset; check fee payer balance and RPC connectivity

Debug Headers

Enable debug headers to diagnose routing and payment issues:

curl -X POST http://localhost:8402/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Solvela-Debug: true" \
  -H "PAYMENT-SIGNATURE: <payment>" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "Hello"}]}'

The response includes:

X-Solvela-Request-Id: 550e8400-e29b-41d4-a716-446655440000
X-Solvela-Model: google/gemini-2.5-flash
X-Solvela-Profile: auto
X-Solvela-Tier: simple
X-Solvela-Score: -0.3500
X-Solvela-Provider: google
X-Solvela-Cache: miss
X-Solvela-Payment-Status: verified
X-Solvela-Token-Estimate-In: 10
X-Solvela-Token-Estimate-Out: 1000
X-Solvela-Latency-Ms: 1247

X-Solvela-Model is the model the request actually routed to (already resolved from an alias or a profile like auto); the routing decision is broken out across X-Solvela-Profile, X-Solvela-Tier, and X-Solvela-Score. Input and output token estimates are separate headers (-In / -Out).

The legacy X-RCR-* header set (the X-RCR-Debug request flag plus an X-RCR- mirror of every response header above) is still emitted for backward compatibility with pre-rebrand clients. X-Solvela-Request-Id (and its X-RCR-Request-Id mirror) is always returned, not gated by the debug flag.

Health Check Interpretation

curl -s http://localhost:8402/health | jq

Response	Meaning
`{"status": "ok"}`	Gateway is running and ready to serve requests
Connection refused	Gateway process is not running
Timeout	Gateway is starting up or overloaded

For deeper diagnostics, check the escrow health endpoint:

curl -s http://localhost:8402/v1/escrow/health \
  -H "Authorization: Bearer <admin-token>" | jq

Key fields to check:

status: "ok", "degraded" (claim processor not running or no fee payer wallets), or "down" (escrow disabled)
claims.failed > 0: some claims are failing; check fee payer balance
claims.pending_in_queue > 50: claim processing is falling behind
fee_payer_wallets == 0: no fee payer keys loaded — claims cannot be submitted

CLI Diagnostics

The solvela doctor command runs a quick diagnostic:

cargo run -p solvela-cli -- doctor

Checks performed:

Gateway reachable -- connects to the gateway's /health endpoint (OK / ERROR / FAIL)
Wallet file -- checks that ~/.solvela/wallet.json exists (FOUND / NOT FOUND); it does not validate the key
Models available -- fetches the model list from /v1/models (OK (N models) / UNAVAILABLE)
SOLANA_WALLET_KEY -- reports whether the env var is set (SET / NOT SET)

It does not test Solana RPC connectivity, USDC balance, or the 402/200 payment cycle — use solvela wallet status and a real request for those.

Log Levels

Set via RUST_LOG:

# Recommended for production
RUST_LOG=gateway=info,tower_http=info

# For debugging request flow
RUST_LOG=gateway=debug,tower_http=debug

# For debugging payment verification
RUST_LOG=gateway=debug,x402=debug

# For debugging smart routing
RUST_LOG=gateway=debug,router=debug

# Maximum verbosity (noisy)
RUST_LOG=debug

The gateway uses structured logging via tracing:

INFO gateway::routes::chat: processing request wallet=7YkAz... model=openai/gpt-4o
INFO gateway::providers::openai: forwarding to OpenAI model=gpt-4o tokens_est=150
INFO gateway::usage: spend logged wallet=7YkAz... cost_usdc=0.006563 model=openai/gpt-4o

Database Troubleshooting

If PostgreSQL-dependent features (stats, spend logging) are not working:

# Check PostgreSQL is running
docker compose ps postgres

# Check connectivity
psql postgres://solvela:solvela_dev_password@localhost:5432/solvela -c "SELECT 1"

# Check migrations ran
psql postgres://solvela:solvela_dev_password@localhost:5432/solvela \
  -c "\dt"

# Check spend logs
psql postgres://solvela:solvela_dev_password@localhost:5432/solvela \
  -c "SELECT COUNT(*) FROM spend_logs"

Redis Troubleshooting

If caching or replay protection is not working:

# Check Redis is running
docker compose ps redis

# Check connectivity
redis-cli -h localhost ping

# Check cache keys (prefixes: solvela:cache: for response cache, solvela:txn: for replay protection)
redis-cli -h localhost keys "solvela:*" | head -20

# Check memory usage
redis-cli -h localhost info memory | grep used_memory_human