Production Architecture

Queues, persistence, background jobs, concurrency, rate limits, and cost controls for a real agent service.

A CLI run() is a lab. Production is: accept a job, persist it, run the loop where it cannot take the website down, and refuse to light money on fire.

Queue and persist

The HTTP request should write a job row (queued) and return an id. Do not run the model on the request thread.
A worker process loads the job, runs Agent.ask, stores messages and the trace id, marks done or failed.
Postgres or SQLite is fine. The point is crash-safe: restart the worker and it picks queued / running again.

Concurrency and rate limits

One worker per process is a valid start. If you fan out, cap concurrent model calls (a semaphore). Cap requests per user per minute. When the provider returns 429, apply the same retry backoff you wrote for tools — then requeue, do not tight-loop.

Cost controls

max_steps and a session token budget — hard stop, recorded on the job.
A cheaper model for planning, a stronger one for the last structured submit — optional, not required.
Kill switches: disable dangerous tools in config without a deploy if you can.

# systemd: one worker, env file, restart on failure
# nginx: terminate TLS, proxy /jobs to your app on 127.0.0.1
# the app: POST /jobs -> {id}; GET /jobs/{id} -> status + result JSON

This matches how this site already ships processes: a unit, a reverse proxy, secrets in an env file. Next: assemble a deployable agent and score it.