title: Scaling workers description: How to run Dendrux across many processes: build the store once at startup, size pools, use PgBouncer, migrate as a single job, schedule sweeps and know every bound.

Scaling workers

Dendrux is a library. You own the processes, the database pool, the migration job and the schedule. The database coordinates claims: any worker can resume any run, and Dendrux keeps no state that matters in process memory. This page lists what to build once, what to size, and which limits bound the work a process accepts.

Build the store once at startup

Build the store when a worker starts and pass it to every Agent that worker serves. Constructing an Agent does no I/O and is cheap; the engine and its pool are not.

from sqlalchemy.ext.asyncio import create_async_engine
from dendrux import Agent
from dendrux.runtime.state import SQLAlchemyStateStore
from dendrux.store import RunStore
 
engine = create_async_engine(
    "postgresql+asyncpg://user:password@database/app",
    pool_size=5,
    max_overflow=0,
    pool_timeout=10,
    pool_pre_ping=True,
    pool_recycle=1800,
)
state = SQLAlchemyStateStore(engine)
reads = RunStore.from_engine(engine)
agent = Agent(
    name="worker",
    prompt="Help the user.",
    provider="openai:gpt-4o-mini",
    state_store=state,
)
# Start application requests here, then drain them before shutdown.
await agent.close()
await reads.close()
await engine.dispose()

SQLAlchemyStateStore runs the agents; RunStore.from_engine supplies the read facade over the same engine. You keep ownership of the engine and dispose it after all work and readers have stopped. Avoid holding a database session while you await an agent run, a tool or a browser.

Build the engine in the worker process, after any fork, on the event loop that will use it. Async Postgres pools belong to their creating loop; using one from another loop is refused with a clear error. Build one store per worker loop instead of calling asyncio.run per request. See also Web endpoints.

What happens if you pass database_url= per request

An Agent(database_url=...) with the same normalised URL and the same database_options reuses one process-wide engine:

• Concurrent first callers share one initialisation. Each initialised Agent holds one reference.
• Agent.close() releases the reference; the engine is disposed when the last holder closes or the process exits. Close every Agent, including after failed requests (async with Agent(...) does this).
• Different options select different pools on purpose.
• Equal in-memory SQLite URLs share one database while the pool lives. Use a distinct engine or a temporary file when tests need isolation.
• A pool whose owning event loop has closed is retired lazily on the next matching acquisition or release. Orphaned connections are released by garbage collection, so they can linger until then and briefly exceed the new pool's limits. Each retirement logs one WARNING with code=shared_engine_loop_closed, without a URL or credentials. No timer or background task is involved: await agent.close() before you close the loop for prompt cleanup.
• At process exit pools are retired synchronously and the operating system closes the sockets.

This stops a per-message Agent from opening a new pool each time, but an explicit state_store= stays the recommended path: you choose the pool size and you see the lifecycle. The DENDRUX_DATABASE_URL environment fallback (an Agent with neither state_store= nor database_url=) still works in 0.5 and warns; it owns one shared reference until reset_engine() or process exit, and it is removed in 0.6. A store created by RunStore.from_database_url owns its engine and must also be closed.

Pool sizing per worker

Each worker process has its own pools. Add pool_size + max_overflow across every process, reader, maintenance job and other application, and keep the total under the database's connection limit with headroom for migrations and operations. Choose a finite pool_timeout so overload waits briefly and fails visibly instead of building an unlimited backlog. A smaller dedicated reader pool keeps viewer traffic from competing with running agents.

Managed database idle timeouts

For non-SQLite URLs Dendrux defaults pool_pre_ping to True and pool_recycle to 1800 seconds. Pre-ping checks a connection at checkout and replaces a disconnected idle one; recycling replaces old connections at checkout. Pick a recycle interval shorter than your managed database's or proxy's idle cutoff. database_options= overrides either setting on Agent, RunStore.from_database_url, sweep and the environment engine.

Pre-ping cannot recover a transaction killed while it is executing. Dendrux retries persistence writes where that is safe; tools with external side effects still need their own idempotency. Apply database timeouts and monitor failures instead of retrying a whole run blindly. A failed database operation inside a run ends it with error_info.category == "persistence_error" and a fixed message that carries no SQL, parameters or driver text; engines Dendrux creates hide bound parameters.

PgBouncer in transaction mode

Disable both asyncpg statement caches when a transaction-mode pooler sits in front of the database. Keep runtime work transactional: Dendrux uses no session-level SET, session advisory lock, temporary table or LISTEN on the pooled path. The one advisory lock it takes, when it inserts an inbox note, is transaction-scoped and released with the transaction.

options = dict(
    pool_size=5,
    max_overflow=0,
    connect_args=dict(
        statement_cache_size=0,
        prepared_statement_cache_size=0,
    ),
)

Pass these as database_options= (or as keyword arguments to create_async_engine). Migrations are the exception; see the next section.

Migrations: one job, direct connection

Run dendrux db migrate once per deploy, as a single job, over a direct Postgres connection (not through PgBouncer, and not from every pod at startup). Run it before you start 0.5 workers. A second concurrent runner refuses with a retry message instead of racing.

• The migration uses session-level locks and lock_timeout settings, which is why it needs a direct session connection.
• Short exclusive steps wait at most 5 seconds for each lock and then fail with a retry message. Finish the long-running transaction that blocked it and run the job again. The index builds run concurrently and a timed-out build is repaired on the retry. The 5 second budget is fixed; schedule the job in a quiet window.
• The status_changed_at backfill is online, in 5,000-row commits, and works best on Postgres 12 or later. The concurrent index build on a large run_events table can take minutes.
• dendrux db check compares the database with this version's head without changing anything (exit 0 current, 1 behind, 2 unknown). A non-SQLite database that was not migrated fails at startup with SchemaOutOfDateError.
• Do not start abandoned-waiting sweeps until the migration has finished.

The revision list and rollback notes are on Upgrading to 0.5.0.

Sweep cadence

sweep() finds stale running rows and abandoned waiting rows. Call it from your own maintenance job, at startup and at an interval that suits the application; Dendrux never schedules it.

• Set stale_running beyond the longest legitimate gap between progress updates (model calls and tools). A long-running tool can call heartbeat() to report progress; progress writes only touch running rows, so a stuck tool thread cannot keep a paused or finished run alive.
• max_runs (default 10,000, ceiling 100,000) bounds how many candidates one call inspects per category, and batch_size (default 100, ceiling 1,000) sets the page size. Fresh running rows and compare-and-swap attempts that lose a race count against the bound. When a category hits its bound, Dendrux logs a structured sweep_cap_reached warning with the category, the cap and the number inspected. The return value has no "more results" flag and there is no cursor, so a short result does not prove the backlog is empty: repeat the call, or raise max_runs.
• A waiting run is abandoned only when both its last status change (status_changed_at) and updated_at are older than abandoned_waiting. The second check protects pauses written by 0.4.1 workers during a rolling deploy; a recent metadata edit delays abandonment by one window.
• Swept waiting rows get the safe error Run swept as abandoned_waiting together with failure_reason.
• Candidate scans use indexes on status and transition time. Heartbeat timestamps are not indexed, so progress writes stay eligible for Postgres HOT updates.

Live readers and bounded work

Every queue Dendrux keeps has a default and an override. Size these with your worker count:

The follower cap is per scope, not per user. A string tenant_id is one scope, all untenanted (None) followers are one scope, and all UNSCOPED followers are one scope. A single-tenant app with more than 100 live readers per process builds its own feed with max_followers_per_tenant=None; see Build your own read routes.

Followers of one run share one poller, so database load grows with watched runs, not with open connections. A follower costs a small constant number of statements however many pauses the run has, and get_pending is bounded whatever the size of the stored pause: one projected read for a client-tool or requested pause, three bounded reads (summary, active-pause sequence, recorded model arguments) for an approval or input pause. Attaching the pending pause to a result adds one projected read for a client-tool or requested pause and three for an approval or input pause, once per pause; Agent.get_result adds three reads, or five for an approval or input pause.

Event numbering is allocated inside the insert statement, so a writer never reads the event log first. Notes are capped by one conditional insert, so concurrent submit_note calls cannot exceed max_notes_per_run.

Read replicas

Read replicas are your deployment choice. Keep claims, tool submissions, cancellation, migrations and sweeps on the writer. Replica lag can delay pending state and event visibility, so a UI must not treat a replica's stale pause as authority to submit a decision. Submissions validate the current pause id on the writer through expected_pause_id. Choose where authorized reads and follow polling run, how lag is monitored, and when the app routes a read back to the writer.

Browser results

RunResult.to_dict(), Agent.get_result() and ToolResult.from_dict() are the allowlisted shapes that cross to a browser. They are described on the Agent reference, and the redaction rules for readers are on the RunStore reference.