Migrate, roll every worker and reader, activate format 2, then enable recovery and live updates. Includes every changed default and rollback rules.

Upgrading to 0.6.0

0.6.0 adds explicit recovery after a worker stops, durable execution checkpoints, optional live updates for follow (Postgres or Redis hints), a batched follow reader and trace continuity across processes. Applications own their workers, maintenance jobs and read routes.

Activation is one-way. Take a database backup before activation. Once activation starts, rollback means restoring that backup. There is no deactivation or CLI downgrade command. Do not let a 0.5 worker or reader access an activated database.

New PostgreSQL databases come up active at format 2 after dendrux db migrate. Here, new means there is no alembic_version table and none of Dendrux's tables before migration; unrelated application tables are allowed. Existing databases need the activate step below, even with zero runs or when already at the current revision.

Provider SDKs. Provider extras now require anthropic>=0.77.0,<1.0 and openai>=1.97.2,<3.0. Older SDKs lack the required Responses API stream handling and Anthropic thinking/structured-output support. If you pin older SDKs, update those pins and reinstall the matching extras before upgrading to 0.6.0.

Upgrade in order

1. Migrate. Install dendrux[postgres]==0.6.0 in the migration job environment. Back up the database, then run dendrux db migrate once over a direct Postgres connection, before starting 0.6 workers. From the 0.5 head e8f9a0b1c2d3, revision f9a0b1c2d3e4 adds recovery columns and dendrux_settings. It leaves the global storage format at 1 and does not build the new unique indexes. Older workers can keep working during this additive step. If upgrading from an earlier release, complete the 0.5 upgrade first.
2. Roll every worker and reader to 0.6. Include read-only services, scripts, scheduled jobs and dashboards built on RunStore. Keep recovery and live updates off during the roll. Apply the default changes below in your application. Check the schema with dendrux db check and inspect dendrux db status; migration alone does not activate format 2.
3. Back up again, then activate. Confirm that no old worker or reader remains. Take the pre-activation backup, then run dendrux db activate once over a direct Postgres connection. The command checks the schema, removes identical duplicate rows, builds three unique indexes and sets the global format to 2. It reads the whole react_traces, tool_calls and run_events tables, builds the indexes concurrently on Postgres and drops the old indexes; run it in a quiet period and check free disk for the builds first. It refuses conflicting duplicates rather than choosing between different contents. Progress is durable; fix the reported condition and rerun the command after a failure. dendrux db status should show storage_format=2 activate_step=done.
4. Enable recovery and live updates explicitly. Call sweep(on_stale="reclaim", ...) from your own maintenance job and dispatch the returned items to agent.reclaim(..., tenant_id=..., item=...). Use a stale_running threshold at least three times Agent.lease_interval. Live updates are optional and off by default; follow polls without them. To turn them on, build the store yourself with live_updates=True (Postgres) or live_updates=RedisUpdates(...) (any store; pip install "dendrux[redis]") on every writer and reader, wrap the reader's store in RunStore(store), and close the store with await store.close() at shutdown. They cannot be set through Agent(database_url=...). See Recovery and Live updates.

The CLI reads the database URL from DENDRUX_DATABASE_URL; keep using your deployment's credential injection for that variable.

dendrux db migrate
dendrux db check
dendrux db status
# Roll every worker and reader, then take the pre-activation backup.
dendrux db activate
dendrux db status

An unstamped database with the complete expected schema can activate too. Use dendrux db check --allow-unstamped to check its structure; a stamped database must be at the current migration head. Do not stamp an incomplete schema to make the check pass.

SQLite files

SQLite adds the new columns on first use, but an existing file stays at format 1 until explicit activation. Stop every other process using the file before dendrux db activate. The command takes an exclusive lock and refuses a live competing transaction. That lock cannot prove that no idle process will write later. Back up the file before activation and restart only 0.6 processes afterwards.

A fresh database created by Dendrux's create_all path starts active at format 2. New PostgreSQL databases also start active after dendrux db migrate. Existing PostgreSQL databases and migrated SQLite files stay at format 1 until explicit activation.

A run keeps its storage format

agent_runs.storage_format is chosen when the run is created. Activation changes new runs only. A format-1 run stays format 1 across every pause and resume; there is no conversion job.

Format-2 runs store conversation history in traces and execution state in a bounded checkpoint. Pause data uses schema_version=2 and omits the growing history and step list. Readers must understand seed, turn, rewrite and state traces. RunStore.get_traces and dendrux traces expose the run's own user, assistant and tool rows in their original order, with dense positions and paging over those rows only. System prompts, seeded conversation prefixes, rewrites and state rows remain internal. Compaction does not remove earlier public audit rows. They return the provider form, including PII placeholders; the reverse mapping remains a separate trusted read.

reclaim() refuses format-1 runs with RunNotReclaimableError(reason="legacy_run"). Under sweep(on_stale="reclaim"), stale legacy runs end as error with failure_reason="stale_running_legacy". They appear in SweepResult.stale_running, not reclaimable.

Every default change

The store used for the call decides whether strict tenancy applies, even if the Agent itself has no store. A delegated child can inherit scope when its outer caller deliberately chose a tenant, including explicit tenant_id=None, or the parent already resolved scope under strict rules. A stored child under a parent whose tenant was merely omitted raises TenantScopeRequiredError; pass tenant_id on the outer run.

Default local database location

When DENDRUX_DATABASE_URL is unset, runtime database helpers, every CLI command and migrations use ./dendrux.db in the current working directory. The runtime helper previously used ~/.dendrux/dendrux.db. This does not enable implicit persistence on Agent; continue passing state_store= or database_url=.

If the local file is absent and the old file exists, database resolution logs local_database_default_changed. No data is moved automatically. To retain the old database, set an explicit absolute URL before running commands or migrations:

export DENDRUX_DATABASE_URL="sqlite+aiosqlite:///$HOME/.dendrux/dendrux.db"

Use the same working directory or explicit URL for your application and CLI. Back up the existing file before upgrading.

Other compatibility changes

Until 1.0, minor releases can change behaviour directly. These changes have no warning cycle.

Store classes: SQLAlchemyStateStore is removed

SQL persistence is now one base class and three concrete stores, all importable from dendrux.store, and each takes an optional live_updates= argument. SQLAlchemyStateStore is removed in 0.6.0 with no alias: importing it raises ImportError and names the replacements. Applications that only use Agent(database_url=), RunStore or the CLI need no change.

StoreCapabilities gains a deployment field and the StateStore protocol gains update_run_meta. A store that declares its capabilities without deployment still runs, with a warning: add "single_process" for in-memory SQLite, "single_host" for a SQLite file shared by processes, or "multi_server" for a database server. The field is descriptive and changes no runtime behaviour.

MySQL 8.4 and MariaDB 10.11 are new in 0.6.0 and need no migration from 0.5, since 0.5 never ran on them. See MySQL and MariaDB.

Database retries

Built-in SQL stores retry recognized database failures before COMMIT at their transaction boundaries. Each failed attempt rolls back before a retry. Raw driver retries apply only inside these guarded SQL scopes.

Eligibility requires a SQLAlchemy or typed persistence error, or native-driver provenance, plus transient classification. Provenance identifies a driver execution path. Application callbacks inside a driver and SQLAlchemy-wrapped serializer errors can qualify. SQLAlchemy pool timeouts retain their retry policy.

A driver or OS failure during COMMIT, including a bare timeout, can leave the outcome unclear. A write with a stable identity reads back its rows. Proven success returns the result. Safe, proven absence permits a retry. Otherwise the store raises non-retryable StoreOutcomeUnknown. Claims, reclaims, submit_and_claim and releases never infer ownership from a later read. PostgreSQL readback waits for a concurrent writer for at most about five seconds. Do not replay an unknown outcome blindly. See Errors.

At most three retries wait 0.1, 0.2 and 0.4 seconds. The 0.7 seconds covers backoff only; driver operations add their own duration. A short refused-connection window can be bridged. Longer outages can exhaust the budget. A failed write makes agent.run raise PersistenceError; a stream ends with run_error. After work starts, the run is saved as error if the database answers again in time. Otherwise it keeps its last committed state. Unconsumed continuations can instead release back to their pause; see Continuation setup failures. Restart terminal-error runs as appropriate. Use sweep and reclaim for runs left running. Each retry logs Transient DB failure n/3 with the operation and run id, without SQL or driver text. This does not promise every reader or sweep survives a restart.

Recorder and runner scopes keep their previous SQLAlchemy and typed-persistence retry policy. They do not newly retry raw driver failures from custom stores. A custom store must resolve an ambiguous COMMIT itself. Do not report it as retryable without proof or a safe stable-identity protocol. Wrapping a landed commit failure in SQLAlchemy OperationalError can still cause replay under the existing policy.

Normalized SQL errors discard internal traceback frames that could retain driver text. Direct PostgresUpdates send, receive and sender-close failures use plain builtin exceptions with fixed text and no raw exception chain. TLS failures keep the tls kind. last_error_type keeps a validated original class name, rather than the original exception class. Native driver exception handlers therefore do not catch these normalized direct-hook failures. Missing drivers retain the installation hint for dendrux[postgres] or dendrux[mysql].

Continuation setup failures

In 0.5.0, execution-setup failures such as a failed trace-index read ended resume, resume_stream, submit_tool_results, submit_input, submit_approval and submit_resume as error when failure persistence succeeded. Earlier owned-pause validation failures could already release the claim. In 0.6.0, continuations (resume, resume_stream, submit_* and reclaim) can release their exact claim when an ordinary exception occurs before submission consumption or work starts. This includes failed setup reads and writes such as run.resumed, save_trace and consume_submission, provided consumption has not committed. StoreOutcomeUnknown during execution setup does not release. Cancellation keeps its existing precedence.

A released continuation returns to its pause. Reclaim releases to running, for a later lease expiry and sweep. No automatic timeout or retry ends a released pause: resubmit the original input/results/decisions explicitly, or call cancel_run if you give up. Permanent setup failures will fail again until fixed. Monitor exceptions, run_error stream events and run.resume_aborted events; a monitor counting only rows with status=error will miss these attempts.

For a confirmed release to a non-running status, started observers receive on_run_finished with that observed status. A released reclaim still reports failure hooks because its row remains running. Failure hooks also run when release is unavailable, fails, or its status cannot be read. For a claimed pause, a store without a release method or an integer claim epoch falls back to terminal error persistence. Reclaim remains running and retryable after lease expiry even without a release method. An attempted release that fails leaves the last durable state for sweep/reclaim; stores without recovery use sweep(on_stale="error").

resume_stream still emits run_error for the failed attempt, but its run_result.status (and stream.result.status) reflects the observed row after release, which can be a pause or running. If the cleanup status read fails, running is the conservative fallback, not proof of current ownership. Read the store before retrying. The error event keeps the failure information and does not supply a pending payload or pause token.

After an outcome commits, an optional projection failure cannot hide it. A pause result may have pending=None and pause_id=None; reload with RunStore.get_pending before submitting, and handle a pause that has since changed. recovery=None can mean its projection failed on a committed-outcome or cancel_run response; result_read_failed logs that case. get_result remains a strict read and raises if recovery or pending cannot be read.

Cancellation during create or claim

With built-in SQL stores, cancelling a continuation before its claim issues COMMIT leaves the original pause intact. The same request can be retried. Cancelling a new run before its create issues COMMIT leaves no new run. From COMMIT onward, cleanup uses the acquired epoch to end the run as cancelled, including cancellation during governance setup. It does not release that claim back to the pause. An ambiguous outcome or failed cleanup can leave the last committed state for recovery.

Create, claim and reclaim store calls run in a joined child task. A custom store sees the caller's inherited ContextVar values, but writes to those variables do not reach the caller. Check task-scoped session handling too. Provider iteration and lifecycle callbacks stay in the caller's task.

The private handoff deadline is about five seconds. It covers only the create or claim handoff, relies on store and driver cancellation, and can be exceeded by a network black hole at COMMIT. Later governance and provider cleanup are outside it. See Cancelling while a run is created or claimed for recovery and legacy-store limits.

Paused runs with a token budget

count_cache_tokens affects the governance digest when a budget has max_tokens. A run paused under 0.5's default count can raise GovernanceMismatchError when resumed under 0.6's cache-inclusive default and default on_governance_change="raise".

Drain those runs before upgrading, or explicitly keep Budget(..., count_cache_tokens=False) for the upgrade window. If you intend to change the accounting while continuing existing runs, use Agent(..., on_governance_change="warn") while they drain and review the token limit. An existing token budget that already chose count_cache_tokens=True keeps that choice. Budget amounts remain outside the digest so raising a paused run's limit is still allowed.

Bounded reads

Four RunStore list reads and get_subtree can return as much as a run holds. 0.6.0 keeps them working and warns, so you can fix your calls before 0.7 changes the defaults.

What to do:

• Pass limit= (1 to 1,000) on every list read. Page events with after_sequence_index= (the last sequence_index you got) and the other three with offset= (plus the rows you got). A full-history export loops until a page comes back empty; the loops are on the RunStore reference.
• Pass max_nodes= to get_subtree and handle SubtreeTooLargeError, raised when the descendants (root excluded) reach the bound. It bounds the node count, not memory.
• get_traces pages the public audit rows directly. Reconstruction limits do not apply to public reads. During execution, rows beyond the first unfinished tool slot are withheld so returned positions stay stable; an empty page can mean that work is still pending. Once the run reaches success, error, cancelled or max_iterations, all committed public rows are visible even if unfinished reservations remain.
• Find the calls that still warn with python -W error::FutureWarning over your tests. The sweep() warning about omitting on_stale= is separate.
• Zero and negative limits now read one row on every SQL path, including the format-2 get_traces, which used to return an empty list.

Close the reader before you dispose its engine, in async with or a finally. Closing the last follower leaves the shared coordinator running until it idles out after about 60 seconds or the reader closes; disposing the engine first can make its next read fail with a pool error and log follow_read_failed. See Scaling workers.

Other changes to check

• agent.reclaim() always requires an explicit tenant_id, including when strict_tenancy=False. Prefer the tenant_id on the returned sweep item. Explicit None selects untenanted runs; trusted cross-tenant code can pass UNSCOPED.
• Custom state stores declare capabilities. An undeclared store keeps the old signatures and format 1. Subclass overrides must accept and forward the new keyword parameters or narrow their declaration. A mismatch raises StoreCapabilityError at Agent construction. See the capability matrix.
• Format-2 provider options must be JSON-only and fit 2 KiB. An assistant batch has at most 64 pending calls by default (Limits.max_pending_calls). Checkpoints fit 32 KiB; history rebuild defaults to 16 MiB and 20,000 rows. Over-limit history is refused rather than truncated. A stored conversation rewrite also has hard caps of 999 messages and 1 MiB of UTF-8 JSON. Exceeding either raises PersistenceError(code="checkpoint_too_large") before provider dispatch. Raising checkpoint or rebuild limits does not raise these rewrite caps.
• Every approved tool execution on a format-2 run adds one checkpoint before dispatch. A decision authorizes one attempt; a second crash may require approval again. Mark a tool idempotent=True only when repeating its external effect is safe.
• Dendrux reports only a fixed list of provider error codes. A valid but unmapped vendor or custom-provider code maps to provider_unknown. A malformed code becomes None and is discarded from provider_code_raw too. Package-issued codes retain their meaning. Do not export provider_code_raw to clients or telemetry.
• A non-streaming socket read timeout reports timeout_reason=None. call_deadline means Dendrux's own deadline, not a socket timeout.
• Non-streaming provider responses have a 16 MiB byte limit before SDK parsing. Exceeding it raises non-retryable ProviderResponseError(provider_code="response_size_limit"). Set max_response_bytes= on the provider constructor to override the limit.
• max_stream_bytes also limits one decoded SSE event, in addition to the cumulative retained stream budget. A low custom limit may now reject a final event that repeats a response. Compressed responses accept gzip or deflate; unsupported or stacked encodings raise response_encoding_unsupported.
• The byte limits count decoded body or SSE event bytes, including any bytes after a compressed end marker. Compressed wire bytes are also bounded, with a small overhead allowance of the greater of 1 KiB or 1% of the configured limit, rounded down to whole bytes. Streaming wire accounting resets at decoded SSE event boundaries. A compressed response with heavy padding or frequent compression flushes can be refused even when its decoded bytes fit the limit.
• Cascade cancellation stays inside the root run's tenant even when called unscoped. Traversal is capped by Budget.max_subtree_nodes (default 10,000); cascade_truncated means remaining branches need attention. Finished runs keep their cancel_requested value. On SQLite, delegated-child creation briefly acquires the write lock before checking the parent.
• Live updates are off by default and belong to the store: live_updates=True (Postgres only) or an instance such as PostgresUpdates(...) or RedisUpdates(...). Only RunStore(store) listens; RunStore.from_engine and RunStore.from_database_url keep polling. Agent(database_url=..., database_options=...), sweep(database_url=...), get_engine and RunStore.from_database_url refuse a live_updates key with a ValueError. A reader built with from_engine or from_database_url polls without warning, so use RunStore(store) to listen. A hint is a lossy wake-up, never data: the database stays the truth. See Live updates for the transports, the failure behaviour and the alert codes.
• Followers are served differently. One coordinator task per RunStore and event loop reads all watched runs in batches (up to 200 runs and 2,000 rows per query), using a token bucket that holds up to 20 reads and refills at 20 reads per second, so a full bucket can allow about 40 reads within a second; 0.5 ran one poller per watched run. The ordering, heartbeats, caps and redaction of follow are unchanged. With healthy live updates a safety poll runs every 10 seconds; without them an idle run is read every 0.5 seconds, backing off to 2 seconds. stream_events keeps its own per-caller polling; prefer follow.
• PostgresUpdates uses connections outside your pool: one sender per store that announces and one listener per store whose readers follow. Count them against the database's connection limit.
• Resumes, submissions and reclaim use the stored trace link to stay in one trace when an OpenTelemetry notifier is supplied. In a trace, a failed tool's stack shows where Dendrux reported the failure, rather than where the tool itself crashed.
• Structured logs carry stable codes. Strict field checking is opt-in with DENDRUX_STRICT_LOG_FIELDS=1; normal operation drops invalid fields. See Log codes.

Storage-format compatibility

Fresh and activated databases create format-2 runs. dendrux db migrate activates new PostgreSQL databases automatically; existing PostgreSQL databases still need dendrux db activate after every worker and reader is upgraded. Existing format-1 runs retain their format. Human-input pauses expose the question and suggested options through RunResult.pending, RunResult.to_dict(), Agent.get_result() and RunStore.get_pending() in both formats. Re-pausing replaces the question and options with those for the new pause.

Format 2 saves a bounded display copy in pause_data.clarification, without a schema migration. Limits(max_pause_summary_bytes=65536) sets the UTF-8 JSON byte budget (without ASCII escaping); the supported range is 128 bytes to 512 KiB. The question has priority, followed by options in order. Excess text ends with [truncated]; options that do not fit are omitted. The pause still succeeds and emits the pause_summary_truncated log code, with no question or option text in the log. Only when the question itself is cut is the clarification turn used for subsequent model calls and recovery also bounded and its duplicated raw-response step field omitted. Omitting excess options preserves the original model response. This display bound is excluded from governance: raising it does not invalidate an existing pause, and reclaim uses the configured bound. The original LLM interaction record is retained. This prevents a long question alone from making the pause unresumable. Saved questions follow the same model-text and PII rules as other saved model output.

RunStore.get_traces() returns the run's own user, assistant and tool rows in both formats. It excludes system prompts, seeded conversation prefixes and recovery state. order_index is the zero-based position among these rows, and limit and offset count only these rows. Retry runs show their new turns, not the predecessor's seed. Compaction does not remove earlier audit rows. The system prompt remains stored for internal recovery and reconstruction. Public SQL reads push filtering, offset and limit into the database and load only the requested page. They have no recovery row or byte limit. Rows after the first unfinished tool slot remain hidden until its observation is committed, keeping offset cursors stable.

These format-2 differences are intentional:

RunResult also has an optional recovery attribute (normally None), and ToolCall has an optional raw_length attribute for malformed arguments. Their dataclass fields and representations have grown; the wire result omits recovery when none occurred. RunAlreadyTerminalError now also inherits from RunControlError and remains a RuntimeError. Avoid parsing exception or dataclass representations as a protocol.

Rollback

Before activation starts, the migration is additive and old workers can still use format-1 data. Alembic can downgrade that revision only while activation has never started. The migration refuses downgrade as soon as activate_step exists, even if activation did not finish.

The optional dendrux db dedupe command also writes that marker. Back up before using it and treat the same downgrade boundary as crossed.

After that point, stop the application and restore the pre-activation backup before deploying an older version. Rehearse backup and restore on a copy of your data before the production upgrade.