How a running or paused dendrux run is cleanly cancelled from any process, without mid-step preemption or race bugs.

Cancellation

agent.cancel_run(run_id) terminates a run. It works whether the run is currently paused in the DB, running on another server, or executing inside a process the current Agent owns. It uses three mechanisms in parallel: an atomic CAS for paused runs, a cooperative flag for running runs anywhere, and an in-process interrupt for runs this Agent instance is executing (run() and stream()). Whichever condition applies wins, and the run ends in status cancelled.

A run on another server is never preempted: it finishes its current model call or tool and cancels at the next checkpoint. Only a run executing in the same process, on the same Agent instance, is interrupted mid-step.

Two scenarios, one API

There is one method on Agent for cancellation, and its behavior depends on what state the run is in when the call arrives:

The caller does not have to know which branch applies. cancel_run sets the cooperative flag, attempts the paused-state CAS, and fires the in-process interrupt in the same call. Exactly one path produces the terminal run.cancelled event.

Before 0.3.0 a batch run() was not interruptible by cancel_run: a model call in the air always completed first. If your code relied on that, see Interrupting run().

cancel_run returning is a receipt, not a confirmation. If the returned RunResult has a non-terminal status, meta["cancel_requested"] is True and the run is on its way to cancelled. Confirmation is the terminal signal: RUN_CANCELLED on the stream you are consuming, the run.cancelled event in the log, or status == "cancelled" from RunStore.get_run (which also exposes cancel_requested).

Cancelling a paused run

This is the deterministic path. The run is not doing anything, so we can immediately flip the status.

The example below starts a run that pauses for approval, then calls cancel_run instead of submit_approval:

import asyncio
 
from dendrux import Agent, tool
from dendrux.llm.anthropic import AnthropicProvider
 
 
@tool(target="server")
async def refund(order_id: int) -> str:
    """Refund an order by its id."""
    return f"Refunded order {order_id}"
 
 
async def main() -> None:
    async with Agent(
        name="Support",
        provider=AnthropicProvider(model="claude-haiku-4-5"),
        prompt="You are a support agent. Use the refund tool when a refund is requested.",
        tools=[refund],
        require_approval=["refund"],            # refund pauses the run for approval
        database_url="sqlite+aiosqlite:///cancel.db",
    ) as agent:
        result = await agent.run("Refund order 42 using the refund tool.")
        print(result.status.value)              # waiting_approval
 
        cancelled = await agent.cancel_run(result.run_id)
        print(cancelled.status.value)           # cancelled  (instead of approving)
 
        # idempotent: cancelling a terminal run is a no-op, never raises
        again = await agent.cancel_run(result.run_id)
        print(again.status.value)               # cancelled
 
 
asyncio.run(main())

The DB state before and after, pulled directly from SQLite, shows the single-statement transition:

first.run returned: status=waiting_approval
 
STATE AT PAUSE (waiting_approval)
  status: waiting_approval
  cancel_requested: 0
  iteration_count: 1
  pause_data: <set>
  run_events:
    seq=0  run.started
    seq=1  llm.completed
    seq=2  approval.requested
    seq=3  run.paused
 
cancel_run returned: status=cancelled
 
STATE AFTER CANCEL
  status: cancelled
  cancel_requested: 0
  iteration_count: 1
  pause_data: null
  run_events:
    seq=0  run.started
    seq=1  llm.completed
    seq=2  approval.requested
    seq=3  run.paused
    seq=4  run.cancelled

Three things worth noting in that diff.

1. status went straight from waiting_approval to cancelled without an intermediate running state. One SQL statement made the transition.
2. pause_data was cleared to JSON null as part of the same UPDATE. There is no stale handoff data sitting on a terminal row.
3. A run.cancelled event was appended at seq=4, continuing the existing per-run monotonic sequence. SSE clients that were streaming this run see the cancel arrive as a normal event.

Cancelling a running run

When the run is actively iterating, cancel cannot finalize atomically without fighting the runner. Instead, cancel_run sets the cancel_requested flag and returns. The runner itself is responsible for seeing the flag and stopping.

The runner observes the durable flag at two checkpoints:

1. Top of each iteration. Before building messages or calling the LLM, the loop reads is_cancel_requested(run_id). If true, the loop returns with status CANCELLED and the runner finalizes.
2. Pre-pause. If the iteration body decides to pause (a client tool needs running, approval is required, clarification is needed), the runner checks the flag one more time before writing pause_data. This prevents a cancel that arrived mid-iteration from settling the run into a paused state with stale pause_data.

For a run that is executing somewhere this Agent instance cannot reach, tool executions are not preempted. A tool already running finishes normally, its result is recorded, and the cancel is honored at the next checkpoint. Aborting a tool mid-execution can leak side effects, and waiting one boundary is cheaper than cleaning that up. In-process interrupts (below) are the deliberate exception: you opt into them by calling cancel_run on the instance that owns the run.

Interrupting run()

When run() starts, the runner registers the task it executes on with the owning Agent (right after the run row is created) and unregisters it when run() ends. cancel_run(run_id) on the same Agent instance, in the same process, then cancels that task in addition to setting the durable flag, so an in-flight provider call is dropped instead of being waited for.

The contract:

• run() returns RunResult(status=CANCELLED) with meta["interrupted"] = True and meta["partial_output"] = False. A batch call has no partial text, so answer is None.
• The row ends cancelled, and run.cancelled carries {"reason": "cancel_requested", "interrupted": true, "partial_output": false}.
• In-flight tools are interrupted with CancelledError, like any asyncio task cancellation. A tool that must not be abandoned halfway should shield its critical section (asyncio.shield) or be written so that running it again is safe.
• If the cancel arrives before registration, after the provider's last event, or after the run finished, the durable flag decides: see Which wins.
• Another Agent instance, or another process, only sets the durable flag. It does not interrupt this run.
• Usage for the abandoned call is not recorded (providers report tokens only at the end).

cancel_run with cascade=True cancels delegated children too; if a child is running in this process, it is interrupted the same way.

task.cancel() versus cancel_run

Cancelling the task that awaits run() yourself, including via asyncio.wait_for(agent.run(...), timeout), is a different thing from cancel_run(). The runner catches CancelledError, atomically transitions a persisted running row to cancelled, emits run.cancelled with reason task_cancelled if that transition wins, closes run lifecycle hooks, and re-raises. wait_for exposes this as TimeoutError. Startup hooks after run creation and error handling are covered too. The runner waits for the cancellation status/event writes even if another cancellation arrives during cleanup. Each recorder/notifier receives at most one closing hook; callbacks stay in the original run task. Already paused or terminal rows are preserved. No RunResult is returned and in-flight model text is not captured. Unlike cooperative cancel_run() on another instance, cancelling an asyncio task can interrupt awaited tool execution.

The runner tells the two apart: a cancellation it caused through cancel_run is absorbed, while one your own code caused is re-raised so structured concurrency keeps working.

examples/33_cost_cancel_idempotency.py demonstrates this live: a wait_for timeout on run() leaves the row cancelled with a single run.cancelled event, and the same script covers stream idempotency and the final-turn answer contract.

Interrupting a streamed run

A chat turn is usually one iteration, so "observed at the next iteration" would make cancel_run useless for exactly the runs people most want to interrupt. Streamed runs therefore get an in-process interrupt.

When stream() or resume_stream() starts driving the loop, the runner registers an asyncio.Event for that run on the owning Agent. cancel_run on the same instance sets it. Inside the loop, every pull from the provider stream is raced against that event, so an interrupt is honoured while the model is still silent, not just when the next token arrives. On interrupt the loop:

1. Closes the provider stream (the HTTP connection is dropped; the vendor stops generating).
2. Reports the abandoned call through on_llm_call_failed with a StreamInterruptedError, so notifiers can close the span.
3. Yields RUN_CANCELLED whose run_result.answer is the current turn’s partial text, with meta["interrupted"] = True and meta["partial_output"] saying whether answer carries anything.

The runner persists that outcome exactly like any other terminal result: status cancelled, answer set to the partial text, and a run.cancelled event carrying {"reason": "cancel_requested", "interrupted": true, "partial_output": true}.

async with agent.stream("Write me a long essay") as stream:
    consumer = asyncio.create_task(consume(stream))     # your code reads events
 
    ...
    await agent.cancel_run(stream.run_id)               # from anywhere in this process
    events = await consumer                             # ends with RUN_CANCELLED
    partial = events[-1].run_result.answer              # partial text from the current turn

The interrupt lives on the Agent instance, so it only reaches streams that instance is driving. cancel_run from another process (or another Agent object) still sets the durable flag; that run finishes its current model call and cancels at the next checkpoint. If you need cross-process mid-stream interruption, call cancel_run on the instance that owns the stream, for example by routing the request to the server that holds the SSE connection.

The abandoned model call reports no usage (providers send token counts only at the end of a stream), so usage on the cancelled result covers completed iterations only. The interrupted call is not stored as an llm_interactions row; the partial answer on the run row and the run.cancelled payload are the audit trail.

The partial answer is not the full transcript. The buffer resets at each TOOL_RESULT, so earlier assistant commentary is excluded and the answer may be None immediately after a tool result. On success, answer likewise contains only the final answer turn. Accumulate stream deltas across turns for the full live transcript, or use RunStore.get_traces() for recorded assistant messages.

Abandoning a stream

Closing the RunStream (aclose(), leaving async with, or break followed by garbage collection) is the other way a streamed run ends early. The runner's cleanup finalizes the row as cancelled and persists the same partial answer, with run.cancelled carrying {"reason": "stream_closed", "partial_output": ...}. The loop's generator is closed deterministically in the same task, so provider connections and telemetry context are released before aclose() returns.

Prefer cancel_run when the interruption is a product action (a user hit stop, another agent took the floor): it goes through the loop, closes the LLM lifecycle pair, and gives the consumer a terminal RUN_CANCELLED event to react to. Closing the stream is for when the consumer itself is going away.

Cascade

Delegated child runs have their own rows and their own cancel flags. cancel_run(run_id, cascade=True) cancels the whole subtree:

result = await agent.cancel_run(run_id, cascade=True)
print(result.meta["cascade"])
# [{'run_id': 'child-2', 'outcome': 'cancelled'},
#  {'run_id': 'child-1', 'outcome': 'cancel_requested'},
#  {'run_id': 'root',    'outcome': 'cancelled'}]

• It works on a point-in-time snapshot of the descendants, deepest first with run_id last. A child created after the snapshot is not included.
• Each run's outcome is listed in meta["cascade"]: cancelled (finalized now), cancel_requested (it is running; it stops at its next checkpoint, or is interrupted at once if it is running in this process) or terminal (already finished).
• Paused descendants (waiting_* or paused) are finalized in the same call.
• cascade=False (the default) cancels only run_id; its children keep running.
• tenant_id= scopes the call: the root must belong to exactly that tenant (or RunNotFoundError), and the cascade only reaches descendants of the same tenant.
• Stores written for earlier releases that do not implement get_subtree raise NotImplementedError naming the missing method when cascade=True is used.

External callers only set flags and finalize paused rows. The run.cancelled event of a running descendant is written by that run's own runner when it observes the flag.

Which wins: completion or cancellation

Exactly one terminal status is ever written, because every finalize is an atomic compare-and-swap. The rule for what that status is:

• Cancel arrives before the provider's next event (including while the model is silent): cancellation wins. The result is cancelled with a partial answer.
• Cancel arrives after the provider's final event has been received (the loop is recording the completed call, or executing a tool): the current iteration finishes. If it produces the final answer, the run is success and cancel_requested is cleared. If it pauses or continues, cancellation is honoured at the pre-pause or top-of-iteration checkpoint.
• Cancel arrives after the run's last checkpoint: it does not stop a run that then completes successfully. If the final answer is already being produced, the run finishes as success and the request is cleared; the work and its side effects happened, so reporting cancelled would be wrong.
• Cancel arrives after the run is terminal: no-op.

In other words, a model response that is fully in hand is never thrown away. Partial output is preserved only when there was genuinely more to come.

Why the flag is race-safe

A naive implementation of "is it still cancellable?" would read the status, check it is not terminal, then write cancel_requested = true. That introduces a window where the run can finish between the read and the write.

Dendrux folds both into one SQL statement:

UPDATE agent_runs
   SET cancel_requested = true,
       updated_at = now()
 WHERE id = :run_id
   AND status NOT IN ('success', 'error', 'cancelled', 'max_iterations');

The terminal-status guard is part of the WHERE clause. If the run has already finished, rowcount is 0, and the flag is never set. If the run finishes between the SELECT-ish part of the UPDATE and the commit, the UPDATE simply does nothing, because the row's status no longer matches.

The paired CAS finalize does the same trick on the other end:

UPDATE agent_runs
   SET status = 'cancelled',
       pause_data = NULL,
       cancel_requested = false,
       iteration_count = :n,
       updated_at = now()
 WHERE id = :run_id
   AND status IN ('waiting_approval', 'waiting_client_tool', 'waiting_human_input', 'paused');

If the run is running (not paused), this matches 0 rows and has no effect. The cooperative flag picks up the work. If the run is paused, this finalizes in one round-trip. The caller gets a boolean back that tells them whether this particular CAS "won" or not, and exactly one of the two paths produces the run.cancelled event so the timeline never has a duplicate.

Why cooperative instead of preemptive

The alternative to a flag + checkpoints is to cancel the asyncio task running the loop. Python can raise CancelledError into the coroutine, and the runner would stop immediately.

Dendrux does use in-process signals where it owns a handle: cancel_run cancels a locally running run() or submit_* task and sets the interrupt for a locally streamed run. But those are additive, not primary. The cooperative flag is the source of truth, for three reasons:

1. Cross-process cancel works at all. A preemptive asyncio cancel cannot reach a run executing on another server. A flag on the DB row can.
2. Tool executions are respected. A CancelledError raised inside a tool orphans whatever the tool was doing. The flag waits until the tool returns. (An in-process interrupt trades that away on purpose; see Interrupting run().)
3. Bounded observation points. The loop observes cancellation at the top of an iteration, before pausing, and between provider stream events. Nothing else observes it, which keeps the loop's invariants easy to hold.

The in-process stream interrupt is the one place Dendrux does preempt a provider call, and it does so at a boundary the loop controls: between two stream events, after closing the provider stream, with the partial output captured. That is very different from throwing CancelledError into arbitrary code.

Other ways a run ends as cancelled

• Stale sweep. sweep(stale_running=...) finalizes a stale running row that has a cancel pending as cancelled instead of error (the request is honored; failure_reason still records the stale classification).
• Budget. A budget with on_exceed="cancel" ends the run cancelled with failure_reason="budget_exceeded" and a run.cancelled event whose reason is budget_exceeded. It never reads as a user cancellation.

Undelivered inbox notes on a cancelled run are listed in meta["undelivered_notes"] of the result.

The result is a cancel that is deterministic for paused runs, immediate for runs this process owns, observable-within-one-iteration for everything else, and idempotent for terminal ones. One call, regardless of where the run actually lives.