Log codes and what to do
Generated by
packages/python/tools/gen_log_codes.py. Edit the registry, then regenerate.
Coded diagnostics use log_code. Lines emitted by the helper carry
code and kind, plus run_id and tenant_id when set.
Other fields are declared per code. Identifiers are validated; text uses closed enums.
With DENDRUX_STRICT_LOG_FIELDS=1, invalid fields raise. Otherwise they are dropped
with one log_field_dropped diagnostic. Unregistered code objects always raise.
Drop diagnostics contain only a count and validated identifiers, never rejected values.
Vendor messages and payloads are never log fields. Custom provider names log as custom.
Provider codes use closed per-provider tables.
Present unknown codes become provider_unknown.
Missing or invalid codes remain unset. Category and HTTP status remain independent.
Governance
| Code | Level | Fields | Fires when | Do |
|---|---|---|---|---|
governance_changed | WARNING | code, kind, run_id, tenant_id | Continuation detects changed governance in warning mode. | Review the change and use strict policy enforcement. |
governance_legacy | INFO | code, kind, run_id, tenant_id | A legacy run lacks governance pins. | Review the policy before continuing it. |
governance_mismatch | WARNING | code, kind, run_id, tenant_id | A stored governance policy cannot be continued. | Restore the original policy or start a new run. |
metadata_too_large | WARNING | bytes, code, entry, kind, limit, run_id, tenant_id | Metadata exceeds its byte bound. | Reduce metadata or raise the bound. |
reserved_metadata_key | WARNING | code, entry, kind, run_id, tenant_id | User metadata contains a reserved key. | Remove reserved metadata keys. |
tenant_scope_required | WARNING | code, entry, kind, run_id, tenant_id | A strict tenancy entry has no explicit tenant scope. | Pass the intended tenant scope. |
Live Updates
| Code | Level | Fields | Fires when | Do |
|---|---|---|---|---|
follow_coordinator_restart | WARNING | attempt, code, kind, run_id, tenant_id | A follower restarts a stopped coordinator. | Inspect store connectivity and coordinator failures. |
follow_read_failed | WARNING | code, count, error_kind, kind, run_id, tenant_id | A coordinator chunk read fails. | Check store connectivity; other chunks continue. |
live_updates_bad_message | WARNING | code, count, kind, run_id, tenant_id | A received message fails bounded wire validation. | Check channel isolation and sender limits. |
live_updates_close_timeout | WARNING | code, kind, run_id, tenant_id, timeout_s | Resource shutdown exhausts its deadline. | Check cancellation cooperation in the transport. |
live_updates_degraded | WARNING | code, error_kind, error_type, kind, run_id, tenant_id | The listener loses its subscription; readers poll. | Check bus connectivity; alert after one minute. |
live_updates_many_stores | WARNING | code, count, database_key, kind, run_id, tenant_id | Many open stores announce to one database. | Share a store within each process and close unused stores. |
live_updates_missed_hint | WARNING | code, count, kind, run_id, tenant_id | A healthy safety poll finds events no hint announced (first discovery per coordinator and generation; heuristic). | Check that writers and readers use the same bus channel. |
live_updates_overflow | WARNING | code, count, kind, limit, run_id, tenant_id | Pending announcements exceed their bound. | Share stores or raise the pending bound; readers recover by polling. |
live_updates_oversize | WARNING | code, count, kind, limit, run_id, tenant_id | A live-update hint exceeded the message byte limit and was dropped. | Use channel isolation to exclude foreign publishers; for local hints, increase max_message_bytes within the transport limit, or use shorter run ids. |
live_updates_recovered | INFO | code, kind, run_id, tenant_id | The listener confirms its subscription. | Confirm reader latency returns to normal. |
live_updates_send_failed | WARNING | code, count, error_kind, error_type, kind, run_id, tenant_id | An announcement fails and sending enters backoff. | Check bus connectivity; the database commit is unaffected. |
live_updates_send_recovered | INFO | code, kind, run_id, tenant_id | The first announcement succeeds after a send failure. | Confirm pending announcements drain. |
live_updates_tls_unknown | WARNING | code, kind, run_id, tenant_id | The live-update connection has no explicit TLS settings for a remote host. | Pass explicit connection TLS settings. |
live_updates_unavailable | WARNING | code, error_kind, error_type, kind, run_id, tenant_id | The listener cannot become ready. | Check the live-update transport; polling remains available. |
live_updates_wrong_loop | WARNING | code, count, kind, run_id, tenant_id | An instance is used from another event loop. | Use one instance per store and event loop. |
MCP
| Code | Level | Fields | Fires when | Do |
|---|---|---|---|---|
invalid_schema | WARNING | code, kind, run_id, tenant_id | An MCP tool has an invalid schema. | Fix the upstream tool schema. |
mcp_cleanup_failed | WARNING | code, count, kind, run_id, tenant_id | An MCP connection cleanup operation fails. | Inspect transport cleanup and repeated failure counts. |
mcp_credential_resolution_failed | DEBUG | code, count, kind, run_id, tenant_id | MCP credential resolution fails. | Check the credential provider configuration. |
mcp_discovery_byte_cap | WARNING | code, kind, pages, run_id, tenant_id | MCP discovery exceeds the byte cap. | Inspect the upstream server or adjust the discovery bound. |
mcp_discovery_cursor_length | WARNING | code, kind, pages, run_id, tenant_id | MCP discovery exceeds the cursor bound. | Inspect the upstream server or adjust the discovery bound. |
mcp_discovery_deadline | WARNING | code, kind, pages, run_id, tenant_id | MCP discovery exceeds its deadline. | Inspect the upstream server or adjust the discovery bound. |
mcp_discovery_page_cap | WARNING | code, kind, pages, run_id, tenant_id | MCP discovery exceeds the page cap. | Inspect the upstream server or adjust the discovery bound. |
mcp_discovery_repeated_cursor | WARNING | code, kind, pages, run_id, tenant_id | MCP discovery repeats its cursor. | Inspect the upstream server or adjust the discovery bound. |
mcp_discovery_tool_cap | WARNING | code, kind, pages, run_id, tenant_id | MCP discovery exceeds the tool cap. | Inspect the upstream server or adjust the discovery bound. |
mcp_oauth_close_failed | DEBUG | code, count, kind, run_id, tenant_id | Closing an OAuth response or transport fails. | Inspect the OAuth transport lifecycle. |
mcp_observer_dispose_failed | DEBUG | code, count, kind, run_id, tenant_id | Disposing an MCP observer awaitable fails. | Use synchronous observers that return no awaitable. |
mcp_transport_cleanup_failed | DEBUG | code, count, kind, run_id, tenant_id | Cleanup after an MCP connect failure also fails. | Inspect transport cleanup and cancellation handling. |
mcp_transport_failed | DEBUG | code, count, kind, run_id, tenant_id | An MCP transport operation fails. | Inspect typed failure metadata and transport configuration. |
mcp_view_relinked | INFO | code, kind, run_id, tenant_id | An MCP view changes its registered link. | Review the updated MCP configuration. |
Migration
| Code | Level | Fields | Fires when | Do |
|---|---|---|---|---|
activation_conflict | WARNING | code, index, kind, run_id, table, tenant_id | Activation detects conflicting duplicate rows. | Resolve the bounded diagnostic before retrying activation. |
migration_lock_timeout | WARNING | code, kind, revision, run_id, step, tenant_id, timeout_seconds | A migration cannot acquire its lock in time. | Retry when the table is quiet. |
Observability
| Code | Level | Fields | Fires when | Do |
|---|---|---|---|---|
log_field_dropped | WARNING | code, count, kind, run_id, tenant_id | A structured log field fails closed validation. | Fix the emitting call; values are deliberately omitted. |
trace_link_invalid | WARNING | code, kind, run_id, tenant_id | A trace link fails W3C validation. | Fix the notifier trace propagation. |
Provider
| Code | Level | Fields | Fires when | Do |
|---|---|---|---|---|
api_error | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider reports an internal failure or unavailable service. | Check provider status and retry only when the error is retryable. |
auth | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects credentials without a vendor code. | Check provider credentials and account permissions. |
authentication_error | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects credentials or access permissions. | Check the configured credentials and model permissions. |
billing_error | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider account reaches a credit or usage limit. | Restore provider credits or account quota before retrying. |
billing_hard_limit_reached | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider account reaches a credit or usage limit. | Restore provider credits or account quota before retrying. |
call_deadline | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The overall provider call deadline expires. | Reduce retry/response time or raise the call deadline. |
content_filter | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider content filter rejects the request or response. | Review the input against provider content policy. |
content_policy_violation | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider content policy rejects input or output. | Review the request against provider content policy. |
context_length | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | A request exceeds the context window without a vendor code. | Reduce conversation history or requested output tokens. |
context_length_exceeded | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The request exceeds the model context window. | Reduce conversation history, tool results or requested output tokens. |
credits | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider account reaches a credit limit without a vendor code. | Restore provider credits or quota before retrying. |
deadline_during_backoff | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider call deadline expires during retry backoff. | Inspect the preceding HTTP status and provider availability. |
empty_completion | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | A provider completes without a usable semantic response. | Inspect provider behavior and request configuration before retrying. |
image_content_policy_violation | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider content policy rejects input or output. | Review the request against provider content policy. |
image_parse_error | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects image format, content, size or location. | Check the image descriptor, supported formats and provider size limits. |
image_too_large | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects image format, content, size or location. | Check the image descriptor, supported formats and provider size limits. |
image_too_small | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects image format, content, size or location. | Check the image descriptor, supported formats and provider size limits. |
insufficient_credits | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider account reaches a credit or usage limit. | Restore provider credits or account quota before retrying. |
insufficient_quota | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider account reaches a credit or usage limit. | Restore provider credits or account quota before retrying. |
invalid_api_key | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects credentials or access permissions. | Check the configured credentials and model permissions. |
invalid_base64_image | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | A provider failure projects the declared invalid_base64_image code. | Inspect the safe category and status; apply the provider retry guidance. |
invalid_image | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects image format, content, size or location. | Check the image descriptor, supported formats and provider size limits. |
invalid_image_format | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects image format, content, size or location. | Check the image descriptor, supported formats and provider size limits. |
invalid_image_url | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects image format, content, size or location. | Check the image descriptor, supported formats and provider size limits. |
invalid_prompt | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects request parameters, resources or request limits. | Check request parameters, resource identifiers and provider limits. |
invalid_request_error | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects request parameters, resources or request limits. | Check request parameters, resource identifiers and provider limits. |
invalid_value | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects request parameters, resources or request limits. | Check request parameters, resource identifiers and provider limits. |
model_not_found | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The configured provider model is unavailable to the account. | Check the model identifier and account access. |
not_found_error | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects request parameters, resources or request limits. | Check request parameters, resource identifiers and provider limits. |
organization_usage_limit_exceeded | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider account reaches a credit or usage limit. | Restore provider credits or account quota before retrying. |
overloaded_error | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider reports an internal failure or unavailable service. | Check provider status and retry only when the error is retryable. |
permission_denied | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects credentials or access permissions. | Check the configured credentials and model permissions. |
permission_error | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects credentials or access permissions. | Check the configured credentials and model permissions. |
provider_kwargs_invalid | WARNING | code, kind, run_id, tenant_id | Provider options fail closed validation. | Use only supported provider options. |
provider_kwargs_too_large | WARNING | bytes, code, kind, limit, run_id, tenant_id | Provider options exceed the checkpoint byte bound. | Reduce provider options. |
provider_timeout | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider reports a request timeout. | Inspect provider latency and the configured request deadlines. |
provider_unknown | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | A vendor code has no closed mapping. | Inspect the provider category and status; never log vendor text. |
rate_limit | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rate-limits a request without a vendor code. | Reduce request rate and honor retry-after guidance. |
rate_limit_error | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rate-limits the request. | Reduce request rate and honor retry-after guidance. |
rate_limit_exceeded | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rate-limits the request. | Reduce request rate and honor retry-after guidance. |
refusal | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The model refuses to answer without a vendor code. | Review the request against provider policy. |
rejected | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects a request without a vendor code. | Check request parameters and provider limits. |
request_too_large | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects request parameters, resources or request limits. | Check request parameters, resource identifiers and provider limits. |
response_encoding_unsupported | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider returns an unsupported response content encoding. | Use gzip or deflate responses; inspect the provider or gateway encoding configuration. |
response_size_limit | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | A provider response exceeds its configured byte bound. | Reduce response size or raise the response bound deliberately. |
server_error | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider reports an internal failure or unavailable service. | Check provider status and retry only when the error is retryable. |
service_unavailable | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider reports an internal failure or unavailable service. | Check provider status and retry only when the error is retryable. |
stream_idle | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider stream exceeds its semantic idle deadline. | Inspect provider responsiveness and idle deadline sizing. |
stream_incomplete | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | A provider stream ends prematurely without a vendor code. | Inspect provider connectivity before retrying. |
stream_size_limit | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | A provider response exceeds its configured byte bound. | Reduce response size or raise the response bound deliberately. |
stream_tool_call_limit | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | A provider stream exceeds the distinct tool-call cap. | Reduce tool fanout or raise the stream tool-call bound. |
stream_total | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider stream exceeds its total deadline. | Reduce response length or raise the total deadline. |
timeout | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | A provider request times out without a vendor code. | Inspect the timeout reason and configured deadlines. |
timeout_error | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | Anthropic reports that the request timed out. | Inspect provider availability and the request deadline before retrying. |
token_limit_exceeded | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects request parameters, resources or request limits. | Check request parameters, resource identifiers and provider limits. |
unavailable | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider is unavailable without a vendor code. | Check provider status and retry only when retryable. |
unknown | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | An unclassified provider failure has no vendor code. | Inspect the safe category, status and provider availability. |
unsupported_image | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider rejects image format, content, size or location. | Check the image descriptor, supported formats and provider size limits. |
usage_invalid | WARNING | code, kind, run_id, tenant_id | Provider usage cannot be trusted. | Treat accounting as unknown and inspect the adapter. |
vector_store_timeout | WARNING | category, code, kind, provider, retryable, run_id, status, tenant_id, timeout_reason | The provider reports an internal failure or unavailable service. | Check provider status and retry only when the error is retryable. |
Recovery
| Code | Level | Fields | Fires when | Do |
|---|---|---|---|---|
agent_mismatch | WARNING | code, kind, run_id, tenant_id | The recovery agent differs from the saved agent. | Recreate the original agent configuration. |
already_reclaimed | WARNING | code, kind, run_id, tenant_id | Another caller already claimed the recovery generation. | Use the winning worker and avoid duplicate recovery. |
checkpoint_too_large | WARNING | bytes, code, kind, limit, run_id, tenant_id | A checkpoint exceeds its byte bound. | Reduce run state before continuing. |
fenced_finish_failed | WARNING | code, kind, run_id, tenant_id | A fenced worker cannot finish its lifecycle recorder. | Inspect store connectivity and the winning worker. |
history_too_large | WARNING | bytes, code, count, kind, limit, run_id, tenant_id | Conversation rebuild exceeds its bound. | Reduce history or raise the documented rebuild bound. |
lease_fresh | WARNING | code, kind, run_id, tenant_id | Reclaim targets a run whose lease has not expired. | Sweep stale leases before attempting reclaim. |
lease_interval | WARNING | code, kind, run_id, tenant_id | The lease interval is below its floor. | Use a lease interval of at least five seconds. |
lease_lost | WARNING | code, kind, run_id, tenant_id | Consecutive lease renewals fail. | Restore store connectivity and inspect recovery eligibility. |
lease_renew_failed | WARNING | attempt, code, kind, run_id, tenant_id | A lease renewal fails or exceeds its deadline. | Check store latency before the lease is lost. |
legacy_run | WARNING | code, kind, run_id, tenant_id | The run uses a storage format that cannot be reclaimed. | Allow the legacy run to finish or start a new run. |
legacy_run_swept | WARNING | code, kind, run_id, tenant_id | A stale format-1 run is finalised. | Inspect the stopped run; format-1 runs cannot be reclaimed. |
loop_unknown | WARNING | code, kind, run_id, tenant_id | A recovery marker has an unknown loop. | Use a supported recovery loop and inspect worker versions. |
loop_unsupported | WARNING | code, kind, run_id, tenant_id | The saved loop does not support recovery. | Use a supported loop for new recoverable runs. |
never_started | WARNING | code, kind, run_id, tenant_id | Sweep finalizes a stale run that never started. | Inspect worker startup and retry with a new run. |
not_running | WARNING | code, kind, run_id, tenant_id | Reclaim targets a run that is no longer running. | Read the current result before retrying recovery. |
pause_format_unsupported | WARNING | code, kind, run_id, tenant_id | A stored pause format is unsupported. | Upgrade every worker before continuing the run. |
pause_summary_truncated | WARNING | code, kind, run_id, tenant_id | A human-input display summary exceeded its byte bound. | Read the full model turn or raise max_pause_summary_bytes. |
reclaim_limit | WARNING | attempt, code, kind, run_id, tenant_id | A run reaches its reclaim attempt limit. | Alert at any rate and inspect worker crashes. |
recovery_call_missing | WARNING | code, kind, run_id, tenant_id | Recovery cannot locate a checkpointed tool call. | Inspect durable tool rows before restarting the run. |
recovery_scan_limit | WARNING | code, kind, limit, run_id, tenant_id | A recovery scan bound exceeds the supported maximum. | Reduce the scan bound. |
recovery_state_invalid | WARNING | code, kind, run_id, tenant_id | Recovery cannot rebuild the saved checkpoint. | Inspect checkpoint compatibility and start a new run if necessary. |
recovery_state_missing | WARNING | code, kind, run_id, tenant_id | Recovery cannot locate required checkpoint state. | Inspect checkpoint persistence before restarting the run. |
rewrite_invalid | WARNING | code, kind, run_id, tenant_id | A conversation rewrite has invalid metadata. | Inspect the conversation writer. |
run_fenced | WARNING | code, kind, run_id, tenant_id | An old worker writes after another claim wins. | Expected after reclaim; alert on a sustained high rate. |
run_finalized | WARNING | code, kind, run_id, tenant_id | Recovery finalizes a run instead of continuing it. | Read the final status and its recovery reason. |
run_not_found | WARNING | code, kind, run_id, tenant_id | Recovery cannot find a run in the tenant scope. | Verify the run identifier and tenant scope. |
run_not_reclaimable | WARNING | code, kind, reason, run_id, tenant_id | A reclaim request cannot safely continue the run. | Inspect the refusal reason before retrying. |
skill_mismatch | WARNING | code, kind, run_id, tenant_id | Recovery skill pins differ from the saved pins. | Restore the pinned skill versions before retrying. |
stale_running | WARNING | code, gap_s, kind, run_id, tenant_id | A stale run is finalised under the error policy. | Inspect worker health and lease sizing. |
stale_running_legacy | WARNING | code, kind, run_id, tenant_id | A legacy stale run cannot be reclaimed. | Inspect the stopped run and worker health. |
submission_generation | WARNING | code, kind, run_id, tenant_id | A submission belongs to a different pause generation. | Reload the pending pause and resubmit. |
submission_missing | WARNING | code, kind, run_id, tenant_id | A dispatch has no durable approval or submission. | Inspect the pause and durable submission. |
sweep_cap_reached | WARNING | cap, category, code, inspected, kind, run_id, tenant_id | A sweep reaches its inspection bound. | Run another sweep or raise the bound. |
tool_replayed | WARNING | code, kind, run_id, tenant_id | Recovery replays an interrupted idempotent tool. | Monitor repeated worker crashes and tool idempotence. |
Runtime
| Code | Level | Fields | Fires when | Do |
|---|---|---|---|---|
cancel_persist_failed | ERROR | code, kind, run_id, tenant_id | Cancellation cannot be persisted. | Retry cancellation after restoring the store. |
cancel_requested | WARNING | code, kind, run_id, tenant_id | A stale run has a durable cancellation request. | No action unless cancellation is unexpected. |
cascade_truncated | WARNING | code, count, kind, limit, run_id, tenant_id | Cancellation discovers more descendants than the subtree traversal bound. | Inspect the bounded subtree and cancel remaining subtrees or raise the bound. |
database_url_env_ignored | WARNING | code, kind, run_id, tenant_id | An Agent without persistence ignores DENDRUX_DATABASE_URL. | Pass state_store or database_url explicitly to persist Agent runs. |
input_size_limit | WARNING | bytes, code, entry, field, kind, limit, run_id, tenant_id | An input exceeds its configured bound. | Reduce the input or raise the bound deliberately. |
invalid_limit | WARNING | code, kind, run_id, tenant_id | A recovery limit is invalid. | Use a positive limit within the documented bound. |
legacy_claim_validation | WARNING | code, kind, run_id, tenant_id | A legacy store needs claim-time validation. | Implement the current store capabilities when upgrading. |
local_database_default_changed | WARNING | code, kind, run_id, tenant_id | ./dendrux.db is absent; ~/.dendrux/dendrux.db exists and is no longer the default. | Set DENDRUX_DATABASE_URL to use ~/.dendrux/dendrux.db, or start with ./dendrux.db. |
pause_data_not_jsonb | WARNING | code, kind, run_id, tenant_id | A stored pause is not a JSON object. | Inspect the stored pause format and writer versions. |
pause_id_mismatch | WARNING | code, kind, run_id, tenant_id | A submission targets another pause generation. | Reload the pending pause and resubmit. |
pending_limit | WARNING | code, count, kind, limit, run_id, tenant_id | A model produces too many pending calls. | Reduce tool fanout or raise the pending call bound. |
read_argument_invalid | WARNING | argument, code, kind, run_id, tenant_id | A read argument is invalid. | Correct the named argument. |
result_read_failed | WARNING | code, kind, run_id, tenant_id | An optional read fails while returning a known durable outcome. | The outcome is preserved; reload missing recovery or pending information from the store. |
run_cancel_recovered | WARNING | code, kind, run_id, tenant_id | A Python 3.11 persisted run recovers cancellation at a dispatch boundary. | Expected when a database wait loses cancellation; Python 3.12+ is unaffected. |
run_cancelled | WARNING | code, kind, run_id, tenant_id | Child admission is refused under a cancelled parent. | Stop creating children for the parent. |
run_failed | ERROR | code, error_type, kind, run_id, tenant_id | A run task fails. | Inspect its safe result and trace. |
skills_session_unknown | WARNING | code, count, kind, run_id, tenant_id | A skill session references unknown state. | Restore the pinned skill definitions. |
subtree_limit | WARNING | code, kind, limit, run_id, tenant_id | A descendant traversal or public subtree read reaches its node bound. | Inspect the subtree and raise its bound if appropriate. |
wire_answer_omitted | WARNING | code, kind, run_id, tenant_id | A wire answer exceeds its encoding bound. | Read the bounded result through the store. |
Store
| Code | Level | Fields | Fires when | Do |
|---|---|---|---|---|
follow_capacity | WARNING | code, kind, limit, run_id, tenant_id | The process follower cap is reached. | Reduce followers or raise the configured cap. |
follow_feed_error | WARNING | code, kind, run_id, tenant_id | A live event feed fails. | Check store connectivity and retry the feed. |
follow_shutdown_pending | WARNING | code, kind, run_id, tenant_id | A feed is still stopping after its cleanup deadline. | Inspect slow store operations. |
follow_tenant_capacity | WARNING | code, kind, limit, run_id, tenant_id | A tenant follower cap is reached. | Reduce tenant followers or raise its cap. |
idempotency_conflict_legacy | WARNING | code, kind, run_id, tenant_id | A legacy write cannot use the current unique index. | Finish the rolling upgrade and activate the schema. |
llm_failed_write | WARNING | code, kind, run_id, tenant_id | Best-effort provider-failure evidence cannot be written. | Inspect store connectivity; the safe run result remains authoritative. |
note_cap_unenforced | WARNING | code, kind, run_id, tenant_id | A custom store cannot enforce the note cap. | Implement bounded note writes in the custom store. |
note_limit | WARNING | code, kind, limit, run_id, tenant_id | A run reaches the note cap. | Consume notes or raise the configured cap. |
persistence_error | WARNING | code, kind, run_id, tenant_id | A durable operation fails. | Restore store connectivity and inspect the run result. |
shared_engine_loop_closed | WARNING | code, kind, run_id, tenant_id | A pooled engine belongs to a closed event loop. | Use an engine owned by the current loop. |
store_capability | WARNING | code, kind, run_id, tenant_id | A declared store capability has an incompatible method. | Implement the frozen store contract. |
store_outcome_unknown | WARNING | code, kind, run_id, tenant_id | A commit acknowledgement is lost and the operation outcome cannot be proven. | Inspect persisted state before attempting the operation again. |
store_statement_limit | WARNING | bytes, code, kind, limit, run_id, tenant_id | A configured limit or a statement is larger than the database accepts. | Lower the Agent's Limits, shrink the payload or raise the server's max_allowed_packet. |
write_conflict | WARNING | code, count, kind, run_id, table, tenant_id | A retried identity has conflicting content. | Inspect writer consistency; do not retry blindly. |
Tool
| Code | Level | Fields | Fires when | Do |
|---|---|---|---|---|
cancelled | WARNING | code, kind, reason, run_id, tenant_id | A sibling tool stops before completion. | Inspect the batch result before retrying. |
invalid_tool_result | WARNING | code, kind, run_id, tenant_id | A client tool result fails validation. | Submit a valid result for the pending call. |
invalid_tool_schema | WARNING | code, kind, run_id, tenant_id | A tool call targets a schema that cannot be validated. | Fix the declared tool schema. |
outcome_unknown | WARNING | code, kind, reason, run_id, tenant_id | An interrupted tool may have applied its effects. | Check the side effect before retrying. |
size_limit | WARNING | bytes, code, kind, limit, run_id, tenant_id, truncated | A tool result exceeds its byte cap. | Reduce the result or use payload offload. |
timeout_not_started | WARNING | code, kind, run_id, tenant_id, timeout_s, zombie_threads | A queued tool times out before starting. | Retry after reducing tool pool load. |
timeout_outcome_unknown | WARNING | code, kind, run_id, tenant_id, timeout_s, zombie_threads | A running tool exceeds its deadline. | Check the side effect before retrying. |
tool_pool_saturated | WARNING | code, kind, run_id, tenant_id | The synchronous tool pool cannot accept work. | Reduce concurrency or raise the pool bound. |
tool_row_may_be_missing | WARNING | code, kind, run_id, tenant_id | A finished tool is interrupted while its row is written. | Check the side effect before retrying. |
tool_zombie_finished | INFO | code, kind, run_id, tenant_id, zombies | An abandoned synchronous tool finally exits. | Inspect repeated long-running tools. |
What to alert on
cascade_truncated: inspect the subtree and cancel remaining branches or raise its bound.reclaim_limitat any rate: inspect the run and worker crashes.run_fencedabove a small sustained rate: inspect worker turnover and lease sizing.live_updates_degradedlasting more than one minute: inspect the listener connection.sweep_cap_reachedon every sweep: run more often or raise the scan bound.follow_capacityorfollow_tenant_capacity: reduce followers or raise their cap.log_field_dropped: fix the emitting call without adding raw values to diagnostics.