Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Retries and circuit breakers

The one connector type that retries, the loop it runs, and the breaker that sheds load from a failing dependency.

Retries

Only http connectors retry. A retry block on any other type is refused with 400 on create and update — it would otherwise be silently ignored. No other connector type re-drives a failed call: a call that timed out may already have been applied.

The retry object accepts exactly two keys; an unknown key is refused:

FieldTypeRequiredDefaultDescription
max_retriesintegerno3Retry attempts after the first request. Values above 16 are refused
retry_delay_msintegerno1000Delay before the first retry, in milliseconds

The retry loop behaves as follows:

  • Backoff is exponential. The delay doubles on each attempt and is capped at 60 seconds.
  • The whole loop shares one deadline: timeout_ms × (max_retries + 1), measured from the first attempt, backoff included. timeout_ms is the http_call task’s timeout.
  • Idempotent methods only: GET, PUT, and DELETE retry. POST and PATCH retry only when the connector sets retry_non_idempotent: true (default false); their budget is otherwise exactly one timeout_ms.
  • Retryable errors: HTTP status ≥ 500, 429, 408, status 0 (no response), timeouts, and I/O errors. Everything else fails immediately.

Warning

A timed-out POST may already have been applied, so re-sending it can duplicate the side effect. Enable retry_non_idempotent only when the endpoint honors an idempotency key the workflow sets in headers.

Circuit breakers

Circuit breakers shed load from a failing dependency. They are global and off by default: the settings live under [engine.circuit_breaker] in the Configuration Reference, not in connector config.

When enabled, breakers behave as follows:

  • One breaker per channel:connector pair, per node. State is in-process and never shared across a cluster.
  • Every connector-backed task function passes through its breaker, not only http_call.
  • Only retryable failures count. A call the backend rejected — a syntax error, a constraint violation — says nothing about the dependency’s health and never trips the breaker.
  • The breaker opens after failure_threshold consecutive retryable failures. While open, calls fail immediately with 503 CIRCUIT_OPEN (error codes).
  • Half-open admits a single probe. After recovery_timeout_secs, one request is let through. Success closes the breaker; failure reopens it.
  • The breaker map is bounded at max_breakers entries with LRU eviction. Eviction prefers a closed victim: evicting an open breaker would re-admit full load to a dependency still known to be broken. Only when every breaker is open does plain LRU apply, with a warning.

List and reset breakers through the Admin API.

  • Connector types: every type, and the shared blocks all of them carry.
  • http: the retry and retry_non_idempotent fields.
  • Engine settings: where the breaker settings live.
  • http_call: the task timeout the whole loop is measured against.

Last verified 14 September 2026