Rate limits and retries

A failed attempt is either retried or finished. The engine decides from the error the executor reported. A check failure is finished: the same code over the same inputs fails the same check, so it is not tried again.

Ordinary failures

A retryable failure is a rate limit, an upstream outage (500, 502, 503, 504), or a dropped connection. A 400, a missing page, and a TerminalError thrown from run are terminal. The attempt status becomes failed, and error_class is retryable or terminal.

The ordinary budget counts retryable failures that are not rate limits. A source is tried 3 times. A raw asset, a model, and a destination are tried once. When the budget is spent, the error message gains a suffix that names that number:

(gave up after 3 attempts)

A model, a destination, or a raw asset uses (gave up after 1 attempts). That is the string the engine writes.

Rate limits

A connector’s HTTP client waits out a 429 inside the attempt, for up to 30 seconds. Retry-After sets the wait. With no header, the client waits 1 second, then 4 seconds, then 10 seconds, and then hands the error to the engine. A Retry-After that would push the wait past 30 seconds is not slept in the client. The engine receives it instead.

The engine treats that error as a rate limit when it carries retry_after_ms. Rate limits use their own budget of 8 retries, and they do not spend the ordinary attempt budget. Each wait is the server’s Retry-After, capped at 15 minutes. With no delay on the error, the next attempt starts immediately. Past 8 retries the message is:

(gave up after 8 rate-limit retries)

Read the result

puppi logs <run-id>

Each attempt prints its status, error_class when one was set, the error message, and the ctx.log lines. A retry is a new attempt on the same asset, with a higher number. See Read a run.