Rate limits are the constraint nobody plans for

By Anwar Benhamada · August 6, 2026

Your automation works beautifully. You run it against ten items. Then you point it at two thousand and everything falls over — not because the logic is wrong, but because you hit a ceiling nobody mentioned in the demo.

Rate limits are the most predictable scaling problem in this category and the least planned-for.

The four limits, and which one bites first

Most providers enforce several simultaneously:

Requests per minute (RPM) — how often you can ask
Tokens per minute (TPM) — how much you can process
Concurrent requests — how many can be in flight
Daily or monthly caps — the hard ceiling

TPM usually bites first, and it’s the one people forget, because a workflow with large inputs hits the token ceiling long before the request ceiling. Ten requests a minute sounds generous until each carries 30k tokens of context.

Wrapper tools — the ones with a nice UI over someone else’s model — add a fifth limit: their own, which is usually stricter than the underlying provider’s and much less documented.

What happens when you hit one

Three behaviours, in descending order of pleasantness:

429 with Retry-After — the good case. Honour the header and back off.
429 with nothing — you’re guessing at backoff.
Silent degradation — the request succeeds but something is quietly worse: truncated context, a fallback model, a queued response that arrives late.

That third one is the dangerous one. Your pipeline reports success and the output quality drops for reasons nothing logs. If you’re running anything unattended, you need a quality check that catches this, because the error handling never will.

Design around them before you need to

Batch, don’t stream. If you’re processing a list, queue it and drain at a controlled rate rather than firing everything and handling failures. A queue with a fixed drain rate is simple and never surprises you.

Exponential backoff with jitter. Retry at 1s, 2s, 4s, 8s — plus a random offset. Without jitter, all your retries collide again in lockstep, which is how one rate-limit event becomes a cascade.

Cap total retries. Infinite retry against a daily cap burns your quota on requests that cannot succeed until tomorrow.

Make it idempotent. You will re-run a partially failed batch. If re-running duplicates work or double-charges you, that’s worse than the original failure.

Track your own usage. Don’t discover the monthly cap by hitting it. Log tokens consumed and compare against the ceiling as you go.

Estimate before you build

Rough arithmetic, done in ten minutes, prevents most of this:

items × tokens per item = total tokens
total tokens ÷ TPM limit = minimum minutes

If that number is “eleven hours”, you’ve learned something important before writing any code — either you need a higher tier, a smaller model for the bulk of the work, or a different architecture entirely.

Do this for your realistic worst case, not your test case. The gap between them is where the surprise lives.

Questions to ask before committing

Rate limits are frequently absent from pricing pages and buried in docs, so ask directly:

What are the RPM, TPM and concurrency limits on this tier?
What happens when I exceed them — error, queue, or degraded output?
Do limits reset per minute, per day, or per billing month?
Can they be raised, and what does that cost?
Are limits per account, per key, or per workspace?

That last one matters more than it sounds. If limits are per account and you’re running three workflows, they’re competing with each other — and the symptom is one workflow mysteriously slowing down when an unrelated one is busy.

The pattern that scales

Two tiers of model. Use a cheap fast model for the bulk of the work and escalate only the cases that need it. Most workloads have a long tail of easy items and a small head of hard ones, and paying premium rates for the easy tail is where both cost and rate-limit pressure come from.

That single change usually buys more headroom than any tier upgrade, and it costs less.