The technical part of integrating a data API is short: create a key, send the request the docs show you, check the status code. Getting started and Authentication already cover that.
The expensive part is the five decisions that come before it. What they have in common: each one is cheap to make now and painful to change once you are running.
Decision 1: how to estimate call volume
This number picks your call pack, and it sets how tight the other four decisions need to be.
The most common error is counting a report as one call. A weekly competitor report is usually 20 ASINs × 3 field groups × once a week = 60 calls — and if you also pull twelve months of history to compare against, the first run can be several hundred.
Estimate it as entities × field groups × frequency, then budget the initial backfill separately as a one-off. First run and steady state are often an order of magnitude apart, and estimating only the steady state blows the budget on day one.
Decision 2: which fields to store and which to fetch live
One question decides it: how often does this field change?
| Rate of change | Typical fields | What to do |
|---|---|---|
| Effectively static | ASIN, brand, category, launch date | Store it, fetch only when missing |
| Daily | BSR, rating count, sales estimates | Store it plus a scheduled refresh |
| Continuous | Price, coupons, availability | Fetch live, never cache overnight |
Both extremes waste money. Store nothing and you pay repeatedly for data that never changed. Store everything and you make decisions on a stale price.
Designing this as two tables — slow fields in one, fast fields in a timestamped log — is far easier than splitting them later. A number with no timestamp cannot be judged fit for use three months on.
Decision 3: retries have to be classified, not uniform
This is the one most often written wrong. Failures are not one thing, so retry logic should not be one branch.
| Status | Typical error code | Worth retrying? | What to do |
|---|---|---|---|
400 | VALIDATION_ERROR | No | Retrying an unchanged request just fails again |
401 | INVALID_API_KEY, API_KEY_EXPIRED | No | Replace the key; do not back off and retry |
402 | INSUFFICIENT_CREDITS | No | Out of balance — no number of retries succeeds |
403 | ENDPOINT_NOT_INCLUDED | No | The plan does not include that endpoint |
413 | — | No | Body over 64 KB; split the batch |
429 | RATE_LIMIT_EXCEEDED, CONCURRENCY_LIMIT_EXCEEDED | Yes | Back off per Retry-After |
503, 504 | SERVICE_BUSY, SERVICE_TIMEOUT | Yes | Exponential backoff with jitter |
402 and 429 look alike and are not the same problem. 429 means "too fast right now, wait" — backing off clears it. 402 means the balance is gone; what it needs is an alert and a top-up, not a retry loop. Put both in the same catch branch and the day your credits run out, your jobs spin quietly and produce nothing.
The two error codes under 429 differ too. RATE_LIMIT_EXCEEDED means too many requests per minute; CONCURRENCY_LIMIT_EXCEEDED means too many in flight at once. The first is fixed by slowing down, the second by reducing parallelism — and slowing down does not reduce parallelism.
Three response headers exist for exactly this:
| Header | Meaning |
|---|---|
X-RateLimit-Limit | Requests allowed in the current minute window |
X-RateLimit-Remaining | Requests left in the current window |
X-RateLimit-Reset | Window reset time, Unix seconds |
Do not wait for the 429. Slow down when X-RateLimit-Remaining drops below a threshold and you get more throughput than backing off after being limited.
As for whether a failed request consumes credits — do not guess. Signed-in accounts can see request metadata on the usage page, where redacted bodies and responses are kept for seven days. Trigger a failure on purpose before launch and check it against the call detail, which beats any assumption.
Errors and rate limitsThe full status code table, error code list, rate limit headers and how long call details are retainedDecision 4: concurrency is an account budget, not a per-job setting
This is the least intuitive rule, and the one that bites when you scale:
RPM, concurrency and credits aggregate per account. Creating more API keys does not grant extra quota.
The instinct is to hand every job its own key so they run independently. They do not. Three scheduled jobs opening ten connections each put thirty in flight against one account ceiling, and whoever starts first pushes the others into CONCURRENCY_LIMIT_EXCEEDED.
So concurrency has to be designed as one shared limiter, not per job:
- Route every call through one client wrapper, with the limiter inside the wrapper rather than in business code
- Give jobs priorities — a live lookup outranks a nightly backfill
- Rate-limit backfills explicitly; they are not urgent, and they are the likeliest to eat the whole quota
One more ceiling worth knowing up front: a request body over 64 KB returns 413. When a batch endpoint takes a list of ASINs, that limit decides the maximum batch size, and it belongs in your batching logic rather than in a production incident.
Decision 5: how the team manages keys
Since quota is shared at the account level anyway, extra keys are not about quota. Their real value is two other things: attribution and independent revocation.
Issue keys by purpose, not by person:
- Split them as
prod-api,cron-backfill,dev-localand the call detail tells you which path misbehaved - When one path goes wrong, disable that key and the rest keep running
- When someone leaves, the question is which keys they touched, not whether to rotate the whole account
Three hard lines: keys live in server-side environment variables only, never in the repository, never in front-end code. Anything that needs data in a browser should go through your own backend.
What the five have in common
None of them is a hard technical problem. Each is a choice whose cost is asymmetric to reverse. Underestimate volume and you find out in month one. Choose the wrong caching split and you find out when numbers stop reconciling. Write one retry branch and you find out the day credits run out. Set concurrency per job and you find out when the third job ships. Issue keys per person and you find out at the first departure.
Half an hour on these five before the first request goes in is cheaper than coming back for them later.
When you move on to picking endpoints, The complete guide to Amazon data APIs breaks the 46 endpoints down by the job each one does. If you are still deciding whether a third-party API is the right road at all, start with How to choose an Amazon data API.
Questions
Do more API keys raise my limits? No. RPM, concurrency and credits all aggregate per account. Multiple keys buy attribution and independent revocation, not quota.
How long should I wait after a 429?
Follow Retry-After when it is present. Without it, use exponential backoff with random jitter so several jobs do not retry in lockstep.
How do I confirm whether failed requests consume credits? Check the call detail on the usage page. Redacted request metadata is kept for seven days, so trigger one failure before launch and read it back.
How many items fit in one batch request? There is no single number — it is bounded by the 64 KB body limit. Measure it once against your actual parameter sizes and leave headroom in the batching logic.