Start with what a scraper genuinely solves, or the rest of this reads like a pitch.
For one-off, small-batch, simple-field needs, writing your own script is usually the fastest road. Titles and prices for a hundred ASINs, needed once and never again: half a day to write, ten minutes to run, faster than evaluating an API, signing up and integrating. Building it yourself is the right call there.
The costs go wrong in the other case: the script works, and someone says "let's run it daily."
From that moment it stops being a one-off script and becomes a data pipeline that needs maintaining — and four costs start their clock.
Cost 1: anti-bot upkeep is recurring, not one-time
That the first version ran successfully carries less information than most people assume.
What it proves is that this page structure, at this frequency, from this location, today, could be read. Of those four conditions, only the first is under your control — and it changes too.
The real spend is on stability. The script was fine yesterday; today it returns empty arrays, throws no error, and quietly delivers a third less data. There is no stack trace for this class of problem, only a person comparing outputs. And it happens on a random schedule, usually not during working hours on a weekday.
The shape of this cost: development hours are one-time, investigation hours are recurring, and you do not control the interval. The second item is what budgets miss, because at proposal time it does not exist yet.
This post does not cover any method of evading detection. If a plan's feasibility depends on getting around another party's technical measures, what needs adjusting is the plan's scope, not the technique.
Cost 2: you have to define the field semantics yourself
The most underestimated one, because it does not look like a cost.
The page has a BSR. The page does not have "monthly sales."
What you can scrape is the model's input, not its output. To get units from a rank you need a rank-to-sales conversion curve, and fitting that curve requires samples labeled with real unit sales — which exist only inside sellers' own accounts. This is not an engineering difficulty. You simply do not have the samples.
The same problem repeats across many fields:
| You think you are scraping one field | What you must actually decide first |
|---|---|
| Price | List price, deal price, or after a coupon applies? Sellers surface these differently |
| BSR | Top-level rank or subcategory rank? The category path has several levels |
| Sales | Not on the page. Either skip it or build a model |
| Rating count | Merged across the parent, or the single child |
| Category | The breadcrumb display name or the node ID? Display names change; IDs are steadier |
Every row is a definition a human has to settle. An API ships a contract with defined semantics; a scraper ships whatever the page happened to show that day. Two people scraping the same page get different numbers with neither of them having written a bug — that is normal in a self-built pipeline.
Worse, these definitions usually go unwritten. They live in the head of whoever wrote the selectors. Three months on, nobody can say what that column actually means.
For the line between factual fields and derived ones, What Amazon sales data can and cannot tell you goes deeper.
Cost 3: compliance and account exposure
This one is not a technical problem, which is why it is often missing from an engineering assessment.
What has to be weighed: the platform's terms of service, the permitted scope of use, and how any of it relates to your own seller account. The party carrying the risk is the business, not the person writing the script — yet the discussion at kickoff is often engineering-only.
A practical move: before starting, get the business side to answer plainly who carries it if this collection path creates an account-level problem. The answer does not need to be complicated, but somebody needs to give one. If nobody will, the risk has not actually been accepted — it has only been deferred.
Cost 4: when the maintainer leaves, the data stops
The first three can be solved with money and time. This one cannot.
Very little of the real knowledge in a scraper lives in the code: why a selector is written that way, which field differs on which marketplace, which failures to retry and which to skip, how the last redesign was patched. None of it gets documented, because at the time of writing it was all obvious.
Once the person who wrote it leaves, the next page redesign is the end of the line. Not because it cannot be fixed, but because the fix costs enough that nobody approves it: whoever inherits it has to reverse-engineer the whole thing, and the only specification is the code.
This risk scales inversely with team size. On a team of three, exactly one person usually understands the collection layer.
The shape of the four costs
| One-time | Recurring | Unpredictable | |
|---|---|---|---|
| Anti-bot upkeep | First version | Investigation and repair | When redesigns land |
| Field semantics | Defining fields | Checking for drift | Page structure changes |
| Compliance | One assessment | — | When exposure materializes |
| Maintainer | — | Handover and docs | When they leave |
The third column is the point of that table. Budgetable costs are not the dangerous ones; unpredictable costs are. A third-party API's recurring cost sits in the second column — it follows call volume, it can be calculated, it scales linearly with the business. A self-built pipeline's main cost sits in the third.
That does not make building it yourself more expensive. It makes it expensive differently: not a larger bill, but a bill whose arrival date you do not know.
When building it yourself still holds up
One test carries most of the weight: does anyone but you need this field?
- A common need → an endpoint almost certainly covers it, and scraping reinvents the wheel
- Genuinely specific to your business → building holds up, because there is no second option
Two more conditions make it safer: the need is one-off or low-frequency, and the fields are simple enough not to require defining semantics. All three met, write the script. Any one of them missing, check the catalog first.
API catalog46 endpoints grouped by products, keywords, markets, traffic and brands — check whether the field you need is already coveredIf you have not settled on a road yet, How to choose an Amazon data API compares what official SP-API, third-party endpoints and self-collection each give you. This post expands the self-collection column of that comparison.
Questions
Is small-scale scraping worth it? For one-off needs with simple fields, yes. Once it becomes a daily scheduled job, these four costs start their clock.
Is scraping cheaper than calling an API? Cheaper up front, because it is only development hours. Once investigation time, semantic drift and maintainer risk are counted, usually not. The key difference is not the total — it is the predictability.
Can I scrape sales figures? That field is not on the page. What you can scrape is BSR, and going from BSR to units requires a conversion curve, which requires samples labeled with real sales. Those samples are not in public data.
Can I use both? Yes, and it is common. Route the common fields through an API and collect only the long-tail fields nobody else needs. That keeps the self-built part small enough that all four costs stay manageable.