You, at 2am · Whatever you have time to maintain
Scraping card data yourself vs using an API
An honest look at what it costs to scrape and normalise trading card game data in-house: the initial build, the ongoing maintenance, and when doing it yourself is genuinely the right call.
What Building it yourself does well
- Zero vendor cost and no rate limits beyond what the sources impose.
- Total control over schema, storage and refresh cadence.
- No dependency on a company that could change its pricing or shut down.
- For one game with modest needs, a weekend of work can genuinely be enough.
Where TCGGraph differs
- Each game is a separate integration with its own quirks, and publishers change their card lists without notice.
- Set releases and spoiler seasons are the moments your scraper breaks and also the moments your users care most.
- Price history cannot be backfilled. If you start collecting today, you have no history for a year.
- Reconciling the same card across languages and printings is the genuinely hard part, and it never stops needing attention.
Side by side
Feature comparison
| Feature | Building it yourself | TCGGraph |
|---|---|---|
| Up-front cost | 2–6 weeks of engineering | An afternoon |
| Ongoing maintenance | Every set, every game | None |
| Price history | Starts empty | Back to 2019 |
| European prices (Cardmarket) | A second integration | Same card object |
| Multi-language linking | Hard | |
| Breaks during spoiler season | Usually | |
| You own the data | Bulk exports from Growth up |
When to use Building it yourself instead
If you need one game, shallow data and have no deadline, build it. Scryfall and pokemontcg.io already solve most of that for free. Buy instead when you need several games, when price history matters, or when the maintenance would land on someone whose time is worth more than the subscription.
Signals
When it is worth switching
None of these are reasons on their own. Two or three together usually are.
- You are spending more time on ingestion than on the product the data is for.
- A source changed its HTML or its rate limits and your pipeline broke silently, and you found out from a user.
- You have added a second game and discovered the first game's normalisation assumptions do not transfer.
- Somebody has to be on call for a data pipeline that is not your business.
Migration
Moving off Building it yourself, step by step
The honest version, including the parts that are not one-to-one.
Step 1
Cost the thing you already built
Ingestion for one game is a weekend. Eight games, with per-game price sources, image hosting, deduplication across reprints, legality tracking and a monitor that tells you when a source silently changes shape, is not a weekend. Most teams underestimate this by an order of magnitude, and the estimate that matters is maintenance, not the initial build.
Step 2
Keep your identifiers, change your source
If you already store upstream identifiers — Scryfall oracle ids, Konami passcodes, set codes — you can keep them. We carry the common external ids on every card, so a migration is usually a join rather than a re-import.
Step 3
Move the schedule to webhooks
The cron job that re-scrapes everything nightly becomes a webhook subscription for the events you care about: set published, legality changed, price threshold crossed. Less work, fresher data, and no nightly window where you are hammering somebody else's site.
Step 4
Keep the export path open
Bulk exports in JSON, NDJSON and Parquet are included from Growth up, and they are complete rather than crippled. If you ever want to go back to running your own store, the door is not locked behind you — which is the main thing anyone who has been burned by a data vendor wants to hear.
Field mapping
What each field becomes
The part that takes the longest is discovering that Building it yourself spells the same concept differently. Here it is, written down.
| Building it yourself | TCGGraph | Notes |
|---|---|---|
| Per-source ad hoc schema | One schema, 8 games | Core fields identical; game specifics namespaced. |
| Nightly full re-scrape | Webhooks + conditional GETs | 304 responses cost zero credits. |
| Your image hosting bill | Unmetered image CDN | Never counted against credits. |
| Hand-rolled dedupe | printings[] per card | Reprints modelled rather than merged. |
| One price source, if any | 5 sources, daily history | Kept separate, not blended. |
| You are on call | 99.95% SLA on Growth | Service credits if we miss it. |
FAQ
Common questions
Anything else, email hello@tcggraph.com. We will tell you if the answer is that you should stay where you are.
More comparisons