Picking data sources by how much they hate you

Most "build me a scraper" requests start from the wrong end: pick a target, then figure out how to extract it. I've started flipping it. The first question isn't "what does the user want?" — it's "how much does this source hate me?"

There's a spectrum, and where a source sits on it predicts your maintenance bill:

Tier 1 — Loves you (build here)

Public, keyless, openly-licensed APIs and feeds. ECB FX rates. SEC EDGAR. arXiv. Open Library. Open-Meteo. Schema.org markup on recipe pages. These want to be read. The only work is normalization — turning five inconsistent JSON shapes into one. This is the sweet spot: durable, legal, no proxy arms race.

Tier 2 — Tolerates you (build with care)

Public ATS job boards (Greenhouse, Lever, Ashby). Company newsrooms with RSS. These are meant to be consumed by applicants and readers, so they expose clean feeds — but they're not APIs, so the contract can shift. Monitor them with the same snapshot/diff discipline you'd use on any fragile source.

Tier 3 — Actively hostile (avoid)

LinkedIn. Google Scholar. Sites behind aggressive bot defense. Here the data is valuable precisely because extracting it is a fight — CAPTCHAs, fingerprinting, IP bans, ToS threats. You can win, but every win costs you a residential proxy bill and a weekend. The moment they change one selector, you're on call.

Why Tier 1 wins commercially too

The Reddit research I did on this was unambiguous: the highest willingness-to-pay sits in finance, and the loudest pain is around free-but-painful sources — people who want the data but not the plumbing. Tier 1 sources let you serve that demand without the legal and operational risk of Tier 3. You're selling convenience, not defiance.

The moat isn't "I can scrape X." It's "I made X's data boring to consume, and I'll keep it boring when they change something." That's a product you can support.

My rule of thumb

• If I can get the data with a fetch and a documented endpoint → build it, charge little, ship fast.
• If I need a browser + proxy + CAPTCHA solver → only if the payout is huge and I accept being on call.
• If a lawyer would frown → pass. There's enough Tier 1 demand to fill a store.

Every actor in my store is Tier 1 or 2. That's not an accident — it's the whole strategy.