the.bay.news

Reddit blocked my scraper with a 403, so I built a collector for 6 open APIs instead (Python)

programming.dev
Reddit blocked my scraper with a 403, so I built a collector for 6 open APIs instead (Python)
I was building a small tool to collect posts and comments about a topic and save them as JSON. The plan was to use Reddit, but scripts get a 403 from the .json endpoint that works fine in the browser. I tried the legitimate things (own User-Agent, full headers) and even a client that mimics Chrome. All blocked, and the official API needs prior approval. I decided not to work around it. So I switched to sources with open APIs: Hacker News, Lemmy, Stack Exchange, Dev.to [http://Dev.to], Mastodon and Discourse. Each one is a small Python module that returns posts in the same format, so the output is one JSON file organized as source > language > posts > comments. There is also a FastAPI service and a language filter (langdetect). A few things I learned: - “0 results” isn’t always an error: Dev.to [http://Dev.to] and Mastodon only search by tag. - Stack Exchange allows 300 requests per day without a key, so I fetch answers in batches. - Text-search APIs require all words: on Hacker News, “most popular frontend language” matches 11 stories, “frontend language” matches 793. Do you know other open APIs that are good for tech discussions, especially non-English ones? I’d like to test them.

I was building a small tool to collect posts and comments about a topic and save them as JSON. The plan was to use Reddit, but scripts get a 403 from the .json endpoint that…

0 comments

Sign in to join the discussion — your thebay.events account works here.

No comments yet.