Web scraping is a powerful way to access otherwise unavailable data, but it’s becoming more complex as websites deploy defenses like Captchas and anti-bot systems. At SWR Data Lab, we’ve tackled this across investigations ranging from Google price comparisons to healthcare platforms and social media scraping, each requiring a different approach. In this session, we share a practical decision framework for choosing the right scraping strategy based on robustness, cost, and maintainability.
In this session, we will present a decision framework for selecting the right scraping strategy based on our learnings. Rather than promoting a single tool, we want to focus on choosing the right approach for your use case, considering robustness, cost, and maintainability in a newsroom context. Using real examples, we walk through our workflow: from analyzing sites with dev tools to selecting between HTTP scraping, browser automation, and advanced tools—along with best practices and when paid services are worth it.
To follow along, you should have some experience in scraping and, ideally, Python. The participants will be able to extend their toolkit, make smarter choices in their scraping workflow, and handle real-world obstacles efficiently. No special tools are required to follow along
Stephanie Jauss is a data reporter at the German public broadcaster SWR. She studied Computer Science and Media in Stuttgart as well as Investigative Journalism in Gothenburg.
