Web Scraping & Data Collection
Collect the data your startup runs on — API-first, compliant, deduplicated and monitored. Build a polite, resilient scraping pipeline (fetch → parse → normalise → store), enrich records to your ICP, and schedule it to run hands-free with alerts. Tutor: Ada.
Digital course. By purchasing you request that delivery begins immediately and acknowledge that your statutory right of withdrawal expires once delivery has begun (§ 356 (4) BGB).
- Lifetime access to the full course
- Build-along Workbook — Claude Code right in your browser
- Progress tracking, topic by topic
- Certificate of completion when you finish
- Taught on real Kaern software & founder playbooks
After you get access, your course lives in My Courses — log in any time with your email.
Already bought it? Log in to read it.
The data you need — competitor prices, leads, market signals — is sitting on public websites. In this course you build a polite, legal, self-healing scraper fleet that collects it while you sleep.
Introduction
You won't watch Ada talk — you'll build. By the final module you will run scheduled scrapers that fetch public pages politely, extract structured data with an LLM, store clean deduplicated datasets, and alert you when a site changes. You'll also know exactly where the legal and ethical lines are, because a scraper that gets you blocked or sued is worse than no scraper.
Module 1 — Targets, Rules and the Legal Line
Before writing code you choose targets worth scraping and learn the rules that keep you welcome.
Learning Objectives
- Pick three data targets tied to a real business decision
- Read robots.txt and terms of service and decide what's in bounds
- Apply the politeness rules: identify yourself, rate-limit, cache
- Know the GDPR line: public business data vs personal data
Lesson 1.1 — Scrape for a decision, not for sport
Bad scraping projects start with "let's collect everything". Good ones start with a decision: "I'll reprice weekly against these five competitors" or "I need every bike shop in Bavaria with a phone number". Write the decision first, then the minimum data that powers it.
Pick three targets for the course: one price/product page, one directory or listing site, one news or content source. These three shapes cover most business scraping.
Lesson 1.2 — robots.txt, ToS and being a good guest
Fetch robots.txt on each target — it's the site telling you where bots are welcome. Read the terms of service with your AI assistant summarizing the scraping-relevant clauses. Public facts (prices, addresses) are generally fair game in the EU; hammering servers or scraping personal profiles is not.
Politeness rules you'll code in Module 2: identify with an honest User-Agent, one request every few seconds, cache everything, never scrape behind a login you weren't given.
Lesson 1.3 — The GDPR line
Business data (company names, public prices, office addresses) is safe territory. Personal data — even publicly posted — triggers GDPR duties the moment you store it. Names of private individuals, personal emails and phone numbers need a documented legitimate interest and a deletion policy.
Rule of thumb for this course: firmographic yes, personal only when it's clearly business-role data (info@ addresses, office phones), and document why you keep it.
🔒 That’s the end of your free lesson
Unlock the full Web Scraping & Data Collection — every remaining module, your build-along Workbook, progress tracking, and a certificate when you finish.
Already bought it? Log in to read the rest.