Web Scraping — practical data extraction and parsing

Course overview

Description generated based on course syllabus and open data.

The program covers legal aspects of web scraping, tool selection, extraction of structured and semi-structured data (RegEx, CSS, XPath), working with proxies, headers, queues, and headless browsers for stable parsing.

Web Scraping: scope and learning format

Lectures and hands-on practice on crawling, collecting, cleaning, and storing data. Covers common constraints, CAPTCHA, rate limits, and strategies like throttle, retries, and fingerprint rotation.

Web Scraping: who it suits and who it does not

Suitable for

• Developers who automate information gathering, working with HTML/DOM and HTTP.
• Analysts and DS specialists building datasets and monitoring changes.
• Professionals with Python basics who need a data extraction tool.

Not suitable for

• Those expecting turnkey “one-click” services without configuration.
• Those unwilling to consider legal and ethical constraints of data collection.

Web Scraping: problem → expected outcome

• Fragmented HTML/JS content → structured tables/JSON for further analysis.
• Anti-bot and blockers → controlled strategies for proxies, headers, delays, fingerprint rotation.
• Large volumes → task queues, parallel crawling, incremental updates.
• Unstable markup → robust selectors, validation, and fallback templates.

Web Scraping vs alternatives: approach comparison

Manual data collection

Accurate but slow and unscalable; web scraping automates repetitive steps.

Public APIs

Best when available; web scraping is relevant when APIs are missing or limited.

Off‑the‑shelf aggregators/tools

Quick start with less control; custom scrapers are more flexible for specific cases.

Crawling without parsing

Yields links, not data; parsing structures content for analytics.

Learning outcomes: Web Scraping competencies

• Designing scrapers for target sources: from DOM mapping to crawl plans.
• Extraction via RegEx, CSS, XPath; handling cookies, sessions, headers.
• Anti-bot evasion: IP/UA rotation, proxy pools, timing, headless browsers.
• Persisting data to CSV/JSON/DB, logging, retries, and parser testing.
• Assessing legal risks, rate limits, robots.txt, and fair-use practices.